2024-05-27T09:08:18.568 INFO:root:teuthology version: 0.0.1.dev291+ge7ae758 2024-05-27T09:08:18.568 DEBUG:teuthology.run:Teuthology command: teuthology --name vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/fs/misc}} --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679 -- /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/orig.config.yaml 2024-05-27T09:08:18.630 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T09:08:18.724 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679 branch: wip-vshankar-testing-20240521.105740-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/fs/misc}} email: vshankar@redhat.com first_in_suite: false job_id: '7727679' kernel: client: branch: testing flavor: default kdb: 1 sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240521.105740-debug ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request sha1: a6507388dd5057528934822c0163b0c347ef1d5d subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: a6507388dd5057528934822c0163b0c347ef1d5d kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240521.105740-debug sha1: a6507388dd5057528934822c0163b0c347ef1d5d owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2274 sha1: a6507388dd5057528934822c0163b0c347ef1d5d sleep_before_teardown: 0 subset: 442/1024 suite: fs suite_branch: wip-vshankar-testing-20240521.105740-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: a6507388dd5057528934822c0163b0c347ef1d5d syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi044.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPHeN/D5EzaFUJSh/xMNad76AA8yoA/PKCoFbkZ7fsbeclOxYg6RPscTopqFFtvRyyAR/Xj1z32iRYFhFI0MBmI= smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGSz+t1h6L2a6y0htBxkxt1oEMChGI6sky+RerZTIPZble9KtmaxkVzbsdXWSNJE0nJlyF2+9QSw0Qzv8AG1wp8= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKPn6M/+p4ZmCw0YGNhAQi/Ofa5XLmxznHvU3T3sldp/tz5GeEPqjt9bQgSKBZ9wqi7DyMuaJ2lo+zwlmuxw0bU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - fs/misc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e7ae7580c417d6ab27fda98fecadca5d9f85d63e timestamp: 2024-05-27_06:46:53 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-27T09:08:18.725 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa; will attempt to use it 2024-05-27T09:08:18.725 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa/tasks 2024-05-27T09:08:18.726 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-27T09:08:18.727 INFO:teuthology.task.internal:Checking packages... 2024-05-27T09:08:18.748 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a6507388dd5057528934822c0163b0c347ef1d5d' 2024-05-27T09:08:18.748 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-27T09:08:18.748 INFO:teuthology.packaging:ref: None 2024-05-27T09:08:18.748 INFO:teuthology.packaging:tag: None 2024-05-27T09:08:18.748 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240521.105740-debug 2024-05-27T09:08:18.748 INFO:teuthology.packaging:sha1: a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:08:18.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240521.105740-debug 2024-05-27T09:08:18.971 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3868.ga6507388 2024-05-27T09:08:18.973 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-27T09:08:18.981 INFO:teuthology.task.internal:no buildpackages task found 2024-05-27T09:08:18.981 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-27T09:08:19.010 INFO:teuthology.task.internal:Saving configuration 2024-05-27T09:08:19.033 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-27T09:08:19.044 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-27T09:08:19.068 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 09:02:39.472062', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPHeN/D5EzaFUJSh/xMNad76AA8yoA/PKCoFbkZ7fsbeclOxYg6RPscTopqFFtvRyyAR/Xj1z32iRYFhFI0MBmI='} 2024-05-27T09:08:19.089 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 09:02:39.473136', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGSz+t1h6L2a6y0htBxkxt1oEMChGI6sky+RerZTIPZble9KtmaxkVzbsdXWSNJE0nJlyF2+9QSw0Qzv8AG1wp8='} 2024-05-27T09:08:19.112 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 09:02:39.470073', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKPn6M/+p4ZmCw0YGNhAQi/Ofa5XLmxznHvU3T3sldp/tz5GeEPqjt9bQgSKBZ9wqi7DyMuaJ2lo+zwlmuxw0bU='} 2024-05-27T09:08:19.112 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-27T09:08:19.122 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-27T09:08:19.122 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-27T09:08:19.122 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-27T09:08:19.123 INFO:teuthology.run_tasks:Running task console_log... 2024-05-27T09:08:19.214 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa26c3c5090>, signals=[15]) 2024-05-27T09:08:19.214 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-27T09:08:19.230 INFO:teuthology.task.internal:Opening connections... 2024-05-27T09:08:19.230 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi044.front.sepia.ceph.com 2024-05-27T09:08:19.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T09:08:19.319 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-05-27T09:08:19.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T09:08:19.404 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-05-27T09:08:19.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T09:08:19.482 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-27T09:08:19.488 DEBUG:teuthology.orchestra.run.smithi044:> uname -m 2024-05-27T09:08:19.507 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2024-05-27T09:08:19.507 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release 2024-05-27T09:08:19.566 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream" 2024-05-27T09:08:19.566 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="9" 2024-05-27T09:08:19.567 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos" 2024-05-27T09:08:19.567 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora" 2024-05-27T09:08:19.567 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="9" 2024-05-27T09:08:19.567 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el9" 2024-05-27T09:08:19.567 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T09:08:19.567 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2024-05-27T09:08:19.568 INFO:teuthology.orchestra.run.smithi044.stdout:LOGO="fedora-logo-icon" 2024-05-27T09:08:19.568 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T09:08:19.568 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/" 2024-05-27T09:08:19.568 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T09:08:19.568 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T09:08:19.568 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T09:08:19.569 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2024-05-27T09:08:19.608 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-05-27T09:08:19.626 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-05-27T09:08:19.627 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-05-27T09:08:19.685 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-05-27T09:08:19.685 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2024-05-27T09:08:19.685 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-05-27T09:08:19.685 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-05-27T09:08:19.685 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2024-05-27T09:08:19.685 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2024-05-27T09:08:19.685 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T09:08:19.685 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-05-27T09:08:19.685 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2024-05-27T09:08:19.686 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T09:08:19.686 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-05-27T09:08:19.686 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T09:08:19.686 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T09:08:19.686 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T09:08:19.686 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-05-27T09:08:19.707 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-05-27T09:08:19.726 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-05-27T09:08:19.726 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-05-27T09:08:19.783 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-05-27T09:08:19.783 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-05-27T09:08:19.783 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-05-27T09:08:19.784 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-05-27T09:08:19.784 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-05-27T09:08:19.784 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-05-27T09:08:19.784 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T09:08:19.784 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-05-27T09:08:19.784 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-05-27T09:08:19.784 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T09:08:19.784 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-05-27T09:08:19.784 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T09:08:19.784 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T09:08:19.784 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T09:08:19.785 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-05-27T09:08:19.807 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-27T09:08:19.824 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-27T09:08:19.835 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-27T09:08:19.835 DEBUG:teuthology.orchestra.run.smithi044:> test '!' -e /home/ubuntu/cephtest 2024-05-27T09:08:19.838 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-05-27T09:08:19.840 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-05-27T09:08:19.856 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-27T09:08:19.866 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-27T09:08:19.866 DEBUG:teuthology.orchestra.run.smithi044:> test -z $(ls -A /var/lib/ceph) 2024-05-27T09:08:19.896 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-05-27T09:08:19.899 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-05-27T09:08:19.989 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-27T09:08:20.071 INFO:teuthology.run_tasks:Running task kernel... 2024-05-27T09:08:20.088 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'} 2024-05-27T09:08:20.088 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-05-27T09:08:20.089 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-05-27T09:08:20.089 DEBUG:teuthology.orchestra.run.smithi044:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T09:08:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:08:20.107 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-05-27T09:08:20.326 DEBUG:teuthology.task.kernel:sha1 for client.0 is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T09:08:20.326 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-27T09:08:20.326 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-05-27T09:08:20.345 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-437.el9.x86_64 2024-05-27T09:08:20.345 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T09:08:20.345 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-27T09:08:20.346 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum upgrade -y linux-firmware 2024-05-27T09:08:23.873 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.1 MB 00:01 2024-05-27T09:08:27.674 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 19 MB 00:02 2024-05-27T09:08:32.858 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.2 MB 00:01 2024-05-27T09:08:35.150 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 17 kB 00:01 2024-05-27T09:08:36.743 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-27T09:08:42.201 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-27T09:08:44.264 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-05-27T09:08:44.265 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-05-27T09:08:44.265 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-05-27T09:08:44.357 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on client.0... 2024-05-27T09:08:44.357 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T09:08:44.587 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ 2024-05-27T09:08:44.587 DEBUG:teuthology.orchestra.run.smithi044:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_gb9459a26b088-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-27T09:08:44.590 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi044.front.sepia.ceph.com to complete... 2024-05-27T09:08:56.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27 09:08:56 URL:https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/kernel-6.9.0_gb9459a26b088-1.x86_64.rpm [356863406/356863406] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-27T09:08:56.686 INFO:teuthology.orchestra.run.smithi044.stderr:FINISHED --2024-05-27 09:08:56-- 2024-05-27T09:08:56.686 INFO:teuthology.orchestra.run.smithi044.stderr:Total wall clock time: 12s 2024-05-27T09:08:56.686 INFO:teuthology.orchestra.run.smithi044.stderr:Downloaded: 1 files, 340M in 12s (28.9 MB/s) 2024-05-27T09:08:56.688 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on client.0... 2024-05-27T09:08:56.688 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-27T09:08:58.874 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying... ######################################## 2024-05-27T09:08:58.987 INFO:teuthology.orchestra.run.smithi044.stdout:Preparing... ######################################## 2024-05-27T09:08:59.100 INFO:teuthology.orchestra.run.smithi044.stdout:Updating / installing... 2024-05-27T09:09:07.381 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-6.9.0_gb9459a26b088-1 ######################################## 2024-05-27T09:09:47.887 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi044.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-27T09:09:47.887 DEBUG:teuthology.orchestra.run.smithi044:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-27T09:09:47.968 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/System.map-6.9.0-gb9459a26b088 2024-05-27T09:09:47.969 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/config-6.9.0-gb9459a26b088 2024-05-27T09:09:47.969 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T09:09:47.969 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/System.map 2024-05-27T09:09:47.969 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/config 2024-05-27T09:09:47.969 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel 2024-05-27T09:09:47.969 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch 2024-05-27T09:09:47.969 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86 2024-05-27T09:09:47.969 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto 2024-05-27T09:09:47.969 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-27T09:09:47.969 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-27T09:09:47.970 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-27T09:09:47.971 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-27T09:09:47.972 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-27T09:09:47.973 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-27T09:09:47.974 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-27T09:09:47.974 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-27T09:09:47.974 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-27T09:09:47.974 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-27T09:09:47.974 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-27T09:09:47.974 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-27T09:09:47.974 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-27T09:09:47.974 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-27T09:09:47.974 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-27T09:09:47.974 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-27T09:09:47.974 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-27T09:09:47.975 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-27T09:09:47.976 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-27T09:09:47.976 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-27T09:09:47.976 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-27T09:09:47.976 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-27T09:09:47.976 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers 2024-05-27T09:09:47.976 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi 2024-05-27T09:09:47.976 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-27T09:09:47.976 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-27T09:09:47.976 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-27T09:09:47.976 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ata_generic.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-27T09:09:47.977 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-27T09:09:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-27T09:09:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-27T09:09:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-27T09:09:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-27T09:09:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-27T09:09:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-27T09:09:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-27T09:09:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-27T09:09:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-27T09:09:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-27T09:09:47.979 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-27T09:09:47.979 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-27T09:09:47.979 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-27T09:09:47.979 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-27T09:09:47.979 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-27T09:09:47.979 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-27T09:09:47.979 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-27T09:09:47.979 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-27T09:09:47.979 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-27T09:09:47.979 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-27T09:09:47.980 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-27T09:09:47.980 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-27T09:09:47.980 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-27T09:09:47.980 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-27T09:09:47.980 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-27T09:09:47.980 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-27T09:09:47.980 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-27T09:09:47.980 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-27T09:09:47.980 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sis.ko 2024-05-27T09:09:47.980 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-27T09:09:47.980 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-27T09:09:47.981 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-27T09:09:47.981 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-27T09:09:47.981 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-27T09:09:47.981 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-27T09:09:47.981 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-27T09:09:47.981 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-27T09:09:47.981 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-27T09:09:47.981 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-27T09:09:47.981 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-27T09:09:47.981 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-27T09:09:47.981 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-27T09:09:47.982 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-27T09:09:47.982 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-27T09:09:47.982 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-27T09:09:47.982 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-27T09:09:47.982 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay 2024-05-27T09:09:47.982 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-27T09:09:47.982 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base 2024-05-27T09:09:47.982 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap 2024-05-27T09:09:47.982 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-27T09:09:47.982 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-27T09:09:47.982 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma 2024-05-27T09:09:47.983 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-27T09:09:47.983 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block 2024-05-27T09:09:47.983 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe 2024-05-27T09:09:47.983 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-27T09:09:47.983 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/brd.ko 2024-05-27T09:09:47.983 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd 2024-05-27T09:09:47.983 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-27T09:09:47.983 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-27T09:09:47.983 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/loop.ko 2024-05-27T09:09:47.983 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx 2024-05-27T09:09:47.984 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-27T09:09:47.984 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-27T09:09:47.984 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk 2024-05-27T09:09:47.984 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-27T09:09:47.984 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-27T09:09:47.984 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-27T09:09:47.984 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/virtio_blk.ko 2024-05-27T09:09:47.984 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback 2024-05-27T09:09:47.984 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-27T09:09:47.984 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkfront.ko 2024-05-27T09:09:47.984 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram 2024-05-27T09:09:47.985 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-27T09:09:47.985 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char 2024-05-27T09:09:47.985 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-27T09:09:47.985 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random 2024-05-27T09:09:47.985 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-27T09:09:47.985 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-27T09:09:47.985 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-27T09:09:47.985 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-27T09:09:47.985 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-27T09:09:47.985 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi 2024-05-27T09:09:47.985 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-27T09:09:47.986 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-27T09:09:47.986 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-27T09:09:47.986 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-27T09:09:47.986 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-27T09:09:47.986 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-27T09:09:47.986 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-27T09:09:47.986 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave 2024-05-27T09:09:47.986 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-27T09:09:47.986 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-27T09:09:47.986 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm.ko 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/uv_mmtimer.ko 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/virtio_console.ko 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-27T09:09:47.987 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-27T09:09:47.988 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-27T09:09:47.988 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-27T09:09:47.988 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-27T09:09:47.988 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-27T09:09:47.988 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto 2024-05-27T09:09:47.988 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp 2024-05-27T09:09:47.988 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-27T09:09:47.988 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-27T09:09:47.988 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel 2024-05-27T09:09:47.989 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat 2024-05-27T09:09:47.989 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common 2024-05-27T09:09:47.989 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-27T09:09:47.989 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-27T09:09:47.989 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-27T09:09:47.989 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-27T09:09:47.989 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-27T09:09:47.989 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio 2024-05-27T09:09:47.989 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-27T09:09:47.989 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca 2024-05-27T09:09:47.990 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-27T09:09:47.990 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq 2024-05-27T09:09:47.990 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-27T09:09:47.990 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma 2024-05-27T09:09:47.990 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw 2024-05-27T09:09:47.990 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-27T09:09:47.990 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat 2024-05-27T09:09:47.990 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-27T09:09:47.990 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac 2024-05-27T09:09:47.991 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-27T09:09:47.991 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-27T09:09:47.991 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-27T09:09:47.991 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-27T09:09:47.991 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-27T09:09:47.991 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-27T09:09:47.991 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-27T09:09:47.991 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-27T09:09:47.991 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-27T09:09:47.991 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-27T09:09:47.991 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/edd.ko 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-27T09:09:47.992 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-27T09:09:47.993 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-27T09:09:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-27T09:09:47.995 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-27T09:09:47.996 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-27T09:09:47.997 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-27T09:09:47.998 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-27T09:09:47.999 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-27T09:09:48.000 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-27T09:09:48.001 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-27T09:09:48.002 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-27T09:09:48.003 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-27T09:09:48.004 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-27T09:09:48.005 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-27T09:09:48.006 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos 2024-05-27T09:09:48.007 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-27T09:09:48.008 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-dev.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-27T09:09:48.009 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-27T09:09:48.010 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-27T09:09:48.010 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-27T09:09:48.010 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband 2024-05-27T09:09:48.010 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core 2024-05-27T09:09:48.010 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-27T09:09:48.010 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-27T09:09:48.010 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-27T09:09:48.010 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-27T09:09:48.010 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-27T09:09:48.010 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-27T09:09:48.010 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib 2024-05-27T09:09:48.012 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-27T09:09:48.013 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-27T09:09:48.014 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-27T09:09:48.015 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-27T09:09:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-27T09:09:48.017 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-27T09:09:48.018 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mfd-core.ko 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl 2024-05-27T09:09:48.019 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-27T09:09:48.020 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-27T09:09:48.021 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-27T09:09:48.022 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-27T09:09:48.023 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-27T09:09:48.024 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-27T09:09:48.025 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-27T09:09:48.026 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip 2024-05-27T09:09:48.027 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-27T09:09:48.028 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-27T09:09:48.029 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-27T09:09:48.030 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi 2024-05-27T09:09:48.031 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-27T09:09:48.032 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-27T09:09:48.033 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun 2024-05-27T09:09:48.034 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-27T09:09:48.035 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-27T09:09:48.036 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/net_failover.ko 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy 2024-05-27T09:09:48.037 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/dp83640.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-27T09:09:48.038 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-27T09:09:48.039 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slhc.ko 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-27T09:09:48.040 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tun.ko 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/virtio_net.ko 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netfront.ko 2024-05-27T09:09:48.041 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug 2024-05-27T09:09:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-27T09:09:48.043 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-27T09:09:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps 2024-05-27T09:09:48.045 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-27T09:09:48.046 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-27T09:09:48.047 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-27T09:09:48.048 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx 2024-05-27T09:09:48.049 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i 2024-05-27T09:09:48.050 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-27T09:09:48.051 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-27T09:09:48.052 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid 2024-05-27T09:09:48.053 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-27T09:09:48.054 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-27T09:09:48.055 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-27T09:09:48.056 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-27T09:09:48.057 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-27T09:09:48.058 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs 2024-05-27T09:09:48.059 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-27T09:09:48.060 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-27T09:09:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-27T09:09:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-27T09:09:48.063 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-27T09:09:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-27T09:09:48.065 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-27T09:09:48.066 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-27T09:09:48.067 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio 2024-05-27T09:09:48.068 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio.ko 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_balloon.ko 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_mmio.ko 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci.ko 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_ring.ko 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-27T09:09:48.069 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-27T09:09:48.070 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-27T09:09:48.071 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-27T09:09:48.072 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs 2024-05-27T09:09:48.073 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm 2024-05-27T09:09:48.074 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/fat.ko 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/vfat.ko 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/fuse.ko 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-27T09:09:48.075 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs 2024-05-27T09:09:48.076 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-27T09:09:48.077 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-27T09:09:48.078 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-27T09:09:48.079 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-27T09:09:48.080 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-27T09:09:48.081 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-27T09:09:48.082 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-27T09:09:48.083 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu/rcutorture.ko 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-27T09:09:48.084 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/asn1_encoder.ko 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-27T09:09:48.085 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd/zstd_compress.ko 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-27T09:09:48.086 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q 2024-05-27T09:09:48.087 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-27T09:09:48.089 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p 2024-05-27T09:09:48.089 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-27T09:09:48.089 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-27T09:09:48.089 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-27T09:09:48.089 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-27T09:09:48.089 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv 2024-05-27T09:09:48.089 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-27T09:09:48.089 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-27T09:09:48.090 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-27T09:09:48.091 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/failover.ko 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver/dns_resolver.ko 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa 2024-05-27T09:09:48.092 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-27T09:09:48.093 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-27T09:09:48.094 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-27T09:09:48.095 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-27T09:09:48.096 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-27T09:09:48.097 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-27T09:09:48.098 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-27T09:09:48.099 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-27T09:09:48.100 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-27T09:09:48.101 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-27T09:09:48.102 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-27T09:09:48.103 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-27T09:09:48.104 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-27T09:09:48.105 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-27T09:09:48.106 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_quota.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tunnel.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-27T09:09:48.107 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-27T09:09:48.108 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-27T09:09:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-27T09:09:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-27T09:09:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-27T09:09:48.112 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched 2024-05-27T09:09:48.113 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-27T09:09:48.114 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-27T09:09:48.115 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-27T09:09:48.116 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix 2024-05-27T09:09:48.117 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys 2024-05-27T09:09:48.118 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys/trusted.ko 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-27T09:09:48.119 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-27T09:09:48.120 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-27T09:09:48.120 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T09:09:48.124 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-27T09:09:48.124 DEBUG:teuthology.orchestra.run.smithi044:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-27T09:09:48.148 INFO:teuthology.orchestra.run.smithi044.stdout:postinstall scriptlet (using /bin/sh): 2024-05-27T09:09:48.148 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T09:09:48.148 INFO:teuthology.orchestra.run.smithi044.stdout: /usr/bin/kernel-install add 6.9.0-gb9459a26b088 /lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T09:09:48.148 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-05-27T09:09:48.148 INFO:teuthology.orchestra.run.smithi044.stdout:for file in vmlinuz System.map config; do 2024-05-27T09:09:48.148 INFO:teuthology.orchestra.run.smithi044.stdout: if ! cmp --silent "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088"; then 2024-05-27T09:09:48.148 INFO:teuthology.orchestra.run.smithi044.stdout: cp "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088" 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout: fi 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:done 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -d "/lib/modules/6.9.0-gb9459a26b088/dtb" ] && \ 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout: ! diff -rq "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" >/dev/null 2>&1; then 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout: rm -rf "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout: cp -r "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:if [ ! -e "/lib/modules/6.9.0-gb9459a26b088/modules.dep" ]; then 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout: /usr/sbin/depmod 6.9.0-gb9459a26b088 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-install remove 6.9.0-gb9459a26b088 2024-05-27T09:09:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-05-27T09:09:48.150 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -qi grub2-tools 2024-05-27T09:09:48.224 INFO:teuthology.orchestra.run.smithi044.stdout:Name : grub2-tools 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Epoch : 1 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Version : 2.06 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Release : 68.el9 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Architecture: x86_64 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Group : Unspecified 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Size : 8264254 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:License : GPLv3+ 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Packager : builder@centos.org 2024-05-27T09:09:48.225 INFO:teuthology.orchestra.run.smithi044.stdout:Vendor : CentOS 2024-05-27T09:09:48.226 INFO:teuthology.orchestra.run.smithi044.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-27T09:09:48.226 INFO:teuthology.orchestra.run.smithi044.stdout:Summary : Support tools for GRUB. 2024-05-27T09:09:48.226 INFO:teuthology.orchestra.run.smithi044.stdout:Description : 2024-05-27T09:09:48.226 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:09:48.226 INFO:teuthology.orchestra.run.smithi044.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-27T09:09:48.226 INFO:teuthology.orchestra.run.smithi044.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-27T09:09:48.226 INFO:teuthology.orchestra.run.smithi044.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-27T09:09:48.226 INFO:teuthology.orchestra.run.smithi044.stdout:hardware devices. 2024-05-27T09:09:48.226 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:09:48.226 INFO:teuthology.orchestra.run.smithi044.stdout:This subpackage provides tools for support of all platforms. 2024-05-27T09:09:48.227 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-27T09:09:48.227 INFO:teuthology.task.kernel:Updating grub on smithi044 to boot 6.9.0-gb9459a26b088 2024-05-27T09:09:48.227 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-27T09:09:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:Generating grub configuration file ... 2024-05-27T09:09:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-27T09:09:50.148 INFO:teuthology.orchestra.run.smithi044.stderr:done 2024-05-27T09:09:50.151 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-05-27T09:09:50.166 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.TWEYdy5Boc 2024-05-27T09:09:50.167 DEBUG:teuthology.orchestra.run.smithi044:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TWEYdy5Boc 2024-05-27T09:09:50.233 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.TWEYdy5Boc 2024-05-27T09:09:50.376 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.TWEYdy5Boc is 6KB 2024-05-27T09:09:50.424 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.TWEYdy5Boc 2024-05-27T09:09:50.439 DEBUG:teuthology.orchestra.run.smithi044:> sudo /bin/ls /boot/loader/entries || true 2024-05-27T09:09:50.507 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-27T09:09:50.507 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-27T09:09:50.508 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-27T09:09:50.508 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088.conf 2024-05-27T09:09:50.509 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088 2024-05-27T09:09:50.660 DEBUG:teuthology.orchestra.run.smithi044:> sudo shutdown -r now 2024-05-27T09:10:20.663 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-27T09:10:20.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-27T09:10:20.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T09:10:39.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.44 2024-05-27T09:10:48.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-27T09:10:48.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T09:11:48.139 DEBUG:teuthology.orchestra.remote:timed out 2024-05-27T09:12:00.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-27T09:12:00.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T09:12:00.507 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-05-27T09:12:01.118 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-27T09:12:01.119 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-gb9459a26b088"... 2024-05-27T09:12:01.119 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-05-27T09:12:01.174 INFO:teuthology.orchestra.run.smithi044.stdout:6.9.0-gb9459a26b088 2024-05-27T09:12:01.175 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-27T09:12:01.175 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T09:12:01.175 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-27T09:12:02.176 DEBUG:teuthology.parallel:result is None 2024-05-27T09:12:02.176 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-27T09:12:02.183 INFO:teuthology.task.internal:Creating test directory... 2024-05-27T09:12:02.184 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T09:12:02.191 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T09:12:02.194 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T09:12:02.244 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-27T09:12:02.251 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-27T09:12:02.269 INFO:teuthology.task.internal:Creating archive directory... 2024-05-27T09:12:02.269 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T09:12:02.275 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T09:12:02.277 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T09:12:02.364 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-27T09:12:02.371 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-27T09:12:02.371 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T09:12:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:12:02.388 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T09:12:02.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:12:02.405 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T09:12:02.412 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T09:12:02.430 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T09:12:02.481 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T09:12:02.492 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T09:12:02.500 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T09:12:02.514 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T09:12:02.530 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T09:12:02.583 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T09:12:02.588 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-27T09:12:02.595 INFO:teuthology.task.internal:Configuring sudo... 2024-05-27T09:12:02.595 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T09:12:02.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T09:12:02.603 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T09:12:02.685 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-27T09:12:02.694 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-27T09:12:02.695 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T09:12:02.701 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T09:12:02.704 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T09:12:02.751 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T09:12:02.837 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T09:12:02.934 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T09:12:03.021 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T09:12:03.109 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:12:03.109 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T09:12:03.201 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T09:12:03.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T09:12:03.302 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T09:12:03.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T09:12:03.431 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-27T09:12:03.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T09:12:03.499 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T09:12:03.540 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T09:12:03.598 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T09:12:03.682 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T09:12:03.723 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-27T09:12:03.723 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T09:12:03.788 DEBUG:teuthology.orchestra.run.smithi044:> sudo service rsyslog restart 2024-05-27T09:12:03.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-05-27T09:12:03.798 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-05-27T09:12:03.834 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T09:12:03.862 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T09:12:03.905 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T09:12:04.378 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-27T09:12:04.418 INFO:teuthology.task.internal:Starting timer... 2024-05-27T09:12:04.418 INFO:teuthology.run_tasks:Running task pcp... 2024-05-27T09:12:04.545 INFO:teuthology.run_tasks:Running task selinux... 2024-05-27T09:12:04.567 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-27T09:12:04.568 DEBUG:teuthology.orchestra.run.smithi044:> sudo service auditd rotate 2024-05-27T09:12:04.684 INFO:teuthology.orchestra.run.smithi044.stdout:Rotating logs: 2024-05-27T09:12:04.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2024-05-27T09:12:04.753 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2024-05-27T09:12:04.755 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-05-27T09:12:04.809 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-05-27T09:12:04.811 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-27T09:12:04.811 DEBUG:teuthology.orchestra.run.smithi044:> /usr/sbin/getenforce 2024-05-27T09:12:04.897 INFO:teuthology.orchestra.run.smithi044.stdout:Permissive 2024-05-27T09:12:04.897 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2024-05-27T09:12:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2024-05-27T09:12:04.936 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-05-27T09:12:04.967 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-05-27T09:12:04.968 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi044.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-05-27T09:12:04.968 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T09:12:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:12:05.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T09:12:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:12:05.061 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T09:12:05.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:12:05.087 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-27T09:12:05.088 DEBUG:teuthology.orchestra.run.smithi044:> sudo /usr/sbin/setenforce permissive 2024-05-27T09:12:05.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2024-05-27T09:12:05.201 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-05-27T09:12:05.228 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-27T09:12:05.237 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-27T09:12:05.241 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-27T09:12:05.388 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-27T09:12:05.399 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-27T09:12:05.400 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi044.front.sepia.ceph.com,smithi050.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-27T09:18:32.952 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi044.front.sepia.ceph.com'), Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-05-27T09:18:32.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-27T09:18:32.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T09:18:33.063 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-05-27T09:18:33.210 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-05-27T09:18:33.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-05-27T09:18:33.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T09:18:33.288 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-05-27T09:18:33.366 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-05-27T09:18:33.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-05-27T09:18:33.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T09:18:33.444 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-05-27T09:18:33.529 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-05-27T09:18:33.530 INFO:teuthology.run_tasks:Running task clock... 2024-05-27T09:18:33.541 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-27T09:18:33.542 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T09:18:33.542 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T09:18:33.550 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T09:18:33.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T09:18:33.552 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T09:18:33.552 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T09:18:33.583 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T09:18:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T09:18:33.599 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T09:18:33.612 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T09:18:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2024-05-27T09:18:33.635 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2024-05-27T09:18:33.635 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-05-27T09:18:33.641 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T09:18:33.645 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-05-27T09:18:33.649 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T09:18:33.659 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T09:18:33.663 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T09:18:33.672 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T09:18:33.684 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T09:18:33.721 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-05-27T09:18:33.724 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T09:18:33.724 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-05-27T09:18:33.724 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:33.724 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:33.724 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:33.724 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:33.725 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-05-27T09:18:33.729 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T09:18:33.729 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-05-27T09:18:33.729 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:33.729 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:33.729 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:33.729 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:33.762 INFO:teuthology.orchestra.run.smithi044.stderr:sudo: ntpd: command not found 2024-05-27T09:18:33.796 INFO:teuthology.orchestra.run.smithi044.stdout:506 Cannot talk to daemon 2024-05-27T09:18:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T09:18:33.884 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T09:18:34.043 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-05-27T09:18:34.053 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T09:18:34.053 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-05-27T09:18:34.053 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:34.053 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:34.053 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:34.053 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:18:34.055 INFO:teuthology.run_tasks:Running task install... 2024-05-27T09:18:34.063 DEBUG:teuthology.task.install:project ceph 2024-05-27T09:18:34.063 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a6507388dd5057528934822c0163b0c347ef1d5d'}} 2024-05-27T09:18:34.063 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a6507388dd5057528934822c0163b0c347ef1d5d'} 2024-05-27T09:18:34.063 INFO:teuthology.task.install:Using flavor: default 2024-05-27T09:18:34.068 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-27T09:18:34.069 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-27T09:18:34.069 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a6507388dd5057528934822c0163b0c347ef1d5d', 'tag': None, 'wait_for_package': False} 2024-05-27T09:18:34.070 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:18:34.071 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a6507388dd5057528934822c0163b0c347ef1d5d', 'tag': None, 'wait_for_package': False} 2024-05-27T09:18:34.072 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:18:34.072 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a6507388dd5057528934822c0163b0c347ef1d5d', 'tag': None, 'wait_for_package': False} 2024-05-27T09:18:34.072 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:18:34.219 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/ 2024-05-27T09:18:34.220 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3868.ga6507388 2024-05-27T09:18:34.294 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/ 2024-05-27T09:18:34.294 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3868.ga6507388 2024-05-27T09:18:34.306 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/ 2024-05-27T09:18:34.306 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3868.ga6507388 2024-05-27T09:18:34.345 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T09:18:34.345 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-27T09:18:34.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T09:18:34.374 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T09:18:34.374 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a6507388dd5057528934822c0163b0c347ef1d5d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T09:18:34.415 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T09:18:34.415 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-27T09:18:34.415 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T09:18:34.442 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T09:18:34.442 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a6507388dd5057528934822c0163b0c347ef1d5d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T09:18:34.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T09:18:34.467 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T09:18:34.467 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:18:34.467 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T09:18:34.508 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T09:18:34.520 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T09:18:34.560 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T09:18:34.560 DEBUG:teuthology.orchestra.run.smithi044:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a6507388dd5057528934822c0163b0c347ef1d5d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T09:18:34.583 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T09:18:34.584 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2024-05-27T09:18:34.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2024-05-27T09:18:34.646 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-05-27T09:18:34.648 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-05-27T09:18:34.691 DEBUG:teuthology.orchestra.run.smithi044:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T09:18:34.811 DEBUG:teuthology.orchestra.run.smithi044:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T09:18:34.820 INFO:teuthology.orchestra.run.smithi050.stdout:45 files removed 2024-05-27T09:18:34.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T09:18:34.870 INFO:teuthology.orchestra.run.smithi189.stdout:45 files removed 2024-05-27T09:18:34.897 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T09:18:34.897 INFO:teuthology.orchestra.run.smithi044.stdout:check_obsoletes = 1 2024-05-27T09:18:34.902 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum clean all 2024-05-27T09:18:35.237 INFO:teuthology.orchestra.run.smithi044.stdout:45 files removed 2024-05-27T09:18:35.276 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T09:18:35.437 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 265 kB/s | 84 kB 00:00 2024-05-27T09:18:35.568 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 205 kB/s | 84 kB 00:00 2024-05-27T09:18:35.887 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-05-27T09:18:36.001 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-05-27T09:18:36.134 INFO:teuthology.orchestra.run.smithi044.stdout:ceph packages for x86_64 204 kB/s | 84 kB 00:00 2024-05-27T09:18:36.255 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-05-27T09:18:36.386 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-05-27T09:18:36.677 INFO:teuthology.orchestra.run.smithi044.stdout:ceph noarch packages 27 kB/s | 12 kB 00:00 2024-05-27T09:18:36.969 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.1 MB 00:00 2024-05-27T09:18:37.302 INFO:teuthology.orchestra.run.smithi044.stdout:ceph source packages 3.5 kB/s | 1.9 kB 00:00 2024-05-27T09:18:38.078 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.1 MB 00:00 2024-05-27T09:18:38.155 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.1 MB 00:01 2024-05-27T09:18:39.948 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-05-27T09:18:40.065 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-05-27T09:18:40.278 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 19 MB 00:02 2024-05-27T09:18:44.382 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-05-27T09:18:45.078 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 5.8 MB/s | 6.2 MB 00:01 2024-05-27T09:18:46.025 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 17 kB 00:00 2024-05-27T09:18:46.695 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 17 kB 00:00 2024-05-27T09:18:47.010 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-05-27T09:18:47.329 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.2 MB 00:03 2024-05-27T09:18:48.813 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 17 kB 00:00 2024-05-27T09:18:49.070 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 21 MB 00:02 2024-05-27T09:18:50.242 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-05-27T09:18:52.368 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-05-27T09:18:54.613 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T09:18:54.613 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T09:18:54.625 INFO:teuthology.orchestra.run.smithi050.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T09:18:54.625 INFO:teuthology.orchestra.run.smithi050.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T09:18:54.626 INFO:teuthology.orchestra.run.smithi050.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T09:18:54.627 INFO:teuthology.orchestra.run.smithi050.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T09:18:54.629 INFO:teuthology.orchestra.run.smithi050.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T09:18:54.631 INFO:teuthology.orchestra.run.smithi050.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T09:18:54.632 INFO:teuthology.orchestra.run.smithi050.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T09:18:54.699 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-27T09:18:54.710 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-05-27T09:18:54.710 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-27T09:18:54.710 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:19.0.0-3868.ga6507388.el9 ceph 6.6 k 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:19.0.0-3868.ga6507388.el9 ceph 5.4 M 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:19.0.0-3868.ga6507388.el9 ceph 863 k 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3868.ga6507388.el9 ceph 147 k 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:19.0.0-3868.ga6507388.el9 ceph 1.5 M 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 143 k 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 3.5 M 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 7.4 M 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 49 k 2024-05-27T09:18:54.711 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:19.0.0-3868.ga6507388.el9 ceph 10 M 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:19.0.0-3868.ga6507388.el9 ceph 43 M 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 273 k 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 764 k 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-mirror x86_64 2:19.0.0-3868.ga6507388.el9 ceph 224 k 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-shell x86_64 2:19.0.0-3868.ga6507388.el9 ceph 27 k 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-top noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 18 k 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:19.0.0-3868.ga6507388.el9 ceph 32 k 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 756 k 2024-05-27T09:18:54.712 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:19.0.0-3868.ga6507388.el9 ceph 127 k 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:19.0.0-3868.ga6507388.el9 ceph 161 k 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:19.0.0-3868.ga6507388.el9 ceph 321 k 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:19.0.0-3868.ga6507388.el9 ceph 301 k 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:19.0.0-3868.ga6507388.el9 ceph 100 k 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:19.0.0-3868.ga6507388.el9 ceph 86 k 2024-05-27T09:18:54.713 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:19.0.0-3868.ga6507388.el9 ceph 3.0 M 2024-05-27T09:18:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:19.0.0-3868.ga6507388.el9 ceph 170 k 2024-05-27T09:18:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T09:18:54.714 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2024-05-27T09:18:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T09:18:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T09:18:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T09:18:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T09:18:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T09:18:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T09:18:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T09:18:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 3.3 M 2024-05-27T09:18:54.715 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 3.1 M 2024-05-27T09:18:54.715 INFO:teuthology.orchestra.run.smithi050.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T09:18:54.715 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T09:18:54.715 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-05-27T09:18:54.715 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T09:18:54.715 INFO:teuthology.orchestra.run.smithi050.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T09:18:54.715 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T09:18:54.715 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:19.0.0-3868.ga6507388.el9 ceph 20 M 2024-05-27T09:18:54.715 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 34 k 2024-05-27T09:18:54.715 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:19.0.0-3868.ga6507388.el9 ceph 2.3 M 2024-05-27T09:18:54.715 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 274 k 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:19.0.0-3868.ga6507388.el9 ceph 4.9 M 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:19.0.0-3868.ga6507388.el9 ceph 18 M 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 16 k 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:19.0.0-3868.ga6507388.el9 ceph 25 k 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:19.0.0-3868.ga6507388.el9 ceph 157 k 2024-05-27T09:18:54.716 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 497 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 5.2 M 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T09:18:54.717 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T09:18:54.718 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T09:18:54.718 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T09:18:54.718 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T09:18:54.718 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T09:18:54.718 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T09:18:54.718 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T09:18:54.718 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T09:18:54.718 INFO:teuthology.orchestra.run.smithi050.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T09:18:54.718 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T09:18:54.718 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T09:18:54.718 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:19.0.0-3868.ga6507388.el9 ceph 45 k 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:19.0.0-3868.ga6507388.el9 ceph 136 k 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T09:18:54.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T09:18:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T09:18:54.721 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T09:18:54.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T09:18:54.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T09:18:54.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T09:18:54.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T09:18:54.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T09:18:54.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T09:18:54.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T09:18:54.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T09:18:54.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T09:18:54.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T09:18:54.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T09:18:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout:Install 148 Packages 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 11 Packages 2024-05-27T09:18:54.724 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:18:54.725 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 219 M 2024-05-27T09:18:54.725 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-27T09:18:55.228 INFO:teuthology.orchestra.run.smithi050.stdout:(1/159): ceph-19.0.0-3868.ga6507388.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-05-27T09:18:55.461 INFO:teuthology.orchestra.run.smithi050.stdout:(2/159): ceph-fuse-19.0.0-3868.ga6507388.el9.x8 3.6 MB/s | 863 kB 00:00 2024-05-27T09:18:55.475 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-05-27T09:18:55.489 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-05-27T09:18:55.537 INFO:teuthology.orchestra.run.smithi050.stdout:(3/159): ceph-immutable-object-cache-19.0.0-386 1.9 MB/s | 147 kB 00:00 2024-05-27T09:18:55.954 INFO:teuthology.orchestra.run.smithi050.stdout:(4/159): ceph-mds-19.0.0-3868.ga6507388.el9.x86 5.4 MB/s | 2.3 MB 00:00 2024-05-27T09:18:56.237 INFO:teuthology.orchestra.run.smithi050.stdout:(5/159): ceph-mgr-19.0.0-3868.ga6507388.el9.x86 5.3 MB/s | 1.5 MB 00:00 2024-05-27T09:18:56.679 INFO:teuthology.orchestra.run.smithi050.stdout:(6/159): ceph-common-19.0.0-3868.ga6507388.el9. 12 MB/s | 20 MB 00:01 2024-05-27T09:18:56.763 INFO:teuthology.orchestra.run.smithi050.stdout:(7/159): ceph-base-19.0.0-3868.ga6507388.el9.x8 3.1 MB/s | 5.4 MB 00:01 2024-05-27T09:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout:(8/159): ceph-mon-19.0.0-3868.ga6507388.el9.x86 6.2 MB/s | 4.9 MB 00:00 2024-05-27T09:18:57.097 INFO:teuthology.orchestra.run.smithi050.stdout:(9/159): ceph-selinux-19.0.0-3868.ga6507388.el9 337 kB/s | 25 kB 00:00 2024-05-27T09:18:57.666 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T09:18:57.666 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T09:18:57.677 INFO:teuthology.orchestra.run.smithi189.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T09:18:57.678 INFO:teuthology.orchestra.run.smithi189.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T09:18:57.679 INFO:teuthology.orchestra.run.smithi189.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T09:18:57.680 INFO:teuthology.orchestra.run.smithi189.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T09:18:57.681 INFO:teuthology.orchestra.run.smithi189.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T09:18:57.684 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T09:18:57.685 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T09:18:57.749 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-05-27T09:18:57.769 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:19.0.0-3868.ga6507388.el9 ceph 6.6 k 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:19.0.0-3868.ga6507388.el9 ceph 5.4 M 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:19.0.0-3868.ga6507388.el9 ceph 863 k 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3868.ga6507388.el9 ceph 147 k 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:19.0.0-3868.ga6507388.el9 ceph 1.5 M 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 143 k 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 3.5 M 2024-05-27T09:18:57.770 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 7.4 M 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 49 k 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:19.0.0-3868.ga6507388.el9 ceph 10 M 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:19.0.0-3868.ga6507388.el9 ceph 43 M 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 273 k 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 764 k 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror x86_64 2:19.0.0-3868.ga6507388.el9 ceph 224 k 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell x86_64 2:19.0.0-3868.ga6507388.el9 ceph 27 k 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 18 k 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:19.0.0-3868.ga6507388.el9 ceph 32 k 2024-05-27T09:18:57.771 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 756 k 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:19.0.0-3868.ga6507388.el9 ceph 127 k 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:19.0.0-3868.ga6507388.el9 ceph 161 k 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:19.0.0-3868.ga6507388.el9 ceph 321 k 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:19.0.0-3868.ga6507388.el9 ceph 301 k 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:19.0.0-3868.ga6507388.el9 ceph 100 k 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:19.0.0-3868.ga6507388.el9 ceph 86 k 2024-05-27T09:18:57.772 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:19.0.0-3868.ga6507388.el9 ceph 3.0 M 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:19.0.0-3868.ga6507388.el9 ceph 170 k 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 3.3 M 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 3.1 M 2024-05-27T09:18:57.773 INFO:teuthology.orchestra.run.smithi189.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T09:18:57.774 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T09:18:57.774 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-05-27T09:18:57.774 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T09:18:57.774 INFO:teuthology.orchestra.run.smithi189.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T09:18:57.775 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T09:18:57.775 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:19.0.0-3868.ga6507388.el9 ceph 20 M 2024-05-27T09:18:57.775 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 34 k 2024-05-27T09:18:57.775 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:19.0.0-3868.ga6507388.el9 ceph 2.3 M 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 274 k 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:19.0.0-3868.ga6507388.el9 ceph 4.9 M 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:19.0.0-3868.ga6507388.el9 ceph 18 M 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 16 k 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:19.0.0-3868.ga6507388.el9 ceph 25 k 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T09:18:57.776 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:19.0.0-3868.ga6507388.el9 ceph 157 k 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 497 k 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 5.2 M 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T09:18:57.777 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T09:18:57.778 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:19.0.0-3868.ga6507388.el9 ceph 45 k 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:19.0.0-3868.ga6507388.el9 ceph 136 k 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T09:18:57.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T09:18:57.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T09:18:57.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T09:18:57.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T09:18:57.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T09:18:57.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T09:18:57.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T09:18:57.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T09:18:57.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T09:18:57.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T09:18:57.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T09:18:57.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T09:18:57.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T09:18:57.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout:Installing weak dependencies: 2024-05-27T09:18:57.784 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T09:18:57.785 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:18:57.785 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-05-27T09:18:57.785 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-05-27T09:18:57.785 INFO:teuthology.orchestra.run.smithi189.stdout:Install 148 Packages 2024-05-27T09:18:57.785 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 11 Packages 2024-05-27T09:18:57.785 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:18:57.786 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 219 M 2024-05-27T09:18:57.786 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-05-27T09:18:57.830 INFO:teuthology.orchestra.run.smithi050.stdout:(10/159): ceph-osd-19.0.0-3868.ga6507388.el9.x8 15 MB/s | 18 MB 00:01 2024-05-27T09:18:57.889 INFO:teuthology.orchestra.run.smithi050.stdout:(11/159): cephfs-mirror-19.0.0-3868.ga6507388.e 3.8 MB/s | 224 kB 00:00 2024-05-27T09:18:57.939 INFO:teuthology.orchestra.run.smithi050.stdout:(12/159): cephfs-shell-19.0.0-3868.ga6507388.el 543 kB/s | 27 kB 00:00 2024-05-27T09:18:57.989 INFO:teuthology.orchestra.run.smithi050.stdout:(13/159): libcephfs-devel-19.0.0-3868.ga6507388 635 kB/s | 32 kB 00:00 2024-05-27T09:18:58.042 INFO:teuthology.orchestra.run.smithi044.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T09:18:58.044 INFO:teuthology.orchestra.run.smithi044.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T09:18:58.065 INFO:teuthology.orchestra.run.smithi044.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T09:18:58.067 INFO:teuthology.orchestra.run.smithi044.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T09:18:58.068 INFO:teuthology.orchestra.run.smithi044.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T09:18:58.069 INFO:teuthology.orchestra.run.smithi044.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T09:18:58.072 INFO:teuthology.orchestra.run.smithi044.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T09:18:58.076 INFO:teuthology.orchestra.run.smithi044.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T09:18:58.077 INFO:teuthology.orchestra.run.smithi044.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T09:18:58.090 INFO:teuthology.orchestra.run.smithi050.stdout:(14/159): libcephfs2-19.0.0-3868.ga6507388.el9. 7.4 MB/s | 756 kB 00:00 2024-05-27T09:18:58.155 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-05-27T09:18:58.156 INFO:teuthology.orchestra.run.smithi050.stdout:(15/159): libcephsqlite-19.0.0-3868.ga6507388.e 2.3 MB/s | 157 kB 00:00 2024-05-27T09:18:58.172 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-05-27T09:18:58.172 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-05-27T09:18:58.172 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-05-27T09:18:58.172 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-05-27T09:18:58.172 INFO:teuthology.orchestra.run.smithi044.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T09:18:58.172 INFO:teuthology.orchestra.run.smithi044.stdout: ceph x86_64 2:19.0.0-3868.ga6507388.el9 ceph 6.6 k 2024-05-27T09:18:58.172 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base x86_64 2:19.0.0-3868.ga6507388.el9 ceph 5.4 M 2024-05-27T09:18:58.172 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse x86_64 2:19.0.0-3868.ga6507388.el9 ceph 863 k 2024-05-27T09:18:58.172 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3868.ga6507388.el9 ceph 147 k 2024-05-27T09:18:58.172 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr x86_64 2:19.0.0-3868.ga6507388.el9 ceph 1.5 M 2024-05-27T09:18:58.172 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 143 k 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 3.5 M 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 7.4 M 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 49 k 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw x86_64 2:19.0.0-3868.ga6507388.el9 ceph 10 M 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test x86_64 2:19.0.0-3868.ga6507388.el9 ceph 43 M 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 273 k 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 764 k 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-mirror x86_64 2:19.0.0-3868.ga6507388.el9 ceph 224 k 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-shell x86_64 2:19.0.0-3868.ga6507388.el9 ceph 27 k 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-top noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 18 k 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T09:18:58.173 INFO:teuthology.orchestra.run.smithi044.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T09:18:58.174 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel x86_64 2:19.0.0-3868.ga6507388.el9 ceph 32 k 2024-05-27T09:18:58.174 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 756 k 2024-05-27T09:18:58.174 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel x86_64 2:19.0.0-3868.ga6507388.el9 ceph 127 k 2024-05-27T09:18:58.174 INFO:teuthology.orchestra.run.smithi044.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T09:18:58.174 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T09:18:58.174 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T09:18:58.174 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T09:18:58.174 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T09:18:58.174 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T09:18:58.174 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs x86_64 2:19.0.0-3868.ga6507388.el9 ceph 161 k 2024-05-27T09:18:58.174 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados x86_64 2:19.0.0-3868.ga6507388.el9 ceph 321 k 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd x86_64 2:19.0.0-3868.ga6507388.el9 ceph 301 k 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw x86_64 2:19.0.0-3868.ga6507388.el9 ceph 100 k 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse x86_64 2:19.0.0-3868.ga6507388.el9 ceph 86 k 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror x86_64 2:19.0.0-3868.ga6507388.el9 ceph 3.0 M 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd x86_64 2:19.0.0-3868.ga6507388.el9 ceph 170 k 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrading: 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T09:18:58.175 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: librados2 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 3.3 M 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 3.1 M 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common x86_64 2:19.0.0-3868.ga6507388.el9 ceph 20 M 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 34 k 2024-05-27T09:18:58.176 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds x86_64 2:19.0.0-3868.ga6507388.el9 ceph 2.3 M 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 274 k 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon x86_64 2:19.0.0-3868.ga6507388.el9 ceph 4.9 M 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd x86_64 2:19.0.0-3868.ga6507388.el9 ceph 18 M 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3868.ga6507388.el9 ceph-noarch 16 k 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux x86_64 2:19.0.0-3868.ga6507388.el9 ceph 25 k 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T09:18:58.177 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite x86_64 2:19.0.0-3868.ga6507388.el9 ceph 157 k 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 497 k 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2 x86_64 2:19.0.0-3868.ga6507388.el9 ceph 5.2 M 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T09:18:58.178 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T09:18:58.179 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse x86_64 2:19.0.0-3868.ga6507388.el9 ceph 45 k 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common x86_64 2:19.0.0-3868.ga6507388.el9 ceph 136 k 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T09:18:58.180 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T09:18:58.181 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T09:18:58.182 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T09:18:58.183 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T09:18:58.184 INFO:teuthology.orchestra.run.smithi044.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T09:18:58.185 INFO:teuthology.orchestra.run.smithi044.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T09:18:58.185 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T09:18:58.185 INFO:teuthology.orchestra.run.smithi044.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T09:18:58.185 INFO:teuthology.orchestra.run.smithi044.stdout:Installing weak dependencies: 2024-05-27T09:18:58.185 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T09:18:58.185 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:18:58.185 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-05-27T09:18:58.185 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-05-27T09:18:58.185 INFO:teuthology.orchestra.run.smithi044.stdout:Install 148 Packages 2024-05-27T09:18:58.185 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrade 11 Packages 2024-05-27T09:18:58.185 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:18:58.186 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 219 M 2024-05-27T09:18:58.186 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-05-27T09:18:58.215 INFO:teuthology.orchestra.run.smithi050.stdout:(16/159): librados-devel-19.0.0-3868.ga6507388. 2.1 MB/s | 127 kB 00:00 2024-05-27T09:18:58.299 INFO:teuthology.orchestra.run.smithi050.stdout:(17/159): libradosstriper1-19.0.0-3868.ga650738 5.8 MB/s | 497 kB 00:00 2024-05-27T09:18:58.691 INFO:teuthology.orchestra.run.smithi050.stdout:(18/159): librgw2-19.0.0-3868.ga6507388.el9.x86 13 MB/s | 5.2 MB 00:00 2024-05-27T09:18:58.707 INFO:teuthology.orchestra.run.smithi189.stdout:(1/159): ceph-19.0.0-3868.ga6507388.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-05-27T09:18:58.749 INFO:teuthology.orchestra.run.smithi050.stdout:(19/159): python3-ceph-argparse-19.0.0-3868.ga6 775 kB/s | 45 kB 00:00 2024-05-27T09:18:58.867 INFO:teuthology.orchestra.run.smithi050.stdout:(20/159): ceph-radosgw-19.0.0-3868.ga6507388.el 4.8 MB/s | 10 MB 00:02 2024-05-27T09:18:58.883 INFO:teuthology.orchestra.run.smithi050.stdout:(21/159): python3-ceph-common-19.0.0-3868.ga650 1.0 MB/s | 136 kB 00:00 2024-05-27T09:18:58.975 INFO:teuthology.orchestra.run.smithi050.stdout:(22/159): python3-cephfs-19.0.0-3868.ga6507388. 1.5 MB/s | 161 kB 00:00 2024-05-27T09:18:59.000 INFO:teuthology.orchestra.run.smithi050.stdout:(23/159): python3-rados-19.0.0-3868.ga6507388.e 2.7 MB/s | 321 kB 00:00 2024-05-27T09:18:59.084 INFO:teuthology.orchestra.run.smithi050.stdout:(24/159): python3-rbd-19.0.0-3868.ga6507388.el9 2.7 MB/s | 301 kB 00:00 2024-05-27T09:18:59.110 INFO:teuthology.orchestra.run.smithi050.stdout:(25/159): python3-rgw-19.0.0-3868.ga6507388.el9 916 kB/s | 100 kB 00:00 2024-05-27T09:18:59.176 INFO:teuthology.orchestra.run.smithi050.stdout:(26/159): rbd-fuse-19.0.0-3868.ga6507388.el9.x8 935 kB/s | 86 kB 00:00 2024-05-27T09:18:59.341 INFO:teuthology.orchestra.run.smithi189.stdout:(2/159): ceph-fuse-19.0.0-3868.ga6507388.el9.x8 1.3 MB/s | 863 kB 00:00 2024-05-27T09:18:59.382 INFO:teuthology.orchestra.run.smithi044.stdout:(1/159): ceph-19.0.0-3868.ga6507388.el9.x86_64. 14 kB/s | 6.6 kB 00:00 2024-05-27T09:18:59.460 INFO:teuthology.orchestra.run.smithi050.stdout:(27/159): rbd-mirror-19.0.0-3868.ga6507388.el9. 8.7 MB/s | 3.0 MB 00:00 2024-05-27T09:18:59.508 INFO:teuthology.orchestra.run.smithi189.stdout:(3/159): ceph-immutable-object-cache-19.0.0-386 883 kB/s | 147 kB 00:00 2024-05-27T09:18:59.519 INFO:teuthology.orchestra.run.smithi050.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3868.g 589 kB/s | 34 kB 00:00 2024-05-27T09:18:59.611 INFO:teuthology.orchestra.run.smithi050.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3868.ga650738 1.5 MB/s | 143 kB 00:00 2024-05-27T09:18:59.953 INFO:teuthology.orchestra.run.smithi050.stdout:(30/159): rbd-nbd-19.0.0-3868.ga6507388.el9.x86 219 kB/s | 170 kB 00:00 2024-05-27T09:19:00.049 INFO:teuthology.orchestra.run.smithi044.stdout:(2/159): ceph-fuse-19.0.0-3868.ga6507388.el9.x8 1.3 MB/s | 863 kB 00:00 2024-05-27T09:19:00.103 INFO:teuthology.orchestra.run.smithi050.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3868.ga6507 7.2 MB/s | 3.5 MB 00:00 2024-05-27T09:19:00.141 INFO:teuthology.orchestra.run.smithi044.stdout:(3/159): ceph-immutable-object-cache-19.0.0-386 1.6 MB/s | 147 kB 00:00 2024-05-27T09:19:00.187 INFO:teuthology.orchestra.run.smithi050.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3868.ga6 3.2 MB/s | 274 kB 00:00 2024-05-27T09:19:00.287 INFO:teuthology.orchestra.run.smithi050.stdout:(33/159): ceph-mgr-rook-19.0.0-3868.ga6507388.e 494 kB/s | 49 kB 00:00 2024-05-27T09:19:00.346 INFO:teuthology.orchestra.run.smithi050.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3868.ga 282 kB/s | 16 kB 00:00 2024-05-27T09:19:00.413 INFO:teuthology.orchestra.run.smithi050.stdout:(35/159): ceph-volume-19.0.0-3868.ga6507388.el9 4.0 MB/s | 273 kB 00:00 2024-05-27T09:19:00.475 INFO:teuthology.orchestra.run.smithi189.stdout:(4/159): ceph-common-19.0.0-3868.ga6507388.el9. 9.3 MB/s | 20 MB 00:02 2024-05-27T09:19:00.538 INFO:teuthology.orchestra.run.smithi050.stdout:(36/159): cephadm-19.0.0-3868.ga6507388.el9.noa 6.0 MB/s | 764 kB 00:00 2024-05-27T09:19:00.638 INFO:teuthology.orchestra.run.smithi050.stdout:(37/159): cephfs-top-19.0.0-3868.ga6507388.el9. 182 kB/s | 18 kB 00:00 2024-05-27T09:19:00.675 INFO:teuthology.orchestra.run.smithi189.stdout:(5/159): ceph-mgr-19.0.0-3868.ga6507388.el9.x86 7.5 MB/s | 1.5 MB 00:00 2024-05-27T09:19:00.717 INFO:teuthology.orchestra.run.smithi189.stdout:(6/159): ceph-mds-19.0.0-3868.ga6507388.el9.x86 1.9 MB/s | 2.3 MB 00:01 2024-05-27T09:19:00.797 INFO:teuthology.orchestra.run.smithi044.stdout:(4/159): ceph-mds-19.0.0-3868.ga6507388.el9.x86 3.5 MB/s | 2.3 MB 00:00 2024-05-27T09:19:01.092 INFO:teuthology.orchestra.run.smithi189.stdout:(7/159): ceph-base-19.0.0-3868.ga6507388.el9.x8 2.0 MB/s | 5.4 MB 00:02 2024-05-27T09:19:01.193 INFO:teuthology.orchestra.run.smithi189.stdout:(8/159): ceph-mon-19.0.0-3868.ga6507388.el9.x86 9.4 MB/s | 4.9 MB 00:00 2024-05-27T09:19:01.199 INFO:teuthology.orchestra.run.smithi044.stdout:(5/159): ceph-mgr-19.0.0-3868.ga6507388.el9.x86 3.8 MB/s | 1.5 MB 00:00 2024-05-27T09:19:01.251 INFO:teuthology.orchestra.run.smithi189.stdout:(9/159): ceph-selinux-19.0.0-3868.ga6507388.el9 433 kB/s | 25 kB 00:00 2024-05-27T09:19:01.330 INFO:teuthology.orchestra.run.smithi050.stdout:(38/159): fuse-2.9.9-15.el9.x86_64.rpm 116 kB/s | 80 kB 00:00 2024-05-27T09:19:01.864 INFO:teuthology.orchestra.run.smithi050.stdout:(39/159): ceph-mgr-diskprediction-local-19.0.0- 3.9 MB/s | 7.4 MB 00:01 2024-05-27T09:19:02.091 INFO:teuthology.orchestra.run.smithi044.stdout:(6/159): ceph-mon-19.0.0-3868.ga6507388.el9.x86 5.5 MB/s | 4.9 MB 00:00 2024-05-27T09:19:02.322 INFO:teuthology.orchestra.run.smithi050.stdout:(40/159): libgfortran-11.4.1-3.el9.x86_64.rpm 810 kB/s | 803 kB 00:00 2024-05-27T09:19:02.575 INFO:teuthology.orchestra.run.smithi044.stdout:(7/159): ceph-base-19.0.0-3868.ga6507388.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-05-27T09:19:02.664 INFO:teuthology.orchestra.run.smithi050.stdout:(41/159): libquadmath-11.4.1-3.el9.x86_64.rpm 244 kB/s | 195 kB 00:00 2024-05-27T09:19:03.148 INFO:teuthology.orchestra.run.smithi050.stdout:(42/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 307 kB/s | 253 kB 00:00 2024-05-27T09:19:03.682 INFO:teuthology.orchestra.run.smithi050.stdout:(43/159): python3-cryptography-36.0.1-4.el9.x86 1.2 MB/s | 1.2 MB 00:01 2024-05-27T09:19:03.882 INFO:teuthology.orchestra.run.smithi050.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 145 kB/s | 106 kB 00:00 2024-05-27T09:19:04.432 INFO:teuthology.orchestra.run.smithi050.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 180 kB/s | 135 kB 00:00 2024-05-27T09:19:04.517 INFO:teuthology.orchestra.run.smithi044.stdout:(8/159): ceph-osd-19.0.0-3868.ga6507388.el9.x86 7.3 MB/s | 18 MB 00:02 2024-05-27T09:19:04.616 INFO:teuthology.orchestra.run.smithi050.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 170 kB/s | 125 kB 00:00 2024-05-27T09:19:04.617 INFO:teuthology.orchestra.run.smithi044.stdout:(9/159): ceph-selinux-19.0.0-3868.ga6507388.el9 255 kB/s | 25 kB 00:00 2024-05-27T09:19:04.841 INFO:teuthology.orchestra.run.smithi050.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 3.0 MB/s | 689 kB 00:00 2024-05-27T09:19:04.959 INFO:teuthology.orchestra.run.smithi189.stdout:(10/159): ceph-radosgw-19.0.0-3868.ga6507388.el 2.6 MB/s | 10 MB 00:03 2024-05-27T09:19:04.991 INFO:teuthology.orchestra.run.smithi050.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 6.1 MB/s | 943 kB 00:00 2024-05-27T09:19:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-05-27T09:19:05.083 INFO:teuthology.orchestra.run.smithi050.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 519 kB/s | 22 kB 00:00 2024-05-27T09:19:05.159 INFO:teuthology.orchestra.run.smithi189.stdout:(11/159): cephfs-mirror-19.0.0-3868.ga6507388.e 1.1 MB/s | 224 kB 00:00 2024-05-27T09:19:05.167 INFO:teuthology.orchestra.run.smithi050.stdout:(51/159): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-05-27T09:19:05.217 INFO:teuthology.orchestra.run.smithi050.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 656 kB/s | 33 kB 00:00 2024-05-27T09:19:05.243 INFO:teuthology.orchestra.run.smithi050.stdout:(53/159): python3-urllib3-1.26.5-5.el9.noarch.r 265 kB/s | 215 kB 00:00 2024-05-27T09:19:05.268 INFO:teuthology.orchestra.run.smithi189.stdout:(12/159): cephfs-shell-19.0.0-3868.ga6507388.el 250 kB/s | 27 kB 00:00 2024-05-27T09:19:05.327 INFO:teuthology.orchestra.run.smithi050.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 219 kB/s | 18 kB 00:00 2024-05-27T09:19:05.377 INFO:teuthology.orchestra.run.smithi050.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-05-27T09:19:05.485 INFO:teuthology.orchestra.run.smithi050.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-05-27T09:19:05.536 INFO:teuthology.orchestra.run.smithi050.stdout:(57/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-05-27T09:19:05.595 INFO:teuthology.orchestra.run.smithi050.stdout:(58/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.9 MB/s | 3.0 MB 00:00 2024-05-27T09:19:05.643 INFO:teuthology.orchestra.run.smithi189.stdout:(13/159): ceph-test-19.0.0-3868.ga6507388.el9.x 9.9 MB/s | 43 MB 00:04 2024-05-27T09:19:05.653 INFO:teuthology.orchestra.run.smithi050.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-05-27T09:19:05.669 INFO:teuthology.orchestra.run.smithi189.stdout:(14/159): libcephfs-devel-19.0.0-3868.ga6507388 79 kB/s | 32 kB 00:00 2024-05-27T09:19:05.712 INFO:teuthology.orchestra.run.smithi050.stdout:(60/159): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-05-27T09:19:05.737 INFO:teuthology.orchestra.run.smithi050.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 304 kB/s | 25 kB 00:00 2024-05-27T09:19:05.763 INFO:teuthology.orchestra.run.smithi050.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-05-27T09:19:05.785 INFO:teuthology.orchestra.run.smithi189.stdout:(15/159): libcephfs2-19.0.0-3868.ga6507388.el9. 5.2 MB/s | 756 kB 00:00 2024-05-27T09:19:05.796 INFO:teuthology.orchestra.run.smithi050.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-05-27T09:19:05.821 INFO:teuthology.orchestra.run.smithi050.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-05-27T09:19:05.827 INFO:teuthology.orchestra.run.smithi189.stdout:(16/159): libcephsqlite-19.0.0-3868.ga6507388.e 992 kB/s | 157 kB 00:00 2024-05-27T09:19:05.847 INFO:teuthology.orchestra.run.smithi050.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 700 kB/s | 35 kB 00:00 2024-05-27T09:19:05.853 INFO:teuthology.orchestra.run.smithi189.stdout:(17/159): librados-devel-19.0.0-3868.ga6507388. 1.9 MB/s | 127 kB 00:00 2024-05-27T09:19:06.087 INFO:teuthology.orchestra.run.smithi189.stdout:(18/159): libradosstriper1-19.0.0-3868.ga650738 1.9 MB/s | 497 kB 00:00 2024-05-27T09:19:06.187 INFO:teuthology.orchestra.run.smithi189.stdout:(19/159): python3-ceph-argparse-19.0.0-3868.ga6 452 kB/s | 45 kB 00:00 2024-05-27T09:19:06.295 INFO:teuthology.orchestra.run.smithi189.stdout:(20/159): python3-ceph-common-19.0.0-3868.ga650 1.2 MB/s | 136 kB 00:00 2024-05-27T09:19:06.322 INFO:teuthology.orchestra.run.smithi050.stdout:(66/159): openblas-openmp-0.3.21-2.el9.x86_64.r 9.5 MB/s | 4.7 MB 00:00 2024-05-27T09:19:06.404 INFO:teuthology.orchestra.run.smithi189.stdout:(21/159): python3-cephfs-19.0.0-3868.ga6507388. 1.5 MB/s | 161 kB 00:00 2024-05-27T09:19:06.464 INFO:teuthology.orchestra.run.smithi050.stdout:(67/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.2 MB/s | 4.4 MB 00:00 2024-05-27T09:19:06.548 INFO:teuthology.orchestra.run.smithi050.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 7.1 MB/s | 1.6 MB 00:00 2024-05-27T09:19:06.606 INFO:teuthology.orchestra.run.smithi050.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-05-27T09:19:06.621 INFO:teuthology.orchestra.run.smithi189.stdout:(22/159): librgw2-19.0.0-3868.ga6507388.el9.x86 6.7 MB/s | 5.2 MB 00:00 2024-05-27T09:19:06.640 INFO:teuthology.orchestra.run.smithi050.stdout:(70/159): postgresql-contrib-13.14-1.el9.x86_64 5.1 MB/s | 911 kB 00:00 2024-05-27T09:19:06.646 INFO:teuthology.orchestra.run.smithi189.stdout:(23/159): python3-rados-19.0.0-3868.ga6507388.e 1.3 MB/s | 321 kB 00:00 2024-05-27T09:19:06.697 INFO:teuthology.orchestra.run.smithi189.stdout:(24/159): python3-rbd-19.0.0-3868.ga6507388.el9 3.9 MB/s | 301 kB 00:00 2024-05-27T09:19:06.747 INFO:teuthology.orchestra.run.smithi189.stdout:(25/159): python3-rgw-19.0.0-3868.ga6507388.el9 992 kB/s | 100 kB 00:00 2024-05-27T09:19:06.772 INFO:teuthology.orchestra.run.smithi189.stdout:(26/159): rbd-fuse-19.0.0-3868.ga6507388.el9.x8 1.1 MB/s | 86 kB 00:00 2024-05-27T09:19:06.790 INFO:teuthology.orchestra.run.smithi050.stdout:(71/159): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-05-27T09:19:06.856 INFO:teuthology.orchestra.run.smithi189.stdout:(27/159): rbd-nbd-19.0.0-3868.ga6507388.el9.x86 2.0 MB/s | 170 kB 00:00 2024-05-27T09:19:06.923 INFO:teuthology.orchestra.run.smithi189.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3868.g 513 kB/s | 34 kB 00:00 2024-05-27T09:19:06.998 INFO:teuthology.orchestra.run.smithi189.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3868.ga650738 1.9 MB/s | 143 kB 00:00 2024-05-27T09:19:07.216 INFO:teuthology.orchestra.run.smithi050.stdout:(72/159): ceph-test-19.0.0-3868.ga6507388.el9.x 4.3 MB/s | 43 MB 00:10 2024-05-27T09:19:07.241 INFO:teuthology.orchestra.run.smithi050.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 194 kB/s | 87 kB 00:00 2024-05-27T09:19:07.299 INFO:teuthology.orchestra.run.smithi050.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.0 MB/s | 61 kB 00:00 2024-05-27T09:19:07.383 INFO:teuthology.orchestra.run.smithi050.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-05-27T09:19:07.457 INFO:teuthology.orchestra.run.smithi189.stdout:(30/159): ceph-mgr-dashboard-19.0.0-3868.ga6507 7.7 MB/s | 3.5 MB 00:00 2024-05-27T09:19:07.458 INFO:teuthology.orchestra.run.smithi050.stdout:(76/159): postgresql-server-13.14-1.el9.x86_64. 6.8 MB/s | 5.8 MB 00:00 2024-05-27T09:19:07.508 INFO:teuthology.orchestra.run.smithi050.stdout:(77/159): python3-jinja2-2.11.3-6.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-05-27T09:19:07.525 INFO:teuthology.orchestra.run.smithi050.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 382 kB/s | 26 kB 00:00 2024-05-27T09:19:07.533 INFO:teuthology.orchestra.run.smithi044.stdout:(10/159): ceph-radosgw-19.0.0-3868.ga6507388.el 2.0 MB/s | 10 MB 00:04 2024-05-27T09:19:07.559 INFO:teuthology.orchestra.run.smithi050.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-05-27T09:19:07.576 INFO:teuthology.orchestra.run.smithi050.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-05-27T09:19:07.618 INFO:teuthology.orchestra.run.smithi050.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 592 kB/s | 35 kB 00:00 2024-05-27T09:19:07.699 INFO:teuthology.orchestra.run.smithi189.stdout:(31/159): rbd-mirror-19.0.0-3868.ga6507388.el9. 3.2 MB/s | 3.0 MB 00:00 2024-05-27T09:19:07.700 INFO:teuthology.orchestra.run.smithi044.stdout:(11/159): cephfs-mirror-19.0.0-3868.ga6507388.e 1.3 MB/s | 224 kB 00:00 2024-05-27T09:19:07.702 INFO:teuthology.orchestra.run.smithi050.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-05-27T09:19:07.761 INFO:teuthology.orchestra.run.smithi050.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-05-27T09:19:07.809 INFO:teuthology.orchestra.run.smithi044.stdout:(12/159): cephfs-shell-19.0.0-3868.ga6507388.el 252 kB/s | 27 kB 00:00 2024-05-27T09:19:07.819 INFO:teuthology.orchestra.run.smithi050.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-05-27T09:19:07.849 INFO:teuthology.orchestra.run.smithi189.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3868.ga6 1.8 MB/s | 274 kB 00:00 2024-05-27T09:19:07.878 INFO:teuthology.orchestra.run.smithi050.stdout:(85/159): python3-prettytable-0.7.2-27.el9.noar 728 kB/s | 42 kB 00:00 2024-05-27T09:19:07.909 INFO:teuthology.orchestra.run.smithi044.stdout:(13/159): libcephfs-devel-19.0.0-3868.ga6507388 319 kB/s | 32 kB 00:00 2024-05-27T09:19:07.941 INFO:teuthology.orchestra.run.smithi189.stdout:(33/159): ceph-mgr-rook-19.0.0-3868.ga6507388.e 539 kB/s | 49 kB 00:00 2024-05-27T09:19:07.970 INFO:teuthology.orchestra.run.smithi050.stdout:(86/159): python3-protobuf-3.14.0-13.el9.noarch 2.9 MB/s | 269 kB 00:00 2024-05-27T09:19:08.028 INFO:teuthology.orchestra.run.smithi050.stdout:(87/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-05-27T09:19:08.033 INFO:teuthology.orchestra.run.smithi189.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3868.ga 179 kB/s | 16 kB 00:00 2024-05-27T09:19:08.112 INFO:teuthology.orchestra.run.smithi050.stdout:(88/159): python3-pyasn1-modules-0.4.8-6.el9.no 3.3 MB/s | 279 kB 00:00 2024-05-27T09:19:08.212 INFO:teuthology.orchestra.run.smithi050.stdout:(89/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.0 MB/s | 5.1 MB 00:00 2024-05-27T09:19:08.216 INFO:teuthology.orchestra.run.smithi189.stdout:(35/159): ceph-volume-19.0.0-3868.ga6507388.el9 1.5 MB/s | 273 kB 00:00 2024-05-27T09:19:08.238 INFO:teuthology.orchestra.run.smithi050.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 407 kB/s | 51 kB 00:00 2024-05-27T09:19:08.263 INFO:teuthology.orchestra.run.smithi050.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-27T09:19:08.300 INFO:teuthology.orchestra.run.smithi189.stdout:(36/159): ceph-mgr-diskprediction-local-19.0.0- 8.8 MB/s | 7.4 MB 00:00 2024-05-27T09:19:08.318 INFO:teuthology.orchestra.run.smithi044.stdout:(14/159): libcephfs2-19.0.0-3868.ga6507388.el9. 1.8 MB/s | 756 kB 00:00 2024-05-27T09:19:08.346 INFO:teuthology.orchestra.run.smithi050.stdout:(92/159): python3-babel-2.9.1-2.el9.noarch.rpm 5.3 MB/s | 6.0 MB 00:01 2024-05-27T09:19:08.372 INFO:teuthology.orchestra.run.smithi050.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-05-27T09:19:08.430 INFO:teuthology.orchestra.run.smithi050.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-05-27T09:19:08.456 INFO:teuthology.orchestra.run.smithi050.stdout:(95/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 406 kB/s | 44 kB 00:00 2024-05-27T09:19:08.475 INFO:teuthology.orchestra.run.smithi189.stdout:(37/159): ceph-osd-19.0.0-3868.ga6507388.el9.x8 2.3 MB/s | 18 MB 00:07 2024-05-27T09:19:08.489 INFO:teuthology.orchestra.run.smithi050.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 614 kB/s | 36 kB 00:00 2024-05-27T09:19:08.492 INFO:teuthology.orchestra.run.smithi189.stdout:(38/159): cephfs-top-19.0.0-3868.ga6507388.el9. 95 kB/s | 18 kB 00:00 2024-05-27T09:19:08.515 INFO:teuthology.orchestra.run.smithi050.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 810 kB/s | 47 kB 00:00 2024-05-27T09:19:08.526 INFO:teuthology.orchestra.run.smithi189.stdout:(39/159): cephadm-19.0.0-3868.ga6507388.el9.noa 2.4 MB/s | 764 kB 00:00 2024-05-27T09:19:08.548 INFO:teuthology.orchestra.run.smithi050.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-05-27T09:19:08.574 INFO:teuthology.orchestra.run.smithi050.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 984 kB/s | 58 kB 00:00 2024-05-27T09:19:08.607 INFO:teuthology.orchestra.run.smithi050.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-05-27T09:19:08.633 INFO:teuthology.orchestra.run.smithi050.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-27T09:19:08.658 INFO:teuthology.orchestra.run.smithi050.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 897 kB/s | 45 kB 00:00 2024-05-27T09:19:08.742 INFO:teuthology.orchestra.run.smithi050.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-05-27T09:19:08.777 INFO:teuthology.orchestra.run.smithi044.stdout:(15/159): ceph-test-19.0.0-3868.ga6507388.el9.x 10 MB/s | 43 MB 00:04 2024-05-27T09:19:08.810 INFO:teuthology.orchestra.run.smithi044.stdout:(16/159): libcephsqlite-19.0.0-3868.ga6507388.e 320 kB/s | 157 kB 00:00 2024-05-27T09:19:08.844 INFO:teuthology.orchestra.run.smithi044.stdout:(17/159): librados-devel-19.0.0-3868.ga6507388. 1.9 MB/s | 127 kB 00:00 2024-05-27T09:19:08.867 INFO:teuthology.orchestra.run.smithi050.stdout:(104/159): protobuf-compiler-3.14.0-13.el9.x86_ 4.0 MB/s | 863 kB 00:00 2024-05-27T09:19:08.926 INFO:teuthology.orchestra.run.smithi189.stdout:(40/159): fuse-2.9.9-15.el9.x86_64.rpm 178 kB/s | 80 kB 00:00 2024-05-27T09:19:09.002 INFO:teuthology.orchestra.run.smithi044.stdout:(18/159): libradosstriper1-19.0.0-3868.ga650738 2.5 MB/s | 497 kB 00:00 2024-05-27T09:19:09.035 INFO:teuthology.orchestra.run.smithi189.stdout:(41/159): libquadmath-11.4.1-3.el9.x86_64.rpm 384 kB/s | 195 kB 00:00 2024-05-27T09:19:09.095 INFO:teuthology.orchestra.run.smithi044.stdout:(19/159): python3-ceph-argparse-19.0.0-3868.ga6 496 kB/s | 45 kB 00:00 2024-05-27T09:19:09.186 INFO:teuthology.orchestra.run.smithi044.stdout:(20/159): python3-ceph-common-19.0.0-3868.ga650 1.5 MB/s | 136 kB 00:00 2024-05-27T09:19:09.252 INFO:teuthology.orchestra.run.smithi189.stdout:(42/159): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-05-27T09:19:09.277 INFO:teuthology.orchestra.run.smithi189.stdout:(43/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 723 kB/s | 253 kB 00:00 2024-05-27T09:19:09.295 INFO:teuthology.orchestra.run.smithi044.stdout:(21/159): librgw2-19.0.0-3868.ga6507388.el9.x86 11 MB/s | 5.2 MB 00:00 2024-05-27T09:19:09.317 INFO:teuthology.orchestra.run.smithi050.stdout:(105/159): abseil-cpp-20211102.0-3.el9.x86_64.r 953 kB/s | 548 kB 00:00 2024-05-27T09:19:09.329 INFO:teuthology.orchestra.run.smithi044.stdout:(22/159): python3-cephfs-19.0.0-3868.ga6507388. 1.1 MB/s | 161 kB 00:00 2024-05-27T09:19:09.351 INFO:teuthology.orchestra.run.smithi050.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 638 kB/s | 309 kB 00:00 2024-05-27T09:19:09.360 INFO:teuthology.orchestra.run.smithi189.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 982 kB/s | 106 kB 00:00 2024-05-27T09:19:09.371 INFO:teuthology.orchestra.run.smithi044.stdout:(23/159): python3-rados-19.0.0-3868.ga6507388.e 4.2 MB/s | 321 kB 00:00 2024-05-27T09:19:09.386 INFO:teuthology.orchestra.run.smithi189.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-05-27T09:19:09.436 INFO:teuthology.orchestra.run.smithi189.stdout:(46/159): python3-cryptography-36.0.1-4.el9.x86 3.1 MB/s | 1.2 MB 00:00 2024-05-27T09:19:09.438 INFO:teuthology.orchestra.run.smithi044.stdout:(24/159): python3-rbd-19.0.0-3868.ga6507388.el9 2.7 MB/s | 301 kB 00:00 2024-05-27T09:19:09.462 INFO:teuthology.orchestra.run.smithi189.stdout:(47/159): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-05-27T09:19:09.463 INFO:teuthology.orchestra.run.smithi044.stdout:(25/159): python3-rgw-19.0.0-3868.ga6507388.el9 1.1 MB/s | 100 kB 00:00 2024-05-27T09:19:09.495 INFO:teuthology.orchestra.run.smithi189.stdout:(48/159): python3-urllib3-1.26.5-5.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-05-27T09:19:09.497 INFO:teuthology.orchestra.run.smithi044.stdout:(26/159): rbd-fuse-19.0.0-3868.ga6507388.el9.x8 1.5 MB/s | 86 kB 00:00 2024-05-27T09:19:09.568 INFO:teuthology.orchestra.run.smithi050.stdout:(107/159): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-05-27T09:19:09.595 INFO:teuthology.orchestra.run.smithi189.stdout:(49/159): automake-1.16.2-8.el9.noarch.rpm 4.2 MB/s | 689 kB 00:00 2024-05-27T09:19:09.621 INFO:teuthology.orchestra.run.smithi189.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 854 kB/s | 107 kB 00:00 2024-05-27T09:19:09.646 INFO:teuthology.orchestra.run.smithi189.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 431 kB/s | 22 kB 00:00 2024-05-27T09:19:09.680 INFO:teuthology.orchestra.run.smithi189.stdout:(52/159): bison-3.7.4-5.el9.x86_64.rpm 4.2 MB/s | 943 kB 00:00 2024-05-27T09:19:09.705 INFO:teuthology.orchestra.run.smithi189.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-05-27T09:19:09.730 INFO:teuthology.orchestra.run.smithi189.stdout:(54/159): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-05-27T09:19:09.755 INFO:teuthology.orchestra.run.smithi044.stdout:(27/159): ceph-common-19.0.0-3868.ga6507388.el9 1.8 MB/s | 20 MB 00:10 2024-05-27T09:19:09.756 INFO:teuthology.orchestra.run.smithi189.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-05-27T09:19:09.781 INFO:teuthology.orchestra.run.smithi189.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-05-27T09:19:09.789 INFO:teuthology.orchestra.run.smithi044.stdout:(28/159): rbd-nbd-19.0.0-3868.ga6507388.el9.x86 584 kB/s | 170 kB 00:00 2024-05-27T09:19:09.806 INFO:teuthology.orchestra.run.smithi189.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-05-27T09:19:09.814 INFO:teuthology.orchestra.run.smithi044.stdout:(29/159): ceph-grafana-dashboards-19.0.0-3868.g 597 kB/s | 34 kB 00:00 2024-05-27T09:19:09.831 INFO:teuthology.orchestra.run.smithi189.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 901 kB/s | 45 kB 00:00 2024-05-27T09:19:09.865 INFO:teuthology.orchestra.run.smithi189.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-05-27T09:19:09.899 INFO:teuthology.orchestra.run.smithi189.stdout:(60/159): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-05-27T09:19:09.914 INFO:teuthology.orchestra.run.smithi044.stdout:(30/159): rbd-mirror-19.0.0-3868.ga6507388.el9. 6.8 MB/s | 3.0 MB 00:00 2024-05-27T09:19:09.924 INFO:teuthology.orchestra.run.smithi189.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 434 kB/s | 25 kB 00:00 2024-05-27T09:19:09.935 INFO:teuthology.orchestra.run.smithi050.stdout:(108/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.6 MB/s | 16 MB 00:01 2024-05-27T09:19:09.939 INFO:teuthology.orchestra.run.smithi044.stdout:(31/159): ceph-mgr-cephadm-19.0.0-3868.ga650738 955 kB/s | 143 kB 00:00 2024-05-27T09:19:09.949 INFO:teuthology.orchestra.run.smithi189.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-05-27T09:19:09.960 INFO:teuthology.orchestra.run.smithi050.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 66 kB/s | 26 kB 00:00 2024-05-27T09:19:09.974 INFO:teuthology.orchestra.run.smithi189.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-05-27T09:19:10.000 INFO:teuthology.orchestra.run.smithi189.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-05-27T09:19:10.025 INFO:teuthology.orchestra.run.smithi189.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 702 kB/s | 35 kB 00:00 2024-05-27T09:19:10.074 INFO:teuthology.orchestra.run.smithi044.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3868.ga6 2.0 MB/s | 274 kB 00:00 2024-05-27T09:19:10.075 INFO:teuthology.orchestra.run.smithi189.stdout:(66/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.6 MB/s | 3.0 MB 00:00 2024-05-27T09:19:10.151 INFO:teuthology.orchestra.run.smithi189.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-05-27T09:19:10.157 INFO:teuthology.orchestra.run.smithi044.stdout:(33/159): ceph-mgr-rook-19.0.0-3868.ga6507388.e 602 kB/s | 49 kB 00:00 2024-05-27T09:19:10.218 INFO:teuthology.orchestra.run.smithi189.stdout:(68/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-05-27T09:19:10.241 INFO:teuthology.orchestra.run.smithi044.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3868.ga 200 kB/s | 16 kB 00:00 2024-05-27T09:19:10.269 INFO:teuthology.orchestra.run.smithi050.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 146 kB/s | 49 kB 00:00 2024-05-27T09:19:10.276 INFO:teuthology.orchestra.run.smithi189.stdout:(69/159): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-05-27T09:19:10.294 INFO:teuthology.orchestra.run.smithi050.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 202 kB/s | 67 kB 00:00 2024-05-27T09:19:10.301 INFO:teuthology.orchestra.run.smithi189.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-05-27T09:19:10.335 INFO:teuthology.orchestra.run.smithi189.stdout:(71/159): postgresql-contrib-13.14-1.el9.x86_64 4.8 MB/s | 911 kB 00:00 2024-05-27T09:19:10.377 INFO:teuthology.orchestra.run.smithi189.stdout:(72/159): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-05-27T09:19:10.402 INFO:teuthology.orchestra.run.smithi189.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-05-27T09:19:10.453 INFO:teuthology.orchestra.run.smithi189.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-05-27T09:19:10.491 INFO:teuthology.orchestra.run.smithi044.stdout:(35/159): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-05-27T09:19:10.525 INFO:teuthology.orchestra.run.smithi044.stdout:(36/159): ceph-volume-19.0.0-3868.ga6507388.el9 965 kB/s | 273 kB 00:00 2024-05-27T09:19:10.537 INFO:teuthology.orchestra.run.smithi189.stdout:(75/159): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-05-27T09:19:10.553 INFO:teuthology.orchestra.run.smithi189.stdout:(76/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-05-27T09:19:10.592 INFO:teuthology.orchestra.run.smithi044.stdout:(37/159): cephadm-19.0.0-3868.ga6507388.el9.noa 7.5 MB/s | 764 kB 00:00 2024-05-27T09:19:10.617 INFO:teuthology.orchestra.run.smithi044.stdout:(38/159): cephfs-top-19.0.0-3868.ga6507388.el9. 200 kB/s | 18 kB 00:00 2024-05-27T09:19:10.678 INFO:teuthology.orchestra.run.smithi050.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 370 kB/s | 151 kB 00:00 2024-05-27T09:19:10.679 INFO:teuthology.orchestra.run.smithi189.stdout:(77/159): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 6.0 MB 00:00 2024-05-27T09:19:10.704 INFO:teuthology.orchestra.run.smithi189.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 170 kB/s | 26 kB 00:00 2024-05-27T09:19:10.730 INFO:teuthology.orchestra.run.smithi189.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-05-27T09:19:10.734 INFO:teuthology.orchestra.run.smithi044.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 571 kB/s | 80 kB 00:00 2024-05-27T09:19:10.753 INFO:teuthology.orchestra.run.smithi050.stdout:(113/159): libarrow-9.0.0-11.el9.x86_64.rpm 3.1 MB/s | 4.4 MB 00:01 2024-05-27T09:19:10.755 INFO:teuthology.orchestra.run.smithi189.stdout:(80/159): python3-jinja2-2.11.3-6.el9.noarch.rp 1.1 MB/s | 248 kB 00:00 2024-05-27T09:19:10.797 INFO:teuthology.orchestra.run.smithi189.stdout:(81/159): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-05-27T09:19:10.822 INFO:teuthology.orchestra.run.smithi189.stdout:(82/159): python3-markupsafe-1.1.1-12.el9.x86_6 378 kB/s | 35 kB 00:00 2024-05-27T09:19:10.826 INFO:teuthology.orchestra.run.smithi044.stdout:(40/159): ceph-mgr-dashboard-19.0.0-3868.ga6507 3.5 MB/s | 3.5 MB 00:01 2024-05-27T09:19:10.851 INFO:teuthology.orchestra.run.smithi044.stdout:(41/159): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-05-27T09:19:10.864 INFO:teuthology.orchestra.run.smithi189.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 5.3 MB/s | 222 kB 00:00 2024-05-27T09:19:10.889 INFO:teuthology.orchestra.run.smithi189.stdout:(84/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-05-27T09:19:10.901 INFO:teuthology.orchestra.run.smithi044.stdout:(42/159): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-05-27T09:19:10.914 INFO:teuthology.orchestra.run.smithi189.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-27T09:19:10.936 INFO:teuthology.orchestra.run.smithi050.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.3 MB/s | 839 kB 00:00 2024-05-27T09:19:10.939 INFO:teuthology.orchestra.run.smithi189.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 842 kB/s | 42 kB 00:00 2024-05-27T09:19:10.951 INFO:teuthology.orchestra.run.smithi044.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-27T09:19:10.965 INFO:teuthology.orchestra.run.smithi189.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-05-27T09:19:10.990 INFO:teuthology.orchestra.run.smithi189.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-05-27T09:19:11.010 INFO:teuthology.orchestra.run.smithi044.stdout:(44/159): python3-cryptography-36.0.1-4.el9.x86 7.8 MB/s | 1.2 MB 00:00 2024-05-27T09:19:11.015 INFO:teuthology.orchestra.run.smithi189.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-05-27T09:19:11.041 INFO:teuthology.orchestra.run.smithi189.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-27T09:19:11.045 INFO:teuthology.orchestra.run.smithi044.stdout:(45/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-05-27T09:19:11.062 INFO:teuthology.orchestra.run.smithi050.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 141 kB/s | 43 kB 00:00 2024-05-27T09:19:11.069 INFO:teuthology.orchestra.run.smithi044.stdout:(46/159): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-05-27T09:19:11.103 INFO:teuthology.orchestra.run.smithi044.stdout:(47/159): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-05-27T09:19:11.108 INFO:teuthology.orchestra.run.smithi189.stdout:(91/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-05-27T09:19:11.133 INFO:teuthology.orchestra.run.smithi189.stdout:(92/159): python3-requests-oauthlib-1.3.0-12.el 457 kB/s | 54 kB 00:00 2024-05-27T09:19:11.137 INFO:teuthology.orchestra.run.smithi044.stdout:(48/159): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-05-27T09:19:11.158 INFO:teuthology.orchestra.run.smithi189.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 828 kB/s | 42 kB 00:00 2024-05-27T09:19:11.170 INFO:teuthology.orchestra.run.smithi050.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 138 kB/s | 32 kB 00:00 2024-05-27T09:19:11.184 INFO:teuthology.orchestra.run.smithi189.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 883 kB/s | 44 kB 00:00 2024-05-27T09:19:11.209 INFO:teuthology.orchestra.run.smithi189.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-27T09:19:11.213 INFO:teuthology.orchestra.run.smithi050.stdout:(117/159): python3-asyncssh-2.13.2-5.el9.noarch 1.0 MB/s | 548 kB 00:00 2024-05-27T09:19:11.234 INFO:teuthology.orchestra.run.smithi189.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-05-27T09:19:11.260 INFO:teuthology.orchestra.run.smithi189.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-05-27T09:19:11.262 INFO:teuthology.orchestra.run.smithi044.stdout:(49/159): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 689 kB 00:00 2024-05-27T09:19:11.285 INFO:teuthology.orchestra.run.smithi189.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-05-27T09:19:11.287 INFO:teuthology.orchestra.run.smithi044.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 715 kB/s | 107 kB 00:00 2024-05-27T09:19:11.310 INFO:teuthology.orchestra.run.smithi189.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-27T09:19:11.312 INFO:teuthology.orchestra.run.smithi050.stdout:(118/159): python3-certifi-2021.10.8-2.el9.noar 60 kB/s | 15 kB 00:00 2024-05-27T09:19:11.321 INFO:teuthology.orchestra.run.smithi044.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 376 kB/s | 22 kB 00:00 2024-05-27T09:19:11.335 INFO:teuthology.orchestra.run.smithi189.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-05-27T09:19:11.361 INFO:teuthology.orchestra.run.smithi189.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-27T09:19:11.371 INFO:teuthology.orchestra.run.smithi044.stdout:(52/159): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 943 kB 00:00 2024-05-27T09:19:11.386 INFO:teuthology.orchestra.run.smithi189.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 892 kB/s | 45 kB 00:00 2024-05-27T09:19:11.396 INFO:teuthology.orchestra.run.smithi044.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 438 kB/s | 33 kB 00:00 2024-05-27T09:19:11.430 INFO:teuthology.orchestra.run.smithi044.stdout:(54/159): flex-2.6.4-9.el9.x86_64.rpm 2.2 MB/s | 313 kB 00:00 2024-05-27T09:19:11.456 INFO:teuthology.orchestra.run.smithi044.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 315 kB/s | 18 kB 00:00 2024-05-27T09:19:11.481 INFO:teuthology.orchestra.run.smithi044.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 226 kB/s | 11 kB 00:00 2024-05-27T09:19:11.506 INFO:teuthology.orchestra.run.smithi044.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 160 kB 00:00 2024-05-27T09:19:11.531 INFO:teuthology.orchestra.run.smithi044.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 913 kB/s | 45 kB 00:00 2024-05-27T09:19:11.590 INFO:teuthology.orchestra.run.smithi044.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 662 kB 00:00 2024-05-27T09:19:11.632 INFO:teuthology.orchestra.run.smithi044.stdout:(60/159): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 582 kB 00:00 2024-05-27T09:19:11.645 INFO:teuthology.orchestra.run.smithi189.stdout:(103/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-05-27T09:19:11.646 INFO:teuthology.orchestra.run.smithi050.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 361 kB/s | 172 kB 00:00 2024-05-27T09:19:11.657 INFO:teuthology.orchestra.run.smithi044.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 383 kB/s | 25 kB 00:00 2024-05-27T09:19:11.670 INFO:teuthology.orchestra.run.smithi189.stdout:(104/159): lua-devel-5.4.4-4.el9.x86_64.rpm 72 kB/s | 22 kB 00:00 2024-05-27T09:19:11.683 INFO:teuthology.orchestra.run.smithi044.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 957 kB/s | 47 kB 00:00 2024-05-27T09:19:11.716 INFO:teuthology.orchestra.run.smithi044.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-05-27T09:19:11.750 INFO:teuthology.orchestra.run.smithi044.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-05-27T09:19:11.775 INFO:teuthology.orchestra.run.smithi044.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 610 kB/s | 35 kB 00:00 2024-05-27T09:19:11.780 INFO:teuthology.orchestra.run.smithi050.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 632 kB/s | 358 kB 00:00 2024-05-27T09:19:11.814 INFO:teuthology.orchestra.run.smithi050.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 605 kB/s | 303 kB 00:00 2024-05-27T09:19:11.859 INFO:teuthology.orchestra.run.smithi044.stdout:(66/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.1 MB/s | 3.0 MB 00:00 2024-05-27T09:19:12.084 INFO:teuthology.orchestra.run.smithi044.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 7.1 MB/s | 1.6 MB 00:00 2024-05-27T09:19:12.106 INFO:teuthology.orchestra.run.smithi050.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 448 kB/s | 206 kB 00:00 2024-05-27T09:19:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-05-27T09:19:12.201 INFO:teuthology.orchestra.run.smithi044.stdout:(68/159): openblas-openmp-0.3.21-2.el9.x86_64.r 11 MB/s | 4.7 MB 00:00 2024-05-27T09:19:12.206 INFO:teuthology.orchestra.run.smithi050.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 369 kB/s | 144 kB 00:00 2024-05-27T09:19:12.301 INFO:teuthology.orchestra.run.smithi044.stdout:(69/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 8.4 MB/s | 4.4 MB 00:00 2024-05-27T09:19:12.304 INFO:teuthology.orchestra.run.smithi189.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 487 kB/s | 309 kB 00:00 2024-05-27T09:19:12.306 INFO:teuthology.orchestra.run.smithi050.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 53 kB/s | 11 kB 00:00 2024-05-27T09:19:12.335 INFO:teuthology.orchestra.run.smithi044.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-05-27T09:19:12.346 INFO:teuthology.orchestra.run.smithi189.stdout:(107/159): abseil-cpp-20211102.0-3.el9.x86_64.r 782 kB/s | 548 kB 00:00 2024-05-27T09:19:12.394 INFO:teuthology.orchestra.run.smithi044.stdout:(71/159): postgresql-contrib-13.14-1.el9.x86_64 2.9 MB/s | 911 kB 00:00 2024-05-27T09:19:12.446 INFO:teuthology.orchestra.run.smithi189.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-05-27T09:19:12.452 INFO:teuthology.orchestra.run.smithi044.stdout:(72/159): protobuf-3.14.0-13.el9.x86_64.rpm 8.7 MB/s | 1.0 MB 00:00 2024-05-27T09:19:12.477 INFO:teuthology.orchestra.run.smithi044.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-05-27T09:19:12.481 INFO:teuthology.orchestra.run.smithi050.stdout:(125/159): python3-jaraco-classes-3.2.1-5.el9.n 64 kB/s | 18 kB 00:00 2024-05-27T09:19:12.488 INFO:teuthology.orchestra.run.smithi189.stdout:(109/159): grpc-data-1.46.7-10.el9.noarch.rpm 55 kB/s | 19 kB 00:00 2024-05-27T09:19:12.528 INFO:teuthology.orchestra.run.smithi044.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-05-27T09:19:12.538 INFO:teuthology.orchestra.run.smithi189.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 529 kB/s | 49 kB 00:00 2024-05-27T09:19:12.548 INFO:teuthology.orchestra.run.smithi050.stdout:(126/159): python3-jaraco-collections-3.0.0-8.e 96 kB/s | 23 kB 00:00 2024-05-27T09:19:12.587 INFO:teuthology.orchestra.run.smithi044.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 4.2 MB/s | 244 kB 00:00 2024-05-27T09:19:12.645 INFO:teuthology.orchestra.run.smithi044.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-05-27T09:19:12.655 INFO:teuthology.orchestra.run.smithi189.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-05-27T09:19:12.689 INFO:teuthology.orchestra.run.smithi189.stdout:(112/159): libunwind-1.6.2-1.el9.x86_64.rpm 336 kB/s | 67 kB 00:00 2024-05-27T09:19:12.724 INFO:teuthology.orchestra.run.smithi050.stdout:(127/159): python3-jaraco-functools-3.5.0-2.el9 80 kB/s | 19 kB 00:00 2024-05-27T09:19:12.754 INFO:teuthology.orchestra.run.smithi044.stdout:(77/159): postgresql-server-13.14-1.el9.x86_64. 13 MB/s | 5.8 MB 00:00 2024-05-27T09:19:12.756 INFO:teuthology.orchestra.run.smithi189.stdout:(113/159): libarrow-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-05-27T09:19:12.779 INFO:teuthology.orchestra.run.smithi044.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 193 kB/s | 26 kB 00:00 2024-05-27T09:19:12.799 INFO:teuthology.orchestra.run.smithi050.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 79 kB/s | 20 kB 00:00 2024-05-27T09:19:12.813 INFO:teuthology.orchestra.run.smithi044.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 332 kB/s | 19 kB 00:00 2024-05-27T09:19:12.840 INFO:teuthology.orchestra.run.smithi189.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.5 MB/s | 839 kB 00:00 2024-05-27T09:19:12.846 INFO:teuthology.orchestra.run.smithi044.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-05-27T09:19:12.865 INFO:teuthology.orchestra.run.smithi189.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 400 kB/s | 43 kB 00:00 2024-05-27T09:19:12.872 INFO:teuthology.orchestra.run.smithi044.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 602 kB/s | 35 kB 00:00 2024-05-27T09:19:12.930 INFO:teuthology.orchestra.run.smithi044.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-05-27T09:19:12.940 INFO:teuthology.orchestra.run.smithi189.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-05-27T09:19:12.966 INFO:teuthology.orchestra.run.smithi189.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-05-27T09:19:13.008 INFO:teuthology.orchestra.run.smithi189.stdout:(118/159): python3-asyncssh-2.13.2-5.el9.noarch 1.7 MB/s | 548 kB 00:00 2024-05-27T09:19:13.041 INFO:teuthology.orchestra.run.smithi189.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-05-27T09:19:13.047 INFO:teuthology.orchestra.run.smithi044.stdout:(83/159): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-05-27T09:19:13.066 INFO:teuthology.orchestra.run.smithi189.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-05-27T09:19:13.081 INFO:teuthology.orchestra.run.smithi044.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 222 kB 00:00 2024-05-27T09:19:13.106 INFO:teuthology.orchestra.run.smithi044.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-05-27T09:19:13.131 INFO:teuthology.orchestra.run.smithi044.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 856 kB/s | 42 kB 00:00 2024-05-27T09:19:13.133 INFO:teuthology.orchestra.run.smithi189.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-05-27T09:19:13.149 INFO:teuthology.orchestra.run.smithi050.stdout:(129/159): python3-grpcio-1.46.7-10.el9.x86_64. 1.5 MB/s | 2.0 MB 00:01 2024-05-27T09:19:13.158 INFO:teuthology.orchestra.run.smithi189.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 1.7 MB/s | 206 kB 00:00 2024-05-27T09:19:13.173 INFO:teuthology.orchestra.run.smithi044.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 269 kB 00:00 2024-05-27T09:19:13.174 INFO:teuthology.orchestra.run.smithi050.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 123 kB/s | 46 kB 00:00 2024-05-27T09:19:13.207 INFO:teuthology.orchestra.run.smithi044.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-05-27T09:19:13.209 INFO:teuthology.orchestra.run.smithi189.stdout:(123/159): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-05-27T09:19:13.234 INFO:teuthology.orchestra.run.smithi189.stdout:(124/159): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-05-27T09:19:13.240 INFO:teuthology.orchestra.run.smithi044.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-05-27T09:19:13.259 INFO:teuthology.orchestra.run.smithi189.stdout:(125/159): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-05-27T09:19:13.283 INFO:teuthology.orchestra.run.smithi044.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 686 kB/s | 51 kB 00:00 2024-05-27T09:19:13.308 INFO:teuthology.orchestra.run.smithi044.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 812 kB/s | 54 kB 00:00 2024-05-27T09:19:13.309 INFO:teuthology.orchestra.run.smithi189.stdout:(126/159): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-05-27T09:19:13.335 INFO:teuthology.orchestra.run.smithi189.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-05-27T09:19:13.360 INFO:teuthology.orchestra.run.smithi189.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-05-27T09:19:13.383 INFO:teuthology.orchestra.run.smithi044.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 560 kB/s | 42 kB 00:00 2024-05-27T09:19:13.410 INFO:teuthology.orchestra.run.smithi189.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 195 kB/s | 20 kB 00:00 2024-05-27T09:19:13.466 INFO:teuthology.orchestra.run.smithi050.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 249 kB/s | 79 kB 00:00 2024-05-27T09:19:13.469 INFO:teuthology.orchestra.run.smithi189.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 427 kB/s | 46 kB 00:00 2024-05-27T09:19:13.492 INFO:teuthology.orchestra.run.smithi050.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 182 kB/s | 58 kB 00:00 2024-05-27T09:19:13.500 INFO:teuthology.orchestra.run.smithi044.stdout:(93/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.8 MB/s | 5.1 MB 00:00 2024-05-27T09:19:13.502 INFO:teuthology.orchestra.run.smithi189.stdout:(131/159): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-05-27T09:19:13.526 INFO:teuthology.orchestra.run.smithi044.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 313 kB/s | 44 kB 00:00 2024-05-27T09:19:13.528 INFO:teuthology.orchestra.run.smithi189.stdout:(132/159): python3-more-itertools-8.12.0-2.el9. 671 kB/s | 79 kB 00:00 2024-05-27T09:19:13.559 INFO:teuthology.orchestra.run.smithi044.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 3.8 MB/s | 221 kB 00:00 2024-05-27T09:19:13.579 INFO:teuthology.orchestra.run.smithi189.stdout:(133/159): python3-natsort-7.1.1-5.el9.noarch.r 529 kB/s | 58 kB 00:00 2024-05-27T09:19:13.601 INFO:teuthology.orchestra.run.smithi044.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 482 kB/s | 36 kB 00:00 2024-05-27T09:19:13.612 INFO:teuthology.orchestra.run.smithi189.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-05-27T09:19:13.625 INFO:teuthology.orchestra.run.smithi050.stdout:(133/159): python3-kubernetes-26.1.0-2.el9.noar 1.1 MB/s | 1.0 MB 00:00 2024-05-27T09:19:13.626 INFO:teuthology.orchestra.run.smithi044.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 712 kB/s | 47 kB 00:00 2024-05-27T09:19:13.654 INFO:teuthology.orchestra.run.smithi189.stdout:(135/159): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-05-27T09:19:13.660 INFO:teuthology.orchestra.run.smithi044.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 5.2 MB/s | 305 kB 00:00 2024-05-27T09:19:13.677 INFO:teuthology.orchestra.run.smithi044.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-27T09:19:13.679 INFO:teuthology.orchestra.run.smithi189.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 896 kB/s | 90 kB 00:00 2024-05-27T09:19:13.710 INFO:teuthology.orchestra.run.smithi044.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.9 MB/s | 94 kB 00:00 2024-05-27T09:19:13.713 INFO:teuthology.orchestra.run.smithi189.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-05-27T09:19:13.727 INFO:teuthology.orchestra.run.smithi044.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-05-27T09:19:13.734 INFO:teuthology.orchestra.run.smithi050.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 68 kB/s | 16 kB 00:00 2024-05-27T09:19:13.755 INFO:teuthology.orchestra.run.smithi189.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-05-27T09:19:13.778 INFO:teuthology.orchestra.run.smithi044.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 672 kB/s | 45 kB 00:00 2024-05-27T09:19:13.789 INFO:teuthology.orchestra.run.smithi189.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-05-27T09:19:13.814 INFO:teuthology.orchestra.run.smithi189.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-05-27T09:19:13.856 INFO:teuthology.orchestra.run.smithi189.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-05-27T09:19:13.889 INFO:teuthology.orchestra.run.smithi189.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-05-27T09:19:13.923 INFO:teuthology.orchestra.run.smithi189.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-05-27T09:19:13.926 INFO:teuthology.orchestra.run.smithi050.stdout:(135/159): python3-pecan-1.4.2-3.el9.noarch.rpm 593 kB/s | 272 kB 00:00 2024-05-27T09:19:13.951 INFO:teuthology.orchestra.run.smithi050.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 276 kB/s | 90 kB 00:00 2024-05-27T09:19:13.957 INFO:teuthology.orchestra.run.smithi189.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 892 kB/s | 90 kB 00:00 2024-05-27T09:19:13.985 INFO:teuthology.orchestra.run.smithi050.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 112 kB/s | 28 kB 00:00 2024-05-27T09:19:14.032 INFO:teuthology.orchestra.run.smithi189.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-05-27T09:19:14.049 INFO:teuthology.orchestra.run.smithi189.stdout:(146/159): python3-zc-lockfile-2.0-10.el9.noarc 159 kB/s | 20 kB 00:00 2024-05-27T09:19:14.070 INFO:teuthology.orchestra.run.smithi044.stdout:(103/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-05-27T09:19:14.074 INFO:teuthology.orchestra.run.smithi189.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-05-27T09:19:14.177 INFO:teuthology.orchestra.run.smithi050.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 123 kB/s | 31 kB 00:00 2024-05-27T09:19:14.266 INFO:teuthology.orchestra.run.smithi189.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-05-27T09:19:14.319 INFO:teuthology.orchestra.run.smithi050.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 177 kB/s | 59 kB 00:00 2024-05-27T09:19:14.428 INFO:teuthology.orchestra.run.smithi050.stdout:(140/159): python3-routes-2.5.1-5.el9.noarch.rp 396 kB/s | 188 kB 00:00 2024-05-27T09:19:14.453 INFO:teuthology.orchestra.run.smithi050.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 130 kB/s | 36 kB 00:00 2024-05-27T09:19:14.470 INFO:teuthology.orchestra.run.smithi044.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-05-27T09:19:14.496 INFO:teuthology.orchestra.run.smithi044.stdout:(105/159): lua-devel-5.4.4-4.el9.x86_64.rpm 29 kB/s | 22 kB 00:00 2024-05-27T09:19:14.546 INFO:teuthology.orchestra.run.smithi044.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 4.2 MB/s | 309 kB 00:00 2024-05-27T09:19:14.638 INFO:teuthology.orchestra.run.smithi044.stdout:(107/159): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-05-27T09:19:14.645 INFO:teuthology.orchestra.run.smithi050.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 157 kB/s | 51 kB 00:00 2024-05-27T09:19:14.721 INFO:teuthology.orchestra.run.smithi044.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 314 kB/s | 26 kB 00:00 2024-05-27T09:19:14.749 INFO:teuthology.orchestra.run.smithi189.stdout:(149/159): librbd1-19.0.0-3868.ga6507388.el9.x8 4.6 MB/s | 3.1 MB 00:00 2024-05-27T09:19:14.808 INFO:teuthology.orchestra.run.smithi189.stdout:(150/159): librados2-19.0.0-3868.ga6507388.el9. 4.4 MB/s | 3.3 MB 00:00 2024-05-27T09:19:14.813 INFO:teuthology.orchestra.run.smithi044.stdout:(109/159): libarrow-9.0.0-11.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-05-27T09:19:14.839 INFO:teuthology.orchestra.run.smithi044.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 418 kB/s | 49 kB 00:00 2024-05-27T09:19:14.845 INFO:teuthology.orchestra.run.smithi050.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 229 kB/s | 90 kB 00:00 2024-05-27T09:19:14.872 INFO:teuthology.orchestra.run.smithi044.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-05-27T09:19:14.887 INFO:teuthology.orchestra.run.smithi050.stdout:(144/159): python3-webob-1.8.7-6.el9.noarch.rpm 500 kB/s | 230 kB 00:00 2024-05-27T09:19:14.956 INFO:teuthology.orchestra.run.smithi044.stdout:(112/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.9 MB/s | 839 kB 00:00 2024-05-27T09:19:14.981 INFO:teuthology.orchestra.run.smithi044.stdout:(113/159): luarocks-3.9.2-5.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-05-27T09:19:15.040 INFO:teuthology.orchestra.run.smithi044.stdout:(114/159): python3-asyncssh-2.13.2-5.el9.noarch 6.5 MB/s | 548 kB 00:00 2024-05-27T09:19:15.073 INFO:teuthology.orchestra.run.smithi044.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 476 kB/s | 43 kB 00:00 2024-05-27T09:19:15.099 INFO:teuthology.orchestra.run.smithi044.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 558 kB/s | 32 kB 00:00 2024-05-27T09:19:15.104 INFO:teuthology.orchestra.run.smithi050.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 77 kB/s | 20 kB 00:00 2024-05-27T09:19:15.141 INFO:teuthology.orchestra.run.smithi044.stdout:(117/159): protobuf-compiler-3.14.0-13.el9.x86_ 634 kB/s | 863 kB 00:01 2024-05-27T09:19:15.150 INFO:teuthology.orchestra.run.smithi189.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 121 kB/s | 41 kB 00:00 2024-05-27T09:19:15.166 INFO:teuthology.orchestra.run.smithi044.stdout:(118/159): python3-certifi-2021.10.8-2.el9.noar 164 kB/s | 15 kB 00:00 2024-05-27T09:19:15.199 INFO:teuthology.orchestra.run.smithi044.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-05-27T09:19:15.200 INFO:teuthology.orchestra.run.smithi189.stdout:(152/159): NetworkManager-1.47.90-1.el9.x86_64. 2.5 MB/s | 2.3 MB 00:00 2024-05-27T09:19:15.258 INFO:teuthology.orchestra.run.smithi044.stdout:(120/159): python3-google-auth-2.29.0-1.el9.noa 3.5 MB/s | 206 kB 00:00 2024-05-27T09:19:15.287 INFO:teuthology.orchestra.run.smithi050.stdout:(146/159): python3-werkzeug-2.0.3-3.el9.1.noarc 665 kB/s | 427 kB 00:00 2024-05-27T09:19:15.292 INFO:teuthology.orchestra.run.smithi044.stdout:(121/159): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-05-27T09:19:15.309 INFO:teuthology.orchestra.run.smithi189.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 339 kB/s | 37 kB 00:00 2024-05-27T09:19:15.313 INFO:teuthology.orchestra.run.smithi050.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 450 kB/s | 192 kB 00:00 2024-05-27T09:19:15.417 INFO:teuthology.orchestra.run.smithi044.stdout:(122/159): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-05-27T09:19:15.418 INFO:teuthology.orchestra.run.smithi189.stdout:(154/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.9 MB/s | 207 kB 00:00 2024-05-27T09:19:15.451 INFO:teuthology.orchestra.run.smithi044.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 916 kB/s | 144 kB 00:00 2024-05-27T09:19:15.468 INFO:teuthology.orchestra.run.smithi044.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 217 kB/s | 11 kB 00:00 2024-05-27T09:19:15.493 INFO:teuthology.orchestra.run.smithi189.stdout:(155/159): NetworkManager-tui-1.47.90-1.el9.x86 724 kB/s | 248 kB 00:00 2024-05-27T09:19:15.526 INFO:teuthology.orchestra.run.smithi189.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 2.3 MB/s | 260 kB 00:00 2024-05-27T09:19:15.601 INFO:teuthology.orchestra.run.smithi044.stdout:(125/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 699 kB/s | 303 kB 00:00 2024-05-27T09:19:15.652 INFO:teuthology.orchestra.run.smithi189.stdout:(157/159): NetworkManager-libnm-1.47.90-1.el9.x 2.0 MB/s | 1.8 MB 00:00 2024-05-27T09:19:15.719 INFO:teuthology.orchestra.run.smithi044.stdout:(126/159): python3-jaraco-classes-3.2.1-5.el9.n 66 kB/s | 18 kB 00:00 2024-05-27T09:19:15.730 INFO:teuthology.orchestra.run.smithi050.stdout:(148/159): librados2-19.0.0-3868.ga6507388.el9. 7.5 MB/s | 3.3 MB 00:00 2024-05-27T09:19:15.743 INFO:teuthology.orchestra.run.smithi044.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 84 kB/s | 23 kB 00:00 2024-05-27T09:19:15.769 INFO:teuthology.orchestra.run.smithi044.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 116 kB/s | 19 kB 00:00 2024-05-27T09:19:15.802 INFO:teuthology.orchestra.run.smithi044.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 342 kB/s | 20 kB 00:00 2024-05-27T09:19:15.828 INFO:teuthology.orchestra.run.smithi044.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 798 kB/s | 46 kB 00:00 2024-05-27T09:19:15.870 INFO:teuthology.orchestra.run.smithi044.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-05-27T09:19:15.895 INFO:teuthology.orchestra.run.smithi044.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 867 kB/s | 58 kB 00:00 2024-05-27T09:19:15.902 INFO:teuthology.orchestra.run.smithi189.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 3.3 MB/s | 1.3 MB 00:00 2024-05-27T09:19:15.913 INFO:teuthology.orchestra.run.smithi050.stdout:(149/159): librbd1-19.0.0-3868.ga6507388.el9.x8 5.2 MB/s | 3.1 MB 00:00 2024-05-27T09:19:15.945 INFO:teuthology.orchestra.run.smithi044.stdout:(133/159): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-05-27T09:19:15.971 INFO:teuthology.orchestra.run.smithi044.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 220 kB/s | 16 kB 00:00 2024-05-27T09:19:16.113 INFO:teuthology.orchestra.run.smithi044.stdout:(135/159): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-05-27T09:19:16.138 INFO:teuthology.orchestra.run.smithi044.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 170 kB/s | 28 kB 00:00 2024-05-27T09:19:16.163 INFO:teuthology.orchestra.run.smithi044.stdout:(137/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 415 kB/s | 90 kB 00:00 2024-05-27T09:19:16.188 INFO:teuthology.orchestra.run.smithi044.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 412 kB/s | 31 kB 00:00 2024-05-27T09:19:16.222 INFO:teuthology.orchestra.run.smithi044.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-05-27T09:19:16.247 INFO:teuthology.orchestra.run.smithi044.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 617 kB/s | 36 kB 00:00 2024-05-27T09:19:16.269 INFO:teuthology.orchestra.run.smithi189.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 3.2 MB/s | 2.4 MB 00:00 2024-05-27T09:19:16.272 INFO:teuthology.orchestra.run.smithi044.stdout:(141/159): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-05-27T09:19:16.275 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-05-27T09:19:16.275 INFO:teuthology.orchestra.run.smithi189.stdout:Total 12 MB/s | 219 MB 00:18 2024-05-27T09:19:16.298 INFO:teuthology.orchestra.run.smithi044.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 683 kB/s | 51 kB 00:00 2024-05-27T09:19:16.313 INFO:teuthology.orchestra.run.smithi050.stdout:(150/159): thrift-0.15.0-2.el9.x86_64.rpm 1.3 MB/s | 1.6 MB 00:01 2024-05-27T09:19:16.331 INFO:teuthology.orchestra.run.smithi044.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-05-27T09:19:16.357 INFO:teuthology.orchestra.run.smithi044.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-05-27T09:19:16.390 INFO:teuthology.orchestra.run.smithi044.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 346 kB/s | 20 kB 00:00 2024-05-27T09:19:16.424 INFO:teuthology.orchestra.run.smithi044.stdout:(146/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-05-27T09:19:16.474 INFO:teuthology.orchestra.run.smithi044.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-05-27T09:19:16.550 INFO:teuthology.orchestra.run.smithi044.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-05-27T09:19:16.834 INFO:teuthology.orchestra.run.smithi044.stdout:(149/159): NetworkManager-1.47.90-1.el9.x86_64. 8.2 MB/s | 2.3 MB 00:00 2024-05-27T09:19:16.939 INFO:teuthology.orchestra.run.smithi050.stdout:(151/159): NetworkManager-1.47.90-1.el9.x86_64. 1.9 MB/s | 2.3 MB 00:01 2024-05-27T09:19:16.950 INFO:teuthology.orchestra.run.smithi044.stdout:(150/159): librados2-19.0.0-3868.ga6507388.el9. 6.3 MB/s | 3.3 MB 00:00 2024-05-27T09:19:16.973 INFO:teuthology.orchestra.run.smithi050.stdout:(152/159): NetworkManager-team-1.47.90-1.el9.x8 63 kB/s | 41 kB 00:00 2024-05-27T09:19:17.017 INFO:teuthology.orchestra.run.smithi044.stdout:(151/159): NetworkManager-libnm-1.47.90-1.el9.x 9.9 MB/s | 1.8 MB 00:00 2024-05-27T09:19:17.076 INFO:teuthology.orchestra.run.smithi044.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 4.2 MB/s | 248 kB 00:00 2024-05-27T09:19:17.081 INFO:teuthology.orchestra.run.smithi050.stdout:(153/159): NetworkManager-libnm-1.47.90-1.el9.x 1.5 MB/s | 1.8 MB 00:01 2024-05-27T09:19:17.109 INFO:teuthology.orchestra.run.smithi044.stdout:(153/159): NetworkManager-team-1.47.90-1.el9.x8 263 kB/s | 41 kB 00:00 2024-05-27T09:19:17.134 INFO:teuthology.orchestra.run.smithi044.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 636 kB/s | 37 kB 00:00 2024-05-27T09:19:17.193 INFO:teuthology.orchestra.run.smithi044.stdout:(155/159): elfutils-libs-0.191-4.el9.x86_64.rpm 4.4 MB/s | 260 kB 00:00 2024-05-27T09:19:17.227 INFO:teuthology.orchestra.run.smithi044.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.7 MB/s | 207 kB 00:00 2024-05-27T09:19:17.294 INFO:teuthology.orchestra.run.smithi044.stdout:(157/159): openssl-3.2.1-1.el9.x86_64.rpm 14 MB/s | 1.3 MB 00:00 2024-05-27T09:19:17.402 INFO:teuthology.orchestra.run.smithi044.stdout:(158/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-05-27T09:19:17.512 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-05-27T09:19:17.568 INFO:teuthology.orchestra.run.smithi044.stdout:(159/159): librbd1-19.0.0-3868.ga6507388.el9.x8 2.8 MB/s | 3.1 MB 00:01 2024-05-27T09:19:17.574 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-05-27T09:19:17.575 INFO:teuthology.orchestra.run.smithi044.stdout:Total 11 MB/s | 219 MB 00:19 2024-05-27T09:19:17.665 INFO:teuthology.orchestra.run.smithi050.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 53 kB/s | 37 kB 00:00 2024-05-27T09:19:17.668 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-05-27T09:19:17.668 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-05-27T09:19:17.790 INFO:teuthology.orchestra.run.smithi050.stdout:(155/159): NetworkManager-tui-1.47.90-1.el9.x86 291 kB/s | 248 kB 00:00 2024-05-27T09:19:17.915 INFO:teuthology.orchestra.run.smithi050.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 248 kB/s | 207 kB 00:00 2024-05-27T09:19:18.557 INFO:teuthology.orchestra.run.smithi050.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 291 kB/s | 260 kB 00:00 2024-05-27T09:19:18.815 INFO:teuthology.orchestra.run.smithi050.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 1.3 MB/s | 1.3 MB 00:01 2024-05-27T09:19:19.041 INFO:teuthology.orchestra.run.smithi050.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.2 MB/s | 2.4 MB 00:01 2024-05-27T09:19:19.045 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-27T09:19:19.045 INFO:teuthology.orchestra.run.smithi050.stdout:Total 9.0 MB/s | 219 MB 00:24 2024-05-27T09:19:19.525 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-05-27T09:19:19.525 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-05-27T09:19:20.244 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-27T09:19:20.418 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-27T09:19:20.419 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-27T09:19:20.896 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-05-27T09:19:21.168 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-05-27T09:19:21.168 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-05-27T09:19:22.294 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-27T09:19:22.294 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-27T09:19:23.915 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-05-27T09:19:23.916 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-05-27T09:19:25.840 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-05-27T09:19:25.956 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T09:19:26.156 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T09:19:26.219 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T09:19:26.258 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T09:19:26.283 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T09:19:26.327 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T09:19:26.715 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T09:19:26.792 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:19.0.0-3868.ga6507388.el9.x86_64 8/170 2024-05-27T09:19:26.916 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:19.0.0-3868.ga6507388.el9.x86_64 8/170 2024-05-27T09:19:26.951 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 9/170 2024-05-27T09:19:27.042 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 9/170 2024-05-27T09:19:27.072 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:19.0.0-3868.ga6507388.el9.x86_64 10/170 2024-05-27T09:19:27.105 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T09:19:27.286 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T09:19:27.337 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T09:19:27.365 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 14/170 2024-05-27T09:19:27.452 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 14/170 2024-05-27T09:19:27.475 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86 15/170 2024-05-27T09:19:27.537 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86 15/170 2024-05-27T09:19:27.559 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T09:19:27.577 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T09:19:27.785 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T09:19:27.915 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T09:19:28.065 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T09:19:28.125 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T09:19:28.235 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T09:19:28.327 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T09:19:28.532 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T09:19:28.616 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-27T09:19:28.653 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T09:19:28.681 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T09:19:28.716 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T09:19:28.737 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T09:19:28.790 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T09:19:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T09:19:28.832 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T09:19:28.884 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T09:19:28.889 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T09:19:28.916 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T09:19:28.930 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T09:19:28.944 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T09:19:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T09:19:29.000 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T09:19:29.014 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T09:19:29.067 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T09:19:29.097 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T09:19:29.142 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T09:19:29.218 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T09:19:29.252 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:19.0.0-3868.ga6507388.el9. 33/170 2024-05-27T09:19:29.287 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:19.0.0-3868.ga6507388.el 34/170 2024-05-27T09:19:29.316 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:19.0.0-3868.ga6507388.el9.x86_6 35/170 2024-05-27T09:19:29.351 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T09:19:29.387 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T09:19:29.431 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T09:19:29.493 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:19.0.0-3868.ga6507388.el9.x86_64 8/170 2024-05-27T09:19:29.505 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T09:19:29.641 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T09:19:29.691 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T09:19:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:19.0.0-3868.ga6507388.el9.x86_64 8/170 2024-05-27T09:19:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 9/170 2024-05-27T09:19:29.908 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 9/170 2024-05-27T09:19:29.938 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:19.0.0-3868.ga6507388.el9.x86_64 10/170 2024-05-27T09:19:29.972 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T09:19:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T09:19:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T09:19:30.231 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T09:19:30.257 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 14/170 2024-05-27T09:19:30.315 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T09:19:30.343 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T09:19:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 14/170 2024-05-27T09:19:30.379 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T09:19:30.391 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86 15/170 2024-05-27T09:19:30.420 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T09:19:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86 15/170 2024-05-27T09:19:30.458 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:19.0.0-3868.ga6507388.el9.x86_6 46/170 2024-05-27T09:19:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T09:19:30.500 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T09:19:30.500 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T09:19:30.648 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T09:19:30.699 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-05-27T09:19:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T09:19:30.711 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T09:19:30.804 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T09:19:30.809 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T09:19:30.829 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T09:19:30.868 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T09:19:30.894 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T09:19:30.940 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T09:19:30.971 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T09:19:30.990 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T09:19:31.007 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T09:19:31.008 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T09:19:31.042 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T09:19:31.055 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T09:19:31.064 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T09:19:31.091 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T09:19:31.103 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T09:19:31.134 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T09:19:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T09:19:31.176 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T09:19:31.212 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T09:19:31.226 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T09:19:31.314 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T09:19:31.361 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T09:19:31.411 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T09:19:31.434 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T09:19:31.506 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T09:19:31.536 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T09:19:31.576 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T09:19:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T09:19:31.602 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T09:19:31.626 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T09:19:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T09:19:31.660 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T09:19:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T09:19:31.702 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T09:19:31.751 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T09:19:31.768 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librados2-2:19.0.0-3868.ga6507388.el9.x86_64 8/170 2024-05-27T09:19:31.791 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T09:19:31.829 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T09:19:31.835 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T09:19:31.868 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T09:19:31.868 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T09:19:31.896 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T09:19:31.896 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T09:19:31.930 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T09:19:31.946 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:19.0.0-3868.ga6507388.el9.x86_64 8/170 2024-05-27T09:19:31.968 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T09:19:31.969 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T09:19:31.977 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 9/170 2024-05-27T09:19:32.001 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T09:19:32.020 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T09:19:32.030 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T09:19:32.050 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T09:19:32.065 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T09:19:32.094 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T09:19:32.096 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 9/170 2024-05-27T09:19:32.098 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T09:19:32.135 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rados-2:19.0.0-3868.ga6507388.el9.x86_64 10/170 2024-05-27T09:19:32.142 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T09:19:32.174 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T09:19:32.181 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T09:19:32.214 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:19.0.0-3868.ga6507388.el9. 33/170 2024-05-27T09:19:32.230 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T09:19:32.258 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:19.0.0-3868.ga6507388.el 34/170 2024-05-27T09:19:32.261 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T09:19:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:19.0.0-3868.ga6507388.el9.x86_6 35/170 2024-05-27T09:19:32.313 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T09:19:32.345 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T09:19:32.360 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T09:19:32.392 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T09:19:32.402 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T09:19:32.491 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T09:19:32.505 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T09:19:32.539 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T09:19:32.623 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T09:19:32.661 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T09:19:32.714 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T09:19:32.800 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T09:19:32.824 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 14/170 2024-05-27T09:19:33.000 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 14/170 2024-05-27T09:19:33.025 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86 15/170 2024-05-27T09:19:33.120 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86 15/170 2024-05-27T09:19:33.152 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T09:19:33.231 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T09:19:33.277 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T09:19:33.280 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T09:19:33.327 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T09:19:33.361 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T09:19:33.378 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T09:19:33.412 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T09:19:33.449 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T09:19:33.463 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T09:19:33.498 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T09:19:33.516 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T09:19:33.528 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:19.0.0-3868.ga6507388.el9.x86_6 46/170 2024-05-27T09:19:33.541 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T09:19:33.561 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T09:19:33.562 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T09:19:33.650 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T09:19:33.680 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T09:19:33.726 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T09:19:33.789 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T09:19:33.860 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T09:19:33.868 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T09:19:33.885 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T09:19:33.886 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T09:19:33.947 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T09:19:33.964 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T09:19:34.021 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T09:19:34.041 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T09:19:34.066 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T09:19:34.103 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T09:19:34.138 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T09:19:34.142 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T09:19:34.177 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T09:19:34.221 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T09:19:34.263 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T09:19:34.313 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T09:19:34.407 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T09:19:34.411 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T09:19:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T09:19:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T09:19:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T09:19:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T09:19:34.625 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T09:19:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T09:19:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T09:19:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T09:19:34.736 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T09:19:34.744 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T09:19:34.792 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T09:19:34.930 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T09:19:34.943 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T09:19:34.971 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T09:19:34.984 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T09:19:34.998 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T09:19:35.029 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T09:19:35.033 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T09:19:35.107 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T09:19:35.133 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T09:19:35.181 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T09:19:35.245 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T09:19:35.279 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T09:19:35.292 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T09:19:35.324 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T09:19:35.444 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T09:19:35.501 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T09:19:35.519 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T09:19:35.527 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T09:19:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T09:19:35.575 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T09:19:35.603 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T09:19:35.632 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T09:19:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T09:19:35.729 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T09:19:35.748 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T09:19:35.776 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T09:19:35.788 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-common-2:19.0.0-3868.ga6507388.el9. 33/170 2024-05-27T09:19:35.806 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T09:19:35.840 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-argparse-2:19.0.0-3868.ga6507388.el 34/170 2024-05-27T09:19:35.854 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T09:19:35.879 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cephfs-2:19.0.0-3868.ga6507388.el9.x86_6 35/170 2024-05-27T09:19:35.967 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T09:19:35.990 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T09:19:36.008 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T09:19:36.026 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T09:19:36.057 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T09:19:36.091 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T09:19:36.119 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T09:19:36.152 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T09:19:36.159 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T09:19:36.337 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T09:19:36.401 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T09:19:36.474 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T09:19:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 103/170 2024-05-27T09:19:36.781 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 103/170 2024-05-27T09:19:36.808 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T09:19:36.811 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:19.0.0-3868.ga6507388.el9.x86_64 104/170 2024-05-27T09:19:36.848 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T09:19:36.855 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T09:19:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T09:19:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T09:19:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T09:19:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T09:19:37.114 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T09:19:37.190 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T09:19:37.218 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T09:19:37.225 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T09:19:37.260 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T09:19:37.301 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T09:19:37.338 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T09:19:37.364 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T09:19:37.368 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 107/170 2024-05-27T09:19:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T09:19:37.419 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados-devel-2:19.0.0-3868.ga6507388.el9.x86_6 46/170 2024-05-27T09:19:37.422 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 107/170 2024-05-27T09:19:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T09:19:37.444 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:19.0.0-3868.ga6507388.el9.x86_64 108/170 2024-05-27T09:19:37.496 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T09:19:37.519 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T09:19:37.721 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T09:19:37.780 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T09:19:37.915 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T09:19:37.996 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T09:19:38.116 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T09:19:38.155 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T09:19:38.187 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T09:19:38.213 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T09:19:38.253 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T09:19:38.294 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T09:19:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T09:19:38.357 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T09:19:38.425 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T09:19:38.635 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T09:19:38.688 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T09:19:38.745 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T09:19:38.882 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T09:19:38.914 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T09:19:38.960 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T09:19:38.995 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T09:19:39.028 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T09:19:39.229 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T09:19:39.415 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T09:19:39.469 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T09:19:39.508 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T09:19:39.559 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T09:19:39.626 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T09:19:39.654 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T09:19:39.702 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T09:19:39.714 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 109/170 2024-05-27T09:19:39.736 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T09:19:39.749 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 109/170 2024-05-27T09:19:39.751 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T09:19:39.773 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T09:19:39.805 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T09:19:39.806 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T09:19:39.856 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T09:19:39.921 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T09:19:39.958 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T09:19:40.105 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T09:19:40.140 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T09:19:40.163 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T09:19:40.185 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 103/170 2024-05-27T09:19:40.207 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T09:19:40.260 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 103/170 2024-05-27T09:19:40.289 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T09:19:40.290 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:19.0.0-3868.ga6507388.el9.x86_64 104/170 2024-05-27T09:19:40.309 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 109/170 2024-05-27T09:19:40.327 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T09:19:40.344 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 110/170 2024-05-27T09:19:40.433 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 110/170 2024-05-27T09:19:40.433 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T09:19:40.433 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T09:19:40.433 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:19:40.454 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T09:19:40.463 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 111/170 2024-05-27T09:19:40.530 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T09:19:40.842 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T09:19:40.871 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 107/170 2024-05-27T09:19:40.927 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 107/170 2024-05-27T09:19:40.957 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:19.0.0-3868.ga6507388.el9.x86_64 108/170 2024-05-27T09:19:41.539 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T09:19:41.596 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T09:19:41.629 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T09:19:41.762 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T09:19:41.813 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T09:19:41.867 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T09:19:41.980 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T09:19:42.031 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T09:19:42.281 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T09:19:42.319 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T09:19:42.452 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T09:19:43.221 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 109/170 2024-05-27T09:19:43.262 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 109/170 2024-05-27T09:19:43.405 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T09:19:43.548 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T09:19:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 109/170 2024-05-27T09:19:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 110/170 2024-05-27T09:19:43.984 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 110/170 2024-05-27T09:19:43.985 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T09:19:43.985 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T09:19:43.985 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:19:44.017 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 111/170 2024-05-27T09:19:45.252 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T09:19:45.352 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T09:19:45.388 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T09:19:45.423 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T09:19:45.822 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T09:19:45.864 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 103/170 2024-05-27T09:19:46.013 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 103/170 2024-05-27T09:19:46.038 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rbd-2:19.0.0-3868.ga6507388.el9.x86_64 104/170 2024-05-27T09:19:46.094 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T09:19:46.698 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T09:19:46.737 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 107/170 2024-05-27T09:19:46.818 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 107/170 2024-05-27T09:19:46.844 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rgw-2:19.0.0-3868.ga6507388.el9.x86_64 108/170 2024-05-27T09:19:49.640 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 109/170 2024-05-27T09:19:49.674 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 109/170 2024-05-27T09:19:50.366 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 109/170 2024-05-27T09:19:50.411 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 110/170 2024-05-27T09:19:50.578 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 110/170 2024-05-27T09:19:50.578 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T09:19:50.578 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T09:19:50.578 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:19:50.620 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 111/170 2024-05-27T09:19:52.079 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 111/170 2024-05-27T09:19:52.080 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-05-27T09:19:52.080 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-05-27T09:19:52.080 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-05-27T09:19:52.080 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-05-27T09:19:52.080 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-05-27T09:19:52.080 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-05-27T09:19:52.080 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-05-27T09:19:52.080 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:19:52.256 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 112/170 2024-05-27T09:19:52.273 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 112/170 2024-05-27T09:19:52.273 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:52.273 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T09:19:52.273 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:19:53.842 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 113/170 2024-05-27T09:19:53.869 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 113/170 2024-05-27T09:19:53.869 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:53.869 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T09:19:53.869 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T09:19:53.869 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T09:19:53.869 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:19:54.101 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 114/170 2024-05-27T09:19:54.124 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 114/170 2024-05-27T09:19:54.124 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:54.125 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T09:19:54.125 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T09:19:54.125 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T09:19:54.125 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:19:54.585 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 115/170 2024-05-27T09:19:54.609 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 115/170 2024-05-27T09:19:54.609 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:54.609 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T09:19:54.609 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T09:19:54.609 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T09:19:54.609 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:19:54.707 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T09:19:54.765 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T09:19:54.820 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T09:19:55.009 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T09:19:55.053 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T09:19:55.132 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T09:19:55.221 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T09:19:55.370 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T09:19:55.428 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T09:19:55.468 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T09:19:55.504 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T09:19:55.956 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:19.0.0-3868.ga6507388.el9.noarch 127/170 2024-05-27T09:19:55.986 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:19.0.0-3868.ga6507388.el9.noarch 127/170 2024-05-27T09:19:56.026 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3868.ga6507388.e 128/170 2024-05-27T09:19:56.088 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3868.ga6507388. 129/170 2024-05-27T09:19:56.125 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noa 130/170 2024-05-27T09:19:56.397 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 111/170 2024-05-27T09:19:56.397 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2024-05-27T09:19:56.397 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2024-05-27T09:19:56.398 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2024-05-27T09:19:56.398 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2024-05-27T09:19:56.398 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2024-05-27T09:19:56.398 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2024-05-27T09:19:56.398 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2024-05-27T09:19:56.398 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:19:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 112/170 2024-05-27T09:19:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 112/170 2024-05-27T09:19:56.591 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:56.592 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T09:19:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:19:57.175 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noa 130/170 2024-05-27T09:19:57.219 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.n 131/170 2024-05-27T09:19:58.190 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.n 131/170 2024-05-27T09:19:58.219 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3868.ga65 132/170 2024-05-27T09:19:58.220 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 113/170 2024-05-27T09:19:58.247 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 113/170 2024-05-27T09:19:58.247 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:58.247 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T09:19:58.247 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T09:19:58.247 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T09:19:58.248 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:19:58.332 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3868.ga65 132/170 2024-05-27T09:19:58.488 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 114/170 2024-05-27T09:19:58.494 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3868.ga6507388.el 133/170 2024-05-27T09:19:58.511 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 114/170 2024-05-27T09:19:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T09:19:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T09:19:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T09:19:58.511 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:19:58.529 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 134/170 2024-05-27T09:19:58.553 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 134/170 2024-05-27T09:19:58.553 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:58.553 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T09:19:58.553 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T09:19:58.554 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T09:19:58.554 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:19:58.597 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 135/170 2024-05-27T09:19:58.609 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 135/170 2024-05-27T09:19:58.706 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:19.0.0-3868.ga6507388.el9.x86_64 136/170 2024-05-27T09:19:58.754 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 137/170 2024-05-27T09:19:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 115/170 2024-05-27T09:19:59.020 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 115/170 2024-05-27T09:19:59.021 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:59.021 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T09:19:59.021 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T09:19:59.021 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T09:19:59.021 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:19:59.145 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T09:19:59.203 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T09:19:59.267 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T09:19:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T09:19:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T09:19:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T09:19:59.695 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T09:19:59.704 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T09:19:59.733 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 139/170 2024-05-27T09:19:59.756 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 139/170 2024-05-27T09:19:59.756 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:59.756 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T09:19:59.756 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T09:19:59.757 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T09:19:59.757 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:19:59.800 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3868.ga6507 140/170 2024-05-27T09:19:59.822 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3868.ga6507 140/170 2024-05-27T09:19:59.823 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:59.823 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T09:19:59.823 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:19:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T09:19:59.867 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 141/170 2024-05-27T09:19:59.890 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T09:19:59.890 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 141/170 2024-05-27T09:19:59.891 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:19:59.891 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T09:19:59.891 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T09:19:59.891 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T09:19:59.891 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:19:59.922 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T09:19:59.949 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T09:20:00.201 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 142/170 2024-05-27T09:20:00.236 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 142/170 2024-05-27T09:20:00.236 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:00.237 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T09:20:00.237 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T09:20:00.237 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T09:20:00.237 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:20:00.371 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.0.0-3868.ga6507388.el9.noarch 127/170 2024-05-27T09:20:00.397 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:19.0.0-3868.ga6507388.el9.noarch 127/170 2024-05-27T09:20:00.430 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3868.ga6507388.e 128/170 2024-05-27T09:20:00.494 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3868.ga6507388. 129/170 2024-05-27T09:20:00.520 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noa 130/170 2024-05-27T09:20:01.626 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noa 130/170 2024-05-27T09:20:01.672 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.n 131/170 2024-05-27T09:20:02.688 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.n 131/170 2024-05-27T09:20:02.714 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3868.ga65 132/170 2024-05-27T09:20:02.833 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3868.ga65 132/170 2024-05-27T09:20:03.005 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3868.ga6507388.el 133/170 2024-05-27T09:20:03.033 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 134/170 2024-05-27T09:20:03.057 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 134/170 2024-05-27T09:20:03.057 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:03.057 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T09:20:03.057 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T09:20:03.057 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T09:20:03.057 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:20:03.108 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 135/170 2024-05-27T09:20:03.119 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 135/170 2024-05-27T09:20:03.230 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:19.0.0-3868.ga6507388.el9.x86_64 136/170 2024-05-27T09:20:03.283 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 137/170 2024-05-27T09:20:04.273 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T09:20:04.311 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 139/170 2024-05-27T09:20:04.335 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 139/170 2024-05-27T09:20:04.335 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:04.335 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T09:20:04.335 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T09:20:04.335 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T09:20:04.335 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:20:04.378 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3868.ga6507 140/170 2024-05-27T09:20:04.401 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3868.ga6507 140/170 2024-05-27T09:20:04.401 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:04.401 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T09:20:04.401 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:20:04.462 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 141/170 2024-05-27T09:20:04.486 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 141/170 2024-05-27T09:20:04.486 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:04.486 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T09:20:04.486 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T09:20:04.486 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T09:20:04.486 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:20:04.504 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:19.0.0-3868.ga6507388.el9.x86_64 143/170 2024-05-27T09:20:04.554 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 144/170 2024-05-27T09:20:04.604 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:19.0.0-3868.ga6507388.el9.x86_64 145/170 2024-05-27T09:20:04.635 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T09:20:04.636 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 111/170 2024-05-27T09:20:04.636 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /sys 2024-05-27T09:20:04.636 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /proc 2024-05-27T09:20:04.636 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /mnt 2024-05-27T09:20:04.636 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /var/tmp 2024-05-27T09:20:04.636 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /home 2024-05-27T09:20:04.636 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /root 2024-05-27T09:20:04.636 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /tmp 2024-05-27T09:20:04.636 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:04.663 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T09:20:04.821 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 142/170 2024-05-27T09:20:04.845 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 142/170 2024-05-27T09:20:04.845 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:04.845 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T09:20:04.845 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T09:20:04.845 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T09:20:04.846 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:20:04.852 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-shell-2:19.0.0-3868.ga6507388.el9.x86_64 148/170 2024-05-27T09:20:04.881 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 112/170 2024-05-27T09:20:04.892 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T09:20:04.924 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:19.0.0-3868.ga6507388.el9.x86_ 150/170 2024-05-27T09:20:04.932 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 112/170 2024-05-27T09:20:04.932 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:04.933 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T09:20:04.933 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:04.965 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T09:20:04.994 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T09:20:05.041 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T09:20:05.652 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T09:20:05.697 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T09:20:05.717 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T09:20:06.559 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-top-2:19.0.0-3868.ga6507388.el9.noarch 155/170 2024-05-27T09:20:06.651 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T09:20:06.692 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T09:20:06.741 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 113/170 2024-05-27T09:20:06.774 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T09:20:06.804 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T09:20:06.832 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 113/170 2024-05-27T09:20:06.832 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:06.833 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T09:20:06.833 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T09:20:06.833 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T09:20:06.833 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:06.837 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T09:20:06.871 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T09:20:06.897 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T09:20:06.897 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T09:20:06.973 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T09:20:06.998 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T09:20:06.998 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T09:20:07.082 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T09:20:07.107 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T09:20:07.108 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 114/170 2024-05-27T09:20:07.113 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T09:20:07.113 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T09:20:07.181 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 114/170 2024-05-27T09:20:07.181 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:07.182 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T09:20:07.182 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T09:20:07.182 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T09:20:07.182 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:07.309 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T09:20:07.310 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T09:20:07.402 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T09:20:07.426 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T09:20:07.490 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T09:20:07.725 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 115/170 2024-05-27T09:20:07.798 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 115/170 2024-05-27T09:20:07.798 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:07.798 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T09:20:07.798 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T09:20:07.799 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T09:20:07.799 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:07.943 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T09:20:08.007 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T09:20:08.083 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T09:20:08.331 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T09:20:08.422 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T09:20:08.518 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T09:20:08.629 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T09:20:08.798 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T09:20:08.855 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T09:20:08.895 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T09:20:08.928 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T09:20:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:19.0.0-3868.ga6507388.el9.x86_64 143/170 2024-05-27T09:20:09.400 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 144/170 2024-05-27T09:20:09.441 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:19.0.0-3868.ga6507388.el9.x86_64 145/170 2024-05-27T09:20:09.472 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T09:20:09.500 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T09:20:09.520 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephadm-2:19.0.0-3868.ga6507388.el9.noarch 127/170 2024-05-27T09:20:09.565 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephadm-2:19.0.0-3868.ga6507388.el9.noarch 127/170 2024-05-27T09:20:09.622 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3868.ga6507388.e 128/170 2024-05-27T09:20:09.692 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephfs-shell-2:19.0.0-3868.ga6507388.el9.x86_64 148/170 2024-05-27T09:20:09.735 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3868.ga6507388. 129/170 2024-05-27T09:20:09.746 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T09:20:09.776 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noa 130/170 2024-05-27T09:20:09.778 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:19.0.0-3868.ga6507388.el9.x86_ 150/170 2024-05-27T09:20:09.819 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T09:20:09.848 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T09:20:09.903 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T09:20:10.474 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T09:20:10.509 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T09:20:10.529 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T09:20:11.139 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noa 130/170 2024-05-27T09:20:11.177 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.n 131/170 2024-05-27T09:20:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephfs-top-2:19.0.0-3868.ga6507388.el9.noarch 155/170 2024-05-27T09:20:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T09:20:11.573 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T09:20:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T09:20:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T09:20:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T09:20:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T09:20:11.769 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T09:20:11.769 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T09:20:11.853 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T09:20:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T09:20:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T09:20:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T09:20:11.995 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T09:20:12.000 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T09:20:12.000 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T09:20:12.189 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T09:20:12.189 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T09:20:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T09:20:12.306 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T09:20:12.343 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.n 131/170 2024-05-27T09:20:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T09:20:12.544 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3868.ga65 132/170 2024-05-27T09:20:12.732 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3868.ga65 132/170 2024-05-27T09:20:12.928 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3868.ga6507388.el 133/170 2024-05-27T09:20:12.969 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 134/170 2024-05-27T09:20:13.046 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 134/170 2024-05-27T09:20:13.046 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:13.046 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T09:20:13.046 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T09:20:13.046 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T09:20:13.046 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:13.104 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 135/170 2024-05-27T09:20:13.146 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 135/170 2024-05-27T09:20:13.261 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-2:19.0.0-3868.ga6507388.el9.x86_64 136/170 2024-05-27T09:20:13.302 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T09:20:13.302 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:19.0.0-3868.ga6507388.el9.x86_64 1/170 2024-05-27T09:20:13.302 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 2/170 2024-05-27T09:20:13.302 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 3/170 2024-05-27T09:20:13.302 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 4/170 2024-05-27T09:20:13.302 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3868.ga6507 5/170 2024-05-27T09:20:13.302 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 6/170 2024-05-27T09:20:13.302 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 7/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 8/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 9/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 10/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 11/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:19.0.0-3868.ga6507388.el9.x86_64 12/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 13/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-shell-2:19.0.0-3868.ga6507388.el9.x86_64 14/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:19.0.0-3868.ga6507388.el9.x86_ 15/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 16/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 17/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:19.0.0-3868.ga6507388.el9.x86_6 18/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86 19/170 2024-05-27T09:20:13.303 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 20/170 2024-05-27T09:20:13.304 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3868.ga6507388.el 21/170 2024-05-27T09:20:13.304 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:19.0.0-3868.ga6507388.el9. 22/170 2024-05-27T09:20:13.304 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:19.0.0-3868.ga6507388.el9.x86_6 23/170 2024-05-27T09:20:13.304 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:19.0.0-3868.ga6507388.el9.x86_64 24/170 2024-05-27T09:20:13.304 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:19.0.0-3868.ga6507388.el9.x86_64 25/170 2024-05-27T09:20:13.304 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:19.0.0-3868.ga6507388.el9.x86_64 26/170 2024-05-27T09:20:13.304 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 27/170 2024-05-27T09:20:13.316 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 28/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:19.0.0-3868.ga6507388.el9.x86_64 29/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3868.ga6507388. 30/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noa 31/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.n 32/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3868.ga65 33/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3868.ga6507388.el 34/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 35/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3868.ga6507388.e 36/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 37/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:19.0.0-3868.ga6507388.el9.noarch 38/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-top-2:19.0.0-3868.ga6507388.el9.noarch 39/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T09:20:13.317 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T09:20:13.318 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T09:20:13.318 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T09:20:13.318 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T09:20:13.318 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T09:20:13.318 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T09:20:13.318 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T09:20:13.318 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T09:20:13.318 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T09:20:13.318 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T09:20:13.318 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T09:20:13.318 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T09:20:13.319 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T09:20:13.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T09:20:13.321 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T09:20:13.322 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T09:20:13.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T09:20:13.324 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T09:20:13.325 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:19.0.0-3868.ga6507388.el9.x86_64 149/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 151/170 2024-05-27T09:20:13.326 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T09:20:13.327 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T09:20:13.328 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T09:20:13.328 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T09:20:13.328 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T09:20:13.328 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T09:20:13.337 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 137/170 2024-05-27T09:20:14.415 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T09:20:14.448 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 139/170 2024-05-27T09:20:14.524 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 139/170 2024-05-27T09:20:14.524 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:14.524 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T09:20:14.524 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T09:20:14.524 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T09:20:14.524 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:14.573 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3868.ga6507 140/170 2024-05-27T09:20:14.650 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3868.ga6507 140/170 2024-05-27T09:20:14.650 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:14.651 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T09:20:14.651 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:14.733 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 141/170 2024-05-27T09:20:14.804 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 141/170 2024-05-27T09:20:14.804 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:14.804 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T09:20:14.804 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T09:20:14.804 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T09:20:14.804 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:15.217 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 142/170 2024-05-27T09:20:15.289 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 142/170 2024-05-27T09:20:15.289 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T09:20:15.289 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T09:20:15.289 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T09:20:15.289 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T09:20:15.289 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:15.885 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T09:20:15.886 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:20:15.886 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-05-27T09:20:15.886 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:15.886 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:15.886 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:15.886 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:15.886 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T09:20:15.886 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T09:20:15.886 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T09:20:15.886 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.887 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.888 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T09:20:15.889 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T09:20:15.890 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T09:20:15.890 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T09:20:15.890 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T09:20:15.890 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T09:20:15.890 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.890 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.890 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.890 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T09:20:15.890 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T09:20:15.891 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T09:20:15.891 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T09:20:15.891 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T09:20:15.891 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.891 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.891 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T09:20:15.891 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.891 INFO:teuthology.orchestra.run.smithi189.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T09:20:15.891 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T09:20:15.891 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T09:20:15.892 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T09:20:15.893 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T09:20:15.893 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T09:20:15.893 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T09:20:15.893 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T09:20:15.893 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T09:20:15.893 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T09:20:15.893 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.893 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.893 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.893 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T09:20:15.893 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T09:20:15.894 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T09:20:15.895 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T09:20:15.895 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T09:20:15.895 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T09:20:15.895 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T09:20:15.895 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T09:20:15.895 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T09:20:15.895 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T09:20:15.895 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T09:20:15.895 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T09:20:15.895 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T09:20:15.895 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T09:20:15.896 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T09:20:15.896 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T09:20:15.896 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T09:20:15.896 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T09:20:15.896 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T09:20:15.896 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T09:20:15.896 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T09:20:15.896 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T09:20:15.896 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T09:20:15.896 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T09:20:15.896 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T09:20:15.897 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.898 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.899 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:15.899 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T09:20:15.899 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T09:20:15.899 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T09:20:15.899 INFO:teuthology.orchestra.run.smithi189.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T09:20:15.899 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T09:20:15.899 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T09:20:15.899 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T09:20:15.899 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:20:15.900 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-05-27T09:20:16.336 DEBUG:teuthology.parallel:result is None 2024-05-27T09:20:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T09:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:19.0.0-3868.ga6507388.el9.x86_64 1/170 2024-05-27T09:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 2/170 2024-05-27T09:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 3/170 2024-05-27T09:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 4/170 2024-05-27T09:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3868.ga6507 5/170 2024-05-27T09:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 6/170 2024-05-27T09:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 7/170 2024-05-27T09:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 8/170 2024-05-27T09:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 9/170 2024-05-27T09:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 10/170 2024-05-27T09:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 11/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:19.0.0-3868.ga6507388.el9.x86_64 12/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 13/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephfs-shell-2:19.0.0-3868.ga6507388.el9.x86_64 14/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:19.0.0-3868.ga6507388.el9.x86_ 15/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 16/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 17/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:19.0.0-3868.ga6507388.el9.x86_6 18/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86 19/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 20/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3868.ga6507388.el 21/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:19.0.0-3868.ga6507388.el9. 22/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:19.0.0-3868.ga6507388.el9.x86_6 23/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:19.0.0-3868.ga6507388.el9.x86_64 24/170 2024-05-27T09:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:19.0.0-3868.ga6507388.el9.x86_64 25/170 2024-05-27T09:20:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:19.0.0-3868.ga6507388.el9.x86_64 26/170 2024-05-27T09:20:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 27/170 2024-05-27T09:20:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 28/170 2024-05-27T09:20:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:19.0.0-3868.ga6507388.el9.x86_64 29/170 2024-05-27T09:20:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3868.ga6507388. 30/170 2024-05-27T09:20:18.232 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noa 31/170 2024-05-27T09:20:18.232 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.n 32/170 2024-05-27T09:20:18.232 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3868.ga65 33/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3868.ga6507388.el 34/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 35/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3868.ga6507388.e 36/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 37/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:19.0.0-3868.ga6507388.el9.noarch 38/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephfs-top-2:19.0.0-3868.ga6507388.el9.noarch 39/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T09:20:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T09:20:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T09:20:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T09:20:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T09:20:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T09:20:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T09:20:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T09:20:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T09:20:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:19.0.0-3868.ga6507388.el9.x86_64 149/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 151/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T09:20:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T09:20:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T09:20:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T09:20:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T09:20:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T09:20:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T09:20:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T09:20:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T09:20:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T09:20:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T09:20:20.248 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-test-2:19.0.0-3868.ga6507388.el9.x86_64 143/170 2024-05-27T09:20:20.302 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 144/170 2024-05-27T09:20:20.357 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-nbd-2:19.0.0-3868.ga6507388.el9.x86_64 145/170 2024-05-27T09:20:20.396 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T09:20:20.434 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T09:20:20.742 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-shell-2:19.0.0-3868.ga6507388.el9.x86_64 148/170 2024-05-27T09:20:20.801 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T09:20:20.825 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs-devel-2:19.0.0-3868.ga6507388.el9.x86_ 150/170 2024-05-27T09:20:20.873 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T09:20:20.873 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:20:20.873 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2024-05-27T09:20:20.873 INFO:teuthology.orchestra.run.smithi050.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:20.873 INFO:teuthology.orchestra.run.smithi050.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:20.873 INFO:teuthology.orchestra.run.smithi050.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:20.873 INFO:teuthology.orchestra.run.smithi050.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:20.873 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T09:20:20.874 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T09:20:20.874 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T09:20:20.874 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.874 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.874 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T09:20:20.874 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T09:20:20.874 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-27T09:20:20.874 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T09:20:20.874 INFO:teuthology.orchestra.run.smithi050.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T09:20:20.874 INFO:teuthology.orchestra.run.smithi050.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T09:20:20.874 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-shell-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.876 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-top-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T09:20:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T09:20:20.879 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.880 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T09:20:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T09:20:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T09:20:20.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T09:20:20.884 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T09:20:20.886 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T09:20:20.886 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T09:20:20.886 INFO:teuthology.orchestra.run.smithi050.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T09:20:20.886 INFO:teuthology.orchestra.run.smithi050.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T09:20:20.886 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T09:20:20.886 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T09:20:20.887 INFO:teuthology.orchestra.run.smithi050.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T09:20:20.887 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-27T09:20:20.887 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-27T09:20:20.923 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T09:20:21.002 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T09:20:21.312 DEBUG:teuthology.parallel:result is None 2024-05-27T09:20:21.935 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T09:20:21.980 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T09:20:22.043 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T09:20:23.532 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-top-2:19.0.0-3868.ga6507388.el9.noarch 155/170 2024-05-27T09:20:23.637 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T09:20:23.698 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T09:20:23.844 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T09:20:23.880 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T09:20:23.913 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T09:20:23.947 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T09:20:23.988 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T09:20:23.988 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T09:20:24.142 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T09:20:24.201 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T09:20:24.202 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T09:20:24.322 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T09:20:24.360 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T09:20:24.394 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T09:20:24.395 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T09:20:24.723 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T09:20:24.723 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T09:20:24.869 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T09:20:24.913 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T09:20:25.047 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T09:20:25.339 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T09:20:25.339 INFO:teuthology.orchestra.run.smithi044.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-27T09:20:25.340 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:31.148 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-2:19.0.0-3868.ga6507388.el9.x86_64 1/170 2024-05-27T09:20:31.148 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 2/170 2024-05-27T09:20:31.148 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 3/170 2024-05-27T09:20:31.148 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 4/170 2024-05-27T09:20:31.148 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3868.ga6507 5/170 2024-05-27T09:20:31.149 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 6/170 2024-05-27T09:20:31.149 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 7/170 2024-05-27T09:20:31.149 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 8/170 2024-05-27T09:20:31.149 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 9/170 2024-05-27T09:20:31.149 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 10/170 2024-05-27T09:20:31.149 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 11/170 2024-05-27T09:20:31.149 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-test-2:19.0.0-3868.ga6507388.el9.x86_64 12/170 2024-05-27T09:20:31.149 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 13/170 2024-05-27T09:20:31.149 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-shell-2:19.0.0-3868.ga6507388.el9.x86_64 14/170 2024-05-27T09:20:31.149 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs-devel-2:19.0.0-3868.ga6507388.el9.x86_ 15/170 2024-05-27T09:20:31.149 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 16/170 2024-05-27T09:20:31.150 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 17/170 2024-05-27T09:20:31.150 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados-devel-2:19.0.0-3868.ga6507388.el9.x86_6 18/170 2024-05-27T09:20:31.150 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86 19/170 2024-05-27T09:20:31.150 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 20/170 2024-05-27T09:20:31.150 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3868.ga6507388.el 21/170 2024-05-27T09:20:31.150 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-common-2:19.0.0-3868.ga6507388.el9. 22/170 2024-05-27T09:20:31.150 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cephfs-2:19.0.0-3868.ga6507388.el9.x86_6 23/170 2024-05-27T09:20:31.150 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rados-2:19.0.0-3868.ga6507388.el9.x86_64 24/170 2024-05-27T09:20:31.150 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rbd-2:19.0.0-3868.ga6507388.el9.x86_64 25/170 2024-05-27T09:20:31.150 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rgw-2:19.0.0-3868.ga6507388.el9.x86_64 26/170 2024-05-27T09:20:31.150 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 27/170 2024-05-27T09:20:31.164 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 28/170 2024-05-27T09:20:31.164 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-nbd-2:19.0.0-3868.ga6507388.el9.x86_64 29/170 2024-05-27T09:20:31.164 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3868.ga6507388. 30/170 2024-05-27T09:20:31.164 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noa 31/170 2024-05-27T09:20:31.164 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.n 32/170 2024-05-27T09:20:31.164 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3868.ga65 33/170 2024-05-27T09:20:31.164 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3868.ga6507388.el 34/170 2024-05-27T09:20:31.164 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 35/170 2024-05-27T09:20:31.164 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3868.ga6507388.e 36/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 37/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephadm-2:19.0.0-3868.ga6507388.el9.noarch 38/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-top-2:19.0.0-3868.ga6507388.el9.noarch 39/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T09:20:31.165 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T09:20:31.166 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T09:20:31.167 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T09:20:31.168 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T09:20:31.169 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T09:20:31.170 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T09:20:31.171 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T09:20:31.172 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:19.0.0-3868.ga6507388.el9.x86_64 149/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 151/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T09:20:31.173 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T09:20:31.174 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T09:20:31.175 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T09:20:33.477 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T09:20:33.477 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:33.477 INFO:teuthology.orchestra.run.smithi044.stdout:Upgraded: 2024-05-27T09:20:33.477 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-05-27T09:20:33.478 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.479 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:33.480 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-shell-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-top-2:19.0.0-3868.ga6507388.el9.noarch 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T09:20:33.481 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T09:20:33.482 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T09:20:33.483 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.484 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T09:20:33.485 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T09:20:33.486 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T09:20:33.487 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T09:20:33.488 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-2:19.0.0-3868.ga6507388.el9.x86_64 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T09:20:33.489 INFO:teuthology.orchestra.run.smithi044.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T09:20:33.490 INFO:teuthology.orchestra.run.smithi044.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T09:20:33.490 INFO:teuthology.orchestra.run.smithi044.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T09:20:33.490 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T09:20:33.490 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T09:20:33.490 INFO:teuthology.orchestra.run.smithi044.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T09:20:33.490 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:20:33.490 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-05-27T09:20:34.250 DEBUG:teuthology.parallel:result is None 2024-05-27T09:20:34.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:20:34.395 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T09:20:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:19.0.0-3868.ga6507388.el9 2024-05-27T09:20:34.449 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3868.ga6507388.el9 2024-05-27T09:20:34.450 INFO:teuthology.task.install:The correct ceph version 19.0.0-3868.ga6507388 is installed. 2024-05-27T09:20:34.451 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:20:34.673 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T09:20:34.700 INFO:teuthology.orchestra.run.smithi050.stdout:19.0.0-3868.ga6507388.el9 2024-05-27T09:20:34.700 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3868.ga6507388.el9 2024-05-27T09:20:34.701 INFO:teuthology.task.install:The correct ceph version 19.0.0-3868.ga6507388 is installed. 2024-05-27T09:20:34.702 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:20:34.929 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T09:20:34.955 INFO:teuthology.orchestra.run.smithi189.stdout:19.0.0-3868.ga6507388.el9 2024-05-27T09:20:34.955 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3868.ga6507388.el9 2024-05-27T09:20:34.955 INFO:teuthology.task.install:The correct ceph version 19.0.0-3868.ga6507388 is installed. 2024-05-27T09:20:34.957 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-27T09:20:34.957 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:20:34.957 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T09:20:35.032 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-27T09:20:35.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T09:20:35.064 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-27T09:20:35.064 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T09:20:35.094 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-27T09:20:35.094 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:20:35.095 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T09:20:35.159 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T09:20:35.264 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-27T09:20:35.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T09:20:35.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T09:20:35.362 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-27T09:20:35.362 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T09:20:35.391 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T09:20:35.459 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-27T09:20:35.459 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:20:35.459 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T09:20:35.527 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T09:20:35.632 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-27T09:20:35.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T09:20:35.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T09:20:35.722 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-27T09:20:35.722 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T09:20:35.747 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T09:20:35.812 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-27T09:20:35.812 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:20:35.812 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T09:20:35.873 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T09:20:35.975 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-27T09:20:35.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T09:20:36.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T09:20:36.064 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-27T09:20:36.064 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T09:20:36.088 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T09:20:36.152 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-27T09:20:36.254 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'a6507388dd5057528934822c0163b0c347ef1d5d', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-05-27T09:20:36.254 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:20:36.255 INFO:tasks.cephadm:Cluster fsid is 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:20:36.255 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-27T09:20:36.255 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.44', 'mon.b': '172.21.15.50', 'mon.c': '172.21.15.189'} 2024-05-27T09:20:36.255 INFO:tasks.cephadm:First mon is mon.a on smithi044 2024-05-27T09:20:36.256 INFO:tasks.cephadm:First mgr is x 2024-05-27T09:20:36.256 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-27T09:20:36.256 DEBUG:teuthology.orchestra.run.smithi044:> sudo hostname $(hostname -s) 2024-05-27T09:20:36.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo hostname $(hostname -s) 2024-05-27T09:20:36.346 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-05-27T09:20:36.370 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-27T09:20:36.370 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:20:36.593 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'a6507388dd5057528934822c0163b0c347ef1d5d', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79602/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3868-ga6507388', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3868.ga6507388'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-21 12:50:55.456707', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240521.105740-debug', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'a6507388dd5057528934822c0163b0c347ef1d5d', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79604/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3868-ga6507388', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3868.ga6507388'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-21 12:34:17.608165', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240521.105740-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-05-27T09:20:36.811 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-vshankar-testing-20240521.105740-debug, sha1 a6507388dd5057528934822c0163b0c347ef1d5d from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:20:36.813 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/x86_64/flavors/default/cephadm 2024-05-27T09:20:36.813 INFO:tasks.cephadm:Downloading cephadm from url: https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/x86_64/flavors/default/cephadm 2024-05-27T09:20:36.813 DEBUG:teuthology.orchestra.run.smithi044:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T09:20:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:-rw-r--r--. 1 ubuntu ubuntu 782785 May 27 09:20 /home/ubuntu/cephtest/cephadm 2024-05-27T09:20:37.426 DEBUG:teuthology.orchestra.run.smithi050:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T09:20:37.865 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r--. 1 ubuntu ubuntu 782785 May 27 09:20 /home/ubuntu/cephtest/cephadm 2024-05-27T09:20:37.866 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.105740-debug/a6507388dd5057528934822c0163b0c347ef1d5d/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T09:20:38.318 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 782785 May 27 09:20 /home/ubuntu/cephtest/cephadm 2024-05-27T09:20:38.319 DEBUG:teuthology.orchestra.run.smithi044:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T09:20:38.366 DEBUG:teuthology.orchestra.run.smithi050:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T09:20:38.386 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T09:20:38.421 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d on all hosts... 2024-05-27T09:20:38.422 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d pull 2024-05-27T09:20:38.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d pull 2024-05-27T09:20:38.432 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d pull 2024-05-27T09:20:39.316 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d... 2024-05-27T09:20:39.353 INFO:teuthology.orchestra.run.smithi050.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d... 2024-05-27T09:20:40.006 INFO:teuthology.orchestra.run.smithi044.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d... 2024-05-27T09:21:09.877 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-05-27T09:21:09.878 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev)", 2024-05-27T09:21:09.878 INFO:teuthology.orchestra.run.smithi050.stdout: "image_id": "850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76", 2024-05-27T09:21:09.878 INFO:teuthology.orchestra.run.smithi050.stdout: "repo_digests": [ 2024-05-27T09:21:09.878 INFO:teuthology.orchestra.run.smithi050.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6" 2024-05-27T09:21:09.878 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-05-27T09:21:09.878 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-05-27T09:21:18.848 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-05-27T09:21:18.849 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev)", 2024-05-27T09:21:18.849 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76", 2024-05-27T09:21:18.849 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-05-27T09:21:18.849 INFO:teuthology.orchestra.run.smithi189.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6" 2024-05-27T09:21:18.849 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-05-27T09:21:18.849 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-05-27T09:21:23.614 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:21:23.614 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev)", 2024-05-27T09:21:23.614 INFO:teuthology.orchestra.run.smithi044.stdout: "image_id": "850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76", 2024-05-27T09:21:23.614 INFO:teuthology.orchestra.run.smithi044.stdout: "repo_digests": [ 2024-05-27T09:21:23.614 INFO:teuthology.orchestra.run.smithi044.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6" 2024-05-27T09:21:23.615 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:21:23.615 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:21:23.655 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph 2024-05-27T09:21:23.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph 2024-05-27T09:21:23.766 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-05-27T09:21:23.798 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 777 /etc/ceph 2024-05-27T09:21:23.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /etc/ceph 2024-05-27T09:21:23.912 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-05-27T09:21:23.941 INFO:tasks.cephadm:Writing seed config... 2024-05-27T09:21:23.942 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-27T09:21:23.942 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-27T09:21:23.942 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-27T09:21:23.942 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-27T09:21:23.943 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-27T09:21:23.944 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-27T09:21:23.945 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-05-27T09:21:23.946 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-27T09:21:23.946 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-27T09:21:23.946 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-27T09:21:23.946 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-27T09:21:23.946 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-27T09:21:23.946 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-27T09:21:23.946 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-27T09:21:23.946 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-05-27T09:21:23.947 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:21:23.947 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-27T09:21:24.007 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 604b015a-1c0a-11ef-bc9b-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-27T09:21:24.007 DEBUG:teuthology.orchestra.run.smithi044:mon.a> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.a.service 2024-05-27T09:21:24.055 DEBUG:teuthology.orchestra.run.smithi044:mgr.x> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.x.service 2024-05-27T09:21:24.101 INFO:tasks.cephadm:Bootstrapping... 2024-05-27T09:21:24.101 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d -v bootstrap --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.44 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-27T09:21:24.373 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-05-27T09:21:24.373 INFO:teuthology.orchestra.run.smithi044.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d', '-v', 'bootstrap', '--fsid', '604b015a-1c0a-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.44', '--skip-admin-label'] 2024-05-27T09:21:24.373 INFO:teuthology.orchestra.run.smithi044.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-27T09:21:24.374 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying podman|docker is present... 2024-05-27T09:21:24.441 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 5.0.2 2024-05-27T09:21:24.442 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying lvm2 is present... 2024-05-27T09:21:24.442 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying time synchronization is in place... 2024-05-27T09:21:24.462 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-27T09:21:24.462 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-27T09:21:24.480 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-27T09:21:24.480 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-05-27T09:21:24.499 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout enabled 2024-05-27T09:21:24.517 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout active 2024-05-27T09:21:24.517 INFO:teuthology.orchestra.run.smithi044.stdout:Unit chronyd.service is enabled and running 2024-05-27T09:21:24.517 INFO:teuthology.orchestra.run.smithi044.stdout:Repeating the final host check... 2024-05-27T09:21:24.577 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 5.0.2 2024-05-27T09:21:24.577 INFO:teuthology.orchestra.run.smithi044.stdout:podman (/bin/podman) version 5.0.2 is present 2024-05-27T09:21:24.577 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl is present 2024-05-27T09:21:24.577 INFO:teuthology.orchestra.run.smithi044.stdout:lvcreate is present 2024-05-27T09:21:24.592 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-27T09:21:24.592 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-27T09:21:24.605 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-27T09:21:24.605 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-05-27T09:21:24.621 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout enabled 2024-05-27T09:21:24.635 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout active 2024-05-27T09:21:24.635 INFO:teuthology.orchestra.run.smithi044.stdout:Unit chronyd.service is enabled and running 2024-05-27T09:21:24.635 INFO:teuthology.orchestra.run.smithi044.stdout:Host looks OK 2024-05-27T09:21:24.635 INFO:teuthology.orchestra.run.smithi044.stdout:Cluster fsid: 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:21:24.636 INFO:teuthology.orchestra.run.smithi044.stdout:Acquiring lock 140253469128016 on /run/cephadm/604b015a-1c0a-11ef-bc9b-c7b262605968.lock 2024-05-27T09:21:24.636 INFO:teuthology.orchestra.run.smithi044.stdout:Lock 140253469128016 acquired on /run/cephadm/604b015a-1c0a-11ef-bc9b-c7b262605968.lock 2024-05-27T09:21:24.636 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying IP 172.21.15.44 port 3300 ... 2024-05-27T09:21:24.637 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying IP 172.21.15.44 port 6789 ... 2024-05-27T09:21:24.637 INFO:teuthology.orchestra.run.smithi044.stdout:Base mon IP(s) is [172.21.15.44:3300, 172.21.15.44:6789], mon addrv is [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-05-27T09:21:24.645 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.44 metric 100 2024-05-27T09:21:24.645 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.44 metric 100 2024-05-27T09:21:24.653 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-27T09:21:24.653 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-05-27T09:21:24.653 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-05-27T09:21:24.660 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-27T09:21:24.660 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-27T09:21:24.661 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-27T09:21:24.661 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-05-27T09:21:24.661 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-05-27T09:21:24.661 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-27T09:21:24.661 INFO:teuthology.orchestra.run.smithi044.stdout:Mon IP `172.21.15.44` is in CIDR network `172.21.0.0/20` 2024-05-27T09:21:24.661 INFO:teuthology.orchestra.run.smithi044.stdout:Mon IP `172.21.15.44` is in CIDR network `172.21.0.0/20` 2024-05-27T09:21:24.662 INFO:teuthology.orchestra.run.smithi044.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-27T09:21:24.662 INFO:teuthology.orchestra.run.smithi044.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-27T09:21:24.663 INFO:teuthology.orchestra.run.smithi044.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d... 2024-05-27T09:21:25.317 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 2024-05-27T09:21:25.317 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d... 2024-05-27T09:21:25.317 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Getting image source signatures 2024-05-27T09:21:25.317 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying blob sha256:6ac36e7c57f227110df5cd65cfdca1cc05e0b17df6748637a6aa04c5c80f89c6 2024-05-27T09:21:25.318 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying blob sha256:30c527fbaaeaa4fed4aa057eca2023a2923eff3c212014d5d990bfbe5e3b29dc 2024-05-27T09:21:25.318 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying config sha256:850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 2024-05-27T09:21:25.318 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-27T09:21:26.567 INFO:teuthology.orchestra.run.smithi044.stdout:ceph: stdout ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev) 2024-05-27T09:21:26.567 INFO:teuthology.orchestra.run.smithi044.stdout:Ceph version: ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev) 2024-05-27T09:21:26.567 INFO:teuthology.orchestra.run.smithi044.stdout:Extracting ceph user uid/gid from container image... 2024-05-27T09:21:27.787 INFO:teuthology.orchestra.run.smithi044.stdout:stat: stdout 167 167 2024-05-27T09:21:27.788 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial keys... 2024-05-27T09:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQCYUFRmPh3qExAAOvLwRs/wAVmPkh2KBIWwUg== 2024-05-27T09:21:30.214 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQCZUFRmAKkkJBAAwFYSGkHceJhXSrKl1bcVNg== 2024-05-27T09:21:31.384 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQCaUFRm+FfbKxAA5MlFTuLGfFi3iEePyvHU7A== 2024-05-27T09:21:31.386 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial monmap... 2024-05-27T09:21:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-27T09:21:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-27T09:21:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:21:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-27T09:21:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool for a [v2:172.21.15.44:3300,v1:172.21.15.44:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-27T09:21:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:setting min_mon_release = quincy 2024-05-27T09:21:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: set fsid to 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:21:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-27T09:21:32.566 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:21:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:Creating mon... 2024-05-27T09:21:33.693 INFO:teuthology.orchestra.run.smithi044.stdout:create mon.a on 2024-05-27T09:21:34.091 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-27T09:21:34.342 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-27T09:21:34.651 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-604b015a-1c0a-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-604b015a-1c0a-11ef-bc9b-c7b262605968.target. 2024-05-27T09:21:34.651 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-604b015a-1c0a-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-604b015a-1c0a-11ef-bc9b-c7b262605968.target. 2024-05-27T09:21:35.071 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.a 2024-05-27T09:21:35.071 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to reset failed state of unit ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.a.service: Unit ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.a.service not loaded. 2024-05-27T09:21:35.391 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-604b015a-1c0a-11ef-bc9b-c7b262605968.target.wants/ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.a.service → /etc/systemd/system/ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@.service. 2024-05-27T09:21:36.082 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:35 smithi044 podman[31230]: 2024-05-27 09:21:35.844108879 +0000 UTC m=+0.161757158 container create 79c61d4d327a2a1679a6853da3a0047d1b54414c6a1024e579b5b7a685230e1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521) 2024-05-27T09:21:36.082 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:36 smithi044 podman[31230]: 2024-05-27 09:21:36.080941685 +0000 UTC m=+0.398589992 container init 79c61d4d327a2a1679a6853da3a0047d1b54414c6a1024e579b5b7a685230e1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-27T09:21:36.214 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T09:21:36.215 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-05-27T09:21:36.234 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T09:21:36.235 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-05-27T09:21:36.235 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-05-27T09:21:36.235 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to enable service . firewalld.service is not available 2024-05-27T09:21:36.236 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mon to start... 2024-05-27T09:21:36.236 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mon... 2024-05-27T09:21:36.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:36 smithi044 podman[31230]: 2024-05-27 09:21:36.091881628 +0000 UTC m=+0.409529935 container start 79c61d4d327a2a1679a6853da3a0047d1b54414c6a1024e579b5b7a685230e1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:21:36.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:36 smithi044 bash[31230]: 79c61d4d327a2a1679a6853da3a0047d1b54414c6a1024e579b5b7a685230e1a 2024-05-27T09:21:36.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:36 smithi044 systemd[1]: Started Ceph mon.a for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:21:36.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:36 smithi044 ceph-mon[31248]: mkfs 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:21:36.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:36 smithi044 ceph-mon[31248]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-27T09:21:37.645 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-27T09:21:37.645 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: monmap epoch 1 2024-05-27T09:21:37.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:21:37.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: last_changed 2024-05-27T09:21:31.930626+0000 2024-05-27T09:21:37.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: created 2024-05-27T09:21:31.930626+0000 2024-05-27T09:21:37.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: min_mon_release 19 (squid) 2024-05-27T09:21:37.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: election_strategy: 1 2024-05-27T09:21:37.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-05-27T09:21:37.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: fsmap 2024-05-27T09:21:37.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: osdmap e1: 0 total, 0 up, 0 in 2024-05-27T09:21:37.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: mgrmap e1: no daemons active 2024-05-27T09:21:37.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:37 smithi044 ceph-mon[31248]: from='client.? 172.21.15.44:0/253040899' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-27T09:21:37.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout cluster: 2024-05-27T09:21:37.661 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout id: 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:21:37.661 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-27T09:21:37.661 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:21:37.661 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout services: 2024-05-27T09:21:37.661 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.750585s) 2024-05-27T09:21:37.661 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-27T09:21:37.661 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-27T09:21:37.661 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:21:37.662 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout data: 2024-05-27T09:21:37.662 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-27T09:21:37.662 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-27T09:21:37.662 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-27T09:21:37.662 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout pgs: 2024-05-27T09:21:37.662 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:21:37.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.911+0000 7f0d52800700 1 Processor -- start 2024-05-27T09:21:37.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.912+0000 7f0d52800700 1 -- start start 2024-05-27T09:21:37.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.912+0000 7f0d52800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c10c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:21:37.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.912+0000 7f0d52800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4c10ca50 con 0x7f0d4c10c050 2024-05-27T09:21:37.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.913+0000 7f0d4b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c10c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:21:37.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.913+0000 7f0d4b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c10c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45876/0 (socket says 172.21.15.44:45876) 2024-05-27T09:21:37.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.913+0000 7f0d4b400700 1 -- 172.21.15.44:0/2163659737 learned_addr learned my addr 172.21.15.44:0/2163659737 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:21:37.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.914+0000 7f0d4b400700 1 -- 172.21.15.44:0/2163659737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4c10d180 con 0x7f0d4c10c050 2024-05-27T09:21:37.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.914+0000 7f0d4b400700 1 --2- 172.21.15.44:0/2163659737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c10c460 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0098a0 tx=0x7f0d3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92966881571f605 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:21:37.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.915+0000 7f0d4a000700 1 -- 172.21.15.44:0/2163659737 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c004160 con 0x7f0d4c10c050 2024-05-27T09:21:37.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.915+0000 7f0d4a000700 1 -- 172.21.15.44:0/2163659737 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d3c004300 con 0x7f0d4c10c050 2024-05-27T09:21:37.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.915+0000 7f0d4a000700 1 -- 172.21.15.44:0/2163659737 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c0036a0 con 0x7f0d4c10c050 2024-05-27T09:21:37.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.915+0000 7f0d52800700 1 -- 172.21.15.44:0/2163659737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 msgr2=0x7f0d4c10c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:21:37.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.915+0000 7f0d52800700 1 --2- 172.21.15.44:0/2163659737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c10c460 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0098a0 tx=0x7f0d3c009bb0 comp rx=0 tx=0).stop 2024-05-27T09:21:37.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.916+0000 7f0d52800700 1 -- 172.21.15.44:0/2163659737 shutdown_connections 2024-05-27T09:21:37.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.916+0000 7f0d52800700 1 --2- 172.21.15.44:0/2163659737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c10c460 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:21:37.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.916+0000 7f0d52800700 1 -- 172.21.15.44:0/2163659737 >> 172.21.15.44:0/2163659737 conn(0x7f0d4c07f410 msgr2=0x7f0d4c109d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:21:37.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.916+0000 7f0d52800700 1 -- 172.21.15.44:0/2163659737 shutdown_connections 2024-05-27T09:21:37.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.916+0000 7f0d52800700 1 -- 172.21.15.44:0/2163659737 wait complete. 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.917+0000 7f0d52800700 1 Processor -- start 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.917+0000 7f0d52800700 1 -- start start 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.918+0000 7f0d52800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.918+0000 7f0d52800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4c10d6b0 con 0x7f0d4c10c050 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.919+0000 7f0d4b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.919+0000 7f0d4b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c19fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45892/0 (socket says 172.21.15.44:45892) 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.919+0000 7f0d4b400700 1 -- 172.21.15.44:0/253040899 learned_addr learned my addr 172.21.15.44:0/253040899 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.919+0000 7f0d4b400700 1 -- 172.21.15.44:0/253040899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4c1a0010 con 0x7f0d4c10c050 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.920+0000 7f0d4b400700 1 --2- 172.21.15.44:0/253040899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c19fad0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c000c00 tx=0x7f0d3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.920+0000 7f0d43e00700 1 -- 172.21.15.44:0/253040899 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c003db0 con 0x7f0d4c10c050 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.920+0000 7f0d43e00700 1 -- 172.21.15.44:0/253040899 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d3c011630 con 0x7f0d4c10c050 2024-05-27T09:21:37.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.920+0000 7f0d43e00700 1 -- 172.21.15.44:0/253040899 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c011950 con 0x7f0d4c10c050 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.920+0000 7f0d52800700 1 -- 172.21.15.44:0/253040899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d4c1a02a0 con 0x7f0d4c10c050 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.920+0000 7f0d52800700 1 -- 172.21.15.44:0/253040899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d4c1a2eb0 con 0x7f0d4c10c050 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.921+0000 7f0d43e00700 1 -- 172.21.15.44:0/253040899 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0d3c011af0 con 0x7f0d4c10c050 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.921+0000 7f0d43e00700 1 -- 172.21.15.44:0/253040899 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0d3c01c450 con 0x7f0d4c10c050 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.922+0000 7f0d52800700 1 -- 172.21.15.44:0/253040899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d4c064e10 con 0x7f0d4c10c050 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:36.925+0000 7f0d43e00700 1 -- 172.21.15.44:0/253040899 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f0d3c004260 con 0x7f0d4c10c050 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:37.005+0000 7f0d52800700 1 -- 172.21.15.44:0/253040899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f0d4c02e690 con 0x7f0d4c10c050 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:37.006+0000 7f0d43e00700 1 -- 172.21.15.44:0/253040899 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f0d3c01b450 con 0x7f0d4c10c050 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:37.008+0000 7f0d52800700 1 -- 172.21.15.44:0/253040899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 msgr2=0x7f0d4c19fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:37.008+0000 7f0d52800700 1 --2- 172.21.15.44:0/253040899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c19fad0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c000c00 tx=0x7f0d3c004750 comp rx=0 tx=0).stop 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:37.008+0000 7f0d52800700 1 -- 172.21.15.44:0/253040899 shutdown_connections 2024-05-27T09:21:37.670 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:37.008+0000 7f0d52800700 1 --2- 172.21.15.44:0/253040899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d4c10c050 0x7f0d4c19fad0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:21:37.671 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:37.008+0000 7f0d52800700 1 -- 172.21.15.44:0/253040899 >> 172.21.15.44:0/253040899 conn(0x7f0d4c07f410 msgr2=0x7f0d4c196260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:21:37.671 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:37.008+0000 7f0d52800700 1 -- 172.21.15.44:0/253040899 shutdown_connections 2024-05-27T09:21:37.671 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:37.008+0000 7f0d52800700 1 -- 172.21.15.44:0/253040899 wait complete. 2024-05-27T09:21:37.671 INFO:teuthology.orchestra.run.smithi044.stdout:mon is available 2024-05-27T09:21:37.671 INFO:teuthology.orchestra.run.smithi044.stdout:Assimilating anything we can from ceph.conf... 2024-05-27T09:21:38.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:38 smithi044 ceph-mon[31248]: from='client.? 172.21.15.44:0/2432195447' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-27T09:21:38.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:38 smithi044 ceph-mon[31248]: from='client.? 172.21.15.44:0/2432195447' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-27T09:21:39.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:21:39.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [global] 2024-05-27T09:21:39.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout fsid = 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:21:39.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-27T09:21:39.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-05-27T09:21:39.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-27T09:21:39.057 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-27T09:21:39.057 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-27T09:21:39.057 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-27T09:21:39.057 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:21:39.057 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-27T09:21:39.057 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-27T09:21:39.057 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:21:39.057 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [osd] 2024-05-27T09:21:39.057 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-05-27T09:21:39.057 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-27T09:21:39.057 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-27T09:21:39.058 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-27T09:21:39.058 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-27T09:21:39.058 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-27T09:21:39.058 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.344+0000 7f4ef3200700 1 Processor -- start 2024-05-27T09:21:39.058 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.344+0000 7f4ef3200700 1 -- start start 2024-05-27T09:21:39.058 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.344+0000 7f4ef3200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef407d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:21:39.058 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.344+0000 7f4ef3200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef407d850 con 0x7f4ef407ee50 2024-05-27T09:21:39.058 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.345+0000 7f4ef1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef407d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:21:39.058 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.345+0000 7f4ef1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef407d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45894/0 (socket says 172.21.15.44:45894) 2024-05-27T09:21:39.058 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.345+0000 7f4ef1e00700 1 -- 172.21.15.44:0/696129735 learned_addr learned my addr 172.21.15.44:0/696129735 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.346+0000 7f4ef1e00700 1 -- 172.21.15.44:0/696129735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef407d9d0 con 0x7f4ef407ee50 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.346+0000 7f4ef1e00700 1 --2- 172.21.15.44:0/696129735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef407d280 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40098a0 tx=0x7f4ee4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee2dfa36608b477 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.346+0000 7f4ef0a00700 1 -- 172.21.15.44:0/696129735 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ee4004160 con 0x7f4ef407ee50 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.346+0000 7f4ef0a00700 1 -- 172.21.15.44:0/696129735 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee4004300 con 0x7f4ef407ee50 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.346+0000 7f4ef0a00700 1 -- 172.21.15.44:0/696129735 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ee40036a0 con 0x7f4ef407ee50 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.347+0000 7f4ef3200700 1 -- 172.21.15.44:0/696129735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 msgr2=0x7f4ef407d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.347+0000 7f4ef3200700 1 --2- 172.21.15.44:0/696129735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef407d280 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40098a0 tx=0x7f4ee4009bb0 comp rx=0 tx=0).stop 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.347+0000 7f4ef3200700 1 -- 172.21.15.44:0/696129735 shutdown_connections 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.347+0000 7f4ef3200700 1 --2- 172.21.15.44:0/696129735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef407d280 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.347+0000 7f4ef3200700 1 -- 172.21.15.44:0/696129735 >> 172.21.15.44:0/696129735 conn(0x7f4ef4105220 msgr2=0x7f4ef4107670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.347+0000 7f4ef3200700 1 -- 172.21.15.44:0/696129735 shutdown_connections 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.347+0000 7f4ef3200700 1 -- 172.21.15.44:0/696129735 wait complete. 2024-05-27T09:21:39.059 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.349+0000 7f4ef3200700 1 Processor -- start 2024-05-27T09:21:39.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.349+0000 7f4ef3200700 1 -- start start 2024-05-27T09:21:39.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.349+0000 7f4ef3200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef419f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:21:39.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.349+0000 7f4ef3200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef407df00 con 0x7f4ef407ee50 2024-05-27T09:21:39.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.349+0000 7f4ef1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef419f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:21:39.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.349+0000 7f4ef1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef419f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45904/0 (socket says 172.21.15.44:45904) 2024-05-27T09:21:39.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.349+0000 7f4ef1e00700 1 -- 172.21.15.44:0/2432195447 learned_addr learned my addr 172.21.15.44:0/2432195447 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:21:39.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.350+0000 7f4ef1e00700 1 -- 172.21.15.44:0/2432195447 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef419fdb0 con 0x7f4ef407ee50 2024-05-27T09:21:39.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.350+0000 7f4ef1e00700 1 --2- 172.21.15.44:0/2432195447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef419f870 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4000c00 tx=0x7f4ee4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:21:39.060 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.350+0000 7f4eeaa00700 1 -- 172.21.15.44:0/2432195447 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ee4003db0 con 0x7f4ef407ee50 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.351+0000 7f4eeaa00700 1 -- 172.21.15.44:0/2432195447 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee4011630 con 0x7f4ef407ee50 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.351+0000 7f4eeaa00700 1 -- 172.21.15.44:0/2432195447 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ee4011950 con 0x7f4ef407ee50 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.351+0000 7f4ef3200700 1 -- 172.21.15.44:0/2432195447 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef41a0040 con 0x7f4ef407ee50 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.351+0000 7f4ef3200700 1 -- 172.21.15.44:0/2432195447 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef41a2c50 con 0x7f4ef407ee50 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.351+0000 7f4eeaa00700 1 -- 172.21.15.44:0/2432195447 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4ee4011af0 con 0x7f4ef407ee50 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.351+0000 7f4eeaa00700 1 -- 172.21.15.44:0/2432195447 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4ee401c450 con 0x7f4ef407ee50 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.352+0000 7f4ef3200700 1 -- 172.21.15.44:0/2432195447 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ef4064e10 con 0x7f4ef407ee50 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.355+0000 7f4eeaa00700 1 -- 172.21.15.44:0/2432195447 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f4ee4032020 con 0x7f4ef407ee50 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.433+0000 7f4ef3200700 1 -- 172.21.15.44:0/2432195447 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f4ef402e8e0 con 0x7f4ef407ee50 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.442+0000 7f4eeaa00700 1 -- 172.21.15.44:0/2432195447 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+492 (secure 0 0 0) 0x7f4ee4004260 con 0x7f4ef407ee50 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.443+0000 7f4ef3200700 1 -- 172.21.15.44:0/2432195447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 msgr2=0x7f4ef419f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.443+0000 7f4ef3200700 1 --2- 172.21.15.44:0/2432195447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef419f870 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4000c00 tx=0x7f4ee4004750 comp rx=0 tx=0).stop 2024-05-27T09:21:39.061 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.443+0000 7f4ef3200700 1 -- 172.21.15.44:0/2432195447 shutdown_connections 2024-05-27T09:21:39.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.443+0000 7f4ef3200700 1 --2- 172.21.15.44:0/2432195447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ef407ee50 0x7f4ef419f870 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:21:39.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.443+0000 7f4ef3200700 1 -- 172.21.15.44:0/2432195447 >> 172.21.15.44:0/2432195447 conn(0x7f4ef4105220 msgr2=0x7f4ef4107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:21:39.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.443+0000 7f4ef3200700 1 -- 172.21.15.44:0/2432195447 shutdown_connections 2024-05-27T09:21:39.062 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:38.443+0000 7f4ef3200700 1 -- 172.21.15.44:0/2432195447 wait complete. 2024-05-27T09:21:39.062 INFO:teuthology.orchestra.run.smithi044.stdout:Generating new minimal ceph.conf... 2024-05-27T09:21:40.127 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:39 smithi044 ceph-mon[31248]: from='client.? 172.21.15.44:0/2211388871' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:21:40.480 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.724+0000 7f928fc00700 1 Processor -- start 2024-05-27T09:21:40.480 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.724+0000 7f928fc00700 1 -- start start 2024-05-27T09:21:40.480 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.725+0000 7f928fc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f928810c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:21:40.480 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.725+0000 7f928fc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f928810ca50 con 0x7f928810c050 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.725+0000 7f928d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f928810c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.725+0000 7f928d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f928810c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45912/0 (socket says 172.21.15.44:45912) 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.725+0000 7f928d200700 1 -- 172.21.15.44:0/2283323572 learned_addr learned my addr 172.21.15.44:0/2283323572 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.726+0000 7f928d200700 1 -- 172.21.15.44:0/2283323572 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928810d180 con 0x7f928810c050 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.726+0000 7f928d200700 1 --2- 172.21.15.44:0/2283323572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f928810c460 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f927c0098a0 tx=0x7f927c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af37e8e9425b4f45 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.726+0000 7f9287400700 1 -- 172.21.15.44:0/2283323572 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f927c004160 con 0x7f928810c050 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.726+0000 7f9287400700 1 -- 172.21.15.44:0/2283323572 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f927c004300 con 0x7f928810c050 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.726+0000 7f9287400700 1 -- 172.21.15.44:0/2283323572 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f927c011af0 con 0x7f928810c050 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.727+0000 7f928fc00700 1 -- 172.21.15.44:0/2283323572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 msgr2=0x7f928810c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.727+0000 7f928fc00700 1 --2- 172.21.15.44:0/2283323572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f928810c460 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f927c0098a0 tx=0x7f927c009bb0 comp rx=0 tx=0).stop 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.727+0000 7f928fc00700 1 -- 172.21.15.44:0/2283323572 shutdown_connections 2024-05-27T09:21:40.481 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.727+0000 7f928fc00700 1 --2- 172.21.15.44:0/2283323572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f928810c460 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.727+0000 7f928fc00700 1 -- 172.21.15.44:0/2283323572 >> 172.21.15.44:0/2283323572 conn(0x7f928807f410 msgr2=0x7f9288109d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.728+0000 7f928fc00700 1 -- 172.21.15.44:0/2283323572 shutdown_connections 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.728+0000 7f928fc00700 1 -- 172.21.15.44:0/2283323572 wait complete. 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.729+0000 7f928fc00700 1 Processor -- start 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.729+0000 7f928fc00700 1 -- start start 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.730+0000 7f928fc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f92881a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.730+0000 7f928fc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9288110530 con 0x7f928810c050 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.730+0000 7f928d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f92881a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.730+0000 7f928d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f92881a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45920/0 (socket says 172.21.15.44:45920) 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.730+0000 7f928d200700 1 -- 172.21.15.44:0/2211388871 learned_addr learned my addr 172.21.15.44:0/2211388871 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.731+0000 7f928d200700 1 -- 172.21.15.44:0/2211388871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92881a0990 con 0x7f928810c050 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.731+0000 7f928d200700 1 --2- 172.21.15.44:0/2211388871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f92881a0450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f927c011890 tx=0x7f927c011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:21:40.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.731+0000 7f9285600700 1 -- 172.21.15.44:0/2211388871 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f927c0039e0 con 0x7f928810c050 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.732+0000 7f9285600700 1 -- 172.21.15.44:0/2211388871 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f927c003b80 con 0x7f928810c050 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.732+0000 7f9285600700 1 -- 172.21.15.44:0/2211388871 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f927c0246e0 con 0x7f928810c050 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.732+0000 7f928fc00700 1 -- 172.21.15.44:0/2211388871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92881a0c20 con 0x7f928810c050 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.732+0000 7f928fc00700 1 -- 172.21.15.44:0/2211388871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92881a3830 con 0x7f928810c050 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.732+0000 7f9285600700 1 -- 172.21.15.44:0/2211388871 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f927c022020 con 0x7f928810c050 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.732+0000 7f9285600700 1 -- 172.21.15.44:0/2211388871 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f927c02dc40 con 0x7f928810c050 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.733+0000 7f928fc00700 1 -- 172.21.15.44:0/2211388871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9288064e10 con 0x7f928810c050 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.736+0000 7f9285600700 1 -- 172.21.15.44:0/2211388871 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f927c01aa70 con 0x7f928810c050 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.814+0000 7f928fc00700 1 -- 172.21.15.44:0/2211388871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f92881a3e30 con 0x7f928810c050 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.815+0000 7f9285600700 1 -- 172.21.15.44:0/2211388871 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f927c024e20 con 0x7f928810c050 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.817+0000 7f928fc00700 1 -- 172.21.15.44:0/2211388871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 msgr2=0x7f92881a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:21:40.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.817+0000 7f928fc00700 1 --2- 172.21.15.44:0/2211388871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f92881a0450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f927c011890 tx=0x7f927c011970 comp rx=0 tx=0).stop 2024-05-27T09:21:40.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.817+0000 7f928fc00700 1 -- 172.21.15.44:0/2211388871 shutdown_connections 2024-05-27T09:21:40.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.817+0000 7f928fc00700 1 --2- 172.21.15.44:0/2211388871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810c050 0x7f92881a0450 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:21:40.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.817+0000 7f928fc00700 1 -- 172.21.15.44:0/2211388871 >> 172.21.15.44:0/2211388871 conn(0x7f928807f410 msgr2=0x7f9288109240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:21:40.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.817+0000 7f928fc00700 1 -- 172.21.15.44:0/2211388871 shutdown_connections 2024-05-27T09:21:40.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:39.817+0000 7f928fc00700 1 -- 172.21.15.44:0/2211388871 wait complete. 2024-05-27T09:21:40.484 INFO:teuthology.orchestra.run.smithi044.stdout:Restarting the monitor... 2024-05-27T09:21:40.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:40 smithi044 systemd[1]: Stopping Ceph mon.a for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:21:41.315 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:40 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a[31244]: 2024-05-27T09:21:40.991+0000 7f40db800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:21:41.315 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:40 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a[31244]: 2024-05-27T09:21:40.991+0000 7f40db800700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-27T09:21:41.315 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:41 smithi044 podman[31493]: 2024-05-27 09:21:41.063569714 +0000 UTC m=+0.221253566 container stop 79c61d4d327a2a1679a6853da3a0047d1b54414c6a1024e579b5b7a685230e1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:21:41.315 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:41 smithi044 podman[31493]: 2024-05-27 09:21:41.091007929 +0000 UTC m=+0.248691765 container died 79c61d4d327a2a1679a6853da3a0047d1b54414c6a1024e579b5b7a685230e1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-27T09:21:41.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:41 smithi044 podman[31493]: 2024-05-27 09:21:41.533914432 +0000 UTC m=+0.691598294 container cleanup 79c61d4d327a2a1679a6853da3a0047d1b54414c6a1024e579b5b7a685230e1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-27T09:21:41.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:41 smithi044 bash[31493]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a 2024-05-27T09:21:50.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:50 smithi044 podman[31507]: 2024-05-27 09:21:50.09401766 +0000 UTC m=+9.007930380 container remove 79c61d4d327a2a1679a6853da3a0047d1b54414c6a1024e579b5b7a685230e1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, GIT_BRANCH=HEAD) 2024-05-27T09:21:50.723 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:50 smithi044 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-27T09:21:50.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:50 smithi044 systemd[1]: Stopped Ceph mon.a for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:21:50.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:50 smithi044 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.a.service: Consumed 2.006s CPU time. 2024-05-27T09:21:50.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:50 smithi044 systemd[1]: Starting Ceph mon.a for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:21:51.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:50 smithi044 podman[31599]: 2024-05-27 09:21:50.77167347 +0000 UTC m=+0.038919202 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:21:52.092 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:51 smithi044 podman[31599]: 2024-05-27 09:21:51.829988479 +0000 UTC m=+1.097234223 container create b84ccfae169a68c88fdcfff87c89693b8d29e829b2905953c90493f103ffe83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T09:21:52.214 INFO:teuthology.orchestra.run.smithi044.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-27T09:21:52.375 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 podman[31599]: 2024-05-27 09:21:52.091749802 +0000 UTC m=+1.358995593 container init b84ccfae169a68c88fdcfff87c89693b8d29e829b2905953c90493f103ffe83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=) 2024-05-27T09:21:52.375 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 podman[31599]: 2024-05-27 09:21:52.104472992 +0000 UTC m=+1.371718773 container start b84ccfae169a68c88fdcfff87c89693b8d29e829b2905953c90493f103ffe83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:21:52.375 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 bash[31599]: b84ccfae169a68c88fdcfff87c89693b8d29e829b2905953c90493f103ffe83a 2024-05-27T09:21:52.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 systemd[1]: Started Ceph mon.a for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:21:52.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T09:21:52.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev), process ceph-mon, pid 2 2024-05-27T09:21:52.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: pidfile_write: ignore empty --pid-file 2024-05-27T09:21:52.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: load: jerasure load: lrc 2024-05-27T09:21:52.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: RocksDB version: 7.9.2 2024-05-27T09:21:52.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Git sha 0 2024-05-27T09:21:52.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Compile date 2024-05-21 11:39:56 2024-05-27T09:21:52.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: DB SUMMARY 2024-05-27T09:21:52.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: DB Session ID: PAYG090COAQL49YWL63B 2024-05-27T09:21:52.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: CURRENT file: CURRENT 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106026 ; 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.error_if_exists: 0 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.create_if_missing: 0 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.paranoid_checks: 1 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.env: 0x55f75d093500 2024-05-27T09:21:52.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T09:21:52.378 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.info_log: 0x55f78ccea040 2024-05-27T09:21:52.378 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T09:21:52.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.statistics: (nil) 2024-05-27T09:21:52.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.use_fsync: 0 2024-05-27T09:21:52.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_log_file_size: 0 2024-05-27T09:21:52.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T09:21:52.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T09:21:52.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T09:21:52.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T09:21:52.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.allow_fallocate: 1 2024-05-27T09:21:52.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T09:21:52.379 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T09:21:52.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.use_direct_reads: 0 2024-05-27T09:21:52.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T09:21:52.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T09:21:52.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.db_log_dir: 2024-05-27T09:21:52.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.wal_dir: 2024-05-27T09:21:52.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T09:21:52.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T09:21:52.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T09:21:52.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T09:21:52.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T09:21:52.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.write_buffer_manager: 0x55f78cbb48c0 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T09:21:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.unordered_write: 0 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.row_cache: None 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.wal_filter: None 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.two_write_queues: 0 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.wal_compression: 0 2024-05-27T09:21:52.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.atomic_flush: 0 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.log_readahead_size: 0 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T09:21:52.383 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_background_jobs: 2 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_background_compactions: -1 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_subcompactions: 1 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_open_files: -1 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T09:21:52.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_background_flushes: -1 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Compression algorithms supported: 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: kZSTD supported: 0 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: kXpressCompression supported: 0 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: kZlibCompression supported: 1 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: kSnappyCompression supported: 1 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: kLZ4Compression supported: 1 2024-05-27T09:21:52.385 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: kBZip2Compression supported: 0 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.merge_operator: 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_filter: None 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T09:21:52.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f78ccea200) 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks: 1 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_top_level_index_and_filter: 1 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_type: 0 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_index_type: 0 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_shortening: 1 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: checksum: 4 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: no_block_cache: 0 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache: 0x55f78cbe71f0 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_name: BinnedLRUCache 2024-05-27T09:21:52.387 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_options: 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: capacity : 536870912 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: num_shard_bits : 4 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: strict_capacity_limit : 0 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: high_pri_pool_ratio: 0.000 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_compressed: (nil) 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: persistent_cache: (nil) 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size: 4096 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size_deviation: 10 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_restart_interval: 16 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_block_restart_interval: 1 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: metadata_block_size: 4096 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: partition_filters: 0 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: use_delta_encoding: 1 2024-05-27T09:21:52.388 INFO:journalctl@ceph.mon.a.smithi044.stdout: filter_policy: bloomfilter 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout: whole_key_filtering: 1 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout: verify_compression: 0 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout: read_amp_bytes_per_bit: 0 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout: format_version: 5 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_index_compression: 1 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_align: 0 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout: max_auto_readahead_size: 262144 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout: prepopulate_block_cache: 0 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout: initial_auto_readahead_size: 8192 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compression: NoCompression 2024-05-27T09:21:52.389 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.num_levels: 7 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T09:21:52.390 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T09:21:52.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T09:21:52.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T09:21:52.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T09:21:52.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T09:21:52.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T09:21:52.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T09:21:52.393 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T09:21:52.393 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T09:21:52.393 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T09:21:52.393 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T09:21:52.393 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T09:21:52.393 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T09:21:52.393 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T09:21:52.393 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T09:21:52.393 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T09:21:52.393 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T09:21:52.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.inplace_update_support: 0 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.bloom_locality: 0 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.max_successive_merges: 0 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T09:21:52.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T09:21:52.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.ttl: 2592000 2024-05-27T09:21:52.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T09:21:52.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T09:21:52.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T09:21:52.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.enable_blob_files: false 2024-05-27T09:21:52.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.min_blob_size: 0 2024-05-27T09:21:52.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T09:21:52.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T09:21:52.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T09:21:52.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T09:21:52.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cdfc36cb-db5d-4ffd-9875-24f7ace614b2 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716801712243228, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716801712246972, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101078, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 330, "table_properties": {"data_size": 98967, "index_size": 303, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14405, "raw_average_key_size": 49, "raw_value_size": 91146, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 289, "num_filter_entries": 289, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716801712, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cdfc36cb-db5d-4ffd-9875-24f7ace614b2", "db_session_id": "PAYG090COAQL49YWL63B", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716801712247197, "job": 1, "event": "recovery_finished"} 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T09:21:52.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f78cbcbc00 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: DB pointer 0x55f7919be000 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** DB Stats ** 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T09:21:52.398 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: L0 2/0 100.57 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 50.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: Sum 2/0 100.57 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 50.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 50.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 50.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T09:21:52.399 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative compaction: 0.00 GB write, 8.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval compaction: 0.00 GB write, 8.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: Block cache BinnedLRUCache@0x55f78cbe71f0#2 capacity: 512.00 MB usage: 1.53 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.03 KB,0.000196695%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-27T09:21:52.400 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: starting mon.a rank 0 at public addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] at bind addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: mon.a@-1(???) e1 preinit fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: mon.a@-1(???).mds e1 new map 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: mon.a@-1(???).mds e1 print_map 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout: e1 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout: legacy client fscid: -1 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout: No filesystems configured 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T09:21:52.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T09:21:52.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T09:21:52.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T09:21:52.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-27T09:21:52.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-27T09:21:52.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: monmap epoch 1 2024-05-27T09:21:52.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:21:52.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: last_changed 2024-05-27T09:21:31.930626+0000 2024-05-27T09:21:52.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: created 2024-05-27T09:21:31.930626+0000 2024-05-27T09:21:52.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: min_mon_release 19 (squid) 2024-05-27T09:21:52.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: election_strategy: 1 2024-05-27T09:21:52.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-05-27T09:21:52.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: fsmap 2024-05-27T09:21:52.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: osdmap e1: 0 total, 0 up, 0 in 2024-05-27T09:21:52.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:52 smithi044 ceph-mon[31618]: mgrmap e1: no daemons active 2024-05-27T09:21:53.594 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.972+0000 7fa953a00700 1 Processor -- start 2024-05-27T09:21:53.594 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.972+0000 7fa953a00700 1 -- start start 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.973+0000 7fa953a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c07d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.973+0000 7fa953a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c07d780 con 0x7fa94c07ee10 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.974+0000 7fa951000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c07d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.974+0000 7fa951000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c07d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56916/0 (socket says 172.21.15.44:56916) 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.974+0000 7fa951000700 1 -- 172.21.15.44:0/2169873833 learned_addr learned my addr 172.21.15.44:0/2169873833 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.974+0000 7fa951000700 1 -- 172.21.15.44:0/2169873833 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c07d900 con 0x7fa94c07ee10 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.974+0000 7fa951000700 1 --2- 172.21.15.44:0/2169873833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c07d240 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa9400098a0 tx=0x7fa940009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5342de3749364e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.975+0000 7fa94b400700 1 -- 172.21.15.44:0/2169873833 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa940004160 con 0x7fa94c07ee10 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.975+0000 7fa94b400700 1 -- 172.21.15.44:0/2169873833 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa940004300 con 0x7fa94c07ee10 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.975+0000 7fa94b400700 1 -- 172.21.15.44:0/2169873833 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa940011af0 con 0x7fa94c07ee10 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.976+0000 7fa953a00700 1 -- 172.21.15.44:0/2169873833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 msgr2=0x7fa94c07d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:21:53.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.976+0000 7fa953a00700 1 --2- 172.21.15.44:0/2169873833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c07d240 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa9400098a0 tx=0x7fa940009bb0 comp rx=0 tx=0).stop 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.976+0000 7fa953a00700 1 -- 172.21.15.44:0/2169873833 shutdown_connections 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.976+0000 7fa953a00700 1 --2- 172.21.15.44:0/2169873833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c07d240 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.976+0000 7fa953a00700 1 -- 172.21.15.44:0/2169873833 >> 172.21.15.44:0/2169873833 conn(0x7fa94c1053d0 msgr2=0x7fa94c107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.976+0000 7fa953a00700 1 -- 172.21.15.44:0/2169873833 shutdown_connections 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.976+0000 7fa953a00700 1 -- 172.21.15.44:0/2169873833 wait complete. 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.977+0000 7fa953a00700 1 Processor -- start 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.977+0000 7fa953a00700 1 -- start start 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.978+0000 7fa953a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c1a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.978+0000 7fa953a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c110310 con 0x7fa94c07ee10 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.978+0000 7fa951000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c1a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.978+0000 7fa951000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c1a0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56928/0 (socket says 172.21.15.44:56928) 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.978+0000 7fa951000700 1 -- 172.21.15.44:0/2322768364 learned_addr learned my addr 172.21.15.44:0/2322768364 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:21:53.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.978+0000 7fa951000700 1 -- 172.21.15.44:0/2322768364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c1a0870 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.979+0000 7fa951000700 1 --2- 172.21.15.44:0/2322768364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c1a0330 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa9400117b0 tx=0x7fa940011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.979+0000 7fa949600700 1 -- 172.21.15.44:0/2322768364 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa940003af0 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.979+0000 7fa949600700 1 -- 172.21.15.44:0/2322768364 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa940020070 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.979+0000 7fa949600700 1 -- 172.21.15.44:0/2322768364 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa940019630 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.979+0000 7fa953a00700 1 -- 172.21.15.44:0/2322768364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c1a0b00 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.979+0000 7fa953a00700 1 -- 172.21.15.44:0/2322768364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c1a3710 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.980+0000 7fa949600700 1 -- 172.21.15.44:0/2322768364 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa940019bf0 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.980+0000 7fa949600700 1 -- 172.21.15.44:0/2322768364 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa94000b760 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.980+0000 7fa953a00700 1 -- 172.21.15.44:0/2322768364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa94c064e10 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:52.984+0000 7fa949600700 1 -- 172.21.15.44:0/2322768364 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fa9400042a0 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:53.063+0000 7fa953a00700 1 -- 172.21.15.44:0/2322768364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fa94c1a3d70 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:53.066+0000 7fa949600700 1 -- 172.21.15.44:0/2322768364 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fa94001a9c0 con 0x7fa94c07ee10 2024-05-27T09:21:53.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:53.068+0000 7fa953a00700 1 -- 172.21.15.44:0/2322768364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 msgr2=0x7fa94c1a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:21:53.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:53.068+0000 7fa953a00700 1 --2- 172.21.15.44:0/2322768364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c1a0330 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa9400117b0 tx=0x7fa940011b70 comp rx=0 tx=0).stop 2024-05-27T09:21:53.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:53.068+0000 7fa953a00700 1 -- 172.21.15.44:0/2322768364 shutdown_connections 2024-05-27T09:21:53.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:53.068+0000 7fa953a00700 1 --2- 172.21.15.44:0/2322768364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa94c07ee10 0x7fa94c1a0330 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:21:53.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:53.068+0000 7fa953a00700 1 -- 172.21.15.44:0/2322768364 >> 172.21.15.44:0/2322768364 conn(0x7fa94c1053d0 msgr2=0x7fa94c105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:21:53.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:53.068+0000 7fa953a00700 1 -- 172.21.15.44:0/2322768364 shutdown_connections 2024-05-27T09:21:53.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:53.068+0000 7fa953a00700 1 -- 172.21.15.44:0/2322768364 wait complete. 2024-05-27T09:21:53.598 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-27T09:21:53.622 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-27T09:21:53.622 INFO:teuthology.orchestra.run.smithi044.stdout:Creating mgr... 2024-05-27T09:21:53.623 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-27T09:21:53.624 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-27T09:21:54.134 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.x 2024-05-27T09:21:54.134 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to reset failed state of unit ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.x.service: Unit ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.x.service not loaded. 2024-05-27T09:21:54.389 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-604b015a-1c0a-11ef-bc9b-c7b262605968.target.wants/ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@.service. 2024-05-27T09:21:54.422 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:54 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2322768364' entity='client.admin' 2024-05-27T09:21:54.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:54 smithi044 systemd[1]: Starting Ceph mgr.x for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:21:55.064 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:54 smithi044 podman[31841]: 2024-05-27 09:21:54.711424791 +0000 UTC m=+0.036536835 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:21:55.064 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:54 smithi044 podman[31841]: 2024-05-27 09:21:54.807942644 +0000 UTC m=+0.133054725 container create 310da0308fc5104ca2b057b9aea4d1e5c20e666a22cd17b534a1b279374bd34f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T09:21:55.064 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:55 smithi044 podman[31841]: 2024-05-27 09:21:55.062023782 +0000 UTC m=+0.387135933 container init 310da0308fc5104ca2b057b9aea4d1e5c20e666a22cd17b534a1b279374bd34f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:21:55.189 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T09:21:55.189 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-05-27T09:21:55.207 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T09:21:55.208 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-05-27T09:21:55.208 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-05-27T09:21:55.208 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to enable service . firewalld.service is not available 2024-05-27T09:21:55.226 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T09:21:55.226 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-05-27T09:21:55.242 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T09:21:55.242 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-05-27T09:21:55.242 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-05-27T09:21:55.242 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-27T09:21:55.242 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr to start... 2024-05-27T09:21:55.242 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr... 2024-05-27T09:21:55.401 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:55 smithi044 podman[31841]: 2024-05-27 09:21:55.071729419 +0000 UTC m=+0.396841470 container start 310da0308fc5104ca2b057b9aea4d1e5c20e666a22cd17b534a1b279374bd34f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T09:21:55.401 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:55 smithi044 bash[31841]: 310da0308fc5104ca2b057b9aea4d1e5c20e666a22cd17b534a1b279374bd34f 2024-05-27T09:21:55.401 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:55 smithi044 systemd[1]: Started Ceph mgr.x for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:21:55.673 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:55 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:55.404+0000 7fd17f051200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T09:21:55.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:55 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:55.674+0000 7fd17f051200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:21:55.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:55 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:55.817+0000 7fd17f051200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T09:21:56.251 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:56 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1242908078' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T09:21:56.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:56 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:56.251+0000 7fd17f051200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T09:21:56.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:56 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:56.386+0000 7fd17f051200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T09:21:57.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:21:57.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-05-27T09:21:57.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:21:57.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T09:21:57.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T09:21:57.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T09:21:57.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T09:21:57.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:21:57.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T09:21:57.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T09:21:57.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T09:21:57.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T09:21:57.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T09:21:57.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T09:21:57.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T09:21:57.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:21:57.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T09:21:57.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T09:21:57.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T09:21:57.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T09:21:57.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T09:21:57.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T09:21:57.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T09:21:57.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T09:21:36.259306+0000", 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T09:21:57.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.901+0000 7fa7b9000700 1 Processor -- start 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.902+0000 7fa7b9000700 1 -- start start 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.902+0000 7fa7b9000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b4109010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.902+0000 7fa7b9000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b4109600 con 0x7fa7b4108c00 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.903+0000 7fa7b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b4109010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.903+0000 7fa7b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b4109010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56960/0 (socket says 172.21.15.44:56960) 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.903+0000 7fa7b1e00700 1 -- 172.21.15.44:0/1795629045 learned_addr learned my addr 172.21.15.44:0/1795629045 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.903+0000 7fa7b1e00700 1 -- 172.21.15.44:0/1795629045 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b4109dd0 con 0x7fa7b4108c00 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.904+0000 7fa7b1e00700 1 --2- 172.21.15.44:0/1795629045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b4109010 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40098a0 tx=0x7fa7a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6fec873e678abe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.904+0000 7fa7b0a00700 1 -- 172.21.15.44:0/1795629045 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a4004160 con 0x7fa7b4108c00 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.904+0000 7fa7b0a00700 1 -- 172.21.15.44:0/1795629045 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa7a4004300 con 0x7fa7b4108c00 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.904+0000 7fa7b0a00700 1 -- 172.21.15.44:0/1795629045 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a4011af0 con 0x7fa7b4108c00 2024-05-27T09:21:57.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.905+0000 7fa7b9000700 1 -- 172.21.15.44:0/1795629045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 msgr2=0x7fa7b4109010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:21:57.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.905+0000 7fa7b9000700 1 --2- 172.21.15.44:0/1795629045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b4109010 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40098a0 tx=0x7fa7a4009bb0 comp rx=0 tx=0).stop 2024-05-27T09:21:57.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.906+0000 7fa7b9000700 1 -- 172.21.15.44:0/1795629045 shutdown_connections 2024-05-27T09:21:57.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.906+0000 7fa7b9000700 1 --2- 172.21.15.44:0/1795629045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b4109010 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:21:57.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.906+0000 7fa7b9000700 1 -- 172.21.15.44:0/1795629045 >> 172.21.15.44:0/1795629045 conn(0x7fa7b41042e0 msgr2=0x7fa7b4106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:21:57.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.906+0000 7fa7b9000700 1 -- 172.21.15.44:0/1795629045 shutdown_connections 2024-05-27T09:21:57.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.906+0000 7fa7b9000700 1 -- 172.21.15.44:0/1795629045 wait complete. 2024-05-27T09:21:57.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.907+0000 7fa7b9000700 1 Processor -- start 2024-05-27T09:21:57.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.908+0000 7fa7b9000700 1 -- start start 2024-05-27T09:21:57.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.908+0000 7fa7b9000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b419e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.908+0000 7fa7b9000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b410d180 con 0x7fa7b4108c00 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.908+0000 7fa7b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b419e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.908+0000 7fa7b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b419e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56968/0 (socket says 172.21.15.44:56968) 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.908+0000 7fa7b1e00700 1 -- 172.21.15.44:0/1242908078 learned_addr learned my addr 172.21.15.44:0/1242908078 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.909+0000 7fa7b1e00700 1 -- 172.21.15.44:0/1242908078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b419e690 con 0x7fa7b4108c00 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.909+0000 7fa7b1e00700 1 --2- 172.21.15.44:0/1242908078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b419e150 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4011890 tx=0x7fa7a4011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.910+0000 7fa7aaa00700 1 -- 172.21.15.44:0/1242908078 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a40039e0 con 0x7fa7b4108c00 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.910+0000 7fa7aaa00700 1 -- 172.21.15.44:0/1242908078 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa7a4003b80 con 0x7fa7b4108c00 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.910+0000 7fa7b9000700 1 -- 172.21.15.44:0/1242908078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b419e920 con 0x7fa7b4108c00 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.910+0000 7fa7aaa00700 1 -- 172.21.15.44:0/1242908078 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a40246e0 con 0x7fa7b4108c00 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.910+0000 7fa7b9000700 1 -- 172.21.15.44:0/1242908078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b41a1530 con 0x7fa7b4108c00 2024-05-27T09:21:57.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.910+0000 7fa7aaa00700 1 -- 172.21.15.44:0/1242908078 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa7a4022020 con 0x7fa7b4108c00 2024-05-27T09:21:57.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.911+0000 7fa7aaa00700 1 -- 172.21.15.44:0/1242908078 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa7a402dc60 con 0x7fa7b4108c00 2024-05-27T09:21:57.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.911+0000 7fa7b9000700 1 -- 172.21.15.44:0/1242908078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b4064e10 con 0x7fa7b4108c00 2024-05-27T09:21:57.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.914+0000 7fa7aaa00700 1 -- 172.21.15.44:0/1242908078 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fa7a401aa70 con 0x7fa7b4108c00 2024-05-27T09:21:57.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.995+0000 7fa7b9000700 1 -- 172.21.15.44:0/1242908078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa7b402e8e0 con 0x7fa7b4108c00 2024-05-27T09:21:57.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.995+0000 7fa7aaa00700 1 -- 172.21.15.44:0/1242908078 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa7a402d3f0 con 0x7fa7b4108c00 2024-05-27T09:21:57.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.996+0000 7fa7b9000700 1 -- 172.21.15.44:0/1242908078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 msgr2=0x7fa7b419e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:21:57.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.996+0000 7fa7b9000700 1 --2- 172.21.15.44:0/1242908078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b419e150 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4011890 tx=0x7fa7a4011970 comp rx=0 tx=0).stop 2024-05-27T09:21:57.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.996+0000 7fa7b9000700 1 -- 172.21.15.44:0/1242908078 shutdown_connections 2024-05-27T09:21:57.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.996+0000 7fa7b9000700 1 --2- 172.21.15.44:0/1242908078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7b4108c00 0x7fa7b419e150 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:21:57.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.996+0000 7fa7b9000700 1 -- 172.21.15.44:0/1242908078 >> 172.21.15.44:0/1242908078 conn(0x7fa7b41042e0 msgr2=0x7fa7b4194700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:21:57.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.997+0000 7fa7b9000700 1 -- 172.21.15.44:0/1242908078 shutdown_connections 2024-05-27T09:21:57.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:55.997+0000 7fa7b9000700 1 -- 172.21.15.44:0/1242908078 wait complete. 2024-05-27T09:21:57.161 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (1/15)... 2024-05-27T09:21:58.475 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:58 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:58.063+0000 7fd17f051200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T09:21:58.475 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:58 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:58.301+0000 7fd17f051200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T09:21:58.924 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:58 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:58.522+0000 7fd17f051200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T09:21:59.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:58 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:58.923+0000 7fd17f051200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T09:21:59.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:59 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:59.051+0000 7fd17f051200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T09:21:59.579 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:59 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:59.318+0000 7fd17f051200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T09:21:59.897 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:21:59 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:21:59.595+0000 7fd17f051200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:22:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:21:59 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2771384682' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T09:22:00.582 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:00.441+0000 7fd17f051200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T09:22:00.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:22:00.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-05-27T09:22:00.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:22:00.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T09:22:00.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T09:22:00.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T09:22:00.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T09:22:00.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:00.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T09:22:00.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T09:22:00.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-05-27T09:22:00.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:22:00.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T09:22:00.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T09:22:00.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:00.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T09:22:00.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T09:21:36.259306+0000", 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.806+0000 7fa86da00700 1 Processor -- start 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.806+0000 7fa86da00700 1 -- start start 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.806+0000 7fa86da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa868109f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.806+0000 7fa86da00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810a540 con 0x7fa868109b40 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.807+0000 7fa867e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa868109f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.807+0000 7fa867e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa868109f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56982/0 (socket says 172.21.15.44:56982) 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.807+0000 7fa867e00700 1 -- 172.21.15.44:0/2342739427 learned_addr learned my addr 172.21.15.44:0/2342739427 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:00.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.807+0000 7fa867e00700 1 -- 172.21.15.44:0/2342739427 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86810ad10 con 0x7fa868109b40 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.808+0000 7fa867e00700 1 --2- 172.21.15.44:0/2342739427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa868109f50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa8580098a0 tx=0x7fa858009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c3cc34661a9ced server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.808+0000 7fa866a00700 1 -- 172.21.15.44:0/2342739427 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa858004160 con 0x7fa868109b40 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.808+0000 7fa866a00700 1 -- 172.21.15.44:0/2342739427 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa858004300 con 0x7fa868109b40 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.808+0000 7fa866a00700 1 -- 172.21.15.44:0/2342739427 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa858011af0 con 0x7fa868109b40 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.809+0000 7fa86da00700 1 -- 172.21.15.44:0/2342739427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 msgr2=0x7fa868109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.809+0000 7fa86da00700 1 --2- 172.21.15.44:0/2342739427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa868109f50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa8580098a0 tx=0x7fa858009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.809+0000 7fa86da00700 1 -- 172.21.15.44:0/2342739427 shutdown_connections 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.809+0000 7fa86da00700 1 --2- 172.21.15.44:0/2342739427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa868109f50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.809+0000 7fa86da00700 1 -- 172.21.15.44:0/2342739427 >> 172.21.15.44:0/2342739427 conn(0x7fa868105220 msgr2=0x7fa868107670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.809+0000 7fa86da00700 1 -- 172.21.15.44:0/2342739427 shutdown_connections 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.809+0000 7fa86da00700 1 -- 172.21.15.44:0/2342739427 wait complete. 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.810+0000 7fa86da00700 1 Processor -- start 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.811+0000 7fa86da00700 1 -- start start 2024-05-27T09:22:00.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.811+0000 7fa86da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa86807ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.811+0000 7fa86da00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810e0c0 con 0x7fa868109b40 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.811+0000 7fa867e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa86807ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.811+0000 7fa867e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa86807ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56990/0 (socket says 172.21.15.44:56990) 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.811+0000 7fa867e00700 1 -- 172.21.15.44:0/2771384682 learned_addr learned my addr 172.21.15.44:0/2771384682 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.812+0000 7fa867e00700 1 -- 172.21.15.44:0/2771384682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86807efb0 con 0x7fa868109b40 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.812+0000 7fa867e00700 1 --2- 172.21.15.44:0/2771384682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa86807ea70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa858011890 tx=0x7fa858011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.812+0000 7fa864c00700 1 -- 172.21.15.44:0/2771384682 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8580039e0 con 0x7fa868109b40 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.812+0000 7fa864c00700 1 -- 172.21.15.44:0/2771384682 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa858003b80 con 0x7fa868109b40 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.812+0000 7fa864c00700 1 -- 172.21.15.44:0/2771384682 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8580246e0 con 0x7fa868109b40 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.812+0000 7fa86da00700 1 -- 172.21.15.44:0/2771384682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86807d200 con 0x7fa868109b40 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.812+0000 7fa86da00700 1 -- 172.21.15.44:0/2771384682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa86807d6e0 con 0x7fa868109b40 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.813+0000 7fa864c00700 1 -- 172.21.15.44:0/2771384682 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa858022020 con 0x7fa868109b40 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.813+0000 7fa864c00700 1 -- 172.21.15.44:0/2771384682 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa85802dc40 con 0x7fa868109b40 2024-05-27T09:22:00.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.814+0000 7fa86da00700 1 -- 172.21.15.44:0/2771384682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa868064e10 con 0x7fa868109b40 2024-05-27T09:22:00.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.816+0000 7fa864c00700 1 -- 172.21.15.44:0/2771384682 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fa858043ec0 con 0x7fa868109b40 2024-05-27T09:22:00.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.896+0000 7fa86da00700 1 -- 172.21.15.44:0/2771384682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa86807d960 con 0x7fa868109b40 2024-05-27T09:22:00.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.897+0000 7fa864c00700 1 -- 172.21.15.44:0/2771384682 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa858032020 con 0x7fa868109b40 2024-05-27T09:22:00.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.899+0000 7fa86da00700 1 -- 172.21.15.44:0/2771384682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 msgr2=0x7fa86807ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:00.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.899+0000 7fa86da00700 1 --2- 172.21.15.44:0/2771384682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa86807ea70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa858011890 tx=0x7fa858011970 comp rx=0 tx=0).stop 2024-05-27T09:22:00.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.899+0000 7fa86da00700 1 -- 172.21.15.44:0/2771384682 shutdown_connections 2024-05-27T09:22:00.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.899+0000 7fa86da00700 1 --2- 172.21.15.44:0/2771384682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa868109b40 0x7fa86807ea70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:00.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.899+0000 7fa86da00700 1 -- 172.21.15.44:0/2771384682 >> 172.21.15.44:0/2771384682 conn(0x7fa868105220 msgr2=0x7fa868106c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:00.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.899+0000 7fa86da00700 1 -- 172.21.15.44:0/2771384682 shutdown_connections 2024-05-27T09:22:00.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:21:59.899+0000 7fa86da00700 1 -- 172.21.15.44:0/2771384682 wait complete. 2024-05-27T09:22:00.609 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (2/15)... 2024-05-27T09:22:00.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:00.744+0000 7fd17f051200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T09:22:01.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:01 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:01.322+0000 7fd17f051200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T09:22:01.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:01 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:01.457+0000 7fd17f051200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T09:22:02.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:01 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:01.935+0000 7fd17f051200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T09:22:02.723 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:02 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:02.410+0000 7fd17f051200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T09:22:02.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:02 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:02.541+0000 7fd17f051200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T09:22:03.014 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:02 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:02.889+0000 7fd17f051200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T09:22:03.321 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:03 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:03.081+0000 7fd17f051200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T09:22:03.322 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:03 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:03.218+0000 7fd17f051200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T09:22:03.637 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:03 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4009101254' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T09:22:03.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:03 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:03.636+0000 7fd17f051200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T09:22:03.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:03 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:03.765+0000 7fd17f051200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T09:22:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:22:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-05-27T09:22:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:22:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T09:22:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T09:22:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T09:22:04.055 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T09:22:04.055 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:04.055 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T09:22:04.055 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T09:22:04.056 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T09:22:04.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T09:22:04.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T09:22:04.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T09:22:04.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T09:22:04.064 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T09:22:04.065 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:04.066 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T09:21:36.259306+0000", 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.230+0000 7f689ae00700 1 Processor -- start 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.230+0000 7f689ae00700 1 -- start start 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.231+0000 7f689ae00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689410cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.231+0000 7f689ae00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f689407d6d0 con 0x7f689410a8c0 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.231+0000 7f6893e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689410cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.231+0000 7f6893e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689410cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60176/0 (socket says 172.21.15.44:60176) 2024-05-27T09:22:04.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.231+0000 7f6893e00700 1 -- 172.21.15.44:0/1028643573 learned_addr learned my addr 172.21.15.44:0/1028643573 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.232+0000 7f6893e00700 1 -- 172.21.15.44:0/1028643573 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689410d220 con 0x7f689410a8c0 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.232+0000 7f6893e00700 1 --2- 172.21.15.44:0/1028643573 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689410cce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f68840098a0 tx=0x7f6884009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e822b3aadc17a11 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.232+0000 7f6892a00700 1 -- 172.21.15.44:0/1028643573 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6884004160 con 0x7f689410a8c0 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.232+0000 7f6892a00700 1 -- 172.21.15.44:0/1028643573 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6884004300 con 0x7f689410a8c0 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.232+0000 7f6892a00700 1 -- 172.21.15.44:0/1028643573 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6884011af0 con 0x7f689410a8c0 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.233+0000 7f689ae00700 1 -- 172.21.15.44:0/1028643573 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 msgr2=0x7f689410cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.233+0000 7f689ae00700 1 --2- 172.21.15.44:0/1028643573 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689410cce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f68840098a0 tx=0x7f6884009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.233+0000 7f689ae00700 1 -- 172.21.15.44:0/1028643573 shutdown_connections 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.233+0000 7f689ae00700 1 --2- 172.21.15.44:0/1028643573 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689410cce0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.233+0000 7f689ae00700 1 -- 172.21.15.44:0/1028643573 >> 172.21.15.44:0/1028643573 conn(0x7f68941042c0 msgr2=0x7f6894106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.233+0000 7f689ae00700 1 -- 172.21.15.44:0/1028643573 shutdown_connections 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.233+0000 7f689ae00700 1 -- 172.21.15.44:0/1028643573 wait complete. 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.234+0000 7f689ae00700 1 Processor -- start 2024-05-27T09:22:04.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.235+0000 7f689ae00700 1 -- start start 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.235+0000 7f689ae00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689419e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.235+0000 7f689ae00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f689410e180 con 0x7f689410a8c0 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.235+0000 7f6893e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689419e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.235+0000 7f6893e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689419e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60186/0 (socket says 172.21.15.44:60186) 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.235+0000 7f6893e00700 1 -- 172.21.15.44:0/4009101254 learned_addr learned my addr 172.21.15.44:0/4009101254 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.236+0000 7f6893e00700 1 -- 172.21.15.44:0/4009101254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689419e6e0 con 0x7f689410a8c0 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.236+0000 7f6893e00700 1 --2- 172.21.15.44:0/4009101254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689419e1a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f68840117b0 tx=0x7f6884011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.236+0000 7f6890c00700 1 -- 172.21.15.44:0/4009101254 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6884003af0 con 0x7f689410a8c0 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.236+0000 7f6890c00700 1 -- 172.21.15.44:0/4009101254 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6884020070 con 0x7f689410a8c0 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.236+0000 7f689ae00700 1 -- 172.21.15.44:0/4009101254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f689419e970 con 0x7f689410a8c0 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.236+0000 7f6890c00700 1 -- 172.21.15.44:0/4009101254 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6884019630 con 0x7f689410a8c0 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.237+0000 7f689ae00700 1 -- 172.21.15.44:0/4009101254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68941a1580 con 0x7f689410a8c0 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.237+0000 7f6890c00700 1 -- 172.21.15.44:0/4009101254 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6884019c30 con 0x7f689410a8c0 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.237+0000 7f6890c00700 1 -- 172.21.15.44:0/4009101254 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f688400b720 con 0x7f689410a8c0 2024-05-27T09:22:04.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.238+0000 7f689ae00700 1 -- 172.21.15.44:0/4009101254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6894064e10 con 0x7f689410a8c0 2024-05-27T09:22:04.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.241+0000 7f6890c00700 1 -- 172.21.15.44:0/4009101254 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f68840042a0 con 0x7f689410a8c0 2024-05-27T09:22:04.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.321+0000 7f689ae00700 1 -- 172.21.15.44:0/4009101254 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f68941a1b70 con 0x7f689410a8c0 2024-05-27T09:22:04.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.322+0000 7f6890c00700 1 -- 172.21.15.44:0/4009101254 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f688401a9c0 con 0x7f689410a8c0 2024-05-27T09:22:04.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.323+0000 7f689ae00700 1 -- 172.21.15.44:0/4009101254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 msgr2=0x7f689419e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:04.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.323+0000 7f689ae00700 1 --2- 172.21.15.44:0/4009101254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689419e1a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f68840117b0 tx=0x7f6884011b70 comp rx=0 tx=0).stop 2024-05-27T09:22:04.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.323+0000 7f689ae00700 1 -- 172.21.15.44:0/4009101254 shutdown_connections 2024-05-27T09:22:04.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.323+0000 7f689ae00700 1 --2- 172.21.15.44:0/4009101254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f689410a8c0 0x7f689419e1a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:04.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.323+0000 7f689ae00700 1 -- 172.21.15.44:0/4009101254 >> 172.21.15.44:0/4009101254 conn(0x7f68941042c0 msgr2=0x7f6894105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:04.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.324+0000 7f689ae00700 1 -- 172.21.15.44:0/4009101254 shutdown_connections 2024-05-27T09:22:04.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:03.324+0000 7f689ae00700 1 -- 172.21.15.44:0/4009101254 wait complete. 2024-05-27T09:22:04.070 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (3/15)... 2024-05-27T09:22:04.871 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:04 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:04.572+0000 7fd17f051200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T09:22:05.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:04 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:04.870+0000 7fd17f051200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T09:22:05.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:04 smithi044 ceph-mon[31618]: Activating manager daemon x 2024-05-27T09:22:05.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:04 smithi044 ceph-mon[31618]: mgrmap e2: x(active, starting, since 0.0157023s) 2024-05-27T09:22:05.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:04 smithi044 ceph-mon[31618]: from='mgr.14100 172.21.15.44:0/14406258' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T09:22:05.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:04 smithi044 ceph-mon[31618]: from='mgr.14100 172.21.15.44:0/14406258' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T09:22:05.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:04 smithi044 ceph-mon[31618]: from='mgr.14100 172.21.15.44:0/14406258' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T09:22:05.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:04 smithi044 ceph-mon[31618]: from='mgr.14100 172.21.15.44:0/14406258' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:22:05.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:04 smithi044 ceph-mon[31618]: from='mgr.14100 172.21.15.44:0/14406258' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T09:22:05.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:04 smithi044 ceph-mon[31618]: Manager daemon x is now available 2024-05-27T09:22:05.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:04 smithi044 ceph-mon[31618]: from='mgr.14100 172.21.15.44:0/14406258' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T09:22:05.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:04 smithi044 ceph-mon[31618]: from='mgr.14100 172.21.15.44:0/14406258' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T09:22:06.222 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:05 smithi044 ceph-mon[31618]: from='mgr.14100 172.21.15.44:0/14406258' entity='mgr.x' 2024-05-27T09:22:06.222 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:05 smithi044 ceph-mon[31618]: from='mgr.14100 172.21.15.44:0/14406258' entity='mgr.x' 2024-05-27T09:22:06.222 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:05 smithi044 ceph-mon[31618]: from='mgr.14100 172.21.15.44:0/14406258' entity='mgr.x' 2024-05-27T09:22:06.222 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:05 smithi044 ceph-mon[31618]: mgrmap e3: x(active, since 1.02341s) 2024-05-27T09:22:07.648 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:22:07.649 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-05-27T09:22:07.649 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:22:07.649 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T09:22:07.649 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T09:22:07.650 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T09:22:07.651 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T09:22:07.651 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:07.651 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T09:22:07.651 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T09:22:07.651 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-05-27T09:22:07.651 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:22:07.651 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T09:22:07.652 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T09:22:07.652 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:22:07.652 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-05-27T09:22:07.652 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T09:22:07.652 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:07.653 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T09:22:07.653 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T09:22:07.653 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:07.653 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T09:22:07.653 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:07.654 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T09:22:07.654 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T09:22:07.654 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T09:22:07.654 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T09:22:07.654 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T09:22:07.658 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T09:22:07.658 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:07.658 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T09:22:07.658 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T09:22:07.658 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T09:22:07.658 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T09:22:07.658 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T09:22:07.658 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T09:22:07.658 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T09:22:07.658 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T09:22:07.659 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T09:21:36.259306+0000", 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.734+0000 7f83c8c00700 1 Processor -- start 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.734+0000 7f83c8c00700 1 -- start start 2024-05-27T09:22:07.660 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.735+0000 7f83c8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c4080530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:07.661 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.735+0000 7f83c8c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f83c4080b40 con 0x7f83c4080120 2024-05-27T09:22:07.661 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.736+0000 7f83c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c4080530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:07.666 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.736+0000 7f83c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c4080530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60252/0 (socket says 172.21.15.44:60252) 2024-05-27T09:22:07.666 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.736+0000 7f83c1e00700 1 -- 172.21.15.44:0/2138715939 learned_addr learned my addr 172.21.15.44:0/2138715939 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:07.666 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.736+0000 7f83c1e00700 1 -- 172.21.15.44:0/2138715939 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c40812a0 con 0x7f83c4080120 2024-05-27T09:22:07.666 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.736+0000 7f83c1e00700 1 --2- 172.21.15.44:0/2138715939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c4080530 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f83b40098a0 tx=0x7f83b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08348ca143620d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:07.666 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.737+0000 7f83c0a00700 1 -- 172.21.15.44:0/2138715939 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83b4004160 con 0x7f83c4080120 2024-05-27T09:22:07.666 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.737+0000 7f83c0a00700 1 -- 172.21.15.44:0/2138715939 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f83b4004300 con 0x7f83c4080120 2024-05-27T09:22:07.666 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.737+0000 7f83c0a00700 1 -- 172.21.15.44:0/2138715939 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83b4011af0 con 0x7f83c4080120 2024-05-27T09:22:07.666 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.738+0000 7f83c8c00700 1 -- 172.21.15.44:0/2138715939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 msgr2=0x7f83c4080530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:07.666 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.738+0000 7f83c8c00700 1 --2- 172.21.15.44:0/2138715939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c4080530 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f83b40098a0 tx=0x7f83b4009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.738+0000 7f83c8c00700 1 -- 172.21.15.44:0/2138715939 shutdown_connections 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.738+0000 7f83c8c00700 1 --2- 172.21.15.44:0/2138715939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c4080530 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.738+0000 7f83c8c00700 1 -- 172.21.15.44:0/2138715939 >> 172.21.15.44:0/2138715939 conn(0x7f83c407f410 msgr2=0x7f83c4109d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.738+0000 7f83c8c00700 1 -- 172.21.15.44:0/2138715939 shutdown_connections 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.738+0000 7f83c8c00700 1 -- 172.21.15.44:0/2138715939 wait complete. 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.739+0000 7f83c8c00700 1 Processor -- start 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.739+0000 7f83c8c00700 1 -- start start 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.740+0000 7f83c8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c41a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.740+0000 7f83c8c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f83c4082660 con 0x7f83c4080120 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.740+0000 7f83c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c41a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.740+0000 7f83c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c41a4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60266/0 (socket says 172.21.15.44:60266) 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.740+0000 7f83c1e00700 1 -- 172.21.15.44:0/1370343897 learned_addr learned my addr 172.21.15.44:0/1370343897 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:07.667 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.740+0000 7f83c1e00700 1 -- 172.21.15.44:0/1370343897 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c41a4d40 con 0x7f83c4080120 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.741+0000 7f83c1e00700 1 --2- 172.21.15.44:0/1370343897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c41a4800 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f83b40117b0 tx=0x7f83b4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.741+0000 7f83baa00700 1 -- 172.21.15.44:0/1370343897 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83b4003af0 con 0x7f83c4080120 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.741+0000 7f83baa00700 1 -- 172.21.15.44:0/1370343897 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f83b4020070 con 0x7f83c4080120 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.741+0000 7f83c8c00700 1 -- 172.21.15.44:0/1370343897 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c41a4fd0 con 0x7f83c4080120 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.741+0000 7f83baa00700 1 -- 172.21.15.44:0/1370343897 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83b4019630 con 0x7f83c4080120 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.741+0000 7f83c8c00700 1 -- 172.21.15.44:0/1370343897 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c41a7be0 con 0x7f83c4080120 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.743+0000 7f83c8c00700 1 -- 172.21.15.44:0/1370343897 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83c4064e10 con 0x7f83c4080120 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.743+0000 7f83baa00700 1 -- 172.21.15.44:0/1370343897 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 3) ==== 50105+0+0 (secure 0 0 0) 0x7f83b401a9c0 con 0x7f83c4080120 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.744+0000 7f83baa00700 1 --2- 172.21.15.44:0/1370343897 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f83a803e350 0x7f83a8040860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.744+0000 7f83baa00700 1 -- 172.21.15.44:0/1370343897 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f83b4053870 con 0x7f83c4080120 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.746+0000 7f83c1400700 1 --2- 172.21.15.44:0/1370343897 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f83a803e350 0x7f83a8040860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.747+0000 7f83c1400700 1 --2- 172.21.15.44:0/1370343897 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f83a803e350 0x7f83a8040860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f83b0006fd0 tx=0x7f83b0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.751+0000 7f83baa00700 1 -- 172.21.15.44:0/1370343897 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f83b40042a0 con 0x7f83c4080120 2024-05-27T09:22:07.668 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:06.941+0000 7f83baa00700 1 -- 172.21.15.44:0/1370343897 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f83b401a430 con 0x7f83c4080120 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.085+0000 7f83c8c00700 1 -- 172.21.15.44:0/1370343897 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f83c402e880 con 0x7f83c4080120 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.086+0000 7f83baa00700 1 -- 172.21.15.44:0/1370343897 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f83b40172b0 con 0x7f83c4080120 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.089+0000 7f83c8c00700 1 -- 172.21.15.44:0/1370343897 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f83a803e350 msgr2=0x7f83a8040860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.089+0000 7f83c8c00700 1 --2- 172.21.15.44:0/1370343897 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f83a803e350 0x7f83a8040860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f83b0006fd0 tx=0x7f83b0006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.089+0000 7f83c8c00700 1 -- 172.21.15.44:0/1370343897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 msgr2=0x7f83c41a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.089+0000 7f83c8c00700 1 --2- 172.21.15.44:0/1370343897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c41a4800 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f83b40117b0 tx=0x7f83b4011b70 comp rx=0 tx=0).stop 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.089+0000 7f83c8c00700 1 -- 172.21.15.44:0/1370343897 shutdown_connections 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.089+0000 7f83c8c00700 1 --2- 172.21.15.44:0/1370343897 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f83a803e350 0x7f83a8040860 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.089+0000 7f83c8c00700 1 --2- 172.21.15.44:0/1370343897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83c4080120 0x7f83c41a4800 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.089+0000 7f83c8c00700 1 -- 172.21.15.44:0/1370343897 >> 172.21.15.44:0/1370343897 conn(0x7f83c407f410 msgr2=0x7f83c419af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.090+0000 7f83c8c00700 1 -- 172.21.15.44:0/1370343897 shutdown_connections 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:07.090+0000 7f83c8c00700 1 -- 172.21.15.44:0/1370343897 wait complete. 2024-05-27T09:22:07.669 INFO:teuthology.orchestra.run.smithi044.stdout:mgr is available 2024-05-27T09:22:08.197 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:07 smithi044 ceph-mon[31618]: mgrmap e4: x(active, since 2s) 2024-05-27T09:22:08.198 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:07 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1370343897' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T09:22:09.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:08 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/467075843' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-27T09:22:09.327 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:22:09.327 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [global] 2024-05-27T09:22:09.327 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout fsid = 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:22:09.327 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-27T09:22:09.327 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-27T09:22:09.328 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-27T09:22:09.329 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-27T09:22:09.329 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-27T09:22:09.330 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:22:09.330 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-27T09:22:09.330 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-27T09:22:09.330 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-05-27T09:22:09.330 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [osd] 2024-05-27T09:22:09.330 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-05-27T09:22:09.330 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-27T09:22:09.330 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-27T09:22:09.330 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-27T09:22:09.330 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-27T09:22:09.330 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-27T09:22:09.331 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.351+0000 7fa29ce00700 1 Processor -- start 2024-05-27T09:22:09.331 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.351+0000 7fa29ce00700 1 -- start start 2024-05-27T09:22:09.331 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.352+0000 7fa29ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa29810c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:09.331 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.352+0000 7fa29ce00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa29810c7e0 con 0x7fa29810bde0 2024-05-27T09:22:09.335 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.352+0000 7fa297400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa29810c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:09.335 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.352+0000 7fa297400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa29810c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60282/0 (socket says 172.21.15.44:60282) 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.352+0000 7fa297400700 1 -- 172.21.15.44:0/1904717191 learned_addr learned my addr 172.21.15.44:0/1904717191 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.353+0000 7fa297400700 1 -- 172.21.15.44:0/1904717191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa29810cfb0 con 0x7fa29810bde0 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.353+0000 7fa297400700 1 --2- 172.21.15.44:0/1904717191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa29810c1f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa2880098a0 tx=0x7fa288009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a16e47d019c73d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.354+0000 7fa296000700 1 -- 172.21.15.44:0/1904717191 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa288004160 con 0x7fa29810bde0 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.354+0000 7fa296000700 1 -- 172.21.15.44:0/1904717191 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa288004300 con 0x7fa29810bde0 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.354+0000 7fa296000700 1 -- 172.21.15.44:0/1904717191 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa288011af0 con 0x7fa29810bde0 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.355+0000 7fa29ce00700 1 -- 172.21.15.44:0/1904717191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 msgr2=0x7fa29810c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.355+0000 7fa29ce00700 1 --2- 172.21.15.44:0/1904717191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa29810c1f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa2880098a0 tx=0x7fa288009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.355+0000 7fa29ce00700 1 -- 172.21.15.44:0/1904717191 shutdown_connections 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.355+0000 7fa29ce00700 1 --2- 172.21.15.44:0/1904717191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa29810c1f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.355+0000 7fa29ce00700 1 -- 172.21.15.44:0/1904717191 >> 172.21.15.44:0/1904717191 conn(0x7fa29807f190 msgr2=0x7fa298109d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:09.336 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.355+0000 7fa29ce00700 1 -- 172.21.15.44:0/1904717191 shutdown_connections 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.355+0000 7fa29ce00700 1 -- 172.21.15.44:0/1904717191 wait complete. 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.356+0000 7fa29ce00700 1 Processor -- start 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.356+0000 7fa29ce00700 1 -- start start 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.357+0000 7fa29ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa2981a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.357+0000 7fa29ce00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa298110360 con 0x7fa29810bde0 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.357+0000 7fa297400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa2981a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.357+0000 7fa297400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa2981a0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60298/0 (socket says 172.21.15.44:60298) 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.357+0000 7fa297400700 1 -- 172.21.15.44:0/467075843 learned_addr learned my addr 172.21.15.44:0/467075843 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.357+0000 7fa297400700 1 -- 172.21.15.44:0/467075843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2981a07a0 con 0x7fa29810bde0 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.358+0000 7fa297400700 1 --2- 172.21.15.44:0/467075843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa2981a0260 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa288009fb0 tx=0x7fa288011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.358+0000 7fa28fe00700 1 -- 172.21.15.44:0/467075843 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2880036a0 con 0x7fa29810bde0 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.358+0000 7fa28fe00700 1 -- 172.21.15.44:0/467075843 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa288003cd0 con 0x7fa29810bde0 2024-05-27T09:22:09.337 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.358+0000 7fa29ce00700 1 -- 172.21.15.44:0/467075843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2981a0a30 con 0x7fa29810bde0 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.358+0000 7fa28fe00700 1 -- 172.21.15.44:0/467075843 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa28801a430 con 0x7fa29810bde0 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.358+0000 7fa29ce00700 1 -- 172.21.15.44:0/467075843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2981a3640 con 0x7fa29810bde0 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.359+0000 7fa29ce00700 1 -- 172.21.15.44:0/467075843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa298064e10 con 0x7fa29810bde0 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.360+0000 7fa28fe00700 1 -- 172.21.15.44:0/467075843 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7fa288024440 con 0x7fa29810bde0 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.360+0000 7fa28fe00700 1 --2- 172.21.15.44:0/467075843 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7fa27c03e420 0x7fa27c040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.360+0000 7fa28fe00700 1 -- 172.21.15.44:0/467075843 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa2880535b0 con 0x7fa29810bde0 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.361+0000 7fa296a00700 1 --2- 172.21.15.44:0/467075843 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7fa27c03e420 0x7fa27c040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.361+0000 7fa296a00700 1 --2- 172.21.15.44:0/467075843 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7fa27c03e420 0x7fa27c040930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa284006fd0 tx=0x7fa284006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.366+0000 7fa28fe00700 1 -- 172.21.15.44:0/467075843 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa288024810 con 0x7fa29810bde0 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.612+0000 7fa29ce00700 1 -- 172.21.15.44:0/467075843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fa2981a3810 con 0x7fa29810bde0 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.615+0000 7fa28fe00700 1 -- 172.21.15.44:0/467075843 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+436 (secure 0 0 0) 0x7fa288052fd0 con 0x7fa29810bde0 2024-05-27T09:22:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.618+0000 7fa29ce00700 1 -- 172.21.15.44:0/467075843 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7fa27c03e420 msgr2=0x7fa27c040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:09.339 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.618+0000 7fa29ce00700 1 --2- 172.21.15.44:0/467075843 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7fa27c03e420 0x7fa27c040930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa284006fd0 tx=0x7fa284006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:09.339 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.618+0000 7fa29ce00700 1 -- 172.21.15.44:0/467075843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 msgr2=0x7fa2981a0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:09.339 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.618+0000 7fa29ce00700 1 --2- 172.21.15.44:0/467075843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa2981a0260 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa288009fb0 tx=0x7fa288011890 comp rx=0 tx=0).stop 2024-05-27T09:22:09.339 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.619+0000 7fa29ce00700 1 -- 172.21.15.44:0/467075843 shutdown_connections 2024-05-27T09:22:09.339 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.619+0000 7fa29ce00700 1 --2- 172.21.15.44:0/467075843 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7fa27c03e420 0x7fa27c040930 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:09.339 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.619+0000 7fa29ce00700 1 --2- 172.21.15.44:0/467075843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa29810bde0 0x7fa2981a0260 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:09.339 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.619+0000 7fa29ce00700 1 -- 172.21.15.44:0/467075843 >> 172.21.15.44:0/467075843 conn(0x7fa29807f190 msgr2=0x7fa2981091e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:09.339 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.619+0000 7fa29ce00700 1 -- 172.21.15.44:0/467075843 shutdown_connections 2024-05-27T09:22:09.339 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:08.619+0000 7fa29ce00700 1 -- 172.21.15.44:0/467075843 wait complete. 2024-05-27T09:22:09.339 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling cephadm module... 2024-05-27T09:22:10.626 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:10 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4187990452' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-27T09:22:10.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:10 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: ignoring --setuser ceph since I am not root 2024-05-27T09:22:10.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:10 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: ignoring --setgroup ceph since I am not root 2024-05-27T09:22:10.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:10 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:10.813+0000 7faa712c7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T09:22:11.138 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.035+0000 7f5b02c00700 1 Processor -- start 2024-05-27T09:22:11.138 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.035+0000 7f5b02c00700 1 -- start start 2024-05-27T09:22:11.138 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.036+0000 7f5b02c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:11.138 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.036+0000 7f5b02c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc10a5e0 con 0x7f5afc109be0 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.036+0000 7f5b01800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc109ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.036+0000 7f5b01800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc109ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60312/0 (socket says 172.21.15.44:60312) 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.036+0000 7f5b01800700 1 -- 172.21.15.44:0/2809025330 learned_addr learned my addr 172.21.15.44:0/2809025330 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.037+0000 7f5b01800700 1 -- 172.21.15.44:0/2809025330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5afc10adb0 con 0x7f5afc109be0 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.037+0000 7f5b01800700 1 --2- 172.21.15.44:0/2809025330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc109ff0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5af00098a0 tx=0x7f5af0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d949b6c187ed974 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.037+0000 7f5afbe00700 1 -- 172.21.15.44:0/2809025330 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5af0004160 con 0x7f5afc109be0 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.038+0000 7f5afbe00700 1 -- 172.21.15.44:0/2809025330 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5af0004300 con 0x7f5afc109be0 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.038+0000 7f5afbe00700 1 -- 172.21.15.44:0/2809025330 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5af0011af0 con 0x7f5afc109be0 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.038+0000 7f5b02c00700 1 -- 172.21.15.44:0/2809025330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 msgr2=0x7f5afc109ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.038+0000 7f5b02c00700 1 --2- 172.21.15.44:0/2809025330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc109ff0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5af00098a0 tx=0x7f5af0009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.038+0000 7f5b02c00700 1 -- 172.21.15.44:0/2809025330 shutdown_connections 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.038+0000 7f5b02c00700 1 --2- 172.21.15.44:0/2809025330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc109ff0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:11.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.038+0000 7f5b02c00700 1 -- 172.21.15.44:0/2809025330 >> 172.21.15.44:0/2809025330 conn(0x7f5afc105180 msgr2=0x7f5afc107630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.039+0000 7f5b02c00700 1 -- 172.21.15.44:0/2809025330 shutdown_connections 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.039+0000 7f5b02c00700 1 -- 172.21.15.44:0/2809025330 wait complete. 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.040+0000 7f5b02c00700 1 Processor -- start 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.040+0000 7f5b02c00700 1 -- start start 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.040+0000 7f5b02c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc1a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.040+0000 7f5b02c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc10e160 con 0x7f5afc109be0 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.040+0000 7f5b01800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc1a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.041+0000 7f5b01800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc1a02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60314/0 (socket says 172.21.15.44:60314) 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.041+0000 7f5b01800700 1 -- 172.21.15.44:0/4187990452 learned_addr learned my addr 172.21.15.44:0/4187990452 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.041+0000 7f5b01800700 1 -- 172.21.15.44:0/4187990452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5afc1a0820 con 0x7f5afc109be0 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.041+0000 7f5b01800700 1 --2- 172.21.15.44:0/4187990452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc1a02e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5af0011890 tx=0x7f5af0011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:11.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.042+0000 7f5afa000700 1 -- 172.21.15.44:0/4187990452 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5af00039e0 con 0x7f5afc109be0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.042+0000 7f5afa000700 1 -- 172.21.15.44:0/4187990452 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5af0003b80 con 0x7f5afc109be0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.042+0000 7f5afa000700 1 -- 172.21.15.44:0/4187990452 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5af00246e0 con 0x7f5afc109be0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.042+0000 7f5b02c00700 1 -- 172.21.15.44:0/4187990452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5afc1a0ab0 con 0x7f5afc109be0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.042+0000 7f5b02c00700 1 -- 172.21.15.44:0/4187990452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5afc1a36c0 con 0x7f5afc109be0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.043+0000 7f5b02c00700 1 -- 172.21.15.44:0/4187990452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5afc064e10 con 0x7f5afc109be0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.043+0000 7f5afa000700 1 -- 172.21.15.44:0/4187990452 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f5af0022020 con 0x7f5afc109be0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.044+0000 7f5afa000700 1 --2- 172.21.15.44:0/4187990452 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f5ae403e470 0x7f5ae4040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.044+0000 7f5afa000700 1 -- 172.21.15.44:0/4187990452 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5af0053340 con 0x7f5afc109be0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.044+0000 7f5b00e00700 1 --2- 172.21.15.44:0/4187990452 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f5ae403e470 0x7f5ae4040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.045+0000 7f5b00e00700 1 --2- 172.21.15.44:0/4187990452 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f5ae403e470 0x7f5ae4040980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8006fd0 tx=0x7f5ae8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.050+0000 7f5afa000700 1 -- 172.21.15.44:0/4187990452 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5af00212c0 con 0x7f5afc109be0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.358+0000 7f5b02c00700 1 -- 172.21.15.44:0/4187990452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f5afc02e5e0 con 0x7f5afc109be0 2024-05-27T09:22:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.412+0000 7f5afa000700 1 -- 172.21.15.44:0/4187990452 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f5afc02e5e0 con 0x7f5afc109be0 2024-05-27T09:22:11.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.415+0000 7f5afa000700 1 -- 172.21.15.44:0/4187990452 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f5af002cd00 con 0x7f5afc109be0 2024-05-27T09:22:11.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.416+0000 7f5b02c00700 1 -- 172.21.15.44:0/4187990452 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f5ae403e470 msgr2=0x7f5ae4040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:11.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.416+0000 7f5b02c00700 1 --2- 172.21.15.44:0/4187990452 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f5ae403e470 0x7f5ae4040980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8006fd0 tx=0x7f5ae8006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:11.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.416+0000 7f5b02c00700 1 -- 172.21.15.44:0/4187990452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 msgr2=0x7f5afc1a02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:11.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.416+0000 7f5b02c00700 1 --2- 172.21.15.44:0/4187990452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc1a02e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5af0011890 tx=0x7f5af0011970 comp rx=0 tx=0).stop 2024-05-27T09:22:11.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.417+0000 7f5b02c00700 1 -- 172.21.15.44:0/4187990452 shutdown_connections 2024-05-27T09:22:11.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.417+0000 7f5b02c00700 1 --2- 172.21.15.44:0/4187990452 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f5ae403e470 0x7f5ae4040980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:11.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.417+0000 7f5b02c00700 1 --2- 172.21.15.44:0/4187990452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5afc109be0 0x7f5afc1a02e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:11.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.417+0000 7f5b02c00700 1 -- 172.21.15.44:0/4187990452 >> 172.21.15.44:0/4187990452 conn(0x7f5afc105180 msgr2=0x7f5afc07db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:11.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.418+0000 7f5b02c00700 1 -- 172.21.15.44:0/4187990452 shutdown_connections 2024-05-27T09:22:11.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:10.418+0000 7f5b02c00700 1 -- 172.21.15.44:0/4187990452 wait complete. 2024-05-27T09:22:11.422 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:11 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:11.076+0000 7faa712c7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:22:11.422 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:11 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:11.219+0000 7faa712c7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T09:22:11.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:11 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4187990452' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-27T09:22:11.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:11 smithi044 ceph-mon[31618]: mgrmap e5: x(active, since 5s) 2024-05-27T09:22:11.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:11 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:11.638+0000 7faa712c7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T09:22:12.076 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:11 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:11.762+0000 7faa712c7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T09:22:12.722 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:12 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4114674640' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-27T09:22:12.737 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-05-27T09:22:12.737 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-27T09:22:12.737 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T09:22:12.737 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-27T09:22:12.737 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-27T09:22:12.737 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-05-27T09:22:12.737 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.767+0000 7f828c400700 1 Processor -- start 2024-05-27T09:22:12.737 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.767+0000 7f828c400700 1 -- start start 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.768+0000 7f828c400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f828407d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.768+0000 7f828c400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f828407d760 con 0x7f828407edf0 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.768+0000 7f8289a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f828407d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.768+0000 7f8289a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f828407d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60336/0 (socket says 172.21.15.44:60336) 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.768+0000 7f8289a00700 1 -- 172.21.15.44:0/725236045 learned_addr learned my addr 172.21.15.44:0/725236045 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.769+0000 7f8289a00700 1 -- 172.21.15.44:0/725236045 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828407d8e0 con 0x7f828407edf0 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.769+0000 7f8289a00700 1 --2- 172.21.15.44:0/725236045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f828407d220 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f82780098a0 tx=0x7f8278009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3247433d00025d81 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.769+0000 7f8283e00700 1 -- 172.21.15.44:0/725236045 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8278004160 con 0x7f828407edf0 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.769+0000 7f8283e00700 1 -- 172.21.15.44:0/725236045 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8278004300 con 0x7f828407edf0 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.769+0000 7f8283e00700 1 -- 172.21.15.44:0/725236045 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8278011af0 con 0x7f828407edf0 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.770+0000 7f828c400700 1 -- 172.21.15.44:0/725236045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 msgr2=0x7f828407d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:12.738 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.770+0000 7f828c400700 1 --2- 172.21.15.44:0/725236045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f828407d220 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f82780098a0 tx=0x7f8278009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.771+0000 7f828c400700 1 -- 172.21.15.44:0/725236045 shutdown_connections 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.771+0000 7f828c400700 1 --2- 172.21.15.44:0/725236045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f828407d220 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.771+0000 7f828c400700 1 -- 172.21.15.44:0/725236045 >> 172.21.15.44:0/725236045 conn(0x7f82841053d0 msgr2=0x7f8284107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.771+0000 7f828c400700 1 -- 172.21.15.44:0/725236045 shutdown_connections 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.771+0000 7f828c400700 1 -- 172.21.15.44:0/725236045 wait complete. 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.772+0000 7f828c400700 1 Processor -- start 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.772+0000 7f828c400700 1 -- start start 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.772+0000 7f828c400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f82841a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.772+0000 7f828c400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8284110330 con 0x7f828407edf0 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.773+0000 7f8289a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f82841a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.773+0000 7f8289a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f82841a0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60352/0 (socket says 172.21.15.44:60352) 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.773+0000 7f8289a00700 1 -- 172.21.15.44:0/4114674640 learned_addr learned my addr 172.21.15.44:0/4114674640 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:12.739 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.773+0000 7f8289a00700 1 -- 172.21.15.44:0/4114674640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82841a08d0 con 0x7f828407edf0 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.773+0000 7f8289a00700 1 --2- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f82841a0390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f82780117b0 tx=0x7f8278011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.774+0000 7f8282000700 1 -- 172.21.15.44:0/4114674640 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8278003af0 con 0x7f828407edf0 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.774+0000 7f8282000700 1 -- 172.21.15.44:0/4114674640 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8278020070 con 0x7f828407edf0 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.774+0000 7f8282000700 1 -- 172.21.15.44:0/4114674640 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8278019630 con 0x7f828407edf0 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.774+0000 7f828c400700 1 -- 172.21.15.44:0/4114674640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82841a0b60 con 0x7f828407edf0 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.774+0000 7f828c400700 1 -- 172.21.15.44:0/4114674640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82841a3770 con 0x7f828407edf0 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.775+0000 7f828c400700 1 -- 172.21.15.44:0/4114674640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8284064e10 con 0x7f828407edf0 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.776+0000 7f8282000700 1 -- 172.21.15.44:0/4114674640 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f827801a9c0 con 0x7f828407edf0 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.777+0000 7f8282000700 1 --2- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f826c03e4c0 0x7f826c0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.777+0000 7f8282000700 1 -- 172.21.15.44:0/4114674640 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8278053fc0 con 0x7f828407edf0 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.777+0000 7f8289000700 1 -- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f826c03e4c0 msgr2=0x7f826c0409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2049000735 2024-05-27T09:22:12.740 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.777+0000 7f8289000700 1 --2- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f826c03e4c0 0x7f826c0409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.782+0000 7f8282000700 1 -- 172.21.15.44:0/4114674640 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8278025e20 con 0x7f828407edf0 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.978+0000 7f8289000700 1 -- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f826c03e4c0 msgr2=0x7f826c0409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2049000735 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:11.978+0000 7f8289000700 1 --2- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f826c03e4c0 0x7f826c0409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.075+0000 7f828c400700 1 -- 172.21.15.44:0/4114674640 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f82841a3940 con 0x7f828407edf0 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.076+0000 7f8282000700 1 -- 172.21.15.44:0/4114674640 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f827800b3e0 con 0x7f828407edf0 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.079+0000 7f828c400700 1 -- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f826c03e4c0 msgr2=0x7f826c0409d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.079+0000 7f828c400700 1 --2- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f826c03e4c0 0x7f826c0409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.079+0000 7f828c400700 1 -- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 msgr2=0x7f82841a0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.079+0000 7f828c400700 1 --2- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f82841a0390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f82780117b0 tx=0x7f8278011b70 comp rx=0 tx=0).stop 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.080+0000 7f828c400700 1 -- 172.21.15.44:0/4114674640 shutdown_connections 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.080+0000 7f828c400700 1 --2- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7f826c03e4c0 0x7f826c0409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:12.741 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.080+0000 7f828c400700 1 --2- 172.21.15.44:0/4114674640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f828407edf0 0x7f82841a0390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:12.742 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.080+0000 7f828c400700 1 -- 172.21.15.44:0/4114674640 >> 172.21.15.44:0/4114674640 conn(0x7f82841053d0 msgr2=0x7f8284105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:12.742 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.080+0000 7f828c400700 1 -- 172.21.15.44:0/4114674640 shutdown_connections 2024-05-27T09:22:12.742 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:12.080+0000 7f828c400700 1 -- 172.21.15.44:0/4114674640 wait complete. 2024-05-27T09:22:12.742 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for the mgr to restart... 2024-05-27T09:22:12.742 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr epoch 5... 2024-05-27T09:22:13.631 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:13 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:13.393+0000 7faa712c7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T09:22:13.959 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:13 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:13.630+0000 7faa712c7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T09:22:13.959 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:13 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:13.852+0000 7faa712c7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T09:22:14.643 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:14.252+0000 7faa712c7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T09:22:14.643 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:14.381+0000 7faa712c7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T09:22:14.910 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:14.642+0000 7faa712c7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T09:22:14.910 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:14.909+0000 7faa712c7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:22:16.036 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:15 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:15.735+0000 7faa712c7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T09:22:16.036 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:16 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:16.035+0000 7faa712c7200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T09:22:16.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:16 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:16.612+0000 7faa712c7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T09:22:16.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:16 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:16.746+0000 7faa712c7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T09:22:17.702 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:17 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:17.225+0000 7faa712c7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T09:22:17.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:17 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:17.701+0000 7faa712c7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T09:22:17.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:17 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:17.832+0000 7faa712c7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T09:22:18.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:18 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:18.170+0000 7faa712c7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T09:22:18.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:18 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:18.358+0000 7faa712c7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T09:22:18.902 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:18 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:18.492+0000 7faa712c7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T09:22:19.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:18 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:18.901+0000 7faa712c7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T09:22:19.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:19 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:19.032+0000 7faa712c7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T09:22:20.126 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:19 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:19.827+0000 7faa712c7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T09:22:20.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:20 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:20.125+0000 7faa712c7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T09:22:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:20 smithi044 ceph-mon[31618]: Active manager daemon x restarted 2024-05-27T09:22:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:20 smithi044 ceph-mon[31618]: Activating manager daemon x 2024-05-27T09:22:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:20 smithi044 ceph-mon[31618]: osdmap e2: 0 total, 0 up, 0 in 2024-05-27T09:22:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:20 smithi044 ceph-mon[31618]: mgrmap e6: x(active, starting, since 0.00805863s) 2024-05-27T09:22:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:20 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:22:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:20 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T09:22:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:20 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T09:22:20.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:20 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T09:22:20.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:20 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T09:22:20.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:20 smithi044 ceph-mon[31618]: Manager daemon x is now available 2024-05-27T09:22:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:21 smithi044 ceph-mon[31618]: Found migration_current of "None". Setting to last migration. 2024-05-27T09:22:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:21 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:21 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:21 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:22:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:21 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:22:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:21 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T09:22:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:21 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T09:22:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:21 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:21 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:21 smithi044 ceph-mon[31618]: mgrmap e7: x(active, since 1.01961s) 2024-05-27T09:22:21.850 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-05-27T09:22:21.850 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-27T09:22:21.851 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-27T09:22:21.851 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-05-27T09:22:21.851 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.346+0000 7ff53f200700 1 Processor -- start 2024-05-27T09:22:21.851 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.346+0000 7ff53f200700 1 -- start start 2024-05-27T09:22:21.851 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.347+0000 7ff53f200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff540109f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:21.851 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.347+0000 7ff53f200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff54010a540 con 0x7ff540109b40 2024-05-27T09:22:21.851 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.347+0000 7ff53de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff540109f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:21.851 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.347+0000 7ff53de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff540109f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54110/0 (socket says 172.21.15.44:54110) 2024-05-27T09:22:21.851 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.347+0000 7ff53de00700 1 -- 172.21.15.44:0/2172982827 learned_addr learned my addr 172.21.15.44:0/2172982827 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:21.851 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.348+0000 7ff53de00700 1 -- 172.21.15.44:0/2172982827 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54010ad10 con 0x7ff540109b40 2024-05-27T09:22:21.851 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.348+0000 7ff53de00700 1 --2- 172.21.15.44:0/2172982827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff540109f50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff5300098a0 tx=0x7ff530009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327a3f39a5b16a32 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.348+0000 7ff53ca00700 1 -- 172.21.15.44:0/2172982827 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff530004160 con 0x7ff540109b40 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.349+0000 7ff53ca00700 1 -- 172.21.15.44:0/2172982827 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff530004300 con 0x7ff540109b40 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.349+0000 7ff53ca00700 1 -- 172.21.15.44:0/2172982827 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff530011af0 con 0x7ff540109b40 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.349+0000 7ff53f200700 1 -- 172.21.15.44:0/2172982827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 msgr2=0x7ff540109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.349+0000 7ff53f200700 1 --2- 172.21.15.44:0/2172982827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff540109f50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff5300098a0 tx=0x7ff530009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.350+0000 7ff53f200700 1 -- 172.21.15.44:0/2172982827 shutdown_connections 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.350+0000 7ff53f200700 1 --2- 172.21.15.44:0/2172982827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff540109f50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.350+0000 7ff53f200700 1 -- 172.21.15.44:0/2172982827 >> 172.21.15.44:0/2172982827 conn(0x7ff540105220 msgr2=0x7ff540107670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.350+0000 7ff53f200700 1 -- 172.21.15.44:0/2172982827 shutdown_connections 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.350+0000 7ff53f200700 1 -- 172.21.15.44:0/2172982827 wait complete. 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.351+0000 7ff53f200700 1 Processor -- start 2024-05-27T09:22:21.852 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.351+0000 7ff53f200700 1 -- start start 2024-05-27T09:22:21.853 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.351+0000 7ff53f200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff5401a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:21.853 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.351+0000 7ff53f200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff54010e0c0 con 0x7ff540109b40 2024-05-27T09:22:21.853 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.352+0000 7ff53de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff5401a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:21.853 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.352+0000 7ff53de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff5401a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54114/0 (socket says 172.21.15.44:54114) 2024-05-27T09:22:21.853 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.352+0000 7ff53de00700 1 -- 172.21.15.44:0/3917946546 learned_addr learned my addr 172.21.15.44:0/3917946546 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:21.853 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.353+0000 7ff53de00700 1 -- 172.21.15.44:0/3917946546 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5401a06e0 con 0x7ff540109b40 2024-05-27T09:22:21.853 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.353+0000 7ff53de00700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff5401a01a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff530011890 tx=0x7ff530011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:21.853 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.353+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5300039e0 con 0x7ff540109b40 2024-05-27T09:22:21.853 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.353+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff530003b80 con 0x7ff540109b40 2024-05-27T09:22:21.853 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.353+0000 7ff53f200700 1 -- 172.21.15.44:0/3917946546 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5401a0970 con 0x7ff540109b40 2024-05-27T09:22:21.853 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.354+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5300246e0 con 0x7ff540109b40 2024-05-27T09:22:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.354+0000 7ff53f200700 1 -- 172.21.15.44:0/3917946546 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5401a3580 con 0x7ff540109b40 2024-05-27T09:22:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.355+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7ff530022020 con 0x7ff540109b40 2024-05-27T09:22:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.356+0000 7ff536a00700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 0x7ff524040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.356+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 --> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff524041090 con 0x7ff52403e470 2024-05-27T09:22:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.356+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff530053550 con 0x7ff540109b40 2024-05-27T09:22:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.356+0000 7ff53d400700 1 -- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 msgr2=0x7ff524040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2049000735 2024-05-27T09:22:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.356+0000 7ff53d400700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 0x7ff524040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T09:22:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.557+0000 7ff53d400700 1 -- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 msgr2=0x7ff524040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2049000735 2024-05-27T09:22:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.557+0000 7ff53d400700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 0x7ff524040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T09:22:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.958+0000 7ff53d400700 1 -- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 msgr2=0x7ff524040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2049000735 2024-05-27T09:22:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:13.958+0000 7ff53d400700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 0x7ff524040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:14.759+0000 7ff53d400700 1 -- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 msgr2=0x7ff524040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2049000735 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:14.759+0000 7ff53d400700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 0x7ff524040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:16.361+0000 7ff53d400700 1 -- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 msgr2=0x7ff524040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2049000735 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:16.361+0000 7ff53d400700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 0x7ff524040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:19.564+0000 7ff53d400700 1 -- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 msgr2=0x7ff524040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2049000735 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:19.565+0000 7ff53d400700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 0x7ff524040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:20.134+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mgrmap(e 6) ==== 49989+0+0 (secure 0 0 0) 0x7ff530024a90 con 0x7ff540109b40 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:20.135+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 msgr2=0x7ff524040980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:20.135+0000 7ff536a00700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 0x7ff524040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.141+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7ff530054f30 con 0x7ff540109b40 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.141+0000 7ff536a00700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff524041db0 0x7ff5240441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:21.855 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.141+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 --> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff53002ae70 con 0x7ff524041db0 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.144+0000 7ff53d400700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff524041db0 0x7ff5240441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.147+0000 7ff53d400700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff524041db0 0x7ff5240441d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff52c003a10 tx=0x7ff52c009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.149+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 <== mgr.14120 v2:172.21.15.44:6800/4254617371 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7ff53002ae70 con 0x7ff524041db0 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.160+0000 7ff52be00700 1 -- 172.21.15.44:0/3917946546 --> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff540064e10 con 0x7ff524041db0 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.161+0000 7ff536a00700 1 -- 172.21.15.44:0/3917946546 <== mgr.14120 v2:172.21.15.44:6800/4254617371 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7ff540064e10 con 0x7ff524041db0 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.161+0000 7ff52be00700 1 -- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff524041db0 msgr2=0x7ff5240441d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.161+0000 7ff52be00700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff524041db0 0x7ff5240441d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff52c003a10 tx=0x7ff52c009290 comp rx=0 tx=0).stop 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.161+0000 7ff52be00700 1 -- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 msgr2=0x7ff5401a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.161+0000 7ff52be00700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff5401a01a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff530011890 tx=0x7ff530011970 comp rx=0 tx=0).stop 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.161+0000 7ff52be00700 1 -- 172.21.15.44:0/3917946546 shutdown_connections 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.161+0000 7ff52be00700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff524041db0 0x7ff5240441d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:21.856 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.161+0000 7ff52be00700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff540109b40 0x7ff5401a01a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:21.857 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.161+0000 7ff52be00700 1 --2- 172.21.15.44:0/3917946546 >> [v2:172.21.15.44:6800/2049000735,v1:172.21.15.44:6801/2049000735] conn(0x7ff52403e470 0x7ff524040980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:21.857 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.161+0000 7ff52be00700 1 -- 172.21.15.44:0/3917946546 >> 172.21.15.44:0/3917946546 conn(0x7ff540105220 msgr2=0x7ff54007d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:21.857 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.162+0000 7ff52be00700 1 -- 172.21.15.44:0/3917946546 shutdown_connections 2024-05-27T09:22:21.857 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:21.162+0000 7ff52be00700 1 -- 172.21.15.44:0/3917946546 wait complete. 2024-05-27T09:22:21.857 INFO:teuthology.orchestra.run.smithi044.stdout:mgr epoch 5 is available 2024-05-27T09:22:21.857 INFO:teuthology.orchestra.run.smithi044.stdout:Setting orchestrator backend to cephadm... 2024-05-27T09:22:22.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:22 smithi044 ceph-mon[31618]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-27T09:22:22.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:22 smithi044 ceph-mon[31618]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-27T09:22:22.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:22 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:22:23.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:23 smithi044 ceph-mon[31618]: [27/May/2024:09:22:21] ENGINE Bus STARTING 2024-05-27T09:22:23.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:23 smithi044 ceph-mon[31618]: [27/May/2024:09:22:21] ENGINE Serving on http://172.21.15.44:8765 2024-05-27T09:22:23.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:23 smithi044 ceph-mon[31618]: [27/May/2024:09:22:22] ENGINE Serving on https://172.21.15.44:7150 2024-05-27T09:22:23.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:23 smithi044 ceph-mon[31618]: [27/May/2024:09:22:22] ENGINE Bus STARTED 2024-05-27T09:22:23.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:23 smithi044 ceph-mon[31618]: mgrmap e8: x(active, since 2s) 2024-05-27T09:22:23.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:23 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:23.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:23 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:22:23.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.537+0000 7fed88a00700 1 Processor -- start 2024-05-27T09:22:23.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.537+0000 7fed88a00700 1 -- start start 2024-05-27T09:22:23.459 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.538+0000 7fed88a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed8410c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:23.459 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.538+0000 7fed88a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fed8410c7d0 con 0x7fed8410bdd0 2024-05-27T09:22:23.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.538+0000 7fed82800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed8410c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:23.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.539+0000 7fed82800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed8410c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54200/0 (socket says 172.21.15.44:54200) 2024-05-27T09:22:23.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.539+0000 7fed82800700 1 -- 172.21.15.44:0/4053366862 learned_addr learned my addr 172.21.15.44:0/4053366862 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:23.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.539+0000 7fed82800700 1 -- 172.21.15.44:0/4053366862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8410cf00 con 0x7fed8410bdd0 2024-05-27T09:22:23.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.539+0000 7fed82800700 1 --2- 172.21.15.44:0/4053366862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed8410c1e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fed740098a0 tx=0x7fed74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee267736862b547 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:23.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.540+0000 7fed81400700 1 -- 172.21.15.44:0/4053366862 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed74004160 con 0x7fed8410bdd0 2024-05-27T09:22:23.462 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.540+0000 7fed81400700 1 -- 172.21.15.44:0/4053366862 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed74004300 con 0x7fed8410bdd0 2024-05-27T09:22:23.462 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.540+0000 7fed81400700 1 -- 172.21.15.44:0/4053366862 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed74011af0 con 0x7fed8410bdd0 2024-05-27T09:22:23.462 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.541+0000 7fed88a00700 1 -- 172.21.15.44:0/4053366862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 msgr2=0x7fed8410c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:23.462 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.541+0000 7fed88a00700 1 --2- 172.21.15.44:0/4053366862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed8410c1e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fed740098a0 tx=0x7fed74009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:23.462 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.541+0000 7fed88a00700 1 -- 172.21.15.44:0/4053366862 shutdown_connections 2024-05-27T09:22:23.462 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.541+0000 7fed88a00700 1 --2- 172.21.15.44:0/4053366862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed8410c1e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:23.462 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.541+0000 7fed88a00700 1 -- 172.21.15.44:0/4053366862 >> 172.21.15.44:0/4053366862 conn(0x7fed841074c0 msgr2=0x7fed841098e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:23.462 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.541+0000 7fed88a00700 1 -- 172.21.15.44:0/4053366862 shutdown_connections 2024-05-27T09:22:23.462 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.541+0000 7fed88a00700 1 -- 172.21.15.44:0/4053366862 wait complete. 2024-05-27T09:22:23.462 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.542+0000 7fed88a00700 1 Processor -- start 2024-05-27T09:22:23.463 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.542+0000 7fed88a00700 1 -- start start 2024-05-27T09:22:23.463 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.543+0000 7fed88a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed841a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:23.463 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.543+0000 7fed88a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fed841102b0 con 0x7fed8410bdd0 2024-05-27T09:22:23.463 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.543+0000 7fed82800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed841a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:23.463 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.543+0000 7fed82800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed841a01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54214/0 (socket says 172.21.15.44:54214) 2024-05-27T09:22:23.464 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.543+0000 7fed82800700 1 -- 172.21.15.44:0/911654579 learned_addr learned my addr 172.21.15.44:0/911654579 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.543+0000 7fed82800700 1 -- 172.21.15.44:0/911654579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed841a0730 con 0x7fed8410bdd0 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.544+0000 7fed82800700 1 --2- 172.21.15.44:0/911654579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed841a01f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed740090e0 tx=0x7fed74011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.544+0000 7fed7b400700 1 -- 172.21.15.44:0/911654579 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed740039e0 con 0x7fed8410bdd0 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.544+0000 7fed7b400700 1 -- 172.21.15.44:0/911654579 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed74003b80 con 0x7fed8410bdd0 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.544+0000 7fed7b400700 1 -- 172.21.15.44:0/911654579 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed74024690 con 0x7fed8410bdd0 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.544+0000 7fed88a00700 1 -- 172.21.15.44:0/911654579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed841a09c0 con 0x7fed8410bdd0 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.544+0000 7fed88a00700 1 -- 172.21.15.44:0/911654579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed841a35d0 con 0x7fed8410bdd0 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.545+0000 7fed88a00700 1 -- 172.21.15.44:0/911654579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed84064e10 con 0x7fed8410bdd0 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.546+0000 7fed7b400700 1 -- 172.21.15.44:0/911654579 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fed74022020 con 0x7fed8410bdd0 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.546+0000 7fed7b400700 1 --2- 172.21.15.44:0/911654579 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fed6803e4c0 0x7fed680409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.546+0000 7fed7b400700 1 -- 172.21.15.44:0/911654579 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fed74053720 con 0x7fed8410bdd0 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.546+0000 7fed81e00700 1 --2- 172.21.15.44:0/911654579 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fed6803e4c0 0x7fed680409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:23.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.547+0000 7fed81e00700 1 --2- 172.21.15.44:0/911654579 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fed6803e4c0 0x7fed680409d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fed70006fd0 tx=0x7fed70006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:23.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.552+0000 7fed7b400700 1 -- 172.21.15.44:0/911654579 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fed74016260 con 0x7fed8410bdd0 2024-05-27T09:22:23.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.811+0000 7fed88a00700 1 -- 172.21.15.44:0/911654579 --> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fed841094f0 con 0x7fed6803e4c0 2024-05-27T09:22:23.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.824+0000 7fed7b400700 1 -- 172.21.15.44:0/911654579 <== mgr.14120 v2:172.21.15.44:6800/4254617371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fed841094f0 con 0x7fed6803e4c0 2024-05-27T09:22:23.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.827+0000 7fed88a00700 1 -- 172.21.15.44:0/911654579 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fed6803e4c0 msgr2=0x7fed680409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:23.467 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.827+0000 7fed88a00700 1 --2- 172.21.15.44:0/911654579 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fed6803e4c0 0x7fed680409d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fed70006fd0 tx=0x7fed70006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:23.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.827+0000 7fed88a00700 1 -- 172.21.15.44:0/911654579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 msgr2=0x7fed841a01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:23.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.827+0000 7fed88a00700 1 --2- 172.21.15.44:0/911654579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed841a01f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed740090e0 tx=0x7fed74011940 comp rx=0 tx=0).stop 2024-05-27T09:22:23.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.828+0000 7fed88a00700 1 -- 172.21.15.44:0/911654579 shutdown_connections 2024-05-27T09:22:23.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.828+0000 7fed88a00700 1 --2- 172.21.15.44:0/911654579 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fed6803e4c0 0x7fed680409d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:23.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.828+0000 7fed88a00700 1 --2- 172.21.15.44:0/911654579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed8410bdd0 0x7fed841a01f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:23.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.828+0000 7fed88a00700 1 -- 172.21.15.44:0/911654579 >> 172.21.15.44:0/911654579 conn(0x7fed841074c0 msgr2=0x7fed84108da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:23.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.828+0000 7fed88a00700 1 -- 172.21.15.44:0/911654579 shutdown_connections 2024-05-27T09:22:23.468 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:22.828+0000 7fed88a00700 1 -- 172.21.15.44:0/911654579 wait complete. 2024-05-27T09:22:24.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:24 smithi044 ceph-mon[31618]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:22:25.178 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-27T09:22:25.178 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.272+0000 7f3f36800700 1 Processor -- start 2024-05-27T09:22:25.178 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.272+0000 7f3f36800700 1 -- start start 2024-05-27T09:22:25.178 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.273+0000 7f3f36800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f3007d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:25.178 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.273+0000 7f3f36800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3007d760 con 0x7f3f3007edf0 2024-05-27T09:22:25.178 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.273+0000 7f3f2f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f3007d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:25.178 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.273+0000 7f3f2f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f3007d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53070/0 (socket says 172.21.15.44:53070) 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.273+0000 7f3f2f400700 1 -- 172.21.15.44:0/1876719618 learned_addr learned my addr 172.21.15.44:0/1876719618 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.274+0000 7f3f2f400700 1 -- 172.21.15.44:0/1876719618 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3007d8e0 con 0x7f3f3007edf0 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.274+0000 7f3f2f400700 1 --2- 172.21.15.44:0/1876719618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f3007d220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3f200098a0 tx=0x7f3f20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457b1fd316e179dc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.275+0000 7f3f2e000700 1 -- 172.21.15.44:0/1876719618 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f20004160 con 0x7f3f3007edf0 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.275+0000 7f3f2e000700 1 -- 172.21.15.44:0/1876719618 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3f20004300 con 0x7f3f3007edf0 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.275+0000 7f3f2e000700 1 -- 172.21.15.44:0/1876719618 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f20011af0 con 0x7f3f3007edf0 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.276+0000 7f3f36800700 1 -- 172.21.15.44:0/1876719618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 msgr2=0x7f3f3007d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.276+0000 7f3f36800700 1 --2- 172.21.15.44:0/1876719618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f3007d220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3f200098a0 tx=0x7f3f20009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.276+0000 7f3f36800700 1 -- 172.21.15.44:0/1876719618 shutdown_connections 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.276+0000 7f3f36800700 1 --2- 172.21.15.44:0/1876719618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f3007d220 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.276+0000 7f3f36800700 1 -- 172.21.15.44:0/1876719618 >> 172.21.15.44:0/1876719618 conn(0x7f3f301053d0 msgr2=0x7f3f30107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:25.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.276+0000 7f3f36800700 1 -- 172.21.15.44:0/1876719618 shutdown_connections 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.276+0000 7f3f36800700 1 -- 172.21.15.44:0/1876719618 wait complete. 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.277+0000 7f3f36800700 1 Processor -- start 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.277+0000 7f3f36800700 1 -- start start 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.278+0000 7f3f36800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f301a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.278+0000 7f3f36800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f30110330 con 0x7f3f3007edf0 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.278+0000 7f3f2f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f301a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.278+0000 7f3f2f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f301a0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53082/0 (socket says 172.21.15.44:53082) 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.278+0000 7f3f2f400700 1 -- 172.21.15.44:0/776989034 learned_addr learned my addr 172.21.15.44:0/776989034 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.278+0000 7f3f2f400700 1 -- 172.21.15.44:0/776989034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f301a08d0 con 0x7f3f3007edf0 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.279+0000 7f3f2f400700 1 --2- 172.21.15.44:0/776989034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f301a0390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3f200117b0 tx=0x7f3f20011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.279+0000 7f3f27e00700 1 -- 172.21.15.44:0/776989034 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f200039e0 con 0x7f3f3007edf0 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.279+0000 7f3f27e00700 1 -- 172.21.15.44:0/776989034 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3f20020070 con 0x7f3f3007edf0 2024-05-27T09:22:25.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.279+0000 7f3f36800700 1 -- 172.21.15.44:0/776989034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f301a0b60 con 0x7f3f3007edf0 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.279+0000 7f3f27e00700 1 -- 172.21.15.44:0/776989034 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f20019570 con 0x7f3f3007edf0 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.279+0000 7f3f36800700 1 -- 172.21.15.44:0/776989034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f301a3770 con 0x7f3f3007edf0 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.281+0000 7f3f36800700 1 -- 172.21.15.44:0/776989034 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f30064e10 con 0x7f3f3007edf0 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.281+0000 7f3f27e00700 1 -- 172.21.15.44:0/776989034 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f3f20019820 con 0x7f3f3007edf0 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.281+0000 7f3f27e00700 1 --2- 172.21.15.44:0/776989034 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f3f1403e4c0 0x7f3f140409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.282+0000 7f3f27e00700 1 -- 172.21.15.44:0/776989034 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3f20053b40 con 0x7f3f3007edf0 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.282+0000 7f3f2ea00700 1 --2- 172.21.15.44:0/776989034 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f3f1403e4c0 0x7f3f140409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.283+0000 7f3f2ea00700 1 --2- 172.21.15.44:0/776989034 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f3f1403e4c0 0x7f3f140409d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c006fd0 tx=0x7f3f1c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.291+0000 7f3f27e00700 1 -- 172.21.15.44:0/776989034 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3f20018d60 con 0x7f3f3007edf0 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.530+0000 7f3f36800700 1 -- 172.21.15.44:0/776989034 --> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f3f30106520 con 0x7f3f1403e4c0 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.532+0000 7f3f27e00700 1 -- 172.21.15.44:0/776989034 <== mgr.14120 v2:172.21.15.44:6800/4254617371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f3f30106520 con 0x7f3f1403e4c0 2024-05-27T09:22:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.535+0000 7f3f36800700 1 -- 172.21.15.44:0/776989034 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f3f1403e4c0 msgr2=0x7f3f140409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.535+0000 7f3f36800700 1 --2- 172.21.15.44:0/776989034 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f3f1403e4c0 0x7f3f140409d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c006fd0 tx=0x7f3f1c006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.535+0000 7f3f36800700 1 -- 172.21.15.44:0/776989034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 msgr2=0x7f3f301a0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.535+0000 7f3f36800700 1 --2- 172.21.15.44:0/776989034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f301a0390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3f200117b0 tx=0x7f3f20011940 comp rx=0 tx=0).stop 2024-05-27T09:22:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.535+0000 7f3f36800700 1 -- 172.21.15.44:0/776989034 shutdown_connections 2024-05-27T09:22:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.535+0000 7f3f36800700 1 --2- 172.21.15.44:0/776989034 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f3f1403e4c0 0x7f3f140409d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.535+0000 7f3f36800700 1 --2- 172.21.15.44:0/776989034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f3007edf0 0x7f3f301a0390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.535+0000 7f3f36800700 1 -- 172.21.15.44:0/776989034 >> 172.21.15.44:0/776989034 conn(0x7f3f301053d0 msgr2=0x7f3f30105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.535+0000 7f3f36800700 1 -- 172.21.15.44:0/776989034 shutdown_connections 2024-05-27T09:22:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:24.536+0000 7f3f36800700 1 -- 172.21.15.44:0/776989034 wait complete. 2024-05-27T09:22:25.182 INFO:teuthology.orchestra.run.smithi044.stdout:Generating ssh key... 2024-05-27T09:22:25.469 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:25 smithi044 ceph-mon[31618]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:22:26.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: Generating public/private rsa key pair. 2024-05-27T09:22:26.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: Your identification has been saved in /tmp/tmpucupzx27/key. 2024-05-27T09:22:26.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: Your public key has been saved in /tmp/tmpucupzx27/key.pub. 2024-05-27T09:22:26.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: The key fingerprint is: 2024-05-27T09:22:26.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: SHA256:HGKbsvi6v8+u4xk7dQmC+XSPlZyps+Fv1i8JpJOw8NM ceph-604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:22:26.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: The key's randomart image is: 2024-05-27T09:22:26.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: +---[RSA 3072]----+ 2024-05-27T09:22:26.725 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: | | 2024-05-27T09:22:26.725 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: | | 2024-05-27T09:22:26.725 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: | o o..+ | 2024-05-27T09:22:26.725 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: | + +.o=B. | 2024-05-27T09:22:26.725 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: | =.*oXS. | 2024-05-27T09:22:26.725 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: | .=oE = | 2024-05-27T09:22:26.725 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: | . o+ * o . | 2024-05-27T09:22:26.725 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: | .o=o o + | 2024-05-27T09:22:26.725 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: | o=OB++. o. | 2024-05-27T09:22:26.725 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: +----[SHA256]-----+ 2024-05-27T09:22:27.288 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.885+0000 7fe8fac00700 1 Processor -- start 2024-05-27T09:22:27.288 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.886+0000 7fe8fac00700 1 -- start start 2024-05-27T09:22:27.288 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.886+0000 7fe8fac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f407d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:27.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.886+0000 7fe8fac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f407d780 con 0x7fe8f407ee10 2024-05-27T09:22:27.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.887+0000 7fe8f3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f407d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:27.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.887+0000 7fe8f3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f407d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53094/0 (socket says 172.21.15.44:53094) 2024-05-27T09:22:27.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.887+0000 7fe8f3e00700 1 -- 172.21.15.44:0/1019552604 learned_addr learned my addr 172.21.15.44:0/1019552604 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:27.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.887+0000 7fe8f3e00700 1 -- 172.21.15.44:0/1019552604 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f407d900 con 0x7fe8f407ee10 2024-05-27T09:22:27.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.887+0000 7fe8f3e00700 1 --2- 172.21.15.44:0/1019552604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f407d240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40098a0 tx=0x7fe8e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c638dafe6482a33c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:27.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.888+0000 7fe8f3400700 1 -- 172.21.15.44:0/1019552604 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8e4004160 con 0x7fe8f407ee10 2024-05-27T09:22:27.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.888+0000 7fe8f3400700 1 -- 172.21.15.44:0/1019552604 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe8e4004300 con 0x7fe8f407ee10 2024-05-27T09:22:27.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.888+0000 7fe8f3400700 1 -- 172.21.15.44:0/1019552604 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8e4011af0 con 0x7fe8f407ee10 2024-05-27T09:22:27.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.888+0000 7fe8fac00700 1 -- 172.21.15.44:0/1019552604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 msgr2=0x7fe8f407d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:27.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.888+0000 7fe8fac00700 1 --2- 172.21.15.44:0/1019552604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f407d240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40098a0 tx=0x7fe8e4009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:27.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.889+0000 7fe8fac00700 1 -- 172.21.15.44:0/1019552604 shutdown_connections 2024-05-27T09:22:27.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.889+0000 7fe8fac00700 1 --2- 172.21.15.44:0/1019552604 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f407d240 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:27.291 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.889+0000 7fe8fac00700 1 -- 172.21.15.44:0/1019552604 >> 172.21.15.44:0/1019552604 conn(0x7fe8f41053d0 msgr2=0x7fe8f4107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:27.291 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.889+0000 7fe8fac00700 1 -- 172.21.15.44:0/1019552604 shutdown_connections 2024-05-27T09:22:27.291 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.889+0000 7fe8fac00700 1 -- 172.21.15.44:0/1019552604 wait complete. 2024-05-27T09:22:27.291 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.890+0000 7fe8fac00700 1 Processor -- start 2024-05-27T09:22:27.291 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.890+0000 7fe8fac00700 1 -- start start 2024-05-27T09:22:27.291 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.890+0000 7fe8fac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f41a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:27.291 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.890+0000 7fe8fac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f4110310 con 0x7fe8f407ee10 2024-05-27T09:22:27.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.891+0000 7fe8f3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f41a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:27.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.891+0000 7fe8f3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f41a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53102/0 (socket says 172.21.15.44:53102) 2024-05-27T09:22:27.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.891+0000 7fe8f3e00700 1 -- 172.21.15.44:0/181676689 learned_addr learned my addr 172.21.15.44:0/181676689 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:27.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.891+0000 7fe8f3e00700 1 -- 172.21.15.44:0/181676689 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f41a08a0 con 0x7fe8f407ee10 2024-05-27T09:22:27.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.891+0000 7fe8f3e00700 1 --2- 172.21.15.44:0/181676689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f41a0360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40117b0 tx=0x7fe8e4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:27.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.892+0000 7fe8f1600700 1 -- 172.21.15.44:0/181676689 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8e4003af0 con 0x7fe8f407ee10 2024-05-27T09:22:27.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.892+0000 7fe8f1600700 1 -- 172.21.15.44:0/181676689 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe8e4020070 con 0x7fe8f407ee10 2024-05-27T09:22:27.293 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.892+0000 7fe8f1600700 1 -- 172.21.15.44:0/181676689 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8e4019630 con 0x7fe8f407ee10 2024-05-27T09:22:27.293 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.892+0000 7fe8fac00700 1 -- 172.21.15.44:0/181676689 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8f41a0b30 con 0x7fe8f407ee10 2024-05-27T09:22:27.293 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.892+0000 7fe8fac00700 1 -- 172.21.15.44:0/181676689 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8f41a3740 con 0x7fe8f407ee10 2024-05-27T09:22:27.293 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.893+0000 7fe8fac00700 1 -- 172.21.15.44:0/181676689 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8f4064e10 con 0x7fe8f407ee10 2024-05-27T09:22:27.293 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.893+0000 7fe8f1600700 1 -- 172.21.15.44:0/181676689 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fe8e401a430 con 0x7fe8f407ee10 2024-05-27T09:22:27.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.894+0000 7fe8f1600700 1 --2- 172.21.15.44:0/181676689 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fe8d803e4c0 0x7fe8d80409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:27.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.894+0000 7fe8f1600700 1 -- 172.21.15.44:0/181676689 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe8e4053ba0 con 0x7fe8f407ee10 2024-05-27T09:22:27.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.894+0000 7fe8ebe00700 1 --2- 172.21.15.44:0/181676689 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fe8d803e4c0 0x7fe8d80409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:27.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.895+0000 7fe8ebe00700 1 --2- 172.21.15.44:0/181676689 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fe8d803e4c0 0x7fe8d80409d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc006fd0 tx=0x7fe8dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:27.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:25.899+0000 7fe8f1600700 1 -- 172.21.15.44:0/181676689 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe8e4021de0 con 0x7fe8f407ee10 2024-05-27T09:22:27.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.136+0000 7fe8fac00700 1 -- 172.21.15.44:0/181676689 --> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fe8f4106490 con 0x7fe8d803e4c0 2024-05-27T09:22:27.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.471+0000 7fe8f1600700 1 -- 172.21.15.44:0/181676689 <== mgr.14120 v2:172.21.15.44:6800/4254617371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe8f4106490 con 0x7fe8d803e4c0 2024-05-27T09:22:27.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.474+0000 7fe8fac00700 1 -- 172.21.15.44:0/181676689 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fe8d803e4c0 msgr2=0x7fe8d80409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:27.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.474+0000 7fe8fac00700 1 --2- 172.21.15.44:0/181676689 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fe8d803e4c0 0x7fe8d80409d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc006fd0 tx=0x7fe8dc006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:27.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.474+0000 7fe8fac00700 1 -- 172.21.15.44:0/181676689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 msgr2=0x7fe8f41a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:27.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.474+0000 7fe8fac00700 1 --2- 172.21.15.44:0/181676689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f41a0360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40117b0 tx=0x7fe8e4011b70 comp rx=0 tx=0).stop 2024-05-27T09:22:27.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.474+0000 7fe8fac00700 1 -- 172.21.15.44:0/181676689 shutdown_connections 2024-05-27T09:22:27.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.474+0000 7fe8fac00700 1 --2- 172.21.15.44:0/181676689 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fe8d803e4c0 0x7fe8d80409d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:27.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.474+0000 7fe8fac00700 1 --2- 172.21.15.44:0/181676689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8f407ee10 0x7fe8f41a0360 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:27.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.474+0000 7fe8fac00700 1 -- 172.21.15.44:0/181676689 >> 172.21.15.44:0/181676689 conn(0x7fe8f41053d0 msgr2=0x7fe8f4105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:27.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.474+0000 7fe8fac00700 1 -- 172.21.15.44:0/181676689 shutdown_connections 2024-05-27T09:22:27.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:26.474+0000 7fe8fac00700 1 -- 172.21.15.44:0/181676689 wait complete. 2024-05-27T09:22:27.571 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:27 smithi044 ceph-mon[31618]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:22:27.572 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:27 smithi044 ceph-mon[31618]: Generating ssh key... 2024-05-27T09:22:27.572 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:27 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:27.572 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:27 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:28.855 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:28 smithi044 ceph-mon[31618]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:22:28.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuxp6aHomdMxuJOuYLG3+1iZ9y3KLZcfEx2oO133X/JZWr8OqfLfmpO13TEc9tnu39303H1PKIyjN0fmrjmGk0ii/2ygAQoH0/ir2+iukOWTC+OyD+06PFNcASJSLm8Hhxk6w/MSBxagIy0uUSXv/ads/DYlAoyU98wkhI7nYghRcsCemrLMtMmlPO1LQS5fE99FqX/+LOGuUKnN71+abp3GuRfB+IeF9sliBtgX90iqthPc8r+NGskCJ5VQ4BwcPgVLSsO/w/TLvjBxNk1RbiZ1zRIADMsoFe3/0YIA8NpASlPNmkyyfWaLZqPatkKTcUmSa51RCCZrk5EE+9ci79zrUQcwERUE4eW/n457mJ29/nWosNwXW5rmaaxURcEu+E1IU4NeL5ADPqX3RKPaiTDX5wIfpDG6DB/fjVZoExsPh2y6FQ/9wmcmaC7p0OGs63JdPV8fh6/1xutwCGEEHzD7WWVGAv277vDOzpXwJjJWkUG7GhBI4yj1iD+qNp398= ceph-604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:22:28.872 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.987+0000 7f2aae400700 1 Processor -- start 2024-05-27T09:22:28.872 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.987+0000 7f2aae400700 1 -- start start 2024-05-27T09:22:28.872 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.988+0000 7f2aae400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa8109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:28.872 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.988+0000 7f2aae400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa810a5e0 con 0x7f2aa8109be0 2024-05-27T09:22:28.872 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.988+0000 7f2aad000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa8109ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:28.872 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.988+0000 7f2aad000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa8109ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53112/0 (socket says 172.21.15.44:53112) 2024-05-27T09:22:28.872 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.988+0000 7f2aad000700 1 -- 172.21.15.44:0/312104194 learned_addr learned my addr 172.21.15.44:0/312104194 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:28.876 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.989+0000 7f2aad000700 1 -- 172.21.15.44:0/312104194 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa810adb0 con 0x7f2aa8109be0 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.989+0000 7f2aad000700 1 --2- 172.21.15.44:0/312104194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa8109ff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0098a0 tx=0x7f2a9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc388423f428b9a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.990+0000 7f2aa7400700 1 -- 172.21.15.44:0/312104194 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a9c004160 con 0x7f2aa8109be0 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.990+0000 7f2aa7400700 1 -- 172.21.15.44:0/312104194 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2a9c004300 con 0x7f2aa8109be0 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.990+0000 7f2aa7400700 1 -- 172.21.15.44:0/312104194 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a9c011af0 con 0x7f2aa8109be0 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.991+0000 7f2aae400700 1 -- 172.21.15.44:0/312104194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 msgr2=0x7f2aa8109ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.991+0000 7f2aae400700 1 --2- 172.21.15.44:0/312104194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa8109ff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0098a0 tx=0x7f2a9c009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.991+0000 7f2aae400700 1 -- 172.21.15.44:0/312104194 shutdown_connections 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.991+0000 7f2aae400700 1 --2- 172.21.15.44:0/312104194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa8109ff0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.991+0000 7f2aae400700 1 -- 172.21.15.44:0/312104194 >> 172.21.15.44:0/312104194 conn(0x7f2aa8105180 msgr2=0x7f2aa8107630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.991+0000 7f2aae400700 1 -- 172.21.15.44:0/312104194 shutdown_connections 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.991+0000 7f2aae400700 1 -- 172.21.15.44:0/312104194 wait complete. 2024-05-27T09:22:28.877 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.993+0000 7f2aae400700 1 Processor -- start 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.993+0000 7f2aae400700 1 -- start start 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.993+0000 7f2aae400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa81a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.993+0000 7f2aae400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa810e160 con 0x7f2aa8109be0 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.994+0000 7f2aad000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa81a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.994+0000 7f2aad000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa81a02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53128/0 (socket says 172.21.15.44:53128) 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.994+0000 7f2aad000700 1 -- 172.21.15.44:0/2669154615 learned_addr learned my addr 172.21.15.44:0/2669154615 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.994+0000 7f2aad000700 1 -- 172.21.15.44:0/2669154615 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa81a0820 con 0x7f2aa8109be0 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.994+0000 7f2aad000700 1 --2- 172.21.15.44:0/2669154615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa81a02e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c011890 tx=0x7f2a9c011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.995+0000 7f2aa5600700 1 -- 172.21.15.44:0/2669154615 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a9c0039e0 con 0x7f2aa8109be0 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.995+0000 7f2aa5600700 1 -- 172.21.15.44:0/2669154615 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2a9c003b80 con 0x7f2aa8109be0 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.995+0000 7f2aae400700 1 -- 172.21.15.44:0/2669154615 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aa81a0ab0 con 0x7f2aa8109be0 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.995+0000 7f2aa5600700 1 -- 172.21.15.44:0/2669154615 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a9c0246e0 con 0x7f2aa8109be0 2024-05-27T09:22:28.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.995+0000 7f2aae400700 1 -- 172.21.15.44:0/2669154615 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aa81a36c0 con 0x7f2aa8109be0 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.996+0000 7f2aa5600700 1 -- 172.21.15.44:0/2669154615 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f2a9c022020 con 0x7f2aa8109be0 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.996+0000 7f2aae400700 1 -- 172.21.15.44:0/2669154615 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aa8064e10 con 0x7f2aa8109be0 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.997+0000 7f2aa5600700 1 --2- 172.21.15.44:0/2669154615 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f2a9003e470 0x7f2a90040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.997+0000 7f2aa5600700 1 -- 172.21.15.44:0/2669154615 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2a9c053790 con 0x7f2aa8109be0 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.997+0000 7f2aa7e00700 1 --2- 172.21.15.44:0/2669154615 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f2a9003e470 0x7f2a90040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:27.998+0000 7f2aa7e00700 1 --2- 172.21.15.44:0/2669154615 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f2a9003e470 0x7f2a90040980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2a94006fd0 tx=0x7f2a94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.004+0000 7f2aa5600700 1 -- 172.21.15.44:0/2669154615 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2a9c00edf0 con 0x7f2aa8109be0 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.246+0000 7f2aae400700 1 -- 172.21.15.44:0/2669154615 --> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f2aa802e8c0 con 0x7f2a9003e470 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.248+0000 7f2aa5600700 1 -- 172.21.15.44:0/2669154615 <== mgr.14120 v2:172.21.15.44:6800/4254617371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f2aa802e8c0 con 0x7f2a9003e470 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.251+0000 7f2aae400700 1 -- 172.21.15.44:0/2669154615 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f2a9003e470 msgr2=0x7f2a90040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.251+0000 7f2aae400700 1 --2- 172.21.15.44:0/2669154615 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f2a9003e470 0x7f2a90040980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2a94006fd0 tx=0x7f2a94006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.251+0000 7f2aae400700 1 -- 172.21.15.44:0/2669154615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 msgr2=0x7f2aa81a02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:28.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.251+0000 7f2aae400700 1 --2- 172.21.15.44:0/2669154615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa81a02e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c011890 tx=0x7f2a9c011970 comp rx=0 tx=0).stop 2024-05-27T09:22:28.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.251+0000 7f2aae400700 1 -- 172.21.15.44:0/2669154615 shutdown_connections 2024-05-27T09:22:28.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.251+0000 7f2aae400700 1 --2- 172.21.15.44:0/2669154615 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f2a9003e470 0x7f2a90040980 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:28.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.251+0000 7f2aae400700 1 --2- 172.21.15.44:0/2669154615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2aa8109be0 0x7f2aa81a02e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:28.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.251+0000 7f2aae400700 1 -- 172.21.15.44:0/2669154615 >> 172.21.15.44:0/2669154615 conn(0x7f2aa8105180 msgr2=0x7f2aa807db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:28.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.251+0000 7f2aae400700 1 -- 172.21.15.44:0/2669154615 shutdown_connections 2024-05-27T09:22:28.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:28.252+0000 7f2aae400700 1 -- 172.21.15.44:0/2669154615 wait complete. 2024-05-27T09:22:28.880 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-27T09:22:28.881 INFO:teuthology.orchestra.run.smithi044.stdout:Adding key to root@localhost authorized_keys... 2024-05-27T09:22:28.881 INFO:teuthology.orchestra.run.smithi044.stdout:Adding host smithi044... 2024-05-27T09:22:30.946 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:30 smithi044 ceph-mon[31618]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi044", "addr": "172.21.15.44", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:22:31.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:31 smithi044 ceph-mon[31618]: Deploying cephadm binary to smithi044 2024-05-27T09:22:37.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Added host 'smithi044' with addr '172.21.15.44' 2024-05-27T09:22:37.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.579+0000 7fcfbc200700 1 Processor -- start 2024-05-27T09:22:37.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.579+0000 7fcfbc200700 1 -- start start 2024-05-27T09:22:37.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.579+0000 7fcfbc200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb410a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:37.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.579+0000 7fcfbc200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb410a7b0 con 0x7fcfb4109db0 2024-05-27T09:22:37.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.580+0000 7fcfb9800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb410a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:37.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.580+0000 7fcfb9800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb410a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53142/0 (socket says 172.21.15.44:53142) 2024-05-27T09:22:37.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.580+0000 7fcfb9800700 1 -- 172.21.15.44:0/962878312 learned_addr learned my addr 172.21.15.44:0/962878312 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:37.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.580+0000 7fcfb9800700 1 -- 172.21.15.44:0/962878312 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb410af80 con 0x7fcfb4109db0 2024-05-27T09:22:37.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.581+0000 7fcfb9800700 1 --2- 172.21.15.44:0/962878312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb410a1c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80098a0 tx=0x7fcfa8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce55672f5889801 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:37.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.581+0000 7fcfb3e00700 1 -- 172.21.15.44:0/962878312 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfa8004160 con 0x7fcfb4109db0 2024-05-27T09:22:37.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.581+0000 7fcfb3e00700 1 -- 172.21.15.44:0/962878312 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcfa8004300 con 0x7fcfb4109db0 2024-05-27T09:22:37.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.581+0000 7fcfb3e00700 1 -- 172.21.15.44:0/962878312 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfa8011af0 con 0x7fcfb4109db0 2024-05-27T09:22:37.086 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.582+0000 7fcfbc200700 1 -- 172.21.15.44:0/962878312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 msgr2=0x7fcfb410a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:37.086 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.582+0000 7fcfbc200700 1 --2- 172.21.15.44:0/962878312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb410a1c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80098a0 tx=0x7fcfa8009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:37.086 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.582+0000 7fcfbc200700 1 -- 172.21.15.44:0/962878312 shutdown_connections 2024-05-27T09:22:37.086 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.582+0000 7fcfbc200700 1 --2- 172.21.15.44:0/962878312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb410a1c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:37.086 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.582+0000 7fcfbc200700 1 -- 172.21.15.44:0/962878312 >> 172.21.15.44:0/962878312 conn(0x7fcfb41053d0 msgr2=0x7fcfb4107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:37.086 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.582+0000 7fcfbc200700 1 -- 172.21.15.44:0/962878312 shutdown_connections 2024-05-27T09:22:37.086 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.582+0000 7fcfbc200700 1 -- 172.21.15.44:0/962878312 wait complete. 2024-05-27T09:22:37.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.584+0000 7fcfbc200700 1 Processor -- start 2024-05-27T09:22:37.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.584+0000 7fcfbc200700 1 -- start start 2024-05-27T09:22:37.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.584+0000 7fcfbc200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb419e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:37.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.584+0000 7fcfbc200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb410e330 con 0x7fcfb4109db0 2024-05-27T09:22:37.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.584+0000 7fcfb9800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb419e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:37.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.584+0000 7fcfb9800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb419e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53158/0 (socket says 172.21.15.44:53158) 2024-05-27T09:22:37.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.584+0000 7fcfb9800700 1 -- 172.21.15.44:0/3473160758 learned_addr learned my addr 172.21.15.44:0/3473160758 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:37.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.585+0000 7fcfb9800700 1 -- 172.21.15.44:0/3473160758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb419e750 con 0x7fcfb4109db0 2024-05-27T09:22:37.088 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.585+0000 7fcfb9800700 1 --2- 172.21.15.44:0/3473160758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb419e210 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8011890 tx=0x7fcfa8011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:37.088 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.585+0000 7fcfb2000700 1 -- 172.21.15.44:0/3473160758 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfa80039e0 con 0x7fcfb4109db0 2024-05-27T09:22:37.088 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.586+0000 7fcfb2000700 1 -- 172.21.15.44:0/3473160758 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcfa8003b80 con 0x7fcfb4109db0 2024-05-27T09:22:37.088 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.586+0000 7fcfb2000700 1 -- 172.21.15.44:0/3473160758 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfa80246e0 con 0x7fcfb4109db0 2024-05-27T09:22:37.088 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.586+0000 7fcfbc200700 1 -- 172.21.15.44:0/3473160758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb419e9e0 con 0x7fcfb4109db0 2024-05-27T09:22:37.088 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.586+0000 7fcfbc200700 1 -- 172.21.15.44:0/3473160758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb41a15f0 con 0x7fcfb4109db0 2024-05-27T09:22:37.088 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.587+0000 7fcfbc200700 1 -- 172.21.15.44:0/3473160758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfb4064e10 con 0x7fcfb4109db0 2024-05-27T09:22:37.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.587+0000 7fcfb2000700 1 -- 172.21.15.44:0/3473160758 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fcfa8022020 con 0x7fcfb4109db0 2024-05-27T09:22:37.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.588+0000 7fcfb2000700 1 --2- 172.21.15.44:0/3473160758 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fcf9c03e4c0 0x7fcf9c0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:37.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.588+0000 7fcfb2000700 1 -- 172.21.15.44:0/3473160758 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcfa80536e0 con 0x7fcfb4109db0 2024-05-27T09:22:37.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.588+0000 7fcfb8e00700 1 --2- 172.21.15.44:0/3473160758 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fcf9c03e4c0 0x7fcf9c0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:37.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.589+0000 7fcfb8e00700 1 --2- 172.21.15.44:0/3473160758 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fcf9c03e4c0 0x7fcf9c0409d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0006fd0 tx=0x7fcfa0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:37.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.594+0000 7fcfb2000700 1 -- 172.21.15.44:0/3473160758 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcfa80212c0 con 0x7fcfb4109db0 2024-05-27T09:22:37.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:29.833+0000 7fcfbc200700 1 -- 172.21.15.44:0/3473160758 --> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi044", "addr": "172.21.15.44", "target": ["mon-mgr", ""]}) -- 0x7fcfb4107440 con 0x7fcf9c03e4c0 2024-05-27T09:22:37.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:36.312+0000 7fcfb2000700 1 -- 172.21.15.44:0/3473160758 <== mgr.14120 v2:172.21.15.44:6800/4254617371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fcfb4107440 con 0x7fcf9c03e4c0 2024-05-27T09:22:37.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:36.316+0000 7fcfbc200700 1 -- 172.21.15.44:0/3473160758 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fcf9c03e4c0 msgr2=0x7fcf9c0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:37.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:36.316+0000 7fcfbc200700 1 --2- 172.21.15.44:0/3473160758 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fcf9c03e4c0 0x7fcf9c0409d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0006fd0 tx=0x7fcfa0006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:37.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:36.316+0000 7fcfbc200700 1 -- 172.21.15.44:0/3473160758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 msgr2=0x7fcfb419e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:37.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:36.316+0000 7fcfbc200700 1 --2- 172.21.15.44:0/3473160758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb419e210 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8011890 tx=0x7fcfa8011970 comp rx=0 tx=0).stop 2024-05-27T09:22:37.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:36.316+0000 7fcfbc200700 1 -- 172.21.15.44:0/3473160758 shutdown_connections 2024-05-27T09:22:37.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:36.316+0000 7fcfbc200700 1 --2- 172.21.15.44:0/3473160758 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fcf9c03e4c0 0x7fcf9c0409d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:37.091 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:36.316+0000 7fcfbc200700 1 --2- 172.21.15.44:0/3473160758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfb4109db0 0x7fcfb419e210 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:37.091 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:36.316+0000 7fcfbc200700 1 -- 172.21.15.44:0/3473160758 >> 172.21.15.44:0/3473160758 conn(0x7fcfb41053d0 msgr2=0x7fcfb4106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:37.091 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:36.317+0000 7fcfbc200700 1 -- 172.21.15.44:0/3473160758 shutdown_connections 2024-05-27T09:22:37.091 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:36.317+0000 7fcfbc200700 1 -- 172.21.15.44:0/3473160758 wait complete. 2024-05-27T09:22:37.091 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying unmanaged mon service... 2024-05-27T09:22:37.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:37 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:37.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:37 smithi044 ceph-mon[31618]: Added host smithi044 2024-05-27T09:22:37.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:37 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:22:39.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-27T09:22:39.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.820+0000 7f26da000700 1 Processor -- start 2024-05-27T09:22:39.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.820+0000 7f26da000700 1 -- start start 2024-05-27T09:22:39.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.820+0000 7f26da000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d410c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:39.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.820+0000 7f26da000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26d410c820 con 0x7f26d410be00 2024-05-27T09:22:39.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.821+0000 7f26d3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d410c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:39.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.821+0000 7f26d3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d410c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49356/0 (socket says 172.21.15.44:49356) 2024-05-27T09:22:39.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.821+0000 7f26d3400700 1 -- 172.21.15.44:0/1799341801 learned_addr learned my addr 172.21.15.44:0/1799341801 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:39.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.822+0000 7f26d3400700 1 -- 172.21.15.44:0/1799341801 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d410d080 con 0x7f26d410be00 2024-05-27T09:22:39.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.822+0000 7f26d3400700 1 --2- 172.21.15.44:0/1799341801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d410c210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f26c40098a0 tx=0x7f26c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f4cd7c1bec9457 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:39.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.823+0000 7f26d2000700 1 -- 172.21.15.44:0/1799341801 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c4004160 con 0x7f26d410be00 2024-05-27T09:22:39.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.823+0000 7f26d2000700 1 -- 172.21.15.44:0/1799341801 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f26c4004300 con 0x7f26d410be00 2024-05-27T09:22:39.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.823+0000 7f26d2000700 1 -- 172.21.15.44:0/1799341801 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c4011af0 con 0x7f26d410be00 2024-05-27T09:22:39.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.823+0000 7f26da000700 1 -- 172.21.15.44:0/1799341801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 msgr2=0x7f26d410c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:39.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.824+0000 7f26da000700 1 --2- 172.21.15.44:0/1799341801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d410c210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f26c40098a0 tx=0x7f26c4009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:39.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.824+0000 7f26da000700 1 -- 172.21.15.44:0/1799341801 shutdown_connections 2024-05-27T09:22:39.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.824+0000 7f26da000700 1 --2- 172.21.15.44:0/1799341801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d410c210 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:39.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.824+0000 7f26da000700 1 -- 172.21.15.44:0/1799341801 >> 172.21.15.44:0/1799341801 conn(0x7f26d407f480 msgr2=0x7f26d4109f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:39.290 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.824+0000 7f26da000700 1 -- 172.21.15.44:0/1799341801 shutdown_connections 2024-05-27T09:22:39.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.824+0000 7f26da000700 1 -- 172.21.15.44:0/1799341801 wait complete. 2024-05-27T09:22:39.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.825+0000 7f26da000700 1 Processor -- start 2024-05-27T09:22:39.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.825+0000 7f26da000700 1 -- start start 2024-05-27T09:22:39.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.826+0000 7f26da000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d41a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:39.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.826+0000 7f26da000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26d41104c0 con 0x7f26d410be00 2024-05-27T09:22:39.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.826+0000 7f26d3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d41a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:39.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.826+0000 7f26d3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d41a47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49370/0 (socket says 172.21.15.44:49370) 2024-05-27T09:22:39.294 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.826+0000 7f26d3400700 1 -- 172.21.15.44:0/3758019993 learned_addr learned my addr 172.21.15.44:0/3758019993 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.826+0000 7f26d3400700 1 -- 172.21.15.44:0/3758019993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d41a4cf0 con 0x7f26d410be00 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.827+0000 7f26d3400700 1 --2- 172.21.15.44:0/3758019993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d41a47b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f26c4015000 tx=0x7f26c4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.827+0000 7f26cbe00700 1 -- 172.21.15.44:0/3758019993 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c40039e0 con 0x7f26d410be00 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.827+0000 7f26cbe00700 1 -- 172.21.15.44:0/3758019993 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f26c4003b80 con 0x7f26d410be00 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.827+0000 7f26cbe00700 1 -- 172.21.15.44:0/3758019993 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c40246e0 con 0x7f26d410be00 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.827+0000 7f26da000700 1 -- 172.21.15.44:0/3758019993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26d41a4f80 con 0x7f26d410be00 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.828+0000 7f26da000700 1 -- 172.21.15.44:0/3758019993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26d41a7b90 con 0x7f26d410be00 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.829+0000 7f26da000700 1 -- 172.21.15.44:0/3758019993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26d4064e10 con 0x7f26d410be00 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.830+0000 7f26cbe00700 1 -- 172.21.15.44:0/3758019993 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f26c4022020 con 0x7f26d410be00 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.830+0000 7f26cbe00700 1 --2- 172.21.15.44:0/3758019993 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f26b803e4c0 0x7f26b80409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.830+0000 7f26cbe00700 1 -- 172.21.15.44:0/3758019993 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f26c4053ce0 con 0x7f26d410be00 2024-05-27T09:22:39.295 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.831+0000 7f26d2a00700 1 --2- 172.21.15.44:0/3758019993 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f26b803e4c0 0x7f26b80409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.831+0000 7f26d2a00700 1 --2- 172.21.15.44:0/3758019993 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f26b803e4c0 0x7f26b80409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f26c0006fd0 tx=0x7f26c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:37.836+0000 7f26cbe00700 1 -- 172.21.15.44:0/3758019993 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f26c402cd60 con 0x7f26d410be00 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.098+0000 7f26da000700 1 -- 172.21.15.44:0/3758019993 --> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f26d4082d40 con 0x7f26b803e4c0 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.106+0000 7f26cbe00700 1 -- 172.21.15.44:0/3758019993 <== mgr.14120 v2:172.21.15.44:6800/4254617371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f26d4082d40 con 0x7f26b803e4c0 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.109+0000 7f26da000700 1 -- 172.21.15.44:0/3758019993 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f26b803e4c0 msgr2=0x7f26b80409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.109+0000 7f26da000700 1 --2- 172.21.15.44:0/3758019993 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f26b803e4c0 0x7f26b80409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f26c0006fd0 tx=0x7f26c0006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.109+0000 7f26da000700 1 -- 172.21.15.44:0/3758019993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 msgr2=0x7f26d41a47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.109+0000 7f26da000700 1 --2- 172.21.15.44:0/3758019993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d41a47b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f26c4015000 tx=0x7f26c4011940 comp rx=0 tx=0).stop 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.109+0000 7f26da000700 1 -- 172.21.15.44:0/3758019993 shutdown_connections 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.109+0000 7f26da000700 1 --2- 172.21.15.44:0/3758019993 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f26b803e4c0 0x7f26b80409d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.109+0000 7f26da000700 1 --2- 172.21.15.44:0/3758019993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26d410be00 0x7f26d41a47b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.109+0000 7f26da000700 1 -- 172.21.15.44:0/3758019993 >> 172.21.15.44:0/3758019993 conn(0x7f26d407f480 msgr2=0x7f26d4109610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:39.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.110+0000 7f26da000700 1 -- 172.21.15.44:0/3758019993 shutdown_connections 2024-05-27T09:22:39.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:38.110+0000 7f26da000700 1 -- 172.21.15.44:0/3758019993 wait complete. 2024-05-27T09:22:39.297 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying unmanaged mgr service... 2024-05-27T09:22:39.430 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:39 smithi044 ceph-mon[31618]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:22:39.430 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:39 smithi044 ceph-mon[31618]: Saving service mon spec with placement count:5 2024-05-27T09:22:39.430 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:39 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:41.142 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:40 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:41.143 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:40 smithi044 ceph-mon[31618]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:22:41.143 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:40 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:41.143 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:40 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:41.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-27T09:22:41.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.467+0000 7f0f43c00700 1 Processor -- start 2024-05-27T09:22:41.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.468+0000 7f0f43c00700 1 -- start start 2024-05-27T09:22:41.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.468+0000 7f0f43c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c10a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:41.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.468+0000 7f0f43c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3c10a730 con 0x7f0f3c109d30 2024-05-27T09:22:41.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.469+0000 7f0f41200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c10a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:41.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.469+0000 7f0f41200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c10a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49380/0 (socket says 172.21.15.44:49380) 2024-05-27T09:22:41.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.469+0000 7f0f41200700 1 -- 172.21.15.44:0/875469915 learned_addr learned my addr 172.21.15.44:0/875469915 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:41.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.469+0000 7f0f41200700 1 -- 172.21.15.44:0/875469915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3c10af00 con 0x7f0f3c109d30 2024-05-27T09:22:41.470 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.470+0000 7f0f41200700 1 --2- 172.21.15.44:0/875469915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c10a140 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f300098a0 tx=0x7f0f30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f3c07cb1f658da server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:41.471 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.470+0000 7f0f3b400700 1 -- 172.21.15.44:0/875469915 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f30004160 con 0x7f0f3c109d30 2024-05-27T09:22:41.471 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.470+0000 7f0f3b400700 1 -- 172.21.15.44:0/875469915 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f30004300 con 0x7f0f3c109d30 2024-05-27T09:22:41.471 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.470+0000 7f0f3b400700 1 -- 172.21.15.44:0/875469915 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f30011a20 con 0x7f0f3c109d30 2024-05-27T09:22:41.475 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.471+0000 7f0f43c00700 1 -- 172.21.15.44:0/875469915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 msgr2=0x7f0f3c10a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:41.476 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.471+0000 7f0f43c00700 1 --2- 172.21.15.44:0/875469915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c10a140 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f300098a0 tx=0x7f0f30009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:41.476 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.471+0000 7f0f43c00700 1 -- 172.21.15.44:0/875469915 shutdown_connections 2024-05-27T09:22:41.476 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.471+0000 7f0f43c00700 1 --2- 172.21.15.44:0/875469915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c10a140 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:41.476 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.471+0000 7f0f43c00700 1 -- 172.21.15.44:0/875469915 >> 172.21.15.44:0/875469915 conn(0x7f0f3c1053d0 msgr2=0x7f0f3c107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:41.476 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.471+0000 7f0f43c00700 1 -- 172.21.15.44:0/875469915 shutdown_connections 2024-05-27T09:22:41.476 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.471+0000 7f0f43c00700 1 -- 172.21.15.44:0/875469915 wait complete. 2024-05-27T09:22:41.476 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.472+0000 7f0f43c00700 1 Processor -- start 2024-05-27T09:22:41.476 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.473+0000 7f0f43c00700 1 -- start start 2024-05-27T09:22:41.476 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.473+0000 7f0f43c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c1a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:41.477 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.473+0000 7f0f43c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3c10e2b0 con 0x7f0f3c109d30 2024-05-27T09:22:41.477 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.473+0000 7f0f41200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c1a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:41.477 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.473+0000 7f0f41200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c1a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49388/0 (socket says 172.21.15.44:49388) 2024-05-27T09:22:41.477 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.473+0000 7f0f41200700 1 -- 172.21.15.44:0/3997655750 learned_addr learned my addr 172.21.15.44:0/3997655750 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:41.477 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.474+0000 7f0f41200700 1 -- 172.21.15.44:0/3997655750 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3c1a08b0 con 0x7f0f3c109d30 2024-05-27T09:22:41.477 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.474+0000 7f0f41200700 1 --2- 172.21.15.44:0/3997655750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c1a0370 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f30003e70 tx=0x7f0f30011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:41.477 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.474+0000 7f0f39600700 1 -- 172.21.15.44:0/3997655750 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f30003aa0 con 0x7f0f3c109d30 2024-05-27T09:22:41.477 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.474+0000 7f0f39600700 1 -- 172.21.15.44:0/3997655750 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f3001a8e0 con 0x7f0f3c109d30 2024-05-27T09:22:41.477 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.474+0000 7f0f39600700 1 -- 172.21.15.44:0/3997655750 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f30024870 con 0x7f0f3c109d30 2024-05-27T09:22:41.477 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.475+0000 7f0f43c00700 1 -- 172.21.15.44:0/3997655750 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f3c1a0b40 con 0x7f0f3c109d30 2024-05-27T09:22:41.477 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.475+0000 7f0f43c00700 1 -- 172.21.15.44:0/3997655750 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f3c1a3750 con 0x7f0f3c109d30 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.476+0000 7f0f43c00700 1 -- 172.21.15.44:0/3997655750 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f3c064e10 con 0x7f0f3c109d30 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.476+0000 7f0f39600700 1 -- 172.21.15.44:0/3997655750 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f0f30022020 con 0x7f0f3c109d30 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.477+0000 7f0f39600700 1 --2- 172.21.15.44:0/3997655750 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f0f2403e4c0 0x7f0f240409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.477+0000 7f0f39600700 1 -- 172.21.15.44:0/3997655750 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0f30053840 con 0x7f0f3c109d30 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.478+0000 7f0f3be00700 1 --2- 172.21.15.44:0/3997655750 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f0f2403e4c0 0x7f0f240409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.478+0000 7f0f3be00700 1 --2- 172.21.15.44:0/3997655750 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f0f2403e4c0 0x7f0f240409d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f28006fd0 tx=0x7f0f28006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.482+0000 7f0f39600700 1 -- 172.21.15.44:0/3997655750 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0f300212c0 con 0x7f0f3c109d30 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.746+0000 7f0f43c00700 1 -- 172.21.15.44:0/3997655750 --> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f0f3c1a39e0 con 0x7f0f2403e4c0 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.756+0000 7f0f39600700 1 -- 172.21.15.44:0/3997655750 <== mgr.14120 v2:172.21.15.44:6800/4254617371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0f3c1a39e0 con 0x7f0f2403e4c0 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.759+0000 7f0f43c00700 1 -- 172.21.15.44:0/3997655750 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f0f2403e4c0 msgr2=0x7f0f240409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.759+0000 7f0f43c00700 1 --2- 172.21.15.44:0/3997655750 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f0f2403e4c0 0x7f0f240409d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f28006fd0 tx=0x7f0f28006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:41.478 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.759+0000 7f0f43c00700 1 -- 172.21.15.44:0/3997655750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 msgr2=0x7f0f3c1a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:41.479 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.759+0000 7f0f43c00700 1 --2- 172.21.15.44:0/3997655750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c1a0370 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f30003e70 tx=0x7f0f30011aa0 comp rx=0 tx=0).stop 2024-05-27T09:22:41.479 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.759+0000 7f0f43c00700 1 -- 172.21.15.44:0/3997655750 shutdown_connections 2024-05-27T09:22:41.479 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.759+0000 7f0f43c00700 1 --2- 172.21.15.44:0/3997655750 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f0f2403e4c0 0x7f0f240409d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:41.479 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.759+0000 7f0f43c00700 1 --2- 172.21.15.44:0/3997655750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f3c109d30 0x7f0f3c1a0370 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:41.479 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.759+0000 7f0f43c00700 1 -- 172.21.15.44:0/3997655750 >> 172.21.15.44:0/3997655750 conn(0x7f0f3c1053d0 msgr2=0x7f0f3c105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:41.479 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.759+0000 7f0f43c00700 1 -- 172.21.15.44:0/3997655750 shutdown_connections 2024-05-27T09:22:41.479 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:40.760+0000 7f0f43c00700 1 -- 172.21.15.44:0/3997655750 wait complete. 2024-05-27T09:22:42.124 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:41 smithi044 ceph-mon[31618]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:22:42.124 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:41 smithi044 ceph-mon[31618]: Saving service mgr spec with placement count:2 2024-05-27T09:22:43.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.123+0000 7f5a66c00700 1 Processor -- start 2024-05-27T09:22:43.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.123+0000 7f5a66c00700 1 -- start start 2024-05-27T09:22:43.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.124+0000 7f5a66c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6010a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:43.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.124+0000 7f5a66c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6010a730 con 0x7f5a60109d30 2024-05-27T09:22:43.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.124+0000 7f5a5fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6010a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:43.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.124+0000 7f5a5fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6010a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49400/0 (socket says 172.21.15.44:49400) 2024-05-27T09:22:43.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.124+0000 7f5a5fe00700 1 -- 172.21.15.44:0/1187877442 learned_addr learned my addr 172.21.15.44:0/1187877442 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.125+0000 7f5a5fe00700 1 -- 172.21.15.44:0/1187877442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6010af00 con 0x7f5a60109d30 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.125+0000 7f5a5fe00700 1 --2- 172.21.15.44:0/1187877442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6010a140 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5a500098a0 tx=0x7f5a50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a950bfa8c36d5b12 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.125+0000 7f5a5ea00700 1 -- 172.21.15.44:0/1187877442 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a50004160 con 0x7f5a60109d30 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.126+0000 7f5a5ea00700 1 -- 172.21.15.44:0/1187877442 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a50004300 con 0x7f5a60109d30 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.126+0000 7f5a5ea00700 1 -- 172.21.15.44:0/1187877442 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a50011a20 con 0x7f5a60109d30 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.126+0000 7f5a66c00700 1 -- 172.21.15.44:0/1187877442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 msgr2=0x7f5a6010a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.126+0000 7f5a66c00700 1 --2- 172.21.15.44:0/1187877442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6010a140 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5a500098a0 tx=0x7f5a50009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.126+0000 7f5a66c00700 1 -- 172.21.15.44:0/1187877442 shutdown_connections 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.126+0000 7f5a66c00700 1 --2- 172.21.15.44:0/1187877442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6010a140 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.126+0000 7f5a66c00700 1 -- 172.21.15.44:0/1187877442 >> 172.21.15.44:0/1187877442 conn(0x7f5a601053d0 msgr2=0x7f5a60107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.127+0000 7f5a66c00700 1 -- 172.21.15.44:0/1187877442 shutdown_connections 2024-05-27T09:22:43.426 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.127+0000 7f5a66c00700 1 -- 172.21.15.44:0/1187877442 wait complete. 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.128+0000 7f5a66c00700 1 Processor -- start 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.128+0000 7f5a66c00700 1 -- start start 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.129+0000 7f5a66c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6019e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.129+0000 7f5a66c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6010e2b0 con 0x7f5a60109d30 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.129+0000 7f5a5fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6019e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.129+0000 7f5a5fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6019e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49408/0 (socket says 172.21.15.44:49408) 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.129+0000 7f5a5fe00700 1 -- 172.21.15.44:0/4249791438 learned_addr learned my addr 172.21.15.44:0/4249791438 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.130+0000 7f5a5fe00700 1 -- 172.21.15.44:0/4249791438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6019e680 con 0x7f5a60109d30 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.130+0000 7f5a5fe00700 1 --2- 172.21.15.44:0/4249791438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6019e140 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5a50003e70 tx=0x7f5a50011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.130+0000 7f5a5cc00700 1 -- 172.21.15.44:0/4249791438 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a50003aa0 con 0x7f5a60109d30 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.131+0000 7f5a5cc00700 1 -- 172.21.15.44:0/4249791438 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a5001a8e0 con 0x7f5a60109d30 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.131+0000 7f5a5cc00700 1 -- 172.21.15.44:0/4249791438 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a50024870 con 0x7f5a60109d30 2024-05-27T09:22:43.427 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.131+0000 7f5a66c00700 1 -- 172.21.15.44:0/4249791438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a6019e910 con 0x7f5a60109d30 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.131+0000 7f5a66c00700 1 -- 172.21.15.44:0/4249791438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a601a1520 con 0x7f5a60109d30 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.132+0000 7f5a5cc00700 1 -- 172.21.15.44:0/4249791438 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f5a50022020 con 0x7f5a60109d30 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.132+0000 7f5a66c00700 1 -- 172.21.15.44:0/4249791438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a60064e10 con 0x7f5a60109d30 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.133+0000 7f5a5cc00700 1 --2- 172.21.15.44:0/4249791438 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f5a4403e4c0 0x7f5a440409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.133+0000 7f5a5cc00700 1 -- 172.21.15.44:0/4249791438 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5a500538d0 con 0x7f5a60109d30 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.133+0000 7f5a5f400700 1 --2- 172.21.15.44:0/4249791438 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f5a4403e4c0 0x7f5a440409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.134+0000 7f5a5f400700 1 --2- 172.21.15.44:0/4249791438 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f5a4403e4c0 0x7f5a440409d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c006fd0 tx=0x7f5a4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.139+0000 7f5a5cc00700 1 -- 172.21.15.44:0/4249791438 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5a50029df0 con 0x7f5a60109d30 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.389+0000 7f5a66c00700 1 -- 172.21.15.44:0/4249791438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f5a601a1b20 con 0x7f5a60109d30 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.394+0000 7f5a5cc00700 1 -- 172.21.15.44:0/4249791438 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f5a50053220 con 0x7f5a60109d30 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.398+0000 7f5a66c00700 1 -- 172.21.15.44:0/4249791438 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f5a4403e4c0 msgr2=0x7f5a440409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:43.428 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.398+0000 7f5a66c00700 1 --2- 172.21.15.44:0/4249791438 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f5a4403e4c0 0x7f5a440409d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c006fd0 tx=0x7f5a4c006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:43.429 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.398+0000 7f5a66c00700 1 -- 172.21.15.44:0/4249791438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 msgr2=0x7f5a6019e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:43.429 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.398+0000 7f5a66c00700 1 --2- 172.21.15.44:0/4249791438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6019e140 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5a50003e70 tx=0x7f5a50011aa0 comp rx=0 tx=0).stop 2024-05-27T09:22:43.429 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.398+0000 7f5a66c00700 1 -- 172.21.15.44:0/4249791438 shutdown_connections 2024-05-27T09:22:43.429 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.398+0000 7f5a66c00700 1 --2- 172.21.15.44:0/4249791438 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f5a4403e4c0 0x7f5a440409d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:43.429 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.398+0000 7f5a66c00700 1 --2- 172.21.15.44:0/4249791438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5a60109d30 0x7f5a6019e140 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:43.429 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.398+0000 7f5a66c00700 1 -- 172.21.15.44:0/4249791438 >> 172.21.15.44:0/4249791438 conn(0x7f5a601053d0 msgr2=0x7f5a60105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:43.429 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.399+0000 7f5a66c00700 1 -- 172.21.15.44:0/4249791438 shutdown_connections 2024-05-27T09:22:43.429 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:42.399+0000 7f5a66c00700 1 -- 172.21.15.44:0/4249791438 wait complete. 2024-05-27T09:22:43.709 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:43 smithi044 ceph-mon[31618]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:22:43.709 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:43 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4249791438' entity='client.admin' 2024-05-27T09:22:43.709 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:43 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:45.035 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.076+0000 7ff0b7800700 1 Processor -- start 2024-05-27T09:22:45.035 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.076+0000 7ff0b7800700 1 -- start start 2024-05-27T09:22:45.035 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.076+0000 7ff0b7800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b0080530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:45.035 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.076+0000 7ff0b7800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b0080b40 con 0x7ff0b0080120 2024-05-27T09:22:45.035 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.077+0000 7ff0b4e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b0080530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.077+0000 7ff0b4e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b0080530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54322/0 (socket says 172.21.15.44:54322) 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.077+0000 7ff0b4e00700 1 -- 172.21.15.44:0/1289542269 learned_addr learned my addr 172.21.15.44:0/1289542269 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.078+0000 7ff0b4e00700 1 -- 172.21.15.44:0/1289542269 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b00812a0 con 0x7ff0b0080120 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.078+0000 7ff0b4e00700 1 --2- 172.21.15.44:0/1289542269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b0080530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff0a40098a0 tx=0x7ff0a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acebee1ab70d21b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.078+0000 7ff0af400700 1 -- 172.21.15.44:0/1289542269 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0a4004160 con 0x7ff0b0080120 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.079+0000 7ff0af400700 1 -- 172.21.15.44:0/1289542269 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0a4004300 con 0x7ff0b0080120 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.079+0000 7ff0af400700 1 -- 172.21.15.44:0/1289542269 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0a4011a20 con 0x7ff0b0080120 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.079+0000 7ff0b7800700 1 -- 172.21.15.44:0/1289542269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 msgr2=0x7ff0b0080530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.079+0000 7ff0b7800700 1 --2- 172.21.15.44:0/1289542269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b0080530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff0a40098a0 tx=0x7ff0a4009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.080+0000 7ff0b7800700 1 -- 172.21.15.44:0/1289542269 shutdown_connections 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.080+0000 7ff0b7800700 1 --2- 172.21.15.44:0/1289542269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b0080530 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.080+0000 7ff0b7800700 1 -- 172.21.15.44:0/1289542269 >> 172.21.15.44:0/1289542269 conn(0x7ff0b007f410 msgr2=0x7ff0b0109d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:45.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.080+0000 7ff0b7800700 1 -- 172.21.15.44:0/1289542269 shutdown_connections 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.080+0000 7ff0b7800700 1 -- 172.21.15.44:0/1289542269 wait complete. 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.081+0000 7ff0b7800700 1 Processor -- start 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.081+0000 7ff0b7800700 1 -- start start 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.081+0000 7ff0b7800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b01a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.081+0000 7ff0b7800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b0082660 con 0x7ff0b0080120 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.082+0000 7ff0b4e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b01a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.082+0000 7ff0b4e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b01a4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54324/0 (socket says 172.21.15.44:54324) 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.082+0000 7ff0b4e00700 1 -- 172.21.15.44:0/205741803 learned_addr learned my addr 172.21.15.44:0/205741803 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.082+0000 7ff0b4e00700 1 -- 172.21.15.44:0/205741803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b01a4cb0 con 0x7ff0b0080120 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.082+0000 7ff0b4e00700 1 --2- 172.21.15.44:0/205741803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b01a4770 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4000c00 tx=0x7ff0a4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.083+0000 7ff0ad600700 1 -- 172.21.15.44:0/205741803 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0a4003aa0 con 0x7ff0b0080120 2024-05-27T09:22:45.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.083+0000 7ff0ad600700 1 -- 172.21.15.44:0/205741803 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0a401b8f0 con 0x7ff0b0080120 2024-05-27T09:22:45.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.083+0000 7ff0b7800700 1 -- 172.21.15.44:0/205741803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b01a4f40 con 0x7ff0b0080120 2024-05-27T09:22:45.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.083+0000 7ff0ad600700 1 -- 172.21.15.44:0/205741803 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0a4023890 con 0x7ff0b0080120 2024-05-27T09:22:45.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.083+0000 7ff0b7800700 1 -- 172.21.15.44:0/205741803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b01a7b50 con 0x7ff0b0080120 2024-05-27T09:22:45.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.084+0000 7ff0b7800700 1 -- 172.21.15.44:0/205741803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0b0064e10 con 0x7ff0b0080120 2024-05-27T09:22:45.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.085+0000 7ff0ad600700 1 -- 172.21.15.44:0/205741803 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7ff0a401b440 con 0x7ff0b0080120 2024-05-27T09:22:45.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.085+0000 7ff0ad600700 1 --2- 172.21.15.44:0/205741803 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff09803e470 0x7ff098040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:45.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.085+0000 7ff0ad600700 1 -- 172.21.15.44:0/205741803 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff0a4053ed0 con 0x7ff0b0080120 2024-05-27T09:22:45.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.085+0000 7ff0afe00700 1 --2- 172.21.15.44:0/205741803 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff09803e470 0x7ff098040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:45.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.086+0000 7ff0afe00700 1 --2- 172.21.15.44:0/205741803 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff09803e470 0x7ff098040980 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff09c006fd0 tx=0x7ff09c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:45.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.091+0000 7ff0ad600700 1 -- 172.21.15.44:0/205741803 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff0a401f030 con 0x7ff0b0080120 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.332+0000 7ff0b7800700 1 -- 172.21.15.44:0/205741803 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7ff0b002e850 con 0x7ff0b0080120 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.335+0000 7ff0ad600700 1 -- 172.21.15.44:0/205741803 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7ff0a4016260 con 0x7ff0b0080120 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.339+0000 7ff0b7800700 1 -- 172.21.15.44:0/205741803 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff09803e470 msgr2=0x7ff098040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.339+0000 7ff0b7800700 1 --2- 172.21.15.44:0/205741803 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff09803e470 0x7ff098040980 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff09c006fd0 tx=0x7ff09c006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.339+0000 7ff0b7800700 1 -- 172.21.15.44:0/205741803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 msgr2=0x7ff0b01a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.339+0000 7ff0b7800700 1 --2- 172.21.15.44:0/205741803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b01a4770 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4000c00 tx=0x7ff0a4011aa0 comp rx=0 tx=0).stop 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.339+0000 7ff0b7800700 1 -- 172.21.15.44:0/205741803 shutdown_connections 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.339+0000 7ff0b7800700 1 --2- 172.21.15.44:0/205741803 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7ff09803e470 0x7ff098040980 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.339+0000 7ff0b7800700 1 --2- 172.21.15.44:0/205741803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0b0080120 0x7ff0b01a4770 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.339+0000 7ff0b7800700 1 -- 172.21.15.44:0/205741803 >> 172.21.15.44:0/205741803 conn(0x7ff0b007f410 msgr2=0x7ff0b019af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.340+0000 7ff0b7800700 1 -- 172.21.15.44:0/205741803 shutdown_connections 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:44.340+0000 7ff0b7800700 1 -- 172.21.15.44:0/205741803 wait complete. 2024-05-27T09:22:45.039 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling the dashboard module... 2024-05-27T09:22:45.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:45 smithi044 ceph-mon[31618]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:22:45.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:45 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/205741803' entity='client.admin' 2024-05-27T09:22:46.807 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:46 smithi044 ceph-mon[31618]: from='mgr.14120 172.21.15.44:0/1004056801' entity='mgr.x' 2024-05-27T09:22:46.807 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:46 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/206606903' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-27T09:22:47.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:46 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: ignoring --setuser ceph since I am not root 2024-05-27T09:22:47.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:46 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: ignoring --setgroup ceph since I am not root 2024-05-27T09:22:47.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:46.999+0000 7f6e6d161200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T09:22:47.579 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:47.270+0000 7f6e6d161200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:22:47.579 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:47.410+0000 7f6e6d161200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T09:22:47.836 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:47.824+0000 7f6e6d161200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T09:22:47.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:47 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/206606903' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-27T09:22:47.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:47 smithi044 ceph-mon[31618]: mgrmap e9: x(active, since 26s) 2024-05-27T09:22:48.167 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:47.944+0000 7f6e6d161200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T09:22:48.175 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.710+0000 7fc1e1200700 1 Processor -- start 2024-05-27T09:22:48.175 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.710+0000 7fc1e1200700 1 -- start start 2024-05-27T09:22:48.175 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.711+0000 7fc1e1200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc10a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:48.176 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.711+0000 7fc1e1200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc10a6d0 con 0x7fc1dc109cd0 2024-05-27T09:22:48.176 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.712+0000 7fc1d9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc10a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:48.176 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.712+0000 7fc1d9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc10a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54338/0 (socket says 172.21.15.44:54338) 2024-05-27T09:22:48.176 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.712+0000 7fc1d9e00700 1 -- 172.21.15.44:0/2320674340 learned_addr learned my addr 172.21.15.44:0/2320674340 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:48.176 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.712+0000 7fc1d9e00700 1 -- 172.21.15.44:0/2320674340 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc10aea0 con 0x7fc1dc109cd0 2024-05-27T09:22:48.177 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.713+0000 7fc1d9e00700 1 --2- 172.21.15.44:0/2320674340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc10a0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0098a0 tx=0x7fc1cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e413ecb621c8a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:48.177 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.713+0000 7fc1d9400700 1 -- 172.21.15.44:0/2320674340 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1cc004160 con 0x7fc1dc109cd0 2024-05-27T09:22:48.177 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.713+0000 7fc1d9400700 1 -- 172.21.15.44:0/2320674340 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1cc004300 con 0x7fc1dc109cd0 2024-05-27T09:22:48.177 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.713+0000 7fc1d9400700 1 -- 172.21.15.44:0/2320674340 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1cc011a20 con 0x7fc1dc109cd0 2024-05-27T09:22:48.178 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.714+0000 7fc1e1200700 1 -- 172.21.15.44:0/2320674340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 msgr2=0x7fc1dc10a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:48.178 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.714+0000 7fc1e1200700 1 --2- 172.21.15.44:0/2320674340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc10a0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0098a0 tx=0x7fc1cc009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:48.178 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.714+0000 7fc1e1200700 1 -- 172.21.15.44:0/2320674340 shutdown_connections 2024-05-27T09:22:48.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.714+0000 7fc1e1200700 1 --2- 172.21.15.44:0/2320674340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc10a0e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:48.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.714+0000 7fc1e1200700 1 -- 172.21.15.44:0/2320674340 >> 172.21.15.44:0/2320674340 conn(0x7fc1dc105370 msgr2=0x7fc1dc1077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:48.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.714+0000 7fc1e1200700 1 -- 172.21.15.44:0/2320674340 shutdown_connections 2024-05-27T09:22:48.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.714+0000 7fc1e1200700 1 -- 172.21.15.44:0/2320674340 wait complete. 2024-05-27T09:22:48.179 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.715+0000 7fc1e1200700 1 Processor -- start 2024-05-27T09:22:48.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.715+0000 7fc1e1200700 1 -- start start 2024-05-27T09:22:48.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.716+0000 7fc1e1200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc19e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:48.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.716+0000 7fc1e1200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc10e250 con 0x7fc1dc109cd0 2024-05-27T09:22:48.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.716+0000 7fc1d9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc19e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:48.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.716+0000 7fc1d9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc19e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54352/0 (socket says 172.21.15.44:54352) 2024-05-27T09:22:48.180 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.716+0000 7fc1d9e00700 1 -- 172.21.15.44:0/206606903 learned_addr learned my addr 172.21.15.44:0/206606903 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:48.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.717+0000 7fc1d9e00700 1 -- 172.21.15.44:0/206606903 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc19e690 con 0x7fc1dc109cd0 2024-05-27T09:22:48.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.717+0000 7fc1d9e00700 1 --2- 172.21.15.44:0/206606903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc19e150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009130 tx=0x7fc1cc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:48.181 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.717+0000 7fc1d3400700 1 -- 172.21.15.44:0/206606903 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1cc003aa0 con 0x7fc1dc109cd0 2024-05-27T09:22:48.182 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.718+0000 7fc1d3400700 1 -- 172.21.15.44:0/206606903 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1cc01a8e0 con 0x7fc1dc109cd0 2024-05-27T09:22:48.182 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.718+0000 7fc1e1200700 1 -- 172.21.15.44:0/206606903 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1dc19e920 con 0x7fc1dc109cd0 2024-05-27T09:22:48.183 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.718+0000 7fc1d3400700 1 -- 172.21.15.44:0/206606903 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1cc0248f0 con 0x7fc1dc109cd0 2024-05-27T09:22:48.183 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.718+0000 7fc1e1200700 1 -- 172.21.15.44:0/206606903 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1dc1a1530 con 0x7fc1dc109cd0 2024-05-27T09:22:48.183 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.719+0000 7fc1e1200700 1 -- 172.21.15.44:0/206606903 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1dc064e10 con 0x7fc1dc109cd0 2024-05-27T09:22:48.183 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.719+0000 7fc1d3400700 1 -- 172.21.15.44:0/206606903 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fc1cc022020 con 0x7fc1dc109cd0 2024-05-27T09:22:48.183 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.720+0000 7fc1d3400700 1 --2- 172.21.15.44:0/206606903 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fc1c003e4c0 0x7fc1c00409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:48.183 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.720+0000 7fc1d3400700 1 -- 172.21.15.44:0/206606903 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc1cc0538b0 con 0x7fc1dc109cd0 2024-05-27T09:22:48.183 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.720+0000 7fc1d1400700 1 --2- 172.21.15.44:0/206606903 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fc1c003e4c0 0x7fc1c00409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:48.185 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.721+0000 7fc1d1400700 1 --2- 172.21.15.44:0/206606903 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fc1c003e4c0 0x7fc1c00409d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4006fd0 tx=0x7fc1c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:48.185 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:45.725+0000 7fc1d3400700 1 -- 172.21.15.44:0/206606903 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc1cc029df0 con 0x7fc1dc109cd0 2024-05-27T09:22:48.185 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.031+0000 7fc1e1200700 1 -- 172.21.15.44:0/206606903 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fc1dc1a1b30 con 0x7fc1dc109cd0 2024-05-27T09:22:48.185 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.579+0000 7fc1d3400700 1 -- 172.21.15.44:0/206606903 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fc1cc053200 con 0x7fc1dc109cd0 2024-05-27T09:22:48.185 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.582+0000 7fc1d3400700 1 -- 172.21.15.44:0/206606903 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7fc1cc055cf0 con 0x7fc1dc109cd0 2024-05-27T09:22:48.185 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.584+0000 7fc1e1200700 1 -- 172.21.15.44:0/206606903 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fc1c003e4c0 msgr2=0x7fc1c00409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:48.185 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.584+0000 7fc1e1200700 1 --2- 172.21.15.44:0/206606903 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fc1c003e4c0 0x7fc1c00409d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4006fd0 tx=0x7fc1c4006e40 comp rx=0 tx=0).stop 2024-05-27T09:22:48.185 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.584+0000 7fc1e1200700 1 -- 172.21.15.44:0/206606903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 msgr2=0x7fc1dc19e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:48.185 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.584+0000 7fc1e1200700 1 --2- 172.21.15.44:0/206606903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc19e150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009130 tx=0x7fc1cc011aa0 comp rx=0 tx=0).stop 2024-05-27T09:22:48.185 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.584+0000 7fc1e1200700 1 -- 172.21.15.44:0/206606903 shutdown_connections 2024-05-27T09:22:48.186 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.584+0000 7fc1e1200700 1 --2- 172.21.15.44:0/206606903 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7fc1c003e4c0 0x7fc1c00409d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:48.186 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.584+0000 7fc1e1200700 1 --2- 172.21.15.44:0/206606903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1dc109cd0 0x7fc1dc19e150 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:48.186 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.584+0000 7fc1e1200700 1 -- 172.21.15.44:0/206606903 >> 172.21.15.44:0/206606903 conn(0x7fc1dc105370 msgr2=0x7fc1dc105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:48.186 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.585+0000 7fc1e1200700 1 -- 172.21.15.44:0/206606903 shutdown_connections 2024-05-27T09:22:48.186 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:46.586+0000 7fc1e1200700 1 -- 172.21.15.44:0/206606903 wait complete. 2024-05-27T09:22:49.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:49 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/35515545' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-27T09:22:49.876 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:49 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:49.636+0000 7f6e6d161200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T09:22:49.902 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-05-27T09:22:49.902 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-27T09:22:49.902 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T09:22:49.902 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-27T09:22:49.902 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-27T09:22:49.902 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-05-27T09:22:49.902 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.850+0000 7f90dd800700 1 Processor -- start 2024-05-27T09:22:49.902 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.851+0000 7f90dd800700 1 -- start start 2024-05-27T09:22:49.902 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.851+0000 7f90dd800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d807d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:49.902 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.851+0000 7f90dd800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f90d807d830 con 0x7f90d807ee30 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.852+0000 7f90d6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d807d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.852+0000 7f90d6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d807d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54384/0 (socket says 172.21.15.44:54384) 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.852+0000 7f90d6a00700 1 -- 172.21.15.44:0/749576545 learned_addr learned my addr 172.21.15.44:0/749576545 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.853+0000 7f90d6a00700 1 -- 172.21.15.44:0/749576545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d807d9b0 con 0x7f90d807ee30 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.853+0000 7f90d6a00700 1 --2- 172.21.15.44:0/749576545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d807d260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f90c80098a0 tx=0x7f90c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e31c5bf52bbd6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.853+0000 7f90d5600700 1 -- 172.21.15.44:0/749576545 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90c8004160 con 0x7f90d807ee30 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.853+0000 7f90d5600700 1 -- 172.21.15.44:0/749576545 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90c8004300 con 0x7f90d807ee30 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.853+0000 7f90d5600700 1 -- 172.21.15.44:0/749576545 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90c8011a20 con 0x7f90d807ee30 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.854+0000 7f90dd800700 1 -- 172.21.15.44:0/749576545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 msgr2=0x7f90d807d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.854+0000 7f90dd800700 1 --2- 172.21.15.44:0/749576545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d807d260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f90c80098a0 tx=0x7f90c8009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.854+0000 7f90dd800700 1 -- 172.21.15.44:0/749576545 shutdown_connections 2024-05-27T09:22:49.903 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.854+0000 7f90dd800700 1 --2- 172.21.15.44:0/749576545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d807d260 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:49.904 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.854+0000 7f90dd800700 1 -- 172.21.15.44:0/749576545 >> 172.21.15.44:0/749576545 conn(0x7f90d81053d0 msgr2=0x7f90d8107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:49.904 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.855+0000 7f90dd800700 1 -- 172.21.15.44:0/749576545 shutdown_connections 2024-05-27T09:22:49.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.855+0000 7f90dd800700 1 -- 172.21.15.44:0/749576545 wait complete. 2024-05-27T09:22:49.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.856+0000 7f90dd800700 1 Processor -- start 2024-05-27T09:22:49.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.856+0000 7f90dd800700 1 -- start start 2024-05-27T09:22:49.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.856+0000 7f90dd800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d81a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:49.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.856+0000 7f90dd800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f90d8110390 con 0x7f90d807ee30 2024-05-27T09:22:49.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.856+0000 7f90d6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d81a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:49.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.856+0000 7f90d6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d81a0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54400/0 (socket says 172.21.15.44:54400) 2024-05-27T09:22:49.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.856+0000 7f90d6a00700 1 -- 172.21.15.44:0/35515545 learned_addr learned my addr 172.21.15.44:0/35515545 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:49.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.857+0000 7f90d6a00700 1 -- 172.21.15.44:0/35515545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d81a0840 con 0x7f90d807ee30 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.857+0000 7f90d6a00700 1 --2- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d81a0300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f90c8000c00 tx=0x7f90c8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.857+0000 7f90cf400700 1 -- 172.21.15.44:0/35515545 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90c8003aa0 con 0x7f90d807ee30 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.857+0000 7f90cf400700 1 -- 172.21.15.44:0/35515545 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90c801b8f0 con 0x7f90d807ee30 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.857+0000 7f90cf400700 1 -- 172.21.15.44:0/35515545 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90c8023890 con 0x7f90d807ee30 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.857+0000 7f90dd800700 1 -- 172.21.15.44:0/35515545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90d81a0ad0 con 0x7f90d807ee30 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.858+0000 7f90dd800700 1 -- 172.21.15.44:0/35515545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90d81a36e0 con 0x7f90d807ee30 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.859+0000 7f90cf400700 1 -- 172.21.15.44:0/35515545 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f90c801b440 con 0x7f90d807ee30 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.859+0000 7f90dd800700 1 -- 172.21.15.44:0/35515545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90d8064e10 con 0x7f90d807ee30 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.859+0000 7f90cf400700 1 --2- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f90bc03e520 0x7f90bc040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.860+0000 7f90cf400700 1 -- 172.21.15.44:0/35515545 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f90c8053ed0 con 0x7f90d807ee30 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.860+0000 7f90d6000700 1 -- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f90bc03e520 msgr2=0x7f90bc040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4254617371 2024-05-27T09:22:49.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.860+0000 7f90d6000700 1 --2- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f90bc03e520 0x7f90bc040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:48.866+0000 7f90cf400700 1 -- 172.21.15.44:0/35515545 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f90c801f030 con 0x7f90d807ee30 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.061+0000 7f90d6000700 1 -- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f90bc03e520 msgr2=0x7f90bc040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4254617371 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.061+0000 7f90d6000700 1 --2- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f90bc03e520 0x7f90bc040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.176+0000 7f90dd800700 1 -- 172.21.15.44:0/35515545 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f90d81a38b0 con 0x7f90d807ee30 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.177+0000 7f90cf400700 1 -- 172.21.15.44:0/35515545 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f90d81a38b0 con 0x7f90d807ee30 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.180+0000 7f90dd800700 1 -- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f90bc03e520 msgr2=0x7f90bc040a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.180+0000 7f90dd800700 1 --2- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f90bc03e520 0x7f90bc040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.180+0000 7f90dd800700 1 -- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 msgr2=0x7f90d81a0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.180+0000 7f90dd800700 1 --2- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d81a0300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f90c8000c00 tx=0x7f90c8011aa0 comp rx=0 tx=0).stop 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.180+0000 7f90dd800700 1 -- 172.21.15.44:0/35515545 shutdown_connections 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.180+0000 7f90dd800700 1 --2- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f90bc03e520 0x7f90bc040a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:49.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.180+0000 7f90dd800700 1 --2- 172.21.15.44:0/35515545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90d807ee30 0x7f90d81a0300 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:49.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.180+0000 7f90dd800700 1 -- 172.21.15.44:0/35515545 >> 172.21.15.44:0/35515545 conn(0x7f90d81053d0 msgr2=0x7f90d8105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:49.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.180+0000 7f90dd800700 1 -- 172.21.15.44:0/35515545 shutdown_connections 2024-05-27T09:22:49.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:49.180+0000 7f90dd800700 1 -- 172.21.15.44:0/35515545 wait complete. 2024-05-27T09:22:49.911 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for the mgr to restart... 2024-05-27T09:22:49.911 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr epoch 9... 2024-05-27T09:22:50.212 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:49 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:49.876+0000 7f6e6d161200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T09:22:50.212 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:50 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:50.104+0000 7f6e6d161200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T09:22:50.792 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:50 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:50.520+0000 7f6e6d161200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T09:22:50.792 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:50 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:50.650+0000 7f6e6d161200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T09:22:51.185 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:50 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:50.913+0000 7f6e6d161200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T09:22:51.185 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:51 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:51.184+0000 7f6e6d161200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:22:52.314 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:52 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:52.011+0000 7f6e6d161200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T09:22:52.314 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:52 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:52.313+0000 7f6e6d161200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T09:22:52.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:52 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:52.911+0000 7f6e6d161200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T09:22:53.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:53 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:53.048+0000 7f6e6d161200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T09:22:53.916 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:53 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:53.533+0000 7f6e6d161200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T09:22:54.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:54.011+0000 7f6e6d161200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T09:22:54.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:54.143+0000 7f6e6d161200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T09:22:54.808 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:54.484+0000 7f6e6d161200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T09:22:54.808 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:54.672+0000 7f6e6d161200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T09:22:54.808 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:54.806+0000 7f6e6d161200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T09:22:55.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:55 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:55.219+0000 7f6e6d161200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T09:22:55.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:55 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:55.349+0000 7f6e6d161200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T09:22:56.448 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:56 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:56.146+0000 7f6e6d161200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T09:22:56.726 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:22:56 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:22:56.447+0000 7f6e6d161200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T09:22:56.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:56 smithi044 ceph-mon[31618]: Active manager daemon x restarted 2024-05-27T09:22:56.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:56 smithi044 ceph-mon[31618]: Activating manager daemon x 2024-05-27T09:22:56.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:56 smithi044 ceph-mon[31618]: osdmap e3: 0 total, 0 up, 0 in 2024-05-27T09:22:56.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:56 smithi044 ceph-mon[31618]: mgrmap e10: x(active, starting, since 0.0127275s) 2024-05-27T09:22:56.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:22:56.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T09:22:56.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T09:22:56.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T09:22:56.728 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T09:22:56.728 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:56 smithi044 ceph-mon[31618]: Manager daemon x is now available 2024-05-27T09:22:57.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:22:57.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T09:22:57.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T09:22:57.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:57 smithi044 ceph-mon[31618]: [27/May/2024:09:22:56] ENGINE Bus STARTING 2024-05-27T09:22:57.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:57 smithi044 ceph-mon[31618]: mgrmap e11: x(active, since 1.02167s) 2024-05-27T09:22:58.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-05-27T09:22:58.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-27T09:22:58.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-27T09:22:58.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-05-27T09:22:58.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.581+0000 7f3657200700 1 Processor -- start 2024-05-27T09:22:58.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.581+0000 7f3657200700 1 -- start start 2024-05-27T09:22:58.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.581+0000 7f3657200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f365810ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:58.150 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.581+0000 7f3657200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f365807ce80 con 0x7f365810a610 2024-05-27T09:22:58.150 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.582+0000 7f3655e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f365810ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:58.150 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.582+0000 7f3655e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f365810ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54414/0 (socket says 172.21.15.44:54414) 2024-05-27T09:22:58.150 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.582+0000 7f3655e00700 1 -- 172.21.15.44:0/2552533454 learned_addr learned my addr 172.21.15.44:0/2552533454 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:58.150 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.582+0000 7f3655e00700 1 -- 172.21.15.44:0/2552533454 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365810cf70 con 0x7f365810a610 2024-05-27T09:22:58.150 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.583+0000 7f3655e00700 1 --2- 172.21.15.44:0/2552533454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f365810ca30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f36480098a0 tx=0x7f3648009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7731c72ef2134 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:58.150 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.583+0000 7f3654a00700 1 -- 172.21.15.44:0/2552533454 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3648004160 con 0x7f365810a610 2024-05-27T09:22:58.151 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.583+0000 7f3654a00700 1 -- 172.21.15.44:0/2552533454 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3648004300 con 0x7f365810a610 2024-05-27T09:22:58.151 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.583+0000 7f3654a00700 1 -- 172.21.15.44:0/2552533454 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3648011a20 con 0x7f365810a610 2024-05-27T09:22:58.151 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.584+0000 7f3657200700 1 -- 172.21.15.44:0/2552533454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 msgr2=0x7f365810ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:58.151 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.584+0000 7f3657200700 1 --2- 172.21.15.44:0/2552533454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f365810ca30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f36480098a0 tx=0x7f3648009bb0 comp rx=0 tx=0).stop 2024-05-27T09:22:58.151 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.584+0000 7f3657200700 1 -- 172.21.15.44:0/2552533454 shutdown_connections 2024-05-27T09:22:58.151 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.584+0000 7f3657200700 1 --2- 172.21.15.44:0/2552533454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f365810ca30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:58.151 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.584+0000 7f3657200700 1 -- 172.21.15.44:0/2552533454 >> 172.21.15.44:0/2552533454 conn(0x7f3658104150 msgr2=0x7f36581065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:58.151 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.584+0000 7f3657200700 1 -- 172.21.15.44:0/2552533454 shutdown_connections 2024-05-27T09:22:58.151 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.585+0000 7f3657200700 1 -- 172.21.15.44:0/2552533454 wait complete. 2024-05-27T09:22:58.151 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.585+0000 7f3657200700 1 Processor -- start 2024-05-27T09:22:58.151 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.586+0000 7f3657200700 1 -- start start 2024-05-27T09:22:58.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.586+0000 7f3657200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f36581a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:58.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.586+0000 7f3657200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f365810df70 con 0x7f365810a610 2024-05-27T09:22:58.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.586+0000 7f3655e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f36581a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:58.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.586+0000 7f3655e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f36581a0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54430/0 (socket says 172.21.15.44:54430) 2024-05-27T09:22:58.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.586+0000 7f3655e00700 1 -- 172.21.15.44:0/884892944 learned_addr learned my addr 172.21.15.44:0/884892944 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:22:58.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.587+0000 7f3655e00700 1 -- 172.21.15.44:0/884892944 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36581a06b0 con 0x7f365810a610 2024-05-27T09:22:58.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.587+0000 7f3655e00700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f36581a0170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3648009520 tx=0x7f3648011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:58.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.587+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36480039e0 con 0x7f365810a610 2024-05-27T09:22:58.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.587+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f364801b8f0 con 0x7f365810a610 2024-05-27T09:22:58.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.587+0000 7f3657200700 1 -- 172.21.15.44:0/884892944 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36581a0940 con 0x7f365810a610 2024-05-27T09:22:58.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.588+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36480197d0 con 0x7f365810a610 2024-05-27T09:22:58.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.588+0000 7f3657200700 1 -- 172.21.15.44:0/884892944 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36581a3550 con 0x7f365810a610 2024-05-27T09:22:58.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.589+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f3648019a40 con 0x7f365810a610 2024-05-27T09:22:58.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.590+0000 7f364ea00700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 0x7f363c040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.590+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 --> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f363c041140 con 0x7f363c03e520 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.590+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f36480541b0 con 0x7f365810a610 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.590+0000 7f3655400700 1 -- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 msgr2=0x7f363c040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4254617371 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.590+0000 7f3655400700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 0x7f363c040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.791+0000 7f3655400700 1 -- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 msgr2=0x7f363c040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4254617371 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:50.791+0000 7f3655400700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 0x7f363c040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:51.192+0000 7f3655400700 1 -- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 msgr2=0x7f363c040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4254617371 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:51.192+0000 7f3655400700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 0x7f363c040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:51.993+0000 7f3655400700 1 -- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 msgr2=0x7f363c040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4254617371 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:51.993+0000 7f3655400700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 0x7f363c040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:53.595+0000 7f3655400700 1 -- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 msgr2=0x7f363c040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4254617371 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:53.595+0000 7f3655400700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 0x7f363c040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-27T09:22:58.159 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:56.460+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mgrmap(e 10) ==== 50002+0+0 (secure 0 0 0) 0x7f364802aa00 con 0x7f365810a610 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:56.460+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 msgr2=0x7f363c040a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:56.460+0000 7f364ea00700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 0x7f363c040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.463+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7f3648055b60 con 0x7f365810a610 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.463+0000 7f364ea00700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f3648053d40 0x7f363c044000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.463+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3648024e70 con 0x7f3648053d40 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.464+0000 7f3655400700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f3648053d40 0x7f363c044000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.466+0000 7f3655400700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f3648053d40 0x7f363c044000 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3644003a10 tx=0x7f3644009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.467+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f3648024e70 con 0x7f3648053d40 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.477+0000 7f3657200700 1 -- 172.21.15.44:0/884892944 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f3658064e10 con 0x7f3648053d40 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.478+0000 7f364ea00700 1 -- 172.21.15.44:0/884892944 <== mgr.14152 v2:172.21.15.44:6800/4077502151 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f3658064e10 con 0x7f3648053d40 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.479+0000 7f3657200700 1 -- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f3648053d40 msgr2=0x7f363c044000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:58.160 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.479+0000 7f3657200700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f3648053d40 0x7f363c044000 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3644003a10 tx=0x7f3644009270 comp rx=0 tx=0).stop 2024-05-27T09:22:58.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.479+0000 7f3657200700 1 -- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 msgr2=0x7f36581a0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:22:58.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.479+0000 7f3657200700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f36581a0170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3648009520 tx=0x7f3648011aa0 comp rx=0 tx=0).stop 2024-05-27T09:22:58.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.479+0000 7f3657200700 1 -- 172.21.15.44:0/884892944 shutdown_connections 2024-05-27T09:22:58.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.479+0000 7f3657200700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f3648053d40 0x7f363c044000 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:58.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.479+0000 7f3657200700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f365810a610 0x7f36581a0170 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:58.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.479+0000 7f3657200700 1 --2- 172.21.15.44:0/884892944 >> [v2:172.21.15.44:6800/4254617371,v1:172.21.15.44:6801/4254617371] conn(0x7f363c03e520 0x7f363c040a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:22:58.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.479+0000 7f3657200700 1 -- 172.21.15.44:0/884892944 >> 172.21.15.44:0/884892944 conn(0x7f3658104150 msgr2=0x7f3658108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:22:58.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.479+0000 7f3657200700 1 -- 172.21.15.44:0/884892944 shutdown_connections 2024-05-27T09:22:58.161 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:57.480+0000 7f3657200700 1 -- 172.21.15.44:0/884892944 wait complete. 2024-05-27T09:22:58.162 INFO:teuthology.orchestra.run.smithi044.stdout:mgr epoch 9 is available 2024-05-27T09:22:58.162 INFO:teuthology.orchestra.run.smithi044.stdout:Generating a dashboard self-signed certificate... 2024-05-27T09:22:58.660 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:58 smithi044 ceph-mon[31618]: [27/May/2024:09:22:56] ENGINE Serving on http://172.21.15.44:8765 2024-05-27T09:22:58.660 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:58 smithi044 ceph-mon[31618]: [27/May/2024:09:22:57] ENGINE Serving on https://172.21.15.44:7150 2024-05-27T09:22:58.661 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:58 smithi044 ceph-mon[31618]: [27/May/2024:09:22:57] ENGINE Bus STARTED 2024-05-27T09:22:58.661 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:58 smithi044 ceph-mon[31618]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-27T09:22:58.661 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:58 smithi044 ceph-mon[31618]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-27T09:22:59.547 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:59 smithi044 ceph-mon[31618]: mgrmap e12: x(active, since 2s) 2024-05-27T09:22:59.548 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:22:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:00.391 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-27T09:23:00.391 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.286+0000 7fd875a00700 1 Processor -- start 2024-05-27T09:23:00.391 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.287+0000 7fd875a00700 1 -- start start 2024-05-27T09:23:00.391 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.287+0000 7fd875a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd87010cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:00.391 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.287+0000 7fd875a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd87007d8b0 con 0x7fd87010a7e0 2024-05-27T09:23:00.392 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.288+0000 7fd86ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd87010cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:00.392 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.288+0000 7fd86ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd87010cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59128/0 (socket says 172.21.15.44:59128) 2024-05-27T09:23:00.392 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.288+0000 7fd86ea00700 1 -- 172.21.15.44:0/3755480323 learned_addr learned my addr 172.21.15.44:0/3755480323 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:00.392 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.289+0000 7fd86ea00700 1 -- 172.21.15.44:0/3755480323 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87010d140 con 0x7fd87010a7e0 2024-05-27T09:23:00.392 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.289+0000 7fd86ea00700 1 --2- 172.21.15.44:0/3755480323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd87010cc00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd8600098a0 tx=0x7fd860009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70d7ec1f1c746bf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:00.392 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.289+0000 7fd86e000700 1 -- 172.21.15.44:0/3755480323 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd860004160 con 0x7fd87010a7e0 2024-05-27T09:23:00.393 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.289+0000 7fd86e000700 1 -- 172.21.15.44:0/3755480323 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd860004300 con 0x7fd87010a7e0 2024-05-27T09:23:00.393 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.289+0000 7fd86e000700 1 -- 172.21.15.44:0/3755480323 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd860011a20 con 0x7fd87010a7e0 2024-05-27T09:23:00.393 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.290+0000 7fd875a00700 1 -- 172.21.15.44:0/3755480323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 msgr2=0x7fd87010cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:00.393 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.290+0000 7fd875a00700 1 --2- 172.21.15.44:0/3755480323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd87010cc00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd8600098a0 tx=0x7fd860009bb0 comp rx=0 tx=0).stop 2024-05-27T09:23:00.393 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.290+0000 7fd875a00700 1 -- 172.21.15.44:0/3755480323 shutdown_connections 2024-05-27T09:23:00.393 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.290+0000 7fd875a00700 1 --2- 172.21.15.44:0/3755480323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd87010cc00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:00.394 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.290+0000 7fd875a00700 1 -- 172.21.15.44:0/3755480323 >> 172.21.15.44:0/3755480323 conn(0x7fd8701042e0 msgr2=0x7fd870106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:00.394 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.290+0000 7fd875a00700 1 -- 172.21.15.44:0/3755480323 shutdown_connections 2024-05-27T09:23:00.394 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.290+0000 7fd875a00700 1 -- 172.21.15.44:0/3755480323 wait complete. 2024-05-27T09:23:00.394 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.291+0000 7fd875a00700 1 Processor -- start 2024-05-27T09:23:00.394 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.292+0000 7fd875a00700 1 -- start start 2024-05-27T09:23:00.394 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.292+0000 7fd875a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd870109f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:00.394 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.292+0000 7fd875a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd87010e140 con 0x7fd87010a7e0 2024-05-27T09:23:00.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.292+0000 7fd86ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd870109f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:00.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.292+0000 7fd86ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd870109f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59130/0 (socket says 172.21.15.44:59130) 2024-05-27T09:23:00.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.292+0000 7fd86ea00700 1 -- 172.21.15.44:0/361389716 learned_addr learned my addr 172.21.15.44:0/361389716 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:00.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.293+0000 7fd86ea00700 1 -- 172.21.15.44:0/361389716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87010a4a0 con 0x7fd87010a7e0 2024-05-27T09:23:00.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.293+0000 7fd86ea00700 1 --2- 172.21.15.44:0/361389716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd870109f60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd8600119f0 tx=0x7fd860011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:00.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.293+0000 7fd867400700 1 -- 172.21.15.44:0/361389716 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8600039e0 con 0x7fd87010a7e0 2024-05-27T09:23:00.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.294+0000 7fd867400700 1 -- 172.21.15.44:0/361389716 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd86001b8f0 con 0x7fd87010a7e0 2024-05-27T09:23:00.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.294+0000 7fd875a00700 1 -- 172.21.15.44:0/361389716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8701086f0 con 0x7fd87010a7e0 2024-05-27T09:23:00.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.294+0000 7fd867400700 1 -- 172.21.15.44:0/361389716 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8600237a0 con 0x7fd87010a7e0 2024-05-27T09:23:00.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.294+0000 7fd875a00700 1 -- 172.21.15.44:0/361389716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd870108bd0 con 0x7fd87010a7e0 2024-05-27T09:23:00.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.295+0000 7fd875a00700 1 -- 172.21.15.44:0/361389716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd870064e10 con 0x7fd87010a7e0 2024-05-27T09:23:00.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.295+0000 7fd867400700 1 -- 172.21.15.44:0/361389716 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7fd86002c3f0 con 0x7fd87010a7e0 2024-05-27T09:23:00.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.296+0000 7fd867400700 1 --2- 172.21.15.44:0/361389716 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd85403e470 0x7fd854040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:00.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.296+0000 7fd867400700 1 -- 172.21.15.44:0/361389716 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd860054140 con 0x7fd87010a7e0 2024-05-27T09:23:00.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.297+0000 7fd867e00700 1 --2- 172.21.15.44:0/361389716 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd85403e470 0x7fd854040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:00.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.298+0000 7fd867e00700 1 --2- 172.21.15.44:0/361389716 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd85403e470 0x7fd854040980 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd858006fd0 tx=0x7fd858006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:00.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.302+0000 7fd867400700 1 -- 172.21.15.44:0/361389716 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd86001f030 con 0x7fd87010a7e0 2024-05-27T09:23:00.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.546+0000 7fd875a00700 1 -- 172.21.15.44:0/361389716 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fd870109300 con 0x7fd85403e470 2024-05-27T09:23:00.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.734+0000 7fd867400700 1 -- 172.21.15.44:0/361389716 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fd870109300 con 0x7fd85403e470 2024-05-27T09:23:00.398 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.739+0000 7fd875a00700 1 -- 172.21.15.44:0/361389716 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd85403e470 msgr2=0x7fd854040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:00.398 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.739+0000 7fd875a00700 1 --2- 172.21.15.44:0/361389716 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd85403e470 0x7fd854040980 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd858006fd0 tx=0x7fd858006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:00.398 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.739+0000 7fd875a00700 1 -- 172.21.15.44:0/361389716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 msgr2=0x7fd870109f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:00.398 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.739+0000 7fd875a00700 1 --2- 172.21.15.44:0/361389716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd870109f60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd8600119f0 tx=0x7fd860011ad0 comp rx=0 tx=0).stop 2024-05-27T09:23:00.398 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.739+0000 7fd875a00700 1 -- 172.21.15.44:0/361389716 shutdown_connections 2024-05-27T09:23:00.398 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.739+0000 7fd875a00700 1 --2- 172.21.15.44:0/361389716 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd85403e470 0x7fd854040980 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:00.399 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.739+0000 7fd875a00700 1 --2- 172.21.15.44:0/361389716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd87010a7e0 0x7fd870109f60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:00.399 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.739+0000 7fd875a00700 1 -- 172.21.15.44:0/361389716 >> 172.21.15.44:0/361389716 conn(0x7fd8701042e0 msgr2=0x7fd870105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:00.399 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.739+0000 7fd875a00700 1 -- 172.21.15.44:0/361389716 shutdown_connections 2024-05-27T09:23:00.399 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:22:59.739+0000 7fd875a00700 1 -- 172.21.15.44:0/361389716 wait complete. 2024-05-27T09:23:00.399 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial admin user... 2024-05-27T09:23:00.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:00 smithi044 ceph-mon[31618]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:23:00.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:00 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:00.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:00 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:02.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$xrP9zz/SkNTioCFTLpedT.ebSO0OpwwNpDSoHnFyVUrOLnlhNT7nG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1716801781, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-27T09:23:02.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.075+0000 7ff376600700 1 Processor -- start 2024-05-27T09:23:02.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.075+0000 7ff376600700 1 -- start start 2024-05-27T09:23:02.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.076+0000 7ff376600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff37010cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.076+0000 7ff376600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff37007ce80 con 0x7ff37010a820 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.076+0000 7ff36f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff37010cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.076+0000 7ff36f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff37010cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59142/0 (socket says 172.21.15.44:59142) 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.076+0000 7ff36f400700 1 -- 172.21.15.44:0/1769141271 learned_addr learned my addr 172.21.15.44:0/1769141271 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.077+0000 7ff36f400700 1 -- 172.21.15.44:0/1769141271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37010d180 con 0x7ff37010a820 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.077+0000 7ff36f400700 1 --2- 172.21.15.44:0/1769141271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff37010cc40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff3600098a0 tx=0x7ff360009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd07c6bb9f09fff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.077+0000 7ff36e000700 1 -- 172.21.15.44:0/1769141271 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff360004160 con 0x7ff37010a820 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.077+0000 7ff36e000700 1 -- 172.21.15.44:0/1769141271 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff360004300 con 0x7ff37010a820 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.078+0000 7ff36e000700 1 -- 172.21.15.44:0/1769141271 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff360011a20 con 0x7ff37010a820 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.079+0000 7ff376600700 1 -- 172.21.15.44:0/1769141271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 msgr2=0x7ff37010cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.079+0000 7ff376600700 1 --2- 172.21.15.44:0/1769141271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff37010cc40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff3600098a0 tx=0x7ff360009bb0 comp rx=0 tx=0).stop 2024-05-27T09:23:02.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.079+0000 7ff376600700 1 -- 172.21.15.44:0/1769141271 shutdown_connections 2024-05-27T09:23:02.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.079+0000 7ff376600700 1 --2- 172.21.15.44:0/1769141271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff37010cc40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:02.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.079+0000 7ff376600700 1 -- 172.21.15.44:0/1769141271 >> 172.21.15.44:0/1769141271 conn(0x7ff370104340 msgr2=0x7ff370106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:02.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.079+0000 7ff376600700 1 -- 172.21.15.44:0/1769141271 shutdown_connections 2024-05-27T09:23:02.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.079+0000 7ff376600700 1 -- 172.21.15.44:0/1769141271 wait complete. 2024-05-27T09:23:02.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.080+0000 7ff376600700 1 Processor -- start 2024-05-27T09:23:02.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.080+0000 7ff376600700 1 -- start start 2024-05-27T09:23:02.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.081+0000 7ff376600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff370109fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.081+0000 7ff376600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff37010e180 con 0x7ff37010a820 2024-05-27T09:23:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.081+0000 7ff36f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff370109fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.081+0000 7ff36f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff370109fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59144/0 (socket says 172.21.15.44:59144) 2024-05-27T09:23:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.081+0000 7ff36f400700 1 -- 172.21.15.44:0/1130504919 learned_addr learned my addr 172.21.15.44:0/1130504919 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.082+0000 7ff36f400700 1 -- 172.21.15.44:0/1130504919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37010a4e0 con 0x7ff37010a820 2024-05-27T09:23:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.082+0000 7ff36f400700 1 --2- 172.21.15.44:0/1130504919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff370109fa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff360000c00 tx=0x7ff360011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.082+0000 7ff367e00700 1 -- 172.21.15.44:0/1130504919 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff360003aa0 con 0x7ff37010a820 2024-05-27T09:23:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.082+0000 7ff367e00700 1 -- 172.21.15.44:0/1130504919 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff36001b8f0 con 0x7ff37010a820 2024-05-27T09:23:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.082+0000 7ff367e00700 1 -- 172.21.15.44:0/1130504919 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff360023890 con 0x7ff37010a820 2024-05-27T09:23:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.082+0000 7ff376600700 1 -- 172.21.15.44:0/1130504919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff370108730 con 0x7ff37010a820 2024-05-27T09:23:02.307 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.082+0000 7ff376600700 1 -- 172.21.15.44:0/1130504919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff370108c10 con 0x7ff37010a820 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.084+0000 7ff376600700 1 -- 172.21.15.44:0/1130504919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff370064e10 con 0x7ff37010a820 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.084+0000 7ff367e00700 1 -- 172.21.15.44:0/1130504919 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7ff36002c3f0 con 0x7ff37010a820 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.085+0000 7ff367e00700 1 --2- 172.21.15.44:0/1130504919 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff35403e520 0x7ff354040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.085+0000 7ff367e00700 1 -- 172.21.15.44:0/1130504919 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff3600540c0 con 0x7ff37010a820 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.085+0000 7ff36ea00700 1 --2- 172.21.15.44:0/1130504919 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff35403e520 0x7ff354040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.086+0000 7ff36ea00700 1 --2- 172.21.15.44:0/1130504919 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff35403e520 0x7ff354040a30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff35c006fd0 tx=0x7ff35c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.091+0000 7ff367e00700 1 -- 172.21.15.44:0/1130504919 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff36001f030 con 0x7ff37010a820 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.353+0000 7ff376600700 1 -- 172.21.15.44:0/1130504919 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7ff370109320 con 0x7ff35403e520 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.618+0000 7ff367e00700 1 -- 172.21.15.44:0/1130504919 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7ff370109320 con 0x7ff35403e520 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.621+0000 7ff376600700 1 -- 172.21.15.44:0/1130504919 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff35403e520 msgr2=0x7ff354040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.622+0000 7ff376600700 1 --2- 172.21.15.44:0/1130504919 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff35403e520 0x7ff354040a30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff35c006fd0 tx=0x7ff35c006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:02.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.622+0000 7ff376600700 1 -- 172.21.15.44:0/1130504919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 msgr2=0x7ff370109fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:02.309 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.622+0000 7ff376600700 1 --2- 172.21.15.44:0/1130504919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff370109fa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff360000c00 tx=0x7ff360011aa0 comp rx=0 tx=0).stop 2024-05-27T09:23:02.309 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.622+0000 7ff376600700 1 -- 172.21.15.44:0/1130504919 shutdown_connections 2024-05-27T09:23:02.309 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.622+0000 7ff376600700 1 --2- 172.21.15.44:0/1130504919 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff35403e520 0x7ff354040a30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:02.309 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.622+0000 7ff376600700 1 --2- 172.21.15.44:0/1130504919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff37010a820 0x7ff370109fa0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:02.309 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.622+0000 7ff376600700 1 -- 172.21.15.44:0/1130504919 >> 172.21.15.44:0/1130504919 conn(0x7ff370104340 msgr2=0x7ff370105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:02.309 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.623+0000 7ff376600700 1 -- 172.21.15.44:0/1130504919 shutdown_connections 2024-05-27T09:23:02.309 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:01.623+0000 7ff376600700 1 -- 172.21.15.44:0/1130504919 wait complete. 2024-05-27T09:23:02.309 INFO:teuthology.orchestra.run.smithi044.stdout:Fetching dashboard port number... 2024-05-27T09:23:02.594 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:02 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:02.594 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:02 smithi044 ceph-mon[31618]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:23:02.594 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:02 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:03.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:03 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2464443394' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-27T09:23:04.043 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 8443 2024-05-27T09:23:04.044 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.959+0000 7ffa40800700 1 Processor -- start 2024-05-27T09:23:04.045 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.960+0000 7ffa40800700 1 -- start start 2024-05-27T09:23:04.045 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.960+0000 7ffa40800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa3810ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:04.045 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.960+0000 7ffa40800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3807ce00 con 0x7ffa3810a8a0 2024-05-27T09:23:04.045 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.961+0000 7ffa3de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa3810ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:04.045 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.961+0000 7ffa3de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa3810ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44302/0 (socket says 172.21.15.44:44302) 2024-05-27T09:23:04.045 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.961+0000 7ffa3de00700 1 -- 172.21.15.44:0/3877841670 learned_addr learned my addr 172.21.15.44:0/3877841670 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:04.045 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.961+0000 7ffa3de00700 1 -- 172.21.15.44:0/3877841670 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3810d200 con 0x7ffa3810a8a0 2024-05-27T09:23:04.045 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.961+0000 7ffa3de00700 1 --2- 172.21.15.44:0/3877841670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa3810ccc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0098a0 tx=0x7ffa2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3251484626b8c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:04.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.962+0000 7ffa3ca00700 1 -- 172.21.15.44:0/3877841670 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa2c004160 con 0x7ffa3810a8a0 2024-05-27T09:23:04.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.962+0000 7ffa3ca00700 1 -- 172.21.15.44:0/3877841670 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa2c004300 con 0x7ffa3810a8a0 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.962+0000 7ffa3ca00700 1 -- 172.21.15.44:0/3877841670 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa2c011a20 con 0x7ffa3810a8a0 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.963+0000 7ffa40800700 1 -- 172.21.15.44:0/3877841670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 msgr2=0x7ffa3810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.963+0000 7ffa40800700 1 --2- 172.21.15.44:0/3877841670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa3810ccc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0098a0 tx=0x7ffa2c009bb0 comp rx=0 tx=0).stop 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.963+0000 7ffa40800700 1 -- 172.21.15.44:0/3877841670 shutdown_connections 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.963+0000 7ffa40800700 1 --2- 172.21.15.44:0/3877841670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa3810ccc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.963+0000 7ffa40800700 1 -- 172.21.15.44:0/3877841670 >> 172.21.15.44:0/3877841670 conn(0x7ffa38104340 msgr2=0x7ffa38106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.963+0000 7ffa40800700 1 -- 172.21.15.44:0/3877841670 shutdown_connections 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.963+0000 7ffa40800700 1 -- 172.21.15.44:0/3877841670 wait complete. 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.964+0000 7ffa40800700 1 Processor -- start 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.964+0000 7ffa40800700 1 -- start start 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.965+0000 7ffa40800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa381a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.965+0000 7ffa40800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3810e1a0 con 0x7ffa3810a8a0 2024-05-27T09:23:04.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.965+0000 7ffa3de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa381a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.965+0000 7ffa3de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa381a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44304/0 (socket says 172.21.15.44:44304) 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.965+0000 7ffa3de00700 1 -- 172.21.15.44:0/2464443394 learned_addr learned my addr 172.21.15.44:0/2464443394 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.966+0000 7ffa3de00700 1 -- 172.21.15.44:0/2464443394 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa381a0830 con 0x7ffa3810a8a0 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.966+0000 7ffa3de00700 1 --2- 172.21.15.44:0/2464443394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa381a02f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c000c00 tx=0x7ffa2c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.966+0000 7ffa32a00700 1 -- 172.21.15.44:0/2464443394 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa2c003aa0 con 0x7ffa3810a8a0 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.966+0000 7ffa32a00700 1 -- 172.21.15.44:0/2464443394 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa2c01b8f0 con 0x7ffa3810a8a0 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.966+0000 7ffa40800700 1 -- 172.21.15.44:0/2464443394 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa381a0ac0 con 0x7ffa3810a8a0 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.966+0000 7ffa32a00700 1 -- 172.21.15.44:0/2464443394 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa2c023890 con 0x7ffa3810a8a0 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.966+0000 7ffa40800700 1 -- 172.21.15.44:0/2464443394 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa381a36d0 con 0x7ffa3810a8a0 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.968+0000 7ffa40800700 1 -- 172.21.15.44:0/2464443394 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa38064e10 con 0x7ffa3810a8a0 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.968+0000 7ffa32a00700 1 -- 172.21.15.44:0/2464443394 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7ffa2c01b440 con 0x7ffa3810a8a0 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.968+0000 7ffa32a00700 1 --2- 172.21.15.44:0/2464443394 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa2003e520 0x7ffa20040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:04.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.968+0000 7ffa32a00700 1 -- 172.21.15.44:0/2464443394 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ffa2c053fe0 con 0x7ffa3810a8a0 2024-05-27T09:23:04.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.969+0000 7ffa3d400700 1 --2- 172.21.15.44:0/2464443394 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa2003e520 0x7ffa20040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:04.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.969+0000 7ffa3d400700 1 --2- 172.21.15.44:0/2464443394 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa2003e520 0x7ffa20040a30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffa28006fd0 tx=0x7ffa28006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:04.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:02.974+0000 7ffa32a00700 1 -- 172.21.15.44:0/2464443394 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ffa2c01f030 con 0x7ffa3810a8a0 2024-05-27T09:23:04.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.217+0000 7ffa40800700 1 -- 172.21.15.44:0/2464443394 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7ffa381a38a0 con 0x7ffa3810a8a0 2024-05-27T09:23:04.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.218+0000 7ffa32a00700 1 -- 172.21.15.44:0/2464443394 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7ffa2c00be50 con 0x7ffa3810a8a0 2024-05-27T09:23:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.222+0000 7ffa40800700 1 -- 172.21.15.44:0/2464443394 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa2003e520 msgr2=0x7ffa20040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.222+0000 7ffa40800700 1 --2- 172.21.15.44:0/2464443394 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa2003e520 0x7ffa20040a30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffa28006fd0 tx=0x7ffa28006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.222+0000 7ffa40800700 1 -- 172.21.15.44:0/2464443394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 msgr2=0x7ffa381a02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.222+0000 7ffa40800700 1 --2- 172.21.15.44:0/2464443394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa381a02f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c000c00 tx=0x7ffa2c011aa0 comp rx=0 tx=0).stop 2024-05-27T09:23:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.223+0000 7ffa40800700 1 -- 172.21.15.44:0/2464443394 shutdown_connections 2024-05-27T09:23:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.223+0000 7ffa40800700 1 --2- 172.21.15.44:0/2464443394 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa2003e520 0x7ffa20040a30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.223+0000 7ffa40800700 1 --2- 172.21.15.44:0/2464443394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3810a8a0 0x7ffa381a02f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.223+0000 7ffa40800700 1 -- 172.21.15.44:0/2464443394 >> 172.21.15.44:0/2464443394 conn(0x7ffa38104340 msgr2=0x7ffa38108790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.223+0000 7ffa40800700 1 -- 172.21.15.44:0/2464443394 shutdown_connections 2024-05-27T09:23:04.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:03.223+0000 7ffa40800700 1 -- 172.21.15.44:0/2464443394 wait complete. 2024-05-27T09:23:04.081 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T09:23:04.082 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-05-27T09:23:04.104 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T09:23:04.104 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-05-27T09:23:04.104 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-05-27T09:23:04.104 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-27T09:23:04.108 INFO:teuthology.orchestra.run.smithi044.stdout:Ceph Dashboard is now available at: 2024-05-27T09:23:04.108 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:04.108 INFO:teuthology.orchestra.run.smithi044.stdout: URL: https://smithi044.front.sepia.ceph.com:8443/ 2024-05-27T09:23:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: User: admin 2024-05-27T09:23:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: Password: r35xml3v0k 2024-05-27T09:23:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:04.109 INFO:teuthology.orchestra.run.smithi044.stdout:Saving cluster configuration to /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config directory 2024-05-27T09:23:04.110 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling autotune for osd_memory_target 2024-05-27T09:23:04.560 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:04 smithi044 ceph-mon[31618]: mgrmap e13: x(active, since 6s) 2024-05-27T09:23:06.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.247+0000 7fc61ac00700 1 Processor -- start 2024-05-27T09:23:06.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.247+0000 7fc61ac00700 1 -- start start 2024-05-27T09:23:06.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.248+0000 7fc61ac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc61410a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:06.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.248+0000 7fc61ac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc61410a7b0 con 0x7fc614109db0 2024-05-27T09:23:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.248+0000 7fc613e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc61410a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.248+0000 7fc613e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc61410a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44310/0 (socket says 172.21.15.44:44310) 2024-05-27T09:23:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.248+0000 7fc613e00700 1 -- 172.21.15.44:0/2137878742 learned_addr learned my addr 172.21.15.44:0/2137878742 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.249+0000 7fc613e00700 1 -- 172.21.15.44:0/2137878742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61410af80 con 0x7fc614109db0 2024-05-27T09:23:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.249+0000 7fc613e00700 1 --2- 172.21.15.44:0/2137878742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc61410a1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc6040098a0 tx=0x7fc604009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212d92cbe94d024 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:06.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.250+0000 7fc612a00700 1 -- 172.21.15.44:0/2137878742 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc604004160 con 0x7fc614109db0 2024-05-27T09:23:06.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.250+0000 7fc612a00700 1 -- 172.21.15.44:0/2137878742 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc604004300 con 0x7fc614109db0 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.250+0000 7fc612a00700 1 -- 172.21.15.44:0/2137878742 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc604011a20 con 0x7fc614109db0 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.250+0000 7fc61ac00700 1 -- 172.21.15.44:0/2137878742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 msgr2=0x7fc61410a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.250+0000 7fc61ac00700 1 --2- 172.21.15.44:0/2137878742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc61410a1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc6040098a0 tx=0x7fc604009bb0 comp rx=0 tx=0).stop 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.251+0000 7fc61ac00700 1 -- 172.21.15.44:0/2137878742 shutdown_connections 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.251+0000 7fc61ac00700 1 --2- 172.21.15.44:0/2137878742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc61410a1c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.251+0000 7fc61ac00700 1 -- 172.21.15.44:0/2137878742 >> 172.21.15.44:0/2137878742 conn(0x7fc6141053d0 msgr2=0x7fc614107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.251+0000 7fc61ac00700 1 -- 172.21.15.44:0/2137878742 shutdown_connections 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.251+0000 7fc61ac00700 1 -- 172.21.15.44:0/2137878742 wait complete. 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.252+0000 7fc61ac00700 1 Processor -- start 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.252+0000 7fc61ac00700 1 -- start start 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.252+0000 7fc61ac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc6141a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:06.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.252+0000 7fc61ac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc61410e330 con 0x7fc614109db0 2024-05-27T09:23:06.112 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.253+0000 7fc613e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc6141a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:06.112 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.253+0000 7fc613e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc6141a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44316/0 (socket says 172.21.15.44:44316) 2024-05-27T09:23:06.113 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.253+0000 7fc613e00700 1 -- 172.21.15.44:0/705969112 learned_addr learned my addr 172.21.15.44:0/705969112 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:06.113 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.253+0000 7fc613e00700 1 -- 172.21.15.44:0/705969112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6141a0930 con 0x7fc614109db0 2024-05-27T09:23:06.113 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.254+0000 7fc613e00700 1 --2- 172.21.15.44:0/705969112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc6141a03f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc604003e70 tx=0x7fc604011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:06.113 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.254+0000 7fc610c00700 1 -- 172.21.15.44:0/705969112 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc604003aa0 con 0x7fc614109db0 2024-05-27T09:23:06.113 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.254+0000 7fc610c00700 1 -- 172.21.15.44:0/705969112 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc60401a8e0 con 0x7fc614109db0 2024-05-27T09:23:06.113 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.254+0000 7fc61ac00700 1 -- 172.21.15.44:0/705969112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6141a0bc0 con 0x7fc614109db0 2024-05-27T09:23:06.113 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.254+0000 7fc610c00700 1 -- 172.21.15.44:0/705969112 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc604024870 con 0x7fc614109db0 2024-05-27T09:23:06.113 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.254+0000 7fc61ac00700 1 -- 172.21.15.44:0/705969112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6141a37d0 con 0x7fc614109db0 2024-05-27T09:23:06.113 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.255+0000 7fc61ac00700 1 -- 172.21.15.44:0/705969112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc614064e10 con 0x7fc614109db0 2024-05-27T09:23:06.113 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.255+0000 7fc610c00700 1 -- 172.21.15.44:0/705969112 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fc604022020 con 0x7fc614109db0 2024-05-27T09:23:06.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.256+0000 7fc610c00700 1 --2- 172.21.15.44:0/705969112 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc5f803e560 0x7fc5f8040a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:06.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.256+0000 7fc610c00700 1 -- 172.21.15.44:0/705969112 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc604053a70 con 0x7fc614109db0 2024-05-27T09:23:06.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.256+0000 7fc613400700 1 --2- 172.21.15.44:0/705969112 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc5f803e560 0x7fc5f8040a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:06.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.257+0000 7fc613400700 1 --2- 172.21.15.44:0/705969112 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc5f803e560 0x7fc5f8040a70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:06.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.261+0000 7fc610c00700 1 -- 172.21.15.44:0/705969112 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc604016da0 con 0x7fc614109db0 2024-05-27T09:23:06.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.496+0000 7fc61ac00700 1 -- 172.21.15.44:0/705969112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fc6141a39a0 con 0x7fc614109db0 2024-05-27T09:23:06.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.497+0000 7fc610c00700 1 -- 172.21.15.44:0/705969112 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fc60401ad20 con 0x7fc614109db0 2024-05-27T09:23:06.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.500+0000 7fc61ac00700 1 -- 172.21.15.44:0/705969112 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc5f803e560 msgr2=0x7fc5f8040a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:06.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.500+0000 7fc61ac00700 1 --2- 172.21.15.44:0/705969112 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc5f803e560 0x7fc5f8040a70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:06.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.500+0000 7fc61ac00700 1 -- 172.21.15.44:0/705969112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 msgr2=0x7fc6141a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:06.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.500+0000 7fc61ac00700 1 --2- 172.21.15.44:0/705969112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc6141a03f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc604003e70 tx=0x7fc604011aa0 comp rx=0 tx=0).stop 2024-05-27T09:23:06.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.501+0000 7fc61ac00700 1 -- 172.21.15.44:0/705969112 shutdown_connections 2024-05-27T09:23:06.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.501+0000 7fc61ac00700 1 --2- 172.21.15.44:0/705969112 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc5f803e560 0x7fc5f8040a70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:06.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.501+0000 7fc61ac00700 1 --2- 172.21.15.44:0/705969112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc614109db0 0x7fc6141a03f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:06.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.501+0000 7fc61ac00700 1 -- 172.21.15.44:0/705969112 >> 172.21.15.44:0/705969112 conn(0x7fc6141053d0 msgr2=0x7fc614106ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:06.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.501+0000 7fc61ac00700 1 -- 172.21.15.44:0/705969112 shutdown_connections 2024-05-27T09:23:06.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:05.501+0000 7fc61ac00700 1 -- 172.21.15.44:0/705969112 wait complete. 2024-05-27T09:23:07.869 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.805+0000 7f33cb200700 1 Processor -- start 2024-05-27T09:23:07.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.806+0000 7f33cb200700 1 -- start start 2024-05-27T09:23:07.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.806+0000 7f33cb200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c4111d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:07.870 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.806+0000 7f33cb200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f33c4112380 con 0x7f33c4111980 2024-05-27T09:23:07.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.807+0000 7f33c3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c4111d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:07.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.807+0000 7f33c3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c4111d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44320/0 (socket says 172.21.15.44:44320) 2024-05-27T09:23:07.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.807+0000 7f33c3e00700 1 -- 172.21.15.44:0/648703787 learned_addr learned my addr 172.21.15.44:0/648703787 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:07.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.807+0000 7f33c3e00700 1 -- 172.21.15.44:0/648703787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c4110560 con 0x7f33c4111980 2024-05-27T09:23:07.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.808+0000 7f33c3e00700 1 --2- 172.21.15.44:0/648703787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c4111d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f33b40098a0 tx=0x7f33b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2faf8240d953 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:07.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.808+0000 7f33c3400700 1 -- 172.21.15.44:0/648703787 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33b4004160 con 0x7f33c4111980 2024-05-27T09:23:07.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.808+0000 7f33c3400700 1 -- 172.21.15.44:0/648703787 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33b4004300 con 0x7f33c4111980 2024-05-27T09:23:07.871 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.808+0000 7f33c3400700 1 -- 172.21.15.44:0/648703787 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33b4011a20 con 0x7f33c4111980 2024-05-27T09:23:07.873 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.809+0000 7f33cb200700 1 -- 172.21.15.44:0/648703787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 msgr2=0x7f33c4111d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.809+0000 7f33cb200700 1 --2- 172.21.15.44:0/648703787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c4111d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f33b40098a0 tx=0x7f33b4009bb0 comp rx=0 tx=0).stop 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.809+0000 7f33cb200700 1 -- 172.21.15.44:0/648703787 shutdown_connections 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.809+0000 7f33cb200700 1 --2- 172.21.15.44:0/648703787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c4111d90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.809+0000 7f33cb200700 1 -- 172.21.15.44:0/648703787 >> 172.21.15.44:0/648703787 conn(0x7f33c407f410 msgr2=0x7f33c40818c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.809+0000 7f33cb200700 1 -- 172.21.15.44:0/648703787 shutdown_connections 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.809+0000 7f33cb200700 1 -- 172.21.15.44:0/648703787 wait complete. 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.811+0000 7f33cb200700 1 Processor -- start 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.811+0000 7f33cb200700 1 -- start start 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.811+0000 7f33cb200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c41a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.811+0000 7f33cb200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f33c4114920 con 0x7f33c4111980 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.811+0000 7f33c3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c41a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:07.874 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.812+0000 7f33c3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c41a4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44326/0 (socket says 172.21.15.44:44326) 2024-05-27T09:23:07.875 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.812+0000 7f33c3e00700 1 -- 172.21.15.44:0/1398427132 learned_addr learned my addr 172.21.15.44:0/1398427132 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:07.875 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.812+0000 7f33c3e00700 1 -- 172.21.15.44:0/1398427132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c41a4d40 con 0x7f33c4111980 2024-05-27T09:23:07.875 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.812+0000 7f33c3e00700 1 --2- 172.21.15.44:0/1398427132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c41a4800 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f33b4003e70 tx=0x7f33b4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:07.875 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.813+0000 7f33c1600700 1 -- 172.21.15.44:0/1398427132 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33b4003aa0 con 0x7f33c4111980 2024-05-27T09:23:07.875 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.813+0000 7f33c1600700 1 -- 172.21.15.44:0/1398427132 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33b401a8e0 con 0x7f33c4111980 2024-05-27T09:23:07.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.813+0000 7f33cb200700 1 -- 172.21.15.44:0/1398427132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33c41a4fd0 con 0x7f33c4111980 2024-05-27T09:23:07.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.813+0000 7f33c1600700 1 -- 172.21.15.44:0/1398427132 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33b4024870 con 0x7f33c4111980 2024-05-27T09:23:07.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.813+0000 7f33cb200700 1 -- 172.21.15.44:0/1398427132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33c41a7be0 con 0x7f33c4111980 2024-05-27T09:23:07.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.814+0000 7f33c1600700 1 -- 172.21.15.44:0/1398427132 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f33b4022020 con 0x7f33c4111980 2024-05-27T09:23:07.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.814+0000 7f33cb200700 1 -- 172.21.15.44:0/1398427132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33c4064e10 con 0x7f33c4111980 2024-05-27T09:23:07.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.815+0000 7f33c1600700 1 --2- 172.21.15.44:0/1398427132 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33a803e560 0x7f33a8040a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:07.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.815+0000 7f33c1600700 1 -- 172.21.15.44:0/1398427132 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f33b4053c40 con 0x7f33c4111980 2024-05-27T09:23:07.878 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.815+0000 7f33bbe00700 1 --2- 172.21.15.44:0/1398427132 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33a803e560 0x7f33a8040a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.816+0000 7f33bbe00700 1 --2- 172.21.15.44:0/1398427132 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33a803e560 0x7f33a8040a70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f33ac006fd0 tx=0x7f33ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:06.821+0000 7f33c1600700 1 -- 172.21.15.44:0/1398427132 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f33b400b3e0 con 0x7f33c4111980 2024-05-27T09:23:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.179+0000 7f33cb200700 1 -- 172.21.15.44:0/1398427132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f33c41a7db0 con 0x7f33c4111980 2024-05-27T09:23:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.183+0000 7f33c1600700 1 -- 172.21.15.44:0/1398427132 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f33b401aa90 con 0x7f33c4111980 2024-05-27T09:23:07.879 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.188+0000 7f33cb200700 1 -- 172.21.15.44:0/1398427132 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33a803e560 msgr2=0x7f33a8040a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.188+0000 7f33cb200700 1 --2- 172.21.15.44:0/1398427132 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33a803e560 0x7f33a8040a70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f33ac006fd0 tx=0x7f33ac006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.188+0000 7f33cb200700 1 -- 172.21.15.44:0/1398427132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 msgr2=0x7f33c41a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.188+0000 7f33cb200700 1 --2- 172.21.15.44:0/1398427132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c41a4800 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f33b4003e70 tx=0x7f33b4011aa0 comp rx=0 tx=0).stop 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.188+0000 7f33cb200700 1 -- 172.21.15.44:0/1398427132 shutdown_connections 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.188+0000 7f33cb200700 1 --2- 172.21.15.44:0/1398427132 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33a803e560 0x7f33a8040a70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.188+0000 7f33cb200700 1 --2- 172.21.15.44:0/1398427132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33c4111980 0x7f33c41a4800 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.188+0000 7f33cb200700 1 -- 172.21.15.44:0/1398427132 >> 172.21.15.44:0/1398427132 conn(0x7f33c407f410 msgr2=0x7f33c407fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.188+0000 7f33cb200700 1 -- 172.21.15.44:0/1398427132 shutdown_connections 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-05-27T09:23:07.188+0000 7f33cb200700 1 -- 172.21.15.44:0/1398427132 wait complete. 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:07.880 INFO:teuthology.orchestra.run.smithi044.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout:Or, if you are only running a single cluster on this host: 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout: ceph telemetry on 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout:For more information see: 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:07.881 INFO:teuthology.orchestra.run.smithi044.stdout:Bootstrap complete. 2024-05-27T09:23:07.962 INFO:tasks.cephadm:Fetching config... 2024-05-27T09:23:07.962 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:23:07.962 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-27T09:23:08.013 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-27T09:23:08.013 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:23:08.013 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-27T09:23:08.065 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-27T09:23:08.065 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:23:08.065 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-27T09:23:08.135 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-27T09:23:08.135 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:23:08.135 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-27T09:23:08.234 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-27T09:23:08.234 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuxp6aHomdMxuJOuYLG3+1iZ9y3KLZcfEx2oO133X/JZWr8OqfLfmpO13TEc9tnu39303H1PKIyjN0fmrjmGk0ii/2ygAQoH0/ir2+iukOWTC+OyD+06PFNcASJSLm8Hhxk6w/MSBxagIy0uUSXv/ads/DYlAoyU98wkhI7nYghRcsCemrLMtMmlPO1LQS5fE99FqX/+LOGuUKnN71+abp3GuRfB+IeF9sliBtgX90iqthPc8r+NGskCJ5VQ4BwcPgVLSsO/w/TLvjBxNk1RbiZ1zRIADMsoFe3/0YIA8NpASlPNmkyyfWaLZqPatkKTcUmSa51RCCZrk5EE+9ci79zrUQcwERUE4eW/n457mJ29/nWosNwXW5rmaaxURcEu+E1IU4NeL5ADPqX3RKPaiTDX5wIfpDG6DB/fjVZoExsPh2y6FQ/9wmcmaC7p0OGs63JdPV8fh6/1xutwCGEEHzD7WWVGAv277vDOzpXwJjJWkUG7GhBI4yj1iD+qNp398= ceph-604b015a-1c0a-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T09:23:08.351 INFO:teuthology.orchestra.run.smithi044.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuxp6aHomdMxuJOuYLG3+1iZ9y3KLZcfEx2oO133X/JZWr8OqfLfmpO13TEc9tnu39303H1PKIyjN0fmrjmGk0ii/2ygAQoH0/ir2+iukOWTC+OyD+06PFNcASJSLm8Hhxk6w/MSBxagIy0uUSXv/ads/DYlAoyU98wkhI7nYghRcsCemrLMtMmlPO1LQS5fE99FqX/+LOGuUKnN71+abp3GuRfB+IeF9sliBtgX90iqthPc8r+NGskCJ5VQ4BwcPgVLSsO/w/TLvjBxNk1RbiZ1zRIADMsoFe3/0YIA8NpASlPNmkyyfWaLZqPatkKTcUmSa51RCCZrk5EE+9ci79zrUQcwERUE4eW/n457mJ29/nWosNwXW5rmaaxURcEu+E1IU4NeL5ADPqX3RKPaiTDX5wIfpDG6DB/fjVZoExsPh2y6FQ/9wmcmaC7p0OGs63JdPV8fh6/1xutwCGEEHzD7WWVGAv277vDOzpXwJjJWkUG7GhBI4yj1iD+qNp398= ceph-604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:23:08.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:08 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1398427132' entity='client.admin' 2024-05-27T09:23:08.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuxp6aHomdMxuJOuYLG3+1iZ9y3KLZcfEx2oO133X/JZWr8OqfLfmpO13TEc9tnu39303H1PKIyjN0fmrjmGk0ii/2ygAQoH0/ir2+iukOWTC+OyD+06PFNcASJSLm8Hhxk6w/MSBxagIy0uUSXv/ads/DYlAoyU98wkhI7nYghRcsCemrLMtMmlPO1LQS5fE99FqX/+LOGuUKnN71+abp3GuRfB+IeF9sliBtgX90iqthPc8r+NGskCJ5VQ4BwcPgVLSsO/w/TLvjBxNk1RbiZ1zRIADMsoFe3/0YIA8NpASlPNmkyyfWaLZqPatkKTcUmSa51RCCZrk5EE+9ci79zrUQcwERUE4eW/n457mJ29/nWosNwXW5rmaaxURcEu+E1IU4NeL5ADPqX3RKPaiTDX5wIfpDG6DB/fjVZoExsPh2y6FQ/9wmcmaC7p0OGs63JdPV8fh6/1xutwCGEEHzD7WWVGAv277vDOzpXwJjJWkUG7GhBI4yj1iD+qNp398= ceph-604b015a-1c0a-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T09:23:08.430 INFO:teuthology.orchestra.run.smithi050.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuxp6aHomdMxuJOuYLG3+1iZ9y3KLZcfEx2oO133X/JZWr8OqfLfmpO13TEc9tnu39303H1PKIyjN0fmrjmGk0ii/2ygAQoH0/ir2+iukOWTC+OyD+06PFNcASJSLm8Hhxk6w/MSBxagIy0uUSXv/ads/DYlAoyU98wkhI7nYghRcsCemrLMtMmlPO1LQS5fE99FqX/+LOGuUKnN71+abp3GuRfB+IeF9sliBtgX90iqthPc8r+NGskCJ5VQ4BwcPgVLSsO/w/TLvjBxNk1RbiZ1zRIADMsoFe3/0YIA8NpASlPNmkyyfWaLZqPatkKTcUmSa51RCCZrk5EE+9ci79zrUQcwERUE4eW/n457mJ29/nWosNwXW5rmaaxURcEu+E1IU4NeL5ADPqX3RKPaiTDX5wIfpDG6DB/fjVZoExsPh2y6FQ/9wmcmaC7p0OGs63JdPV8fh6/1xutwCGEEHzD7WWVGAv277vDOzpXwJjJWkUG7GhBI4yj1iD+qNp398= ceph-604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:23:08.441 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuxp6aHomdMxuJOuYLG3+1iZ9y3KLZcfEx2oO133X/JZWr8OqfLfmpO13TEc9tnu39303H1PKIyjN0fmrjmGk0ii/2ygAQoH0/ir2+iukOWTC+OyD+06PFNcASJSLm8Hhxk6w/MSBxagIy0uUSXv/ads/DYlAoyU98wkhI7nYghRcsCemrLMtMmlPO1LQS5fE99FqX/+LOGuUKnN71+abp3GuRfB+IeF9sliBtgX90iqthPc8r+NGskCJ5VQ4BwcPgVLSsO/w/TLvjBxNk1RbiZ1zRIADMsoFe3/0YIA8NpASlPNmkyyfWaLZqPatkKTcUmSa51RCCZrk5EE+9ci79zrUQcwERUE4eW/n457mJ29/nWosNwXW5rmaaxURcEu+E1IU4NeL5ADPqX3RKPaiTDX5wIfpDG6DB/fjVZoExsPh2y6FQ/9wmcmaC7p0OGs63JdPV8fh6/1xutwCGEEHzD7WWVGAv277vDOzpXwJjJWkUG7GhBI4yj1iD+qNp398= ceph-604b015a-1c0a-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T09:23:08.484 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuxp6aHomdMxuJOuYLG3+1iZ9y3KLZcfEx2oO133X/JZWr8OqfLfmpO13TEc9tnu39303H1PKIyjN0fmrjmGk0ii/2ygAQoH0/ir2+iukOWTC+OyD+06PFNcASJSLm8Hhxk6w/MSBxagIy0uUSXv/ads/DYlAoyU98wkhI7nYghRcsCemrLMtMmlPO1LQS5fE99FqX/+LOGuUKnN71+abp3GuRfB+IeF9sliBtgX90iqthPc8r+NGskCJ5VQ4BwcPgVLSsO/w/TLvjBxNk1RbiZ1zRIADMsoFe3/0YIA8NpASlPNmkyyfWaLZqPatkKTcUmSa51RCCZrk5EE+9ci79zrUQcwERUE4eW/n457mJ29/nWosNwXW5rmaaxURcEu+E1IU4NeL5ADPqX3RKPaiTDX5wIfpDG6DB/fjVZoExsPh2y6FQ/9wmcmaC7p0OGs63JdPV8fh6/1xutwCGEEHzD7WWVGAv277vDOzpXwJjJWkUG7GhBI4yj1iD+qNp398= ceph-604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:23:08.496 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-27T09:23:08.932 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:23:11.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.838+0000 7fa4c8000700 1 -- 172.21.15.44:0/4248423607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4c0104710 msgr2=0x7fa4c0106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:11.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.838+0000 7fa4c8000700 1 --2- 172.21.15.44:0/4248423607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4c0104710 0x7fa4c0106b30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009910 tx=0x7fa4b4009c20 comp rx=0 tx=0).stop 2024-05-27T09:23:11.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.838+0000 7fa4c8000700 1 -- 172.21.15.44:0/4248423607 shutdown_connections 2024-05-27T09:23:11.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.838+0000 7fa4c8000700 1 --2- 172.21.15.44:0/4248423607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4c0104710 0x7fa4c0106b30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:11.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.838+0000 7fa4c8000700 1 -- 172.21.15.44:0/4248423607 >> 172.21.15.44:0/4248423607 conn(0x7fa4c00fe130 msgr2=0x7fa4c01005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:11.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.838+0000 7fa4c8000700 1 -- 172.21.15.44:0/4248423607 shutdown_connections 2024-05-27T09:23:11.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.838+0000 7fa4c8000700 1 -- 172.21.15.44:0/4248423607 wait complete. 2024-05-27T09:23:11.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.839+0000 7fa4c8000700 1 Processor -- start 2024-05-27T09:23:11.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.839+0000 7fa4c8000700 1 -- start start 2024-05-27T09:23:11.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.840+0000 7fa4c8000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4c0104710 0x7fa4c019b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:11.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.840+0000 7fa4c8000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c0109b80 con 0x7fa4c0104710 2024-05-27T09:23:11.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.840+0000 7fa4c5600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4c0104710 0x7fa4c019b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:11.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.840+0000 7fa4c5600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4c0104710 0x7fa4c019b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44340/0 (socket says 172.21.15.44:44340) 2024-05-27T09:23:11.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.840+0000 7fa4c5600700 1 -- 172.21.15.44:0/1283322697 learned_addr learned my addr 172.21.15.44:0/1283322697 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:11.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.841+0000 7fa4c5600700 1 -- 172.21.15.44:0/1283322697 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c019be50 con 0x7fa4c0104710 2024-05-27T09:23:11.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.841+0000 7fa4c5600700 1 --2- 172.21.15.44:0/1283322697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4c0104710 0x7fa4c019b910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40050d0 tx=0x7fa4b4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:11.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.841+0000 7fa4be000700 1 -- 172.21.15.44:0/1283322697 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4b4027070 con 0x7fa4c0104710 2024-05-27T09:23:11.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.841+0000 7fa4be000700 1 -- 172.21.15.44:0/1283322697 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4b400be10 con 0x7fa4c0104710 2024-05-27T09:23:11.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.841+0000 7fa4be000700 1 -- 172.21.15.44:0/1283322697 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4b4022d80 con 0x7fa4c0104710 2024-05-27T09:23:11.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.841+0000 7fa4c8000700 1 -- 172.21.15.44:0/1283322697 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c019c0e0 con 0x7fa4c0104710 2024-05-27T09:23:11.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.842+0000 7fa4c8000700 1 -- 172.21.15.44:0/1283322697 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c019ecf0 con 0x7fa4c0104710 2024-05-27T09:23:11.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.843+0000 7fa4c8000700 1 -- 172.21.15.44:0/1283322697 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c010a300 con 0x7fa4c0104710 2024-05-27T09:23:11.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.843+0000 7fa4be000700 1 -- 172.21.15.44:0/1283322697 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fa4b4022470 con 0x7fa4c0104710 2024-05-27T09:23:11.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.844+0000 7fa4be000700 1 --2- 172.21.15.44:0/1283322697 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa4a803e570 0x7fa4a8040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:11.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.844+0000 7fa4be000700 1 -- 172.21.15.44:0/1283322697 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa4b401e030 con 0x7fa4c0104710 2024-05-27T09:23:11.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.844+0000 7fa4c4c00700 1 --2- 172.21.15.44:0/1283322697 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa4a803e570 0x7fa4a8040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:11.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.845+0000 7fa4c4c00700 1 --2- 172.21.15.44:0/1283322697 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa4a803e570 0x7fa4a8040a80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac006fd0 tx=0x7fa4ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:11.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:11.850+0000 7fa4be000700 1 -- 172.21.15.44:0/1283322697 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa4b4013070 con 0x7fa4c0104710 2024-05-27T09:23:12.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.090+0000 7fa4c8000700 1 -- 172.21.15.44:0/1283322697 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fa4c002e650 con 0x7fa4c0104710 2024-05-27T09:23:12.091 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:12 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:12.091 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:12 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:12.091 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:12 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:23:12.091 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:12 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:12.091 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:12 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:23:12.091 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:12 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:12.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.095+0000 7fa4be000700 1 -- 172.21.15.44:0/1283322697 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fa4b400de00 con 0x7fa4c0104710 2024-05-27T09:23:12.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.099+0000 7fa4c8000700 1 -- 172.21.15.44:0/1283322697 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa4a803e570 msgr2=0x7fa4a8040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:12.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.099+0000 7fa4c8000700 1 --2- 172.21.15.44:0/1283322697 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa4a803e570 0x7fa4a8040a80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac006fd0 tx=0x7fa4ac006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:12.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.099+0000 7fa4c8000700 1 -- 172.21.15.44:0/1283322697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4c0104710 msgr2=0x7fa4c019b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:12.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.099+0000 7fa4c8000700 1 --2- 172.21.15.44:0/1283322697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4c0104710 0x7fa4c019b910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40050d0 tx=0x7fa4b4009f80 comp rx=0 tx=0).stop 2024-05-27T09:23:12.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.099+0000 7fa4c8000700 1 -- 172.21.15.44:0/1283322697 shutdown_connections 2024-05-27T09:23:12.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.099+0000 7fa4c8000700 1 --2- 172.21.15.44:0/1283322697 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa4a803e570 0x7fa4a8040a80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:12.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.099+0000 7fa4c8000700 1 --2- 172.21.15.44:0/1283322697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4c0104710 0x7fa4c019b910 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:12.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.099+0000 7fa4c8000700 1 -- 172.21.15.44:0/1283322697 >> 172.21.15.44:0/1283322697 conn(0x7fa4c00fe130 msgr2=0x7fa4c01005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:12.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.099+0000 7fa4c8000700 1 -- 172.21.15.44:0/1283322697 shutdown_connections 2024-05-27T09:23:12.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:12.099+0000 7fa4c8000700 1 -- 172.21.15.44:0/1283322697 wait complete. 2024-05-27T09:23:12.841 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-27T09:23:12.841 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-27T09:23:13.272 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:23:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:13 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1283322697' entity='client.admin' 2024-05-27T09:23:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:23:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:15.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.908+0000 7fe73dc00700 1 -- 172.21.15.44:0/3255018967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe738105dd0 msgr2=0x7fe7381061e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:15.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.908+0000 7fe73dc00700 1 --2- 172.21.15.44:0/3255018967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe738105dd0 0x7fe7381061e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe728009910 tx=0x7fe728009c20 comp rx=0 tx=0).stop 2024-05-27T09:23:15.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.908+0000 7fe73dc00700 1 -- 172.21.15.44:0/3255018967 shutdown_connections 2024-05-27T09:23:15.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.908+0000 7fe73dc00700 1 --2- 172.21.15.44:0/3255018967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe738105dd0 0x7fe7381061e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:15.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.908+0000 7fe73dc00700 1 -- 172.21.15.44:0/3255018967 >> 172.21.15.44:0/3255018967 conn(0x7fe738101410 msgr2=0x7fe7381038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:15.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.908+0000 7fe73dc00700 1 -- 172.21.15.44:0/3255018967 shutdown_connections 2024-05-27T09:23:15.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.909+0000 7fe73dc00700 1 -- 172.21.15.44:0/3255018967 wait complete. 2024-05-27T09:23:15.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.910+0000 7fe73dc00700 1 Processor -- start 2024-05-27T09:23:15.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.910+0000 7fe73dc00700 1 -- start start 2024-05-27T09:23:15.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.910+0000 7fe73dc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe738105dd0 0x7fe73819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:15.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.910+0000 7fe73dc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe73810bd90 con 0x7fe738105dd0 2024-05-27T09:23:15.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.911+0000 7fe736a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe738105dd0 0x7fe73819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:15.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.911+0000 7fe736a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe738105dd0 0x7fe73819b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46316/0 (socket says 172.21.15.44:46316) 2024-05-27T09:23:15.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.911+0000 7fe736a00700 1 -- 172.21.15.44:0/688642656 learned_addr learned my addr 172.21.15.44:0/688642656 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:15.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.911+0000 7fe736a00700 1 -- 172.21.15.44:0/688642656 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73819be30 con 0x7fe738105dd0 2024-05-27T09:23:15.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.911+0000 7fe736a00700 1 --2- 172.21.15.44:0/688642656 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe738105dd0 0x7fe73819b8f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe728006010 tx=0x7fe728004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:15.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.912+0000 7fe72f400700 1 -- 172.21.15.44:0/688642656 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe728026070 con 0x7fe738105dd0 2024-05-27T09:23:15.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.912+0000 7fe72f400700 1 -- 172.21.15.44:0/688642656 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe72800bc00 con 0x7fe738105dd0 2024-05-27T09:23:15.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.912+0000 7fe73dc00700 1 -- 172.21.15.44:0/688642656 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe73819c0c0 con 0x7fe738105dd0 2024-05-27T09:23:15.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.912+0000 7fe72f400700 1 -- 172.21.15.44:0/688642656 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe728003970 con 0x7fe738105dd0 2024-05-27T09:23:15.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.912+0000 7fe73dc00700 1 -- 172.21.15.44:0/688642656 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe73819ecd0 con 0x7fe738105dd0 2024-05-27T09:23:15.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.913+0000 7fe73dc00700 1 -- 172.21.15.44:0/688642656 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe73810c510 con 0x7fe738105dd0 2024-05-27T09:23:15.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.913+0000 7fe72f400700 1 -- 172.21.15.44:0/688642656 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fe728029070 con 0x7fe738105dd0 2024-05-27T09:23:15.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.914+0000 7fe72f400700 1 --2- 172.21.15.44:0/688642656 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe71c03e570 0x7fe71c040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:15.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.914+0000 7fe72f400700 1 -- 172.21.15.44:0/688642656 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe72801e070 con 0x7fe738105dd0 2024-05-27T09:23:15.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.914+0000 7fe736000700 1 --2- 172.21.15.44:0/688642656 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe71c03e570 0x7fe71c040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:15.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.915+0000 7fe736000700 1 --2- 172.21.15.44:0/688642656 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe71c03e570 0x7fe71c040a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe720006fd0 tx=0x7fe720006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:15.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:15.920+0000 7fe72f400700 1 -- 172.21.15.44:0/688642656 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe728013070 con 0x7fe738105dd0 2024-05-27T09:23:16.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.159+0000 7fe73dc00700 1 -- 172.21.15.44:0/688642656 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fe73802e6f0 con 0x7fe71c03e570 2024-05-27T09:23:16.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.166+0000 7fe72f400700 1 -- 172.21.15.44:0/688642656 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe73802e6f0 con 0x7fe71c03e570 2024-05-27T09:23:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.169+0000 7fe73dc00700 1 -- 172.21.15.44:0/688642656 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe71c03e570 msgr2=0x7fe71c040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.169+0000 7fe73dc00700 1 --2- 172.21.15.44:0/688642656 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe71c03e570 0x7fe71c040a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe720006fd0 tx=0x7fe720006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.169+0000 7fe73dc00700 1 -- 172.21.15.44:0/688642656 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe738105dd0 msgr2=0x7fe73819b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.169+0000 7fe73dc00700 1 --2- 172.21.15.44:0/688642656 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe738105dd0 0x7fe73819b8f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe728006010 tx=0x7fe728004fa0 comp rx=0 tx=0).stop 2024-05-27T09:23:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.170+0000 7fe73dc00700 1 -- 172.21.15.44:0/688642656 shutdown_connections 2024-05-27T09:23:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.170+0000 7fe73dc00700 1 --2- 172.21.15.44:0/688642656 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe71c03e570 0x7fe71c040a80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.170+0000 7fe73dc00700 1 --2- 172.21.15.44:0/688642656 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe738105dd0 0x7fe73819b8f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.170+0000 7fe73dc00700 1 -- 172.21.15.44:0/688642656 >> 172.21.15.44:0/688642656 conn(0x7fe738101410 msgr2=0x7fe738102110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.170+0000 7fe73dc00700 1 -- 172.21.15.44:0/688642656 shutdown_connections 2024-05-27T09:23:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:16.170+0000 7fe73dc00700 1 -- 172.21.15.44:0/688642656 wait complete. 2024-05-27T09:23:16.951 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi050 2024-05-27T09:23:16.951 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-27T09:23:16.951 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/etc/ceph/ceph.conf 2024-05-27T09:23:16.970 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-27T09:23:16.970 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:23:17.028 INFO:tasks.cephadm:Adding host smithi050 to orchestrator... 2024-05-27T09:23:17.029 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch host add smithi050 2024-05-27T09:23:17.448 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:23:17.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:17 smithi044 ceph-mon[31618]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:23:17.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:17.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:23:17.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:23:17.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:23:17.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:17 smithi044 ceph-mon[31618]: Updating smithi044:/etc/ceph/ceph.conf 2024-05-27T09:23:17.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:17 smithi044 ceph-mon[31618]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:18.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:18 smithi044 ceph-mon[31618]: Updating smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:23:18.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:18 smithi044 ceph-mon[31618]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:23:19.202 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:19 smithi044 ceph-mon[31618]: Updating smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:23:19.202 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:19 smithi044 ceph-mon[31618]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:19.203 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:19.203 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:19.203 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:19.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.578+0000 7fec2f400700 1 -- 172.21.15.44:0/4056081959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec2810a3c0 msgr2=0x7fec2810a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:19.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.578+0000 7fec2f400700 1 --2- 172.21.15.44:0/4056081959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec2810a3c0 0x7fec2810a7d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009910 tx=0x7fec1c009c20 comp rx=0 tx=0).stop 2024-05-27T09:23:19.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.579+0000 7fec2f400700 1 -- 172.21.15.44:0/4056081959 shutdown_connections 2024-05-27T09:23:19.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.579+0000 7fec2f400700 1 --2- 172.21.15.44:0/4056081959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec2810a3c0 0x7fec2810a7d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:19.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.579+0000 7fec2f400700 1 -- 172.21.15.44:0/4056081959 >> 172.21.15.44:0/4056081959 conn(0x7fec280799e0 msgr2=0x7fec281085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:19.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.579+0000 7fec2f400700 1 -- 172.21.15.44:0/4056081959 shutdown_connections 2024-05-27T09:23:19.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.579+0000 7fec2f400700 1 -- 172.21.15.44:0/4056081959 wait complete. 2024-05-27T09:23:19.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.580+0000 7fec2f400700 1 Processor -- start 2024-05-27T09:23:19.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.580+0000 7fec2f400700 1 -- start start 2024-05-27T09:23:19.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.581+0000 7fec2f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec2810a3c0 0x7fec2819ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:19.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.581+0000 7fec2f400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fec281146b0 con 0x7fec2810a3c0 2024-05-27T09:23:19.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.581+0000 7fec2ca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec2810a3c0 0x7fec2819ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:19.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.581+0000 7fec2ca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec2810a3c0 0x7fec2819ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46326/0 (socket says 172.21.15.44:46326) 2024-05-27T09:23:19.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.581+0000 7fec2ca00700 1 -- 172.21.15.44:0/408922331 learned_addr learned my addr 172.21.15.44:0/408922331 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:19.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.581+0000 7fec2ca00700 1 -- 172.21.15.44:0/408922331 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec281a04d0 con 0x7fec2810a3c0 2024-05-27T09:23:19.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.582+0000 7fec2ca00700 1 --2- 172.21.15.44:0/408922331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec2810a3c0 0x7fec2819ff90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fec1c006010 tx=0x7fec1c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:19.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.582+0000 7fec25600700 1 -- 172.21.15.44:0/408922331 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec1c026070 con 0x7fec2810a3c0 2024-05-27T09:23:19.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.582+0000 7fec25600700 1 -- 172.21.15.44:0/408922331 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec1c00bba0 con 0x7fec2810a3c0 2024-05-27T09:23:19.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.582+0000 7fec25600700 1 -- 172.21.15.44:0/408922331 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec1c003910 con 0x7fec2810a3c0 2024-05-27T09:23:19.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.582+0000 7fec2f400700 1 -- 172.21.15.44:0/408922331 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec281a07c0 con 0x7fec2810a3c0 2024-05-27T09:23:19.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.582+0000 7fec2f400700 1 -- 172.21.15.44:0/408922331 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec281a4770 con 0x7fec2810a3c0 2024-05-27T09:23:19.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.583+0000 7fec2f400700 1 -- 172.21.15.44:0/408922331 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec2810b870 con 0x7fec2810a3c0 2024-05-27T09:23:19.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.583+0000 7fec25600700 1 -- 172.21.15.44:0/408922331 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fec1c029070 con 0x7fec2810a3c0 2024-05-27T09:23:19.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.584+0000 7fec25600700 1 --2- 172.21.15.44:0/408922331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fec1003e570 0x7fec10040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:19.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.584+0000 7fec25600700 1 -- 172.21.15.44:0/408922331 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fec1c01e070 con 0x7fec2810a3c0 2024-05-27T09:23:19.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.584+0000 7fec27e00700 1 --2- 172.21.15.44:0/408922331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fec1003e570 0x7fec10040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:19.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.585+0000 7fec27e00700 1 --2- 172.21.15.44:0/408922331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fec1003e570 0x7fec10040a80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fec14006fd0 tx=0x7fec14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:19.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.590+0000 7fec25600700 1 -- 172.21.15.44:0/408922331 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fec1c013070 con 0x7fec2810a3c0 2024-05-27T09:23:19.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:19.831+0000 7fec2f400700 1 -- 172.21.15.44:0/408922331 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi050", "target": ["mon-mgr", ""]}) -- 0x7fec281a12a0 con 0x7fec1003e570 2024-05-27T09:23:21.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:21 smithi044 ceph-mon[31618]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi050", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:23:21.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:21 smithi044 ceph-mon[31618]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:21.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:21 smithi044 ceph-mon[31618]: Deploying cephadm binary to smithi050 2024-05-27T09:23:23.782 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:23 smithi044 ceph-mon[31618]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:24.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:24.788+0000 7fec25600700 1 -- 172.21.15.44:0/408922331 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fec281a12a0 con 0x7fec1003e570 2024-05-27T09:23:24.789 INFO:teuthology.orchestra.run.smithi044.stdout:Added host 'smithi050' with addr '172.21.15.50' 2024-05-27T09:23:24.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:24.792+0000 7fec2f400700 1 -- 172.21.15.44:0/408922331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fec1003e570 msgr2=0x7fec10040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:24.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:24.793+0000 7fec2f400700 1 --2- 172.21.15.44:0/408922331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fec1003e570 0x7fec10040a80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fec14006fd0 tx=0x7fec14006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:24.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:24.793+0000 7fec2f400700 1 -- 172.21.15.44:0/408922331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec2810a3c0 msgr2=0x7fec2819ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:24.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:24.793+0000 7fec2f400700 1 --2- 172.21.15.44:0/408922331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec2810a3c0 0x7fec2819ff90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fec1c006010 tx=0x7fec1c005780 comp rx=0 tx=0).stop 2024-05-27T09:23:24.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:24.793+0000 7fec2f400700 1 -- 172.21.15.44:0/408922331 shutdown_connections 2024-05-27T09:23:24.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:24.793+0000 7fec2f400700 1 --2- 172.21.15.44:0/408922331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fec1003e570 0x7fec10040a80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:24.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:24.793+0000 7fec2f400700 1 --2- 172.21.15.44:0/408922331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec2810a3c0 0x7fec2819ff90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:24.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:24.793+0000 7fec2f400700 1 -- 172.21.15.44:0/408922331 >> 172.21.15.44:0/408922331 conn(0x7fec280799e0 msgr2=0x7fec2810bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:24.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:24.793+0000 7fec2f400700 1 -- 172.21.15.44:0/408922331 shutdown_connections 2024-05-27T09:23:24.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:24.793+0000 7fec2f400700 1 -- 172.21.15.44:0/408922331 wait complete. 2024-05-27T09:23:25.442 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-27T09:23:25.855 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:23:26.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:25 smithi044 ceph-mon[31618]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:26.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:26.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:25 smithi044 ceph-mon[31618]: Added host smithi050 2024-05-27T09:23:26.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:23:26.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:27.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:26 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:27.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:26 smithi044 ceph-mon[31618]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:28.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.130+0000 7f2b8b200700 1 -- 172.21.15.44:0/1456051918 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b84104d10 msgr2=0x7f2b84105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.130+0000 7f2b8b200700 1 --2- 172.21.15.44:0/1456051918 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b84104d10 0x7f2b84105120 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b74009910 tx=0x7f2b74009c20 comp rx=0 tx=0).stop 2024-05-27T09:23:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.130+0000 7f2b8b200700 1 -- 172.21.15.44:0/1456051918 shutdown_connections 2024-05-27T09:23:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.130+0000 7f2b8b200700 1 --2- 172.21.15.44:0/1456051918 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b84104d10 0x7f2b84105120 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.130+0000 7f2b8b200700 1 -- 172.21.15.44:0/1456051918 >> 172.21.15.44:0/1456051918 conn(0x7f2b840fe560 msgr2=0x7f2b841009b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.131+0000 7f2b8b200700 1 -- 172.21.15.44:0/1456051918 shutdown_connections 2024-05-27T09:23:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.131+0000 7f2b8b200700 1 -- 172.21.15.44:0/1456051918 wait complete. 2024-05-27T09:23:28.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.132+0000 7f2b8b200700 1 Processor -- start 2024-05-27T09:23:28.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.132+0000 7f2b8b200700 1 -- start start 2024-05-27T09:23:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.133+0000 7f2b8b200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b84104d10 0x7f2b841a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.133+0000 7f2b8b200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8410e100 con 0x7f2b84104d10 2024-05-27T09:23:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.133+0000 7f2b83e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b84104d10 0x7f2b841a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.133+0000 7f2b83e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b84104d10 0x7f2b841a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52764/0 (socket says 172.21.15.44:52764) 2024-05-27T09:23:28.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.133+0000 7f2b83e00700 1 -- 172.21.15.44:0/4058833840 learned_addr learned my addr 172.21.15.44:0/4058833840 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:28.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.134+0000 7f2b83e00700 1 -- 172.21.15.44:0/4058833840 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b841a1ce0 con 0x7f2b84104d10 2024-05-27T09:23:28.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.134+0000 7f2b83e00700 1 --2- 172.21.15.44:0/4058833840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b84104d10 0x7f2b841a17a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b740056d0 tx=0x7f2b740057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:28.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.134+0000 7f2b80c00700 1 -- 172.21.15.44:0/4058833840 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b74026070 con 0x7f2b84104d10 2024-05-27T09:23:28.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.134+0000 7f2b80c00700 1 -- 172.21.15.44:0/4058833840 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b74004370 con 0x7f2b84104d10 2024-05-27T09:23:28.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.134+0000 7f2b80c00700 1 -- 172.21.15.44:0/4058833840 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b740039b0 con 0x7f2b84104d10 2024-05-27T09:23:28.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.134+0000 7f2b8b200700 1 -- 172.21.15.44:0/4058833840 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b841a1f70 con 0x7f2b84104d10 2024-05-27T09:23:28.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.135+0000 7f2b8b200700 1 -- 172.21.15.44:0/4058833840 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b8419b9c0 con 0x7f2b84104d10 2024-05-27T09:23:28.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.136+0000 7f2b8b200700 1 -- 172.21.15.44:0/4058833840 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b8406c410 con 0x7f2b84104d10 2024-05-27T09:23:28.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.136+0000 7f2b80c00700 1 -- 172.21.15.44:0/4058833840 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f2b74029070 con 0x7f2b84104d10 2024-05-27T09:23:28.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.137+0000 7f2b80c00700 1 --2- 172.21.15.44:0/4058833840 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2b6803e570 0x7f2b68040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:28.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.137+0000 7f2b80c00700 1 -- 172.21.15.44:0/4058833840 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2b7401e070 con 0x7f2b84104d10 2024-05-27T09:23:28.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.137+0000 7f2b83400700 1 --2- 172.21.15.44:0/4058833840 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2b6803e570 0x7f2b68040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:28.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.138+0000 7f2b83400700 1 --2- 172.21.15.44:0/4058833840 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2b6803e570 0x7f2b68040a80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2b70006fd0 tx=0x7f2b70006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:28.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.143+0000 7f2b80c00700 1 -- 172.21.15.44:0/4058833840 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2b74013070 con 0x7f2b84104d10 2024-05-27T09:23:28.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.386+0000 7f2b8b200700 1 -- 172.21.15.44:0/4058833840 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2b8419cef0 con 0x7f2b6803e570 2024-05-27T09:23:28.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.389+0000 7f2b80c00700 1 -- 172.21.15.44:0/4058833840 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f2b8419cef0 con 0x7f2b6803e570 2024-05-27T09:23:28.390 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:28.390 INFO:teuthology.orchestra.run.smithi044.stdout:[{"addr": "172.21.15.44", "hostname": "smithi044", "labels": [], "status": ""}, {"addr": "172.21.15.50", "hostname": "smithi050", "labels": [], "status": ""}] 2024-05-27T09:23:28.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.393+0000 7f2b8b200700 1 -- 172.21.15.44:0/4058833840 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2b6803e570 msgr2=0x7f2b68040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:28.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.393+0000 7f2b8b200700 1 --2- 172.21.15.44:0/4058833840 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2b6803e570 0x7f2b68040a80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2b70006fd0 tx=0x7f2b70006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:28.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.393+0000 7f2b8b200700 1 -- 172.21.15.44:0/4058833840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b84104d10 msgr2=0x7f2b841a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:28.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.393+0000 7f2b8b200700 1 --2- 172.21.15.44:0/4058833840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b84104d10 0x7f2b841a17a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b740056d0 tx=0x7f2b740057b0 comp rx=0 tx=0).stop 2024-05-27T09:23:28.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.393+0000 7f2b8b200700 1 -- 172.21.15.44:0/4058833840 shutdown_connections 2024-05-27T09:23:28.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.393+0000 7f2b8b200700 1 --2- 172.21.15.44:0/4058833840 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2b6803e570 0x7f2b68040a80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:28.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.393+0000 7f2b8b200700 1 --2- 172.21.15.44:0/4058833840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b84104d10 0x7f2b841a17a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:28.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.393+0000 7f2b8b200700 1 -- 172.21.15.44:0/4058833840 >> 172.21.15.44:0/4058833840 conn(0x7f2b840fe560 msgr2=0x7f2b84109e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:28.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.393+0000 7f2b8b200700 1 -- 172.21.15.44:0/4058833840 shutdown_connections 2024-05-27T09:23:28.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:28.393+0000 7f2b8b200700 1 -- 172.21.15.44:0/4058833840 wait complete. 2024-05-27T09:23:28.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:28 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:29.119 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-05-27T09:23:29.119 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-27T09:23:29.119 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-05-27T09:23:29.138 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-27T09:23:29.138 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:23:29.195 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-05-27T09:23:29.196 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch host add smithi189 2024-05-27T09:23:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:23:29.950 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:29 smithi044 ceph-mon[31618]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:23:29.951 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:29 smithi044 ceph-mon[31618]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:31.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.690+0000 7f9dcd000700 1 -- 172.21.15.44:0/2479812902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dc8103de0 msgr2=0x7f9dc81041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:31.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.690+0000 7f9dcd000700 1 --2- 172.21.15.44:0/2479812902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dc8103de0 0x7f9dc81041f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9db8009910 tx=0x7f9db8009c20 comp rx=0 tx=0).stop 2024-05-27T09:23:31.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.690+0000 7f9dcd000700 1 -- 172.21.15.44:0/2479812902 shutdown_connections 2024-05-27T09:23:31.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.690+0000 7f9dcd000700 1 --2- 172.21.15.44:0/2479812902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dc8103de0 0x7f9dc81041f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:31.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.690+0000 7f9dcd000700 1 -- 172.21.15.44:0/2479812902 >> 172.21.15.44:0/2479812902 conn(0x7f9dc80ff690 msgr2=0x7f9dc8101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:31.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.690+0000 7f9dcd000700 1 -- 172.21.15.44:0/2479812902 shutdown_connections 2024-05-27T09:23:31.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.690+0000 7f9dcd000700 1 -- 172.21.15.44:0/2479812902 wait complete. 2024-05-27T09:23:31.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.691+0000 7f9dcd000700 1 Processor -- start 2024-05-27T09:23:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.691+0000 7f9dcd000700 1 -- start start 2024-05-27T09:23:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.692+0000 7f9dcd000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dc8103de0 0x7f9dc81a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.692+0000 7f9dcd000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc810e0d0 con 0x7f9dc8103de0 2024-05-27T09:23:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.692+0000 7f9dc5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dc8103de0 0x7f9dc81a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.692+0000 7f9dc5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dc8103de0 0x7f9dc81a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52772/0 (socket says 172.21.15.44:52772) 2024-05-27T09:23:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.692+0000 7f9dc5e00700 1 -- 172.21.15.44:0/3242427810 learned_addr learned my addr 172.21.15.44:0/3242427810 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:31.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.693+0000 7f9dc5e00700 1 -- 172.21.15.44:0/3242427810 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc81a1c90 con 0x7f9dc8103de0 2024-05-27T09:23:31.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.693+0000 7f9dc5e00700 1 --2- 172.21.15.44:0/3242427810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dc8103de0 0x7f9dc81a1750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9db8006010 tx=0x7f9db8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:31.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.693+0000 7f9dbf400700 1 -- 172.21.15.44:0/3242427810 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9db8026070 con 0x7f9dc8103de0 2024-05-27T09:23:31.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.693+0000 7f9dbf400700 1 -- 172.21.15.44:0/3242427810 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9db800bba0 con 0x7f9dc8103de0 2024-05-27T09:23:31.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.693+0000 7f9dbf400700 1 -- 172.21.15.44:0/3242427810 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9db8003910 con 0x7f9dc8103de0 2024-05-27T09:23:31.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.694+0000 7f9dcd000700 1 -- 172.21.15.44:0/3242427810 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc81a1f80 con 0x7f9dc8103de0 2024-05-27T09:23:31.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.694+0000 7f9dcd000700 1 -- 172.21.15.44:0/3242427810 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc819b970 con 0x7f9dc8103de0 2024-05-27T09:23:31.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.695+0000 7f9dcd000700 1 -- 172.21.15.44:0/3242427810 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc8105290 con 0x7f9dc8103de0 2024-05-27T09:23:31.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.695+0000 7f9dbf400700 1 -- 172.21.15.44:0/3242427810 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f9db8029070 con 0x7f9dc8103de0 2024-05-27T09:23:31.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.696+0000 7f9dbf400700 1 --2- 172.21.15.44:0/3242427810 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dac03e570 0x7f9dac040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:31.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.696+0000 7f9dbf400700 1 -- 172.21.15.44:0/3242427810 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9db801e070 con 0x7f9dc8103de0 2024-05-27T09:23:31.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.696+0000 7f9dbd400700 1 --2- 172.21.15.44:0/3242427810 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dac03e570 0x7f9dac040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.697+0000 7f9dbd400700 1 --2- 172.21.15.44:0/3242427810 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dac03e570 0x7f9dac040a80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9db0006fd0 tx=0x7f9db0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:31.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.701+0000 7f9dbf400700 1 -- 172.21.15.44:0/3242427810 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9db8013070 con 0x7f9dc8103de0 2024-05-27T09:23:31.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:31.949+0000 7f9dcd000700 1 -- 172.21.15.44:0/3242427810 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}) -- 0x7f9dc819cea0 con 0x7f9dac03e570 2024-05-27T09:23:31.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:31 smithi044 ceph-mon[31618]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:31.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:31.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:31.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:31.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:31.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi050", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:23:31.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:23:31.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:23:31.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:31 smithi044 ceph-mon[31618]: Updating smithi050:/etc/ceph/ceph.conf 2024-05-27T09:23:33.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:32 smithi044 ceph-mon[31618]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:23:33.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:32 smithi044 ceph-mon[31618]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:23:33.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:32 smithi044 ceph-mon[31618]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:23:33.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:32 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:33.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:32 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:33.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:32 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:34.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:33 smithi044 ceph-mon[31618]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:23:34.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:33 smithi044 ceph-mon[31618]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:34.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:33 smithi044 ceph-mon[31618]: Deploying cephadm binary to smithi189 2024-05-27T09:23:36.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:35 smithi044 ceph-mon[31618]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:36.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:36.961+0000 7f9dbf400700 1 -- 172.21.15.44:0/3242427810 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f9dc819cea0 con 0x7f9dac03e570 2024-05-27T09:23:36.963 INFO:teuthology.orchestra.run.smithi044.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-05-27T09:23:36.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:36.966+0000 7f9dcd000700 1 -- 172.21.15.44:0/3242427810 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dac03e570 msgr2=0x7f9dac040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:36.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:36.966+0000 7f9dcd000700 1 --2- 172.21.15.44:0/3242427810 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dac03e570 0x7f9dac040a80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9db0006fd0 tx=0x7f9db0006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:36.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:36.966+0000 7f9dcd000700 1 -- 172.21.15.44:0/3242427810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dc8103de0 msgr2=0x7f9dc81a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:36.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:36.966+0000 7f9dcd000700 1 --2- 172.21.15.44:0/3242427810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dc8103de0 0x7f9dc81a1750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9db8006010 tx=0x7f9db8005780 comp rx=0 tx=0).stop 2024-05-27T09:23:36.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:36.966+0000 7f9dcd000700 1 -- 172.21.15.44:0/3242427810 shutdown_connections 2024-05-27T09:23:36.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:36.966+0000 7f9dcd000700 1 --2- 172.21.15.44:0/3242427810 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dac03e570 0x7f9dac040a80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:36.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:36.966+0000 7f9dcd000700 1 --2- 172.21.15.44:0/3242427810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dc8103de0 0x7f9dc81a1750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:36.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:36.966+0000 7f9dcd000700 1 -- 172.21.15.44:0/3242427810 >> 172.21.15.44:0/3242427810 conn(0x7f9dc80ff690 msgr2=0x7f9dc8075bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:36.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:36.967+0000 7f9dcd000700 1 -- 172.21.15.44:0/3242427810 shutdown_connections 2024-05-27T09:23:36.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:36.967+0000 7f9dcd000700 1 -- 172.21.15.44:0/3242427810 wait complete. 2024-05-27T09:23:37.661 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-27T09:23:37.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:37 smithi044 ceph-mon[31618]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:37.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:37.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:23:37.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:38.120 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:23:38.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:38 smithi044 ceph-mon[31618]: Added host smithi189 2024-05-27T09:23:38.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:38 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:39 smithi044 ceph-mon[31618]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:40.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.225+0000 7ff169200700 1 -- 172.21.15.44:0/3533756515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff164075d20 msgr2=0x7ff164076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:40.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.225+0000 7ff169200700 1 --2- 172.21.15.44:0/3533756515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff164075d20 0x7ff164076130 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff154009910 tx=0x7ff154009c20 comp rx=0 tx=0).stop 2024-05-27T09:23:40.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.225+0000 7ff169200700 1 -- 172.21.15.44:0/3533756515 shutdown_connections 2024-05-27T09:23:40.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.225+0000 7ff169200700 1 --2- 172.21.15.44:0/3533756515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff164075d20 0x7ff164076130 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:40.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.225+0000 7ff169200700 1 -- 172.21.15.44:0/3533756515 >> 172.21.15.44:0/3533756515 conn(0x7ff1640ff690 msgr2=0x7ff164101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:40.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.225+0000 7ff169200700 1 -- 172.21.15.44:0/3533756515 shutdown_connections 2024-05-27T09:23:40.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.225+0000 7ff169200700 1 -- 172.21.15.44:0/3533756515 wait complete. 2024-05-27T09:23:40.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.227+0000 7ff169200700 1 Processor -- start 2024-05-27T09:23:40.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.227+0000 7ff169200700 1 -- start start 2024-05-27T09:23:40.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.228+0000 7ff169200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff164075d20 0x7ff1641a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.228+0000 7ff169200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff164110330 con 0x7ff164075d20 2024-05-27T09:23:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.228+0000 7ff161e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff164075d20 0x7ff1641a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.229+0000 7ff161e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff164075d20 0x7ff1641a1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45236/0 (socket says 172.21.15.44:45236) 2024-05-27T09:23:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.229+0000 7ff161e00700 1 -- 172.21.15.44:0/3123373374 learned_addr learned my addr 172.21.15.44:0/3123373374 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.229+0000 7ff161e00700 1 -- 172.21.15.44:0/3123373374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1641a1cb0 con 0x7ff164075d20 2024-05-27T09:23:40.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.229+0000 7ff161e00700 1 --2- 172.21.15.44:0/3123373374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff164075d20 0x7ff1641a1770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff154006010 tx=0x7ff154005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:40.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.230+0000 7ff15aa00700 1 -- 172.21.15.44:0/3123373374 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff154026070 con 0x7ff164075d20 2024-05-27T09:23:40.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.230+0000 7ff169200700 1 -- 172.21.15.44:0/3123373374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1641a1f40 con 0x7ff164075d20 2024-05-27T09:23:40.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.230+0000 7ff15aa00700 1 -- 172.21.15.44:0/3123373374 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff15400bc00 con 0x7ff164075d20 2024-05-27T09:23:40.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.230+0000 7ff169200700 1 -- 172.21.15.44:0/3123373374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff16419b990 con 0x7ff164075d20 2024-05-27T09:23:40.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.230+0000 7ff15aa00700 1 -- 172.21.15.44:0/3123373374 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff154003910 con 0x7ff164075d20 2024-05-27T09:23:40.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.231+0000 7ff15aa00700 1 -- 172.21.15.44:0/3123373374 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7ff154029070 con 0x7ff164075d20 2024-05-27T09:23:40.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.232+0000 7ff169200700 1 -- 172.21.15.44:0/3123373374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff144005330 con 0x7ff164075d20 2024-05-27T09:23:40.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.232+0000 7ff15aa00700 1 --2- 172.21.15.44:0/3123373374 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff14803e510 0x7ff148040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:40.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.232+0000 7ff15aa00700 1 -- 172.21.15.44:0/3123373374 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff15401e070 con 0x7ff164075d20 2024-05-27T09:23:40.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.232+0000 7ff161400700 1 --2- 172.21.15.44:0/3123373374 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff14803e510 0x7ff148040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:40.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.233+0000 7ff161400700 1 --2- 172.21.15.44:0/3123373374 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff14803e510 0x7ff148040a20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff150006fd0 tx=0x7ff150006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.238+0000 7ff15aa00700 1 -- 172.21.15.44:0/3123373374 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff154013070 con 0x7ff164075d20 2024-05-27T09:23:40.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.484+0000 7ff169200700 1 -- 172.21.15.44:0/3123373374 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff144000bc0 con 0x7ff14803e510 2024-05-27T09:23:40.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.486+0000 7ff15aa00700 1 -- 172.21.15.44:0/3123373374 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7ff144000bc0 con 0x7ff14803e510 2024-05-27T09:23:40.488 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:23:40.489 INFO:teuthology.orchestra.run.smithi044.stdout:[{"addr": "172.21.15.44", "hostname": "smithi044", "labels": [], "status": ""}, {"addr": "172.21.15.50", "hostname": "smithi050", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-05-27T09:23:40.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.490+0000 7ff169200700 1 -- 172.21.15.44:0/3123373374 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff14803e510 msgr2=0x7ff148040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:40.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.490+0000 7ff169200700 1 --2- 172.21.15.44:0/3123373374 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff14803e510 0x7ff148040a20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff150006fd0 tx=0x7ff150006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:40.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.490+0000 7ff169200700 1 -- 172.21.15.44:0/3123373374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff164075d20 msgr2=0x7ff1641a1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:40.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.490+0000 7ff169200700 1 --2- 172.21.15.44:0/3123373374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff164075d20 0x7ff1641a1770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff154006010 tx=0x7ff154005780 comp rx=0 tx=0).stop 2024-05-27T09:23:40.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.491+0000 7ff169200700 1 -- 172.21.15.44:0/3123373374 shutdown_connections 2024-05-27T09:23:40.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.491+0000 7ff169200700 1 --2- 172.21.15.44:0/3123373374 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff14803e510 0x7ff148040a20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:40.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.491+0000 7ff169200700 1 --2- 172.21.15.44:0/3123373374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff164075d20 0x7ff1641a1770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:40.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.491+0000 7ff169200700 1 -- 172.21.15.44:0/3123373374 >> 172.21.15.44:0/3123373374 conn(0x7ff1640ff690 msgr2=0x7ff164107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:40.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.491+0000 7ff169200700 1 -- 172.21.15.44:0/3123373374 shutdown_connections 2024-05-27T09:23:40.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:40.491+0000 7ff169200700 1 -- 172.21.15.44:0/3123373374 wait complete. 2024-05-27T09:23:41.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:41.245 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-27T09:23:41.246 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-05-27T09:23:41.693 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:23:42.034 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:41 smithi044 ceph-mon[31618]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:42.034 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:41 smithi044 ceph-mon[31618]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:23:43.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.937+0000 7f748c200700 1 -- 172.21.15.44:0/2537448686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7484105ff0 msgr2=0x7f7484106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:43.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.937+0000 7f748c200700 1 --2- 172.21.15.44:0/2537448686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7484105ff0 0x7f7484106400 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7478009910 tx=0x7f7478009c20 comp rx=0 tx=0).stop 2024-05-27T09:23:43.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:43 smithi044 ceph-mon[31618]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:43.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:43.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:43.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:43.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:43.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:23:43.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:23:43.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:23:43.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:43 smithi044 ceph-mon[31618]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-27T09:23:43.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.937+0000 7f748c200700 1 -- 172.21.15.44:0/2537448686 shutdown_connections 2024-05-27T09:23:43.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.937+0000 7f748c200700 1 --2- 172.21.15.44:0/2537448686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7484105ff0 0x7f7484106400 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:43.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.937+0000 7f748c200700 1 -- 172.21.15.44:0/2537448686 >> 172.21.15.44:0/2537448686 conn(0x7f74841018a0 msgr2=0x7f7484103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:43.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.937+0000 7f748c200700 1 -- 172.21.15.44:0/2537448686 shutdown_connections 2024-05-27T09:23:43.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.937+0000 7f748c200700 1 -- 172.21.15.44:0/2537448686 wait complete. 2024-05-27T09:23:43.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.939+0000 7f748c200700 1 Processor -- start 2024-05-27T09:23:43.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.939+0000 7f748c200700 1 -- start start 2024-05-27T09:23:43.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.939+0000 7f748c200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7484105ff0 0x7f74841a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:43.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.939+0000 7f748c200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f74841102e0 con 0x7f7484105ff0 2024-05-27T09:23:43.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.940+0000 7f7489800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7484105ff0 0x7f74841a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:43.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.940+0000 7f7489800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7484105ff0 0x7f74841a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40194/0 (socket says 172.21.15.44:40194) 2024-05-27T09:23:43.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.940+0000 7f7489800700 1 -- 172.21.15.44:0/1394052851 learned_addr learned my addr 172.21.15.44:0/1394052851 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:23:43.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.940+0000 7f7489800700 1 -- 172.21.15.44:0/1394052851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74841a1c60 con 0x7f7484105ff0 2024-05-27T09:23:43.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.941+0000 7f7489800700 1 --2- 172.21.15.44:0/1394052851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7484105ff0 0x7f74841a1720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7478006010 tx=0x7f7478005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:43.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.941+0000 7f7482000700 1 -- 172.21.15.44:0/1394052851 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7478026070 con 0x7f7484105ff0 2024-05-27T09:23:43.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.941+0000 7f7482000700 1 -- 172.21.15.44:0/1394052851 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f747800bba0 con 0x7f7484105ff0 2024-05-27T09:23:43.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.941+0000 7f7482000700 1 -- 172.21.15.44:0/1394052851 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7478003910 con 0x7f7484105ff0 2024-05-27T09:23:43.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.941+0000 7f748c200700 1 -- 172.21.15.44:0/1394052851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74841a1f50 con 0x7f7484105ff0 2024-05-27T09:23:43.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.941+0000 7f748c200700 1 -- 172.21.15.44:0/1394052851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f748419b940 con 0x7f7484105ff0 2024-05-27T09:23:43.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.943+0000 7f748c200700 1 -- 172.21.15.44:0/1394052851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74841074a0 con 0x7f7484105ff0 2024-05-27T09:23:43.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.943+0000 7f7482000700 1 -- 172.21.15.44:0/1394052851 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f7478029070 con 0x7f7484105ff0 2024-05-27T09:23:43.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.944+0000 7f7482000700 1 --2- 172.21.15.44:0/1394052851 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f746c03e570 0x7f746c040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:43.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.944+0000 7f7482000700 1 -- 172.21.15.44:0/1394052851 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f747801e070 con 0x7f7484105ff0 2024-05-27T09:23:43.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.944+0000 7f7488e00700 1 --2- 172.21.15.44:0/1394052851 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f746c03e570 0x7f746c040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:43.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.945+0000 7f7488e00700 1 --2- 172.21.15.44:0/1394052851 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f746c03e570 0x7f746c040a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7470006fd0 tx=0x7f7470006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:43.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:43.950+0000 7f7482000700 1 -- 172.21.15.44:0/1394052851 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7478013070 con 0x7f7484105ff0 2024-05-27T09:23:44.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.201+0000 7f748c200700 1 -- 172.21.15.44:0/1394052851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f748419ce70 con 0x7f7484105ff0 2024-05-27T09:23:44.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.966+0000 7f7482000700 1 -- 172.21.15.44:0/1394052851 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f747800e3e0 con 0x7f7484105ff0 2024-05-27T09:23:44.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.971+0000 7f748c200700 1 -- 172.21.15.44:0/1394052851 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f746c03e570 msgr2=0x7f746c040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:44.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.971+0000 7f748c200700 1 --2- 172.21.15.44:0/1394052851 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f746c03e570 0x7f746c040a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7470006fd0 tx=0x7f7470006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:44.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.971+0000 7f748c200700 1 -- 172.21.15.44:0/1394052851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7484105ff0 msgr2=0x7f74841a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:44.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.971+0000 7f748c200700 1 --2- 172.21.15.44:0/1394052851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7484105ff0 0x7f74841a1720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7478006010 tx=0x7f7478005780 comp rx=0 tx=0).stop 2024-05-27T09:23:44.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.971+0000 7f748c200700 1 -- 172.21.15.44:0/1394052851 shutdown_connections 2024-05-27T09:23:44.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.971+0000 7f748c200700 1 --2- 172.21.15.44:0/1394052851 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f746c03e570 0x7f746c040a80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:44.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.971+0000 7f748c200700 1 --2- 172.21.15.44:0/1394052851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7484105ff0 0x7f74841a1720 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:44.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.971+0000 7f748c200700 1 -- 172.21.15.44:0/1394052851 >> 172.21.15.44:0/1394052851 conn(0x7f74841018a0 msgr2=0x7f7484103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:44.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.971+0000 7f748c200700 1 -- 172.21.15.44:0/1394052851 shutdown_connections 2024-05-27T09:23:44.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:23:44.971+0000 7f748c200700 1 -- 172.21.15.44:0/1394052851 wait complete. 2024-05-27T09:23:44.973 INFO:teuthology.orchestra.run.smithi044.stderr:adjusted tunables profile to default 2024-05-27T09:23:45.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:44 smithi044 ceph-mon[31618]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:23:45.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:44 smithi044 ceph-mon[31618]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:23:45.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:44 smithi044 ceph-mon[31618]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:23:45.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:45.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:45.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:45.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:44 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1394052851' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T09:23:45.770 INFO:tasks.cephadm:Adding mon.a on smithi044 2024-05-27T09:23:45.770 INFO:tasks.cephadm:Adding mon.b on smithi050 2024-05-27T09:23:45.770 INFO:tasks.cephadm:Adding mon.c on smithi189 2024-05-27T09:23:45.771 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch apply mon '3;smithi044:172.21.15.44=a;smithi050:172.21.15.50=b;smithi189:172.21.15.189=c' 2024-05-27T09:23:45.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:45 smithi044 ceph-mon[31618]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:45.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:45 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1394052851' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T09:23:45.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:45 smithi044 ceph-mon[31618]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T09:23:46.005 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:23:47.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.423+0000 7fa074b7d700 1 -- 172.21.15.189:0/1560256522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa070109c30 msgr2=0x7fa07010a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:47.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.423+0000 7fa074b7d700 1 --2- 172.21.15.189:0/1560256522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa070109c30 0x7fa07010a040 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa060009910 tx=0x7fa060009c20 comp rx=0 tx=0).stop 2024-05-27T09:23:47.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.423+0000 7fa074b7d700 1 -- 172.21.15.189:0/1560256522 shutdown_connections 2024-05-27T09:23:47.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.423+0000 7fa074b7d700 1 --2- 172.21.15.189:0/1560256522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa070109c30 0x7fa07010a040 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:47.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.423+0000 7fa074b7d700 1 -- 172.21.15.189:0/1560256522 >> 172.21.15.189:0/1560256522 conn(0x7fa0700ff670 msgr2=0x7fa070101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:47.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.423+0000 7fa074b7d700 1 -- 172.21.15.189:0/1560256522 shutdown_connections 2024-05-27T09:23:47.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.423+0000 7fa074b7d700 1 -- 172.21.15.189:0/1560256522 wait complete. 2024-05-27T09:23:47.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.424+0000 7fa074b7d700 1 Processor -- start 2024-05-27T09:23:47.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.424+0000 7fa074b7d700 1 -- start start 2024-05-27T09:23:47.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.424+0000 7fa074b7d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa070109c30 0x7fa07019f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:47.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.424+0000 7fa074b7d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa07010e110 con 0x7fa070109c30 2024-05-27T09:23:47.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.425+0000 7fa06e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa070109c30 0x7fa07019f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:47.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.425+0000 7fa06e59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa070109c30 0x7fa07019f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.189:54372/0 (socket says 172.21.15.189:54372) 2024-05-27T09:23:47.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.425+0000 7fa06e59c700 1 -- 172.21.15.189:0/324031161 learned_addr learned my addr 172.21.15.189:0/324031161 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-27T09:23:47.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.426+0000 7fa06e59c700 1 -- 172.21.15.189:0/324031161 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa07019fb20 con 0x7fa070109c30 2024-05-27T09:23:47.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.426+0000 7fa06e59c700 1 --2- 172.21.15.189:0/324031161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa070109c30 0x7fa07019f5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0600056d0 tx=0x7fa0600057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:47.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.426+0000 7fa05f7fe700 1 -- 172.21.15.189:0/324031161 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa060027070 con 0x7fa070109c30 2024-05-27T09:23:47.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.426+0000 7fa074b7d700 1 -- 172.21.15.189:0/324031161 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa07019fdb0 con 0x7fa070109c30 2024-05-27T09:23:47.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.426+0000 7fa074b7d700 1 -- 172.21.15.189:0/324031161 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa070199870 con 0x7fa070109c30 2024-05-27T09:23:47.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.427+0000 7fa05f7fe700 1 -- 172.21.15.189:0/324031161 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0600047d0 con 0x7fa070109c30 2024-05-27T09:23:47.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.427+0000 7fa05f7fe700 1 -- 172.21.15.189:0/324031161 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa060031060 con 0x7fa070109c30 2024-05-27T09:23:47.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.428+0000 7fa074b7d700 1 -- 172.21.15.189:0/324031161 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa050005330 con 0x7fa070109c30 2024-05-27T09:23:47.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.428+0000 7fa05f7fe700 1 -- 172.21.15.189:0/324031161 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fa06002a070 con 0x7fa070109c30 2024-05-27T09:23:47.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.428+0000 7fa05f7fe700 1 --2- 172.21.15.189:0/324031161 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa05803e510 0x7fa058040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:47.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.429+0000 7fa05f7fe700 1 -- 172.21.15.189:0/324031161 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa06001e030 con 0x7fa070109c30 2024-05-27T09:23:47.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.429+0000 7fa06dd9b700 1 --2- 172.21.15.189:0/324031161 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa05803e510 0x7fa058040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:47.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.430+0000 7fa06dd9b700 1 --2- 172.21.15.189:0/324031161 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa05803e510 0x7fa058040a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa064006fd0 tx=0x7fa064006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:47.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.445+0000 7fa05f7fe700 1 -- 172.21.15.189:0/324031161 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa060013070 con 0x7fa070109c30 2024-05-27T09:23:47.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.658+0000 7fa074b7d700 1 -- 172.21.15.189:0/324031161 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi050:172.21.15.50=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}) -- 0x7fa050000c80 con 0x7fa05803e510 2024-05-27T09:23:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.675+0000 7fa05f7fe700 1 -- 172.21.15.189:0/324031161 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa050000c80 con 0x7fa05803e510 2024-05-27T09:23:47.675 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mon update... 2024-05-27T09:23:47.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.677+0000 7fa074b7d700 1 -- 172.21.15.189:0/324031161 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa05803e510 msgr2=0x7fa058040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:47.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.677+0000 7fa074b7d700 1 --2- 172.21.15.189:0/324031161 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa05803e510 0x7fa058040a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa064006fd0 tx=0x7fa064006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:47.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.677+0000 7fa074b7d700 1 -- 172.21.15.189:0/324031161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa070109c30 msgr2=0x7fa07019f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:47.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.677+0000 7fa074b7d700 1 --2- 172.21.15.189:0/324031161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa070109c30 0x7fa07019f5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0600056d0 tx=0x7fa0600057b0 comp rx=0 tx=0).stop 2024-05-27T09:23:47.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.677+0000 7fa074b7d700 1 -- 172.21.15.189:0/324031161 shutdown_connections 2024-05-27T09:23:47.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.677+0000 7fa074b7d700 1 --2- 172.21.15.189:0/324031161 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa05803e510 0x7fa058040a20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:47.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.677+0000 7fa074b7d700 1 --2- 172.21.15.189:0/324031161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa070109c30 0x7fa07019f5e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:47.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.677+0000 7fa074b7d700 1 -- 172.21.15.189:0/324031161 >> 172.21.15.189:0/324031161 conn(0x7fa0700ff670 msgr2=0x7fa070100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:47.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.677+0000 7fa074b7d700 1 -- 172.21.15.189:0/324031161 shutdown_connections 2024-05-27T09:23:47.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:47.677+0000 7fa074b7d700 1 -- 172.21.15.189:0/324031161 wait complete. 2024-05-27T09:23:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:47 smithi044 ceph-mon[31618]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:23:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:23:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:23:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:23:48.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:23:48.295 DEBUG:teuthology.orchestra.run.smithi050:mon.b> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.b.service 2024-05-27T09:23:48.298 DEBUG:teuthology.orchestra.run.smithi189:mon.c> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.c.service 2024-05-27T09:23:48.300 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T09:23:48.300 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T09:23:48.872 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:23:49.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:48 smithi044 ceph-mon[31618]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi050:172.21.15.50=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:23:49.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:48 smithi044 ceph-mon[31618]: Saving service mon spec with placement smithi044:172.21.15.44=a;smithi050:172.21.15.50=b;smithi189:172.21.15.189=c;count:3 2024-05-27T09:23:49.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:48 smithi044 ceph-mon[31618]: Deploying daemon mon.c on smithi189 2024-05-27T09:23:50.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:49 smithi044 ceph-mon[31618]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:51.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.541+0000 7f4775997700 1 -- 172.21.15.189:0/94477205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4770108b90 msgr2=0x7f4770108fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:51.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.541+0000 7f4775997700 1 --2- 172.21.15.189:0/94477205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4770108b90 0x7f4770108fa0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4760009910 tx=0x7f4760009c20 comp rx=0 tx=0).stop 2024-05-27T09:23:51.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.541+0000 7f4775997700 1 -- 172.21.15.189:0/94477205 shutdown_connections 2024-05-27T09:23:51.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.541+0000 7f4775997700 1 --2- 172.21.15.189:0/94477205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4770108b90 0x7f4770108fa0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:51.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.541+0000 7f4775997700 1 -- 172.21.15.189:0/94477205 >> 172.21.15.189:0/94477205 conn(0x7f47700fde20 msgr2=0x7f4770100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:51.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.541+0000 7f4775997700 1 -- 172.21.15.189:0/94477205 shutdown_connections 2024-05-27T09:23:51.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.541+0000 7f4775997700 1 -- 172.21.15.189:0/94477205 wait complete. 2024-05-27T09:23:51.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.541+0000 7f4775997700 1 Processor -- start 2024-05-27T09:23:51.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.542+0000 7f4775997700 1 -- start start 2024-05-27T09:23:51.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.542+0000 7f4775997700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4770108b90 0x7f4770103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:51.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.542+0000 7f4775997700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f477010e210 con 0x7f4770108b90 2024-05-27T09:23:51.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.542+0000 7f476effd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4770108b90 0x7f4770103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:51.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.542+0000 7f476effd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4770108b90 0x7f4770103160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.189:54388/0 (socket says 172.21.15.189:54388) 2024-05-27T09:23:51.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.542+0000 7f476effd700 1 -- 172.21.15.189:0/217415574 learned_addr learned my addr 172.21.15.189:0/217415574 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-27T09:23:51.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.543+0000 7f476effd700 1 -- 172.21.15.189:0/217415574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4770104fb0 con 0x7f4770108b90 2024-05-27T09:23:51.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.544+0000 7f476effd700 1 --2- 172.21.15.189:0/217415574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4770108b90 0x7f4770103160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f476000b5c0 tx=0x7f4760005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:51.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.544+0000 7f4774995700 1 -- 172.21.15.189:0/217415574 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4760026070 con 0x7f4770108b90 2024-05-27T09:23:51.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.544+0000 7f4775997700 1 -- 172.21.15.189:0/217415574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47701036a0 con 0x7f4770108b90 2024-05-27T09:23:51.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.544+0000 7f4775997700 1 -- 172.21.15.189:0/217415574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4770103aa0 con 0x7f4770108b90 2024-05-27T09:23:51.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.544+0000 7f4774995700 1 -- 172.21.15.189:0/217415574 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4760004710 con 0x7f4770108b90 2024-05-27T09:23:51.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.544+0000 7f4774995700 1 -- 172.21.15.189:0/217415574 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4760030060 con 0x7f4770108b90 2024-05-27T09:23:51.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.545+0000 7f4775997700 1 -- 172.21.15.189:0/217415574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f475c005330 con 0x7f4770108b90 2024-05-27T09:23:51.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.546+0000 7f4774995700 1 -- 172.21.15.189:0/217415574 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f4760029070 con 0x7f4770108b90 2024-05-27T09:23:51.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.546+0000 7f4774995700 1 --2- 172.21.15.189:0/217415574 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f475803e510 0x7f4758040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:51.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.546+0000 7f4774995700 1 -- 172.21.15.189:0/217415574 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f476001e070 con 0x7f4770108b90 2024-05-27T09:23:51.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.547+0000 7f476e7fc700 1 --2- 172.21.15.189:0/217415574 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f475803e510 0x7f4758040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:51.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.548+0000 7f476e7fc700 1 --2- 172.21.15.189:0/217415574 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f475803e510 0x7f4758040a20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4764006fd0 tx=0x7f4764006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:51.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.562+0000 7f4774995700 1 -- 172.21.15.189:0/217415574 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4760013070 con 0x7f4770108b90 2024-05-27T09:23:51.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.873+0000 7f4775997700 1 -- 172.21.15.189:0/217415574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f475c005620 con 0x7f4770108b90 2024-05-27T09:23:51.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.875+0000 7f4774995700 1 -- 172.21.15.189:0/217415574 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f476000ebd0 con 0x7f4770108b90 2024-05-27T09:23:51.875 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:23:51.875 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","modified":"2024-05-27T09:21:31.930626Z","created":"2024-05-27T09:21:31.930626Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-27T09:23:51.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.877+0000 7f4775997700 1 -- 172.21.15.189:0/217415574 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f475803e510 msgr2=0x7f4758040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:51.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.877+0000 7f4775997700 1 --2- 172.21.15.189:0/217415574 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f475803e510 0x7f4758040a20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4764006fd0 tx=0x7f4764006e40 comp rx=0 tx=0).stop 2024-05-27T09:23:51.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.877+0000 7f4775997700 1 -- 172.21.15.189:0/217415574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4770108b90 msgr2=0x7f4770103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:51.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.877+0000 7f4775997700 1 --2- 172.21.15.189:0/217415574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4770108b90 0x7f4770103160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f476000b5c0 tx=0x7f4760005780 comp rx=0 tx=0).stop 2024-05-27T09:23:51.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.877+0000 7f4775997700 1 -- 172.21.15.189:0/217415574 shutdown_connections 2024-05-27T09:23:51.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.877+0000 7f4775997700 1 --2- 172.21.15.189:0/217415574 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f475803e510 0x7f4758040a20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:51.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.877+0000 7f4775997700 1 --2- 172.21.15.189:0/217415574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4770108b90 0x7f4770103160 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:51.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.877+0000 7f4775997700 1 -- 172.21.15.189:0/217415574 >> 172.21.15.189:0/217415574 conn(0x7f47700fde20 msgr2=0x7f4770105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:51.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.878+0000 7f4775997700 1 -- 172.21.15.189:0/217415574 shutdown_connections 2024-05-27T09:23:51.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:51.878+0000 7f4775997700 1 -- 172.21.15.189:0/217415574 wait complete. 2024-05-27T09:23:51.879 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-05-27T09:23:52.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:51 smithi044 ceph-mon[31618]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:53.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:52 smithi189 ceph-mon[36671]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-27T09:23:53.126 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:52 smithi044 ceph-mon[31618]: from='client.? 172.21.15.189:0/217415574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T09:23:53.619 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T09:23:53.620 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T09:23:53.860 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:23:58.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:58.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:23:58.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: mon.a calling monitor election 2024-05-27T09:23:58.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:58.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: mon.c calling monitor election 2024-05-27T09:23:58.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: monmap epoch 2 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: last_changed 2024-05-27T09:23:52.805643+0000 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: created 2024-05-27T09:21:31.930626+0000 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: min_mon_release 19 (squid) 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: election_strategy: 1 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-05-27T09:23:58.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-27T09:23:58.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: fsmap 2024-05-27T09:23:58.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T09:23:58.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: mgrmap e13: x(active, since 61s) 2024-05-27T09:23:58.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: overall HEALTH_OK 2024-05-27T09:23:58.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:58.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:23:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: mon.a calling monitor election 2024-05-27T09:23:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: mon.c calling monitor election 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: monmap epoch 2 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:23:58.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: last_changed 2024-05-27T09:23:52.805643+0000 2024-05-27T09:23:58.340 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: created 2024-05-27T09:21:31.930626+0000 2024-05-27T09:23:58.340 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: min_mon_release 19 (squid) 2024-05-27T09:23:58.340 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: election_strategy: 1 2024-05-27T09:23:58.340 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-05-27T09:23:58.340 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-27T09:23:58.340 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: fsmap 2024-05-27T09:23:58.340 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T09:23:58.340 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: mgrmap e13: x(active, since 61s) 2024-05-27T09:23:58.340 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: overall HEALTH_OK 2024-05-27T09:23:58.340 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:58.340 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:58.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.365+0000 7f8cc08b8700 1 -- 172.21.15.189:0/3904858209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ca80035c0 msgr2=0x7f8ca8005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:58.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.365+0000 7f8cc08b8700 1 --2- 172.21.15.189:0/3904858209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ca80035c0 0x7f8ca8005aa0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc06b4c0 tx=0x7f8cac00feb0 comp rx=0 tx=0).stop 2024-05-27T09:23:58.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.365+0000 7f8cc08b8700 1 -- 172.21.15.189:0/3904858209 shutdown_connections 2024-05-27T09:23:58.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.365+0000 7f8cc08b8700 1 --2- 172.21.15.189:0/3904858209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ca80035c0 0x7f8ca8005aa0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:58.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.365+0000 7f8cc08b8700 1 -- 172.21.15.189:0/3904858209 >> 172.21.15.189:0/3904858209 conn(0x7f8cbc077e80 msgr2=0x7f8cbc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:58.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.365+0000 7f8cc08b8700 1 -- 172.21.15.189:0/3904858209 shutdown_connections 2024-05-27T09:23:58.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.365+0000 7f8cc08b8700 1 -- 172.21.15.189:0/3904858209 wait complete. 2024-05-27T09:23:58.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.366+0000 7f8cc08b8700 1 Processor -- start 2024-05-27T09:23:58.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.366+0000 7f8cc08b8700 1 -- start start 2024-05-27T09:23:58.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cc08b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca80035c0 0x7f8cbc19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:58.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cc08b8700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8cbc19fd70 0x7f8cbc199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cc08b8700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc10ad00 con 0x7f8cbc19fd70 2024-05-27T09:23:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cc08b8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc10ab80 con 0x7f8ca80035c0 2024-05-27T09:23:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cba59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca80035c0 0x7f8cbc19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cba59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca80035c0 0x7f8cbc19f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47636/0 (socket says 172.21.15.189:47636) 2024-05-27T09:23:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cba59c700 1 -- 172.21.15.189:0/2298025542 learned_addr learned my addr 172.21.15.189:0/2298025542 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-27T09:23:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cba59c700 1 -- 172.21.15.189:0/2298025542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca80035c0 msgr2=0x7f8cbc19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-27T09:23:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cba59c700 1 -- 172.21.15.189:0/2298025542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca80035c0 msgr2=0x7f8cbc19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-27T09:23:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cba59c700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca80035c0 0x7f8cbc19f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T09:23:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cba59c700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca80035c0 0x7f8cbc19f830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T09:23:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.367+0000 7f8cb1d9b700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8cbc19fd70 0x7f8cbc199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:58.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.368+0000 7f8cb1d9b700 1 -- 172.21.15.189:0/2298025542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca80035c0 msgr2=0x7f8cbc19f830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:23:58.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.368+0000 7f8cb1d9b700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca80035c0 0x7f8cbc19f830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:58.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.368+0000 7f8cb1d9b700 1 -- 172.21.15.189:0/2298025542 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc1a0340 con 0x7f8cbc19fd70 2024-05-27T09:23:58.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.369+0000 7f8cb1d9b700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8cbc19fd70 0x7f8cbc199a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8ca400b4d0 tx=0x7f8ca400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:58.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.369+0000 7f8cb3fff700 1 -- 172.21.15.189:0/2298025542 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8ca4011630 con 0x7f8cbc19fd70 2024-05-27T09:23:58.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.369+0000 7f8cc08b8700 1 -- 172.21.15.189:0/2298025542 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cbc199ff0 con 0x7f8cbc19fd70 2024-05-27T09:23:58.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.369+0000 7f8cc08b8700 1 -- 172.21.15.189:0/2298025542 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cbc19a580 con 0x7f8cbc19fd70 2024-05-27T09:23:58.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.369+0000 7f8cb3fff700 1 -- 172.21.15.189:0/2298025542 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ca4011c80 con 0x7f8cbc19fd70 2024-05-27T09:23:58.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.369+0000 7f8cb3fff700 1 -- 172.21.15.189:0/2298025542 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8ca401adc0 con 0x7f8cbc19fd70 2024-05-27T09:23:58.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.370+0000 7f8cc08b8700 1 -- 172.21.15.189:0/2298025542 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ca8002160 con 0x7f8cbc19fd70 2024-05-27T09:23:58.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.371+0000 7f8cb3fff700 1 -- 172.21.15.189:0/2298025542 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f8ca4018020 con 0x7f8cbc19fd70 2024-05-27T09:23:58.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.371+0000 7f8cb3fff700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8c9c03e2b0 0x7f8c9c0407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:23:58.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.371+0000 7f8cb3fff700 1 -- 172.21.15.189:0/2298025542 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8ca4052170 con 0x7f8cbc19fd70 2024-05-27T09:23:58.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.372+0000 7f8cba59c700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8c9c03e2b0 0x7f8c9c0407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:23:58.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.373+0000 7f8cba59c700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8c9c03e2b0 0x7f8c9c0407c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00fa60 tx=0x7f8cac005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:23:58.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.388+0000 7f8cb3fff700 1 -- 172.21.15.189:0/2298025542 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8ca401f020 con 0x7f8cbc19fd70 2024-05-27T09:23:58.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.677+0000 7f8cc08b8700 1 -- 172.21.15.189:0/2298025542 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f8ca8002c50 con 0x7f8cbc19fd70 2024-05-27T09:23:58.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.678+0000 7f8cb3fff700 1 -- 172.21.15.189:0/2298025542 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f8ca403f410 con 0x7f8cbc19fd70 2024-05-27T09:23:58.678 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:23:58.679 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","modified":"2024-05-27T09:23:52.805643Z","created":"2024-05-27T09:21:31.930626Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-27T09:23:58.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.680+0000 7f8cc08b8700 1 -- 172.21.15.189:0/2298025542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8c9c03e2b0 msgr2=0x7f8c9c0407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:58.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.680+0000 7f8cc08b8700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8c9c03e2b0 0x7f8c9c0407c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00fa60 tx=0x7f8cac005960 comp rx=0 tx=0).stop 2024-05-27T09:23:58.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.680+0000 7f8cc08b8700 1 -- 172.21.15.189:0/2298025542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8cbc19fd70 msgr2=0x7f8cbc199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:23:58.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.680+0000 7f8cc08b8700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8cbc19fd70 0x7f8cbc199a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8ca400b4d0 tx=0x7f8ca400b890 comp rx=0 tx=0).stop 2024-05-27T09:23:58.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.680+0000 7f8cc08b8700 1 -- 172.21.15.189:0/2298025542 shutdown_connections 2024-05-27T09:23:58.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.680+0000 7f8cc08b8700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8c9c03e2b0 0x7f8c9c0407c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:58.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.680+0000 7f8cc08b8700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca80035c0 0x7f8cbc19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:58.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.680+0000 7f8cc08b8700 1 --2- 172.21.15.189:0/2298025542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8cbc19fd70 0x7f8cbc199a50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:23:58.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.680+0000 7f8cc08b8700 1 -- 172.21.15.189:0/2298025542 >> 172.21.15.189:0/2298025542 conn(0x7f8cbc077e80 msgr2=0x7f8cbc1071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:23:58.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.680+0000 7f8cc08b8700 1 -- 172.21.15.189:0/2298025542 shutdown_connections 2024-05-27T09:23:58.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:23:58.680+0000 7f8cc08b8700 1 -- 172.21.15.189:0/2298025542 wait complete. 2024-05-27T09:23:58.682 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-05-27T09:23:58.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:58 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:58.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:58 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:23:58.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:58 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:23:58.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:58 smithi189 ceph-mon[36671]: Deploying daemon mon.b on smithi050 2024-05-27T09:23:58.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:58 smithi189 ceph-mon[36671]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:58.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:58 smithi189 ceph-mon[36671]: from='client.? 172.21.15.189:0/2298025542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T09:23:58.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:23:58 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:23:59.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:58 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:23:59.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:58 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:23:59.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:58 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:23:59.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:58 smithi044 ceph-mon[31618]: Deploying daemon mon.b on smithi050 2024-05-27T09:23:59.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:58 smithi044 ceph-mon[31618]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:23:59.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:58 smithi044 ceph-mon[31618]: from='client.? 172.21.15.189:0/2298025542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T09:23:59.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:23:58 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:00.223 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:23:59 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:23:59.806+0000 7f6e31e00700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-27T09:24:00.282 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T09:24:00.286 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T09:24:00.530 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:24:01.714 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:01 smithi189 ceph-mon[36671]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:01.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.943+0000 7f5d239ed700 1 -- 172.21.15.189:0/1509811022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d1c102c10 msgr2=0x7f5d1c103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:01.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.943+0000 7f5d239ed700 1 --2- 172.21.15.189:0/1509811022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d1c102c10 0x7f5d1c103020 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5d18009910 tx=0x7f5d18009c20 comp rx=0 tx=0).stop 2024-05-27T09:24:01.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.943+0000 7f5d239ed700 1 -- 172.21.15.189:0/1509811022 shutdown_connections 2024-05-27T09:24:01.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.943+0000 7f5d239ed700 1 --2- 172.21.15.189:0/1509811022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d1c102c10 0x7f5d1c103020 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:01.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.943+0000 7f5d239ed700 1 -- 172.21.15.189:0/1509811022 >> 172.21.15.189:0/1509811022 conn(0x7f5d1c0797d0 msgr2=0x7f5d1c079c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:24:01.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.943+0000 7f5d239ed700 1 -- 172.21.15.189:0/1509811022 shutdown_connections 2024-05-27T09:24:01.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.943+0000 7f5d239ed700 1 -- 172.21.15.189:0/1509811022 wait complete. 2024-05-27T09:24:01.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.943+0000 7f5d239ed700 1 Processor -- start 2024-05-27T09:24:01.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.943+0000 7f5d239ed700 1 -- start start 2024-05-27T09:24:01.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d239ed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d1c102c10 0x7f5d1c1a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:01.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d239ed700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d1c1a07e0 0x7f5d1c1a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:01.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d239ed700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c114810 con 0x7f5d1c1a07e0 2024-05-27T09:24:01.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d239ed700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c114690 con 0x7f5d1c102c10 2024-05-27T09:24:01.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d21789700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d1c102c10 0x7f5d1c1a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:01.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d21789700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d1c102c10 0x7f5d1c1a02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47640/0 (socket says 172.21.15.189:47640) 2024-05-27T09:24:01.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d21789700 1 -- 172.21.15.189:0/600027892 learned_addr learned my addr 172.21.15.189:0/600027892 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-27T09:24:01.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d21789700 1 -- 172.21.15.189:0/600027892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d1c102c10 msgr2=0x7f5d1c1a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-27T09:24:01.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d21789700 1 -- 172.21.15.189:0/600027892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d1c102c10 msgr2=0x7f5d1c1a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-27T09:24:01.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d21789700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d1c102c10 0x7f5d1c1a02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T09:24:01.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d21789700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d1c102c10 0x7f5d1c1a02a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T09:24:01.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.944+0000 7f5d20f88700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d1c1a07e0 0x7f5d1c1a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:01.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.945+0000 7f5d20f88700 1 -- 172.21.15.189:0/600027892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d1c102c10 msgr2=0x7f5d1c1a02a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:24:01.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.945+0000 7f5d20f88700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d1c102c10 0x7f5d1c1a02a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:01.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.945+0000 7f5d20f88700 1 -- 172.21.15.189:0/600027892 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1c1a0de0 con 0x7f5d1c1a07e0 2024-05-27T09:24:01.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.945+0000 7f5d20f88700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d1c1a07e0 0x7f5d1c1a4ad0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c00b4d0 tx=0x7f5d0c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:24:01.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.946+0000 7f5d127fc700 1 -- 172.21.15.189:0/600027892 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f5d0c011630 con 0x7f5d1c1a07e0 2024-05-27T09:24:01.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.946+0000 7f5d127fc700 1 -- 172.21.15.189:0/600027892 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d0c011c80 con 0x7f5d1c1a07e0 2024-05-27T09:24:01.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.946+0000 7f5d239ed700 1 -- 172.21.15.189:0/600027892 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d1c1a5010 con 0x7f5d1c1a07e0 2024-05-27T09:24:01.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.946+0000 7f5d127fc700 1 -- 172.21.15.189:0/600027892 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f5d0c010da0 con 0x7f5d1c1a07e0 2024-05-27T09:24:01.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.946+0000 7f5d239ed700 1 -- 172.21.15.189:0/600027892 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d1c1a55a0 con 0x7f5d1c1a07e0 2024-05-27T09:24:01.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.947+0000 7f5d239ed700 1 -- 172.21.15.189:0/600027892 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d1c1a5910 con 0x7f5d1c1a07e0 2024-05-27T09:24:01.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.948+0000 7f5d127fc700 1 -- 172.21.15.189:0/600027892 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f5d0c018020 con 0x7f5d1c1a07e0 2024-05-27T09:24:01.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.948+0000 7f5d127fc700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d0803e2b0 0x7f5d080407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:01.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.948+0000 7f5d127fc700 1 -- 172.21.15.189:0/600027892 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5d0c052570 con 0x7f5d1c1a07e0 2024-05-27T09:24:01.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.949+0000 7f5d21789700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d0803e2b0 0x7f5d080407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:01.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.950+0000 7f5d21789700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d0803e2b0 0x7f5d080407c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5d18000c00 tx=0x7f5d1801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:24:01.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:01.964+0000 7f5d127fc700 1 -- 172.21.15.189:0/600027892 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5d0c01f020 con 0x7f5d1c1a07e0 2024-05-27T09:24:01.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:01 smithi044 ceph-mon[31618]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.020 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-27T09:24:02.020 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-27T09:24:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:02.029+0000 7f5d127fc700 1 -- 172.21.15.189:0/600027892 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c022580 con 0x7f5d1c1a07e0 2024-05-27T09:24:02.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:02.255+0000 7f5d239ed700 1 -- 172.21.15.189:0/600027892 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5d1c1a1600 con 0x7f5d1c1a07e0 2024-05-27T09:24:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-27T09:24:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-27T09:24:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-27T09:24:02.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-27T09:24:02.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T09:24:02.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T09:24:02.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T09:24:02.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T09:24:02.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Added host smithi050 2024-05-27T09:24:02.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:02.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi050", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:02.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Updating smithi050:/etc/ceph/ceph.conf 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Deploying cephadm binary to smithi189 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Added host smithi189 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:24:02.299 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:02.299 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:02.299 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-27T09:24:02.299 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:24:02.299 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:24:02.299 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:24:02.299 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.299 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.299 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.300 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1394052851' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T09:24:02.300 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.300 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1394052851' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T09:24:02.300 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T09:24:02.300 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.300 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.301 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:02.301 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:02.301 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:02.301 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.301 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:24:02.301 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:02.301 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi050:172.21.15.50=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:24:02.301 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Saving service mon spec with placement smithi044:172.21.15.44=a;smithi050:172.21.15.50=b;smithi189:172.21.15.189=c;count:3 2024-05-27T09:24:02.301 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Deploying daemon mon.c on smithi189 2024-05-27T09:24:02.301 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.301 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='client.? 172.21.15.189:0/217415574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.a calling monitor election 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.c calling monitor election 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.302 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: monmap epoch 2 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: last_changed 2024-05-27T09:23:52.805643+0000 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: created 2024-05-27T09:21:31.930626+0000 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: min_mon_release 19 (squid) 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: election_strategy: 1 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: fsmap 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mgrmap e13: x(active, since 61s) 2024-05-27T09:24:02.303 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: overall HEALTH_OK 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: Deploying daemon mon.b on smithi050 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='client.? 172.21.15.189:0/2298025542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 ceph-mon[36371]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 bash[36353]: c51b05fd7b42cf0e1610a373598af8b2330671bd24c118dd93fadf4fd654103b 2024-05-27T09:24:02.304 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:02 smithi050 systemd[1]: Started Ceph mon.b for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:24:07.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.051+0000 7f5d127fc700 1 -- 172.21.15.189:0/600027892 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f5d0c050f40 con 0x7f5d1c1a07e0 2024-05-27T09:24:07.053 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-05-27T09:24:07.053 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":3,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","modified":"2024-05-27T09:24:02.026636Z","created":"2024-05-27T09:21:31.930626Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-27T09:24:07.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.054+0000 7f5d239ed700 1 -- 172.21.15.189:0/600027892 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d0803e2b0 msgr2=0x7f5d080407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:07.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.054+0000 7f5d239ed700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d0803e2b0 0x7f5d080407c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5d18000c00 tx=0x7f5d1801a040 comp rx=0 tx=0).stop 2024-05-27T09:24:07.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.054+0000 7f5d239ed700 1 -- 172.21.15.189:0/600027892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d1c1a07e0 msgr2=0x7f5d1c1a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:07.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.054+0000 7f5d239ed700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d1c1a07e0 0x7f5d1c1a4ad0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c00b4d0 tx=0x7f5d0c00b890 comp rx=0 tx=0).stop 2024-05-27T09:24:07.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.054+0000 7f5d239ed700 1 -- 172.21.15.189:0/600027892 shutdown_connections 2024-05-27T09:24:07.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.055+0000 7f5d239ed700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d0803e2b0 0x7f5d080407c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:07.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.055+0000 7f5d239ed700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d1c102c10 0x7f5d1c1a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:07.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.055+0000 7f5d239ed700 1 --2- 172.21.15.189:0/600027892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d1c1a07e0 0x7f5d1c1a4ad0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:07.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.055+0000 7f5d239ed700 1 -- 172.21.15.189:0/600027892 >> 172.21.15.189:0/600027892 conn(0x7f5d1c0797d0 msgr2=0x7f5d1c0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:24:07.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.055+0000 7f5d239ed700 1 -- 172.21.15.189:0/600027892 shutdown_connections 2024-05-27T09:24:07.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:07.055+0000 7f5d239ed700 1 -- 172.21.15.189:0/600027892 wait complete. 2024-05-27T09:24:07.056 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 3 2024-05-27T09:24:07.311 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:24:07.311 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: mon.a calling monitor election 2024-05-27T09:24:07.311 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: mon.c calling monitor election 2024-05-27T09:24:07.311 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.311 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:07.311 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: from='client.? 172.21.15.189:0/600027892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T09:24:07.311 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:07.311 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.311 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.311 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: monmap epoch 3 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: last_changed 2024-05-27T09:24:02.026636+0000 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: created 2024-05-27T09:21:31.930626+0000 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: min_mon_release 19 (squid) 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: election_strategy: 1 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-05-27T09:24:07.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-27T09:24:07.313 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: 2: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-05-27T09:24:07.313 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: fsmap 2024-05-27T09:24:07.313 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T09:24:07.313 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: mgrmap e13: x(active, since 70s) 2024-05-27T09:24:07.313 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: overall HEALTH_OK 2024-05-27T09:24:07.313 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:07.313 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:07 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:07.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:24:07.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: mon.a calling monitor election 2024-05-27T09:24:07.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: mon.c calling monitor election 2024-05-27T09:24:07.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:07.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: from='client.? 172.21.15.189:0/600027892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T09:24:07.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:07.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:07.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:07.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:07.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T09:24:07.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: monmap epoch 3 2024-05-27T09:24:07.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: last_changed 2024-05-27T09:24:02.026636+0000 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: created 2024-05-27T09:21:31.930626+0000 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: min_mon_release 19 (squid) 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: election_strategy: 1 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: 2: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: fsmap 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: mgrmap e13: x(active, since 70s) 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: overall HEALTH_OK 2024-05-27T09:24:07.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:07.479 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:07 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:07.624 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-27T09:24:07.625 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-05-27T09:24:08.052 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:24:08.409 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:08.409 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:08.409 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:08.409 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:08.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:08.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:08.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:08.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:09.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:24:09.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: mon.a calling monitor election 2024-05-27T09:24:09.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: mon.c calling monitor election 2024-05-27T09:24:09.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:09.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:24:09.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='client.? 172.21.15.189:0/600027892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: monmap epoch 3 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: last_changed 2024-05-27T09:24:02.026636+0000 2024-05-27T09:24:09.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: created 2024-05-27T09:21:31.930626+0000 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: min_mon_release 19 (squid) 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: election_strategy: 1 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: 2: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: fsmap 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: mgrmap e13: x(active, since 70s) 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: overall HEALTH_OK 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:09.296 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:09.297 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:09.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: mon.b calling monitor election 2024-05-27T09:24:09.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: mon.b calling monitor election 2024-05-27T09:24:09.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: mon.c calling monitor election 2024-05-27T09:24:09.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: mon.a calling monitor election 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: monmap epoch 3 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: last_changed 2024-05-27T09:24:02.026636+0000 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: created 2024-05-27T09:21:31.930626+0000 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: min_mon_release 19 (squid) 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: election_strategy: 1 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: 2: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: fsmap 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: mgrmap e13: x(active, since 72s) 2024-05-27T09:24:09.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: overall HEALTH_OK 2024-05-27T09:24:09.912 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:09.912 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:09.912 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:09.912 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: mon.b calling monitor election 2024-05-27T09:24:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: mon.b calling monitor election 2024-05-27T09:24:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: mon.c calling monitor election 2024-05-27T09:24:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: mon.a calling monitor election 2024-05-27T09:24:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T09:24:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: monmap epoch 3 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: last_changed 2024-05-27T09:24:02.026636+0000 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: created 2024-05-27T09:21:31.930626+0000 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: min_mon_release 19 (squid) 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: election_strategy: 1 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: 2: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: fsmap 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: mgrmap e13: x(active, since 72s) 2024-05-27T09:24:10.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: overall HEALTH_OK 2024-05-27T09:24:10.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: mon.b calling monitor election 2024-05-27T09:24:10.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: mon.b calling monitor election 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: mon.c calling monitor election 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: mon.a calling monitor election 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: monmap epoch 3 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: last_changed 2024-05-27T09:24:02.026636+0000 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: created 2024-05-27T09:21:31.930626+0000 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: min_mon_release 19 (squid) 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: election_strategy: 1 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-05-27T09:24:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: 2: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-05-27T09:24:10.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: fsmap 2024-05-27T09:24:10.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T09:24:10.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: mgrmap e13: x(active, since 72s) 2024-05-27T09:24:10.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: overall HEALTH_OK 2024-05-27T09:24:10.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.149+0000 7fb36fe00700 1 -- 172.21.15.44:0/1160338100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368104d40 msgr2=0x7fb368105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:10.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.149+0000 7fb36fe00700 1 --2- 172.21.15.44:0/1160338100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368104d40 0x7fb368105150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009910 tx=0x7fb35c009c20 comp rx=0 tx=0).stop 2024-05-27T09:24:10.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.149+0000 7fb36fe00700 1 -- 172.21.15.44:0/1160338100 shutdown_connections 2024-05-27T09:24:10.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.149+0000 7fb36fe00700 1 --2- 172.21.15.44:0/1160338100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368104d40 0x7fb368105150 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:10.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.149+0000 7fb36fe00700 1 -- 172.21.15.44:0/1160338100 >> 172.21.15.44:0/1160338100 conn(0x7fb3680785a0 msgr2=0x7fb368078a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:24:10.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.149+0000 7fb36fe00700 1 -- 172.21.15.44:0/1160338100 shutdown_connections 2024-05-27T09:24:10.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.149+0000 7fb36fe00700 1 -- 172.21.15.44:0/1160338100 wait complete. 2024-05-27T09:24:10.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.150+0000 7fb36fe00700 1 Processor -- start 2024-05-27T09:24:10.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.151+0000 7fb36fe00700 1 -- start start 2024-05-27T09:24:10.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.151+0000 7fb36fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb368104d40 0x7fb3681a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:10.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.151+0000 7fb36fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3681a07c0 0x7fb3681a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:10.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.151+0000 7fb36fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3681a4ff0 0x7fb3681a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:10.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.151+0000 7fb36fe00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb36810e130 con 0x7fb3681a4ff0 2024-05-27T09:24:10.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36fe00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb36810dfb0 con 0x7fb3681a07c0 2024-05-27T09:24:10.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb36810e2b0 con 0x7fb368104d40 2024-05-27T09:24:10.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb368104d40 0x7fb3681a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:10.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb368104d40 0x7fb3681a0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:54188/0 (socket says 172.21.15.44:54188) 2024-05-27T09:24:10.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36d400700 1 -- 172.21.15.44:0/2850136377 learned_addr learned my addr 172.21.15.44:0/2850136377 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:24:10.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36de00700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3681a4ff0 0x7fb3681a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:10.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36ca00700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3681a07c0 0x7fb3681a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:10.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36d400700 1 -- 172.21.15.44:0/2850136377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3681a07c0 msgr2=0x7fb3681a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:10.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36d400700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3681a07c0 0x7fb3681a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:10.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36d400700 1 -- 172.21.15.44:0/2850136377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3681a4ff0 msgr2=0x7fb3681a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:10.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36d400700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3681a4ff0 0x7fb3681a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:10.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36d400700 1 -- 172.21.15.44:0/2850136377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3681acd60 con 0x7fb368104d40 2024-05-27T09:24:10.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.152+0000 7fb36ca00700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3681a07c0 0x7fb3681a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:24:10.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.153+0000 7fb36d400700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb368104d40 0x7fb3681a0280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb35c004f80 tx=0x7fb35c004fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:24:10.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.153+0000 7fb366000700 1 -- 172.21.15.44:0/2850136377 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c026070 con 0x7fb368104d40 2024-05-27T09:24:10.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.153+0000 7fb366000700 1 -- 172.21.15.44:0/2850136377 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb35c00b750 con 0x7fb368104d40 2024-05-27T09:24:10.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.153+0000 7fb366000700 1 -- 172.21.15.44:0/2850136377 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c003970 con 0x7fb368104d40 2024-05-27T09:24:10.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.154+0000 7fb36fe00700 1 -- 172.21.15.44:0/2850136377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3681acf30 con 0x7fb368104d40 2024-05-27T09:24:10.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.154+0000 7fb36fe00700 1 -- 172.21.15.44:0/2850136377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3681ad330 con 0x7fb368104d40 2024-05-27T09:24:10.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.155+0000 7fb36fe00700 1 -- 172.21.15.44:0/2850136377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb368064e10 con 0x7fb368104d40 2024-05-27T09:24:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.155+0000 7fb366000700 1 -- 172.21.15.44:0/2850136377 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fb35c003b10 con 0x7fb368104d40 2024-05-27T09:24:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.156+0000 7fb366000700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb35003e9a0 0x7fb350040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.156+0000 7fb366000700 1 -- 172.21.15.44:0/2850136377 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb35c053910 con 0x7fb368104d40 2024-05-27T09:24:10.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.156+0000 7fb36ca00700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb35003e9a0 0x7fb350040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.157+0000 7fb36ca00700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb35003e9a0 0x7fb350040eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb368101d00 tx=0x7fb354008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:24:10.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.161+0000 7fb366000700 1 -- 172.21.15.44:0/2850136377 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb35c013070 con 0x7fb368104d40 2024-05-27T09:24:10.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.403+0000 7fb36fe00700 1 -- 172.21.15.44:0/2850136377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fb3681ad780 con 0x7fb368104d40 2024-05-27T09:24:10.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.404+0000 7fb366000700 1 -- 172.21.15.44:0/2850136377 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7fb35c02f020 con 0x7fb368104d40 2024-05-27T09:24:10.405 INFO:teuthology.orchestra.run.smithi044.stdout:# minimal ceph.conf for 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:24:10.405 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-05-27T09:24:10.405 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:24:10.405 INFO:teuthology.orchestra.run.smithi044.stdout: mon_host = [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-05-27T09:24:10.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.407+0000 7fb36fe00700 1 -- 172.21.15.44:0/2850136377 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb35003e9a0 msgr2=0x7fb350040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:10.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.407+0000 7fb36fe00700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb35003e9a0 0x7fb350040eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb368101d00 tx=0x7fb354008040 comp rx=0 tx=0).stop 2024-05-27T09:24:10.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.407+0000 7fb36fe00700 1 -- 172.21.15.44:0/2850136377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb368104d40 msgr2=0x7fb3681a0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:10.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.407+0000 7fb36fe00700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb368104d40 0x7fb3681a0280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb35c004f80 tx=0x7fb35c004fb0 comp rx=0 tx=0).stop 2024-05-27T09:24:10.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.407+0000 7fb36fe00700 1 -- 172.21.15.44:0/2850136377 shutdown_connections 2024-05-27T09:24:10.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.407+0000 7fb36fe00700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb35003e9a0 0x7fb350040eb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:10.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.407+0000 7fb36fe00700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3681a4ff0 0x7fb3681a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:10.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.407+0000 7fb36fe00700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb368104d40 0x7fb3681a0280 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:10.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.407+0000 7fb36fe00700 1 --2- 172.21.15.44:0/2850136377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3681a07c0 0x7fb3681a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:10.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.407+0000 7fb36fe00700 1 -- 172.21.15.44:0/2850136377 >> 172.21.15.44:0/2850136377 conn(0x7fb3680785a0 msgr2=0x7fb368102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:24:10.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.407+0000 7fb36fe00700 1 -- 172.21.15.44:0/2850136377 shutdown_connections 2024-05-27T09:24:10.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:10.408+0000 7fb36fe00700 1 -- 172.21.15.44:0/2850136377 wait complete. 2024-05-27T09:24:10.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:24:10.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:24:10.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: Updating smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:24:10.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:10.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2850136377' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:10.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:10.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:24:10.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T09:24:10.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:10 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:11.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:24:11.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:24:11.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: Updating smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:24:11.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:11.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2850136377' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:11.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:24:11.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T09:24:11.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:10 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:11.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:24:11.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:24:11.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: Updating smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:24:11.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:24:11.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2850136377' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:11.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:24:11.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T09:24:11.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:10 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:11.183 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-27T09:24:11.183 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:24:11.184 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T09:24:11.247 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:24:11.247 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:24:11.371 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-27T09:24:11.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T09:24:11.403 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-27T09:24:11.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:24:11.472 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-27T09:24:11.473 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T09:24:11.505 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-27T09:24:11.505 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:24:11.575 INFO:tasks.cephadm:Adding mgr.x on smithi044 2024-05-27T09:24:11.575 INFO:tasks.cephadm:Adding mgr.y on smithi050 2024-05-27T09:24:11.575 INFO:tasks.cephadm:Adding mgr.z on smithi189 2024-05-27T09:24:11.576 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch apply mgr '3;smithi044=x;smithi050=y;smithi189=z' 2024-05-27T09:24:11.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:11 smithi189 ceph-mon[36671]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:11.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:11 smithi189 ceph-mon[36671]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T09:24:11.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:11 smithi189 ceph-mon[36671]: Reconfiguring daemon mon.a on smithi044 2024-05-27T09:24:11.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:11 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:11.849 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:24:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:11 smithi044 ceph-mon[31618]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:11 smithi044 ceph-mon[31618]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T09:24:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:11 smithi044 ceph-mon[31618]: Reconfiguring daemon mon.a on smithi044 2024-05-27T09:24:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:11 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:12.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:11 smithi050 ceph-mon[36371]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:12.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:11 smithi050 ceph-mon[36371]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T09:24:12.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:11 smithi050 ceph-mon[36371]: Reconfiguring daemon mon.a on smithi044 2024-05-27T09:24:12.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:11 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:13.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.280+0000 7fcf97d28700 1 -- 172.21.15.189:0/1749376142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf90102be0 msgr2=0x7fcf90102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:13.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.280+0000 7fcf97d28700 1 --2- 172.21.15.189:0/1749376142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf90102be0 0x7fcf90102ff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00b0a0 tx=0x7fcf8c00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:24:13.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.280+0000 7fcf97d28700 1 -- 172.21.15.189:0/1749376142 shutdown_connections 2024-05-27T09:24:13.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.280+0000 7fcf97d28700 1 --2- 172.21.15.189:0/1749376142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf90102be0 0x7fcf90102ff0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:13.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.280+0000 7fcf97d28700 1 -- 172.21.15.189:0/1749376142 >> 172.21.15.189:0/1749376142 conn(0x7fcf900fe240 msgr2=0x7fcf901006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:24:13.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.280+0000 7fcf97d28700 1 -- 172.21.15.189:0/1749376142 shutdown_connections 2024-05-27T09:24:13.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.280+0000 7fcf97d28700 1 -- 172.21.15.189:0/1749376142 wait complete. 2024-05-27T09:24:13.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.280+0000 7fcf97d28700 1 Processor -- start 2024-05-27T09:24:13.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.280+0000 7fcf97d28700 1 -- start start 2024-05-27T09:24:13.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf97d28700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf90102be0 0x7fcf9006c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:13.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf97d28700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf9019c6f0 0x7fcf9019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:13.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf97d28700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf9019f050 0x7fcf901a1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf97d28700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcf90108b60 con 0x7fcf9019f050 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf97d28700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcf901089e0 con 0x7fcf90102be0 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf97d28700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcf90108ce0 con 0x7fcf9019c6f0 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf952c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf9019c6f0 0x7fcf9019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf952c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf9019c6f0 0x7fcf9019eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58154/0 (socket says 172.21.15.189:58154) 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf952c3700 1 -- 172.21.15.189:0/1799032295 learned_addr learned my addr 172.21.15.189:0/1799032295 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf95ac4700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf90102be0 0x7fcf9006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf952c3700 1 -- 172.21.15.189:0/1799032295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf90102be0 msgr2=0x7fcf9006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf952c3700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf90102be0 0x7fcf9006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf952c3700 1 -- 172.21.15.189:0/1799032295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf9019f050 msgr2=0x7fcf901a1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf952c3700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf9019f050 0x7fcf901a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf952c3700 1 -- 172.21.15.189:0/1799032295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf901a1cc0 con 0x7fcf9019c6f0 2024-05-27T09:24:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf95ac4700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf90102be0 0x7fcf9006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:24:13.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.281+0000 7fcf952c3700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf9019c6f0 0x7fcf9019eb10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000b780 tx=0x7fcf8000bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:24:13.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.282+0000 7fcf86ffd700 1 -- 172.21.15.189:0/1799032295 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf80004230 con 0x7fcf9019c6f0 2024-05-27T09:24:13.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.282+0000 7fcf86ffd700 1 -- 172.21.15.189:0/1799032295 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf800043d0 con 0x7fcf9019c6f0 2024-05-27T09:24:13.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.282+0000 7fcf97d28700 1 -- 172.21.15.189:0/1799032295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf901a1fb0 con 0x7fcf9019c6f0 2024-05-27T09:24:13.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.282+0000 7fcf86ffd700 1 -- 172.21.15.189:0/1799032295 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf80011bd0 con 0x7fcf9019c6f0 2024-05-27T09:24:13.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.282+0000 7fcf97d28700 1 -- 172.21.15.189:0/1799032295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf901a24e0 con 0x7fcf9019c6f0 2024-05-27T09:24:13.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.283+0000 7fcf86ffd700 1 -- 172.21.15.189:0/1799032295 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fcf8001a400 con 0x7fcf9019c6f0 2024-05-27T09:24:13.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.283+0000 7fcf86ffd700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fcf7c03e9a0 0x7fcf7c040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:13.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.283+0000 7fcf86ffd700 1 -- 172.21.15.189:0/1799032295 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcf80053340 con 0x7fcf9019c6f0 2024-05-27T09:24:13.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.283+0000 7fcf97d28700 1 -- 172.21.15.189:0/1799032295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf90064e10 con 0x7fcf9019c6f0 2024-05-27T09:24:13.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.283+0000 7fcf95ac4700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fcf7c03e9a0 0x7fcf7c040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:13.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.284+0000 7fcf95ac4700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fcf7c03e9a0 0x7fcf7c040eb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009fd0 tx=0x7fcf8c00b7d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:24:13.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.288+0000 7fcf86ffd700 1 -- 172.21.15.189:0/1799032295 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcf80020740 con 0x7fcf9019c6f0 2024-05-27T09:24:13.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.504+0000 7fcf97d28700 1 -- 172.21.15.189:0/1799032295 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi050=y;smithi189=z", "target": ["mon-mgr", ""]}) -- 0x7fcf90100030 con 0x7fcf7c03e9a0 2024-05-27T09:24:13.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.519+0000 7fcf86ffd700 1 -- 172.21.15.189:0/1799032295 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fcf90100030 con 0x7fcf7c03e9a0 2024-05-27T09:24:13.519 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mgr update... 2024-05-27T09:24:13.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 -- 172.21.15.189:0/1799032295 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fcf7c03e9a0 msgr2=0x7fcf7c040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:13.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fcf7c03e9a0 0x7fcf7c040eb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009fd0 tx=0x7fcf8c00b7d0 comp rx=0 tx=0).stop 2024-05-27T09:24:13.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 -- 172.21.15.189:0/1799032295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf9019c6f0 msgr2=0x7fcf9019eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:13.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf9019c6f0 0x7fcf9019eb10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000b780 tx=0x7fcf8000bb40 comp rx=0 tx=0).stop 2024-05-27T09:24:13.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 -- 172.21.15.189:0/1799032295 shutdown_connections 2024-05-27T09:24:13.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fcf7c03e9a0 0x7fcf7c040eb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:13.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf9019f050 0x7fcf901a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:13.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf90102be0 0x7fcf9006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:13.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 --2- 172.21.15.189:0/1799032295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf9019c6f0 0x7fcf9019eb10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:13.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 -- 172.21.15.189:0/1799032295 >> 172.21.15.189:0/1799032295 conn(0x7fcf900fe240 msgr2=0x7fcf900ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:24:13.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 -- 172.21.15.189:0/1799032295 shutdown_connections 2024-05-27T09:24:13.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:24:13.521+0000 7fcf97d28700 1 -- 172.21.15.189:0/1799032295 wait complete. 2024-05-27T09:24:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:13 smithi050 ceph-mon[36371]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:13 smithi050 ceph-mon[36371]: Reconfiguring mon.b (monmap changed)... 2024-05-27T09:24:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:24:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T09:24:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:13.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:13 smithi050 ceph-mon[36371]: Reconfiguring daemon mon.b on smithi050 2024-05-27T09:24:13.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:13 smithi189 ceph-mon[36671]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:13.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:13.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:13.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:13 smithi189 ceph-mon[36671]: Reconfiguring mon.b (monmap changed)... 2024-05-27T09:24:13.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:24:13.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T09:24:13.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:13.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:13 smithi189 ceph-mon[36671]: Reconfiguring daemon mon.b on smithi050 2024-05-27T09:24:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:13 smithi044 ceph-mon[31618]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:13 smithi044 ceph-mon[31618]: Reconfiguring mon.b (monmap changed)... 2024-05-27T09:24:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:24:13.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T09:24:13.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:13.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:13 smithi044 ceph-mon[31618]: Reconfiguring daemon mon.b on smithi050 2024-05-27T09:24:14.121 DEBUG:teuthology.orchestra.run.smithi050:mgr.y> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.y.service 2024-05-27T09:24:14.123 DEBUG:teuthology.orchestra.run.smithi189:mgr.z> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.z.service 2024-05-27T09:24:14.125 INFO:tasks.cephadm:Deploying OSDs... 2024-05-27T09:24:14.126 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:24:14.126 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T09:24:14.187 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T09:24:14.187 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_1 2024-05-27T09:24:14.239 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T09:24:14.239 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:14.239 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-05-27T09:24:14.239 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:14.239 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:14.239 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-05-27 09:23:12.737245975 +0000 2024-05-27T09:24:14.240 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-05-27 09:17:33.419526291 +0000 2024-05-27T09:24:14.240 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-05-27 09:17:33.419526291 +0000 2024-05-27T09:24:14.240 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-05-27 09:17:33.419526291 +0000 2024-05-27T09:24:14.240 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T09:24:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-05-27T09:24:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-05-27T09:24:14.316 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000366301 s, 1.4 MB/s 2024-05-27T09:24:14.320 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T09:24:14.380 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_2 2024-05-27T09:24:14.432 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T09:24:14.433 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:14.433 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-05-27T09:24:14.433 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:14.433 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:14.433 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-05-27 09:23:12.841244357 +0000 2024-05-27T09:24:14.433 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-05-27 09:17:34.010516801 +0000 2024-05-27T09:24:14.433 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-05-27 09:17:34.010516801 +0000 2024-05-27T09:24:14.433 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-05-27 09:17:34.010516801 +0000 2024-05-27T09:24:14.433 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T09:24:14.509 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-05-27T09:24:14.510 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-05-27T09:24:14.510 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000369399 s, 1.4 MB/s 2024-05-27T09:24:14.512 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T09:24:14.569 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_3 2024-05-27T09:24:14.621 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T09:24:14.622 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:14.622 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-05-27T09:24:14.622 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:14.622 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:14.622 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-05-27 09:23:12.916243190 +0000 2024-05-27T09:24:14.622 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-05-27 09:17:34.581507632 +0000 2024-05-27T09:24:14.622 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-05-27 09:17:34.581507632 +0000 2024-05-27T09:24:14.622 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-05-27 09:17:34.581507632 +0000 2024-05-27T09:24:14.623 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T09:24:14.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:14 smithi044 ceph-mon[31618]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi050=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:24:14.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:14 smithi044 ceph-mon[31618]: Saving service mgr spec with placement smithi044=x;smithi050=y;smithi189=z;count:3 2024-05-27T09:24:14.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:14.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:14.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:14.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:24:14.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T09:24:14.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:14.699 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-05-27T09:24:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-05-27T09:24:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000381746 s, 1.3 MB/s 2024-05-27T09:24:14.703 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T09:24:14.761 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_4 2024-05-27T09:24:14.771 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:14 smithi189 ceph-mon[36671]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi050=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:24:14.771 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:14 smithi189 ceph-mon[36671]: Saving service mgr spec with placement smithi044=x;smithi050=y;smithi189=z;count:3 2024-05-27T09:24:14.771 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:14.771 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:14.771 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:14.771 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:24:14.771 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T09:24:14.771 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:14.793 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:14 smithi050 ceph-mon[36371]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi050=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:24:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:14 smithi050 ceph-mon[36371]: Saving service mgr spec with placement smithi044=x;smithi050=y;smithi189=z;count:3 2024-05-27T09:24:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T09:24:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T09:24:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:14.814 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T09:24:14.814 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:14.814 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-05-27T09:24:14.814 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:14.814 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:14.814 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-05-27 09:23:12.979242210 +0000 2024-05-27T09:24:14.814 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-05-27 09:17:35.197497740 +0000 2024-05-27T09:24:14.815 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-05-27 09:17:35.197497740 +0000 2024-05-27T09:24:14.815 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-05-27 09:17:35.197497740 +0000 2024-05-27T09:24:14.815 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T09:24:14.899 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-05-27T09:24:14.900 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-05-27T09:24:14.900 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000363788 s, 1.4 MB/s 2024-05-27T09:24:14.904 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T09:24:14.961 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-27T09:24:14.962 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T09:24:14.981 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T09:24:14.981 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-05-27T09:24:15.040 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T09:24:15.040 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:15.040 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-27T09:24:15.040 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:15.040 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:15.040 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-05-27 09:24:01.482472069 +0000 2024-05-27T09:24:15.040 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-05-27 09:16:23.596392811 +0000 2024-05-27T09:24:15.040 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-05-27 09:16:23.596392811 +0000 2024-05-27T09:24:15.041 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-05-27 09:16:23.596392811 +0000 2024-05-27T09:24:15.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T09:24:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-05-27T09:24:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-05-27T09:24:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000113857 s, 4.5 MB/s 2024-05-27T09:24:15.109 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T09:24:15.168 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-05-27T09:24:15.225 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T09:24:15.225 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:15.225 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-27T09:24:15.225 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:15.225 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:15.225 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-05-27 09:24:01.482472069 +0000 2024-05-27T09:24:15.225 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-05-27 09:16:23.918384502 +0000 2024-05-27T09:24:15.225 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-05-27 09:16:23.918384502 +0000 2024-05-27T09:24:15.225 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-05-27 09:16:23.918384502 +0000 2024-05-27T09:24:15.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T09:24:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-05-27T09:24:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-05-27T09:24:15.291 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000118961 s, 4.3 MB/s 2024-05-27T09:24:15.292 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T09:24:15.349 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-05-27T09:24:15.405 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T09:24:15.405 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:15.405 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-27T09:24:15.405 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:15.405 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:15.405 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-05-27 09:24:01.482472069 +0000 2024-05-27T09:24:15.405 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-05-27 09:16:24.247376014 +0000 2024-05-27T09:24:15.406 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-05-27 09:16:24.247376014 +0000 2024-05-27T09:24:15.406 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-05-27 09:16:24.247376014 +0000 2024-05-27T09:24:15.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T09:24:15.469 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-05-27T09:24:15.469 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-05-27T09:24:15.469 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.00010153 s, 5.0 MB/s 2024-05-27T09:24:15.470 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T09:24:15.527 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-05-27T09:24:15.583 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T09:24:15.584 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:15.584 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-05-27T09:24:15.584 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:15.584 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:15.584 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-05-27 09:24:01.483472044 +0000 2024-05-27T09:24:15.584 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-05-27 09:16:24.652365565 +0000 2024-05-27T09:24:15.584 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-05-27 09:16:24.653365539 +0000 2024-05-27T09:24:15.584 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-05-27 09:16:24.652365565 +0000 2024-05-27T09:24:15.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T09:24:15.648 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-05-27T09:24:15.648 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-05-27T09:24:15.648 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000111911 s, 4.6 MB/s 2024-05-27T09:24:15.649 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T09:24:15.705 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-05-27T09:24:15.706 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T09:24:15.721 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T09:24:15.721 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-05-27T09:24:15.776 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T09:24:15.777 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:15.777 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-27T09:24:15.777 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:15.777 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:15.777 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-05-27 09:23:51.712459606 +0000 2024-05-27T09:24:15.777 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-05-27 09:16:03.137733120 +0000 2024-05-27T09:24:15.777 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-05-27 09:16:03.137733120 +0000 2024-05-27T09:24:15.777 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-05-27 09:16:03.137733120 +0000 2024-05-27T09:24:15.778 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T09:24:15.793 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:15 smithi050 ceph-mon[36371]: Reconfiguring mon.c (monmap changed)... 2024-05-27T09:24:15.793 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:15 smithi050 ceph-mon[36371]: Reconfiguring daemon mon.c on smithi189 2024-05-27T09:24:15.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:15 smithi050 ceph-mon[36371]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:15.836 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:15 smithi189 ceph-mon[36671]: Reconfiguring mon.c (monmap changed)... 2024-05-27T09:24:15.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:15 smithi189 ceph-mon[36671]: Reconfiguring daemon mon.c on smithi189 2024-05-27T09:24:15.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:15 smithi189 ceph-mon[36671]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:15.839 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-05-27T09:24:15.839 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-05-27T09:24:15.839 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000196695 s, 2.6 MB/s 2024-05-27T09:24:15.840 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T09:24:15.896 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-05-27T09:24:15.952 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T09:24:15.952 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:15.952 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-27T09:24:15.952 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:15.952 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:15.952 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-05-27 09:23:51.713459586 +0000 2024-05-27T09:24:15.952 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-05-27 09:16:03.465726611 +0000 2024-05-27T09:24:15.952 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-05-27 09:16:03.465726611 +0000 2024-05-27T09:24:15.952 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-05-27 09:16:03.465726611 +0000 2024-05-27T09:24:15.953 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T09:24:15.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:15 smithi044 ceph-mon[31618]: Reconfiguring mon.c (monmap changed)... 2024-05-27T09:24:15.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:15 smithi044 ceph-mon[31618]: Reconfiguring daemon mon.c on smithi189 2024-05-27T09:24:15.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:15 smithi044 ceph-mon[31618]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:16.015 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-05-27T09:24:16.015 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-05-27T09:24:16.015 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000197153 s, 2.6 MB/s 2024-05-27T09:24:16.016 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T09:24:16.072 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-05-27T09:24:16.127 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T09:24:16.127 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-05-27T09:24:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-05-27 09:23:51.713459586 +0000 2024-05-27T09:24:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-05-27 09:16:03.794720081 +0000 2024-05-27T09:24:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-05-27 09:16:03.794720081 +0000 2024-05-27T09:24:16.128 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-05-27 09:16:03.794720081 +0000 2024-05-27T09:24:16.128 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T09:24:16.189 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-05-27T09:24:16.190 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-05-27T09:24:16.190 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000194007 s, 2.6 MB/s 2024-05-27T09:24:16.191 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T09:24:16.248 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-05-27T09:24:16.304 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T09:24:16.304 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T09:24:16.304 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-05-27T09:24:16.304 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:24:16.304 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T09:24:16.304 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-05-27 09:23:51.714459567 +0000 2024-05-27T09:24:16.304 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-05-27 09:16:04.121713591 +0000 2024-05-27T09:24:16.304 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-05-27 09:16:04.121713591 +0000 2024-05-27T09:24:16.305 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-05-27 09:16:04.121713591 +0000 2024-05-27T09:24:16.305 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T09:24:16.367 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-05-27T09:24:16.367 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-05-27T09:24:16.367 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000203743 s, 2.5 MB/s 2024-05-27T09:24:16.368 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T09:24:16.424 INFO:tasks.cephadm:Deploying osd.0 on smithi044 with /dev/vg_nvme/lv_4... 2024-05-27T09:24:16.424 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T09:24:16.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:16 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:16.855 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:16 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:16.855 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:16 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:16.855 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:16 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:16.855 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:16 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:16.855 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:16 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:16.855 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:16 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T09:24:16.856 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:16 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T09:24:16.856 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:16 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T09:24:16.856 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:16 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:16.856 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:16 smithi189 ceph-mon[36671]: Deploying daemon mgr.z on smithi189 2024-05-27T09:24:16.868 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:24:16.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:16 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:16.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:16 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:16.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:16 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:16.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:16 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:16.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:16 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:16.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:16 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:16.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:16 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T09:24:16.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:16 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T09:24:16.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:16 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T09:24:16.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:16 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:16.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:16 smithi044 ceph-mon[31618]: Deploying daemon mgr.z on smithi189 2024-05-27T09:24:17.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:16 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:17.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:16 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:17.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:16 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:17.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:16 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:17.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:16 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:17.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:16 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:17.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:16 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T09:24:17.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:16 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T09:24:17.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:16 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T09:24:17.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:16 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:17.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:16 smithi050 ceph-mon[36371]: Deploying daemon mgr.z on smithi189 2024-05-27T09:24:18.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:17 smithi044 ceph-mon[31618]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:17 smithi050 ceph-mon[36371]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:18.064 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:17 smithi189 ceph-mon[36671]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:18.064 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:17 smithi189 systemd[1]: Starting Ceph mgr.z for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:24:18.338 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:18 smithi189 podman[38514]: 2024-05-27 09:24:17.966147082 +0000 UTC m=+0.016868022 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:24:18.338 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:18 smithi189 podman[38514]: 2024-05-27 09:24:18.069249396 +0000 UTC m=+0.119970336 container create 672bcb45c16631b6b9c4dbc1629ade9eadbaaf33f1598a491ffd2723bd85705e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:24:18.338 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:18 smithi189 podman[38514]: 2024-05-27 09:24:18.305144391 +0000 UTC m=+0.355865324 container init 672bcb45c16631b6b9c4dbc1629ade9eadbaaf33f1598a491ffd2723bd85705e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T09:24:18.338 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:18 smithi189 podman[38514]: 2024-05-27 09:24:18.308355742 +0000 UTC m=+0.359076683 container start 672bcb45c16631b6b9c4dbc1629ade9eadbaaf33f1598a491ffd2723bd85705e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-27T09:24:18.656 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:18 smithi189 ceph-mgr[38533]: -- 172.21.15.189:0/3677586561 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5651426165a0 con 0x5651473a0480 2024-05-27T09:24:18.657 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:18 smithi189 bash[38514]: 672bcb45c16631b6b9c4dbc1629ade9eadbaaf33f1598a491ffd2723bd85705e 2024-05-27T09:24:18.657 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:18 smithi189 systemd[1]: Started Ceph mgr.z for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:24:18.657 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:18.472+0000 7fe2302ab200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T09:24:18.657 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:18.656+0000 7fe2302ab200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:24:19.063 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:18.758+0000 7fe2302ab200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T09:24:19.064 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:19 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:19.063+0000 7fe2302ab200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T09:24:19.337 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:19 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:19.147+0000 7fe2302ab200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T09:24:19.621 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T09:24:19.621 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T09:24:19.621 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-05-27T09:24:19.621 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-05-27T09:24:19.621 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0344962 s, 304 MB/s 2024-05-27T09:24:19.621 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-05-27T09:24:19.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:19.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:19 smithi044 ceph-mon[31618]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:19.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:19.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:19.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T09:24:19.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T09:24:19.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T09:24:19.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:19.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:19 smithi044 ceph-mon[31618]: Deploying daemon mgr.y on smithi050 2024-05-27T09:24:19.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:19.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:19 smithi050 ceph-mon[36371]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:19.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:19.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:19.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T09:24:19.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T09:24:19.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T09:24:19.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:19.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:19 smithi050 ceph-mon[36371]: Deploying daemon mgr.y on smithi050 2024-05-27T09:24:19.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:19.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:19 smithi189 ceph-mon[36671]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:19.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:19.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:19.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T09:24:19.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T09:24:19.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T09:24:19.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:19.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:19 smithi189 ceph-mon[36671]: Deploying daemon mgr.y on smithi050 2024-05-27T09:24:20.355 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_4 2024-05-27T09:24:20.544 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:20 smithi050 systemd[1]: Starting Ceph mgr.y for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:24:20.588 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:20 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:20.333+0000 7fe2302ab200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T09:24:20.588 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:20 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:20.501+0000 7fe2302ab200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T09:24:20.812 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:24:20.924 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:20 smithi050 podman[37729]: 2024-05-27 09:24:20.647009262 +0000 UTC m=+0.107646971 container create 5211fc522376d1889f81f5b0b67a6c2a82dfb2c39eff84e8e7afc96a2132346a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-27T09:24:20.925 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:20 smithi050 podman[37729]: 2024-05-27 09:24:20.557737548 +0000 UTC m=+0.018375272 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:24:20.925 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:20 smithi050 podman[37729]: 2024-05-27 09:24:20.840999844 +0000 UTC m=+0.301637562 container init 5211fc522376d1889f81f5b0b67a6c2a82dfb2c39eff84e8e7afc96a2132346a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2) 2024-05-27T09:24:20.925 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:20 smithi050 podman[37729]: 2024-05-27 09:24:20.843571379 +0000 UTC m=+0.304209097 container start 5211fc522376d1889f81f5b0b67a6c2a82dfb2c39eff84e8e7afc96a2132346a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux ) 2024-05-27T09:24:20.925 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:20 smithi050 ceph-mgr[37748]: -- 172.21.15.50:0/1327950787 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ab201465a0 con 0x55ab24ed0480 2024-05-27T09:24:20.925 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:20 smithi050 bash[37729]: 5211fc522376d1889f81f5b0b67a6c2a82dfb2c39eff84e8e7afc96a2132346a 2024-05-27T09:24:20.958 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:20 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:20.672+0000 7fe2302ab200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T09:24:21.205 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:20 smithi050 systemd[1]: Started Ceph mgr.y for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:24:21.205 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:21 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:21.010+0000 7f1ad9a1a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T09:24:21.205 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:21 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:21.203+0000 7f1ad9a1a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:24:21.255 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:20 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:20.957+0000 7fe2302ab200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T09:24:21.256 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:21 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:21.050+0000 7fe2302ab200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T09:24:21.256 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:21 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:21.255+0000 7fe2302ab200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T09:24:21.541 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:21 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:21.312+0000 7f1ad9a1a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T09:24:21.542 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:21 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:21.453+0000 7fe2302ab200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:24:21.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:21 smithi050 ceph-mon[36371]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:21.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:21.794 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:21 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:21.646+0000 7f1ad9a1a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T09:24:21.794 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:21 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:21.737+0000 7f1ad9a1a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T09:24:21.803 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:21 smithi044 ceph-mon[31618]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:21.803 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.804 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.804 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.804 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.804 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:21.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:21 smithi189 ceph-mon[36671]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:21.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:21.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:22.564 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:22.091+0000 7fe2302ab200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T09:24:22.564 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:22.321+0000 7fe2302ab200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T09:24:22.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:22 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:22.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:22 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:22.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:22 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:22.837 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:22.752+0000 7fe2302ab200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T09:24:22.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:22 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:22.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:22 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:22.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:22 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:23.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.026+0000 7f520cc00700 1 -- 172.21.15.44:0/4260901703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52081059a0 msgr2=0x7f5208105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:23.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.026+0000 7f520cc00700 1 --2- 172.21.15.44:0/4260901703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52081059a0 0x7f5208105db0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f51f80098f0 tx=0x7f51f8009c00 comp rx=0 tx=0).stop 2024-05-27T09:24:23.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.027+0000 7f520cc00700 1 -- 172.21.15.44:0/4260901703 shutdown_connections 2024-05-27T09:24:23.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.027+0000 7f520cc00700 1 --2- 172.21.15.44:0/4260901703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52081049d0 0x7f520810c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:23.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.027+0000 7f520cc00700 1 --2- 172.21.15.44:0/4260901703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52081059a0 0x7f5208105db0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:23.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.027+0000 7f520cc00700 1 --2- 172.21.15.44:0/4260901703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5208103f50 0x7f5208104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:23.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.027+0000 7f520cc00700 1 -- 172.21.15.44:0/4260901703 >> 172.21.15.44:0/4260901703 conn(0x7f52080ff9c0 msgr2=0x7f5208101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:24:23.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.027+0000 7f520cc00700 1 -- 172.21.15.44:0/4260901703 shutdown_connections 2024-05-27T09:24:23.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.027+0000 7f520cc00700 1 -- 172.21.15.44:0/4260901703 wait complete. 2024-05-27T09:24:23.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.028+0000 7f520cc00700 1 Processor -- start 2024-05-27T09:24:23.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.028+0000 7f520cc00700 1 -- start start 2024-05-27T09:24:23.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.029+0000 7f520cc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5208103f50 0x7f520819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:23.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.029+0000 7f520cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52081049d0 0x7f520819b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:23.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.029+0000 7f520cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52081059a0 0x7f52081a1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:23.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.029+0000 7f520cc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f520810ff10 con 0x7f52081049d0 2024-05-27T09:24:23.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.029+0000 7f520cc00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f520810fd90 con 0x7f5208103f50 2024-05-27T09:24:23.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.029+0000 7f5205e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5208103f50 0x7f520819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:23.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.029+0000 7f5205e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5208103f50 0x7f520819ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:41508/0 (socket says 172.21.15.44:41508) 2024-05-27T09:24:23.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.029+0000 7f5205e00700 1 -- 172.21.15.44:0/1723572403 learned_addr learned my addr 172.21.15.44:0/1723572403 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:24:23.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.029+0000 7f51ffe00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52081049d0 0x7f520819b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:23.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.029+0000 7f520cc00700 1 -- 172.21.15.44:0/1723572403 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5208110090 con 0x7f52081059a0 2024-05-27T09:24:23.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.029+0000 7f5206800700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52081059a0 0x7f52081a1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:23.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.030+0000 7f5205e00700 1 -- 172.21.15.44:0/1723572403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52081059a0 msgr2=0x7f52081a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:23.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.030+0000 7f5205e00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52081059a0 0x7f52081a1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:23.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.030+0000 7f5205e00700 1 -- 172.21.15.44:0/1723572403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52081049d0 msgr2=0x7f520819b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:23.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.030+0000 7f5205e00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52081049d0 0x7f520819b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:23.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.030+0000 7f5205e00700 1 -- 172.21.15.44:0/1723572403 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52081a2660 con 0x7f5208103f50 2024-05-27T09:24:23.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.030+0000 7f5205e00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5208103f50 0x7f520819ab00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51f8000c00 tx=0x7f51f8005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:24:23.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.030+0000 7f51fea00700 1 -- 172.21.15.44:0/1723572403 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f80053e0 con 0x7f5208103f50 2024-05-27T09:24:23.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.031+0000 7f51fea00700 1 -- 172.21.15.44:0/1723572403 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51f8005580 con 0x7f5208103f50 2024-05-27T09:24:23.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.031+0000 7f520cc00700 1 -- 172.21.15.44:0/1723572403 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52081a28f0 con 0x7f5208103f50 2024-05-27T09:24:23.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.031+0000 7f51fea00700 1 -- 172.21.15.44:0/1723572403 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8003a40 con 0x7f5208103f50 2024-05-27T09:24:23.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.031+0000 7f520cc00700 1 -- 172.21.15.44:0/1723572403 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52081a2d40 con 0x7f5208103f50 2024-05-27T09:24:23.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.032+0000 7f51fea00700 1 -- 172.21.15.44:0/1723572403 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f51f8003c40 con 0x7f5208103f50 2024-05-27T09:24:23.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.032+0000 7f520cc00700 1 -- 172.21.15.44:0/1723572403 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52080512b0 con 0x7f5208103f50 2024-05-27T09:24:23.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.033+0000 7f51fea00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51ec03e750 0x7f51ec040c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:23.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.033+0000 7f51fea00700 1 -- 172.21.15.44:0/1723572403 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f51f80539f0 con 0x7f5208103f50 2024-05-27T09:24:23.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.033+0000 7f51ffe00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51ec03e750 0x7f51ec040c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:23.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.034+0000 7f51ffe00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51ec03e750 0x7f51ec040c60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f520819bbb0 tx=0x7f51f0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:24:23.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.039+0000 7f51fea00700 1 -- 172.21.15.44:0/1723572403 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f51f8013030 con 0x7f5208103f50 2024-05-27T09:24:23.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:22 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:23.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:22 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:23.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:22 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:23.221 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:22.847+0000 7fe2302ab200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T09:24:23.221 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:23 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:23.220+0000 7fe2302ab200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T09:24:23.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:23.293+0000 7f520cc00700 1 -- 172.21.15.44:0/1723572403 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f520802ea40 con 0x7f51ec03e750 2024-05-27T09:24:23.405 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:23 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:23.051+0000 7f1ad9a1a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T09:24:23.405 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:23 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:23.227+0000 7f1ad9a1a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T09:24:23.405 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:23 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:23.404+0000 7f1ad9a1a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T09:24:23.717 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:23.717 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:23.717 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:23.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:23.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:23.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:23.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T09:24:23.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T09:24:23.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:23.719 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:24:23.719 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:24:23.719 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:23.837 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:23 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:23.569+0000 7fe2302ab200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T09:24:23.837 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:23 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:23.661+0000 7fe2302ab200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T09:24:23.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:23.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:23.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:23.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:23.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:23.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:23.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T09:24:23.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T09:24:23.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:23.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:24:23.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:24:23.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:24.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:24.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:24.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:24.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:24.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:24.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:24.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T09:24:24.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T09:24:24.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:24.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:24:24.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:24:24.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:24.013 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:23 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:23.703+0000 7f1ad9a1a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T09:24:24.013 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:23 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:23.799+0000 7f1ad9a1a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T09:24:24.013 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:24 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:24.010+0000 7f1ad9a1a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T09:24:24.294 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:24 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:24.211+0000 7f1ad9a1a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:24:24.337 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:23 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:23.901+0000 7fe2302ab200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T09:24:24.337 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:24.033+0000 7fe2302ab200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T09:24:24.337 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:24.127+0000 7fe2302ab200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T09:24:24.724 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:24.453+0000 7fe2302ab200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T09:24:24.724 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:24.544+0000 7fe2302ab200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T09:24:24.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:24 smithi044 ceph-mon[31618]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T09:24:24.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:24 smithi044 ceph-mon[31618]: Reconfiguring daemon mgr.x on smithi044 2024-05-27T09:24:24.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:24 smithi044 ceph-mon[31618]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:24:25.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:24 smithi050 ceph-mon[36371]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T09:24:25.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:24 smithi050 ceph-mon[36371]: Reconfiguring daemon mgr.x on smithi044 2024-05-27T09:24:25.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:24 smithi050 ceph-mon[36371]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:24:25.044 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:24 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:24.839+0000 7f1ad9a1a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T09:24:25.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:24 smithi189 ceph-mon[36671]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T09:24:25.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:24 smithi189 ceph-mon[36671]: Reconfiguring daemon mgr.x on smithi044 2024-05-27T09:24:25.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:24 smithi189 ceph-mon[36671]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:24:25.514 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:25 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:25.070+0000 7f1ad9a1a200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T09:24:25.587 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:25 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:25.142+0000 7fe2302ab200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T09:24:25.588 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:24:25 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:24:25.357+0000 7fe2302ab200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T09:24:25.794 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:25 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:25.512+0000 7f1ad9a1a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T09:24:25.794 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:25 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:25.612+0000 7f1ad9a1a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T09:24:25.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:25.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:25.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:25.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:25.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:25.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:25.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:25.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T09:24:25.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: Standby manager daemon z started 2024-05-27T09:24:25.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:24:25.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T09:24:25.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:25 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:24:25.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:25.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:25.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:25.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:25.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:25.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:25.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:25.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T09:24:25.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: Standby manager daemon z started 2024-05-27T09:24:25.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:24:25.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T09:24:25.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:25 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:24:26.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T09:24:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: Standby manager daemon z started 2024-05-27T09:24:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:24:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T09:24:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:25 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.189:0/3894071986' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:24:26.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:26.148+0000 7f51fea00700 1 -- 172.21.15.44:0/1723572403 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 14) ==== 99937+0+0 (secure 0 0 0) 0x7f51f802c440 con 0x7f5208103f50 2024-05-27T09:24:26.293 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:25 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:25.995+0000 7f1ad9a1a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T09:24:26.720 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:26 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:26.369+0000 7f1ad9a1a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T09:24:26.720 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:26 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:26.465+0000 7f1ad9a1a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T09:24:27.044 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:26 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:26.718+0000 7f1ad9a1a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T09:24:27.063 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:26 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:26.857+0000 7f1ad9a1a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T09:24:27.064 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:26 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:26.957+0000 7f1ad9a1a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T09:24:27.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:27 smithi044 ceph-mon[31618]: mgrmap e14: x(active, since 89s), standbys: z 2024-05-27T09:24:27.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:27 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T09:24:27.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:27 smithi044 ceph-mon[31618]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:27.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:27 smithi050 ceph-mon[36371]: mgrmap e14: x(active, since 89s), standbys: z 2024-05-27T09:24:27.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:27 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T09:24:27.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:27 smithi050 ceph-mon[36371]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:27.544 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:27 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:27.295+0000 7f1ad9a1a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T09:24:27.544 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:27 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:27.390+0000 7f1ad9a1a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T09:24:27.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:27 smithi189 ceph-mon[36671]: mgrmap e14: x(active, since 89s), standbys: z 2024-05-27T09:24:27.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:27 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T09:24:27.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:27 smithi189 ceph-mon[36671]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:28.294 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:28 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:28.020+0000 7f1ad9a1a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T09:24:28.294 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:24:28 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:24:28.245+0000 7f1ad9a1a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T09:24:28.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:28.318+0000 7f51fea00700 1 -- 172.21.15.44:0/1723572403 <== mon.2 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f51f80223d0 con 0x7f5208103f50 2024-05-27T09:24:28.568 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:28 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T09:24:28.568 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:28 smithi050 ceph-mon[36371]: Standby manager daemon y started 2024-05-27T09:24:28.568 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:28 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:24:28.568 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:28 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T09:24:28.568 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:28 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:24:28.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:28 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T09:24:28.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:28 smithi189 ceph-mon[36671]: Standby manager daemon y started 2024-05-27T09:24:28.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:28 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:24:28.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:28 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T09:24:28.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:28 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:24:28.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:28 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T09:24:28.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:28 smithi044 ceph-mon[31618]: Standby manager daemon y started 2024-05-27T09:24:28.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:28 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:24:28.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:28 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T09:24:28.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:28 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.50:0/3600339493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:24:29.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:29 smithi189 ceph-mon[36671]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-05-27T09:24:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:29 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T09:24:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:29 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/4137098698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8959d30f-ab21-41e8-b900-1f5ef3426892"}]: dispatch 2024-05-27T09:24:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:29 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/4137098698' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8959d30f-ab21-41e8-b900-1f5ef3426892"}]': finished 2024-05-27T09:24:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:29 smithi189 ceph-mon[36671]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T09:24:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:29 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:29 smithi189 ceph-mon[36671]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:29.723 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:29 smithi044 ceph-mon[31618]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-05-27T09:24:29.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:29 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T09:24:29.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:29 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4137098698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8959d30f-ab21-41e8-b900-1f5ef3426892"}]: dispatch 2024-05-27T09:24:29.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:29 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4137098698' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8959d30f-ab21-41e8-b900-1f5ef3426892"}]': finished 2024-05-27T09:24:29.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:29 smithi044 ceph-mon[31618]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T09:24:29.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:29 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:29.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:29 smithi044 ceph-mon[31618]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:29.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:29 smithi050 ceph-mon[36371]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-05-27T09:24:29.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:29 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T09:24:29.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:29 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/4137098698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8959d30f-ab21-41e8-b900-1f5ef3426892"}]: dispatch 2024-05-27T09:24:29.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:29 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/4137098698' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8959d30f-ab21-41e8-b900-1f5ef3426892"}]': finished 2024-05-27T09:24:29.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:29 smithi050 ceph-mon[36371]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T09:24:29.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:29 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:29.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:29 smithi050 ceph-mon[36371]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:30.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:30 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/12464494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:24:30.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:30 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/12464494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:24:30.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:30 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/12464494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:24:31.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:31 smithi044 ceph-mon[31618]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:31.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:31 smithi050 ceph-mon[36371]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:31.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:31 smithi189 ceph-mon[36671]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:33.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:33 smithi189 ceph-mon[36671]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:33.926 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:33 smithi044 ceph-mon[31618]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:34.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:33 smithi050 ceph-mon[36371]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:35.658 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:35 smithi044 ceph-mon[31618]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:35.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:35 smithi189 ceph-mon[36671]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:36.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:35 smithi050 ceph-mon[36371]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:37.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:37 smithi189 ceph-mon[36671]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:37.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:37 smithi044 ceph-mon[31618]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:38.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:37 smithi050 ceph-mon[36371]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:38.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:38 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T09:24:38.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:38 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:38.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:38 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T09:24:38.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:38 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:39.016 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:38 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T09:24:39.016 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:38 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:39.813 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:39 smithi044 ceph-mon[31618]: Deploying daemon osd.0 on smithi044 2024-05-27T09:24:39.813 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:39 smithi044 ceph-mon[31618]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:39.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:39 smithi189 ceph-mon[36671]: Deploying daemon osd.0 on smithi044 2024-05-27T09:24:39.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:39 smithi189 ceph-mon[36671]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:40.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:39 smithi050 ceph-mon[36371]: Deploying daemon osd.0 on smithi044 2024-05-27T09:24:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:39 smithi050 ceph-mon[36371]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:41.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:41 smithi044 ceph-mon[31618]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:41.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:41 smithi189 ceph-mon[36671]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:42.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:41 smithi050 ceph-mon[36371]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:43.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:43 smithi044 ceph-mon[31618]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:43.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:43 smithi189 ceph-mon[36671]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:44.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:43 smithi050 ceph-mon[36371]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:45.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:45 smithi189 ceph-mon[36671]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:45.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:45 smithi044 ceph-mon[31618]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:46.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:45 smithi050 ceph-mon[36371]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:47.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:47 smithi189 ceph-mon[36671]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:47.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:47 smithi044 ceph-mon[31618]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:48.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:47 smithi050 ceph-mon[36371]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:49.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:49.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:49.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:49.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:49 smithi044 ceph-mon[31618]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:49.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:49.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:49.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:49.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:49 smithi050 ceph-mon[36371]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:49 smithi189 ceph-mon[36671]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:51.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:51 smithi189 ceph-mon[36671]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:51.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:51 smithi044 ceph-mon[31618]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:52.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:51 smithi050 ceph-mon[36371]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:52.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:52.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:52.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:52.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:52.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:52.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:52 smithi044 ceph-mon[31618]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T09:24:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:52 smithi050 ceph-mon[36371]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T09:24:53.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:53.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:53.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:53.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:53.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:53.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:52 smithi189 ceph-mon[36671]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T09:24:53.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.292+0000 7f51fea00700 1 -- 172.21.15.44:0/1723572403 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f51f001b3f0 con 0x7f51ec03e750 2024-05-27T09:24:53.295 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 0 on host 'smithi044' 2024-05-27T09:24:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 -- 172.21.15.44:0/1723572403 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51ec03e750 msgr2=0x7f51ec040c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51ec03e750 0x7f51ec040c60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f520819bbb0 tx=0x7f51f0009310 comp rx=0 tx=0).stop 2024-05-27T09:24:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 -- 172.21.15.44:0/1723572403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5208103f50 msgr2=0x7f520819ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5208103f50 0x7f520819ab00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51f8000c00 tx=0x7f51f8005270 comp rx=0 tx=0).stop 2024-05-27T09:24:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 -- 172.21.15.44:0/1723572403 shutdown_connections 2024-05-27T09:24:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51ec03e750 0x7f51ec040c60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52081059a0 0x7f52081a1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:53.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5208103f50 0x7f520819ab00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 --2- 172.21.15.44:0/1723572403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52081049d0 0x7f520819b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 -- 172.21.15.44:0/1723572403 >> 172.21.15.44:0/1723572403 conn(0x7f52080ff9c0 msgr2=0x7f5208108730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:24:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 -- 172.21.15.44:0/1723572403 shutdown_connections 2024-05-27T09:24:53.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:24:53.297+0000 7f520cc00700 1 -- 172.21.15.44:0/1723572403 wait complete. 2024-05-27T09:24:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:53 smithi044 ceph-mon[31618]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:53 smithi044 ceph-mon[31618]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T09:24:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:53 smithi044 ceph-mon[31618]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T09:24:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:53 smithi044 ceph-mon[31618]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:24:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:53 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:53 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:53 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:53 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:53 smithi189 ceph-mon[36671]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:53 smithi189 ceph-mon[36671]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T09:24:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:53 smithi189 ceph-mon[36671]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T09:24:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:53 smithi189 ceph-mon[36671]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:24:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:53 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:53 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:53 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:53 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:54.090 DEBUG:teuthology.orchestra.run.smithi044:osd.0> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.0.service 2024-05-27T09:24:54.094 INFO:tasks.cephadm:Deploying osd.1 on smithi050 with /dev/vg_nvme/lv_4... 2024-05-27T09:24:54.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T09:24:54.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:53 smithi050 ceph-mon[36371]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:54.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:53 smithi050 ceph-mon[36371]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T09:24:54.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:53 smithi050 ceph-mon[36371]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T09:24:54.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:53 smithi050 ceph-mon[36371]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:24:54.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:53 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:54.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:53 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:54.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:53 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:54.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:53 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:24:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:24:54.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:54 smithi044 ceph-mon[31618]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:24:54.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:54 smithi044 ceph-mon[31618]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T09:24:54.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:54 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:54.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:54 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:55.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:54 smithi050 ceph-mon[36371]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:24:55.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:54 smithi050 ceph-mon[36371]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T09:24:55.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:54 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:55.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:54 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:55.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:54 smithi189 ceph-mon[36671]: from='osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:24:55.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:54 smithi189 ceph-mon[36671]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T09:24:55.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:54 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:55.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:54 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:55.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:55 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:24:55.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:55 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:24:55.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:55 smithi044 ceph-mon[31618]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:55.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:55 smithi044 ceph-mon[31618]: osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534] boot 2024-05-27T09:24:55.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:55 smithi044 ceph-mon[31618]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T09:24:55.989 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:55 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:55 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:24:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:55 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:24:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:55 smithi050 ceph-mon[36371]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:55 smithi050 ceph-mon[36371]: osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534] boot 2024-05-27T09:24:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:55 smithi050 ceph-mon[36371]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T09:24:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:55 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:55 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:24:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:55 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:24:56.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:55 smithi189 ceph-mon[36671]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T09:24:56.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:55 smithi189 ceph-mon[36671]: osd.0 [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534] boot 2024-05-27T09:24:56.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:55 smithi189 ceph-mon[36671]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T09:24:56.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:55 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:24:56.280 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T09:24:56.280 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T09:24:56.280 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-05-27T09:24:56.280 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-05-27T09:24:56.280 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00844341 s, 1.2 GB/s 2024-05-27T09:24:56.280 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-05-27T09:24:56.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_4 2024-05-27T09:24:57.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:56 smithi189 ceph-mon[36671]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T09:24:57.130 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:24:57.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:56 smithi044 ceph-mon[31618]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T09:24:57.252 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:56 smithi050 ceph-mon[36371]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T09:24:58.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:57 smithi189 ceph-mon[36671]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:24:58.212 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:57 smithi050 ceph-mon[36371]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:24:58.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:57 smithi044 ceph-mon[31618]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:24:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.691+0000 7f473a78c700 1 -- 172.21.15.50:0/2850010014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4734107460 msgr2=0x7f4734109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.691+0000 7f473a78c700 1 --2- 172.21.15.50:0/2850010014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4734107460 0x7f4734109940 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4728009990 tx=0x7f4728009ca0 comp rx=0 tx=0).stop 2024-05-27T09:24:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.691+0000 7f473a78c700 1 -- 172.21.15.50:0/2850010014 shutdown_connections 2024-05-27T09:24:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.691+0000 7f473a78c700 1 --2- 172.21.15.50:0/2850010014 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4734109fd0 0x7f473410c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.691+0000 7f473a78c700 1 --2- 172.21.15.50:0/2850010014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473406ccf0 0x7f4734106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.691+0000 7f473a78c700 1 --2- 172.21.15.50:0/2850010014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4734107460 0x7f4734109940 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.691+0000 7f473a78c700 1 -- 172.21.15.50:0/2850010014 >> 172.21.15.50:0/2850010014 conn(0x7f47340fe860 msgr2=0x7f4734100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:24:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.691+0000 7f473a78c700 1 -- 172.21.15.50:0/2850010014 shutdown_connections 2024-05-27T09:24:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.691+0000 7f473a78c700 1 -- 172.21.15.50:0/2850010014 wait complete. 2024-05-27T09:24:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.691+0000 7f473a78c700 1 Processor -- start 2024-05-27T09:24:58.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f473a78c700 1 -- start start 2024-05-27T09:24:58.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f473a78c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f473406ccf0 0x7f47341046e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f473a78c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4734107460 0x7f4734102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f473a78c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4734109fd0 0x7f47341032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f473a78c700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f473410fde0 con 0x7f473406ccf0 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f473a78c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f473410fc60 con 0x7f4734107460 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f473a78c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f473410ff60 con 0x7f4734109fd0 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f47337fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4734107460 0x7f4734102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f47337fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4734107460 0x7f4734102d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55668/0 (socket says 172.21.15.50:55668) 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f47337fe700 1 -- 172.21.15.50:0/3336032808 learned_addr learned my addr 172.21.15.50:0/3336032808 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f4738d29700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4734109fd0 0x7f47341032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f47337fe700 1 -- 172.21.15.50:0/3336032808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4734109fd0 msgr2=0x7f47341032d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f47337fe700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4734109fd0 0x7f47341032d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f47337fe700 1 -- 172.21.15.50:0/3336032808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f473406ccf0 msgr2=0x7f47341046e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:24:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f47337fe700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f473406ccf0 0x7f47341046e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:24:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f47337fe700 1 -- 172.21.15.50:0/3336032808 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4734103a30 con 0x7f4734107460 2024-05-27T09:24:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.692+0000 7f4738d29700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4734109fd0 0x7f47341032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:24:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.693+0000 7f47337fe700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4734107460 0x7f4734102d90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4728006010 tx=0x7f47280047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:24:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.693+0000 7f47317fa700 1 -- 172.21.15.50:0/3336032808 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4728027070 con 0x7f4734107460 2024-05-27T09:24:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.693+0000 7f473a78c700 1 -- 172.21.15.50:0/3336032808 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47341a4a00 con 0x7f4734107460 2024-05-27T09:24:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.693+0000 7f47317fa700 1 -- 172.21.15.50:0/3336032808 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4728004140 con 0x7f4734107460 2024-05-27T09:24:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.693+0000 7f47317fa700 1 -- 172.21.15.50:0/3336032808 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472800f460 con 0x7f4734107460 2024-05-27T09:24:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.693+0000 7f473a78c700 1 -- 172.21.15.50:0/3336032808 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47341a4e20 con 0x7f4734107460 2024-05-27T09:24:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.694+0000 7f473a78c700 1 -- 172.21.15.50:0/3336032808 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47340512b0 con 0x7f4734107460 2024-05-27T09:24:58.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.695+0000 7f47317fa700 1 -- 172.21.15.50:0/3336032808 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f472802a070 con 0x7f4734107460 2024-05-27T09:24:58.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.696+0000 7f47317fa700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f471c0b27f0 0x7f471c0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:24:58.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.696+0000 7f47317fa700 1 -- 172.21.15.50:0/3336032808 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f47280e1620 con 0x7f4734107460 2024-05-27T09:24:58.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.697+0000 7f4733fff700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f471c0b27f0 0x7f471c0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:24:58.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.698+0000 7f4733fff700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f471c0b27f0 0x7f471c0b4d00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4724005950 tx=0x7f47240058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:24:58.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.701+0000 7f47317fa700 1 -- 172.21.15.50:0/3336032808 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4728013070 con 0x7f4734107460 2024-05-27T09:24:58.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:24:58.926+0000 7f473a78c700 1 -- 172.21.15.50:0/3336032808 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4734063bf0 con 0x7f471c0b27f0 2024-05-27T09:24:59.877 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:24:59.878 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi044 2024-05-27T09:24:59.878 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:59.878 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:59.878 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:24:59.878 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi044 to 17094M 2024-05-27T09:24:59.878 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:59.878 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:24:59.878 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:24:59.878 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:24:59.878 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:24:59.878 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:24:59.879 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:24:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi044 2024-05-27T09:25:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:25:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi044 to 17094M 2024-05-27T09:25:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:25:00.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:25:00.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:24:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi044 2024-05-27T09:25:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:25:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi044 to 17094M 2024-05-27T09:25:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:25:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:25:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:24:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:01.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:00 smithi050 ceph-mon[36371]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:25:01.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:00 smithi189 ceph-mon[36671]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:25:01.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:00 smithi044 ceph-mon[31618]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:25:02.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:01 smithi189 ceph-mon[36671]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:02.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:01 smithi189 ceph-mon[36671]: from='client.? 172.21.15.50:0/453256232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f7adef-a968-40d3-9722-d17f5d4f9670"}]: dispatch 2024-05-27T09:25:02.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:01 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f7adef-a968-40d3-9722-d17f5d4f9670"}]: dispatch 2024-05-27T09:25:02.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:01 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55f7adef-a968-40d3-9722-d17f5d4f9670"}]': finished 2024-05-27T09:25:02.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:01 smithi189 ceph-mon[36671]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T09:25:02.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:01 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:02.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:01 smithi044 ceph-mon[31618]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:02.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:01 smithi044 ceph-mon[31618]: from='client.? 172.21.15.50:0/453256232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f7adef-a968-40d3-9722-d17f5d4f9670"}]: dispatch 2024-05-27T09:25:02.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:01 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f7adef-a968-40d3-9722-d17f5d4f9670"}]: dispatch 2024-05-27T09:25:02.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:01 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55f7adef-a968-40d3-9722-d17f5d4f9670"}]': finished 2024-05-27T09:25:02.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:01 smithi044 ceph-mon[31618]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T09:25:02.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:01 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:02.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:01 smithi050 ceph-mon[36371]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:01 smithi050 ceph-mon[36371]: from='client.? 172.21.15.50:0/453256232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f7adef-a968-40d3-9722-d17f5d4f9670"}]: dispatch 2024-05-27T09:25:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:01 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f7adef-a968-40d3-9722-d17f5d4f9670"}]: dispatch 2024-05-27T09:25:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:01 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55f7adef-a968-40d3-9722-d17f5d4f9670"}]': finished 2024-05-27T09:25:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:01 smithi050 ceph-mon[36371]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T09:25:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:01 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:03.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:02 smithi189 ceph-mon[36671]: from='client.? 172.21.15.50:0/2544286854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:25:03.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:02 smithi044 ceph-mon[31618]: from='client.? 172.21.15.50:0/2544286854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:25:03.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:02 smithi050 ceph-mon[36371]: from='client.? 172.21.15.50:0/2544286854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:25:04.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:03 smithi189 ceph-mon[36671]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:03 smithi044 ceph-mon[31618]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:04.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:03 smithi050 ceph-mon[36371]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:05.981 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:05 smithi050 ceph-mon[36371]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:06.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:05 smithi189 ceph-mon[36671]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:05 smithi044 ceph-mon[31618]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:08.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:07 smithi050 ceph-mon[36371]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:08.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:07 smithi189 ceph-mon[36671]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:08.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:07 smithi044 ceph-mon[31618]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:09.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:08 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T09:25:09.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:08 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:09.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T09:25:09.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:09.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T09:25:09.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:09 smithi050 ceph-mon[36371]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:09 smithi050 ceph-mon[36371]: Deploying daemon osd.1 on smithi050 2024-05-27T09:25:10.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:09 smithi189 ceph-mon[36671]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:10.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:09 smithi189 ceph-mon[36671]: Deploying daemon osd.1 on smithi050 2024-05-27T09:25:10.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:09 smithi044 ceph-mon[31618]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:10.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:09 smithi044 ceph-mon[31618]: Deploying daemon osd.1 on smithi050 2024-05-27T09:25:12.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:11 smithi050 ceph-mon[36371]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:12.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:11 smithi189 ceph-mon[36671]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:12.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:11 smithi044 ceph-mon[31618]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:13.882 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:13 smithi050 ceph-mon[36371]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:13 smithi044 ceph-mon[31618]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:14.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:13 smithi189 ceph-mon[36671]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:15.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:15.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:15.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:15 smithi044 ceph-mon[31618]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:15.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:15.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:15.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:15.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:15 smithi050 ceph-mon[36371]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:15.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:15.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:15.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:15.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:15 smithi189 ceph-mon[36671]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:15.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:17.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:17 smithi050 ceph-mon[36371]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:17.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:17.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:17.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:17.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:17.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:17.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:17 smithi189 ceph-mon[36671]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:17.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:17.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:17.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:17.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:17.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:17 smithi044 ceph-mon[31618]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:18.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.212+0000 7f47317fa700 1 -- 172.21.15.50:0/3336032808 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4734063bf0 con 0x7f471c0b27f0 2024-05-27T09:25:18.212 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 1 on host 'smithi050' 2024-05-27T09:25:18.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 -- 172.21.15.50:0/3336032808 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f471c0b27f0 msgr2=0x7f471c0b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:25:18.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f471c0b27f0 0x7f471c0b4d00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4724005950 tx=0x7f47240058e0 comp rx=0 tx=0).stop 2024-05-27T09:25:18.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 -- 172.21.15.50:0/3336032808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4734107460 msgr2=0x7f4734102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:25:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4734107460 0x7f4734102d90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4728006010 tx=0x7f47280047b0 comp rx=0 tx=0).stop 2024-05-27T09:25:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 -- 172.21.15.50:0/3336032808 shutdown_connections 2024-05-27T09:25:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f471c0b27f0 0x7f471c0b4d00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4734109fd0 0x7f47341032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f473406ccf0 0x7f47341046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 --2- 172.21.15.50:0/3336032808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4734107460 0x7f4734102d90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 -- 172.21.15.50:0/3336032808 >> 172.21.15.50:0/3336032808 conn(0x7f47340fe860 msgr2=0x7f47341058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:25:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 -- 172.21.15.50:0/3336032808 shutdown_connections 2024-05-27T09:25:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:25:18.215+0000 7f473a78c700 1 -- 172.21.15.50:0/3336032808 wait complete. 2024-05-27T09:25:18.789 DEBUG:teuthology.orchestra.run.smithi050:osd.1> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.1.service 2024-05-27T09:25:18.792 INFO:tasks.cephadm:Deploying osd.2 on smithi189 with /dev/vg_nvme/lv_4... 2024-05-27T09:25:18.792 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T09:25:19.034 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:25:19.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:19.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:19.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:19.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:19 smithi044 ceph-mon[31618]: from='osd.1 [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T09:25:19.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:19 smithi044 ceph-mon[31618]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T09:25:19.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:19 smithi044 ceph-mon[31618]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:19.482 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:19.482 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:19.482 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:19.482 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:19 smithi189 ceph-mon[36671]: from='osd.1 [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T09:25:19.482 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:19 smithi189 ceph-mon[36671]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T09:25:19.482 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:19 smithi189 ceph-mon[36671]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:19.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:19.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:19.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:19.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:19 smithi050 ceph-mon[36371]: from='osd.1 [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T09:25:19.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:19 smithi050 ceph-mon[36371]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T09:25:19.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:19 smithi050 ceph-mon[36371]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:20.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:20 smithi189 ceph-mon[36671]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T09:25:20.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:20 smithi189 ceph-mon[36671]: from='osd.1 [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:25:20.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:20 smithi189 ceph-mon[36671]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T09:25:20.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:20 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:20.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:20 smithi189 ceph-mon[36671]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:25:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:20 smithi044 ceph-mon[31618]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T09:25:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:20 smithi044 ceph-mon[31618]: from='osd.1 [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:25:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:20 smithi044 ceph-mon[31618]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T09:25:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:20 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:20 smithi044 ceph-mon[31618]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:25:20.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:20 smithi050 ceph-mon[36371]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T09:25:20.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:20 smithi050 ceph-mon[36371]: from='osd.1 [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:25:20.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:20 smithi050 ceph-mon[36371]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T09:25:20.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:20 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:20.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:20 smithi050 ceph-mon[36371]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:25:20.545 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:25:20 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1[40952]: 2024-05-27T09:25:20.203+0000 7fb9d7dd1700 -1 osd.1 0 waiting for initial osdmap 2024-05-27T09:25:20.545 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:25:20 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1[40952]: 2024-05-27T09:25:20.207+0000 7fb9d3b86700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T09:25:20.871 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T09:25:20.872 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T09:25:20.872 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-05-27T09:25:20.872 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-05-27T09:25:20.872 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113737 s, 922 MB/s 2024-05-27T09:25:20.872 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-05-27T09:25:21.467 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:21 smithi189 ceph-mon[36671]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:25:21.467 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:21 smithi189 ceph-mon[36671]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T09:25:21.468 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:21.468 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:21 smithi189 ceph-mon[36671]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:21.468 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:21 smithi044 ceph-mon[31618]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:25:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:21 smithi044 ceph-mon[31618]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T09:25:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:21 smithi044 ceph-mon[31618]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:21.494 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_4 2024-05-27T09:25:21.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:21 smithi050 ceph-mon[36371]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:25:21.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:21 smithi050 ceph-mon[36371]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T09:25:21.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:21.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:21 smithi050 ceph-mon[36371]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T09:25:21.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:21.738 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:25:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:25:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:25:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: osd.1 [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354] boot 2024-05-27T09:25:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T09:25:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi050 2024-05-27T09:25:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:25:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi050 to 17093M 2024-05-27T09:25:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:22.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:22.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:22 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:25:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:25:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: osd.1 [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354] boot 2024-05-27T09:25:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T09:25:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi050 2024-05-27T09:25:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:25:22.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi050 to 17093M 2024-05-27T09:25:22.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:22.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:22.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:22.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:22 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:22.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:25:22.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:25:22.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: osd.1 [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354] boot 2024-05-27T09:25:22.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T09:25:22.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:25:22.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi050 2024-05-27T09:25:22.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:22.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:22.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:25:22.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi050 to 17093M 2024-05-27T09:25:22.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:22.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:22.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:22.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:22 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:23.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.253+0000 7fc93d75b700 1 -- 172.21.15.189:0/866358849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc938104350 msgr2=0x7fc938104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:25:23.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.253+0000 7fc93d75b700 1 --2- 172.21.15.189:0/866358849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc938104350 0x7fc938104760 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009990 tx=0x7fc92c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:25:23.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.253+0000 7fc93d75b700 1 -- 172.21.15.189:0/866358849 shutdown_connections 2024-05-27T09:25:23.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.253+0000 7fc93d75b700 1 --2- 172.21.15.189:0/866358849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc938104cf0 0x7fc93810b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:23.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.253+0000 7fc93d75b700 1 --2- 172.21.15.189:0/866358849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc938103220 0x7fc938103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:23.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.253+0000 7fc93d75b700 1 --2- 172.21.15.189:0/866358849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc938104350 0x7fc938104760 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.253+0000 7fc93d75b700 1 -- 172.21.15.189:0/866358849 >> 172.21.15.189:0/866358849 conn(0x7fc9380fe860 msgr2=0x7fc938100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:25:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.253+0000 7fc93d75b700 1 -- 172.21.15.189:0/866358849 shutdown_connections 2024-05-27T09:25:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.253+0000 7fc93d75b700 1 -- 172.21.15.189:0/866358849 wait complete. 2024-05-27T09:25:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc93d75b700 1 Processor -- start 2024-05-27T09:25:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc93d75b700 1 -- start start 2024-05-27T09:25:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc93d75b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc938103220 0x7fc93806cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:25:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc93d75b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc938104350 0x7fc93819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:25:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc93d75b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc938104cf0 0x7fc9381a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:25:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc93d75b700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc93810ed50 con 0x7fc938103220 2024-05-27T09:25:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc93d75b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc93810ebd0 con 0x7fc938104cf0 2024-05-27T09:25:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc93d75b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc93810eed0 con 0x7fc938104350 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc9367fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc938104350 0x7fc93819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc9367fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc938104350 0x7fc93819f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:43112/0 (socket says 172.21.15.189:43112) 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc9367fc700 1 -- 172.21.15.189:0/3605861095 learned_addr learned my addr 172.21.15.189:0/3605861095 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.254+0000 7fc9377fe700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc938104cf0 0x7fc9381a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.255+0000 7fc9367fc700 1 -- 172.21.15.189:0/3605861095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc938104cf0 msgr2=0x7fc9381a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.255+0000 7fc9367fc700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc938104cf0 0x7fc9381a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.255+0000 7fc9367fc700 1 -- 172.21.15.189:0/3605861095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc938103220 msgr2=0x7fc93806cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.255+0000 7fc9367fc700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc938103220 0x7fc93806cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.255+0000 7fc9367fc700 1 -- 172.21.15.189:0/3605861095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9381a24f0 con 0x7fc938104350 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.255+0000 7fc9367fc700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc938104350 0x7fc93819f730 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00b760 tx=0x7fc92c004790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.255+0000 7fc91ffff700 1 -- 172.21.15.189:0/3605861095 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c00fb90 con 0x7fc938104350 2024-05-27T09:25:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.255+0000 7fc93d75b700 1 -- 172.21.15.189:0/3605861095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9381a2780 con 0x7fc938104350 2024-05-27T09:25:23.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.255+0000 7fc91ffff700 1 -- 172.21.15.189:0/3605861095 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc92c00fd30 con 0x7fc938104350 2024-05-27T09:25:23.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.255+0000 7fc91ffff700 1 -- 172.21.15.189:0/3605861095 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c0173f0 con 0x7fc938104350 2024-05-27T09:25:23.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.255+0000 7fc93d75b700 1 -- 172.21.15.189:0/3605861095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9381a2c00 con 0x7fc938104350 2024-05-27T09:25:23.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.256+0000 7fc93d75b700 1 -- 172.21.15.189:0/3605861095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9380512b0 con 0x7fc938104350 2024-05-27T09:25:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.258+0000 7fc91ffff700 1 -- 172.21.15.189:0/3605861095 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7fc92c028060 con 0x7fc938104350 2024-05-27T09:25:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.258+0000 7fc91ffff700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc9200b27f0 0x7fc9200b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:25:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.258+0000 7fc91ffff700 1 -- 172.21.15.189:0/3605861095 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fc92c0e16a0 con 0x7fc938104350 2024-05-27T09:25:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.259+0000 7fc936ffd700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc9200b27f0 0x7fc9200b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:25:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.260+0000 7fc936ffd700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc9200b27f0 0x7fc9200b4d00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc938104250 tx=0x7fc93000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:25:23.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.263+0000 7fc91ffff700 1 -- 172.21.15.189:0/3605861095 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc92c013070 con 0x7fc938104350 2024-05-27T09:25:23.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:23.473+0000 7fc93d75b700 1 -- 172.21.15.189:0/3605861095 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc938063bf0 con 0x7fc9200b27f0 2024-05-27T09:25:23.474 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:23 smithi189 ceph-mon[36671]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T09:25:23.474 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:23 smithi189 ceph-mon[36671]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:23.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:23 smithi050 ceph-mon[36371]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T09:25:23.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:23 smithi050 ceph-mon[36371]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:23.626 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:23 smithi044 ceph-mon[31618]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T09:25:23.626 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:23 smithi044 ceph-mon[31618]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:24.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:24 smithi189 ceph-mon[36671]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:25:24.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:24 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:25:24.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:24 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:25:24.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:24 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:24.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:24 smithi050 ceph-mon[36371]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:25:24.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:24 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:25:24.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:24 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:25:24.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:24 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:24.661 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:24 smithi044 ceph-mon[31618]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:25:24.662 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:24 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:25:24.662 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:24 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:25:24.662 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:24 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:25.360 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:25 smithi044 ceph-mon[31618]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:25.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:25 smithi050 ceph-mon[36371]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:25.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:25 smithi189 ceph-mon[36671]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:26.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:26 smithi050 ceph-mon[36371]: from='client.? 172.21.15.189:0/633870414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b3e59b2-65cf-40df-ab3c-eee28b4157d6"}]: dispatch 2024-05-27T09:25:26.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:26 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b3e59b2-65cf-40df-ab3c-eee28b4157d6"}]: dispatch 2024-05-27T09:25:26.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:26 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b3e59b2-65cf-40df-ab3c-eee28b4157d6"}]': finished 2024-05-27T09:25:26.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:26 smithi050 ceph-mon[36371]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T09:25:26.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:26 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:26.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:26 smithi189 ceph-mon[36671]: from='client.? 172.21.15.189:0/633870414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b3e59b2-65cf-40df-ab3c-eee28b4157d6"}]: dispatch 2024-05-27T09:25:26.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:26 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b3e59b2-65cf-40df-ab3c-eee28b4157d6"}]: dispatch 2024-05-27T09:25:26.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:26 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b3e59b2-65cf-40df-ab3c-eee28b4157d6"}]': finished 2024-05-27T09:25:26.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:26 smithi189 ceph-mon[36671]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T09:25:26.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:26 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:26.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:26 smithi044 ceph-mon[31618]: from='client.? 172.21.15.189:0/633870414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b3e59b2-65cf-40df-ab3c-eee28b4157d6"}]: dispatch 2024-05-27T09:25:26.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:26 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b3e59b2-65cf-40df-ab3c-eee28b4157d6"}]: dispatch 2024-05-27T09:25:26.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:26 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b3e59b2-65cf-40df-ab3c-eee28b4157d6"}]': finished 2024-05-27T09:25:26.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:26 smithi044 ceph-mon[31618]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T09:25:26.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:26 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:27.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:27 smithi050 ceph-mon[36371]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:27.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:27 smithi050 ceph-mon[36371]: from='client.? 172.21.15.189:0/2298782141' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:25:27.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:27 smithi189 ceph-mon[36671]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:27.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:27 smithi189 ceph-mon[36671]: from='client.? 172.21.15.189:0/2298782141' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:25:27.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:27 smithi044 ceph-mon[31618]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:27.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:27 smithi044 ceph-mon[31618]: from='client.? 172.21.15.189:0/2298782141' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:25:29.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:29 smithi189 ceph-mon[36671]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:29.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:29 smithi044 ceph-mon[31618]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:30.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:29 smithi050 ceph-mon[36371]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:31.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:31 smithi189 ceph-mon[36671]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:31.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:31 smithi044 ceph-mon[31618]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:32.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:31 smithi050 ceph-mon[36371]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:33.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:33 smithi189 ceph-mon[36671]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:33.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:33 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T09:25:33.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:33 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:33.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:33 smithi044 ceph-mon[31618]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:33.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:33 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T09:25:33.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:33 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:34.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:33 smithi050 ceph-mon[36371]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:34.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:33 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T09:25:34.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:33 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:34.746 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:34 smithi189 ceph-mon[36671]: Deploying daemon osd.2 on smithi189 2024-05-27T09:25:34.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:34 smithi044 ceph-mon[31618]: Deploying daemon osd.2 on smithi189 2024-05-27T09:25:35.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:34 smithi050 ceph-mon[36371]: Deploying daemon osd.2 on smithi189 2024-05-27T09:25:35.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:35 smithi189 ceph-mon[36671]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:35.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:35 smithi044 ceph-mon[31618]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:36.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:35 smithi050 ceph-mon[36371]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:37.769 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:37 smithi189 ceph-mon[36671]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:37.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:37 smithi044 ceph-mon[31618]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:38.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:37 smithi050 ceph-mon[36371]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:39.829 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:39 smithi189 ceph-mon[36671]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:39.829 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:39.829 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:39.829 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:39 smithi044 ceph-mon[31618]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:39 smithi050 ceph-mon[36371]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:41 smithi050 ceph-mon[36371]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:41.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:41 smithi189 ceph-mon[36671]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:41.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:41 smithi044 ceph-mon[31618]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:41.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:41.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:41.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:41.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:41.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:42.208 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 2 on host 'smithi189' 2024-05-27T09:25:42.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.207+0000 7fc91ffff700 1 -- 172.21.15.189:0/3605861095 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc938063bf0 con 0x7fc9200b27f0 2024-05-27T09:25:42.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 -- 172.21.15.189:0/3605861095 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc9200b27f0 msgr2=0x7fc9200b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:25:42.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc9200b27f0 0x7fc9200b4d00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc938104250 tx=0x7fc93000b040 comp rx=0 tx=0).stop 2024-05-27T09:25:42.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 -- 172.21.15.189:0/3605861095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc938104350 msgr2=0x7fc93819f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:25:42.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc938104350 0x7fc93819f730 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00b760 tx=0x7fc92c004790 comp rx=0 tx=0).stop 2024-05-27T09:25:42.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 -- 172.21.15.189:0/3605861095 shutdown_connections 2024-05-27T09:25:42.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc9200b27f0 0x7fc9200b4d00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:42.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc938104cf0 0x7fc9381a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:42.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc938103220 0x7fc93806cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:42.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 --2- 172.21.15.189:0/3605861095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc938104350 0x7fc93819f730 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:42.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 -- 172.21.15.189:0/3605861095 >> 172.21.15.189:0/3605861095 conn(0x7fc9380fe860 msgr2=0x7fc938107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:25:42.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 -- 172.21.15.189:0/3605861095 shutdown_connections 2024-05-27T09:25:42.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:25:42.210+0000 7fc93d75b700 1 -- 172.21.15.189:0/3605861095 wait complete. 2024-05-27T09:25:42.813 DEBUG:teuthology.orchestra.run.smithi189:osd.2> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.2.service 2024-05-27T09:25:42.815 INFO:tasks.cephadm:Deploying osd.3 on smithi044 with /dev/vg_nvme/lv_3... 2024-05-27T09:25:42.816 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T09:25:43.287 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:25:43.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:43.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:43.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:43.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:43 smithi044 ceph-mon[31618]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:43.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:43 smithi044 ceph-mon[31618]: from='osd.2 [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T09:25:43.446 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:43 smithi044 ceph-mon[31618]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T09:25:43.525 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:43.525 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:43.525 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:43.525 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:43 smithi189 ceph-mon[36671]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:43.525 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:43 smithi189 ceph-mon[36671]: from='osd.2 [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T09:25:43.525 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:43 smithi189 ceph-mon[36671]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T09:25:43.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:43.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:43.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:25:43.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:43 smithi050 ceph-mon[36371]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:43.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:43 smithi050 ceph-mon[36371]: from='osd.2 [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T09:25:43.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:43 smithi050 ceph-mon[36371]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T09:25:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:44 smithi189 ceph-mon[36671]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T09:25:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:44 smithi189 ceph-mon[36671]: from='osd.2 [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:25:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:44 smithi189 ceph-mon[36671]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T09:25:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:44 smithi189 ceph-mon[36671]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:25:44.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:44 smithi044 ceph-mon[31618]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T09:25:44.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:44 smithi044 ceph-mon[31618]: from='osd.2 [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:25:44.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:44 smithi044 ceph-mon[31618]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T09:25:44.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:44.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:44 smithi044 ceph-mon[31618]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:25:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:44 smithi050 ceph-mon[36371]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T09:25:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:44 smithi050 ceph-mon[36371]: from='osd.2 [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:25:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:44 smithi050 ceph-mon[36371]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T09:25:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:44 smithi050 ceph-mon[36371]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:25:45.228 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:45 smithi189 ceph-mon[36671]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:25:45.228 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:45 smithi189 ceph-mon[36671]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T09:25:45.228 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:45 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:45.228 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:45 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:45.228 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:45 smithi189 ceph-mon[36671]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:45.228 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:45 smithi189 ceph-mon[36671]: from='osd.2 ' entity='osd.2' 2024-05-27T09:25:45.228 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:45 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:45.229 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:25:45 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2[41765]: 2024-05-27T09:25:45.108+0000 7f979f3f3700 -1 osd.2 0 waiting for initial osdmap 2024-05-27T09:25:45.229 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:25:45 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2[41765]: 2024-05-27T09:25:45.187+0000 7f979699f700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T09:25:45.313 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:45 smithi044 ceph-mon[31618]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:25:45.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:45 smithi044 ceph-mon[31618]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T09:25:45.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:45 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:45.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:45 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:45.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:45 smithi044 ceph-mon[31618]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:45.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:45 smithi044 ceph-mon[31618]: from='osd.2 ' entity='osd.2' 2024-05-27T09:25:45.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:45 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:45 smithi050 ceph-mon[36371]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:25:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:45 smithi050 ceph-mon[36371]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T09:25:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:45 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:45 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:45 smithi050 ceph-mon[36371]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T09:25:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:45 smithi050 ceph-mon[36371]: from='osd.2 ' entity='osd.2' 2024-05-27T09:25:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:45 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:46.452 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T09:25:46.452 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T09:25:46.452 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-05-27T09:25:46.452 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-05-27T09:25:46.452 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0340375 s, 308 MB/s 2024-05-27T09:25:46.452 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-05-27T09:25:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:25:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:25:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi189 2024-05-27T09:25:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:25:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi189 to 17088M 2024-05-27T09:25:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:46.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: osd.2 [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374] boot 2024-05-27T09:25:46.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T09:25:46.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:46.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi189 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi189 to 17088M 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: osd.2 [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374] boot 2024-05-27T09:25:46.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T09:25:46.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:46.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:25:46.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:25:46.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi189 2024-05-27T09:25:46.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:25:46.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi189 to 17088M 2024-05-27T09:25:46.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:46.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:25:46.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:46.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: osd.2 [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374] boot 2024-05-27T09:25:46.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T09:25:46.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:25:47.221 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_3 2024-05-27T09:25:47.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:47 smithi044 ceph-mon[31618]: pgmap v93: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:47.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:25:47.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:47 smithi050 ceph-mon[36371]: pgmap v93: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:47.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:47 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:25:47.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:47 smithi189 ceph-mon[36671]: pgmap v93: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:47.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:47 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:25:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:25:48.409 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T09:25:48.409 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:48 smithi044 ceph-mon[31618]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T09:25:48.409 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:25:48.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T09:25:48.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:48 smithi050 ceph-mon[36371]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T09:25:48.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:25:48.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T09:25:48.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:48 smithi189 ceph-mon[36671]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T09:25:48.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:25:48.725 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51192]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T09:25:48.725 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51192]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:48.725 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51192]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:48.725 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51192]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:48.725 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51196]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T09:25:48.725 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51196]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:48.725 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51196]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:48.725 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51196]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:48.893 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44636]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T09:25:48.893 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44636]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:48.893 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44636]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:48.893 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44636]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:48.894 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44639]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T09:25:48.894 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44639]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:48.894 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44639]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:48.894 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44639]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:49.088 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45413]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T09:25:49.088 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45413]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:49.088 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45413]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:49.088 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45413]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:49.088 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45416]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T09:25:49.088 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45416]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:49.088 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45416]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:49.089 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45416]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:49.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45419]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T09:25:49.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45419]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:49.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45419]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:49.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45419]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:49.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45422]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T09:25:49.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45422]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:49.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45422]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:49.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:48 smithi189 sudo[45422]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:49.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51200]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T09:25:49.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51200]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:49.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51200]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:49.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51200]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:49.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51203]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T09:25:49.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51203]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:49.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51203]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:49.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:48 smithi044 sudo[51203]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:49.289 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44642]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T09:25:49.289 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44642]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:49.289 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44642]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:49.289 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44642]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:49.289 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44645]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T09:25:49.289 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44645]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T09:25:49.289 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44645]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T09:25:49.290 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:48 smithi050 sudo[44645]: pam_unix(sudo:session): session closed for user root 2024-05-27T09:25:49.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T09:25:49.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T09:25:49.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:49.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:49.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:49.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:49.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T09:25:49.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T09:25:49.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:25:49.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:25:49.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:25:49.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T09:25:49.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:25:49.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:25:49.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:25:49.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T09:25:49.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T09:25:49.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:25:49.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:25:49.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:25:49.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T09:25:49.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:49 smithi050 ceph-mon[36371]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T09:25:49.554 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T09:25:49.554 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T09:25:49.554 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:49.554 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:49.554 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:49.555 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:49.555 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T09:25:49.555 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T09:25:49.555 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:25:49.555 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:25:49.555 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:25:49.555 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T09:25:49.556 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:25:49.556 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:25:49.556 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:25:49.557 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T09:25:49.557 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T09:25:49.557 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:25:49.557 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:25:49.557 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:25:49.558 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T09:25:49.558 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:49 smithi044 ceph-mon[31618]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T09:25:49.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T09:25:49.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T09:25:49.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:49.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:49.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:49.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:25:49.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T09:25:49.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T09:25:49.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:25:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:25:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:25:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T09:25:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:25:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:25:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:25:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T09:25:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T09:25:49.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:25:49.590 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:25:49.590 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:25:49.590 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T09:25:49.590 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:49 smithi189 ceph-mon[36671]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T09:25:49.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.805+0000 7f6b36c00700 1 -- 172.21.15.44:0/551162711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b300776d0 msgr2=0x7f6b30075ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:25:49.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.805+0000 7f6b36c00700 1 --2- 172.21.15.44:0/551162711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b300776d0 0x7f6b30075ba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6b20009990 tx=0x7f6b20009ca0 comp rx=0 tx=0).stop 2024-05-27T09:25:49.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.805+0000 7f6b36c00700 1 -- 172.21.15.44:0/551162711 shutdown_connections 2024-05-27T09:25:49.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.805+0000 7f6b36c00700 1 --2- 172.21.15.44:0/551162711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b3010c1e0 0x7f6b3010e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:49.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.805+0000 7f6b36c00700 1 --2- 172.21.15.44:0/551162711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b300776d0 0x7f6b30075ba0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:49.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.805+0000 7f6b36c00700 1 --2- 172.21.15.44:0/551162711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b300760e0 0x7f6b30076570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:49.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.805+0000 7f6b36c00700 1 -- 172.21.15.44:0/551162711 >> 172.21.15.44:0/551162711 conn(0x7f6b300ff9b0 msgr2=0x7f6b30101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:25:49.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.805+0000 7f6b36c00700 1 -- 172.21.15.44:0/551162711 shutdown_connections 2024-05-27T09:25:49.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.806+0000 7f6b36c00700 1 -- 172.21.15.44:0/551162711 wait complete. 2024-05-27T09:25:49.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.807+0000 7f6b36c00700 1 Processor -- start 2024-05-27T09:25:49.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.807+0000 7f6b36c00700 1 -- start start 2024-05-27T09:25:49.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.808+0000 7f6b36c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b300760e0 0x7f6b3019cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:25:49.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.808+0000 7f6b36c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b300776d0 0x7f6b3019d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:25:49.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.808+0000 7f6b36c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b3010c1e0 0x7f6b301a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:25:49.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.808+0000 7f6b36c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6b30111fc0 con 0x7f6b3010c1e0 2024-05-27T09:25:49.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.808+0000 7f6b36c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6b30111e40 con 0x7f6b300776d0 2024-05-27T09:25:49.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.808+0000 7f6b36c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6b30112140 con 0x7f6b300760e0 2024-05-27T09:25:49.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.808+0000 7f6b2fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b300760e0 0x7f6b3019cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:25:49.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.808+0000 7f6b2fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b300760e0 0x7f6b3019cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:58840/0 (socket says 172.21.15.44:58840) 2024-05-27T09:25:49.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.808+0000 7f6b2fe00700 1 -- 172.21.15.44:0/533796733 learned_addr learned my addr 172.21.15.44:0/533796733 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:25:49.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.808+0000 7f6b2f400700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b300776d0 0x7f6b3019d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:25:49.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.808+0000 7f6b34c00700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b3010c1e0 0x7f6b301a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:25:49.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.809+0000 7f6b2fe00700 1 -- 172.21.15.44:0/533796733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b300776d0 msgr2=0x7f6b3019d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:25:49.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.809+0000 7f6b2fe00700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b300776d0 0x7f6b3019d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:49.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.809+0000 7f6b2fe00700 1 -- 172.21.15.44:0/533796733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b3010c1e0 msgr2=0x7f6b301a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:25:49.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.809+0000 7f6b2fe00700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b3010c1e0 0x7f6b301a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:25:49.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.809+0000 7f6b2fe00700 1 -- 172.21.15.44:0/533796733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b301a46a0 con 0x7f6b300760e0 2024-05-27T09:25:49.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.809+0000 7f6b2fe00700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b300760e0 0x7f6b3019cb40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000b760 tx=0x7f6b200057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:25:49.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.809+0000 7f6b2cc00700 1 -- 172.21.15.44:0/533796733 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b20026070 con 0x7f6b300760e0 2024-05-27T09:25:49.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.809+0000 7f6b2cc00700 1 -- 172.21.15.44:0/533796733 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b20021410 con 0x7f6b300760e0 2024-05-27T09:25:49.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.809+0000 7f6b2cc00700 1 -- 172.21.15.44:0/533796733 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b20017650 con 0x7f6b300760e0 2024-05-27T09:25:49.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.809+0000 7f6b36c00700 1 -- 172.21.15.44:0/533796733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b301a4930 con 0x7f6b300760e0 2024-05-27T09:25:49.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.810+0000 7f6b36c00700 1 -- 172.21.15.44:0/533796733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b301a4d30 con 0x7f6b300760e0 2024-05-27T09:25:49.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.811+0000 7f6b36c00700 1 -- 172.21.15.44:0/533796733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b300512b0 con 0x7f6b300760e0 2024-05-27T09:25:49.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.813+0000 7f6b2cc00700 1 -- 172.21.15.44:0/533796733 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f6b20029070 con 0x7f6b300760e0 2024-05-27T09:25:49.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.815+0000 7f6b2cc00700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6b140b2900 0x7f6b140b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:25:49.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.815+0000 7f6b2cc00700 1 -- 172.21.15.44:0/533796733 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f6b200e1e20 con 0x7f6b300760e0 2024-05-27T09:25:49.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.815+0000 7f6b2f400700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6b140b2900 0x7f6b140b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:25:49.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.816+0000 7f6b2f400700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6b140b2900 0x7f6b140b4e10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b3019dbf0 tx=0x7f6b1c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:25:49.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:49.817+0000 7f6b2cc00700 1 -- 172.21.15.44:0/533796733 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6b2005edf0 con 0x7f6b300760e0 2024-05-27T09:25:50.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:25:50.066+0000 7f6b36c00700 1 -- 172.21.15.44:0/533796733 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6b30063cb0 con 0x7f6b140b2900 2024-05-27T09:25:50.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:50 smithi189 ceph-mon[36671]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-27T09:25:50.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:50 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:25:50.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:50 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:25:50.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:50 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:50.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:50 smithi044 ceph-mon[31618]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-27T09:25:50.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:50 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:25:50.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:50 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:25:50.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:50 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:51.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:50 smithi050 ceph-mon[36371]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-27T09:25:51.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:50 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:25:51.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:50 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:25:51.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:50 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:25:51.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:51 smithi044 ceph-mon[31618]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:25:51.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:51 smithi044 ceph-mon[31618]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:51.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:51 smithi189 ceph-mon[36671]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:25:51.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:51 smithi189 ceph-mon[36671]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:52.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:51 smithi050 ceph-mon[36371]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:25:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:51 smithi050 ceph-mon[36371]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:53.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:53 smithi189 ceph-mon[36671]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:53.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:53 smithi044 ceph-mon[31618]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:54.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:53 smithi050 ceph-mon[36371]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:54.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:54 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2456492449' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9928a4-b7d8-436e-893b-5146bbbd7f4a"}]: dispatch 2024-05-27T09:25:54.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:54 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2456492449' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c9928a4-b7d8-436e-893b-5146bbbd7f4a"}]': finished 2024-05-27T09:25:54.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:54 smithi189 ceph-mon[36671]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T09:25:54.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:54 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:25:54.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:54 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2456492449' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9928a4-b7d8-436e-893b-5146bbbd7f4a"}]: dispatch 2024-05-27T09:25:54.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:54 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2456492449' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c9928a4-b7d8-436e-893b-5146bbbd7f4a"}]': finished 2024-05-27T09:25:54.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:54 smithi044 ceph-mon[31618]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T09:25:54.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:54 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:25:55.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:54 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2456492449' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9928a4-b7d8-436e-893b-5146bbbd7f4a"}]: dispatch 2024-05-27T09:25:55.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:54 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2456492449' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c9928a4-b7d8-436e-893b-5146bbbd7f4a"}]': finished 2024-05-27T09:25:55.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:54 smithi050 ceph-mon[36371]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T09:25:55.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:54 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:25:55.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:55 smithi189 ceph-mon[36671]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:55.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:55 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/670329806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:25:55.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:55 smithi044 ceph-mon[31618]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:55.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:55 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/670329806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:25:56.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:55 smithi050 ceph-mon[36371]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:55 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/670329806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:25:57.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:57 smithi189 ceph-mon[36671]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:57.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:57 smithi044 ceph-mon[31618]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:58.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:57 smithi050 ceph-mon[36371]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:59.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:25:59 smithi044 ceph-mon[31618]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:25:59.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:25:59 smithi189 ceph-mon[36671]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:00.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:25:59 smithi050 ceph-mon[36371]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:01.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:01 smithi044 ceph-mon[31618]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:01.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:01 smithi189 ceph-mon[36671]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:01.846 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:01 smithi050 ceph-mon[36371]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:03.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:03 smithi189 ceph-mon[36671]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:03.901 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:03 smithi044 ceph-mon[31618]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:04.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:03 smithi050 ceph-mon[36371]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:04.723 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:04 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T09:26:04.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:04 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:05.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:04 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T09:26:05.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:04 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:05.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:04 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T09:26:05.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:04 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:05.818 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:05 smithi044 ceph-mon[31618]: Deploying daemon osd.3 on smithi044 2024-05-27T09:26:05.818 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:05 smithi044 ceph-mon[31618]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:06.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:05 smithi050 ceph-mon[36371]: Deploying daemon osd.3 on smithi044 2024-05-27T09:26:06.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:05 smithi050 ceph-mon[36371]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:06.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:05 smithi189 ceph-mon[36671]: Deploying daemon osd.3 on smithi044 2024-05-27T09:26:06.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:05 smithi189 ceph-mon[36671]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:07.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:07 smithi044 ceph-mon[31618]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:08.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:07 smithi050 ceph-mon[36371]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:08.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:07 smithi189 ceph-mon[36671]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:09.890 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:09 smithi189 ceph-mon[36671]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:09.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:09 smithi044 ceph-mon[31618]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:10.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:09 smithi050 ceph-mon[36371]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:11 smithi044 ceph-mon[31618]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:12.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:11 smithi050 ceph-mon[36371]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:12.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:11 smithi189 ceph-mon[36671]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:13 smithi044 ceph-mon[31618]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:14.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:13 smithi050 ceph-mon[36371]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:14.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:13 smithi189 ceph-mon[36671]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:14.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:14.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:14.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:15.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:15.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:15.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:15.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:15.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:15.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:15.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:15 smithi044 ceph-mon[31618]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:16.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:15 smithi050 ceph-mon[36371]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:16.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:15 smithi189 ceph-mon[36671]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:17.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:17 smithi044 ceph-mon[31618]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:17 smithi050 ceph-mon[36371]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:18.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:17 smithi189 ceph-mon[36671]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:18.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:18.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:18 smithi050 ceph-mon[36371]: from='osd.3 [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T09:26:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:18 smithi050 ceph-mon[36371]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T09:26:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:18 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:18 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:18 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:19.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:18 smithi189 ceph-mon[36671]: from='osd.3 [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T09:26:19.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:18 smithi189 ceph-mon[36671]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T09:26:19.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:18 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:19.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:18 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:19.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:18 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:19.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:18 smithi044 ceph-mon[31618]: from='osd.3 [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T09:26:19.106 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:18 smithi044 ceph-mon[31618]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T09:26:19.106 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:18 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:19.106 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:18 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:19.106 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:18 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.223+0000 7f6b2cc00700 1 -- 172.21.15.44:0/533796733 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6b30063cb0 con 0x7f6b140b2900 2024-05-27T09:26:19.225 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 3 on host 'smithi044' 2024-05-27T09:26:19.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.227+0000 7f6b36c00700 1 -- 172.21.15.44:0/533796733 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6b140b2900 msgr2=0x7f6b140b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:26:19.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.227+0000 7f6b36c00700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6b140b2900 0x7f6b140b4e10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b3019dbf0 tx=0x7f6b1c009260 comp rx=0 tx=0).stop 2024-05-27T09:26:19.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.227+0000 7f6b36c00700 1 -- 172.21.15.44:0/533796733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b300760e0 msgr2=0x7f6b3019cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:26:19.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.227+0000 7f6b36c00700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b300760e0 0x7f6b3019cb40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000b760 tx=0x7f6b200057c0 comp rx=0 tx=0).stop 2024-05-27T09:26:19.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.227+0000 7f6b36c00700 1 -- 172.21.15.44:0/533796733 shutdown_connections 2024-05-27T09:26:19.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.227+0000 7f6b36c00700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6b140b2900 0x7f6b140b4e10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:19.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.227+0000 7f6b36c00700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6b3010c1e0 0x7f6b301a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:19.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.227+0000 7f6b36c00700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b300760e0 0x7f6b3019cb40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:19.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.227+0000 7f6b36c00700 1 --2- 172.21.15.44:0/533796733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b300776d0 0x7f6b3019d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:19.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.227+0000 7f6b36c00700 1 -- 172.21.15.44:0/533796733 >> 172.21.15.44:0/533796733 conn(0x7f6b300ff9b0 msgr2=0x7f6b30108170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:26:19.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.228+0000 7f6b36c00700 1 -- 172.21.15.44:0/533796733 shutdown_connections 2024-05-27T09:26:19.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:26:19.228+0000 7f6b36c00700 1 -- 172.21.15.44:0/533796733 wait complete. 2024-05-27T09:26:19.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:19 smithi044 ceph-mon[31618]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:19.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:19 smithi044 ceph-mon[31618]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T09:26:19.883 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:19 smithi044 ceph-mon[31618]: from='osd.3 [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:26:19.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:19 smithi044 ceph-mon[31618]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T09:26:19.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:19.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:19 smithi044 ceph-mon[31618]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:26:19.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:19.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:19.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:19 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:19.926 DEBUG:teuthology.orchestra.run.smithi044:osd.3> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.3.service 2024-05-27T09:26:19.930 INFO:tasks.cephadm:Deploying osd.4 on smithi050 with /dev/vg_nvme/lv_3... 2024-05-27T09:26:19.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T09:26:20.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:19 smithi050 ceph-mon[36371]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:20.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:19 smithi050 ceph-mon[36371]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T09:26:20.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:19 smithi050 ceph-mon[36371]: from='osd.3 [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:26:20.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:19 smithi050 ceph-mon[36371]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T09:26:20.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:20.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:19 smithi050 ceph-mon[36371]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:26:20.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:20.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:20.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:19 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:20.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:19 smithi189 ceph-mon[36671]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:20.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:19 smithi189 ceph-mon[36671]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T09:26:20.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:19 smithi189 ceph-mon[36671]: from='osd.3 [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:26:20.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:19 smithi189 ceph-mon[36671]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T09:26:20.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:20.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:19 smithi189 ceph-mon[36671]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:26:20.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:20.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:20.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:19 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:20.186 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:26:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:20 smithi050 ceph-mon[36371]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:26:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:20 smithi050 ceph-mon[36371]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T09:26:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:20 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:20 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:21.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:20 smithi044 ceph-mon[31618]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:26:21.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:20 smithi044 ceph-mon[31618]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T09:26:21.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:20 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:21.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:20 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:21.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:20 smithi189 ceph-mon[36671]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:26:21.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:20 smithi189 ceph-mon[36671]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T09:26:21.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:20 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:21.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:20 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:21 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:26:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:21 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:26:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:21 smithi050 ceph-mon[36371]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:21 smithi050 ceph-mon[36371]: osd.3 [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697] boot 2024-05-27T09:26:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:21 smithi050 ceph-mon[36371]: osdmap e25: 4 total, 4 up, 4 in 2024-05-27T09:26:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:21 smithi050 ceph-mon[36371]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T09:26:22.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:21 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:26:22.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:21 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:26:22.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:21 smithi044 ceph-mon[31618]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:22.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:21 smithi044 ceph-mon[31618]: osd.3 [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697] boot 2024-05-27T09:26:22.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:21 smithi044 ceph-mon[31618]: osdmap e25: 4 total, 4 up, 4 in 2024-05-27T09:26:22.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:22.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:21 smithi044 ceph-mon[31618]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T09:26:22.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:21 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:26:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:21 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:26:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:21 smithi189 ceph-mon[36671]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T09:26:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:21 smithi189 ceph-mon[36671]: osd.3 [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697] boot 2024-05-27T09:26:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:21 smithi189 ceph-mon[36671]: osdmap e25: 4 total, 4 up, 4 in 2024-05-27T09:26:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:26:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:21 smithi189 ceph-mon[36671]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T09:26:22.160 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T09:26:22.160 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T09:26:22.160 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-05-27T09:26:22.160 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-05-27T09:26:22.160 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0083237 s, 1.3 GB/s 2024-05-27T09:26:22.160 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-05-27T09:26:22.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_3 2024-05-27T09:26:22.975 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:26:24.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:23 smithi050 ceph-mon[36371]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:24.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:23 smithi050 ceph-mon[36371]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T09:26:24.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:23 smithi189 ceph-mon[36671]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:24.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:23 smithi189 ceph-mon[36671]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T09:26:24.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:23 smithi044 ceph-mon[31618]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:24.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:23 smithi044 ceph-mon[31618]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T09:26:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.532+0000 7fe53942e700 1 -- 172.21.15.50:0/3820460856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe534104360 msgr2=0x7fe534104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:26:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.532+0000 7fe53942e700 1 --2- 172.21.15.50:0/3820460856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe534104360 0x7fe534104770 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe524009990 tx=0x7fe524009ca0 comp rx=0 tx=0).stop 2024-05-27T09:26:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.532+0000 7fe53942e700 1 -- 172.21.15.50:0/3820460856 shutdown_connections 2024-05-27T09:26:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.532+0000 7fe53942e700 1 --2- 172.21.15.50:0/3820460856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe534105e30 0x7fe53410c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.532+0000 7fe53942e700 1 --2- 172.21.15.50:0/3820460856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe534104360 0x7fe534104770 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.532+0000 7fe53942e700 1 --2- 172.21.15.50:0/3820460856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe534105490 0x7fe5341058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.532+0000 7fe53942e700 1 -- 172.21.15.50:0/3820460856 >> 172.21.15.50:0/3820460856 conn(0x7fe5340ff9c0 msgr2=0x7fe534101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:26:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.533+0000 7fe53942e700 1 -- 172.21.15.50:0/3820460856 shutdown_connections 2024-05-27T09:26:24.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.533+0000 7fe53942e700 1 -- 172.21.15.50:0/3820460856 wait complete. 2024-05-27T09:26:24.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.533+0000 7fe53942e700 1 Processor -- start 2024-05-27T09:26:24.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe53942e700 1 -- start start 2024-05-27T09:26:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe53942e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe534104360 0x7fe534078560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:26:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe53942e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe534105490 0x7fe534076c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:26:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe53942e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe534105e30 0x7fe534077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:26:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe53942e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe53410fe20 con 0x7fe534104360 2024-05-27T09:26:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe53942e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe53410fca0 con 0x7fe534105e30 2024-05-27T09:26:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe53942e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe53410ffa0 con 0x7fe534105490 2024-05-27T09:26:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe534105e30 0x7fe534077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:26:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe534105e30 0x7fe534077150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53758/0 (socket says 172.21.15.50:53758) 2024-05-27T09:26:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe5337fe700 1 -- 172.21.15.50:0/1593766138 learned_addr learned my addr 172.21.15.50:0/1593766138 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-05-27T09:26:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe5327fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe534105490 0x7fe534076c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:26:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe5337fe700 1 -- 172.21.15.50:0/1593766138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe534105490 msgr2=0x7fe534076c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:26:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe5337fe700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe534105490 0x7fe534076c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe5337fe700 1 -- 172.21.15.50:0/1593766138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe534104360 msgr2=0x7fe534078560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:26:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe5337fe700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe534104360 0x7fe534078560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.534+0000 7fe5337fe700 1 -- 172.21.15.50:0/1593766138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5340778b0 con 0x7fe534105e30 2024-05-27T09:26:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.535+0000 7fe5337fe700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe534105e30 0x7fe534077150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe52c00b970 tx=0x7fe52c00bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:26:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.535+0000 7fe51bfff700 1 -- 172.21.15.50:0/1593766138 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c019070 con 0x7fe534105e30 2024-05-27T09:26:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.535+0000 7fe51bfff700 1 -- 172.21.15.50:0/1593766138 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe52c00cde0 con 0x7fe534105e30 2024-05-27T09:26:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.535+0000 7fe53942e700 1 -- 172.21.15.50:0/1593766138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5341a4a60 con 0x7fe534105e30 2024-05-27T09:26:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.535+0000 7fe51bfff700 1 -- 172.21.15.50:0/1593766138 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c007370 con 0x7fe534105e30 2024-05-27T09:26:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.535+0000 7fe53942e700 1 -- 172.21.15.50:0/1593766138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5341a4fc0 con 0x7fe534105e30 2024-05-27T09:26:24.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.536+0000 7fe53942e700 1 -- 172.21.15.50:0/1593766138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5340512b0 con 0x7fe534105e30 2024-05-27T09:26:24.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.537+0000 7fe51bfff700 1 -- 172.21.15.50:0/1593766138 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe52c01c070 con 0x7fe534105e30 2024-05-27T09:26:24.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.538+0000 7fe51bfff700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe51c0b2900 0x7fe51c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:26:24.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.538+0000 7fe51bfff700 1 -- 172.21.15.50:0/1593766138 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3669+0+0 (secure 0 0 0) 0x7fe52c0e0bc0 con 0x7fe534105e30 2024-05-27T09:26:24.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.539+0000 7fe532ffd700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe51c0b2900 0x7fe51c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:26:24.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.540+0000 7fe532ffd700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe51c0b2900 0x7fe51c0b4e10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe52400b5c0 tx=0x7fe524003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:26:24.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.542+0000 7fe51bfff700 1 -- 172.21.15.50:0/1593766138 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe52c023070 con 0x7fe534105e30 2024-05-27T09:26:24.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:24.766+0000 7fe53942e700 1 -- 172.21.15.50:0/1593766138 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe534063bf0 con 0x7fe51c0b2900 2024-05-27T09:26:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:26:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:26:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:26:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi044 2024-05-27T09:26:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:26.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:26.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:26.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:26.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi044 to 8547M 2024-05-27T09:26:26.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:26.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:26.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:26.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:26.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:26.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:26:26.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:26:26.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:26:26.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:26.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi044 2024-05-27T09:26:26.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:26.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:26.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:26.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:26.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi044 to 8547M 2024-05-27T09:26:26.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:26.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:26.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:26.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:26:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:26:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:26:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi044 2024-05-27T09:26:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:26.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:26.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:26.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:26.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi044 to 8547M 2024-05-27T09:26:26.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:26.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:26.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:26.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:28.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:27 smithi050 ceph-mon[36371]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:28.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:27 smithi050 ceph-mon[36371]: from='client.? 172.21.15.50:0/2312381387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88aa8a99-e01b-45d4-bac3-df51aaf8b94e"}]: dispatch 2024-05-27T09:26:28.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:27 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88aa8a99-e01b-45d4-bac3-df51aaf8b94e"}]: dispatch 2024-05-27T09:26:28.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:27 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88aa8a99-e01b-45d4-bac3-df51aaf8b94e"}]': finished 2024-05-27T09:26:28.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:27 smithi050 ceph-mon[36371]: osdmap e28: 5 total, 4 up, 5 in 2024-05-27T09:26:28.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:27 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:28.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:27 smithi189 ceph-mon[36671]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:27 smithi189 ceph-mon[36671]: from='client.? 172.21.15.50:0/2312381387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88aa8a99-e01b-45d4-bac3-df51aaf8b94e"}]: dispatch 2024-05-27T09:26:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:27 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88aa8a99-e01b-45d4-bac3-df51aaf8b94e"}]: dispatch 2024-05-27T09:26:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:27 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88aa8a99-e01b-45d4-bac3-df51aaf8b94e"}]': finished 2024-05-27T09:26:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:27 smithi189 ceph-mon[36671]: osdmap e28: 5 total, 4 up, 5 in 2024-05-27T09:26:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:27 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:27 smithi044 ceph-mon[31618]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:27 smithi044 ceph-mon[31618]: from='client.? 172.21.15.50:0/2312381387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88aa8a99-e01b-45d4-bac3-df51aaf8b94e"}]: dispatch 2024-05-27T09:26:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:27 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88aa8a99-e01b-45d4-bac3-df51aaf8b94e"}]: dispatch 2024-05-27T09:26:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:27 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88aa8a99-e01b-45d4-bac3-df51aaf8b94e"}]': finished 2024-05-27T09:26:28.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:27 smithi044 ceph-mon[31618]: osdmap e28: 5 total, 4 up, 5 in 2024-05-27T09:26:28.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:27 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:29.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:28 smithi189 ceph-mon[36671]: from='client.? 172.21.15.50:0/1023969787' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:26:29.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:28 smithi044 ceph-mon[31618]: from='client.? 172.21.15.50:0/1023969787' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:26:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:28 smithi050 ceph-mon[36371]: from='client.? 172.21.15.50:0/1023969787' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:26:30.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:29 smithi189 ceph-mon[36671]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-27T09:26:30.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:29 smithi044 ceph-mon[31618]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-27T09:26:30.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:29 smithi050 ceph-mon[36371]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-27T09:26:31.957 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:31 smithi050 ceph-mon[36371]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T09:26:32.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:31 smithi189 ceph-mon[36671]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T09:26:32.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:31 smithi044 ceph-mon[31618]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T09:26:33.934 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:33 smithi050 ceph-mon[36371]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T09:26:34.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:33 smithi189 ceph-mon[36671]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T09:26:34.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:33 smithi044 ceph-mon[31618]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T09:26:35.946 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:35 smithi050 ceph-mon[36371]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:26:35.946 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:35 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T09:26:35.946 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:35 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:36.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:35 smithi044 ceph-mon[31618]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:26:36.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:35 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T09:26:36.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:35 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:36.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:35 smithi189 ceph-mon[36671]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:26:36.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:35 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T09:26:36.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:35 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:37.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:36 smithi189 ceph-mon[36671]: Deploying daemon osd.4 on smithi050 2024-05-27T09:26:37.197 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:36 smithi050 ceph-mon[36371]: Deploying daemon osd.4 on smithi050 2024-05-27T09:26:37.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:36 smithi044 ceph-mon[31618]: Deploying daemon osd.4 on smithi050 2024-05-27T09:26:38.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:37 smithi050 ceph-mon[36371]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:26:38.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:37 smithi189 ceph-mon[36671]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:26:38.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:37 smithi044 ceph-mon[31618]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:26:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:39 smithi050 ceph-mon[36371]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:40.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:39 smithi189 ceph-mon[36671]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:40.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:39 smithi044 ceph-mon[31618]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:41.101 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:41.101 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:41.102 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:41.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:41.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:41.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:41.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:41.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:41.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:41.926 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:41 smithi050 ceph-mon[36371]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:42.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:41 smithi044 ceph-mon[31618]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:42.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:41 smithi189 ceph-mon[36671]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:43.929 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:43 smithi050 ceph-mon[36371]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:43.929 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:43.929 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:43.929 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:43.929 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:43.929 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:44.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:43 smithi044 ceph-mon[31618]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:44.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:44.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:44.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:44.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:44.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:44.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.309+0000 7fe51bfff700 1 -- 172.21.15.50:0/1593766138 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe534063bf0 con 0x7fe51c0b2900 2024-05-27T09:26:44.310 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 4 on host 'smithi050' 2024-05-27T09:26:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.311+0000 7fe53942e700 1 -- 172.21.15.50:0/1593766138 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe51c0b2900 msgr2=0x7fe51c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:26:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.311+0000 7fe53942e700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe51c0b2900 0x7fe51c0b4e10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe52400b5c0 tx=0x7fe524003860 comp rx=0 tx=0).stop 2024-05-27T09:26:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.311+0000 7fe53942e700 1 -- 172.21.15.50:0/1593766138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe534105e30 msgr2=0x7fe534077150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:26:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.311+0000 7fe53942e700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe534105e30 0x7fe534077150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe52c00b970 tx=0x7fe52c00bc80 comp rx=0 tx=0).stop 2024-05-27T09:26:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.312+0000 7fe53942e700 1 -- 172.21.15.50:0/1593766138 shutdown_connections 2024-05-27T09:26:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.312+0000 7fe53942e700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe51c0b2900 0x7fe51c0b4e10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.312+0000 7fe53942e700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe534105e30 0x7fe534077150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.312+0000 7fe53942e700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe534104360 0x7fe534078560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.312+0000 7fe53942e700 1 --2- 172.21.15.50:0/1593766138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe534105490 0x7fe534076c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:44.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.312+0000 7fe53942e700 1 -- 172.21.15.50:0/1593766138 >> 172.21.15.50:0/1593766138 conn(0x7fe5340ff9c0 msgr2=0x7fe5341086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:26:44.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.312+0000 7fe53942e700 1 -- 172.21.15.50:0/1593766138 shutdown_connections 2024-05-27T09:26:44.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:26:44.312+0000 7fe53942e700 1 -- 172.21.15.50:0/1593766138 wait complete. 2024-05-27T09:26:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:43 smithi189 ceph-mon[36671]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:44.919 DEBUG:teuthology.orchestra.run.smithi050:osd.4> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.4.service 2024-05-27T09:26:44.921 INFO:tasks.cephadm:Deploying osd.5 on smithi189 with /dev/vg_nvme/lv_3... 2024-05-27T09:26:44.922 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T09:26:45.166 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:26:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:45 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:45 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:45 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:45 smithi050 ceph-mon[36371]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:45 smithi050 ceph-mon[36371]: from='osd.4 [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T09:26:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:45 smithi050 ceph-mon[36371]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T09:26:45.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:45 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:45.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:45 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:45.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:45 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:45.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:45 smithi189 ceph-mon[36671]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:45.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:45 smithi189 ceph-mon[36671]: from='osd.4 [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T09:26:45.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:45 smithi189 ceph-mon[36671]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T09:26:45.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:45 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:45.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:45 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:26:45.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:45 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:45.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:45 smithi044 ceph-mon[31618]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:45.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:45 smithi044 ceph-mon[31618]: from='osd.4 [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T09:26:45.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:45 smithi044 ceph-mon[31618]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T09:26:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:46 smithi050 ceph-mon[36371]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T09:26:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:46 smithi050 ceph-mon[36371]: from='osd.4 [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:26:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:46 smithi050 ceph-mon[36371]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T09:26:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:46.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:46 smithi050 ceph-mon[36371]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:26:46.545 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:26:46 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:26:46.277+0000 7f247acf8700 -1 osd.4 0 waiting for initial osdmap 2024-05-27T09:26:46.545 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:26:46 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:26:46.284+0000 7f2476aad700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T09:26:46.584 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:46 smithi189 ceph-mon[36671]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T09:26:46.584 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:46 smithi189 ceph-mon[36671]: from='osd.4 [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:26:46.584 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:46 smithi189 ceph-mon[36671]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T09:26:46.584 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:46.584 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:46 smithi189 ceph-mon[36671]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:26:46.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:46 smithi044 ceph-mon[31618]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T09:26:46.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:46 smithi044 ceph-mon[31618]: from='osd.4 [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:26:46.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:46 smithi044 ceph-mon[31618]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T09:26:46.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:46.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:46 smithi044 ceph-mon[31618]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:26:47.482 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T09:26:47.482 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T09:26:47.482 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-05-27T09:26:47.482 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-05-27T09:26:47.482 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114366 s, 917 MB/s 2024-05-27T09:26:47.482 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-05-27T09:26:47.483 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:47 smithi050 ceph-mon[36371]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:26:47.483 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:47 smithi050 ceph-mon[36371]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T09:26:47.483 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:47 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:47.483 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:47 smithi050 ceph-mon[36371]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:47.483 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:47 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:47.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:47 smithi189 ceph-mon[36671]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:26:47.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:47 smithi189 ceph-mon[36671]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T09:26:47.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:47 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:47.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:47 smithi189 ceph-mon[36671]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:47.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:47 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:47.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:47 smithi044 ceph-mon[31618]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:26:47.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:47 smithi044 ceph-mon[31618]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T09:26:47.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:47.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:47 smithi044 ceph-mon[31618]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T09:26:47.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:48.112 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_3 2024-05-27T09:26:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: osd.4 [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660] boot 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: osdmap e31: 5 total, 5 up, 5 in 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:48.351 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:26:48.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:26:48.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:26:48.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: osd.4 [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660] boot 2024-05-27T09:26:48.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: osdmap e31: 5 total, 5 up, 5 in 2024-05-27T09:26:48.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:48.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:48.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:48.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:48.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:48.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:48.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:48.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:48.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:48.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:26:48.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:26:48.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: osd.4 [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660] boot 2024-05-27T09:26:48.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: osdmap e31: 5 total, 5 up, 5 in 2024-05-27T09:26:48.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:26:48.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:48.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:48.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:48.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:26:48.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:48.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:48.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:26:48.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:26:49.458 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:49 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi050 2024-05-27T09:26:49.459 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:49 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi050 to 8546M 2024-05-27T09:26:49.459 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:49 smithi189 ceph-mon[36671]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T09:26:49.459 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:49 smithi189 ceph-mon[36671]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:49.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:49 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi050 2024-05-27T09:26:49.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:49 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi050 to 8546M 2024-05-27T09:26:49.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:49 smithi044 ceph-mon[31618]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T09:26:49.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:49 smithi044 ceph-mon[31618]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:49.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:49 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi050 2024-05-27T09:26:49.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:49 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi050 to 8546M 2024-05-27T09:26:49.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:49 smithi050 ceph-mon[36371]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T09:26:49.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:49 smithi050 ceph-mon[36371]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:49.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.968+0000 7f0c42a88700 1 -- 172.21.15.189:0/588220115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c103f50 msgr2=0x7f0c3c104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:26:49.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.968+0000 7f0c42a88700 1 --2- 172.21.15.189:0/588220115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c103f50 0x7f0c3c104360 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009990 tx=0x7f0c28009ca0 comp rx=0 tx=0).stop 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.968+0000 7f0c42a88700 1 -- 172.21.15.189:0/588220115 shutdown_connections 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.968+0000 7f0c42a88700 1 --2- 172.21.15.189:0/588220115 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c3c1049d0 0x7f0c3c10c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.968+0000 7f0c42a88700 1 --2- 172.21.15.189:0/588220115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c3c1059a0 0x7f0c3c105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.968+0000 7f0c42a88700 1 --2- 172.21.15.189:0/588220115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c103f50 0x7f0c3c104360 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.968+0000 7f0c42a88700 1 -- 172.21.15.189:0/588220115 >> 172.21.15.189:0/588220115 conn(0x7f0c3c0ff9c0 msgr2=0x7f0c3c101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.968+0000 7f0c42a88700 1 -- 172.21.15.189:0/588220115 shutdown_connections 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.968+0000 7f0c42a88700 1 -- 172.21.15.189:0/588220115 wait complete. 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c42a88700 1 Processor -- start 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c42a88700 1 -- start start 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c42a88700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c3c103f50 0x7f0c3c19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c42a88700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c1049d0 0x7f0c3c19afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:26:49.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c42a88700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c3c1059a0 0x7f0c3c1a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c42a88700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c10ffc0 con 0x7f0c3c103f50 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c42a88700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c10fe40 con 0x7f0c3c1059a0 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c42a88700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c110140 con 0x7f0c3c1049d0 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c3bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c1049d0 0x7f0c3c19afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c3bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c1049d0 0x7f0c3c19afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44128/0 (socket says 172.21.15.189:44128) 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c3bfff700 1 -- 172.21.15.189:0/2109994096 learned_addr learned my addr 172.21.15.189:0/2109994096 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c41025700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c3c1059a0 0x7f0c3c1a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c3bfff700 1 -- 172.21.15.189:0/2109994096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c3c1059a0 msgr2=0x7f0c3c1a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c3bfff700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c3c1059a0 0x7f0c3c1a1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c3bfff700 1 -- 172.21.15.189:0/2109994096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c3c103f50 msgr2=0x7f0c3c19aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c3bfff700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c3c103f50 0x7f0c3c19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:26:49.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.969+0000 7f0c3bfff700 1 -- 172.21.15.189:0/2109994096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c3c1a25d0 con 0x7f0c3c1049d0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.970+0000 7f0c3bfff700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c1049d0 0x7f0c3c19afb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0c28006010 tx=0x7f0c28009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.970+0000 7f0c39ffb700 1 -- 172.21.15.189:0/2109994096 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c28027070 con 0x7f0c3c1049d0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.970+0000 7f0c39ffb700 1 -- 172.21.15.189:0/2109994096 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c28022470 con 0x7f0c3c1049d0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.970+0000 7f0c42a88700 1 -- 172.21.15.189:0/2109994096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c3c1a2860 con 0x7f0c3c1049d0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.970+0000 7f0c39ffb700 1 -- 172.21.15.189:0/2109994096 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2800f6c0 con 0x7f0c3c1049d0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.970+0000 7f0c42a88700 1 -- 172.21.15.189:0/2109994096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c3c1a2d30 con 0x7f0c3c1049d0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.971+0000 7f0c42a88700 1 -- 172.21.15.189:0/2109994096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c3c0512b0 con 0x7f0c3c1049d0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.972+0000 7f0c39ffb700 1 -- 172.21.15.189:0/2109994096 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f0c2802a070 con 0x7f0c3c1049d0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.973+0000 7f0c39ffb700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0c2c0b2880 0x7f0c2c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.973+0000 7f0c39ffb700 1 -- 172.21.15.189:0/2109994096 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4101+0+0 (secure 0 0 0) 0x7f0c280e2620 con 0x7f0c3c1049d0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.974+0000 7f0c40824700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0c2c0b2880 0x7f0c2c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.975+0000 7f0c40824700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0c2c0b2880 0x7f0c2c0b4d90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c105390 tx=0x7f0c30005ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:26:49.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:49.978+0000 7f0c39ffb700 1 -- 172.21.15.189:0/2109994096 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0c28013070 con 0x7f0c3c1049d0 2024-05-27T09:26:50.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:26:50.187+0000 7f0c42a88700 1 -- 172.21.15.189:0/2109994096 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0c3c063bf0 con 0x7f0c2c0b2880 2024-05-27T09:26:50.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:50 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:26:50.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:50 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:26:50.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:50 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:50.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:50 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:26:50.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:50 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:26:50.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:50 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:50.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:50 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:26:50.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:50 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:26:50.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:50 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:26:51.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:51 smithi189 ceph-mon[36671]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:26:51.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:51 smithi189 ceph-mon[36671]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:51.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:51 smithi044 ceph-mon[31618]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:26:51.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:51 smithi044 ceph-mon[31618]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:51.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:51 smithi050 ceph-mon[36371]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:26:51.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:51 smithi050 ceph-mon[36371]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:53 smithi044 ceph-mon[31618]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:53 smithi044 ceph-mon[31618]: from='client.? 172.21.15.189:0/4014426570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b075f197-da86-4cca-b088-d6912121fbad"}]: dispatch 2024-05-27T09:26:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:53 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b075f197-da86-4cca-b088-d6912121fbad"}]: dispatch 2024-05-27T09:26:53.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:53 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b075f197-da86-4cca-b088-d6912121fbad"}]': finished 2024-05-27T09:26:53.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:53 smithi044 ceph-mon[31618]: osdmap e33: 6 total, 5 up, 6 in 2024-05-27T09:26:53.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:53 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:26:53.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:53 smithi044 ceph-mon[31618]: from='client.? 172.21.15.189:0/2051562279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:26:54.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:53 smithi050 ceph-mon[36371]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:54.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:53 smithi050 ceph-mon[36371]: from='client.? 172.21.15.189:0/4014426570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b075f197-da86-4cca-b088-d6912121fbad"}]: dispatch 2024-05-27T09:26:54.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:53 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b075f197-da86-4cca-b088-d6912121fbad"}]: dispatch 2024-05-27T09:26:54.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:53 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b075f197-da86-4cca-b088-d6912121fbad"}]': finished 2024-05-27T09:26:54.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:53 smithi050 ceph-mon[36371]: osdmap e33: 6 total, 5 up, 6 in 2024-05-27T09:26:54.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:53 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:26:54.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:53 smithi050 ceph-mon[36371]: from='client.? 172.21.15.189:0/2051562279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:26:54.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:53 smithi189 ceph-mon[36671]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:53 smithi189 ceph-mon[36671]: from='client.? 172.21.15.189:0/4014426570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b075f197-da86-4cca-b088-d6912121fbad"}]: dispatch 2024-05-27T09:26:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:53 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b075f197-da86-4cca-b088-d6912121fbad"}]: dispatch 2024-05-27T09:26:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:53 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b075f197-da86-4cca-b088-d6912121fbad"}]': finished 2024-05-27T09:26:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:53 smithi189 ceph-mon[36671]: osdmap e33: 6 total, 5 up, 6 in 2024-05-27T09:26:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:53 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:26:54.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:53 smithi189 ceph-mon[36671]: from='client.? 172.21.15.189:0/2051562279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:26:55.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:55 smithi044 ceph-mon[31618]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:56.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:55 smithi050 ceph-mon[36371]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:55 smithi189 ceph-mon[36671]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:57.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:57 smithi189 ceph-mon[36671]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:57.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:57 smithi044 ceph-mon[31618]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:58.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:57 smithi050 ceph-mon[36371]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:59.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:26:59 smithi189 ceph-mon[36671]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:26:59.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:26:59 smithi044 ceph-mon[31618]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:26:59 smithi050 ceph-mon[36371]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:00.879 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:00 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T09:27:00.879 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:00 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:00.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:00 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T09:27:00.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:00 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:01.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:00 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T09:27:01.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:00 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:01.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:01 smithi189 ceph-mon[36671]: Deploying daemon osd.5 on smithi189 2024-05-27T09:27:01.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:01 smithi189 ceph-mon[36671]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:01.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:01 smithi050 ceph-mon[36371]: Deploying daemon osd.5 on smithi189 2024-05-27T09:27:01.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:01 smithi050 ceph-mon[36371]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:01.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:01 smithi044 ceph-mon[31618]: Deploying daemon osd.5 on smithi189 2024-05-27T09:27:01.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:01 smithi044 ceph-mon[31618]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:03.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:03 smithi189 ceph-mon[36671]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:03.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:03 smithi044 ceph-mon[31618]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:04.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:03 smithi050 ceph-mon[36371]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:05.921 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:05 smithi189 ceph-mon[36671]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:05.921 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:05 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:05.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:05 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:05.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:05 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:05.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:05 smithi044 ceph-mon[31618]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:05.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:05 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:05.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:05 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:05.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:05 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:06.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:05 smithi050 ceph-mon[36371]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:06.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:05 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:06.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:05 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:06.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:05 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:07.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:07 smithi189 ceph-mon[36671]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:07.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:07 smithi044 ceph-mon[31618]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:08.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:07 smithi050 ceph-mon[36371]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:09.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.124+0000 7f0c39ffb700 1 -- 172.21.15.189:0/2109994096 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0c3c063bf0 con 0x7f0c2c0b2880 2024-05-27T09:27:09.125 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 5 on host 'smithi189' 2024-05-27T09:27:09.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 -- 172.21.15.189:0/2109994096 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0c2c0b2880 msgr2=0x7f0c2c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:27:09.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0c2c0b2880 0x7f0c2c0b4d90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c105390 tx=0x7f0c30005ca0 comp rx=0 tx=0).stop 2024-05-27T09:27:09.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 -- 172.21.15.189:0/2109994096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c1049d0 msgr2=0x7f0c3c19afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:27:09.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c1049d0 0x7f0c3c19afb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0c28006010 tx=0x7f0c28009f90 comp rx=0 tx=0).stop 2024-05-27T09:27:09.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 -- 172.21.15.189:0/2109994096 shutdown_connections 2024-05-27T09:27:09.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0c2c0b2880 0x7f0c2c0b4d90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:09.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c3c1059a0 0x7f0c3c1a1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:09.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c3c103f50 0x7f0c3c19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:09.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 --2- 172.21.15.189:0/2109994096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c1049d0 0x7f0c3c19afb0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:09.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 -- 172.21.15.189:0/2109994096 >> 172.21.15.189:0/2109994096 conn(0x7f0c3c0ff9c0 msgr2=0x7f0c3c1086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:27:09.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 -- 172.21.15.189:0/2109994096 shutdown_connections 2024-05-27T09:27:09.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:27:09.127+0000 7f0c42a88700 1 -- 172.21.15.189:0/2109994096 wait complete. 2024-05-27T09:27:09.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:09.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:09.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:09 smithi189 ceph-mon[36671]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:09.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:09.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:09.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:09.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:09 smithi044 ceph-mon[31618]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:09.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:09 smithi050 ceph-mon[36371]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:09.651 DEBUG:teuthology.orchestra.run.smithi189:osd.5> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.5.service 2024-05-27T09:27:09.653 INFO:tasks.cephadm:Deploying osd.6 on smithi044 with /dev/vg_nvme/lv_2... 2024-05-27T09:27:09.654 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T09:27:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:27:10.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:10 smithi044 ceph-mon[31618]: from='osd.5 [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T09:27:10.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:10 smithi044 ceph-mon[31618]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T09:27:10.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:10 smithi050 ceph-mon[36371]: from='osd.5 [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T09:27:10.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:10 smithi050 ceph-mon[36371]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T09:27:10.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:10 smithi189 ceph-mon[36671]: from='osd.5 [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T09:27:10.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:10 smithi189 ceph-mon[36671]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T09:27:11.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:11 smithi044 ceph-mon[31618]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T09:27:11.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:11 smithi044 ceph-mon[31618]: from='osd.5 [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:27:11.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:11 smithi044 ceph-mon[31618]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T09:27:11.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:11 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:27:11.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:11 smithi044 ceph-mon[31618]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:27:11.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:11 smithi044 ceph-mon[31618]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:11.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:11 smithi050 ceph-mon[36371]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T09:27:11.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:11 smithi050 ceph-mon[36371]: from='osd.5 [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:27:11.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:11 smithi050 ceph-mon[36371]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T09:27:11.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:11 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:27:11.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:11 smithi050 ceph-mon[36371]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:27:11.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:11 smithi050 ceph-mon[36371]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:11 smithi189 ceph-mon[36671]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T09:27:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:11 smithi189 ceph-mon[36671]: from='osd.5 [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:27:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:11 smithi189 ceph-mon[36671]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T09:27:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:11 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:27:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:11 smithi189 ceph-mon[36671]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:27:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:11 smithi189 ceph-mon[36671]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-05-27T09:27:12.226 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:27:12 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:27:12.119+0000 7f391bc1d700 -1 osd.5 0 waiting for initial osdmap 2024-05-27T09:27:12.227 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:27:12 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:27:12.181+0000 7f39139ca700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T09:27:12.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:12 smithi044 ceph-mon[31618]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:27:12.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:12 smithi044 ceph-mon[31618]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T09:27:12.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:12 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:27:12.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:12 smithi044 ceph-mon[31618]: from='osd.5 ' entity='osd.5' 2024-05-27T09:27:12.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:12 smithi044 ceph-mon[31618]: osd.5 [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881] boot 2024-05-27T09:27:12.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:12 smithi044 ceph-mon[31618]: osdmap e36: 6 total, 6 up, 6 in 2024-05-27T09:27:12.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:12 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:27:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:12 smithi050 ceph-mon[36371]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:27:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:12 smithi050 ceph-mon[36371]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T09:27:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:12 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:27:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:12 smithi050 ceph-mon[36371]: from='osd.5 ' entity='osd.5' 2024-05-27T09:27:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:12 smithi050 ceph-mon[36371]: osd.5 [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881] boot 2024-05-27T09:27:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:12 smithi050 ceph-mon[36371]: osdmap e36: 6 total, 6 up, 6 in 2024-05-27T09:27:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:12 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:27:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:12 smithi189 ceph-mon[36671]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:27:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:12 smithi189 ceph-mon[36671]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T09:27:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:12 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:27:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:12 smithi189 ceph-mon[36671]: from='osd.5 ' entity='osd.5' 2024-05-27T09:27:12.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:12 smithi189 ceph-mon[36671]: osd.5 [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881] boot 2024-05-27T09:27:12.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:12 smithi189 ceph-mon[36671]: osdmap e36: 6 total, 6 up, 6 in 2024-05-27T09:27:12.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:12 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:27:13.038 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T09:27:13.038 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T09:27:13.038 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-05-27T09:27:13.038 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-05-27T09:27:13.038 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0345235 s, 304 MB/s 2024-05-27T09:27:13.038 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-05-27T09:27:13.466 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:27:13.466 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:27:13.466 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T09:27:13.466 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 538 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:13.466 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi189 2024-05-27T09:27:13.467 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.467 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.467 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:13.467 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:13.467 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi189 to 8544M 2024-05-27T09:27:13.467 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.467 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:13.467 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:13.467 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:27:13.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 538 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi189 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi189 to 8544M 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.678 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:27:13.678 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:27:13.678 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T09:27:13.678 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 538 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:13.678 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi189 2024-05-27T09:27:13.678 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.678 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:13.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:13.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi189 to 8544M 2024-05-27T09:27:13.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:13.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:13.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:13.962 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_2 2024-05-27T09:27:14.418 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:27:14.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:14 smithi050 ceph-mon[36371]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T09:27:14.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:14 smithi044 ceph-mon[31618]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T09:27:14.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:14 smithi189 ceph-mon[36671]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T09:27:15.693 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:15 smithi044 ceph-mon[31618]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 538 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:15.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:15 smithi050 ceph-mon[36371]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 538 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:15.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:15 smithi189 ceph-mon[36671]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 538 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.612+0000 7f0542400700 1 -- 172.21.15.44:0/1036988885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f053c104330 msgr2=0x7f053c104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:27:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.612+0000 7f0542400700 1 --2- 172.21.15.44:0/1036988885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f053c104330 0x7f053c104740 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f052c009990 tx=0x7f052c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:27:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.612+0000 7f0542400700 1 -- 172.21.15.44:0/1036988885 shutdown_connections 2024-05-27T09:27:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.612+0000 7f0542400700 1 --2- 172.21.15.44:0/1036988885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f053c105e00 0x7f053c10c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.612+0000 7f0542400700 1 --2- 172.21.15.44:0/1036988885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f053c104330 0x7f053c104740 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.612+0000 7f0542400700 1 --2- 172.21.15.44:0/1036988885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f053c105460 0x7f053c105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:16.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.612+0000 7f0542400700 1 -- 172.21.15.44:0/1036988885 >> 172.21.15.44:0/1036988885 conn(0x7f053c0ff9b0 msgr2=0x7f053c101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:27:16.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.612+0000 7f0542400700 1 -- 172.21.15.44:0/1036988885 shutdown_connections 2024-05-27T09:27:16.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.612+0000 7f0542400700 1 -- 172.21.15.44:0/1036988885 wait complete. 2024-05-27T09:27:16.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.614+0000 7f0542400700 1 Processor -- start 2024-05-27T09:27:16.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.614+0000 7f0542400700 1 -- start start 2024-05-27T09:27:16.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.614+0000 7f0542400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f053c104330 0x7f053c19cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:27:16.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.614+0000 7f0542400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f053c105460 0x7f053c19d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:27:16.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f0542400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f053c105e00 0x7f053c1a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:27:16.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f0542400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f053c10fd80 con 0x7f053c105460 2024-05-27T09:27:16.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f0542400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f053c10fc00 con 0x7f053c105e00 2024-05-27T09:27:16.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f0542400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f053c10ff00 con 0x7f053c104330 2024-05-27T09:27:16.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f053b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f053c104330 0x7f053c19cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:27:16.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f0532a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f053c105460 0x7f053c19d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:27:16.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f053b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f053c104330 0x7f053c19cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59466/0 (socket says 172.21.15.44:59466) 2024-05-27T09:27:16.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f053b400700 1 -- 172.21.15.44:0/1734067141 learned_addr learned my addr 172.21.15.44:0/1734067141 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:27:16.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f053be00700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f053c105e00 0x7f053c1a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:27:16.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f053b400700 1 -- 172.21.15.44:0/1734067141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f053c105e00 msgr2=0x7f053c1a4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:27:16.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f053b400700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f053c105e00 0x7f053c1a4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:16.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f053b400700 1 -- 172.21.15.44:0/1734067141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f053c105460 msgr2=0x7f053c19d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:27:16.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f053b400700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f053c105460 0x7f053c19d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:16.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f053b400700 1 -- 172.21.15.44:0/1734067141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c1a4690 con 0x7f053c104330 2024-05-27T09:27:16.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.615+0000 7f0532a00700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f053c105460 0x7f053c19d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:27:16.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.616+0000 7f053b400700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f053c104330 0x7f053c19cb30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f052c00b6c0 tx=0x7f052c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:27:16.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.616+0000 7f0538c00700 1 -- 172.21.15.44:0/1734067141 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c026070 con 0x7f053c104330 2024-05-27T09:27:16.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.616+0000 7f0542400700 1 -- 172.21.15.44:0/1734067141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f053c1a4920 con 0x7f053c104330 2024-05-27T09:27:16.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.616+0000 7f0538c00700 1 -- 172.21.15.44:0/1734067141 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f052c021410 con 0x7f053c104330 2024-05-27T09:27:16.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.616+0000 7f0538c00700 1 -- 172.21.15.44:0/1734067141 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c017650 con 0x7f053c104330 2024-05-27T09:27:16.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.616+0000 7f0542400700 1 -- 172.21.15.44:0/1734067141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f053c1a4dd0 con 0x7f053c104330 2024-05-27T09:27:16.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.618+0000 7f0542400700 1 -- 172.21.15.44:0/1734067141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f053c0512b0 con 0x7f053c104330 2024-05-27T09:27:16.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.620+0000 7f0538c00700 1 -- 172.21.15.44:0/1734067141 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f052c029070 con 0x7f053c104330 2024-05-27T09:27:16.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.622+0000 7f0538c00700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f05200b2880 0x7f05200b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:27:16.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.622+0000 7f0538c00700 1 -- 172.21.15.44:0/1734067141 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4533+0+0 (secure 0 0 0) 0x7f052c0e26b0 con 0x7f053c104330 2024-05-27T09:27:16.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.622+0000 7f0532a00700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f05200b2880 0x7f05200b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:27:16.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.623+0000 7f0532a00700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f05200b2880 0x7f05200b4d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f053c19dbe0 tx=0x7f0524009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:27:16.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.624+0000 7f0538c00700 1 -- 172.21.15.44:0/1734067141 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f052c013070 con 0x7f053c104330 2024-05-27T09:27:16.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:16.870+0000 7f0542400700 1 -- 172.21.15.44:0/1734067141 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f053c077100 con 0x7f05200b2880 2024-05-27T09:27:17.868 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:17 smithi044 ceph-mon[31618]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 537 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:17.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:17 smithi044 ceph-mon[31618]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:27:17.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:27:17.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:27:17.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:17 smithi050 ceph-mon[36371]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 537 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:17 smithi050 ceph-mon[36371]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:27:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:27:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:27:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:18.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:17 smithi189 ceph-mon[36671]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 537 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:18.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:17 smithi189 ceph-mon[36671]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:27:18.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:27:18.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:27:18.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:19.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:19 smithi044 ceph-mon[31618]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-05-27T09:27:20.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:19 smithi050 ceph-mon[36371]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-05-27T09:27:20.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:19 smithi189 ceph-mon[36671]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-05-27T09:27:21.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:21 smithi044 ceph-mon[31618]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-27T09:27:21.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:21 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1306498502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24e70958-59a8-415b-95f9-1b52a7dd8263"}]: dispatch 2024-05-27T09:27:21.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:21 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1306498502' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24e70958-59a8-415b-95f9-1b52a7dd8263"}]': finished 2024-05-27T09:27:21.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:21 smithi044 ceph-mon[31618]: osdmap e39: 7 total, 6 up, 7 in 2024-05-27T09:27:21.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:21 smithi050 ceph-mon[36371]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-27T09:27:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:21 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1306498502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24e70958-59a8-415b-95f9-1b52a7dd8263"}]: dispatch 2024-05-27T09:27:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:21 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1306498502' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24e70958-59a8-415b-95f9-1b52a7dd8263"}]': finished 2024-05-27T09:27:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:21 smithi050 ceph-mon[36371]: osdmap e39: 7 total, 6 up, 7 in 2024-05-27T09:27:22.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:21 smithi189 ceph-mon[36671]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-27T09:27:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:21 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1306498502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24e70958-59a8-415b-95f9-1b52a7dd8263"}]: dispatch 2024-05-27T09:27:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:21 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1306498502' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24e70958-59a8-415b-95f9-1b52a7dd8263"}]': finished 2024-05-27T09:27:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:21 smithi189 ceph-mon[36671]: osdmap e39: 7 total, 6 up, 7 in 2024-05-27T09:27:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:22.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:22 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2846670081' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:27:23.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:22 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2846670081' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:27:23.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:22 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2846670081' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:27:23.968 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:23 smithi044 ceph-mon[31618]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-27T09:27:24.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:23 smithi050 ceph-mon[36371]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-27T09:27:24.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:23 smithi189 ceph-mon[36671]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-27T09:27:25.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:25 smithi044 ceph-mon[31618]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:27:26.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:25 smithi050 ceph-mon[36371]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:27:26.090 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:25 smithi189 ceph-mon[36671]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:27:27.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:27 smithi044 ceph-mon[31618]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:27:28.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:27 smithi050 ceph-mon[36371]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:27:28.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:27 smithi189 ceph-mon[36671]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:27:29.949 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:29 smithi044 ceph-mon[31618]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:30.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:29 smithi050 ceph-mon[36371]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:30.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:29 smithi189 ceph-mon[36671]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:31.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:31 smithi044 ceph-mon[31618]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:31.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T09:27:31.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:31 smithi050 ceph-mon[36371]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:31 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T09:27:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:31 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:32.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:31 smithi189 ceph-mon[36671]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:32.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:31 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T09:27:32.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:31 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:32.810 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:32 smithi044 ceph-mon[31618]: Deploying daemon osd.6 on smithi044 2024-05-27T09:27:33.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:32 smithi050 ceph-mon[36371]: Deploying daemon osd.6 on smithi044 2024-05-27T09:27:33.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:32 smithi189 ceph-mon[36671]: Deploying daemon osd.6 on smithi044 2024-05-27T09:27:33.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:33 smithi044 ceph-mon[31618]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:34.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:33 smithi050 ceph-mon[36371]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:34.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:33 smithi189 ceph-mon[36671]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:35.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:35 smithi044 ceph-mon[31618]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:36.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:35 smithi050 ceph-mon[36371]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:36.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:35 smithi189 ceph-mon[36671]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:37.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:37 smithi044 ceph-mon[31618]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:38.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:37 smithi050 ceph-mon[36371]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:38.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:37 smithi189 ceph-mon[36671]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:39.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:39 smithi044 ceph-mon[31618]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:40.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:39 smithi050 ceph-mon[36371]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:40.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:39 smithi189 ceph-mon[36671]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:42.126 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:41 smithi044 ceph-mon[31618]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:42.126 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:42.126 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:42.126 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:42.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:41 smithi050 ceph-mon[36371]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:42.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:42.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:42.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:42.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:41 smithi189 ceph-mon[36671]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:42.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:42.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:42.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:43 smithi044 ceph-mon[31618]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:44.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:43 smithi050 ceph-mon[36371]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:43 smithi189 ceph-mon[36671]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:46.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:45 smithi050 ceph-mon[36371]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:46.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:45 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:46.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:45 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:46.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:45 smithi050 ceph-mon[36371]: from='osd.6 [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T09:27:46.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:45 smithi050 ceph-mon[36371]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T09:27:46.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:45 smithi189 ceph-mon[36671]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:46.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:45 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:46.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:45 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:46.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:45 smithi189 ceph-mon[36671]: from='osd.6 [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T09:27:46.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:45 smithi189 ceph-mon[36671]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T09:27:46.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:45 smithi044 ceph-mon[31618]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:46.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:45 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:46.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:45 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:46.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:45 smithi044 ceph-mon[31618]: from='osd.6 [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T09:27:46.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:45 smithi044 ceph-mon[31618]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T09:27:46.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.400+0000 7f0538c00700 1 -- 172.21.15.44:0/1734067141 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f053c077100 con 0x7f05200b2880 2024-05-27T09:27:46.403 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 6 on host 'smithi044' 2024-05-27T09:27:46.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.405+0000 7f0542400700 1 -- 172.21.15.44:0/1734067141 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f05200b2880 msgr2=0x7f05200b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:27:46.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.405+0000 7f0542400700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f05200b2880 0x7f05200b4d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f053c19dbe0 tx=0x7f0524009260 comp rx=0 tx=0).stop 2024-05-27T09:27:46.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.405+0000 7f0542400700 1 -- 172.21.15.44:0/1734067141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f053c104330 msgr2=0x7f053c19cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:27:46.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.405+0000 7f0542400700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f053c104330 0x7f053c19cb30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f052c00b6c0 tx=0x7f052c009580 comp rx=0 tx=0).stop 2024-05-27T09:27:46.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.405+0000 7f0542400700 1 -- 172.21.15.44:0/1734067141 shutdown_connections 2024-05-27T09:27:46.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.405+0000 7f0542400700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f05200b2880 0x7f05200b4d90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:46.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.405+0000 7f0542400700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f053c105e00 0x7f053c1a4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:46.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.405+0000 7f0542400700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f053c104330 0x7f053c19cb30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:46.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.405+0000 7f0542400700 1 --2- 172.21.15.44:0/1734067141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f053c105460 0x7f053c19d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:46.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.405+0000 7f0542400700 1 -- 172.21.15.44:0/1734067141 >> 172.21.15.44:0/1734067141 conn(0x7f053c0ff9b0 msgr2=0x7f053c0759e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:27:46.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.405+0000 7f0542400700 1 -- 172.21.15.44:0/1734067141 shutdown_connections 2024-05-27T09:27:46.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:27:46.406+0000 7f0542400700 1 -- 172.21.15.44:0/1734067141 wait complete. 2024-05-27T09:27:47.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:46 smithi044 ceph-mon[31618]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T09:27:47.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:46 smithi044 ceph-mon[31618]: from='osd.6 [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:27:47.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:46 smithi044 ceph-mon[31618]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T09:27:47.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:47.116 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:46 smithi044 ceph-mon[31618]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:27:47.116 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:47.116 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:47.116 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:47.116 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:47.116 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:47.116 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:47.179 DEBUG:teuthology.orchestra.run.smithi044:osd.6> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.6.service 2024-05-27T09:27:47.184 INFO:tasks.cephadm:Deploying osd.7 on smithi050 with /dev/vg_nvme/lv_2... 2024-05-27T09:27:47.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T09:27:47.211 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:46 smithi050 ceph-mon[36371]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T09:27:47.212 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:46 smithi050 ceph-mon[36371]: from='osd.6 [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:27:47.212 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:46 smithi050 ceph-mon[36371]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T09:27:47.212 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:47.212 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:46 smithi050 ceph-mon[36371]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:27:47.212 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:47.212 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:47.212 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:47.212 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:47.212 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:47.212 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:47.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:46 smithi189 ceph-mon[36671]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T09:27:47.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:46 smithi189 ceph-mon[36671]: from='osd.6 [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:27:47.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:46 smithi189 ceph-mon[36671]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T09:27:47.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:47.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:46 smithi189 ceph-mon[36671]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:27:47.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:47.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:47.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:47.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:47.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:47.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:27:47.439 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:27:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:47 smithi044 ceph-mon[31618]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:47 smithi044 ceph-mon[31618]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:27:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:47 smithi044 ceph-mon[31618]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T09:27:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:47 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:48.225 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:27:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:27:47.931+0000 7f3283800700 -1 osd.6 0 waiting for initial osdmap 2024-05-27T09:27:48.225 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:27:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:27:47.959+0000 7f3279600700 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T09:27:48.278 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:47 smithi050 ceph-mon[36371]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:48.278 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:47 smithi050 ceph-mon[36371]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:27:48.278 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:47 smithi050 ceph-mon[36371]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T09:27:48.278 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:47 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:48.278 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:47 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:47 smithi189 ceph-mon[36671]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:47 smithi189 ceph-mon[36671]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:27:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:47 smithi189 ceph-mon[36671]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T09:27:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:47 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:48.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:47 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:48.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:48 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:27:48.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:48 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:27:48.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:48.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:48 smithi044 ceph-mon[31618]: from='osd.6 ' entity='osd.6' 2024-05-27T09:27:49.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:48 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:27:49.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:48 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:27:49.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:49.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:48 smithi050 ceph-mon[36371]: from='osd.6 ' entity='osd.6' 2024-05-27T09:27:49.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:48 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:27:49.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:48 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:27:49.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:49.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:48 smithi189 ceph-mon[36671]: from='osd.6 ' entity='osd.6' 2024-05-27T09:27:49.424 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T09:27:49.425 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T09:27:49.425 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-05-27T09:27:49.425 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-05-27T09:27:49.425 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00848608 s, 1.2 GB/s 2024-05-27T09:27:49.425 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 2024-05-27T09:27:49.425 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-05-27T09:27:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:49 smithi050 ceph-mon[36371]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:49 smithi050 ceph-mon[36371]: osd.6 [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643] boot 2024-05-27T09:27:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:49 smithi050 ceph-mon[36371]: osdmap e42: 7 total, 7 up, 7 in 2024-05-27T09:27:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:50.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_2 2024-05-27T09:27:50.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:49 smithi044 ceph-mon[31618]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:50.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:50.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:49 smithi044 ceph-mon[31618]: osd.6 [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643] boot 2024-05-27T09:27:50.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:49 smithi044 ceph-mon[31618]: osdmap e42: 7 total, 7 up, 7 in 2024-05-27T09:27:50.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:50.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:49 smithi189 ceph-mon[36671]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T09:27:50.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:50.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:49 smithi189 ceph-mon[36671]: osd.6 [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643] boot 2024-05-27T09:27:50.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:49 smithi189 ceph-mon[36671]: osdmap e42: 7 total, 7 up, 7 in 2024-05-27T09:27:50.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:27:50.385 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:27:51.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:50 smithi050 ceph-mon[36371]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T09:27:51.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:50 smithi050 ceph-mon[36371]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:51.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:50 smithi044 ceph-mon[31618]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T09:27:51.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:50 smithi044 ceph-mon[31618]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:51.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:50 smithi189 ceph-mon[36671]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T09:27:51.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:50 smithi189 ceph-mon[36671]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:51.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.946+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/3127239301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18107780 msgr2=0x7f5b18109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:27:51.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.946+0000 7f5b1dc0d700 1 --2- 172.21.15.50:0/3127239301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18107780 0x7f5b18109ba0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c009990 tx=0x7f5b0c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:27:51.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.946+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/3127239301 shutdown_connections 2024-05-27T09:27:51.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.946+0000 7f5b1dc0d700 1 --2- 172.21.15.50:0/3127239301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b1810a0e0 0x7f5b1810c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:51.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.946+0000 7f5b1dc0d700 1 --2- 172.21.15.50:0/3127239301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b18104e20 0x7f5b18107240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:51.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.946+0000 7f5b1dc0d700 1 --2- 172.21.15.50:0/3127239301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18107780 0x7f5b18109ba0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:51.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.946+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/3127239301 >> 172.21.15.50:0/3127239301 conn(0x7f5b180fe860 msgr2=0x7f5b18100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.946+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/3127239301 shutdown_connections 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.946+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/3127239301 wait complete. 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.947+0000 7f5b1dc0d700 1 Processor -- start 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.947+0000 7f5b1dc0d700 1 -- start start 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.947+0000 7f5b1dc0d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b18104e20 0x7f5b1819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.947+0000 7f5b1dc0d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18107780 0x7f5b1819ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.947+0000 7f5b1dc0d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b1810a0e0 0x7f5b181a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.947+0000 7f5b1dc0d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1810fe20 con 0x7f5b1810a0e0 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.947+0000 7f5b1dc0d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1810fca0 con 0x7f5b18107780 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.947+0000 7f5b1dc0d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1810ffa0 con 0x7f5b18104e20 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.947+0000 7f5b16ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18107780 0x7f5b1819ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:27:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.947+0000 7f5b16ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18107780 0x7f5b1819ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45504/0 (socket says 172.21.15.50:45504) 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.947+0000 7f5b16ffd700 1 -- 172.21.15.50:0/2318087408 learned_addr learned my addr 172.21.15.50:0/2318087408 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b177fe700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b18104e20 0x7f5b1819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b16ffd700 1 -- 172.21.15.50:0/2318087408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b18104e20 msgr2=0x7f5b1819a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b16ffd700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b18104e20 0x7f5b1819a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b16ffd700 1 -- 172.21.15.50:0/2318087408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b1810a0e0 msgr2=0x7f5b181a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b16ffd700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b1810a0e0 0x7f5b181a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b16ffd700 1 -- 172.21.15.50:0/2318087408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b181a24a0 con 0x7f5b18107780 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b16ffd700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18107780 0x7f5b1819ae80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c006010 tx=0x7f5b0c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b14ff9700 1 -- 172.21.15.50:0/2318087408 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c027070 con 0x7f5b18107780 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b14ff9700 1 -- 172.21.15.50:0/2318087408 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b0c0040c0 con 0x7f5b18107780 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/2318087408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b181a2730 con 0x7f5b18107780 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b14ff9700 1 -- 172.21.15.50:0/2318087408 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c022d90 con 0x7f5b18107780 2024-05-27T09:27:51.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.948+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/2318087408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b181a2c00 con 0x7f5b18107780 2024-05-27T09:27:51.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.949+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/2318087408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b180512b0 con 0x7f5b18107780 2024-05-27T09:27:51.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.951+0000 7f5b14ff9700 1 -- 172.21.15.50:0/2318087408 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5b0c02a070 con 0x7f5b18107780 2024-05-27T09:27:51.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.951+0000 7f5b14ff9700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5b000b2880 0x7f5b000b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:27:51.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.952+0000 7f5b14ff9700 1 -- 172.21.15.50:0/2318087408 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4965+0+0 (secure 0 0 0) 0x7f5b0c0e25f0 con 0x7f5b18107780 2024-05-27T09:27:51.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.952+0000 7f5b177fe700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5b000b2880 0x7f5b000b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:27:51.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.953+0000 7f5b177fe700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5b000b2880 0x7f5b000b4d90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b1806ca50 tx=0x7f5b08009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:27:51.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:51.955+0000 7f5b14ff9700 1 -- 172.21.15.50:0/2318087408 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5b0c05fdf0 con 0x7f5b18107780 2024-05-27T09:27:52.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:27:52.177+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/2318087408 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5b18063cb0 con 0x7f5b000b2880 2024-05-27T09:27:52.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:27:52.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:52.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:27:52.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:52.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:52.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:52.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:52.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:52.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:27:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:27:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:52.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:52.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:52.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:27:52.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:52.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:27:52.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:52.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:52.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:52.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:52.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:27:52.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:53.492 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:53 smithi050 ceph-mon[36371]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:27:53.492 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:53 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi044 2024-05-27T09:27:53.493 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:53 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi044 to 5698M 2024-05-27T09:27:53.493 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:53 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:53.493 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:53 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:53.493 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:53 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:53.493 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:53 smithi050 ceph-mon[36371]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:53.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:53 smithi189 ceph-mon[36671]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:27:53.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:53 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi044 2024-05-27T09:27:53.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:53 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi044 to 5698M 2024-05-27T09:27:53.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:53 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:53.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:53 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:53.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:53 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:53.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:53 smithi189 ceph-mon[36671]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:53.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:53 smithi044 ceph-mon[31618]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:27:53.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:53 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi044 2024-05-27T09:27:53.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:53 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi044 to 5698M 2024-05-27T09:27:53.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:53 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:27:53.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:53 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:27:53.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:53 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:27:53.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:53 smithi044 ceph-mon[31618]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:55.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:55 smithi044 ceph-mon[31618]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:55.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:55 smithi044 ceph-mon[31618]: from='client.? 172.21.15.50:0/4129758477' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd1d6765-ace5-45f8-a07b-6cd583f32f7d"}]: dispatch 2024-05-27T09:27:55.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:55 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd1d6765-ace5-45f8-a07b-6cd583f32f7d"}]: dispatch 2024-05-27T09:27:55.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:55 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd1d6765-ace5-45f8-a07b-6cd583f32f7d"}]': finished 2024-05-27T09:27:55.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:55 smithi044 ceph-mon[31618]: osdmap e44: 8 total, 7 up, 8 in 2024-05-27T09:27:55.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:55 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:27:55.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:55 smithi044 ceph-mon[31618]: from='client.? 172.21.15.50:0/1809927725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:27:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:55 smithi050 ceph-mon[36371]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:55 smithi050 ceph-mon[36371]: from='client.? 172.21.15.50:0/4129758477' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd1d6765-ace5-45f8-a07b-6cd583f32f7d"}]: dispatch 2024-05-27T09:27:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:55 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd1d6765-ace5-45f8-a07b-6cd583f32f7d"}]: dispatch 2024-05-27T09:27:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:55 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd1d6765-ace5-45f8-a07b-6cd583f32f7d"}]': finished 2024-05-27T09:27:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:55 smithi050 ceph-mon[36371]: osdmap e44: 8 total, 7 up, 8 in 2024-05-27T09:27:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:55 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:27:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:55 smithi050 ceph-mon[36371]: from='client.? 172.21.15.50:0/1809927725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:27:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:55 smithi189 ceph-mon[36671]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:55 smithi189 ceph-mon[36671]: from='client.? 172.21.15.50:0/4129758477' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd1d6765-ace5-45f8-a07b-6cd583f32f7d"}]: dispatch 2024-05-27T09:27:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:55 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd1d6765-ace5-45f8-a07b-6cd583f32f7d"}]: dispatch 2024-05-27T09:27:56.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:55 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd1d6765-ace5-45f8-a07b-6cd583f32f7d"}]': finished 2024-05-27T09:27:56.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:55 smithi189 ceph-mon[36671]: osdmap e44: 8 total, 7 up, 8 in 2024-05-27T09:27:56.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:55 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:27:56.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:55 smithi189 ceph-mon[36671]: from='client.? 172.21.15.50:0/1809927725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:27:57.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:57 smithi044 ceph-mon[31618]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:58.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:57 smithi050 ceph-mon[36371]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:58.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:57 smithi189 ceph-mon[36671]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:59.921 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:27:59 smithi050 ceph-mon[36371]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:27:59.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:27:59 smithi044 ceph-mon[31618]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:00.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:27:59 smithi189 ceph-mon[36671]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:01.643 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:01 smithi050 ceph-mon[36371]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:01.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:01 smithi044 ceph-mon[31618]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:02.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:01 smithi189 ceph-mon[36671]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:02.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T09:28:02.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:02 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:02.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:02 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T09:28:02.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:02 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:03.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:02 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T09:28:03.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:02 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:03.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:03 smithi050 ceph-mon[36371]: Deploying daemon osd.7 on smithi050 2024-05-27T09:28:03.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:03 smithi050 ceph-mon[36371]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:03.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:03 smithi044 ceph-mon[31618]: Deploying daemon osd.7 on smithi050 2024-05-27T09:28:03.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:03 smithi044 ceph-mon[31618]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:04.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:03 smithi189 ceph-mon[36671]: Deploying daemon osd.7 on smithi050 2024-05-27T09:28:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:03 smithi189 ceph-mon[36671]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:05.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:05 smithi050 ceph-mon[36371]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:05.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:05 smithi044 ceph-mon[31618]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:06.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:05 smithi189 ceph-mon[36671]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:07.793 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:07 smithi050 ceph-mon[36371]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:07.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:07 smithi044 ceph-mon[31618]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:08.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:07 smithi189 ceph-mon[36671]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:09.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:08 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:09.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:08 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:09.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:08 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:09.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:09.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:09.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:09.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:09.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:09.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:09 smithi050 ceph-mon[36371]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:10.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:09 smithi189 ceph-mon[36671]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:10.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:09 smithi044 ceph-mon[31618]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:11.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.572+0000 7f5b14ff9700 1 -- 172.21.15.50:0/2318087408 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5b18063cb0 con 0x7f5b000b2880 2024-05-27T09:28:11.573 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 7 on host 'smithi050' 2024-05-27T09:28:11.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.574+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/2318087408 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5b000b2880 msgr2=0x7f5b000b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:28:11.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.574+0000 7f5b1dc0d700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5b000b2880 0x7f5b000b4d90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b1806ca50 tx=0x7f5b08009190 comp rx=0 tx=0).stop 2024-05-27T09:28:11.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.574+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/2318087408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18107780 msgr2=0x7f5b1819ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:28:11.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.574+0000 7f5b1dc0d700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18107780 0x7f5b1819ae80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c006010 tx=0x7f5b0c0047b0 comp rx=0 tx=0).stop 2024-05-27T09:28:11.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.574+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/2318087408 shutdown_connections 2024-05-27T09:28:11.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.574+0000 7f5b1dc0d700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5b000b2880 0x7f5b000b4d90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:11.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.574+0000 7f5b1dc0d700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b1810a0e0 0x7f5b181a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:11.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.574+0000 7f5b1dc0d700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b18104e20 0x7f5b1819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:11.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.574+0000 7f5b1dc0d700 1 --2- 172.21.15.50:0/2318087408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18107780 0x7f5b1819ae80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:11.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.574+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/2318087408 >> 172.21.15.50:0/2318087408 conn(0x7f5b180fe860 msgr2=0x7f5b18105910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:28:11.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.575+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/2318087408 shutdown_connections 2024-05-27T09:28:11.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:28:11.575+0000 7f5b1dc0d700 1 -- 172.21.15.50:0/2318087408 wait complete. 2024-05-27T09:28:11.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:11 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:11.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:11 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:11.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:11 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:11.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:11 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:11.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:11 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:11.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:11 smithi050 ceph-mon[36371]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:11.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:11 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:11.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:11 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:11.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:11 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:11.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:11 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:11.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:11 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:11.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:11 smithi189 ceph-mon[36671]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:11 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:11 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:11 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:11 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:11 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:11 smithi044 ceph-mon[31618]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:12.185 DEBUG:teuthology.orchestra.run.smithi050:osd.7> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.7.service 2024-05-27T09:28:12.187 INFO:tasks.cephadm:Deploying osd.8 on smithi189 with /dev/vg_nvme/lv_2... 2024-05-27T09:28:12.188 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T09:28:12.434 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:28:12.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:12 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:12.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:12 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:12.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:12 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:12.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:12 smithi189 ceph-mon[36671]: from='osd.7 [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T09:28:12.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:12 smithi189 ceph-mon[36671]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T09:28:12.793 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:12 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:12.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:12 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:12.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:12 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:12.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:12 smithi050 ceph-mon[36371]: from='osd.7 [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T09:28:12.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:12 smithi050 ceph-mon[36371]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T09:28:12.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:12 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:12.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:12 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:12.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:12 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:12.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:12 smithi044 ceph-mon[31618]: from='osd.7 [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T09:28:12.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:12 smithi044 ceph-mon[31618]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T09:28:13.778 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:13 smithi189 ceph-mon[36671]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T09:28:13.779 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:13 smithi189 ceph-mon[36671]: from='osd.7 [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:28:13.779 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:13 smithi189 ceph-mon[36671]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T09:28:13.779 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:28:13.779 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:13 smithi189 ceph-mon[36671]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:28:13.779 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:13 smithi189 ceph-mon[36671]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:13 smithi050 ceph-mon[36371]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T09:28:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:13 smithi050 ceph-mon[36371]: from='osd.7 [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:28:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:13 smithi050 ceph-mon[36371]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T09:28:13.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:28:13.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:13 smithi050 ceph-mon[36371]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:28:13.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:13 smithi050 ceph-mon[36371]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:13.795 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:28:13 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:28:13.544+0000 7fa610b2b700 -1 osd.7 0 waiting for initial osdmap 2024-05-27T09:28:13.795 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:28:13 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:28:13.557+0000 7fa60c8e0700 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T09:28:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:13 smithi044 ceph-mon[31618]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T09:28:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:13 smithi044 ceph-mon[31618]: from='osd.7 [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:28:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:13 smithi044 ceph-mon[31618]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T09:28:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:28:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:13 smithi044 ceph-mon[31618]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:28:13.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:13 smithi044 ceph-mon[31618]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:14.331 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T09:28:14.331 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T09:28:14.331 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-05-27T09:28:14.331 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-05-27T09:28:14.331 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011427 s, 918 MB/s 2024-05-27T09:28:14.332 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-05-27T09:28:14.626 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:14 smithi189 ceph-mon[36671]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:28:14.626 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:14 smithi189 ceph-mon[36671]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T09:28:14.626 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:28:14.663 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:14 smithi050 ceph-mon[36371]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:28:14.663 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:14 smithi050 ceph-mon[36371]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T09:28:14.663 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:28:14.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:14 smithi044 ceph-mon[31618]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:28:14.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:14 smithi044 ceph-mon[31618]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T09:28:14.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:28:15.109 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_2 2024-05-27T09:28:15.348 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:28:15.764 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:28:15.764 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:28:15.764 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: osd.7 [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446] boot 2024-05-27T09:28:15.764 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: osdmap e47: 8 total, 8 up, 8 in 2024-05-27T09:28:15.764 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:28:15.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:15.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:15.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:15.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:15.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:15.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:15.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:15.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:15.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:15.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:15.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:15 smithi189 ceph-mon[36671]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T09:28:15.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: osd.7 [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446] boot 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: osdmap e47: 8 total, 8 up, 8 in 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:15.976 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:15.976 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:15 smithi044 ceph-mon[31618]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T09:28:16.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:28:16.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:28:16.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: osd.7 [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446] boot 2024-05-27T09:28:16.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: osdmap e47: 8 total, 8 up, 8 in 2024-05-27T09:28:16.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:28:16.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-05-27T09:28:16.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:16.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:16.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:16.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:16.047 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:16.047 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:16.047 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:16.047 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:16.047 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:16.047 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:15 smithi050 ceph-mon[36371]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T09:28:16.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:16 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi050 2024-05-27T09:28:16.688 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:16 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi050 to 5697M 2024-05-27T09:28:16.688 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:16 smithi189 ceph-mon[36671]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T09:28:16.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.907+0000 7f2947483700 1 -- 172.21.15.189:0/424901631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2940104330 msgr2=0x7f2940104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:28:16.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.907+0000 7f2947483700 1 --2- 172.21.15.189:0/424901631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2940104330 0x7f2940104740 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2930009990 tx=0x7f2930009ca0 comp rx=0 tx=0).stop 2024-05-27T09:28:16.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.907+0000 7f2947483700 1 -- 172.21.15.189:0/424901631 shutdown_connections 2024-05-27T09:28:16.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.907+0000 7f2947483700 1 --2- 172.21.15.189:0/424901631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2940105e00 0x7f294010c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:16.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.907+0000 7f2947483700 1 --2- 172.21.15.189:0/424901631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2940104330 0x7f2940104740 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:16.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.907+0000 7f2947483700 1 --2- 172.21.15.189:0/424901631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2940105460 0x7f2940105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:16.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.907+0000 7f2947483700 1 -- 172.21.15.189:0/424901631 >> 172.21.15.189:0/424901631 conn(0x7f29400ff9b0 msgr2=0x7f2940101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:28:16.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.908+0000 7f2947483700 1 -- 172.21.15.189:0/424901631 shutdown_connections 2024-05-27T09:28:16.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.908+0000 7f2947483700 1 -- 172.21.15.189:0/424901631 wait complete. 2024-05-27T09:28:16.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.908+0000 7f2947483700 1 Processor -- start 2024-05-27T09:28:16.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.908+0000 7f2947483700 1 -- start start 2024-05-27T09:28:16.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.908+0000 7f2947483700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2940104330 0x7f294019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:28:16.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.908+0000 7f2947483700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2940105460 0x7f294019afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.908+0000 7f2947483700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2940105e00 0x7f29401a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2947483700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f294010fe70 con 0x7f2940104330 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2947483700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f294010fcf0 con 0x7f2940105460 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2947483700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f294010fff0 con 0x7f2940105e00 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2945a20700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2940105e00 0x7f29401a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2945a20700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2940105e00 0x7f29401a1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60938/0 (socket says 172.21.15.189:60938) 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2945a20700 1 -- 172.21.15.189:0/2656215869 learned_addr learned my addr 172.21.15.189:0/2656215869 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2944a1e700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2940105460 0x7f294019afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2945a20700 1 -- 172.21.15.189:0/2656215869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2940105460 msgr2=0x7f294019afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2945a20700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2940105460 0x7f294019afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2945a20700 1 -- 172.21.15.189:0/2656215869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2940104330 msgr2=0x7f294019aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:28:16.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2945a20700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2940104330 0x7f294019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:16.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2945a20700 1 -- 172.21.15.189:0/2656215869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29401a25b0 con 0x7f2940105e00 2024-05-27T09:28:16.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2944a1e700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2940105460 0x7f294019afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:28:16.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2945a20700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2940105e00 0x7f29401a1f30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f293800b970 tx=0x7f293800bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:28:16.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f29367fc700 1 -- 172.21.15.189:0/2656215869 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938019070 con 0x7f2940105e00 2024-05-27T09:28:16.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f29367fc700 1 -- 172.21.15.189:0/2656215869 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2938014440 con 0x7f2940105e00 2024-05-27T09:28:16.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2947483700 1 -- 172.21.15.189:0/2656215869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29401a28a0 con 0x7f2940105e00 2024-05-27T09:28:16.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f29367fc700 1 -- 172.21.15.189:0/2656215869 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293800ce60 con 0x7f2940105e00 2024-05-27T09:28:16.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.909+0000 7f2947483700 1 -- 172.21.15.189:0/2656215869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29401a2e60 con 0x7f2940105e00 2024-05-27T09:28:16.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.910+0000 7f2947483700 1 -- 172.21.15.189:0/2656215869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29400512b0 con 0x7f2940105e00 2024-05-27T09:28:16.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.912+0000 7f29367fc700 1 -- 172.21.15.189:0/2656215869 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f293801c070 con 0x7f2940105e00 2024-05-27T09:28:16.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.912+0000 7f29367fc700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f292c0b2900 0x7f292c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:28:16.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.913+0000 7f29367fc700 1 -- 172.21.15.189:0/2656215869 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5397+0+0 (secure 0 0 0) 0x7f29380e0f20 con 0x7f2940105e00 2024-05-27T09:28:16.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.913+0000 7f294521f700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f292c0b2900 0x7f292c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:28:16.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.914+0000 7f294521f700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f292c0b2900 0x7f292c0b4e10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f293000b5c0 tx=0x7f2930003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:28:16.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:16.916+0000 7f29367fc700 1 -- 172.21.15.189:0/2656215869 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f29380a97a0 con 0x7f2940105e00 2024-05-27T09:28:16.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:16 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi050 2024-05-27T09:28:16.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:16 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi050 to 5697M 2024-05-27T09:28:16.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:16 smithi044 ceph-mon[31618]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T09:28:17.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:16 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi050 2024-05-27T09:28:17.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:16 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi050 to 5697M 2024-05-27T09:28:17.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:16 smithi050 ceph-mon[36371]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T09:28:17.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:17.129+0000 7f2947483700 1 -- 172.21.15.189:0/2656215869 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2940063bf0 con 0x7f292c0b2900 2024-05-27T09:28:17.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:17 smithi189 ceph-mon[36671]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:17.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:28:17.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:28:17.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:17 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:17 smithi044 ceph-mon[31618]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:28:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:28:17.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:17 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:17 smithi050 ceph-mon[36371]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:28:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:28:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:17 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:18.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:18 smithi189 ceph-mon[36671]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:28:18.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:18 smithi044 ceph-mon[31618]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:28:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:18 smithi050 ceph-mon[36371]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:28:19.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:19 smithi189 ceph-mon[36671]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:19.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:19 smithi044 ceph-mon[31618]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:20.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:19 smithi050 ceph-mon[36371]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:20.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:20 smithi044 ceph-mon[31618]: from='client.? 172.21.15.189:0/1507514859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2602577e-c4b3-433e-a443-a71e4aa793a4"}]: dispatch 2024-05-27T09:28:20.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:20 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2602577e-c4b3-433e-a443-a71e4aa793a4"}]: dispatch 2024-05-27T09:28:20.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:20 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2602577e-c4b3-433e-a443-a71e4aa793a4"}]': finished 2024-05-27T09:28:20.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:20 smithi044 ceph-mon[31618]: osdmap e50: 9 total, 8 up, 9 in 2024-05-27T09:28:20.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:20 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:20.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:20 smithi044 ceph-mon[31618]: from='client.? 172.21.15.189:0/1689421513' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:28:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:20 smithi050 ceph-mon[36371]: from='client.? 172.21.15.189:0/1507514859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2602577e-c4b3-433e-a443-a71e4aa793a4"}]: dispatch 2024-05-27T09:28:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:20 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2602577e-c4b3-433e-a443-a71e4aa793a4"}]: dispatch 2024-05-27T09:28:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:20 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2602577e-c4b3-433e-a443-a71e4aa793a4"}]': finished 2024-05-27T09:28:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:20 smithi050 ceph-mon[36371]: osdmap e50: 9 total, 8 up, 9 in 2024-05-27T09:28:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:20 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:20 smithi050 ceph-mon[36371]: from='client.? 172.21.15.189:0/1689421513' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:28:21.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:20 smithi189 ceph-mon[36671]: from='client.? 172.21.15.189:0/1507514859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2602577e-c4b3-433e-a443-a71e4aa793a4"}]: dispatch 2024-05-27T09:28:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:20 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2602577e-c4b3-433e-a443-a71e4aa793a4"}]: dispatch 2024-05-27T09:28:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:20 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2602577e-c4b3-433e-a443-a71e4aa793a4"}]': finished 2024-05-27T09:28:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:20 smithi189 ceph-mon[36671]: osdmap e50: 9 total, 8 up, 9 in 2024-05-27T09:28:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:20 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:20 smithi189 ceph-mon[36671]: from='client.? 172.21.15.189:0/1689421513' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:28:22.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:21 smithi044 ceph-mon[31618]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T09:28:22.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:21 smithi050 ceph-mon[36371]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T09:28:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:21 smithi189 ceph-mon[36671]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T09:28:24.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:23 smithi189 ceph-mon[36671]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-27T09:28:24.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:23 smithi044 ceph-mon[31618]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-27T09:28:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:23 smithi050 ceph-mon[36371]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-27T09:28:26.014 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:25 smithi189 ceph-mon[36671]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T09:28:26.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:25 smithi044 ceph-mon[31618]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T09:28:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:25 smithi050 ceph-mon[36371]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T09:28:28.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:27 smithi189 ceph-mon[36671]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:28:28.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:27 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T09:28:28.012 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:27 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:27 smithi044 ceph-mon[31618]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:28:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:27 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T09:28:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:27 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:28.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:27 smithi050 ceph-mon[36371]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:28:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:27 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T09:28:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:27 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:29.178 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:28 smithi189 ceph-mon[36671]: Deploying daemon osd.8 on smithi189 2024-05-27T09:28:29.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:28 smithi044 ceph-mon[31618]: Deploying daemon osd.8 on smithi189 2024-05-27T09:28:29.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:28 smithi050 ceph-mon[36371]: Deploying daemon osd.8 on smithi189 2024-05-27T09:28:30.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:29 smithi044 ceph-mon[31618]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:28:30.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:29 smithi189 ceph-mon[36671]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:28:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:29 smithi050 ceph-mon[36371]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:28:32.170 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:31 smithi189 ceph-mon[36671]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:32.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:31 smithi044 ceph-mon[31618]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:32.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:31 smithi050 ceph-mon[36371]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:33.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:33 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:33.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:33 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:33.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:33 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:33.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:33 smithi044 ceph-mon[31618]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:33.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:33 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:33.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:33 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:33.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:33 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:33.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:33 smithi189 ceph-mon[36671]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:33.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:33 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:33.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:33 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:33.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:33 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:33.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:33 smithi050 ceph-mon[36371]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:35.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:35 smithi189 ceph-mon[36671]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:35.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:35 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:35.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:35 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:35.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:35 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:35.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:35 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:35.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:35 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:35.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:35 smithi044 ceph-mon[31618]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:35.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:35 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:35.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:35 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:35.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:35 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:35.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:35 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:35.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:35 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:36.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:35 smithi050 ceph-mon[36371]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:36.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:35 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:36.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:35 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:36.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:35 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:36.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:35 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:36.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:35 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:36.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.265+0000 7f29367fc700 1 -- 172.21.15.189:0/2656215869 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2940063bf0 con 0x7f292c0b2900 2024-05-27T09:28:36.266 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 8 on host 'smithi189' 2024-05-27T09:28:36.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 -- 172.21.15.189:0/2656215869 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f292c0b2900 msgr2=0x7f292c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:28:36.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f292c0b2900 0x7f292c0b4e10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f293000b5c0 tx=0x7f2930003860 comp rx=0 tx=0).stop 2024-05-27T09:28:36.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 -- 172.21.15.189:0/2656215869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2940105e00 msgr2=0x7f29401a1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:28:36.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2940105e00 0x7f29401a1f30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f293800b970 tx=0x7f293800bd30 comp rx=0 tx=0).stop 2024-05-27T09:28:36.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 -- 172.21.15.189:0/2656215869 shutdown_connections 2024-05-27T09:28:36.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f292c0b2900 0x7f292c0b4e10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:36.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2940105e00 0x7f29401a1f30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:36.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2940104330 0x7f294019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:36.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 --2- 172.21.15.189:0/2656215869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2940105460 0x7f294019afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:36.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 -- 172.21.15.189:0/2656215869 >> 172.21.15.189:0/2656215869 conn(0x7f29400ff9b0 msgr2=0x7f2940108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:28:36.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 -- 172.21.15.189:0/2656215869 shutdown_connections 2024-05-27T09:28:36.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:28:36.268+0000 7f2947483700 1 -- 172.21.15.189:0/2656215869 wait complete. 2024-05-27T09:28:37.025 DEBUG:teuthology.orchestra.run.smithi189:osd.8> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.8.service 2024-05-27T09:28:37.028 INFO:tasks.cephadm:Deploying osd.9 on smithi044 with /dev/vg_nvme/lv_1... 2024-05-27T09:28:37.028 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T09:28:37.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:37.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:37.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:37.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:37 smithi189 ceph-mon[36671]: from='osd.8 [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T09:28:37.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:37 smithi189 ceph-mon[36671]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T09:28:37.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:37 smithi189 ceph-mon[36671]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:37 smithi044 ceph-mon[31618]: from='osd.8 [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T09:28:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:37 smithi044 ceph-mon[31618]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T09:28:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:37 smithi044 ceph-mon[31618]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:37.509 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:28:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:28:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:37 smithi050 ceph-mon[36371]: from='osd.8 [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T09:28:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:37 smithi050 ceph-mon[36371]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T09:28:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:37 smithi050 ceph-mon[36371]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:38.383 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:38 smithi189 ceph-mon[36671]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T09:28:38.383 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:38 smithi189 ceph-mon[36671]: from='osd.8 [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:28:38.383 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:38 smithi189 ceph-mon[36671]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T09:28:38.383 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:38 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:38.383 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:38 smithi189 ceph-mon[36671]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:28:38.383 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:28:38 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:28:38.244+0000 7fa875490700 -1 osd.8 0 waiting for initial osdmap 2024-05-27T09:28:38.384 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:28:38 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:28:38.257+0000 7fa871245700 -1 osd.8 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T09:28:38.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:38 smithi050 ceph-mon[36371]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T09:28:38.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:38 smithi050 ceph-mon[36371]: from='osd.8 [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:28:38.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:38 smithi050 ceph-mon[36371]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T09:28:38.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:38 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:38.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:38 smithi050 ceph-mon[36371]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:28:38.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:38 smithi044 ceph-mon[31618]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T09:28:38.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:38 smithi044 ceph-mon[31618]: from='osd.8 [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:28:38.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:38 smithi044 ceph-mon[31618]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T09:28:38.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:38 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:38.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:38 smithi044 ceph-mon[31618]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:28:39.400 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:39 smithi044 ceph-mon[31618]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:28:39.400 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:39 smithi044 ceph-mon[31618]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T09:28:39.400 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:39.400 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:39 smithi044 ceph-mon[31618]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:39.400 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:39 smithi044 ceph-mon[31618]: osd.8 [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694] boot 2024-05-27T09:28:39.400 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:39 smithi044 ceph-mon[31618]: osdmap e53: 9 total, 9 up, 9 in 2024-05-27T09:28:39.400 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:39.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:39 smithi189 ceph-mon[36671]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:28:39.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:39 smithi189 ceph-mon[36671]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T09:28:39.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:39.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:39 smithi189 ceph-mon[36671]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:39.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:39 smithi189 ceph-mon[36671]: osd.8 [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694] boot 2024-05-27T09:28:39.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:39 smithi189 ceph-mon[36671]: osdmap e53: 9 total, 9 up, 9 in 2024-05-27T09:28:39.510 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:39.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:39 smithi050 ceph-mon[36371]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:28:39.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:39 smithi050 ceph-mon[36371]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T09:28:39.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:39.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:39 smithi050 ceph-mon[36371]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-05-27T09:28:39.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:39 smithi050 ceph-mon[36371]: osd.8 [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694] boot 2024-05-27T09:28:39.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:39 smithi050 ceph-mon[36371]: osdmap e53: 9 total, 9 up, 9 in 2024-05-27T09:28:39.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:28:40.285 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T09:28:40.285 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T09:28:40.285 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-05-27T09:28:40.285 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-05-27T09:28:40.286 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0395013 s, 265 MB/s 2024-05-27T09:28:40.286 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-05-27T09:28:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:28:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:28:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi189 2024-05-27T09:28:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi189 to 5696M 2024-05-27T09:28:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:40.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:40.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:40 smithi050 ceph-mon[36371]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T09:28:40.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi189 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi189 to 5696M 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:40 smithi189 ceph-mon[36671]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T09:28:40.710 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:28:40.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:28:40.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi189 2024-05-27T09:28:40.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:40.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:40.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:28:40.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi189 to 5696M 2024-05-27T09:28:40.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:40.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:28:40.712 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:28:40.712 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:40 smithi044 ceph-mon[31618]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T09:28:41.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:41 smithi189 ceph-mon[36671]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:41.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:41 smithi189 ceph-mon[36671]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T09:28:41.357 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_1 2024-05-27T09:28:41.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:41 smithi050 ceph-mon[36371]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:41.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:41 smithi050 ceph-mon[36371]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T09:28:41.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:41 smithi044 ceph-mon[31618]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:41.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:41 smithi044 ceph-mon[31618]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T09:28:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:28:43.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:43 smithi050 ceph-mon[36371]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:43 smithi044 ceph-mon[31618]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:44.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:43 smithi189 ceph-mon[36671]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:44.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.186+0000 7fc3b2c00700 1 -- 172.21.15.44:0/1599165753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3ac10c250 msgr2=0x7fc3ac112b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:28:44.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.186+0000 7fc3b2c00700 1 --2- 172.21.15.44:0/1599165753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3ac10c250 0x7fc3ac112b00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400b0a0 tx=0x7fc3a400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:28:44.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.186+0000 7fc3b2c00700 1 -- 172.21.15.44:0/1599165753 shutdown_connections 2024-05-27T09:28:44.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.186+0000 7fc3b2c00700 1 --2- 172.21.15.44:0/1599165753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3ac10c250 0x7fc3ac112b00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:44.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.186+0000 7fc3b2c00700 1 --2- 172.21.15.44:0/1599165753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3ac10a780 0x7fc3ac10ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:44.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.186+0000 7fc3b2c00700 1 --2- 172.21.15.44:0/1599165753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3ac10b8b0 0x7fc3ac10bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:44.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.186+0000 7fc3b2c00700 1 -- 172.21.15.44:0/1599165753 >> 172.21.15.44:0/1599165753 conn(0x7fc3ac079830 msgr2=0x7fc3ac1085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:28:44.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.186+0000 7fc3b2c00700 1 -- 172.21.15.44:0/1599165753 shutdown_connections 2024-05-27T09:28:44.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.187+0000 7fc3b2c00700 1 -- 172.21.15.44:0/1599165753 wait complete. 2024-05-27T09:28:44.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.188+0000 7fc3b2c00700 1 Processor -- start 2024-05-27T09:28:44.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.188+0000 7fc3b2c00700 1 -- start start 2024-05-27T09:28:44.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.188+0000 7fc3b2c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3ac10a780 0x7fc3ac1a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:28:44.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.188+0000 7fc3b2c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3ac10b8b0 0x7fc3ac1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:28:44.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.188+0000 7fc3b2c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3ac10c250 0x7fc3ac1a8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:28:44.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.188+0000 7fc3b2c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ac1161f0 con 0x7fc3ac10c250 2024-05-27T09:28:44.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.188+0000 7fc3b2c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ac116070 con 0x7fc3ac10a780 2024-05-27T09:28:44.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.188+0000 7fc3b2c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ac116370 con 0x7fc3ac10b8b0 2024-05-27T09:28:44.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b2200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3ac10c250 0x7fc3ac1a8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:28:44.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b1800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3ac10a780 0x7fc3ac1a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:28:44.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b2200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3ac10c250 0x7fc3ac1a8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42782/0 (socket says 172.21.15.44:42782) 2024-05-27T09:28:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b2200700 1 -- 172.21.15.44:0/2755699554 learned_addr learned my addr 172.21.15.44:0/2755699554 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:28:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b0e00700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3ac10b8b0 0x7fc3ac1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:28:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b1800700 1 -- 172.21.15.44:0/2755699554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3ac10b8b0 msgr2=0x7fc3ac1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:28:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b1800700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3ac10b8b0 0x7fc3ac1a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b1800700 1 -- 172.21.15.44:0/2755699554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3ac10c250 msgr2=0x7fc3ac1a8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:28:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b1800700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3ac10c250 0x7fc3ac1a8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:28:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b1800700 1 -- 172.21.15.44:0/2755699554 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ac1a87e0 con 0x7fc3ac10a780 2024-05-27T09:28:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b2200700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3ac10c250 0x7fc3ac1a8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:28:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.189+0000 7fc3b1800700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3ac10a780 0x7fc3ac1a0c70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac10a480 tx=0x7fc3a000eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:28:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.190+0000 7fc3aa000700 1 -- 172.21.15.44:0/2755699554 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0004220 con 0x7fc3ac10a780 2024-05-27T09:28:44.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.190+0000 7fc3b2c00700 1 -- 172.21.15.44:0/2755699554 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3ac1a8ad0 con 0x7fc3ac10a780 2024-05-27T09:28:44.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.190+0000 7fc3aa000700 1 -- 172.21.15.44:0/2755699554 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3a00043c0 con 0x7fc3ac10a780 2024-05-27T09:28:44.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.190+0000 7fc3aa000700 1 -- 172.21.15.44:0/2755699554 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0003be0 con 0x7fc3ac10a780 2024-05-27T09:28:44.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.190+0000 7fc3b2c00700 1 -- 172.21.15.44:0/2755699554 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3ac1a9060 con 0x7fc3ac10a780 2024-05-27T09:28:44.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.191+0000 7fc3b2c00700 1 -- 172.21.15.44:0/2755699554 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3ac0512b0 con 0x7fc3ac10a780 2024-05-27T09:28:44.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.194+0000 7fc3aa000700 1 -- 172.21.15.44:0/2755699554 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc3a0046910 con 0x7fc3ac10a780 2024-05-27T09:28:44.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.195+0000 7fc3aa000700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc3940b2880 0x7fc3940b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:28:44.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.196+0000 7fc3aa000700 1 -- 172.21.15.44:0/2755699554 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5829+0+0 (secure 0 0 0) 0x7fc3a00e1c50 con 0x7fc3ac10a780 2024-05-27T09:28:44.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.196+0000 7fc3b0e00700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc3940b2880 0x7fc3940b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:28:44.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.196+0000 7fc3b0e00700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc3940b2880 0x7fc3940b4d90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009c60 tx=0x7fc39c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:28:44.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.198+0000 7fc3aa000700 1 -- 172.21.15.44:0/2755699554 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc3a00aa280 con 0x7fc3ac10a780 2024-05-27T09:28:44.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:28:44.448+0000 7fc3b2c00700 1 -- 172.21.15.44:0/2755699554 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc3ac063bf0 con 0x7fc3940b2880 2024-05-27T09:28:44.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:28:44.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:28:44.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:45.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:28:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:28:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:45.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:28:45.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:28:45.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:28:45.761 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:45 smithi044 ceph-mon[31618]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:28:45.761 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:45 smithi044 ceph-mon[31618]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T09:28:46.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:45 smithi050 ceph-mon[36371]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:28:46.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:45 smithi050 ceph-mon[36371]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T09:28:46.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:45 smithi189 ceph-mon[36671]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:28:46.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:45 smithi189 ceph-mon[36671]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T09:28:47.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:47 smithi044 ceph-mon[31618]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-27T09:28:48.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:47 smithi050 ceph-mon[36371]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-27T09:28:48.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:47 smithi189 ceph-mon[36671]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-27T09:28:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:49 smithi044 ceph-mon[31618]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-27T09:28:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:49 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1540858211' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0307f939-76d2-4b3d-89ae-31581649f5e5"}]: dispatch 2024-05-27T09:28:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:49 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0307f939-76d2-4b3d-89ae-31581649f5e5"}]: dispatch 2024-05-27T09:28:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:49 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0307f939-76d2-4b3d-89ae-31581649f5e5"}]': finished 2024-05-27T09:28:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:49 smithi044 ceph-mon[31618]: osdmap e56: 10 total, 9 up, 10 in 2024-05-27T09:28:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:28:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:49 smithi050 ceph-mon[36371]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-27T09:28:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:49 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1540858211' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0307f939-76d2-4b3d-89ae-31581649f5e5"}]: dispatch 2024-05-27T09:28:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:49 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0307f939-76d2-4b3d-89ae-31581649f5e5"}]: dispatch 2024-05-27T09:28:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:49 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0307f939-76d2-4b3d-89ae-31581649f5e5"}]': finished 2024-05-27T09:28:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:49 smithi050 ceph-mon[36371]: osdmap e56: 10 total, 9 up, 10 in 2024-05-27T09:28:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:28:50.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:49 smithi189 ceph-mon[36671]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-27T09:28:50.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:49 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1540858211' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0307f939-76d2-4b3d-89ae-31581649f5e5"}]: dispatch 2024-05-27T09:28:50.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:49 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0307f939-76d2-4b3d-89ae-31581649f5e5"}]: dispatch 2024-05-27T09:28:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:49 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0307f939-76d2-4b3d-89ae-31581649f5e5"}]': finished 2024-05-27T09:28:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:49 smithi189 ceph-mon[36671]: osdmap e56: 10 total, 9 up, 10 in 2024-05-27T09:28:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:28:50.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:50 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1382889696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:28:51.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:50 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1382889696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:28:51.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:50 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1382889696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:28:51.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:51 smithi044 ceph-mon[31618]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-27T09:28:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:51 smithi050 ceph-mon[36371]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-27T09:28:52.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:51 smithi189 ceph-mon[36671]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-27T09:28:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:53 smithi044 ceph-mon[31618]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:28:54.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:53 smithi050 ceph-mon[36371]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:28:54.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:53 smithi189 ceph-mon[36671]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:28:55.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:55 smithi044 ceph-mon[31618]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:56.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:55 smithi050 ceph-mon[36371]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:55 smithi189 ceph-mon[36671]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:57.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:57 smithi044 ceph-mon[31618]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:58.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:57 smithi050 ceph-mon[36371]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:58.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:57 smithi189 ceph-mon[36671]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:59.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:59 smithi044 ceph-mon[31618]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:28:59.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T09:28:59.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:28:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:59 smithi050 ceph-mon[36371]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T09:29:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:28:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:00.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:59 smithi189 ceph-mon[36671]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:00.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T09:29:00.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:28:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:00.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:00 smithi044 ceph-mon[31618]: Deploying daemon osd.9 on smithi044 2024-05-27T09:29:01.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:00 smithi050 ceph-mon[36371]: Deploying daemon osd.9 on smithi044 2024-05-27T09:29:01.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:00 smithi189 ceph-mon[36671]: Deploying daemon osd.9 on smithi044 2024-05-27T09:29:01.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:01 smithi044 ceph-mon[31618]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:02.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:01 smithi050 ceph-mon[36371]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:02.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:01 smithi189 ceph-mon[36671]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:03.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:03 smithi044 ceph-mon[31618]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:04.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:03 smithi050 ceph-mon[36371]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:04.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:03 smithi189 ceph-mon[36671]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:05.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:05 smithi044 ceph-mon[31618]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:06.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:05 smithi050 ceph-mon[36371]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:06.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:05 smithi189 ceph-mon[36671]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:07.941 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:07 smithi044 ceph-mon[31618]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:08.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:07 smithi050 ceph-mon[36371]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:08.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:07 smithi189 ceph-mon[36671]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:09.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:09.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:09.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:09.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:09 smithi044 ceph-mon[31618]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:09.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:09.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:09.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:09.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:09 smithi050 ceph-mon[36371]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:09.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:09.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:09.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:09.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:09 smithi189 ceph-mon[36671]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:11.804 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:11 smithi044 ceph-mon[31618]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:12.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:11 smithi050 ceph-mon[36371]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:12.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:11 smithi189 ceph-mon[36671]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:13.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:13 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:13 smithi189 ceph-mon[36671]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:13 smithi189 ceph-mon[36671]: from='osd.9 [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T09:29:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:13 smithi189 ceph-mon[36671]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T09:29:13.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:13.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:13 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:13.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:13 smithi044 ceph-mon[31618]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:13.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:13 smithi044 ceph-mon[31618]: from='osd.9 [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T09:29:13.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:13 smithi044 ceph-mon[31618]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T09:29:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:13 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:13 smithi050 ceph-mon[36371]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:13 smithi050 ceph-mon[36371]: from='osd.9 [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T09:29:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:13 smithi050 ceph-mon[36371]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T09:29:14.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.357+0000 7fc3aa000700 1 -- 172.21.15.44:0/2755699554 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc3ac063bf0 con 0x7fc3940b2880 2024-05-27T09:29:14.360 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 9 on host 'smithi044' 2024-05-27T09:29:14.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.363+0000 7fc3b2c00700 1 -- 172.21.15.44:0/2755699554 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc3940b2880 msgr2=0x7fc3940b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:29:14.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.363+0000 7fc3b2c00700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc3940b2880 0x7fc3940b4d90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009c60 tx=0x7fc39c009260 comp rx=0 tx=0).stop 2024-05-27T09:29:14.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.363+0000 7fc3b2c00700 1 -- 172.21.15.44:0/2755699554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3ac10a780 msgr2=0x7fc3ac1a0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:29:14.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.363+0000 7fc3b2c00700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3ac10a780 0x7fc3ac1a0c70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac10a480 tx=0x7fc3a000eca0 comp rx=0 tx=0).stop 2024-05-27T09:29:14.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.363+0000 7fc3b2c00700 1 -- 172.21.15.44:0/2755699554 shutdown_connections 2024-05-27T09:29:14.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.363+0000 7fc3b2c00700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc3940b2880 0x7fc3940b4d90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:14.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.363+0000 7fc3b2c00700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3ac10c250 0x7fc3ac1a8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:14.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.363+0000 7fc3b2c00700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3ac10a780 0x7fc3ac1a0c70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:14.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.363+0000 7fc3b2c00700 1 --2- 172.21.15.44:0/2755699554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3ac10b8b0 0x7fc3ac1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:14.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.363+0000 7fc3b2c00700 1 -- 172.21.15.44:0/2755699554 >> 172.21.15.44:0/2755699554 conn(0x7fc3ac079830 msgr2=0x7fc3ac10eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:29:14.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.364+0000 7fc3b2c00700 1 -- 172.21.15.44:0/2755699554 shutdown_connections 2024-05-27T09:29:14.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:29:14.364+0000 7fc3b2c00700 1 -- 172.21.15.44:0/2755699554 wait complete. 2024-05-27T09:29:14.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:14 smithi044 ceph-mon[31618]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T09:29:14.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:14 smithi044 ceph-mon[31618]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T09:29:14.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:14.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:14 smithi044 ceph-mon[31618]: from='osd.9 [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:29:14.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:14 smithi044 ceph-mon[31618]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:29:14.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:14.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:14.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:14.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:14.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:14.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:14 smithi050 ceph-mon[36371]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T09:29:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:14 smithi050 ceph-mon[36371]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T09:29:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:14 smithi050 ceph-mon[36371]: from='osd.9 [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:29:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:14 smithi050 ceph-mon[36371]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:29:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:14.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:14.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:14.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:14.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:14.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:14 smithi189 ceph-mon[36671]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T09:29:14.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:14 smithi189 ceph-mon[36671]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T09:29:14.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:14.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:14 smithi189 ceph-mon[36671]: from='osd.9 [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:29:14.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:14 smithi189 ceph-mon[36671]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-05-27T09:29:14.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:14.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:14.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:14.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:14.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:14.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:15.243 DEBUG:teuthology.orchestra.run.smithi044:osd.9> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.9.service 2024-05-27T09:29:15.254 INFO:tasks.cephadm:Deploying osd.10 on smithi050 with /dev/vg_nvme/lv_1... 2024-05-27T09:29:15.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T09:29:15.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:15 smithi044 ceph-mon[31618]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:29:15.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:15 smithi044 ceph-mon[31618]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T09:29:15.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:15.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:15.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:15 smithi044 ceph-mon[31618]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:29:15.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:15 smithi050 ceph-mon[36371]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:29:15.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:15 smithi050 ceph-mon[36371]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T09:29:15.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:15.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:15.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:15 smithi050 ceph-mon[36371]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:15.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:15 smithi189 ceph-mon[36671]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-05-27T09:29:15.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:15 smithi189 ceph-mon[36671]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T09:29:15.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:15.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:15.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:15 smithi189 ceph-mon[36671]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-05-27T09:29:16.608 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:16 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:29:16.608 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:16 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:29:16.608 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:16 smithi044 ceph-mon[31618]: osd.9 [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032] boot 2024-05-27T09:29:16.608 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:16 smithi044 ceph-mon[31618]: osdmap e59: 10 total, 10 up, 10 in 2024-05-27T09:29:16.608 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:16 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:16.648 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:16 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:29:16.649 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:16 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:29:16.649 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:16 smithi050 ceph-mon[36371]: osd.9 [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032] boot 2024-05-27T09:29:16.649 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:16 smithi050 ceph-mon[36371]: osdmap e59: 10 total, 10 up, 10 in 2024-05-27T09:29:16.649 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:16 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:16.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:16 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:29:16.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:16 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:29:16.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:16 smithi189 ceph-mon[36671]: osd.9 [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032] boot 2024-05-27T09:29:16.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:16 smithi189 ceph-mon[36671]: osdmap e59: 10 total, 10 up, 10 in 2024-05-27T09:29:16.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:16 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:29:17.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:17 smithi044 ceph-mon[31618]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T09:29:17.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:17 smithi044 ceph-mon[31618]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T09:29:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T09:29:17.521 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-05-27T09:29:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-05-27T09:29:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0135905 s, 772 MB/s 2024-05-27T09:29:17.522 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-05-27T09:29:17.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:17 smithi050 ceph-mon[36371]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T09:29:17.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:17 smithi050 ceph-mon[36371]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:17.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:17 smithi189 ceph-mon[36671]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T09:29:17.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:17 smithi189 ceph-mon[36671]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:18.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_1 2024-05-27T09:29:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:29:19.896 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:19 smithi050 ceph-mon[36371]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:19.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:19 smithi044 ceph-mon[31618]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:20.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:19 smithi189 ceph-mon[36671]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:20.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.115+0000 7f2139c6d700 1 -- 172.21.15.50:0/851792588 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2134104d10 msgr2=0x7f213410b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:29:20.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.115+0000 7f2139c6d700 1 --2- 172.21.15.50:0/851792588 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2134104d10 0x7f213410b5c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f212c00b0a0 tx=0x7f212c00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:29:20.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.115+0000 7f2139c6d700 1 -- 172.21.15.50:0/851792588 shutdown_connections 2024-05-27T09:29:20.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.115+0000 7f2139c6d700 1 --2- 172.21.15.50:0/851792588 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2134104d10 0x7f213410b5c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.115+0000 7f2139c6d700 1 --2- 172.21.15.50:0/851792588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2134103240 0x7f2134103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.115+0000 7f2139c6d700 1 --2- 172.21.15.50:0/851792588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2134104370 0x7f2134104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.115+0000 7f2139c6d700 1 -- 172.21.15.50:0/851792588 >> 172.21.15.50:0/851792588 conn(0x7f21340fe880 msgr2=0x7f2134100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:29:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.115+0000 7f2139c6d700 1 -- 172.21.15.50:0/851792588 shutdown_connections 2024-05-27T09:29:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.115+0000 7f2139c6d700 1 -- 172.21.15.50:0/851792588 wait complete. 2024-05-27T09:29:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2139c6d700 1 Processor -- start 2024-05-27T09:29:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2139c6d700 1 -- start start 2024-05-27T09:29:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2139c6d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2134103240 0x7f2134076440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:29:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2139c6d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2134104370 0x7f213419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:29:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2139c6d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2134104d10 0x7f21341a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:29:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2139c6d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f213410ed40 con 0x7f2134103240 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2139c6d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f213410ebc0 con 0x7f2134104d10 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2139c6d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f213410eec0 con 0x7f2134104370 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2133fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2134104d10 0x7f21341a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2133fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2134104d10 0x7f21341a1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55454/0 (socket says 172.21.15.50:55454) 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2133fff700 1 -- 172.21.15.50:0/2073209667 learned_addr learned my addr 172.21.15.50:0/2073209667 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.116+0000 7f2132ffd700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2134104370 0x7f213419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.117+0000 7f2133fff700 1 -- 172.21.15.50:0/2073209667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2134104370 msgr2=0x7f213419f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.117+0000 7f2133fff700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2134104370 0x7f213419f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.117+0000 7f2133fff700 1 -- 172.21.15.50:0/2073209667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2134103240 msgr2=0x7f2134076440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.117+0000 7f2133fff700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2134103240 0x7f2134076440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.117+0000 7f2133fff700 1 -- 172.21.15.50:0/2073209667 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21341a2490 con 0x7f2134104d10 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.117+0000 7f2133fff700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2134104d10 0x7f21341a1c60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f212c0062a0 tx=0x7f212c004e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:29:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.117+0000 7f2130ff9700 1 -- 172.21.15.50:0/2073209667 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c021070 con 0x7f2134104d10 2024-05-27T09:29:20.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.117+0000 7f2130ff9700 1 -- 172.21.15.50:0/2073209667 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f212c007c80 con 0x7f2134104d10 2024-05-27T09:29:20.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.117+0000 7f2139c6d700 1 -- 172.21.15.50:0/2073209667 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21341a2720 con 0x7f2134104d10 2024-05-27T09:29:20.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.117+0000 7f2130ff9700 1 -- 172.21.15.50:0/2073209667 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c004850 con 0x7f2134104d10 2024-05-27T09:29:20.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.117+0000 7f2139c6d700 1 -- 172.21.15.50:0/2073209667 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21341a2ce0 con 0x7f2134104d10 2024-05-27T09:29:20.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.118+0000 7f2139c6d700 1 -- 172.21.15.50:0/2073209667 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21340512b0 con 0x7f2134104d10 2024-05-27T09:29:20.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.120+0000 7f2130ff9700 1 -- 172.21.15.50:0/2073209667 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f212c003a00 con 0x7f2134104d10 2024-05-27T09:29:20.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.120+0000 7f2130ff9700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f211c0b2880 0x7f211c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:29:20.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.121+0000 7f2130ff9700 1 -- 172.21.15.50:0/2073209667 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6261+0+0 (secure 0 0 0) 0x7f212c00f020 con 0x7f2134104d10 2024-05-27T09:29:20.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.121+0000 7f21337fe700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f211c0b2880 0x7f211c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:29:20.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.122+0000 7f21337fe700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f211c0b2880 0x7f211c0b4d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2134104270 tx=0x7f2124009470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:29:20.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.124+0000 7f2130ff9700 1 -- 172.21.15.50:0/2073209667 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f212c02a020 con 0x7f2134104d10 2024-05-27T09:29:20.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:20.352+0000 7f2139c6d700 1 -- 172.21.15.50:0/2073209667 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2134063bf0 con 0x7f211c0b2880 2024-05-27T09:29:21.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi044 2024-05-27T09:29:21.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi044 to 4273M 2024-05-27T09:29:21.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:21.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:21.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:29:21.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:29:21.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:29:21.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:21.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:21 smithi050 ceph-mon[36371]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi044 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi044 to 4273M 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:29:21.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:29:21.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:29:21.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:21.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:21 smithi044 ceph-mon[31618]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:21.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi044 2024-05-27T09:29:21.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:21.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi044 to 4273M 2024-05-27T09:29:21.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:21.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:21.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:21.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:29:21.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:29:21.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:29:21.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:21.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:21 smithi189 ceph-mon[36671]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:23.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:23 smithi050 ceph-mon[36371]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:23.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:23 smithi050 ceph-mon[36371]: from='client.? 172.21.15.50:0/1241420339' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f094a1ca-1b0a-47a4-974e-5620cd898ea4"}]: dispatch 2024-05-27T09:29:23.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:23 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f094a1ca-1b0a-47a4-974e-5620cd898ea4"}]: dispatch 2024-05-27T09:29:23.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:23 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f094a1ca-1b0a-47a4-974e-5620cd898ea4"}]': finished 2024-05-27T09:29:23.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:23 smithi050 ceph-mon[36371]: osdmap e61: 11 total, 10 up, 11 in 2024-05-27T09:29:23.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:23.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:23 smithi044 ceph-mon[31618]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:23.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:23 smithi044 ceph-mon[31618]: from='client.? 172.21.15.50:0/1241420339' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f094a1ca-1b0a-47a4-974e-5620cd898ea4"}]: dispatch 2024-05-27T09:29:23.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:23 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f094a1ca-1b0a-47a4-974e-5620cd898ea4"}]: dispatch 2024-05-27T09:29:23.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:23 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f094a1ca-1b0a-47a4-974e-5620cd898ea4"}]': finished 2024-05-27T09:29:23.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:23 smithi044 ceph-mon[31618]: osdmap e61: 11 total, 10 up, 11 in 2024-05-27T09:29:23.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:24.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:23 smithi189 ceph-mon[36671]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:24.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:23 smithi189 ceph-mon[36671]: from='client.? 172.21.15.50:0/1241420339' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f094a1ca-1b0a-47a4-974e-5620cd898ea4"}]: dispatch 2024-05-27T09:29:24.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:23 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f094a1ca-1b0a-47a4-974e-5620cd898ea4"}]: dispatch 2024-05-27T09:29:24.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:23 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f094a1ca-1b0a-47a4-974e-5620cd898ea4"}]': finished 2024-05-27T09:29:24.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:23 smithi189 ceph-mon[36671]: osdmap e61: 11 total, 10 up, 11 in 2024-05-27T09:29:24.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:24.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:24 smithi044 ceph-mon[31618]: from='client.? 172.21.15.50:0/2555970702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:29:25.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:24 smithi050 ceph-mon[36371]: from='client.? 172.21.15.50:0/2555970702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:29:25.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:24 smithi189 ceph-mon[36671]: from='client.? 172.21.15.50:0/2555970702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:29:25.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:25 smithi044 ceph-mon[31618]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:25 smithi050 ceph-mon[36371]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:26.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:25 smithi189 ceph-mon[36671]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:27.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:27 smithi044 ceph-mon[31618]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:28.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:27 smithi050 ceph-mon[36371]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:28.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:27 smithi189 ceph-mon[36671]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:29.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:29 smithi050 ceph-mon[36371]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:29.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:29 smithi044 ceph-mon[31618]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:30.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:29 smithi189 ceph-mon[36671]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:30.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:30 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T09:29:30.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:30 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:31.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:30 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T09:29:31.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:30 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:31.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:30 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T09:29:31.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:30 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:31.793 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:31 smithi050 ceph-mon[36371]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:31.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:31 smithi050 ceph-mon[36371]: Deploying daemon osd.10 on smithi050 2024-05-27T09:29:31.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:31 smithi044 ceph-mon[31618]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:31.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:31 smithi044 ceph-mon[31618]: Deploying daemon osd.10 on smithi050 2024-05-27T09:29:32.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:31 smithi189 ceph-mon[36671]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:31 smithi189 ceph-mon[36671]: Deploying daemon osd.10 on smithi050 2024-05-27T09:29:33.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:33 smithi044 ceph-mon[31618]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:33.987 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:33 smithi050 ceph-mon[36371]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:34.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:33 smithi189 ceph-mon[36671]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:35.962 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:35 smithi050 ceph-mon[36371]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:35.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:35 smithi044 ceph-mon[31618]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:36.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:35 smithi189 ceph-mon[36671]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:37.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:37 smithi044 ceph-mon[31618]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:37 smithi050 ceph-mon[36371]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:37.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:37.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:37.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:37.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:37 smithi189 ceph-mon[36671]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:39.760 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:39 smithi050 ceph-mon[36371]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:39.760 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:39.760 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:39.760 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:39.760 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:39.760 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:39 smithi044 ceph-mon[31618]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:40.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:39 smithi189 ceph-mon[36671]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:40.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:40.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:40.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:40.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:40.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:40.089 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 10 on host 'smithi050' 2024-05-27T09:29:40.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.088+0000 7f2130ff9700 1 -- 172.21.15.50:0/2073209667 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2134063bf0 con 0x7f211c0b2880 2024-05-27T09:29:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.090+0000 7f2139c6d700 1 -- 172.21.15.50:0/2073209667 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f211c0b2880 msgr2=0x7f211c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:29:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.090+0000 7f2139c6d700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f211c0b2880 0x7f211c0b4d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2134104270 tx=0x7f2124009470 comp rx=0 tx=0).stop 2024-05-27T09:29:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.090+0000 7f2139c6d700 1 -- 172.21.15.50:0/2073209667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2134104d10 msgr2=0x7f21341a1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:29:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.090+0000 7f2139c6d700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2134104d10 0x7f21341a1c60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f212c0062a0 tx=0x7f212c004e40 comp rx=0 tx=0).stop 2024-05-27T09:29:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.091+0000 7f2139c6d700 1 -- 172.21.15.50:0/2073209667 shutdown_connections 2024-05-27T09:29:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.091+0000 7f2139c6d700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f211c0b2880 0x7f211c0b4d90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.091+0000 7f2139c6d700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2134104d10 0x7f21341a1c60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.091+0000 7f2139c6d700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2134103240 0x7f2134076440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:40.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.091+0000 7f2139c6d700 1 --2- 172.21.15.50:0/2073209667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2134104370 0x7f213419f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:40.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.091+0000 7f2139c6d700 1 -- 172.21.15.50:0/2073209667 >> 172.21.15.50:0/2073209667 conn(0x7f21340fe880 msgr2=0x7f21341075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:29:40.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.091+0000 7f2139c6d700 1 -- 172.21.15.50:0/2073209667 shutdown_connections 2024-05-27T09:29:40.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-27T09:29:40.091+0000 7f2139c6d700 1 -- 172.21.15.50:0/2073209667 wait complete. 2024-05-27T09:29:40.667 DEBUG:teuthology.orchestra.run.smithi050:osd.10> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.10.service 2024-05-27T09:29:40.669 INFO:tasks.cephadm:Deploying osd.11 on smithi189 with /dev/vg_nvme/lv_1... 2024-05-27T09:29:40.669 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T09:29:40.913 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:29:41.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:41.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:41.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:41.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:41 smithi050 ceph-mon[36371]: from='osd.10 [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T09:29:41.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:41 smithi050 ceph-mon[36371]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T09:29:41.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:41 smithi050 ceph-mon[36371]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:41.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:41.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:41.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:41.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:41 smithi189 ceph-mon[36671]: from='osd.10 [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T09:29:41.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:41 smithi189 ceph-mon[36671]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T09:29:41.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:41 smithi189 ceph-mon[36671]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:41.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:41.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:29:41.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:41.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:41 smithi044 ceph-mon[31618]: from='osd.10 [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T09:29:41.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:41 smithi044 ceph-mon[31618]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T09:29:41.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:41 smithi044 ceph-mon[31618]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:42.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:42 smithi050 ceph-mon[36371]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T09:29:42.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:42 smithi050 ceph-mon[36371]: from='osd.10 [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:29:42.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:42 smithi050 ceph-mon[36371]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T09:29:42.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:42 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:42.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:42 smithi050 ceph-mon[36371]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:29:42.275 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:42 smithi189 ceph-mon[36671]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T09:29:42.275 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:42 smithi189 ceph-mon[36671]: from='osd.10 [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:29:42.275 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:42 smithi189 ceph-mon[36671]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T09:29:42.275 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:42 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:42.275 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:42 smithi189 ceph-mon[36671]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:29:42.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:42 smithi044 ceph-mon[31618]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T09:29:42.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:42 smithi044 ceph-mon[31618]: from='osd.10 [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:29:42.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:42 smithi044 ceph-mon[31618]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T09:29:42.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:42 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:42.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:42 smithi044 ceph-mon[31618]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-05-27T09:29:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T09:29:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T09:29:42.830 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-05-27T09:29:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-05-27T09:29:42.830 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113662 s, 923 MB/s 2024-05-27T09:29:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-05-27T09:29:43.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:43 smithi050 ceph-mon[36371]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:29:43.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:43 smithi050 ceph-mon[36371]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T09:29:43.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:43.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:43 smithi050 ceph-mon[36371]: osdmap e64: 11 total, 10 up, 11 in 2024-05-27T09:29:43.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:43.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:43 smithi050 ceph-mon[36371]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:43.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:43 smithi050 ceph-mon[36371]: from='osd.10 ' entity='osd.10' 2024-05-27T09:29:43.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:43.295 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:29:42 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:29:42.865+0000 7fc4813ef700 -1 osd.10 0 waiting for initial osdmap 2024-05-27T09:29:43.295 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:29:42 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:29:42.881+0000 7fc47919c700 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T09:29:43.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:43 smithi189 ceph-mon[36671]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:29:43.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:43 smithi189 ceph-mon[36671]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T09:29:43.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:43.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:43 smithi189 ceph-mon[36671]: osdmap e64: 11 total, 10 up, 11 in 2024-05-27T09:29:43.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:43.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:43 smithi189 ceph-mon[36671]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:43.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:43 smithi189 ceph-mon[36671]: from='osd.10 ' entity='osd.10' 2024-05-27T09:29:43.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:43.440 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_1 2024-05-27T09:29:43.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:43 smithi044 ceph-mon[31618]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-05-27T09:29:43.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:43 smithi044 ceph-mon[31618]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T09:29:43.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:43.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:43 smithi044 ceph-mon[31618]: osdmap e64: 11 total, 10 up, 11 in 2024-05-27T09:29:43.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:43.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:43 smithi044 ceph-mon[31618]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T09:29:43.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:43 smithi044 ceph-mon[31618]: from='osd.10 ' entity='osd.10' 2024-05-27T09:29:43.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:43.682 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:29:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:29:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:29:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: osd.10 [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028] boot 2024-05-27T09:29:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T09:29:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi050 2024-05-27T09:29:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi050 to 4273M 2024-05-27T09:29:44.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:44.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:44.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:44.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:44.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:29:44.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:29:44.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: osd.10 [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028] boot 2024-05-27T09:29:44.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T09:29:44.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:44.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi050 2024-05-27T09:29:44.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:44.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:44.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi050 to 4273M 2024-05-27T09:29:44.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:44.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:44.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:44.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:29:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:29:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: osd.10 [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028] boot 2024-05-27T09:29:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T09:29:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:29:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi050 2024-05-27T09:29:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:44.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:44.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:29:44.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi050 to 4273M 2024-05-27T09:29:44.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:44.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:44.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:29:44.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:29:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.198+0000 7f7d0f248700 1 -- 172.21.15.189:0/2952909262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d08104370 msgr2=0x7f7d08104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:29:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.198+0000 7f7d0f248700 1 --2- 172.21.15.189:0/2952909262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d08104370 0x7f7d08104780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009990 tx=0x7f7cf4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:29:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.198+0000 7f7d0f248700 1 -- 172.21.15.189:0/2952909262 shutdown_connections 2024-05-27T09:29:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.198+0000 7f7d0f248700 1 --2- 172.21.15.189:0/2952909262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d08104d10 0x7f7d0810b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.198+0000 7f7d0f248700 1 --2- 172.21.15.189:0/2952909262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d08103240 0x7f7d08103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.198+0000 7f7d0f248700 1 --2- 172.21.15.189:0/2952909262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d08104370 0x7f7d08104780 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.198+0000 7f7d0f248700 1 -- 172.21.15.189:0/2952909262 >> 172.21.15.189:0/2952909262 conn(0x7f7d080fe880 msgr2=0x7f7d08100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:29:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.198+0000 7f7d0f248700 1 -- 172.21.15.189:0/2952909262 shutdown_connections 2024-05-27T09:29:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.198+0000 7f7d0f248700 1 -- 172.21.15.189:0/2952909262 wait complete. 2024-05-27T09:29:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d0f248700 1 Processor -- start 2024-05-27T09:29:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d0f248700 1 -- start start 2024-05-27T09:29:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d0f248700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d08103240 0x7f7d0810de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:29:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d0f248700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d08104370 0x7f7d0810c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:29:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d0f248700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d08104d10 0x7f7d0810ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:29:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d0f248700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0810fe20 con 0x7f7d08104d10 2024-05-27T09:29:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d0f248700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0810fca0 con 0x7f7d08103240 2024-05-27T09:29:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d0f248700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0810ffa0 con 0x7f7d08104370 2024-05-27T09:29:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d08104370 0x7f7d0810c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:29:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d08104370 0x7f7d0810c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52056/0 (socket says 172.21.15.189:52056) 2024-05-27T09:29:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d07fff700 1 -- 172.21.15.189:0/2226877269 learned_addr learned my addr 172.21.15.189:0/2226877269 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-27T09:29:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d0cfe4700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d08103240 0x7f7d0810de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d07fff700 1 -- 172.21.15.189:0/2226877269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d08103240 msgr2=0x7f7d0810de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d07fff700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d08103240 0x7f7d0810de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d07fff700 1 -- 172.21.15.189:0/2226877269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d08104d10 msgr2=0x7f7d0810ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d07fff700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d08104d10 0x7f7d0810ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.199+0000 7f7d07fff700 1 -- 172.21.15.189:0/2226877269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0810d160 con 0x7f7d08104370 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.200+0000 7f7d07fff700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d08104370 0x7f7d0810c4c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4006010 tx=0x7f7cf40047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.200+0000 7f7d05ffb700 1 -- 172.21.15.189:0/2226877269 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4027070 con 0x7f7d08104370 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.200+0000 7f7d05ffb700 1 -- 172.21.15.189:0/2226877269 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cf4004140 con 0x7f7d08104370 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.200+0000 7f7d05ffb700 1 -- 172.21.15.189:0/2226877269 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf400f460 con 0x7f7d08104370 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.200+0000 7f7d0f248700 1 -- 172.21.15.189:0/2226877269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d081a4a30 con 0x7f7d08104370 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.200+0000 7f7d0f248700 1 -- 172.21.15.189:0/2226877269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d081a4dd0 con 0x7f7d08104370 2024-05-27T09:29:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.201+0000 7f7d0f248700 1 -- 172.21.15.189:0/2226877269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d080512f0 con 0x7f7d08104370 2024-05-27T09:29:45.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.203+0000 7f7d05ffb700 1 -- 172.21.15.189:0/2226877269 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f7cf402a070 con 0x7f7d08104370 2024-05-27T09:29:45.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.203+0000 7f7d05ffb700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f7cf80b2880 0x7f7cf80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:29:45.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.203+0000 7f7d05ffb700 1 -- 172.21.15.189:0/2226877269 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6694+0+0 (secure 0 0 0) 0x7f7cf40e2e10 con 0x7f7d08104370 2024-05-27T09:29:45.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.204+0000 7f7d0cfe4700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f7cf80b2880 0x7f7cf80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:29:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.205+0000 7f7d0cfe4700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f7cf80b2880 0x7f7cf80b4d90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc005950 tx=0x7f7cfc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:29:45.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.207+0000 7f7d05ffb700 1 -- 172.21.15.189:0/2226877269 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7cf4013070 con 0x7f7d08104370 2024-05-27T09:29:45.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:29:45.421+0000 7f7d0f248700 1 -- 172.21.15.189:0/2226877269 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7d08063bf0 con 0x7f7cf80b2880 2024-05-27T09:29:45.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:45 smithi044 ceph-mon[31618]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T09:29:45.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:45 smithi044 ceph-mon[31618]: pgmap v260: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:29:45.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:45 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:29:45.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:45 smithi050 ceph-mon[36371]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T09:29:45.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:45 smithi050 ceph-mon[36371]: pgmap v260: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:29:45.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:45 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:29:45.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:45 smithi189 ceph-mon[36671]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T09:29:45.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:45 smithi189 ceph-mon[36671]: pgmap v260: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:29:45.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:45 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T09:29:46.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:46 smithi044 ceph-mon[31618]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:29:46.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:29:46.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:46.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:46 smithi044 ceph-mon[31618]: osdmap e67: 11 total, 11 up, 11 in 2024-05-27T09:29:46.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:46 smithi050 ceph-mon[36371]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:29:46.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:29:46.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:46.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:46 smithi050 ceph-mon[36371]: osdmap e67: 11 total, 11 up, 11 in 2024-05-27T09:29:46.803 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:46 smithi189 ceph-mon[36671]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:29:46.803 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T09:29:46.803 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:46.803 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:46 smithi189 ceph-mon[36671]: osdmap e67: 11 total, 11 up, 11 in 2024-05-27T09:29:47.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:47 smithi189 ceph-mon[36671]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:29:47.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:47 smithi044 ceph-mon[31618]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:29:47.793 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:47 smithi050 ceph-mon[36371]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:29:48.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:48 smithi044 ceph-mon[31618]: from='client.? 172.21.15.189:0/2721372711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b9794fa-856f-4343-945e-4be444c0cd9b"}]: dispatch 2024-05-27T09:29:48.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:48 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b9794fa-856f-4343-945e-4be444c0cd9b"}]: dispatch 2024-05-27T09:29:48.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:48 smithi044 ceph-mon[31618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b9794fa-856f-4343-945e-4be444c0cd9b"}]': finished 2024-05-27T09:29:48.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:48 smithi044 ceph-mon[31618]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T09:29:48.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:48 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:29:48.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:48 smithi050 ceph-mon[36371]: from='client.? 172.21.15.189:0/2721372711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b9794fa-856f-4343-945e-4be444c0cd9b"}]: dispatch 2024-05-27T09:29:48.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:48 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b9794fa-856f-4343-945e-4be444c0cd9b"}]: dispatch 2024-05-27T09:29:48.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:48 smithi050 ceph-mon[36371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b9794fa-856f-4343-945e-4be444c0cd9b"}]': finished 2024-05-27T09:29:48.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:48 smithi050 ceph-mon[36371]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T09:29:48.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:48 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:29:48.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:48 smithi189 ceph-mon[36671]: from='client.? 172.21.15.189:0/2721372711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b9794fa-856f-4343-945e-4be444c0cd9b"}]: dispatch 2024-05-27T09:29:48.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:48 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b9794fa-856f-4343-945e-4be444c0cd9b"}]: dispatch 2024-05-27T09:29:48.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:48 smithi189 ceph-mon[36671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b9794fa-856f-4343-945e-4be444c0cd9b"}]': finished 2024-05-27T09:29:48.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:48 smithi189 ceph-mon[36671]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T09:29:48.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:48 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:29:49.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:49 smithi050 ceph-mon[36371]: pgmap v264: 1 pgs: 1 peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:29:49.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:49 smithi050 ceph-mon[36371]: from='client.? 172.21.15.189:0/3570174926' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:29:49.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:49 smithi189 ceph-mon[36671]: pgmap v264: 1 pgs: 1 peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:29:49.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:49 smithi189 ceph-mon[36671]: from='client.? 172.21.15.189:0/3570174926' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:29:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:49 smithi044 ceph-mon[31618]: pgmap v264: 1 pgs: 1 peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:29:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:49 smithi044 ceph-mon[31618]: from='client.? 172.21.15.189:0/3570174926' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T09:29:51.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:51 smithi044 ceph-mon[31618]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-05-27T09:29:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:51 smithi050 ceph-mon[36371]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-05-27T09:29:52.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:51 smithi189 ceph-mon[36671]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-05-27T09:29:53.781 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:53 smithi189 ceph-mon[36671]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T09:29:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:53 smithi044 ceph-mon[31618]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T09:29:54.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:53 smithi050 ceph-mon[36371]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T09:29:55.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:55 smithi189 ceph-mon[36671]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-27T09:29:55.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:55 smithi044 ceph-mon[31618]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-27T09:29:56.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:55 smithi050 ceph-mon[36371]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-27T09:29:56.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:56 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T09:29:56.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:56 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:56.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:56 smithi189 ceph-mon[36671]: Deploying daemon osd.11 on smithi189 2024-05-27T09:29:56.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T09:29:56.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:56.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:56 smithi044 ceph-mon[31618]: Deploying daemon osd.11 on smithi189 2024-05-27T09:29:57.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:56 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T09:29:57.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:56 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:29:57.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:56 smithi050 ceph-mon[36371]: Deploying daemon osd.11 on smithi189 2024-05-27T09:29:57.817 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:57 smithi189 ceph-mon[36671]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:29:57.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:57 smithi044 ceph-mon[31618]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:29:58.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:57 smithi050 ceph-mon[36371]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T09:29:59.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:29:59 smithi189 ceph-mon[36671]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-27T09:29:59.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:29:59 smithi044 ceph-mon[31618]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-27T09:30:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:29:59 smithi050 ceph-mon[36371]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-27T09:30:00.936 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:00 smithi189 ceph-mon[36671]: overall HEALTH_OK 2024-05-27T09:30:00.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:00 smithi044 ceph-mon[31618]: overall HEALTH_OK 2024-05-27T09:30:01.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:00 smithi050 ceph-mon[36371]: overall HEALTH_OK 2024-05-27T09:30:01.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:01 smithi189 ceph-mon[36671]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-27T09:30:01.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:01 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:01.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:01 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:01.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:01 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:30:01.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:01 smithi044 ceph-mon[31618]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-27T09:30:01.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:01 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:01.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:01 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:01.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:01 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:30:02.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:01 smithi050 ceph-mon[36371]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-27T09:30:02.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:01 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:02.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:01 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:02.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:01 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:30:03.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:03 smithi189 ceph-mon[36671]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:03.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:03 smithi044 ceph-mon[31618]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:04.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:03 smithi050 ceph-mon[36371]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:04.864 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 11 on host 'smithi189' 2024-05-27T09:30:04.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.863+0000 7f7d05ffb700 1 -- 172.21.15.189:0/2226877269 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f7d08063bf0 con 0x7f7cf80b2880 2024-05-27T09:30:04.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 -- 172.21.15.189:0/2226877269 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f7cf80b2880 msgr2=0x7f7cf80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:04.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f7cf80b2880 0x7f7cf80b4d90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc005950 tx=0x7f7cfc0058e0 comp rx=0 tx=0).stop 2024-05-27T09:30:04.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 -- 172.21.15.189:0/2226877269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d08104370 msgr2=0x7f7d0810c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:04.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d08104370 0x7f7d0810c4c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4006010 tx=0x7f7cf40047b0 comp rx=0 tx=0).stop 2024-05-27T09:30:04.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 -- 172.21.15.189:0/2226877269 shutdown_connections 2024-05-27T09:30:04.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f7cf80b2880 0x7f7cf80b4d90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:04.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7d08104d10 0x7f7d0810ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:04.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d08103240 0x7f7d0810de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:04.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 --2- 172.21.15.189:0/2226877269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d08104370 0x7f7d0810c4c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:04.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 -- 172.21.15.189:0/2226877269 >> 172.21.15.189:0/2226877269 conn(0x7f7d080fe880 msgr2=0x7f7d081075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:04.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 -- 172.21.15.189:0/2226877269 shutdown_connections 2024-05-27T09:30:04.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:04.865+0000 7f7d0f248700 1 -- 172.21.15.189:0/2226877269 wait complete. 2024-05-27T09:30:05.075 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:05 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.076 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:05 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.076 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:05 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:05.076 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:05 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:30:05.076 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:05 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.076 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:05 smithi189 ceph-mon[36671]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:05.076 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:05 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.076 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:05 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:30:05.076 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:05 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:05 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:05 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:05 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:05.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:05 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:30:05.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:05 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:05 smithi050 ceph-mon[36371]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:05.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:05 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:05 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:30:05.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:05 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.473 DEBUG:teuthology.orchestra.run.smithi189:osd.11> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.11.service 2024-05-27T09:30:05.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:05 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:05 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:05 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:05.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:05 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:30:05.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:05 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:05 smithi044 ceph-mon[31618]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:05.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:05 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:05 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:30:05.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:05 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:05.477 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-27T09:30:05.478 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-05-27T09:30:05.946 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:30:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:06 smithi189 ceph-mon[36671]: from='osd.11 [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T09:30:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:06 smithi189 ceph-mon[36671]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T09:30:06.340 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:06 smithi044 ceph-mon[31618]: from='osd.11 [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T09:30:06.340 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:06 smithi044 ceph-mon[31618]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T09:30:06.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:06 smithi050 ceph-mon[36371]: from='osd.11 [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T09:30:06.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:06 smithi050 ceph-mon[36371]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T09:30:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:07 smithi044 ceph-mon[31618]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T09:30:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:07 smithi044 ceph-mon[31618]: from='osd.11 [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:30:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:07 smithi044 ceph-mon[31618]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T09:30:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:07 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:07 smithi044 ceph-mon[31618]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:30:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:07 smithi044 ceph-mon[31618]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:07 smithi189 ceph-mon[36671]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T09:30:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:07 smithi189 ceph-mon[36671]: from='osd.11 [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:30:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:07 smithi189 ceph-mon[36671]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T09:30:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:07 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:07 smithi189 ceph-mon[36671]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:30:07.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:07 smithi189 ceph-mon[36671]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:07.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:07 smithi050 ceph-mon[36371]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T09:30:07.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:07 smithi050 ceph-mon[36371]: from='osd.11 [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:30:07.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:07 smithi050 ceph-mon[36371]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T09:30:07.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:07 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:07.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:07 smithi050 ceph-mon[36371]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-05-27T09:30:07.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:07 smithi050 ceph-mon[36371]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:08.109 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:30:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:30:08.005+0000 7ffa80b0e700 -1 osd.11 0 waiting for initial osdmap 2024-05-27T09:30:08.109 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:30:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:30:08.064+0000 7ffa788bb700 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T09:30:08.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.191+0000 7ffa43c00700 1 -- 172.21.15.44:0/858477569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3c10c210 msgr2=0x7ffa3c10e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:08.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.191+0000 7ffa43c00700 1 --2- 172.21.15.44:0/858477569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3c10c210 0x7ffa3c10e6f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400b0a0 tx=0x7ffa3400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:30:08.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.191+0000 7ffa43c00700 1 -- 172.21.15.44:0/858477569 shutdown_connections 2024-05-27T09:30:08.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.191+0000 7ffa43c00700 1 --2- 172.21.15.44:0/858477569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3c10c210 0x7ffa3c10e6f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:08.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.191+0000 7ffa43c00700 1 --2- 172.21.15.44:0/858477569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa3c077650 0x7ffa3c075b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:08.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.191+0000 7ffa43c00700 1 --2- 172.21.15.44:0/858477569 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa3c076060 0x7ffa3c076470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:08.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.191+0000 7ffa43c00700 1 -- 172.21.15.44:0/858477569 >> 172.21.15.44:0/858477569 conn(0x7ffa3c0ff9b0 msgr2=0x7ffa3c101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:08.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.191+0000 7ffa43c00700 1 -- 172.21.15.44:0/858477569 shutdown_connections 2024-05-27T09:30:08.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.191+0000 7ffa43c00700 1 -- 172.21.15.44:0/858477569 wait complete. 2024-05-27T09:30:08.194 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:08 smithi044 ceph-mon[31618]: purged_snaps scrub starts 2024-05-27T09:30:08.194 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:08 smithi044 ceph-mon[31618]: purged_snaps scrub ok 2024-05-27T09:30:08.194 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:08 smithi044 ceph-mon[31618]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:30:08.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:08 smithi044 ceph-mon[31618]: osdmap e70: 12 total, 11 up, 12 in 2024-05-27T09:30:08.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:08.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:08 smithi044 ceph-mon[31618]: from='osd.11 ' entity='osd.11' 2024-05-27T09:30:08.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:08.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.193+0000 7ffa43c00700 1 Processor -- start 2024-05-27T09:30:08.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.193+0000 7ffa43c00700 1 -- start start 2024-05-27T09:30:08.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.194+0000 7ffa43c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3c076060 0x7ffa3c19cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:08.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.194+0000 7ffa43c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa3c077650 0x7ffa3c19d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:08.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.194+0000 7ffa43c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa3c10c210 0x7ffa3c1a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:08.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.194+0000 7ffa43c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c111f90 con 0x7ffa3c076060 2024-05-27T09:30:08.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.194+0000 7ffa43c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c111e10 con 0x7ffa3c10c210 2024-05-27T09:30:08.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.194+0000 7ffa43c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c112110 con 0x7ffa3c077650 2024-05-27T09:30:08.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.194+0000 7ffa41200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3c076060 0x7ffa3c19cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:08.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.194+0000 7ffa41200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3c076060 0x7ffa3c19cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49530/0 (socket says 172.21.15.44:49530) 2024-05-27T09:30:08.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.194+0000 7ffa41200700 1 -- 172.21.15.44:0/1214188645 learned_addr learned my addr 172.21.15.44:0/1214188645 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:30:08.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.194+0000 7ffa33e00700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa3c077650 0x7ffa3c19d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:08.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.194+0000 7ffa41c00700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa3c10c210 0x7ffa3c1a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:08.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.195+0000 7ffa41200700 1 -- 172.21.15.44:0/1214188645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa3c077650 msgr2=0x7ffa3c19d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:08.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.195+0000 7ffa41200700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa3c077650 0x7ffa3c19d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:08.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.195+0000 7ffa41200700 1 -- 172.21.15.44:0/1214188645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa3c10c210 msgr2=0x7ffa3c1a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:08.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.195+0000 7ffa41200700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa3c10c210 0x7ffa3c1a3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:08.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.195+0000 7ffa41200700 1 -- 172.21.15.44:0/1214188645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c1a45f0 con 0x7ffa3c076060 2024-05-27T09:30:08.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.195+0000 7ffa41200700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3c076060 0x7ffa3c19cb00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00e970 tx=0x7ffa2c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:08.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.196+0000 7ffa3a000700 1 -- 172.21.15.44:0/1214188645 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c004a30 con 0x7ffa3c076060 2024-05-27T09:30:08.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.196+0000 7ffa3a000700 1 -- 172.21.15.44:0/1214188645 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa2c00cc90 con 0x7ffa3c076060 2024-05-27T09:30:08.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.196+0000 7ffa43c00700 1 -- 172.21.15.44:0/1214188645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa3c1a48e0 con 0x7ffa3c076060 2024-05-27T09:30:08.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.196+0000 7ffa3a000700 1 -- 172.21.15.44:0/1214188645 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c005610 con 0x7ffa3c076060 2024-05-27T09:30:08.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.196+0000 7ffa43c00700 1 -- 172.21.15.44:0/1214188645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa3c1a4e70 con 0x7ffa3c076060 2024-05-27T09:30:08.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.197+0000 7ffa43c00700 1 -- 172.21.15.44:0/1214188645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa3c0512b0 con 0x7ffa3c076060 2024-05-27T09:30:08.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.200+0000 7ffa3a000700 1 -- 172.21.15.44:0/1214188645 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ffa2c00c7e0 con 0x7ffa3c076060 2024-05-27T09:30:08.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.201+0000 7ffa3a000700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa1c0b2900 0x7ffa1c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:08.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.202+0000 7ffa3a000700 1 -- 172.21.15.44:0/1214188645 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6847+0+0 (secure 0 0 0) 0x7ffa2c0e2000 con 0x7ffa3c076060 2024-05-27T09:30:08.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.202+0000 7ffa33e00700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa1c0b2900 0x7ffa1c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:08.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.203+0000 7ffa33e00700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa1c0b2900 0x7ffa1c0b4e10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c19dbb0 tx=0x7ffa24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:08.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.204+0000 7ffa3a000700 1 -- 172.21.15.44:0/1214188645 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ffa2c054df0 con 0x7ffa3c076060 2024-05-27T09:30:08.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.443+0000 7ffa43c00700 1 -- 172.21.15.44:0/1214188645 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7ffa3c02e5e0 con 0x7ffa3c076060 2024-05-27T09:30:08.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.444+0000 7ffa3a000700 1 -- 172.21.15.44:0/1214188645 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) ==== 74+0+133 (secure 0 0 0) 0x7ffa2c0aa240 con 0x7ffa3c076060 2024-05-27T09:30:08.445 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:30:08.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.447+0000 7ffa43c00700 1 -- 172.21.15.44:0/1214188645 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa1c0b2900 msgr2=0x7ffa1c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:08.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.447+0000 7ffa43c00700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa1c0b2900 0x7ffa1c0b4e10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c19dbb0 tx=0x7ffa24009260 comp rx=0 tx=0).stop 2024-05-27T09:30:08.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.447+0000 7ffa43c00700 1 -- 172.21.15.44:0/1214188645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3c076060 msgr2=0x7ffa3c19cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:08.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.447+0000 7ffa43c00700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3c076060 0x7ffa3c19cb00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00e970 tx=0x7ffa2c00ec80 comp rx=0 tx=0).stop 2024-05-27T09:30:08.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.447+0000 7ffa43c00700 1 -- 172.21.15.44:0/1214188645 shutdown_connections 2024-05-27T09:30:08.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.447+0000 7ffa43c00700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ffa1c0b2900 0x7ffa1c0b4e10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:08.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.447+0000 7ffa43c00700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa3c10c210 0x7ffa3c1a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:08.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.447+0000 7ffa43c00700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa3c076060 0x7ffa3c19cb00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:08.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.447+0000 7ffa43c00700 1 --2- 172.21.15.44:0/1214188645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa3c077650 0x7ffa3c19d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:08.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.447+0000 7ffa43c00700 1 -- 172.21.15.44:0/1214188645 >> 172.21.15.44:0/1214188645 conn(0x7ffa3c0ff9b0 msgr2=0x7ffa3c1081f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:08.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.447+0000 7ffa43c00700 1 -- 172.21.15.44:0/1214188645 shutdown_connections 2024-05-27T09:30:08.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:08.448+0000 7ffa43c00700 1 -- 172.21.15.44:0/1214188645 wait complete. 2024-05-27T09:30:08.453 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:08 smithi189 ceph-mon[36671]: purged_snaps scrub starts 2024-05-27T09:30:08.453 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:08 smithi189 ceph-mon[36671]: purged_snaps scrub ok 2024-05-27T09:30:08.453 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:08 smithi189 ceph-mon[36671]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:30:08.454 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:08 smithi189 ceph-mon[36671]: osdmap e70: 12 total, 11 up, 12 in 2024-05-27T09:30:08.454 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:08.454 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:08 smithi189 ceph-mon[36671]: from='osd.11 ' entity='osd.11' 2024-05-27T09:30:08.454 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:08 smithi050 ceph-mon[36371]: purged_snaps scrub starts 2024-05-27T09:30:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:08 smithi050 ceph-mon[36371]: purged_snaps scrub ok 2024-05-27T09:30:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:08 smithi050 ceph-mon[36371]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-05-27T09:30:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:08 smithi050 ceph-mon[36371]: osdmap e70: 12 total, 11 up, 12 in 2024-05-27T09:30:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:08 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:08 smithi050 ceph-mon[36371]: from='osd.11 ' entity='osd.11' 2024-05-27T09:30:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:08 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:09.270 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":70,"num_osds":12,"num_up_osds":11,"osd_up_since":1716802183,"num_in_osds":12,"osd_in_since":1716802188,"num_remapped_pgs":0} 2024-05-27T09:30:09.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1214188645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T09:30:09.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: Detected new or changed devices on smithi189 2024-05-27T09:30:09.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi189 to 4272M 2024-05-27T09:30:09.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:09.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:30:09.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:09.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: osd.11 [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633] boot 2024-05-27T09:30:09.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T09:30:09.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1214188645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T09:30:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: Detected new or changed devices on smithi189 2024-05-27T09:30:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi189 to 4272M 2024-05-27T09:30:09.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:09.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:30:09.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:09.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: osd.11 [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633] boot 2024-05-27T09:30:09.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T09:30:09.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:09.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1214188645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T09:30:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: Detected new or changed devices on smithi189 2024-05-27T09:30:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:30:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi189 to 4272M 2024-05-27T09:30:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:30:09.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:09.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-05-27T09:30:09.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: osd.11 [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633] boot 2024-05-27T09:30:09.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T09:30:09.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:30:10.272 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-05-27T09:30:10.764 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:30:11.203 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:11 smithi044 ceph-mon[31618]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T09:30:11.203 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:11 smithi044 ceph-mon[31618]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:11.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:11 smithi050 ceph-mon[36371]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T09:30:11.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:11 smithi050 ceph-mon[36371]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:11.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:11 smithi189 ceph-mon[36671]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T09:30:11.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:11 smithi189 ceph-mon[36671]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:13.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.373+0000 7ff162c00700 1 -- 172.21.15.44:0/3113787422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff15c104c40 msgr2=0x7ff15c107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:13.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.373+0000 7ff162c00700 1 --2- 172.21.15.44:0/3113787422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff15c104c40 0x7ff15c107060 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff15000b0a0 tx=0x7ff15000b3b0 comp rx=0 tx=0).stop 2024-05-27T09:30:13.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.373+0000 7ff162c00700 1 -- 172.21.15.44:0/3113787422 shutdown_connections 2024-05-27T09:30:13.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.373+0000 7ff162c00700 1 --2- 172.21.15.44:0/3113787422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15c109f00 0x7ff15c10c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:13.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.373+0000 7ff162c00700 1 --2- 172.21.15.44:0/3113787422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff15c104c40 0x7ff15c107060 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:13.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.373+0000 7ff162c00700 1 --2- 172.21.15.44:0/3113787422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c1075a0 0x7ff15c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:13.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.373+0000 7ff162c00700 1 -- 172.21.15.44:0/3113787422 >> 172.21.15.44:0/3113787422 conn(0x7ff15c0fe680 msgr2=0x7ff15c100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:13.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.373+0000 7ff162c00700 1 -- 172.21.15.44:0/3113787422 shutdown_connections 2024-05-27T09:30:13.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.373+0000 7ff162c00700 1 -- 172.21.15.44:0/3113787422 wait complete. 2024-05-27T09:30:13.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.374+0000 7ff162c00700 1 Processor -- start 2024-05-27T09:30:13.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.374+0000 7ff162c00700 1 -- start start 2024-05-27T09:30:13.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.375+0000 7ff162c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c104c40 0x7ff15c1044a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:13.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.375+0000 7ff162c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff15c1075a0 0x7ff15c102b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:13.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.375+0000 7ff162c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15c109f00 0x7ff15c103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.375+0000 7ff162c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c10fbd0 con 0x7ff15c1075a0 2024-05-27T09:30:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.375+0000 7ff162c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c10fa50 con 0x7ff15c104c40 2024-05-27T09:30:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.375+0000 7ff162c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c10fd50 con 0x7ff15c109f00 2024-05-27T09:30:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.375+0000 7ff162200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15c109f00 0x7ff15c103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.375+0000 7ff162200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15c109f00 0x7ff15c103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43522/0 (socket says 172.21.15.44:43522) 2024-05-27T09:30:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.375+0000 7ff162200700 1 -- 172.21.15.44:0/2533790251 learned_addr learned my addr 172.21.15.44:0/2533790251 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:30:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.375+0000 7ff161800700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c104c40 0x7ff15c1044a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff160e00700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff15c1075a0 0x7ff15c102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff162200700 1 -- 172.21.15.44:0/2533790251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c104c40 msgr2=0x7ff15c1044a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff162200700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c104c40 0x7ff15c1044a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff162200700 1 -- 172.21.15.44:0/2533790251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff15c1075a0 msgr2=0x7ff15c102b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff162200700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff15c1075a0 0x7ff15c102b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff162200700 1 -- 172.21.15.44:0/2533790251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15c1037f0 con 0x7ff15c109f00 2024-05-27T09:30:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff160e00700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff15c1075a0 0x7ff15c102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:30:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff162200700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15c109f00 0x7ff15c103090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff15400ea80 tx=0x7ff15400ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff15a000700 1 -- 172.21.15.44:0/2533790251 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154004220 con 0x7ff15c109f00 2024-05-27T09:30:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff15a000700 1 -- 172.21.15.44:0/2533790251 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff15400ce00 con 0x7ff15c109f00 2024-05-27T09:30:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff162c00700 1 -- 172.21.15.44:0/2533790251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff15c1a4880 con 0x7ff15c109f00 2024-05-27T09:30:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.376+0000 7ff15a000700 1 -- 172.21.15.44:0/2533790251 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154003dd0 con 0x7ff15c109f00 2024-05-27T09:30:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.377+0000 7ff162c00700 1 -- 172.21.15.44:0/2533790251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15c1a4d60 con 0x7ff15c109f00 2024-05-27T09:30:13.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.378+0000 7ff162c00700 1 -- 172.21.15.44:0/2533790251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff15c0512b0 con 0x7ff15c109f00 2024-05-27T09:30:13.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.380+0000 7ff15a000700 1 -- 172.21.15.44:0/2533790251 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ff15400c950 con 0x7ff15c109f00 2024-05-27T09:30:13.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.381+0000 7ff15a000700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff1440b2900 0x7ff1440b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:13.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.382+0000 7ff15a000700 1 -- 172.21.15.44:0/2533790251 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ff1540e1f70 con 0x7ff15c109f00 2024-05-27T09:30:13.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.382+0000 7ff161800700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff1440b2900 0x7ff1440b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:13.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.382+0000 7ff161800700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff1440b2900 0x7ff1440b4e10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff150009fd0 tx=0x7ff15000b700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:13.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.384+0000 7ff15a000700 1 -- 172.21.15.44:0/2533790251 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff1540aa090 con 0x7ff15c109f00 2024-05-27T09:30:13.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.622+0000 7ff162c00700 1 -- 172.21.15.44:0/2533790251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7ff15c02e5e0 con 0x7ff15c109f00 2024-05-27T09:30:13.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.623+0000 7ff15a000700 1 -- 172.21.15.44:0/2533790251 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7ff1540a9990 con 0x7ff15c109f00 2024-05-27T09:30:13.623 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:30:13.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 -- 172.21.15.44:0/2533790251 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff1440b2900 msgr2=0x7ff1440b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:13.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff1440b2900 0x7ff1440b4e10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff150009fd0 tx=0x7ff15000b700 comp rx=0 tx=0).stop 2024-05-27T09:30:13.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 -- 172.21.15.44:0/2533790251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15c109f00 msgr2=0x7ff15c103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:13.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15c109f00 0x7ff15c103090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff15400ea80 tx=0x7ff15400ed90 comp rx=0 tx=0).stop 2024-05-27T09:30:13.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 -- 172.21.15.44:0/2533790251 shutdown_connections 2024-05-27T09:30:13.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff1440b2900 0x7ff1440b4e10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:13.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15c109f00 0x7ff15c103090 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:13.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff15c104c40 0x7ff15c1044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:13.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 --2- 172.21.15.44:0/2533790251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff15c1075a0 0x7ff15c102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:13.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 -- 172.21.15.44:0/2533790251 >> 172.21.15.44:0/2533790251 conn(0x7ff15c0fe680 msgr2=0x7ff15c105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:13.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 -- 172.21.15.44:0/2533790251 shutdown_connections 2024-05-27T09:30:13.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:13.626+0000 7ff162c00700 1 -- 172.21.15.44:0/2533790251 wait complete. 2024-05-27T09:30:13.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:13 smithi189 ceph-mon[36671]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:13.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:13 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2533790251' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T09:30:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:13 smithi044 ceph-mon[31618]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:13.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:13 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2533790251' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T09:30:14.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:13 smithi050 ceph-mon[36371]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:14.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:13 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2533790251' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T09:30:14.567 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1716802209,"num_in_osds":12,"osd_in_since":1716802188,"num_remapped_pgs":0} 2024-05-27T09:30:14.568 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T09:30:15.043 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:30:15.954 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:15 smithi044 ceph-mon[31618]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:16.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:15 smithi050 ceph-mon[36371]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:16.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:15 smithi189 ceph-mon[36671]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.192+0000 7fa53b800700 1 -- 172.21.15.44:0/2210077738 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa534108730 msgr2=0x7fa534110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.192+0000 7fa53b800700 1 --2- 172.21.15.44:0/2210077738 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa534108730 0x7fa534110090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00b0d0 tx=0x7fa52c00b3e0 comp rx=0 tx=0).stop 2024-05-27T09:30:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.192+0000 7fa53b800700 1 -- 172.21.15.44:0/2210077738 shutdown_connections 2024-05-27T09:30:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.192+0000 7fa53b800700 1 --2- 172.21.15.44:0/2210077738 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa534108730 0x7fa534110090 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.192+0000 7fa53b800700 1 --2- 172.21.15.44:0/2210077738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5341034b0 0x7fa5341038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.192+0000 7fa53b800700 1 --2- 172.21.15.44:0/2210077738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa534107de0 0x7fa5341081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.192+0000 7fa53b800700 1 -- 172.21.15.44:0/2210077738 >> 172.21.15.44:0/2210077738 conn(0x7fa5340fedb0 msgr2=0x7fa5341011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.192+0000 7fa53b800700 1 -- 172.21.15.44:0/2210077738 shutdown_connections 2024-05-27T09:30:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.192+0000 7fa53b800700 1 -- 172.21.15.44:0/2210077738 wait complete. 2024-05-27T09:30:17.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.193+0000 7fa53b800700 1 Processor -- start 2024-05-27T09:30:17.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.193+0000 7fa53b800700 1 -- start start 2024-05-27T09:30:17.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.194+0000 7fa53b800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5341034b0 0x7fa5340ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:17.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.194+0000 7fa53b800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa534107de0 0x7fa5340ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:17.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.194+0000 7fa53b800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa534108730 0x7fa5340ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:17.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.194+0000 7fa53b800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa5341133b0 con 0x7fa5341034b0 2024-05-27T09:30:17.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.194+0000 7fa53b800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa534113230 con 0x7fa534108730 2024-05-27T09:30:17.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.194+0000 7fa53b800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa534113530 con 0x7fa534107de0 2024-05-27T09:30:17.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa533e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa534107de0 0x7fa5340ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:17.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa539800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa534108730 0x7fa5340ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:17.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa538e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5341034b0 0x7fa5340ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:17.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa533e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa534107de0 0x7fa5340ff7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43532/0 (socket says 172.21.15.44:43532) 2024-05-27T09:30:17.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa533e00700 1 -- 172.21.15.44:0/17873461 learned_addr learned my addr 172.21.15.44:0/17873461 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:30:17.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa538e00700 1 -- 172.21.15.44:0/17873461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa534107de0 msgr2=0x7fa5340ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:17.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa538e00700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa534107de0 0x7fa5340ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:17.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa538e00700 1 -- 172.21.15.44:0/17873461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa534108730 msgr2=0x7fa5340ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:17.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa538e00700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa534108730 0x7fa5340ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:17.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa538e00700 1 -- 172.21.15.44:0/17873461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5341a9190 con 0x7fa5341034b0 2024-05-27T09:30:17.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa533e00700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa534107de0 0x7fa5340ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:30:17.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.195+0000 7fa539800700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa534108730 0x7fa5340ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:30:17.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.196+0000 7fa538e00700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5341034b0 0x7fa5340ff290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa52800e9f0 tx=0x7fa52800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:17.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.196+0000 7fa531600700 1 -- 172.21.15.44:0/17873461 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52800c9c0 con 0x7fa5341034b0 2024-05-27T09:30:17.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.196+0000 7fa531600700 1 -- 172.21.15.44:0/17873461 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa52800cb60 con 0x7fa5341034b0 2024-05-27T09:30:17.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.196+0000 7fa53b800700 1 -- 172.21.15.44:0/17873461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5341a9480 con 0x7fa5341034b0 2024-05-27T09:30:17.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.196+0000 7fa531600700 1 -- 172.21.15.44:0/17873461 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa528018680 con 0x7fa5341034b0 2024-05-27T09:30:17.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.196+0000 7fa53b800700 1 -- 172.21.15.44:0/17873461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5341a9a10 con 0x7fa5341034b0 2024-05-27T09:30:17.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.197+0000 7fa53b800700 1 -- 172.21.15.44:0/17873461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5340512b0 con 0x7fa5341034b0 2024-05-27T09:30:17.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.200+0000 7fa531600700 1 -- 172.21.15.44:0/17873461 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa528010430 con 0x7fa5341034b0 2024-05-27T09:30:17.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.201+0000 7fa531600700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa51c0b2900 0x7fa51c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:17.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.202+0000 7fa531600700 1 -- 172.21.15.44:0/17873461 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fa5280e2260 con 0x7fa5341034b0 2024-05-27T09:30:17.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.202+0000 7fa533e00700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa51c0b2900 0x7fa51c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:17.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.203+0000 7fa533e00700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa51c0b2900 0x7fa51c0b4e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa5341099f0 tx=0x7fa520009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:17.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.205+0000 7fa531600700 1 -- 172.21.15.44:0/17873461 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa5280aa380 con 0x7fa5341034b0 2024-05-27T09:30:17.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.443+0000 7fa53b800700 1 -- 172.21.15.44:0/17873461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa53402e5e0 con 0x7fa5341034b0 2024-05-27T09:30:17.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.445+0000 7fa531600700 1 -- 172.21.15.44:0/17873461 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18397 (secure 0 0 0) 0x7fa5280b2400 con 0x7fa5341034b0 2024-05-27T09:30:17.446 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:30:17.447 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":72,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:30:10.013417+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:30:17.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 -- 172.21.15.44:0/17873461 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa51c0b2900 msgr2=0x7fa51c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:17.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa51c0b2900 0x7fa51c0b4e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa5341099f0 tx=0x7fa520009260 comp rx=0 tx=0).stop 2024-05-27T09:30:17.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 -- 172.21.15.44:0/17873461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5341034b0 msgr2=0x7fa5340ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:17.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5341034b0 0x7fa5340ff290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa52800e9f0 tx=0x7fa52800ed00 comp rx=0 tx=0).stop 2024-05-27T09:30:17.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 -- 172.21.15.44:0/17873461 shutdown_connections 2024-05-27T09:30:17.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa51c0b2900 0x7fa51c0b4e10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:17.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa534108730 0x7fa5340ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:17.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5341034b0 0x7fa5340ff290 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:17.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 --2- 172.21.15.44:0/17873461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa534107de0 0x7fa5340ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:17.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 -- 172.21.15.44:0/17873461 >> 172.21.15.44:0/17873461 conn(0x7fa5340fedb0 msgr2=0x7fa53406b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:17.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 -- 172.21.15.44:0/17873461 shutdown_connections 2024-05-27T09:30:17.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:17.448+0000 7fa53b800700 1 -- 172.21.15.44:0/17873461 wait complete. 2024-05-27T09:30:17.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:17 smithi044 ceph-mon[31618]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:17.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:17 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/17873461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:30:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:17 smithi050 ceph-mon[36371]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:17 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/17873461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:30:18.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:17 smithi189 ceph-mon[36671]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:18.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:17 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/17873461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:30:18.220 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T09:25:46.636320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T09:30:18.221 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-27T09:30:18.699 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:30:19.893 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:19 smithi044 ceph-mon[31618]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:20.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:19 smithi050 ceph-mon[36371]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:20.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:19 smithi189 ceph-mon[36671]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.765+0000 7f67ba200700 1 -- 172.21.15.44:0/1460527153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67b4108550 msgr2=0x7f67b410feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.765+0000 7f67ba200700 1 --2- 172.21.15.44:0/1460527153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67b4108550 0x7f67b410feb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009f90 tx=0x7f67ac00b330 comp rx=0 tx=0).stop 2024-05-27T09:30:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.765+0000 7f67ba200700 1 -- 172.21.15.44:0/1460527153 shutdown_connections 2024-05-27T09:30:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.765+0000 7f67ba200700 1 --2- 172.21.15.44:0/1460527153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67b4108550 0x7f67b410feb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.765+0000 7f67ba200700 1 --2- 172.21.15.44:0/1460527153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67b41032d0 0x7f67b41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:20.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.765+0000 7f67ba200700 1 --2- 172.21.15.44:0/1460527153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67b4107c00 0x7f67b4108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:20.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.765+0000 7f67ba200700 1 -- 172.21.15.44:0/1460527153 >> 172.21.15.44:0/1460527153 conn(0x7f67b4078ab0 msgr2=0x7f67b4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:20.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.765+0000 7f67ba200700 1 -- 172.21.15.44:0/1460527153 shutdown_connections 2024-05-27T09:30:20.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.765+0000 7f67ba200700 1 -- 172.21.15.44:0/1460527153 wait complete. 2024-05-27T09:30:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.766+0000 7f67ba200700 1 Processor -- start 2024-05-27T09:30:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.767+0000 7f67ba200700 1 -- start start 2024-05-27T09:30:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.767+0000 7f67ba200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67b41032d0 0x7f67b41a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.767+0000 7f67ba200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67b4107c00 0x7f67b41a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.767+0000 7f67ba200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67b4108550 0x7f67b419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.767+0000 7f67ba200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f67b4113180 con 0x7f67b41032d0 2024-05-27T09:30:20.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.767+0000 7f67ba200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f67b4113000 con 0x7f67b4107c00 2024-05-27T09:30:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.767+0000 7f67ba200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f67b4113300 con 0x7f67b4108550 2024-05-27T09:30:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.767+0000 7f67b8e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67b41032d0 0x7f67b41a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.767+0000 7f67b8e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67b41032d0 0x7f67b41a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51024/0 (socket says 172.21.15.44:51024) 2024-05-27T09:30:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.767+0000 7f67b8e00700 1 -- 172.21.15.44:0/3614386271 learned_addr learned my addr 172.21.15.44:0/3614386271 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:30:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.768+0000 7f67b3e00700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67b4107c00 0x7f67b41a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:20.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.768+0000 7f67b9800700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67b4108550 0x7f67b419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.768+0000 7f67b8e00700 1 -- 172.21.15.44:0/3614386271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67b4108550 msgr2=0x7f67b419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.768+0000 7f67b8e00700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67b4108550 0x7f67b419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.768+0000 7f67b8e00700 1 -- 172.21.15.44:0/3614386271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67b4107c00 msgr2=0x7f67b41a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.768+0000 7f67b8e00700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67b4107c00 0x7f67b41a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.768+0000 7f67b8e00700 1 -- 172.21.15.44:0/3614386271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b419b1b0 con 0x7f67b41032d0 2024-05-27T09:30:20.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.768+0000 7f67b9800700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67b4108550 0x7f67b419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:30:20.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.768+0000 7f67b8e00700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67b41032d0 0x7f67b41a0890 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f67a800eb30 tx=0x7f67a800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:20.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.769+0000 7f67b1600700 1 -- 172.21.15.44:0/3614386271 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a800cbd0 con 0x7f67b41032d0 2024-05-27T09:30:20.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.769+0000 7f67b1600700 1 -- 172.21.15.44:0/3614386271 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67a80044d0 con 0x7f67b41032d0 2024-05-27T09:30:20.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.769+0000 7f67b1600700 1 -- 172.21.15.44:0/3614386271 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a8003e30 con 0x7f67b41032d0 2024-05-27T09:30:20.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.769+0000 7f67ba200700 1 -- 172.21.15.44:0/3614386271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b419b4a0 con 0x7f67b41032d0 2024-05-27T09:30:20.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.769+0000 7f67ba200700 1 -- 172.21.15.44:0/3614386271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b41a7740 con 0x7f67b41032d0 2024-05-27T09:30:20.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.770+0000 7f67ba200700 1 -- 172.21.15.44:0/3614386271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b40512b0 con 0x7f67b41032d0 2024-05-27T09:30:20.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.773+0000 7f67b1600700 1 -- 172.21.15.44:0/3614386271 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f67a8004000 con 0x7f67b41032d0 2024-05-27T09:30:20.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.774+0000 7f67b1600700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f679c0b2900 0x7f679c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.774+0000 7f67b1600700 1 -- 172.21.15.44:0/3614386271 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f67a80e2100 con 0x7f67b41032d0 2024-05-27T09:30:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.774+0000 7f67b3e00700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f679c0b2900 0x7f679c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:20.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.775+0000 7f67b3e00700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f679c0b2900 0x7f679c0b4e10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009750 tx=0x7f67a0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:20.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:20.780+0000 7f67b1600700 1 -- 172.21.15.44:0/3614386271 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f67a80042f0 con 0x7f67b41032d0 2024-05-27T09:30:21.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.018+0000 7f67ba200700 1 -- 172.21.15.44:0/3614386271 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f67b402e5e0 con 0x7f67b41032d0 2024-05-27T09:30:21.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.019+0000 7f67b1600700 1 -- 172.21.15.44:0/3614386271 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f67a80aa220 con 0x7f67b41032d0 2024-05-27T09:30:21.019 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-05-27T09:30:21.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.022+0000 7f67ba200700 1 -- 172.21.15.44:0/3614386271 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f679c0b2900 msgr2=0x7f679c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:21.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.022+0000 7f67ba200700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f679c0b2900 0x7f679c0b4e10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009750 tx=0x7f67a0009310 comp rx=0 tx=0).stop 2024-05-27T09:30:21.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.022+0000 7f67ba200700 1 -- 172.21.15.44:0/3614386271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67b41032d0 msgr2=0x7f67b41a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:21.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.022+0000 7f67ba200700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67b41032d0 0x7f67b41a0890 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f67a800eb30 tx=0x7f67a800ee40 comp rx=0 tx=0).stop 2024-05-27T09:30:21.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.022+0000 7f67ba200700 1 -- 172.21.15.44:0/3614386271 shutdown_connections 2024-05-27T09:30:21.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.022+0000 7f67ba200700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f679c0b2900 0x7f679c0b4e10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:21.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.022+0000 7f67ba200700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67b4108550 0x7f67b419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:21.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.022+0000 7f67ba200700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67b41032d0 0x7f67b41a0890 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:21.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.022+0000 7f67ba200700 1 --2- 172.21.15.44:0/3614386271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67b4107c00 0x7f67b41a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:21.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.022+0000 7f67ba200700 1 -- 172.21.15.44:0/3614386271 >> 172.21.15.44:0/3614386271 conn(0x7f67b4078ab0 msgr2=0x7f67b406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:21.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.022+0000 7f67ba200700 1 -- 172.21.15.44:0/3614386271 shutdown_connections 2024-05-27T09:30:21.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:21.023+0000 7f67ba200700 1 -- 172.21.15.44:0/3614386271 wait complete. 2024-05-27T09:30:21.771 INFO:tasks.cephadm:Adding mds.a on smithi044 2024-05-27T09:30:21.772 INFO:tasks.cephadm:Adding mds.d on smithi044 2024-05-27T09:30:21.772 INFO:tasks.cephadm:Adding mds.g on smithi044 2024-05-27T09:30:21.772 INFO:tasks.cephadm:Adding mds.j on smithi044 2024-05-27T09:30:21.772 INFO:tasks.cephadm:Adding mds.b on smithi050 2024-05-27T09:30:21.772 INFO:tasks.cephadm:Adding mds.e on smithi050 2024-05-27T09:30:21.772 INFO:tasks.cephadm:Adding mds.h on smithi050 2024-05-27T09:30:21.772 INFO:tasks.cephadm:Adding mds.k on smithi050 2024-05-27T09:30:21.772 INFO:tasks.cephadm:Adding mds.c on smithi189 2024-05-27T09:30:21.772 INFO:tasks.cephadm:Adding mds.f on smithi189 2024-05-27T09:30:21.772 INFO:tasks.cephadm:Adding mds.i on smithi189 2024-05-27T09:30:21.772 INFO:tasks.cephadm:Adding mds.l on smithi189 2024-05-27T09:30:21.773 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph orch apply mds all '12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l' 2024-05-27T09:30:21.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:21 smithi044 ceph-mon[31618]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:21.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:21 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3614386271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T09:30:22.024 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:30:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:21 smithi050 ceph-mon[36371]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:21 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3614386271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T09:30:22.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:21 smithi189 ceph-mon[36671]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:22.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:21 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3614386271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T09:30:23.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.613+0000 7f9ddfb52700 1 -- 172.21.15.189:0/2446644484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dd8104cf0 msgr2=0x7f9dd810b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:23.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.613+0000 7f9ddfb52700 1 --2- 172.21.15.189:0/2446644484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dd8104cf0 0x7f9dd810b5a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000b0a0 tx=0x7f9dd000b3b0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.613+0000 7f9ddfb52700 1 -- 172.21.15.189:0/2446644484 shutdown_connections 2024-05-27T09:30:23.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.613+0000 7f9ddfb52700 1 --2- 172.21.15.189:0/2446644484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dd8104cf0 0x7f9dd810b5a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.613+0000 7f9ddfb52700 1 --2- 172.21.15.189:0/2446644484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dd8103220 0x7f9dd8103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.613+0000 7f9ddfb52700 1 --2- 172.21.15.189:0/2446644484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd8104350 0x7f9dd8104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.613+0000 7f9ddfb52700 1 -- 172.21.15.189:0/2446644484 >> 172.21.15.189:0/2446644484 conn(0x7f9dd80fe860 msgr2=0x7f9dd8100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.613+0000 7f9ddfb52700 1 -- 172.21.15.189:0/2446644484 shutdown_connections 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.613+0000 7f9ddfb52700 1 -- 172.21.15.189:0/2446644484 wait complete. 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.614+0000 7f9ddfb52700 1 Processor -- start 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.614+0000 7f9ddfb52700 1 -- start start 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.614+0000 7f9ddfb52700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dd8103220 0x7f9dd806caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.614+0000 7f9ddfb52700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dd8104350 0x7f9dd819f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.614+0000 7f9ddfb52700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd8104cf0 0x7f9dd81a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.614+0000 7f9ddfb52700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd810ed00 con 0x7f9dd8104350 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.614+0000 7f9ddfb52700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd810eb80 con 0x7f9dd8104cf0 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.614+0000 7f9ddfb52700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd810ee80 con 0x7f9dd8103220 2024-05-27T09:30:23.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.614+0000 7f9ddd8ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dd8103220 0x7f9dd806caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.614+0000 7f9ddd8ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dd8103220 0x7f9dd806caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49700/0 (socket says 172.21.15.189:49700) 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.614+0000 7f9ddd8ee700 1 -- 172.21.15.189:0/1017475252 learned_addr learned my addr 172.21.15.189:0/1017475252 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9dde0ef700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd8104cf0 0x7f9dd81a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9ddd8ee700 1 -- 172.21.15.189:0/1017475252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd8104cf0 msgr2=0x7f9dd81a1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9ddd8ee700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd8104cf0 0x7f9dd81a1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9ddd8ee700 1 -- 172.21.15.189:0/1017475252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dd8104350 msgr2=0x7f9dd819f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9ddd8ee700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dd8104350 0x7f9dd819f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9ddd8ee700 1 -- 172.21.15.189:0/1017475252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd81a2470 con 0x7f9dd8103220 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9ddd8ee700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dd8103220 0x7f9dd806caf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400b4d0 tx=0x7f9dd400b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9dceffd700 1 -- 172.21.15.189:0/1017475252 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4015020 con 0x7f9dd8103220 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9ddfb52700 1 -- 172.21.15.189:0/1017475252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd81a2760 con 0x7f9dd8103220 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9dceffd700 1 -- 172.21.15.189:0/1017475252 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dd4011ce0 con 0x7f9dd8103220 2024-05-27T09:30:23.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9dceffd700 1 -- 172.21.15.189:0/1017475252 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4010cf0 con 0x7f9dd8103220 2024-05-27T09:30:23.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.615+0000 7f9ddfb52700 1 -- 172.21.15.189:0/1017475252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd81a2d70 con 0x7f9dd8103220 2024-05-27T09:30:23.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.616+0000 7f9ddfb52700 1 -- 172.21.15.189:0/1017475252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dd80512b0 con 0x7f9dd8103220 2024-05-27T09:30:23.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.619+0000 7f9dceffd700 1 -- 172.21.15.189:0/1017475252 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9dd4018020 con 0x7f9dd8103220 2024-05-27T09:30:23.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.619+0000 7f9dceffd700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dc40b2900 0x7f9dc40b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:23.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.619+0000 7f9dceffd700 1 -- 172.21.15.189:0/1017475252 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f9dd40e19d0 con 0x7f9dd8103220 2024-05-27T09:30:23.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.620+0000 7f9ddd0ed700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dc40b2900 0x7f9dc40b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:23.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.621+0000 7f9ddd0ed700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dc40b2900 0x7f9dc40b4e10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8005950 tx=0x7f9dc80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:23.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.623+0000 7f9dceffd700 1 -- 172.21.15.189:0/1017475252 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9dd401f020 con 0x7f9dd8103220 2024-05-27T09:30:23.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.835+0000 7f9ddfb52700 1 -- 172.21.15.189:0/1017475252 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}) -- 0x7f9dd8063df0 con 0x7f9dc40b2900 2024-05-27T09:30:23.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.853+0000 7f9dceffd700 1 -- 172.21.15.189:0/1017475252 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f9dd8063df0 con 0x7f9dc40b2900 2024-05-27T09:30:23.854 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mds.all update... 2024-05-27T09:30:23.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 -- 172.21.15.189:0/1017475252 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dc40b2900 msgr2=0x7f9dc40b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:23.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dc40b2900 0x7f9dc40b4e10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8005950 tx=0x7f9dc80058e0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 -- 172.21.15.189:0/1017475252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dd8103220 msgr2=0x7f9dd806caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:23.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dd8103220 0x7f9dd806caf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400b4d0 tx=0x7f9dd400b7e0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 -- 172.21.15.189:0/1017475252 shutdown_connections 2024-05-27T09:30:23.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9dc40b2900 0x7f9dc40b4e10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd8104cf0 0x7f9dd81a1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dd8103220 0x7f9dd806caf0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 --2- 172.21.15.189:0/1017475252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9dd8104350 0x7f9dd819f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:23.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 -- 172.21.15.189:0/1017475252 >> 172.21.15.189:0/1017475252 conn(0x7f9dd80fe860 msgr2=0x7f9dd8107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:23.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 -- 172.21.15.189:0/1017475252 shutdown_connections 2024-05-27T09:30:23.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-05-27T09:30:23.856+0000 7f9ddfb52700 1 -- 172.21.15.189:0/1017475252 wait complete. 2024-05-27T09:30:23.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:23 smithi044 ceph-mon[31618]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:24.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:23 smithi050 ceph-mon[36371]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:24.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:23 smithi189 ceph-mon[36671]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:24.568 DEBUG:teuthology.orchestra.run.smithi044:mds.a> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.a.service 2024-05-27T09:30:24.576 DEBUG:teuthology.orchestra.run.smithi044:mds.d> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.d.service 2024-05-27T09:30:24.622 DEBUG:teuthology.orchestra.run.smithi044:mds.g> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.g.service 2024-05-27T09:30:24.667 DEBUG:teuthology.orchestra.run.smithi044:mds.j> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.j.service 2024-05-27T09:30:24.713 DEBUG:teuthology.orchestra.run.smithi050:mds.b> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.b.service 2024-05-27T09:30:24.717 DEBUG:teuthology.orchestra.run.smithi050:mds.e> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.e.service 2024-05-27T09:30:24.760 DEBUG:teuthology.orchestra.run.smithi050:mds.h> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.h.service 2024-05-27T09:30:24.803 DEBUG:teuthology.orchestra.run.smithi050:mds.k> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.k.service 2024-05-27T09:30:24.846 DEBUG:teuthology.orchestra.run.smithi189:mds.c> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.c.service 2024-05-27T09:30:24.848 DEBUG:teuthology.orchestra.run.smithi189:mds.f> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.f.service 2024-05-27T09:30:24.891 DEBUG:teuthology.orchestra.run.smithi189:mds.i> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.i.service 2024-05-27T09:30:24.934 DEBUG:teuthology.orchestra.run.smithi189:mds.l> sudo journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.l.service 2024-05-27T09:30:24.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:24 smithi044 ceph-mon[31618]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:30:24.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:24 smithi044 ceph-mon[31618]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-05-27T09:30:24.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:24 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:24.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:24 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:30:24.977 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-27T09:30:24.977 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:30:25.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:24 smithi050 ceph-mon[36371]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:30:25.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:24 smithi050 ceph-mon[36371]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-05-27T09:30:25.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:24 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:25.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:24 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:30:25.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:24 smithi189 ceph-mon[36671]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:30:25.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:24 smithi189 ceph-mon[36671]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-05-27T09:30:25.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:24 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:25.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:24 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:30:25.428 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:30:26.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:25 smithi189 ceph-mon[36671]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:30:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:26.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:25 smithi044 ceph-mon[31618]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:30:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:26.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:26.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:25 smithi050 ceph-mon[36371]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:30:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:27.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:26 smithi044 ceph-mon[31618]: Deploying daemon mds.l on smithi189 2024-05-27T09:30:27.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:26 smithi189 ceph-mon[36671]: Deploying daemon mds.l on smithi189 2024-05-27T09:30:27.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:26 smithi050 ceph-mon[36371]: Deploying daemon mds.l on smithi189 2024-05-27T09:30:27.489 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:30:27 smithi189 systemd[1]: Starting Ceph mds.l for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:30:27.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.592+0000 7fa280000700 1 -- 172.21.15.44:0/3398695634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa278107de0 msgr2=0x7fa2781081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:27.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.592+0000 7fa280000700 1 --2- 172.21.15.44:0/3398695634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa278107de0 0x7fa2781081f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa2640099c0 tx=0x7fa264009cd0 comp rx=0 tx=0).stop 2024-05-27T09:30:27.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.592+0000 7fa280000700 1 -- 172.21.15.44:0/3398695634 shutdown_connections 2024-05-27T09:30:27.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.592+0000 7fa280000700 1 --2- 172.21.15.44:0/3398695634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa278108730 0x7fa278110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:27.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.592+0000 7fa280000700 1 --2- 172.21.15.44:0/3398695634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2781034b0 0x7fa2781038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:27.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.592+0000 7fa280000700 1 --2- 172.21.15.44:0/3398695634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa278107de0 0x7fa2781081f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:27.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.592+0000 7fa280000700 1 -- 172.21.15.44:0/3398695634 >> 172.21.15.44:0/3398695634 conn(0x7fa2780fedb0 msgr2=0x7fa2781011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:27.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.592+0000 7fa280000700 1 -- 172.21.15.44:0/3398695634 shutdown_connections 2024-05-27T09:30:27.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.592+0000 7fa280000700 1 -- 172.21.15.44:0/3398695634 wait complete. 2024-05-27T09:30:27.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.593+0000 7fa280000700 1 Processor -- start 2024-05-27T09:30:27.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.593+0000 7fa280000700 1 -- start start 2024-05-27T09:30:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.594+0000 7fa280000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2781034b0 0x7fa2780ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:27.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.594+0000 7fa280000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa278107de0 0x7fa2780ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.594+0000 7fa280000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa278108730 0x7fa2780ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.594+0000 7fa280000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa278113390 con 0x7fa278108730 2024-05-27T09:30:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.594+0000 7fa280000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa278113210 con 0x7fa2781034b0 2024-05-27T09:30:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.594+0000 7fa280000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa278113510 con 0x7fa278107de0 2024-05-27T09:30:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.594+0000 7fa27e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa278108730 0x7fa2780ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.594+0000 7fa27d600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2781034b0 0x7fa2780ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.594+0000 7fa27e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa278108730 0x7fa2780ffd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47268/0 (socket says 172.21.15.44:47268) 2024-05-27T09:30:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.594+0000 7fa27e000700 1 -- 172.21.15.44:0/557839338 learned_addr learned my addr 172.21.15.44:0/557839338 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:30:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.595+0000 7fa27e000700 1 -- 172.21.15.44:0/557839338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa278107de0 msgr2=0x7fa2780ff7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T09:30:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.595+0000 7fa27e000700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa278107de0 0x7fa2780ff7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:27.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.595+0000 7fa27e000700 1 -- 172.21.15.44:0/557839338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2781034b0 msgr2=0x7fa2780ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:27.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.595+0000 7fa27e000700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2781034b0 0x7fa2780ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:27.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.595+0000 7fa27e000700 1 -- 172.21.15.44:0/557839338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2781a9210 con 0x7fa278108730 2024-05-27T09:30:27.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.595+0000 7fa27d600700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2781034b0 0x7fa2780ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:30:27.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.595+0000 7fa27e000700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa278108730 0x7fa2780ffd10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa27000ef30 tx=0x7fa27000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:27.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.596+0000 7fa276000700 1 -- 172.21.15.44:0/557839338 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2700092e0 con 0x7fa278108730 2024-05-27T09:30:27.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.596+0000 7fa276000700 1 -- 172.21.15.44:0/557839338 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa270004bb0 con 0x7fa278108730 2024-05-27T09:30:27.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.596+0000 7fa276000700 1 -- 172.21.15.44:0/557839338 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa270004280 con 0x7fa278108730 2024-05-27T09:30:27.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.596+0000 7fa280000700 1 -- 172.21.15.44:0/557839338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2781a94a0 con 0x7fa278108730 2024-05-27T09:30:27.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.596+0000 7fa280000700 1 -- 172.21.15.44:0/557839338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2781a9990 con 0x7fa278108730 2024-05-27T09:30:27.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.597+0000 7fa280000700 1 -- 172.21.15.44:0/557839338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2780512b0 con 0x7fa278108730 2024-05-27T09:30:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.601+0000 7fa276000700 1 -- 172.21.15.44:0/557839338 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa27001b030 con 0x7fa278108730 2024-05-27T09:30:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.602+0000 7fa276000700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa2600b2880 0x7fa2600b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.603+0000 7fa276000700 1 -- 172.21.15.44:0/557839338 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fa2700e2280 con 0x7fa278108730 2024-05-27T09:30:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.603+0000 7fa27d600700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa2600b2880 0x7fa2600b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:27.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.603+0000 7fa27d600700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa2600b2880 0x7fa2600b4d90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa26c011490 tx=0x7fa26c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:27.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.605+0000 7fa276000700 1 -- 172.21.15.44:0/557839338 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa270022020 con 0x7fa278108730 2024-05-27T09:30:27.755 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:30:27 smithi189 podman[66916]: 2024-05-27 09:30:27.392104241 +0000 UTC m=+0.017714572 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:30:27.755 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:30:27 smithi189 podman[66916]: 2024-05-27 09:30:27.502683631 +0000 UTC m=+0.128293951 container create 2f8c7b3cbe9889510e721163688d567f97631a9c30da43ad66bdf33bb59bd15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-27T09:30:27.755 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:30:27 smithi189 podman[66916]: 2024-05-27 09:30:27.671668906 +0000 UTC m=+0.297279226 container init 2f8c7b3cbe9889510e721163688d567f97631a9c30da43ad66bdf33bb59bd15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True) 2024-05-27T09:30:27.755 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:30:27 smithi189 podman[66916]: 2024-05-27 09:30:27.674420577 +0000 UTC m=+0.300030900 container start 2f8c7b3cbe9889510e721163688d567f97631a9c30da43ad66bdf33bb59bd15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-27T09:30:27.755 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:30:27 smithi189 ceph-mds[66935]: -- 172.21.15.189:0/288609038 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5584911d1a40 con 0x558491219180 2024-05-27T09:30:27.755 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:30:27 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-l[66930]: starting mds.l at 2024-05-27T09:30:27.755 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:30:27 smithi189 bash[66916]: 2f8c7b3cbe9889510e721163688d567f97631a9c30da43ad66bdf33bb59bd15f 2024-05-27T09:30:27.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.949+0000 7fa280000700 1 -- 172.21.15.44:0/557839338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2781007f0 con 0x7fa278108730 2024-05-27T09:30:27.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.950+0000 7fa276000700 1 -- 172.21.15.44:0/557839338 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7fa2781007f0 con 0x7fa278108730 2024-05-27T09:30:27.952 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:30:27.952 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T09:30:27.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.953+0000 7fa280000700 1 -- 172.21.15.44:0/557839338 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa2600b2880 msgr2=0x7fa2600b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:27.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.953+0000 7fa280000700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa2600b2880 0x7fa2600b4d90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa26c011490 tx=0x7fa26c009260 comp rx=0 tx=0).stop 2024-05-27T09:30:27.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.954+0000 7fa280000700 1 -- 172.21.15.44:0/557839338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa278108730 msgr2=0x7fa2780ffd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:27.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.954+0000 7fa280000700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa278108730 0x7fa2780ffd10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa27000ef30 tx=0x7fa27000c3a0 comp rx=0 tx=0).stop 2024-05-27T09:30:27.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.954+0000 7fa280000700 1 -- 172.21.15.44:0/557839338 shutdown_connections 2024-05-27T09:30:27.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.954+0000 7fa280000700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa2600b2880 0x7fa2600b4d90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:27.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.954+0000 7fa280000700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa278108730 0x7fa2780ffd10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:27.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.954+0000 7fa280000700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2781034b0 0x7fa2780ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:27.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.954+0000 7fa280000700 1 --2- 172.21.15.44:0/557839338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa278107de0 0x7fa2780ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:27.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.954+0000 7fa280000700 1 -- 172.21.15.44:0/557839338 >> 172.21.15.44:0/557839338 conn(0x7fa2780fedb0 msgr2=0x7fa27806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:27.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.954+0000 7fa280000700 1 -- 172.21.15.44:0/557839338 shutdown_connections 2024-05-27T09:30:27.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:27.954+0000 7fa280000700 1 -- 172.21.15.44:0/557839338 wait complete. 2024-05-27T09:30:27.957 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 2 2024-05-27T09:30:28.087 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:30:27 smithi189 systemd[1]: Started Ceph mds.l for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:30:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:27 smithi189 ceph-mon[36671]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:27 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6832/742901973,v1:172.21.15.189:6833/742901973] up:boot 2024-05-27T09:30:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:27 smithi189 ceph-mon[36671]: fsmap 1 up:standby 2024-05-27T09:30:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:27 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T09:30:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:27 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:27 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:27 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:27 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:27 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:28.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:27 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:27 smithi044 ceph-mon[31618]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:28.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:27 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6832/742901973,v1:172.21.15.189:6833/742901973] up:boot 2024-05-27T09:30:28.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:27 smithi044 ceph-mon[31618]: fsmap 1 up:standby 2024-05-27T09:30:28.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:27 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T09:30:28.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:27 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:28.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:27 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:28.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:27 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:28.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:27 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:28.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:27 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:28.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:27 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:28.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:27 smithi050 ceph-mon[36371]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:27 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6832/742901973,v1:172.21.15.189:6833/742901973] up:boot 2024-05-27T09:30:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:27 smithi050 ceph-mon[36371]: fsmap 1 up:standby 2024-05-27T09:30:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:27 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T09:30:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:27 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:27 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:27 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:27 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:27 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:27 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:28.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:28 smithi044 ceph-mon[31618]: Deploying daemon mds.k on smithi050 2024-05-27T09:30:28.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:28 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/557839338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:28 smithi050 ceph-mon[36371]: Deploying daemon mds.k on smithi050 2024-05-27T09:30:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:28 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/557839338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:29.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:28 smithi189 ceph-mon[36671]: Deploying daemon mds.k on smithi050 2024-05-27T09:30:29.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:28 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/557839338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:30.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:29 smithi044 ceph-mon[31618]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:30.266 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:29 smithi050 ceph-mon[36371]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:30.266 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:30:30 smithi050 systemd[1]: Starting Ceph mds.k for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:30:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:29 smithi189 ceph-mon[36671]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:30.527 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:30:30 smithi050 podman[65774]: 2024-05-27 09:30:30.168708038 +0000 UTC m=+0.018413427 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:30:30.527 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:30:30 smithi050 podman[65774]: 2024-05-27 09:30:30.295331373 +0000 UTC m=+0.145036801 container create 166e6764c810be15ce17ed86f563136d99faa547986e59c9abee82b072e5754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:30:30.527 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:30:30 smithi050 podman[65774]: 2024-05-27 09:30:30.49024191 +0000 UTC m=+0.339947287 container init 166e6764c810be15ce17ed86f563136d99faa547986e59c9abee82b072e5754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-27T09:30:30.527 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:30:30 smithi050 podman[65774]: 2024-05-27 09:30:30.493166753 +0000 UTC m=+0.342872142 container start 166e6764c810be15ce17ed86f563136d99faa547986e59c9abee82b072e5754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-05-27T09:30:30.717 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:30:30.794 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:30:30 smithi050 ceph-mds[65792]: -- 172.21.15.50:0/3400001223 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562b189eda40 con 0x562b18a35600 2024-05-27T09:30:30.794 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:30:30 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-k[65788]: starting mds.k at 2024-05-27T09:30:30.794 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:30:30 smithi050 bash[65774]: 166e6764c810be15ce17ed86f563136d99faa547986e59c9abee82b072e5754f 2024-05-27T09:30:30.794 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:30:30 smithi050 systemd[1]: Started Ceph mds.k for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:30:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:30:31.897 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:31 smithi044 ceph-mon[31618]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:31.897 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:31.897 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:31.897 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:31.897 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:31.897 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:31.897 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:31.897 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:31 smithi044 ceph-mon[31618]: Deploying daemon mds.j on smithi044 2024-05-27T09:30:31.897 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:31 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6832/4273379740,v1:172.21.15.50:6833/4273379740] up:boot 2024-05-27T09:30:31.898 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:31 smithi044 ceph-mon[31618]: fsmap 2 up:standby 2024-05-27T09:30:31.898 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:31 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T09:30:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:31 smithi050 ceph-mon[36371]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:31 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:31 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:31 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:31 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:31 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:31 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:31 smithi050 ceph-mon[36371]: Deploying daemon mds.j on smithi044 2024-05-27T09:30:32.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:31 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6832/4273379740,v1:172.21.15.50:6833/4273379740] up:boot 2024-05-27T09:30:32.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:31 smithi050 ceph-mon[36371]: fsmap 2 up:standby 2024-05-27T09:30:32.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:31 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T09:30:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:31 smithi189 ceph-mon[36671]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:31 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:31 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:31 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:31 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:31 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:31 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:31 smithi189 ceph-mon[36671]: Deploying daemon mds.j on smithi044 2024-05-27T09:30:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:31 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6832/4273379740,v1:172.21.15.50:6833/4273379740] up:boot 2024-05-27T09:30:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:31 smithi189 ceph-mon[36671]: fsmap 2 up:standby 2024-05-27T09:30:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:31 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T09:30:33.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:33 smithi044 ceph-mon[31618]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:33.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.596+0000 7faa5b600700 1 -- 172.21.15.44:0/1180958032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa5410e8d0 msgr2=0x7faa54110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:33.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.596+0000 7faa5b600700 1 --2- 172.21.15.44:0/1180958032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa5410e8d0 0x7faa54110db0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faa4c009990 tx=0x7faa4c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:30:33.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.597+0000 7faa5b600700 1 -- 172.21.15.44:0/1180958032 shutdown_connections 2024-05-27T09:30:33.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.597+0000 7faa5b600700 1 --2- 172.21.15.44:0/1180958032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa5410e8d0 0x7faa54110db0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.597+0000 7faa5b600700 1 --2- 172.21.15.44:0/1180958032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa541054f0 0x7faa54105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.597+0000 7faa5b600700 1 --2- 172.21.15.44:0/1180958032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa54105e40 0x7faa5410e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.597+0000 7faa5b600700 1 -- 172.21.15.44:0/1180958032 >> 172.21.15.44:0/1180958032 conn(0x7faa540fedb0 msgr2=0x7faa541011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.597+0000 7faa5b600700 1 -- 172.21.15.44:0/1180958032 shutdown_connections 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.597+0000 7faa5b600700 1 -- 172.21.15.44:0/1180958032 wait complete. 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.598+0000 7faa5b600700 1 Processor -- start 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.598+0000 7faa5b600700 1 -- start start 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.599+0000 7faa5b600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa541054f0 0x7faa541031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.599+0000 7faa5b600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa54105e40 0x7faa54103720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.599+0000 7faa5b600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa5410e8d0 0x7faa54103c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.599+0000 7faa5b600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faa54114480 con 0x7faa5410e8d0 2024-05-27T09:30:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.599+0000 7faa5b600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faa54114300 con 0x7faa541054f0 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.599+0000 7faa5b600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faa54114600 con 0x7faa54105e40 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.599+0000 7faa58c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa541054f0 0x7faa541031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.599+0000 7faa58c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa541054f0 0x7faa541031e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47094/0 (socket says 172.21.15.44:47094) 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.599+0000 7faa58c00700 1 -- 172.21.15.44:0/1004430644 learned_addr learned my addr 172.21.15.44:0/1004430644 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa59600700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa5410e8d0 0x7faa54103c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa53e00700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa54105e40 0x7faa54103720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa58c00700 1 -- 172.21.15.44:0/1004430644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa54105e40 msgr2=0x7faa54103720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa58c00700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa54105e40 0x7faa54103720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa58c00700 1 -- 172.21.15.44:0/1004430644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa5410e8d0 msgr2=0x7faa54103c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa58c00700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa5410e8d0 0x7faa54103c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa58c00700 1 -- 172.21.15.44:0/1004430644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa541a9110 con 0x7faa541054f0 2024-05-27T09:30:33.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa59600700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa5410e8d0 0x7faa54103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa58c00700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa541054f0 0x7faa541031e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa4800ef10 tx=0x7faa4800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa51600700 1 -- 172.21.15.44:0/1004430644 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4800cad0 con 0x7faa541054f0 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa51600700 1 -- 172.21.15.44:0/1004430644 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa48019070 con 0x7faa541054f0 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa51600700 1 -- 172.21.15.44:0/1004430644 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa480146e0 con 0x7faa541054f0 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.600+0000 7faa5b600700 1 -- 172.21.15.44:0/1004430644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa541a93a0 con 0x7faa541054f0 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.601+0000 7faa5b600700 1 -- 172.21.15.44:0/1004430644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa541a9890 con 0x7faa541054f0 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.602+0000 7faa5b600700 1 -- 172.21.15.44:0/1004430644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa540512b0 con 0x7faa541054f0 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.604+0000 7faa51600700 1 -- 172.21.15.44:0/1004430644 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7faa48012050 con 0x7faa541054f0 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.605+0000 7faa51600700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7faa3c0b2900 0x7faa3c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.606+0000 7faa51600700 1 -- 172.21.15.44:0/1004430644 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7faa480e1e30 con 0x7faa541054f0 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.606+0000 7faa53e00700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7faa3c0b2900 0x7faa3c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:33.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.607+0000 7faa53e00700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7faa3c0b2900 0x7faa3c0b4e10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7faa541022f0 tx=0x7faa40009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:33.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.610+0000 7faa51600700 1 -- 172.21.15.44:0/1004430644 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7faa48010040 con 0x7faa541054f0 2024-05-27T09:30:33.885 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:33 smithi050 ceph-mon[36371]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:33.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.911+0000 7faa5b600700 1 -- 172.21.15.44:0/1004430644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa541a9b90 con 0x7faa541054f0 2024-05-27T09:30:33.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.912+0000 7faa51600700 1 -- 172.21.15.44:0/1004430644 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7faa480a9f50 con 0x7faa541054f0 2024-05-27T09:30:33.913 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:30:33.913 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T09:30:33.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.916+0000 7faa5b600700 1 -- 172.21.15.44:0/1004430644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7faa3c0b2900 msgr2=0x7faa3c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:33.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.916+0000 7faa5b600700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7faa3c0b2900 0x7faa3c0b4e10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7faa541022f0 tx=0x7faa40009260 comp rx=0 tx=0).stop 2024-05-27T09:30:33.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.916+0000 7faa5b600700 1 -- 172.21.15.44:0/1004430644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa541054f0 msgr2=0x7faa541031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:33.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.916+0000 7faa5b600700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa541054f0 0x7faa541031e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa4800ef10 tx=0x7faa4800c3a0 comp rx=0 tx=0).stop 2024-05-27T09:30:33.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.916+0000 7faa5b600700 1 -- 172.21.15.44:0/1004430644 shutdown_connections 2024-05-27T09:30:33.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.917+0000 7faa5b600700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7faa3c0b2900 0x7faa3c0b4e10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:33.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.917+0000 7faa5b600700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa5410e8d0 0x7faa54103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:33.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.917+0000 7faa5b600700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa541054f0 0x7faa541031e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:33.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.917+0000 7faa5b600700 1 --2- 172.21.15.44:0/1004430644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa54105e40 0x7faa54103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:33.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.917+0000 7faa5b600700 1 -- 172.21.15.44:0/1004430644 >> 172.21.15.44:0/1004430644 conn(0x7faa540fedb0 msgr2=0x7faa540ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:33.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.918+0000 7faa5b600700 1 -- 172.21.15.44:0/1004430644 shutdown_connections 2024-05-27T09:30:33.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:33.918+0000 7faa5b600700 1 -- 172.21.15.44:0/1004430644 wait complete. 2024-05-27T09:30:33.920 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 3 2024-05-27T09:30:34.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:33 smithi189 ceph-mon[36671]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:34.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:34 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1004430644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:35.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:34 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1004430644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:35.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:34 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1004430644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:35.224 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:30:35 smithi044 systemd[1]: Starting Ceph mds.j for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:30:35.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:35 smithi044 ceph-mon[31618]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:35.830 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:30:35 smithi044 podman[72055]: 2024-05-27 09:30:35.475145136 +0000 UTC m=+0.040261108 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:30:35.831 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:30:35 smithi044 podman[72055]: 2024-05-27 09:30:35.590507514 +0000 UTC m=+0.155623518 container create b861e9014d39d2a93efcde4fd5685d818f3fdd98ae82ce6999ad371cfe8516ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-27T09:30:35.831 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:30:35 smithi044 podman[72055]: 2024-05-27 09:30:35.819073031 +0000 UTC m=+0.384189058 container init b861e9014d39d2a93efcde4fd5685d818f3fdd98ae82ce6999ad371cfe8516ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-27T09:30:35.831 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:30:35 smithi044 podman[72055]: 2024-05-27 09:30:35.829422282 +0000 UTC m=+0.394538255 container start b861e9014d39d2a93efcde4fd5685d818f3fdd98ae82ce6999ad371cfe8516ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-27T09:30:36.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:35 smithi050 ceph-mon[36371]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:36.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:35 smithi189 ceph-mon[36671]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:36.224 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:30:35 smithi044 bash[72055]: b861e9014d39d2a93efcde4fd5685d818f3fdd98ae82ce6999ad371cfe8516ef 2024-05-27T09:30:36.224 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:30:35 smithi044 systemd[1]: Started Ceph mds.j for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:30:36.224 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:30:35 smithi044 ceph-mds[72074]: -- 172.21.15.44:0/1624516654 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5640b1edfa40 con 0x5640b1f27180 2024-05-27T09:30:36.224 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:30:35 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-j[72070]: starting mds.j at 2024-05-27T09:30:37.287 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:30:37.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:37.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:37.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:37.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:37.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:37.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:37.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:37 smithi050 ceph-mon[36371]: Deploying daemon mds.d on smithi044 2024-05-27T09:30:37.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:37 smithi050 ceph-mon[36371]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:37.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:37 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.44:6834/907929206,v1:172.21.15.44:6835/907929206] up:boot 2024-05-27T09:30:37.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:37 smithi050 ceph-mon[36371]: fsmap 3 up:standby 2024-05-27T09:30:37.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:37 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T09:30:37.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:37.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:37.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:37.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:37.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:37.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:37.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:37 smithi189 ceph-mon[36671]: Deploying daemon mds.d on smithi044 2024-05-27T09:30:37.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:37 smithi189 ceph-mon[36671]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:37.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:37 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.44:6834/907929206,v1:172.21.15.44:6835/907929206] up:boot 2024-05-27T09:30:37.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:37 smithi189 ceph-mon[36671]: fsmap 3 up:standby 2024-05-27T09:30:37.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:37 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T09:30:37.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:37.367 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:37.367 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:37.367 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:37.367 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:37.367 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:37.367 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:37 smithi044 ceph-mon[31618]: Deploying daemon mds.d on smithi044 2024-05-27T09:30:37.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:37 smithi044 ceph-mon[31618]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:37.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:37 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.44:6834/907929206,v1:172.21.15.44:6835/907929206] up:boot 2024-05-27T09:30:37.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:37 smithi044 ceph-mon[31618]: fsmap 3 up:standby 2024-05-27T09:30:37.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:37 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T09:30:38.295 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:30:39.225 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:30:39 smithi044 systemd[1]: Starting Ceph mds.d for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:30:39.934 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:39 smithi044 ceph-mon[31618]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:39 smithi050 ceph-mon[36371]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:40.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:39 smithi189 ceph-mon[36671]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:40.186 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:30:39 smithi044 podman[72460]: 2024-05-27 09:30:39.933368819 +0000 UTC m=+0.123694950 container create dd646d323f4b969fa42a95f101fd5fcc58c98431e9cde0b4b18cd803d5d6d4c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-d, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T09:30:40.187 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:30:39 smithi044 podman[72460]: 2024-05-27 09:30:39.843888024 +0000 UTC m=+0.034214153 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:30:40.474 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:30:40 smithi044 podman[72460]: 2024-05-27 09:30:40.188799934 +0000 UTC m=+0.379126203 container init dd646d323f4b969fa42a95f101fd5fcc58c98431e9cde0b4b18cd803d5d6d4c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-27T09:30:40.475 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:30:40 smithi044 podman[72460]: 2024-05-27 09:30:40.199146913 +0000 UTC m=+0.389473086 container start dd646d323f4b969fa42a95f101fd5fcc58c98431e9cde0b4b18cd803d5d6d4c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-27T09:30:40.475 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:30:40 smithi044 bash[72460]: dd646d323f4b969fa42a95f101fd5fcc58c98431e9cde0b4b18cd803d5d6d4c4 2024-05-27T09:30:40.475 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:30:40 smithi044 systemd[1]: Started Ceph mds.d for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:30:40.475 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:30:40 smithi044 ceph-mds[72498]: -- 172.21.15.44:0/2328037746 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5560d4383a40 con 0x5560d43cb180 2024-05-27T09:30:40.475 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:30:40 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-d[72491]: starting mds.d at 2024-05-27T09:30:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.973+0000 7f915d000700 1 -- 172.21.15.44:0/3874140037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91581045e0 msgr2=0x7f91581049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.973+0000 7f915d000700 1 --2- 172.21.15.44:0/3874140037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91581045e0 0x7f91581049f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f91480099c0 tx=0x7f9148009cd0 comp rx=0 tx=0).stop 2024-05-27T09:30:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.973+0000 7f915d000700 1 -- 172.21.15.44:0/3874140037 shutdown_connections 2024-05-27T09:30:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.973+0000 7f915d000700 1 --2- 172.21.15.44:0/3874140037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9158109860 0x7f91581111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.973+0000 7f915d000700 1 --2- 172.21.15.44:0/3874140037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91581045e0 0x7f91581049f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.973+0000 7f915d000700 1 --2- 172.21.15.44:0/3874140037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9158108f10 0x7f9158109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.973+0000 7f915d000700 1 -- 172.21.15.44:0/3874140037 >> 172.21.15.44:0/3874140037 conn(0x7f9158100030 msgr2=0x7f9158102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.973+0000 7f915d000700 1 -- 172.21.15.44:0/3874140037 shutdown_connections 2024-05-27T09:30:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.973+0000 7f915d000700 1 -- 172.21.15.44:0/3874140037 wait complete. 2024-05-27T09:30:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.975+0000 7f915d000700 1 Processor -- start 2024-05-27T09:30:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.975+0000 7f915d000700 1 -- start start 2024-05-27T09:30:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.976+0000 7f915d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91581045e0 0x7f915819aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.976+0000 7f915d000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9158108f10 0x7f915819b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.976+0000 7f915d000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9158109860 0x7f915819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.976+0000 7f915d000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f91581144e0 con 0x7f9158109860 2024-05-27T09:30:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.976+0000 7f915d000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9158114360 con 0x7f9158108f10 2024-05-27T09:30:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.976+0000 7f915d000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9158114660 con 0x7f91581045e0 2024-05-27T09:30:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.976+0000 7f914d400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9158108f10 0x7f915819b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.976+0000 7f9156800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9158109860 0x7f915819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.976+0000 7f914d400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9158108f10 0x7f915819b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47168/0 (socket says 172.21.15.44:47168) 2024-05-27T09:30:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.976+0000 7f914d400700 1 -- 172.21.15.44:0/1806506999 learned_addr learned my addr 172.21.15.44:0/1806506999 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:30:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.976+0000 7f9155e00700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91581045e0 0x7f915819aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:40.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.977+0000 7f914d400700 1 -- 172.21.15.44:0/1806506999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91581045e0 msgr2=0x7f915819aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.977+0000 7f914d400700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91581045e0 0x7f915819aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.977+0000 7f914d400700 1 -- 172.21.15.44:0/1806506999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9158109860 msgr2=0x7f915819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.977+0000 7f914d400700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9158109860 0x7f915819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.977+0000 7f914d400700 1 -- 172.21.15.44:0/1806506999 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915819fd80 con 0x7f9158108f10 2024-05-27T09:30:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.977+0000 7f9156800700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9158109860 0x7f915819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:30:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.977+0000 7f914d400700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9158108f10 0x7f915819b450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f914000e970 tx=0x7f914000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.977+0000 7f914f400700 1 -- 172.21.15.44:0/1806506999 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140004a30 con 0x7f9158108f10 2024-05-27T09:30:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.978+0000 7f914f400700 1 -- 172.21.15.44:0/1806506999 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f914000cc90 con 0x7f9158108f10 2024-05-27T09:30:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.978+0000 7f915d000700 1 -- 172.21.15.44:0/1806506999 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91581a0070 con 0x7f9158108f10 2024-05-27T09:30:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.978+0000 7f914f400700 1 -- 172.21.15.44:0/1806506999 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140005610 con 0x7f9158108f10 2024-05-27T09:30:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.978+0000 7f915d000700 1 -- 172.21.15.44:0/1806506999 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91581a7990 con 0x7f9158108f10 2024-05-27T09:30:40.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.979+0000 7f915d000700 1 -- 172.21.15.44:0/1806506999 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91580512b0 con 0x7f9158108f10 2024-05-27T09:30:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.981+0000 7f914f400700 1 -- 172.21.15.44:0/1806506999 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f914000c7e0 con 0x7f9158108f10 2024-05-27T09:30:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.983+0000 7f914f400700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f913c0b2880 0x7f913c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.983+0000 7f914f400700 1 -- 172.21.15.44:0/1806506999 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f91400e2220 con 0x7f9158108f10 2024-05-27T09:30:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.983+0000 7f9155e00700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f913c0b2880 0x7f913c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.984+0000 7f9155e00700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f913c0b2880 0x7f913c0b4d90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9148003760 tx=0x7f914801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:40.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:40.986+0000 7f914f400700 1 -- 172.21.15.44:0/1806506999 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9140054df0 con 0x7f9158108f10 2024-05-27T09:30:41.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.275+0000 7f915d000700 1 -- 172.21.15.44:0/1806506999 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f915819c210 con 0x7f9158108f10 2024-05-27T09:30:41.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.276+0000 7f914f400700 1 -- 172.21.15.44:0/1806506999 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3865 (secure 0 0 0) 0x7f9140091090 con 0x7f9158108f10 2024-05-27T09:30:41.276 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:30:41.276 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T09:30:41.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.279+0000 7f915d000700 1 -- 172.21.15.44:0/1806506999 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f913c0b2880 msgr2=0x7f913c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:41.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.279+0000 7f915d000700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f913c0b2880 0x7f913c0b4d90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9148003760 tx=0x7f914801a040 comp rx=0 tx=0).stop 2024-05-27T09:30:41.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.279+0000 7f915d000700 1 -- 172.21.15.44:0/1806506999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9158108f10 msgr2=0x7f915819b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:41.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.279+0000 7f915d000700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9158108f10 0x7f915819b450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f914000e970 tx=0x7f914000ec80 comp rx=0 tx=0).stop 2024-05-27T09:30:41.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.279+0000 7f915d000700 1 -- 172.21.15.44:0/1806506999 shutdown_connections 2024-05-27T09:30:41.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.279+0000 7f915d000700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f913c0b2880 0x7f913c0b4d90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:41.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.279+0000 7f915d000700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9158109860 0x7f915819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:41.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.279+0000 7f915d000700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91581045e0 0x7f915819aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:41.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.279+0000 7f915d000700 1 --2- 172.21.15.44:0/1806506999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9158108f10 0x7f915819b450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:41.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.279+0000 7f915d000700 1 -- 172.21.15.44:0/1806506999 >> 172.21.15.44:0/1806506999 conn(0x7f9158100030 msgr2=0x7f9158108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:41.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.280+0000 7f915d000700 1 -- 172.21.15.44:0/1806506999 shutdown_connections 2024-05-27T09:30:41.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:41.280+0000 7f915d000700 1 -- 172.21.15.44:0/1806506999 wait complete. 2024-05-27T09:30:41.282 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 5 2024-05-27T09:30:41.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:41.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:41.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:41.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:41.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:41.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:41.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: Deploying daemon mds.e on smithi050 2024-05-27T09:30:41.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:41.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.44:6836/3467964892,v1:172.21.15.44:6837/3467964892] up:boot 2024-05-27T09:30:41.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: fsmap 4 up:standby 2024-05-27T09:30:41.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T09:30:41.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:41 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1806506999' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:41.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: Deploying daemon mds.e on smithi050 2024-05-27T09:30:41.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:41.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.44:6836/3467964892,v1:172.21.15.44:6837/3467964892] up:boot 2024-05-27T09:30:41.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: fsmap 4 up:standby 2024-05-27T09:30:41.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T09:30:41.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:41 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1806506999' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: Deploying daemon mds.e on smithi050 2024-05-27T09:30:41.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:41.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.44:6836/3467964892,v1:172.21.15.44:6837/3467964892] up:boot 2024-05-27T09:30:41.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: fsmap 4 up:standby 2024-05-27T09:30:41.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T09:30:41.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:41 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1806506999' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:42.785 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:30:42 smithi050 systemd[1]: Starting Ceph mds.e for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:30:43.039 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:30:42 smithi050 podman[66086]: 2024-05-27 09:30:42.784985799 +0000 UTC m=+0.113771572 container create 9e88f59da4bece838578c28d8f3285bfce288600bf52ab36adbae7018259f203 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-05-27T09:30:43.039 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:30:42 smithi050 podman[66086]: 2024-05-27 09:30:42.688740328 +0000 UTC m=+0.017526102 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:30:43.039 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:30:42 smithi050 podman[66086]: 2024-05-27 09:30:42.963342123 +0000 UTC m=+0.292127911 container init 9e88f59da4bece838578c28d8f3285bfce288600bf52ab36adbae7018259f203 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:30:43.039 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:30:42 smithi050 podman[66086]: 2024-05-27 09:30:42.965985339 +0000 UTC m=+0.294771117 container start 9e88f59da4bece838578c28d8f3285bfce288600bf52ab36adbae7018259f203 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-27T09:30:43.039 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:30:42 smithi050 ceph-mds[66105]: -- 172.21.15.50:0/572339189 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a2107cfa40 con 0x55a210817180 2024-05-27T09:30:43.039 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:30:43 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e[66101]: starting mds.e at 2024-05-27T09:30:43.039 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:30:43 smithi050 bash[66086]: 9e88f59da4bece838578c28d8f3285bfce288600bf52ab36adbae7018259f203 2024-05-27T09:30:43.293 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:30:43 smithi050 systemd[1]: Started Ceph mds.e for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:30:43.958 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:43 smithi050 ceph-mon[36371]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:43.958 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:43.958 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:43.958 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:43.958 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:43.958 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:43.958 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:43 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:43 smithi044 ceph-mon[31618]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:43 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:43.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:43 smithi189 ceph-mon[36671]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:43.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:43.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:44.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:44.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:44.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:44.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:43 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:44.066 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:30:44.535 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:30:44.941 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:44 smithi044 ceph-mon[31618]: Deploying daemon mds.f on smithi189 2024-05-27T09:30:44.942 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:44 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6834/1092513236,v1:172.21.15.50:6835/1092513236] up:boot 2024-05-27T09:30:44.942 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:44 smithi044 ceph-mon[31618]: fsmap 5 up:standby 2024-05-27T09:30:44.942 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:44 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T09:30:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:44 smithi050 ceph-mon[36371]: Deploying daemon mds.f on smithi189 2024-05-27T09:30:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:44 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6834/1092513236,v1:172.21.15.50:6835/1092513236] up:boot 2024-05-27T09:30:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:44 smithi050 ceph-mon[36371]: fsmap 5 up:standby 2024-05-27T09:30:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:44 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T09:30:45.077 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:44 smithi189 ceph-mon[36671]: Deploying daemon mds.f on smithi189 2024-05-27T09:30:45.077 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:44 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6834/1092513236,v1:172.21.15.50:6835/1092513236] up:boot 2024-05-27T09:30:45.077 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:44 smithi189 ceph-mon[36671]: fsmap 5 up:standby 2024-05-27T09:30:45.077 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:44 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T09:30:45.338 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:30:45 smithi189 systemd[1]: Starting Ceph mds.f for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:30:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:45 smithi044 ceph-mon[31618]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:45.774 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:45 smithi189 ceph-mon[36671]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:45.774 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:30:45 smithi189 podman[67233]: 2024-05-27 09:30:45.378694608 +0000 UTC m=+0.017475301 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:30:45.774 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:30:45 smithi189 podman[67233]: 2024-05-27 09:30:45.480290102 +0000 UTC m=+0.119070784 container create 4d1bcbfcf9a87dbbb87c71b3eb218a2eea56b27b2e064f1bd2114be588fcac51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T09:30:45.775 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:30:45 smithi189 podman[67233]: 2024-05-27 09:30:45.68276005 +0000 UTC m=+0.321540738 container init 4d1bcbfcf9a87dbbb87c71b3eb218a2eea56b27b2e064f1bd2114be588fcac51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-05-27T09:30:45.775 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:30:45 smithi189 podman[67233]: 2024-05-27 09:30:45.685542265 +0000 UTC m=+0.324322959 container start 4d1bcbfcf9a87dbbb87c71b3eb218a2eea56b27b2e064f1bd2114be588fcac51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:30:45.775 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:30:45 smithi189 ceph-mds[67252]: -- 172.21.15.189:0/3178426393 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56175825da40 con 0x5617582a5a80 2024-05-27T09:30:45.775 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:30:45 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[67248]: starting mds.f at 2024-05-27T09:30:45.775 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:30:45 smithi189 bash[67233]: 4d1bcbfcf9a87dbbb87c71b3eb218a2eea56b27b2e064f1bd2114be588fcac51 2024-05-27T09:30:46.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:45 smithi050 ceph-mon[36371]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:46.087 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:30:45 smithi189 systemd[1]: Started Ceph mds.f for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:30:46.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.758+0000 7fe965a00700 1 -- 172.21.15.44:0/492705805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe960108d20 msgr2=0x7fe960109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:46.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.758+0000 7fe965a00700 1 --2- 172.21.15.44:0/492705805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe960108d20 0x7fe960109130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe94800b0d0 tx=0x7fe94800b3e0 comp rx=0 tx=0).stop 2024-05-27T09:30:46.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.758+0000 7fe965a00700 1 -- 172.21.15.44:0/492705805 shutdown_connections 2024-05-27T09:30:46.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.758+0000 7fe965a00700 1 --2- 172.21.15.44:0/492705805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe960109670 0x7fe960110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:46.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.758+0000 7fe965a00700 1 --2- 172.21.15.44:0/492705805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9601043f0 0x7fe960104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:46.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.758+0000 7fe965a00700 1 --2- 172.21.15.44:0/492705805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe960108d20 0x7fe960109130 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:46.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.758+0000 7fe965a00700 1 -- 172.21.15.44:0/492705805 >> 172.21.15.44:0/492705805 conn(0x7fe960079d00 msgr2=0x7fe96007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:46.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.758+0000 7fe965a00700 1 -- 172.21.15.44:0/492705805 shutdown_connections 2024-05-27T09:30:46.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.758+0000 7fe965a00700 1 -- 172.21.15.44:0/492705805 wait complete. 2024-05-27T09:30:46.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.759+0000 7fe965a00700 1 Processor -- start 2024-05-27T09:30:46.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.759+0000 7fe965a00700 1 -- start start 2024-05-27T09:30:46.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.760+0000 7fe965a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9601043f0 0x7fe9601a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:46.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.760+0000 7fe965a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe960108d20 0x7fe9601a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:46.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.760+0000 7fe965a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe960109670 0x7fe96019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:46.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.760+0000 7fe965a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe960114280 con 0x7fe9601043f0 2024-05-27T09:30:46.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.760+0000 7fe965a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe960114100 con 0x7fe960109670 2024-05-27T09:30:46.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.760+0000 7fe965a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe960114400 con 0x7fe960108d20 2024-05-27T09:30:46.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.760+0000 7fe95fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9601043f0 0x7fe9601a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:46.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.760+0000 7fe957400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe960108d20 0x7fe9601a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.760+0000 7fe95fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9601043f0 0x7fe9601a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57700/0 (socket says 172.21.15.44:57700) 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.760+0000 7fe95fe00700 1 -- 172.21.15.44:0/3566572770 learned_addr learned my addr 172.21.15.44:0/3566572770 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.761+0000 7fe965000700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe960109670 0x7fe96019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.761+0000 7fe957400700 1 -- 172.21.15.44:0/3566572770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe960109670 msgr2=0x7fe96019aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.761+0000 7fe957400700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe960109670 0x7fe96019aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.761+0000 7fe957400700 1 -- 172.21.15.44:0/3566572770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9601043f0 msgr2=0x7fe9601a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.761+0000 7fe957400700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9601043f0 0x7fe9601a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.761+0000 7fe957400700 1 -- 172.21.15.44:0/3566572770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe96019b150 con 0x7fe960108d20 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.761+0000 7fe965000700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe960109670 0x7fe96019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.761+0000 7fe957400700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe960108d20 0x7fe9601a0dc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe948000f80 tx=0x7fe948007a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.762+0000 7fe95d600700 1 -- 172.21.15.44:0/3566572770 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe948003c00 con 0x7fe960108d20 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.762+0000 7fe95d600700 1 -- 172.21.15.44:0/3566572770 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe948022070 con 0x7fe960108d20 2024-05-27T09:30:46.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.762+0000 7fe95d600700 1 -- 172.21.15.44:0/3566572770 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94801b6c0 con 0x7fe960108d20 2024-05-27T09:30:46.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.762+0000 7fe965a00700 1 -- 172.21.15.44:0/3566572770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe96019b3e0 con 0x7fe960108d20 2024-05-27T09:30:46.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.762+0000 7fe965a00700 1 -- 172.21.15.44:0/3566572770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe96019b870 con 0x7fe960108d20 2024-05-27T09:30:46.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.763+0000 7fe965a00700 1 -- 172.21.15.44:0/3566572770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9601058a0 con 0x7fe960108d20 2024-05-27T09:30:46.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.766+0000 7fe95d600700 1 -- 172.21.15.44:0/3566572770 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe94801d470 con 0x7fe960108d20 2024-05-27T09:30:46.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.767+0000 7fe95d600700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe9440b2900 0x7fe9440b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:46.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.767+0000 7fe95d600700 1 -- 172.21.15.44:0/3566572770 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fe94800f020 con 0x7fe960108d20 2024-05-27T09:30:46.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.767+0000 7fe95fe00700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe9440b2900 0x7fe9440b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:46.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.768+0000 7fe95fe00700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe9440b2900 0x7fe9440b4e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe96019bbe0 tx=0x7fe950007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:46.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:46.772+0000 7fe95d600700 1 -- 172.21.15.44:0/3566572770 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe9480aad40 con 0x7fe960108d20 2024-05-27T09:30:47.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.064+0000 7fe965a00700 1 -- 172.21.15.44:0/3566572770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe96002e5e0 con 0x7fe960108d20 2024-05-27T09:30:47.066 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:47.066 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:47.066 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:47.066 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:47.066 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:47.066 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:47.066 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:46 smithi044 ceph-mon[31618]: Deploying daemon mds.c on smithi189 2024-05-27T09:30:47.066 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:46 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6834/2673083839,v1:172.21.15.189:6835/2673083839] up:boot 2024-05-27T09:30:47.067 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:46 smithi044 ceph-mon[31618]: fsmap 6 up:standby 2024-05-27T09:30:47.067 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:46 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:30:47.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.065+0000 7fe95d600700 1 -- 172.21.15.44:0/3566572770 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7fe9480aa640 con 0x7fe960108d20 2024-05-27T09:30:47.067 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:30:47.067 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-05-27T09:30:47.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.068+0000 7fe965a00700 1 -- 172.21.15.44:0/3566572770 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe9440b2900 msgr2=0x7fe9440b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:47.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.068+0000 7fe965a00700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe9440b2900 0x7fe9440b4e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe96019bbe0 tx=0x7fe950007480 comp rx=0 tx=0).stop 2024-05-27T09:30:47.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.068+0000 7fe965a00700 1 -- 172.21.15.44:0/3566572770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe960108d20 msgr2=0x7fe9601a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:47.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.068+0000 7fe965a00700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe960108d20 0x7fe9601a0dc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe948000f80 tx=0x7fe948007a20 comp rx=0 tx=0).stop 2024-05-27T09:30:47.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.068+0000 7fe965a00700 1 -- 172.21.15.44:0/3566572770 shutdown_connections 2024-05-27T09:30:47.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.068+0000 7fe965a00700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe9440b2900 0x7fe9440b4e10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:47.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.068+0000 7fe965a00700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe960109670 0x7fe96019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:47.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.068+0000 7fe965a00700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9601043f0 0x7fe9601a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:47.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.068+0000 7fe965a00700 1 --2- 172.21.15.44:0/3566572770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe960108d20 0x7fe9601a0dc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:47.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.068+0000 7fe965a00700 1 -- 172.21.15.44:0/3566572770 >> 172.21.15.44:0/3566572770 conn(0x7fe960079d00 msgr2=0x7fe960101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:47.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.069+0000 7fe965a00700 1 -- 172.21.15.44:0/3566572770 shutdown_connections 2024-05-27T09:30:47.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:47.069+0000 7fe965a00700 1 -- 172.21.15.44:0/3566572770 wait complete. 2024-05-27T09:30:47.071 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 7 2024-05-27T09:30:47.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:47.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:47.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:47.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:47.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:47.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:47.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:46 smithi189 ceph-mon[36671]: Deploying daemon mds.c on smithi189 2024-05-27T09:30:47.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:46 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6834/2673083839,v1:172.21.15.189:6835/2673083839] up:boot 2024-05-27T09:30:47.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:46 smithi189 ceph-mon[36671]: fsmap 6 up:standby 2024-05-27T09:30:47.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:46 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:30:47.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:47.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:47.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:47.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:47.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:47.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:47.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:46 smithi050 ceph-mon[36371]: Deploying daemon mds.c on smithi189 2024-05-27T09:30:47.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:46 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6834/2673083839,v1:172.21.15.189:6835/2673083839] up:boot 2024-05-27T09:30:47.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:46 smithi050 ceph-mon[36371]: fsmap 6 up:standby 2024-05-27T09:30:47.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:46 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:30:48.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:47 smithi189 ceph-mon[36671]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:48.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:47 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3566572770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:48.088 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:30:47 smithi189 systemd[1]: Starting Ceph mds.c for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:30:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:47 smithi044 ceph-mon[31618]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:47 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3566572770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:48.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:47 smithi050 ceph-mon[36371]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:48.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:47 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3566572770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:48.498 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:30:48 smithi189 podman[67537]: 2024-05-27 09:30:48.131144279 +0000 UTC m=+0.017800193 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:30:48.498 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:30:48 smithi189 podman[67537]: 2024-05-27 09:30:48.243126062 +0000 UTC m=+0.129782019 container create 9889fd83bbe81c8c6788dcb7d2ef6aabf433e84410a5cad19776ffc2e461df8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-27T09:30:48.498 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:30:48 smithi189 podman[67537]: 2024-05-27 09:30:48.462156018 +0000 UTC m=+0.348811933 container init 9889fd83bbe81c8c6788dcb7d2ef6aabf433e84410a5cad19776ffc2e461df8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-27T09:30:48.498 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:30:48 smithi189 podman[67537]: 2024-05-27 09:30:48.464737378 +0000 UTC m=+0.351393299 container start 9889fd83bbe81c8c6788dcb7d2ef6aabf433e84410a5cad19776ffc2e461df8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c, ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T09:30:48.837 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:30:48 smithi189 ceph-mds[67556]: -- 172.21.15.189:0/1613949309 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55729101ba40 con 0x557291063600 2024-05-27T09:30:48.838 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:30:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c[67552]: starting mds.c at 2024-05-27T09:30:48.838 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:30:48 smithi189 bash[67537]: 9889fd83bbe81c8c6788dcb7d2ef6aabf433e84410a5cad19776ffc2e461df8a 2024-05-27T09:30:48.838 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:30:48 smithi189 systemd[1]: Started Ceph mds.c for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:30:49.678 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:49.678 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:49.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:49.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:49.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:49.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:49.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:49.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: Deploying daemon mds.h on smithi050 2024-05-27T09:30:49.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] up:boot 2024-05-27T09:30:49.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: fsmap 7 up:standby 2024-05-27T09:30:49.679 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:30:49.680 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:49 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:30:49.901 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:30:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: Deploying daemon mds.h on smithi050 2024-05-27T09:30:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] up:boot 2024-05-27T09:30:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: fsmap 7 up:standby 2024-05-27T09:30:49.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:30:49.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:49 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: Deploying daemon mds.h on smithi050 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] up:boot 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: fsmap 7 up:standby 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:30:50.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:49 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:30:50.381 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:30:51.026 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:30:50 smithi050 systemd[1]: Starting Ceph mds.h for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:30:51.294 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:30:51 smithi050 podman[66391]: 2024-05-27 09:30:51.025558158 +0000 UTC m=+0.104158341 container create 9d1fd8d5f05aab4d3a98300187dc17e53241c6a5b3735a48b3ea54e601aa2f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-27T09:30:51.294 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:30:51 smithi050 podman[66391]: 2024-05-27 09:30:50.939363135 +0000 UTC m=+0.017963294 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:30:51.294 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:30:51 smithi050 podman[66391]: 2024-05-27 09:30:51.236425472 +0000 UTC m=+0.315025612 container init 9d1fd8d5f05aab4d3a98300187dc17e53241c6a5b3735a48b3ea54e601aa2f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, CEPH_POINT_RELEASE=) 2024-05-27T09:30:51.294 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:30:51 smithi050 podman[66391]: 2024-05-27 09:30:51.239017816 +0000 UTC m=+0.317617964 container start 9d1fd8d5f05aab4d3a98300187dc17e53241c6a5b3735a48b3ea54e601aa2f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-27T09:30:51.295 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:30:51 smithi050 ceph-mds[66410]: -- 172.21.15.50:0/1409702141 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565410a73a40 con 0x565410abb180 2024-05-27T09:30:51.295 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:30:51 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h[66406]: starting mds.h at 2024-05-27T09:30:51.617 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:30:51 smithi050 bash[66391]: 9d1fd8d5f05aab4d3a98300187dc17e53241c6a5b3735a48b3ea54e601aa2f59 2024-05-27T09:30:51.618 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:30:51 smithi050 systemd[1]: Started Ceph mds.h for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:30:51.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:51 smithi044 ceph-mon[31618]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:51.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:51 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:51.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:51 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:51.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:51 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:51.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:51 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:51.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:51 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:51.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:51 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:51 smithi050 ceph-mon[36371]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:51 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:51 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:51 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:51 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:51 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:51 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:52.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:51 smithi189 ceph-mon[36671]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:52.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:51 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:52.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:51 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:52.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:51 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:52.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:51 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:52.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:51 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:52.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:51 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:52.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.655+0000 7f67d6a00700 1 -- 172.21.15.44:0/32303755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67d0108f10 msgr2=0x7f67d0109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:52.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.655+0000 7f67d6a00700 1 --2- 172.21.15.44:0/32303755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67d0108f10 0x7f67d0109320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f67b8009990 tx=0x7f67b8009ca0 comp rx=0 tx=0).stop 2024-05-27T09:30:52.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.655+0000 7f67d6a00700 1 -- 172.21.15.44:0/32303755 shutdown_connections 2024-05-27T09:30:52.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.655+0000 7f67d6a00700 1 --2- 172.21.15.44:0/32303755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67d0109860 0x7f67d01111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:52.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.655+0000 7f67d6a00700 1 --2- 172.21.15.44:0/32303755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d01045e0 0x7f67d01049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:52.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.655+0000 7f67d6a00700 1 --2- 172.21.15.44:0/32303755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67d0108f10 0x7f67d0109320 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:52.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.655+0000 7f67d6a00700 1 -- 172.21.15.44:0/32303755 >> 172.21.15.44:0/32303755 conn(0x7f67d0100030 msgr2=0x7f67d0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:52.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.655+0000 7f67d6a00700 1 -- 172.21.15.44:0/32303755 shutdown_connections 2024-05-27T09:30:52.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.655+0000 7f67d6a00700 1 -- 172.21.15.44:0/32303755 wait complete. 2024-05-27T09:30:52.658 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:52 smithi044 ceph-mon[31618]: Deploying daemon mds.i on smithi189 2024-05-27T09:30:52.658 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:52 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:boot 2024-05-27T09:30:52.659 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:52 smithi044 ceph-mon[31618]: fsmap 8 up:standby 2024-05-27T09:30:52.659 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:52 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T09:30:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.657+0000 7f67d6a00700 1 Processor -- start 2024-05-27T09:30:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.657+0000 7f67d6a00700 1 -- start start 2024-05-27T09:30:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.657+0000 7f67d6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d01045e0 0x7f67d019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.657+0000 7f67d6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67d0108f10 0x7f67d019b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.657+0000 7f67d6a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67d0109860 0x7f67d019f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67d6a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f67d0114470 con 0x7f67d0108f10 2024-05-27T09:30:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67d6a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f67d01142f0 con 0x7f67d0109860 2024-05-27T09:30:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67d6a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f67d01145f0 con 0x7f67d01045e0 2024-05-27T09:30:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67c7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67d0108f10 0x7f67d019b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67cfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d01045e0 0x7f67d019ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67cfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d01045e0 0x7f67d019ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:37432/0 (socket says 172.21.15.44:37432) 2024-05-27T09:30:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67cfe00700 1 -- 172.21.15.44:0/1523032283 learned_addr learned my addr 172.21.15.44:0/1523032283 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:30:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67d4a00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67d0109860 0x7f67d019f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67cfe00700 1 -- 172.21.15.44:0/1523032283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67d0109860 msgr2=0x7f67d019f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:52.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67cfe00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67d0109860 0x7f67d019f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:52.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67cfe00700 1 -- 172.21.15.44:0/1523032283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67d0108f10 msgr2=0x7f67d019b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:52.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67cfe00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67d0108f10 0x7f67d019b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:52.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67cfe00700 1 -- 172.21.15.44:0/1523032283 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d019fcf0 con 0x7f67d01045e0 2024-05-27T09:30:52.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.658+0000 7f67d4a00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67d0109860 0x7f67d019f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:30:52.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.659+0000 7f67cfe00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d01045e0 0x7f67d019ae20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f67c000eb30 tx=0x7f67c000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:52.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.659+0000 7f67cd600700 1 -- 172.21.15.44:0/1523032283 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c000cc90 con 0x7f67d01045e0 2024-05-27T09:30:52.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.659+0000 7f67cd600700 1 -- 172.21.15.44:0/1523032283 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67c00044d0 con 0x7f67d01045e0 2024-05-27T09:30:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.659+0000 7f67d6a00700 1 -- 172.21.15.44:0/1523032283 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d019ffe0 con 0x7f67d01045e0 2024-05-27T09:30:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.659+0000 7f67cd600700 1 -- 172.21.15.44:0/1523032283 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0010430 con 0x7f67d01045e0 2024-05-27T09:30:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.659+0000 7f67d6a00700 1 -- 172.21.15.44:0/1523032283 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d01a7900 con 0x7f67d01045e0 2024-05-27T09:30:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.660+0000 7f67d6a00700 1 -- 172.21.15.44:0/1523032283 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67d00512b0 con 0x7f67d01045e0 2024-05-27T09:30:52.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.662+0000 7f67cd600700 1 -- 172.21.15.44:0/1523032283 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f67c0003ea0 con 0x7f67d01045e0 2024-05-27T09:30:52.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.664+0000 7f67cd600700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f67b40b2880 0x7f67b40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:52.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.664+0000 7f67cd600700 1 -- 172.21.15.44:0/1523032283 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f67c00e20b0 con 0x7f67d01045e0 2024-05-27T09:30:52.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.664+0000 7f67c7e00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f67b40b2880 0x7f67b40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:52.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.665+0000 7f67c7e00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f67b40b2880 0x7f67b40b4d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f67b800b2e0 tx=0x7f67b8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.668+0000 7f67cd600700 1 -- 172.21.15.44:0/1523032283 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f67c00aa1d0 con 0x7f67d01045e0 2024-05-27T09:30:52.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:52 smithi189 ceph-mon[36671]: Deploying daemon mds.i on smithi189 2024-05-27T09:30:52.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:52 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:boot 2024-05-27T09:30:52.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:52 smithi189 ceph-mon[36671]: fsmap 8 up:standby 2024-05-27T09:30:52.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:52 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T09:30:52.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.960+0000 7f67d6a00700 1 -- 172.21.15.44:0/1523032283 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67d019c180 con 0x7f67d01045e0 2024-05-27T09:30:52.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.961+0000 7f67cd600700 1 -- 172.21.15.44:0/1523032283 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7f67c00a9ad0 con 0x7f67d01045e0 2024-05-27T09:30:52.962 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:30:52.963 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-27T09:30:52.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.965+0000 7f67d6a00700 1 -- 172.21.15.44:0/1523032283 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f67b40b2880 msgr2=0x7f67b40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:52.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.965+0000 7f67d6a00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f67b40b2880 0x7f67b40b4d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f67b800b2e0 tx=0x7f67b8005960 comp rx=0 tx=0).stop 2024-05-27T09:30:52.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.965+0000 7f67d6a00700 1 -- 172.21.15.44:0/1523032283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d01045e0 msgr2=0x7f67d019ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:52.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.965+0000 7f67d6a00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d01045e0 0x7f67d019ae20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f67c000eb30 tx=0x7f67c000ee40 comp rx=0 tx=0).stop 2024-05-27T09:30:52.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.965+0000 7f67d6a00700 1 -- 172.21.15.44:0/1523032283 shutdown_connections 2024-05-27T09:30:52.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.965+0000 7f67d6a00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f67b40b2880 0x7f67b40b4d90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:52.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.965+0000 7f67d6a00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67d0109860 0x7f67d019f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:52.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.965+0000 7f67d6a00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d01045e0 0x7f67d019ae20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:52.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.965+0000 7f67d6a00700 1 --2- 172.21.15.44:0/1523032283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67d0108f10 0x7f67d019b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:52.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.965+0000 7f67d6a00700 1 -- 172.21.15.44:0/1523032283 >> 172.21.15.44:0/1523032283 conn(0x7f67d0100030 msgr2=0x7f67d0108650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:52.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.966+0000 7f67d6a00700 1 -- 172.21.15.44:0/1523032283 shutdown_connections 2024-05-27T09:30:52.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:52.966+0000 7f67d6a00700 1 -- 172.21.15.44:0/1523032283 wait complete. 2024-05-27T09:30:52.968 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 9 2024-05-27T09:30:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:52 smithi050 ceph-mon[36371]: Deploying daemon mds.i on smithi189 2024-05-27T09:30:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:52 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:boot 2024-05-27T09:30:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:52 smithi050 ceph-mon[36371]: fsmap 8 up:standby 2024-05-27T09:30:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:52 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T09:30:53.645 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:30:53 smithi189 systemd[1]: Starting Ceph mds.i for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:30:53.968 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:53 smithi189 ceph-mon[36671]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:53.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:53 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1523032283' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:53.969 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:30:53 smithi189 podman[67843]: 2024-05-27 09:30:53.68699191 +0000 UTC m=+0.017295152 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:30:53.969 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:30:53 smithi189 podman[67843]: 2024-05-27 09:30:53.790658932 +0000 UTC m=+0.120962160 container create 05084cdaaf8ecba1b4f72d59ce45a1f33201ef66927b997adc959422eedf1e89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T09:30:53.969 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:30:53 smithi189 podman[67843]: 2024-05-27 09:30:53.968660054 +0000 UTC m=+0.298963286 container init 05084cdaaf8ecba1b4f72d59ce45a1f33201ef66927b997adc959422eedf1e89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:30:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:53 smithi044 ceph-mon[31618]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:53 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1523032283' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:54.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:53 smithi050 ceph-mon[36371]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:54.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:53 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1523032283' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:30:54.337 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:30:53 smithi189 podman[67843]: 2024-05-27 09:30:53.971644431 +0000 UTC m=+0.301947666 container start 05084cdaaf8ecba1b4f72d59ce45a1f33201ef66927b997adc959422eedf1e89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-05-27T09:30:54.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:30:54 smithi189 ceph-mds[67862]: -- 172.21.15.189:0/352099822 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d369bdfa40 con 0x55d369c27a80 2024-05-27T09:30:54.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:30:54 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: starting mds.i at 2024-05-27T09:30:54.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:30:54 smithi189 bash[67843]: 05084cdaaf8ecba1b4f72d59ce45a1f33201ef66927b997adc959422eedf1e89 2024-05-27T09:30:54.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:30:54 smithi189 systemd[1]: Started Ceph mds.i for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:30:55.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:55 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:55.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:55 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:55.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:55 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:55.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:55 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:55.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:55 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:55.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:55 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:55.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:55 smithi189 ceph-mon[36671]: Deploying daemon mds.g on smithi044 2024-05-27T09:30:55.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:55 smithi189 ceph-mon[36671]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:55.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:55 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] up:boot 2024-05-27T09:30:55.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:55 smithi189 ceph-mon[36671]: fsmap 9 up:standby 2024-05-27T09:30:55.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:55 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:30:55.362 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:55 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:55.362 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:55 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:55.362 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:55 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:55.362 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:55 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:55.362 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:55 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:55.362 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:55 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:55.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:55 smithi044 ceph-mon[31618]: Deploying daemon mds.g on smithi044 2024-05-27T09:30:55.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:55 smithi044 ceph-mon[31618]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:55.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:55 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] up:boot 2024-05-27T09:30:55.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:55 smithi044 ceph-mon[31618]: fsmap 9 up:standby 2024-05-27T09:30:55.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:55 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:30:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:55 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:55 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:55 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:55 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:55 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:55 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:55 smithi050 ceph-mon[36371]: Deploying daemon mds.g on smithi044 2024-05-27T09:30:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:55 smithi050 ceph-mon[36371]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:55 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] up:boot 2024-05-27T09:30:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:55 smithi050 ceph-mon[36371]: fsmap 9 up:standby 2024-05-27T09:30:55.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:55 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:30:55.788 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:30:56.491 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:30:57.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:57 smithi044 ceph-mon[31618]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:58.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:57 smithi050 ceph-mon[36371]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:58.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:57 smithi189 ceph-mon[36671]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:58.280 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:30:58 smithi044 podman[73229]: 2024-05-27 09:30:58.054968015 +0000 UTC m=+0.162376447 container create 35e81d080ca225250602f940cb6dafe2ba812955b6eee09bcdf7bb8405937d1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-g, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T09:30:58.544 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:30:58 smithi044 podman[73229]: 2024-05-27 09:30:58.401356074 +0000 UTC m=+0.508764513 container init 35e81d080ca225250602f940cb6dafe2ba812955b6eee09bcdf7bb8405937d1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux ) 2024-05-27T09:30:58.544 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:30:58 smithi044 podman[73229]: 2024-05-27 09:30:58.416274042 +0000 UTC m=+0.523682501 container start 35e81d080ca225250602f940cb6dafe2ba812955b6eee09bcdf7bb8405937d1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-g, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-27T09:30:58.803 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:30:58 smithi044 ceph-mds[73264]: -- 172.21.15.44:0/1441266248 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f17e203a40 con 0x55f17e24ba80 2024-05-27T09:30:58.804 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:30:58 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-g[73258]: starting mds.g at 2024-05-27T09:30:58.804 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:30:58 smithi044 bash[73229]: 35e81d080ca225250602f940cb6dafe2ba812955b6eee09bcdf7bb8405937d1e 2024-05-27T09:30:58.804 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:30:58 smithi044 systemd[1]: Started Ceph mds.g for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:30:58.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.964+0000 7fc31a400700 1 -- 172.21.15.44:0/2427226540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc314109850 msgr2=0x7fc3141111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:58.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.964+0000 7fc31a400700 1 --2- 172.21.15.44:0/2427226540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc314109850 0x7fc3141111b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00b0d0 tx=0x7fc30c00b3e0 comp rx=0 tx=0).stop 2024-05-27T09:30:58.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.965+0000 7fc31a400700 1 -- 172.21.15.44:0/2427226540 shutdown_connections 2024-05-27T09:30:58.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.965+0000 7fc31a400700 1 --2- 172.21.15.44:0/2427226540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc314109850 0x7fc3141111b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:58.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.965+0000 7fc31a400700 1 --2- 172.21.15.44:0/2427226540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3141045d0 0x7fc3141049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:58.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.965+0000 7fc31a400700 1 --2- 172.21.15.44:0/2427226540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314108f00 0x7fc314109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:58.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.965+0000 7fc31a400700 1 -- 172.21.15.44:0/2427226540 >> 172.21.15.44:0/2427226540 conn(0x7fc3140ffee0 msgr2=0x7fc314102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:58.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.965+0000 7fc31a400700 1 -- 172.21.15.44:0/2427226540 shutdown_connections 2024-05-27T09:30:58.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.965+0000 7fc31a400700 1 -- 172.21.15.44:0/2427226540 wait complete. 2024-05-27T09:30:58.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.966+0000 7fc31a400700 1 Processor -- start 2024-05-27T09:30:58.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.966+0000 7fc31a400700 1 -- start start 2024-05-27T09:30:58.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc31a400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3141045d0 0x7fc31419cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:58.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc31a400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314108f00 0x7fc31419d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:58.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc31a400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314109850 0x7fc3141a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:58.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc31a400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc314114480 con 0x7fc3141045d0 2024-05-27T09:30:58.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc31a400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc314114300 con 0x7fc314109850 2024-05-27T09:30:58.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc31a400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc314114600 con 0x7fc314108f00 2024-05-27T09:30:58.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc313400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3141045d0 0x7fc31419cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:58.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc30be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314108f00 0x7fc31419d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:58.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc30be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314108f00 0x7fc31419d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39994/0 (socket says 172.21.15.44:39994) 2024-05-27T09:30:58.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc30be00700 1 -- 172.21.15.44:0/3931090648 learned_addr learned my addr 172.21.15.44:0/3931090648 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:30:58.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc313400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3141045d0 0x7fc31419cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34666/0 (socket says 172.21.15.44:34666) 2024-05-27T09:30:58.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.967+0000 7fc313e00700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314109850 0x7fc3141a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:58.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.968+0000 7fc30be00700 1 -- 172.21.15.44:0/3931090648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314109850 msgr2=0x7fc3141a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:58.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.968+0000 7fc30be00700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314109850 0x7fc3141a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:58.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.968+0000 7fc30be00700 1 -- 172.21.15.44:0/3931090648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3141045d0 msgr2=0x7fc31419cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:58.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.968+0000 7fc30be00700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3141045d0 0x7fc31419cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:58.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.968+0000 7fc30be00700 1 -- 172.21.15.44:0/3931090648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3141a1ea0 con 0x7fc314108f00 2024-05-27T09:30:58.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.968+0000 7fc30be00700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314108f00 0x7fc31419d570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00ea30 tx=0x7fc2fc00ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:58.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.968+0000 7fc310c00700 1 -- 172.21.15.44:0/3931090648 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc00cbd0 con 0x7fc314108f00 2024-05-27T09:30:58.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.968+0000 7fc310c00700 1 -- 172.21.15.44:0/3931090648 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2fc0044d0 con 0x7fc314108f00 2024-05-27T09:30:58.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.968+0000 7fc31a400700 1 -- 172.21.15.44:0/3931090648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3141a2190 con 0x7fc314108f00 2024-05-27T09:30:58.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.968+0000 7fc310c00700 1 -- 172.21.15.44:0/3931090648 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc003dc0 con 0x7fc314108f00 2024-05-27T09:30:58.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.969+0000 7fc31a400700 1 -- 172.21.15.44:0/3931090648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3141a9ab0 con 0x7fc314108f00 2024-05-27T09:30:58.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.970+0000 7fc31a400700 1 -- 172.21.15.44:0/3931090648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3140512b0 con 0x7fc314108f00 2024-05-27T09:30:58.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.972+0000 7fc310c00700 1 -- 172.21.15.44:0/3931090648 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc2fc004000 con 0x7fc314108f00 2024-05-27T09:30:58.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.973+0000 7fc310c00700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc2f80b2900 0x7fc2f80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:30:58.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.974+0000 7fc310c00700 1 -- 172.21.15.44:0/3931090648 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fc2fc0e1e30 con 0x7fc314108f00 2024-05-27T09:30:58.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.974+0000 7fc313400700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc2f80b2900 0x7fc2f80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:30:58.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.974+0000 7fc313400700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc2f80b2900 0x7fc2f80b4e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc304009c60 tx=0x7fc304009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:30:58.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:58.978+0000 7fc310c00700 1 -- 172.21.15.44:0/3931090648 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc2fc0042f0 con 0x7fc314108f00 2024-05-27T09:30:59.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.269+0000 7fc31a400700 1 -- 172.21.15.44:0/3931090648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3141a9c80 con 0x7fc314108f00 2024-05-27T09:30:59.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.270+0000 7fc310c00700 1 -- 172.21.15.44:0/3931090648 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8824 (secure 0 0 0) 0x7fc2fc0a9f50 con 0x7fc314108f00 2024-05-27T09:30:59.272 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:30:59.272 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-27T09:30:59.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 -- 172.21.15.44:0/3931090648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc2f80b2900 msgr2=0x7fc2f80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:59.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc2f80b2900 0x7fc2f80b4e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc304009c60 tx=0x7fc304009260 comp rx=0 tx=0).stop 2024-05-27T09:30:59.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 -- 172.21.15.44:0/3931090648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314108f00 msgr2=0x7fc31419d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:30:59.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314108f00 0x7fc31419d570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00ea30 tx=0x7fc2fc00ed40 comp rx=0 tx=0).stop 2024-05-27T09:30:59.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 -- 172.21.15.44:0/3931090648 shutdown_connections 2024-05-27T09:30:59.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc2f80b2900 0x7fc2f80b4e10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:59.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314109850 0x7fc3141a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:59.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3141045d0 0x7fc31419cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:59.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 --2- 172.21.15.44:0/3931090648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314108f00 0x7fc31419d570 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:30:59.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 -- 172.21.15.44:0/3931090648 >> 172.21.15.44:0/3931090648 conn(0x7fc3140ffee0 msgr2=0x7fc31410a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:30:59.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 -- 172.21.15.44:0/3931090648 shutdown_connections 2024-05-27T09:30:59.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:30:59.274+0000 7fc31a400700 1 -- 172.21.15.44:0/3931090648 wait complete. 2024-05-27T09:30:59.277 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 11 2024-05-27T09:30:59.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:30:59.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.44:6838/285214695,v1:172.21.15.44:6839/285214695] up:boot 2024-05-27T09:30:59.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: fsmap 10 up:standby 2024-05-27T09:30:59.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T09:30:59.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:59.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:59.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:30:59.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:30:59.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:30:59.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:30:59.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: Deploying daemon mds.a on smithi044 2024-05-27T09:30:59.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:30:59 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3931090648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:31:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.44:6838/285214695,v1:172.21.15.44:6839/285214695] up:boot 2024-05-27T09:31:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: fsmap 10 up:standby 2024-05-27T09:31:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T09:31:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:31:00.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:31:00.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:31:00.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: Deploying daemon mds.a on smithi044 2024-05-27T09:31:00.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:30:59 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3931090648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:31:00.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.44:6838/285214695,v1:172.21.15.44:6839/285214695] up:boot 2024-05-27T09:31:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: fsmap 10 up:standby 2024-05-27T09:31:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T09:31:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:31:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:31:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:31:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: Deploying daemon mds.a on smithi044 2024-05-27T09:31:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:30:59 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3931090648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:31:01.944 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:01 smithi044 ceph-mon[31618]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:02.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:01 smithi050 ceph-mon[36371]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:02.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:01 smithi189 ceph-mon[36671]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:02.174 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:31:02.873 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:31:02 smithi044 podman[73621]: 2024-05-27 09:31:02.633458751 +0000 UTC m=+0.157591936 container create 7658c9f94714258bd2b5523be4ddf958efce2f979936dd90311552ec16c552e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-27T09:31:02.873 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:31:02 smithi044 podman[73621]: 2024-05-27 09:31:02.870893517 +0000 UTC m=+0.395026709 container init 7658c9f94714258bd2b5523be4ddf958efce2f979936dd90311552ec16c552e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-27T09:31:03.050 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:03.201 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:31:02 smithi044 podman[73621]: 2024-05-27 09:31:02.880987679 +0000 UTC m=+0.405120837 container start 7658c9f94714258bd2b5523be4ddf958efce2f979936dd90311552ec16c552e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T09:31:03.201 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:31:02 smithi044 bash[73621]: 7658c9f94714258bd2b5523be4ddf958efce2f979936dd90311552ec16c552e2 2024-05-27T09:31:03.201 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:31:02 smithi044 systemd[1]: Started Ceph mds.a for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:31:03.201 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:31:03 smithi044 ceph-mds[73659]: -- 172.21.15.44:0/286879445 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561f94071a40 con 0x561f940b9180 2024-05-27T09:31:03.201 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:31:03 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-a[73653]: starting mds.a at 2024-05-27T09:31:03.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:03 smithi044 ceph-mon[31618]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:03.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:03 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:03.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:03 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:03.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:03 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:03.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:03 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:31:03.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:03 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:31:03.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:03 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:31:03.989 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:03 smithi050 ceph-mon[36371]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:03.990 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:03 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:03.990 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:03 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:03.990 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:03 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:03.990 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:03 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:31:03.990 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:03 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:31:03.990 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:03 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:31:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:03 smithi189 ceph-mon[36671]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:03 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:03 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:03 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:03 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T09:31:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:03 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T09:31:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:03 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:31:04.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:04 smithi044 ceph-mon[31618]: Deploying daemon mds.b on smithi050 2024-05-27T09:31:04.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:04 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.44:6840/3128909996,v1:172.21.15.44:6841/3128909996] up:boot 2024-05-27T09:31:04.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:04 smithi044 ceph-mon[31618]: fsmap 11 up:standby 2024-05-27T09:31:04.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:04 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T09:31:05.023 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:04 smithi050 ceph-mon[36371]: Deploying daemon mds.b on smithi050 2024-05-27T09:31:05.023 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:04 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.44:6840/3128909996,v1:172.21.15.44:6841/3128909996] up:boot 2024-05-27T09:31:05.023 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:04 smithi050 ceph-mon[36371]: fsmap 11 up:standby 2024-05-27T09:31:05.023 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:04 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T09:31:05.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:04 smithi189 ceph-mon[36671]: Deploying daemon mds.b on smithi050 2024-05-27T09:31:05.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:04 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.44:6840/3128909996,v1:172.21.15.44:6841/3128909996] up:boot 2024-05-27T09:31:05.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:04 smithi189 ceph-mon[36671]: fsmap 11 up:standby 2024-05-27T09:31:05.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:04 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T09:31:05.294 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:31:05 smithi050 systemd[1]: Starting Ceph mds.b for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:31:05.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.363+0000 7f8cef000700 1 -- 172.21.15.44:0/4243434199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ce810c5b0 msgr2=0x7f8ce810c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:05.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.363+0000 7f8cef000700 1 --2- 172.21.15.44:0/4243434199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ce810c5b0 0x7f8ce810c9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8cd80098f0 tx=0x7f8cd8009c00 comp rx=0 tx=0).stop 2024-05-27T09:31:05.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.363+0000 7f8cef000700 1 -- 172.21.15.44:0/4243434199 shutdown_connections 2024-05-27T09:31:05.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.363+0000 7f8cef000700 1 --2- 172.21.15.44:0/4243434199 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ce8106ec0 0x7f8ce8113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:05.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.363+0000 7f8cef000700 1 --2- 172.21.15.44:0/4243434199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ce810c5b0 0x7f8ce810c9c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:05.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.363+0000 7f8cef000700 1 --2- 172.21.15.44:0/4243434199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ce8106570 0x7f8ce8106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:05.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.363+0000 7f8cef000700 1 -- 172.21.15.44:0/4243434199 >> 172.21.15.44:0/4243434199 conn(0x7f8ce8102070 msgr2=0x7f8ce81044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:05.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.363+0000 7f8cef000700 1 -- 172.21.15.44:0/4243434199 shutdown_connections 2024-05-27T09:31:05.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.363+0000 7f8cef000700 1 -- 172.21.15.44:0/4243434199 wait complete. 2024-05-27T09:31:05.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.365+0000 7f8cef000700 1 Processor -- start 2024-05-27T09:31:05.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.365+0000 7f8cef000700 1 -- start start 2024-05-27T09:31:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.365+0000 7f8cef000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ce8106570 0x7f8ce819d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8cef000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ce8106ec0 0x7f8ce819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8cef000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ce810c5b0 0x7f8ce81a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:05.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8cef000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce8116710 con 0x7f8ce8106ec0 2024-05-27T09:31:05.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8cef000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce8116590 con 0x7f8ce8106570 2024-05-27T09:31:05.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8cef000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce8116890 con 0x7f8ce810c5b0 2024-05-27T09:31:05.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8ce7e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ce8106570 0x7f8ce819d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:05.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8ce7e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ce8106570 0x7f8ce819d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47166/0 (socket says 172.21.15.44:47166) 2024-05-27T09:31:05.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8ce7e00700 1 -- 172.21.15.44:0/2418917978 learned_addr learned my addr 172.21.15.44:0/2418917978 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:05.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8ce7400700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ce8106ec0 0x7f8ce819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:05.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8ced000700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ce810c5b0 0x7f8ce81a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:05.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8ce7e00700 1 -- 172.21.15.44:0/2418917978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ce810c5b0 msgr2=0x7f8ce81a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8ce7e00700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ce810c5b0 0x7f8ce81a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8ce7e00700 1 -- 172.21.15.44:0/2418917978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ce8106ec0 msgr2=0x7f8ce819d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8ce7e00700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ce8106ec0 0x7f8ce819d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8ce7e00700 1 -- 172.21.15.44:0/2418917978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce81a1f40 con 0x7f8ce8106570 2024-05-27T09:31:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.366+0000 7f8ced000700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ce810c5b0 0x7f8ce81a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:31:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.367+0000 7f8ce7e00700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ce8106570 0x7f8ce819d0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8009230 tx=0x7f8cd800f990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:05.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.367+0000 7f8ce4c00700 1 -- 172.21.15.44:0/2418917978 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8027070 con 0x7f8ce8106570 2024-05-27T09:31:05.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.367+0000 7f8ce4c00700 1 -- 172.21.15.44:0/2418917978 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cd80055f0 con 0x7f8ce8106570 2024-05-27T09:31:05.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.367+0000 7f8cef000700 1 -- 172.21.15.44:0/2418917978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ce81a21d0 con 0x7f8ce8106570 2024-05-27T09:31:05.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.367+0000 7f8ce4c00700 1 -- 172.21.15.44:0/2418917978 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8004930 con 0x7f8ce8106570 2024-05-27T09:31:05.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.368+0000 7f8cef000700 1 -- 172.21.15.44:0/2418917978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ce81a9b50 con 0x7f8ce8106570 2024-05-27T09:31:05.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.369+0000 7f8cef000700 1 -- 172.21.15.44:0/2418917978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ce8107c90 con 0x7f8ce8106570 2024-05-27T09:31:05.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.371+0000 7f8ce4c00700 1 -- 172.21.15.44:0/2418917978 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8cd802a070 con 0x7f8ce8106570 2024-05-27T09:31:05.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.372+0000 7f8ce4c00700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8ccc0b2900 0x7f8ccc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:05.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.373+0000 7f8ce4c00700 1 -- 172.21.15.44:0/2418917978 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f8cd80e27b0 con 0x7f8ce8106570 2024-05-27T09:31:05.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.373+0000 7f8ce7400700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8ccc0b2900 0x7f8ccc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:05.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.373+0000 7f8ce7400700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8ccc0b2900 0x7f8ccc0b4e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8103c50 tx=0x7f8cd4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:05.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.377+0000 7f8ce4c00700 1 -- 172.21.15.44:0/2418917978 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8cd805fdf0 con 0x7f8ce8106570 2024-05-27T09:31:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.665+0000 7f8cef000700 1 -- 172.21.15.44:0/2418917978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ce819e3d0 con 0x7f8ce8106570 2024-05-27T09:31:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.666+0000 7f8ce4c00700 1 -- 172.21.15.44:0/2418917978 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7f8cd8013030 con 0x7f8ce8106570 2024-05-27T09:31:05.668 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:31:05.668 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-27T09:31:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 -- 172.21.15.44:0/2418917978 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8ccc0b2900 msgr2=0x7f8ccc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8ccc0b2900 0x7f8ccc0b4e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8103c50 tx=0x7f8cd4009260 comp rx=0 tx=0).stop 2024-05-27T09:31:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 -- 172.21.15.44:0/2418917978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ce8106570 msgr2=0x7f8ce819d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ce8106570 0x7f8ce819d0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8009230 tx=0x7f8cd800f990 comp rx=0 tx=0).stop 2024-05-27T09:31:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 -- 172.21.15.44:0/2418917978 shutdown_connections 2024-05-27T09:31:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8ccc0b2900 0x7f8ccc0b4e10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ce810c5b0 0x7f8ce81a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ce8106570 0x7f8ce819d0c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 --2- 172.21.15.44:0/2418917978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ce8106ec0 0x7f8ce819d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 -- 172.21.15.44:0/2418917978 >> 172.21.15.44:0/2418917978 conn(0x7f8ce8102070 msgr2=0x7f8ce810a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 -- 172.21.15.44:0/2418917978 shutdown_connections 2024-05-27T09:31:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:05.670+0000 7f8cef000700 1 -- 172.21.15.44:0/2418917978 wait complete. 2024-05-27T09:31:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 12 2024-05-27T09:31:05.712 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:31:05 smithi050 podman[66706]: 2024-05-27 09:31:05.42976643 +0000 UTC m=+0.114785240 container create 8b7b22588b732a688b4845dd8457da3babc752f80dd4c3ed3d6e9e0ca05050c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738) 2024-05-27T09:31:05.712 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:31:05 smithi050 podman[66706]: 2024-05-27 09:31:05.332613624 +0000 UTC m=+0.017632427 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:31:05.712 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:31:05 smithi050 podman[66706]: 2024-05-27 09:31:05.623749054 +0000 UTC m=+0.308767842 container init 8b7b22588b732a688b4845dd8457da3babc752f80dd4c3ed3d6e9e0ca05050c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T09:31:05.712 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:31:05 smithi050 podman[66706]: 2024-05-27 09:31:05.626437879 +0000 UTC m=+0.311456668 container start 8b7b22588b732a688b4845dd8457da3babc752f80dd4c3ed3d6e9e0ca05050c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-27T09:31:05.712 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:31:05 smithi050 ceph-mds[66725]: -- 172.21.15.50:0/3100393111 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56232374fa40 con 0x562323797180 2024-05-27T09:31:05.712 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:31:05 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b[66721]: starting mds.b at 2024-05-27T09:31:05.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:05 smithi044 ceph-mon[31618]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:05.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:05 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2418917978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:31:06.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:05 smithi050 ceph-mon[36371]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:06.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:05 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2418917978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:31:06.044 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:31:05 smithi050 bash[66706]: 8b7b22588b732a688b4845dd8457da3babc752f80dd4c3ed3d6e9e0ca05050c2 2024-05-27T09:31:06.044 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:31:05 smithi050 systemd[1]: Started Ceph mds.b for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:31:06.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:05 smithi189 ceph-mon[36671]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:06.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:05 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2418917978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:31:06.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:06 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] up:boot 2024-05-27T09:31:06.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:06 smithi044 ceph-mon[31618]: fsmap 12 up:standby 2024-05-27T09:31:06.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:06 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T09:31:06.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:06 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:06.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:06 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:06.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:06 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:06.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:06 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:06.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:06 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:06.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:06 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:31:06.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:06 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:06.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:06 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] up:boot 2024-05-27T09:31:06.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:06 smithi189 ceph-mon[36671]: fsmap 12 up:standby 2024-05-27T09:31:06.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:06 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T09:31:06.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:06 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:06.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:06 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:06.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:06 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:06.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:06 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:06.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:06 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:06.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:06 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:31:06.990 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:06 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:07.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:06 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] up:boot 2024-05-27T09:31:07.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:06 smithi050 ceph-mon[36371]: fsmap 12 up:standby 2024-05-27T09:31:07.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:06 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T09:31:07.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:06 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:07.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:06 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:07.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:06 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:07.012 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:06 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:07.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:06 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:07.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:06 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:31:07.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:06 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:07.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:07 smithi044 ceph-mon[31618]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:08.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:07 smithi050 ceph-mon[36371]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:08.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:07 smithi189 ceph-mon[36671]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:08.606 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:31:08.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:08.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:08.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:08.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:08 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:09.054 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:09.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:08 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:09.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:08 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:09.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:08 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:09.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:08 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:09.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:09.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:09.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:09.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:08 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:10.173 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:09 smithi189 ceph-mon[36671]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:10.173 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:10.173 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:10.173 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:31:10.174 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:31:10.174 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:10.174 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:09 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:31:10.214 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:09 smithi044 ceph-mon[31618]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:10.214 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:10.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:10.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:31:10.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:31:10.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:10.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:09 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:31:10.227 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:09 smithi050 ceph-mon[36371]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:10.227 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:10.227 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:10.227 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:31:10.227 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:31:10.227 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:10.227 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:09 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:31:11.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.215+0000 7fd697a00700 1 -- 172.21.15.44:0/1501147891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6900764a0 msgr2=0x7fd690114e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:11.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.215+0000 7fd697a00700 1 --2- 172.21.15.44:0/1501147891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6900764a0 0x7fd690114e20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009990 tx=0x7fd67c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:31:11.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.216+0000 7fd697a00700 1 -- 172.21.15.44:0/1501147891 shutdown_connections 2024-05-27T09:31:11.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.216+0000 7fd697a00700 1 --2- 172.21.15.44:0/1501147891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd690115360 0x7fd690117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:11.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.216+0000 7fd697a00700 1 --2- 172.21.15.44:0/1501147891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd690075b50 0x7fd690075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:11.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.216+0000 7fd697a00700 1 --2- 172.21.15.44:0/1501147891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6900764a0 0x7fd690114e20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:11.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.216+0000 7fd697a00700 1 -- 172.21.15.44:0/1501147891 >> 172.21.15.44:0/1501147891 conn(0x7fd690100030 msgr2=0x7fd690102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:11.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.216+0000 7fd697a00700 1 -- 172.21.15.44:0/1501147891 shutdown_connections 2024-05-27T09:31:11.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.216+0000 7fd697a00700 1 -- 172.21.15.44:0/1501147891 wait complete. 2024-05-27T09:31:11.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.217+0000 7fd697a00700 1 Processor -- start 2024-05-27T09:31:11.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.217+0000 7fd697a00700 1 -- start start 2024-05-27T09:31:11.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.218+0000 7fd697a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd690075b50 0x7fd6901a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:11.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.218+0000 7fd697a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6900764a0 0x7fd6901a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.218+0000 7fd697a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd690115360 0x7fd6901a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.218+0000 7fd697a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd69011aab0 con 0x7fd6900764a0 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.218+0000 7fd697a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd69011a930 con 0x7fd690075b50 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.218+0000 7fd697a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd69011ac30 con 0x7fd690115360 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.218+0000 7fd68fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6900764a0 0x7fd6901a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.218+0000 7fd695000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd690075b50 0x7fd6901a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.218+0000 7fd68fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6900764a0 0x7fd6901a19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35032/0 (socket says 172.21.15.44:35032) 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.218+0000 7fd68fe00700 1 -- 172.21.15.44:0/709182809 learned_addr learned my addr 172.21.15.44:0/709182809 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.218+0000 7fd695a00700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd690115360 0x7fd6901a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.219+0000 7fd68fe00700 1 -- 172.21.15.44:0/709182809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd690115360 msgr2=0x7fd6901a5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.219+0000 7fd68fe00700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd690115360 0x7fd6901a5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:11.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.219+0000 7fd68fe00700 1 -- 172.21.15.44:0/709182809 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd690075b50 msgr2=0x7fd6901a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:11.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.219+0000 7fd68fe00700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd690075b50 0x7fd6901a1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:11.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.219+0000 7fd68fe00700 1 -- 172.21.15.44:0/709182809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6901a6310 con 0x7fd6900764a0 2024-05-27T09:31:11.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.220+0000 7fd68fe00700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6900764a0 0x7fd6901a19e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd67c000c00 tx=0x7fd67c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:11.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.220+0000 7fd68d600700 1 -- 172.21.15.44:0/709182809 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c027070 con 0x7fd6900764a0 2024-05-27T09:31:11.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.220+0000 7fd68d600700 1 -- 172.21.15.44:0/709182809 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd67c022470 con 0x7fd6900764a0 2024-05-27T09:31:11.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.220+0000 7fd68d600700 1 -- 172.21.15.44:0/709182809 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c017650 con 0x7fd6900764a0 2024-05-27T09:31:11.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.220+0000 7fd697a00700 1 -- 172.21.15.44:0/709182809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6901a65a0 con 0x7fd6900764a0 2024-05-27T09:31:11.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.220+0000 7fd697a00700 1 -- 172.21.15.44:0/709182809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6901adf20 con 0x7fd6900764a0 2024-05-27T09:31:11.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.222+0000 7fd697a00700 1 -- 172.21.15.44:0/709182809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd690077290 con 0x7fd6900764a0 2024-05-27T09:31:11.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.225+0000 7fd68d600700 1 -- 172.21.15.44:0/709182809 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fd67c02a070 con 0x7fd6900764a0 2024-05-27T09:31:11.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.226+0000 7fd68d600700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd6780b2900 0x7fd6780b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:11.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.226+0000 7fd68d600700 1 -- 172.21.15.44:0/709182809 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fd67c0e2af0 con 0x7fd6900764a0 2024-05-27T09:31:11.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.227+0000 7fd695000700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd6780b2900 0x7fd6780b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:11.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.228+0000 7fd695000700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd6780b2900 0x7fd6780b4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd684009680 tx=0x7fd684009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:11.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.234+0000 7fd68d600700 1 -- 172.21.15.44:0/709182809 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd67c013070 con 0x7fd6900764a0 2024-05-27T09:31:11.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.547+0000 7fd697a00700 1 -- 172.21.15.44:0/709182809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6901a27a0 con 0x7fd6900764a0 2024-05-27T09:31:11.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.548+0000 7fd68d600700 1 -- 172.21.15.44:0/709182809 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10476 (secure 0 0 0) 0x7fd67c0aaca0 con 0x7fd6900764a0 2024-05-27T09:31:11.549 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:31:11.550 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24542,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-27T09:31:11.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.550+0000 7fd697a00700 1 -- 172.21.15.44:0/709182809 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd6780b2900 msgr2=0x7fd6780b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:11.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.550+0000 7fd697a00700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd6780b2900 0x7fd6780b4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd684009680 tx=0x7fd684009260 comp rx=0 tx=0).stop 2024-05-27T09:31:11.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.550+0000 7fd697a00700 1 -- 172.21.15.44:0/709182809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6900764a0 msgr2=0x7fd6901a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:11.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.550+0000 7fd697a00700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6900764a0 0x7fd6901a19e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd67c000c00 tx=0x7fd67c0057c0 comp rx=0 tx=0).stop 2024-05-27T09:31:11.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.550+0000 7fd697a00700 1 -- 172.21.15.44:0/709182809 shutdown_connections 2024-05-27T09:31:11.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.550+0000 7fd697a00700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd6780b2900 0x7fd6780b4e10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:11.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.550+0000 7fd697a00700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd690115360 0x7fd6901a5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:11.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.551+0000 7fd697a00700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd690075b50 0x7fd6901a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:11.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.551+0000 7fd697a00700 1 --2- 172.21.15.44:0/709182809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6900764a0 0x7fd6901a19e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:11.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.551+0000 7fd697a00700 1 -- 172.21.15.44:0/709182809 >> 172.21.15.44:0/709182809 conn(0x7fd690100030 msgr2=0x7fd690100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:11.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.551+0000 7fd697a00700 1 -- 172.21.15.44:0/709182809 shutdown_connections 2024-05-27T09:31:11.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:11.551+0000 7fd697a00700 1 -- 172.21.15.44:0/709182809 wait complete. 2024-05-27T09:31:11.556 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-05-27T09:31:12.202 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:11 smithi044 ceph-mon[31618]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:12.202 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:11 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.202 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:11 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.202 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:11 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.203 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:11 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.203 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:11 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/709182809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:31:12.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:11 smithi050 ceph-mon[36371]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:12.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:11 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:11 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:11 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:11 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:11 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/709182809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:31:12.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:11 smithi189 ceph-mon[36671]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:12.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:11 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:11 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:11 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:11 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:12.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:11 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/709182809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:31:12.752 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-05-27T09:31:13.186 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:13.678 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:13 smithi044 ceph-mon[31618]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:13.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:13 smithi050 ceph-mon[36371]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:13.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:13 smithi189 ceph-mon[36671]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:14.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:14.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:14.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:31:14.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:31:14.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:15.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:15.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:15.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:31:15.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:31:15.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:15.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:15.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:15.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:31:15.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:31:15.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:31:15.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.488+0000 7fd372a00700 1 -- 172.21.15.44:0/162953267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd36c10b890 msgr2=0x7fd36c1131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:15.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.488+0000 7fd372a00700 1 --2- 172.21.15.44:0/162953267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd36c10b890 0x7fd36c1131f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd36400b0a0 tx=0x7fd36400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:31:15.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.488+0000 7fd372a00700 1 -- 172.21.15.44:0/162953267 shutdown_connections 2024-05-27T09:31:15.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.488+0000 7fd372a00700 1 --2- 172.21.15.44:0/162953267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd36c10b890 0x7fd36c1131f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:15.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.488+0000 7fd372a00700 1 --2- 172.21.15.44:0/162953267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd36c106610 0x7fd36c106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:15.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.488+0000 7fd372a00700 1 --2- 172.21.15.44:0/162953267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd36c10af40 0x7fd36c10b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:15.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.488+0000 7fd372a00700 1 -- 172.21.15.44:0/162953267 >> 172.21.15.44:0/162953267 conn(0x7fd36c102020 msgr2=0x7fd36c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:15.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.488+0000 7fd372a00700 1 -- 172.21.15.44:0/162953267 shutdown_connections 2024-05-27T09:31:15.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.488+0000 7fd372a00700 1 -- 172.21.15.44:0/162953267 wait complete. 2024-05-27T09:31:15.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.490+0000 7fd372a00700 1 Processor -- start 2024-05-27T09:31:15.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.491+0000 7fd372a00700 1 -- start start 2024-05-27T09:31:15.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.491+0000 7fd372a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd36c106610 0x7fd36c19cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:15.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.491+0000 7fd372a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd36c10af40 0x7fd36c19d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:15.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.491+0000 7fd372a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd36c10b890 0x7fd36c1a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:15.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.491+0000 7fd372a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd36c116510 con 0x7fd36c106610 2024-05-27T09:31:15.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.491+0000 7fd372a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd36c116390 con 0x7fd36c10af40 2024-05-27T09:31:15.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.491+0000 7fd372a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd36c116690 con 0x7fd36c10b890 2024-05-27T09:31:15.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.492+0000 7fd370c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd36c10af40 0x7fd36c19d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:15.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.492+0000 7fd370c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd36c10af40 0x7fd36c19d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45934/0 (socket says 172.21.15.44:45934) 2024-05-27T09:31:15.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.492+0000 7fd371600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd36c106610 0x7fd36c19cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:15.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.492+0000 7fd370c00700 1 -- 172.21.15.44:0/2177162685 learned_addr learned my addr 172.21.15.44:0/2177162685 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:15.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.492+0000 7fd372000700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd36c10b890 0x7fd36c1a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.492+0000 7fd370c00700 1 -- 172.21.15.44:0/2177162685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd36c10b890 msgr2=0x7fd36c1a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.492+0000 7fd370c00700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd36c10b890 0x7fd36c1a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.492+0000 7fd370c00700 1 -- 172.21.15.44:0/2177162685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd36c106610 msgr2=0x7fd36c19cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.492+0000 7fd370c00700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd36c106610 0x7fd36c19cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.492+0000 7fd370c00700 1 -- 172.21.15.44:0/2177162685 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36c1a1c90 con 0x7fd36c10af40 2024-05-27T09:31:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.492+0000 7fd372000700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd36c10b890 0x7fd36c1a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:31:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.493+0000 7fd370c00700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd36c10af40 0x7fd36c19d360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd35800e970 tx=0x7fd35800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.493+0000 7fd36a000700 1 -- 172.21.15.44:0/2177162685 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35800cbd0 con 0x7fd36c10af40 2024-05-27T09:31:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.493+0000 7fd36a000700 1 -- 172.21.15.44:0/2177162685 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd35800cd70 con 0x7fd36c10af40 2024-05-27T09:31:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.493+0000 7fd372a00700 1 -- 172.21.15.44:0/2177162685 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd36c1a1f80 con 0x7fd36c10af40 2024-05-27T09:31:15.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.493+0000 7fd36a000700 1 -- 172.21.15.44:0/2177162685 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd358005680 con 0x7fd36c10af40 2024-05-27T09:31:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.493+0000 7fd372a00700 1 -- 172.21.15.44:0/2177162685 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd36c1a98a0 con 0x7fd36c10af40 2024-05-27T09:31:15.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.494+0000 7fd372a00700 1 -- 172.21.15.44:0/2177162685 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd36c0512b0 con 0x7fd36c10af40 2024-05-27T09:31:15.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.497+0000 7fd36a000700 1 -- 172.21.15.44:0/2177162685 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fd358004750 con 0x7fd36c10af40 2024-05-27T09:31:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.498+0000 7fd36a000700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd3540b2900 0x7fd3540b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.498+0000 7fd36a000700 1 -- 172.21.15.44:0/2177162685 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fd3580e2180 con 0x7fd36c10af40 2024-05-27T09:31:15.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.499+0000 7fd371600700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd3540b2900 0x7fd3540b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:15.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.499+0000 7fd371600700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd3540b2900 0x7fd3540b4e10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd360009c60 tx=0x7fd360009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:15.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.501+0000 7fd36a000700 1 -- 172.21.15.44:0/2177162685 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd358054df0 con 0x7fd36c10af40 2024-05-27T09:31:15.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.743+0000 7fd372a00700 1 -- 172.21.15.44:0/2177162685 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd36c02e5e0 con 0x7fd36c10af40 2024-05-27T09:31:15.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.744+0000 7fd36a000700 1 -- 172.21.15.44:0/2177162685 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7fd3580aa2a0 con 0x7fd36c10af40 2024-05-27T09:31:15.746 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:31:15.746 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-05-27T09:31:15.746 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:31:15.747 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2024-05-27T09:31:15.747 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2024-05-27T09:31:15.747 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:31:15.747 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-05-27T09:31:15.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.748+0000 7fd372a00700 1 -- 172.21.15.44:0/2177162685 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd3540b2900 msgr2=0x7fd3540b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:15.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.748+0000 7fd372a00700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd3540b2900 0x7fd3540b4e10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd360009c60 tx=0x7fd360009260 comp rx=0 tx=0).stop 2024-05-27T09:31:15.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.748+0000 7fd372a00700 1 -- 172.21.15.44:0/2177162685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd36c10af40 msgr2=0x7fd36c19d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:15.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.748+0000 7fd372a00700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd36c10af40 0x7fd36c19d360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd35800e970 tx=0x7fd35800ec80 comp rx=0 tx=0).stop 2024-05-27T09:31:15.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.748+0000 7fd372a00700 1 -- 172.21.15.44:0/2177162685 shutdown_connections 2024-05-27T09:31:15.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.748+0000 7fd372a00700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd3540b2900 0x7fd3540b4e10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:15.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.748+0000 7fd372a00700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd36c10b890 0x7fd36c1a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:15.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.748+0000 7fd372a00700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd36c106610 0x7fd36c19cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:15.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.748+0000 7fd372a00700 1 --2- 172.21.15.44:0/2177162685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd36c10af40 0x7fd36c19d360 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:15.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.748+0000 7fd372a00700 1 -- 172.21.15.44:0/2177162685 >> 172.21.15.44:0/2177162685 conn(0x7fd36c102020 msgr2=0x7fd36c10a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:15.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.749+0000 7fd372a00700 1 -- 172.21.15.44:0/2177162685 shutdown_connections 2024-05-27T09:31:15.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:15.749+0000 7fd372a00700 1 -- 172.21.15.44:0/2177162685 wait complete. 2024-05-27T09:31:16.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:15 smithi189 ceph-mon[36671]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:16.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:15 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2177162685' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T09:31:16.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:15 smithi044 ceph-mon[31618]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:16.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:15 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2177162685' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T09:31:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:15 smithi050 ceph-mon[36371]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:15 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2177162685' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T09:31:16.511 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-27T09:31:16.512 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-27T09:31:16.996 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:18.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:17 smithi189 ceph-mon[36671]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:18.151 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:17 smithi044 ceph-mon[31618]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:18.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:17 smithi050 ceph-mon[36371]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:19.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.192+0000 7fc4a2c00700 1 -- 172.21.15.44:0/495712724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc49c10b8b0 msgr2=0x7fc49c113210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:19.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.192+0000 7fc4a2c00700 1 --2- 172.21.15.44:0/495712724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc49c10b8b0 0x7fc49c113210 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc49400b0d0 tx=0x7fc49400b3e0 comp rx=0 tx=0).stop 2024-05-27T09:31:19.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.192+0000 7fc4a2c00700 1 -- 172.21.15.44:0/495712724 shutdown_connections 2024-05-27T09:31:19.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.192+0000 7fc4a2c00700 1 --2- 172.21.15.44:0/495712724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc49c10b8b0 0x7fc49c113210 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:19.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.192+0000 7fc4a2c00700 1 --2- 172.21.15.44:0/495712724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc49c106630 0x7fc49c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:19.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.192+0000 7fc4a2c00700 1 --2- 172.21.15.44:0/495712724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c10af60 0x7fc49c10b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:19.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.192+0000 7fc4a2c00700 1 -- 172.21.15.44:0/495712724 >> 172.21.15.44:0/495712724 conn(0x7fc49c101ec0 msgr2=0x7fc49c104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:19.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.193+0000 7fc4a2c00700 1 -- 172.21.15.44:0/495712724 shutdown_connections 2024-05-27T09:31:19.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.193+0000 7fc4a2c00700 1 -- 172.21.15.44:0/495712724 wait complete. 2024-05-27T09:31:19.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.194+0000 7fc4a2c00700 1 Processor -- start 2024-05-27T09:31:19.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.194+0000 7fc4a2c00700 1 -- start start 2024-05-27T09:31:19.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.195+0000 7fc4a2c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc49c106630 0x7fc49c19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:19.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.195+0000 7fc4a2c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc49c10af60 0x7fc49c19d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:19.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.195+0000 7fc4a2c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c10b8b0 0x7fc49c1a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:19.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.195+0000 7fc4a2c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1164e0 con 0x7fc49c10af60 2024-05-27T09:31:19.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.195+0000 7fc4a2c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c116360 con 0x7fc49c106630 2024-05-27T09:31:19.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.195+0000 7fc4a2c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c116660 con 0x7fc49c10b8b0 2024-05-27T09:31:19.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.195+0000 7fc4a1800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc49c106630 0x7fc49c19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:19.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.195+0000 7fc4a1800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc49c106630 0x7fc49c19cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45954/0 (socket says 172.21.15.44:45954) 2024-05-27T09:31:19.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.195+0000 7fc4a1800700 1 -- 172.21.15.44:0/2196847723 learned_addr learned my addr 172.21.15.44:0/2196847723 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:19.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.195+0000 7fc4a0e00700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc49c10af60 0x7fc49c19d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:19.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.196+0000 7fc4a2200700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c10b8b0 0x7fc49c1a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:19.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.196+0000 7fc4a1800700 1 -- 172.21.15.44:0/2196847723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c10b8b0 msgr2=0x7fc49c1a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:19.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.196+0000 7fc4a1800700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c10b8b0 0x7fc49c1a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:19.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.196+0000 7fc4a1800700 1 -- 172.21.15.44:0/2196847723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc49c10af60 msgr2=0x7fc49c19d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:19.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.196+0000 7fc4a1800700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc49c10af60 0x7fc49c19d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:19.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.196+0000 7fc4a1800700 1 -- 172.21.15.44:0/2196847723 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49c1a1ca0 con 0x7fc49c106630 2024-05-27T09:31:19.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.196+0000 7fc4a1800700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc49c106630 0x7fc49c19cdd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc49000e9a0 tx=0x7fc49000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:19.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.197+0000 7fc49a000700 1 -- 172.21.15.44:0/2196847723 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49000ca10 con 0x7fc49c106630 2024-05-27T09:31:19.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.197+0000 7fc49a000700 1 -- 172.21.15.44:0/2196847723 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc49000cbb0 con 0x7fc49c106630 2024-05-27T09:31:19.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.197+0000 7fc4a2c00700 1 -- 172.21.15.44:0/2196847723 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc49c1a1f90 con 0x7fc49c106630 2024-05-27T09:31:19.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.197+0000 7fc49a000700 1 -- 172.21.15.44:0/2196847723 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc490018680 con 0x7fc49c106630 2024-05-27T09:31:19.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.197+0000 7fc4a2c00700 1 -- 172.21.15.44:0/2196847723 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc49c1a98b0 con 0x7fc49c106630 2024-05-27T09:31:19.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.198+0000 7fc4a2c00700 1 -- 172.21.15.44:0/2196847723 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc49c0512b0 con 0x7fc49c106630 2024-05-27T09:31:19.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.200+0000 7fc49a000700 1 -- 172.21.15.44:0/2196847723 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc49004aab0 con 0x7fc49c106630 2024-05-27T09:31:19.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.202+0000 7fc49a000700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc4840b2900 0x7fc4840b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:19.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.202+0000 7fc49a000700 1 -- 172.21.15.44:0/2196847723 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fc4900e1fe0 con 0x7fc49c106630 2024-05-27T09:31:19.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.202+0000 7fc4a0e00700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc4840b2900 0x7fc4840b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:19.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.203+0000 7fc4a0e00700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc4840b2900 0x7fc4840b4e10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc49c105910 tx=0x7fc488009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:19.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.205+0000 7fc49a000700 1 -- 172.21.15.44:0/2196847723 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc4900aa100 con 0x7fc49c106630 2024-05-27T09:31:19.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.441+0000 7fc4a2c00700 1 -- 172.21.15.44:0/2196847723 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc49c02e5e0 con 0x7fc49c106630 2024-05-27T09:31:19.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.829+0000 7fc49a000700 1 -- 172.21.15.44:0/2196847723 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7fc4900b2180 con 0x7fc49c106630 2024-05-27T09:31:19.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.839+0000 7fc4a2c00700 1 -- 172.21.15.44:0/2196847723 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc4840b2900 msgr2=0x7fc4840b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:19.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.839+0000 7fc4a2c00700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc4840b2900 0x7fc4840b4e10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc49c105910 tx=0x7fc488009260 comp rx=0 tx=0).stop 2024-05-27T09:31:19.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.839+0000 7fc4a2c00700 1 -- 172.21.15.44:0/2196847723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc49c106630 msgr2=0x7fc49c19cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:19.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.839+0000 7fc4a2c00700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc49c106630 0x7fc49c19cdd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc49000e9a0 tx=0x7fc49000ecb0 comp rx=0 tx=0).stop 2024-05-27T09:31:19.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.840+0000 7fc4a2c00700 1 -- 172.21.15.44:0/2196847723 shutdown_connections 2024-05-27T09:31:19.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.840+0000 7fc4a2c00700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc4840b2900 0x7fc4840b4e10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:19.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.840+0000 7fc4a2c00700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c10b8b0 0x7fc49c1a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:19.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.840+0000 7fc4a2c00700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc49c106630 0x7fc49c19cdd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:19.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.840+0000 7fc4a2c00700 1 --2- 172.21.15.44:0/2196847723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc49c10af60 0x7fc49c19d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:19.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.840+0000 7fc4a2c00700 1 -- 172.21.15.44:0/2196847723 >> 172.21.15.44:0/2196847723 conn(0x7fc49c101ec0 msgr2=0x7fc49c10a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:19.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.842+0000 7fc4a2c00700 1 -- 172.21.15.44:0/2196847723 shutdown_connections 2024-05-27T09:31:19.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:19.842+0000 7fc4a2c00700 1 -- 172.21.15.44:0/2196847723 wait complete. 2024-05-27T09:31:19.845 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2024-05-27T09:31:20.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:19 smithi044 ceph-mon[31618]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:20.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:19 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2196847723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T09:31:20.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:19 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T09:31:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:19 smithi050 ceph-mon[36371]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:19 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2196847723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T09:31:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:19 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T09:31:20.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:19 smithi189 ceph-mon[36671]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:20.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:19 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2196847723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T09:31:20.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:19 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T09:31:20.711 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-27T09:31:20.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:20 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T09:31:20.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:20 smithi044 ceph-mon[31618]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T09:31:20.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:20 smithi044 ceph-mon[31618]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T09:31:21.201 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:21.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:20 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T09:31:21.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:20 smithi050 ceph-mon[36371]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T09:31:21.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:20 smithi050 ceph-mon[36371]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T09:31:21.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:20 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T09:31:21.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:20 smithi189 ceph-mon[36671]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T09:31:21.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:20 smithi189 ceph-mon[36671]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T09:31:21.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:21 smithi044 ceph-mon[31618]: pgmap v315: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:21.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:21 smithi044 ceph-mon[31618]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T09:31:21.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:21 smithi044 ceph-mon[31618]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T09:31:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:21 smithi050 ceph-mon[36371]: pgmap v315: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:21 smithi050 ceph-mon[36371]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T09:31:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:21 smithi050 ceph-mon[36371]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T09:31:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:21 smithi189 ceph-mon[36671]: pgmap v315: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:21 smithi189 ceph-mon[36671]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T09:31:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:21 smithi189 ceph-mon[36671]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T09:31:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.480+0000 7f1bdf000700 1 -- 172.21.15.44:0/1577758532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd81084f0 msgr2=0x7f1bd8108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.481+0000 7f1bdf000700 1 --2- 172.21.15.44:0/1577758532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd81084f0 0x7f1bd8108900 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc0074e0 tx=0x7f1bcc00af90 comp rx=0 tx=0).stop 2024-05-27T09:31:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.481+0000 7f1bdf000700 1 -- 172.21.15.44:0/1577758532 shutdown_connections 2024-05-27T09:31:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.481+0000 7f1bdf000700 1 --2- 172.21.15.44:0/1577758532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd8102cf0 0x7f1bd810f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.481+0000 7f1bdf000700 1 --2- 172.21.15.44:0/1577758532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd81084f0 0x7f1bd8108900 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.481+0000 7f1bdf000700 1 --2- 172.21.15.44:0/1577758532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd81023a0 0x7f1bd81027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.481+0000 7f1bdf000700 1 -- 172.21.15.44:0/1577758532 >> 172.21.15.44:0/1577758532 conn(0x7f1bd8077f40 msgr2=0x7f1bd8078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.481+0000 7f1bdf000700 1 -- 172.21.15.44:0/1577758532 shutdown_connections 2024-05-27T09:31:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.481+0000 7f1bdf000700 1 -- 172.21.15.44:0/1577758532 wait complete. 2024-05-27T09:31:23.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.482+0000 7f1bdf000700 1 Processor -- start 2024-05-27T09:31:23.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.482+0000 7f1bdf000700 1 -- start start 2024-05-27T09:31:23.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.483+0000 7f1bdf000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd81023a0 0x7f1bd819cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:23.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.483+0000 7f1bdf000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd8102cf0 0x7f1bd819d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:23.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.483+0000 7f1bdf000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd81084f0 0x7f1bd81a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:23.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.483+0000 7f1bdf000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd8112200 con 0x7f1bd81084f0 2024-05-27T09:31:23.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.483+0000 7f1bdf000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd8112080 con 0x7f1bd8102cf0 2024-05-27T09:31:23.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.483+0000 7f1bdf000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd8112380 con 0x7f1bd81023a0 2024-05-27T09:31:23.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.483+0000 7f1bddc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd81023a0 0x7f1bd819cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:23.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.483+0000 7f1bde600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd81084f0 0x7f1bd81a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:23.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.483+0000 7f1bddc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd81023a0 0x7f1bd819cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49346/0 (socket says 172.21.15.44:49346) 2024-05-27T09:31:23.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.483+0000 7f1bddc00700 1 -- 172.21.15.44:0/3020676266 learned_addr learned my addr 172.21.15.44:0/3020676266 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:23.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.483+0000 7f1bdd200700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd8102cf0 0x7f1bd819d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:23.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.484+0000 7f1bddc00700 1 -- 172.21.15.44:0/3020676266 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd8102cf0 msgr2=0x7f1bd819d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:23.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.484+0000 7f1bddc00700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd8102cf0 0x7f1bd819d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:23.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.484+0000 7f1bddc00700 1 -- 172.21.15.44:0/3020676266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd81084f0 msgr2=0x7f1bd81a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:23.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.484+0000 7f1bddc00700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd81084f0 0x7f1bd81a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:23.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.484+0000 7f1bddc00700 1 -- 172.21.15.44:0/3020676266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd81a1e80 con 0x7f1bd81023a0 2024-05-27T09:31:23.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.484+0000 7f1bddc00700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd81023a0 0x7f1bd819cfb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc005b40 tx=0x7f1bcc007e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:23.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.484+0000 7f1bd6000700 1 -- 172.21.15.44:0/3020676266 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc018070 con 0x7f1bd81023a0 2024-05-27T09:31:23.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.484+0000 7f1bd6000700 1 -- 172.21.15.44:0/3020676266 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bcc003a70 con 0x7f1bd81023a0 2024-05-27T09:31:23.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.484+0000 7f1bd6000700 1 -- 172.21.15.44:0/3020676266 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc00a330 con 0x7f1bd81023a0 2024-05-27T09:31:23.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.484+0000 7f1bdf000700 1 -- 172.21.15.44:0/3020676266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd81a2110 con 0x7f1bd81023a0 2024-05-27T09:31:23.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.485+0000 7f1bdf000700 1 -- 172.21.15.44:0/3020676266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd81a2600 con 0x7f1bd81023a0 2024-05-27T09:31:23.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.486+0000 7f1bdf000700 1 -- 172.21.15.44:0/3020676266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd8103ac0 con 0x7f1bd81023a0 2024-05-27T09:31:23.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.488+0000 7f1bd6000700 1 -- 172.21.15.44:0/3020676266 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f1bcc003f40 con 0x7f1bd81023a0 2024-05-27T09:31:23.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.489+0000 7f1bd6000700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f1bc00b2900 0x7f1bc00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:23.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.490+0000 7f1bd6000700 1 -- 172.21.15.44:0/3020676266 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7497+0+0 (secure 0 0 0) 0x7f1bcc010020 con 0x7f1bd81023a0 2024-05-27T09:31:23.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.490+0000 7f1bdd200700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f1bc00b2900 0x7f1bc00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:23.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.491+0000 7f1bdd200700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f1bc00b2900 0x7f1bc00b4e10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1bd81001d0 tx=0x7f1bc4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:23.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.495+0000 7f1bd6000700 1 -- 172.21.15.44:0/3020676266 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1bcc02c020 con 0x7f1bd81023a0 2024-05-27T09:31:23.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.731+0000 7f1bdf000700 1 -- 172.21.15.44:0/3020676266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f1bd81a9a90 con 0x7f1bd81023a0 2024-05-27T09:31:23.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.873+0000 7f1bd6000700 1 -- 172.21.15.44:0/3020676266 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v76) ==== 170+0+0 (secure 0 0 0) 0x7f1bcc0aa660 con 0x7f1bd81023a0 2024-05-27T09:31:23.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.882+0000 7f1bdf000700 1 -- 172.21.15.44:0/3020676266 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f1bc00b2900 msgr2=0x7f1bc00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.882+0000 7f1bdf000700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f1bc00b2900 0x7f1bc00b4e10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1bd81001d0 tx=0x7f1bc4009260 comp rx=0 tx=0).stop 2024-05-27T09:31:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.882+0000 7f1bdf000700 1 -- 172.21.15.44:0/3020676266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd81023a0 msgr2=0x7f1bd819cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.882+0000 7f1bdf000700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd81023a0 0x7f1bd819cfb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc005b40 tx=0x7f1bcc007e80 comp rx=0 tx=0).stop 2024-05-27T09:31:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.882+0000 7f1bdf000700 1 -- 172.21.15.44:0/3020676266 shutdown_connections 2024-05-27T09:31:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.882+0000 7f1bdf000700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f1bc00b2900 0x7f1bc00b4e10 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1bd81001d0 tx=0x7f1bc4009260 comp rx=0 tx=0).stop 2024-05-27T09:31:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.882+0000 7f1bdf000700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd81084f0 0x7f1bd81a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.882+0000 7f1bdf000700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd81023a0 0x7f1bd819cfb0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc005b40 tx=0x7f1bcc007e80 comp rx=0 tx=0).stop 2024-05-27T09:31:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.882+0000 7f1bdf000700 1 --2- 172.21.15.44:0/3020676266 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd8102cf0 0x7f1bd819d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:23.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.882+0000 7f1bdf000700 1 -- 172.21.15.44:0/3020676266 >> 172.21.15.44:0/3020676266 conn(0x7f1bd8077f40 msgr2=0x7f1bd81008a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:23.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.884+0000 7f1bdf000700 1 -- 172.21.15.44:0/3020676266 shutdown_connections 2024-05-27T09:31:23.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:23.885+0000 7f1bdf000700 1 -- 172.21.15.44:0/3020676266 wait complete. 2024-05-27T09:31:23.890 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2024-05-27T09:31:24.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:23 smithi044 ceph-mon[31618]: pgmap v318: 33 pgs: 8 creating+peering, 10 active+clean, 15 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:24.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:23 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3020676266' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T09:31:24.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:23 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T09:31:24.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:23 smithi050 ceph-mon[36371]: pgmap v318: 33 pgs: 8 creating+peering, 10 active+clean, 15 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:23 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3020676266' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T09:31:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:23 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T09:31:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:23 smithi189 ceph-mon[36671]: pgmap v318: 33 pgs: 8 creating+peering, 10 active+clean, 15 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:23 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3020676266' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T09:31:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:23 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T09:31:24.828 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-27T09:31:25.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:24 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T09:31:25.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:24 smithi044 ceph-mon[31618]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T09:31:25.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:24 smithi044 ceph-mon[31618]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T09:31:25.288 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:25.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:24 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T09:31:25.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:24 smithi050 ceph-mon[36371]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T09:31:25.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:24 smithi050 ceph-mon[36371]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T09:31:25.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:24 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T09:31:25.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:24 smithi189 ceph-mon[36671]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T09:31:25.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:24 smithi189 ceph-mon[36671]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T09:31:25.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:25 smithi044 ceph-mon[31618]: pgmap v320: 97 pgs: 31 creating+peering, 25 active+clean, 41 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:25.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:25 smithi044 ceph-mon[31618]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T09:31:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:25 smithi050 ceph-mon[36371]: pgmap v320: 97 pgs: 31 creating+peering, 25 active+clean, 41 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:25 smithi050 ceph-mon[36371]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T09:31:26.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:25 smithi189 ceph-mon[36671]: pgmap v320: 97 pgs: 31 creating+peering, 25 active+clean, 41 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:26.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:25 smithi189 ceph-mon[36671]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T09:31:27.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.438+0000 7f3342400700 1 -- 172.21.15.44:0/570965817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f333c1045e0 msgr2=0x7f333c1049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:27.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.438+0000 7f3342400700 1 --2- 172.21.15.44:0/570965817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f333c1045e0 0x7f333c1049f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f332c0099c0 tx=0x7f332c009cd0 comp rx=0 tx=0).stop 2024-05-27T09:31:27.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.438+0000 7f3342400700 1 -- 172.21.15.44:0/570965817 shutdown_connections 2024-05-27T09:31:27.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.438+0000 7f3342400700 1 --2- 172.21.15.44:0/570965817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f333c109860 0x7f333c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:27.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.438+0000 7f3342400700 1 --2- 172.21.15.44:0/570965817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f333c1045e0 0x7f333c1049f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:27.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.438+0000 7f3342400700 1 --2- 172.21.15.44:0/570965817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f333c108f10 0x7f333c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:27.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.438+0000 7f3342400700 1 -- 172.21.15.44:0/570965817 >> 172.21.15.44:0/570965817 conn(0x7f333c100030 msgr2=0x7f333c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:27.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.439+0000 7f3342400700 1 -- 172.21.15.44:0/570965817 shutdown_connections 2024-05-27T09:31:27.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.439+0000 7f3342400700 1 -- 172.21.15.44:0/570965817 wait complete. 2024-05-27T09:31:27.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.440+0000 7f3342400700 1 Processor -- start 2024-05-27T09:31:27.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.440+0000 7f3342400700 1 -- start start 2024-05-27T09:31:27.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.440+0000 7f3342400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f333c1045e0 0x7f333c19aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:27.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.440+0000 7f3342400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f333c108f10 0x7f333c19b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:27.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.440+0000 7f3342400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f333c109860 0x7f333c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:27.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.440+0000 7f3342400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1144e0 con 0x7f333c109860 2024-05-27T09:31:27.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.440+0000 7f3342400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f333c114360 con 0x7f333c1045e0 2024-05-27T09:31:27.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.440+0000 7f3342400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f333c114660 con 0x7f333c108f10 2024-05-27T09:31:27.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.441+0000 7f333b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f333c1045e0 0x7f333c19aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:27.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.441+0000 7f333b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f333c1045e0 0x7f333c19aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:40878/0 (socket says 172.21.15.44:40878) 2024-05-27T09:31:27.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.441+0000 7f333b400700 1 -- 172.21.15.44:0/1300139722 learned_addr learned my addr 172.21.15.44:0/1300139722 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:27.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.441+0000 7f333be00700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f333c109860 0x7f333c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:27.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.441+0000 7f3333e00700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f333c108f10 0x7f333c19b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:27.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.441+0000 7f333b400700 1 -- 172.21.15.44:0/1300139722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f333c108f10 msgr2=0x7f333c19b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:27.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.441+0000 7f333b400700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f333c108f10 0x7f333c19b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:27.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.441+0000 7f333b400700 1 -- 172.21.15.44:0/1300139722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f333c109860 msgr2=0x7f333c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:27.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.441+0000 7f333b400700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f333c109860 0x7f333c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:27.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.441+0000 7f333b400700 1 -- 172.21.15.44:0/1300139722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333c19fd80 con 0x7f333c1045e0 2024-05-27T09:31:27.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.442+0000 7f333b400700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f333c1045e0 0x7f333c19aeb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f332c005910 tx=0x7f332c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:27.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.442+0000 7f3338c00700 1 -- 172.21.15.44:0/1300139722 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c026070 con 0x7f333c1045e0 2024-05-27T09:31:27.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.442+0000 7f3338c00700 1 -- 172.21.15.44:0/1300139722 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f332c003c70 con 0x7f333c1045e0 2024-05-27T09:31:27.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.442+0000 7f3342400700 1 -- 172.21.15.44:0/1300139722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f333c1a0010 con 0x7f333c1045e0 2024-05-27T09:31:27.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.442+0000 7f3338c00700 1 -- 172.21.15.44:0/1300139722 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c00fa40 con 0x7f333c1045e0 2024-05-27T09:31:27.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.442+0000 7f3342400700 1 -- 172.21.15.44:0/1300139722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f333c1a04f0 con 0x7f333c1045e0 2024-05-27T09:31:27.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.443+0000 7f3331600700 1 -- 172.21.15.44:0/1300139722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f333c0512b0 con 0x7f333c1045e0 2024-05-27T09:31:27.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.446+0000 7f3338c00700 1 -- 172.21.15.44:0/1300139722 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f332c029070 con 0x7f333c1045e0 2024-05-27T09:31:27.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.447+0000 7f3338c00700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33200b2880 0x7f33200b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:27.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.448+0000 7f3338c00700 1 -- 172.21.15.44:0/1300139722 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7879+0+0 (secure 0 0 0) 0x7f332c0e28c0 con 0x7f333c1045e0 2024-05-27T09:31:27.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.448+0000 7f3333e00700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33200b2880 0x7f33200b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:27.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.449+0000 7f3333e00700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33200b2880 0x7f33200b4d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f333c103640 tx=0x7f3324009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:27.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.453+0000 7f3338c00700 1 -- 172.21.15.44:0/1300139722 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f332c013070 con 0x7f333c1045e0 2024-05-27T09:31:27.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.744+0000 7f3331600700 1 -- 172.21.15.44:0/1300139722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f333c0643c0 con 0x7f333c1045e0 2024-05-27T09:31:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.935+0000 7f3338c00700 1 -- 172.21.15.44:0/1300139722 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f332c0aa780 con 0x7f333c1045e0 2024-05-27T09:31:27.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.943+0000 7f3331600700 1 -- 172.21.15.44:0/1300139722 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33200b2880 msgr2=0x7f33200b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:27.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.943+0000 7f3331600700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33200b2880 0x7f33200b4d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f333c103640 tx=0x7f3324009260 comp rx=0 tx=0).stop 2024-05-27T09:31:27.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.943+0000 7f3331600700 1 -- 172.21.15.44:0/1300139722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f333c1045e0 msgr2=0x7f333c19aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:27.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.943+0000 7f3331600700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f333c1045e0 0x7f333c19aeb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f332c005910 tx=0x7f332c004270 comp rx=0 tx=0).stop 2024-05-27T09:31:27.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.944+0000 7f3331600700 1 -- 172.21.15.44:0/1300139722 shutdown_connections 2024-05-27T09:31:27.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.944+0000 7f3331600700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f33200b2880 0x7f33200b4d90 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f333c103640 tx=0x7f3324009260 comp rx=0 tx=0).stop 2024-05-27T09:31:27.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.944+0000 7f3331600700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f333c109860 0x7f333c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:27.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.944+0000 7f3331600700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f333c1045e0 0x7f333c19aeb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:27.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.944+0000 7f3331600700 1 --2- 172.21.15.44:0/1300139722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f333c108f10 0x7f333c19b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:27.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.944+0000 7f3331600700 1 -- 172.21.15.44:0/1300139722 >> 172.21.15.44:0/1300139722 conn(0x7f333c100030 msgr2=0x7f333c108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:27.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.945+0000 7f3331600700 1 -- 172.21.15.44:0/1300139722 shutdown_connections 2024-05-27T09:31:27.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:27.945+0000 7f3331600700 1 -- 172.21.15.44:0/1300139722 wait complete. 2024-05-27T09:31:27.951 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 2 and data pool 3 2024-05-27T09:31:28.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:27 smithi044 ceph-mon[31618]: pgmap v323: 97 pgs: 4 creating+activating, 23 creating+peering, 57 active+clean, 13 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:28.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:27 smithi044 ceph-mon[31618]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T09:31:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:27 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1300139722' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T09:31:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:27 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T09:31:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:27 smithi044 ceph-mon[31618]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T09:31:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:27 smithi050 ceph-mon[36371]: pgmap v323: 97 pgs: 4 creating+activating, 23 creating+peering, 57 active+clean, 13 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:27 smithi050 ceph-mon[36371]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T09:31:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:27 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1300139722' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T09:31:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:27 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T09:31:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:27 smithi050 ceph-mon[36371]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T09:31:28.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:27 smithi189 ceph-mon[36671]: pgmap v323: 97 pgs: 4 creating+activating, 23 creating+peering, 57 active+clean, 13 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:31:28.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:27 smithi189 ceph-mon[36671]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T09:31:28.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:27 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1300139722' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T09:31:28.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:27 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T09:31:28.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:27 smithi189 ceph-mon[36671]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T09:31:28.743 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T09:31:29.193 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:29.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:28 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T09:31:29.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:28 smithi044 ceph-mon[31618]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T09:31:29.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:28 smithi044 ceph-mon[31618]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T09:31:29.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:28 smithi044 ceph-mon[31618]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T09:31:29.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:28 smithi044 ceph-mon[31618]: pgmap v325: 97 pgs: 4 creating+activating, 23 creating+peering, 70 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-27T09:31:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:28 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T09:31:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:28 smithi050 ceph-mon[36371]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T09:31:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:28 smithi050 ceph-mon[36371]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T09:31:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:28 smithi050 ceph-mon[36371]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T09:31:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:28 smithi050 ceph-mon[36371]: pgmap v325: 97 pgs: 4 creating+activating, 23 creating+peering, 70 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-27T09:31:29.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:28 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T09:31:29.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:28 smithi189 ceph-mon[36671]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T09:31:29.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:28 smithi189 ceph-mon[36671]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T09:31:29.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:28 smithi189 ceph-mon[36671]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T09:31:29.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:28 smithi189 ceph-mon[36671]: pgmap v325: 97 pgs: 4 creating+activating, 23 creating+peering, 70 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-27T09:31:30.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:29 smithi044 ceph-mon[31618]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T09:31:30.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:29 smithi044 ceph-mon[31618]: Cluster is now healthy 2024-05-27T09:31:30.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:29 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] up:active 2024-05-27T09:31:30.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:29 smithi044 ceph-mon[31618]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:30.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:29 smithi044 ceph-mon[31618]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:29 smithi050 ceph-mon[36371]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T09:31:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:29 smithi050 ceph-mon[36371]: Cluster is now healthy 2024-05-27T09:31:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:29 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] up:active 2024-05-27T09:31:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:29 smithi050 ceph-mon[36371]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:29 smithi050 ceph-mon[36371]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:29 smithi189 ceph-mon[36671]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T09:31:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:29 smithi189 ceph-mon[36671]: Cluster is now healthy 2024-05-27T09:31:30.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:29 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] up:active 2024-05-27T09:31:30.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:29 smithi189 ceph-mon[36671]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:30.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:29 smithi189 ceph-mon[36671]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:30.977 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:31:30 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b[66721]: 2024-05-27T09:31:30.712+0000 7fc6ea134700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T09:31:31.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.273+0000 7f1221200700 1 -- 172.21.15.44:0/2731612673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f121c110c00 msgr2=0x7f121c113020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:31.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.273+0000 7f1221200700 1 --2- 172.21.15.44:0/2731612673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f121c110c00 0x7f121c113020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f121400b0a0 tx=0x7f121400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:31:31.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.273+0000 7f1221200700 1 -- 172.21.15.44:0/2731612673 shutdown_connections 2024-05-27T09:31:31.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.273+0000 7f1221200700 1 --2- 172.21.15.44:0/2731612673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f121c110c00 0x7f121c113020 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:31.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.273+0000 7f1221200700 1 --2- 172.21.15.44:0/2731612673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f121c076be0 0x7f121c076ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:31.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.273+0000 7f1221200700 1 --2- 172.21.15.44:0/2731612673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f121c077530 0x7f121c1106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:31.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.273+0000 7f1221200700 1 -- 172.21.15.44:0/2731612673 >> 172.21.15.44:0/2731612673 conn(0x7f121c0ffec0 msgr2=0x7f121c102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:31.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:30 smithi044 ceph-mon[31618]: pgmap v326: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 6 op/s 2024-05-27T09:31:31.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.273+0000 7f1221200700 1 -- 172.21.15.44:0/2731612673 shutdown_connections 2024-05-27T09:31:31.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.274+0000 7f1221200700 1 -- 172.21.15.44:0/2731612673 wait complete. 2024-05-27T09:31:31.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.275+0000 7f1221200700 1 Processor -- start 2024-05-27T09:31:31.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.275+0000 7f1221200700 1 -- start start 2024-05-27T09:31:31.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.276+0000 7f1221200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f121c076be0 0x7f121c19d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:31.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.276+0000 7f1221200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f121c077530 0x7f121c19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:31.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.276+0000 7f1221200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f121c110c00 0x7f121c1a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:31.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.276+0000 7f1221200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f121c1166e0 con 0x7f121c110c00 2024-05-27T09:31:31.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.276+0000 7f1221200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f121c116560 con 0x7f121c076be0 2024-05-27T09:31:31.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.276+0000 7f1221200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f121c116860 con 0x7f121c077530 2024-05-27T09:31:31.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.276+0000 7f121a800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f121c110c00 0x7f121c1a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:31.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.276+0000 7f1219e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f121c076be0 0x7f121c19d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:31.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.276+0000 7f121a800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f121c110c00 0x7f121c1a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56576/0 (socket says 172.21.15.44:56576) 2024-05-27T09:31:31.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.276+0000 7f121a800700 1 -- 172.21.15.44:0/3350334577 learned_addr learned my addr 172.21.15.44:0/3350334577 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:31.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.276+0000 7f1213e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f121c077530 0x7f121c19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:31.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f121a800700 1 -- 172.21.15.44:0/3350334577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f121c077530 msgr2=0x7f121c19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:31.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f121a800700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f121c077530 0x7f121c19d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:31.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f121a800700 1 -- 172.21.15.44:0/3350334577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f121c076be0 msgr2=0x7f121c19d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:31.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f121a800700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f121c076be0 0x7f121c19d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:31.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f121a800700 1 -- 172.21.15.44:0/3350334577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121c1a1fa0 con 0x7f121c110c00 2024-05-27T09:31:31.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f1219e00700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f121c076be0 0x7f121c19d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:31:31.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f121a800700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f121c110c00 0x7f121c1a1900 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f121400b070 tx=0x7f1214009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:31.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f1212a00700 1 -- 172.21.15.44:0/3350334577 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214021070 con 0x7f121c110c00 2024-05-27T09:31:31.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f1221200700 1 -- 172.21.15.44:0/3350334577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f121c1a2230 con 0x7f121c110c00 2024-05-27T09:31:31.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f1212a00700 1 -- 172.21.15.44:0/3350334577 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1214004e40 con 0x7f121c110c00 2024-05-27T09:31:31.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f1212a00700 1 -- 172.21.15.44:0/3350334577 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214004630 con 0x7f121c110c00 2024-05-27T09:31:31.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.277+0000 7f1221200700 1 -- 172.21.15.44:0/3350334577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f121c1a9bb0 con 0x7f121c110c00 2024-05-27T09:31:31.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.279+0000 7f1221200700 1 -- 172.21.15.44:0/3350334577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f121c0512b0 con 0x7f121c110c00 2024-05-27T09:31:31.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.281+0000 7f1212a00700 1 -- 172.21.15.44:0/3350334577 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f1214007980 con 0x7f121c110c00 2024-05-27T09:31:31.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.283+0000 7f1212a00700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12000b2880 0x7f12000b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:31.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.283+0000 7f1212a00700 1 -- 172.21.15.44:0/3350334577 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f121400f020 con 0x7f121c110c00 2024-05-27T09:31:31.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.283+0000 7f1219e00700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12000b2880 0x7f12000b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:31.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.284+0000 7f1219e00700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12000b2880 0x7f12000b4d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f120c011490 tx=0x7f120c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:31.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.287+0000 7f1212a00700 1 -- 172.21.15.44:0/3350334577 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f121402a020 con 0x7f121c110c00 2024-05-27T09:31:31.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:30 smithi050 ceph-mon[36371]: pgmap v326: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 6 op/s 2024-05-27T09:31:31.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:30 smithi189 ceph-mon[36671]: pgmap v326: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 6 op/s 2024-05-27T09:31:31.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.524+0000 7f1221200700 1 -- 172.21.15.44:0/3350334577 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f121c19e430 con 0x7f121c110c00 2024-05-27T09:31:31.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.529+0000 7f1212a00700 1 -- 172.21.15.44:0/3350334577 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22160 (secure 0 0 0) 0x7f121402e3e0 con 0x7f121c110c00 2024-05-27T09:31:31.530 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:31:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":79,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:31:27.747239+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:31:31.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 -- 172.21.15.44:0/3350334577 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12000b2880 msgr2=0x7f12000b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:31.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12000b2880 0x7f12000b4d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f120c011490 tx=0x7f120c009260 comp rx=0 tx=0).stop 2024-05-27T09:31:31.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 -- 172.21.15.44:0/3350334577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f121c110c00 msgr2=0x7f121c1a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:31.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f121c110c00 0x7f121c1a1900 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f121400b070 tx=0x7f1214009c90 comp rx=0 tx=0).stop 2024-05-27T09:31:31.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 -- 172.21.15.44:0/3350334577 shutdown_connections 2024-05-27T09:31:31.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12000b2880 0x7f12000b4d90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:31.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f121c110c00 0x7f121c1a1900 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:31.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f121c076be0 0x7f121c19d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:31.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 --2- 172.21.15.44:0/3350334577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f121c077530 0x7f121c19d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:31.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 -- 172.21.15.44:0/3350334577 >> 172.21.15.44:0/3350334577 conn(0x7f121c0ffec0 msgr2=0x7f121c1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:31.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 -- 172.21.15.44:0/3350334577 shutdown_connections 2024-05-27T09:31:31.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:31.534+0000 7f1221200700 1 -- 172.21.15.44:0/3350334577 wait complete. 2024-05-27T09:31:32.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:31 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3350334577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:31:32.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:31 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3350334577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:31:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:31 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3350334577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:31:32.351 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T09:31:32.816 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:33.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:32 smithi044 ceph-mon[31618]: pgmap v327: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T09:31:33.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:32 smithi050 ceph-mon[36371]: pgmap v327: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T09:31:33.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:32 smithi189 ceph-mon[36671]: pgmap v327: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T09:31:34.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.836+0000 7fd7f5200700 1 -- 172.21.15.44:0/461629279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f010a980 msgr2=0x7fd7f010ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:34.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.836+0000 7fd7f5200700 1 --2- 172.21.15.44:0/461629279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f010a980 0x7fd7f010ad90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8009990 tx=0x7fd7d8009ca0 comp rx=0 tx=0).stop 2024-05-27T09:31:34.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.836+0000 7fd7f5200700 1 -- 172.21.15.44:0/461629279 shutdown_connections 2024-05-27T09:31:34.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.836+0000 7fd7f5200700 1 --2- 172.21.15.44:0/461629279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f010b2d0 0x7fd7f0117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:34.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.836+0000 7fd7f5200700 1 --2- 172.21.15.44:0/461629279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f0110960 0x7fd7f0110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:34.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.836+0000 7fd7f5200700 1 --2- 172.21.15.44:0/461629279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f010a980 0x7fd7f010ad90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:34.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.836+0000 7fd7f5200700 1 -- 172.21.15.44:0/461629279 >> 172.21.15.44:0/461629279 conn(0x7fd7f0077c70 msgr2=0x7fd7f00780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.836+0000 7fd7f5200700 1 -- 172.21.15.44:0/461629279 shutdown_connections 2024-05-27T09:31:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.837+0000 7fd7f5200700 1 -- 172.21.15.44:0/461629279 wait complete. 2024-05-27T09:31:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.838+0000 7fd7f5200700 1 Processor -- start 2024-05-27T09:31:34.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.838+0000 7fd7f5200700 1 -- start start 2024-05-27T09:31:34.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.838+0000 7fd7f5200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f010a980 0x7fd7f01a14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:34.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.838+0000 7fd7f5200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f010b2d0 0x7fd7f01a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:34.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.838+0000 7fd7f5200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f0110960 0x7fd7f01a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:34.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.838+0000 7fd7f5200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f011aad0 con 0x7fd7f0110960 2024-05-27T09:31:34.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.838+0000 7fd7f5200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f011a950 con 0x7fd7f010b2d0 2024-05-27T09:31:34.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.838+0000 7fd7f5200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f011ac50 con 0x7fd7f010a980 2024-05-27T09:31:34.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.839+0000 7fd7e5400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f010b2d0 0x7fd7f01a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:34.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.839+0000 7fd7e5400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f010b2d0 0x7fd7f01a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34754/0 (socket says 172.21.15.44:34754) 2024-05-27T09:31:34.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.839+0000 7fd7e5400700 1 -- 172.21.15.44:0/2728564166 learned_addr learned my addr 172.21.15.44:0/2728564166 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:34.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.839+0000 7fd7ede00700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f010a980 0x7fd7f01a14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:34.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.839+0000 7fd7ee800700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f0110960 0x7fd7f01a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:34.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.839+0000 7fd7e5400700 1 -- 172.21.15.44:0/2728564166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f010a980 msgr2=0x7fd7f01a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:34.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.839+0000 7fd7e5400700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f010a980 0x7fd7f01a14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.839+0000 7fd7e5400700 1 -- 172.21.15.44:0/2728564166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f0110960 msgr2=0x7fd7f01a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.839+0000 7fd7e5400700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f0110960 0x7fd7f01a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.839+0000 7fd7e5400700 1 -- 172.21.15.44:0/2728564166 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f01a6370 con 0x7fd7f010b2d0 2024-05-27T09:31:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.839+0000 7fd7ede00700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f010a980 0x7fd7f01a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:31:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.840+0000 7fd7e5400700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f010b2d0 0x7fd7f01a1a40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8000c00 tx=0x7fd7d80047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:34.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.840+0000 7fd7e7400700 1 -- 172.21.15.44:0/2728564166 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8026070 con 0x7fd7f010b2d0 2024-05-27T09:31:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.840+0000 7fd7e7400700 1 -- 172.21.15.44:0/2728564166 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7d8021410 con 0x7fd7f010b2d0 2024-05-27T09:31:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.840+0000 7fd7f5200700 1 -- 172.21.15.44:0/2728564166 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f01a6600 con 0x7fd7f010b2d0 2024-05-27T09:31:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.840+0000 7fd7e7400700 1 -- 172.21.15.44:0/2728564166 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8017650 con 0x7fd7f010b2d0 2024-05-27T09:31:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.840+0000 7fd7f5200700 1 -- 172.21.15.44:0/2728564166 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f01adf80 con 0x7fd7f010b2d0 2024-05-27T09:31:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.841+0000 7fd7f5200700 1 -- 172.21.15.44:0/2728564166 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f010c070 con 0x7fd7f010b2d0 2024-05-27T09:31:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.844+0000 7fd7e7400700 1 -- 172.21.15.44:0/2728564166 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fd7d8029070 con 0x7fd7f010b2d0 2024-05-27T09:31:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.845+0000 7fd7e7400700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd7d40b2900 0x7fd7d40b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.845+0000 7fd7e7400700 1 -- 172.21.15.44:0/2728564166 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7fd7d80e2ea0 con 0x7fd7f010b2d0 2024-05-27T09:31:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.845+0000 7fd7ede00700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd7d40b2900 0x7fd7d40b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:34.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.846+0000 7fd7ede00700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd7d40b2900 0x7fd7d40b4e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd7f007d0e0 tx=0x7fd7e00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:34.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:34.850+0000 7fd7e7400700 1 -- 172.21.15.44:0/2728564166 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd7d8013070 con 0x7fd7f010b2d0 2024-05-27T09:31:35.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.089+0000 7fd7f5200700 1 -- 172.21.15.44:0/2728564166 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7f00512b0 con 0x7fd7f010b2d0 2024-05-27T09:31:35.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.091+0000 7fd7e7400700 1 -- 172.21.15.44:0/2728564166 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22160 (secure 0 0 0) 0x7fd7d80aad00 con 0x7fd7f010b2d0 2024-05-27T09:31:35.093 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:31:35.093 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":79,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:31:27.747239+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:31:35.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.094+0000 7fd7f5200700 1 -- 172.21.15.44:0/2728564166 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd7d40b2900 msgr2=0x7fd7d40b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:35.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.094+0000 7fd7f5200700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd7d40b2900 0x7fd7d40b4e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd7f007d0e0 tx=0x7fd7e00073d0 comp rx=0 tx=0).stop 2024-05-27T09:31:35.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.095+0000 7fd7f5200700 1 -- 172.21.15.44:0/2728564166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f010b2d0 msgr2=0x7fd7f01a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.095+0000 7fd7f5200700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f010b2d0 0x7fd7f01a1a40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8000c00 tx=0x7fd7d80047b0 comp rx=0 tx=0).stop 2024-05-27T09:31:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.095+0000 7fd7f5200700 1 -- 172.21.15.44:0/2728564166 shutdown_connections 2024-05-27T09:31:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.095+0000 7fd7f5200700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd7d40b2900 0x7fd7d40b4e10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.095+0000 7fd7f5200700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f0110960 0x7fd7f01a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.095+0000 7fd7f5200700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f010a980 0x7fd7f01a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.095+0000 7fd7f5200700 1 --2- 172.21.15.44:0/2728564166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f010b2d0 0x7fd7f01a1a40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.095+0000 7fd7f5200700 1 -- 172.21.15.44:0/2728564166 >> 172.21.15.44:0/2728564166 conn(0x7fd7f0077c70 msgr2=0x7fd7f007b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.095+0000 7fd7f5200700 1 -- 172.21.15.44:0/2728564166 shutdown_connections 2024-05-27T09:31:35.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:35.095+0000 7fd7f5200700 1 -- 172.21.15.44:0/2728564166 wait complete. 2024-05-27T09:31:35.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:35 smithi044 ceph-mon[31618]: pgmap v328: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T09:31:35.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:35 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2728564166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:31:35.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:35 smithi050 ceph-mon[36371]: pgmap v328: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T09:31:35.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:35 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2728564166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:31:35.833 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-05-27T09:31:36.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:35 smithi189 ceph-mon[36671]: pgmap v328: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T09:31:36.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:35 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2728564166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:31:36.309 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:37.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:37 smithi044 ceph-mon[31618]: pgmap v329: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-27T09:31:38.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:37 smithi050 ceph-mon[36371]: pgmap v329: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-27T09:31:38.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:37 smithi189 ceph-mon[36671]: pgmap v329: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-27T09:31:38.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.335+0000 7fb31c600700 1 -- 172.21.15.44:0/2385861774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb31410e8e0 msgr2=0x7fb314110dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:38.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.335+0000 7fb31c600700 1 --2- 172.21.15.44:0/2385861774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb31410e8e0 0x7fb314110dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00b0a0 tx=0x7fb30c00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:31:38.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.335+0000 7fb31c600700 1 -- 172.21.15.44:0/2385861774 shutdown_connections 2024-05-27T09:31:38.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.335+0000 7fb31c600700 1 --2- 172.21.15.44:0/2385861774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb31410e8e0 0x7fb314110dc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:38.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.335+0000 7fb31c600700 1 --2- 172.21.15.44:0/2385861774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3141054b0 0x7fb3141058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:38.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.335+0000 7fb31c600700 1 --2- 172.21.15.44:0/2385861774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb314105e00 0x7fb31410e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:38.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.335+0000 7fb31c600700 1 -- 172.21.15.44:0/2385861774 >> 172.21.15.44:0/2385861774 conn(0x7fb3140fedb0 msgr2=0x7fb3141011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:38.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.335+0000 7fb31c600700 1 -- 172.21.15.44:0/2385861774 shutdown_connections 2024-05-27T09:31:38.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.335+0000 7fb31c600700 1 -- 172.21.15.44:0/2385861774 wait complete. 2024-05-27T09:31:38.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.336+0000 7fb31c600700 1 Processor -- start 2024-05-27T09:31:38.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.337+0000 7fb31c600700 1 -- start start 2024-05-27T09:31:38.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.337+0000 7fb31c600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3141054b0 0x7fb3141031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:38.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.337+0000 7fb31c600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb314105e00 0x7fb3141036e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:38.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.337+0000 7fb31c600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb31410e8e0 0x7fb314103c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:38.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.337+0000 7fb31c600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb314114440 con 0x7fb3141054b0 2024-05-27T09:31:38.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.337+0000 7fb31c600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb3141142c0 con 0x7fb31410e8e0 2024-05-27T09:31:38.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.337+0000 7fb31c600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb3141145c0 con 0x7fb314105e00 2024-05-27T09:31:38.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb319c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3141054b0 0x7fb3141031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:38.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb319c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3141054b0 0x7fb3141031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41242/0 (socket says 172.21.15.44:41242) 2024-05-27T09:31:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb319c00700 1 -- 172.21.15.44:0/2553932696 learned_addr learned my addr 172.21.15.44:0/2553932696 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb31a600700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb31410e8e0 0x7fb314103c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb319200700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb314105e00 0x7fb3141036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb31a600700 1 -- 172.21.15.44:0/2553932696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb314105e00 msgr2=0x7fb3141036e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb31a600700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb314105e00 0x7fb3141036e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb31a600700 1 -- 172.21.15.44:0/2553932696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3141054b0 msgr2=0x7fb3141031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb31a600700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3141054b0 0x7fb3141031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb31a600700 1 -- 172.21.15.44:0/2553932696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3141a9130 con 0x7fb31410e8e0 2024-05-27T09:31:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb319200700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb314105e00 0x7fb3141036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:31:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.338+0000 7fb319c00700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3141054b0 0x7fb3141031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:31:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.339+0000 7fb31a600700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb31410e8e0 0x7fb314103c20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00b070 tx=0x7fb30c004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:38.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.339+0000 7fb312000700 1 -- 172.21.15.44:0/2553932696 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c021070 con 0x7fb31410e8e0 2024-05-27T09:31:38.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.339+0000 7fb312000700 1 -- 172.21.15.44:0/2553932696 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb30c007e30 con 0x7fb31410e8e0 2024-05-27T09:31:38.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.339+0000 7fb312000700 1 -- 172.21.15.44:0/2553932696 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c0045d0 con 0x7fb31410e8e0 2024-05-27T09:31:38.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.339+0000 7fb31c600700 1 -- 172.21.15.44:0/2553932696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3141a93c0 con 0x7fb31410e8e0 2024-05-27T09:31:38.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.339+0000 7fb31c600700 1 -- 172.21.15.44:0/2553932696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3141a98b0 con 0x7fb31410e8e0 2024-05-27T09:31:38.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.340+0000 7fb31c600700 1 -- 172.21.15.44:0/2553932696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3140512b0 con 0x7fb31410e8e0 2024-05-27T09:31:38.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.343+0000 7fb312000700 1 -- 172.21.15.44:0/2553932696 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fb30c003c00 con 0x7fb31410e8e0 2024-05-27T09:31:38.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.344+0000 7fb312000700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb2fc0b2900 0x7fb2fc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:38.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.345+0000 7fb312000700 1 -- 172.21.15.44:0/2553932696 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7fb30c00f020 con 0x7fb31410e8e0 2024-05-27T09:31:38.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.345+0000 7fb319c00700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb2fc0b2900 0x7fb2fc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:38.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.346+0000 7fb319c00700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb2fc0b2900 0x7fb2fc0b4e10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb308011550 tx=0x7fb308009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:38.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.347+0000 7fb312000700 1 -- 172.21.15.44:0/2553932696 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb30c02a020 con 0x7fb31410e8e0 2024-05-27T09:31:38.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.634+0000 7fb31c600700 1 -- 172.21.15.44:0/2553932696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb31402e7a0 con 0x7fb31410e8e0 2024-05-27T09:31:38.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.699+0000 7fb312000700 1 -- 172.21.15.44:0/2553932696 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7fb30c0ab540 con 0x7fb31410e8e0 2024-05-27T09:31:38.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 -- 172.21.15.44:0/2553932696 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb2fc0b2900 msgr2=0x7fb2fc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:38.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb2fc0b2900 0x7fb2fc0b4e10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb308011550 tx=0x7fb308009260 comp rx=0 tx=0).stop 2024-05-27T09:31:38.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 -- 172.21.15.44:0/2553932696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb31410e8e0 msgr2=0x7fb314103c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:38.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb31410e8e0 0x7fb314103c20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00b070 tx=0x7fb30c004d00 comp rx=0 tx=0).stop 2024-05-27T09:31:38.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 -- 172.21.15.44:0/2553932696 shutdown_connections 2024-05-27T09:31:38.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb2fc0b2900 0x7fb2fc0b4e10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:38.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb31410e8e0 0x7fb314103c20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:38.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3141054b0 0x7fb3141031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:38.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 --2- 172.21.15.44:0/2553932696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb314105e00 0x7fb3141036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:38.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 -- 172.21.15.44:0/2553932696 >> 172.21.15.44:0/2553932696 conn(0x7fb3140fedb0 msgr2=0x7fb3140ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:38.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 -- 172.21.15.44:0/2553932696 shutdown_connections 2024-05-27T09:31:38.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:38.703+0000 7fb31c600700 1 -- 172.21.15.44:0/2553932696 wait complete. 2024-05-27T09:31:38.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:38 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2553932696' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T09:31:38.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:38 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T09:31:39.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:38 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2553932696' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T09:31:39.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:38 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T09:31:39.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:38 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2553932696' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T09:31:39.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:38 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T09:31:39.613 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-05-27T09:31:39.613 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-05-27T09:31:39.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:39 smithi044 ceph-mon[31618]: pgmap v330: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-27T09:31:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:39 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T09:31:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:39 smithi044 ceph-mon[31618]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:39 smithi050 ceph-mon[36371]: pgmap v330: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-27T09:31:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:39 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T09:31:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:39 smithi050 ceph-mon[36371]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:40.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:39 smithi189 ceph-mon[36671]: pgmap v330: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-27T09:31:40.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:39 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T09:31:40.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:39 smithi189 ceph-mon[36671]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:40.112 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:41.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:41 smithi044 ceph-mon[31618]: pgmap v331: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-05-27T09:31:42.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:41 smithi050 ceph-mon[36371]: pgmap v331: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-05-27T09:31:42.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:41 smithi189 ceph-mon[36671]: pgmap v331: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-05-27T09:31:42.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.244+0000 7fddba600700 1 -- 172.21.15.44:0/2812804243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddb4109850 msgr2=0x7fddb41111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:42.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.244+0000 7fddba600700 1 --2- 172.21.15.44:0/2812804243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddb4109850 0x7fddb41111b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fddac00b0d0 tx=0x7fddac00b3e0 comp rx=0 tx=0).stop 2024-05-27T09:31:42.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.244+0000 7fddba600700 1 -- 172.21.15.44:0/2812804243 shutdown_connections 2024-05-27T09:31:42.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.244+0000 7fddba600700 1 --2- 172.21.15.44:0/2812804243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddb4109850 0x7fddb41111b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:42.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.244+0000 7fddba600700 1 --2- 172.21.15.44:0/2812804243 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb41045d0 0x7fddb41049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:42.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.244+0000 7fddba600700 1 --2- 172.21.15.44:0/2812804243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb4108f00 0x7fddb4109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:42.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.244+0000 7fddba600700 1 -- 172.21.15.44:0/2812804243 >> 172.21.15.44:0/2812804243 conn(0x7fddb40ffec0 msgr2=0x7fddb4102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:42.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.244+0000 7fddba600700 1 -- 172.21.15.44:0/2812804243 shutdown_connections 2024-05-27T09:31:42.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.244+0000 7fddba600700 1 -- 172.21.15.44:0/2812804243 wait complete. 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.246+0000 7fddba600700 1 Processor -- start 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.246+0000 7fddba600700 1 -- start start 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.247+0000 7fddba600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb41045d0 0x7fddb419d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.247+0000 7fddba600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb4108f00 0x7fddb419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.247+0000 7fddba600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddb4109850 0x7fddb41a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.247+0000 7fddba600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fddb4114480 con 0x7fddb4109850 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.247+0000 7fddba600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fddb4114300 con 0x7fddb41045d0 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.247+0000 7fddba600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fddb4114600 con 0x7fddb4108f00 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.247+0000 7fddb2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb4108f00 0x7fddb419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.247+0000 7fddb2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb4108f00 0x7fddb419d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:52356/0 (socket says 172.21.15.44:52356) 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.247+0000 7fddb2a00700 1 -- 172.21.15.44:0/4229451227 learned_addr learned my addr 172.21.15.44:0/4229451227 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.247+0000 7fddb3400700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb41045d0 0x7fddb419d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:42.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.247+0000 7fddb3e00700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddb4109850 0x7fddb41a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:42.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.248+0000 7fddb2a00700 1 -- 172.21.15.44:0/4229451227 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb41045d0 msgr2=0x7fddb419d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:42.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.248+0000 7fddb2a00700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb41045d0 0x7fddb419d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:42.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.248+0000 7fddb2a00700 1 -- 172.21.15.44:0/4229451227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddb4109850 msgr2=0x7fddb41a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:42.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.248+0000 7fddb2a00700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddb4109850 0x7fddb41a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:42.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.248+0000 7fddb2a00700 1 -- 172.21.15.44:0/4229451227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb41a1f20 con 0x7fddb4108f00 2024-05-27T09:31:42.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.248+0000 7fddb2a00700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb4108f00 0x7fddb419d5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdda000e970 tx=0x7fdda000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:42.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.249+0000 7fddabe00700 1 -- 172.21.15.44:0/4229451227 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda0004a30 con 0x7fddb4108f00 2024-05-27T09:31:42.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.249+0000 7fddabe00700 1 -- 172.21.15.44:0/4229451227 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdda000cc90 con 0x7fddb4108f00 2024-05-27T09:31:42.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.249+0000 7fddba600700 1 -- 172.21.15.44:0/4229451227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddb41a2210 con 0x7fddb4108f00 2024-05-27T09:31:42.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.249+0000 7fddabe00700 1 -- 172.21.15.44:0/4229451227 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda0005610 con 0x7fddb4108f00 2024-05-27T09:31:42.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.249+0000 7fddba600700 1 -- 172.21.15.44:0/4229451227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddb41a9b30 con 0x7fddb4108f00 2024-05-27T09:31:42.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.250+0000 7fddba600700 1 -- 172.21.15.44:0/4229451227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddb40512b0 con 0x7fddb4108f00 2024-05-27T09:31:42.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.253+0000 7fddabe00700 1 -- 172.21.15.44:0/4229451227 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fdda000c7e0 con 0x7fddb4108f00 2024-05-27T09:31:42.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.255+0000 7fddabe00700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdd980b2880 0x7fdd980b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:42.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.255+0000 7fddabe00700 1 -- 172.21.15.44:0/4229451227 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7fdda00e2250 con 0x7fddb4108f00 2024-05-27T09:31:42.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.255+0000 7fddb3400700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdd980b2880 0x7fdd980b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:42.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.256+0000 7fddb3400700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdd980b2880 0x7fdd980b4d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009680 tx=0x7fdda4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:42.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.259+0000 7fddabe00700 1 -- 172.21.15.44:0/4229451227 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdda00aa020 con 0x7fddb4108f00 2024-05-27T09:31:42.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.546+0000 7fddba600700 1 -- 172.21.15.44:0/4229451227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fddb41a9d00 con 0x7fddb4108f00 2024-05-27T09:31:42.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.721+0000 7fddabe00700 1 -- 172.21.15.44:0/4229451227 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fdda00a9920 con 0x7fddb4108f00 2024-05-27T09:31:42.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 -- 172.21.15.44:0/4229451227 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdd980b2880 msgr2=0x7fdd980b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:42.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdd980b2880 0x7fdd980b4d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009680 tx=0x7fdda4009260 comp rx=0 tx=0).stop 2024-05-27T09:31:42.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 -- 172.21.15.44:0/4229451227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb4108f00 msgr2=0x7fddb419d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:42.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb4108f00 0x7fddb419d5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdda000e970 tx=0x7fdda000ec80 comp rx=0 tx=0).stop 2024-05-27T09:31:42.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 -- 172.21.15.44:0/4229451227 shutdown_connections 2024-05-27T09:31:42.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdd980b2880 0x7fdd980b4d90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:42.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fddb4109850 0x7fddb41a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:42.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb41045d0 0x7fddb419d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:42.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 --2- 172.21.15.44:0/4229451227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb4108f00 0x7fddb419d5f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:42.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 -- 172.21.15.44:0/4229451227 >> 172.21.15.44:0/4229451227 conn(0x7fddb40ffec0 msgr2=0x7fddb4110e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:42.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 -- 172.21.15.44:0/4229451227 shutdown_connections 2024-05-27T09:31:42.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:42.725+0000 7fddba600700 1 -- 172.21.15.44:0/4229451227 wait complete. 2024-05-27T09:31:42.798 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:42 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4229451227' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-27T09:31:42.798 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:42 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-27T09:31:42.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:42 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/4229451227' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-27T09:31:42.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:42 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-27T09:31:43.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:42 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/4229451227' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-27T09:31:43.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:42 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-27T09:31:43.489 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-05-27T09:31:43.934 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:43.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:43 smithi044 ceph-mon[31618]: pgmap v332: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 0 op/s 2024-05-27T09:31:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:43 smithi044 ceph-mon[31618]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T09:31:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:43 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-27T09:31:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:43 smithi044 ceph-mon[31618]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T09:31:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:43 smithi044 ceph-mon[31618]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:43 smithi044 ceph-mon[31618]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-05-27T09:31:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:43 smithi044 ceph-mon[31618]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-27T09:31:44.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:43 smithi050 ceph-mon[36371]: pgmap v332: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 0 op/s 2024-05-27T09:31:44.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:43 smithi050 ceph-mon[36371]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T09:31:44.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:43 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-27T09:31:44.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:43 smithi050 ceph-mon[36371]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T09:31:44.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:43 smithi050 ceph-mon[36371]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:44.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:43 smithi050 ceph-mon[36371]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-05-27T09:31:44.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:43 smithi050 ceph-mon[36371]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-27T09:31:44.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:43 smithi189 ceph-mon[36671]: pgmap v332: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 0 op/s 2024-05-27T09:31:44.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:43 smithi189 ceph-mon[36671]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T09:31:44.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:43 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-27T09:31:44.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:43 smithi189 ceph-mon[36671]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T09:31:44.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:43 smithi189 ceph-mon[36671]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T09:31:44.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:43 smithi189 ceph-mon[36671]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-05-27T09:31:44.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:43 smithi189 ceph-mon[36671]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-27T09:31:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:44 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] up:active 2024-05-27T09:31:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:44 smithi050 ceph-mon[36371]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T09:31:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:44 smithi050 ceph-mon[36371]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T09:31:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:44 smithi050 ceph-mon[36371]: Cluster is now healthy 2024-05-27T09:31:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:44 smithi050 ceph-mon[36371]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-05-27T09:31:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:44 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-05-27T09:31:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:44 smithi050 ceph-mon[36371]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-27T09:31:45.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:44 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] up:active 2024-05-27T09:31:45.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:44 smithi189 ceph-mon[36671]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T09:31:45.090 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:44 smithi189 ceph-mon[36671]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T09:31:45.090 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:44 smithi189 ceph-mon[36671]: Cluster is now healthy 2024-05-27T09:31:45.090 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:44 smithi189 ceph-mon[36671]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-05-27T09:31:45.090 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:44 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-05-27T09:31:45.090 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:44 smithi189 ceph-mon[36671]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-27T09:31:45.149 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:44 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] up:active 2024-05-27T09:31:45.149 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:44 smithi044 ceph-mon[31618]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T09:31:45.149 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:44 smithi044 ceph-mon[31618]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T09:31:45.149 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:44 smithi044 ceph-mon[31618]: Cluster is now healthy 2024-05-27T09:31:45.149 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:44 smithi044 ceph-mon[31618]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-05-27T09:31:45.149 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:44 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-05-27T09:31:45.149 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:44 smithi044 ceph-mon[31618]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-27T09:31:45.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:45 smithi050 ceph-mon[36371]: pgmap v333: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-05-27T09:31:45.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:45 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] up:active 2024-05-27T09:31:45.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:45 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-05-27T09:31:45.794 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:31:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b[66721]: 2024-05-27T09:31:45.712+0000 7fc6ea134700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-05-27T09:31:46.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.085+0000 7f2510400700 1 -- 172.21.15.44:0/3413966465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250810c5b0 msgr2=0x7f250810c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:46.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.085+0000 7f2510400700 1 --2- 172.21.15.44:0/3413966465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250810c5b0 0x7f250810c9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009990 tx=0x7f24fc009ca0 comp rx=0 tx=0).stop 2024-05-27T09:31:46.088 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:45 smithi044 ceph-mon[31618]: pgmap v333: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-05-27T09:31:46.088 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:45 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] up:active 2024-05-27T09:31:46.088 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:45 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-05-27T09:31:46.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.085+0000 7f2510400700 1 -- 172.21.15.44:0/3413966465 shutdown_connections 2024-05-27T09:31:46.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.085+0000 7f2510400700 1 --2- 172.21.15.44:0/3413966465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2508106ec0 0x7f2508113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:46.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.085+0000 7f2510400700 1 --2- 172.21.15.44:0/3413966465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250810c5b0 0x7f250810c9c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.085+0000 7f2510400700 1 --2- 172.21.15.44:0/3413966465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2508106570 0x7f2508106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.085+0000 7f2510400700 1 -- 172.21.15.44:0/3413966465 >> 172.21.15.44:0/3413966465 conn(0x7f2508102070 msgr2=0x7f25081044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.086+0000 7f2510400700 1 -- 172.21.15.44:0/3413966465 shutdown_connections 2024-05-27T09:31:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.086+0000 7f2510400700 1 -- 172.21.15.44:0/3413966465 wait complete. 2024-05-27T09:31:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.087+0000 7f2510400700 1 Processor -- start 2024-05-27T09:31:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.088+0000 7f2510400700 1 -- start start 2024-05-27T09:31:46.090 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:45 smithi189 ceph-mon[36671]: pgmap v333: 97 pgs: 97 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-05-27T09:31:46.090 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:45 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] up:active 2024-05-27T09:31:46.090 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:45 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-05-27T09:31:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.088+0000 7f2510400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2508106570 0x7f250819d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.088+0000 7f2510400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2508106ec0 0x7f250819d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.088+0000 7f2510400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f250810c5b0 0x7f25081a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.088+0000 7f2510400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2508116710 con 0x7f250810c5b0 2024-05-27T09:31:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.088+0000 7f2510400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2508116590 con 0x7f2508106570 2024-05-27T09:31:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.088+0000 7f2510400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2508116890 con 0x7f2508106ec0 2024-05-27T09:31:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.089+0000 7f250da00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2508106570 0x7f250819d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.089+0000 7f250da00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2508106570 0x7f250819d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47208/0 (socket says 172.21.15.44:47208) 2024-05-27T09:31:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.089+0000 7f250da00700 1 -- 172.21.15.44:0/2514465244 learned_addr learned my addr 172.21.15.44:0/2514465244 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.089+0000 7f250d000700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2508106ec0 0x7f250819d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.089+0000 7f250e400700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f250810c5b0 0x7f25081a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.089+0000 7f250da00700 1 -- 172.21.15.44:0/2514465244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2508106ec0 msgr2=0x7f250819d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.089+0000 7f250da00700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2508106ec0 0x7f250819d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.089+0000 7f250da00700 1 -- 172.21.15.44:0/2514465244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f250810c5b0 msgr2=0x7f25081a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.089+0000 7f250da00700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f250810c5b0 0x7f25081a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.089+0000 7f250da00700 1 -- 172.21.15.44:0/2514465244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25081a1f50 con 0x7f2508106570 2024-05-27T09:31:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.089+0000 7f250e400700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f250810c5b0 0x7f25081a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:31:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.090+0000 7f250da00700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2508106570 0x7f250819d080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f24fc000c00 tx=0x7f24fc0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.090+0000 7f2506000700 1 -- 172.21.15.44:0/2514465244 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24fc026070 con 0x7f2508106570 2024-05-27T09:31:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.090+0000 7f2506000700 1 -- 172.21.15.44:0/2514465244 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24fc021410 con 0x7f2508106570 2024-05-27T09:31:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.090+0000 7f2506000700 1 -- 172.21.15.44:0/2514465244 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24fc017650 con 0x7f2508106570 2024-05-27T09:31:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.090+0000 7f2510400700 1 -- 172.21.15.44:0/2514465244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25081a21e0 con 0x7f2508106570 2024-05-27T09:31:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.090+0000 7f2510400700 1 -- 172.21.15.44:0/2514465244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25081a2670 con 0x7f2508106570 2024-05-27T09:31:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.091+0000 7f2510400700 1 -- 172.21.15.44:0/2514465244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25080512b0 con 0x7f2508106570 2024-05-27T09:31:46.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.094+0000 7f2506000700 1 -- 172.21.15.44:0/2514465244 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f24fc029070 con 0x7f2508106570 2024-05-27T09:31:46.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.095+0000 7f2506000700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f24f00b2900 0x7f24f00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:46.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.095+0000 7f2506000700 1 -- 172.21.15.44:0/2514465244 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f24fc0e30a0 con 0x7f2508106570 2024-05-27T09:31:46.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.096+0000 7f250d000700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f24f00b2900 0x7f24f00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:46.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.096+0000 7f250d000700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f24f00b2900 0x7f24f00b4e10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2508103c50 tx=0x7f24f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:46.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.100+0000 7f2506000700 1 -- 172.21.15.44:0/2514465244 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f24fc013070 con 0x7f2508106570 2024-05-27T09:31:46.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.389+0000 7f2510400700 1 -- 172.21.15.44:0/2514465244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f250802e5e0 con 0x7f2508106570 2024-05-27T09:31:46.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.768+0000 7f2506000700 1 -- 172.21.15.44:0/2514465244 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f24fc0aaf00 con 0x7f2508106570 2024-05-27T09:31:46.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.773+0000 7f2510400700 1 -- 172.21.15.44:0/2514465244 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f24f00b2900 msgr2=0x7f24f00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:46.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.773+0000 7f2510400700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f24f00b2900 0x7f24f00b4e10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2508103c50 tx=0x7f24f4009260 comp rx=0 tx=0).stop 2024-05-27T09:31:46.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.773+0000 7f2510400700 1 -- 172.21.15.44:0/2514465244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2508106570 msgr2=0x7f250819d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:46.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.773+0000 7f2510400700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2508106570 0x7f250819d080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f24fc000c00 tx=0x7f24fc0047b0 comp rx=0 tx=0).stop 2024-05-27T09:31:46.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.773+0000 7f2510400700 1 -- 172.21.15.44:0/2514465244 shutdown_connections 2024-05-27T09:31:46.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.773+0000 7f2510400700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f24f00b2900 0x7f24f00b4e10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:46.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.773+0000 7f2510400700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f250810c5b0 0x7f25081a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:46.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.773+0000 7f2510400700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2508106570 0x7f250819d080 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:46.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.773+0000 7f2510400700 1 --2- 172.21.15.44:0/2514465244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2508106ec0 0x7f250819d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:46.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.773+0000 7f2510400700 1 -- 172.21.15.44:0/2514465244 >> 172.21.15.44:0/2514465244 conn(0x7f2508102070 msgr2=0x7f2508104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:46.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.773+0000 7f2510400700 1 -- 172.21.15.44:0/2514465244 shutdown_connections 2024-05-27T09:31:46.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:46.774+0000 7f2510400700 1 -- 172.21.15.44:0/2514465244 wait complete. 2024-05-27T09:31:47.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:46 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2514465244' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T09:31:47.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:46 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T09:31:47.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:46 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2514465244' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T09:31:47.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:46 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T09:31:47.152 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:46 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2514465244' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T09:31:47.152 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:46 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T09:31:47.588 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-05-27T09:31:48.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:47 smithi050 ceph-mon[36371]: pgmap v334: 97 pgs: 97 active+clean; 580 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T09:31:48.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:47 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T09:31:48.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:47 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-05-27T09:31:48.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:47 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:31:48.069 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:48.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:47 smithi189 ceph-mon[36671]: pgmap v334: 97 pgs: 97 active+clean; 580 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T09:31:48.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:47 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T09:31:48.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:47 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-05-27T09:31:48.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:47 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:31:48.219 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:47 smithi044 ceph-mon[31618]: pgmap v334: 97 pgs: 97 active+clean; 580 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T09:31:48.219 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:47 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T09:31:48.220 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:47 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-05-27T09:31:48.220 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:47 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:31:49.866 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:49 smithi044 ceph-mon[31618]: pgmap v335: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-27T09:31:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:49 smithi050 ceph-mon[36371]: pgmap v335: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-27T09:31:50.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:49 smithi189 ceph-mon[36671]: pgmap v335: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-27T09:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.121+0000 7fbd35c00700 1 -- 172.21.15.44:0/625684302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd30110960 msgr2=0x7fbd30110d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.121+0000 7fbd35c00700 1 --2- 172.21.15.44:0/625684302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd30110960 0x7fbd30110d70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009990 tx=0x7fbd20009ca0 comp rx=0 tx=0).stop 2024-05-27T09:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.121+0000 7fbd35c00700 1 -- 172.21.15.44:0/625684302 shutdown_connections 2024-05-27T09:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.121+0000 7fbd35c00700 1 --2- 172.21.15.44:0/625684302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3010b2d0 0x7fbd30117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.121+0000 7fbd35c00700 1 --2- 172.21.15.44:0/625684302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd30110960 0x7fbd30110d70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.121+0000 7fbd35c00700 1 --2- 172.21.15.44:0/625684302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd3010a980 0x7fbd3010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.121+0000 7fbd35c00700 1 -- 172.21.15.44:0/625684302 >> 172.21.15.44:0/625684302 conn(0x7fbd30077c70 msgr2=0x7fbd300780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.121+0000 7fbd35c00700 1 -- 172.21.15.44:0/625684302 shutdown_connections 2024-05-27T09:31:50.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.121+0000 7fbd35c00700 1 -- 172.21.15.44:0/625684302 wait complete. 2024-05-27T09:31:50.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.122+0000 7fbd35c00700 1 Processor -- start 2024-05-27T09:31:50.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.123+0000 7fbd35c00700 1 -- start start 2024-05-27T09:31:50.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.123+0000 7fbd35c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd3010a980 0x7fbd301a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:50.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.123+0000 7fbd35c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3010b2d0 0x7fbd301a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:50.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.123+0000 7fbd35c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd30110960 0x7fbd301a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:50.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.123+0000 7fbd35c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3011aab0 con 0x7fbd30110960 2024-05-27T09:31:50.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.123+0000 7fbd35c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3011a930 con 0x7fbd3010b2d0 2024-05-27T09:31:50.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.123+0000 7fbd35c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3011ac30 con 0x7fbd3010a980 2024-05-27T09:31:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.124+0000 7fbd26000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3010b2d0 0x7fbd301a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.124+0000 7fbd2ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd3010a980 0x7fbd301a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.124+0000 7fbd2f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd30110960 0x7fbd301a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.124+0000 7fbd2ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd3010a980 0x7fbd301a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:48346/0 (socket says 172.21.15.44:48346) 2024-05-27T09:31:50.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.124+0000 7fbd2ea00700 1 -- 172.21.15.44:0/1776093239 learned_addr learned my addr 172.21.15.44:0/1776093239 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:50.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.124+0000 7fbd2ea00700 1 -- 172.21.15.44:0/1776093239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3010b2d0 msgr2=0x7fbd301a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:50.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.124+0000 7fbd2ea00700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3010b2d0 0x7fbd301a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:50.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.124+0000 7fbd2ea00700 1 -- 172.21.15.44:0/1776093239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd30110960 msgr2=0x7fbd301a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:50.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.124+0000 7fbd2ea00700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd30110960 0x7fbd301a5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:50.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.124+0000 7fbd2ea00700 1 -- 172.21.15.44:0/1776093239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd301a63f0 con 0x7fbd3010a980 2024-05-27T09:31:50.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.125+0000 7fbd2ea00700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd3010a980 0x7fbd301a1520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbd20000c00 tx=0x7fbd200057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:50.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.125+0000 7fbd27e00700 1 -- 172.21.15.44:0/1776093239 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20027070 con 0x7fbd3010a980 2024-05-27T09:31:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.125+0000 7fbd35c00700 1 -- 172.21.15.44:0/1776093239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd301a6680 con 0x7fbd3010a980 2024-05-27T09:31:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.125+0000 7fbd27e00700 1 -- 172.21.15.44:0/1776093239 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd20022470 con 0x7fbd3010a980 2024-05-27T09:31:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.125+0000 7fbd27e00700 1 -- 172.21.15.44:0/1776093239 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20017650 con 0x7fbd3010a980 2024-05-27T09:31:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.125+0000 7fbd35c00700 1 -- 172.21.15.44:0/1776093239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd301a6b60 con 0x7fbd3010a980 2024-05-27T09:31:50.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.126+0000 7fbd35c00700 1 -- 172.21.15.44:0/1776093239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd300512b0 con 0x7fbd3010a980 2024-05-27T09:31:50.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.129+0000 7fbd27e00700 1 -- 172.21.15.44:0/1776093239 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fbd2002a070 con 0x7fbd3010a980 2024-05-27T09:31:50.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.130+0000 7fbd27e00700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbd140b2880 0x7fbd140b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:50.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.130+0000 7fbd27e00700 1 -- 172.21.15.44:0/1776093239 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7fbd200e3070 con 0x7fbd3010a980 2024-05-27T09:31:50.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.130+0000 7fbd26000700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbd140b2880 0x7fbd140b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:50.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.131+0000 7fbd26000700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbd140b2880 0x7fbd140b4d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd3007b0c0 tx=0x7fbd1800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:50.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.132+0000 7fbd27e00700 1 -- 172.21.15.44:0/1776093239 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbd20013070 con 0x7fbd3010a980 2024-05-27T09:31:50.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.421+0000 7fbd35c00700 1 -- 172.21.15.44:0/1776093239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fbd3002e5e0 con 0x7fbd3010a980 2024-05-27T09:31:50.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.792+0000 7fbd27e00700 1 -- 172.21.15.44:0/1776093239 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7fbd200aaed0 con 0x7fbd3010a980 2024-05-27T09:31:50.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 -- 172.21.15.44:0/1776093239 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbd140b2880 msgr2=0x7fbd140b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:50.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbd140b2880 0x7fbd140b4d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd3007b0c0 tx=0x7fbd1800b040 comp rx=0 tx=0).stop 2024-05-27T09:31:50.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 -- 172.21.15.44:0/1776093239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd3010a980 msgr2=0x7fbd301a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:50.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd3010a980 0x7fbd301a1520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbd20000c00 tx=0x7fbd200057c0 comp rx=0 tx=0).stop 2024-05-27T09:31:50.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 -- 172.21.15.44:0/1776093239 shutdown_connections 2024-05-27T09:31:50.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbd140b2880 0x7fbd140b4d90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:50.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd30110960 0x7fbd301a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:50.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd3010a980 0x7fbd301a1520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:50.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 --2- 172.21.15.44:0/1776093239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd3010b2d0 0x7fbd301a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:50.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 -- 172.21.15.44:0/1776093239 >> 172.21.15.44:0/1776093239 conn(0x7fbd30077c70 msgr2=0x7fbd3007bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:50.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 -- 172.21.15.44:0/1776093239 shutdown_connections 2024-05-27T09:31:50.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:50.797+0000 7fbd35c00700 1 -- 172.21.15.44:0/1776093239 wait complete. 2024-05-27T09:31:51.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:50 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1776093239' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T09:31:51.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:50 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T09:31:51.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:50 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1776093239' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T09:31:51.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:50 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T09:31:51.198 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:50 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1776093239' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T09:31:51.199 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:50 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T09:31:51.558 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-05-27T09:31:51.558 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-05-27T09:31:52.038 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:52.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:51 smithi050 ceph-mon[36371]: pgmap v336: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:31:52.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:51 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T09:31:52.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:51 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:31:52.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:51 smithi189 ceph-mon[36671]: pgmap v336: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:31:52.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:51 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T09:31:52.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:51 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:31:52.192 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:51 smithi044 ceph-mon[31618]: pgmap v336: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:31:52.193 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:51 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T09:31:52.193 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:51 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:31:54.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:53 smithi189 ceph-mon[36671]: pgmap v337: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-27T09:31:54.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.100+0000 7f2f6e400700 1 -- 172.21.15.44:0/1575739441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f68105e40 msgr2=0x7f2f6810e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:54.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.100+0000 7f2f6e400700 1 --2- 172.21.15.44:0/1575739441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f68105e40 0x7f2f6810e390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2f54009990 tx=0x7f2f54009ca0 comp rx=0 tx=0).stop 2024-05-27T09:31:54.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:53 smithi044 ceph-mon[31618]: pgmap v337: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-27T09:31:54.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.100+0000 7f2f6e400700 1 -- 172.21.15.44:0/1575739441 shutdown_connections 2024-05-27T09:31:54.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.100+0000 7f2f6e400700 1 --2- 172.21.15.44:0/1575739441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6810e8d0 0x7f2f68110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:54.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.100+0000 7f2f6e400700 1 --2- 172.21.15.44:0/1575739441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f681054f0 0x7f2f68105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:54.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.100+0000 7f2f6e400700 1 --2- 172.21.15.44:0/1575739441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f68105e40 0x7f2f6810e390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:54.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.100+0000 7f2f6e400700 1 -- 172.21.15.44:0/1575739441 >> 172.21.15.44:0/1575739441 conn(0x7f2f680fedb0 msgr2=0x7f2f681011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:54.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.101+0000 7f2f6e400700 1 -- 172.21.15.44:0/1575739441 shutdown_connections 2024-05-27T09:31:54.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.101+0000 7f2f6e400700 1 -- 172.21.15.44:0/1575739441 wait complete. 2024-05-27T09:31:54.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.102+0000 7f2f6e400700 1 Processor -- start 2024-05-27T09:31:54.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.102+0000 7f2f6e400700 1 -- start start 2024-05-27T09:31:54.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.103+0000 7f2f6e400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f681054f0 0x7f2f6819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:54.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.103+0000 7f2f6e400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f68105e40 0x7f2f6819d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:54.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.103+0000 7f2f6e400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6810e8d0 0x7f2f681a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:54.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.103+0000 7f2f6e400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2f68114430 con 0x7f2f68105e40 2024-05-27T09:31:54.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.103+0000 7f2f6e400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2f681142b0 con 0x7f2f681054f0 2024-05-27T09:31:54.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.103+0000 7f2f6e400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2f681145b0 con 0x7f2f6810e8d0 2024-05-27T09:31:54.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.103+0000 7f2f66a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f68105e40 0x7f2f6819d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:54.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.103+0000 7f2f67400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f681054f0 0x7f2f6819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:54.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.103+0000 7f2f66a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f68105e40 0x7f2f6819d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56002/0 (socket says 172.21.15.44:56002) 2024-05-27T09:31:54.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.103+0000 7f2f67e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6810e8d0 0x7f2f681a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:54.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.103+0000 7f2f66a00700 1 -- 172.21.15.44:0/502940542 learned_addr learned my addr 172.21.15.44:0/502940542 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:54.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.104+0000 7f2f67e00700 1 -- 172.21.15.44:0/502940542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f681054f0 msgr2=0x7f2f6819cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:54.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.104+0000 7f2f67e00700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f681054f0 0x7f2f6819cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:54.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.104+0000 7f2f67e00700 1 -- 172.21.15.44:0/502940542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f68105e40 msgr2=0x7f2f6819d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:54.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.104+0000 7f2f67e00700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f68105e40 0x7f2f6819d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:54.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.104+0000 7f2f67e00700 1 -- 172.21.15.44:0/502940542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f681a1eb0 con 0x7f2f6810e8d0 2024-05-27T09:31:54.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.104+0000 7f2f67e00700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6810e8d0 0x7f2f681a1810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000efc0 tx=0x7f2f6000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:54.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.104+0000 7f2f5fe00700 1 -- 172.21.15.44:0/502940542 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f600092e0 con 0x7f2f6810e8d0 2024-05-27T09:31:54.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.104+0000 7f2f5fe00700 1 -- 172.21.15.44:0/502940542 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f60004bb0 con 0x7f2f6810e8d0 2024-05-27T09:31:54.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.104+0000 7f2f6e400700 1 -- 172.21.15.44:0/502940542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f681a2140 con 0x7f2f6810e8d0 2024-05-27T09:31:54.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.104+0000 7f2f5fe00700 1 -- 172.21.15.44:0/502940542 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f60004020 con 0x7f2f6810e8d0 2024-05-27T09:31:54.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.105+0000 7f2f6e400700 1 -- 172.21.15.44:0/502940542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f681a2630 con 0x7f2f6810e8d0 2024-05-27T09:31:54.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.106+0000 7f2f6e400700 1 -- 172.21.15.44:0/502940542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f680512b0 con 0x7f2f6810e8d0 2024-05-27T09:31:54.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.108+0000 7f2f5fe00700 1 -- 172.21.15.44:0/502940542 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f2f6001b030 con 0x7f2f6810e8d0 2024-05-27T09:31:54.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.109+0000 7f2f5fe00700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f4c0b2880 0x7f2f4c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:54.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.110+0000 7f2f5fe00700 1 -- 172.21.15.44:0/502940542 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f2f600e1bc0 con 0x7f2f6810e8d0 2024-05-27T09:31:54.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.110+0000 7f2f67400700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f4c0b2880 0x7f2f4c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:54.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.111+0000 7f2f67400700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f4c0b2880 0x7f2f4c0b4d90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009c60 tx=0x7f2f58009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:54.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.114+0000 7f2f5fe00700 1 -- 172.21.15.44:0/502940542 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2f60022020 con 0x7f2f6810e8d0 2024-05-27T09:31:54.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:53 smithi050 ceph-mon[36371]: pgmap v337: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-27T09:31:54.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.386+0000 7f2f6e400700 1 -- 172.21.15.44:0/502940542 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f2f68103880 con 0x7f2f4c0b2880 2024-05-27T09:31:54.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.426+0000 7f2f5fe00700 1 -- 172.21.15.44:0/502940542 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2f68103880 con 0x7f2f4c0b2880 2024-05-27T09:31:54.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 -- 172.21.15.44:0/502940542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f4c0b2880 msgr2=0x7f2f4c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f4c0b2880 0x7f2f4c0b4d90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009c60 tx=0x7f2f58009260 comp rx=0 tx=0).stop 2024-05-27T09:31:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 -- 172.21.15.44:0/502940542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6810e8d0 msgr2=0x7f2f681a1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6810e8d0 0x7f2f681a1810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000efc0 tx=0x7f2f6000c3a0 comp rx=0 tx=0).stop 2024-05-27T09:31:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 -- 172.21.15.44:0/502940542 shutdown_connections 2024-05-27T09:31:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f4c0b2880 0x7f2f4c0b4d90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6810e8d0 0x7f2f681a1810 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f681054f0 0x7f2f6819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 --2- 172.21.15.44:0/502940542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f68105e40 0x7f2f6819d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 -- 172.21.15.44:0/502940542 >> 172.21.15.44:0/502940542 conn(0x7f2f680fedb0 msgr2=0x7f2f68103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 -- 172.21.15.44:0/502940542 shutdown_connections 2024-05-27T09:31:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:54.430+0000 7f2f6e400700 1 -- 172.21.15.44:0/502940542 wait complete. 2024-05-27T09:31:54.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:31:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:31:54.398+0000 7f6dc1600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:31:54.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:31:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:31:54.398+0000 7f6dc1600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:31:54.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:31:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:31:54.398+0000 7f6dc1600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:31:54.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:31:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:31:54.398+0000 7f6dc1600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:31:54.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:31:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:31:54.398+0000 7f6dc1600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:31:55.187 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-27T09:31:55.661 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:31:56.131 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:55 smithi044 ceph-mon[31618]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:31:56.131 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:55 smithi044 ceph-mon[31618]: pgmap v338: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:31:56.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:55 smithi050 ceph-mon[36371]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:31:56.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:55 smithi050 ceph-mon[36371]: pgmap v338: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:31:56.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:55 smithi189 ceph-mon[36671]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:31:56.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:55 smithi189 ceph-mon[36671]: pgmap v338: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:31:57.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:56 smithi044 ceph-mon[31618]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-27T09:31:57.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-05-27T09:31:57.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-05-27T09:31:57.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-05-27T09:31:57.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-05-27T09:31:57.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-05-27T09:31:57.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-05-27T09:31:57.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 6]}]: dispatch 2024-05-27T09:31:57.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:56 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-27T09:31:57.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:56 smithi050 ceph-mon[36371]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-27T09:31:57.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:56 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-05-27T09:31:57.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:56 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-05-27T09:31:57.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:56 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-05-27T09:31:57.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:56 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-05-27T09:31:57.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:56 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-05-27T09:31:57.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:56 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-05-27T09:31:57.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:56 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 6]}]: dispatch 2024-05-27T09:31:57.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:56 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-27T09:31:57.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:56 smithi189 ceph-mon[36671]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-27T09:31:57.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:56 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-05-27T09:31:57.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:56 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-05-27T09:31:57.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:56 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-05-27T09:31:57.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:56 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-05-27T09:31:57.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:56 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-05-27T09:31:57.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:56 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-05-27T09:31:57.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:56 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 6]}]: dispatch 2024-05-27T09:31:57.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:56 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-27T09:31:57.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.854+0000 7f7087c00700 1 -- 172.21.15.44:0/927947257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7080108730 msgr2=0x7f7080110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:57.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.854+0000 7f7087c00700 1 --2- 172.21.15.44:0/927947257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7080108730 0x7f7080110090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f707800b0a0 tx=0x7f707800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:31:57.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.854+0000 7f7087c00700 1 -- 172.21.15.44:0/927947257 shutdown_connections 2024-05-27T09:31:57.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.854+0000 7f7087c00700 1 --2- 172.21.15.44:0/927947257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7080108730 0x7f7080110090 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:57.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.854+0000 7f7087c00700 1 --2- 172.21.15.44:0/927947257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70801034b0 0x7f70801038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:57.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.854+0000 7f7087c00700 1 --2- 172.21.15.44:0/927947257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7080107de0 0x7f70801081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:57.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.854+0000 7f7087c00700 1 -- 172.21.15.44:0/927947257 >> 172.21.15.44:0/927947257 conn(0x7f70800fedb0 msgr2=0x7f70801011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:57.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.855+0000 7f7087c00700 1 -- 172.21.15.44:0/927947257 shutdown_connections 2024-05-27T09:31:57.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.855+0000 7f7087c00700 1 -- 172.21.15.44:0/927947257 wait complete. 2024-05-27T09:31:57.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.857+0000 7f7087c00700 1 Processor -- start 2024-05-27T09:31:57.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.857+0000 7f7087c00700 1 -- start start 2024-05-27T09:31:57.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.857+0000 7f7087c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70801034b0 0x7f70801999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:57.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.857+0000 7f7087c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7080107de0 0x7f7080199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:57.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.857+0000 7f7087c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7080108730 0x7f708019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:57.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.857+0000 7f7087c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7080113360 con 0x7f70801034b0 2024-05-27T09:31:57.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.857+0000 7f7087c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f70801131e0 con 0x7f7080107de0 2024-05-27T09:31:57.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.857+0000 7f7087c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f70801134e0 con 0x7f7080108730 2024-05-27T09:31:57.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f707fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7080107de0 0x7f7080199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:57.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f7085200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70801034b0 0x7f70801999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:57.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f707fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7080107de0 0x7f7080199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53784/0 (socket says 172.21.15.44:53784) 2024-05-27T09:31:57.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f707fe00700 1 -- 172.21.15.44:0/1959050595 learned_addr learned my addr 172.21.15.44:0/1959050595 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:31:57.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f7085200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70801034b0 0x7f70801999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56024/0 (socket says 172.21.15.44:56024) 2024-05-27T09:31:57.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f7085c00700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7080108730 0x7f708019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:57.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f707fe00700 1 -- 172.21.15.44:0/1959050595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7080108730 msgr2=0x7f708019a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:57.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f707fe00700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7080108730 0x7f708019a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:57.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f707fe00700 1 -- 172.21.15.44:0/1959050595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70801034b0 msgr2=0x7f70801999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:57.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f707fe00700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70801034b0 0x7f70801999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:57.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f707fe00700 1 -- 172.21.15.44:0/1959050595 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70801a90d0 con 0x7f7080107de0 2024-05-27T09:31:57.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f7085c00700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7080108730 0x7f708019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:31:57.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.858+0000 7f7085200700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70801034b0 0x7f70801999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:31:57.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.859+0000 7f707fe00700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7080107de0 0x7f7080199f10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f706c00e970 tx=0x7f706c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:57.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.862+0000 7f707d600700 1 -- 172.21.15.44:0/1959050595 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c00cbd0 con 0x7f7080107de0 2024-05-27T09:31:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.862+0000 7f7087c00700 1 -- 172.21.15.44:0/1959050595 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70801a93c0 con 0x7f7080107de0 2024-05-27T09:31:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.862+0000 7f707d600700 1 -- 172.21.15.44:0/1959050595 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f706c00cd70 con 0x7f7080107de0 2024-05-27T09:31:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.862+0000 7f707d600700 1 -- 172.21.15.44:0/1959050595 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c005680 con 0x7f7080107de0 2024-05-27T09:31:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.863+0000 7f7087c00700 1 -- 172.21.15.44:0/1959050595 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70801a9950 con 0x7f7080107de0 2024-05-27T09:31:57.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.865+0000 7f7087c00700 1 -- 172.21.15.44:0/1959050595 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7080104960 con 0x7f7080107de0 2024-05-27T09:31:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.869+0000 7f707d600700 1 -- 172.21.15.44:0/1959050595 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f706c004750 con 0x7f7080107de0 2024-05-27T09:31:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.872+0000 7f707d600700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f70680b2990 0x7f70680b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:31:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.872+0000 7f707d600700 1 -- 172.21.15.44:0/1959050595 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f706c0e2790 con 0x7f7080107de0 2024-05-27T09:31:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.872+0000 7f7085200700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f70680b2990 0x7f70680b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:31:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.873+0000 7f7085200700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f70680b2990 0x7f70680b4ea0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7074011490 tx=0x7f7074009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:31:57.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:57.878+0000 7f707d600700 1 -- 172.21.15.44:0/1959050595 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f706c054df0 con 0x7f7080107de0 2024-05-27T09:31:58.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.151+0000 7f7087c00700 1 -- 172.21.15.44:0/1959050595 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f7080063bf0 con 0x7f70680b2990 2024-05-27T09:31:58.153 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:57 smithi044 ceph-mon[31618]: pgmap v339: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-05-27T09:31:58.153 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-05-27T09:31:58.153 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-05-27T09:31:58.153 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-05-27T09:31:58.153 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-05-27T09:31:58.153 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-05-27T09:31:58.153 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-05-27T09:31:58.153 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 6]}]': finished 2024-05-27T09:31:58.153 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:57 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-27T09:31:58.153 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:57 smithi044 ceph-mon[31618]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T09:31:58.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.200+0000 7f707d600700 1 -- 172.21.15.44:0/1959050595 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7080063bf0 con 0x7f70680b2990 2024-05-27T09:31:58.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.204+0000 7f7087c00700 1 -- 172.21.15.44:0/1959050595 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f70680b2990 msgr2=0x7f70680b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:58.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.204+0000 7f7087c00700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f70680b2990 0x7f70680b4ea0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7074011490 tx=0x7f7074009260 comp rx=0 tx=0).stop 2024-05-27T09:31:58.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.204+0000 7f7087c00700 1 -- 172.21.15.44:0/1959050595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7080107de0 msgr2=0x7f7080199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:31:58.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.204+0000 7f7087c00700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7080107de0 0x7f7080199f10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f706c00e970 tx=0x7f706c00ec80 comp rx=0 tx=0).stop 2024-05-27T09:31:58.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.204+0000 7f7087c00700 1 -- 172.21.15.44:0/1959050595 shutdown_connections 2024-05-27T09:31:58.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.204+0000 7f7087c00700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f70680b2990 0x7f70680b4ea0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:58.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.204+0000 7f7087c00700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7080108730 0x7f708019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:58.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.204+0000 7f7087c00700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70801034b0 0x7f70801999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:58.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.204+0000 7f7087c00700 1 --2- 172.21.15.44:0/1959050595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7080107de0 0x7f7080199f10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:31:58.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.204+0000 7f7087c00700 1 -- 172.21.15.44:0/1959050595 >> 172.21.15.44:0/1959050595 conn(0x7f70800fedb0 msgr2=0x7f7080107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:31:58.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.205+0000 7f7087c00700 1 -- 172.21.15.44:0/1959050595 shutdown_connections 2024-05-27T09:31:58.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:31:58.205+0000 7f7087c00700 1 -- 172.21.15.44:0/1959050595 wait complete. 2024-05-27T09:31:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:57 smithi050 ceph-mon[36371]: pgmap v339: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-05-27T09:31:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:57 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-05-27T09:31:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:57 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-05-27T09:31:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:57 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-05-27T09:31:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:57 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-05-27T09:31:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:57 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-05-27T09:31:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:57 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-05-27T09:31:58.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:57 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 6]}]': finished 2024-05-27T09:31:58.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:57 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-27T09:31:58.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:57 smithi050 ceph-mon[36371]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T09:31:58.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:57 smithi189 ceph-mon[36671]: pgmap v339: 97 pgs: 97 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-05-27T09:31:58.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-05-27T09:31:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-05-27T09:31:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-05-27T09:31:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-05-27T09:31:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-05-27T09:31:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-05-27T09:31:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 6]}]': finished 2024-05-27T09:31:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:57 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-27T09:31:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:57 smithi189 ceph-mon[36671]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T09:31:59.010 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-27T09:31:59.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:58 smithi044 ceph-mon[31618]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T09:31:59.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:58 smithi050 ceph-mon[36371]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T09:31:59.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:58 smithi189 ceph-mon[36671]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T09:31:59.440 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:59 smithi044 ceph-mon[31618]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:31:59 smithi044 ceph-mon[31618]: pgmap v342: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 383 B/s wr, 5 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:59 smithi050 ceph-mon[36371]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:31:59 smithi050 ceph-mon[36371]: pgmap v342: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 383 B/s wr, 5 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:00.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:59 smithi189 ceph-mon[36671]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:00.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:31:59 smithi189 ceph-mon[36671]: pgmap v342: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 383 B/s wr, 5 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:01.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.509+0000 7f96d8400700 1 -- 172.21.15.44:0/1122504894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d0075b50 msgr2=0x7f96d0075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:01.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.509+0000 7f96d8400700 1 --2- 172.21.15.44:0/1122504894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d0075b50 0x7f96d0075f60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009990 tx=0x7f96c4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:32:01.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.510+0000 7f96d8400700 1 -- 172.21.15.44:0/1122504894 shutdown_connections 2024-05-27T09:32:01.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.510+0000 7f96d8400700 1 --2- 172.21.15.44:0/1122504894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d0115360 0x7f96d0117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:01.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.510+0000 7f96d8400700 1 --2- 172.21.15.44:0/1122504894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d0075b50 0x7f96d0075f60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:01.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.510+0000 7f96d8400700 1 --2- 172.21.15.44:0/1122504894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96d00764a0 0x7f96d0114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:01.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.510+0000 7f96d8400700 1 -- 172.21.15.44:0/1122504894 >> 172.21.15.44:0/1122504894 conn(0x7f96d0100030 msgr2=0x7f96d0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:01.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.510+0000 7f96d8400700 1 -- 172.21.15.44:0/1122504894 shutdown_connections 2024-05-27T09:32:01.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.510+0000 7f96d8400700 1 -- 172.21.15.44:0/1122504894 wait complete. 2024-05-27T09:32:01.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.511+0000 7f96d8400700 1 Processor -- start 2024-05-27T09:32:01.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.511+0000 7f96d8400700 1 -- start start 2024-05-27T09:32:01.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.512+0000 7f96d8400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d0075b50 0x7f96d01a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:01.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.512+0000 7f96d8400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00764a0 0x7f96d01a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:01.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.512+0000 7f96d8400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96d0115360 0x7f96d01a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:01.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.512+0000 7f96d8400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f96d011aa60 con 0x7f96d0115360 2024-05-27T09:32:01.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.512+0000 7f96d8400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f96d011a8e0 con 0x7f96d0075b50 2024-05-27T09:32:01.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.512+0000 7f96d8400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f96d011abe0 con 0x7f96d00764a0 2024-05-27T09:32:01.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.512+0000 7f96d5a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d0075b50 0x7f96d01a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:01.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.512+0000 7f96d5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00764a0 0x7f96d01a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:01.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.512+0000 7f96d5a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d0075b50 0x7f96d01a1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53800/0 (socket says 172.21.15.44:53800) 2024-05-27T09:32:01.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.512+0000 7f96d5a00700 1 -- 172.21.15.44:0/2831124351 learned_addr learned my addr 172.21.15.44:0/2831124351 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:01.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.512+0000 7f96d6400700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96d0115360 0x7f96d01a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:01.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.513+0000 7f96d5a00700 1 -- 172.21.15.44:0/2831124351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00764a0 msgr2=0x7f96d01a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:01.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.513+0000 7f96d5a00700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00764a0 0x7f96d01a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:01.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.513+0000 7f96d5a00700 1 -- 172.21.15.44:0/2831124351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96d0115360 msgr2=0x7f96d01a5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:01.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.513+0000 7f96d5a00700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96d0115360 0x7f96d01a5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:01.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.513+0000 7f96d5a00700 1 -- 172.21.15.44:0/2831124351 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d01a62e0 con 0x7f96d0075b50 2024-05-27T09:32:01.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.513+0000 7f96d5a00700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d0075b50 0x7f96d01a1410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f96c4000c00 tx=0x7f96c400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:01.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.513+0000 7f96ce000700 1 -- 172.21.15.44:0/2831124351 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c4027070 con 0x7f96d0075b50 2024-05-27T09:32:01.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.514+0000 7f96ce000700 1 -- 172.21.15.44:0/2831124351 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96c4022470 con 0x7f96d0075b50 2024-05-27T09:32:01.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.514+0000 7f96ce000700 1 -- 172.21.15.44:0/2831124351 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c4017650 con 0x7f96d0075b50 2024-05-27T09:32:01.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.514+0000 7f96d8400700 1 -- 172.21.15.44:0/2831124351 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d01a6570 con 0x7f96d0075b50 2024-05-27T09:32:01.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.514+0000 7f96d8400700 1 -- 172.21.15.44:0/2831124351 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d01a6a50 con 0x7f96d0075b50 2024-05-27T09:32:01.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.515+0000 7f96d8400700 1 -- 172.21.15.44:0/2831124351 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d00512b0 con 0x7f96d0075b50 2024-05-27T09:32:01.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.518+0000 7f96ce000700 1 -- 172.21.15.44:0/2831124351 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f96c402a070 con 0x7f96d0075b50 2024-05-27T09:32:01.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.519+0000 7f96ce000700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f96b80b2910 0x7f96b80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:01.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.519+0000 7f96ce000700 1 -- 172.21.15.44:0/2831124351 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f96c40e3030 con 0x7f96d0075b50 2024-05-27T09:32:01.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.520+0000 7f96d5000700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f96b80b2910 0x7f96b80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:01.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.520+0000 7f96d5000700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f96b80b2910 0x7f96b80b4e20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009680 tx=0x7f96bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:01.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.524+0000 7f96ce000700 1 -- 172.21.15.44:0/2831124351 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f96c4013070 con 0x7f96d0075b50 2024-05-27T09:32:01.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.797+0000 7f96d8400700 1 -- 172.21.15.44:0/2831124351 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f96d0063bf0 con 0x7f96b80b2910 2024-05-27T09:32:01.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.811+0000 7f96ce000700 1 -- 172.21.15.44:0/2831124351 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f96d0063bf0 con 0x7f96b80b2910 2024-05-27T09:32:01.812 INFO:teuthology.orchestra.run.smithi044.stdout:/volumes/qa/sv_0/a8464d66-8227-47f6-9472-37f047d14363 2024-05-27T09:32:01.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.814+0000 7f96d8400700 1 -- 172.21.15.44:0/2831124351 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f96b80b2910 msgr2=0x7f96b80b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:01.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.814+0000 7f96d8400700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f96b80b2910 0x7f96b80b4e20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009680 tx=0x7f96bc009260 comp rx=0 tx=0).stop 2024-05-27T09:32:01.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.814+0000 7f96d8400700 1 -- 172.21.15.44:0/2831124351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d0075b50 msgr2=0x7f96d01a1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:01.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.814+0000 7f96d8400700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d0075b50 0x7f96d01a1410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f96c4000c00 tx=0x7f96c400bfd0 comp rx=0 tx=0).stop 2024-05-27T09:32:01.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.814+0000 7f96d8400700 1 -- 172.21.15.44:0/2831124351 shutdown_connections 2024-05-27T09:32:01.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.814+0000 7f96d8400700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f96b80b2910 0x7f96b80b4e20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:01.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.814+0000 7f96d8400700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96d0115360 0x7f96d01a5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:01.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.814+0000 7f96d8400700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96d0075b50 0x7f96d01a1410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:01.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.814+0000 7f96d8400700 1 --2- 172.21.15.44:0/2831124351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00764a0 0x7f96d01a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:01.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.814+0000 7f96d8400700 1 -- 172.21.15.44:0/2831124351 >> 172.21.15.44:0/2831124351 conn(0x7f96d0100030 msgr2=0x7f96d0100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:01.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.815+0000 7f96d8400700 1 -- 172.21.15.44:0/2831124351 shutdown_connections 2024-05-27T09:32:01.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:01.815+0000 7f96d8400700 1 -- 172.21.15.44:0/2831124351 wait complete. 2024-05-27T09:32:02.172 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:01 smithi044 ceph-mon[31618]: pgmap v343: 97 pgs: 1 peering, 96 active+clean; 603 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 9 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:01 smithi050 ceph-mon[36371]: pgmap v343: 97 pgs: 1 peering, 96 active+clean; 603 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 9 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:02.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:01 smithi189 ceph-mon[36671]: pgmap v343: 97 pgs: 1 peering, 96 active+clean; 603 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 9 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:02.489 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-27T09:32:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:03.195 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:02 smithi050 ceph-mon[36371]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:03.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:02 smithi044 ceph-mon[31618]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:03.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:02 smithi189 ceph-mon[36671]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:03 smithi044 ceph-mon[31618]: pgmap v344: 97 pgs: 97 active+clean; 603 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.5 KiB/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:03 smithi050 ceph-mon[36371]: pgmap v344: 97 pgs: 97 active+clean; 603 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.5 KiB/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:04.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:03 smithi189 ceph-mon[36671]: pgmap v344: 97 pgs: 97 active+clean; 603 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.5 KiB/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.170+0000 7f929ac00700 1 -- 172.21.15.44:0/1129984379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9294106570 msgr2=0x7f9294106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.170+0000 7f929ac00700 1 --2- 172.21.15.44:0/1129984379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9294106570 0x7f9294106980 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9280009990 tx=0x7f9280009ca0 comp rx=0 tx=0).stop 2024-05-27T09:32:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.170+0000 7f929ac00700 1 -- 172.21.15.44:0/1129984379 shutdown_connections 2024-05-27T09:32:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.170+0000 7f929ac00700 1 --2- 172.21.15.44:0/1129984379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9294106ec0 0x7f9294113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.170+0000 7f929ac00700 1 --2- 172.21.15.44:0/1129984379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f929410c5b0 0x7f929410c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.170+0000 7f929ac00700 1 --2- 172.21.15.44:0/1129984379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9294106570 0x7f9294106980 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.170+0000 7f929ac00700 1 -- 172.21.15.44:0/1129984379 >> 172.21.15.44:0/1129984379 conn(0x7f9294102070 msgr2=0x7f92941044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.170+0000 7f929ac00700 1 -- 172.21.15.44:0/1129984379 shutdown_connections 2024-05-27T09:32:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.170+0000 7f929ac00700 1 -- 172.21.15.44:0/1129984379 wait complete. 2024-05-27T09:32:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.172+0000 7f929ac00700 1 Processor -- start 2024-05-27T09:32:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.172+0000 7f929ac00700 1 -- start start 2024-05-27T09:32:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.172+0000 7f929ac00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9294106570 0x7f9294110760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.172+0000 7f929ac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9294106ec0 0x7f9294110ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.172+0000 7f929ac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929410c5b0 0x7f92941111e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.172+0000 7f929ac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9294116690 con 0x7f929410c5b0 2024-05-27T09:32:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.172+0000 7f929ac00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9294116510 con 0x7f9294106570 2024-05-27T09:32:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.172+0000 7f929ac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9294116810 con 0x7f9294106ec0 2024-05-27T09:32:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.172+0000 7f9293e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9294106570 0x7f9294110760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.173+0000 7f9293e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9294106570 0x7f9294110760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38600/0 (socket says 172.21.15.44:38600) 2024-05-27T09:32:05.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.173+0000 7f9293e00700 1 -- 172.21.15.44:0/506843960 learned_addr learned my addr 172.21.15.44:0/506843960 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.173+0000 7f9298c00700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929410c5b0 0x7f92941111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.173+0000 7f9293400700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9294106ec0 0x7f9294110ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.173+0000 7f9293e00700 1 -- 172.21.15.44:0/506843960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9294106ec0 msgr2=0x7f9294110ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.173+0000 7f9293e00700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9294106ec0 0x7f9294110ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.173+0000 7f9293e00700 1 -- 172.21.15.44:0/506843960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929410c5b0 msgr2=0x7f92941111e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.173+0000 7f9293e00700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929410c5b0 0x7f92941111e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.173+0000 7f9293e00700 1 -- 172.21.15.44:0/506843960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92941ad560 con 0x7f9294106570 2024-05-27T09:32:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.173+0000 7f9293e00700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9294106570 0x7f9294110760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f928400ea60 tx=0x7f928400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.174+0000 7f9290c00700 1 -- 172.21.15.44:0/506843960 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928400cbd0 con 0x7f9294106570 2024-05-27T09:32:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.174+0000 7f9290c00700 1 -- 172.21.15.44:0/506843960 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92840044d0 con 0x7f9294106570 2024-05-27T09:32:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.174+0000 7f929ac00700 1 -- 172.21.15.44:0/506843960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92941ad850 con 0x7f9294106570 2024-05-27T09:32:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.174+0000 7f9290c00700 1 -- 172.21.15.44:0/506843960 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284010430 con 0x7f9294106570 2024-05-27T09:32:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.174+0000 7f929ac00700 1 -- 172.21.15.44:0/506843960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92941addb0 con 0x7f9294106570 2024-05-27T09:32:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.175+0000 7f929ac00700 1 -- 172.21.15.44:0/506843960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9294107c90 con 0x7f9294106570 2024-05-27T09:32:05.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.177+0000 7f9290c00700 1 -- 172.21.15.44:0/506843960 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9284003e70 con 0x7f9294106570 2024-05-27T09:32:05.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.179+0000 7f9290c00700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f92780b2990 0x7f92780b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:05.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.179+0000 7f9290c00700 1 -- 172.21.15.44:0/506843960 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f92840e24a0 con 0x7f9294106570 2024-05-27T09:32:05.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.179+0000 7f9293400700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f92780b2990 0x7f92780b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:05.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.180+0000 7f9293400700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f92780b2990 0x7f92780b4ea0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9280006010 tx=0x7f9280005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:05.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.182+0000 7f9290c00700 1 -- 172.21.15.44:0/506843960 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f92840aa010 con 0x7f9294106570 2024-05-27T09:32:05.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.452+0000 7f929ac00700 1 -- 172.21.15.44:0/506843960 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9294063bf0 con 0x7f92780b2990 2024-05-27T09:32:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.493+0000 7f9290c00700 1 -- 172.21.15.44:0/506843960 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9294063bf0 con 0x7f92780b2990 2024-05-27T09:32:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 -- 172.21.15.44:0/506843960 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f92780b2990 msgr2=0x7f92780b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f92780b2990 0x7f92780b4ea0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9280006010 tx=0x7f9280005960 comp rx=0 tx=0).stop 2024-05-27T09:32:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 -- 172.21.15.44:0/506843960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9294106570 msgr2=0x7f9294110760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9294106570 0x7f9294110760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f928400ea60 tx=0x7f928400ed70 comp rx=0 tx=0).stop 2024-05-27T09:32:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 -- 172.21.15.44:0/506843960 shutdown_connections 2024-05-27T09:32:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f92780b2990 0x7f92780b4ea0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f929410c5b0 0x7f92941111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9294106570 0x7f9294110760 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 --2- 172.21.15.44:0/506843960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9294106ec0 0x7f9294110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 -- 172.21.15.44:0/506843960 >> 172.21.15.44:0/506843960 conn(0x7f9294102070 msgr2=0x7f9294104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 -- 172.21.15.44:0/506843960 shutdown_connections 2024-05-27T09:32:05.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:05.497+0000 7f929ac00700 1 -- 172.21.15.44:0/506843960 wait complete. 2024-05-27T09:32:06.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:05 smithi044 ceph-mon[31618]: pgmap v345: 97 pgs: 97 active+clean; 608 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:06.280 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-27T09:32:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:05 smithi050 ceph-mon[36371]: pgmap v345: 97 pgs: 97 active+clean; 608 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:05 smithi189 ceph-mon[36671]: pgmap v345: 97 pgs: 97 active+clean; 608 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 72 B/s, 0 objects/s recovering 2024-05-27T09:32:06.743 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:07.202 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:06 smithi044 ceph-mon[31618]: from='client.24638 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:07.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:06 smithi050 ceph-mon[36371]: from='client.24638 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:06 smithi189 ceph-mon[36671]: from='client.24638 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:08.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:07 smithi044 ceph-mon[31618]: pgmap v346: 97 pgs: 97 active+clean; 608 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.6 KiB/s wr, 8 op/s; 59 B/s, 0 objects/s recovering 2024-05-27T09:32:08.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:07 smithi050 ceph-mon[36371]: pgmap v346: 97 pgs: 97 active+clean; 608 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.6 KiB/s wr, 8 op/s; 59 B/s, 0 objects/s recovering 2024-05-27T09:32:08.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:07 smithi189 ceph-mon[36671]: pgmap v346: 97 pgs: 97 active+clean; 608 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.6 KiB/s wr, 8 op/s; 59 B/s, 0 objects/s recovering 2024-05-27T09:32:08.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.951+0000 7fd8d6400700 1 -- 172.21.15.44:0/3968419525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d0105da0 msgr2=0x7fd8d010e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:08.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.951+0000 7fd8d6400700 1 --2- 172.21.15.44:0/3968419525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d0105da0 0x7fd8d010e2f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009990 tx=0x7fd8b8009ca0 comp rx=0 tx=0).stop 2024-05-27T09:32:08.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.951+0000 7fd8d6400700 1 -- 172.21.15.44:0/3968419525 shutdown_connections 2024-05-27T09:32:08.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.951+0000 7fd8d6400700 1 --2- 172.21.15.44:0/3968419525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d010e8c0 0x7fd8d0110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:08.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.951+0000 7fd8d6400700 1 --2- 172.21.15.44:0/3968419525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8d0105450 0x7fd8d0105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:08.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.951+0000 7fd8d6400700 1 --2- 172.21.15.44:0/3968419525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d0105da0 0x7fd8d010e2f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:08.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.951+0000 7fd8d6400700 1 -- 172.21.15.44:0/3968419525 >> 172.21.15.44:0/3968419525 conn(0x7fd8d00fedb0 msgr2=0x7fd8d01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:08.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.952+0000 7fd8d6400700 1 -- 172.21.15.44:0/3968419525 shutdown_connections 2024-05-27T09:32:08.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.952+0000 7fd8d6400700 1 -- 172.21.15.44:0/3968419525 wait complete. 2024-05-27T09:32:08.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.953+0000 7fd8d6400700 1 Processor -- start 2024-05-27T09:32:08.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.953+0000 7fd8d6400700 1 -- start start 2024-05-27T09:32:08.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.954+0000 7fd8d6400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d0105450 0x7fd8d019ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:08.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.954+0000 7fd8d6400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0105da0 0x7fd8d019b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:08.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.954+0000 7fd8d6400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8d010e8c0 0x7fd8d019f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:08.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.954+0000 7fd8d6400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d0114420 con 0x7fd8d010e8c0 2024-05-27T09:32:08.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.954+0000 7fd8d6400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d01142a0 con 0x7fd8d0105450 2024-05-27T09:32:08.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.954+0000 7fd8d6400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d01145a0 con 0x7fd8d0105da0 2024-05-27T09:32:08.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.954+0000 7fd8c6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0105da0 0x7fd8d019b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:08.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.954+0000 7fd8c6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0105da0 0x7fd8d019b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39966/0 (socket says 172.21.15.44:39966) 2024-05-27T09:32:08.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.954+0000 7fd8cfe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8d010e8c0 0x7fd8d019f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:08.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.954+0000 7fd8c6a00700 1 -- 172.21.15.44:0/1790047800 learned_addr learned my addr 172.21.15.44:0/1790047800 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:08.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.954+0000 7fd8cf400700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d0105450 0x7fd8d019ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:08.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.955+0000 7fd8c6a00700 1 -- 172.21.15.44:0/1790047800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d0105450 msgr2=0x7fd8d019ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:08.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.955+0000 7fd8c6a00700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d0105450 0x7fd8d019ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:08.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.955+0000 7fd8c6a00700 1 -- 172.21.15.44:0/1790047800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8d010e8c0 msgr2=0x7fd8d019f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:08.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.955+0000 7fd8c6a00700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8d010e8c0 0x7fd8d019f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:08.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.955+0000 7fd8c6a00700 1 -- 172.21.15.44:0/1790047800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d019fc60 con 0x7fd8d0105da0 2024-05-27T09:32:08.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.955+0000 7fd8c6a00700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0105da0 0x7fd8d019b330 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80057f0 tx=0x7fd8b800bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:08.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.955+0000 7fd8ccc00700 1 -- 172.21.15.44:0/1790047800 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8026070 con 0x7fd8d0105da0 2024-05-27T09:32:08.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.955+0000 7fd8ccc00700 1 -- 172.21.15.44:0/1790047800 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b8021410 con 0x7fd8d0105da0 2024-05-27T09:32:08.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.955+0000 7fd8ccc00700 1 -- 172.21.15.44:0/1790047800 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8017650 con 0x7fd8d0105da0 2024-05-27T09:32:08.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.955+0000 7fd8d6400700 1 -- 172.21.15.44:0/1790047800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d019fef0 con 0x7fd8d0105da0 2024-05-27T09:32:08.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.956+0000 7fd8d6400700 1 -- 172.21.15.44:0/1790047800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d0103070 con 0x7fd8d0105da0 2024-05-27T09:32:08.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.957+0000 7fd8d6400700 1 -- 172.21.15.44:0/1790047800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8d006bee0 con 0x7fd8d0105da0 2024-05-27T09:32:08.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.959+0000 7fd8ccc00700 1 -- 172.21.15.44:0/1790047800 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd8b8029070 con 0x7fd8d0105da0 2024-05-27T09:32:08.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.960+0000 7fd8ccc00700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8b40b2990 0x7fd8b40b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:08.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.961+0000 7fd8ccc00700 1 -- 172.21.15.44:0/1790047800 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fd8b80e31b0 con 0x7fd8d0105da0 2024-05-27T09:32:08.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.961+0000 7fd8cf400700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8b40b2990 0x7fd8b40b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:08.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.962+0000 7fd8cf400700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8b40b2990 0x7fd8b40b4ea0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0009680 tx=0x7fd8c0012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:08.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:08.963+0000 7fd8ccc00700 1 -- 172.21.15.44:0/1790047800 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd8b805edf0 con 0x7fd8d0105da0 2024-05-27T09:32:09.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.239+0000 7fd8d6400700 1 -- 172.21.15.44:0/1790047800 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fd8d019c0f0 con 0x7fd8b40b2990 2024-05-27T09:32:09.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.248+0000 7fd8ccc00700 1 -- 172.21.15.44:0/1790047800 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fd8d019c0f0 con 0x7fd8b40b2990 2024-05-27T09:32:09.250 INFO:teuthology.orchestra.run.smithi044.stdout:/volumes/qa/sv_1/11c31bb5-2e8d-444b-b829-f9cacdbbd223 2024-05-27T09:32:09.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 -- 172.21.15.44:0/1790047800 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8b40b2990 msgr2=0x7fd8b40b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8b40b2990 0x7fd8b40b4ea0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0009680 tx=0x7fd8c0012040 comp rx=0 tx=0).stop 2024-05-27T09:32:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 -- 172.21.15.44:0/1790047800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0105da0 msgr2=0x7fd8d019b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0105da0 0x7fd8d019b330 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80057f0 tx=0x7fd8b800bfd0 comp rx=0 tx=0).stop 2024-05-27T09:32:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 -- 172.21.15.44:0/1790047800 shutdown_connections 2024-05-27T09:32:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8b40b2990 0x7fd8b40b4ea0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8d010e8c0 0x7fd8d019f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:09.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d0105450 0x7fd8d019ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 --2- 172.21.15.44:0/1790047800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0105da0 0x7fd8d019b330 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 -- 172.21.15.44:0/1790047800 >> 172.21.15.44:0/1790047800 conn(0x7fd8d00fedb0 msgr2=0x7fd8d00ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 -- 172.21.15.44:0/1790047800 shutdown_connections 2024-05-27T09:32:09.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:09.251+0000 7fd8d6400700 1 -- 172.21.15.44:0/1790047800 wait complete. 2024-05-27T09:32:09.956 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:09 smithi044 ceph-mon[31618]: pgmap v347: 97 pgs: 97 active+clean; 608 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 53 B/s, 0 objects/s recovering 2024-05-27T09:32:10.036 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:32:10.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:09 smithi050 ceph-mon[36371]: pgmap v347: 97 pgs: 97 active+clean; 608 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 53 B/s, 0 objects/s recovering 2024-05-27T09:32:10.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:09 smithi189 ceph-mon[36671]: pgmap v347: 97 pgs: 97 active+clean; 608 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 53 B/s, 0 objects/s recovering 2024-05-27T09:32:10.493 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:11.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:10 smithi044 ceph-mon[31618]: from='client.24667 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:11.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:10 smithi050 ceph-mon[36371]: from='client.24667 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:11.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:10 smithi189 ceph-mon[36671]: from='client.24667 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:32:12.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:11 smithi044 ceph-mon[31618]: pgmap v348: 97 pgs: 97 active+clean; 631 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.9 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T09:32:12.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:11 smithi189 ceph-mon[36671]: pgmap v348: 97 pgs: 97 active+clean; 631 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.9 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T09:32:12.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:11 smithi050 ceph-mon[36371]: pgmap v348: 97 pgs: 97 active+clean; 631 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.9 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T09:32:12.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.865+0000 7f6964200700 1 -- 172.21.15.44:0/408297328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f695c10b2d0 msgr2=0x7f695c117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:12.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.865+0000 7f6964200700 1 --2- 172.21.15.44:0/408297328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f695c10b2d0 0x7f695c117800 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f695400b0a0 tx=0x7f695400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:32:12.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.866+0000 7f6964200700 1 -- 172.21.15.44:0/408297328 shutdown_connections 2024-05-27T09:32:12.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.866+0000 7f6964200700 1 --2- 172.21.15.44:0/408297328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f695c10b2d0 0x7f695c117800 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:12.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.866+0000 7f6964200700 1 --2- 172.21.15.44:0/408297328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f695c110960 0x7f695c110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:12.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.866+0000 7f6964200700 1 --2- 172.21.15.44:0/408297328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f695c10a980 0x7f695c10ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:12.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.866+0000 7f6964200700 1 -- 172.21.15.44:0/408297328 >> 172.21.15.44:0/408297328 conn(0x7f695c077c70 msgr2=0x7f695c0780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:12.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.866+0000 7f6964200700 1 -- 172.21.15.44:0/408297328 shutdown_connections 2024-05-27T09:32:12.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.866+0000 7f6964200700 1 -- 172.21.15.44:0/408297328 wait complete. 2024-05-27T09:32:12.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.868+0000 7f6964200700 1 Processor -- start 2024-05-27T09:32:12.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.868+0000 7f6964200700 1 -- start start 2024-05-27T09:32:12.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.869+0000 7f6964200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f695c10a980 0x7f695c1a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:12.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.869+0000 7f6964200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f695c10b2d0 0x7f695c1a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:12.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.869+0000 7f6964200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f695c110960 0x7f695c1a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:12.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.869+0000 7f6964200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f695c11aa80 con 0x7f695c10b2d0 2024-05-27T09:32:12.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.869+0000 7f6964200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f695c11a900 con 0x7f695c10a980 2024-05-27T09:32:12.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.869+0000 7f6964200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f695c11ac00 con 0x7f695c110960 2024-05-27T09:32:12.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.869+0000 7f6960e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f695c10b2d0 0x7f695c1a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:12.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.869+0000 7f6962200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f695c110960 0x7f695c1a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:12.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.869+0000 7f6962200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f695c110960 0x7f695c1a5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41344/0 (socket says 172.21.15.44:41344) 2024-05-27T09:32:12.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.869+0000 7f6962200700 1 -- 172.21.15.44:0/1162173558 learned_addr learned my addr 172.21.15.44:0/1162173558 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:12.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.869+0000 7f6961800700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f695c10a980 0x7f695c1a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:12.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.870+0000 7f6962200700 1 -- 172.21.15.44:0/1162173558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f695c10a980 msgr2=0x7f695c1a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:12.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.870+0000 7f6962200700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f695c10a980 0x7f695c1a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:12.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.870+0000 7f6962200700 1 -- 172.21.15.44:0/1162173558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f695c10b2d0 msgr2=0x7f695c1a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:12.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.870+0000 7f6962200700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f695c10b2d0 0x7f695c1a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:12.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.870+0000 7f6962200700 1 -- 172.21.15.44:0/1162173558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695c1a62a0 con 0x7f695c110960 2024-05-27T09:32:12.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.870+0000 7f6962200700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f695c110960 0x7f695c1a5c00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f695400b780 tx=0x7f6954004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:12.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.870+0000 7f695a000700 1 -- 172.21.15.44:0/1162173558 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6954021070 con 0x7f695c110960 2024-05-27T09:32:12.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.871+0000 7f695a000700 1 -- 172.21.15.44:0/1162173558 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6954007e30 con 0x7f695c110960 2024-05-27T09:32:12.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.871+0000 7f695a000700 1 -- 172.21.15.44:0/1162173558 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6954004700 con 0x7f695c110960 2024-05-27T09:32:12.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.871+0000 7f6964200700 1 -- 172.21.15.44:0/1162173558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f695c1a6530 con 0x7f695c110960 2024-05-27T09:32:12.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.871+0000 7f6964200700 1 -- 172.21.15.44:0/1162173558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f695c1a6a20 con 0x7f695c110960 2024-05-27T09:32:12.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.872+0000 7f6964200700 1 -- 172.21.15.44:0/1162173558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f695c0512b0 con 0x7f695c110960 2024-05-27T09:32:12.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.875+0000 7f695a000700 1 -- 172.21.15.44:0/1162173558 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f6954007980 con 0x7f695c110960 2024-05-27T09:32:12.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.876+0000 7f695a000700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f69440b2910 0x7f69440b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:12.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.876+0000 7f695a000700 1 -- 172.21.15.44:0/1162173558 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f69540e32f0 con 0x7f695c110960 2024-05-27T09:32:12.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.876+0000 7f6961800700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f69440b2910 0x7f69440b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:12.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.877+0000 7f6961800700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f69440b2910 0x7f69440b4e20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f695c07d0e0 tx=0x7f6950009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:12.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:12.881+0000 7f695a000700 1 -- 172.21.15.44:0/1162173558 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f69540aaef0 con 0x7f695c110960 2024-05-27T09:32:13.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.178+0000 7f6964200700 1 -- 172.21.15.44:0/1162173558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f695c02e5e0 con 0x7f695c110960 2024-05-27T09:32:13.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.180+0000 7f695a000700 1 -- 172.21.15.44:0/1162173558 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7f69540aa7f0 con 0x7f695c110960 2024-05-27T09:32:13.182 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:32:13.182 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:31:50.423700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:32:13.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.183+0000 7f6964200700 1 -- 172.21.15.44:0/1162173558 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f69440b2910 msgr2=0x7f69440b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:13.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.183+0000 7f6964200700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f69440b2910 0x7f69440b4e20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f695c07d0e0 tx=0x7f6950009260 comp rx=0 tx=0).stop 2024-05-27T09:32:13.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.183+0000 7f6964200700 1 -- 172.21.15.44:0/1162173558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f695c110960 msgr2=0x7f695c1a5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:13.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.183+0000 7f6964200700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f695c110960 0x7f695c1a5c00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f695400b780 tx=0x7f6954004d00 comp rx=0 tx=0).stop 2024-05-27T09:32:13.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.183+0000 7f6964200700 1 -- 172.21.15.44:0/1162173558 shutdown_connections 2024-05-27T09:32:13.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.183+0000 7f6964200700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f69440b2910 0x7f69440b4e20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:13.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.183+0000 7f6964200700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f695c110960 0x7f695c1a5c00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:13.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.183+0000 7f6964200700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f695c10a980 0x7f695c1a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:13.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.183+0000 7f6964200700 1 --2- 172.21.15.44:0/1162173558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f695c10b2d0 0x7f695c1a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:13.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.183+0000 7f6964200700 1 -- 172.21.15.44:0/1162173558 >> 172.21.15.44:0/1162173558 conn(0x7f695c077c70 msgr2=0x7f695c07b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:13.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.184+0000 7f6964200700 1 -- 172.21.15.44:0/1162173558 shutdown_connections 2024-05-27T09:32:13.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:13.184+0000 7f6964200700 1 -- 172.21.15.44:0/1162173558 wait complete. 2024-05-27T09:32:13.187 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-05-27T09:32:14.054 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T09:32:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:13 smithi044 ceph-mon[31618]: pgmap v349: 97 pgs: 97 active+clean; 631 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-05-27T09:32:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:13 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1162173558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:32:14.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:13 smithi050 ceph-mon[36371]: pgmap v349: 97 pgs: 97 active+clean; 631 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-05-27T09:32:14.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:13 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1162173558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:32:14.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:13 smithi189 ceph-mon[36671]: pgmap v349: 97 pgs: 97 active+clean; 631 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-05-27T09:32:14.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:13 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1162173558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:32:14.496 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:15.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:14 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:32:15.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:14 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:32:15.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:14 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:32:16.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:15 smithi044 ceph-mon[31618]: pgmap v350: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-05-27T09:32:16.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:15 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:15 smithi050 ceph-mon[36371]: pgmap v350: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-05-27T09:32:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:15 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:15 smithi189 ceph-mon[36671]: pgmap v350: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-05-27T09:32:16.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:15 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:16.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.804+0000 7f4a9ae00700 1 -- 172.21.15.44:0/774213731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a94110960 msgr2=0x7f4a94110d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:16.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.804+0000 7f4a9ae00700 1 --2- 172.21.15.44:0/774213731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a94110960 0x7f4a94110d70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a840098f0 tx=0x7f4a84009c00 comp rx=0 tx=0).stop 2024-05-27T09:32:16.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.804+0000 7f4a9ae00700 1 -- 172.21.15.44:0/774213731 shutdown_connections 2024-05-27T09:32:16.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.804+0000 7f4a9ae00700 1 --2- 172.21.15.44:0/774213731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a9410b2d0 0x7f4a94117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:16.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.804+0000 7f4a9ae00700 1 --2- 172.21.15.44:0/774213731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a94110960 0x7f4a94110d70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:16.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.804+0000 7f4a9ae00700 1 --2- 172.21.15.44:0/774213731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a9410a980 0x7f4a9410ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:16.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.804+0000 7f4a9ae00700 1 -- 172.21.15.44:0/774213731 >> 172.21.15.44:0/774213731 conn(0x7f4a94077c70 msgr2=0x7f4a940780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:16.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.804+0000 7f4a9ae00700 1 -- 172.21.15.44:0/774213731 shutdown_connections 2024-05-27T09:32:16.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.804+0000 7f4a9ae00700 1 -- 172.21.15.44:0/774213731 wait complete. 2024-05-27T09:32:16.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.805+0000 7f4a9ae00700 1 Processor -- start 2024-05-27T09:32:16.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.806+0000 7f4a9ae00700 1 -- start start 2024-05-27T09:32:16.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.806+0000 7f4a9ae00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a9410a980 0x7f4a941a14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:16.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.806+0000 7f4a9ae00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a9410b2d0 0x7f4a941a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:16.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.806+0000 7f4a9ae00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a94110960 0x7f4a941a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:16.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.806+0000 7f4a9ae00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9411aac0 con 0x7f4a9410b2d0 2024-05-27T09:32:16.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.806+0000 7f4a9ae00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9411a940 con 0x7f4a94110960 2024-05-27T09:32:16.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.806+0000 7f4a9ae00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9411ac40 con 0x7f4a9410a980 2024-05-27T09:32:16.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.807+0000 7f4a93e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a9410a980 0x7f4a941a14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:16.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.807+0000 7f4a93e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a9410a980 0x7f4a941a14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41364/0 (socket says 172.21.15.44:41364) 2024-05-27T09:32:16.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.807+0000 7f4a93e00700 1 -- 172.21.15.44:0/2322716764 learned_addr learned my addr 172.21.15.44:0/2322716764 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:16.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.807+0000 7f4a93400700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a9410b2d0 0x7f4a941a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:16.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.807+0000 7f4a98e00700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a94110960 0x7f4a941a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:16.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.807+0000 7f4a93e00700 1 -- 172.21.15.44:0/2322716764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a94110960 msgr2=0x7f4a941a5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:16.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.807+0000 7f4a93e00700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a94110960 0x7f4a941a5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:16.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.807+0000 7f4a93e00700 1 -- 172.21.15.44:0/2322716764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a9410b2d0 msgr2=0x7f4a941a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:16.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.807+0000 7f4a93e00700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a9410b2d0 0x7f4a941a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:16.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.807+0000 7f4a93e00700 1 -- 172.21.15.44:0/2322716764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a941a6380 con 0x7f4a9410a980 2024-05-27T09:32:16.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.808+0000 7f4a93e00700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a9410a980 0x7f4a941a14b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009230 tx=0x7f4a8400f990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:16.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.808+0000 7f4a90c00700 1 -- 172.21.15.44:0/2322716764 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a84027070 con 0x7f4a9410a980 2024-05-27T09:32:16.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.808+0000 7f4a90c00700 1 -- 172.21.15.44:0/2322716764 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a84005570 con 0x7f4a9410a980 2024-05-27T09:32:16.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.808+0000 7f4a9ae00700 1 -- 172.21.15.44:0/2322716764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a941a6610 con 0x7f4a9410a980 2024-05-27T09:32:16.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.808+0000 7f4a90c00700 1 -- 172.21.15.44:0/2322716764 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a84004930 con 0x7f4a9410a980 2024-05-27T09:32:16.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.808+0000 7f4a9ae00700 1 -- 172.21.15.44:0/2322716764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a941a6af0 con 0x7f4a9410a980 2024-05-27T09:32:16.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.809+0000 7f4a9ae00700 1 -- 172.21.15.44:0/2322716764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a940512b0 con 0x7f4a9410a980 2024-05-27T09:32:16.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.812+0000 7f4a90c00700 1 -- 172.21.15.44:0/2322716764 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4a8402a070 con 0x7f4a9410a980 2024-05-27T09:32:16.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.814+0000 7f4a90c00700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4a780b2910 0x7f4a780b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:16.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.814+0000 7f4a90c00700 1 -- 172.21.15.44:0/2322716764 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f4a840e2dd0 con 0x7f4a9410a980 2024-05-27T09:32:16.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.814+0000 7f4a93400700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4a780b2910 0x7f4a780b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:16.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.815+0000 7f4a93400700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4a780b2910 0x7f4a780b4e20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4a9407b130 tx=0x7f4a80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:16.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:16.818+0000 7f4a90c00700 1 -- 172.21.15.44:0/2322716764 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4a84013030 con 0x7f4a9410a980 2024-05-27T09:32:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.072+0000 7f4a9ae00700 1 -- 172.21.15.44:0/2322716764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a9402e5e0 con 0x7f4a9410a980 2024-05-27T09:32:17.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:16 smithi044 ceph-mon[31618]: pgmap v351: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-05-27T09:32:17.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.074+0000 7f4a90c00700 1 -- 172.21.15.44:0/2322716764 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22611 (secure 0 0 0) 0x7f4a840aa9d0 con 0x7f4a9410a980 2024-05-27T09:32:17.077 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:32:17.077 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":81,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:31:57.855852+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:32:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 -- 172.21.15.44:0/2322716764 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4a780b2910 msgr2=0x7f4a780b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4a780b2910 0x7f4a780b4e20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4a9407b130 tx=0x7f4a80009260 comp rx=0 tx=0).stop 2024-05-27T09:32:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 -- 172.21.15.44:0/2322716764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a9410a980 msgr2=0x7f4a941a14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a9410a980 0x7f4a941a14b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009230 tx=0x7f4a8400f990 comp rx=0 tx=0).stop 2024-05-27T09:32:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 -- 172.21.15.44:0/2322716764 shutdown_connections 2024-05-27T09:32:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4a780b2910 0x7f4a780b4e20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a94110960 0x7f4a941a5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a9410a980 0x7f4a941a14b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 --2- 172.21.15.44:0/2322716764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a9410b2d0 0x7f4a941a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 -- 172.21.15.44:0/2322716764 >> 172.21.15.44:0/2322716764 conn(0x7f4a94077c70 msgr2=0x7f4a9407bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 -- 172.21.15.44:0/2322716764 shutdown_connections 2024-05-27T09:32:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:17.078+0000 7f4a9ae00700 1 -- 172.21.15.44:0/2322716764 wait complete. 2024-05-27T09:32:17.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:16 smithi050 ceph-mon[36371]: pgmap v351: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-05-27T09:32:17.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:16 smithi189 ceph-mon[36671]: pgmap v351: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-05-27T09:32:18.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:17 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2322716764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:32:18.106 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-27T09:32:18.106 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T09:32:18.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:17 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2322716764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:32:18.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:17 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2322716764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:32:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:19.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:18 smithi050 ceph-mon[36371]: pgmap v352: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-05-27T09:32:19.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:18 smithi189 ceph-mon[36671]: pgmap v352: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-05-27T09:32:19.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:18 smithi044 ceph-mon[31618]: pgmap v352: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-05-27T09:32:20.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.857+0000 7f51b3800700 1 -- 172.21.15.44:0/805555848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ac1034b0 msgr2=0x7f51ac1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:20.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.857+0000 7f51b3800700 1 --2- 172.21.15.44:0/805555848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ac1034b0 0x7f51ac1038c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f51a00099c0 tx=0x7f51a0009cd0 comp rx=0 tx=0).stop 2024-05-27T09:32:20.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.857+0000 7f51b3800700 1 -- 172.21.15.44:0/805555848 shutdown_connections 2024-05-27T09:32:20.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.857+0000 7f51b3800700 1 --2- 172.21.15.44:0/805555848 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ac108730 0x7f51ac110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:20.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.857+0000 7f51b3800700 1 --2- 172.21.15.44:0/805555848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ac1034b0 0x7f51ac1038c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:20.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.857+0000 7f51b3800700 1 --2- 172.21.15.44:0/805555848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ac107de0 0x7f51ac1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:20.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.857+0000 7f51b3800700 1 -- 172.21.15.44:0/805555848 >> 172.21.15.44:0/805555848 conn(0x7f51ac0fede0 msgr2=0x7f51ac101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:20.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.857+0000 7f51b3800700 1 -- 172.21.15.44:0/805555848 shutdown_connections 2024-05-27T09:32:20.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.857+0000 7f51b3800700 1 -- 172.21.15.44:0/805555848 wait complete. 2024-05-27T09:32:20.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.858+0000 7f51b3800700 1 Processor -- start 2024-05-27T09:32:20.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.858+0000 7f51b3800700 1 -- start start 2024-05-27T09:32:20.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.859+0000 7f51b3800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ac1034b0 0x7f51ac19adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:20.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.859+0000 7f51b3800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ac107de0 0x7f51ac19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:20.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.859+0000 7f51b3800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ac108730 0x7f51ac19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:20.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.859+0000 7f51b3800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac1133b0 con 0x7f51ac1034b0 2024-05-27T09:32:20.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.859+0000 7f51b3800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac113230 con 0x7f51ac107de0 2024-05-27T09:32:20.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.859+0000 7f51b3800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac113530 con 0x7f51ac108730 2024-05-27T09:32:20.861 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:20 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:20.862 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:20 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:20.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.859+0000 7f51b1800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ac108730 0x7f51ac19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:20.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.860+0000 7f51b1800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ac108730 0x7f51ac19f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41374/0 (socket says 172.21.15.44:41374) 2024-05-27T09:32:20.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.860+0000 7f51b1800700 1 -- 172.21.15.44:0/3271849997 learned_addr learned my addr 172.21.15.44:0/3271849997 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.860+0000 7f51b0e00700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ac1034b0 0x7f51ac19adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.860+0000 7f51abe00700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ac107de0 0x7f51ac19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.860+0000 7f51b1800700 1 -- 172.21.15.44:0/3271849997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ac107de0 msgr2=0x7f51ac19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.860+0000 7f51b1800700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ac107de0 0x7f51ac19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.860+0000 7f51b1800700 1 -- 172.21.15.44:0/3271849997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ac1034b0 msgr2=0x7f51ac19adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.860+0000 7f51b1800700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ac1034b0 0x7f51ac19adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.860+0000 7f51b1800700 1 -- 172.21.15.44:0/3271849997 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ac19fcc0 con 0x7f51ac108730 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.861+0000 7f51b1800700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ac108730 0x7f51ac19f620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51a400ef30 tx=0x7f51a400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.861+0000 7f51a9600700 1 -- 172.21.15.44:0/3271849997 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a40092e0 con 0x7f51ac108730 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.861+0000 7f51b3800700 1 -- 172.21.15.44:0/3271849997 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ac19ff50 con 0x7f51ac108730 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.861+0000 7f51a9600700 1 -- 172.21.15.44:0/3271849997 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51a4004bb0 con 0x7f51ac108730 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.861+0000 7f51a9600700 1 -- 172.21.15.44:0/3271849997 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4004020 con 0x7f51ac108730 2024-05-27T09:32:20.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.861+0000 7f51b3800700 1 -- 172.21.15.44:0/3271849997 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ac1a78d0 con 0x7f51ac108730 2024-05-27T09:32:20.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.863+0000 7f51b3800700 1 -- 172.21.15.44:0/3271849997 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51ac0512b0 con 0x7f51ac108730 2024-05-27T09:32:20.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.865+0000 7f51a9600700 1 -- 172.21.15.44:0/3271849997 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f51a401b030 con 0x7f51ac108730 2024-05-27T09:32:20.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.866+0000 7f51a9600700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51940b2910 0x7f51940b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:20.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.867+0000 7f51a9600700 1 -- 172.21.15.44:0/3271849997 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f51a40e1e40 con 0x7f51ac108730 2024-05-27T09:32:20.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.867+0000 7f51b0e00700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51940b2910 0x7f51940b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:20.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.867+0000 7f51b0e00700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51940b2910 0x7f51940b4e20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f51a0000c00 tx=0x7f51a001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:20.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:20.871+0000 7f51a9600700 1 -- 172.21.15.44:0/3271849997 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f51a4022020 con 0x7f51ac108730 2024-05-27T09:32:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:20 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:20 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:21.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:20 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:21.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:20 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:21.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.121+0000 7f51b3800700 1 -- 172.21.15.44:0/3271849997 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f51ac19c150 con 0x7f51940b2910 2024-05-27T09:32:21.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.134+0000 7f51a9600700 1 -- 172.21.15.44:0/3271849997 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258727 (secure 0 0 0) 0x7f51a00672c0 con 0x7f51940b2910 2024-05-27T09:32:21.141 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:32:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 -- 172.21.15.44:0/3271849997 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51940b2910 msgr2=0x7f51940b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51940b2910 0x7f51940b4e20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f51a0000c00 tx=0x7f51a001a040 comp rx=0 tx=0).stop 2024-05-27T09:32:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 -- 172.21.15.44:0/3271849997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ac108730 msgr2=0x7f51ac19f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:21.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ac108730 0x7f51ac19f620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51a400ef30 tx=0x7f51a400c3a0 comp rx=0 tx=0).stop 2024-05-27T09:32:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 -- 172.21.15.44:0/3271849997 shutdown_connections 2024-05-27T09:32:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f51940b2910 0x7f51940b4e20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ac108730 0x7f51ac19f620 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ac1034b0 0x7f51ac19adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 --2- 172.21.15.44:0/3271849997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ac107de0 0x7f51ac19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 -- 172.21.15.44:0/3271849997 >> 172.21.15.44:0/3271849997 conn(0x7f51ac0fede0 msgr2=0x7f51ac075c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 -- 172.21.15.44:0/3271849997 shutdown_connections 2024-05-27T09:32:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:21.139+0000 7f51b3800700 1 -- 172.21.15.44:0/3271849997 wait complete. 2024-05-27T09:32:21.144 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-05-27T09:32:21.797 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:21 smithi044 ceph-mon[31618]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-05-27T09:32:21.798 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:32:21.798 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:32:21.798 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:21.854 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-05-27T09:32:20.643854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655341,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":304,"num_read_kb":251,"num_write":289,"num_write_kb":2263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":234,"ondisk_log_size":234,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378204,"kb_used_data":12392,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563620,"statfs":{"total":1151940427776,"available":1151553146880,"internally_reserved":0,"allocated":12689408,"data_stored":7136394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32381,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":79,"num_write":20,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005346"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861357+0000","last_change":"2024-05-27T09:31:24.899109+0000","last_active":"2024-05-27T09:31:57.861357+0000","last_peered":"2024-05-27T09:31:57.861357+0000","last_clean":"2024-05-27T09:31:57.861357+0000","last_became_active":"2024-05-27T09:31:24.898783+0000","last_became_peered":"2024-05-27T09:31:24.898783+0000","last_unstale":"2024-05-27T09:31:57.861357+0000","last_undegraded":"2024-05-27T09:31:57.861357+0000","last_fullsized":"2024-05-27T09:31:57.861357+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:01:12.220389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862630+0000","last_change":"2024-05-27T09:31:24.898382+0000","last_active":"2024-05-27T09:31:57.862630+0000","last_peered":"2024-05-27T09:31:57.862630+0000","last_clean":"2024-05-27T09:31:57.862630+0000","last_became_active":"2024-05-27T09:31:24.897975+0000","last_became_peered":"2024-05-27T09:31:24.897975+0000","last_unstale":"2024-05-27T09:31:57.862630+0000","last_undegraded":"2024-05-27T09:31:57.862630+0000","last_fullsized":"2024-05-27T09:31:57.862630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:38.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862172+0000","last_change":"2024-05-27T09:31:24.896122+0000","last_active":"2024-05-27T09:31:57.862172+0000","last_peered":"2024-05-27T09:31:57.862172+0000","last_clean":"2024-05-27T09:31:57.862172+0000","last_became_active":"2024-05-27T09:31:24.890867+0000","last_became_peered":"2024-05-27T09:31:24.890867+0000","last_unstale":"2024-05-27T09:31:57.862172+0000","last_undegraded":"2024-05-27T09:31:57.862172+0000","last_fullsized":"2024-05-27T09:31:57.862172+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:09.982177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.870065+0000","last_change":"2024-05-27T09:31:57.868900+0000","last_active":"2024-05-27T09:31:57.870065+0000","last_peered":"2024-05-27T09:31:57.870065+0000","last_clean":"2024-05-27T09:31:57.870065+0000","last_became_active":"2024-05-27T09:31:57.868418+0000","last_became_peered":"2024-05-27T09:31:57.868418+0000","last_unstale":"2024-05-27T09:31:57.870065+0000","last_undegraded":"2024-05-27T09:31:57.870065+0000","last_fullsized":"2024-05-27T09:31:57.870065+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:08.966704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864519+0000","last_change":"2024-05-27T09:31:24.910283+0000","last_active":"2024-05-27T09:31:57.864519+0000","last_peered":"2024-05-27T09:31:57.864519+0000","last_clean":"2024-05-27T09:31:57.864519+0000","last_became_active":"2024-05-27T09:31:24.909882+0000","last_became_peered":"2024-05-27T09:31:24.909882+0000","last_unstale":"2024-05-27T09:31:57.864519+0000","last_undegraded":"2024-05-27T09:31:57.864519+0000","last_fullsized":"2024-05-27T09:31:57.864519+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:43.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.860783+0000","last_change":"2024-05-27T09:31:24.900934+0000","last_active":"2024-05-27T09:31:57.860783+0000","last_peered":"2024-05-27T09:31:57.860783+0000","last_clean":"2024-05-27T09:31:57.860783+0000","last_became_active":"2024-05-27T09:31:24.900534+0000","last_became_peered":"2024-05-27T09:31:24.900534+0000","last_unstale":"2024-05-27T09:31:57.860783+0000","last_undegraded":"2024-05-27T09:31:57.860783+0000","last_fullsized":"2024-05-27T09:31:57.860783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:53.318327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863567+0000","last_change":"2024-05-27T09:31:24.907080+0000","last_active":"2024-05-27T09:31:57.863567+0000","last_peered":"2024-05-27T09:31:57.863567+0000","last_clean":"2024-05-27T09:31:57.863567+0000","last_became_active":"2024-05-27T09:31:24.906715+0000","last_became_peered":"2024-05-27T09:31:24.906715+0000","last_unstale":"2024-05-27T09:31:57.863567+0000","last_undegraded":"2024-05-27T09:31:57.863567+0000","last_fullsized":"2024-05-27T09:31:57.863567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:36.214659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.201871+0000","last_change":"2024-05-27T09:31:58.200608+0000","last_active":"2024-05-27T09:31:58.201871+0000","last_peered":"2024-05-27T09:31:58.201871+0000","last_clean":"2024-05-27T09:31:58.201871+0000","last_became_active":"2024-05-27T09:31:58.199363+0000","last_became_peered":"2024-05-27T09:31:58.199363+0000","last_unstale":"2024-05-27T09:31:58.201871+0000","last_undegraded":"2024-05-27T09:31:58.201871+0000","last_fullsized":"2024-05-27T09:31:58.201871+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:30:46.495930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864728+0000","last_change":"2024-05-27T09:31:24.892630+0000","last_active":"2024-05-27T09:31:57.864728+0000","last_peered":"2024-05-27T09:31:57.864728+0000","last_clean":"2024-05-27T09:31:57.864728+0000","last_became_active":"2024-05-27T09:31:24.892176+0000","last_became_peered":"2024-05-27T09:31:24.892176+0000","last_unstale":"2024-05-27T09:31:57.864728+0000","last_undegraded":"2024-05-27T09:31:57.864728+0000","last_fullsized":"2024-05-27T09:31:57.864728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:49:20.585601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861253+0000","last_change":"2024-05-27T09:31:24.895281+0000","last_active":"2024-05-27T09:31:57.861253+0000","last_peered":"2024-05-27T09:31:57.861253+0000","last_clean":"2024-05-27T09:31:57.861253+0000","last_became_active":"2024-05-27T09:31:24.894886+0000","last_became_peered":"2024-05-27T09:31:24.894886+0000","last_unstale":"2024-05-27T09:31:57.861253+0000","last_undegraded":"2024-05-27T09:31:57.861253+0000","last_fullsized":"2024-05-27T09:31:57.861253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:41.494434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864466+0000","last_change":"2024-05-27T09:31:24.900167+0000","last_active":"2024-05-27T09:31:57.864466+0000","last_peered":"2024-05-27T09:31:57.864466+0000","last_clean":"2024-05-27T09:31:57.864466+0000","last_became_active":"2024-05-27T09:31:24.899750+0000","last_became_peered":"2024-05-27T09:31:24.899750+0000","last_unstale":"2024-05-27T09:31:57.864466+0000","last_undegraded":"2024-05-27T09:31:57.864466+0000","last_fullsized":"2024-05-27T09:31:57.864466+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:30.602246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.869986+0000","last_change":"2024-05-27T09:31:57.868818+0000","last_active":"2024-05-27T09:31:57.869986+0000","last_peered":"2024-05-27T09:31:57.869986+0000","last_clean":"2024-05-27T09:31:57.869986+0000","last_became_active":"2024-05-27T09:31:57.868322+0000","last_became_peered":"2024-05-27T09:31:57.868322+0000","last_unstale":"2024-05-27T09:31:57.869986+0000","last_undegraded":"2024-05-27T09:31:57.869986+0000","last_fullsized":"2024-05-27T09:31:57.869986+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:35.304850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197377+0000","last_change":"2024-05-27T09:31:24.902537+0000","last_active":"2024-05-27T09:31:58.197377+0000","last_peered":"2024-05-27T09:31:58.197377+0000","last_clean":"2024-05-27T09:31:58.197377+0000","last_became_active":"2024-05-27T09:31:24.902132+0000","last_became_peered":"2024-05-27T09:31:24.902132+0000","last_unstale":"2024-05-27T09:31:58.197377+0000","last_undegraded":"2024-05-27T09:31:58.197377+0000","last_fullsized":"2024-05-27T09:31:58.197377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:20:50.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862152+0000","last_change":"2024-05-27T09:31:24.899182+0000","last_active":"2024-05-27T09:31:57.862152+0000","last_peered":"2024-05-27T09:31:57.862152+0000","last_clean":"2024-05-27T09:31:57.862152+0000","last_became_active":"2024-05-27T09:31:24.898749+0000","last_became_peered":"2024-05-27T09:31:24.898749+0000","last_unstale":"2024-05-27T09:31:57.862152+0000","last_undegraded":"2024-05-27T09:31:57.862152+0000","last_fullsized":"2024-05-27T09:31:57.862152+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:44:44.844357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862064+0000","last_change":"2024-05-27T09:31:24.901115+0000","last_active":"2024-05-27T09:31:57.862064+0000","last_peered":"2024-05-27T09:31:57.862064+0000","last_clean":"2024-05-27T09:31:57.862064+0000","last_became_active":"2024-05-27T09:31:24.897011+0000","last_became_peered":"2024-05-27T09:31:24.897011+0000","last_unstale":"2024-05-27T09:31:57.862064+0000","last_undegraded":"2024-05-27T09:31:57.862064+0000","last_fullsized":"2024-05-27T09:31:57.862064+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:01.130407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863461+0000","last_change":"2024-05-27T09:31:24.903063+0000","last_active":"2024-05-27T09:31:57.863461+0000","last_peered":"2024-05-27T09:31:57.863461+0000","last_clean":"2024-05-27T09:31:57.863461+0000","last_became_active":"2024-05-27T09:31:24.899766+0000","last_became_peered":"2024-05-27T09:31:24.899766+0000","last_unstale":"2024-05-27T09:31:57.863461+0000","last_undegraded":"2024-05-27T09:31:57.863461+0000","last_fullsized":"2024-05-27T09:31:57.863461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:45.984226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861506+0000","last_change":"2024-05-27T09:31:24.915922+0000","last_active":"2024-05-27T09:31:57.861506+0000","last_peered":"2024-05-27T09:31:57.861506+0000","last_clean":"2024-05-27T09:31:57.861506+0000","last_became_active":"2024-05-27T09:31:24.915623+0000","last_became_peered":"2024-05-27T09:31:24.915623+0000","last_unstale":"2024-05-27T09:31:57.861506+0000","last_undegraded":"2024-05-27T09:31:57.861506+0000","last_fullsized":"2024-05-27T09:31:57.861506+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:02:01.006704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864190+0000","last_change":"2024-05-27T09:31:24.908397+0000","last_active":"2024-05-27T09:31:57.864190+0000","last_peered":"2024-05-27T09:31:57.864190+0000","last_clean":"2024-05-27T09:31:57.864190+0000","last_became_active":"2024-05-27T09:31:24.903644+0000","last_became_peered":"2024-05-27T09:31:24.903644+0000","last_unstale":"2024-05-27T09:31:57.864190+0000","last_undegraded":"2024-05-27T09:31:57.864190+0000","last_fullsized":"2024-05-27T09:31:57.864190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:09.281429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.867417+0000","last_change":"2024-05-27T09:31:24.912706+0000","last_active":"2024-05-27T09:31:57.867417+0000","last_peered":"2024-05-27T09:31:57.867417+0000","last_clean":"2024-05-27T09:31:57.867417+0000","last_became_active":"2024-05-27T09:31:24.912357+0000","last_became_peered":"2024-05-27T09:31:24.912357+0000","last_unstale":"2024-05-27T09:31:57.867417+0000","last_undegraded":"2024-05-27T09:31:57.867417+0000","last_fullsized":"2024-05-27T09:31:57.867417+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:23.334041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864061+0000","last_change":"2024-05-27T09:31:20.881652+0000","last_active":"2024-05-27T09:31:57.864061+0000","last_peered":"2024-05-27T09:31:57.864061+0000","last_clean":"2024-05-27T09:31:57.864061+0000","last_became_active":"2024-05-27T09:31:20.880757+0000","last_became_peered":"2024-05-27T09:31:20.880757+0000","last_unstale":"2024-05-27T09:31:57.864061+0000","last_undegraded":"2024-05-27T09:31:57.864061+0000","last_fullsized":"2024-05-27T09:31:57.864061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:01.433755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.194126+0000","last_change":"2024-05-27T09:31:24.908010+0000","last_active":"2024-05-27T09:31:58.194126+0000","last_peered":"2024-05-27T09:31:58.194126+0000","last_clean":"2024-05-27T09:31:58.194126+0000","last_became_active":"2024-05-27T09:31:24.907638+0000","last_became_peered":"2024-05-27T09:31:24.907638+0000","last_unstale":"2024-05-27T09:31:58.194126+0000","last_undegraded":"2024-05-27T09:31:58.194126+0000","last_fullsized":"2024-05-27T09:31:58.194126+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:02:49.207709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.195199+0000","last_change":"2024-05-27T09:31:20.860556+0000","last_active":"2024-05-27T09:31:58.195199+0000","last_peered":"2024-05-27T09:31:58.195199+0000","last_clean":"2024-05-27T09:31:58.195199+0000","last_became_active":"2024-05-27T09:31:20.860172+0000","last_became_peered":"2024-05-27T09:31:20.860172+0000","last_unstale":"2024-05-27T09:31:58.195199+0000","last_undegraded":"2024-05-27T09:31:58.195199+0000","last_fullsized":"2024-05-27T09:31:58.195199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:08:21.545681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862841+0000","last_change":"2024-05-27T09:31:24.910111+0000","last_active":"2024-05-27T09:31:57.862841+0000","last_peered":"2024-05-27T09:31:57.862841+0000","last_clean":"2024-05-27T09:31:57.862841+0000","last_became_active":"2024-05-27T09:31:24.908985+0000","last_became_peered":"2024-05-27T09:31:24.908985+0000","last_unstale":"2024-05-27T09:31:57.862841+0000","last_undegraded":"2024-05-27T09:31:57.862841+0000","last_fullsized":"2024-05-27T09:31:57.862841+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:53:29.645792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864702+0000","last_change":"2024-05-27T09:31:20.879993+0000","last_active":"2024-05-27T09:31:57.864702+0000","last_peered":"2024-05-27T09:31:57.864702+0000","last_clean":"2024-05-27T09:31:57.864702+0000","last_became_active":"2024-05-27T09:31:20.878802+0000","last_became_peered":"2024-05-27T09:31:20.878802+0000","last_unstale":"2024-05-27T09:31:57.864702+0000","last_undegraded":"2024-05-27T09:31:57.864702+0000","last_fullsized":"2024-05-27T09:31:57.864702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:15.555612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864431+0000","last_change":"2024-05-27T09:31:24.903533+0000","last_active":"2024-05-27T09:31:57.864431+0000","last_peered":"2024-05-27T09:31:57.864431+0000","last_clean":"2024-05-27T09:31:57.864431+0000","last_became_active":"2024-05-27T09:31:24.903166+0000","last_became_peered":"2024-05-27T09:31:24.903166+0000","last_unstale":"2024-05-27T09:31:57.864431+0000","last_undegraded":"2024-05-27T09:31:57.864431+0000","last_fullsized":"2024-05-27T09:31:57.864431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:52.539470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.878845+0000","last_change":"2024-05-27T09:31:57.878426+0000","last_active":"2024-05-27T09:31:57.878845+0000","last_peered":"2024-05-27T09:31:57.878845+0000","last_clean":"2024-05-27T09:31:57.878845+0000","last_became_active":"2024-05-27T09:31:57.869146+0000","last_became_peered":"2024-05-27T09:31:57.869146+0000","last_unstale":"2024-05-27T09:31:57.878845+0000","last_undegraded":"2024-05-27T09:31:57.878845+0000","last_fullsized":"2024-05-27T09:31:57.878845+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:34:51.926092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864352+0000","last_change":"2024-05-27T09:31:24.897426+0000","last_active":"2024-05-27T09:31:57.864352+0000","last_peered":"2024-05-27T09:31:57.864352+0000","last_clean":"2024-05-27T09:31:57.864352+0000","last_became_active":"2024-05-27T09:31:24.897080+0000","last_became_peered":"2024-05-27T09:31:24.897080+0000","last_unstale":"2024-05-27T09:31:57.864352+0000","last_undegraded":"2024-05-27T09:31:57.864352+0000","last_fullsized":"2024-05-27T09:31:57.864352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:15:42.428391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":66,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:15.833407+0000","last_change":"2024-05-27T09:31:20.846681+0000","last_active":"2024-05-27T09:32:15.833407+0000","last_peered":"2024-05-27T09:32:15.833407+0000","last_clean":"2024-05-27T09:32:15.833407+0000","last_became_active":"2024-05-27T09:31:20.846302+0000","last_became_peered":"2024-05-27T09:31:20.846302+0000","last_unstale":"2024-05-27T09:32:15.833407+0000","last_undegraded":"2024-05-27T09:32:15.833407+0000","last_fullsized":"2024-05-27T09:32:15.833407+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:19.609612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863508+0000","last_change":"2024-05-27T09:31:24.907209+0000","last_active":"2024-05-27T09:31:57.863508+0000","last_peered":"2024-05-27T09:31:57.863508+0000","last_clean":"2024-05-27T09:31:57.863508+0000","last_became_active":"2024-05-27T09:31:24.906752+0000","last_became_peered":"2024-05-27T09:31:24.906752+0000","last_unstale":"2024-05-27T09:31:57.863508+0000","last_undegraded":"2024-05-27T09:31:57.863508+0000","last_fullsized":"2024-05-27T09:31:57.863508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:53.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.860743+0000","last_change":"2024-05-27T09:31:20.879778+0000","last_active":"2024-05-27T09:31:57.860743+0000","last_peered":"2024-05-27T09:31:57.860743+0000","last_clean":"2024-05-27T09:31:57.860743+0000","last_became_active":"2024-05-27T09:31:20.879259+0000","last_became_peered":"2024-05-27T09:31:20.879259+0000","last_unstale":"2024-05-27T09:31:57.860743+0000","last_undegraded":"2024-05-27T09:31:57.860743+0000","last_fullsized":"2024-05-27T09:31:57.860743+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:20:07.972653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.195516+0000","last_change":"2024-05-27T09:31:58.195516+0000","last_active":"2024-05-27T09:31:58.195516+0000","last_peered":"2024-05-27T09:31:58.195516+0000","last_clean":"2024-05-27T09:31:58.195516+0000","last_became_active":"2024-05-27T09:31:58.195118+0000","last_became_peered":"2024-05-27T09:31:58.195118+0000","last_unstale":"2024-05-27T09:31:58.195516+0000","last_undegraded":"2024-05-27T09:31:58.195516+0000","last_fullsized":"2024-05-27T09:31:58.195516+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:11:23.547605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.196330+0000","last_change":"2024-05-27T09:31:20.863152+0000","last_active":"2024-05-27T09:31:58.196330+0000","last_peered":"2024-05-27T09:31:58.196330+0000","last_clean":"2024-05-27T09:31:58.196330+0000","last_became_active":"2024-05-27T09:31:20.862708+0000","last_became_peered":"2024-05-27T09:31:20.862708+0000","last_unstale":"2024-05-27T09:31:58.196330+0000","last_undegraded":"2024-05-27T09:31:58.196330+0000","last_fullsized":"2024-05-27T09:31:58.196330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:21.645773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.192762+0000","last_change":"2024-05-27T09:31:24.893988+0000","last_active":"2024-05-27T09:31:58.192762+0000","last_peered":"2024-05-27T09:31:58.192762+0000","last_clean":"2024-05-27T09:31:58.192762+0000","last_became_active":"2024-05-27T09:31:24.893504+0000","last_became_peered":"2024-05-27T09:31:24.893504+0000","last_unstale":"2024-05-27T09:31:58.192762+0000","last_undegraded":"2024-05-27T09:31:58.192762+0000","last_fullsized":"2024-05-27T09:31:58.192762+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:47.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863900+0000","last_change":"2024-05-27T09:31:20.845555+0000","last_active":"2024-05-27T09:31:57.863900+0000","last_peered":"2024-05-27T09:31:57.863900+0000","last_clean":"2024-05-27T09:31:57.863900+0000","last_became_active":"2024-05-27T09:31:20.845139+0000","last_became_peered":"2024-05-27T09:31:20.845139+0000","last_unstale":"2024-05-27T09:31:57.863900+0000","last_undegraded":"2024-05-27T09:31:57.863900+0000","last_fullsized":"2024-05-27T09:31:57.863900+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:25.917845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863254+0000","last_change":"2024-05-27T09:31:24.916142+0000","last_active":"2024-05-27T09:31:57.863254+0000","last_peered":"2024-05-27T09:31:57.863254+0000","last_clean":"2024-05-27T09:31:57.863254+0000","last_became_active":"2024-05-27T09:31:24.915709+0000","last_became_peered":"2024-05-27T09:31:24.915709+0000","last_unstale":"2024-05-27T09:31:57.863254+0000","last_undegraded":"2024-05-27T09:31:57.863254+0000","last_fullsized":"2024-05-27T09:31:57.863254+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:38.029648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'3","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:17.835685+0000","last_change":"2024-05-27T09:31:20.859240+0000","last_active":"2024-05-27T09:32:17.835685+0000","last_peered":"2024-05-27T09:32:17.835685+0000","last_clean":"2024-05-27T09:32:17.835685+0000","last_became_active":"2024-05-27T09:31:20.858798+0000","last_became_peered":"2024-05-27T09:31:20.858798+0000","last_unstale":"2024-05-27T09:32:17.835685+0000","last_undegraded":"2024-05-27T09:32:17.835685+0000","last_fullsized":"2024-05-27T09:32:17.835685+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:42.280165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863938+0000","last_change":"2024-05-27T09:31:24.908818+0000","last_active":"2024-05-27T09:31:57.863938+0000","last_peered":"2024-05-27T09:31:57.863938+0000","last_clean":"2024-05-27T09:31:57.863938+0000","last_became_active":"2024-05-27T09:31:24.908410+0000","last_became_peered":"2024-05-27T09:31:24.908410+0000","last_unstale":"2024-05-27T09:31:57.863938+0000","last_undegraded":"2024-05-27T09:31:57.863938+0000","last_fullsized":"2024-05-27T09:31:57.863938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:05:40.326232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864270+0000","last_change":"2024-05-27T09:31:20.848632+0000","last_active":"2024-05-27T09:31:57.864270+0000","last_peered":"2024-05-27T09:31:57.864270+0000","last_clean":"2024-05-27T09:31:57.864270+0000","last_became_active":"2024-05-27T09:31:20.847558+0000","last_became_peered":"2024-05-27T09:31:20.847558+0000","last_unstale":"2024-05-27T09:31:57.864270+0000","last_undegraded":"2024-05-27T09:31:57.864270+0000","last_fullsized":"2024-05-27T09:31:57.864270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:53.930565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862862+0000","last_change":"2024-05-27T09:31:24.901513+0000","last_active":"2024-05-27T09:31:57.862862+0000","last_peered":"2024-05-27T09:31:57.862862+0000","last_clean":"2024-05-27T09:31:57.862862+0000","last_became_active":"2024-05-27T09:31:24.895924+0000","last_became_peered":"2024-05-27T09:31:24.895924+0000","last_unstale":"2024-05-27T09:31:57.862862+0000","last_undegraded":"2024-05-27T09:31:57.862862+0000","last_fullsized":"2024-05-27T09:31:57.862862+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:27:39.939098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864463+0000","last_change":"2024-05-27T09:31:20.855195+0000","last_active":"2024-05-27T09:31:57.864463+0000","last_peered":"2024-05-27T09:31:57.864463+0000","last_clean":"2024-05-27T09:31:57.864463+0000","last_became_active":"2024-05-27T09:31:20.854910+0000","last_became_peered":"2024-05-27T09:31:20.854910+0000","last_unstale":"2024-05-27T09:31:57.864463+0000","last_undegraded":"2024-05-27T09:31:57.864463+0000","last_fullsized":"2024-05-27T09:31:57.864463+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:02.795103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865129+0000","last_change":"2024-05-27T09:31:24.895887+0000","last_active":"2024-05-27T09:31:57.865129+0000","last_peered":"2024-05-27T09:31:57.865129+0000","last_clean":"2024-05-27T09:31:57.865129+0000","last_became_active":"2024-05-27T09:31:24.894622+0000","last_became_peered":"2024-05-27T09:31:24.894622+0000","last_unstale":"2024-05-27T09:31:57.865129+0000","last_undegraded":"2024-05-27T09:31:57.865129+0000","last_fullsized":"2024-05-27T09:31:57.865129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:32.833946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'29","reported_seq":129,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:18.844385+0000","last_change":"2024-05-27T09:31:20.846063+0000","last_active":"2024-05-27T09:32:18.844385+0000","last_peered":"2024-05-27T09:32:18.844385+0000","last_clean":"2024-05-27T09:32:18.844385+0000","last_became_active":"2024-05-27T09:31:20.845401+0000","last_became_peered":"2024-05-27T09:31:20.845401+0000","last_unstale":"2024-05-27T09:32:18.844385+0000","last_undegraded":"2024-05-27T09:32:18.844385+0000","last_fullsized":"2024-05-27T09:32:18.844385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:21:38.460183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60347,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":74,"num_read_kb":96,"num_write":30,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864414+0000","last_change":"2024-05-27T09:31:24.892748+0000","last_active":"2024-05-27T09:31:57.864414+0000","last_peered":"2024-05-27T09:31:57.864414+0000","last_clean":"2024-05-27T09:31:57.864414+0000","last_became_active":"2024-05-27T09:31:24.892394+0000","last_became_peered":"2024-05-27T09:31:24.892394+0000","last_unstale":"2024-05-27T09:31:57.864414+0000","last_undegraded":"2024-05-27T09:31:57.864414+0000","last_fullsized":"2024-05-27T09:31:57.864414+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:39:31.381072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.867076+0000","last_change":"2024-05-27T09:31:20.860789+0000","last_active":"2024-05-27T09:31:57.867076+0000","last_peered":"2024-05-27T09:31:57.867076+0000","last_clean":"2024-05-27T09:31:57.867076+0000","last_became_active":"2024-05-27T09:31:20.860369+0000","last_became_peered":"2024-05-27T09:31:20.860369+0000","last_unstale":"2024-05-27T09:31:57.867076+0000","last_undegraded":"2024-05-27T09:31:57.867076+0000","last_fullsized":"2024-05-27T09:31:57.867076+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:05:35.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.196372+0000","last_change":"2024-05-27T09:31:24.918239+0000","last_active":"2024-05-27T09:31:58.196372+0000","last_peered":"2024-05-27T09:31:58.196372+0000","last_clean":"2024-05-27T09:31:58.196372+0000","last_became_active":"2024-05-27T09:31:24.917750+0000","last_became_peered":"2024-05-27T09:31:24.917750+0000","last_unstale":"2024-05-27T09:31:58.196372+0000","last_undegraded":"2024-05-27T09:31:58.196372+0000","last_fullsized":"2024-05-27T09:31:58.196372+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:07.162994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863930+0000","last_change":"2024-05-27T09:31:20.865308+0000","last_active":"2024-05-27T09:31:57.863930+0000","last_peered":"2024-05-27T09:31:57.863930+0000","last_clean":"2024-05-27T09:31:57.863930+0000","last_became_active":"2024-05-27T09:31:20.864920+0000","last_became_peered":"2024-05-27T09:31:20.864920+0000","last_unstale":"2024-05-27T09:31:57.863930+0000","last_undegraded":"2024-05-27T09:31:57.863930+0000","last_fullsized":"2024-05-27T09:31:57.863930+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:36.773322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863675+0000","last_change":"2024-05-27T09:31:24.902533+0000","last_active":"2024-05-27T09:31:57.863675+0000","last_peered":"2024-05-27T09:31:57.863675+0000","last_clean":"2024-05-27T09:31:57.863675+0000","last_became_active":"2024-05-27T09:31:24.902136+0000","last_became_peered":"2024-05-27T09:31:24.902136+0000","last_unstale":"2024-05-27T09:31:57.863675+0000","last_undegraded":"2024-05-27T09:31:57.863675+0000","last_fullsized":"2024-05-27T09:31:57.863675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:49:32.203577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'4","reported_seq":65,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:16.845266+0000","last_change":"2024-05-27T09:31:20.846060+0000","last_active":"2024-05-27T09:32:16.845266+0000","last_peered":"2024-05-27T09:32:16.845266+0000","last_clean":"2024-05-27T09:32:16.845266+0000","last_became_active":"2024-05-27T09:31:20.845632+0000","last_became_peered":"2024-05-27T09:31:20.845632+0000","last_unstale":"2024-05-27T09:32:16.845266+0000","last_undegraded":"2024-05-27T09:32:16.845266+0000","last_fullsized":"2024-05-27T09:32:16.845266+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:55.166579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863415+0000","last_change":"2024-05-27T09:31:24.895866+0000","last_active":"2024-05-27T09:31:57.863415+0000","last_peered":"2024-05-27T09:31:57.863415+0000","last_clean":"2024-05-27T09:31:57.863415+0000","last_became_active":"2024-05-27T09:31:24.895413+0000","last_became_peered":"2024-05-27T09:31:24.895413+0000","last_unstale":"2024-05-27T09:31:57.863415+0000","last_undegraded":"2024-05-27T09:31:57.863415+0000","last_fullsized":"2024-05-27T09:31:57.863415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:01:30.075710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863490+0000","last_change":"2024-05-27T09:31:20.879086+0000","last_active":"2024-05-27T09:31:57.863490+0000","last_peered":"2024-05-27T09:31:57.863490+0000","last_clean":"2024-05-27T09:31:57.863490+0000","last_became_active":"2024-05-27T09:31:20.878331+0000","last_became_peered":"2024-05-27T09:31:20.878331+0000","last_unstale":"2024-05-27T09:31:57.863490+0000","last_undegraded":"2024-05-27T09:31:57.863490+0000","last_fullsized":"2024-05-27T09:31:57.863490+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:31.169812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863697+0000","last_change":"2024-05-27T09:31:24.895069+0000","last_active":"2024-05-27T09:31:57.863697+0000","last_peered":"2024-05-27T09:31:57.863697+0000","last_clean":"2024-05-27T09:31:57.863697+0000","last_became_active":"2024-05-27T09:31:24.894626+0000","last_became_peered":"2024-05-27T09:31:24.894626+0000","last_unstale":"2024-05-27T09:31:57.863697+0000","last_undegraded":"2024-05-27T09:31:57.863697+0000","last_fullsized":"2024-05-27T09:31:57.863697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:21:22.725268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864050+0000","last_change":"2024-05-27T09:31:20.877416+0000","last_active":"2024-05-27T09:31:57.864050+0000","last_peered":"2024-05-27T09:31:57.864050+0000","last_clean":"2024-05-27T09:31:57.864050+0000","last_became_active":"2024-05-27T09:31:20.876093+0000","last_became_peered":"2024-05-27T09:31:20.876093+0000","last_unstale":"2024-05-27T09:31:57.864050+0000","last_undegraded":"2024-05-27T09:31:57.864050+0000","last_fullsized":"2024-05-27T09:31:57.864050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:34.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861332+0000","last_change":"2024-05-27T09:31:24.898430+0000","last_active":"2024-05-27T09:31:57.861332+0000","last_peered":"2024-05-27T09:31:57.861332+0000","last_clean":"2024-05-27T09:31:57.861332+0000","last_became_active":"2024-05-27T09:31:24.897643+0000","last_became_peered":"2024-05-27T09:31:24.897643+0000","last_unstale":"2024-05-27T09:31:57.861332+0000","last_undegraded":"2024-05-27T09:31:57.861332+0000","last_fullsized":"2024-05-27T09:31:57.861332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:44.485561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863733+0000","last_change":"2024-05-27T09:31:20.858253+0000","last_active":"2024-05-27T09:31:57.863733+0000","last_peered":"2024-05-27T09:31:57.863733+0000","last_clean":"2024-05-27T09:31:57.863733+0000","last_became_active":"2024-05-27T09:31:20.857736+0000","last_became_peered":"2024-05-27T09:31:20.857736+0000","last_unstale":"2024-05-27T09:31:57.863733+0000","last_undegraded":"2024-05-27T09:31:57.863733+0000","last_fullsized":"2024-05-27T09:31:57.863733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:53:48.301763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861178+0000","last_change":"2024-05-27T09:31:24.912788+0000","last_active":"2024-05-27T09:31:57.861178+0000","last_peered":"2024-05-27T09:31:57.861178+0000","last_clean":"2024-05-27T09:31:57.861178+0000","last_became_active":"2024-05-27T09:31:24.912366+0000","last_became_peered":"2024-05-27T09:31:24.912366+0000","last_unstale":"2024-05-27T09:31:57.861178+0000","last_undegraded":"2024-05-27T09:31:57.861178+0000","last_fullsized":"2024-05-27T09:31:57.861178+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:55:03.397578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863841+0000","last_change":"2024-05-27T09:31:20.862188+0000","last_active":"2024-05-27T09:31:57.863841+0000","last_peered":"2024-05-27T09:31:57.863841+0000","last_clean":"2024-05-27T09:31:57.863841+0000","last_became_active":"2024-05-27T09:31:20.861791+0000","last_became_peered":"2024-05-27T09:31:20.861791+0000","last_unstale":"2024-05-27T09:31:57.863841+0000","last_undegraded":"2024-05-27T09:31:57.863841+0000","last_fullsized":"2024-05-27T09:31:57.863841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:16:09.127223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861144+0000","last_change":"2024-05-27T09:31:24.898460+0000","last_active":"2024-05-27T09:31:57.861144+0000","last_peered":"2024-05-27T09:31:57.861144+0000","last_clean":"2024-05-27T09:31:57.861144+0000","last_became_active":"2024-05-27T09:31:24.896787+0000","last_became_peered":"2024-05-27T09:31:24.896787+0000","last_unstale":"2024-05-27T09:31:57.861144+0000","last_undegraded":"2024-05-27T09:31:57.861144+0000","last_fullsized":"2024-05-27T09:31:57.861144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:10.199227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864271+0000","last_change":"2024-05-27T09:31:24.896323+0000","last_active":"2024-05-27T09:31:57.864271+0000","last_peered":"2024-05-27T09:31:57.864271+0000","last_clean":"2024-05-27T09:31:57.864271+0000","last_became_active":"2024-05-27T09:31:24.895919+0000","last_became_peered":"2024-05-27T09:31:24.895919+0000","last_unstale":"2024-05-27T09:31:57.864271+0000","last_undegraded":"2024-05-27T09:31:57.864271+0000","last_fullsized":"2024-05-27T09:31:57.864271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:04.883166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'4","reported_seq":64,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:18.848451+0000","last_change":"2024-05-27T09:31:20.862126+0000","last_active":"2024-05-27T09:32:18.848451+0000","last_peered":"2024-05-27T09:32:18.848451+0000","last_clean":"2024-05-27T09:32:18.848451+0000","last_became_active":"2024-05-27T09:31:20.858964+0000","last_became_peered":"2024-05-27T09:31:20.858964+0000","last_unstale":"2024-05-27T09:32:18.848451+0000","last_undegraded":"2024-05-27T09:32:18.848451+0000","last_fullsized":"2024-05-27T09:32:18.848451+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:51:26.838559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.193948+0000","last_change":"2024-05-27T09:31:24.905549+0000","last_active":"2024-05-27T09:31:58.193948+0000","last_peered":"2024-05-27T09:31:58.193948+0000","last_clean":"2024-05-27T09:31:58.193948+0000","last_became_active":"2024-05-27T09:31:24.905212+0000","last_became_peered":"2024-05-27T09:31:24.905212+0000","last_unstale":"2024-05-27T09:31:58.193948+0000","last_undegraded":"2024-05-27T09:31:58.193948+0000","last_fullsized":"2024-05-27T09:31:58.193948+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:18.267807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865689+0000","last_change":"2024-05-27T09:31:20.848527+0000","last_active":"2024-05-27T09:31:57.865689+0000","last_peered":"2024-05-27T09:31:57.865689+0000","last_clean":"2024-05-27T09:31:57.865689+0000","last_became_active":"2024-05-27T09:31:20.847779+0000","last_became_peered":"2024-05-27T09:31:20.847779+0000","last_unstale":"2024-05-27T09:31:57.865689+0000","last_undegraded":"2024-05-27T09:31:57.865689+0000","last_fullsized":"2024-05-27T09:31:57.865689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:15:01.131306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":102,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863868+0000","last_change":"2024-05-27T09:29:46.120380+0000","last_active":"2024-05-27T09:31:57.863868+0000","last_peered":"2024-05-27T09:31:57.863868+0000","last_clean":"2024-05-27T09:31:57.863868+0000","last_became_active":"2024-05-27T09:29:46.113658+0000","last_became_peered":"2024-05-27T09:29:46.113658+0000","last_unstale":"2024-05-27T09:31:57.863868+0000","last_undegraded":"2024-05-27T09:31:57.863868+0000","last_fullsized":"2024-05-27T09:31:57.863868+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_clean_scrub_stamp":"2024-05-27T09:25:47.244521+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:28:40.424824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.196069+0000","last_change":"2024-05-27T09:31:24.900736+0000","last_active":"2024-05-27T09:31:58.196069+0000","last_peered":"2024-05-27T09:31:58.196069+0000","last_clean":"2024-05-27T09:31:58.196069+0000","last_became_active":"2024-05-27T09:31:24.900354+0000","last_became_peered":"2024-05-27T09:31:24.900354+0000","last_unstale":"2024-05-27T09:31:58.196069+0000","last_undegraded":"2024-05-27T09:31:58.196069+0000","last_fullsized":"2024-05-27T09:31:58.196069+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:39:33.171977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.874220+0000","last_change":"2024-05-27T09:31:57.873603+0000","last_active":"2024-05-27T09:31:57.874220+0000","last_peered":"2024-05-27T09:31:57.874220+0000","last_clean":"2024-05-27T09:31:57.874220+0000","last_became_active":"2024-05-27T09:31:57.868616+0000","last_became_peered":"2024-05-27T09:31:57.868616+0000","last_unstale":"2024-05-27T09:31:57.874220+0000","last_undegraded":"2024-05-27T09:31:57.874220+0000","last_fullsized":"2024-05-27T09:31:57.874220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:02:20.112667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864526+0000","last_change":"2024-05-27T09:31:20.878557+0000","last_active":"2024-05-27T09:31:57.864526+0000","last_peered":"2024-05-27T09:31:57.864526+0000","last_clean":"2024-05-27T09:31:57.864526+0000","last_became_active":"2024-05-27T09:31:20.877969+0000","last_became_peered":"2024-05-27T09:31:20.877969+0000","last_unstale":"2024-05-27T09:31:57.864526+0000","last_undegraded":"2024-05-27T09:31:57.864526+0000","last_fullsized":"2024-05-27T09:31:57.864526+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:46.859645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197143+0000","last_change":"2024-05-27T09:31:24.907906+0000","last_active":"2024-05-27T09:31:58.197143+0000","last_peered":"2024-05-27T09:31:58.197143+0000","last_clean":"2024-05-27T09:31:58.197143+0000","last_became_active":"2024-05-27T09:31:24.907473+0000","last_became_peered":"2024-05-27T09:31:24.907473+0000","last_unstale":"2024-05-27T09:31:58.197143+0000","last_undegraded":"2024-05-27T09:31:58.197143+0000","last_fullsized":"2024-05-27T09:31:58.197143+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:11:38.536919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'11","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:11.947034+0000","last_change":"2024-05-27T09:31:20.881250+0000","last_active":"2024-05-27T09:32:11.947034+0000","last_peered":"2024-05-27T09:32:11.947034+0000","last_clean":"2024-05-27T09:32:11.947034+0000","last_became_active":"2024-05-27T09:31:20.880550+0000","last_became_peered":"2024-05-27T09:31:20.880550+0000","last_unstale":"2024-05-27T09:32:11.947034+0000","last_undegraded":"2024-05-27T09:32:11.947034+0000","last_fullsized":"2024-05-27T09:32:11.947034+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:14.783260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.195493+0000","last_change":"2024-05-27T09:31:24.894554+0000","last_active":"2024-05-27T09:31:58.195493+0000","last_peered":"2024-05-27T09:31:58.195493+0000","last_clean":"2024-05-27T09:31:58.195493+0000","last_became_active":"2024-05-27T09:31:24.893800+0000","last_became_peered":"2024-05-27T09:31:24.893800+0000","last_unstale":"2024-05-27T09:31:58.195493+0000","last_undegraded":"2024-05-27T09:31:58.195493+0000","last_fullsized":"2024-05-27T09:31:58.195493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:13.368892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865447+0000","last_change":"2024-05-27T09:31:21.054476+0000","last_active":"2024-05-27T09:31:57.865447+0000","last_peered":"2024-05-27T09:31:57.865447+0000","last_clean":"2024-05-27T09:31:57.865447+0000","last_became_active":"2024-05-27T09:31:21.054091+0000","last_became_peered":"2024-05-27T09:31:21.054091+0000","last_unstale":"2024-05-27T09:31:57.865447+0000","last_undegraded":"2024-05-27T09:31:57.865447+0000","last_fullsized":"2024-05-27T09:31:57.865447+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:15.909230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862690+0000","last_change":"2024-05-27T09:31:24.913157+0000","last_active":"2024-05-27T09:31:57.862690+0000","last_peered":"2024-05-27T09:31:57.862690+0000","last_clean":"2024-05-27T09:31:57.862690+0000","last_became_active":"2024-05-27T09:31:24.912793+0000","last_became_peered":"2024-05-27T09:31:24.912793+0000","last_unstale":"2024-05-27T09:31:57.862690+0000","last_undegraded":"2024-05-27T09:31:57.862690+0000","last_fullsized":"2024-05-27T09:31:57.862690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:17:36.451069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863635+0000","last_change":"2024-05-27T09:31:20.861349+0000","last_active":"2024-05-27T09:31:57.863635+0000","last_peered":"2024-05-27T09:31:57.863635+0000","last_clean":"2024-05-27T09:31:57.863635+0000","last_became_active":"2024-05-27T09:31:20.860896+0000","last_became_peered":"2024-05-27T09:31:20.860896+0000","last_unstale":"2024-05-27T09:31:57.863635+0000","last_undegraded":"2024-05-27T09:31:57.863635+0000","last_fullsized":"2024-05-27T09:31:57.863635+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:00.772910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863616+0000","last_change":"2024-05-27T09:31:24.893240+0000","last_active":"2024-05-27T09:31:57.863616+0000","last_peered":"2024-05-27T09:31:57.863616+0000","last_clean":"2024-05-27T09:31:57.863616+0000","last_became_active":"2024-05-27T09:31:24.892169+0000","last_became_peered":"2024-05-27T09:31:24.892169+0000","last_unstale":"2024-05-27T09:31:57.863616+0000","last_undegraded":"2024-05-27T09:31:57.863616+0000","last_fullsized":"2024-05-27T09:31:57.863616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:26:59.660725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.193440+0000","last_change":"2024-05-27T09:31:20.864307+0000","last_active":"2024-05-27T09:31:58.193440+0000","last_peered":"2024-05-27T09:31:58.193440+0000","last_clean":"2024-05-27T09:31:58.193440+0000","last_became_active":"2024-05-27T09:31:20.863913+0000","last_became_peered":"2024-05-27T09:31:20.863913+0000","last_unstale":"2024-05-27T09:31:58.193440+0000","last_undegraded":"2024-05-27T09:31:58.193440+0000","last_fullsized":"2024-05-27T09:31:58.193440+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:58:57.656731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.860843+0000","last_change":"2024-05-27T09:31:24.905728+0000","last_active":"2024-05-27T09:31:57.860843+0000","last_peered":"2024-05-27T09:31:57.860843+0000","last_clean":"2024-05-27T09:31:57.860843+0000","last_became_active":"2024-05-27T09:31:24.905365+0000","last_became_peered":"2024-05-27T09:31:24.905365+0000","last_unstale":"2024-05-27T09:31:57.860843+0000","last_undegraded":"2024-05-27T09:31:57.860843+0000","last_fullsized":"2024-05-27T09:31:57.860843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:05.013037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.866878+0000","last_change":"2024-05-27T09:31:20.878478+0000","last_active":"2024-05-27T09:31:57.866878+0000","last_peered":"2024-05-27T09:31:57.866878+0000","last_clean":"2024-05-27T09:31:57.866878+0000","last_became_active":"2024-05-27T09:31:20.877952+0000","last_became_peered":"2024-05-27T09:31:20.877952+0000","last_unstale":"2024-05-27T09:31:57.866878+0000","last_undegraded":"2024-05-27T09:31:57.866878+0000","last_fullsized":"2024-05-27T09:31:57.866878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:16:27.580641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.194183+0000","last_change":"2024-05-27T09:31:24.904967+0000","last_active":"2024-05-27T09:31:58.194183+0000","last_peered":"2024-05-27T09:31:58.194183+0000","last_clean":"2024-05-27T09:31:58.194183+0000","last_became_active":"2024-05-27T09:31:24.904542+0000","last_became_peered":"2024-05-27T09:31:24.904542+0000","last_unstale":"2024-05-27T09:31:58.194183+0000","last_undegraded":"2024-05-27T09:31:58.194183+0000","last_fullsized":"2024-05-27T09:31:58.194183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:01:05.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.875713+0000","last_change":"2024-05-27T09:31:57.875713+0000","last_active":"2024-05-27T09:31:57.875713+0000","last_peered":"2024-05-27T09:31:57.875713+0000","last_clean":"2024-05-27T09:31:57.875713+0000","last_became_active":"2024-05-27T09:31:57.870244+0000","last_became_peered":"2024-05-27T09:31:57.870244+0000","last_unstale":"2024-05-27T09:31:57.875713+0000","last_undegraded":"2024-05-27T09:31:57.875713+0000","last_fullsized":"2024-05-27T09:31:57.875713+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:49.272418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864004+0000","last_change":"2024-05-27T09:31:24.892655+0000","last_active":"2024-05-27T09:31:57.864004+0000","last_peered":"2024-05-27T09:31:57.864004+0000","last_clean":"2024-05-27T09:31:57.864004+0000","last_became_active":"2024-05-27T09:31:24.892112+0000","last_became_peered":"2024-05-27T09:31:24.892112+0000","last_unstale":"2024-05-27T09:31:57.864004+0000","last_undegraded":"2024-05-27T09:31:57.864004+0000","last_fullsized":"2024-05-27T09:31:57.864004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.604118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.876559+0000","last_change":"2024-05-27T09:31:57.875213+0000","last_active":"2024-05-27T09:31:57.876559+0000","last_peered":"2024-05-27T09:31:57.876559+0000","last_clean":"2024-05-27T09:31:57.876559+0000","last_became_active":"2024-05-27T09:31:57.869025+0000","last_became_peered":"2024-05-27T09:31:57.869025+0000","last_unstale":"2024-05-27T09:31:57.876559+0000","last_undegraded":"2024-05-27T09:31:57.876559+0000","last_fullsized":"2024-05-27T09:31:57.876559+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:51.433347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864693+0000","last_change":"2024-05-27T09:31:24.899130+0000","last_active":"2024-05-27T09:31:57.864693+0000","last_peered":"2024-05-27T09:31:57.864693+0000","last_clean":"2024-05-27T09:31:57.864693+0000","last_became_active":"2024-05-27T09:31:24.898722+0000","last_became_peered":"2024-05-27T09:31:24.898722+0000","last_unstale":"2024-05-27T09:31:57.864693+0000","last_undegraded":"2024-05-27T09:31:57.864693+0000","last_fullsized":"2024-05-27T09:31:57.864693+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:51.247275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864231+0000","last_change":"2024-05-27T09:31:20.863757+0000","last_active":"2024-05-27T09:31:57.864231+0000","last_peered":"2024-05-27T09:31:57.864231+0000","last_clean":"2024-05-27T09:31:57.864231+0000","last_became_active":"2024-05-27T09:31:20.863465+0000","last_became_peered":"2024-05-27T09:31:20.863465+0000","last_unstale":"2024-05-27T09:31:57.864231+0000","last_undegraded":"2024-05-27T09:31:57.864231+0000","last_fullsized":"2024-05-27T09:31:57.864231+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:21:45.741822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864260+0000","last_change":"2024-05-27T09:31:24.913749+0000","last_active":"2024-05-27T09:31:57.864260+0000","last_peered":"2024-05-27T09:31:57.864260+0000","last_clean":"2024-05-27T09:31:57.864260+0000","last_became_active":"2024-05-27T09:31:24.913384+0000","last_became_peered":"2024-05-27T09:31:24.913384+0000","last_unstale":"2024-05-27T09:31:57.864260+0000","last_undegraded":"2024-05-27T09:31:57.864260+0000","last_fullsized":"2024-05-27T09:31:57.864260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:24:29.826096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197151+0000","last_change":"2024-05-27T09:31:20.865844+0000","last_active":"2024-05-27T09:31:58.197151+0000","last_peered":"2024-05-27T09:31:58.197151+0000","last_clean":"2024-05-27T09:31:58.197151+0000","last_became_active":"2024-05-27T09:31:20.865364+0000","last_became_peered":"2024-05-27T09:31:20.865364+0000","last_unstale":"2024-05-27T09:31:58.197151+0000","last_undegraded":"2024-05-27T09:31:58.197151+0000","last_fullsized":"2024-05-27T09:31:58.197151+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:11:54.451294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865082+0000","last_change":"2024-05-27T09:31:24.890326+0000","last_active":"2024-05-27T09:31:57.865082+0000","last_peered":"2024-05-27T09:31:57.865082+0000","last_clean":"2024-05-27T09:31:57.865082+0000","last_became_active":"2024-05-27T09:31:24.889789+0000","last_became_peered":"2024-05-27T09:31:24.889789+0000","last_unstale":"2024-05-27T09:31:57.865082+0000","last_undegraded":"2024-05-27T09:31:57.865082+0000","last_fullsized":"2024-05-27T09:31:57.865082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:34:04.252231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.860984+0000","last_change":"2024-05-27T09:31:24.896730+0000","last_active":"2024-05-27T09:31:57.860984+0000","last_peered":"2024-05-27T09:31:57.860984+0000","last_clean":"2024-05-27T09:31:57.860984+0000","last_became_active":"2024-05-27T09:31:24.896365+0000","last_became_peered":"2024-05-27T09:31:24.896365+0000","last_unstale":"2024-05-27T09:31:57.860984+0000","last_undegraded":"2024-05-27T09:31:57.860984+0000","last_fullsized":"2024-05-27T09:31:57.860984+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:37.443788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.866594+0000","last_change":"2024-05-27T09:31:24.899215+0000","last_active":"2024-05-27T09:31:57.866594+0000","last_peered":"2024-05-27T09:31:57.866594+0000","last_clean":"2024-05-27T09:31:57.866594+0000","last_became_active":"2024-05-27T09:31:24.898733+0000","last_became_peered":"2024-05-27T09:31:24.898733+0000","last_unstale":"2024-05-27T09:31:57.866594+0000","last_undegraded":"2024-05-27T09:31:57.866594+0000","last_fullsized":"2024-05-27T09:31:57.866594+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:10.085989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.866737+0000","last_change":"2024-05-27T09:31:24.914922+0000","last_active":"2024-05-27T09:31:57.866737+0000","last_peered":"2024-05-27T09:31:57.866737+0000","last_clean":"2024-05-27T09:31:57.866737+0000","last_became_active":"2024-05-27T09:31:24.914541+0000","last_became_peered":"2024-05-27T09:31:24.914541+0000","last_unstale":"2024-05-27T09:31:57.866737+0000","last_undegraded":"2024-05-27T09:31:57.866737+0000","last_fullsized":"2024-05-27T09:31:57.866737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:30:38.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862333+0000","last_change":"2024-05-27T09:31:24.909628+0000","last_active":"2024-05-27T09:31:57.862333+0000","last_peered":"2024-05-27T09:31:57.862333+0000","last_clean":"2024-05-27T09:31:57.862333+0000","last_became_active":"2024-05-27T09:31:24.902931+0000","last_became_peered":"2024-05-27T09:31:24.902931+0000","last_unstale":"2024-05-27T09:31:57.862333+0000","last_undegraded":"2024-05-27T09:31:57.862333+0000","last_fullsized":"2024-05-27T09:31:57.862333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:06.711316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197466+0000","last_change":"2024-05-27T09:31:24.912099+0000","last_active":"2024-05-27T09:31:58.197466+0000","last_peered":"2024-05-27T09:31:58.197466+0000","last_clean":"2024-05-27T09:31:58.197466+0000","last_became_active":"2024-05-27T09:31:24.911669+0000","last_became_peered":"2024-05-27T09:31:24.911669+0000","last_unstale":"2024-05-27T09:31:58.197466+0000","last_undegraded":"2024-05-27T09:31:58.197466+0000","last_fullsized":"2024-05-27T09:31:58.197466+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:58.068589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861115+0000","last_change":"2024-05-27T09:31:24.891426+0000","last_active":"2024-05-27T09:31:57.861115+0000","last_peered":"2024-05-27T09:31:57.861115+0000","last_clean":"2024-05-27T09:31:57.861115+0000","last_became_active":"2024-05-27T09:31:24.890877+0000","last_became_peered":"2024-05-27T09:31:24.890877+0000","last_unstale":"2024-05-27T09:31:57.861115+0000","last_undegraded":"2024-05-27T09:31:57.861115+0000","last_fullsized":"2024-05-27T09:31:57.861115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:15.338188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.867640+0000","last_change":"2024-05-27T09:31:24.892021+0000","last_active":"2024-05-27T09:31:57.867640+0000","last_peered":"2024-05-27T09:31:57.867640+0000","last_clean":"2024-05-27T09:31:57.867640+0000","last_became_active":"2024-05-27T09:31:24.891607+0000","last_became_peered":"2024-05-27T09:31:24.891607+0000","last_unstale":"2024-05-27T09:31:57.867640+0000","last_undegraded":"2024-05-27T09:31:57.867640+0000","last_fullsized":"2024-05-27T09:31:57.867640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:05.623199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.195340+0000","last_change":"2024-05-27T09:31:24.912166+0000","last_active":"2024-05-27T09:31:58.195340+0000","last_peered":"2024-05-27T09:31:58.195340+0000","last_clean":"2024-05-27T09:31:58.195340+0000","last_became_active":"2024-05-27T09:31:24.907555+0000","last_became_peered":"2024-05-27T09:31:24.907555+0000","last_unstale":"2024-05-27T09:31:58.195340+0000","last_undegraded":"2024-05-27T09:31:58.195340+0000","last_fullsized":"2024-05-27T09:31:58.195340+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:55:33.772321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864057+0000","last_change":"2024-05-27T09:31:24.910525+0000","last_active":"2024-05-27T09:31:57.864057+0000","last_peered":"2024-05-27T09:31:57.864057+0000","last_clean":"2024-05-27T09:31:57.864057+0000","last_became_active":"2024-05-27T09:31:24.910089+0000","last_became_peered":"2024-05-27T09:31:24.910089+0000","last_unstale":"2024-05-27T09:31:57.864057+0000","last_undegraded":"2024-05-27T09:31:57.864057+0000","last_fullsized":"2024-05-27T09:31:57.864057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:19:02.274781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.192977+0000","last_change":"2024-05-27T09:31:24.901125+0000","last_active":"2024-05-27T09:31:58.192977+0000","last_peered":"2024-05-27T09:31:58.192977+0000","last_clean":"2024-05-27T09:31:58.192977+0000","last_became_active":"2024-05-27T09:31:24.900361+0000","last_became_peered":"2024-05-27T09:31:24.900361+0000","last_unstale":"2024-05-27T09:31:58.192977+0000","last_undegraded":"2024-05-27T09:31:58.192977+0000","last_fullsized":"2024-05-27T09:31:58.192977+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:30.116349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864297+0000","last_change":"2024-05-27T09:31:24.917612+0000","last_active":"2024-05-27T09:31:57.864297+0000","last_peered":"2024-05-27T09:31:57.864297+0000","last_clean":"2024-05-27T09:31:57.864297+0000","last_became_active":"2024-05-27T09:31:24.917210+0000","last_became_peered":"2024-05-27T09:31:24.917210+0000","last_unstale":"2024-05-27T09:31:57.864297+0000","last_undegraded":"2024-05-27T09:31:57.864297+0000","last_fullsized":"2024-05-27T09:31:57.864297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:19:00.769114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197309+0000","last_change":"2024-05-27T09:31:24.909654+0000","last_active":"2024-05-27T09:31:58.197309+0000","last_peered":"2024-05-27T09:31:58.197309+0000","last_clean":"2024-05-27T09:31:58.197309+0000","last_became_active":"2024-05-27T09:31:24.909143+0000","last_became_peered":"2024-05-27T09:31:24.909143+0000","last_unstale":"2024-05-27T09:31:58.197309+0000","last_undegraded":"2024-05-27T09:31:58.197309+0000","last_fullsized":"2024-05-27T09:31:58.197309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:42.844614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861202+0000","last_change":"2024-05-27T09:31:24.899031+0000","last_active":"2024-05-27T09:31:57.861202+0000","last_peered":"2024-05-27T09:31:57.861202+0000","last_clean":"2024-05-27T09:31:57.861202+0000","last_became_active":"2024-05-27T09:31:24.898664+0000","last_became_peered":"2024-05-27T09:31:24.898664+0000","last_unstale":"2024-05-27T09:31:57.861202+0000","last_undegraded":"2024-05-27T09:31:57.861202+0000","last_fullsized":"2024-05-27T09:31:57.861202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:09.659646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64739,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":223,"num_read_kb":182,"num_write":164,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":194217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678045,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36200,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708952,"statfs":{"total":95995035648,"available":95957966848,"internally_reserved":0,"allocated":876544,"data_stored":431801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874273,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36872,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708280,"statfs":{"total":95995035648,"available":95957278720,"internally_reserved":0,"allocated":1572864,"data_stored":1084020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070503,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":432037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266733,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":431957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462963,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1486848,"data_stored":1024278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626488,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36200,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708952,"statfs":{"total":95995035648,"available":95957966848,"internally_reserved":0,"allocated":880640,"data_stored":431426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":491566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":901120,"data_stored":432697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":488468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":431737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986244,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":888832,"data_stored":432836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822719,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1499136,"data_stored":1023571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":61506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T09:32:21.865 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T09:32:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:21 smithi050 ceph-mon[36371]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-05-27T09:32:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:32:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:32:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:22.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:21 smithi189 ceph-mon[36671]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-05-27T09:32:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:32:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:32:22.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:32:22.350 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:22.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:22 smithi044 ceph-mon[31618]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:32:23.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:22 smithi050 ceph-mon[36371]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:32:23.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:22 smithi189 ceph-mon[36671]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:32:23.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:23 smithi044 ceph-mon[31618]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-27T09:32:24.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:23 smithi050 ceph-mon[36371]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-27T09:32:24.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:23 smithi189 ceph-mon[36671]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-27T09:32:24.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.515+0000 7f0c19400700 1 -- 172.21.15.44:0/4262290500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c14115360 msgr2=0x7f0c14117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:24.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.515+0000 7f0c19400700 1 --2- 172.21.15.44:0/4262290500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c14115360 0x7f0c14117780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009f90 tx=0x7f0c0c00b330 comp rx=0 tx=0).stop 2024-05-27T09:32:24.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.515+0000 7f0c19400700 1 -- 172.21.15.44:0/4262290500 shutdown_connections 2024-05-27T09:32:24.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.515+0000 7f0c19400700 1 --2- 172.21.15.44:0/4262290500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c14115360 0x7f0c14117780 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:24.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.515+0000 7f0c19400700 1 --2- 172.21.15.44:0/4262290500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c14075b50 0x7f0c14075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:24.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.515+0000 7f0c19400700 1 --2- 172.21.15.44:0/4262290500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c140764a0 0x7f0c14114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:24.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.515+0000 7f0c19400700 1 -- 172.21.15.44:0/4262290500 >> 172.21.15.44:0/4262290500 conn(0x7f0c14100030 msgr2=0x7f0c14102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:24.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.515+0000 7f0c19400700 1 -- 172.21.15.44:0/4262290500 shutdown_connections 2024-05-27T09:32:24.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.515+0000 7f0c19400700 1 -- 172.21.15.44:0/4262290500 wait complete. 2024-05-27T09:32:24.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.517+0000 7f0c19400700 1 Processor -- start 2024-05-27T09:32:24.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.517+0000 7f0c19400700 1 -- start start 2024-05-27T09:32:24.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c19400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c14075b50 0x7f0c141a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:24.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c19400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c140764a0 0x7f0c141a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:24.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c19400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c14115360 0x7f0c141a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:24.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c19400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1411aab0 con 0x7f0c14115360 2024-05-27T09:32:24.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c19400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1411a930 con 0x7f0c140764a0 2024-05-27T09:32:24.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c19400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1411ac30 con 0x7f0c14075b50 2024-05-27T09:32:24.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c13200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c14115360 0x7f0c141a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:24.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c12800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c14075b50 0x7f0c141a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:24.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c13200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c14115360 0x7f0c141a5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46168/0 (socket says 172.21.15.44:46168) 2024-05-27T09:32:24.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c12800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c14075b50 0x7f0c141a13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55860/0 (socket says 172.21.15.44:55860) 2024-05-27T09:32:24.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c13200700 1 -- 172.21.15.44:0/2736996486 learned_addr learned my addr 172.21.15.44:0/2736996486 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:24.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.518+0000 7f0c09e00700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c140764a0 0x7f0c141a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:24.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.519+0000 7f0c09e00700 1 -- 172.21.15.44:0/2736996486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c14075b50 msgr2=0x7f0c141a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:24.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.519+0000 7f0c09e00700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c14075b50 0x7f0c141a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:24.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.519+0000 7f0c09e00700 1 -- 172.21.15.44:0/2736996486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c14115360 msgr2=0x7f0c141a5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:24.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.519+0000 7f0c09e00700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c14115360 0x7f0c141a5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:24.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.519+0000 7f0c09e00700 1 -- 172.21.15.44:0/2736996486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c141a6270 con 0x7f0c140764a0 2024-05-27T09:32:24.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.519+0000 7f0c13200700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c14115360 0x7f0c141a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:32:24.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.519+0000 7f0c09e00700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c140764a0 0x7f0c141a1940 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00e9f0 tx=0x7f0bfc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:24.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.519+0000 7f0c0be00700 1 -- 172.21.15.44:0/2736996486 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc00c9c0 con 0x7f0c140764a0 2024-05-27T09:32:24.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.519+0000 7f0c0be00700 1 -- 172.21.15.44:0/2736996486 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bfc00cb60 con 0x7f0c140764a0 2024-05-27T09:32:24.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.519+0000 7f0c19400700 1 -- 172.21.15.44:0/2736996486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c141a6560 con 0x7f0c140764a0 2024-05-27T09:32:24.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.519+0000 7f0c0be00700 1 -- 172.21.15.44:0/2736996486 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc018680 con 0x7f0c140764a0 2024-05-27T09:32:24.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.520+0000 7f0c19400700 1 -- 172.21.15.44:0/2736996486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c141ade80 con 0x7f0c140764a0 2024-05-27T09:32:24.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.521+0000 7f0c19400700 1 -- 172.21.15.44:0/2736996486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c14077290 con 0x7f0c140764a0 2024-05-27T09:32:24.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.523+0000 7f0c0be00700 1 -- 172.21.15.44:0/2736996486 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0bfc010430 con 0x7f0c140764a0 2024-05-27T09:32:24.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.525+0000 7f0c0be00700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0bf80b2910 0x7f0bf80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:24.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.525+0000 7f0c0be00700 1 -- 172.21.15.44:0/2736996486 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f0bfc0e2820 con 0x7f0c140764a0 2024-05-27T09:32:24.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.525+0000 7f0c12800700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0bf80b2910 0x7f0bf80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:24.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.526+0000 7f0c12800700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0bf80b2910 0x7f0bf80b4e20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009750 tx=0x7f0c04009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:24.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.527+0000 7f0c0be00700 1 -- 172.21.15.44:0/2736996486 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0bfc0aa390 con 0x7f0c140764a0 2024-05-27T09:32:24.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.758+0000 7f0c19400700 1 -- 172.21.15.44:0/2736996486 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0c141a2700 con 0x7f0bf80b2910 2024-05-27T09:32:24.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.781+0000 7f0c0be00700 1 -- 172.21.15.44:0/2736996486 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258702 (secure 0 0 0) 0x7f0c141a2700 con 0x7f0bf80b2910 2024-05-27T09:32:24.783 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:32:24.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.787+0000 7f0c19400700 1 -- 172.21.15.44:0/2736996486 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0bf80b2910 msgr2=0x7f0bf80b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:24.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.787+0000 7f0c19400700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0bf80b2910 0x7f0bf80b4e20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009750 tx=0x7f0c04009310 comp rx=0 tx=0).stop 2024-05-27T09:32:24.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.787+0000 7f0c19400700 1 -- 172.21.15.44:0/2736996486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c140764a0 msgr2=0x7f0c141a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:24.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.787+0000 7f0c19400700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c140764a0 0x7f0c141a1940 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00e9f0 tx=0x7f0bfc00ed00 comp rx=0 tx=0).stop 2024-05-27T09:32:24.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.787+0000 7f0c19400700 1 -- 172.21.15.44:0/2736996486 shutdown_connections 2024-05-27T09:32:24.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.787+0000 7f0c19400700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0bf80b2910 0x7f0bf80b4e20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:24.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.787+0000 7f0c19400700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c14115360 0x7f0c141a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:24.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.787+0000 7f0c19400700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c14075b50 0x7f0c141a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:24.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.787+0000 7f0c19400700 1 --2- 172.21.15.44:0/2736996486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c140764a0 0x7f0c141a1940 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:24.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.787+0000 7f0c19400700 1 -- 172.21.15.44:0/2736996486 >> 172.21.15.44:0/2736996486 conn(0x7f0c14100030 msgr2=0x7f0c141005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:24.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.788+0000 7f0c19400700 1 -- 172.21.15.44:0/2736996486 shutdown_connections 2024-05-27T09:32:24.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:24.788+0000 7f0c19400700 1 -- 172.21.15.44:0/2736996486 wait complete. 2024-05-27T09:32:24.790 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-05-27T09:32:25.496 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-05-27T09:32:24.645715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655341,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":329,"num_read_kb":261,"num_write":289,"num_write_kb":2263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":234,"ondisk_log_size":234,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378208,"kb_used_data":12396,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563616,"statfs":{"total":1151940427776,"available":1151553142784,"internally_reserved":0,"allocated":12693504,"data_stored":7140651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":51,"num_write":8,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005058"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861357+0000","last_change":"2024-05-27T09:31:24.899109+0000","last_active":"2024-05-27T09:31:57.861357+0000","last_peered":"2024-05-27T09:31:57.861357+0000","last_clean":"2024-05-27T09:31:57.861357+0000","last_became_active":"2024-05-27T09:31:24.898783+0000","last_became_peered":"2024-05-27T09:31:24.898783+0000","last_unstale":"2024-05-27T09:31:57.861357+0000","last_undegraded":"2024-05-27T09:31:57.861357+0000","last_fullsized":"2024-05-27T09:31:57.861357+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:01:12.220389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862630+0000","last_change":"2024-05-27T09:31:24.898382+0000","last_active":"2024-05-27T09:31:57.862630+0000","last_peered":"2024-05-27T09:31:57.862630+0000","last_clean":"2024-05-27T09:31:57.862630+0000","last_became_active":"2024-05-27T09:31:24.897975+0000","last_became_peered":"2024-05-27T09:31:24.897975+0000","last_unstale":"2024-05-27T09:31:57.862630+0000","last_undegraded":"2024-05-27T09:31:57.862630+0000","last_fullsized":"2024-05-27T09:31:57.862630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:38.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862172+0000","last_change":"2024-05-27T09:31:24.896122+0000","last_active":"2024-05-27T09:31:57.862172+0000","last_peered":"2024-05-27T09:31:57.862172+0000","last_clean":"2024-05-27T09:31:57.862172+0000","last_became_active":"2024-05-27T09:31:24.890867+0000","last_became_peered":"2024-05-27T09:31:24.890867+0000","last_unstale":"2024-05-27T09:31:57.862172+0000","last_undegraded":"2024-05-27T09:31:57.862172+0000","last_fullsized":"2024-05-27T09:31:57.862172+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:09.982177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.870065+0000","last_change":"2024-05-27T09:31:57.868900+0000","last_active":"2024-05-27T09:31:57.870065+0000","last_peered":"2024-05-27T09:31:57.870065+0000","last_clean":"2024-05-27T09:31:57.870065+0000","last_became_active":"2024-05-27T09:31:57.868418+0000","last_became_peered":"2024-05-27T09:31:57.868418+0000","last_unstale":"2024-05-27T09:31:57.870065+0000","last_undegraded":"2024-05-27T09:31:57.870065+0000","last_fullsized":"2024-05-27T09:31:57.870065+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:08.966704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864519+0000","last_change":"2024-05-27T09:31:24.910283+0000","last_active":"2024-05-27T09:31:57.864519+0000","last_peered":"2024-05-27T09:31:57.864519+0000","last_clean":"2024-05-27T09:31:57.864519+0000","last_became_active":"2024-05-27T09:31:24.909882+0000","last_became_peered":"2024-05-27T09:31:24.909882+0000","last_unstale":"2024-05-27T09:31:57.864519+0000","last_undegraded":"2024-05-27T09:31:57.864519+0000","last_fullsized":"2024-05-27T09:31:57.864519+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:43.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.860783+0000","last_change":"2024-05-27T09:31:24.900934+0000","last_active":"2024-05-27T09:31:57.860783+0000","last_peered":"2024-05-27T09:31:57.860783+0000","last_clean":"2024-05-27T09:31:57.860783+0000","last_became_active":"2024-05-27T09:31:24.900534+0000","last_became_peered":"2024-05-27T09:31:24.900534+0000","last_unstale":"2024-05-27T09:31:57.860783+0000","last_undegraded":"2024-05-27T09:31:57.860783+0000","last_fullsized":"2024-05-27T09:31:57.860783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:53.318327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863567+0000","last_change":"2024-05-27T09:31:24.907080+0000","last_active":"2024-05-27T09:31:57.863567+0000","last_peered":"2024-05-27T09:31:57.863567+0000","last_clean":"2024-05-27T09:31:57.863567+0000","last_became_active":"2024-05-27T09:31:24.906715+0000","last_became_peered":"2024-05-27T09:31:24.906715+0000","last_unstale":"2024-05-27T09:31:57.863567+0000","last_undegraded":"2024-05-27T09:31:57.863567+0000","last_fullsized":"2024-05-27T09:31:57.863567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:36.214659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.201871+0000","last_change":"2024-05-27T09:31:58.200608+0000","last_active":"2024-05-27T09:31:58.201871+0000","last_peered":"2024-05-27T09:31:58.201871+0000","last_clean":"2024-05-27T09:31:58.201871+0000","last_became_active":"2024-05-27T09:31:58.199363+0000","last_became_peered":"2024-05-27T09:31:58.199363+0000","last_unstale":"2024-05-27T09:31:58.201871+0000","last_undegraded":"2024-05-27T09:31:58.201871+0000","last_fullsized":"2024-05-27T09:31:58.201871+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:30:46.495930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864728+0000","last_change":"2024-05-27T09:31:24.892630+0000","last_active":"2024-05-27T09:31:57.864728+0000","last_peered":"2024-05-27T09:31:57.864728+0000","last_clean":"2024-05-27T09:31:57.864728+0000","last_became_active":"2024-05-27T09:31:24.892176+0000","last_became_peered":"2024-05-27T09:31:24.892176+0000","last_unstale":"2024-05-27T09:31:57.864728+0000","last_undegraded":"2024-05-27T09:31:57.864728+0000","last_fullsized":"2024-05-27T09:31:57.864728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:49:20.585601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861253+0000","last_change":"2024-05-27T09:31:24.895281+0000","last_active":"2024-05-27T09:31:57.861253+0000","last_peered":"2024-05-27T09:31:57.861253+0000","last_clean":"2024-05-27T09:31:57.861253+0000","last_became_active":"2024-05-27T09:31:24.894886+0000","last_became_peered":"2024-05-27T09:31:24.894886+0000","last_unstale":"2024-05-27T09:31:57.861253+0000","last_undegraded":"2024-05-27T09:31:57.861253+0000","last_fullsized":"2024-05-27T09:31:57.861253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:41.494434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864466+0000","last_change":"2024-05-27T09:31:24.900167+0000","last_active":"2024-05-27T09:31:57.864466+0000","last_peered":"2024-05-27T09:31:57.864466+0000","last_clean":"2024-05-27T09:31:57.864466+0000","last_became_active":"2024-05-27T09:31:24.899750+0000","last_became_peered":"2024-05-27T09:31:24.899750+0000","last_unstale":"2024-05-27T09:31:57.864466+0000","last_undegraded":"2024-05-27T09:31:57.864466+0000","last_fullsized":"2024-05-27T09:31:57.864466+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:30.602246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.869986+0000","last_change":"2024-05-27T09:31:57.868818+0000","last_active":"2024-05-27T09:31:57.869986+0000","last_peered":"2024-05-27T09:31:57.869986+0000","last_clean":"2024-05-27T09:31:57.869986+0000","last_became_active":"2024-05-27T09:31:57.868322+0000","last_became_peered":"2024-05-27T09:31:57.868322+0000","last_unstale":"2024-05-27T09:31:57.869986+0000","last_undegraded":"2024-05-27T09:31:57.869986+0000","last_fullsized":"2024-05-27T09:31:57.869986+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:35.304850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197377+0000","last_change":"2024-05-27T09:31:24.902537+0000","last_active":"2024-05-27T09:31:58.197377+0000","last_peered":"2024-05-27T09:31:58.197377+0000","last_clean":"2024-05-27T09:31:58.197377+0000","last_became_active":"2024-05-27T09:31:24.902132+0000","last_became_peered":"2024-05-27T09:31:24.902132+0000","last_unstale":"2024-05-27T09:31:58.197377+0000","last_undegraded":"2024-05-27T09:31:58.197377+0000","last_fullsized":"2024-05-27T09:31:58.197377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:20:50.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862152+0000","last_change":"2024-05-27T09:31:24.899182+0000","last_active":"2024-05-27T09:31:57.862152+0000","last_peered":"2024-05-27T09:31:57.862152+0000","last_clean":"2024-05-27T09:31:57.862152+0000","last_became_active":"2024-05-27T09:31:24.898749+0000","last_became_peered":"2024-05-27T09:31:24.898749+0000","last_unstale":"2024-05-27T09:31:57.862152+0000","last_undegraded":"2024-05-27T09:31:57.862152+0000","last_fullsized":"2024-05-27T09:31:57.862152+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:44:44.844357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862064+0000","last_change":"2024-05-27T09:31:24.901115+0000","last_active":"2024-05-27T09:31:57.862064+0000","last_peered":"2024-05-27T09:31:57.862064+0000","last_clean":"2024-05-27T09:31:57.862064+0000","last_became_active":"2024-05-27T09:31:24.897011+0000","last_became_peered":"2024-05-27T09:31:24.897011+0000","last_unstale":"2024-05-27T09:31:57.862064+0000","last_undegraded":"2024-05-27T09:31:57.862064+0000","last_fullsized":"2024-05-27T09:31:57.862064+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:01.130407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863461+0000","last_change":"2024-05-27T09:31:24.903063+0000","last_active":"2024-05-27T09:31:57.863461+0000","last_peered":"2024-05-27T09:31:57.863461+0000","last_clean":"2024-05-27T09:31:57.863461+0000","last_became_active":"2024-05-27T09:31:24.899766+0000","last_became_peered":"2024-05-27T09:31:24.899766+0000","last_unstale":"2024-05-27T09:31:57.863461+0000","last_undegraded":"2024-05-27T09:31:57.863461+0000","last_fullsized":"2024-05-27T09:31:57.863461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:45.984226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861506+0000","last_change":"2024-05-27T09:31:24.915922+0000","last_active":"2024-05-27T09:31:57.861506+0000","last_peered":"2024-05-27T09:31:57.861506+0000","last_clean":"2024-05-27T09:31:57.861506+0000","last_became_active":"2024-05-27T09:31:24.915623+0000","last_became_peered":"2024-05-27T09:31:24.915623+0000","last_unstale":"2024-05-27T09:31:57.861506+0000","last_undegraded":"2024-05-27T09:31:57.861506+0000","last_fullsized":"2024-05-27T09:31:57.861506+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:02:01.006704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864190+0000","last_change":"2024-05-27T09:31:24.908397+0000","last_active":"2024-05-27T09:31:57.864190+0000","last_peered":"2024-05-27T09:31:57.864190+0000","last_clean":"2024-05-27T09:31:57.864190+0000","last_became_active":"2024-05-27T09:31:24.903644+0000","last_became_peered":"2024-05-27T09:31:24.903644+0000","last_unstale":"2024-05-27T09:31:57.864190+0000","last_undegraded":"2024-05-27T09:31:57.864190+0000","last_fullsized":"2024-05-27T09:31:57.864190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:09.281429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.867417+0000","last_change":"2024-05-27T09:31:24.912706+0000","last_active":"2024-05-27T09:31:57.867417+0000","last_peered":"2024-05-27T09:31:57.867417+0000","last_clean":"2024-05-27T09:31:57.867417+0000","last_became_active":"2024-05-27T09:31:24.912357+0000","last_became_peered":"2024-05-27T09:31:24.912357+0000","last_unstale":"2024-05-27T09:31:57.867417+0000","last_undegraded":"2024-05-27T09:31:57.867417+0000","last_fullsized":"2024-05-27T09:31:57.867417+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:23.334041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864061+0000","last_change":"2024-05-27T09:31:20.881652+0000","last_active":"2024-05-27T09:31:57.864061+0000","last_peered":"2024-05-27T09:31:57.864061+0000","last_clean":"2024-05-27T09:31:57.864061+0000","last_became_active":"2024-05-27T09:31:20.880757+0000","last_became_peered":"2024-05-27T09:31:20.880757+0000","last_unstale":"2024-05-27T09:31:57.864061+0000","last_undegraded":"2024-05-27T09:31:57.864061+0000","last_fullsized":"2024-05-27T09:31:57.864061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:01.433755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.194126+0000","last_change":"2024-05-27T09:31:24.908010+0000","last_active":"2024-05-27T09:31:58.194126+0000","last_peered":"2024-05-27T09:31:58.194126+0000","last_clean":"2024-05-27T09:31:58.194126+0000","last_became_active":"2024-05-27T09:31:24.907638+0000","last_became_peered":"2024-05-27T09:31:24.907638+0000","last_unstale":"2024-05-27T09:31:58.194126+0000","last_undegraded":"2024-05-27T09:31:58.194126+0000","last_fullsized":"2024-05-27T09:31:58.194126+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:02:49.207709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.195199+0000","last_change":"2024-05-27T09:31:20.860556+0000","last_active":"2024-05-27T09:31:58.195199+0000","last_peered":"2024-05-27T09:31:58.195199+0000","last_clean":"2024-05-27T09:31:58.195199+0000","last_became_active":"2024-05-27T09:31:20.860172+0000","last_became_peered":"2024-05-27T09:31:20.860172+0000","last_unstale":"2024-05-27T09:31:58.195199+0000","last_undegraded":"2024-05-27T09:31:58.195199+0000","last_fullsized":"2024-05-27T09:31:58.195199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:08:21.545681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862841+0000","last_change":"2024-05-27T09:31:24.910111+0000","last_active":"2024-05-27T09:31:57.862841+0000","last_peered":"2024-05-27T09:31:57.862841+0000","last_clean":"2024-05-27T09:31:57.862841+0000","last_became_active":"2024-05-27T09:31:24.908985+0000","last_became_peered":"2024-05-27T09:31:24.908985+0000","last_unstale":"2024-05-27T09:31:57.862841+0000","last_undegraded":"2024-05-27T09:31:57.862841+0000","last_fullsized":"2024-05-27T09:31:57.862841+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:53:29.645792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864702+0000","last_change":"2024-05-27T09:31:20.879993+0000","last_active":"2024-05-27T09:31:57.864702+0000","last_peered":"2024-05-27T09:31:57.864702+0000","last_clean":"2024-05-27T09:31:57.864702+0000","last_became_active":"2024-05-27T09:31:20.878802+0000","last_became_peered":"2024-05-27T09:31:20.878802+0000","last_unstale":"2024-05-27T09:31:57.864702+0000","last_undegraded":"2024-05-27T09:31:57.864702+0000","last_fullsized":"2024-05-27T09:31:57.864702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:15.555612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864431+0000","last_change":"2024-05-27T09:31:24.903533+0000","last_active":"2024-05-27T09:31:57.864431+0000","last_peered":"2024-05-27T09:31:57.864431+0000","last_clean":"2024-05-27T09:31:57.864431+0000","last_became_active":"2024-05-27T09:31:24.903166+0000","last_became_peered":"2024-05-27T09:31:24.903166+0000","last_unstale":"2024-05-27T09:31:57.864431+0000","last_undegraded":"2024-05-27T09:31:57.864431+0000","last_fullsized":"2024-05-27T09:31:57.864431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:52.539470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.878845+0000","last_change":"2024-05-27T09:31:57.878426+0000","last_active":"2024-05-27T09:31:57.878845+0000","last_peered":"2024-05-27T09:31:57.878845+0000","last_clean":"2024-05-27T09:31:57.878845+0000","last_became_active":"2024-05-27T09:31:57.869146+0000","last_became_peered":"2024-05-27T09:31:57.869146+0000","last_unstale":"2024-05-27T09:31:57.878845+0000","last_undegraded":"2024-05-27T09:31:57.878845+0000","last_fullsized":"2024-05-27T09:31:57.878845+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:34:51.926092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864352+0000","last_change":"2024-05-27T09:31:24.897426+0000","last_active":"2024-05-27T09:31:57.864352+0000","last_peered":"2024-05-27T09:31:57.864352+0000","last_clean":"2024-05-27T09:31:57.864352+0000","last_became_active":"2024-05-27T09:31:24.897080+0000","last_became_peered":"2024-05-27T09:31:24.897080+0000","last_unstale":"2024-05-27T09:31:57.864352+0000","last_undegraded":"2024-05-27T09:31:57.864352+0000","last_fullsized":"2024-05-27T09:31:57.864352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:15:42.428391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":71,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:20.841013+0000","last_change":"2024-05-27T09:31:20.846681+0000","last_active":"2024-05-27T09:32:20.841013+0000","last_peered":"2024-05-27T09:32:20.841013+0000","last_clean":"2024-05-27T09:32:20.841013+0000","last_became_active":"2024-05-27T09:31:20.846302+0000","last_became_peered":"2024-05-27T09:31:20.846302+0000","last_unstale":"2024-05-27T09:32:20.841013+0000","last_undegraded":"2024-05-27T09:32:20.841013+0000","last_fullsized":"2024-05-27T09:32:20.841013+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:19.609612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863508+0000","last_change":"2024-05-27T09:31:24.907209+0000","last_active":"2024-05-27T09:31:57.863508+0000","last_peered":"2024-05-27T09:31:57.863508+0000","last_clean":"2024-05-27T09:31:57.863508+0000","last_became_active":"2024-05-27T09:31:24.906752+0000","last_became_peered":"2024-05-27T09:31:24.906752+0000","last_unstale":"2024-05-27T09:31:57.863508+0000","last_undegraded":"2024-05-27T09:31:57.863508+0000","last_fullsized":"2024-05-27T09:31:57.863508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:53.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.860743+0000","last_change":"2024-05-27T09:31:20.879778+0000","last_active":"2024-05-27T09:31:57.860743+0000","last_peered":"2024-05-27T09:31:57.860743+0000","last_clean":"2024-05-27T09:31:57.860743+0000","last_became_active":"2024-05-27T09:31:20.879259+0000","last_became_peered":"2024-05-27T09:31:20.879259+0000","last_unstale":"2024-05-27T09:31:57.860743+0000","last_undegraded":"2024-05-27T09:31:57.860743+0000","last_fullsized":"2024-05-27T09:31:57.860743+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:20:07.972653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.195516+0000","last_change":"2024-05-27T09:31:58.195516+0000","last_active":"2024-05-27T09:31:58.195516+0000","last_peered":"2024-05-27T09:31:58.195516+0000","last_clean":"2024-05-27T09:31:58.195516+0000","last_became_active":"2024-05-27T09:31:58.195118+0000","last_became_peered":"2024-05-27T09:31:58.195118+0000","last_unstale":"2024-05-27T09:31:58.195516+0000","last_undegraded":"2024-05-27T09:31:58.195516+0000","last_fullsized":"2024-05-27T09:31:58.195516+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:11:23.547605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.196330+0000","last_change":"2024-05-27T09:31:20.863152+0000","last_active":"2024-05-27T09:31:58.196330+0000","last_peered":"2024-05-27T09:31:58.196330+0000","last_clean":"2024-05-27T09:31:58.196330+0000","last_became_active":"2024-05-27T09:31:20.862708+0000","last_became_peered":"2024-05-27T09:31:20.862708+0000","last_unstale":"2024-05-27T09:31:58.196330+0000","last_undegraded":"2024-05-27T09:31:58.196330+0000","last_fullsized":"2024-05-27T09:31:58.196330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:21.645773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.192762+0000","last_change":"2024-05-27T09:31:24.893988+0000","last_active":"2024-05-27T09:31:58.192762+0000","last_peered":"2024-05-27T09:31:58.192762+0000","last_clean":"2024-05-27T09:31:58.192762+0000","last_became_active":"2024-05-27T09:31:24.893504+0000","last_became_peered":"2024-05-27T09:31:24.893504+0000","last_unstale":"2024-05-27T09:31:58.192762+0000","last_undegraded":"2024-05-27T09:31:58.192762+0000","last_fullsized":"2024-05-27T09:31:58.192762+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:47.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863900+0000","last_change":"2024-05-27T09:31:20.845555+0000","last_active":"2024-05-27T09:31:57.863900+0000","last_peered":"2024-05-27T09:31:57.863900+0000","last_clean":"2024-05-27T09:31:57.863900+0000","last_became_active":"2024-05-27T09:31:20.845139+0000","last_became_peered":"2024-05-27T09:31:20.845139+0000","last_unstale":"2024-05-27T09:31:57.863900+0000","last_undegraded":"2024-05-27T09:31:57.863900+0000","last_fullsized":"2024-05-27T09:31:57.863900+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:25.917845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863254+0000","last_change":"2024-05-27T09:31:24.916142+0000","last_active":"2024-05-27T09:31:57.863254+0000","last_peered":"2024-05-27T09:31:57.863254+0000","last_clean":"2024-05-27T09:31:57.863254+0000","last_became_active":"2024-05-27T09:31:24.915709+0000","last_became_peered":"2024-05-27T09:31:24.915709+0000","last_unstale":"2024-05-27T09:31:57.863254+0000","last_undegraded":"2024-05-27T09:31:57.863254+0000","last_fullsized":"2024-05-27T09:31:57.863254+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:38.029648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'3","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:22.843614+0000","last_change":"2024-05-27T09:31:20.859240+0000","last_active":"2024-05-27T09:32:22.843614+0000","last_peered":"2024-05-27T09:32:22.843614+0000","last_clean":"2024-05-27T09:32:22.843614+0000","last_became_active":"2024-05-27T09:31:20.858798+0000","last_became_peered":"2024-05-27T09:31:20.858798+0000","last_unstale":"2024-05-27T09:32:22.843614+0000","last_undegraded":"2024-05-27T09:32:22.843614+0000","last_fullsized":"2024-05-27T09:32:22.843614+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:42.280165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863938+0000","last_change":"2024-05-27T09:31:24.908818+0000","last_active":"2024-05-27T09:31:57.863938+0000","last_peered":"2024-05-27T09:31:57.863938+0000","last_clean":"2024-05-27T09:31:57.863938+0000","last_became_active":"2024-05-27T09:31:24.908410+0000","last_became_peered":"2024-05-27T09:31:24.908410+0000","last_unstale":"2024-05-27T09:31:57.863938+0000","last_undegraded":"2024-05-27T09:31:57.863938+0000","last_fullsized":"2024-05-27T09:31:57.863938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:05:40.326232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864270+0000","last_change":"2024-05-27T09:31:20.848632+0000","last_active":"2024-05-27T09:31:57.864270+0000","last_peered":"2024-05-27T09:31:57.864270+0000","last_clean":"2024-05-27T09:31:57.864270+0000","last_became_active":"2024-05-27T09:31:20.847558+0000","last_became_peered":"2024-05-27T09:31:20.847558+0000","last_unstale":"2024-05-27T09:31:57.864270+0000","last_undegraded":"2024-05-27T09:31:57.864270+0000","last_fullsized":"2024-05-27T09:31:57.864270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:53.930565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862862+0000","last_change":"2024-05-27T09:31:24.901513+0000","last_active":"2024-05-27T09:31:57.862862+0000","last_peered":"2024-05-27T09:31:57.862862+0000","last_clean":"2024-05-27T09:31:57.862862+0000","last_became_active":"2024-05-27T09:31:24.895924+0000","last_became_peered":"2024-05-27T09:31:24.895924+0000","last_unstale":"2024-05-27T09:31:57.862862+0000","last_undegraded":"2024-05-27T09:31:57.862862+0000","last_fullsized":"2024-05-27T09:31:57.862862+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:27:39.939098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864463+0000","last_change":"2024-05-27T09:31:20.855195+0000","last_active":"2024-05-27T09:31:57.864463+0000","last_peered":"2024-05-27T09:31:57.864463+0000","last_clean":"2024-05-27T09:31:57.864463+0000","last_became_active":"2024-05-27T09:31:20.854910+0000","last_became_peered":"2024-05-27T09:31:20.854910+0000","last_unstale":"2024-05-27T09:31:57.864463+0000","last_undegraded":"2024-05-27T09:31:57.864463+0000","last_fullsized":"2024-05-27T09:31:57.864463+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:02.795103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865129+0000","last_change":"2024-05-27T09:31:24.895887+0000","last_active":"2024-05-27T09:31:57.865129+0000","last_peered":"2024-05-27T09:31:57.865129+0000","last_clean":"2024-05-27T09:31:57.865129+0000","last_became_active":"2024-05-27T09:31:24.894622+0000","last_became_peered":"2024-05-27T09:31:24.894622+0000","last_unstale":"2024-05-27T09:31:57.865129+0000","last_undegraded":"2024-05-27T09:31:57.865129+0000","last_fullsized":"2024-05-27T09:31:57.865129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:32.833946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'29","reported_seq":139,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:23.852635+0000","last_change":"2024-05-27T09:31:20.846063+0000","last_active":"2024-05-27T09:32:23.852635+0000","last_peered":"2024-05-27T09:32:23.852635+0000","last_clean":"2024-05-27T09:32:23.852635+0000","last_became_active":"2024-05-27T09:31:20.845401+0000","last_became_peered":"2024-05-27T09:31:20.845401+0000","last_unstale":"2024-05-27T09:32:23.852635+0000","last_undegraded":"2024-05-27T09:32:23.852635+0000","last_fullsized":"2024-05-27T09:32:23.852635+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:21:38.460183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60347,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":84,"num_read_kb":101,"num_write":30,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864414+0000","last_change":"2024-05-27T09:31:24.892748+0000","last_active":"2024-05-27T09:31:57.864414+0000","last_peered":"2024-05-27T09:31:57.864414+0000","last_clean":"2024-05-27T09:31:57.864414+0000","last_became_active":"2024-05-27T09:31:24.892394+0000","last_became_peered":"2024-05-27T09:31:24.892394+0000","last_unstale":"2024-05-27T09:31:57.864414+0000","last_undegraded":"2024-05-27T09:31:57.864414+0000","last_fullsized":"2024-05-27T09:31:57.864414+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:39:31.381072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.867076+0000","last_change":"2024-05-27T09:31:20.860789+0000","last_active":"2024-05-27T09:31:57.867076+0000","last_peered":"2024-05-27T09:31:57.867076+0000","last_clean":"2024-05-27T09:31:57.867076+0000","last_became_active":"2024-05-27T09:31:20.860369+0000","last_became_peered":"2024-05-27T09:31:20.860369+0000","last_unstale":"2024-05-27T09:31:57.867076+0000","last_undegraded":"2024-05-27T09:31:57.867076+0000","last_fullsized":"2024-05-27T09:31:57.867076+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:05:35.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.196372+0000","last_change":"2024-05-27T09:31:24.918239+0000","last_active":"2024-05-27T09:31:58.196372+0000","last_peered":"2024-05-27T09:31:58.196372+0000","last_clean":"2024-05-27T09:31:58.196372+0000","last_became_active":"2024-05-27T09:31:24.917750+0000","last_became_peered":"2024-05-27T09:31:24.917750+0000","last_unstale":"2024-05-27T09:31:58.196372+0000","last_undegraded":"2024-05-27T09:31:58.196372+0000","last_fullsized":"2024-05-27T09:31:58.196372+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:07.162994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863930+0000","last_change":"2024-05-27T09:31:20.865308+0000","last_active":"2024-05-27T09:31:57.863930+0000","last_peered":"2024-05-27T09:31:57.863930+0000","last_clean":"2024-05-27T09:31:57.863930+0000","last_became_active":"2024-05-27T09:31:20.864920+0000","last_became_peered":"2024-05-27T09:31:20.864920+0000","last_unstale":"2024-05-27T09:31:57.863930+0000","last_undegraded":"2024-05-27T09:31:57.863930+0000","last_fullsized":"2024-05-27T09:31:57.863930+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:36.773322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863675+0000","last_change":"2024-05-27T09:31:24.902533+0000","last_active":"2024-05-27T09:31:57.863675+0000","last_peered":"2024-05-27T09:31:57.863675+0000","last_clean":"2024-05-27T09:31:57.863675+0000","last_became_active":"2024-05-27T09:31:24.902136+0000","last_became_peered":"2024-05-27T09:31:24.902136+0000","last_unstale":"2024-05-27T09:31:57.863675+0000","last_undegraded":"2024-05-27T09:31:57.863675+0000","last_fullsized":"2024-05-27T09:31:57.863675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:49:32.203577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'4","reported_seq":70,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:21.855103+0000","last_change":"2024-05-27T09:31:20.846060+0000","last_active":"2024-05-27T09:32:21.855103+0000","last_peered":"2024-05-27T09:32:21.855103+0000","last_clean":"2024-05-27T09:32:21.855103+0000","last_became_active":"2024-05-27T09:31:20.845632+0000","last_became_peered":"2024-05-27T09:31:20.845632+0000","last_unstale":"2024-05-27T09:32:21.855103+0000","last_undegraded":"2024-05-27T09:32:21.855103+0000","last_fullsized":"2024-05-27T09:32:21.855103+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:55.166579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863415+0000","last_change":"2024-05-27T09:31:24.895866+0000","last_active":"2024-05-27T09:31:57.863415+0000","last_peered":"2024-05-27T09:31:57.863415+0000","last_clean":"2024-05-27T09:31:57.863415+0000","last_became_active":"2024-05-27T09:31:24.895413+0000","last_became_peered":"2024-05-27T09:31:24.895413+0000","last_unstale":"2024-05-27T09:31:57.863415+0000","last_undegraded":"2024-05-27T09:31:57.863415+0000","last_fullsized":"2024-05-27T09:31:57.863415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:01:30.075710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863490+0000","last_change":"2024-05-27T09:31:20.879086+0000","last_active":"2024-05-27T09:31:57.863490+0000","last_peered":"2024-05-27T09:31:57.863490+0000","last_clean":"2024-05-27T09:31:57.863490+0000","last_became_active":"2024-05-27T09:31:20.878331+0000","last_became_peered":"2024-05-27T09:31:20.878331+0000","last_unstale":"2024-05-27T09:31:57.863490+0000","last_undegraded":"2024-05-27T09:31:57.863490+0000","last_fullsized":"2024-05-27T09:31:57.863490+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:31.169812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863697+0000","last_change":"2024-05-27T09:31:24.895069+0000","last_active":"2024-05-27T09:31:57.863697+0000","last_peered":"2024-05-27T09:31:57.863697+0000","last_clean":"2024-05-27T09:31:57.863697+0000","last_became_active":"2024-05-27T09:31:24.894626+0000","last_became_peered":"2024-05-27T09:31:24.894626+0000","last_unstale":"2024-05-27T09:31:57.863697+0000","last_undegraded":"2024-05-27T09:31:57.863697+0000","last_fullsized":"2024-05-27T09:31:57.863697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:21:22.725268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864050+0000","last_change":"2024-05-27T09:31:20.877416+0000","last_active":"2024-05-27T09:31:57.864050+0000","last_peered":"2024-05-27T09:31:57.864050+0000","last_clean":"2024-05-27T09:31:57.864050+0000","last_became_active":"2024-05-27T09:31:20.876093+0000","last_became_peered":"2024-05-27T09:31:20.876093+0000","last_unstale":"2024-05-27T09:31:57.864050+0000","last_undegraded":"2024-05-27T09:31:57.864050+0000","last_fullsized":"2024-05-27T09:31:57.864050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:34.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861332+0000","last_change":"2024-05-27T09:31:24.898430+0000","last_active":"2024-05-27T09:31:57.861332+0000","last_peered":"2024-05-27T09:31:57.861332+0000","last_clean":"2024-05-27T09:31:57.861332+0000","last_became_active":"2024-05-27T09:31:24.897643+0000","last_became_peered":"2024-05-27T09:31:24.897643+0000","last_unstale":"2024-05-27T09:31:57.861332+0000","last_undegraded":"2024-05-27T09:31:57.861332+0000","last_fullsized":"2024-05-27T09:31:57.861332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:44.485561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863733+0000","last_change":"2024-05-27T09:31:20.858253+0000","last_active":"2024-05-27T09:31:57.863733+0000","last_peered":"2024-05-27T09:31:57.863733+0000","last_clean":"2024-05-27T09:31:57.863733+0000","last_became_active":"2024-05-27T09:31:20.857736+0000","last_became_peered":"2024-05-27T09:31:20.857736+0000","last_unstale":"2024-05-27T09:31:57.863733+0000","last_undegraded":"2024-05-27T09:31:57.863733+0000","last_fullsized":"2024-05-27T09:31:57.863733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:53:48.301763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861178+0000","last_change":"2024-05-27T09:31:24.912788+0000","last_active":"2024-05-27T09:31:57.861178+0000","last_peered":"2024-05-27T09:31:57.861178+0000","last_clean":"2024-05-27T09:31:57.861178+0000","last_became_active":"2024-05-27T09:31:24.912366+0000","last_became_peered":"2024-05-27T09:31:24.912366+0000","last_unstale":"2024-05-27T09:31:57.861178+0000","last_undegraded":"2024-05-27T09:31:57.861178+0000","last_fullsized":"2024-05-27T09:31:57.861178+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:55:03.397578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863841+0000","last_change":"2024-05-27T09:31:20.862188+0000","last_active":"2024-05-27T09:31:57.863841+0000","last_peered":"2024-05-27T09:31:57.863841+0000","last_clean":"2024-05-27T09:31:57.863841+0000","last_became_active":"2024-05-27T09:31:20.861791+0000","last_became_peered":"2024-05-27T09:31:20.861791+0000","last_unstale":"2024-05-27T09:31:57.863841+0000","last_undegraded":"2024-05-27T09:31:57.863841+0000","last_fullsized":"2024-05-27T09:31:57.863841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:16:09.127223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861144+0000","last_change":"2024-05-27T09:31:24.898460+0000","last_active":"2024-05-27T09:31:57.861144+0000","last_peered":"2024-05-27T09:31:57.861144+0000","last_clean":"2024-05-27T09:31:57.861144+0000","last_became_active":"2024-05-27T09:31:24.896787+0000","last_became_peered":"2024-05-27T09:31:24.896787+0000","last_unstale":"2024-05-27T09:31:57.861144+0000","last_undegraded":"2024-05-27T09:31:57.861144+0000","last_fullsized":"2024-05-27T09:31:57.861144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:10.199227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864271+0000","last_change":"2024-05-27T09:31:24.896323+0000","last_active":"2024-05-27T09:31:57.864271+0000","last_peered":"2024-05-27T09:31:57.864271+0000","last_clean":"2024-05-27T09:31:57.864271+0000","last_became_active":"2024-05-27T09:31:24.895919+0000","last_became_peered":"2024-05-27T09:31:24.895919+0000","last_unstale":"2024-05-27T09:31:57.864271+0000","last_undegraded":"2024-05-27T09:31:57.864271+0000","last_fullsized":"2024-05-27T09:31:57.864271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:04.883166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'4","reported_seq":64,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:18.848451+0000","last_change":"2024-05-27T09:31:20.862126+0000","last_active":"2024-05-27T09:32:18.848451+0000","last_peered":"2024-05-27T09:32:18.848451+0000","last_clean":"2024-05-27T09:32:18.848451+0000","last_became_active":"2024-05-27T09:31:20.858964+0000","last_became_peered":"2024-05-27T09:31:20.858964+0000","last_unstale":"2024-05-27T09:32:18.848451+0000","last_undegraded":"2024-05-27T09:32:18.848451+0000","last_fullsized":"2024-05-27T09:32:18.848451+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:51:26.838559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.193948+0000","last_change":"2024-05-27T09:31:24.905549+0000","last_active":"2024-05-27T09:31:58.193948+0000","last_peered":"2024-05-27T09:31:58.193948+0000","last_clean":"2024-05-27T09:31:58.193948+0000","last_became_active":"2024-05-27T09:31:24.905212+0000","last_became_peered":"2024-05-27T09:31:24.905212+0000","last_unstale":"2024-05-27T09:31:58.193948+0000","last_undegraded":"2024-05-27T09:31:58.193948+0000","last_fullsized":"2024-05-27T09:31:58.193948+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:18.267807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865689+0000","last_change":"2024-05-27T09:31:20.848527+0000","last_active":"2024-05-27T09:31:57.865689+0000","last_peered":"2024-05-27T09:31:57.865689+0000","last_clean":"2024-05-27T09:31:57.865689+0000","last_became_active":"2024-05-27T09:31:20.847779+0000","last_became_peered":"2024-05-27T09:31:20.847779+0000","last_unstale":"2024-05-27T09:31:57.865689+0000","last_undegraded":"2024-05-27T09:31:57.865689+0000","last_fullsized":"2024-05-27T09:31:57.865689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:15:01.131306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":102,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863868+0000","last_change":"2024-05-27T09:29:46.120380+0000","last_active":"2024-05-27T09:31:57.863868+0000","last_peered":"2024-05-27T09:31:57.863868+0000","last_clean":"2024-05-27T09:31:57.863868+0000","last_became_active":"2024-05-27T09:29:46.113658+0000","last_became_peered":"2024-05-27T09:29:46.113658+0000","last_unstale":"2024-05-27T09:31:57.863868+0000","last_undegraded":"2024-05-27T09:31:57.863868+0000","last_fullsized":"2024-05-27T09:31:57.863868+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_clean_scrub_stamp":"2024-05-27T09:25:47.244521+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:28:40.424824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.196069+0000","last_change":"2024-05-27T09:31:24.900736+0000","last_active":"2024-05-27T09:31:58.196069+0000","last_peered":"2024-05-27T09:31:58.196069+0000","last_clean":"2024-05-27T09:31:58.196069+0000","last_became_active":"2024-05-27T09:31:24.900354+0000","last_became_peered":"2024-05-27T09:31:24.900354+0000","last_unstale":"2024-05-27T09:31:58.196069+0000","last_undegraded":"2024-05-27T09:31:58.196069+0000","last_fullsized":"2024-05-27T09:31:58.196069+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:39:33.171977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.874220+0000","last_change":"2024-05-27T09:31:57.873603+0000","last_active":"2024-05-27T09:31:57.874220+0000","last_peered":"2024-05-27T09:31:57.874220+0000","last_clean":"2024-05-27T09:31:57.874220+0000","last_became_active":"2024-05-27T09:31:57.868616+0000","last_became_peered":"2024-05-27T09:31:57.868616+0000","last_unstale":"2024-05-27T09:31:57.874220+0000","last_undegraded":"2024-05-27T09:31:57.874220+0000","last_fullsized":"2024-05-27T09:31:57.874220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:02:20.112667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864526+0000","last_change":"2024-05-27T09:31:20.878557+0000","last_active":"2024-05-27T09:31:57.864526+0000","last_peered":"2024-05-27T09:31:57.864526+0000","last_clean":"2024-05-27T09:31:57.864526+0000","last_became_active":"2024-05-27T09:31:20.877969+0000","last_became_peered":"2024-05-27T09:31:20.877969+0000","last_unstale":"2024-05-27T09:31:57.864526+0000","last_undegraded":"2024-05-27T09:31:57.864526+0000","last_fullsized":"2024-05-27T09:31:57.864526+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:46.859645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197143+0000","last_change":"2024-05-27T09:31:24.907906+0000","last_active":"2024-05-27T09:31:58.197143+0000","last_peered":"2024-05-27T09:31:58.197143+0000","last_clean":"2024-05-27T09:31:58.197143+0000","last_became_active":"2024-05-27T09:31:24.907473+0000","last_became_peered":"2024-05-27T09:31:24.907473+0000","last_unstale":"2024-05-27T09:31:58.197143+0000","last_undegraded":"2024-05-27T09:31:58.197143+0000","last_fullsized":"2024-05-27T09:31:58.197143+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:11:38.536919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'11","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:32:11.947034+0000","last_change":"2024-05-27T09:31:20.881250+0000","last_active":"2024-05-27T09:32:11.947034+0000","last_peered":"2024-05-27T09:32:11.947034+0000","last_clean":"2024-05-27T09:32:11.947034+0000","last_became_active":"2024-05-27T09:31:20.880550+0000","last_became_peered":"2024-05-27T09:31:20.880550+0000","last_unstale":"2024-05-27T09:32:11.947034+0000","last_undegraded":"2024-05-27T09:32:11.947034+0000","last_fullsized":"2024-05-27T09:32:11.947034+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:14.783260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.195493+0000","last_change":"2024-05-27T09:31:24.894554+0000","last_active":"2024-05-27T09:31:58.195493+0000","last_peered":"2024-05-27T09:31:58.195493+0000","last_clean":"2024-05-27T09:31:58.195493+0000","last_became_active":"2024-05-27T09:31:24.893800+0000","last_became_peered":"2024-05-27T09:31:24.893800+0000","last_unstale":"2024-05-27T09:31:58.195493+0000","last_undegraded":"2024-05-27T09:31:58.195493+0000","last_fullsized":"2024-05-27T09:31:58.195493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:13.368892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865447+0000","last_change":"2024-05-27T09:31:21.054476+0000","last_active":"2024-05-27T09:31:57.865447+0000","last_peered":"2024-05-27T09:31:57.865447+0000","last_clean":"2024-05-27T09:31:57.865447+0000","last_became_active":"2024-05-27T09:31:21.054091+0000","last_became_peered":"2024-05-27T09:31:21.054091+0000","last_unstale":"2024-05-27T09:31:57.865447+0000","last_undegraded":"2024-05-27T09:31:57.865447+0000","last_fullsized":"2024-05-27T09:31:57.865447+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:15.909230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862690+0000","last_change":"2024-05-27T09:31:24.913157+0000","last_active":"2024-05-27T09:31:57.862690+0000","last_peered":"2024-05-27T09:31:57.862690+0000","last_clean":"2024-05-27T09:31:57.862690+0000","last_became_active":"2024-05-27T09:31:24.912793+0000","last_became_peered":"2024-05-27T09:31:24.912793+0000","last_unstale":"2024-05-27T09:31:57.862690+0000","last_undegraded":"2024-05-27T09:31:57.862690+0000","last_fullsized":"2024-05-27T09:31:57.862690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:17:36.451069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863635+0000","last_change":"2024-05-27T09:31:20.861349+0000","last_active":"2024-05-27T09:31:57.863635+0000","last_peered":"2024-05-27T09:31:57.863635+0000","last_clean":"2024-05-27T09:31:57.863635+0000","last_became_active":"2024-05-27T09:31:20.860896+0000","last_became_peered":"2024-05-27T09:31:20.860896+0000","last_unstale":"2024-05-27T09:31:57.863635+0000","last_undegraded":"2024-05-27T09:31:57.863635+0000","last_fullsized":"2024-05-27T09:31:57.863635+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:00.772910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863616+0000","last_change":"2024-05-27T09:31:24.893240+0000","last_active":"2024-05-27T09:31:57.863616+0000","last_peered":"2024-05-27T09:31:57.863616+0000","last_clean":"2024-05-27T09:31:57.863616+0000","last_became_active":"2024-05-27T09:31:24.892169+0000","last_became_peered":"2024-05-27T09:31:24.892169+0000","last_unstale":"2024-05-27T09:31:57.863616+0000","last_undegraded":"2024-05-27T09:31:57.863616+0000","last_fullsized":"2024-05-27T09:31:57.863616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:26:59.660725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.193440+0000","last_change":"2024-05-27T09:31:20.864307+0000","last_active":"2024-05-27T09:31:58.193440+0000","last_peered":"2024-05-27T09:31:58.193440+0000","last_clean":"2024-05-27T09:31:58.193440+0000","last_became_active":"2024-05-27T09:31:20.863913+0000","last_became_peered":"2024-05-27T09:31:20.863913+0000","last_unstale":"2024-05-27T09:31:58.193440+0000","last_undegraded":"2024-05-27T09:31:58.193440+0000","last_fullsized":"2024-05-27T09:31:58.193440+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:58:57.656731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.860843+0000","last_change":"2024-05-27T09:31:24.905728+0000","last_active":"2024-05-27T09:31:57.860843+0000","last_peered":"2024-05-27T09:31:57.860843+0000","last_clean":"2024-05-27T09:31:57.860843+0000","last_became_active":"2024-05-27T09:31:24.905365+0000","last_became_peered":"2024-05-27T09:31:24.905365+0000","last_unstale":"2024-05-27T09:31:57.860843+0000","last_undegraded":"2024-05-27T09:31:57.860843+0000","last_fullsized":"2024-05-27T09:31:57.860843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:05.013037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.866878+0000","last_change":"2024-05-27T09:31:20.878478+0000","last_active":"2024-05-27T09:31:57.866878+0000","last_peered":"2024-05-27T09:31:57.866878+0000","last_clean":"2024-05-27T09:31:57.866878+0000","last_became_active":"2024-05-27T09:31:20.877952+0000","last_became_peered":"2024-05-27T09:31:20.877952+0000","last_unstale":"2024-05-27T09:31:57.866878+0000","last_undegraded":"2024-05-27T09:31:57.866878+0000","last_fullsized":"2024-05-27T09:31:57.866878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:16:27.580641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.194183+0000","last_change":"2024-05-27T09:31:24.904967+0000","last_active":"2024-05-27T09:31:58.194183+0000","last_peered":"2024-05-27T09:31:58.194183+0000","last_clean":"2024-05-27T09:31:58.194183+0000","last_became_active":"2024-05-27T09:31:24.904542+0000","last_became_peered":"2024-05-27T09:31:24.904542+0000","last_unstale":"2024-05-27T09:31:58.194183+0000","last_undegraded":"2024-05-27T09:31:58.194183+0000","last_fullsized":"2024-05-27T09:31:58.194183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:01:05.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.875713+0000","last_change":"2024-05-27T09:31:57.875713+0000","last_active":"2024-05-27T09:31:57.875713+0000","last_peered":"2024-05-27T09:31:57.875713+0000","last_clean":"2024-05-27T09:31:57.875713+0000","last_became_active":"2024-05-27T09:31:57.870244+0000","last_became_peered":"2024-05-27T09:31:57.870244+0000","last_unstale":"2024-05-27T09:31:57.875713+0000","last_undegraded":"2024-05-27T09:31:57.875713+0000","last_fullsized":"2024-05-27T09:31:57.875713+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:49.272418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864004+0000","last_change":"2024-05-27T09:31:24.892655+0000","last_active":"2024-05-27T09:31:57.864004+0000","last_peered":"2024-05-27T09:31:57.864004+0000","last_clean":"2024-05-27T09:31:57.864004+0000","last_became_active":"2024-05-27T09:31:24.892112+0000","last_became_peered":"2024-05-27T09:31:24.892112+0000","last_unstale":"2024-05-27T09:31:57.864004+0000","last_undegraded":"2024-05-27T09:31:57.864004+0000","last_fullsized":"2024-05-27T09:31:57.864004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.604118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.876559+0000","last_change":"2024-05-27T09:31:57.875213+0000","last_active":"2024-05-27T09:31:57.876559+0000","last_peered":"2024-05-27T09:31:57.876559+0000","last_clean":"2024-05-27T09:31:57.876559+0000","last_became_active":"2024-05-27T09:31:57.869025+0000","last_became_peered":"2024-05-27T09:31:57.869025+0000","last_unstale":"2024-05-27T09:31:57.876559+0000","last_undegraded":"2024-05-27T09:31:57.876559+0000","last_fullsized":"2024-05-27T09:31:57.876559+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:51.433347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864693+0000","last_change":"2024-05-27T09:31:24.899130+0000","last_active":"2024-05-27T09:31:57.864693+0000","last_peered":"2024-05-27T09:31:57.864693+0000","last_clean":"2024-05-27T09:31:57.864693+0000","last_became_active":"2024-05-27T09:31:24.898722+0000","last_became_peered":"2024-05-27T09:31:24.898722+0000","last_unstale":"2024-05-27T09:31:57.864693+0000","last_undegraded":"2024-05-27T09:31:57.864693+0000","last_fullsized":"2024-05-27T09:31:57.864693+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:51.247275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864231+0000","last_change":"2024-05-27T09:31:20.863757+0000","last_active":"2024-05-27T09:31:57.864231+0000","last_peered":"2024-05-27T09:31:57.864231+0000","last_clean":"2024-05-27T09:31:57.864231+0000","last_became_active":"2024-05-27T09:31:20.863465+0000","last_became_peered":"2024-05-27T09:31:20.863465+0000","last_unstale":"2024-05-27T09:31:57.864231+0000","last_undegraded":"2024-05-27T09:31:57.864231+0000","last_fullsized":"2024-05-27T09:31:57.864231+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:21:45.741822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864260+0000","last_change":"2024-05-27T09:31:24.913749+0000","last_active":"2024-05-27T09:31:57.864260+0000","last_peered":"2024-05-27T09:31:57.864260+0000","last_clean":"2024-05-27T09:31:57.864260+0000","last_became_active":"2024-05-27T09:31:24.913384+0000","last_became_peered":"2024-05-27T09:31:24.913384+0000","last_unstale":"2024-05-27T09:31:57.864260+0000","last_undegraded":"2024-05-27T09:31:57.864260+0000","last_fullsized":"2024-05-27T09:31:57.864260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:24:29.826096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197151+0000","last_change":"2024-05-27T09:31:20.865844+0000","last_active":"2024-05-27T09:31:58.197151+0000","last_peered":"2024-05-27T09:31:58.197151+0000","last_clean":"2024-05-27T09:31:58.197151+0000","last_became_active":"2024-05-27T09:31:20.865364+0000","last_became_peered":"2024-05-27T09:31:20.865364+0000","last_unstale":"2024-05-27T09:31:58.197151+0000","last_undegraded":"2024-05-27T09:31:58.197151+0000","last_fullsized":"2024-05-27T09:31:58.197151+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:11:54.451294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865082+0000","last_change":"2024-05-27T09:31:24.890326+0000","last_active":"2024-05-27T09:31:57.865082+0000","last_peered":"2024-05-27T09:31:57.865082+0000","last_clean":"2024-05-27T09:31:57.865082+0000","last_became_active":"2024-05-27T09:31:24.889789+0000","last_became_peered":"2024-05-27T09:31:24.889789+0000","last_unstale":"2024-05-27T09:31:57.865082+0000","last_undegraded":"2024-05-27T09:31:57.865082+0000","last_fullsized":"2024-05-27T09:31:57.865082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:34:04.252231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.860984+0000","last_change":"2024-05-27T09:31:24.896730+0000","last_active":"2024-05-27T09:31:57.860984+0000","last_peered":"2024-05-27T09:31:57.860984+0000","last_clean":"2024-05-27T09:31:57.860984+0000","last_became_active":"2024-05-27T09:31:24.896365+0000","last_became_peered":"2024-05-27T09:31:24.896365+0000","last_unstale":"2024-05-27T09:31:57.860984+0000","last_undegraded":"2024-05-27T09:31:57.860984+0000","last_fullsized":"2024-05-27T09:31:57.860984+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:37.443788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.866594+0000","last_change":"2024-05-27T09:31:24.899215+0000","last_active":"2024-05-27T09:31:57.866594+0000","last_peered":"2024-05-27T09:31:57.866594+0000","last_clean":"2024-05-27T09:31:57.866594+0000","last_became_active":"2024-05-27T09:31:24.898733+0000","last_became_peered":"2024-05-27T09:31:24.898733+0000","last_unstale":"2024-05-27T09:31:57.866594+0000","last_undegraded":"2024-05-27T09:31:57.866594+0000","last_fullsized":"2024-05-27T09:31:57.866594+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:10.085989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.866737+0000","last_change":"2024-05-27T09:31:24.914922+0000","last_active":"2024-05-27T09:31:57.866737+0000","last_peered":"2024-05-27T09:31:57.866737+0000","last_clean":"2024-05-27T09:31:57.866737+0000","last_became_active":"2024-05-27T09:31:24.914541+0000","last_became_peered":"2024-05-27T09:31:24.914541+0000","last_unstale":"2024-05-27T09:31:57.866737+0000","last_undegraded":"2024-05-27T09:31:57.866737+0000","last_fullsized":"2024-05-27T09:31:57.866737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:30:38.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.862333+0000","last_change":"2024-05-27T09:31:24.909628+0000","last_active":"2024-05-27T09:31:57.862333+0000","last_peered":"2024-05-27T09:31:57.862333+0000","last_clean":"2024-05-27T09:31:57.862333+0000","last_became_active":"2024-05-27T09:31:24.902931+0000","last_became_peered":"2024-05-27T09:31:24.902931+0000","last_unstale":"2024-05-27T09:31:57.862333+0000","last_undegraded":"2024-05-27T09:31:57.862333+0000","last_fullsized":"2024-05-27T09:31:57.862333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:06.711316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197466+0000","last_change":"2024-05-27T09:31:24.912099+0000","last_active":"2024-05-27T09:31:58.197466+0000","last_peered":"2024-05-27T09:31:58.197466+0000","last_clean":"2024-05-27T09:31:58.197466+0000","last_became_active":"2024-05-27T09:31:24.911669+0000","last_became_peered":"2024-05-27T09:31:24.911669+0000","last_unstale":"2024-05-27T09:31:58.197466+0000","last_undegraded":"2024-05-27T09:31:58.197466+0000","last_fullsized":"2024-05-27T09:31:58.197466+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:58.068589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861115+0000","last_change":"2024-05-27T09:31:24.891426+0000","last_active":"2024-05-27T09:31:57.861115+0000","last_peered":"2024-05-27T09:31:57.861115+0000","last_clean":"2024-05-27T09:31:57.861115+0000","last_became_active":"2024-05-27T09:31:24.890877+0000","last_became_peered":"2024-05-27T09:31:24.890877+0000","last_unstale":"2024-05-27T09:31:57.861115+0000","last_undegraded":"2024-05-27T09:31:57.861115+0000","last_fullsized":"2024-05-27T09:31:57.861115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:15.338188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.867640+0000","last_change":"2024-05-27T09:31:24.892021+0000","last_active":"2024-05-27T09:31:57.867640+0000","last_peered":"2024-05-27T09:31:57.867640+0000","last_clean":"2024-05-27T09:31:57.867640+0000","last_became_active":"2024-05-27T09:31:24.891607+0000","last_became_peered":"2024-05-27T09:31:24.891607+0000","last_unstale":"2024-05-27T09:31:57.867640+0000","last_undegraded":"2024-05-27T09:31:57.867640+0000","last_fullsized":"2024-05-27T09:31:57.867640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:05.623199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.195340+0000","last_change":"2024-05-27T09:31:24.912166+0000","last_active":"2024-05-27T09:31:58.195340+0000","last_peered":"2024-05-27T09:31:58.195340+0000","last_clean":"2024-05-27T09:31:58.195340+0000","last_became_active":"2024-05-27T09:31:24.907555+0000","last_became_peered":"2024-05-27T09:31:24.907555+0000","last_unstale":"2024-05-27T09:31:58.195340+0000","last_undegraded":"2024-05-27T09:31:58.195340+0000","last_fullsized":"2024-05-27T09:31:58.195340+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:55:33.772321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864057+0000","last_change":"2024-05-27T09:31:24.910525+0000","last_active":"2024-05-27T09:31:57.864057+0000","last_peered":"2024-05-27T09:31:57.864057+0000","last_clean":"2024-05-27T09:31:57.864057+0000","last_became_active":"2024-05-27T09:31:24.910089+0000","last_became_peered":"2024-05-27T09:31:24.910089+0000","last_unstale":"2024-05-27T09:31:57.864057+0000","last_undegraded":"2024-05-27T09:31:57.864057+0000","last_fullsized":"2024-05-27T09:31:57.864057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:19:02.274781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.192977+0000","last_change":"2024-05-27T09:31:24.901125+0000","last_active":"2024-05-27T09:31:58.192977+0000","last_peered":"2024-05-27T09:31:58.192977+0000","last_clean":"2024-05-27T09:31:58.192977+0000","last_became_active":"2024-05-27T09:31:24.900361+0000","last_became_peered":"2024-05-27T09:31:24.900361+0000","last_unstale":"2024-05-27T09:31:58.192977+0000","last_undegraded":"2024-05-27T09:31:58.192977+0000","last_fullsized":"2024-05-27T09:31:58.192977+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:30.116349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864297+0000","last_change":"2024-05-27T09:31:24.917612+0000","last_active":"2024-05-27T09:31:57.864297+0000","last_peered":"2024-05-27T09:31:57.864297+0000","last_clean":"2024-05-27T09:31:57.864297+0000","last_became_active":"2024-05-27T09:31:24.917210+0000","last_became_peered":"2024-05-27T09:31:24.917210+0000","last_unstale":"2024-05-27T09:31:57.864297+0000","last_undegraded":"2024-05-27T09:31:57.864297+0000","last_fullsized":"2024-05-27T09:31:57.864297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:19:00.769114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197309+0000","last_change":"2024-05-27T09:31:24.909654+0000","last_active":"2024-05-27T09:31:58.197309+0000","last_peered":"2024-05-27T09:31:58.197309+0000","last_clean":"2024-05-27T09:31:58.197309+0000","last_became_active":"2024-05-27T09:31:24.909143+0000","last_became_peered":"2024-05-27T09:31:24.909143+0000","last_unstale":"2024-05-27T09:31:58.197309+0000","last_undegraded":"2024-05-27T09:31:58.197309+0000","last_fullsized":"2024-05-27T09:31:58.197309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:42.844614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.861202+0000","last_change":"2024-05-27T09:31:24.899031+0000","last_active":"2024-05-27T09:31:57.861202+0000","last_peered":"2024-05-27T09:31:57.861202+0000","last_clean":"2024-05-27T09:31:57.861202+0000","last_became_active":"2024-05-27T09:31:24.898664+0000","last_became_peered":"2024-05-27T09:31:24.898664+0000","last_unstale":"2024-05-27T09:31:57.861202+0000","last_undegraded":"2024-05-27T09:31:57.861202+0000","last_fullsized":"2024-05-27T09:31:57.861202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:09.659646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64739,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":248,"num_read_kb":192,"num_write":164,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":194217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678046,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36200,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708952,"statfs":{"total":95995035648,"available":95957966848,"internally_reserved":0,"allocated":876544,"data_stored":431801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874274,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36872,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708280,"statfs":{"total":95995035648,"available":95957278720,"internally_reserved":0,"allocated":1572864,"data_stored":1084020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070503,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":432037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266734,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":431957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462964,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1486848,"data_stored":1024278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626488,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36200,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708952,"statfs":{"total":95995035648,"available":95957966848,"internally_reserved":0,"allocated":880640,"data_stored":431426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":491566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":901120,"data_stored":432697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":492725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":431737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986245,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":888832,"data_stored":432836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822720,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1499136,"data_stored":1023571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":61506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T09:32:25.500 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-27T09:32:25.500 INFO:tasks.cephadm:Setting up client nodes... 2024-05-27T09:32:25.501 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-27T09:32:25.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:25 smithi044 ceph-mon[31618]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-05-27T09:32:25.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:25 smithi044 ceph-mon[31618]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:32:26.014 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:25 smithi050 ceph-mon[36371]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-05-27T09:32:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:25 smithi050 ceph-mon[36371]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:32:26.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:25 smithi189 ceph-mon[36671]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-05-27T09:32:26.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:25 smithi189 ceph-mon[36671]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:32:27.812 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:27 smithi044 ceph-mon[31618]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-27T09:32:28.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:27 smithi050 ceph-mon[36371]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-27T09:32:28.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:27 smithi189 ceph-mon[36671]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-27T09:32:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.402+0000 7f3786000700 1 -- 172.21.15.44:0/1647317285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378006cd10 msgr2=0x7f3780109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.402+0000 7f3786000700 1 --2- 172.21.15.44:0/1647317285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378006cd10 0x7f3780109270 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3770009990 tx=0x7f3770009ca0 comp rx=0 tx=0).stop 2024-05-27T09:32:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.402+0000 7f3786000700 1 -- 172.21.15.44:0/1647317285 shutdown_connections 2024-05-27T09:32:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.402+0000 7f3786000700 1 --2- 172.21.15.44:0/1647317285 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f378010c1f0 0x7f378010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.402+0000 7f3786000700 1 --2- 172.21.15.44:0/1647317285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378006cd10 0x7f3780109270 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.402+0000 7f3786000700 1 --2- 172.21.15.44:0/1647317285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37801097b0 0x7f378010bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.402+0000 7f3786000700 1 -- 172.21.15.44:0/1647317285 >> 172.21.15.44:0/1647317285 conn(0x7f37800fe860 msgr2=0x7f3780100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:28.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.402+0000 7f3786000700 1 -- 172.21.15.44:0/1647317285 shutdown_connections 2024-05-27T09:32:28.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.402+0000 7f3786000700 1 -- 172.21.15.44:0/1647317285 wait complete. 2024-05-27T09:32:28.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.403+0000 7f3786000700 1 Processor -- start 2024-05-27T09:32:28.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.404+0000 7f3786000700 1 -- start start 2024-05-27T09:32:28.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.404+0000 7f3786000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f378006cd10 0x7f378019cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:28.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.404+0000 7f3786000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37801097b0 0x7f378019d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:28.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.404+0000 7f3786000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378010c1f0 0x7f37801a4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:28.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.404+0000 7f3786000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3780111fb0 con 0x7f378010c1f0 2024-05-27T09:32:28.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.404+0000 7f3786000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3780111e30 con 0x7f37801097b0 2024-05-27T09:32:28.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.404+0000 7f3786000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3780112130 con 0x7f378006cd10 2024-05-27T09:32:28.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.404+0000 7f377fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378010c1f0 0x7f37801a4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:28.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.404+0000 7f377ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37801097b0 0x7f378019d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:28.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.404+0000 7f377fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378010c1f0 0x7f37801a4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46194/0 (socket says 172.21.15.44:46194) 2024-05-27T09:32:28.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.405+0000 7f377fe00700 1 -- 172.21.15.44:0/813169888 learned_addr learned my addr 172.21.15.44:0/813169888 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:28.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.405+0000 7f377f400700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f378006cd10 0x7f378019cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:28.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.405+0000 7f377ea00700 1 -- 172.21.15.44:0/813169888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f378006cd10 msgr2=0x7f378019cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.405+0000 7f377ea00700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f378006cd10 0x7f378019cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.405+0000 7f377ea00700 1 -- 172.21.15.44:0/813169888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378010c1f0 msgr2=0x7f37801a4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.405+0000 7f377ea00700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378010c1f0 0x7f37801a4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.405+0000 7f377ea00700 1 -- 172.21.15.44:0/813169888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37801a46b0 con 0x7f37801097b0 2024-05-27T09:32:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.405+0000 7f377f400700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f378006cd10 0x7f378019cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:32:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.405+0000 7f377ea00700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37801097b0 0x7f378019d090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f376c00e9f0 tx=0x7f376c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.406+0000 7f3777e00700 1 -- 172.21.15.44:0/813169888 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c00cbd0 con 0x7f37801097b0 2024-05-27T09:32:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.406+0000 7f3777e00700 1 -- 172.21.15.44:0/813169888 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f376c00cd70 con 0x7f37801097b0 2024-05-27T09:32:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.406+0000 7f3786000700 1 -- 172.21.15.44:0/813169888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37801a49a0 con 0x7f37801097b0 2024-05-27T09:32:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.406+0000 7f3777e00700 1 -- 172.21.15.44:0/813169888 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c018680 con 0x7f37801097b0 2024-05-27T09:32:28.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.406+0000 7f3786000700 1 -- 172.21.15.44:0/813169888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37801a4ee0 con 0x7f37801097b0 2024-05-27T09:32:28.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.407+0000 7f3786000700 1 -- 172.21.15.44:0/813169888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3760005360 con 0x7f37801097b0 2024-05-27T09:32:28.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.410+0000 7f3777e00700 1 -- 172.21.15.44:0/813169888 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f376c018820 con 0x7f37801097b0 2024-05-27T09:32:28.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.412+0000 7f3777e00700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f37640b2910 0x7f37640b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:28.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.412+0000 7f3777e00700 1 -- 172.21.15.44:0/813169888 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f376c0e29a0 con 0x7f37801097b0 2024-05-27T09:32:28.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.412+0000 7f377f400700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f37640b2910 0x7f37640b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:28.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.413+0000 7f377f400700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f37640b2910 0x7f37640b4e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f377000b2e0 tx=0x7f377000b680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.414+0000 7f3777e00700 1 -- 172.21.15.44:0/813169888 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f376c0aa510 con 0x7f37801097b0 2024-05-27T09:32:28.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.756+0000 7f3786000700 1 -- 172.21.15.44:0/813169888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f3760005650 con 0x7f37801097b0 2024-05-27T09:32:28.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.763+0000 7f3777e00700 1 -- 172.21.15.44:0/813169888 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f376c0a9e10 con 0x7f37801097b0 2024-05-27T09:32:28.764 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-05-27T09:32:28.765 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQAsU1RmPywzLRAA2T9mZafq8vO72lQGtDpKrw== 2024-05-27T09:32:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 -- 172.21.15.44:0/813169888 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f37640b2910 msgr2=0x7f37640b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f37640b2910 0x7f37640b4e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f377000b2e0 tx=0x7f377000b680 comp rx=0 tx=0).stop 2024-05-27T09:32:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 -- 172.21.15.44:0/813169888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37801097b0 msgr2=0x7f378019d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37801097b0 0x7f378019d090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f376c00e9f0 tx=0x7f376c00ed00 comp rx=0 tx=0).stop 2024-05-27T09:32:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 -- 172.21.15.44:0/813169888 shutdown_connections 2024-05-27T09:32:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f37640b2910 0x7f37640b4e20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f378010c1f0 0x7f37801a4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f378006cd10 0x7f378019cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 --2- 172.21.15.44:0/813169888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37801097b0 0x7f378019d090 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 -- 172.21.15.44:0/813169888 >> 172.21.15.44:0/813169888 conn(0x7f37800fe860 msgr2=0x7f378010a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 -- 172.21.15.44:0/813169888 shutdown_connections 2024-05-27T09:32:28.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:28.767+0000 7f3786000700 1 -- 172.21.15.44:0/813169888 wait complete. 2024-05-27T09:32:29.572 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:32:29.573 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-27T09:32:29.573 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-27T09:32:29.688 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-27T09:32:29.688 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-27T09:32:29.688 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-05-27T09:32:29.801 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:29 smithi044 ceph-mon[31618]: pgmap v357: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-27T09:32:29.802 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:29 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/813169888' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T09:32:29.802 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:29 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T09:32:29.802 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:29 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T09:32:30.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:29 smithi050 ceph-mon[36371]: pgmap v357: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-27T09:32:30.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:29 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/813169888' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T09:32:30.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:29 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T09:32:30.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:29 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T09:32:30.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:29 smithi189 ceph-mon[36671]: pgmap v357: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-27T09:32:30.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:29 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/813169888' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T09:32:30.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:29 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T09:32:30.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:29 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T09:32:30.178 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:31.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:31 smithi044 ceph-mon[31618]: pgmap v358: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-27T09:32:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:31 smithi050 ceph-mon[36371]: pgmap v358: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-27T09:32:32.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:31 smithi189 ceph-mon[36671]: pgmap v358: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-27T09:32:32.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.456+0000 7f1338800700 1 -- 172.21.15.44:0/1727679423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1330107de0 msgr2=0x7f13301081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:32.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.456+0000 7f1338800700 1 --2- 172.21.15.44:0/1727679423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1330107de0 0x7f13301081f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f13200099c0 tx=0x7f1320009cd0 comp rx=0 tx=0).stop 2024-05-27T09:32:32.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.456+0000 7f1338800700 1 -- 172.21.15.44:0/1727679423 shutdown_connections 2024-05-27T09:32:32.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.456+0000 7f1338800700 1 --2- 172.21.15.44:0/1727679423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1330108730 0x7f1330110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:32.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.456+0000 7f1338800700 1 --2- 172.21.15.44:0/1727679423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13301034b0 0x7f13301038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:32.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.456+0000 7f1338800700 1 --2- 172.21.15.44:0/1727679423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1330107de0 0x7f13301081f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:32.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.456+0000 7f1338800700 1 -- 172.21.15.44:0/1727679423 >> 172.21.15.44:0/1727679423 conn(0x7f13300fede0 msgr2=0x7f1330101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:32.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.456+0000 7f1338800700 1 -- 172.21.15.44:0/1727679423 shutdown_connections 2024-05-27T09:32:32.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.456+0000 7f1338800700 1 -- 172.21.15.44:0/1727679423 wait complete. 2024-05-27T09:32:32.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.457+0000 7f1338800700 1 Processor -- start 2024-05-27T09:32:32.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.458+0000 7f1338800700 1 -- start start 2024-05-27T09:32:32.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.458+0000 7f1338800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13301034b0 0x7f13300ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:32.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.458+0000 7f1338800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1330107de0 0x7f13300ff800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:32.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.458+0000 7f1338800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1330108730 0x7f13300ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:32.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.458+0000 7f1338800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1330113340 con 0x7f1330107de0 2024-05-27T09:32:32.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.458+0000 7f1338800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f13301131c0 con 0x7f13301034b0 2024-05-27T09:32:32.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.458+0000 7f1338800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f13301134c0 con 0x7f1330108730 2024-05-27T09:32:32.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.458+0000 7f1335400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1330107de0 0x7f13300ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:32.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.458+0000 7f1335e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13301034b0 0x7f13300ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:32.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.459+0000 7f1335e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13301034b0 0x7f13300ff2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:41528/0 (socket says 172.21.15.44:41528) 2024-05-27T09:32:32.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.459+0000 7f1335e00700 1 -- 172.21.15.44:0/3289971089 learned_addr learned my addr 172.21.15.44:0/3289971089 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:32.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.459+0000 7f1336800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1330108730 0x7f13300ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:32.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.459+0000 7f1335e00700 1 -- 172.21.15.44:0/3289971089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1330108730 msgr2=0x7f13300ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:32.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.459+0000 7f1335e00700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1330108730 0x7f13300ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:32.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.459+0000 7f1335e00700 1 -- 172.21.15.44:0/3289971089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1330107de0 msgr2=0x7f13300ff800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:32.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.459+0000 7f1335e00700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1330107de0 0x7f13300ff800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:32.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.459+0000 7f1335e00700 1 -- 172.21.15.44:0/3289971089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13301a91a0 con 0x7f13301034b0 2024-05-27T09:32:32.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.460+0000 7f1335e00700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13301034b0 0x7f13300ff2c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f132400e9a0 tx=0x7f132400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:32.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.460+0000 7f132aa00700 1 -- 172.21.15.44:0/3289971089 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132400ca10 con 0x7f13301034b0 2024-05-27T09:32:32.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.460+0000 7f132aa00700 1 -- 172.21.15.44:0/3289971089 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f132400cbb0 con 0x7f13301034b0 2024-05-27T09:32:32.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.460+0000 7f132aa00700 1 -- 172.21.15.44:0/3289971089 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324018680 con 0x7f13301034b0 2024-05-27T09:32:32.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.460+0000 7f1338800700 1 -- 172.21.15.44:0/3289971089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13301a9490 con 0x7f13301034b0 2024-05-27T09:32:32.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.460+0000 7f1338800700 1 -- 172.21.15.44:0/3289971089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13301a9950 con 0x7f13301034b0 2024-05-27T09:32:32.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.461+0000 7f1338800700 1 -- 172.21.15.44:0/3289971089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1330104960 con 0x7f13301034b0 2024-05-27T09:32:32.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.464+0000 7f132aa00700 1 -- 172.21.15.44:0/3289971089 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1324010430 con 0x7f13301034b0 2024-05-27T09:32:32.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.466+0000 7f132aa00700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f13180b2990 0x7f13180b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:32.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.466+0000 7f132aa00700 1 -- 172.21.15.44:0/3289971089 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f13240e2850 con 0x7f13301034b0 2024-05-27T09:32:32.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.466+0000 7f1335400700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f13180b2990 0x7f13180b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:32.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.467+0000 7f1335400700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f13180b2990 0x7f13180b4ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1320000c00 tx=0x7f132001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:32.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.468+0000 7f132aa00700 1 -- 172.21.15.44:0/3289971089 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f13240aa3c0 con 0x7f13301034b0 2024-05-27T09:32:32.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.772+0000 7f1338800700 1 -- 172.21.15.44:0/3289971089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f133002e5e0 con 0x7f13301034b0 2024-05-27T09:32:32.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.779+0000 7f132aa00700 1 -- 172.21.15.44:0/3289971089 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+288646 (secure 0 0 0) 0x7f13240a9cc0 con 0x7f13301034b0 2024-05-27T09:32:32.785 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:32:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.786+0000 7f1338800700 1 -- 172.21.15.44:0/3289971089 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f13180b2990 msgr2=0x7f13180b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.786+0000 7f1338800700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f13180b2990 0x7f13180b4ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1320000c00 tx=0x7f132001a040 comp rx=0 tx=0).stop 2024-05-27T09:32:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.786+0000 7f1338800700 1 -- 172.21.15.44:0/3289971089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13301034b0 msgr2=0x7f13300ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.786+0000 7f1338800700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13301034b0 0x7f13300ff2c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f132400e9a0 tx=0x7f132400ecb0 comp rx=0 tx=0).stop 2024-05-27T09:32:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.786+0000 7f1338800700 1 -- 172.21.15.44:0/3289971089 shutdown_connections 2024-05-27T09:32:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.786+0000 7f1338800700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f13180b2990 0x7f13180b4ea0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.786+0000 7f1338800700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1330108730 0x7f13300ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.786+0000 7f1338800700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13301034b0 0x7f13300ff2c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.786+0000 7f1338800700 1 --2- 172.21.15.44:0/3289971089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1330107de0 0x7f13300ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.786+0000 7f1338800700 1 -- 172.21.15.44:0/3289971089 >> 172.21.15.44:0/3289971089 conn(0x7f13300fede0 msgr2=0x7f1330075d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.786+0000 7f1338800700 1 -- 172.21.15.44:0/3289971089 shutdown_connections 2024-05-27T09:32:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:32.787+0000 7f1338800700 1 -- 172.21.15.44:0/3289971089 wait complete. 2024-05-27T09:32:33.626 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":4077502151},{"type":"v1","addr":"172.21.15.44:6801","nonce":4077502151}]},"active_addr":"172.21.15.44:6801/4077502151","active_change":"2024-05-27T09:22:56.451844+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.44:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":4102964244}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":3716545176}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":2278564126}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":408258823}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":1502422530}]}]} 2024-05-27T09:32:33.632 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-27T09:32:33.632 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-27T09:32:33.632 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T09:32:33.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:33 smithi044 ceph-mon[31618]: pgmap v359: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:33.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:33 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3289971089' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T09:32:34.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:33 smithi050 ceph-mon[36371]: pgmap v359: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:34.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:33 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3289971089' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T09:32:34.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:33 smithi189 ceph-mon[36671]: pgmap v359: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:34.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:33 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3289971089' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T09:32:34.138 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:36.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:35 smithi050 ceph-mon[36371]: pgmap v360: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:32:36.065 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:35 smithi044 ceph-mon[31618]: pgmap v360: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:32:36.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:35 smithi189 ceph-mon[36671]: pgmap v360: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:32:36.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.313+0000 7f4773200700 1 -- 172.21.15.44:0/2848472010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47741109a0 msgr2=0x7f4774112e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:36.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.313+0000 7f4773200700 1 --2- 172.21.15.44:0/2848472010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47741109a0 0x7f4774112e00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f476c00b0a0 tx=0x7f476c00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:32:36.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.313+0000 7f4773200700 1 -- 172.21.15.44:0/2848472010 shutdown_connections 2024-05-27T09:32:36.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.313+0000 7f4773200700 1 --2- 172.21.15.44:0/2848472010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47741109a0 0x7f4774112e00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:36.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.313+0000 7f4773200700 1 --2- 172.21.15.44:0/2848472010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4774075d20 0x7f4774076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:36.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.313+0000 7f4773200700 1 --2- 172.21.15.44:0/2848472010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4774076670 0x7f4774110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:36.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.313+0000 7f4773200700 1 -- 172.21.15.44:0/2848472010 >> 172.21.15.44:0/2848472010 conn(0x7f4774079d00 msgr2=0x7f477407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:36.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.313+0000 7f4773200700 1 -- 172.21.15.44:0/2848472010 shutdown_connections 2024-05-27T09:32:36.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.313+0000 7f4773200700 1 -- 172.21.15.44:0/2848472010 wait complete. 2024-05-27T09:32:36.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.315+0000 7f4773200700 1 Processor -- start 2024-05-27T09:32:36.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.315+0000 7f4773200700 1 -- start start 2024-05-27T09:32:36.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.315+0000 7f4773200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4774075d20 0x7f477419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:36.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.315+0000 7f4773200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4774076670 0x7f477419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:36.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.315+0000 7f4773200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47741109a0 0x7f47741a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:36.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.315+0000 7f4773200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f47741164d0 con 0x7f47741109a0 2024-05-27T09:32:36.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.315+0000 7f4773200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4774116350 con 0x7f4774076670 2024-05-27T09:32:36.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.315+0000 7f4773200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4774116650 con 0x7f4774075d20 2024-05-27T09:32:36.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f4771e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4774075d20 0x7f477419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:36.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f4772800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47741109a0 0x7f47741a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:36.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f4771e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4774075d20 0x7f477419cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34988/0 (socket says 172.21.15.44:34988) 2024-05-27T09:32:36.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f4771e00700 1 -- 172.21.15.44:0/3988226372 learned_addr learned my addr 172.21.15.44:0/3988226372 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:36.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f476be00700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4774076670 0x7f477419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:36.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f4771e00700 1 -- 172.21.15.44:0/3988226372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4774076670 msgr2=0x7f477419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:36.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f4771e00700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4774076670 0x7f477419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:36.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f4771e00700 1 -- 172.21.15.44:0/3988226372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47741109a0 msgr2=0x7f47741a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:36.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f4771e00700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47741109a0 0x7f47741a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:36.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f4771e00700 1 -- 172.21.15.44:0/3988226372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47741a1d90 con 0x7f4774075d20 2024-05-27T09:32:36.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f476be00700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4774076670 0x7f477419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:32:36.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.316+0000 7f4772800700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47741109a0 0x7f47741a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:32:36.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.317+0000 7f4771e00700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4774075d20 0x7f477419cec0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f476400eb30 tx=0x7f476400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:36.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.317+0000 7f476aa00700 1 -- 172.21.15.44:0/3988226372 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476400cbd0 con 0x7f4774075d20 2024-05-27T09:32:36.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.317+0000 7f476aa00700 1 -- 172.21.15.44:0/3988226372 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47640044d0 con 0x7f4774075d20 2024-05-27T09:32:36.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.317+0000 7f4773200700 1 -- 172.21.15.44:0/3988226372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47741a2080 con 0x7f4774075d20 2024-05-27T09:32:36.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.317+0000 7f476aa00700 1 -- 172.21.15.44:0/3988226372 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764003e30 con 0x7f4774075d20 2024-05-27T09:32:36.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.317+0000 7f4773200700 1 -- 172.21.15.44:0/3988226372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47741a99a0 con 0x7f4774075d20 2024-05-27T09:32:36.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.319+0000 7f4773200700 1 -- 172.21.15.44:0/3988226372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4774077290 con 0x7f4774075d20 2024-05-27T09:32:36.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.321+0000 7f476aa00700 1 -- 172.21.15.44:0/3988226372 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4764004000 con 0x7f4774075d20 2024-05-27T09:32:36.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.322+0000 7f476aa00700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f47580b2990 0x7f47580b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:36.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.323+0000 7f476aa00700 1 -- 172.21.15.44:0/3988226372 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f47640e26e0 con 0x7f4774075d20 2024-05-27T09:32:36.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.323+0000 7f476be00700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f47580b2990 0x7f47580b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:36.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.324+0000 7f476be00700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f47580b2990 0x7f47580b4ea0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f477407a210 tx=0x7f475c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:36.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.327+0000 7f476aa00700 1 -- 172.21.15.44:0/3988226372 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f47640042f0 con 0x7f4774075d20 2024-05-27T09:32:36.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.564+0000 7f4773200700 1 -- 172.21.15.44:0/3988226372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f477419e220 con 0x7f4774075d20 2024-05-27T09:32:36.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.566+0000 7f476aa00700 1 -- 172.21.15.44:0/3988226372 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22611 (secure 0 0 0) 0x7f47640aa250 con 0x7f4774075d20 2024-05-27T09:32:36.567 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:32:36.568 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":81,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:31:57.855852+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:32:36.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.569+0000 7f4773200700 1 -- 172.21.15.44:0/3988226372 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f47580b2990 msgr2=0x7f47580b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:36.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.569+0000 7f4773200700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f47580b2990 0x7f47580b4ea0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f477407a210 tx=0x7f475c009260 comp rx=0 tx=0).stop 2024-05-27T09:32:36.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.569+0000 7f4773200700 1 -- 172.21.15.44:0/3988226372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4774075d20 msgr2=0x7f477419cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:36.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.569+0000 7f4773200700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4774075d20 0x7f477419cec0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f476400eb30 tx=0x7f476400ee40 comp rx=0 tx=0).stop 2024-05-27T09:32:36.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.570+0000 7f4773200700 1 -- 172.21.15.44:0/3988226372 shutdown_connections 2024-05-27T09:32:36.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.570+0000 7f4773200700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f47580b2990 0x7f47580b4ea0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:36.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.570+0000 7f4773200700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47741109a0 0x7f47741a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:36.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.570+0000 7f4773200700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4774075d20 0x7f477419cec0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:36.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.570+0000 7f4773200700 1 --2- 172.21.15.44:0/3988226372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4774076670 0x7f477419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:36.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.570+0000 7f4773200700 1 -- 172.21.15.44:0/3988226372 >> 172.21.15.44:0/3988226372 conn(0x7f4774079d00 msgr2=0x7f4774102050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:36.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.570+0000 7f4773200700 1 -- 172.21.15.44:0/3988226372 shutdown_connections 2024-05-27T09:32:36.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:36.570+0000 7f4773200700 1 -- 172.21.15.44:0/3988226372 wait complete. 2024-05-27T09:32:36.938 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:36 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3988226372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:32:37.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:36 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3988226372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:32:37.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:36 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3988226372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:32:37.351 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-27T09:32:37.351 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T09:32:37.826 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:38.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:37 smithi050 ceph-mon[36371]: pgmap v361: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:32:38.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:37 smithi189 ceph-mon[36671]: pgmap v361: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:32:38.135 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:37 smithi044 ceph-mon[31618]: pgmap v361: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:32:39.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.911+0000 7fdb71e00700 1 -- 172.21.15.44:0/2386134135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb6c1034b0 msgr2=0x7fdb6c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:39.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.911+0000 7fdb71e00700 1 --2- 172.21.15.44:0/2386134135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb6c1034b0 0x7fdb6c1038c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c0099c0 tx=0x7fdb5c009cd0 comp rx=0 tx=0).stop 2024-05-27T09:32:39.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.911+0000 7fdb71e00700 1 -- 172.21.15.44:0/2386134135 shutdown_connections 2024-05-27T09:32:39.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.911+0000 7fdb71e00700 1 --2- 172.21.15.44:0/2386134135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c108730 0x7fdb6c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:39.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.911+0000 7fdb71e00700 1 --2- 172.21.15.44:0/2386134135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb6c1034b0 0x7fdb6c1038c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:39.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.911+0000 7fdb71e00700 1 --2- 172.21.15.44:0/2386134135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb6c107de0 0x7fdb6c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:39.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.911+0000 7fdb71e00700 1 -- 172.21.15.44:0/2386134135 >> 172.21.15.44:0/2386134135 conn(0x7fdb6c0fedb0 msgr2=0x7fdb6c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:39.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.911+0000 7fdb71e00700 1 -- 172.21.15.44:0/2386134135 shutdown_connections 2024-05-27T09:32:39.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.912+0000 7fdb71e00700 1 -- 172.21.15.44:0/2386134135 wait complete. 2024-05-27T09:32:39.914 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:39 smithi044 ceph-mon[31618]: pgmap v362: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:39.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.914+0000 7fdb71e00700 1 Processor -- start 2024-05-27T09:32:39.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.914+0000 7fdb71e00700 1 -- start start 2024-05-27T09:32:39.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.915+0000 7fdb71e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb6c1034b0 0x7fdb6c19adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:39.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.915+0000 7fdb71e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c107de0 0x7fdb6c19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:39.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.915+0000 7fdb71e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb6c108730 0x7fdb6c19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:39.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.915+0000 7fdb71e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c1133d0 con 0x7fdb6c107de0 2024-05-27T09:32:39.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.915+0000 7fdb71e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c113250 con 0x7fdb6c108730 2024-05-27T09:32:39.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.915+0000 7fdb71e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c113550 con 0x7fdb6c1034b0 2024-05-27T09:32:39.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.915+0000 7fdb6aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb6c1034b0 0x7fdb6c19adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:39.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.915+0000 7fdb62000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c107de0 0x7fdb6c19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:39.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.915+0000 7fdb6aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb6c1034b0 0x7fdb6c19adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34998/0 (socket says 172.21.15.44:34998) 2024-05-27T09:32:39.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.915+0000 7fdb6aa00700 1 -- 172.21.15.44:0/3736803492 learned_addr learned my addr 172.21.15.44:0/3736803492 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.915+0000 7fdb6b400700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb6c108730 0x7fdb6c19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.916+0000 7fdb6aa00700 1 -- 172.21.15.44:0/3736803492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb6c108730 msgr2=0x7fdb6c19f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.916+0000 7fdb6aa00700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb6c108730 0x7fdb6c19f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.916+0000 7fdb6aa00700 1 -- 172.21.15.44:0/3736803492 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c107de0 msgr2=0x7fdb6c19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.916+0000 7fdb6aa00700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c107de0 0x7fdb6c19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.916+0000 7fdb6aa00700 1 -- 172.21.15.44:0/3736803492 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6c19fcc0 con 0x7fdb6c1034b0 2024-05-27T09:32:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.916+0000 7fdb6b400700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb6c108730 0x7fdb6c19f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:32:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.916+0000 7fdb6aa00700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb6c1034b0 0x7fdb6c19adf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c005910 tx=0x7fdb5c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.916+0000 7fdb70a00700 1 -- 172.21.15.44:0/3736803492 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c026070 con 0x7fdb6c1034b0 2024-05-27T09:32:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.916+0000 7fdb70a00700 1 -- 172.21.15.44:0/3736803492 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb5c003c70 con 0x7fdb6c1034b0 2024-05-27T09:32:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.916+0000 7fdb70a00700 1 -- 172.21.15.44:0/3736803492 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c00fb50 con 0x7fdb6c1034b0 2024-05-27T09:32:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.916+0000 7fdb71e00700 1 -- 172.21.15.44:0/3736803492 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb6c19ff50 con 0x7fdb6c1034b0 2024-05-27T09:32:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.917+0000 7fdb71e00700 1 -- 172.21.15.44:0/3736803492 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb6c1a78d0 con 0x7fdb6c1034b0 2024-05-27T09:32:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.918+0000 7fdb71e00700 1 -- 172.21.15.44:0/3736803492 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb6c0512b0 con 0x7fdb6c1034b0 2024-05-27T09:32:39.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.920+0000 7fdb70a00700 1 -- 172.21.15.44:0/3736803492 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fdb5c029070 con 0x7fdb6c1034b0 2024-05-27T09:32:39.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.922+0000 7fdb70a00700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdb500b2990 0x7fdb500b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:39.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.922+0000 7fdb70a00700 1 -- 172.21.15.44:0/3736803492 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fdb5c0e33c0 con 0x7fdb6c1034b0 2024-05-27T09:32:39.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.922+0000 7fdb62000700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdb500b2990 0x7fdb500b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:39.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.923+0000 7fdb62000700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdb500b2990 0x7fdb500b4ea0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c101f90 tx=0x7fdb54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:39.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:39.924+0000 7fdb70a00700 1 -- 172.21.15.44:0/3736803492 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdb5c013070 con 0x7fdb6c1034b0 2024-05-27T09:32:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:39 smithi050 ceph-mon[36371]: pgmap v362: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:40.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:39 smithi189 ceph-mon[36671]: pgmap v362: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:40.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.161+0000 7fdb71e00700 1 -- 172.21.15.44:0/3736803492 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb6c02e5e0 con 0x7fdb6c1034b0 2024-05-27T09:32:40.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.164+0000 7fdb70a00700 1 -- 172.21.15.44:0/3736803492 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22611 (secure 0 0 0) 0x7fdb5c0aafc0 con 0x7fdb6c1034b0 2024-05-27T09:32:40.165 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:32:40.165 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":81,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:31:57.855852+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:32:40.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.167+0000 7fdb71e00700 1 -- 172.21.15.44:0/3736803492 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdb500b2990 msgr2=0x7fdb500b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.167+0000 7fdb71e00700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdb500b2990 0x7fdb500b4ea0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c101f90 tx=0x7fdb54009260 comp rx=0 tx=0).stop 2024-05-27T09:32:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.167+0000 7fdb71e00700 1 -- 172.21.15.44:0/3736803492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb6c1034b0 msgr2=0x7fdb6c19adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.167+0000 7fdb71e00700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb6c1034b0 0x7fdb6c19adf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c005910 tx=0x7fdb5c004270 comp rx=0 tx=0).stop 2024-05-27T09:32:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.167+0000 7fdb71e00700 1 -- 172.21.15.44:0/3736803492 shutdown_connections 2024-05-27T09:32:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.167+0000 7fdb71e00700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fdb500b2990 0x7fdb500b4ea0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:40.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.167+0000 7fdb71e00700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb6c108730 0x7fdb6c19f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:40.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.167+0000 7fdb71e00700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb6c1034b0 0x7fdb6c19adf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:40.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.167+0000 7fdb71e00700 1 --2- 172.21.15.44:0/3736803492 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb6c107de0 0x7fdb6c19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:40.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.167+0000 7fdb71e00700 1 -- 172.21.15.44:0/3736803492 >> 172.21.15.44:0/3736803492 conn(0x7fdb6c0fedb0 msgr2=0x7fdb6c106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:40.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.168+0000 7fdb71e00700 1 -- 172.21.15.44:0/3736803492 shutdown_connections 2024-05-27T09:32:40.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:40.168+0000 7fdb71e00700 1 -- 172.21.15.44:0/3736803492 wait complete. 2024-05-27T09:32:40.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:40 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3736803492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:32:41.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:40 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3736803492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:32:41.066 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-27T09:32:41.067 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-27T09:32:41.067 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-27T09:32:41.067 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-27T09:32:41.068 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-27T09:32:41.068 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-27T09:32:41.068 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-27T09:32:41.069 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-27T09:32:41.069 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-27T09:32:41.070 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-27T09:32:41.070 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-27T09:32:41.070 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-27T09:32:41.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:40 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3736803492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:32:42.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:41 smithi050 ceph-mon[36371]: pgmap v363: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:32:42.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:41 smithi189 ceph-mon[36671]: pgmap v363: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:32:42.104 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:42.168 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:42.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:41 smithi044 ceph-mon[31618]: pgmap v363: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:32:42.230 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:42.261 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:42.263 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:42.265 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:43 smithi044 ceph-mon[31618]: pgmap v364: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:44.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:43 smithi050 ceph-mon[36371]: pgmap v364: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:44.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:43 smithi189 ceph-mon[36671]: pgmap v364: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:44.115 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:44.117 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:46.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:45 smithi050 ceph-mon[36371]: pgmap v365: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:32:46.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:45 smithi044 ceph-mon[31618]: pgmap v365: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:32:46.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:45 smithi189 ceph-mon[36671]: pgmap v365: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:32:48.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:47 smithi050 ceph-mon[36371]: pgmap v366: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:32:48.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:47 smithi189 ceph-mon[36671]: pgmap v366: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:32:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:47 smithi044 ceph-mon[31618]: pgmap v366: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:32:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.771+0000 7ff142800700 1 -- 172.21.15.44:0/2086529279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff13c109860 msgr2=0x7ff13c1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.771+0000 7ff142800700 1 --2- 172.21.15.44:0/2086529279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff13c109860 0x7ff13c1111c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff134009990 tx=0x7ff134009ca0 comp rx=0 tx=0).stop 2024-05-27T09:32:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.771+0000 7ff142800700 1 -- 172.21.15.44:0/2086529279 shutdown_connections 2024-05-27T09:32:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.771+0000 7ff142800700 1 --2- 172.21.15.44:0/2086529279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff13c109860 0x7ff13c1111c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.771+0000 7ff142800700 1 --2- 172.21.15.44:0/2086529279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff13c1045e0 0x7ff13c1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.771+0000 7ff142800700 1 --2- 172.21.15.44:0/2086529279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c108f10 0x7ff13c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.771+0000 7ff142800700 1 -- 172.21.15.44:0/2086529279 >> 172.21.15.44:0/2086529279 conn(0x7ff13c0ffeb0 msgr2=0x7ff13c102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.771+0000 7ff142800700 1 -- 172.21.15.44:0/2086529279 shutdown_connections 2024-05-27T09:32:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.771+0000 7ff142800700 1 -- 172.21.15.44:0/2086529279 wait complete. 2024-05-27T09:32:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.773+0000 7ff142800700 1 Processor -- start 2024-05-27T09:32:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.773+0000 7ff142800700 1 -- start start 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.773+0000 7ff142800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff13c1045e0 0x7ff13c19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.773+0000 7ff142800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff13c108f10 0x7ff13c19d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.773+0000 7ff142800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c109860 0x7ff13c1a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.773+0000 7ff142800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c114490 con 0x7ff13c108f10 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.773+0000 7ff142800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c114310 con 0x7ff13c1045e0 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.773+0000 7ff142800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c114610 con 0x7ff13c109860 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff132a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff13c108f10 0x7ff13c19d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff13b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff13c1045e0 0x7ff13c19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff13be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c109860 0x7ff13c1a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff132a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff13c108f10 0x7ff13c19d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56542/0 (socket says 172.21.15.44:56542) 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff132a00700 1 -- 172.21.15.44:0/1753435072 learned_addr learned my addr 172.21.15.44:0/1753435072 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:48.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff13be00700 1 -- 172.21.15.44:0/1753435072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff13c1045e0 msgr2=0x7ff13c19cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff13be00700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff13c1045e0 0x7ff13c19cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff13be00700 1 -- 172.21.15.44:0/1753435072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff13c108f10 msgr2=0x7ff13c19d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff13be00700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff13c108f10 0x7ff13c19d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff13be00700 1 -- 172.21.15.44:0/1753435072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c1a1e90 con 0x7ff13c109860 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff132a00700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff13c108f10 0x7ff13c19d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.774+0000 7ff13be00700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c109860 0x7ff13c1a17f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff1340059d0 tx=0x7ff1340057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.775+0000 7ff138c00700 1 -- 172.21.15.44:0/1753435072 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff134026070 con 0x7ff13c109860 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.775+0000 7ff142800700 1 -- 172.21.15.44:0/1753435072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13c1a2120 con 0x7ff13c109860 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.775+0000 7ff138c00700 1 -- 172.21.15.44:0/1753435072 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff13400b610 con 0x7ff13c109860 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.775+0000 7ff138c00700 1 -- 172.21.15.44:0/1753435072 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1340173f0 con 0x7ff13c109860 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.775+0000 7ff142800700 1 -- 172.21.15.44:0/1753435072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff13c1a9aa0 con 0x7ff13c109860 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.776+0000 7ff142800700 1 -- 172.21.15.44:0/1753435072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff13c1080d0 con 0x7ff13c109860 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.779+0000 7ff138c00700 1 -- 172.21.15.44:0/1753435072 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff134029070 con 0x7ff13c109860 2024-05-27T09:32:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.780+0000 7ff138c00700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff1200b2910 0x7ff1200b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.781+0000 7ff138c00700 1 -- 172.21.15.44:0/1753435072 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7ff1340e3510 con 0x7ff13c109860 2024-05-27T09:32:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.781+0000 7ff13b400700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff1200b2910 0x7ff1200b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.781+0000 7ff138c00700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374] conn(0x7ff1200bc4a0 0x7ff1200be990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.781+0000 7ff138c00700 1 -- 172.21.15.44:0/1753435072 --> [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff13402ce80 con 0x7ff1200bc4a0 2024-05-27T09:32:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.781+0000 7ff138c00700 1 -- 172.21.15.44:0/1753435072 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7ff1340e38c0 con 0x7ff13c109860 2024-05-27T09:32:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.782+0000 7ff13b400700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff1200b2910 0x7ff1200b4e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00f540 tx=0x7ff12c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.782+0000 7ff132a00700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374] conn(0x7ff1200bc4a0 0x7ff1200be990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.782+0000 7ff132a00700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374] conn(0x7ff1200bc4a0 0x7ff1200be990 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.783+0000 7ff138c00700 1 -- 172.21.15.44:0/1753435072 <== osd.2 v2:172.21.15.189:6800/1488616374 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff13402ce80 con 0x7ff1200bc4a0 2024-05-27T09:32:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.802+0000 7ff142800700 1 -- 172.21.15.44:0/1753435072 --> [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff13c105a90 con 0x7ff1200bc4a0 2024-05-27T09:32:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.803+0000 7ff138c00700 1 -- 172.21.15.44:0/1753435072 <== osd.2 v2:172.21.15.189:6800/1488616374 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff13c105a90 con 0x7ff1200bc4a0 2024-05-27T09:32:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.804+0000 7ff142800700 1 -- 172.21.15.44:0/1753435072 >> [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374] conn(0x7ff1200bc4a0 msgr2=0x7ff1200be990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.804+0000 7ff142800700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374] conn(0x7ff1200bc4a0 0x7ff1200be990 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.804+0000 7ff142800700 1 -- 172.21.15.44:0/1753435072 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff1200b2910 msgr2=0x7ff1200b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.804+0000 7ff142800700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff1200b2910 0x7ff1200b4e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00f540 tx=0x7ff12c00b040 comp rx=0 tx=0).stop 2024-05-27T09:32:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.804+0000 7ff142800700 1 -- 172.21.15.44:0/1753435072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c109860 msgr2=0x7ff13c1a17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.804+0000 7ff142800700 1 --2- 172.21.15.44:0/1753435072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c109860 0x7ff13c1a17f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff1340059d0 tx=0x7ff1340057c0 comp rx=0 tx=0).stop 2024-05-27T09:32:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.804+0000 7ff13be00700 1 -- 172.21.15.44:0/1753435072 reap_dead start 2024-05-27T09:32:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.804+0000 7ff142800700 1 -- 172.21.15.44:0/1753435072 shutdown_connections 2024-05-27T09:32:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.804+0000 7ff142800700 1 -- 172.21.15.44:0/1753435072 >> 172.21.15.44:0/1753435072 conn(0x7ff13c0ffeb0 msgr2=0x7ff13c110e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.804+0000 7ff142800700 1 -- 172.21.15.44:0/1753435072 shutdown_connections 2024-05-27T09:32:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:47.804+0000 7ff142800700 1 -- 172.21.15.44:0/1753435072 wait complete. 2024-05-27T09:32:49.604 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:49.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.717+0000 7f16e8000700 1 -- 172.21.15.44:0/3554828125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16e0105e40 msgr2=0x7f16e010e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:49.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.717+0000 7f16e8000700 1 --2- 172.21.15.44:0/3554828125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16e0105e40 0x7f16e010e390 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009990 tx=0x7f16cc009ca0 comp rx=0 tx=0).stop 2024-05-27T09:32:49.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.717+0000 7f16e8000700 1 -- 172.21.15.44:0/3554828125 shutdown_connections 2024-05-27T09:32:49.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.717+0000 7f16e8000700 1 --2- 172.21.15.44:0/3554828125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16e010e8d0 0x7f16e0110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:49.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.717+0000 7f16e8000700 1 --2- 172.21.15.44:0/3554828125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e01054f0 0x7f16e0105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:49.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.717+0000 7f16e8000700 1 --2- 172.21.15.44:0/3554828125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16e0105e40 0x7f16e010e390 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:49.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.717+0000 7f16e8000700 1 -- 172.21.15.44:0/3554828125 >> 172.21.15.44:0/3554828125 conn(0x7f16e00fedb0 msgr2=0x7f16e01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:49.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.717+0000 7f16e8000700 1 -- 172.21.15.44:0/3554828125 shutdown_connections 2024-05-27T09:32:49.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.718+0000 7f16e8000700 1 -- 172.21.15.44:0/3554828125 wait complete. 2024-05-27T09:32:49.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.719+0000 7f16e8000700 1 Processor -- start 2024-05-27T09:32:49.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.719+0000 7f16e8000700 1 -- start start 2024-05-27T09:32:49.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e8000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16e01054f0 0x7f16e019ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:49.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e8000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16e0105e40 0x7f16e019b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:49.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e8000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e010e8d0 0x7f16e019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:49.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e8000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f16e0114480 con 0x7f16e0105e40 2024-05-27T09:32:49.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e8000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f16e0114300 con 0x7f16e01054f0 2024-05-27T09:32:49.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e8000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f16e0114600 con 0x7f16e010e8d0 2024-05-27T09:32:49.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e5600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16e01054f0 0x7f16e019ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:49.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e5600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16e01054f0 0x7f16e019ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52182/0 (socket says 172.21.15.44:52182) 2024-05-27T09:32:49.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e5600700 1 -- 172.21.15.44:0/3016164451 learned_addr learned my addr 172.21.15.44:0/3016164451 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:49.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e6000700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e010e8d0 0x7f16e019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:49.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e4c00700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16e0105e40 0x7f16e019b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e5600700 1 -- 172.21.15.44:0/3016164451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e010e8d0 msgr2=0x7f16e019f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e5600700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e010e8d0 0x7f16e019f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e5600700 1 -- 172.21.15.44:0/3016164451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16e0105e40 msgr2=0x7f16e019b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e5600700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16e0105e40 0x7f16e019b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e5600700 1 -- 172.21.15.44:0/3016164451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e019fd40 con 0x7f16e01054f0 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.720+0000 7f16e4c00700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16e0105e40 0x7f16e019b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.721+0000 7f16e5600700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16e01054f0 0x7f16e019ae70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f16d400eb30 tx=0x7f16d400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.721+0000 7f16de000700 1 -- 172.21.15.44:0/3016164451 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d400cc90 con 0x7f16e01054f0 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.721+0000 7f16e8000700 1 -- 172.21.15.44:0/3016164451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16e01a0030 con 0x7f16e01054f0 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.721+0000 7f16de000700 1 -- 172.21.15.44:0/3016164451 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16d40044d0 con 0x7f16e01054f0 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.721+0000 7f16de000700 1 -- 172.21.15.44:0/3016164451 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d4010430 con 0x7f16e01054f0 2024-05-27T09:32:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.721+0000 7f16e8000700 1 -- 172.21.15.44:0/3016164451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16e01a7950 con 0x7f16e01054f0 2024-05-27T09:32:49.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.722+0000 7f16e8000700 1 -- 172.21.15.44:0/3016164451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f16e0063f40 con 0x7f16e01054f0 2024-05-27T09:32:49.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.725+0000 7f16de000700 1 -- 172.21.15.44:0/3016164451 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f16d4003ea0 con 0x7f16e01054f0 2024-05-27T09:32:49.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.727+0000 7f16de000700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f16c80b2990 0x7f16c80b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:49.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.727+0000 7f16de000700 1 -- 172.21.15.44:0/3016164451 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f16d40e2890 con 0x7f16e01054f0 2024-05-27T09:32:49.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.727+0000 7f16e4c00700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f16c80b2990 0x7f16c80b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:49.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.727+0000 7f16de000700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354] conn(0x7f16c80bc520 0x7f16c80bea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:49.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.727+0000 7f16de000700 1 -- 172.21.15.44:0/3016164451 --> [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16c80bf0e0 con 0x7f16c80bc520 2024-05-27T09:32:49.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.727+0000 7f16de000700 1 -- 172.21.15.44:0/3016164451 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f16d40e2c40 con 0x7f16e01054f0 2024-05-27T09:32:49.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.728+0000 7f16e4c00700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f16c80b2990 0x7f16c80b4ea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00b2e0 tx=0x7f16cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:49.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.728+0000 7f16e6000700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354] conn(0x7f16c80bc520 0x7f16c80bea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:49.729 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:49.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.729+0000 7f16e6000700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354] conn(0x7f16c80bc520 0x7f16c80bea10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:49.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.730+0000 7f16de000700 1 -- 172.21.15.44:0/3016164451 <== osd.1 v2:172.21.15.50:6800/1520188354 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16c80bf0e0 con 0x7f16c80bc520 2024-05-27T09:32:49.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.750+0000 7f16e8000700 1 -- 172.21.15.44:0/3016164451 --> [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f16e0064340 con 0x7f16c80bc520 2024-05-27T09:32:49.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.750+0000 7f16de000700 1 -- 172.21.15.44:0/3016164451 <== osd.1 v2:172.21.15.50:6800/1520188354 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f16e0064340 con 0x7f16c80bc520 2024-05-27T09:32:49.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.751+0000 7f16e8000700 1 -- 172.21.15.44:0/3016164451 >> [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354] conn(0x7f16c80bc520 msgr2=0x7f16c80bea10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:49.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.751+0000 7f16e8000700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354] conn(0x7f16c80bc520 0x7f16c80bea10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:49.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.751+0000 7f16e8000700 1 -- 172.21.15.44:0/3016164451 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f16c80b2990 msgr2=0x7f16c80b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:49.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.751+0000 7f16e8000700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f16c80b2990 0x7f16c80b4ea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00b2e0 tx=0x7f16cc005fb0 comp rx=0 tx=0).stop 2024-05-27T09:32:49.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.751+0000 7f16e8000700 1 -- 172.21.15.44:0/3016164451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16e01054f0 msgr2=0x7f16e019ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:49.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.751+0000 7f16e8000700 1 --2- 172.21.15.44:0/3016164451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16e01054f0 0x7f16e019ae70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f16d400eb30 tx=0x7f16d400ee40 comp rx=0 tx=0).stop 2024-05-27T09:32:49.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.751+0000 7f16e6000700 1 -- 172.21.15.44:0/3016164451 reap_dead start 2024-05-27T09:32:49.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.751+0000 7f16e8000700 1 -- 172.21.15.44:0/3016164451 shutdown_connections 2024-05-27T09:32:49.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.751+0000 7f16e8000700 1 -- 172.21.15.44:0/3016164451 >> 172.21.15.44:0/3016164451 conn(0x7f16e00fedb0 msgr2=0x7f16e00ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:49.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.751+0000 7f16e8000700 1 -- 172.21.15.44:0/3016164451 shutdown_connections 2024-05-27T09:32:49.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:49.752+0000 7f16e8000700 1 -- 172.21.15.44:0/3016164451 wait complete. 2024-05-27T09:32:49.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:49 smithi044 ceph-mon[31618]: pgmap v367: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:49 smithi050 ceph-mon[36371]: pgmap v367: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:50.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:49 smithi189 ceph-mon[36671]: pgmap v367: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:50.619 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:32:50.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.853+0000 7f6754600700 1 -- 172.21.15.44:0/1893111159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c103a50 msgr2=0x7f674c1106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:50.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.853+0000 7f6754600700 1 --2- 172.21.15.44:0/1893111159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c103a50 0x7f674c1106e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6738009990 tx=0x7f6738009ca0 comp rx=0 tx=0).stop 2024-05-27T09:32:50.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.853+0000 7f6754600700 1 -- 172.21.15.44:0/1893111159 shutdown_connections 2024-05-27T09:32:50.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.853+0000 7f6754600700 1 --2- 172.21.15.44:0/1893111159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f674c110c20 0x7f674c113040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:50.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.853+0000 7f6754600700 1 --2- 172.21.15.44:0/1893111159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f674c103070 0x7f674c103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:50.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.853+0000 7f6754600700 1 --2- 172.21.15.44:0/1893111159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c103a50 0x7f674c1106e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:50.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.853+0000 7f6754600700 1 -- 172.21.15.44:0/1893111159 >> 172.21.15.44:0/1893111159 conn(0x7f674c0fedb0 msgr2=0x7f674c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:50.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.853+0000 7f6754600700 1 -- 172.21.15.44:0/1893111159 shutdown_connections 2024-05-27T09:32:50.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.853+0000 7f6754600700 1 -- 172.21.15.44:0/1893111159 wait complete. 2024-05-27T09:32:50.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.854+0000 7f6754600700 1 Processor -- start 2024-05-27T09:32:50.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.854+0000 7f6754600700 1 -- start start 2024-05-27T09:32:50.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.855+0000 7f6754600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f674c103070 0x7f674c19d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:50.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.855+0000 7f6754600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c103a50 0x7f674c19d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:50.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.855+0000 7f6754600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f674c110c20 0x7f674c1a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:50.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.855+0000 7f6754600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f674c1166f0 con 0x7f674c103a50 2024-05-27T09:32:50.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.855+0000 7f6754600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f674c116570 con 0x7f674c103070 2024-05-27T09:32:50.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.855+0000 7f6754600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f674c116870 con 0x7f674c110c20 2024-05-27T09:32:50.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.855+0000 7f6751c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f674c103070 0x7f674c19d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:50.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.855+0000 7f6751200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c103a50 0x7f674c19d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:50.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.855+0000 7f6752600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f674c110c20 0x7f674c1a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.855+0000 7f6751c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f674c103070 0x7f674c19d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52204/0 (socket says 172.21.15.44:52204) 2024-05-27T09:32:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.855+0000 7f6751c00700 1 -- 172.21.15.44:0/4156560328 learned_addr learned my addr 172.21.15.44:0/4156560328 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.856+0000 7f6751c00700 1 -- 172.21.15.44:0/4156560328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f674c110c20 msgr2=0x7f674c1a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.856+0000 7f6751c00700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f674c110c20 0x7f674c1a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.856+0000 7f6751c00700 1 -- 172.21.15.44:0/4156560328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c103a50 msgr2=0x7f674c19d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.856+0000 7f6751c00700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c103a50 0x7f674c19d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.856+0000 7f6751c00700 1 -- 172.21.15.44:0/4156560328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c1a1f50 con 0x7f674c103070 2024-05-27T09:32:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.856+0000 7f6751c00700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f674c103070 0x7f674c19d080 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f674000e9a0 tx=0x7f674000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.856+0000 7f674a000700 1 -- 172.21.15.44:0/4156560328 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674000ca10 con 0x7f674c103070 2024-05-27T09:32:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.857+0000 7f674a000700 1 -- 172.21.15.44:0/4156560328 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f674000cbb0 con 0x7f674c103070 2024-05-27T09:32:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.857+0000 7f674a000700 1 -- 172.21.15.44:0/4156560328 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6740018680 con 0x7f674c103070 2024-05-27T09:32:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.857+0000 7f6754600700 1 -- 172.21.15.44:0/4156560328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674c1a2240 con 0x7f674c103070 2024-05-27T09:32:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.857+0000 7f6754600700 1 -- 172.21.15.44:0/4156560328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674c1a9b60 con 0x7f674c103070 2024-05-27T09:32:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.858+0000 7f6754600700 1 -- 172.21.15.44:0/4156560328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f674c063f40 con 0x7f674c103070 2024-05-27T09:32:50.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.860+0000 7f674a000700 1 -- 172.21.15.44:0/4156560328 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f6740010430 con 0x7f674c103070 2024-05-27T09:32:50.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.862+0000 7f674a000700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f67340b2990 0x7f67340b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:50.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.862+0000 7f674a000700 1 -- 172.21.15.44:0/4156560328 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f67400e2ad0 con 0x7f674c103070 2024-05-27T09:32:50.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.862+0000 7f6751200700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f67340b2990 0x7f67340b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:50.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.862+0000 7f674a000700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660] conn(0x7f67340bc520 0x7f67340bea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:50.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.862+0000 7f674a000700 1 -- 172.21.15.44:0/4156560328 --> [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67340bf0e0 con 0x7f67340bc520 2024-05-27T09:32:50.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.862+0000 7f674a000700 1 -- 172.21.15.44:0/4156560328 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f67400e2e80 con 0x7f674c103070 2024-05-27T09:32:50.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.863+0000 7f6752600700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660] conn(0x7f67340bc520 0x7f67340bea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:50.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.863+0000 7f6751200700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f67340b2990 0x7f67340b4ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f673800b2e0 tx=0x7f6738005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:50.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.864+0000 7f6752600700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660] conn(0x7f67340bc520 0x7f67340bea10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:50.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.864+0000 7f674a000700 1 -- 172.21.15.44:0/4156560328 <== osd.4 v2:172.21.15.50:6808/299847660 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67340bf0e0 con 0x7f67340bc520 2024-05-27T09:32:50.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.883+0000 7f6754600700 1 -- 172.21.15.44:0/4156560328 --> [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f674c064410 con 0x7f67340bc520 2024-05-27T09:32:50.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.884+0000 7f674a000700 1 -- 172.21.15.44:0/4156560328 <== osd.4 v2:172.21.15.50:6808/299847660 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f674c064410 con 0x7f67340bc520 2024-05-27T09:32:50.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.885+0000 7f6754600700 1 -- 172.21.15.44:0/4156560328 >> [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660] conn(0x7f67340bc520 msgr2=0x7f67340bea10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:50.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.885+0000 7f6754600700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660] conn(0x7f67340bc520 0x7f67340bea10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.885+0000 7f6754600700 1 -- 172.21.15.44:0/4156560328 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f67340b2990 msgr2=0x7f67340b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.885+0000 7f6754600700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f67340b2990 0x7f67340b4ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f673800b2e0 tx=0x7f6738005fd0 comp rx=0 tx=0).stop 2024-05-27T09:32:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.885+0000 7f6754600700 1 -- 172.21.15.44:0/4156560328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f674c103070 msgr2=0x7f674c19d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.885+0000 7f6754600700 1 --2- 172.21.15.44:0/4156560328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f674c103070 0x7f674c19d080 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f674000e9a0 tx=0x7f674000ecb0 comp rx=0 tx=0).stop 2024-05-27T09:32:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.885+0000 7f6752600700 1 -- 172.21.15.44:0/4156560328 reap_dead start 2024-05-27T09:32:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.885+0000 7f6754600700 1 -- 172.21.15.44:0/4156560328 shutdown_connections 2024-05-27T09:32:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.885+0000 7f6754600700 1 -- 172.21.15.44:0/4156560328 >> 172.21.15.44:0/4156560328 conn(0x7f674c0fedb0 msgr2=0x7f674c112920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:50.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.885+0000 7f6754600700 1 -- 172.21.15.44:0/4156560328 shutdown_connections 2024-05-27T09:32:50.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:50.885+0000 7f6754600700 1 -- 172.21.15.44:0/4156560328 wait complete. 2024-05-27T09:32:52.083 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:51 smithi044 ceph-mon[31618]: pgmap v368: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:32:52.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:51 smithi189 ceph-mon[36671]: pgmap v368: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:32:52.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:51 smithi050 ceph-mon[36371]: pgmap v368: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:32:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.082+0000 7f24bf200700 1 -- 172.21.15.44:0/161487575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c0075b50 msgr2=0x7f24c0075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.082+0000 7f24bf200700 1 --2- 172.21.15.44:0/161487575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c0075b50 0x7f24c0075f60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f24b00098f0 tx=0x7f24b0009c00 comp rx=0 tx=0).stop 2024-05-27T09:32:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.082+0000 7f24bf200700 1 -- 172.21.15.44:0/161487575 shutdown_connections 2024-05-27T09:32:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.082+0000 7f24bf200700 1 --2- 172.21.15.44:0/161487575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24c0115170 0x7f24c0117590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.082+0000 7f24bf200700 1 --2- 172.21.15.44:0/161487575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c0075b50 0x7f24c0075f60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.082+0000 7f24bf200700 1 --2- 172.21.15.44:0/161487575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c00764a0 0x7f24c0114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.082+0000 7f24bf200700 1 -- 172.21.15.44:0/161487575 >> 172.21.15.44:0/161487575 conn(0x7f24c0079d00 msgr2=0x7f24c007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.082+0000 7f24bf200700 1 -- 172.21.15.44:0/161487575 shutdown_connections 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.082+0000 7f24bf200700 1 -- 172.21.15.44:0/161487575 wait complete. 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.083+0000 7f24bf200700 1 Processor -- start 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24bf200700 1 -- start start 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24bf200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c0075b50 0x7f24c01a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24bf200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24c00764a0 0x7f24c01a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24bf200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c0115170 0x7f24c01a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24bf200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f24c011a8d0 con 0x7f24c00764a0 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24bf200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f24c011a750 con 0x7f24c0115170 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24bf200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f24c011aa50 con 0x7f24c0075b50 2024-05-27T09:32:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24bde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c0075b50 0x7f24c01a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24bd400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24c00764a0 0x7f24c01a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24bde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c0075b50 0x7f24c01a12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55664/0 (socket says 172.21.15.44:55664) 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24bde00700 1 -- 172.21.15.44:0/821980014 learned_addr learned my addr 172.21.15.44:0/821980014 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.084+0000 7f24be800700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c0115170 0x7f24c01a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.085+0000 7f24bde00700 1 -- 172.21.15.44:0/821980014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c0115170 msgr2=0x7f24c01a5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.085+0000 7f24bde00700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c0115170 0x7f24c01a5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.085+0000 7f24bde00700 1 -- 172.21.15.44:0/821980014 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24c00764a0 msgr2=0x7f24c01a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.085+0000 7f24bde00700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24c00764a0 0x7f24c01a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.085+0000 7f24bde00700 1 -- 172.21.15.44:0/821980014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c01a6170 con 0x7f24c0075b50 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.085+0000 7f24bd400700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24c00764a0 0x7f24c01a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.085+0000 7f24be800700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c0115170 0x7f24c01a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.085+0000 7f24bde00700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c0075b50 0x7f24c01a12a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009230 tx=0x7f24b000fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.086+0000 7f24b6a00700 1 -- 172.21.15.44:0/821980014 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b00050c0 con 0x7f24c0075b50 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.086+0000 7f24b6a00700 1 -- 172.21.15.44:0/821980014 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24b000b560 con 0x7f24c0075b50 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.086+0000 7f24b6a00700 1 -- 172.21.15.44:0/821980014 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b0003a00 con 0x7f24c0075b50 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.087+0000 7f24bf200700 1 -- 172.21.15.44:0/821980014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c01a6400 con 0x7f24c0075b50 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.087+0000 7f24bf200700 1 -- 172.21.15.44:0/821980014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c01a6860 con 0x7f24c0075b50 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.087+0000 7f24bf200700 1 -- 172.21.15.44:0/821980014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f24c0063f40 con 0x7f24c0075b50 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.090+0000 7f24b6a00700 1 -- 172.21.15.44:0/821980014 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f24b000b710 con 0x7f24c0075b50 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.092+0000 7f24b6a00700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f24a40b2990 0x7f24a40b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.092+0000 7f24b6a00700 1 -- 172.21.15.44:0/821980014 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f24b00e3750 con 0x7f24c0075b50 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.093+0000 7f24bd400700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f24a40b2990 0x7f24a40b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.093+0000 7f24b6a00700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694] conn(0x7f24a40bc520 0x7f24a40bea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.093+0000 7f24b6a00700 1 -- 172.21.15.44:0/821980014 --> [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f24a40bf0e0 con 0x7f24a40bc520 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.093+0000 7f24b6a00700 1 -- 172.21.15.44:0/821980014 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f24b00e3b00 con 0x7f24c0075b50 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.093+0000 7f24bd400700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f24a40b2990 0x7f24a40b4ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f24c0103760 tx=0x7f24ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.093+0000 7f24be800700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694] conn(0x7f24a40bc520 0x7f24a40bea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.094+0000 7f24be800700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694] conn(0x7f24a40bc520 0x7f24a40bea10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.095+0000 7f24b6a00700 1 -- 172.21.15.44:0/821980014 <== osd.8 v2:172.21.15.189:6816/2379603694 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f24a40bf0e0 con 0x7f24a40bc520 2024-05-27T09:32:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.114+0000 7f24bf200700 1 -- 172.21.15.44:0/821980014 --> [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f24c0064410 con 0x7f24a40bc520 2024-05-27T09:32:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.114+0000 7f24b6a00700 1 -- 172.21.15.44:0/821980014 <== osd.8 v2:172.21.15.189:6816/2379603694 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f24c0064410 con 0x7f24a40bc520 2024-05-27T09:32:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.115+0000 7f24bf200700 1 -- 172.21.15.44:0/821980014 >> [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694] conn(0x7f24a40bc520 msgr2=0x7f24a40bea10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.115+0000 7f24bf200700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694] conn(0x7f24a40bc520 0x7f24a40bea10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.115+0000 7f24bf200700 1 -- 172.21.15.44:0/821980014 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f24a40b2990 msgr2=0x7f24a40b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.115+0000 7f24bf200700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f24a40b2990 0x7f24a40b4ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f24c0103760 tx=0x7f24ac009260 comp rx=0 tx=0).stop 2024-05-27T09:32:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.115+0000 7f24bf200700 1 -- 172.21.15.44:0/821980014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c0075b50 msgr2=0x7f24c01a12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.115+0000 7f24bf200700 1 --2- 172.21.15.44:0/821980014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c0075b50 0x7f24c01a12a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009230 tx=0x7f24b000fa70 comp rx=0 tx=0).stop 2024-05-27T09:32:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.115+0000 7f24be800700 1 -- 172.21.15.44:0/821980014 reap_dead start 2024-05-27T09:32:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.115+0000 7f24bf200700 1 -- 172.21.15.44:0/821980014 shutdown_connections 2024-05-27T09:32:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.115+0000 7f24bf200700 1 -- 172.21.15.44:0/821980014 >> 172.21.15.44:0/821980014 conn(0x7f24c0079d00 msgr2=0x7f24c0100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.115+0000 7f24bf200700 1 -- 172.21.15.44:0/821980014 shutdown_connections 2024-05-27T09:32:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:52.115+0000 7f24bf200700 1 -- 172.21.15.44:0/821980014 wait complete. 2024-05-27T09:32:53.907 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:53 smithi044 ceph-mon[31618]: pgmap v369: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:54.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:53 smithi189 ceph-mon[36671]: pgmap v369: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:54.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:53 smithi050 ceph-mon[36371]: pgmap v369: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:32:55.925 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:55 smithi044 ceph-mon[31618]: pgmap v370: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:32:55.992 INFO:teuthology.orchestra.run.smithi044.stdout:77309411416 2024-05-27T09:32:55.992 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-27T09:32:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:55 smithi189 ceph-mon[36671]: pgmap v370: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:32:56.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:55 smithi050 ceph-mon[36371]: pgmap v370: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:32:56.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.709+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3594473727 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9cb0105e60 msgr2=0x7f9cb010e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:56.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.709+0000 7f9cb5a00700 1 --2- 172.21.15.44:0/3594473727 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9cb0105e60 0x7f9cb010e3b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0098f0 tx=0x7f9c9c009c00 comp rx=0 tx=0).stop 2024-05-27T09:32:56.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.709+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3594473727 shutdown_connections 2024-05-27T09:32:56.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.709+0000 7f9cb5a00700 1 --2- 172.21.15.44:0/3594473727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cb010e8f0 0x7f9cb0110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:56.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.709+0000 7f9cb5a00700 1 --2- 172.21.15.44:0/3594473727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cb0105510 0x7f9cb0105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.709+0000 7f9cb5a00700 1 --2- 172.21.15.44:0/3594473727 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9cb0105e60 0x7f9cb010e3b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.709+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3594473727 >> 172.21.15.44:0/3594473727 conn(0x7f9cb00fedb0 msgr2=0x7f9cb01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.709+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3594473727 shutdown_connections 2024-05-27T09:32:56.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.709+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3594473727 wait complete. 2024-05-27T09:32:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.710+0000 7f9cb5a00700 1 Processor -- start 2024-05-27T09:32:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.710+0000 7f9cb5a00700 1 -- start start 2024-05-27T09:32:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.711+0000 7f9cb5a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9cb0105510 0x7f9cb019d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.711+0000 7f9cb5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cb0105e60 0x7f9cb019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.711+0000 7f9cb5a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cb010e8f0 0x7f9cb01a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.711+0000 7f9cb5a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb0114480 con 0x7f9cb010e8f0 2024-05-27T09:32:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.711+0000 7f9cb5a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb0114300 con 0x7f9cb0105510 2024-05-27T09:32:56.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.711+0000 7f9cb5a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb0114600 con 0x7f9cb0105e60 2024-05-27T09:32:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.711+0000 7f9cae000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cb0105e60 0x7f9cb019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.711+0000 7f9caf400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cb010e8f0 0x7f9cb01a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.711+0000 7f9cae000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cb0105e60 0x7f9cb019d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60984/0 (socket says 172.21.15.44:60984) 2024-05-27T09:32:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.711+0000 7f9cae000700 1 -- 172.21.15.44:0/3316043402 learned_addr learned my addr 172.21.15.44:0/3316043402 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:32:56.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.711+0000 7f9caea00700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9cb0105510 0x7f9cb019d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.712+0000 7f9cae000700 1 -- 172.21.15.44:0/3316043402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9cb0105510 msgr2=0x7f9cb019d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.712+0000 7f9cae000700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9cb0105510 0x7f9cb019d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.712+0000 7f9cae000700 1 -- 172.21.15.44:0/3316043402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cb010e8f0 msgr2=0x7f9cb01a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.712+0000 7f9cae000700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cb010e8f0 0x7f9cb01a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.712+0000 7f9cae000700 1 -- 172.21.15.44:0/3316043402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb01a1f20 con 0x7f9cb0105e60 2024-05-27T09:32:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.712+0000 7f9cae000700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cb0105e60 0x7f9cb019d5f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009230 tx=0x7f9c9c00fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:56.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.713+0000 7f9ca7400700 1 -- 172.21.15.44:0/3316043402 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c0050c0 con 0x7f9cb0105e60 2024-05-27T09:32:56.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.713+0000 7f9ca7400700 1 -- 172.21.15.44:0/3316043402 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c9c00b560 con 0x7f9cb0105e60 2024-05-27T09:32:56.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.713+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3316043402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cb01a21b0 con 0x7f9cb0105e60 2024-05-27T09:32:56.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.713+0000 7f9ca7400700 1 -- 172.21.15.44:0/3316043402 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c003a00 con 0x7f9cb0105e60 2024-05-27T09:32:56.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.713+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3316043402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cb01a2690 con 0x7f9cb0105e60 2024-05-27T09:32:56.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.714+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3316043402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9cb0103c30 con 0x7f9cb0105e60 2024-05-27T09:32:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.716+0000 7f9ca7400700 1 -- 172.21.15.44:0/3316043402 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9c9c00b710 con 0x7f9cb0105e60 2024-05-27T09:32:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.718+0000 7f9ca7400700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9c940b2910 0x7f9c940b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.718+0000 7f9ca7400700 1 -- 172.21.15.44:0/3316043402 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f9c9c0e3740 con 0x7f9cb0105e60 2024-05-27T09:32:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.718+0000 7f9caea00700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9c940b2910 0x7f9c940b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.718+0000 7f9ca7400700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028] conn(0x7f9c940bc4a0 0x7f9c940be990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:32:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.718+0000 7f9ca7400700 1 -- 172.21.15.44:0/3316043402 --> [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c940bf060 con 0x7f9c940bc4a0 2024-05-27T09:32:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.718+0000 7f9ca7400700 1 -- 172.21.15.44:0/3316043402 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f9c9c0e3af0 con 0x7f9cb0105e60 2024-05-27T09:32:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.719+0000 7f9caf400700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028] conn(0x7f9c940bc4a0 0x7f9c940be990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:32:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.719+0000 7f9caea00700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9c940b2910 0x7f9c940b4e20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0009c60 tx=0x7f9ca0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.719+0000 7f9caf400700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028] conn(0x7f9c940bc4a0 0x7f9c940be990 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:32:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.720+0000 7f9ca7400700 1 -- 172.21.15.44:0/3316043402 <== osd.10 v2:172.21.15.50:6824/1371655028 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9c940bf060 con 0x7f9c940bc4a0 2024-05-27T09:32:56.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.739+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3316043402 --> [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9cb006be20 con 0x7f9c940bc4a0 2024-05-27T09:32:56.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.740+0000 7f9ca7400700 1 -- 172.21.15.44:0/3316043402 <== osd.10 v2:172.21.15.50:6824/1371655028 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9cb006be20 con 0x7f9c940bc4a0 2024-05-27T09:32:56.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.740+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3316043402 >> [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028] conn(0x7f9c940bc4a0 msgr2=0x7f9c940be990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:56.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.740+0000 7f9cb5a00700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028] conn(0x7f9c940bc4a0 0x7f9c940be990 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:32:56.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.740+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3316043402 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9c940b2910 msgr2=0x7f9c940b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:56.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.740+0000 7f9cb5a00700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9c940b2910 0x7f9c940b4e20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0009c60 tx=0x7f9ca0009260 comp rx=0 tx=0).stop 2024-05-27T09:32:56.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.740+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3316043402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cb0105e60 msgr2=0x7f9cb019d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:32:56.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.740+0000 7f9cb5a00700 1 --2- 172.21.15.44:0/3316043402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cb0105e60 0x7f9cb019d5f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009230 tx=0x7f9c9c00fa70 comp rx=0 tx=0).stop 2024-05-27T09:32:56.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.740+0000 7f9caf400700 1 -- 172.21.15.44:0/3316043402 reap_dead start 2024-05-27T09:32:56.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.740+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3316043402 shutdown_connections 2024-05-27T09:32:56.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.740+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3316043402 >> 172.21.15.44:0/3316043402 conn(0x7f9cb00fedb0 msgr2=0x7f9cb01031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:32:56.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.741+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3316043402 shutdown_connections 2024-05-27T09:32:56.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:56.741+0000 7f9cb5a00700 1 -- 172.21.15.44:0/3316043402 wait complete. 2024-05-27T09:32:57.646 INFO:teuthology.orchestra.run.smithi044.stdout:227633266740 2024-05-27T09:32:57.646 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-27T09:32:57.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:57 smithi044 ceph-mon[31618]: pgmap v371: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T09:32:58.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:57 smithi189 ceph-mon[36671]: pgmap v371: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T09:32:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:57 smithi050 ceph-mon[36371]: pgmap v371: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T09:32:58.730 INFO:teuthology.orchestra.run.smithi044.stdout:133143986251 2024-05-27T09:32:58.730 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-27T09:32:59.979 INFO:teuthology.orchestra.run.smithi044.stdout:55834574941 2024-05-27T09:32:59.979 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-27T09:33:00.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:32:59 smithi044 ceph-mon[31618]: pgmap v372: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-05-27T09:33:00.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:32:59 smithi189 ceph-mon[36671]: pgmap v372: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-05-27T09:33:00.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.783+0000 7feef7000700 1 -- 172.21.15.44:0/1802115733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef0115360 msgr2=0x7feef0117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.783+0000 7feef7000700 1 --2- 172.21.15.44:0/1802115733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef0115360 0x7feef0117780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7feee800b0a0 tx=0x7feee800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.783+0000 7feef7000700 1 -- 172.21.15.44:0/1802115733 shutdown_connections 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.783+0000 7feef7000700 1 --2- 172.21.15.44:0/1802115733 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef0115360 0x7feef0117780 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.783+0000 7feef7000700 1 --2- 172.21.15.44:0/1802115733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feef0075b50 0x7feef0075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.783+0000 7feef7000700 1 --2- 172.21.15.44:0/1802115733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feef00764a0 0x7feef0114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.783+0000 7feef7000700 1 -- 172.21.15.44:0/1802115733 >> 172.21.15.44:0/1802115733 conn(0x7feef0100030 msgr2=0x7feef0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.783+0000 7feef7000700 1 -- 172.21.15.44:0/1802115733 shutdown_connections 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.783+0000 7feef7000700 1 -- 172.21.15.44:0/1802115733 wait complete. 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.785+0000 7feef7000700 1 Processor -- start 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.785+0000 7feef7000700 1 -- start start 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.785+0000 7feef7000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef0075b50 0x7feef01a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.786+0000 7feef7000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feef00764a0 0x7feef01a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:00.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.786+0000 7feef7000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feef0115360 0x7feef01a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:00.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.786+0000 7feef7000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feef011aab0 con 0x7feef0075b50 2024-05-27T09:33:00.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.786+0000 7feef7000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feef011a930 con 0x7feef00764a0 2024-05-27T09:33:00.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.786+0000 7feef7000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feef011ac30 con 0x7feef0115360 2024-05-27T09:33:00.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.786+0000 7feeef400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feef00764a0 0x7feef01a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:00.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.786+0000 7feeefe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef0075b50 0x7feef01a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:00.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.786+0000 7feeef400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feef00764a0 0x7feef01a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47196/0 (socket says 172.21.15.44:47196) 2024-05-27T09:33:00.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.786+0000 7feeef400700 1 -- 172.21.15.44:0/796314717 learned_addr learned my addr 172.21.15.44:0/796314717 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:00.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.786+0000 7feef5000700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feef0115360 0x7feef01a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:00.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feeefe00700 1 -- 172.21.15.44:0/796314717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feef0115360 msgr2=0x7feef01a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feeefe00700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feef0115360 0x7feef01a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feeefe00700 1 -- 172.21.15.44:0/796314717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feef00764a0 msgr2=0x7feef01a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feeefe00700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feef00764a0 0x7feef01a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feeefe00700 1 -- 172.21.15.44:0/796314717 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef01a6250 con 0x7feef0075b50 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feef5000700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feef0115360 0x7feef01a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feeef400700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feef00764a0 0x7feef01a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feeefe00700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef0075b50 0x7feef01a1380 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7feee000e9f0 tx=0x7feee000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feeecc00700 1 -- 172.21.15.44:0/796314717 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee000c9c0 con 0x7feef0075b50 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feef7000700 1 -- 172.21.15.44:0/796314717 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef01a6540 con 0x7feef0075b50 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feeecc00700 1 -- 172.21.15.44:0/796314717 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feee000cb60 con 0x7feef0075b50 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.787+0000 7feeecc00700 1 -- 172.21.15.44:0/796314717 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee0018680 con 0x7feef0075b50 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.788+0000 7feef7000700 1 -- 172.21.15.44:0/796314717 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef01ade60 con 0x7feef0075b50 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.789+0000 7feef7000700 1 -- 172.21.15.44:0/796314717 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feef0063ec0 con 0x7feef0075b50 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.793+0000 7feeecc00700 1 -- 172.21.15.44:0/796314717 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7feee0010430 con 0x7feef0075b50 2024-05-27T09:33:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.795+0000 7feeecc00700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7feed40b2910 0x7feed40b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.795+0000 7feeecc00700 1 -- 172.21.15.44:0/796314717 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7feee00e2b20 con 0x7feef0075b50 2024-05-27T09:33:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.795+0000 7feeecc00700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534] conn(0x7feed40bc4a0 0x7feed40be990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.795+0000 7feeecc00700 1 -- 172.21.15.44:0/796314717 --> [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feed40bf060 con 0x7feed40bc4a0 2024-05-27T09:33:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.795+0000 7feeecc00700 1 -- 172.21.15.44:0/796314717 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7feee00e2ed0 con 0x7feef0075b50 2024-05-27T09:33:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.795+0000 7feeef400700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7feed40b2910 0x7feed40b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:00.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.795+0000 7feef5000700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534] conn(0x7feed40bc4a0 0x7feed40be990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.796+0000 7feeef400700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7feed40b2910 0x7feed40b4e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feef01034f0 tx=0x7feedc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.796+0000 7feef5000700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534] conn(0x7feed40bc4a0 0x7feed40be990 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.797+0000 7feeecc00700 1 -- 172.21.15.44:0/796314717 <== osd.0 v2:172.21.15.44:6802/3103307534 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feed40bf060 con 0x7feed40bc4a0 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.819+0000 7feef7000700 1 -- 172.21.15.44:0/796314717 --> [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feef0064340 con 0x7feed40bc4a0 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.820+0000 7feeecc00700 1 -- 172.21.15.44:0/796314717 <== osd.0 v2:172.21.15.44:6802/3103307534 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7feef0064340 con 0x7feed40bc4a0 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.820+0000 7feef7000700 1 -- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534] conn(0x7feed40bc4a0 msgr2=0x7feed40be990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.820+0000 7feef7000700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534] conn(0x7feed40bc4a0 0x7feed40be990 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.820+0000 7feef7000700 1 -- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7feed40b2910 msgr2=0x7feed40b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.820+0000 7feef7000700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7feed40b2910 0x7feed40b4e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feef01034f0 tx=0x7feedc009260 comp rx=0 tx=0).stop 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.820+0000 7feef7000700 1 -- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef0075b50 msgr2=0x7feef01a1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.820+0000 7feef7000700 1 --2- 172.21.15.44:0/796314717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feef0075b50 0x7feef01a1380 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7feee000e9f0 tx=0x7feee000ed00 comp rx=0 tx=0).stop 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.820+0000 7feef5000700 1 -- 172.21.15.44:0/796314717 reap_dead start 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.821+0000 7feef7000700 1 -- 172.21.15.44:0/796314717 shutdown_connections 2024-05-27T09:33:00.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.821+0000 7feef7000700 1 -- 172.21.15.44:0/796314717 >> 172.21.15.44:0/796314717 conn(0x7feef0100030 msgr2=0x7feef0100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.821+0000 7feef7000700 1 -- 172.21.15.44:0/796314717 shutdown_connections 2024-05-27T09:33:00.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:32:59.821+0000 7feef7000700 1 -- 172.21.15.44:0/796314717 wait complete. 2024-05-27T09:33:00.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:32:59 smithi050 ceph-mon[36371]: pgmap v372: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-05-27T09:33:00.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.296+0000 7f1c17800700 1 -- 172.21.15.44:0/4264704628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c10076440 msgr2=0x7f1c101106b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.296+0000 7f1c17800700 1 --2- 172.21.15.44:0/4264704628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c10076440 0x7f1c101106b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009990 tx=0x7f1bfc009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.296+0000 7f1c17800700 1 -- 172.21.15.44:0/4264704628 shutdown_connections 2024-05-27T09:33:00.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.296+0000 7f1c17800700 1 --2- 172.21.15.44:0/4264704628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c10110bf0 0x7f1c10113010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.296+0000 7f1c17800700 1 --2- 172.21.15.44:0/4264704628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c10075af0 0x7f1c10075f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.296+0000 7f1c17800700 1 --2- 172.21.15.44:0/4264704628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c10076440 0x7f1c101106b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.296+0000 7f1c17800700 1 -- 172.21.15.44:0/4264704628 >> 172.21.15.44:0/4264704628 conn(0x7f1c10100030 msgr2=0x7f1c10102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.296+0000 7f1c17800700 1 -- 172.21.15.44:0/4264704628 shutdown_connections 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.296+0000 7f1c17800700 1 -- 172.21.15.44:0/4264704628 wait complete. 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.297+0000 7f1c17800700 1 Processor -- start 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.298+0000 7f1c17800700 1 -- start start 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.298+0000 7f1c17800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c10075af0 0x7f1c1019d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.298+0000 7f1c17800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c10076440 0x7f1c1019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.298+0000 7f1c17800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c10110bf0 0x7f1c101a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.298+0000 7f1c17800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1c101166b0 con 0x7f1c10110bf0 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.298+0000 7f1c17800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1c10116530 con 0x7f1c10076440 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.298+0000 7f1c17800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1c10116830 con 0x7f1c10075af0 2024-05-27T09:33:00.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.299+0000 7f1c0fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c10076440 0x7f1c1019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.299+0000 7f1c0fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c10076440 0x7f1c1019d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47212/0 (socket says 172.21.15.44:47212) 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.299+0000 7f1c0fe00700 1 -- 172.21.15.44:0/100812039 learned_addr learned my addr 172.21.15.44:0/100812039 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.299+0000 7f1c15800700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c10110bf0 0x7f1c101a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.299+0000 7f1c14e00700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c10075af0 0x7f1c1019d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.299+0000 7f1c0fe00700 1 -- 172.21.15.44:0/100812039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c10075af0 msgr2=0x7f1c1019d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.299+0000 7f1c0fe00700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c10075af0 0x7f1c1019d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.299+0000 7f1c0fe00700 1 -- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c10110bf0 msgr2=0x7f1c101a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.299+0000 7f1c0fe00700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c10110bf0 0x7f1c101a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.299+0000 7f1c0fe00700 1 -- 172.21.15.44:0/100812039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c101a1f10 con 0x7f1c10076440 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.299+0000 7f1c0fe00700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c10076440 0x7f1c1019d5e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc00bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.300+0000 7f1c0d600700 1 -- 172.21.15.44:0/100812039 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc027070 con 0x7f1c10076440 2024-05-27T09:33:00.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.300+0000 7f1c17800700 1 -- 172.21.15.44:0/100812039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c101a21a0 con 0x7f1c10076440 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.300+0000 7f1c17800700 1 -- 172.21.15.44:0/100812039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c101a9b20 con 0x7f1c10076440 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.300+0000 7f1c0d600700 1 -- 172.21.15.44:0/100812039 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bfc022470 con 0x7f1c10076440 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.300+0000 7f1c0d600700 1 -- 172.21.15.44:0/100812039 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc017650 con 0x7f1c10076440 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.301+0000 7f1c17800700 1 -- 172.21.15.44:0/100812039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1c10063f40 con 0x7f1c10076440 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.304+0000 7f1c0d600700 1 -- 172.21.15.44:0/100812039 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1bfc02a070 con 0x7f1c10076440 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.305+0000 7f1c0d600700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f1bf80b2990 0x7f1bf80b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.305+0000 7f1c0d600700 1 -- 172.21.15.44:0/100812039 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f1bfc0e37b0 con 0x7f1c10076440 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.305+0000 7f1c14e00700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f1bf80b2990 0x7f1bf80b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.305+0000 7f1c0d600700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697] conn(0x7f1bf80bc3f0 0x7f1bf80be8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.305+0000 7f1c0d600700 1 -- 172.21.15.44:0/100812039 --> [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1bfc02ee50 con 0x7f1bf80bc3f0 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.305+0000 7f1c0d600700 1 -- 172.21.15.44:0/100812039 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f1bfc0e3b60 con 0x7f1c10076440 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.306+0000 7f1c15800700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697] conn(0x7f1bf80bc3f0 0x7f1bf80be8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.306+0000 7f1c14e00700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f1bf80b2990 0x7f1bf80b4ea0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009c60 tx=0x7f1c04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:00.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.306+0000 7f1c15800700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697] conn(0x7f1bf80bc3f0 0x7f1bf80be8e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:00.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.308+0000 7f1c0d600700 1 -- 172.21.15.44:0/100812039 <== osd.3 v2:172.21.15.44:6810/2748877697 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1bfc02ee50 con 0x7f1bf80bc3f0 2024-05-27T09:33:00.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.326+0000 7f1c17800700 1 -- 172.21.15.44:0/100812039 --> [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1c100643c0 con 0x7f1bf80bc3f0 2024-05-27T09:33:00.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.327+0000 7f1c0d600700 1 -- 172.21.15.44:0/100812039 <== osd.3 v2:172.21.15.44:6810/2748877697 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1c100643c0 con 0x7f1bf80bc3f0 2024-05-27T09:33:00.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.327+0000 7f1c17800700 1 -- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697] conn(0x7f1bf80bc3f0 msgr2=0x7f1bf80be8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.327+0000 7f1c17800700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697] conn(0x7f1bf80bc3f0 0x7f1bf80be8e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.328+0000 7f1c17800700 1 -- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f1bf80b2990 msgr2=0x7f1bf80b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.328+0000 7f1c17800700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f1bf80b2990 0x7f1bf80b4ea0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009c60 tx=0x7f1c04009260 comp rx=0 tx=0).stop 2024-05-27T09:33:00.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.328+0000 7f1c17800700 1 -- 172.21.15.44:0/100812039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c10076440 msgr2=0x7f1c1019d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:00.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.328+0000 7f1c17800700 1 --2- 172.21.15.44:0/100812039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c10076440 0x7f1c1019d5e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc00bfd0 comp rx=0 tx=0).stop 2024-05-27T09:33:00.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.328+0000 7f1c15800700 1 -- 172.21.15.44:0/100812039 reap_dead start 2024-05-27T09:33:00.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.328+0000 7f1c17800700 1 -- 172.21.15.44:0/100812039 shutdown_connections 2024-05-27T09:33:00.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.328+0000 7f1c17800700 1 -- 172.21.15.44:0/100812039 >> 172.21.15.44:0/100812039 conn(0x7f1c10100030 msgr2=0x7f1c101006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:00.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.328+0000 7f1c17800700 1 -- 172.21.15.44:0/100812039 shutdown_connections 2024-05-27T09:33:00.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:00.328+0000 7f1c17800700 1 -- 172.21.15.44:0/100812039 wait complete. 2024-05-27T09:33:02.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:01 smithi189 ceph-mon[36671]: pgmap v373: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:33:02.093 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:01 smithi044 ceph-mon[31618]: pgmap v373: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:33:02.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:01 smithi050 ceph-mon[36371]: pgmap v373: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:33:02.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.488+0000 7f944e000700 1 -- 172.21.15.44:0/2396250145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9448109860 msgr2=0x7f94481111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:02.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.488+0000 7f944e000700 1 --2- 172.21.15.44:0/2396250145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9448109860 0x7f94481111c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f944000b0d0 tx=0x7f944000b3e0 comp rx=0 tx=0).stop 2024-05-27T09:33:02.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.489+0000 7f944e000700 1 -- 172.21.15.44:0/2396250145 shutdown_connections 2024-05-27T09:33:02.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.489+0000 7f944e000700 1 --2- 172.21.15.44:0/2396250145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9448109860 0x7f94481111c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:02.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.489+0000 7f944e000700 1 --2- 172.21.15.44:0/2396250145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94481045e0 0x7f94481049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:02.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.489+0000 7f944e000700 1 --2- 172.21.15.44:0/2396250145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9448108f10 0x7f9448109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.489+0000 7f944e000700 1 -- 172.21.15.44:0/2396250145 >> 172.21.15.44:0/2396250145 conn(0x7f9448100030 msgr2=0x7f9448102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.489+0000 7f944e000700 1 -- 172.21.15.44:0/2396250145 shutdown_connections 2024-05-27T09:33:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.489+0000 7f944e000700 1 -- 172.21.15.44:0/2396250145 wait complete. 2024-05-27T09:33:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.490+0000 7f944e000700 1 Processor -- start 2024-05-27T09:33:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.490+0000 7f944e000700 1 -- start start 2024-05-27T09:33:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.491+0000 7f944e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94481045e0 0x7f9448075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.491+0000 7f944e000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9448108f10 0x7f9448075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.491+0000 7f944e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9448109860 0x7f9448076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.491+0000 7f944e000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9448114490 con 0x7f9448109860 2024-05-27T09:33:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.491+0000 7f944e000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9448114310 con 0x7f9448108f10 2024-05-27T09:33:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.491+0000 7f944e000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9448114610 con 0x7f94481045e0 2024-05-27T09:33:02.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.491+0000 7f9446a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9448108f10 0x7f9448075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:02.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.491+0000 7f9446a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9448108f10 0x7f9448075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47228/0 (socket says 172.21.15.44:47228) 2024-05-27T09:33:02.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.491+0000 7f9446a00700 1 -- 172.21.15.44:0/1252805106 learned_addr learned my addr 172.21.15.44:0/1252805106 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:02.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.491+0000 7f9447400700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94481045e0 0x7f9448075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.491+0000 7f9447e00700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9448109860 0x7f9448076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.492+0000 7f9446a00700 1 -- 172.21.15.44:0/1252805106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94481045e0 msgr2=0x7f9448075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.492+0000 7f9446a00700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94481045e0 0x7f9448075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.492+0000 7f9446a00700 1 -- 172.21.15.44:0/1252805106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9448109860 msgr2=0x7f9448076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.492+0000 7f9446a00700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9448109860 0x7f9448076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.492+0000 7f9446a00700 1 -- 172.21.15.44:0/1252805106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94481a91a0 con 0x7f9448108f10 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.492+0000 7f9446a00700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9448108f10 0x7f9448075ff0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f943400e970 tx=0x7f943400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.492+0000 7f943fe00700 1 -- 172.21.15.44:0/1252805106 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943400cbd0 con 0x7f9448108f10 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.492+0000 7f943fe00700 1 -- 172.21.15.44:0/1252805106 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f943400cd70 con 0x7f9448108f10 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.492+0000 7f944e000700 1 -- 172.21.15.44:0/1252805106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94481a9490 con 0x7f9448108f10 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.493+0000 7f943fe00700 1 -- 172.21.15.44:0/1252805106 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9434010430 con 0x7f9448108f10 2024-05-27T09:33:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.493+0000 7f944e000700 1 -- 172.21.15.44:0/1252805106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94481a99d0 con 0x7f9448108f10 2024-05-27T09:33:02.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.494+0000 7f944e000700 1 -- 172.21.15.44:0/1252805106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9448063ec0 con 0x7f9448108f10 2024-05-27T09:33:02.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.498+0000 7f943fe00700 1 -- 172.21.15.44:0/1252805106 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9434004d70 con 0x7f9448108f10 2024-05-27T09:33:02.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.499+0000 7f943fe00700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f942c0b2910 0x7f942c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:02.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.500+0000 7f943fe00700 1 -- 172.21.15.44:0/1252805106 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f94340e2a80 con 0x7f9448108f10 2024-05-27T09:33:02.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.500+0000 7f9447400700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f942c0b2910 0x7f942c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:02.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.500+0000 7f943fe00700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633] conn(0x7f942c0bc4a0 0x7f942c0be990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:02.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.500+0000 7f943fe00700 1 -- 172.21.15.44:0/1252805106 --> [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f943400be50 con 0x7f942c0bc4a0 2024-05-27T09:33:02.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.500+0000 7f943fe00700 1 -- 172.21.15.44:0/1252805106 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f94340e2e30 con 0x7f9448108f10 2024-05-27T09:33:02.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.500+0000 7f9447e00700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633] conn(0x7f942c0bc4a0 0x7f942c0be990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:02.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.501+0000 7f9447400700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f942c0b2910 0x7f942c0b4e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9438005fd0 tx=0x7f943800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:02.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.501+0000 7f9447e00700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633] conn(0x7f942c0bc4a0 0x7f942c0be990 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:02.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.502+0000 7f943fe00700 1 -- 172.21.15.44:0/1252805106 <== osd.11 v2:172.21.15.189:6824/2712606633 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f943400be50 con 0x7f942c0bc4a0 2024-05-27T09:33:02.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.521+0000 7f944e000700 1 -- 172.21.15.44:0/1252805106 --> [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9448064340 con 0x7f942c0bc4a0 2024-05-27T09:33:02.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.521+0000 7f943fe00700 1 -- 172.21.15.44:0/1252805106 <== osd.11 v2:172.21.15.189:6824/2712606633 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9448064340 con 0x7f942c0bc4a0 2024-05-27T09:33:02.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.522+0000 7f944e000700 1 -- 172.21.15.44:0/1252805106 >> [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633] conn(0x7f942c0bc4a0 msgr2=0x7f942c0be990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:02.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.522+0000 7f944e000700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633] conn(0x7f942c0bc4a0 0x7f942c0be990 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:02.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.522+0000 7f944e000700 1 -- 172.21.15.44:0/1252805106 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f942c0b2910 msgr2=0x7f942c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:02.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.522+0000 7f944e000700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f942c0b2910 0x7f942c0b4e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9438005fd0 tx=0x7f943800a560 comp rx=0 tx=0).stop 2024-05-27T09:33:02.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.522+0000 7f944e000700 1 -- 172.21.15.44:0/1252805106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9448108f10 msgr2=0x7f9448075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:02.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.522+0000 7f944e000700 1 --2- 172.21.15.44:0/1252805106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9448108f10 0x7f9448075ff0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f943400e970 tx=0x7f943400ed30 comp rx=0 tx=0).stop 2024-05-27T09:33:02.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.522+0000 7f9447e00700 1 -- 172.21.15.44:0/1252805106 reap_dead start 2024-05-27T09:33:02.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.522+0000 7f944e000700 1 -- 172.21.15.44:0/1252805106 shutdown_connections 2024-05-27T09:33:02.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.522+0000 7f944e000700 1 -- 172.21.15.44:0/1252805106 >> 172.21.15.44:0/1252805106 conn(0x7f9448100030 msgr2=0x7f9448108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:02.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.523+0000 7f944e000700 1 -- 172.21.15.44:0/1252805106 shutdown_connections 2024-05-27T09:33:02.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:02.523+0000 7f944e000700 1 -- 172.21.15.44:0/1252805106 wait complete. 2024-05-27T09:33:02.635 INFO:teuthology.orchestra.run.smithi044.stdout:107374182481 2024-05-27T09:33:02.636 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-27T09:33:03.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.395+0000 7f5b57e00700 1 -- 172.21.15.44:0/596749501 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b5810e6f0 msgr2=0x7f5b58110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:03.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.395+0000 7f5b57e00700 1 --2- 172.21.15.44:0/596749501 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b5810e6f0 0x7f5b58110bd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000b0a0 tx=0x7f5b5000b3b0 comp rx=0 tx=0).stop 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.395+0000 7f5b57e00700 1 -- 172.21.15.44:0/596749501 shutdown_connections 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.395+0000 7f5b57e00700 1 --2- 172.21.15.44:0/596749501 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b5810e6f0 0x7f5b58110bd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.395+0000 7f5b57e00700 1 --2- 172.21.15.44:0/596749501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b58105280 0x7f5b58105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.395+0000 7f5b57e00700 1 --2- 172.21.15.44:0/596749501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b58105bd0 0x7f5b5810e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.396+0000 7f5b57e00700 1 -- 172.21.15.44:0/596749501 >> 172.21.15.44:0/596749501 conn(0x7f5b58078ab0 msgr2=0x7f5b58078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.396+0000 7f5b57e00700 1 -- 172.21.15.44:0/596749501 shutdown_connections 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.396+0000 7f5b57e00700 1 -- 172.21.15.44:0/596749501 wait complete. 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.397+0000 7f5b57e00700 1 Processor -- start 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.397+0000 7f5b57e00700 1 -- start start 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.398+0000 7f5b57e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b58105280 0x7f5b581a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.398+0000 7f5b57e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b58105bd0 0x7f5b581a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:03.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.398+0000 7f5b57e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b5810e6f0 0x7f5b5819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.398+0000 7f5b57e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b58114250 con 0x7f5b58105280 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.398+0000 7f5b57e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581140d0 con 0x7f5b5810e6f0 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.398+0000 7f5b57e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581143d0 con 0x7f5b58105bd0 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.398+0000 7f5b56a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b58105280 0x7f5b581a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.398+0000 7f5b56000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b58105bd0 0x7f5b581a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.398+0000 7f5b56a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b58105280 0x7f5b581a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45372/0 (socket says 172.21.15.44:45372) 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.398+0000 7f5b56a00700 1 -- 172.21.15.44:0/370188398 learned_addr learned my addr 172.21.15.44:0/370188398 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.398+0000 7f5b57400700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b5810e6f0 0x7f5b5819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.399+0000 7f5b56000700 1 -- 172.21.15.44:0/370188398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b5810e6f0 msgr2=0x7f5b5819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.399+0000 7f5b56000700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b5810e6f0 0x7f5b5819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.399+0000 7f5b56000700 1 -- 172.21.15.44:0/370188398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b58105280 msgr2=0x7f5b581a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:03.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.399+0000 7f5b56000700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b58105280 0x7f5b581a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.399+0000 7f5b56000700 1 -- 172.21.15.44:0/370188398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5819b060 con 0x7f5b58105bd0 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.399+0000 7f5b56000700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b58105bd0 0x7f5b581a0cd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400e9f0 tx=0x7f5b4400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.399+0000 7f5b4f400700 1 -- 172.21.15.44:0/370188398 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4400c9c0 con 0x7f5b58105bd0 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.399+0000 7f5b57e00700 1 -- 172.21.15.44:0/370188398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b5819b350 con 0x7f5b58105bd0 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.399+0000 7f5b4f400700 1 -- 172.21.15.44:0/370188398 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b4400cb60 con 0x7f5b58105bd0 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.399+0000 7f5b4f400700 1 -- 172.21.15.44:0/370188398 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44018680 con 0x7f5b58105bd0 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.400+0000 7f5b57e00700 1 -- 172.21.15.44:0/370188398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b581a7620 con 0x7f5b58105bd0 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.400+0000 7f5b57e00700 1 -- 172.21.15.44:0/370188398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5b58063f40 con 0x7f5b58105bd0 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.404+0000 7f5b4f400700 1 -- 172.21.15.44:0/370188398 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5b44010430 con 0x7f5b58105bd0 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.406+0000 7f5b4f400700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5b3c0b2910 0x7f5b3c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.406+0000 7f5b4f400700 1 -- 172.21.15.44:0/370188398 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f5b440e2ac0 con 0x7f5b58105bd0 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.406+0000 7f5b56a00700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5b3c0b2910 0x7f5b3c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:03.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.406+0000 7f5b4f400700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446] conn(0x7f5b3c0bc4a0 0x7f5b3c0be990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.406+0000 7f5b4f400700 1 -- 172.21.15.44:0/370188398 --> [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b3c0bf060 con 0x7f5b3c0bc4a0 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.406+0000 7f5b4f400700 1 -- 172.21.15.44:0/370188398 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f5b440e2e70 con 0x7f5b58105bd0 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.407+0000 7f5b56a00700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5b3c0b2910 0x7f5b3c0b4e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b5819baf0 tx=0x7f5b48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.407+0000 7f5b57400700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446] conn(0x7f5b3c0bc4a0 0x7f5b3c0be990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.408+0000 7f5b57400700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446] conn(0x7f5b3c0bc4a0 0x7f5b3c0be990 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.409+0000 7f5b4f400700 1 -- 172.21.15.44:0/370188398 <== osd.7 v2:172.21.15.50:6816/1616906446 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5b3c0bf060 con 0x7f5b3c0bc4a0 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.428+0000 7f5b57e00700 1 -- 172.21.15.44:0/370188398 --> [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5b580643c0 con 0x7f5b3c0bc4a0 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.429+0000 7f5b4f400700 1 -- 172.21.15.44:0/370188398 <== osd.7 v2:172.21.15.50:6816/1616906446 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5b580643c0 con 0x7f5b3c0bc4a0 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.429+0000 7f5b57e00700 1 -- 172.21.15.44:0/370188398 >> [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446] conn(0x7f5b3c0bc4a0 msgr2=0x7f5b3c0be990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.429+0000 7f5b57e00700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446] conn(0x7f5b3c0bc4a0 0x7f5b3c0be990 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.429+0000 7f5b57e00700 1 -- 172.21.15.44:0/370188398 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5b3c0b2910 msgr2=0x7f5b3c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.429+0000 7f5b57e00700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5b3c0b2910 0x7f5b3c0b4e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b5819baf0 tx=0x7f5b48009260 comp rx=0 tx=0).stop 2024-05-27T09:33:03.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.429+0000 7f5b57e00700 1 -- 172.21.15.44:0/370188398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b58105bd0 msgr2=0x7f5b581a0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:03.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.429+0000 7f5b57e00700 1 --2- 172.21.15.44:0/370188398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b58105bd0 0x7f5b581a0cd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400e9f0 tx=0x7f5b4400ed00 comp rx=0 tx=0).stop 2024-05-27T09:33:03.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.429+0000 7f5b57400700 1 -- 172.21.15.44:0/370188398 reap_dead start 2024-05-27T09:33:03.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.430+0000 7f5b57e00700 1 -- 172.21.15.44:0/370188398 shutdown_connections 2024-05-27T09:33:03.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.430+0000 7f5b57e00700 1 -- 172.21.15.44:0/370188398 >> 172.21.15.44:0/370188398 conn(0x7f5b58078ab0 msgr2=0x7f5b580ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:03.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.430+0000 7f5b57e00700 1 -- 172.21.15.44:0/370188398 shutdown_connections 2024-05-27T09:33:03.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:03.430+0000 7f5b57e00700 1 -- 172.21.15.44:0/370188398 wait complete. 2024-05-27T09:33:03.957 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:03 smithi044 ceph-mon[31618]: pgmap v374: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-05-27T09:33:04.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:03 smithi189 ceph-mon[36671]: pgmap v374: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-05-27T09:33:04.096 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:03 smithi050 ceph-mon[36371]: pgmap v374: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-05-27T09:33:04.744 INFO:teuthology.orchestra.run.smithi044.stdout:279172874281 2024-05-27T09:33:04.744 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-27T09:33:04.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.569+0000 7ff740400700 1 -- 172.21.15.44:0/3307382131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff738075af0 msgr2=0x7ff738075f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:04.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.569+0000 7ff740400700 1 --2- 172.21.15.44:0/3307382131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff738075af0 0x7ff738075f00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009990 tx=0x7ff72c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:04.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.569+0000 7ff740400700 1 -- 172.21.15.44:0/3307382131 shutdown_connections 2024-05-27T09:33:04.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.569+0000 7ff740400700 1 --2- 172.21.15.44:0/3307382131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff738110bf0 0x7ff738113010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:04.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.569+0000 7ff740400700 1 --2- 172.21.15.44:0/3307382131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff738075af0 0x7ff738075f00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:04.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.569+0000 7ff740400700 1 --2- 172.21.15.44:0/3307382131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff738076440 0x7ff7381106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.569+0000 7ff740400700 1 -- 172.21.15.44:0/3307382131 >> 172.21.15.44:0/3307382131 conn(0x7ff738100030 msgr2=0x7ff738102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.569+0000 7ff740400700 1 -- 172.21.15.44:0/3307382131 shutdown_connections 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.569+0000 7ff740400700 1 -- 172.21.15.44:0/3307382131 wait complete. 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.571+0000 7ff740400700 1 Processor -- start 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.572+0000 7ff740400700 1 -- start start 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.572+0000 7ff740400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff738075af0 0x7ff73819d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.572+0000 7ff740400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff738076440 0x7ff73819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.572+0000 7ff740400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff738110bf0 0x7ff7381a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.572+0000 7ff740400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7381166d0 con 0x7ff738076440 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.572+0000 7ff740400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff738116550 con 0x7ff738110bf0 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.572+0000 7ff740400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff738116850 con 0x7ff738075af0 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73d000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff738076440 0x7ff73819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:04.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73da00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff738075af0 0x7ff73819d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73e400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff738110bf0 0x7ff7381a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73da00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff738075af0 0x7ff73819d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50934/0 (socket says 172.21.15.44:50934) 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73e400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff738110bf0 0x7ff7381a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:57920/0 (socket says 172.21.15.44:57920) 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73da00700 1 -- 172.21.15.44:0/2968403864 learned_addr learned my addr 172.21.15.44:0/2968403864 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73e400700 1 -- 172.21.15.44:0/2968403864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff738075af0 msgr2=0x7ff73819d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73e400700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff738075af0 0x7ff73819d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73e400700 1 -- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff738076440 msgr2=0x7ff73819d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73e400700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff738076440 0x7ff73819d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73e400700 1 -- 172.21.15.44:0/2968403864 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7381a1fb0 con 0x7ff738110bf0 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73da00700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff738075af0 0x7ff73819d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.573+0000 7ff73e400700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff738110bf0 0x7ff7381a1910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff73000ef90 tx=0x7ff73000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.574+0000 7ff736000700 1 -- 172.21.15.44:0/2968403864 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7300092e0 con 0x7ff738110bf0 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.574+0000 7ff740400700 1 -- 172.21.15.44:0/2968403864 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7381a2240 con 0x7ff738110bf0 2024-05-27T09:33:04.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.574+0000 7ff740400700 1 -- 172.21.15.44:0/2968403864 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7381a9bc0 con 0x7ff738110bf0 2024-05-27T09:33:04.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.574+0000 7ff736000700 1 -- 172.21.15.44:0/2968403864 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff730004bb0 con 0x7ff738110bf0 2024-05-27T09:33:04.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.574+0000 7ff736000700 1 -- 172.21.15.44:0/2968403864 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730004020 con 0x7ff738110bf0 2024-05-27T09:33:04.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.575+0000 7ff740400700 1 -- 172.21.15.44:0/2968403864 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff718001030 con 0x7ff738110bf0 2024-05-27T09:33:04.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.578+0000 7ff736000700 1 -- 172.21.15.44:0/2968403864 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff73001b030 con 0x7ff738110bf0 2024-05-27T09:33:04.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.580+0000 7ff736000700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff7200b2910 0x7ff7200b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:04.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.580+0000 7ff736000700 1 -- 172.21.15.44:0/2968403864 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7ff7300e2b00 con 0x7ff738110bf0 2024-05-27T09:33:04.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.580+0000 7ff73da00700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff7200b2910 0x7ff7200b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:04.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.580+0000 7ff736000700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032] conn(0x7ff7200bc4a0 0x7ff7200be990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:04.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.580+0000 7ff736000700 1 -- 172.21.15.44:0/2968403864 --> [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7200bf060 con 0x7ff7200bc4a0 2024-05-27T09:33:04.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.580+0000 7ff736000700 1 -- 172.21.15.44:0/2968403864 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7ff7300e2eb0 con 0x7ff738110bf0 2024-05-27T09:33:04.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.581+0000 7ff73d000700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032] conn(0x7ff7200bc4a0 0x7ff7200be990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:04.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.581+0000 7ff73da00700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff7200b2910 0x7ff7200b4e20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00b2e0 tx=0x7ff72c004710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:04.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.581+0000 7ff73d000700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032] conn(0x7ff7200bc4a0 0x7ff7200be990 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:04.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.582+0000 7ff736000700 1 -- 172.21.15.44:0/2968403864 <== osd.9 v2:172.21.15.44:6826/503459032 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7200bf060 con 0x7ff7200bc4a0 2024-05-27T09:33:04.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.601+0000 7ff740400700 1 -- 172.21.15.44:0/2968403864 --> [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff718002e60 con 0x7ff7200bc4a0 2024-05-27T09:33:04.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.601+0000 7ff736000700 1 -- 172.21.15.44:0/2968403864 <== osd.9 v2:172.21.15.44:6826/503459032 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff718002e60 con 0x7ff7200bc4a0 2024-05-27T09:33:04.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.602+0000 7ff740400700 1 -- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032] conn(0x7ff7200bc4a0 msgr2=0x7ff7200be990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:04.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.602+0000 7ff740400700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032] conn(0x7ff7200bc4a0 0x7ff7200be990 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:04.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.602+0000 7ff740400700 1 -- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff7200b2910 msgr2=0x7ff7200b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:04.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.602+0000 7ff740400700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7ff7200b2910 0x7ff7200b4e20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00b2e0 tx=0x7ff72c004710 comp rx=0 tx=0).stop 2024-05-27T09:33:04.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.602+0000 7ff740400700 1 -- 172.21.15.44:0/2968403864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff738110bf0 msgr2=0x7ff7381a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:04.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.602+0000 7ff740400700 1 --2- 172.21.15.44:0/2968403864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff738110bf0 0x7ff7381a1910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff73000ef90 tx=0x7ff73000c3a0 comp rx=0 tx=0).stop 2024-05-27T09:33:04.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.602+0000 7ff73e400700 1 -- 172.21.15.44:0/2968403864 reap_dead start 2024-05-27T09:33:04.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.602+0000 7ff740400700 1 -- 172.21.15.44:0/2968403864 shutdown_connections 2024-05-27T09:33:04.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.602+0000 7ff740400700 1 -- 172.21.15.44:0/2968403864 >> 172.21.15.44:0/2968403864 conn(0x7ff738100030 msgr2=0x7ff738100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:04.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.602+0000 7ff740400700 1 -- 172.21.15.44:0/2968403864 shutdown_connections 2024-05-27T09:33:04.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:04.602+0000 7ff740400700 1 -- 172.21.15.44:0/2968403864 wait complete. 2024-05-27T09:33:05.698 INFO:teuthology.orchestra.run.smithi044.stdout:201863462972 2024-05-27T09:33:05.698 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-27T09:33:06.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:05 smithi189 ceph-mon[36671]: pgmap v375: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-27T09:33:06.165 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:05 smithi044 ceph-mon[31618]: pgmap v375: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-27T09:33:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:05 smithi050 ceph-mon[36371]: pgmap v375: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-27T09:33:06.339 INFO:teuthology.orchestra.run.smithi044.stdout:304942678054 2024-05-27T09:33:06.339 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-27T09:33:06.732 INFO:teuthology.orchestra.run.smithi044.stdout:34359738466 2024-05-27T09:33:06.732 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-27T09:33:07.352 INFO:teuthology.orchestra.run.smithi044.stdout:253403070512 2024-05-27T09:33:07.352 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-27T09:33:07.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.808+0000 7f3632000700 1 -- 172.21.15.44:0/155357171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f362c075b50 msgr2=0x7f362c075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:07.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.808+0000 7f3632000700 1 --2- 172.21.15.44:0/155357171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f362c075b50 0x7f362c075f60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f361c009990 tx=0x7f361c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.808+0000 7f3632000700 1 -- 172.21.15.44:0/155357171 shutdown_connections 2024-05-27T09:33:07.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.808+0000 7f3632000700 1 --2- 172.21.15.44:0/155357171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f362c115360 0x7f362c117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.808+0000 7f3632000700 1 --2- 172.21.15.44:0/155357171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f362c075b50 0x7f362c075f60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.808+0000 7f3632000700 1 --2- 172.21.15.44:0/155357171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f362c0764a0 0x7f362c114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.808+0000 7f3632000700 1 -- 172.21.15.44:0/155357171 >> 172.21.15.44:0/155357171 conn(0x7f362c100030 msgr2=0x7f362c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:07.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.808+0000 7f3632000700 1 -- 172.21.15.44:0/155357171 shutdown_connections 2024-05-27T09:33:07.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.808+0000 7f3632000700 1 -- 172.21.15.44:0/155357171 wait complete. 2024-05-27T09:33:07.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.810+0000 7f3632000700 1 Processor -- start 2024-05-27T09:33:07.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.810+0000 7f3632000700 1 -- start start 2024-05-27T09:33:07.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.810+0000 7f3632000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f362c075b50 0x7f362c1a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:07.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.810+0000 7f3632000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f362c0764a0 0x7f362c1a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:07.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.810+0000 7f3632000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f362c115360 0x7f362c1a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.810+0000 7f3632000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f362c11aab0 con 0x7f362c0764a0 2024-05-27T09:33:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.810+0000 7f3632000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f362c11a930 con 0x7f362c115360 2024-05-27T09:33:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.810+0000 7f3632000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f362c11ac30 con 0x7f362c075b50 2024-05-27T09:33:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.811+0000 7f362aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f362c0764a0 0x7f362c1a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.811+0000 7f362aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f362c0764a0 0x7f362c1a19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45418/0 (socket says 172.21.15.44:45418) 2024-05-27T09:33:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.811+0000 7f362aa00700 1 -- 172.21.15.44:0/3731367146 learned_addr learned my addr 172.21.15.44:0/3731367146 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.811+0000 7f362b400700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f362c075b50 0x7f362c1a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:07.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.811+0000 7f362be00700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f362c115360 0x7f362c1a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.811+0000 7f362b400700 1 -- 172.21.15.44:0/3731367146 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f362c115360 msgr2=0x7f362c1a5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.811+0000 7f362b400700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f362c115360 0x7f362c1a5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.811+0000 7f362b400700 1 -- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f362c0764a0 msgr2=0x7f362c1a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.811+0000 7f362b400700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f362c0764a0 0x7f362c1a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.811+0000 7f362b400700 1 -- 172.21.15.44:0/3731367146 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f362c1a62d0 con 0x7f362c075b50 2024-05-27T09:33:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.812+0000 7f362b400700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f362c075b50 0x7f362c1a1400 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f361c0098e0 tx=0x7f361c0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.812+0000 7f3623e00700 1 -- 172.21.15.44:0/3731367146 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361c027070 con 0x7f362c075b50 2024-05-27T09:33:07.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.812+0000 7f3623e00700 1 -- 172.21.15.44:0/3731367146 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f361c022470 con 0x7f362c075b50 2024-05-27T09:33:07.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.812+0000 7f3632000700 1 -- 172.21.15.44:0/3731367146 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f362c1a6560 con 0x7f362c075b50 2024-05-27T09:33:07.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.812+0000 7f3623e00700 1 -- 172.21.15.44:0/3731367146 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361c017650 con 0x7f362c075b50 2024-05-27T09:33:07.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.812+0000 7f3632000700 1 -- 172.21.15.44:0/3731367146 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f362c1a6a40 con 0x7f362c075b50 2024-05-27T09:33:07.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.813+0000 7f3632000700 1 -- 172.21.15.44:0/3731367146 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f362c063f40 con 0x7f362c075b50 2024-05-27T09:33:07.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.818+0000 7f3623e00700 1 -- 172.21.15.44:0/3731367146 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f361c02a070 con 0x7f362c075b50 2024-05-27T09:33:07.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.819+0000 7f3623e00700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f36100b2910 0x7f36100b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:07.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.819+0000 7f3623e00700 1 -- 172.21.15.44:0/3731367146 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f361c0e3540 con 0x7f362c075b50 2024-05-27T09:33:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.820+0000 7f362aa00700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f36100b2910 0x7f36100b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.820+0000 7f3623e00700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643] conn(0x7f36100bc4a0 0x7f36100be990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.820+0000 7f3623e00700 1 -- 172.21.15.44:0/3731367146 --> [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f361c012ec0 con 0x7f36100bc4a0 2024-05-27T09:33:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.820+0000 7f3623e00700 1 -- 172.21.15.44:0/3731367146 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f361c0e38f0 con 0x7f362c075b50 2024-05-27T09:33:07.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.820+0000 7f362be00700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643] conn(0x7f36100bc4a0 0x7f36100be990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:07.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.821+0000 7f362aa00700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f36100b2910 0x7f36100b4e20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f362c103490 tx=0x7f3618009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:07.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.821+0000 7f362be00700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643] conn(0x7f36100bc4a0 0x7f36100be990 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:07.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.822+0000 7f3623e00700 1 -- 172.21.15.44:0/3731367146 <== osd.6 v2:172.21.15.44:6818/44684643 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f361c012ec0 con 0x7f36100bc4a0 2024-05-27T09:33:07.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.842+0000 7f3632000700 1 -- 172.21.15.44:0/3731367146 --> [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f362c0643c0 con 0x7f36100bc4a0 2024-05-27T09:33:07.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.843+0000 7f3623e00700 1 -- 172.21.15.44:0/3731367146 <== osd.6 v2:172.21.15.44:6818/44684643 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f362c0643c0 con 0x7f36100bc4a0 2024-05-27T09:33:07.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.844+0000 7f3632000700 1 -- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643] conn(0x7f36100bc4a0 msgr2=0x7f36100be990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:07.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.844+0000 7f3632000700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643] conn(0x7f36100bc4a0 0x7f36100be990 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.844+0000 7f3632000700 1 -- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f36100b2910 msgr2=0x7f36100b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:07.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.844+0000 7f3632000700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f36100b2910 0x7f36100b4e20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f362c103490 tx=0x7f3618009260 comp rx=0 tx=0).stop 2024-05-27T09:33:07.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.844+0000 7f3632000700 1 -- 172.21.15.44:0/3731367146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f362c075b50 msgr2=0x7f362c1a1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:07.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.844+0000 7f3632000700 1 --2- 172.21.15.44:0/3731367146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f362c075b50 0x7f362c1a1400 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f361c0098e0 tx=0x7f361c0057c0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.845+0000 7f362be00700 1 -- 172.21.15.44:0/3731367146 reap_dead start 2024-05-27T09:33:07.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.845+0000 7f3632000700 1 -- 172.21.15.44:0/3731367146 shutdown_connections 2024-05-27T09:33:07.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.845+0000 7f3632000700 1 -- 172.21.15.44:0/3731367146 >> 172.21.15.44:0/3731367146 conn(0x7f362c100030 msgr2=0x7f362c100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:07.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.845+0000 7f3632000700 1 -- 172.21.15.44:0/3731367146 shutdown_connections 2024-05-27T09:33:07.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.845+0000 7f3632000700 1 -- 172.21.15.44:0/3731367146 wait complete. 2024-05-27T09:33:07.883 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:07.888 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:07.897 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:07.924 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:07.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.927+0000 7f7aae200700 1 -- 172.21.15.44:0/2383171897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa81111d0 msgr2=0x7f7aa81178e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:07.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.927+0000 7f7aae200700 1 --2- 172.21.15.44:0/2383171897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa81111d0 0x7f7aa81178e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000b0a0 tx=0x7f7aa000b3b0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.930+0000 7f7aae200700 1 -- 172.21.15.44:0/2383171897 shutdown_connections 2024-05-27T09:33:07.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.930+0000 7f7aae200700 1 --2- 172.21.15.44:0/2383171897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa81111d0 0x7f7aa81178e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.930+0000 7f7aae200700 1 --2- 172.21.15.44:0/2383171897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7aa810b0c0 0x7f7aa81105e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.930+0000 7f7aae200700 1 --2- 172.21.15.44:0/2383171897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7aa810b560 0x7f7aa8110b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.930+0000 7f7aae200700 1 -- 172.21.15.44:0/2383171897 >> 172.21.15.44:0/2383171897 conn(0x7f7aa806f4c0 msgr2=0x7f7aa806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:07.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.931+0000 7f7aae200700 1 -- 172.21.15.44:0/2383171897 shutdown_connections 2024-05-27T09:33:07.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.931+0000 7f7aae200700 1 -- 172.21.15.44:0/2383171897 wait complete. 2024-05-27T09:33:07.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.936+0000 7f7aae200700 1 Processor -- start 2024-05-27T09:33:07.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.936+0000 7f7aae200700 1 -- start start 2024-05-27T09:33:07.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.940+0000 7f7aae200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7aa810b0c0 0x7f7aa81af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:07.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.940+0000 7f7aae200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa81af660 0x7f7aa81a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.940+0000 7f7aae200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7aa81a9e10 0x7f7aa81aa220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.940+0000 7f7aae200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa811bc10 con 0x7f7aa81a9e10 2024-05-27T09:33:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.940+0000 7f7aae200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa811ba90 con 0x7f7aa81af660 2024-05-27T09:33:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.940+0000 7f7aae200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa811bd90 con 0x7f7aa810b0c0 2024-05-27T09:33:07.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.940+0000 7f7aa7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7aa810b0c0 0x7f7aa81af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.941+0000 7f7aa7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7aa810b0c0 0x7f7aa81af120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50974/0 (socket says 172.21.15.44:50974) 2024-05-27T09:33:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.941+0000 7f7aa7400700 1 -- 172.21.15.44:0/1218960887 learned_addr learned my addr 172.21.15.44:0/1218960887 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:07.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.941+0000 7f7aa6a00700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa81af660 0x7f7aa81a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:07.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.942+0000 7f7aa7e00700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7aa81a9e10 0x7f7aa81aa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:07.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.943+0000 7f7aa7e00700 1 -- 172.21.15.44:0/1218960887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7aa810b0c0 msgr2=0x7f7aa81af120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:07.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.943+0000 7f7aa7e00700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7aa810b0c0 0x7f7aa81af120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.943+0000 7f7aa7e00700 1 -- 172.21.15.44:0/1218960887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa81af660 msgr2=0x7f7aa81a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:07.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.943+0000 7f7aa7e00700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa81af660 0x7f7aa81a98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:07.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.943+0000 7f7aa7e00700 1 -- 172.21.15.44:0/1218960887 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa81b5ef0 con 0x7f7aa81a9e10 2024-05-27T09:33:07.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.944+0000 7f7aa7e00700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7aa81a9e10 0x7f7aa81aa220 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7aa00062a0 tx=0x7f7aa0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:07.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.945+0000 7f7a9fe00700 1 -- 172.21.15.44:0/1218960887 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0021070 con 0x7f7aa81a9e10 2024-05-27T09:33:07.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.945+0000 7f7aae200700 1 -- 172.21.15.44:0/1218960887 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa81b60c0 con 0x7f7aa81a9e10 2024-05-27T09:33:07.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.945+0000 7f7a9fe00700 1 -- 172.21.15.44:0/1218960887 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7aa0007e30 con 0x7f7aa81a9e10 2024-05-27T09:33:07.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.945+0000 7f7a9fe00700 1 -- 172.21.15.44:0/1218960887 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0021070 con 0x7f7aa81a9e10 2024-05-27T09:33:07.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.948+0000 7f7aae200700 1 -- 172.21.15.44:0/1218960887 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa81b6520 con 0x7f7aa81a9e10 2024-05-27T09:33:07.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.957+0000 7f7a9d600700 1 -- 172.21.15.44:0/1218960887 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7a80001030 con 0x7f7aa81a9e10 2024-05-27T09:33:07.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.959+0000 7f7a9fe00700 1 -- 172.21.15.44:0/1218960887 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f7aa00043f0 con 0x7f7aa81a9e10 2024-05-27T09:33:07.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.962+0000 7f7a9fe00700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f7a8c0b2b10 0x7f7a8c0b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:07.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.966+0000 7f7a9fe00700 1 -- 172.21.15.44:0/1218960887 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f7aa000f020 con 0x7f7aa81a9e10 2024-05-27T09:33:07.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.966+0000 7f7aa7400700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f7a8c0b2b10 0x7f7a8c0b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:07.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.966+0000 7f7a9fe00700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881] conn(0x7f7a8c0bc6a0 0x7f7a8c0beb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:07.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.966+0000 7f7a9fe00700 1 -- 172.21.15.44:0/1218960887 --> [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7aa0021e80 con 0x7f7a8c0bc6a0 2024-05-27T09:33:07.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.967+0000 7f7a9fe00700 1 -- 172.21.15.44:0/1218960887 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f7aa00e9050 con 0x7f7aa81a9e10 2024-05-27T09:33:07.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.967+0000 7f7aa6a00700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881] conn(0x7f7a8c0bc6a0 0x7f7a8c0beb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:07.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.967+0000 7f7aa7400700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f7a8c0b2b10 0x7f7a8c0b5020 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009680 tx=0x7f7a98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:07.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.968+0000 7f7aa6a00700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881] conn(0x7f7a8c0bc6a0 0x7f7a8c0beb90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:07.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:07.969+0000 7f7a9fe00700 1 -- 172.21.15.44:0/1218960887 <== osd.5 v2:172.21.15.189:6808/2756976881 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7aa0021e80 con 0x7f7a8c0bc6a0 2024-05-27T09:33:08.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.001+0000 7f7a9d600700 1 -- 172.21.15.44:0/1218960887 --> [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7a80002e60 con 0x7f7a8c0bc6a0 2024-05-27T09:33:08.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:07 smithi044 ceph-mon[31618]: pgmap v376: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T09:33:08.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.002+0000 7f7a9fe00700 1 -- 172.21.15.44:0/1218960887 <== osd.5 v2:172.21.15.189:6808/2756976881 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7a80002e60 con 0x7f7a8c0bc6a0 2024-05-27T09:33:08.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.003+0000 7f7a9d600700 1 -- 172.21.15.44:0/1218960887 >> [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881] conn(0x7f7a8c0bc6a0 msgr2=0x7f7a8c0beb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:08.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.003+0000 7f7a9d600700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881] conn(0x7f7a8c0bc6a0 0x7f7a8c0beb90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:08.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.005+0000 7f7a9d600700 1 -- 172.21.15.44:0/1218960887 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f7a8c0b2b10 msgr2=0x7f7a8c0b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:08.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.005+0000 7f7a9d600700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f7a8c0b2b10 0x7f7a8c0b5020 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009680 tx=0x7f7a98009260 comp rx=0 tx=0).stop 2024-05-27T09:33:08.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.005+0000 7f7a9d600700 1 -- 172.21.15.44:0/1218960887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7aa81a9e10 msgr2=0x7f7aa81aa220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:08.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.005+0000 7f7a9d600700 1 --2- 172.21.15.44:0/1218960887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7aa81a9e10 0x7f7aa81aa220 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7aa00062a0 tx=0x7f7aa0004c80 comp rx=0 tx=0).stop 2024-05-27T09:33:08.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.005+0000 7f7aa7e00700 1 -- 172.21.15.44:0/1218960887 reap_dead start 2024-05-27T09:33:08.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.005+0000 7f7a9d600700 1 -- 172.21.15.44:0/1218960887 shutdown_connections 2024-05-27T09:33:08.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.005+0000 7f7a9d600700 1 -- 172.21.15.44:0/1218960887 >> 172.21.15.44:0/1218960887 conn(0x7f7aa806f4c0 msgr2=0x7f7aa8072780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:08.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.007+0000 7f7a9d600700 1 -- 172.21.15.44:0/1218960887 shutdown_connections 2024-05-27T09:33:08.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:08.008+0000 7f7a9d600700 1 -- 172.21.15.44:0/1218960887 wait complete. 2024-05-27T09:33:08.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:07 smithi050 ceph-mon[36371]: pgmap v376: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T09:33:08.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:07 smithi189 ceph-mon[36671]: pgmap v376: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T09:33:10.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:09 smithi044 ceph-mon[31618]: pgmap v377: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-05-27T09:33:10.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:09 smithi050 ceph-mon[36371]: pgmap v377: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-05-27T09:33:10.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:09 smithi189 ceph-mon[36671]: pgmap v377: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-05-27T09:33:11.895 INFO:teuthology.orchestra.run.smithi044.stdout:154618822730 2024-05-27T09:33:11.895 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-27T09:33:12.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:11 smithi044 ceph-mon[31618]: pgmap v378: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-05-27T09:33:12.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:11 smithi050 ceph-mon[36371]: pgmap v378: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-05-27T09:33:12.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:11 smithi189 ceph-mon[36671]: pgmap v378: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-05-27T09:33:13.588 INFO:teuthology.orchestra.run.smithi044.stdout:180388626498 2024-05-27T09:33:13.589 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-27T09:33:13.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.889+0000 7f9e85000700 1 -- 172.21.15.44:0/2738137126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e801034b0 msgr2=0x7f9e801038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:13.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.889+0000 7f9e85000700 1 --2- 172.21.15.44:0/2738137126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e801034b0 0x7f9e801038c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9e700098f0 tx=0x7f9e70009c00 comp rx=0 tx=0).stop 2024-05-27T09:33:13.892 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:13 smithi044 ceph-mon[31618]: pgmap v379: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:13.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.889+0000 7f9e85000700 1 -- 172.21.15.44:0/2738137126 shutdown_connections 2024-05-27T09:33:13.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.889+0000 7f9e85000700 1 --2- 172.21.15.44:0/2738137126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e80108730 0x7f9e80110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:13.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.889+0000 7f9e85000700 1 --2- 172.21.15.44:0/2738137126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e801034b0 0x7f9e801038c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:13.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.889+0000 7f9e85000700 1 --2- 172.21.15.44:0/2738137126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e80107de0 0x7f9e801081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:13.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.889+0000 7f9e85000700 1 -- 172.21.15.44:0/2738137126 >> 172.21.15.44:0/2738137126 conn(0x7f9e800fedb0 msgr2=0x7f9e801011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:13.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.889+0000 7f9e85000700 1 -- 172.21.15.44:0/2738137126 shutdown_connections 2024-05-27T09:33:13.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.889+0000 7f9e85000700 1 -- 172.21.15.44:0/2738137126 wait complete. 2024-05-27T09:33:13.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.890+0000 7f9e85000700 1 Processor -- start 2024-05-27T09:33:13.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.891+0000 7f9e85000700 1 -- start start 2024-05-27T09:33:13.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.891+0000 7f9e85000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e801034b0 0x7f9e8019d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:13.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.891+0000 7f9e85000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e80107de0 0x7f9e8019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:13.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.891+0000 7f9e85000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e80108730 0x7f9e801a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:13.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.891+0000 7f9e85000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9e801133b0 con 0x7f9e801034b0 2024-05-27T09:33:13.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.891+0000 7f9e85000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9e80113230 con 0x7f9e80108730 2024-05-27T09:33:13.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.891+0000 7f9e85000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e80113530 con 0x7f9e80107de0 2024-05-27T09:33:13.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.892+0000 7f9e7de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e801034b0 0x7f9e8019d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:13.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.892+0000 7f9e7de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e801034b0 0x7f9e8019d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38730/0 (socket says 172.21.15.44:38730) 2024-05-27T09:33:13.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.892+0000 7f9e7de00700 1 -- 172.21.15.44:0/82884988 learned_addr learned my addr 172.21.15.44:0/82884988 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:13.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.892+0000 7f9e7e800700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e80108730 0x7f9e801a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:13.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.892+0000 7f9e7d400700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e80107de0 0x7f9e8019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:13.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.892+0000 7f9e7de00700 1 -- 172.21.15.44:0/82884988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e80107de0 msgr2=0x7f9e8019d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:13.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.892+0000 7f9e7de00700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e80107de0 0x7f9e8019d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:13.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.892+0000 7f9e7de00700 1 -- 172.21.15.44:0/82884988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e80108730 msgr2=0x7f9e801a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:13.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.892+0000 7f9e7de00700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e80108730 0x7f9e801a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:13.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.892+0000 7f9e7de00700 1 -- 172.21.15.44:0/82884988 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e801a1f00 con 0x7f9e801034b0 2024-05-27T09:33:13.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.893+0000 7f9e7de00700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e801034b0 0x7f9e8019d030 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009230 tx=0x7f9e7000fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:13.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.893+0000 7f9e76a00700 1 -- 172.21.15.44:0/82884988 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70005140 con 0x7f9e801034b0 2024-05-27T09:33:13.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.893+0000 7f9e85000700 1 -- 172.21.15.44:0/82884988 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e801a2190 con 0x7f9e801034b0 2024-05-27T09:33:13.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.893+0000 7f9e85000700 1 -- 172.21.15.44:0/82884988 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e801a9b10 con 0x7f9e801034b0 2024-05-27T09:33:13.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.894+0000 7f9e76a00700 1 -- 172.21.15.44:0/82884988 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e7000b560 con 0x7f9e801034b0 2024-05-27T09:33:13.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.894+0000 7f9e76a00700 1 -- 172.21.15.44:0/82884988 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70003ae0 con 0x7f9e801034b0 2024-05-27T09:33:13.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.895+0000 7f9e85000700 1 -- 172.21.15.44:0/82884988 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e60005330 con 0x7f9e801034b0 2024-05-27T09:33:13.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.898+0000 7f9e76a00700 1 -- 172.21.15.44:0/82884988 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9e70003c80 con 0x7f9e801034b0 2024-05-27T09:33:13.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.900+0000 7f9e76a00700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9e640b2990 0x7f9e640b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:13.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.900+0000 7f9e76a00700 1 -- 172.21.15.44:0/82884988 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f9e700e33f0 con 0x7f9e801034b0 2024-05-27T09:33:13.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.900+0000 7f9e7d400700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9e640b2990 0x7f9e640b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:13.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.901+0000 7f9e7d400700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9e640b2990 0x7f9e640b4ea0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009750 tx=0x7f9e6c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:13.904+0000 7f9e76a00700 1 -- 172.21.15.44:0/82884988 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9e7005fdf0 con 0x7f9e801034b0 2024-05-27T09:33:14.140 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:13 smithi189 ceph-mon[36671]: pgmap v379: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:14.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.149+0000 7f9e85000700 1 -- 172.21.15.44:0/82884988 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f9e600058f0 con 0x7f9e801034b0 2024-05-27T09:33:14.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.151+0000 7f9e76a00700 1 -- 172.21.15.44:0/82884988 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9e70013030 con 0x7f9e801034b0 2024-05-27T09:33:14.153 INFO:teuthology.orchestra.run.smithi044.stdout:77309411421 2024-05-27T09:33:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.154+0000 7f9e85000700 1 -- 172.21.15.44:0/82884988 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9e640b2990 msgr2=0x7f9e640b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.154+0000 7f9e85000700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9e640b2990 0x7f9e640b4ea0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009750 tx=0x7f9e6c009310 comp rx=0 tx=0).stop 2024-05-27T09:33:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.154+0000 7f9e85000700 1 -- 172.21.15.44:0/82884988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e801034b0 msgr2=0x7f9e8019d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.154+0000 7f9e85000700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e801034b0 0x7f9e8019d030 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009230 tx=0x7f9e7000fa70 comp rx=0 tx=0).stop 2024-05-27T09:33:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.154+0000 7f9e85000700 1 -- 172.21.15.44:0/82884988 shutdown_connections 2024-05-27T09:33:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.154+0000 7f9e85000700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f9e640b2990 0x7f9e640b4ea0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.154+0000 7f9e85000700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e80108730 0x7f9e801a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.154+0000 7f9e85000700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e801034b0 0x7f9e8019d030 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.154+0000 7f9e85000700 1 --2- 172.21.15.44:0/82884988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e80107de0 0x7f9e8019d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.154+0000 7f9e85000700 1 -- 172.21.15.44:0/82884988 >> 172.21.15.44:0/82884988 conn(0x7f9e800fedb0 msgr2=0x7f9e801063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.154+0000 7f9e85000700 1 -- 172.21.15.44:0/82884988 shutdown_connections 2024-05-27T09:33:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.155+0000 7f9e85000700 1 -- 172.21.15.44:0/82884988 wait complete. 2024-05-27T09:33:14.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:13 smithi050 ceph-mon[36371]: pgmap v379: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:14.787 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:14.789 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:14.860 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:14 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/82884988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T09:33:14.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.863+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1145431060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0075af0 msgr2=0x7fe8c0075f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:14.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.863+0000 7fe8c5c00700 1 --2- 172.21.15.44:0/1145431060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0075af0 0x7fe8c0075f00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0009990 tx=0x7fe8b0009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:14.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.863+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1145431060 shutdown_connections 2024-05-27T09:33:14.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.863+0000 7fe8c5c00700 1 --2- 172.21.15.44:0/1145431060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c0110bf0 0x7fe8c0113010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:14.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.863+0000 7fe8c5c00700 1 --2- 172.21.15.44:0/1145431060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0075af0 0x7fe8c0075f00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:14.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.863+0000 7fe8c5c00700 1 --2- 172.21.15.44:0/1145431060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8c0076440 0x7fe8c01106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:14.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.863+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1145431060 >> 172.21.15.44:0/1145431060 conn(0x7fe8c0100030 msgr2=0x7fe8c0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:14.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.863+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1145431060 shutdown_connections 2024-05-27T09:33:14.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.863+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1145431060 wait complete. 2024-05-27T09:33:14.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.865+0000 7fe8c5c00700 1 Processor -- start 2024-05-27T09:33:14.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.865+0000 7fe8c5c00700 1 -- start start 2024-05-27T09:33:14.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.865+0000 7fe8c5c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8c0075af0 0x7fe8c019d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:14.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.865+0000 7fe8c5c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c0076440 0x7fe8c019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:14.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.865+0000 7fe8c5c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0110bf0 0x7fe8c01a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:14.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.865+0000 7fe8c5c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c0116720 con 0x7fe8c0075af0 2024-05-27T09:33:14.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.865+0000 7fe8c5c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c01165a0 con 0x7fe8c0076440 2024-05-27T09:33:14.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.865+0000 7fe8c5c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c01168a0 con 0x7fe8c0110bf0 2024-05-27T09:33:14.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.866+0000 7fe8be000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c0076440 0x7fe8c019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:14.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.866+0000 7fe8bea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8c0075af0 0x7fe8c019d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:14.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.866+0000 7fe8be000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c0076440 0x7fe8c019d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49686/0 (socket says 172.21.15.44:49686) 2024-05-27T09:33:14.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.866+0000 7fe8be000700 1 -- 172.21.15.44:0/1357827699 learned_addr learned my addr 172.21.15.44:0/1357827699 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:14.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.866+0000 7fe8bf400700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0110bf0 0x7fe8c01a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:14.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.866+0000 7fe8bea00700 1 -- 172.21.15.44:0/1357827699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0110bf0 msgr2=0x7fe8c01a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:14.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.866+0000 7fe8bea00700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0110bf0 0x7fe8c01a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:14.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.866+0000 7fe8bea00700 1 -- 172.21.15.44:0/1357827699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c0076440 msgr2=0x7fe8c019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:14.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.866+0000 7fe8bea00700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c0076440 0x7fe8c019d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:14.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.866+0000 7fe8bea00700 1 -- 172.21.15.44:0/1357827699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c01a1f80 con 0x7fe8c0075af0 2024-05-27T09:33:14.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.867+0000 7fe8bea00700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8c0075af0 0x7fe8c019d0b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0000c00 tx=0x7fe8b00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:14.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.867+0000 7fe8b7400700 1 -- 172.21.15.44:0/1357827699 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0026070 con 0x7fe8c0075af0 2024-05-27T09:33:14.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.867+0000 7fe8b7400700 1 -- 172.21.15.44:0/1357827699 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8b0021410 con 0x7fe8c0075af0 2024-05-27T09:33:14.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.867+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1357827699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8c01a2210 con 0x7fe8c0075af0 2024-05-27T09:33:14.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.867+0000 7fe8b7400700 1 -- 172.21.15.44:0/1357827699 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0017650 con 0x7fe8c0075af0 2024-05-27T09:33:14.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.867+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1357827699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8c01a2680 con 0x7fe8c0075af0 2024-05-27T09:33:14.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.868+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1357827699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8c00512b0 con 0x7fe8c0075af0 2024-05-27T09:33:14.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.871+0000 7fe8b7400700 1 -- 172.21.15.44:0/1357827699 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fe8b0029070 con 0x7fe8c0075af0 2024-05-27T09:33:14.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.873+0000 7fe8b7400700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe8a40b2990 0x7fe8a40b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:14.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.874+0000 7fe8b7400700 1 -- 172.21.15.44:0/1357827699 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fe8b00e31d0 con 0x7fe8c0075af0 2024-05-27T09:33:14.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.874+0000 7fe8be000700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe8a40b2990 0x7fe8a40b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:14.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.874+0000 7fe8be000700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe8a40b2990 0x7fe8a40b4ea0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0103480 tx=0x7fe8ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:14.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:14.878+0000 7fe8b7400700 1 -- 172.21.15.44:0/1357827699 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe8b0013070 con 0x7fe8c0075af0 2024-05-27T09:33:15.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.122+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1357827699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fe8c002e5e0 con 0x7fe8c0075af0 2024-05-27T09:33:15.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.123+0000 7fe8b7400700 1 -- 172.21.15.44:0/1357827699 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe8b00aadd0 con 0x7fe8c0075af0 2024-05-27T09:33:15.125 INFO:teuthology.orchestra.run.smithi044.stdout:55834574946 2024-05-27T09:33:15.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.126+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1357827699 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe8a40b2990 msgr2=0x7fe8a40b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.126+0000 7fe8c5c00700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe8a40b2990 0x7fe8a40b4ea0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0103480 tx=0x7fe8ac009260 comp rx=0 tx=0).stop 2024-05-27T09:33:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.126+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1357827699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8c0075af0 msgr2=0x7fe8c019d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.126+0000 7fe8c5c00700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8c0075af0 0x7fe8c019d0b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0000c00 tx=0x7fe8b00047b0 comp rx=0 tx=0).stop 2024-05-27T09:33:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.126+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1357827699 shutdown_connections 2024-05-27T09:33:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.126+0000 7fe8c5c00700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe8a40b2990 0x7fe8a40b4ea0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.126+0000 7fe8c5c00700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0110bf0 0x7fe8c01a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.126+0000 7fe8c5c00700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8c0075af0 0x7fe8c019d0b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.126+0000 7fe8c5c00700 1 --2- 172.21.15.44:0/1357827699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8c0076440 0x7fe8c019d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.126+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1357827699 >> 172.21.15.44:0/1357827699 conn(0x7fe8c0100030 msgr2=0x7fe8c01006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.127+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1357827699 shutdown_connections 2024-05-27T09:33:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:15.127+0000 7fe8c5c00700 1 -- 172.21.15.44:0/1357827699 wait complete. 2024-05-27T09:33:15.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:14 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/82884988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T09:33:15.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:14 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/82884988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T09:33:15.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:15 smithi044 ceph-mon[31618]: pgmap v380: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:15.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:15 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1357827699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T09:33:16.224 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411416 got 77309411421 for osd.2 2024-05-27T09:33:16.224 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:16.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:15 smithi050 ceph-mon[36371]: pgmap v380: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:15 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1357827699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T09:33:16.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:15 smithi189 ceph-mon[36671]: pgmap v380: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:16.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:15 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1357827699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T09:33:16.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.659+0000 7fd844e00700 1 -- 172.21.15.44:0/1640970692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd84010c5b0 msgr2=0x7fd84010c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:16.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.659+0000 7fd844e00700 1 --2- 172.21.15.44:0/1640970692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd84010c5b0 0x7fd84010c9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd830009990 tx=0x7fd830009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:16.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.659+0000 7fd844e00700 1 -- 172.21.15.44:0/1640970692 shutdown_connections 2024-05-27T09:33:16.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.659+0000 7fd844e00700 1 --2- 172.21.15.44:0/1640970692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd840106ec0 0x7fd840113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:16.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.659+0000 7fd844e00700 1 --2- 172.21.15.44:0/1640970692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd84010c5b0 0x7fd84010c9c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:16.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.659+0000 7fd844e00700 1 --2- 172.21.15.44:0/1640970692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd840106570 0x7fd840106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:16.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.659+0000 7fd844e00700 1 -- 172.21.15.44:0/1640970692 >> 172.21.15.44:0/1640970692 conn(0x7fd840102070 msgr2=0x7fd8401044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:16.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.659+0000 7fd844e00700 1 -- 172.21.15.44:0/1640970692 shutdown_connections 2024-05-27T09:33:16.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.659+0000 7fd844e00700 1 -- 172.21.15.44:0/1640970692 wait complete. 2024-05-27T09:33:16.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.661+0000 7fd844e00700 1 Processor -- start 2024-05-27T09:33:16.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.661+0000 7fd844e00700 1 -- start start 2024-05-27T09:33:16.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.662+0000 7fd844e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd840106570 0x7fd84019d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:16.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.662+0000 7fd844e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840106ec0 0x7fd84019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:16.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.662+0000 7fd844e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84010c5b0 0x7fd8401a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:16.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.662+0000 7fd844e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd840116710 con 0x7fd840106ec0 2024-05-27T09:33:16.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.662+0000 7fd844e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd840116590 con 0x7fd840106570 2024-05-27T09:33:16.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.662+0000 7fd844e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd840116890 con 0x7fd84010c5b0 2024-05-27T09:33:16.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.662+0000 7fd83d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840106ec0 0x7fd84019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:16.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.662+0000 7fd83de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd840106570 0x7fd84019d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.662+0000 7fd83d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840106ec0 0x7fd84019d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38776/0 (socket says 172.21.15.44:38776) 2024-05-27T09:33:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.662+0000 7fd83d400700 1 -- 172.21.15.44:0/4144763678 learned_addr learned my addr 172.21.15.44:0/4144763678 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.662+0000 7fd83e800700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84010c5b0 0x7fd8401a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.663+0000 7fd83de00700 1 -- 172.21.15.44:0/4144763678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84010c5b0 msgr2=0x7fd8401a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.663+0000 7fd83de00700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84010c5b0 0x7fd8401a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.663+0000 7fd83de00700 1 -- 172.21.15.44:0/4144763678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840106ec0 msgr2=0x7fd84019d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.663+0000 7fd83de00700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840106ec0 0x7fd84019d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.663+0000 7fd83de00700 1 -- 172.21.15.44:0/4144763678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8401a1f00 con 0x7fd840106570 2024-05-27T09:33:16.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.663+0000 7fd83de00700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd840106570 0x7fd84019d080 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd8300098e0 tx=0x7fd8300057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:16.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.663+0000 7fd836a00700 1 -- 172.21.15.44:0/4144763678 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd830027070 con 0x7fd840106570 2024-05-27T09:33:16.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.664+0000 7fd844e00700 1 -- 172.21.15.44:0/4144763678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8401a2190 con 0x7fd840106570 2024-05-27T09:33:16.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.664+0000 7fd836a00700 1 -- 172.21.15.44:0/4144763678 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd830022470 con 0x7fd840106570 2024-05-27T09:33:16.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.664+0000 7fd836a00700 1 -- 172.21.15.44:0/4144763678 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd830017650 con 0x7fd840106570 2024-05-27T09:33:16.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.664+0000 7fd844e00700 1 -- 172.21.15.44:0/4144763678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8401a9b10 con 0x7fd840106570 2024-05-27T09:33:16.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.665+0000 7fd844e00700 1 -- 172.21.15.44:0/4144763678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8400512b0 con 0x7fd840106570 2024-05-27T09:33:16.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.668+0000 7fd836a00700 1 -- 172.21.15.44:0/4144763678 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd83002a070 con 0x7fd840106570 2024-05-27T09:33:16.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.669+0000 7fd836a00700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8240b2910 0x7fd8240b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:16.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.669+0000 7fd836a00700 1 -- 172.21.15.44:0/4144763678 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fd8300e3230 con 0x7fd840106570 2024-05-27T09:33:16.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.669+0000 7fd83d400700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8240b2910 0x7fd8240b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:16.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.670+0000 7fd83d400700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8240b2910 0x7fd8240b4e20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd840103da0 tx=0x7fd82c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:16.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.672+0000 7fd836a00700 1 -- 172.21.15.44:0/4144763678 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd830013070 con 0x7fd840106570 2024-05-27T09:33:16.882 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:16.892 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:16.894 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:16.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.958+0000 7fd844e00700 1 -- 172.21.15.44:0/4144763678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fd84002e5e0 con 0x7fd840106570 2024-05-27T09:33:16.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.959+0000 7fd836a00700 1 -- 172.21.15.44:0/4144763678 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd8300aae30 con 0x7fd840106570 2024-05-27T09:33:16.961 INFO:teuthology.orchestra.run.smithi044.stdout:227633266745 2024-05-27T09:33:16.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.963+0000 7fd844e00700 1 -- 172.21.15.44:0/4144763678 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8240b2910 msgr2=0x7fd8240b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:16.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.964+0000 7fd844e00700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8240b2910 0x7fd8240b4e20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd840103da0 tx=0x7fd82c009260 comp rx=0 tx=0).stop 2024-05-27T09:33:16.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.964+0000 7fd844e00700 1 -- 172.21.15.44:0/4144763678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd840106570 msgr2=0x7fd84019d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.964+0000 7fd844e00700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd840106570 0x7fd84019d080 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd8300098e0 tx=0x7fd8300057c0 comp rx=0 tx=0).stop 2024-05-27T09:33:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.964+0000 7fd844e00700 1 -- 172.21.15.44:0/4144763678 shutdown_connections 2024-05-27T09:33:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.964+0000 7fd844e00700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd8240b2910 0x7fd8240b4e20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.964+0000 7fd844e00700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84010c5b0 0x7fd8401a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.964+0000 7fd844e00700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd840106570 0x7fd84019d080 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.964+0000 7fd844e00700 1 --2- 172.21.15.44:0/4144763678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840106ec0 0x7fd84019d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.964+0000 7fd844e00700 1 -- 172.21.15.44:0/4144763678 >> 172.21.15.44:0/4144763678 conn(0x7fd840102070 msgr2=0x7fd8401048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:16.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.964+0000 7fd844e00700 1 -- 172.21.15.44:0/4144763678 shutdown_connections 2024-05-27T09:33:16.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:16.965+0000 7fd844e00700 1 -- 172.21.15.44:0/4144763678 wait complete. 2024-05-27T09:33:17.961 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:17 smithi044 ceph-mon[31618]: pgmap v381: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:17.961 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:17 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4144763678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T09:33:18.019 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574941 got 55834574946 for osd.1 2024-05-27T09:33:18.019 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:18.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:17 smithi050 ceph-mon[36371]: pgmap v381: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:18.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:17 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/4144763678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T09:33:18.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:17 smithi189 ceph-mon[36671]: pgmap v381: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:18.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:17 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/4144763678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T09:33:19.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.849+0000 7fd4fda00700 1 -- 172.21.15.44:0/2666346119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f810af40 msgr2=0x7fd4f810b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:19.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.849+0000 7fd4fda00700 1 --2- 172.21.15.44:0/2666346119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f810af40 0x7fd4f810b350 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40099c0 tx=0x7fd4e4009cd0 comp rx=0 tx=0).stop 2024-05-27T09:33:19.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.849+0000 7fd4fda00700 1 -- 172.21.15.44:0/2666346119 shutdown_connections 2024-05-27T09:33:19.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.849+0000 7fd4fda00700 1 --2- 172.21.15.44:0/2666346119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f810b890 0x7fd4f81131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:19.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.849+0000 7fd4fda00700 1 --2- 172.21.15.44:0/2666346119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f8106610 0x7fd4f8106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:19.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.849+0000 7fd4fda00700 1 --2- 172.21.15.44:0/2666346119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f810af40 0x7fd4f810b350 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:19.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.849+0000 7fd4fda00700 1 -- 172.21.15.44:0/2666346119 >> 172.21.15.44:0/2666346119 conn(0x7fd4f8102020 msgr2=0x7fd4f8104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:19.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.849+0000 7fd4fda00700 1 -- 172.21.15.44:0/2666346119 shutdown_connections 2024-05-27T09:33:19.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.849+0000 7fd4fda00700 1 -- 172.21.15.44:0/2666346119 wait complete. 2024-05-27T09:33:19.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.851+0000 7fd4fda00700 1 Processor -- start 2024-05-27T09:33:19.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.851+0000 7fd4fda00700 1 -- start start 2024-05-27T09:33:19.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.851+0000 7fd4fda00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f8106610 0x7fd4f819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:19.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.851+0000 7fd4fda00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f810af40 0x7fd4f819d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:19.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.851+0000 7fd4fda00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f810b890 0x7fd4f81a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:19.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.851+0000 7fd4fda00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f81164f0 con 0x7fd4f8106610 2024-05-27T09:33:19.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.851+0000 7fd4fda00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f8116370 con 0x7fd4f810b890 2024-05-27T09:33:19.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.851+0000 7fd4fda00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f8116670 con 0x7fd4f810af40 2024-05-27T09:33:19.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.852+0000 7fd4f7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f8106610 0x7fd4f819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:19.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.852+0000 7fd4f7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f8106610 0x7fd4f819ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38800/0 (socket says 172.21.15.44:38800) 2024-05-27T09:33:19.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.852+0000 7fd4f7e00700 1 -- 172.21.15.44:0/3593212712 learned_addr learned my addr 172.21.15.44:0/3593212712 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:19.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.852+0000 7fd4f7400700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f810af40 0x7fd4f819d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:19.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.852+0000 7fd4fd000700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f810b890 0x7fd4f81a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:19.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.852+0000 7fd4f7e00700 1 -- 172.21.15.44:0/3593212712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f810af40 msgr2=0x7fd4f819d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:19.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.852+0000 7fd4f7e00700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f810af40 0x7fd4f819d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:19.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.852+0000 7fd4f7e00700 1 -- 172.21.15.44:0/3593212712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f810b890 msgr2=0x7fd4f81a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:19.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.852+0000 7fd4f7e00700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f810b890 0x7fd4f81a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:19.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.852+0000 7fd4f7e00700 1 -- 172.21.15.44:0/3593212712 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f81a1d30 con 0x7fd4f8106610 2024-05-27T09:33:19.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.852+0000 7fd4fd000700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f810b890 0x7fd4f81a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:33:19.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.853+0000 7fd4f7e00700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f8106610 0x7fd4f819ce60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd4e800e9a0 tx=0x7fd4e800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:19.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.853+0000 7fd4f4c00700 1 -- 172.21.15.44:0/3593212712 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e800cbd0 con 0x7fd4f8106610 2024-05-27T09:33:19.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.853+0000 7fd4f4c00700 1 -- 172.21.15.44:0/3593212712 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4e800cd70 con 0x7fd4f8106610 2024-05-27T09:33:19.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.853+0000 7fd4f4c00700 1 -- 172.21.15.44:0/3593212712 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8018680 con 0x7fd4f8106610 2024-05-27T09:33:19.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.853+0000 7fd4fda00700 1 -- 172.21.15.44:0/3593212712 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4f81a2020 con 0x7fd4f8106610 2024-05-27T09:33:19.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.853+0000 7fd4fda00700 1 -- 172.21.15.44:0/3593212712 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4f81a9940 con 0x7fd4f8106610 2024-05-27T09:33:19.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.854+0000 7fd4fda00700 1 -- 172.21.15.44:0/3593212712 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4f80512b0 con 0x7fd4f8106610 2024-05-27T09:33:19.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.857+0000 7fd4f4c00700 1 -- 172.21.15.44:0/3593212712 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd4e8010430 con 0x7fd4f8106610 2024-05-27T09:33:19.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.859+0000 7fd4f4c00700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd4dc0b2990 0x7fd4dc0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:19.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.859+0000 7fd4f4c00700 1 -- 172.21.15.44:0/3593212712 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fd4e80e2aa0 con 0x7fd4f8106610 2024-05-27T09:33:19.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.859+0000 7fd4f7400700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd4dc0b2990 0x7fd4dc0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:19.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.860+0000 7fd4f7400700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd4dc0b2990 0x7fd4dc0b4ea0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4000c00 tx=0x7fd4e401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:19.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:19.862+0000 7fd4f4c00700 1 -- 172.21.15.44:0/3593212712 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd4e80aa610 con 0x7fd4f8106610 2024-05-27T09:33:19.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:19 smithi044 ceph-mon[31618]: pgmap v382: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:20.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.108+0000 7fd4fda00700 1 -- 172.21.15.44:0/3593212712 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd4f802e5e0 con 0x7fd4f8106610 2024-05-27T09:33:20.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.109+0000 7fd4f4c00700 1 -- 172.21.15.44:0/3593212712 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd4e80a9f10 con 0x7fd4f8106610 2024-05-27T09:33:20.111 INFO:teuthology.orchestra.run.smithi044.stdout:133143986257 2024-05-27T09:33:20.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.112+0000 7fd4fda00700 1 -- 172.21.15.44:0/3593212712 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd4dc0b2990 msgr2=0x7fd4dc0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:20.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.112+0000 7fd4fda00700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd4dc0b2990 0x7fd4dc0b4ea0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4000c00 tx=0x7fd4e401a040 comp rx=0 tx=0).stop 2024-05-27T09:33:20.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.112+0000 7fd4fda00700 1 -- 172.21.15.44:0/3593212712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f8106610 msgr2=0x7fd4f819ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:20.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.112+0000 7fd4fda00700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f8106610 0x7fd4f819ce60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd4e800e9a0 tx=0x7fd4e800ecb0 comp rx=0 tx=0).stop 2024-05-27T09:33:20.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.112+0000 7fd4fda00700 1 -- 172.21.15.44:0/3593212712 shutdown_connections 2024-05-27T09:33:20.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.112+0000 7fd4fda00700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fd4dc0b2990 0x7fd4dc0b4ea0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:20.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.112+0000 7fd4fda00700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f810b890 0x7fd4f81a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:20.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.112+0000 7fd4fda00700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f8106610 0x7fd4f819ce60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:20.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.112+0000 7fd4fda00700 1 --2- 172.21.15.44:0/3593212712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f810af40 0x7fd4f819d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:20.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.112+0000 7fd4fda00700 1 -- 172.21.15.44:0/3593212712 >> 172.21.15.44:0/3593212712 conn(0x7fd4f8102020 msgr2=0x7fd4f810a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:20.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.112+0000 7fd4fda00700 1 -- 172.21.15.44:0/3593212712 shutdown_connections 2024-05-27T09:33:20.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:20.113+0000 7fd4fda00700 1 -- 172.21.15.44:0/3593212712 wait complete. 2024-05-27T09:33:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:19 smithi050 ceph-mon[36371]: pgmap v382: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:20.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:19 smithi189 ceph-mon[36671]: pgmap v382: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:21.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:20 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3593212712' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T09:33:21.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:20 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3593212712' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T09:33:21.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:20 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3593212712' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T09:33:21.442 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986251 got 133143986257 for osd.4 2024-05-27T09:33:21.442 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:21.911 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266740 got 227633266745 for osd.8 2024-05-27T09:33:21.911 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:21 smithi044 ceph-mon[31618]: pgmap v383: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-05-27T09:33:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:21 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:33:22.206 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:21 smithi189 ceph-mon[36671]: pgmap v383: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-05-27T09:33:22.206 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:21 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:33:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:21 smithi050 ceph-mon[36371]: pgmap v383: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-05-27T09:33:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:21 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:33:22.971 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:23.305 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:23.306 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:24.095 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:23 smithi044 ceph-mon[31618]: pgmap v384: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:24.095 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:24.095 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:24.096 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:24.096 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:23 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:23 smithi050 ceph-mon[36371]: pgmap v384: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:23 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:23 smithi189 ceph-mon[36671]: pgmap v384: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:24.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:24.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:24.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:23 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:25.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.339+0000 7f44d0a00700 1 -- 172.21.15.44:0/2582025371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44cc108f10 msgr2=0x7f44cc109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:25.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.339+0000 7f44d0a00700 1 --2- 172.21.15.44:0/2582025371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44cc108f10 0x7f44cc109320 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009990 tx=0x7f44b8009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:25.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.339+0000 7f44d0a00700 1 -- 172.21.15.44:0/2582025371 shutdown_connections 2024-05-27T09:33:25.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.339+0000 7f44d0a00700 1 --2- 172.21.15.44:0/2582025371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc109860 0x7f44cc1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:25.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.339+0000 7f44d0a00700 1 --2- 172.21.15.44:0/2582025371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44cc1045e0 0x7f44cc1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:25.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.339+0000 7f44d0a00700 1 --2- 172.21.15.44:0/2582025371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44cc108f10 0x7f44cc109320 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:25.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.339+0000 7f44d0a00700 1 -- 172.21.15.44:0/2582025371 >> 172.21.15.44:0/2582025371 conn(0x7f44cc100030 msgr2=0x7f44cc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:25.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.339+0000 7f44d0a00700 1 -- 172.21.15.44:0/2582025371 shutdown_connections 2024-05-27T09:33:25.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.339+0000 7f44d0a00700 1 -- 172.21.15.44:0/2582025371 wait complete. 2024-05-27T09:33:25.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.340+0000 7f44d0a00700 1 Processor -- start 2024-05-27T09:33:25.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.341+0000 7f44d0a00700 1 -- start start 2024-05-27T09:33:25.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.341+0000 7f44d0a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44cc1045e0 0x7f44cc075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.341+0000 7f44d0a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc108f10 0x7f44cc075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.341+0000 7f44d0a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44cc109860 0x7f44cc076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.341+0000 7f44d0a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc1144c0 con 0x7f44cc109860 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.341+0000 7f44d0a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc114340 con 0x7f44cc1045e0 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.341+0000 7f44d0a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc114640 con 0x7f44cc108f10 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44ca800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44cc109860 0x7f44cc076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44c9e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44cc1045e0 0x7f44cc075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44ca800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44cc109860 0x7f44cc076530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42622/0 (socket says 172.21.15.44:42622) 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44c9e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44cc1045e0 0x7f44cc075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38792/0 (socket says 172.21.15.44:38792) 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44ca800700 1 -- 172.21.15.44:0/1513401230 learned_addr learned my addr 172.21.15.44:0/1513401230 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44c9400700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc108f10 0x7f44cc075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44c9e00700 1 -- 172.21.15.44:0/1513401230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc108f10 msgr2=0x7f44cc075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:25.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44c9e00700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc108f10 0x7f44cc075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:25.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44c9e00700 1 -- 172.21.15.44:0/1513401230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44cc109860 msgr2=0x7f44cc076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:25.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44c9e00700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44cc109860 0x7f44cc076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:25.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44c9e00700 1 -- 172.21.15.44:0/1513401230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44cc1a92b0 con 0x7f44cc1045e0 2024-05-27T09:33:25.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.342+0000 7f44c9e00700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44cc1045e0 0x7f44cc075ab0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00d750 tx=0x7f44bc00da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:25.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.343+0000 7f44c2a00700 1 -- 172.21.15.44:0/1513401230 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc014030 con 0x7f44cc1045e0 2024-05-27T09:33:25.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.343+0000 7f44c2a00700 1 -- 172.21.15.44:0/1513401230 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44bc0044d0 con 0x7f44cc1045e0 2024-05-27T09:33:25.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.343+0000 7f44c2a00700 1 -- 172.21.15.44:0/1513401230 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc005420 con 0x7f44cc1045e0 2024-05-27T09:33:25.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.343+0000 7f44d0a00700 1 -- 172.21.15.44:0/1513401230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44cc1a95a0 con 0x7f44cc1045e0 2024-05-27T09:33:25.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.343+0000 7f44d0a00700 1 -- 172.21.15.44:0/1513401230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44cc1a9a90 con 0x7f44cc1045e0 2024-05-27T09:33:25.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.345+0000 7f44d0a00700 1 -- 172.21.15.44:0/1513401230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44cc0512b0 con 0x7f44cc1045e0 2024-05-27T09:33:25.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.347+0000 7f44c2a00700 1 -- 172.21.15.44:0/1513401230 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f44bc004a50 con 0x7f44cc1045e0 2024-05-27T09:33:25.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.349+0000 7f44c2a00700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f44b00b2990 0x7f44b00b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:25.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.349+0000 7f44c2a00700 1 -- 172.21.15.44:0/1513401230 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f44bc0e1fe0 con 0x7f44cc1045e0 2024-05-27T09:33:25.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.349+0000 7f44c9400700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f44b00b2990 0x7f44b00b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:25.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.350+0000 7f44c9400700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f44b00b2990 0x7f44b00b4ea0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f44cc107950 tx=0x7f44b800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:25.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.354+0000 7f44c2a00700 1 -- 172.21.15.44:0/1513401230 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f44bc0a9b50 con 0x7f44cc1045e0 2024-05-27T09:33:25.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.610+0000 7f44d0a00700 1 -- 172.21.15.44:0/1513401230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f44cc02e5e0 con 0x7f44cc1045e0 2024-05-27T09:33:25.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.611+0000 7f44c2a00700 1 -- 172.21.15.44:0/1513401230 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f44bc0a9450 con 0x7f44cc1045e0 2024-05-27T09:33:25.611 INFO:teuthology.orchestra.run.smithi044.stdout:34359738471 2024-05-27T09:33:25.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.614+0000 7f44d0a00700 1 -- 172.21.15.44:0/1513401230 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f44b00b2990 msgr2=0x7f44b00b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:25.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.614+0000 7f44d0a00700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f44b00b2990 0x7f44b00b4ea0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f44cc107950 tx=0x7f44b800b540 comp rx=0 tx=0).stop 2024-05-27T09:33:25.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.614+0000 7f44d0a00700 1 -- 172.21.15.44:0/1513401230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44cc1045e0 msgr2=0x7f44cc075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:25.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.614+0000 7f44d0a00700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44cc1045e0 0x7f44cc075ab0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00d750 tx=0x7f44bc00da60 comp rx=0 tx=0).stop 2024-05-27T09:33:25.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.614+0000 7f44d0a00700 1 -- 172.21.15.44:0/1513401230 shutdown_connections 2024-05-27T09:33:25.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.614+0000 7f44d0a00700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f44b00b2990 0x7f44b00b4ea0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:25.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.614+0000 7f44d0a00700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44cc109860 0x7f44cc076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:25.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.614+0000 7f44d0a00700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44cc1045e0 0x7f44cc075ab0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:25.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.614+0000 7f44d0a00700 1 --2- 172.21.15.44:0/1513401230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc108f10 0x7f44cc075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:25.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.614+0000 7f44d0a00700 1 -- 172.21.15.44:0/1513401230 >> 172.21.15.44:0/1513401230 conn(0x7f44cc100030 msgr2=0x7f44cc10a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:25.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.615+0000 7f44d0a00700 1 -- 172.21.15.44:0/1513401230 shutdown_connections 2024-05-27T09:33:25.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:25.615+0000 7f44d0a00700 1 -- 172.21.15.44:0/1513401230 wait complete. 2024-05-27T09:33:26.147 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:25 smithi044 ceph-mon[31618]: pgmap v385: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:26.147 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:26.147 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:25 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:26.147 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:25 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1513401230' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T09:33:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:25 smithi050 ceph-mon[36371]: pgmap v385: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:25 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:25 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1513401230' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T09:33:26.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:25 smithi189 ceph-mon[36671]: pgmap v385: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:26.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:26.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:25 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:26.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:25 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1513401230' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T09:33:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.541+0000 7f8a64800700 1 -- 172.21.15.44:0/3912405371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a5c1045e0 msgr2=0x7f8a5c1049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.541+0000 7f8a64800700 1 --2- 172.21.15.44:0/3912405371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a5c1045e0 0x7f8a5c1049f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8a500099c0 tx=0x7f8a50009cd0 comp rx=0 tx=0).stop 2024-05-27T09:33:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.541+0000 7f8a64800700 1 -- 172.21.15.44:0/3912405371 shutdown_connections 2024-05-27T09:33:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.541+0000 7f8a64800700 1 --2- 172.21.15.44:0/3912405371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5c109860 0x7f8a5c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.541+0000 7f8a64800700 1 --2- 172.21.15.44:0/3912405371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a5c1045e0 0x7f8a5c1049f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.541+0000 7f8a64800700 1 --2- 172.21.15.44:0/3912405371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c108f10 0x7f8a5c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:26.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.541+0000 7f8a64800700 1 -- 172.21.15.44:0/3912405371 >> 172.21.15.44:0/3912405371 conn(0x7f8a5c100030 msgr2=0x7f8a5c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:26.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.541+0000 7f8a64800700 1 -- 172.21.15.44:0/3912405371 shutdown_connections 2024-05-27T09:33:26.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.541+0000 7f8a64800700 1 -- 172.21.15.44:0/3912405371 wait complete. 2024-05-27T09:33:26.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.543+0000 7f8a64800700 1 Processor -- start 2024-05-27T09:33:26.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.543+0000 7f8a64800700 1 -- start start 2024-05-27T09:33:26.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.543+0000 7f8a64800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c1045e0 0x7f8a5c19d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:26.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.543+0000 7f8a64800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a5c108f10 0x7f8a5c19d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:26.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.543+0000 7f8a64800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5c109860 0x7f8a5c1a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:26.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.543+0000 7f8a64800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1144e0 con 0x7f8a5c108f10 2024-05-27T09:33:26.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.543+0000 7f8a64800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c114360 con 0x7f8a5c1045e0 2024-05-27T09:33:26.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.543+0000 7f8a64800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c114660 con 0x7f8a5c109860 2024-05-27T09:33:26.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.544+0000 7f8a61e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c1045e0 0x7f8a5c19d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:26.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.544+0000 7f8a61400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a5c108f10 0x7f8a5c19d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:26.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.544+0000 7f8a61e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c1045e0 0x7f8a5c19d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38812/0 (socket says 172.21.15.44:38812) 2024-05-27T09:33:26.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.544+0000 7f8a61e00700 1 -- 172.21.15.44:0/1241146076 learned_addr learned my addr 172.21.15.44:0/1241146076 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:26.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.544+0000 7f8a62800700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5c109860 0x7f8a5c1a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:26.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.544+0000 7f8a61400700 1 -- 172.21.15.44:0/1241146076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5c109860 msgr2=0x7f8a5c1a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:26.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.544+0000 7f8a61400700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5c109860 0x7f8a5c1a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:26.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.544+0000 7f8a61400700 1 -- 172.21.15.44:0/1241146076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c1045e0 msgr2=0x7f8a5c19d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:26.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.544+0000 7f8a61400700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c1045e0 0x7f8a5c19d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:26.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.544+0000 7f8a61400700 1 -- 172.21.15.44:0/1241146076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5c1a2020 con 0x7f8a5c108f10 2024-05-27T09:33:26.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.544+0000 7f8a61e00700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c1045e0 0x7f8a5c19d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:33:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.545+0000 7f8a61400700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a5c108f10 0x7f8a5c19d6f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00e9c0 tx=0x7f8a4c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.545+0000 7f8a56a00700 1 -- 172.21.15.44:0/1241146076 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c00c9c0 con 0x7f8a5c108f10 2024-05-27T09:33:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.545+0000 7f8a56a00700 1 -- 172.21.15.44:0/1241146076 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a4c00cb60 con 0x7f8a5c108f10 2024-05-27T09:33:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.545+0000 7f8a64800700 1 -- 172.21.15.44:0/1241146076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5c1a2310 con 0x7f8a5c108f10 2024-05-27T09:33:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.545+0000 7f8a56a00700 1 -- 172.21.15.44:0/1241146076 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c018680 con 0x7f8a5c108f10 2024-05-27T09:33:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.545+0000 7f8a64800700 1 -- 172.21.15.44:0/1241146076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5c1a9c30 con 0x7f8a5c108f10 2024-05-27T09:33:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.546+0000 7f8a64800700 1 -- 172.21.15.44:0/1241146076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a5c0512b0 con 0x7f8a5c108f10 2024-05-27T09:33:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.550+0000 7f8a56a00700 1 -- 172.21.15.44:0/1241146076 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8a4c010430 con 0x7f8a5c108f10 2024-05-27T09:33:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.551+0000 7f8a56a00700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8a440b2910 0x7f8a440b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.551+0000 7f8a56a00700 1 -- 172.21.15.44:0/1241146076 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f8a4c0e2560 con 0x7f8a5c108f10 2024-05-27T09:33:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.551+0000 7f8a61e00700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8a440b2910 0x7f8a440b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.552+0000 7f8a61e00700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8a440b2910 0x7f8a440b4e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009990 tx=0x7f8a5001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:26.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.556+0000 7f8a56a00700 1 -- 172.21.15.44:0/1241146076 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8a4c0aa0d0 con 0x7f8a5c108f10 2024-05-27T09:33:26.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.839+0000 7f8a64800700 1 -- 172.21.15.44:0/1241146076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f8a5c1a9e00 con 0x7f8a5c108f10 2024-05-27T09:33:26.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.840+0000 7f8a56a00700 1 -- 172.21.15.44:0/1241146076 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f8a4c0b2150 con 0x7f8a5c108f10 2024-05-27T09:33:26.841 INFO:teuthology.orchestra.run.smithi044.stdout:279172874287 2024-05-27T09:33:26.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.843+0000 7f8a64800700 1 -- 172.21.15.44:0/1241146076 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8a440b2910 msgr2=0x7f8a440b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:26.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.843+0000 7f8a64800700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8a440b2910 0x7f8a440b4e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009990 tx=0x7f8a5001a040 comp rx=0 tx=0).stop 2024-05-27T09:33:26.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.843+0000 7f8a64800700 1 -- 172.21.15.44:0/1241146076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a5c108f10 msgr2=0x7f8a5c19d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:26.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.843+0000 7f8a64800700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a5c108f10 0x7f8a5c19d6f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00e9c0 tx=0x7f8a4c00ecd0 comp rx=0 tx=0).stop 2024-05-27T09:33:26.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.843+0000 7f8a64800700 1 -- 172.21.15.44:0/1241146076 shutdown_connections 2024-05-27T09:33:26.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.843+0000 7f8a64800700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8a440b2910 0x7f8a440b4e20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.843+0000 7f8a64800700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5c109860 0x7f8a5c1a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.843+0000 7f8a64800700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c1045e0 0x7f8a5c19d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.843+0000 7f8a64800700 1 --2- 172.21.15.44:0/1241146076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a5c108f10 0x7f8a5c19d6f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.843+0000 7f8a64800700 1 -- 172.21.15.44:0/1241146076 >> 172.21.15.44:0/1241146076 conn(0x7f8a5c100030 msgr2=0x7f8a5c1086d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.844+0000 7f8a64800700 1 -- 172.21.15.44:0/1241146076 shutdown_connections 2024-05-27T09:33:26.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:26.844+0000 7f8a64800700 1 -- 172.21.15.44:0/1241146076 wait complete. 2024-05-27T09:33:27.113 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:26 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1241146076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T09:33:27.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:26 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1241146076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T09:33:27.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:26 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1241146076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T09:33:27.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:27 smithi044 ceph-mon[31618]: pgmap v386: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:27.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:27 smithi044 ceph-mon[31618]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-27T09:33:27.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:27 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1706821760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T09:33:27.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:27 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/745902901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T09:33:28.252 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738466 got 34359738471 for osd.0 2024-05-27T09:33:28.252 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.513+0000 7f88f2200700 1 -- 172.21.15.44:0/3924414575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88ec107de0 msgr2=0x7f88ec1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.513+0000 7f88f2200700 1 --2- 172.21.15.44:0/3924414575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88ec107de0 0x7f88ec1081f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009990 tx=0x7f88d8009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.513+0000 7f88f2200700 1 -- 172.21.15.44:0/3924414575 shutdown_connections 2024-05-27T09:33:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.513+0000 7f88f2200700 1 --2- 172.21.15.44:0/3924414575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec108730 0x7f88ec110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.513+0000 7f88f2200700 1 --2- 172.21.15.44:0/3924414575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ec1034b0 0x7f88ec1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.513+0000 7f88f2200700 1 --2- 172.21.15.44:0/3924414575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88ec107de0 0x7f88ec1081f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.513+0000 7f88f2200700 1 -- 172.21.15.44:0/3924414575 >> 172.21.15.44:0/3924414575 conn(0x7f88ec0fedb0 msgr2=0x7f88ec1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.513+0000 7f88f2200700 1 -- 172.21.15.44:0/3924414575 shutdown_connections 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.513+0000 7f88f2200700 1 -- 172.21.15.44:0/3924414575 wait complete. 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.515+0000 7f88f2200700 1 Processor -- start 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.515+0000 7f88f2200700 1 -- start start 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.515+0000 7f88f2200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ec1034b0 0x7f88ec19ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.515+0000 7f88f2200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88ec107de0 0x7f88ec19b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.515+0000 7f88f2200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec108730 0x7f88ec19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.515+0000 7f88f2200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec113390 con 0x7f88ec1034b0 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.515+0000 7f88f2200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec113210 con 0x7f88ec107de0 2024-05-27T09:33:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.515+0000 7f88f2200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec113510 con 0x7f88ec108730 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.516+0000 7f88eaa00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88ec107de0 0x7f88ec19b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.516+0000 7f88eb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ec1034b0 0x7f88ec19ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.516+0000 7f88eaa00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88ec107de0 0x7f88ec19b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38844/0 (socket says 172.21.15.44:38844) 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.516+0000 7f88eaa00700 1 -- 172.21.15.44:0/745902901 learned_addr learned my addr 172.21.15.44:0/745902901 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.516+0000 7f88ebe00700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec108730 0x7f88ec19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.516+0000 7f88eaa00700 1 -- 172.21.15.44:0/745902901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec108730 msgr2=0x7f88ec19f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.516+0000 7f88eaa00700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec108730 0x7f88ec19f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.516+0000 7f88eaa00700 1 -- 172.21.15.44:0/745902901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ec1034b0 msgr2=0x7f88ec19ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.516+0000 7f88eaa00700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ec1034b0 0x7f88ec19ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.516+0000 7f88eaa00700 1 -- 172.21.15.44:0/745902901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ec19fd00 con 0x7f88ec107de0 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.516+0000 7f88eaa00700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88ec107de0 0x7f88ec19b3d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f88d8000c00 tx=0x7f88d8009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.517+0000 7f88e3e00700 1 -- 172.21.15.44:0/745902901 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d8026070 con 0x7f88ec107de0 2024-05-27T09:33:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.517+0000 7f88f2200700 1 -- 172.21.15.44:0/745902901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ec19ff90 con 0x7f88ec107de0 2024-05-27T09:33:28.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.517+0000 7f88f2200700 1 -- 172.21.15.44:0/745902901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ec0ff1c0 con 0x7f88ec107de0 2024-05-27T09:33:28.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.517+0000 7f88e3e00700 1 -- 172.21.15.44:0/745902901 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88d800b610 con 0x7f88ec107de0 2024-05-27T09:33:28.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.517+0000 7f88e3e00700 1 -- 172.21.15.44:0/745902901 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d8021de0 con 0x7f88ec107de0 2024-05-27T09:33:28.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.518+0000 7f88f2200700 1 -- 172.21.15.44:0/745902901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88ec104960 con 0x7f88ec107de0 2024-05-27T09:33:28.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.521+0000 7f88e3e00700 1 -- 172.21.15.44:0/745902901 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f88d8029070 con 0x7f88ec107de0 2024-05-27T09:33:28.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.522+0000 7f88e3e00700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f88d00b2900 0x7f88d00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:28.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.522+0000 7f88e3e00700 1 -- 172.21.15.44:0/745902901 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f88d80e32c0 con 0x7f88ec107de0 2024-05-27T09:33:28.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.523+0000 7f88eb400700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f88d00b2900 0x7f88d00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:28.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.523+0000 7f88eb400700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f88d00b2900 0x7f88d00b4e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0044f0 tx=0x7f88dc014040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:28.291 INFO:teuthology.orchestra.run.smithi044.stdout:201863462977 2024-05-27T09:33:28.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.524+0000 7f88e3e00700 1 -- 172.21.15.44:0/745902901 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f88d8030080 con 0x7f88ec107de0 2024-05-27T09:33:28.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.785+0000 7f88f2200700 1 -- 172.21.15.44:0/745902901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f88ec19c190 con 0x7f88ec107de0 2024-05-27T09:33:28.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.786+0000 7f88e3e00700 1 -- 172.21.15.44:0/745902901 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f88d8013070 con 0x7f88ec107de0 2024-05-27T09:33:28.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.789+0000 7f88f2200700 1 -- 172.21.15.44:0/745902901 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f88d00b2900 msgr2=0x7f88d00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:28.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.789+0000 7f88f2200700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f88d00b2900 0x7f88d00b4e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0044f0 tx=0x7f88dc014040 comp rx=0 tx=0).stop 2024-05-27T09:33:28.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.789+0000 7f88f2200700 1 -- 172.21.15.44:0/745902901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88ec107de0 msgr2=0x7f88ec19b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:28.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.789+0000 7f88f2200700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88ec107de0 0x7f88ec19b3d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f88d8000c00 tx=0x7f88d8009580 comp rx=0 tx=0).stop 2024-05-27T09:33:28.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.790+0000 7f88f2200700 1 -- 172.21.15.44:0/745902901 shutdown_connections 2024-05-27T09:33:28.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.790+0000 7f88f2200700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f88d00b2900 0x7f88d00b4e10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:28.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.790+0000 7f88f2200700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec108730 0x7f88ec19f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:28.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.790+0000 7f88f2200700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88ec1034b0 0x7f88ec19ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:28.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.790+0000 7f88f2200700 1 --2- 172.21.15.44:0/745902901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88ec107de0 0x7f88ec19b3d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:28.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.790+0000 7f88f2200700 1 -- 172.21.15.44:0/745902901 >> 172.21.15.44:0/745902901 conn(0x7f88ec0fedb0 msgr2=0x7f88ec06b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:28.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.790+0000 7f88f2200700 1 -- 172.21.15.44:0/745902901 shutdown_connections 2024-05-27T09:33:28.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.790+0000 7f88f2200700 1 -- 172.21.15.44:0/745902901 wait complete. 2024-05-27T09:33:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:27 smithi050 ceph-mon[36371]: pgmap v386: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:27 smithi050 ceph-mon[36371]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-27T09:33:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:27 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1706821760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T09:33:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:27 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/745902901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T09:33:28.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:27 smithi189 ceph-mon[36671]: pgmap v386: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:28.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:27 smithi189 ceph-mon[36671]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-27T09:33:28.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:27 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1706821760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T09:33:28.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:27 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/745902901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T09:33:29.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.416+0000 7f5d78600700 1 -- 172.21.15.44:0/2535131004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d70105e60 msgr2=0x7f5d7010e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:29.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.416+0000 7f5d78600700 1 --2- 172.21.15.44:0/2535131004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d70105e60 0x7f5d7010e3b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c009990 tx=0x7f5d5c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:29.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.416+0000 7f5d78600700 1 -- 172.21.15.44:0/2535131004 shutdown_connections 2024-05-27T09:33:29.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.416+0000 7f5d78600700 1 --2- 172.21.15.44:0/2535131004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d7010e8f0 0x7f5d70110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:29.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.416+0000 7f5d78600700 1 --2- 172.21.15.44:0/2535131004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d70105510 0x7f5d70105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:29.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.416+0000 7f5d78600700 1 --2- 172.21.15.44:0/2535131004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d70105e60 0x7f5d7010e3b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:29.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.416+0000 7f5d78600700 1 -- 172.21.15.44:0/2535131004 >> 172.21.15.44:0/2535131004 conn(0x7f5d700fedb0 msgr2=0x7f5d701011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:29.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.416+0000 7f5d78600700 1 -- 172.21.15.44:0/2535131004 shutdown_connections 2024-05-27T09:33:29.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.416+0000 7f5d78600700 1 -- 172.21.15.44:0/2535131004 wait complete. 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.418+0000 7f5d78600700 1 Processor -- start 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.418+0000 7f5d78600700 1 -- start start 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.419+0000 7f5d78600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d70105510 0x7f5d7019ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.419+0000 7f5d78600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d70105e60 0x7f5d7019b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.419+0000 7f5d78600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d7010e8f0 0x7f5d7019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.419+0000 7f5d78600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70114510 con 0x7f5d70105e60 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.419+0000 7f5d78600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70114390 con 0x7f5d7010e8f0 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.419+0000 7f5d78600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70114690 con 0x7f5d70105510 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.419+0000 7f5d75200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d70105e60 0x7f5d7019b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.419+0000 7f5d75200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d70105e60 0x7f5d7019b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42662/0 (socket says 172.21.15.44:42662) 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.419+0000 7f5d75200700 1 -- 172.21.15.44:0/1706821760 learned_addr learned my addr 172.21.15.44:0/1706821760 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.419+0000 7f5d75c00700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d70105510 0x7f5d7019ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.420+0000 7f5d75200700 1 -- 172.21.15.44:0/1706821760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d70105510 msgr2=0x7f5d7019ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.420+0000 7f5d75200700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d70105510 0x7f5d7019ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.420+0000 7f5d75200700 1 -- 172.21.15.44:0/1706821760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d7010e8f0 msgr2=0x7f5d7019f690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.420+0000 7f5d75200700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d7010e8f0 0x7f5d7019f690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.420+0000 7f5d75200700 1 -- 172.21.15.44:0/1706821760 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7019fd30 con 0x7f5d70105e60 2024-05-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.420+0000 7f5d75200700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d70105e60 0x7f5d7019b400 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00b2e0 tx=0x7f5d5c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi044.stdout:107374182487 2024-05-27T09:33:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.420+0000 7f5d6e000700 1 -- 172.21.15.44:0/1706821760 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c026070 con 0x7f5d70105e60 2024-05-27T09:33:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.420+0000 7f5d6e000700 1 -- 172.21.15.44:0/1706821760 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d5c021410 con 0x7f5d70105e60 2024-05-27T09:33:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.420+0000 7f5d78600700 1 -- 172.21.15.44:0/1706821760 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7019ffc0 con 0x7f5d70105e60 2024-05-27T09:33:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.421+0000 7f5d78600700 1 -- 172.21.15.44:0/1706821760 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d701a7940 con 0x7f5d70105e60 2024-05-27T09:33:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.421+0000 7f5d6e000700 1 -- 172.21.15.44:0/1706821760 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c017650 con 0x7f5d70105e60 2024-05-27T09:33:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.422+0000 7f5d78600700 1 -- 172.21.15.44:0/1706821760 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d50005360 con 0x7f5d70105e60 2024-05-27T09:33:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.426+0000 7f5d6e000700 1 -- 172.21.15.44:0/1706821760 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f5d5c029070 con 0x7f5d70105e60 2024-05-27T09:33:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.428+0000 7f5d6e000700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d580b2900 0x7f5d580b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.428+0000 7f5d6e000700 1 -- 172.21.15.44:0/1706821760 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f5d5c0e32a0 con 0x7f5d70105e60 2024-05-27T09:33:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.429+0000 7f5d75c00700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d580b2900 0x7f5d580b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.430+0000 7f5d75c00700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d580b2900 0x7f5d580b4e10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d64006fd0 tx=0x7f5d64009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.431+0000 7f5d6e000700 1 -- 172.21.15.44:0/1706821760 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5d5c013070 con 0x7f5d70105e60 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.680+0000 7f5d78600700 1 -- 172.21.15.44:0/1706821760 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f5d50005920 con 0x7f5d70105e60 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.681+0000 7f5d6e000700 1 -- 172.21.15.44:0/1706821760 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5d5c0ab110 con 0x7f5d70105e60 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.683+0000 7f5d78600700 1 -- 172.21.15.44:0/1706821760 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d580b2900 msgr2=0x7f5d580b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.683+0000 7f5d78600700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d580b2900 0x7f5d580b4e10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d64006fd0 tx=0x7f5d64009190 comp rx=0 tx=0).stop 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.683+0000 7f5d78600700 1 -- 172.21.15.44:0/1706821760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d70105e60 msgr2=0x7f5d7019b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.684+0000 7f5d78600700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d70105e60 0x7f5d7019b400 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00b2e0 tx=0x7f5d5c0057c0 comp rx=0 tx=0).stop 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.684+0000 7f5d78600700 1 -- 172.21.15.44:0/1706821760 shutdown_connections 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.684+0000 7f5d78600700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5d580b2900 0x7f5d580b4e10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.684+0000 7f5d78600700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d7010e8f0 0x7f5d7019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.684+0000 7f5d78600700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d70105510 0x7f5d7019ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:29.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.684+0000 7f5d78600700 1 --2- 172.21.15.44:0/1706821760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d70105e60 0x7f5d7019b400 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:29.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.684+0000 7f5d78600700 1 -- 172.21.15.44:0/1706821760 >> 172.21.15.44:0/1706821760 conn(0x7f5d700fedb0 msgr2=0x7f5d70100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:29.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.684+0000 7f5d78600700 1 -- 172.21.15.44:0/1706821760 shutdown_connections 2024-05-27T09:33:29.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:27.684+0000 7f5d78600700 1 -- 172.21.15.44:0/1706821760 wait complete. 2024-05-27T09:33:29.761 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874281 got 279172874287 for osd.10 2024-05-27T09:33:29.762 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:30.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:29 smithi044 ceph-mon[31618]: pgmap v387: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-27T09:33:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:29 smithi050 ceph-mon[36371]: pgmap v387: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-27T09:33:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:29 smithi189 ceph-mon[36671]: pgmap v387: 97 pgs: 97 active+clean; 640 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-27T09:33:30.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.655+0000 7fc008400700 1 -- 172.21.15.44:0/1453415649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0001054b0 msgr2=0x7fc0001058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:30.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.655+0000 7fc008400700 1 --2- 172.21.15.44:0/1453415649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0001054b0 0x7fc0001058c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009990 tx=0x7fbff0009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:30.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.655+0000 7fc008400700 1 -- 172.21.15.44:0/1453415649 shutdown_connections 2024-05-27T09:33:30.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.655+0000 7fc008400700 1 --2- 172.21.15.44:0/1453415649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc00010e8e0 0x7fc000110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:30.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.655+0000 7fc008400700 1 --2- 172.21.15.44:0/1453415649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0001054b0 0x7fc0001058c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:30.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.655+0000 7fc008400700 1 --2- 172.21.15.44:0/1453415649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc000105e00 0x7fc00010e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:30.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.655+0000 7fc008400700 1 -- 172.21.15.44:0/1453415649 >> 172.21.15.44:0/1453415649 conn(0x7fc0000fedb0 msgr2=0x7fc0001011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:30.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.656+0000 7fc008400700 1 -- 172.21.15.44:0/1453415649 shutdown_connections 2024-05-27T09:33:30.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.656+0000 7fc008400700 1 -- 172.21.15.44:0/1453415649 wait complete. 2024-05-27T09:33:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.657+0000 7fc008400700 1 Processor -- start 2024-05-27T09:33:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.657+0000 7fc008400700 1 -- start start 2024-05-27T09:33:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.657+0000 7fc008400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0001054b0 0x7fc00019ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.657+0000 7fc008400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc000105e00 0x7fc00019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.657+0000 7fc008400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc00010e8e0 0x7fc00019f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc008400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc000114430 con 0x7fc00010e8e0 2024-05-27T09:33:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc008400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc0001142b0 con 0x7fc0001054b0 2024-05-27T09:33:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc008400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc0001145b0 con 0x7fc000105e00 2024-05-27T09:33:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc005a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0001054b0 0x7fc00019ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc005a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0001054b0 0x7fc00019ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38856/0 (socket says 172.21.15.44:38856) 2024-05-27T09:33:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc005a00700 1 -- 172.21.15.44:0/4129781007 learned_addr learned my addr 172.21.15.44:0/4129781007 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc006400700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc00010e8e0 0x7fc00019f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc005000700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc000105e00 0x7fc00019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc006400700 1 -- 172.21.15.44:0/4129781007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc000105e00 msgr2=0x7fc00019b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc006400700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc000105e00 0x7fc00019b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc006400700 1 -- 172.21.15.44:0/4129781007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0001054b0 msgr2=0x7fc00019ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc006400700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0001054b0 0x7fc00019ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc006400700 1 -- 172.21.15.44:0/4129781007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00019fc70 con 0x7fc00010e8e0 2024-05-27T09:33:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.658+0000 7fc005a00700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0001054b0 0x7fc00019ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:33:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.659+0000 7fc006400700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc00010e8e0 0x7fc00019f5d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbff8007f10 tx=0x7fbff800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.659+0000 7fbffe000700 1 -- 172.21.15.44:0/4129781007 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8004db0 con 0x7fc00010e8e0 2024-05-27T09:33:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.659+0000 7fbffe000700 1 -- 172.21.15.44:0/4129781007 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbff8009e20 con 0x7fc00010e8e0 2024-05-27T09:33:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.659+0000 7fc008400700 1 -- 172.21.15.44:0/4129781007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00019ff60 con 0x7fc00010e8e0 2024-05-27T09:33:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.659+0000 7fbffe000700 1 -- 172.21.15.44:0/4129781007 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8015400 con 0x7fc00010e8e0 2024-05-27T09:33:30.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.659+0000 7fc008400700 1 -- 172.21.15.44:0/4129781007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0001030d0 con 0x7fc00010e8e0 2024-05-27T09:33:30.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.661+0000 7fc008400700 1 -- 172.21.15.44:0/4129781007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc00006be80 con 0x7fc00010e8e0 2024-05-27T09:33:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.663+0000 7fbffe000700 1 -- 172.21.15.44:0/4129781007 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fbff8005260 con 0x7fc00010e8e0 2024-05-27T09:33:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.665+0000 7fbffe000700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbfe40b2900 0x7fbfe40b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.665+0000 7fbffe000700 1 -- 172.21.15.44:0/4129781007 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fbff80e1fc0 con 0x7fc00010e8e0 2024-05-27T09:33:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.665+0000 7fc005a00700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbfe40b2900 0x7fbfe40b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.666+0000 7fc005a00700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbfe40b2900 0x7fbfe40b4e10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbff0006010 tx=0x7fbff0003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:30.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.668+0000 7fbffe000700 1 -- 172.21.15.44:0/4129781007 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbff8010070 con 0x7fc00010e8e0 2024-05-27T09:33:30.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.908+0000 7fc008400700 1 -- 172.21.15.44:0/4129781007 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fc00002e5e0 con 0x7fc00010e8e0 2024-05-27T09:33:30.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.909+0000 7fbffe000700 1 -- 172.21.15.44:0/4129781007 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fbff80a9c90 con 0x7fc00010e8e0 2024-05-27T09:33:30.910 INFO:teuthology.orchestra.run.smithi044.stdout:304942678059 2024-05-27T09:33:30.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.912+0000 7fc008400700 1 -- 172.21.15.44:0/4129781007 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbfe40b2900 msgr2=0x7fbfe40b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:30.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.912+0000 7fc008400700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbfe40b2900 0x7fbfe40b4e10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbff0006010 tx=0x7fbff0003680 comp rx=0 tx=0).stop 2024-05-27T09:33:30.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.912+0000 7fc008400700 1 -- 172.21.15.44:0/4129781007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc00010e8e0 msgr2=0x7fc00019f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:30.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.912+0000 7fc008400700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc00010e8e0 0x7fc00019f5d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbff8007f10 tx=0x7fbff800d3b0 comp rx=0 tx=0).stop 2024-05-27T09:33:30.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.912+0000 7fc008400700 1 -- 172.21.15.44:0/4129781007 shutdown_connections 2024-05-27T09:33:30.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.912+0000 7fc008400700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbfe40b2900 0x7fbfe40b4e10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:30.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.912+0000 7fc008400700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc00010e8e0 0x7fc00019f5d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:30.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.912+0000 7fc008400700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0001054b0 0x7fc00019ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:30.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.912+0000 7fc008400700 1 --2- 172.21.15.44:0/4129781007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc000105e00 0x7fc00019b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:30.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.912+0000 7fc008400700 1 -- 172.21.15.44:0/4129781007 >> 172.21.15.44:0/4129781007 conn(0x7fc0000fedb0 msgr2=0x7fc0000ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:30.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.913+0000 7fc008400700 1 -- 172.21.15.44:0/4129781007 shutdown_connections 2024-05-27T09:33:30.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:30.913+0000 7fc008400700 1 -- 172.21.15.44:0/4129781007 wait complete. 2024-05-27T09:33:31.003 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182481 got 107374182487 for osd.3 2024-05-27T09:33:31.003 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:31.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:30 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4129781007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T09:33:31.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:30 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/4129781007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T09:33:31.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:30 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/4129781007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T09:33:32.092 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:31 smithi044 ceph-mon[31618]: pgmap v388: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T09:33:32.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:31 smithi050 ceph-mon[36371]: pgmap v388: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T09:33:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:31 smithi189 ceph-mon[36671]: pgmap v388: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T09:33:32.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.343+0000 7f57bbe00700 1 -- 172.21.15.44:0/1370329004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57b41045e0 msgr2=0x7f57b41049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:32.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.343+0000 7f57bbe00700 1 --2- 172.21.15.44:0/1370329004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57b41045e0 0x7f57b41049f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f57a400b0d0 tx=0x7f57a400b3e0 comp rx=0 tx=0).stop 2024-05-27T09:33:32.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.343+0000 7f57bbe00700 1 -- 172.21.15.44:0/1370329004 shutdown_connections 2024-05-27T09:33:32.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.343+0000 7f57bbe00700 1 --2- 172.21.15.44:0/1370329004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57b4109860 0x7f57b41111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:32.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.343+0000 7f57bbe00700 1 --2- 172.21.15.44:0/1370329004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57b41045e0 0x7f57b41049f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:32.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.343+0000 7f57bbe00700 1 --2- 172.21.15.44:0/1370329004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b4108f10 0x7f57b4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:32.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.343+0000 7f57bbe00700 1 -- 172.21.15.44:0/1370329004 >> 172.21.15.44:0/1370329004 conn(0x7f57b4100030 msgr2=0x7f57b4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:32.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.343+0000 7f57bbe00700 1 -- 172.21.15.44:0/1370329004 shutdown_connections 2024-05-27T09:33:32.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.343+0000 7f57bbe00700 1 -- 172.21.15.44:0/1370329004 wait complete. 2024-05-27T09:33:32.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.344+0000 7f57bbe00700 1 Processor -- start 2024-05-27T09:33:32.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.344+0000 7f57bbe00700 1 -- start start 2024-05-27T09:33:32.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.345+0000 7f57bbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57b41045e0 0x7f57b4075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:32.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.345+0000 7f57bbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57b4108f10 0x7f57b4075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:32.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.345+0000 7f57bbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b4109860 0x7f57b4076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:32.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.345+0000 7f57bbe00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f57b41144e0 con 0x7f57b41045e0 2024-05-27T09:33:32.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.345+0000 7f57bbe00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f57b4114360 con 0x7f57b4109860 2024-05-27T09:33:32.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.345+0000 7f57bbe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f57b4114660 con 0x7f57b4108f10 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.345+0000 7f57b8a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57b4108f10 0x7f57b4075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.345+0000 7f57b9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57b41045e0 0x7f57b4075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.345+0000 7f57b8a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57b4108f10 0x7f57b4075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34742/0 (socket says 172.21.15.44:34742) 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.345+0000 7f57b8a00700 1 -- 172.21.15.44:0/100903916 learned_addr learned my addr 172.21.15.44:0/100903916 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.345+0000 7f57b9e00700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b4109860 0x7f57b4076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.346+0000 7f57b8a00700 1 -- 172.21.15.44:0/100903916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b4109860 msgr2=0x7f57b4076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.346+0000 7f57b8a00700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b4109860 0x7f57b4076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.346+0000 7f57b8a00700 1 -- 172.21.15.44:0/100903916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57b41045e0 msgr2=0x7f57b4075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.346+0000 7f57b8a00700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57b41045e0 0x7f57b4075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.346+0000 7f57b8a00700 1 -- 172.21.15.44:0/100903916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b41a91f0 con 0x7f57b4108f10 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.346+0000 7f57b9400700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57b41045e0 0x7f57b4075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:33:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.346+0000 7f57b8a00700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57b4108f10 0x7f57b4075ff0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f57a000e970 tx=0x7f57a000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.346+0000 7f57b2000700 1 -- 172.21.15.44:0/100903916 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0004a30 con 0x7f57b4108f10 2024-05-27T09:33:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.346+0000 7f57bbe00700 1 -- 172.21.15.44:0/100903916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b41a94e0 con 0x7f57b4108f10 2024-05-27T09:33:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.347+0000 7f57b2000700 1 -- 172.21.15.44:0/100903916 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a000cc90 con 0x7f57b4108f10 2024-05-27T09:33:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.347+0000 7f57b2000700 1 -- 172.21.15.44:0/100903916 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0005610 con 0x7f57b4108f10 2024-05-27T09:33:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.347+0000 7f57bbe00700 1 -- 172.21.15.44:0/100903916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b41a9a70 con 0x7f57b4108f10 2024-05-27T09:33:32.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.348+0000 7f57bbe00700 1 -- 172.21.15.44:0/100903916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b40512b0 con 0x7f57b4108f10 2024-05-27T09:33:32.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.351+0000 7f57b2000700 1 -- 172.21.15.44:0/100903916 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f57a000c7e0 con 0x7f57b4108f10 2024-05-27T09:33:32.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.352+0000 7f57b2000700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f57980b2880 0x7f57980b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:32.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.353+0000 7f57b2000700 1 -- 172.21.15.44:0/100903916 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f57a00e23c0 con 0x7f57b4108f10 2024-05-27T09:33:32.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.353+0000 7f57b9400700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f57980b2880 0x7f57980b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:32.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.354+0000 7f57b9400700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f57980b2880 0x7f57980b4d90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f57a400bfd0 tx=0x7f57a400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:32.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.357+0000 7f57b2000700 1 -- 172.21.15.44:0/100903916 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f57a00aa090 con 0x7f57b4108f10 2024-05-27T09:33:32.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.608+0000 7f57bbe00700 1 -- 172.21.15.44:0/100903916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f57b4076fe0 con 0x7f57b4108f10 2024-05-27T09:33:32.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.608+0000 7f57b2000700 1 -- 172.21.15.44:0/100903916 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f57a00a9990 con 0x7f57b4108f10 2024-05-27T09:33:32.609 INFO:teuthology.orchestra.run.smithi044.stdout:154618822734 2024-05-27T09:33:32.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 -- 172.21.15.44:0/100903916 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f57980b2880 msgr2=0x7f57980b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:32.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f57980b2880 0x7f57980b4d90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f57a400bfd0 tx=0x7f57a400b040 comp rx=0 tx=0).stop 2024-05-27T09:33:32.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 -- 172.21.15.44:0/100903916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57b4108f10 msgr2=0x7f57b4075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:32.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57b4108f10 0x7f57b4075ff0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f57a000e970 tx=0x7f57a000ec80 comp rx=0 tx=0).stop 2024-05-27T09:33:32.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 -- 172.21.15.44:0/100903916 shutdown_connections 2024-05-27T09:33:32.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f57980b2880 0x7f57980b4d90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:32.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b4109860 0x7f57b4076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:32.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57b41045e0 0x7f57b4075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:32.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 --2- 172.21.15.44:0/100903916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57b4108f10 0x7f57b4075ff0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:32.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 -- 172.21.15.44:0/100903916 >> 172.21.15.44:0/100903916 conn(0x7f57b4100030 msgr2=0x7f57b410a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:32.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 -- 172.21.15.44:0/100903916 shutdown_connections 2024-05-27T09:33:32.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:32.612+0000 7f57bbe00700 1 -- 172.21.15.44:0/100903916 wait complete. 2024-05-27T09:33:32.624 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462972 got 201863462977 for osd.7 2024-05-27T09:33:32.624 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:33.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:32 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/100903916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T09:33:33.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:32 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/100903916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T09:33:33.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:32 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/100903916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T09:33:34.011 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:33 smithi044 ceph-mon[31618]: pgmap v389: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T09:33:34.195 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:33 smithi050 ceph-mon[36371]: pgmap v389: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T09:33:34.282 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822730 got 154618822734 for osd.5 2024-05-27T09:33:34.282 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:34.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:33 smithi189 ceph-mon[36671]: pgmap v389: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T09:33:35.467 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678054 got 304942678059 for osd.11 2024-05-27T09:33:35.467 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:35.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.482+0000 7fbc61400700 1 -- 172.21.15.44:0/3831287636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c103a00 msgr2=0x7fbc5c110720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:35.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.482+0000 7fbc61400700 1 --2- 172.21.15.44:0/3831287636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c103a00 0x7fbc5c110720 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbc48009990 tx=0x7fbc48009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.482+0000 7fbc61400700 1 -- 172.21.15.44:0/3831287636 shutdown_connections 2024-05-27T09:33:35.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.482+0000 7fbc61400700 1 --2- 172.21.15.44:0/3831287636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc5c110c60 0x7fbc5c113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.482+0000 7fbc61400700 1 --2- 172.21.15.44:0/3831287636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c1030b0 0x7fbc5c1034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.482+0000 7fbc61400700 1 --2- 172.21.15.44:0/3831287636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c103a00 0x7fbc5c110720 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.482+0000 7fbc61400700 1 -- 172.21.15.44:0/3831287636 >> 172.21.15.44:0/3831287636 conn(0x7fbc5c0fedb0 msgr2=0x7fbc5c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:35.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.482+0000 7fbc61400700 1 -- 172.21.15.44:0/3831287636 shutdown_connections 2024-05-27T09:33:35.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.483+0000 7fbc61400700 1 -- 172.21.15.44:0/3831287636 wait complete. 2024-05-27T09:33:35.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.484+0000 7fbc61400700 1 Processor -- start 2024-05-27T09:33:35.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.484+0000 7fbc61400700 1 -- start start 2024-05-27T09:33:35.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.485+0000 7fbc61400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c1030b0 0x7fbc5c19d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:35.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.485+0000 7fbc61400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc5c103a00 0x7fbc5c19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:35.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.485+0000 7fbc61400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c110c60 0x7fbc5c1a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:35.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.485+0000 7fbc61400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c1166e0 con 0x7fbc5c110c60 2024-05-27T09:33:35.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.485+0000 7fbc61400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c116560 con 0x7fbc5c103a00 2024-05-27T09:33:35.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.485+0000 7fbc61400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c116860 con 0x7fbc5c1030b0 2024-05-27T09:33:35.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.485+0000 7fbc59e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc5c103a00 0x7fbc5c19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:35.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.485+0000 7fbc5b200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c110c60 0x7fbc5c1a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:35.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.485+0000 7fbc59e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc5c103a00 0x7fbc5c19d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:55952/0 (socket says 172.21.15.44:55952) 2024-05-27T09:33:35.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.485+0000 7fbc59e00700 1 -- 172.21.15.44:0/3434268511 learned_addr learned my addr 172.21.15.44:0/3434268511 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:35.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.485+0000 7fbc5a800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c1030b0 0x7fbc5c19d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:35.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.486+0000 7fbc59e00700 1 -- 172.21.15.44:0/3434268511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c1030b0 msgr2=0x7fbc5c19d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:35.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.486+0000 7fbc59e00700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c1030b0 0x7fbc5c19d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.486+0000 7fbc59e00700 1 -- 172.21.15.44:0/3434268511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c110c60 msgr2=0x7fbc5c1a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.486+0000 7fbc59e00700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c110c60 0x7fbc5c1a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.486+0000 7fbc59e00700 1 -- 172.21.15.44:0/3434268511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5c1a1fa0 con 0x7fbc5c103a00 2024-05-27T09:33:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.486+0000 7fbc5a800700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c1030b0 0x7fbc5c19d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:33:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.486+0000 7fbc59e00700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc5c103a00 0x7fbc5c19d670 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbc480098e0 tx=0x7fbc480057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.486+0000 7fbc53400700 1 -- 172.21.15.44:0/3434268511 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48026070 con 0x7fbc5c103a00 2024-05-27T09:33:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.487+0000 7fbc53400700 1 -- 172.21.15.44:0/3434268511 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc48021410 con 0x7fbc5c103a00 2024-05-27T09:33:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.487+0000 7fbc53400700 1 -- 172.21.15.44:0/3434268511 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48017650 con 0x7fbc5c103a00 2024-05-27T09:33:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.487+0000 7fbc61400700 1 -- 172.21.15.44:0/3434268511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc5c1a2230 con 0x7fbc5c103a00 2024-05-27T09:33:35.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.487+0000 7fbc61400700 1 -- 172.21.15.44:0/3434268511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc5c1a9bb0 con 0x7fbc5c103a00 2024-05-27T09:33:35.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.488+0000 7fbc61400700 1 -- 172.21.15.44:0/3434268511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc5c0512b0 con 0x7fbc5c103a00 2024-05-27T09:33:35.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.490+0000 7fbc53400700 1 -- 172.21.15.44:0/3434268511 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fbc48009d10 con 0x7fbc5c103a00 2024-05-27T09:33:35.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.492+0000 7fbc53400700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbc400b2940 0x7fbc400b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:35.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.492+0000 7fbc53400700 1 -- 172.21.15.44:0/3434268511 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fbc480e2e70 con 0x7fbc5c103a00 2024-05-27T09:33:35.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.492+0000 7fbc5a800700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbc400b2940 0x7fbc400b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:35.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.493+0000 7fbc5a800700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbc400b2940 0x7fbc400b4e50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c011490 tx=0x7fbc4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:35.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.494+0000 7fbc53400700 1 -- 172.21.15.44:0/3434268511 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbc48013070 con 0x7fbc5c103a00 2024-05-27T09:33:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.343+0000 7f569e400700 1 -- 172.21.15.44:0/4273037608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5698108f10 msgr2=0x7f5698109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.343+0000 7f569e400700 1 --2- 172.21.15.44:0/4273037608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5698108f10 0x7f5698109320 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5680009990 tx=0x7f5680009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.343+0000 7f569e400700 1 -- 172.21.15.44:0/4273037608 shutdown_connections 2024-05-27T09:33:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.343+0000 7f569e400700 1 --2- 172.21.15.44:0/4273037608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5698109860 0x7f56981111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.343+0000 7f569e400700 1 --2- 172.21.15.44:0/4273037608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56981045e0 0x7f56981049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.343+0000 7f569e400700 1 --2- 172.21.15.44:0/4273037608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5698108f10 0x7f5698109320 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.343+0000 7f569e400700 1 -- 172.21.15.44:0/4273037608 >> 172.21.15.44:0/4273037608 conn(0x7f5698100030 msgr2=0x7f5698102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.343+0000 7f569e400700 1 -- 172.21.15.44:0/4273037608 shutdown_connections 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.343+0000 7f569e400700 1 -- 172.21.15.44:0/4273037608 wait complete. 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.344+0000 7f569e400700 1 Processor -- start 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.344+0000 7f569e400700 1 -- start start 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.345+0000 7f569e400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56981045e0 0x7f569819aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.345+0000 7f569e400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5698108f10 0x7f569819b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.345+0000 7f569e400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5698109860 0x7f569819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.345+0000 7f569e400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f56981144e0 con 0x7f5698109860 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.345+0000 7f569e400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5698114360 con 0x7f5698108f10 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.345+0000 7f569e400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5698114660 con 0x7f56981045e0 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.345+0000 7f568ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5698108f10 0x7f569819b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.345+0000 7f568ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5698108f10 0x7f569819b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:55930/0 (socket says 172.21.15.44:55930) 2024-05-27T09:33:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.345+0000 7f568ea00700 1 -- 172.21.15.44:0/2647150585 learned_addr learned my addr 172.21.15.44:0/2647150585 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.345+0000 7f5697400700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56981045e0 0x7f569819aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.346+0000 7f5697e00700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5698109860 0x7f569819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.346+0000 7f568ea00700 1 -- 172.21.15.44:0/2647150585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56981045e0 msgr2=0x7f569819aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.346+0000 7f568ea00700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56981045e0 0x7f569819aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.346+0000 7f568ea00700 1 -- 172.21.15.44:0/2647150585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5698109860 msgr2=0x7f569819f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.346+0000 7f568ea00700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5698109860 0x7f569819f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.346+0000 7f568ea00700 1 -- 172.21.15.44:0/2647150585 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569819fec0 con 0x7f5698108f10 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.346+0000 7f568ea00700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5698108f10 0x7f569819b590 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5680000c00 tx=0x7f56800057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.347+0000 7f5694c00700 1 -- 172.21.15.44:0/2647150585 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680027070 con 0x7f5698108f10 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.347+0000 7f5694c00700 1 -- 172.21.15.44:0/2647150585 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f568000b610 con 0x7f5698108f10 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.347+0000 7f5694c00700 1 -- 172.21.15.44:0/2647150585 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56800173f0 con 0x7f5698108f10 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.347+0000 7f569e400700 1 -- 172.21.15.44:0/2647150585 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56981a0150 con 0x7f5698108f10 2024-05-27T09:33:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.347+0000 7f569e400700 1 -- 172.21.15.44:0/2647150585 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56981a7ad0 con 0x7f5698108f10 2024-05-27T09:33:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.348+0000 7f569e400700 1 -- 172.21.15.44:0/2647150585 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56980512b0 con 0x7f5698108f10 2024-05-27T09:33:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.353+0000 7f5694c00700 1 -- 172.21.15.44:0/2647150585 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f5680009d10 con 0x7f5698108f10 2024-05-27T09:33:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.354+0000 7f5694c00700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f567c0b2900 0x7f567c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.354+0000 7f5694c00700 1 -- 172.21.15.44:0/2647150585 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f56800e2e60 con 0x7f5698108f10 2024-05-27T09:33:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.354+0000 7f5697400700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f567c0b2900 0x7f567c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.355+0000 7f5697400700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f567c0b2900 0x7f567c0b4e10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5688009680 tx=0x7f5688009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.355+0000 7f5694c00700 1 -- 172.21.15.44:0/2647150585 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5680013070 con 0x7f5698108f10 2024-05-27T09:33:35.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.614+0000 7f569e400700 1 -- 172.21.15.44:0/2647150585 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f569802e5e0 con 0x7f5698108f10 2024-05-27T09:33:35.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.615+0000 7f5694c00700 1 -- 172.21.15.44:0/2647150585 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f56800aabc0 con 0x7f5698108f10 2024-05-27T09:33:35.616 INFO:teuthology.orchestra.run.smithi044.stdout:253403070518 2024-05-27T09:33:35.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 -- 172.21.15.44:0/2647150585 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f567c0b2900 msgr2=0x7f567c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:35.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f567c0b2900 0x7f567c0b4e10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5688009680 tx=0x7f5688009260 comp rx=0 tx=0).stop 2024-05-27T09:33:35.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 -- 172.21.15.44:0/2647150585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5698108f10 msgr2=0x7f569819b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:35.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5698108f10 0x7f569819b590 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5680000c00 tx=0x7f56800057c0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 -- 172.21.15.44:0/2647150585 shutdown_connections 2024-05-27T09:33:35.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f567c0b2900 0x7f567c0b4e10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5698109860 0x7f569819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56981045e0 0x7f569819aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 --2- 172.21.15.44:0/2647150585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5698108f10 0x7f569819b590 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 -- 172.21.15.44:0/2647150585 >> 172.21.15.44:0/2647150585 conn(0x7f5698100030 msgr2=0x7f5698108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:35.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 -- 172.21.15.44:0/2647150585 shutdown_connections 2024-05-27T09:33:35.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.619+0000 7f569e400700 1 -- 172.21.15.44:0/2647150585 wait complete. 2024-05-27T09:33:35.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.749+0000 7fbc61400700 1 -- 172.21.15.44:0/3434268511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fbc5c02e5e0 con 0x7fbc5c103a00 2024-05-27T09:33:35.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.750+0000 7fbc53400700 1 -- 172.21.15.44:0/3434268511 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbc480aabd0 con 0x7fbc5c103a00 2024-05-27T09:33:35.751 INFO:teuthology.orchestra.run.smithi044.stdout:180388626503 2024-05-27T09:33:35.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 -- 172.21.15.44:0/3434268511 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbc400b2940 msgr2=0x7fbc400b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbc400b2940 0x7fbc400b4e50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c011490 tx=0x7fbc4c009260 comp rx=0 tx=0).stop 2024-05-27T09:33:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 -- 172.21.15.44:0/3434268511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc5c103a00 msgr2=0x7fbc5c19d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc5c103a00 0x7fbc5c19d670 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbc480098e0 tx=0x7fbc480057c0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 -- 172.21.15.44:0/3434268511 shutdown_connections 2024-05-27T09:33:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fbc400b2940 0x7fbc400b4e50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc5c110c60 0x7fbc5c1a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c1030b0 0x7fbc5c19d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 --2- 172.21.15.44:0/3434268511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc5c103a00 0x7fbc5c19d670 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 -- 172.21.15.44:0/3434268511 >> 172.21.15.44:0/3434268511 conn(0x7fbc5c0fedb0 msgr2=0x7fbc5c1127c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 -- 172.21.15.44:0/3434268511 shutdown_connections 2024-05-27T09:33:35.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:35.753+0000 7fbc61400700 1 -- 172.21.15.44:0/3434268511 wait complete. 2024-05-27T09:33:36.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:35 smithi044 ceph-mon[31618]: pgmap v390: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T09:33:36.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:35 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2647150585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T09:33:36.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:35 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3434268511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T09:33:36.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:35 smithi050 ceph-mon[36371]: pgmap v390: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T09:33:36.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:35 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2647150585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T09:33:36.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:35 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3434268511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T09:33:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:35 smithi189 ceph-mon[36671]: pgmap v390: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T09:33:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:35 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2647150585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T09:33:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:35 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3434268511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T09:33:37.129 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070512 got 253403070518 for osd.9 2024-05-27T09:33:37.130 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:37.585 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626498 got 180388626503 for osd.6 2024-05-27T09:33:37.585 DEBUG:teuthology.parallel:result is None 2024-05-27T09:33:37.586 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-27T09:33:37.586 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T09:33:38.035 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:38.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:37 smithi044 ceph-mon[31618]: pgmap v391: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T09:33:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:37 smithi050 ceph-mon[36371]: pgmap v391: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T09:33:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:37 smithi189 ceph-mon[36671]: pgmap v391: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T09:33:39.213 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:39.213 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:39.213 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:39 smithi044 ceph-mon[31618]: pgmap v392: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T09:33:39.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:39.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:39.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:39 smithi050 ceph-mon[36371]: pgmap v392: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T09:33:39.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:39.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:39.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:39 smithi189 ceph-mon[36671]: pgmap v392: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T09:33:40.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.233+0000 7f5b1b200700 1 -- 172.21.15.44:0/3067731674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b1410a980 msgr2=0x7f5b1410ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:40.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.233+0000 7f5b1b200700 1 --2- 172.21.15.44:0/3067731674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b1410a980 0x7f5b1410ad90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009990 tx=0x7f5b00009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:40.234 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:33:40.235 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:33:40.235 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:40 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:40.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.233+0000 7f5b1b200700 1 -- 172.21.15.44:0/3067731674 shutdown_connections 2024-05-27T09:33:40.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.233+0000 7f5b1b200700 1 --2- 172.21.15.44:0/3067731674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b1410b2d0 0x7f5b14117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:40.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.233+0000 7f5b1b200700 1 --2- 172.21.15.44:0/3067731674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b14110960 0x7f5b14110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:40.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.233+0000 7f5b1b200700 1 --2- 172.21.15.44:0/3067731674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b1410a980 0x7f5b1410ad90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:40.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.233+0000 7f5b1b200700 1 -- 172.21.15.44:0/3067731674 >> 172.21.15.44:0/3067731674 conn(0x7f5b14077c70 msgr2=0x7f5b140780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:40.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.233+0000 7f5b1b200700 1 -- 172.21.15.44:0/3067731674 shutdown_connections 2024-05-27T09:33:40.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.233+0000 7f5b1b200700 1 -- 172.21.15.44:0/3067731674 wait complete. 2024-05-27T09:33:40.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.235+0000 7f5b1b200700 1 Processor -- start 2024-05-27T09:33:40.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.235+0000 7f5b1b200700 1 -- start start 2024-05-27T09:33:40.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.235+0000 7f5b1b200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b1410a980 0x7f5b141a14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:40.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.235+0000 7f5b1b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b1410b2d0 0x7f5b141a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:40.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.235+0000 7f5b1b200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b14110960 0x7f5b141a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:40.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.235+0000 7f5b1b200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1411aad0 con 0x7f5b1410a980 2024-05-27T09:33:40.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.235+0000 7f5b1b200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1411a950 con 0x7f5b14110960 2024-05-27T09:33:40.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.235+0000 7f5b1b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1411ac50 con 0x7f5b1410b2d0 2024-05-27T09:33:40.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.236+0000 7f5b13400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b1410b2d0 0x7f5b141a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:40.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.236+0000 7f5b13e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b1410a980 0x7f5b141a14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:40.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.236+0000 7f5b13400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b1410b2d0 0x7f5b141a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55424/0 (socket says 172.21.15.44:55424) 2024-05-27T09:33:40.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.236+0000 7f5b13400700 1 -- 172.21.15.44:0/1971942649 learned_addr learned my addr 172.21.15.44:0/1971942649 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:40.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.236+0000 7f5b19200700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b14110960 0x7f5b141a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:40.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.236+0000 7f5b13400700 1 -- 172.21.15.44:0/1971942649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b14110960 msgr2=0x7f5b141a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:40.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.236+0000 7f5b13400700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b14110960 0x7f5b141a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:40.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.236+0000 7f5b13400700 1 -- 172.21.15.44:0/1971942649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b1410a980 msgr2=0x7f5b141a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:40.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.236+0000 7f5b13400700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b1410a980 0x7f5b141a14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.236+0000 7f5b13400700 1 -- 172.21.15.44:0/1971942649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b141a6370 con 0x7f5b1410b2d0 2024-05-27T09:33:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.236+0000 7f5b13e00700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b1410a980 0x7f5b141a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:33:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.237+0000 7f5b13400700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b1410b2d0 0x7f5b141a1a40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5b00000c00 tx=0x7f5b000057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:40.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.237+0000 7f5b10c00700 1 -- 172.21.15.44:0/1971942649 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00027070 con 0x7f5b1410b2d0 2024-05-27T09:33:40.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.237+0000 7f5b1b200700 1 -- 172.21.15.44:0/1971942649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b141a6600 con 0x7f5b1410b2d0 2024-05-27T09:33:40.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.237+0000 7f5b10c00700 1 -- 172.21.15.44:0/1971942649 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b00022470 con 0x7f5b1410b2d0 2024-05-27T09:33:40.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.237+0000 7f5b10c00700 1 -- 172.21.15.44:0/1971942649 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00017650 con 0x7f5b1410b2d0 2024-05-27T09:33:40.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.238+0000 7f5b1b200700 1 -- 172.21.15.44:0/1971942649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b141adf80 con 0x7f5b1410b2d0 2024-05-27T09:33:40.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.239+0000 7f5b1b200700 1 -- 172.21.15.44:0/1971942649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b1410c070 con 0x7f5b1410b2d0 2024-05-27T09:33:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.242+0000 7f5b10c00700 1 -- 172.21.15.44:0/1971942649 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f5b0002a070 con 0x7f5b1410b2d0 2024-05-27T09:33:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.243+0000 7f5b10c00700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5af80b2880 0x7f5af80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.244+0000 7f5b10c00700 1 -- 172.21.15.44:0/1971942649 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f5b000e2ed0 con 0x7f5b1410b2d0 2024-05-27T09:33:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.244+0000 7f5b13e00700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5af80b2880 0x7f5af80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.244+0000 7f5b13e00700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5af80b2880 0x7f5af80b4d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5b1407d0e0 tx=0x7f5b04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:40.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.248+0000 7f5b10c00700 1 -- 172.21.15.44:0/1971942649 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5b00013070 con 0x7f5b1410b2d0 2024-05-27T09:33:40.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.484+0000 7f5b1b200700 1 -- 172.21.15.44:0/1971942649 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5b141a2800 con 0x7f5af80b2880 2024-05-27T09:33:40.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.506+0000 7f5b10c00700 1 -- 172.21.15.44:0/1971942649 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258817 (secure 0 0 0) 0x7f5b141a2800 con 0x7f5af80b2880 2024-05-27T09:33:40.512 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:33:40.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 -- 172.21.15.44:0/1971942649 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5af80b2880 msgr2=0x7f5af80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:40.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5af80b2880 0x7f5af80b4d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5b1407d0e0 tx=0x7f5b04009260 comp rx=0 tx=0).stop 2024-05-27T09:33:40.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 -- 172.21.15.44:0/1971942649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b1410b2d0 msgr2=0x7f5b141a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b1410b2d0 0x7f5b141a1a40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5b00000c00 tx=0x7f5b000057c0 comp rx=0 tx=0).stop 2024-05-27T09:33:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 -- 172.21.15.44:0/1971942649 shutdown_connections 2024-05-27T09:33:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f5af80b2880 0x7f5af80b4d90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b14110960 0x7f5b141a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b1410a980 0x7f5b141a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 --2- 172.21.15.44:0/1971942649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b1410b2d0 0x7f5b141a1a40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 -- 172.21.15.44:0/1971942649 >> 172.21.15.44:0/1971942649 conn(0x7f5b14077c70 msgr2=0x7f5b1407b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 -- 172.21.15.44:0/1971942649 shutdown_connections 2024-05-27T09:33:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:40.512+0000 7f5b1b200700 1 -- 172.21.15.44:0/1971942649 wait complete. 2024-05-27T09:33:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-05-27T09:33:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:33:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:33:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:40 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:40.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:33:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:33:40.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:40 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:33:41.159 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:41 smithi044 ceph-mon[31618]: from='client.24856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:33:41.159 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:41 smithi044 ceph-mon[31618]: pgmap v393: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-27T09:33:41.460 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-05-27T09:33:38.679568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655872,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":766,"num_read_kb":478,"num_write":298,"num_write_kb":2270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":742,"ondisk_log_size":742,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382348,"kb_used_data":12396,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559476,"statfs":{"total":1151940427776,"available":1151548903424,"internally_reserved":0,"allocated":12693504,"data_stored":7142244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":33,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004841"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618115+0000","last_change":"2024-05-27T09:31:24.899109+0000","last_active":"2024-05-27T09:33:26.618115+0000","last_peered":"2024-05-27T09:33:26.618115+0000","last_clean":"2024-05-27T09:33:26.618115+0000","last_became_active":"2024-05-27T09:31:24.898783+0000","last_became_peered":"2024-05-27T09:31:24.898783+0000","last_unstale":"2024-05-27T09:33:26.618115+0000","last_undegraded":"2024-05-27T09:33:26.618115+0000","last_fullsized":"2024-05-27T09:33:26.618115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:01:12.220389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.608195+0000","last_change":"2024-05-27T09:31:24.898382+0000","last_active":"2024-05-27T09:33:26.608195+0000","last_peered":"2024-05-27T09:33:26.608195+0000","last_clean":"2024-05-27T09:33:26.608195+0000","last_became_active":"2024-05-27T09:31:24.897975+0000","last_became_peered":"2024-05-27T09:31:24.897975+0000","last_unstale":"2024-05-27T09:33:26.608195+0000","last_undegraded":"2024-05-27T09:33:26.608195+0000","last_fullsized":"2024-05-27T09:33:26.608195+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:38.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.617056+0000","last_change":"2024-05-27T09:31:24.896122+0000","last_active":"2024-05-27T09:33:26.617056+0000","last_peered":"2024-05-27T09:33:26.617056+0000","last_clean":"2024-05-27T09:33:26.617056+0000","last_became_active":"2024-05-27T09:31:24.890867+0000","last_became_peered":"2024-05-27T09:31:24.890867+0000","last_unstale":"2024-05-27T09:33:26.617056+0000","last_undegraded":"2024-05-27T09:33:26.617056+0000","last_fullsized":"2024-05-27T09:33:26.617056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:09.982177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.621020+0000","last_change":"2024-05-27T09:31:57.868900+0000","last_active":"2024-05-27T09:33:26.621020+0000","last_peered":"2024-05-27T09:33:26.621020+0000","last_clean":"2024-05-27T09:33:26.621020+0000","last_became_active":"2024-05-27T09:31:57.868418+0000","last_became_peered":"2024-05-27T09:31:57.868418+0000","last_unstale":"2024-05-27T09:33:26.621020+0000","last_undegraded":"2024-05-27T09:33:26.621020+0000","last_fullsized":"2024-05-27T09:33:26.621020+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:08.966704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.624169+0000","last_change":"2024-05-27T09:31:24.910283+0000","last_active":"2024-05-27T09:33:26.624169+0000","last_peered":"2024-05-27T09:33:26.624169+0000","last_clean":"2024-05-27T09:33:26.624169+0000","last_became_active":"2024-05-27T09:31:24.909882+0000","last_became_peered":"2024-05-27T09:31:24.909882+0000","last_unstale":"2024-05-27T09:33:26.624169+0000","last_undegraded":"2024-05-27T09:33:26.624169+0000","last_fullsized":"2024-05-27T09:33:26.624169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:43.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.630792+0000","last_change":"2024-05-27T09:31:24.900934+0000","last_active":"2024-05-27T09:33:26.630792+0000","last_peered":"2024-05-27T09:33:26.630792+0000","last_clean":"2024-05-27T09:33:26.630792+0000","last_became_active":"2024-05-27T09:31:24.900534+0000","last_became_peered":"2024-05-27T09:31:24.900534+0000","last_unstale":"2024-05-27T09:33:26.630792+0000","last_undegraded":"2024-05-27T09:33:26.630792+0000","last_fullsized":"2024-05-27T09:33:26.630792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:53.318327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.630621+0000","last_change":"2024-05-27T09:31:24.907080+0000","last_active":"2024-05-27T09:33:26.630621+0000","last_peered":"2024-05-27T09:33:26.630621+0000","last_clean":"2024-05-27T09:33:26.630621+0000","last_became_active":"2024-05-27T09:31:24.906715+0000","last_became_peered":"2024-05-27T09:31:24.906715+0000","last_unstale":"2024-05-27T09:33:26.630621+0000","last_undegraded":"2024-05-27T09:33:26.630621+0000","last_fullsized":"2024-05-27T09:33:26.630621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:36.214659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625890+0000","last_change":"2024-05-27T09:31:58.200608+0000","last_active":"2024-05-27T09:33:26.625890+0000","last_peered":"2024-05-27T09:33:26.625890+0000","last_clean":"2024-05-27T09:33:26.625890+0000","last_became_active":"2024-05-27T09:31:58.199363+0000","last_became_peered":"2024-05-27T09:31:58.199363+0000","last_unstale":"2024-05-27T09:33:26.625890+0000","last_undegraded":"2024-05-27T09:33:26.625890+0000","last_fullsized":"2024-05-27T09:33:26.625890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:30:46.495930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618727+0000","last_change":"2024-05-27T09:31:24.892630+0000","last_active":"2024-05-27T09:33:26.618727+0000","last_peered":"2024-05-27T09:33:26.618727+0000","last_clean":"2024-05-27T09:33:26.618727+0000","last_became_active":"2024-05-27T09:31:24.892176+0000","last_became_peered":"2024-05-27T09:31:24.892176+0000","last_unstale":"2024-05-27T09:33:26.618727+0000","last_undegraded":"2024-05-27T09:33:26.618727+0000","last_fullsized":"2024-05-27T09:33:26.618727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:49:20.585601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.616507+0000","last_change":"2024-05-27T09:31:24.895281+0000","last_active":"2024-05-27T09:33:26.616507+0000","last_peered":"2024-05-27T09:33:26.616507+0000","last_clean":"2024-05-27T09:33:26.616507+0000","last_became_active":"2024-05-27T09:31:24.894886+0000","last_became_peered":"2024-05-27T09:31:24.894886+0000","last_unstale":"2024-05-27T09:33:26.616507+0000","last_undegraded":"2024-05-27T09:33:26.616507+0000","last_fullsized":"2024-05-27T09:33:26.616507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:41.494434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.617116+0000","last_change":"2024-05-27T09:31:24.900167+0000","last_active":"2024-05-27T09:33:26.617116+0000","last_peered":"2024-05-27T09:33:26.617116+0000","last_clean":"2024-05-27T09:33:26.617116+0000","last_became_active":"2024-05-27T09:31:24.899750+0000","last_became_peered":"2024-05-27T09:31:24.899750+0000","last_unstale":"2024-05-27T09:33:26.617116+0000","last_undegraded":"2024-05-27T09:33:26.617116+0000","last_fullsized":"2024-05-27T09:33:26.617116+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:30.602246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.626077+0000","last_change":"2024-05-27T09:31:57.868818+0000","last_active":"2024-05-27T09:33:26.626077+0000","last_peered":"2024-05-27T09:33:26.626077+0000","last_clean":"2024-05-27T09:33:26.626077+0000","last_became_active":"2024-05-27T09:31:57.868322+0000","last_became_peered":"2024-05-27T09:31:57.868322+0000","last_unstale":"2024-05-27T09:33:26.626077+0000","last_undegraded":"2024-05-27T09:33:26.626077+0000","last_fullsized":"2024-05-27T09:33:26.626077+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:35.304850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.634376+0000","last_change":"2024-05-27T09:31:24.902537+0000","last_active":"2024-05-27T09:33:26.634376+0000","last_peered":"2024-05-27T09:33:26.634376+0000","last_clean":"2024-05-27T09:33:26.634376+0000","last_became_active":"2024-05-27T09:31:24.902132+0000","last_became_peered":"2024-05-27T09:31:24.902132+0000","last_unstale":"2024-05-27T09:33:26.634376+0000","last_undegraded":"2024-05-27T09:33:26.634376+0000","last_fullsized":"2024-05-27T09:33:26.634376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:20:50.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.601615+0000","last_change":"2024-05-27T09:31:24.899182+0000","last_active":"2024-05-27T09:33:26.601615+0000","last_peered":"2024-05-27T09:33:26.601615+0000","last_clean":"2024-05-27T09:33:26.601615+0000","last_became_active":"2024-05-27T09:31:24.898749+0000","last_became_peered":"2024-05-27T09:31:24.898749+0000","last_unstale":"2024-05-27T09:33:26.601615+0000","last_undegraded":"2024-05-27T09:33:26.601615+0000","last_fullsized":"2024-05-27T09:33:26.601615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:44:44.844357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.605363+0000","last_change":"2024-05-27T09:31:24.901115+0000","last_active":"2024-05-27T09:33:26.605363+0000","last_peered":"2024-05-27T09:33:26.605363+0000","last_clean":"2024-05-27T09:33:26.605363+0000","last_became_active":"2024-05-27T09:31:24.897011+0000","last_became_peered":"2024-05-27T09:31:24.897011+0000","last_unstale":"2024-05-27T09:33:26.605363+0000","last_undegraded":"2024-05-27T09:33:26.605363+0000","last_fullsized":"2024-05-27T09:33:26.605363+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:01.130407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.617965+0000","last_change":"2024-05-27T09:31:24.903063+0000","last_active":"2024-05-27T09:33:26.617965+0000","last_peered":"2024-05-27T09:33:26.617965+0000","last_clean":"2024-05-27T09:33:26.617965+0000","last_became_active":"2024-05-27T09:31:24.899766+0000","last_became_peered":"2024-05-27T09:31:24.899766+0000","last_unstale":"2024-05-27T09:33:26.617965+0000","last_undegraded":"2024-05-27T09:33:26.617965+0000","last_fullsized":"2024-05-27T09:33:26.617965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:45.984226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.617948+0000","last_change":"2024-05-27T09:31:24.915922+0000","last_active":"2024-05-27T09:33:26.617948+0000","last_peered":"2024-05-27T09:33:26.617948+0000","last_clean":"2024-05-27T09:33:26.617948+0000","last_became_active":"2024-05-27T09:31:24.915623+0000","last_became_peered":"2024-05-27T09:31:24.915623+0000","last_unstale":"2024-05-27T09:33:26.617948+0000","last_undegraded":"2024-05-27T09:33:26.617948+0000","last_fullsized":"2024-05-27T09:33:26.617948+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:02:01.006704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.630130+0000","last_change":"2024-05-27T09:31:24.908397+0000","last_active":"2024-05-27T09:33:26.630130+0000","last_peered":"2024-05-27T09:33:26.630130+0000","last_clean":"2024-05-27T09:33:26.630130+0000","last_became_active":"2024-05-27T09:31:24.903644+0000","last_became_peered":"2024-05-27T09:31:24.903644+0000","last_unstale":"2024-05-27T09:33:26.630130+0000","last_undegraded":"2024-05-27T09:33:26.630130+0000","last_fullsized":"2024-05-27T09:33:26.630130+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:09.281429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.619517+0000","last_change":"2024-05-27T09:31:24.912706+0000","last_active":"2024-05-27T09:33:26.619517+0000","last_peered":"2024-05-27T09:33:26.619517+0000","last_clean":"2024-05-27T09:33:26.619517+0000","last_became_active":"2024-05-27T09:31:24.912357+0000","last_became_peered":"2024-05-27T09:31:24.912357+0000","last_unstale":"2024-05-27T09:33:26.619517+0000","last_undegraded":"2024-05-27T09:33:26.619517+0000","last_fullsized":"2024-05-27T09:33:26.619517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:23.334041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864126+0000","last_change":"2024-05-27T09:31:20.881652+0000","last_active":"2024-05-27T09:31:57.864126+0000","last_peered":"2024-05-27T09:31:57.864126+0000","last_clean":"2024-05-27T09:31:57.864126+0000","last_became_active":"2024-05-27T09:31:20.880757+0000","last_became_peered":"2024-05-27T09:31:20.880757+0000","last_unstale":"2024-05-27T09:31:57.864126+0000","last_undegraded":"2024-05-27T09:31:57.864126+0000","last_fullsized":"2024-05-27T09:31:57.864126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:01.433755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.613188+0000","last_change":"2024-05-27T09:31:24.908010+0000","last_active":"2024-05-27T09:33:26.613188+0000","last_peered":"2024-05-27T09:33:26.613188+0000","last_clean":"2024-05-27T09:33:26.613188+0000","last_became_active":"2024-05-27T09:31:24.907638+0000","last_became_peered":"2024-05-27T09:31:24.907638+0000","last_unstale":"2024-05-27T09:33:26.613188+0000","last_undegraded":"2024-05-27T09:33:26.613188+0000","last_fullsized":"2024-05-27T09:33:26.613188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:02:49.207709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.195330+0000","last_change":"2024-05-27T09:31:20.860556+0000","last_active":"2024-05-27T09:31:58.195330+0000","last_peered":"2024-05-27T09:31:58.195330+0000","last_clean":"2024-05-27T09:31:58.195330+0000","last_became_active":"2024-05-27T09:31:20.860172+0000","last_became_peered":"2024-05-27T09:31:20.860172+0000","last_unstale":"2024-05-27T09:31:58.195330+0000","last_undegraded":"2024-05-27T09:31:58.195330+0000","last_fullsized":"2024-05-27T09:31:58.195330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:08:21.545681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.631188+0000","last_change":"2024-05-27T09:31:24.910111+0000","last_active":"2024-05-27T09:33:26.631188+0000","last_peered":"2024-05-27T09:33:26.631188+0000","last_clean":"2024-05-27T09:33:26.631188+0000","last_became_active":"2024-05-27T09:31:24.908985+0000","last_became_peered":"2024-05-27T09:31:24.908985+0000","last_unstale":"2024-05-27T09:33:26.631188+0000","last_undegraded":"2024-05-27T09:33:26.631188+0000","last_fullsized":"2024-05-27T09:33:26.631188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:53:29.645792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864702+0000","last_change":"2024-05-27T09:31:20.879993+0000","last_active":"2024-05-27T09:31:57.864702+0000","last_peered":"2024-05-27T09:31:57.864702+0000","last_clean":"2024-05-27T09:31:57.864702+0000","last_became_active":"2024-05-27T09:31:20.878802+0000","last_became_peered":"2024-05-27T09:31:20.878802+0000","last_unstale":"2024-05-27T09:31:57.864702+0000","last_undegraded":"2024-05-27T09:31:57.864702+0000","last_fullsized":"2024-05-27T09:31:57.864702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:15.555612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.627583+0000","last_change":"2024-05-27T09:31:24.903533+0000","last_active":"2024-05-27T09:33:26.627583+0000","last_peered":"2024-05-27T09:33:26.627583+0000","last_clean":"2024-05-27T09:33:26.627583+0000","last_became_active":"2024-05-27T09:31:24.903166+0000","last_became_peered":"2024-05-27T09:31:24.903166+0000","last_unstale":"2024-05-27T09:33:26.627583+0000","last_undegraded":"2024-05-27T09:33:26.627583+0000","last_fullsized":"2024-05-27T09:33:26.627583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:52.539470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.878845+0000","last_change":"2024-05-27T09:31:57.878426+0000","last_active":"2024-05-27T09:31:57.878845+0000","last_peered":"2024-05-27T09:31:57.878845+0000","last_clean":"2024-05-27T09:31:57.878845+0000","last_became_active":"2024-05-27T09:31:57.869146+0000","last_became_peered":"2024-05-27T09:31:57.869146+0000","last_unstale":"2024-05-27T09:31:57.878845+0000","last_undegraded":"2024-05-27T09:31:57.878845+0000","last_fullsized":"2024-05-27T09:31:57.878845+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:34:51.926092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.607700+0000","last_change":"2024-05-27T09:31:24.897426+0000","last_active":"2024-05-27T09:33:26.607700+0000","last_peered":"2024-05-27T09:33:26.607700+0000","last_clean":"2024-05-27T09:33:26.607700+0000","last_became_active":"2024-05-27T09:31:24.897080+0000","last_became_peered":"2024-05-27T09:31:24.897080+0000","last_unstale":"2024-05-27T09:33:26.607700+0000","last_undegraded":"2024-05-27T09:33:26.607700+0000","last_fullsized":"2024-05-27T09:33:26.607700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:15:42.428391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'7","reported_seq":148,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:35.962096+0000","last_change":"2024-05-27T09:31:20.846681+0000","last_active":"2024-05-27T09:33:35.962096+0000","last_peered":"2024-05-27T09:33:35.962096+0000","last_clean":"2024-05-27T09:33:35.962096+0000","last_became_active":"2024-05-27T09:31:20.846302+0000","last_became_peered":"2024-05-27T09:31:20.846302+0000","last_unstale":"2024-05-27T09:33:35.962096+0000","last_undegraded":"2024-05-27T09:33:35.962096+0000","last_fullsized":"2024-05-27T09:33:35.962096+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:19.609612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625720+0000","last_change":"2024-05-27T09:31:24.907209+0000","last_active":"2024-05-27T09:33:26.625720+0000","last_peered":"2024-05-27T09:33:26.625720+0000","last_clean":"2024-05-27T09:33:26.625720+0000","last_became_active":"2024-05-27T09:31:24.906752+0000","last_became_peered":"2024-05-27T09:31:24.906752+0000","last_unstale":"2024-05-27T09:33:26.625720+0000","last_undegraded":"2024-05-27T09:33:26.625720+0000","last_fullsized":"2024-05-27T09:33:26.625720+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:53.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.860743+0000","last_change":"2024-05-27T09:31:20.879778+0000","last_active":"2024-05-27T09:31:57.860743+0000","last_peered":"2024-05-27T09:31:57.860743+0000","last_clean":"2024-05-27T09:31:57.860743+0000","last_became_active":"2024-05-27T09:31:20.879259+0000","last_became_peered":"2024-05-27T09:31:20.879259+0000","last_unstale":"2024-05-27T09:31:57.860743+0000","last_undegraded":"2024-05-27T09:31:57.860743+0000","last_fullsized":"2024-05-27T09:31:57.860743+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:20:07.972653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618333+0000","last_change":"2024-05-27T09:31:58.195516+0000","last_active":"2024-05-27T09:33:26.618333+0000","last_peered":"2024-05-27T09:33:26.618333+0000","last_clean":"2024-05-27T09:33:26.618333+0000","last_became_active":"2024-05-27T09:31:58.195118+0000","last_became_peered":"2024-05-27T09:31:58.195118+0000","last_unstale":"2024-05-27T09:33:26.618333+0000","last_undegraded":"2024-05-27T09:33:26.618333+0000","last_fullsized":"2024-05-27T09:33:26.618333+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:11:23.547605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.196330+0000","last_change":"2024-05-27T09:31:20.863152+0000","last_active":"2024-05-27T09:31:58.196330+0000","last_peered":"2024-05-27T09:31:58.196330+0000","last_clean":"2024-05-27T09:31:58.196330+0000","last_became_active":"2024-05-27T09:31:20.862708+0000","last_became_peered":"2024-05-27T09:31:20.862708+0000","last_unstale":"2024-05-27T09:31:58.196330+0000","last_undegraded":"2024-05-27T09:31:58.196330+0000","last_fullsized":"2024-05-27T09:31:58.196330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:21.645773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625895+0000","last_change":"2024-05-27T09:31:24.893988+0000","last_active":"2024-05-27T09:33:26.625895+0000","last_peered":"2024-05-27T09:33:26.625895+0000","last_clean":"2024-05-27T09:33:26.625895+0000","last_became_active":"2024-05-27T09:31:24.893504+0000","last_became_peered":"2024-05-27T09:31:24.893504+0000","last_unstale":"2024-05-27T09:33:26.625895+0000","last_undegraded":"2024-05-27T09:33:26.625895+0000","last_fullsized":"2024-05-27T09:33:26.625895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:47.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863900+0000","last_change":"2024-05-27T09:31:20.845555+0000","last_active":"2024-05-27T09:31:57.863900+0000","last_peered":"2024-05-27T09:31:57.863900+0000","last_clean":"2024-05-27T09:31:57.863900+0000","last_became_active":"2024-05-27T09:31:20.845139+0000","last_became_peered":"2024-05-27T09:31:20.845139+0000","last_unstale":"2024-05-27T09:31:57.863900+0000","last_undegraded":"2024-05-27T09:31:57.863900+0000","last_fullsized":"2024-05-27T09:31:57.863900+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:25.917845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.626906+0000","last_change":"2024-05-27T09:31:24.916142+0000","last_active":"2024-05-27T09:33:26.626906+0000","last_peered":"2024-05-27T09:33:26.626906+0000","last_clean":"2024-05-27T09:33:26.626906+0000","last_became_active":"2024-05-27T09:31:24.915709+0000","last_became_peered":"2024-05-27T09:31:24.915709+0000","last_unstale":"2024-05-27T09:33:26.626906+0000","last_undegraded":"2024-05-27T09:33:26.626906+0000","last_fullsized":"2024-05-27T09:33:26.626906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:38.029648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":138,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:32.956952+0000","last_change":"2024-05-27T09:31:20.859240+0000","last_active":"2024-05-27T09:33:32.956952+0000","last_peered":"2024-05-27T09:33:32.956952+0000","last_clean":"2024-05-27T09:33:32.956952+0000","last_became_active":"2024-05-27T09:31:20.858798+0000","last_became_peered":"2024-05-27T09:31:20.858798+0000","last_unstale":"2024-05-27T09:33:32.956952+0000","last_undegraded":"2024-05-27T09:33:32.956952+0000","last_fullsized":"2024-05-27T09:33:32.956952+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:42.280165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.630863+0000","last_change":"2024-05-27T09:31:24.908818+0000","last_active":"2024-05-27T09:33:26.630863+0000","last_peered":"2024-05-27T09:33:26.630863+0000","last_clean":"2024-05-27T09:33:26.630863+0000","last_became_active":"2024-05-27T09:31:24.908410+0000","last_became_peered":"2024-05-27T09:31:24.908410+0000","last_unstale":"2024-05-27T09:33:26.630863+0000","last_undegraded":"2024-05-27T09:33:26.630863+0000","last_fullsized":"2024-05-27T09:33:26.630863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:05:40.326232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864270+0000","last_change":"2024-05-27T09:31:20.848632+0000","last_active":"2024-05-27T09:31:57.864270+0000","last_peered":"2024-05-27T09:31:57.864270+0000","last_clean":"2024-05-27T09:31:57.864270+0000","last_became_active":"2024-05-27T09:31:20.847558+0000","last_became_peered":"2024-05-27T09:31:20.847558+0000","last_unstale":"2024-05-27T09:31:57.864270+0000","last_undegraded":"2024-05-27T09:31:57.864270+0000","last_fullsized":"2024-05-27T09:31:57.864270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:53.930565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.604997+0000","last_change":"2024-05-27T09:31:24.901513+0000","last_active":"2024-05-27T09:33:26.604997+0000","last_peered":"2024-05-27T09:33:26.604997+0000","last_clean":"2024-05-27T09:33:26.604997+0000","last_became_active":"2024-05-27T09:31:24.895924+0000","last_became_peered":"2024-05-27T09:31:24.895924+0000","last_unstale":"2024-05-27T09:33:26.604997+0000","last_undegraded":"2024-05-27T09:33:26.604997+0000","last_fullsized":"2024-05-27T09:33:26.604997+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:27:39.939098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864463+0000","last_change":"2024-05-27T09:31:20.855195+0000","last_active":"2024-05-27T09:31:57.864463+0000","last_peered":"2024-05-27T09:31:57.864463+0000","last_clean":"2024-05-27T09:31:57.864463+0000","last_became_active":"2024-05-27T09:31:20.854910+0000","last_became_peered":"2024-05-27T09:31:20.854910+0000","last_unstale":"2024-05-27T09:31:57.864463+0000","last_undegraded":"2024-05-27T09:31:57.864463+0000","last_fullsized":"2024-05-27T09:31:57.864463+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:02.795103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.622600+0000","last_change":"2024-05-27T09:31:24.895887+0000","last_active":"2024-05-27T09:33:26.622600+0000","last_peered":"2024-05-27T09:33:26.622600+0000","last_clean":"2024-05-27T09:33:26.622600+0000","last_became_active":"2024-05-27T09:31:24.894622+0000","last_became_peered":"2024-05-27T09:31:24.894622+0000","last_unstale":"2024-05-27T09:33:26.622600+0000","last_undegraded":"2024-05-27T09:33:26.622600+0000","last_fullsized":"2024-05-27T09:33:26.622600+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:32.833946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'32","reported_seq":283,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:33.963795+0000","last_change":"2024-05-27T09:31:20.846063+0000","last_active":"2024-05-27T09:33:33.963795+0000","last_peered":"2024-05-27T09:33:33.963795+0000","last_clean":"2024-05-27T09:33:33.963795+0000","last_became_active":"2024-05-27T09:31:20.845401+0000","last_became_peered":"2024-05-27T09:31:20.845401+0000","last_unstale":"2024-05-27T09:33:33.963795+0000","last_undegraded":"2024-05-27T09:33:33.963795+0000","last_fullsized":"2024-05-27T09:33:33.963795+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:21:38.460183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60596,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":225,"num_read_kb":172,"num_write":33,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618414+0000","last_change":"2024-05-27T09:31:24.892748+0000","last_active":"2024-05-27T09:33:26.618414+0000","last_peered":"2024-05-27T09:33:26.618414+0000","last_clean":"2024-05-27T09:33:26.618414+0000","last_became_active":"2024-05-27T09:31:24.892394+0000","last_became_peered":"2024-05-27T09:31:24.892394+0000","last_unstale":"2024-05-27T09:33:26.618414+0000","last_undegraded":"2024-05-27T09:33:26.618414+0000","last_fullsized":"2024-05-27T09:33:26.618414+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:39:31.381072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.867076+0000","last_change":"2024-05-27T09:31:20.860789+0000","last_active":"2024-05-27T09:31:57.867076+0000","last_peered":"2024-05-27T09:31:57.867076+0000","last_clean":"2024-05-27T09:31:57.867076+0000","last_became_active":"2024-05-27T09:31:20.860369+0000","last_became_peered":"2024-05-27T09:31:20.860369+0000","last_unstale":"2024-05-27T09:31:57.867076+0000","last_undegraded":"2024-05-27T09:31:57.867076+0000","last_fullsized":"2024-05-27T09:31:57.867076+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:05:35.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.620332+0000","last_change":"2024-05-27T09:31:24.918239+0000","last_active":"2024-05-27T09:33:26.620332+0000","last_peered":"2024-05-27T09:33:26.620332+0000","last_clean":"2024-05-27T09:33:26.620332+0000","last_became_active":"2024-05-27T09:31:24.917750+0000","last_became_peered":"2024-05-27T09:31:24.917750+0000","last_unstale":"2024-05-27T09:33:26.620332+0000","last_undegraded":"2024-05-27T09:33:26.620332+0000","last_fullsized":"2024-05-27T09:33:26.620332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:07.162994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863930+0000","last_change":"2024-05-27T09:31:20.865308+0000","last_active":"2024-05-27T09:31:57.863930+0000","last_peered":"2024-05-27T09:31:57.863930+0000","last_clean":"2024-05-27T09:31:57.863930+0000","last_became_active":"2024-05-27T09:31:20.864920+0000","last_became_peered":"2024-05-27T09:31:20.864920+0000","last_unstale":"2024-05-27T09:31:57.863930+0000","last_undegraded":"2024-05-27T09:31:57.863930+0000","last_fullsized":"2024-05-27T09:31:57.863930+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:36.773322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.632615+0000","last_change":"2024-05-27T09:31:24.902533+0000","last_active":"2024-05-27T09:33:26.632615+0000","last_peered":"2024-05-27T09:33:26.632615+0000","last_clean":"2024-05-27T09:33:26.632615+0000","last_became_active":"2024-05-27T09:31:24.902136+0000","last_became_peered":"2024-05-27T09:31:24.902136+0000","last_unstale":"2024-05-27T09:33:26.632615+0000","last_undegraded":"2024-05-27T09:33:26.632615+0000","last_fullsized":"2024-05-27T09:33:26.632615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:49:32.203577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'5","reported_seq":147,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:37.009221+0000","last_change":"2024-05-27T09:31:20.846060+0000","last_active":"2024-05-27T09:33:37.009221+0000","last_peered":"2024-05-27T09:33:37.009221+0000","last_clean":"2024-05-27T09:33:37.009221+0000","last_became_active":"2024-05-27T09:31:20.845632+0000","last_became_peered":"2024-05-27T09:31:20.845632+0000","last_unstale":"2024-05-27T09:33:37.009221+0000","last_undegraded":"2024-05-27T09:33:37.009221+0000","last_fullsized":"2024-05-27T09:33:37.009221+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:55.166579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.619066+0000","last_change":"2024-05-27T09:31:24.895866+0000","last_active":"2024-05-27T09:33:26.619066+0000","last_peered":"2024-05-27T09:33:26.619066+0000","last_clean":"2024-05-27T09:33:26.619066+0000","last_became_active":"2024-05-27T09:31:24.895413+0000","last_became_peered":"2024-05-27T09:31:24.895413+0000","last_unstale":"2024-05-27T09:33:26.619066+0000","last_undegraded":"2024-05-27T09:33:26.619066+0000","last_fullsized":"2024-05-27T09:33:26.619066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:01:30.075710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863490+0000","last_change":"2024-05-27T09:31:20.879086+0000","last_active":"2024-05-27T09:31:57.863490+0000","last_peered":"2024-05-27T09:31:57.863490+0000","last_clean":"2024-05-27T09:31:57.863490+0000","last_became_active":"2024-05-27T09:31:20.878331+0000","last_became_peered":"2024-05-27T09:31:20.878331+0000","last_unstale":"2024-05-27T09:31:57.863490+0000","last_undegraded":"2024-05-27T09:31:57.863490+0000","last_fullsized":"2024-05-27T09:31:57.863490+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:31.169812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.616897+0000","last_change":"2024-05-27T09:31:24.895069+0000","last_active":"2024-05-27T09:33:26.616897+0000","last_peered":"2024-05-27T09:33:26.616897+0000","last_clean":"2024-05-27T09:33:26.616897+0000","last_became_active":"2024-05-27T09:31:24.894626+0000","last_became_peered":"2024-05-27T09:31:24.894626+0000","last_unstale":"2024-05-27T09:33:26.616897+0000","last_undegraded":"2024-05-27T09:33:26.616897+0000","last_fullsized":"2024-05-27T09:33:26.616897+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:21:22.725268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864050+0000","last_change":"2024-05-27T09:31:20.877416+0000","last_active":"2024-05-27T09:31:57.864050+0000","last_peered":"2024-05-27T09:31:57.864050+0000","last_clean":"2024-05-27T09:31:57.864050+0000","last_became_active":"2024-05-27T09:31:20.876093+0000","last_became_peered":"2024-05-27T09:31:20.876093+0000","last_unstale":"2024-05-27T09:31:57.864050+0000","last_undegraded":"2024-05-27T09:31:57.864050+0000","last_fullsized":"2024-05-27T09:31:57.864050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:34.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.610608+0000","last_change":"2024-05-27T09:31:24.898430+0000","last_active":"2024-05-27T09:33:26.610608+0000","last_peered":"2024-05-27T09:33:26.610608+0000","last_clean":"2024-05-27T09:33:26.610608+0000","last_became_active":"2024-05-27T09:31:24.897643+0000","last_became_peered":"2024-05-27T09:31:24.897643+0000","last_unstale":"2024-05-27T09:33:26.610608+0000","last_undegraded":"2024-05-27T09:33:26.610608+0000","last_fullsized":"2024-05-27T09:33:26.610608+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:44.485561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863820+0000","last_change":"2024-05-27T09:31:20.858253+0000","last_active":"2024-05-27T09:31:57.863820+0000","last_peered":"2024-05-27T09:31:57.863820+0000","last_clean":"2024-05-27T09:31:57.863820+0000","last_became_active":"2024-05-27T09:31:20.857736+0000","last_became_peered":"2024-05-27T09:31:20.857736+0000","last_unstale":"2024-05-27T09:31:57.863820+0000","last_undegraded":"2024-05-27T09:31:57.863820+0000","last_fullsized":"2024-05-27T09:31:57.863820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:53:48.301763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625573+0000","last_change":"2024-05-27T09:31:24.912788+0000","last_active":"2024-05-27T09:33:26.625573+0000","last_peered":"2024-05-27T09:33:26.625573+0000","last_clean":"2024-05-27T09:33:26.625573+0000","last_became_active":"2024-05-27T09:31:24.912366+0000","last_became_peered":"2024-05-27T09:31:24.912366+0000","last_unstale":"2024-05-27T09:33:26.625573+0000","last_undegraded":"2024-05-27T09:33:26.625573+0000","last_fullsized":"2024-05-27T09:33:26.625573+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:55:03.397578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863905+0000","last_change":"2024-05-27T09:31:20.862188+0000","last_active":"2024-05-27T09:31:57.863905+0000","last_peered":"2024-05-27T09:31:57.863905+0000","last_clean":"2024-05-27T09:31:57.863905+0000","last_became_active":"2024-05-27T09:31:20.861791+0000","last_became_peered":"2024-05-27T09:31:20.861791+0000","last_unstale":"2024-05-27T09:31:57.863905+0000","last_undegraded":"2024-05-27T09:31:57.863905+0000","last_fullsized":"2024-05-27T09:31:57.863905+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:16:09.127223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.616200+0000","last_change":"2024-05-27T09:31:24.898460+0000","last_active":"2024-05-27T09:33:26.616200+0000","last_peered":"2024-05-27T09:33:26.616200+0000","last_clean":"2024-05-27T09:33:26.616200+0000","last_became_active":"2024-05-27T09:31:24.896787+0000","last_became_peered":"2024-05-27T09:31:24.896787+0000","last_unstale":"2024-05-27T09:33:26.616200+0000","last_undegraded":"2024-05-27T09:33:26.616200+0000","last_fullsized":"2024-05-27T09:33:26.616200+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:10.199227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.605196+0000","last_change":"2024-05-27T09:31:24.896323+0000","last_active":"2024-05-27T09:33:26.605196+0000","last_peered":"2024-05-27T09:33:26.605196+0000","last_clean":"2024-05-27T09:33:26.605196+0000","last_became_active":"2024-05-27T09:31:24.895919+0000","last_became_peered":"2024-05-27T09:31:24.895919+0000","last_unstale":"2024-05-27T09:33:26.605196+0000","last_undegraded":"2024-05-27T09:33:26.605196+0000","last_fullsized":"2024-05-27T09:33:26.605196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:04.883166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":140,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:32.999452+0000","last_change":"2024-05-27T09:31:20.862126+0000","last_active":"2024-05-27T09:33:32.999452+0000","last_peered":"2024-05-27T09:33:32.999452+0000","last_clean":"2024-05-27T09:33:32.999452+0000","last_became_active":"2024-05-27T09:31:20.858964+0000","last_became_peered":"2024-05-27T09:31:20.858964+0000","last_unstale":"2024-05-27T09:33:32.999452+0000","last_undegraded":"2024-05-27T09:33:32.999452+0000","last_fullsized":"2024-05-27T09:33:32.999452+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:51:26.838559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625163+0000","last_change":"2024-05-27T09:31:24.905549+0000","last_active":"2024-05-27T09:33:26.625163+0000","last_peered":"2024-05-27T09:33:26.625163+0000","last_clean":"2024-05-27T09:33:26.625163+0000","last_became_active":"2024-05-27T09:31:24.905212+0000","last_became_peered":"2024-05-27T09:31:24.905212+0000","last_unstale":"2024-05-27T09:33:26.625163+0000","last_undegraded":"2024-05-27T09:33:26.625163+0000","last_fullsized":"2024-05-27T09:33:26.625163+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:18.267807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865753+0000","last_change":"2024-05-27T09:31:20.848527+0000","last_active":"2024-05-27T09:31:57.865753+0000","last_peered":"2024-05-27T09:31:57.865753+0000","last_clean":"2024-05-27T09:31:57.865753+0000","last_became_active":"2024-05-27T09:31:20.847779+0000","last_became_peered":"2024-05-27T09:31:20.847779+0000","last_unstale":"2024-05-27T09:31:57.865753+0000","last_undegraded":"2024-05-27T09:31:57.865753+0000","last_fullsized":"2024-05-27T09:31:57.865753+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:15:01.131306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":102,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863868+0000","last_change":"2024-05-27T09:29:46.120380+0000","last_active":"2024-05-27T09:31:57.863868+0000","last_peered":"2024-05-27T09:31:57.863868+0000","last_clean":"2024-05-27T09:31:57.863868+0000","last_became_active":"2024-05-27T09:29:46.113658+0000","last_became_peered":"2024-05-27T09:29:46.113658+0000","last_unstale":"2024-05-27T09:31:57.863868+0000","last_undegraded":"2024-05-27T09:31:57.863868+0000","last_fullsized":"2024-05-27T09:31:57.863868+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_clean_scrub_stamp":"2024-05-27T09:25:47.244521+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:28:40.424824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.623606+0000","last_change":"2024-05-27T09:31:24.900736+0000","last_active":"2024-05-27T09:33:26.623606+0000","last_peered":"2024-05-27T09:33:26.623606+0000","last_clean":"2024-05-27T09:33:26.623606+0000","last_became_active":"2024-05-27T09:31:24.900354+0000","last_became_peered":"2024-05-27T09:31:24.900354+0000","last_unstale":"2024-05-27T09:33:26.623606+0000","last_undegraded":"2024-05-27T09:33:26.623606+0000","last_fullsized":"2024-05-27T09:33:26.623606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:39:33.171977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.874220+0000","last_change":"2024-05-27T09:31:57.873603+0000","last_active":"2024-05-27T09:31:57.874220+0000","last_peered":"2024-05-27T09:31:57.874220+0000","last_clean":"2024-05-27T09:31:57.874220+0000","last_became_active":"2024-05-27T09:31:57.868616+0000","last_became_peered":"2024-05-27T09:31:57.868616+0000","last_unstale":"2024-05-27T09:31:57.874220+0000","last_undegraded":"2024-05-27T09:31:57.874220+0000","last_fullsized":"2024-05-27T09:31:57.874220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:02:20.112667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864575+0000","last_change":"2024-05-27T09:31:20.878557+0000","last_active":"2024-05-27T09:31:57.864575+0000","last_peered":"2024-05-27T09:31:57.864575+0000","last_clean":"2024-05-27T09:31:57.864575+0000","last_became_active":"2024-05-27T09:31:20.877969+0000","last_became_peered":"2024-05-27T09:31:20.877969+0000","last_unstale":"2024-05-27T09:31:57.864575+0000","last_undegraded":"2024-05-27T09:31:57.864575+0000","last_fullsized":"2024-05-27T09:31:57.864575+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:46.859645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.631540+0000","last_change":"2024-05-27T09:31:24.907906+0000","last_active":"2024-05-27T09:33:26.631540+0000","last_peered":"2024-05-27T09:33:26.631540+0000","last_clean":"2024-05-27T09:33:26.631540+0000","last_became_active":"2024-05-27T09:31:24.907473+0000","last_became_peered":"2024-05-27T09:31:24.907473+0000","last_unstale":"2024-05-27T09:33:26.631540+0000","last_undegraded":"2024-05-27T09:33:26.631540+0000","last_fullsized":"2024-05-27T09:33:26.631540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:11:38.536919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'12","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.957464+0000","last_change":"2024-05-27T09:31:20.881250+0000","last_active":"2024-05-27T09:33:26.957464+0000","last_peered":"2024-05-27T09:33:26.957464+0000","last_clean":"2024-05-27T09:33:26.957464+0000","last_became_active":"2024-05-27T09:31:20.880550+0000","last_became_peered":"2024-05-27T09:31:20.880550+0000","last_unstale":"2024-05-27T09:33:26.957464+0000","last_undegraded":"2024-05-27T09:33:26.957464+0000","last_fullsized":"2024-05-27T09:33:26.957464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:14.783260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.615068+0000","last_change":"2024-05-27T09:31:24.894554+0000","last_active":"2024-05-27T09:33:26.615068+0000","last_peered":"2024-05-27T09:33:26.615068+0000","last_clean":"2024-05-27T09:33:26.615068+0000","last_became_active":"2024-05-27T09:31:24.893800+0000","last_became_peered":"2024-05-27T09:31:24.893800+0000","last_unstale":"2024-05-27T09:33:26.615068+0000","last_undegraded":"2024-05-27T09:33:26.615068+0000","last_fullsized":"2024-05-27T09:33:26.615068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:13.368892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865516+0000","last_change":"2024-05-27T09:31:21.054476+0000","last_active":"2024-05-27T09:31:57.865516+0000","last_peered":"2024-05-27T09:31:57.865516+0000","last_clean":"2024-05-27T09:31:57.865516+0000","last_became_active":"2024-05-27T09:31:21.054091+0000","last_became_peered":"2024-05-27T09:31:21.054091+0000","last_unstale":"2024-05-27T09:31:57.865516+0000","last_undegraded":"2024-05-27T09:31:57.865516+0000","last_fullsized":"2024-05-27T09:31:57.865516+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:15.909230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618501+0000","last_change":"2024-05-27T09:31:24.913157+0000","last_active":"2024-05-27T09:33:26.618501+0000","last_peered":"2024-05-27T09:33:26.618501+0000","last_clean":"2024-05-27T09:33:26.618501+0000","last_became_active":"2024-05-27T09:31:24.912793+0000","last_became_peered":"2024-05-27T09:31:24.912793+0000","last_unstale":"2024-05-27T09:33:26.618501+0000","last_undegraded":"2024-05-27T09:33:26.618501+0000","last_fullsized":"2024-05-27T09:33:26.618501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:17:36.451069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863696+0000","last_change":"2024-05-27T09:31:20.861349+0000","last_active":"2024-05-27T09:31:57.863696+0000","last_peered":"2024-05-27T09:31:57.863696+0000","last_clean":"2024-05-27T09:31:57.863696+0000","last_became_active":"2024-05-27T09:31:20.860896+0000","last_became_peered":"2024-05-27T09:31:20.860896+0000","last_unstale":"2024-05-27T09:31:57.863696+0000","last_undegraded":"2024-05-27T09:31:57.863696+0000","last_fullsized":"2024-05-27T09:31:57.863696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:00.772910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.594260+0000","last_change":"2024-05-27T09:31:24.893240+0000","last_active":"2024-05-27T09:33:26.594260+0000","last_peered":"2024-05-27T09:33:26.594260+0000","last_clean":"2024-05-27T09:33:26.594260+0000","last_became_active":"2024-05-27T09:31:24.892169+0000","last_became_peered":"2024-05-27T09:31:24.892169+0000","last_unstale":"2024-05-27T09:33:26.594260+0000","last_undegraded":"2024-05-27T09:33:26.594260+0000","last_fullsized":"2024-05-27T09:33:26.594260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:26:59.660725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.193611+0000","last_change":"2024-05-27T09:31:20.864307+0000","last_active":"2024-05-27T09:31:58.193611+0000","last_peered":"2024-05-27T09:31:58.193611+0000","last_clean":"2024-05-27T09:31:58.193611+0000","last_became_active":"2024-05-27T09:31:20.863913+0000","last_became_peered":"2024-05-27T09:31:20.863913+0000","last_unstale":"2024-05-27T09:31:58.193611+0000","last_undegraded":"2024-05-27T09:31:58.193611+0000","last_fullsized":"2024-05-27T09:31:58.193611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:58:57.656731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.611493+0000","last_change":"2024-05-27T09:31:24.905728+0000","last_active":"2024-05-27T09:33:26.611493+0000","last_peered":"2024-05-27T09:33:26.611493+0000","last_clean":"2024-05-27T09:33:26.611493+0000","last_became_active":"2024-05-27T09:31:24.905365+0000","last_became_peered":"2024-05-27T09:31:24.905365+0000","last_unstale":"2024-05-27T09:33:26.611493+0000","last_undegraded":"2024-05-27T09:33:26.611493+0000","last_fullsized":"2024-05-27T09:33:26.611493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:05.013037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.866937+0000","last_change":"2024-05-27T09:31:20.878478+0000","last_active":"2024-05-27T09:31:57.866937+0000","last_peered":"2024-05-27T09:31:57.866937+0000","last_clean":"2024-05-27T09:31:57.866937+0000","last_became_active":"2024-05-27T09:31:20.877952+0000","last_became_peered":"2024-05-27T09:31:20.877952+0000","last_unstale":"2024-05-27T09:31:57.866937+0000","last_undegraded":"2024-05-27T09:31:57.866937+0000","last_fullsized":"2024-05-27T09:31:57.866937+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:16:27.580641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.613996+0000","last_change":"2024-05-27T09:31:24.904967+0000","last_active":"2024-05-27T09:33:26.613996+0000","last_peered":"2024-05-27T09:33:26.613996+0000","last_clean":"2024-05-27T09:33:26.613996+0000","last_became_active":"2024-05-27T09:31:24.904542+0000","last_became_peered":"2024-05-27T09:31:24.904542+0000","last_unstale":"2024-05-27T09:33:26.613996+0000","last_undegraded":"2024-05-27T09:33:26.613996+0000","last_fullsized":"2024-05-27T09:33:26.613996+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:01:05.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.875713+0000","last_change":"2024-05-27T09:31:57.875713+0000","last_active":"2024-05-27T09:31:57.875713+0000","last_peered":"2024-05-27T09:31:57.875713+0000","last_clean":"2024-05-27T09:31:57.875713+0000","last_became_active":"2024-05-27T09:31:57.870244+0000","last_became_peered":"2024-05-27T09:31:57.870244+0000","last_unstale":"2024-05-27T09:31:57.875713+0000","last_undegraded":"2024-05-27T09:31:57.875713+0000","last_fullsized":"2024-05-27T09:31:57.875713+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:49.272418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.623162+0000","last_change":"2024-05-27T09:31:24.892655+0000","last_active":"2024-05-27T09:33:26.623162+0000","last_peered":"2024-05-27T09:33:26.623162+0000","last_clean":"2024-05-27T09:33:26.623162+0000","last_became_active":"2024-05-27T09:31:24.892112+0000","last_became_peered":"2024-05-27T09:31:24.892112+0000","last_unstale":"2024-05-27T09:33:26.623162+0000","last_undegraded":"2024-05-27T09:33:26.623162+0000","last_fullsized":"2024-05-27T09:33:26.623162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.604118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.876559+0000","last_change":"2024-05-27T09:31:57.875213+0000","last_active":"2024-05-27T09:31:57.876559+0000","last_peered":"2024-05-27T09:31:57.876559+0000","last_clean":"2024-05-27T09:31:57.876559+0000","last_became_active":"2024-05-27T09:31:57.869025+0000","last_became_peered":"2024-05-27T09:31:57.869025+0000","last_unstale":"2024-05-27T09:31:57.876559+0000","last_undegraded":"2024-05-27T09:31:57.876559+0000","last_fullsized":"2024-05-27T09:31:57.876559+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:51.433347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.619825+0000","last_change":"2024-05-27T09:31:24.899130+0000","last_active":"2024-05-27T09:33:26.619825+0000","last_peered":"2024-05-27T09:33:26.619825+0000","last_clean":"2024-05-27T09:33:26.619825+0000","last_became_active":"2024-05-27T09:31:24.898722+0000","last_became_peered":"2024-05-27T09:31:24.898722+0000","last_unstale":"2024-05-27T09:33:26.619825+0000","last_undegraded":"2024-05-27T09:33:26.619825+0000","last_fullsized":"2024-05-27T09:33:26.619825+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:51.247275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864289+0000","last_change":"2024-05-27T09:31:20.863757+0000","last_active":"2024-05-27T09:31:57.864289+0000","last_peered":"2024-05-27T09:31:57.864289+0000","last_clean":"2024-05-27T09:31:57.864289+0000","last_became_active":"2024-05-27T09:31:20.863465+0000","last_became_peered":"2024-05-27T09:31:20.863465+0000","last_unstale":"2024-05-27T09:31:57.864289+0000","last_undegraded":"2024-05-27T09:31:57.864289+0000","last_fullsized":"2024-05-27T09:31:57.864289+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:21:45.741822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.614368+0000","last_change":"2024-05-27T09:31:24.913749+0000","last_active":"2024-05-27T09:33:26.614368+0000","last_peered":"2024-05-27T09:33:26.614368+0000","last_clean":"2024-05-27T09:33:26.614368+0000","last_became_active":"2024-05-27T09:31:24.913384+0000","last_became_peered":"2024-05-27T09:31:24.913384+0000","last_unstale":"2024-05-27T09:33:26.614368+0000","last_undegraded":"2024-05-27T09:33:26.614368+0000","last_fullsized":"2024-05-27T09:33:26.614368+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:24:29.826096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197246+0000","last_change":"2024-05-27T09:31:20.865844+0000","last_active":"2024-05-27T09:31:58.197246+0000","last_peered":"2024-05-27T09:31:58.197246+0000","last_clean":"2024-05-27T09:31:58.197246+0000","last_became_active":"2024-05-27T09:31:20.865364+0000","last_became_peered":"2024-05-27T09:31:20.865364+0000","last_unstale":"2024-05-27T09:31:58.197246+0000","last_undegraded":"2024-05-27T09:31:58.197246+0000","last_fullsized":"2024-05-27T09:31:58.197246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:11:54.451294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618565+0000","last_change":"2024-05-27T09:31:24.890326+0000","last_active":"2024-05-27T09:33:26.618565+0000","last_peered":"2024-05-27T09:33:26.618565+0000","last_clean":"2024-05-27T09:33:26.618565+0000","last_became_active":"2024-05-27T09:31:24.889789+0000","last_became_peered":"2024-05-27T09:31:24.889789+0000","last_unstale":"2024-05-27T09:33:26.618565+0000","last_undegraded":"2024-05-27T09:33:26.618565+0000","last_fullsized":"2024-05-27T09:33:26.618565+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:34:04.252231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.608200+0000","last_change":"2024-05-27T09:31:24.896730+0000","last_active":"2024-05-27T09:33:26.608200+0000","last_peered":"2024-05-27T09:33:26.608200+0000","last_clean":"2024-05-27T09:33:26.608200+0000","last_became_active":"2024-05-27T09:31:24.896365+0000","last_became_peered":"2024-05-27T09:31:24.896365+0000","last_unstale":"2024-05-27T09:33:26.608200+0000","last_undegraded":"2024-05-27T09:33:26.608200+0000","last_fullsized":"2024-05-27T09:33:26.608200+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:37.443788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.611773+0000","last_change":"2024-05-27T09:31:24.899215+0000","last_active":"2024-05-27T09:33:26.611773+0000","last_peered":"2024-05-27T09:33:26.611773+0000","last_clean":"2024-05-27T09:33:26.611773+0000","last_became_active":"2024-05-27T09:31:24.898733+0000","last_became_peered":"2024-05-27T09:31:24.898733+0000","last_unstale":"2024-05-27T09:33:26.611773+0000","last_undegraded":"2024-05-27T09:33:26.611773+0000","last_fullsized":"2024-05-27T09:33:26.611773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:10.085989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.610776+0000","last_change":"2024-05-27T09:31:24.914922+0000","last_active":"2024-05-27T09:33:26.610776+0000","last_peered":"2024-05-27T09:33:26.610776+0000","last_clean":"2024-05-27T09:33:26.610776+0000","last_became_active":"2024-05-27T09:31:24.914541+0000","last_became_peered":"2024-05-27T09:31:24.914541+0000","last_unstale":"2024-05-27T09:33:26.610776+0000","last_undegraded":"2024-05-27T09:33:26.610776+0000","last_fullsized":"2024-05-27T09:33:26.610776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:30:38.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.632917+0000","last_change":"2024-05-27T09:31:24.909628+0000","last_active":"2024-05-27T09:33:26.632917+0000","last_peered":"2024-05-27T09:33:26.632917+0000","last_clean":"2024-05-27T09:33:26.632917+0000","last_became_active":"2024-05-27T09:31:24.902931+0000","last_became_peered":"2024-05-27T09:31:24.902931+0000","last_unstale":"2024-05-27T09:33:26.632917+0000","last_undegraded":"2024-05-27T09:33:26.632917+0000","last_fullsized":"2024-05-27T09:33:26.632917+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:06.711316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625784+0000","last_change":"2024-05-27T09:31:24.912099+0000","last_active":"2024-05-27T09:33:26.625784+0000","last_peered":"2024-05-27T09:33:26.625784+0000","last_clean":"2024-05-27T09:33:26.625784+0000","last_became_active":"2024-05-27T09:31:24.911669+0000","last_became_peered":"2024-05-27T09:31:24.911669+0000","last_unstale":"2024-05-27T09:33:26.625784+0000","last_undegraded":"2024-05-27T09:33:26.625784+0000","last_fullsized":"2024-05-27T09:33:26.625784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:58.068589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.607629+0000","last_change":"2024-05-27T09:31:24.891426+0000","last_active":"2024-05-27T09:33:26.607629+0000","last_peered":"2024-05-27T09:33:26.607629+0000","last_clean":"2024-05-27T09:33:26.607629+0000","last_became_active":"2024-05-27T09:31:24.890877+0000","last_became_peered":"2024-05-27T09:31:24.890877+0000","last_unstale":"2024-05-27T09:33:26.607629+0000","last_undegraded":"2024-05-27T09:33:26.607629+0000","last_fullsized":"2024-05-27T09:33:26.607629+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:15.338188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.604733+0000","last_change":"2024-05-27T09:31:24.892021+0000","last_active":"2024-05-27T09:33:26.604733+0000","last_peered":"2024-05-27T09:33:26.604733+0000","last_clean":"2024-05-27T09:33:26.604733+0000","last_became_active":"2024-05-27T09:31:24.891607+0000","last_became_peered":"2024-05-27T09:31:24.891607+0000","last_unstale":"2024-05-27T09:33:26.604733+0000","last_undegraded":"2024-05-27T09:33:26.604733+0000","last_fullsized":"2024-05-27T09:33:26.604733+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:05.623199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.612899+0000","last_change":"2024-05-27T09:31:24.912166+0000","last_active":"2024-05-27T09:33:26.612899+0000","last_peered":"2024-05-27T09:33:26.612899+0000","last_clean":"2024-05-27T09:33:26.612899+0000","last_became_active":"2024-05-27T09:31:24.907555+0000","last_became_peered":"2024-05-27T09:31:24.907555+0000","last_unstale":"2024-05-27T09:33:26.612899+0000","last_undegraded":"2024-05-27T09:33:26.612899+0000","last_fullsized":"2024-05-27T09:33:26.612899+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:55:33.772321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.630141+0000","last_change":"2024-05-27T09:31:24.910525+0000","last_active":"2024-05-27T09:33:26.630141+0000","last_peered":"2024-05-27T09:33:26.630141+0000","last_clean":"2024-05-27T09:33:26.630141+0000","last_became_active":"2024-05-27T09:31:24.910089+0000","last_became_peered":"2024-05-27T09:31:24.910089+0000","last_unstale":"2024-05-27T09:33:26.630141+0000","last_undegraded":"2024-05-27T09:33:26.630141+0000","last_fullsized":"2024-05-27T09:33:26.630141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:19:02.274781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.613682+0000","last_change":"2024-05-27T09:31:24.901125+0000","last_active":"2024-05-27T09:33:26.613682+0000","last_peered":"2024-05-27T09:33:26.613682+0000","last_clean":"2024-05-27T09:33:26.613682+0000","last_became_active":"2024-05-27T09:31:24.900361+0000","last_became_peered":"2024-05-27T09:31:24.900361+0000","last_unstale":"2024-05-27T09:33:26.613682+0000","last_undegraded":"2024-05-27T09:33:26.613682+0000","last_fullsized":"2024-05-27T09:33:26.613682+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:30.116349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.622231+0000","last_change":"2024-05-27T09:31:24.917612+0000","last_active":"2024-05-27T09:33:26.622231+0000","last_peered":"2024-05-27T09:33:26.622231+0000","last_clean":"2024-05-27T09:33:26.622231+0000","last_became_active":"2024-05-27T09:31:24.917210+0000","last_became_peered":"2024-05-27T09:31:24.917210+0000","last_unstale":"2024-05-27T09:33:26.622231+0000","last_undegraded":"2024-05-27T09:33:26.622231+0000","last_fullsized":"2024-05-27T09:33:26.622231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:19:00.769114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.633486+0000","last_change":"2024-05-27T09:31:24.909654+0000","last_active":"2024-05-27T09:33:26.633486+0000","last_peered":"2024-05-27T09:33:26.633486+0000","last_clean":"2024-05-27T09:33:26.633486+0000","last_became_active":"2024-05-27T09:31:24.909143+0000","last_became_peered":"2024-05-27T09:31:24.909143+0000","last_unstale":"2024-05-27T09:33:26.633486+0000","last_undegraded":"2024-05-27T09:33:26.633486+0000","last_fullsized":"2024-05-27T09:33:26.633486+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:42.844614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.615000+0000","last_change":"2024-05-27T09:31:24.899031+0000","last_active":"2024-05-27T09:33:26.615000+0000","last_peered":"2024-05-27T09:33:26.615000+0000","last_clean":"2024-05-27T09:33:26.615000+0000","last_became_active":"2024-05-27T09:31:24.898664+0000","last_became_peered":"2024-05-27T09:31:24.898664+0000","last_unstale":"2024-05-27T09:33:26.615000+0000","last_undegraded":"2024-05-27T09:33:26.615000+0000","last_fullsized":"2024-05-27T09:33:26.615000+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:09.659646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65270,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":685,"num_read_kb":409,"num_write":173,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678061,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36192,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708960,"statfs":{"total":95995035648,"available":95957975040,"internally_reserved":0,"allocated":876544,"data_stored":431801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874289,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41032,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704120,"statfs":{"total":95995035648,"available":95953018880,"internally_reserved":0,"allocated":1572864,"data_stored":1084410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070519,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":432037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266749,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":431957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462979,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1486848,"data_stored":1024560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626504,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36196,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708956,"statfs":{"total":95995035648,"available":95957970944,"internally_reserved":0,"allocated":880640,"data_stored":431426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":491815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":432697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411426,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":493115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182489,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":431737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986261,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":432977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822736,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1499136,"data_stored":1023712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":61896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T09:33:41.464 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T09:33:41.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:41 smithi050 ceph-mon[36371]: from='client.24856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:33:41.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:41 smithi050 ceph-mon[36371]: pgmap v393: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-27T09:33:41.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:41 smithi189 ceph-mon[36671]: from='client.24856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:33:41.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:41 smithi189 ceph-mon[36671]: pgmap v393: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-27T09:33:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:43.999 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:43 smithi044 ceph-mon[31618]: pgmap v394: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:44.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:43 smithi050 ceph-mon[36371]: pgmap v394: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:44.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:43 smithi189 ceph-mon[36671]: pgmap v394: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:44.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.367+0000 7f98b2600700 1 -- 172.21.15.44:0/296594279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98ac1054d0 msgr2=0x7f98ac1058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:44.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.367+0000 7f98b2600700 1 --2- 172.21.15.44:0/296594279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98ac1054d0 0x7f98ac1058e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f989c0098f0 tx=0x7f989c009c00 comp rx=0 tx=0).stop 2024-05-27T09:33:44.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.367+0000 7f98b2600700 1 -- 172.21.15.44:0/296594279 shutdown_connections 2024-05-27T09:33:44.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.367+0000 7f98b2600700 1 --2- 172.21.15.44:0/296594279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98ac10e900 0x7f98ac110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:44.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.367+0000 7f98b2600700 1 --2- 172.21.15.44:0/296594279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98ac1054d0 0x7f98ac1058e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:44.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.367+0000 7f98b2600700 1 --2- 172.21.15.44:0/296594279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98ac105e20 0x7f98ac10e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:44.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.367+0000 7f98b2600700 1 -- 172.21.15.44:0/296594279 >> 172.21.15.44:0/296594279 conn(0x7f98ac0fedb0 msgr2=0x7f98ac1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:44.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.367+0000 7f98b2600700 1 -- 172.21.15.44:0/296594279 shutdown_connections 2024-05-27T09:33:44.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.368+0000 7f98b2600700 1 -- 172.21.15.44:0/296594279 wait complete. 2024-05-27T09:33:44.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.369+0000 7f98b2600700 1 Processor -- start 2024-05-27T09:33:44.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.370+0000 7f98b2600700 1 -- start start 2024-05-27T09:33:44.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.370+0000 7f98b2600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98ac1054d0 0x7f98ac1031c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:44.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.370+0000 7f98b2600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98ac105e20 0x7f98ac103700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:44.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.370+0000 7f98b2600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98ac10e900 0x7f98ac103c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:44.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.370+0000 7f98b2600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac114420 con 0x7f98ac1054d0 2024-05-27T09:33:44.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.370+0000 7f98b2600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac1142a0 con 0x7f98ac105e20 2024-05-27T09:33:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.370+0000 7f98b2600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac1145a0 con 0x7f98ac10e900 2024-05-27T09:33:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.371+0000 7f98aaa00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98ac105e20 0x7f98ac103700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.371+0000 7f98ab400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98ac1054d0 0x7f98ac1031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.371+0000 7f98aaa00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98ac105e20 0x7f98ac103700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47862/0 (socket says 172.21.15.44:47862) 2024-05-27T09:33:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.371+0000 7f98aaa00700 1 -- 172.21.15.44:0/357607715 learned_addr learned my addr 172.21.15.44:0/357607715 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.371+0000 7f98abe00700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98ac10e900 0x7f98ac103c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.371+0000 7f98aaa00700 1 -- 172.21.15.44:0/357607715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98ac10e900 msgr2=0x7f98ac103c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.371+0000 7f98aaa00700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98ac10e900 0x7f98ac103c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.371+0000 7f98aaa00700 1 -- 172.21.15.44:0/357607715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98ac1054d0 msgr2=0x7f98ac1031c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.371+0000 7f98aaa00700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98ac1054d0 0x7f98ac1031c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.371+0000 7f98aaa00700 1 -- 172.21.15.44:0/357607715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98ac1a9150 con 0x7f98ac105e20 2024-05-27T09:33:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.372+0000 7f98aaa00700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98ac105e20 0x7f98ac103700 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f989800ec10 tx=0x7f989800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.372+0000 7f98a3e00700 1 -- 172.21.15.44:0/357607715 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989800cc90 con 0x7f98ac105e20 2024-05-27T09:33:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.372+0000 7f98b2600700 1 -- 172.21.15.44:0/357607715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98ac1a9440 con 0x7f98ac105e20 2024-05-27T09:33:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.372+0000 7f98a3e00700 1 -- 172.21.15.44:0/357607715 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f989800ce30 con 0x7f98ac105e20 2024-05-27T09:33:44.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.372+0000 7f98a3e00700 1 -- 172.21.15.44:0/357607715 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9898010430 con 0x7f98ac105e20 2024-05-27T09:33:44.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.372+0000 7f98b2600700 1 -- 172.21.15.44:0/357607715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98ac1a99d0 con 0x7f98ac105e20 2024-05-27T09:33:44.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.373+0000 7f98b2600700 1 -- 172.21.15.44:0/357607715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98ac0512b0 con 0x7f98ac105e20 2024-05-27T09:33:44.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.377+0000 7f98a3e00700 1 -- 172.21.15.44:0/357607715 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f9898004000 con 0x7f98ac105e20 2024-05-27T09:33:44.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.378+0000 7f98a3e00700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f98900b2880 0x7f98900b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:44.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.378+0000 7f98a3e00700 1 -- 172.21.15.44:0/357607715 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f98980e25d0 con 0x7f98ac105e20 2024-05-27T09:33:44.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.379+0000 7f98ab400700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f98900b2880 0x7f98900b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:44.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.379+0000 7f98ab400700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f98900b2880 0x7f98900b4d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f989c009fd0 tx=0x7f989c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:44.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.380+0000 7f98a3e00700 1 -- 172.21.15.44:0/357607715 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f98980aa2a0 con 0x7f98ac105e20 2024-05-27T09:33:44.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.612+0000 7f98b2600700 1 -- 172.21.15.44:0/357607715 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f98ac063bf0 con 0x7f98900b2880 2024-05-27T09:33:44.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.621+0000 7f98a3e00700 1 -- 172.21.15.44:0/357607715 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258815 (secure 0 0 0) 0x7f989c066310 con 0x7f98900b2880 2024-05-27T09:33:44.627 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:33:44.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.629+0000 7f98b2600700 1 -- 172.21.15.44:0/357607715 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f98900b2880 msgr2=0x7f98900b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:44.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.629+0000 7f98b2600700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f98900b2880 0x7f98900b4d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f989c009fd0 tx=0x7f989c019040 comp rx=0 tx=0).stop 2024-05-27T09:33:44.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.629+0000 7f98b2600700 1 -- 172.21.15.44:0/357607715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98ac105e20 msgr2=0x7f98ac103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:44.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.630+0000 7f98b2600700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98ac105e20 0x7f98ac103700 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f989800ec10 tx=0x7f989800ef20 comp rx=0 tx=0).stop 2024-05-27T09:33:44.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.630+0000 7f98b2600700 1 -- 172.21.15.44:0/357607715 shutdown_connections 2024-05-27T09:33:44.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.630+0000 7f98b2600700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f98900b2880 0x7f98900b4d90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:44.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.630+0000 7f98b2600700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98ac10e900 0x7f98ac103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:44.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.630+0000 7f98b2600700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98ac1054d0 0x7f98ac1031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:44.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.630+0000 7f98b2600700 1 --2- 172.21.15.44:0/357607715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98ac105e20 0x7f98ac103700 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:44.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.630+0000 7f98b2600700 1 -- 172.21.15.44:0/357607715 >> 172.21.15.44:0/357607715 conn(0x7f98ac0fedb0 msgr2=0x7f98ac0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:44.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.630+0000 7f98b2600700 1 -- 172.21.15.44:0/357607715 shutdown_connections 2024-05-27T09:33:44.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:44.630+0000 7f98b2600700 1 -- 172.21.15.44:0/357607715 wait complete. 2024-05-27T09:33:44.632 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-05-27T09:33:45.504 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-05-27T09:33:42.681754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655872,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":791,"num_read_kb":493,"num_write":298,"num_write_kb":2270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":742,"ondisk_log_size":742,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382348,"kb_used_data":12396,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559476,"statfs":{"total":1151940427776,"available":1151548903424,"internally_reserved":0,"allocated":12693504,"data_stored":7142244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005582"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618115+0000","last_change":"2024-05-27T09:31:24.899109+0000","last_active":"2024-05-27T09:33:26.618115+0000","last_peered":"2024-05-27T09:33:26.618115+0000","last_clean":"2024-05-27T09:33:26.618115+0000","last_became_active":"2024-05-27T09:31:24.898783+0000","last_became_peered":"2024-05-27T09:31:24.898783+0000","last_unstale":"2024-05-27T09:33:26.618115+0000","last_undegraded":"2024-05-27T09:33:26.618115+0000","last_fullsized":"2024-05-27T09:33:26.618115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:01:12.220389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.608195+0000","last_change":"2024-05-27T09:31:24.898382+0000","last_active":"2024-05-27T09:33:26.608195+0000","last_peered":"2024-05-27T09:33:26.608195+0000","last_clean":"2024-05-27T09:33:26.608195+0000","last_became_active":"2024-05-27T09:31:24.897975+0000","last_became_peered":"2024-05-27T09:31:24.897975+0000","last_unstale":"2024-05-27T09:33:26.608195+0000","last_undegraded":"2024-05-27T09:33:26.608195+0000","last_fullsized":"2024-05-27T09:33:26.608195+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:38.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.617056+0000","last_change":"2024-05-27T09:31:24.896122+0000","last_active":"2024-05-27T09:33:26.617056+0000","last_peered":"2024-05-27T09:33:26.617056+0000","last_clean":"2024-05-27T09:33:26.617056+0000","last_became_active":"2024-05-27T09:31:24.890867+0000","last_became_peered":"2024-05-27T09:31:24.890867+0000","last_unstale":"2024-05-27T09:33:26.617056+0000","last_undegraded":"2024-05-27T09:33:26.617056+0000","last_fullsized":"2024-05-27T09:33:26.617056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:09.982177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.621020+0000","last_change":"2024-05-27T09:31:57.868900+0000","last_active":"2024-05-27T09:33:26.621020+0000","last_peered":"2024-05-27T09:33:26.621020+0000","last_clean":"2024-05-27T09:33:26.621020+0000","last_became_active":"2024-05-27T09:31:57.868418+0000","last_became_peered":"2024-05-27T09:31:57.868418+0000","last_unstale":"2024-05-27T09:33:26.621020+0000","last_undegraded":"2024-05-27T09:33:26.621020+0000","last_fullsized":"2024-05-27T09:33:26.621020+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:08.966704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.624169+0000","last_change":"2024-05-27T09:31:24.910283+0000","last_active":"2024-05-27T09:33:26.624169+0000","last_peered":"2024-05-27T09:33:26.624169+0000","last_clean":"2024-05-27T09:33:26.624169+0000","last_became_active":"2024-05-27T09:31:24.909882+0000","last_became_peered":"2024-05-27T09:31:24.909882+0000","last_unstale":"2024-05-27T09:33:26.624169+0000","last_undegraded":"2024-05-27T09:33:26.624169+0000","last_fullsized":"2024-05-27T09:33:26.624169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:43.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.630792+0000","last_change":"2024-05-27T09:31:24.900934+0000","last_active":"2024-05-27T09:33:26.630792+0000","last_peered":"2024-05-27T09:33:26.630792+0000","last_clean":"2024-05-27T09:33:26.630792+0000","last_became_active":"2024-05-27T09:31:24.900534+0000","last_became_peered":"2024-05-27T09:31:24.900534+0000","last_unstale":"2024-05-27T09:33:26.630792+0000","last_undegraded":"2024-05-27T09:33:26.630792+0000","last_fullsized":"2024-05-27T09:33:26.630792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:53.318327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.630621+0000","last_change":"2024-05-27T09:31:24.907080+0000","last_active":"2024-05-27T09:33:26.630621+0000","last_peered":"2024-05-27T09:33:26.630621+0000","last_clean":"2024-05-27T09:33:26.630621+0000","last_became_active":"2024-05-27T09:31:24.906715+0000","last_became_peered":"2024-05-27T09:31:24.906715+0000","last_unstale":"2024-05-27T09:33:26.630621+0000","last_undegraded":"2024-05-27T09:33:26.630621+0000","last_fullsized":"2024-05-27T09:33:26.630621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:36.214659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625890+0000","last_change":"2024-05-27T09:31:58.200608+0000","last_active":"2024-05-27T09:33:26.625890+0000","last_peered":"2024-05-27T09:33:26.625890+0000","last_clean":"2024-05-27T09:33:26.625890+0000","last_became_active":"2024-05-27T09:31:58.199363+0000","last_became_peered":"2024-05-27T09:31:58.199363+0000","last_unstale":"2024-05-27T09:33:26.625890+0000","last_undegraded":"2024-05-27T09:33:26.625890+0000","last_fullsized":"2024-05-27T09:33:26.625890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:30:46.495930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618727+0000","last_change":"2024-05-27T09:31:24.892630+0000","last_active":"2024-05-27T09:33:26.618727+0000","last_peered":"2024-05-27T09:33:26.618727+0000","last_clean":"2024-05-27T09:33:26.618727+0000","last_became_active":"2024-05-27T09:31:24.892176+0000","last_became_peered":"2024-05-27T09:31:24.892176+0000","last_unstale":"2024-05-27T09:33:26.618727+0000","last_undegraded":"2024-05-27T09:33:26.618727+0000","last_fullsized":"2024-05-27T09:33:26.618727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:49:20.585601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.616507+0000","last_change":"2024-05-27T09:31:24.895281+0000","last_active":"2024-05-27T09:33:26.616507+0000","last_peered":"2024-05-27T09:33:26.616507+0000","last_clean":"2024-05-27T09:33:26.616507+0000","last_became_active":"2024-05-27T09:31:24.894886+0000","last_became_peered":"2024-05-27T09:31:24.894886+0000","last_unstale":"2024-05-27T09:33:26.616507+0000","last_undegraded":"2024-05-27T09:33:26.616507+0000","last_fullsized":"2024-05-27T09:33:26.616507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:41.494434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.617116+0000","last_change":"2024-05-27T09:31:24.900167+0000","last_active":"2024-05-27T09:33:26.617116+0000","last_peered":"2024-05-27T09:33:26.617116+0000","last_clean":"2024-05-27T09:33:26.617116+0000","last_became_active":"2024-05-27T09:31:24.899750+0000","last_became_peered":"2024-05-27T09:31:24.899750+0000","last_unstale":"2024-05-27T09:33:26.617116+0000","last_undegraded":"2024-05-27T09:33:26.617116+0000","last_fullsized":"2024-05-27T09:33:26.617116+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:30.602246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.626077+0000","last_change":"2024-05-27T09:31:57.868818+0000","last_active":"2024-05-27T09:33:26.626077+0000","last_peered":"2024-05-27T09:33:26.626077+0000","last_clean":"2024-05-27T09:33:26.626077+0000","last_became_active":"2024-05-27T09:31:57.868322+0000","last_became_peered":"2024-05-27T09:31:57.868322+0000","last_unstale":"2024-05-27T09:33:26.626077+0000","last_undegraded":"2024-05-27T09:33:26.626077+0000","last_fullsized":"2024-05-27T09:33:26.626077+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:35.304850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.634376+0000","last_change":"2024-05-27T09:31:24.902537+0000","last_active":"2024-05-27T09:33:26.634376+0000","last_peered":"2024-05-27T09:33:26.634376+0000","last_clean":"2024-05-27T09:33:26.634376+0000","last_became_active":"2024-05-27T09:31:24.902132+0000","last_became_peered":"2024-05-27T09:31:24.902132+0000","last_unstale":"2024-05-27T09:33:26.634376+0000","last_undegraded":"2024-05-27T09:33:26.634376+0000","last_fullsized":"2024-05-27T09:33:26.634376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:20:50.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.601615+0000","last_change":"2024-05-27T09:31:24.899182+0000","last_active":"2024-05-27T09:33:26.601615+0000","last_peered":"2024-05-27T09:33:26.601615+0000","last_clean":"2024-05-27T09:33:26.601615+0000","last_became_active":"2024-05-27T09:31:24.898749+0000","last_became_peered":"2024-05-27T09:31:24.898749+0000","last_unstale":"2024-05-27T09:33:26.601615+0000","last_undegraded":"2024-05-27T09:33:26.601615+0000","last_fullsized":"2024-05-27T09:33:26.601615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:44:44.844357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.605363+0000","last_change":"2024-05-27T09:31:24.901115+0000","last_active":"2024-05-27T09:33:26.605363+0000","last_peered":"2024-05-27T09:33:26.605363+0000","last_clean":"2024-05-27T09:33:26.605363+0000","last_became_active":"2024-05-27T09:31:24.897011+0000","last_became_peered":"2024-05-27T09:31:24.897011+0000","last_unstale":"2024-05-27T09:33:26.605363+0000","last_undegraded":"2024-05-27T09:33:26.605363+0000","last_fullsized":"2024-05-27T09:33:26.605363+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:01.130407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.617965+0000","last_change":"2024-05-27T09:31:24.903063+0000","last_active":"2024-05-27T09:33:26.617965+0000","last_peered":"2024-05-27T09:33:26.617965+0000","last_clean":"2024-05-27T09:33:26.617965+0000","last_became_active":"2024-05-27T09:31:24.899766+0000","last_became_peered":"2024-05-27T09:31:24.899766+0000","last_unstale":"2024-05-27T09:33:26.617965+0000","last_undegraded":"2024-05-27T09:33:26.617965+0000","last_fullsized":"2024-05-27T09:33:26.617965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:45.984226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.617948+0000","last_change":"2024-05-27T09:31:24.915922+0000","last_active":"2024-05-27T09:33:26.617948+0000","last_peered":"2024-05-27T09:33:26.617948+0000","last_clean":"2024-05-27T09:33:26.617948+0000","last_became_active":"2024-05-27T09:31:24.915623+0000","last_became_peered":"2024-05-27T09:31:24.915623+0000","last_unstale":"2024-05-27T09:33:26.617948+0000","last_undegraded":"2024-05-27T09:33:26.617948+0000","last_fullsized":"2024-05-27T09:33:26.617948+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:02:01.006704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.630130+0000","last_change":"2024-05-27T09:31:24.908397+0000","last_active":"2024-05-27T09:33:26.630130+0000","last_peered":"2024-05-27T09:33:26.630130+0000","last_clean":"2024-05-27T09:33:26.630130+0000","last_became_active":"2024-05-27T09:31:24.903644+0000","last_became_peered":"2024-05-27T09:31:24.903644+0000","last_unstale":"2024-05-27T09:33:26.630130+0000","last_undegraded":"2024-05-27T09:33:26.630130+0000","last_fullsized":"2024-05-27T09:33:26.630130+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:09.281429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.619609+0000","last_change":"2024-05-27T09:31:24.912706+0000","last_active":"2024-05-27T09:33:26.619609+0000","last_peered":"2024-05-27T09:33:26.619609+0000","last_clean":"2024-05-27T09:33:26.619609+0000","last_became_active":"2024-05-27T09:31:24.912357+0000","last_became_peered":"2024-05-27T09:31:24.912357+0000","last_unstale":"2024-05-27T09:33:26.619609+0000","last_undegraded":"2024-05-27T09:33:26.619609+0000","last_fullsized":"2024-05-27T09:33:26.619609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:23.334041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864126+0000","last_change":"2024-05-27T09:31:20.881652+0000","last_active":"2024-05-27T09:31:57.864126+0000","last_peered":"2024-05-27T09:31:57.864126+0000","last_clean":"2024-05-27T09:31:57.864126+0000","last_became_active":"2024-05-27T09:31:20.880757+0000","last_became_peered":"2024-05-27T09:31:20.880757+0000","last_unstale":"2024-05-27T09:31:57.864126+0000","last_undegraded":"2024-05-27T09:31:57.864126+0000","last_fullsized":"2024-05-27T09:31:57.864126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:01.433755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.613188+0000","last_change":"2024-05-27T09:31:24.908010+0000","last_active":"2024-05-27T09:33:26.613188+0000","last_peered":"2024-05-27T09:33:26.613188+0000","last_clean":"2024-05-27T09:33:26.613188+0000","last_became_active":"2024-05-27T09:31:24.907638+0000","last_became_peered":"2024-05-27T09:31:24.907638+0000","last_unstale":"2024-05-27T09:33:26.613188+0000","last_undegraded":"2024-05-27T09:33:26.613188+0000","last_fullsized":"2024-05-27T09:33:26.613188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:02:49.207709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.195330+0000","last_change":"2024-05-27T09:31:20.860556+0000","last_active":"2024-05-27T09:31:58.195330+0000","last_peered":"2024-05-27T09:31:58.195330+0000","last_clean":"2024-05-27T09:31:58.195330+0000","last_became_active":"2024-05-27T09:31:20.860172+0000","last_became_peered":"2024-05-27T09:31:20.860172+0000","last_unstale":"2024-05-27T09:31:58.195330+0000","last_undegraded":"2024-05-27T09:31:58.195330+0000","last_fullsized":"2024-05-27T09:31:58.195330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:08:21.545681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.631188+0000","last_change":"2024-05-27T09:31:24.910111+0000","last_active":"2024-05-27T09:33:26.631188+0000","last_peered":"2024-05-27T09:33:26.631188+0000","last_clean":"2024-05-27T09:33:26.631188+0000","last_became_active":"2024-05-27T09:31:24.908985+0000","last_became_peered":"2024-05-27T09:31:24.908985+0000","last_unstale":"2024-05-27T09:33:26.631188+0000","last_undegraded":"2024-05-27T09:33:26.631188+0000","last_fullsized":"2024-05-27T09:33:26.631188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:53:29.645792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864702+0000","last_change":"2024-05-27T09:31:20.879993+0000","last_active":"2024-05-27T09:31:57.864702+0000","last_peered":"2024-05-27T09:31:57.864702+0000","last_clean":"2024-05-27T09:31:57.864702+0000","last_became_active":"2024-05-27T09:31:20.878802+0000","last_became_peered":"2024-05-27T09:31:20.878802+0000","last_unstale":"2024-05-27T09:31:57.864702+0000","last_undegraded":"2024-05-27T09:31:57.864702+0000","last_fullsized":"2024-05-27T09:31:57.864702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:15.555612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.627583+0000","last_change":"2024-05-27T09:31:24.903533+0000","last_active":"2024-05-27T09:33:26.627583+0000","last_peered":"2024-05-27T09:33:26.627583+0000","last_clean":"2024-05-27T09:33:26.627583+0000","last_became_active":"2024-05-27T09:31:24.903166+0000","last_became_peered":"2024-05-27T09:31:24.903166+0000","last_unstale":"2024-05-27T09:33:26.627583+0000","last_undegraded":"2024-05-27T09:33:26.627583+0000","last_fullsized":"2024-05-27T09:33:26.627583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:52.539470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.878845+0000","last_change":"2024-05-27T09:31:57.878426+0000","last_active":"2024-05-27T09:31:57.878845+0000","last_peered":"2024-05-27T09:31:57.878845+0000","last_clean":"2024-05-27T09:31:57.878845+0000","last_became_active":"2024-05-27T09:31:57.869146+0000","last_became_peered":"2024-05-27T09:31:57.869146+0000","last_unstale":"2024-05-27T09:31:57.878845+0000","last_undegraded":"2024-05-27T09:31:57.878845+0000","last_fullsized":"2024-05-27T09:31:57.878845+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:34:51.926092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.607700+0000","last_change":"2024-05-27T09:31:24.897426+0000","last_active":"2024-05-27T09:33:26.607700+0000","last_peered":"2024-05-27T09:33:26.607700+0000","last_clean":"2024-05-27T09:33:26.607700+0000","last_became_active":"2024-05-27T09:31:24.897080+0000","last_became_peered":"2024-05-27T09:31:24.897080+0000","last_unstale":"2024-05-27T09:33:26.607700+0000","last_undegraded":"2024-05-27T09:33:26.607700+0000","last_fullsized":"2024-05-27T09:33:26.607700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:15:42.428391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'7","reported_seq":153,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:40.970530+0000","last_change":"2024-05-27T09:31:20.846681+0000","last_active":"2024-05-27T09:33:40.970530+0000","last_peered":"2024-05-27T09:33:40.970530+0000","last_clean":"2024-05-27T09:33:40.970530+0000","last_became_active":"2024-05-27T09:31:20.846302+0000","last_became_peered":"2024-05-27T09:31:20.846302+0000","last_unstale":"2024-05-27T09:33:40.970530+0000","last_undegraded":"2024-05-27T09:33:40.970530+0000","last_fullsized":"2024-05-27T09:33:40.970530+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:19.609612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625720+0000","last_change":"2024-05-27T09:31:24.907209+0000","last_active":"2024-05-27T09:33:26.625720+0000","last_peered":"2024-05-27T09:33:26.625720+0000","last_clean":"2024-05-27T09:33:26.625720+0000","last_became_active":"2024-05-27T09:31:24.906752+0000","last_became_peered":"2024-05-27T09:31:24.906752+0000","last_unstale":"2024-05-27T09:33:26.625720+0000","last_undegraded":"2024-05-27T09:33:26.625720+0000","last_fullsized":"2024-05-27T09:33:26.625720+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:53.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.860743+0000","last_change":"2024-05-27T09:31:20.879778+0000","last_active":"2024-05-27T09:31:57.860743+0000","last_peered":"2024-05-27T09:31:57.860743+0000","last_clean":"2024-05-27T09:31:57.860743+0000","last_became_active":"2024-05-27T09:31:20.879259+0000","last_became_peered":"2024-05-27T09:31:20.879259+0000","last_unstale":"2024-05-27T09:31:57.860743+0000","last_undegraded":"2024-05-27T09:31:57.860743+0000","last_fullsized":"2024-05-27T09:31:57.860743+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:20:07.972653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618333+0000","last_change":"2024-05-27T09:31:58.195516+0000","last_active":"2024-05-27T09:33:26.618333+0000","last_peered":"2024-05-27T09:33:26.618333+0000","last_clean":"2024-05-27T09:33:26.618333+0000","last_became_active":"2024-05-27T09:31:58.195118+0000","last_became_peered":"2024-05-27T09:31:58.195118+0000","last_unstale":"2024-05-27T09:33:26.618333+0000","last_undegraded":"2024-05-27T09:33:26.618333+0000","last_fullsized":"2024-05-27T09:33:26.618333+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:11:23.547605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.196330+0000","last_change":"2024-05-27T09:31:20.863152+0000","last_active":"2024-05-27T09:31:58.196330+0000","last_peered":"2024-05-27T09:31:58.196330+0000","last_clean":"2024-05-27T09:31:58.196330+0000","last_became_active":"2024-05-27T09:31:20.862708+0000","last_became_peered":"2024-05-27T09:31:20.862708+0000","last_unstale":"2024-05-27T09:31:58.196330+0000","last_undegraded":"2024-05-27T09:31:58.196330+0000","last_fullsized":"2024-05-27T09:31:58.196330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:21.645773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625895+0000","last_change":"2024-05-27T09:31:24.893988+0000","last_active":"2024-05-27T09:33:26.625895+0000","last_peered":"2024-05-27T09:33:26.625895+0000","last_clean":"2024-05-27T09:33:26.625895+0000","last_became_active":"2024-05-27T09:31:24.893504+0000","last_became_peered":"2024-05-27T09:31:24.893504+0000","last_unstale":"2024-05-27T09:33:26.625895+0000","last_undegraded":"2024-05-27T09:33:26.625895+0000","last_fullsized":"2024-05-27T09:33:26.625895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:47.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863900+0000","last_change":"2024-05-27T09:31:20.845555+0000","last_active":"2024-05-27T09:31:57.863900+0000","last_peered":"2024-05-27T09:31:57.863900+0000","last_clean":"2024-05-27T09:31:57.863900+0000","last_became_active":"2024-05-27T09:31:20.845139+0000","last_became_peered":"2024-05-27T09:31:20.845139+0000","last_unstale":"2024-05-27T09:31:57.863900+0000","last_undegraded":"2024-05-27T09:31:57.863900+0000","last_fullsized":"2024-05-27T09:31:57.863900+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:25.917845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.626906+0000","last_change":"2024-05-27T09:31:24.916142+0000","last_active":"2024-05-27T09:33:26.626906+0000","last_peered":"2024-05-27T09:33:26.626906+0000","last_clean":"2024-05-27T09:33:26.626906+0000","last_became_active":"2024-05-27T09:31:24.915709+0000","last_became_peered":"2024-05-27T09:31:24.915709+0000","last_unstale":"2024-05-27T09:33:26.626906+0000","last_undegraded":"2024-05-27T09:33:26.626906+0000","last_fullsized":"2024-05-27T09:33:26.626906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:38.029648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":143,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:37.964734+0000","last_change":"2024-05-27T09:31:20.859240+0000","last_active":"2024-05-27T09:33:37.964734+0000","last_peered":"2024-05-27T09:33:37.964734+0000","last_clean":"2024-05-27T09:33:37.964734+0000","last_became_active":"2024-05-27T09:31:20.858798+0000","last_became_peered":"2024-05-27T09:31:20.858798+0000","last_unstale":"2024-05-27T09:33:37.964734+0000","last_undegraded":"2024-05-27T09:33:37.964734+0000","last_fullsized":"2024-05-27T09:33:37.964734+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:42.280165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.630863+0000","last_change":"2024-05-27T09:31:24.908818+0000","last_active":"2024-05-27T09:33:26.630863+0000","last_peered":"2024-05-27T09:33:26.630863+0000","last_clean":"2024-05-27T09:33:26.630863+0000","last_became_active":"2024-05-27T09:31:24.908410+0000","last_became_peered":"2024-05-27T09:31:24.908410+0000","last_unstale":"2024-05-27T09:33:26.630863+0000","last_undegraded":"2024-05-27T09:33:26.630863+0000","last_fullsized":"2024-05-27T09:33:26.630863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:05:40.326232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864270+0000","last_change":"2024-05-27T09:31:20.848632+0000","last_active":"2024-05-27T09:31:57.864270+0000","last_peered":"2024-05-27T09:31:57.864270+0000","last_clean":"2024-05-27T09:31:57.864270+0000","last_became_active":"2024-05-27T09:31:20.847558+0000","last_became_peered":"2024-05-27T09:31:20.847558+0000","last_unstale":"2024-05-27T09:31:57.864270+0000","last_undegraded":"2024-05-27T09:31:57.864270+0000","last_fullsized":"2024-05-27T09:31:57.864270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:53.930565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.604997+0000","last_change":"2024-05-27T09:31:24.901513+0000","last_active":"2024-05-27T09:33:26.604997+0000","last_peered":"2024-05-27T09:33:26.604997+0000","last_clean":"2024-05-27T09:33:26.604997+0000","last_became_active":"2024-05-27T09:31:24.895924+0000","last_became_peered":"2024-05-27T09:31:24.895924+0000","last_unstale":"2024-05-27T09:33:26.604997+0000","last_undegraded":"2024-05-27T09:33:26.604997+0000","last_fullsized":"2024-05-27T09:33:26.604997+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:27:39.939098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864463+0000","last_change":"2024-05-27T09:31:20.855195+0000","last_active":"2024-05-27T09:31:57.864463+0000","last_peered":"2024-05-27T09:31:57.864463+0000","last_clean":"2024-05-27T09:31:57.864463+0000","last_became_active":"2024-05-27T09:31:20.854910+0000","last_became_peered":"2024-05-27T09:31:20.854910+0000","last_unstale":"2024-05-27T09:31:57.864463+0000","last_undegraded":"2024-05-27T09:31:57.864463+0000","last_fullsized":"2024-05-27T09:31:57.864463+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:02.795103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.622600+0000","last_change":"2024-05-27T09:31:24.895887+0000","last_active":"2024-05-27T09:33:26.622600+0000","last_peered":"2024-05-27T09:33:26.622600+0000","last_clean":"2024-05-27T09:33:26.622600+0000","last_became_active":"2024-05-27T09:31:24.894622+0000","last_became_peered":"2024-05-27T09:31:24.894622+0000","last_unstale":"2024-05-27T09:33:26.622600+0000","last_undegraded":"2024-05-27T09:33:26.622600+0000","last_fullsized":"2024-05-27T09:33:26.622600+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:32.833946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'32","reported_seq":293,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:38.971982+0000","last_change":"2024-05-27T09:31:20.846063+0000","last_active":"2024-05-27T09:33:38.971982+0000","last_peered":"2024-05-27T09:33:38.971982+0000","last_clean":"2024-05-27T09:33:38.971982+0000","last_became_active":"2024-05-27T09:31:20.845401+0000","last_became_peered":"2024-05-27T09:31:20.845401+0000","last_unstale":"2024-05-27T09:33:38.971982+0000","last_undegraded":"2024-05-27T09:33:38.971982+0000","last_fullsized":"2024-05-27T09:33:38.971982+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:21:38.460183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60596,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":235,"num_read_kb":177,"num_write":33,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618414+0000","last_change":"2024-05-27T09:31:24.892748+0000","last_active":"2024-05-27T09:33:26.618414+0000","last_peered":"2024-05-27T09:33:26.618414+0000","last_clean":"2024-05-27T09:33:26.618414+0000","last_became_active":"2024-05-27T09:31:24.892394+0000","last_became_peered":"2024-05-27T09:31:24.892394+0000","last_unstale":"2024-05-27T09:33:26.618414+0000","last_undegraded":"2024-05-27T09:33:26.618414+0000","last_fullsized":"2024-05-27T09:33:26.618414+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:39:31.381072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.867322+0000","last_change":"2024-05-27T09:31:20.860789+0000","last_active":"2024-05-27T09:31:57.867322+0000","last_peered":"2024-05-27T09:31:57.867322+0000","last_clean":"2024-05-27T09:31:57.867322+0000","last_became_active":"2024-05-27T09:31:20.860369+0000","last_became_peered":"2024-05-27T09:31:20.860369+0000","last_unstale":"2024-05-27T09:31:57.867322+0000","last_undegraded":"2024-05-27T09:31:57.867322+0000","last_fullsized":"2024-05-27T09:31:57.867322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:05:35.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.620332+0000","last_change":"2024-05-27T09:31:24.918239+0000","last_active":"2024-05-27T09:33:26.620332+0000","last_peered":"2024-05-27T09:33:26.620332+0000","last_clean":"2024-05-27T09:33:26.620332+0000","last_became_active":"2024-05-27T09:31:24.917750+0000","last_became_peered":"2024-05-27T09:31:24.917750+0000","last_unstale":"2024-05-27T09:33:26.620332+0000","last_undegraded":"2024-05-27T09:33:26.620332+0000","last_fullsized":"2024-05-27T09:33:26.620332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:07.162994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863994+0000","last_change":"2024-05-27T09:31:20.865308+0000","last_active":"2024-05-27T09:31:57.863994+0000","last_peered":"2024-05-27T09:31:57.863994+0000","last_clean":"2024-05-27T09:31:57.863994+0000","last_became_active":"2024-05-27T09:31:20.864920+0000","last_became_peered":"2024-05-27T09:31:20.864920+0000","last_unstale":"2024-05-27T09:31:57.863994+0000","last_undegraded":"2024-05-27T09:31:57.863994+0000","last_fullsized":"2024-05-27T09:31:57.863994+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:36.773322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.632615+0000","last_change":"2024-05-27T09:31:24.902533+0000","last_active":"2024-05-27T09:33:26.632615+0000","last_peered":"2024-05-27T09:33:26.632615+0000","last_clean":"2024-05-27T09:33:26.632615+0000","last_became_active":"2024-05-27T09:31:24.902136+0000","last_became_peered":"2024-05-27T09:31:24.902136+0000","last_unstale":"2024-05-27T09:33:26.632615+0000","last_undegraded":"2024-05-27T09:33:26.632615+0000","last_fullsized":"2024-05-27T09:33:26.632615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:49:32.203577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'5","reported_seq":147,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:37.009221+0000","last_change":"2024-05-27T09:31:20.846060+0000","last_active":"2024-05-27T09:33:37.009221+0000","last_peered":"2024-05-27T09:33:37.009221+0000","last_clean":"2024-05-27T09:33:37.009221+0000","last_became_active":"2024-05-27T09:31:20.845632+0000","last_became_peered":"2024-05-27T09:31:20.845632+0000","last_unstale":"2024-05-27T09:33:37.009221+0000","last_undegraded":"2024-05-27T09:33:37.009221+0000","last_fullsized":"2024-05-27T09:33:37.009221+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:55.166579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.619066+0000","last_change":"2024-05-27T09:31:24.895866+0000","last_active":"2024-05-27T09:33:26.619066+0000","last_peered":"2024-05-27T09:33:26.619066+0000","last_clean":"2024-05-27T09:33:26.619066+0000","last_became_active":"2024-05-27T09:31:24.895413+0000","last_became_peered":"2024-05-27T09:31:24.895413+0000","last_unstale":"2024-05-27T09:33:26.619066+0000","last_undegraded":"2024-05-27T09:33:26.619066+0000","last_fullsized":"2024-05-27T09:33:26.619066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:01:30.075710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863490+0000","last_change":"2024-05-27T09:31:20.879086+0000","last_active":"2024-05-27T09:31:57.863490+0000","last_peered":"2024-05-27T09:31:57.863490+0000","last_clean":"2024-05-27T09:31:57.863490+0000","last_became_active":"2024-05-27T09:31:20.878331+0000","last_became_peered":"2024-05-27T09:31:20.878331+0000","last_unstale":"2024-05-27T09:31:57.863490+0000","last_undegraded":"2024-05-27T09:31:57.863490+0000","last_fullsized":"2024-05-27T09:31:57.863490+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:31.169812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.616897+0000","last_change":"2024-05-27T09:31:24.895069+0000","last_active":"2024-05-27T09:33:26.616897+0000","last_peered":"2024-05-27T09:33:26.616897+0000","last_clean":"2024-05-27T09:33:26.616897+0000","last_became_active":"2024-05-27T09:31:24.894626+0000","last_became_peered":"2024-05-27T09:31:24.894626+0000","last_unstale":"2024-05-27T09:33:26.616897+0000","last_undegraded":"2024-05-27T09:33:26.616897+0000","last_fullsized":"2024-05-27T09:33:26.616897+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:21:22.725268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864122+0000","last_change":"2024-05-27T09:31:20.877416+0000","last_active":"2024-05-27T09:31:57.864122+0000","last_peered":"2024-05-27T09:31:57.864122+0000","last_clean":"2024-05-27T09:31:57.864122+0000","last_became_active":"2024-05-27T09:31:20.876093+0000","last_became_peered":"2024-05-27T09:31:20.876093+0000","last_unstale":"2024-05-27T09:31:57.864122+0000","last_undegraded":"2024-05-27T09:31:57.864122+0000","last_fullsized":"2024-05-27T09:31:57.864122+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:34.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.610737+0000","last_change":"2024-05-27T09:31:24.898430+0000","last_active":"2024-05-27T09:33:26.610737+0000","last_peered":"2024-05-27T09:33:26.610737+0000","last_clean":"2024-05-27T09:33:26.610737+0000","last_became_active":"2024-05-27T09:31:24.897643+0000","last_became_peered":"2024-05-27T09:31:24.897643+0000","last_unstale":"2024-05-27T09:33:26.610737+0000","last_undegraded":"2024-05-27T09:33:26.610737+0000","last_fullsized":"2024-05-27T09:33:26.610737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:44.485561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863820+0000","last_change":"2024-05-27T09:31:20.858253+0000","last_active":"2024-05-27T09:31:57.863820+0000","last_peered":"2024-05-27T09:31:57.863820+0000","last_clean":"2024-05-27T09:31:57.863820+0000","last_became_active":"2024-05-27T09:31:20.857736+0000","last_became_peered":"2024-05-27T09:31:20.857736+0000","last_unstale":"2024-05-27T09:31:57.863820+0000","last_undegraded":"2024-05-27T09:31:57.863820+0000","last_fullsized":"2024-05-27T09:31:57.863820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:53:48.301763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625573+0000","last_change":"2024-05-27T09:31:24.912788+0000","last_active":"2024-05-27T09:33:26.625573+0000","last_peered":"2024-05-27T09:33:26.625573+0000","last_clean":"2024-05-27T09:33:26.625573+0000","last_became_active":"2024-05-27T09:31:24.912366+0000","last_became_peered":"2024-05-27T09:31:24.912366+0000","last_unstale":"2024-05-27T09:33:26.625573+0000","last_undegraded":"2024-05-27T09:33:26.625573+0000","last_fullsized":"2024-05-27T09:33:26.625573+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:55:03.397578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863905+0000","last_change":"2024-05-27T09:31:20.862188+0000","last_active":"2024-05-27T09:31:57.863905+0000","last_peered":"2024-05-27T09:31:57.863905+0000","last_clean":"2024-05-27T09:31:57.863905+0000","last_became_active":"2024-05-27T09:31:20.861791+0000","last_became_peered":"2024-05-27T09:31:20.861791+0000","last_unstale":"2024-05-27T09:31:57.863905+0000","last_undegraded":"2024-05-27T09:31:57.863905+0000","last_fullsized":"2024-05-27T09:31:57.863905+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:16:09.127223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.616200+0000","last_change":"2024-05-27T09:31:24.898460+0000","last_active":"2024-05-27T09:33:26.616200+0000","last_peered":"2024-05-27T09:33:26.616200+0000","last_clean":"2024-05-27T09:33:26.616200+0000","last_became_active":"2024-05-27T09:31:24.896787+0000","last_became_peered":"2024-05-27T09:31:24.896787+0000","last_unstale":"2024-05-27T09:33:26.616200+0000","last_undegraded":"2024-05-27T09:33:26.616200+0000","last_fullsized":"2024-05-27T09:33:26.616200+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:10.199227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.605196+0000","last_change":"2024-05-27T09:31:24.896323+0000","last_active":"2024-05-27T09:33:26.605196+0000","last_peered":"2024-05-27T09:33:26.605196+0000","last_clean":"2024-05-27T09:33:26.605196+0000","last_became_active":"2024-05-27T09:31:24.895919+0000","last_became_peered":"2024-05-27T09:31:24.895919+0000","last_unstale":"2024-05-27T09:33:26.605196+0000","last_undegraded":"2024-05-27T09:33:26.605196+0000","last_fullsized":"2024-05-27T09:33:26.605196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:04.883166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":145,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:38.010487+0000","last_change":"2024-05-27T09:31:20.862126+0000","last_active":"2024-05-27T09:33:38.010487+0000","last_peered":"2024-05-27T09:33:38.010487+0000","last_clean":"2024-05-27T09:33:38.010487+0000","last_became_active":"2024-05-27T09:31:20.858964+0000","last_became_peered":"2024-05-27T09:31:20.858964+0000","last_unstale":"2024-05-27T09:33:38.010487+0000","last_undegraded":"2024-05-27T09:33:38.010487+0000","last_fullsized":"2024-05-27T09:33:38.010487+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:51:26.838559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625163+0000","last_change":"2024-05-27T09:31:24.905549+0000","last_active":"2024-05-27T09:33:26.625163+0000","last_peered":"2024-05-27T09:33:26.625163+0000","last_clean":"2024-05-27T09:33:26.625163+0000","last_became_active":"2024-05-27T09:31:24.905212+0000","last_became_peered":"2024-05-27T09:31:24.905212+0000","last_unstale":"2024-05-27T09:33:26.625163+0000","last_undegraded":"2024-05-27T09:33:26.625163+0000","last_fullsized":"2024-05-27T09:33:26.625163+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:18.267807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865753+0000","last_change":"2024-05-27T09:31:20.848527+0000","last_active":"2024-05-27T09:31:57.865753+0000","last_peered":"2024-05-27T09:31:57.865753+0000","last_clean":"2024-05-27T09:31:57.865753+0000","last_became_active":"2024-05-27T09:31:20.847779+0000","last_became_peered":"2024-05-27T09:31:20.847779+0000","last_unstale":"2024-05-27T09:31:57.865753+0000","last_undegraded":"2024-05-27T09:31:57.865753+0000","last_fullsized":"2024-05-27T09:31:57.865753+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:15:01.131306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":102,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863868+0000","last_change":"2024-05-27T09:29:46.120380+0000","last_active":"2024-05-27T09:31:57.863868+0000","last_peered":"2024-05-27T09:31:57.863868+0000","last_clean":"2024-05-27T09:31:57.863868+0000","last_became_active":"2024-05-27T09:29:46.113658+0000","last_became_peered":"2024-05-27T09:29:46.113658+0000","last_unstale":"2024-05-27T09:31:57.863868+0000","last_undegraded":"2024-05-27T09:31:57.863868+0000","last_fullsized":"2024-05-27T09:31:57.863868+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_clean_scrub_stamp":"2024-05-27T09:25:47.244521+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:28:40.424824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.623606+0000","last_change":"2024-05-27T09:31:24.900736+0000","last_active":"2024-05-27T09:33:26.623606+0000","last_peered":"2024-05-27T09:33:26.623606+0000","last_clean":"2024-05-27T09:33:26.623606+0000","last_became_active":"2024-05-27T09:31:24.900354+0000","last_became_peered":"2024-05-27T09:31:24.900354+0000","last_unstale":"2024-05-27T09:33:26.623606+0000","last_undegraded":"2024-05-27T09:33:26.623606+0000","last_fullsized":"2024-05-27T09:33:26.623606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:39:33.171977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.874220+0000","last_change":"2024-05-27T09:31:57.873603+0000","last_active":"2024-05-27T09:31:57.874220+0000","last_peered":"2024-05-27T09:31:57.874220+0000","last_clean":"2024-05-27T09:31:57.874220+0000","last_became_active":"2024-05-27T09:31:57.868616+0000","last_became_peered":"2024-05-27T09:31:57.868616+0000","last_unstale":"2024-05-27T09:31:57.874220+0000","last_undegraded":"2024-05-27T09:31:57.874220+0000","last_fullsized":"2024-05-27T09:31:57.874220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:02:20.112667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864575+0000","last_change":"2024-05-27T09:31:20.878557+0000","last_active":"2024-05-27T09:31:57.864575+0000","last_peered":"2024-05-27T09:31:57.864575+0000","last_clean":"2024-05-27T09:31:57.864575+0000","last_became_active":"2024-05-27T09:31:20.877969+0000","last_became_peered":"2024-05-27T09:31:20.877969+0000","last_unstale":"2024-05-27T09:31:57.864575+0000","last_undegraded":"2024-05-27T09:31:57.864575+0000","last_fullsized":"2024-05-27T09:31:57.864575+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:46.859645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.631540+0000","last_change":"2024-05-27T09:31:24.907906+0000","last_active":"2024-05-27T09:33:26.631540+0000","last_peered":"2024-05-27T09:33:26.631540+0000","last_clean":"2024-05-27T09:33:26.631540+0000","last_became_active":"2024-05-27T09:31:24.907473+0000","last_became_peered":"2024-05-27T09:31:24.907473+0000","last_unstale":"2024-05-27T09:33:26.631540+0000","last_undegraded":"2024-05-27T09:33:26.631540+0000","last_fullsized":"2024-05-27T09:33:26.631540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:11:38.536919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'12","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.957464+0000","last_change":"2024-05-27T09:31:20.881250+0000","last_active":"2024-05-27T09:33:26.957464+0000","last_peered":"2024-05-27T09:33:26.957464+0000","last_clean":"2024-05-27T09:33:26.957464+0000","last_became_active":"2024-05-27T09:31:20.880550+0000","last_became_peered":"2024-05-27T09:31:20.880550+0000","last_unstale":"2024-05-27T09:33:26.957464+0000","last_undegraded":"2024-05-27T09:33:26.957464+0000","last_fullsized":"2024-05-27T09:33:26.957464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:14.783260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.615068+0000","last_change":"2024-05-27T09:31:24.894554+0000","last_active":"2024-05-27T09:33:26.615068+0000","last_peered":"2024-05-27T09:33:26.615068+0000","last_clean":"2024-05-27T09:33:26.615068+0000","last_became_active":"2024-05-27T09:31:24.893800+0000","last_became_peered":"2024-05-27T09:31:24.893800+0000","last_unstale":"2024-05-27T09:33:26.615068+0000","last_undegraded":"2024-05-27T09:33:26.615068+0000","last_fullsized":"2024-05-27T09:33:26.615068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:13.368892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.865516+0000","last_change":"2024-05-27T09:31:21.054476+0000","last_active":"2024-05-27T09:31:57.865516+0000","last_peered":"2024-05-27T09:31:57.865516+0000","last_clean":"2024-05-27T09:31:57.865516+0000","last_became_active":"2024-05-27T09:31:21.054091+0000","last_became_peered":"2024-05-27T09:31:21.054091+0000","last_unstale":"2024-05-27T09:31:57.865516+0000","last_undegraded":"2024-05-27T09:31:57.865516+0000","last_fullsized":"2024-05-27T09:31:57.865516+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:15.909230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618501+0000","last_change":"2024-05-27T09:31:24.913157+0000","last_active":"2024-05-27T09:33:26.618501+0000","last_peered":"2024-05-27T09:33:26.618501+0000","last_clean":"2024-05-27T09:33:26.618501+0000","last_became_active":"2024-05-27T09:31:24.912793+0000","last_became_peered":"2024-05-27T09:31:24.912793+0000","last_unstale":"2024-05-27T09:33:26.618501+0000","last_undegraded":"2024-05-27T09:33:26.618501+0000","last_fullsized":"2024-05-27T09:33:26.618501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:17:36.451069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.863696+0000","last_change":"2024-05-27T09:31:20.861349+0000","last_active":"2024-05-27T09:31:57.863696+0000","last_peered":"2024-05-27T09:31:57.863696+0000","last_clean":"2024-05-27T09:31:57.863696+0000","last_became_active":"2024-05-27T09:31:20.860896+0000","last_became_peered":"2024-05-27T09:31:20.860896+0000","last_unstale":"2024-05-27T09:31:57.863696+0000","last_undegraded":"2024-05-27T09:31:57.863696+0000","last_fullsized":"2024-05-27T09:31:57.863696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:00.772910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.594260+0000","last_change":"2024-05-27T09:31:24.893240+0000","last_active":"2024-05-27T09:33:26.594260+0000","last_peered":"2024-05-27T09:33:26.594260+0000","last_clean":"2024-05-27T09:33:26.594260+0000","last_became_active":"2024-05-27T09:31:24.892169+0000","last_became_peered":"2024-05-27T09:31:24.892169+0000","last_unstale":"2024-05-27T09:33:26.594260+0000","last_undegraded":"2024-05-27T09:33:26.594260+0000","last_fullsized":"2024-05-27T09:33:26.594260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:26:59.660725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.193611+0000","last_change":"2024-05-27T09:31:20.864307+0000","last_active":"2024-05-27T09:31:58.193611+0000","last_peered":"2024-05-27T09:31:58.193611+0000","last_clean":"2024-05-27T09:31:58.193611+0000","last_became_active":"2024-05-27T09:31:20.863913+0000","last_became_peered":"2024-05-27T09:31:20.863913+0000","last_unstale":"2024-05-27T09:31:58.193611+0000","last_undegraded":"2024-05-27T09:31:58.193611+0000","last_fullsized":"2024-05-27T09:31:58.193611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:58:57.656731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.611493+0000","last_change":"2024-05-27T09:31:24.905728+0000","last_active":"2024-05-27T09:33:26.611493+0000","last_peered":"2024-05-27T09:33:26.611493+0000","last_clean":"2024-05-27T09:33:26.611493+0000","last_became_active":"2024-05-27T09:31:24.905365+0000","last_became_peered":"2024-05-27T09:31:24.905365+0000","last_unstale":"2024-05-27T09:33:26.611493+0000","last_undegraded":"2024-05-27T09:33:26.611493+0000","last_fullsized":"2024-05-27T09:33:26.611493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:05.013037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.866937+0000","last_change":"2024-05-27T09:31:20.878478+0000","last_active":"2024-05-27T09:31:57.866937+0000","last_peered":"2024-05-27T09:31:57.866937+0000","last_clean":"2024-05-27T09:31:57.866937+0000","last_became_active":"2024-05-27T09:31:20.877952+0000","last_became_peered":"2024-05-27T09:31:20.877952+0000","last_unstale":"2024-05-27T09:31:57.866937+0000","last_undegraded":"2024-05-27T09:31:57.866937+0000","last_fullsized":"2024-05-27T09:31:57.866937+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:16:27.580641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.614951+0000","last_change":"2024-05-27T09:31:24.904967+0000","last_active":"2024-05-27T09:33:26.614951+0000","last_peered":"2024-05-27T09:33:26.614951+0000","last_clean":"2024-05-27T09:33:26.614951+0000","last_became_active":"2024-05-27T09:31:24.904542+0000","last_became_peered":"2024-05-27T09:31:24.904542+0000","last_unstale":"2024-05-27T09:33:26.614951+0000","last_undegraded":"2024-05-27T09:33:26.614951+0000","last_fullsized":"2024-05-27T09:33:26.614951+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:01:05.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":44,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.876204+0000","last_change":"2024-05-27T09:31:57.875713+0000","last_active":"2024-05-27T09:31:57.876204+0000","last_peered":"2024-05-27T09:31:57.876204+0000","last_clean":"2024-05-27T09:31:57.876204+0000","last_became_active":"2024-05-27T09:31:57.870244+0000","last_became_peered":"2024-05-27T09:31:57.870244+0000","last_unstale":"2024-05-27T09:31:57.876204+0000","last_undegraded":"2024-05-27T09:31:57.876204+0000","last_fullsized":"2024-05-27T09:31:57.876204+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:49.272418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.623162+0000","last_change":"2024-05-27T09:31:24.892655+0000","last_active":"2024-05-27T09:33:26.623162+0000","last_peered":"2024-05-27T09:33:26.623162+0000","last_clean":"2024-05-27T09:33:26.623162+0000","last_became_active":"2024-05-27T09:31:24.892112+0000","last_became_peered":"2024-05-27T09:31:24.892112+0000","last_unstale":"2024-05-27T09:33:26.623162+0000","last_undegraded":"2024-05-27T09:33:26.623162+0000","last_fullsized":"2024-05-27T09:33:26.623162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.604118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.876559+0000","last_change":"2024-05-27T09:31:57.875213+0000","last_active":"2024-05-27T09:31:57.876559+0000","last_peered":"2024-05-27T09:31:57.876559+0000","last_clean":"2024-05-27T09:31:57.876559+0000","last_became_active":"2024-05-27T09:31:57.869025+0000","last_became_peered":"2024-05-27T09:31:57.869025+0000","last_unstale":"2024-05-27T09:31:57.876559+0000","last_undegraded":"2024-05-27T09:31:57.876559+0000","last_fullsized":"2024-05-27T09:31:57.876559+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:51.433347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.620958+0000","last_change":"2024-05-27T09:31:24.899130+0000","last_active":"2024-05-27T09:33:26.620958+0000","last_peered":"2024-05-27T09:33:26.620958+0000","last_clean":"2024-05-27T09:33:26.620958+0000","last_became_active":"2024-05-27T09:31:24.898722+0000","last_became_peered":"2024-05-27T09:31:24.898722+0000","last_unstale":"2024-05-27T09:33:26.620958+0000","last_undegraded":"2024-05-27T09:33:26.620958+0000","last_fullsized":"2024-05-27T09:33:26.620958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:51.247275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:57.864289+0000","last_change":"2024-05-27T09:31:20.863757+0000","last_active":"2024-05-27T09:31:57.864289+0000","last_peered":"2024-05-27T09:31:57.864289+0000","last_clean":"2024-05-27T09:31:57.864289+0000","last_became_active":"2024-05-27T09:31:20.863465+0000","last_became_peered":"2024-05-27T09:31:20.863465+0000","last_unstale":"2024-05-27T09:31:57.864289+0000","last_undegraded":"2024-05-27T09:31:57.864289+0000","last_fullsized":"2024-05-27T09:31:57.864289+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:21:45.741822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.615320+0000","last_change":"2024-05-27T09:31:24.913749+0000","last_active":"2024-05-27T09:33:26.615320+0000","last_peered":"2024-05-27T09:33:26.615320+0000","last_clean":"2024-05-27T09:33:26.615320+0000","last_became_active":"2024-05-27T09:31:24.913384+0000","last_became_peered":"2024-05-27T09:31:24.913384+0000","last_unstale":"2024-05-27T09:33:26.615320+0000","last_undegraded":"2024-05-27T09:33:26.615320+0000","last_fullsized":"2024-05-27T09:33:26.615320+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:24:29.826096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:31:58.197246+0000","last_change":"2024-05-27T09:31:20.865844+0000","last_active":"2024-05-27T09:31:58.197246+0000","last_peered":"2024-05-27T09:31:58.197246+0000","last_clean":"2024-05-27T09:31:58.197246+0000","last_became_active":"2024-05-27T09:31:20.865364+0000","last_became_peered":"2024-05-27T09:31:20.865364+0000","last_unstale":"2024-05-27T09:31:58.197246+0000","last_undegraded":"2024-05-27T09:31:58.197246+0000","last_fullsized":"2024-05-27T09:31:58.197246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:11:54.451294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.618694+0000","last_change":"2024-05-27T09:31:24.890326+0000","last_active":"2024-05-27T09:33:26.618694+0000","last_peered":"2024-05-27T09:33:26.618694+0000","last_clean":"2024-05-27T09:33:26.618694+0000","last_became_active":"2024-05-27T09:31:24.889789+0000","last_became_peered":"2024-05-27T09:31:24.889789+0000","last_unstale":"2024-05-27T09:33:26.618694+0000","last_undegraded":"2024-05-27T09:33:26.618694+0000","last_fullsized":"2024-05-27T09:33:26.618694+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:34:04.252231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.608200+0000","last_change":"2024-05-27T09:31:24.896730+0000","last_active":"2024-05-27T09:33:26.608200+0000","last_peered":"2024-05-27T09:33:26.608200+0000","last_clean":"2024-05-27T09:33:26.608200+0000","last_became_active":"2024-05-27T09:31:24.896365+0000","last_became_peered":"2024-05-27T09:31:24.896365+0000","last_unstale":"2024-05-27T09:33:26.608200+0000","last_undegraded":"2024-05-27T09:33:26.608200+0000","last_fullsized":"2024-05-27T09:33:26.608200+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:37.443788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.611773+0000","last_change":"2024-05-27T09:31:24.899215+0000","last_active":"2024-05-27T09:33:26.611773+0000","last_peered":"2024-05-27T09:33:26.611773+0000","last_clean":"2024-05-27T09:33:26.611773+0000","last_became_active":"2024-05-27T09:31:24.898733+0000","last_became_peered":"2024-05-27T09:31:24.898733+0000","last_unstale":"2024-05-27T09:33:26.611773+0000","last_undegraded":"2024-05-27T09:33:26.611773+0000","last_fullsized":"2024-05-27T09:33:26.611773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:10.085989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.610776+0000","last_change":"2024-05-27T09:31:24.914922+0000","last_active":"2024-05-27T09:33:26.610776+0000","last_peered":"2024-05-27T09:33:26.610776+0000","last_clean":"2024-05-27T09:33:26.610776+0000","last_became_active":"2024-05-27T09:31:24.914541+0000","last_became_peered":"2024-05-27T09:31:24.914541+0000","last_unstale":"2024-05-27T09:33:26.610776+0000","last_undegraded":"2024-05-27T09:33:26.610776+0000","last_fullsized":"2024-05-27T09:33:26.610776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:30:38.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.632917+0000","last_change":"2024-05-27T09:31:24.909628+0000","last_active":"2024-05-27T09:33:26.632917+0000","last_peered":"2024-05-27T09:33:26.632917+0000","last_clean":"2024-05-27T09:33:26.632917+0000","last_became_active":"2024-05-27T09:31:24.902931+0000","last_became_peered":"2024-05-27T09:31:24.902931+0000","last_unstale":"2024-05-27T09:33:26.632917+0000","last_undegraded":"2024-05-27T09:33:26.632917+0000","last_fullsized":"2024-05-27T09:33:26.632917+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:06.711316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.625784+0000","last_change":"2024-05-27T09:31:24.912099+0000","last_active":"2024-05-27T09:33:26.625784+0000","last_peered":"2024-05-27T09:33:26.625784+0000","last_clean":"2024-05-27T09:33:26.625784+0000","last_became_active":"2024-05-27T09:31:24.911669+0000","last_became_peered":"2024-05-27T09:31:24.911669+0000","last_unstale":"2024-05-27T09:33:26.625784+0000","last_undegraded":"2024-05-27T09:33:26.625784+0000","last_fullsized":"2024-05-27T09:33:26.625784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:58.068589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.607629+0000","last_change":"2024-05-27T09:31:24.891426+0000","last_active":"2024-05-27T09:33:26.607629+0000","last_peered":"2024-05-27T09:33:26.607629+0000","last_clean":"2024-05-27T09:33:26.607629+0000","last_became_active":"2024-05-27T09:31:24.890877+0000","last_became_peered":"2024-05-27T09:31:24.890877+0000","last_unstale":"2024-05-27T09:33:26.607629+0000","last_undegraded":"2024-05-27T09:33:26.607629+0000","last_fullsized":"2024-05-27T09:33:26.607629+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:15.338188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.604733+0000","last_change":"2024-05-27T09:31:24.892021+0000","last_active":"2024-05-27T09:33:26.604733+0000","last_peered":"2024-05-27T09:33:26.604733+0000","last_clean":"2024-05-27T09:33:26.604733+0000","last_became_active":"2024-05-27T09:31:24.891607+0000","last_became_peered":"2024-05-27T09:31:24.891607+0000","last_unstale":"2024-05-27T09:33:26.604733+0000","last_undegraded":"2024-05-27T09:33:26.604733+0000","last_fullsized":"2024-05-27T09:33:26.604733+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:05.623199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.612899+0000","last_change":"2024-05-27T09:31:24.912166+0000","last_active":"2024-05-27T09:33:26.612899+0000","last_peered":"2024-05-27T09:33:26.612899+0000","last_clean":"2024-05-27T09:33:26.612899+0000","last_became_active":"2024-05-27T09:31:24.907555+0000","last_became_peered":"2024-05-27T09:31:24.907555+0000","last_unstale":"2024-05-27T09:33:26.612899+0000","last_undegraded":"2024-05-27T09:33:26.612899+0000","last_fullsized":"2024-05-27T09:33:26.612899+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:55:33.772321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.630141+0000","last_change":"2024-05-27T09:31:24.910525+0000","last_active":"2024-05-27T09:33:26.630141+0000","last_peered":"2024-05-27T09:33:26.630141+0000","last_clean":"2024-05-27T09:33:26.630141+0000","last_became_active":"2024-05-27T09:31:24.910089+0000","last_became_peered":"2024-05-27T09:31:24.910089+0000","last_unstale":"2024-05-27T09:33:26.630141+0000","last_undegraded":"2024-05-27T09:33:26.630141+0000","last_fullsized":"2024-05-27T09:33:26.630141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:19:02.274781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.613929+0000","last_change":"2024-05-27T09:31:24.901125+0000","last_active":"2024-05-27T09:33:26.613929+0000","last_peered":"2024-05-27T09:33:26.613929+0000","last_clean":"2024-05-27T09:33:26.613929+0000","last_became_active":"2024-05-27T09:31:24.900361+0000","last_became_peered":"2024-05-27T09:31:24.900361+0000","last_unstale":"2024-05-27T09:33:26.613929+0000","last_undegraded":"2024-05-27T09:33:26.613929+0000","last_fullsized":"2024-05-27T09:33:26.613929+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:30.116349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.627741+0000","last_change":"2024-05-27T09:31:24.917612+0000","last_active":"2024-05-27T09:33:26.627741+0000","last_peered":"2024-05-27T09:33:26.627741+0000","last_clean":"2024-05-27T09:33:26.627741+0000","last_became_active":"2024-05-27T09:31:24.917210+0000","last_became_peered":"2024-05-27T09:31:24.917210+0000","last_unstale":"2024-05-27T09:33:26.627741+0000","last_undegraded":"2024-05-27T09:33:26.627741+0000","last_fullsized":"2024-05-27T09:33:26.627741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:19:00.769114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.634237+0000","last_change":"2024-05-27T09:31:24.909654+0000","last_active":"2024-05-27T09:33:26.634237+0000","last_peered":"2024-05-27T09:33:26.634237+0000","last_clean":"2024-05-27T09:33:26.634237+0000","last_became_active":"2024-05-27T09:31:24.909143+0000","last_became_peered":"2024-05-27T09:31:24.909143+0000","last_unstale":"2024-05-27T09:33:26.634237+0000","last_undegraded":"2024-05-27T09:33:26.634237+0000","last_fullsized":"2024-05-27T09:33:26.634237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:42.844614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-27T09:33:26.619344+0000","last_change":"2024-05-27T09:31:24.899031+0000","last_active":"2024-05-27T09:33:26.619344+0000","last_peered":"2024-05-27T09:33:26.619344+0000","last_clean":"2024-05-27T09:33:26.619344+0000","last_became_active":"2024-05-27T09:31:24.898664+0000","last_became_peered":"2024-05-27T09:31:24.898664+0000","last_unstale":"2024-05-27T09:33:26.619344+0000","last_undegraded":"2024-05-27T09:33:26.619344+0000","last_fullsized":"2024-05-27T09:33:26.619344+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:09.659646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65270,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":710,"num_read_kb":424,"num_write":173,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678062,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36192,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708960,"statfs":{"total":95995035648,"available":95957975040,"internally_reserved":0,"allocated":876544,"data_stored":431801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874290,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41032,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704120,"statfs":{"total":95995035648,"available":95953018880,"internally_reserved":0,"allocated":1572864,"data_stored":1084410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070520,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":432037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266750,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":431957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462980,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1486848,"data_stored":1024560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626505,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36196,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708956,"statfs":{"total":95995035648,"available":95957970944,"internally_reserved":0,"allocated":880640,"data_stored":431426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":491815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":432697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411427,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":493115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182490,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":431737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986261,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":432977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822737,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1499136,"data_stored":1023712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":61896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T09:33:45.508 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-27T09:33:45.509 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-27T09:33:45.509 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-27T09:33:45.509 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-05-27T09:33:46.021 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:46.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:45 smithi050 ceph-mon[36371]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:33:46.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:45 smithi050 ceph-mon[36371]: pgmap v395: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:46.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:45 smithi189 ceph-mon[36671]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:33:46.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:45 smithi189 ceph-mon[36671]: pgmap v395: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:46.169 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:45 smithi044 ceph-mon[31618]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:33:46.169 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:45 smithi044 ceph-mon[31618]: pgmap v395: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:47.855 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:47 smithi044 ceph-mon[31618]: pgmap v396: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:48.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:47 smithi050 ceph-mon[36371]: pgmap v396: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:48.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:47 smithi189 ceph-mon[36671]: pgmap v396: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:48.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.090+0000 7f193ec00700 1 -- 172.21.15.44:0/3356212616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1938076670 msgr2=0x7f1938110460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:48.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.090+0000 7f193ec00700 1 --2- 172.21.15.44:0/3356212616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1938076670 0x7f1938110460 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1924009990 tx=0x7f1924009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:48.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.090+0000 7f193ec00700 1 -- 172.21.15.44:0/3356212616 shutdown_connections 2024-05-27T09:33:48.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.090+0000 7f193ec00700 1 --2- 172.21.15.44:0/3356212616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19381109a0 0x7f1938112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:48.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.090+0000 7f193ec00700 1 --2- 172.21.15.44:0/3356212616 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1938075d20 0x7f1938076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:48.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.090+0000 7f193ec00700 1 --2- 172.21.15.44:0/3356212616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1938076670 0x7f1938110460 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:48.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.090+0000 7f193ec00700 1 -- 172.21.15.44:0/3356212616 >> 172.21.15.44:0/3356212616 conn(0x7f1938079d00 msgr2=0x7f193807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:48.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.090+0000 7f193ec00700 1 -- 172.21.15.44:0/3356212616 shutdown_connections 2024-05-27T09:33:48.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.090+0000 7f193ec00700 1 -- 172.21.15.44:0/3356212616 wait complete. 2024-05-27T09:33:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.091+0000 7f193ec00700 1 Processor -- start 2024-05-27T09:33:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.091+0000 7f193ec00700 1 -- start start 2024-05-27T09:33:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.092+0000 7f193ec00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1938075d20 0x7f193819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.092+0000 7f193ec00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1938076670 0x7f193819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.092+0000 7f193ec00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19381109a0 0x7f19381a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.092+0000 7f193ec00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f19381164f0 con 0x7f1938076670 2024-05-27T09:33:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.092+0000 7f193ec00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1938116370 con 0x7f1938075d20 2024-05-27T09:33:48.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.092+0000 7f193ec00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1938116670 con 0x7f19381109a0 2024-05-27T09:33:48.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.092+0000 7f193ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1938076670 0x7f193819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:48.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.092+0000 7f193d800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1938075d20 0x7f193819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:48.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.092+0000 7f193ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1938076670 0x7f193819d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37566/0 (socket says 172.21.15.44:37566) 2024-05-27T09:33:48.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.092+0000 7f193ce00700 1 -- 172.21.15.44:0/150138605 learned_addr learned my addr 172.21.15.44:0/150138605 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:48.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.092+0000 7f193e200700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19381109a0 0x7f19381a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:48.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.093+0000 7f193d800700 1 -- 172.21.15.44:0/150138605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19381109a0 msgr2=0x7f19381a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:48.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.093+0000 7f193d800700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19381109a0 0x7f19381a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:48.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.093+0000 7f193d800700 1 -- 172.21.15.44:0/150138605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1938076670 msgr2=0x7f193819d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.093+0000 7f193d800700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1938076670 0x7f193819d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.093+0000 7f193d800700 1 -- 172.21.15.44:0/150138605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19381a1d70 con 0x7f1938075d20 2024-05-27T09:33:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.093+0000 7f193e200700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19381109a0 0x7f19381a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:33:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.093+0000 7f193ce00700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1938076670 0x7f193819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:33:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.093+0000 7f193d800700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1938075d20 0x7f193819cea0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f192c00e9f0 tx=0x7f192c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.093+0000 7f1936000700 1 -- 172.21.15.44:0/150138605 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c00c9c0 con 0x7f1938075d20 2024-05-27T09:33:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.093+0000 7f1936000700 1 -- 172.21.15.44:0/150138605 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f192c00cb60 con 0x7f1938075d20 2024-05-27T09:33:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.093+0000 7f1936000700 1 -- 172.21.15.44:0/150138605 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c018680 con 0x7f1938075d20 2024-05-27T09:33:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.094+0000 7f193ec00700 1 -- 172.21.15.44:0/150138605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19381a2060 con 0x7f1938075d20 2024-05-27T09:33:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.094+0000 7f193ec00700 1 -- 172.21.15.44:0/150138605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19381a9980 con 0x7f1938075d20 2024-05-27T09:33:48.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.095+0000 7f193ec00700 1 -- 172.21.15.44:0/150138605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1938077290 con 0x7f1938075d20 2024-05-27T09:33:48.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.097+0000 7f1936000700 1 -- 172.21.15.44:0/150138605 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f192c010430 con 0x7f1938075d20 2024-05-27T09:33:48.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.099+0000 7f1936000700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f19200b2900 0x7f19200b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:48.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.099+0000 7f1936000700 1 -- 172.21.15.44:0/150138605 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f192c0e26b0 con 0x7f1938075d20 2024-05-27T09:33:48.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.099+0000 7f193ce00700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f19200b2900 0x7f19200b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:48.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.100+0000 7f193ce00700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f19200b2900 0x7f19200b4e10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f193807a700 tx=0x7f192400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:48.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.101+0000 7f1936000700 1 -- 172.21.15.44:0/150138605 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f192c0aa380 con 0x7f1938075d20 2024-05-27T09:33:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.393+0000 7f193ec00700 1 -- 172.21.15.44:0/150138605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f193819e230 con 0x7f1938075d20 2024-05-27T09:33:48.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.394+0000 7f1936000700 1 -- 172.21.15.44:0/150138605 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f193819e230 con 0x7f1938075d20 2024-05-27T09:33:48.395 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:33:48.395 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-27T09:33:48.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 -- 172.21.15.44:0/150138605 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f19200b2900 msgr2=0x7f19200b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:48.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f19200b2900 0x7f19200b4e10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f193807a700 tx=0x7f192400b280 comp rx=0 tx=0).stop 2024-05-27T09:33:48.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 -- 172.21.15.44:0/150138605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1938075d20 msgr2=0x7f193819cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:48.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1938075d20 0x7f193819cea0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f192c00e9f0 tx=0x7f192c00ed00 comp rx=0 tx=0).stop 2024-05-27T09:33:48.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 -- 172.21.15.44:0/150138605 shutdown_connections 2024-05-27T09:33:48.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f19200b2900 0x7f19200b4e10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:48.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19381109a0 0x7f19381a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:48.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1938075d20 0x7f193819cea0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:48.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 --2- 172.21.15.44:0/150138605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1938076670 0x7f193819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:48.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 -- 172.21.15.44:0/150138605 >> 172.21.15.44:0/150138605 conn(0x7f1938079d00 msgr2=0x7f1938100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:48.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 -- 172.21.15.44:0/150138605 shutdown_connections 2024-05-27T09:33:48.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:48.398+0000 7f193ec00700 1 -- 172.21.15.44:0/150138605 wait complete. 2024-05-27T09:33:49.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:48 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/150138605' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T09:33:49.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:48 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/150138605' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T09:33:49.159 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:48 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/150138605' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T09:33:49.221 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-27T09:33:49.221 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:33:49.708 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:50.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:49 smithi044 ceph-mon[31618]: pgmap v397: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:50.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:49 smithi050 ceph-mon[36371]: pgmap v397: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:50.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:49 smithi189 ceph-mon[36671]: pgmap v397: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:51.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.810+0000 7fa9c8800700 1 -- 172.21.15.44:0/1358692421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9c010b2d0 msgr2=0x7fa9c0117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:51.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.810+0000 7fa9c8800700 1 --2- 172.21.15.44:0/1358692421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9c010b2d0 0x7fa9c0117800 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00b0a0 tx=0x7fa9bc00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:33:51.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.810+0000 7fa9c8800700 1 -- 172.21.15.44:0/1358692421 shutdown_connections 2024-05-27T09:33:51.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.810+0000 7fa9c8800700 1 --2- 172.21.15.44:0/1358692421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9c010b2d0 0x7fa9c0117800 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:51.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.810+0000 7fa9c8800700 1 --2- 172.21.15.44:0/1358692421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c0110960 0x7fa9c0110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:51.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.810+0000 7fa9c8800700 1 --2- 172.21.15.44:0/1358692421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c010a980 0x7fa9c010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:51.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.810+0000 7fa9c8800700 1 -- 172.21.15.44:0/1358692421 >> 172.21.15.44:0/1358692421 conn(0x7fa9c0077c70 msgr2=0x7fa9c00780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:51.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.810+0000 7fa9c8800700 1 -- 172.21.15.44:0/1358692421 shutdown_connections 2024-05-27T09:33:51.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.810+0000 7fa9c8800700 1 -- 172.21.15.44:0/1358692421 wait complete. 2024-05-27T09:33:51.812 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:51 smithi044 ceph-mon[31618]: pgmap v398: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:33:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.811+0000 7fa9c8800700 1 Processor -- start 2024-05-27T09:33:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.811+0000 7fa9c8800700 1 -- start start 2024-05-27T09:33:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.812+0000 7fa9c8800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c010a980 0x7fa9c01a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.812+0000 7fa9c8800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c010b2d0 0x7fa9c01a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.812+0000 7fa9c8800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9c0110960 0x7fa9c01a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:51.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.812+0000 7fa9c8800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c011aab0 con 0x7fa9c0110960 2024-05-27T09:33:51.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.812+0000 7fa9c8800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c011a930 con 0x7fa9c010b2d0 2024-05-27T09:33:51.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.812+0000 7fa9c8800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c011ac30 con 0x7fa9c010a980 2024-05-27T09:33:51.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.812+0000 7fa9c5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c010a980 0x7fa9c01a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:51.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.812+0000 7fa9c5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c010a980 0x7fa9c01a13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34548/0 (socket says 172.21.15.44:34548) 2024-05-27T09:33:51.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.812+0000 7fa9c5e00700 1 -- 172.21.15.44:0/573031894 learned_addr learned my addr 172.21.15.44:0/573031894 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:51.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.812+0000 7fa9c6800700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9c0110960 0x7fa9c01a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:51.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.812+0000 7fa9c5400700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c010b2d0 0x7fa9c01a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:51.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.813+0000 7fa9c5e00700 1 -- 172.21.15.44:0/573031894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c010b2d0 msgr2=0x7fa9c01a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:51.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.813+0000 7fa9c5e00700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c010b2d0 0x7fa9c01a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.813+0000 7fa9c5e00700 1 -- 172.21.15.44:0/573031894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9c0110960 msgr2=0x7fa9c01a5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.813+0000 7fa9c5e00700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9c0110960 0x7fa9c01a5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.813+0000 7fa9c5e00700 1 -- 172.21.15.44:0/573031894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c01a6270 con 0x7fa9c010a980 2024-05-27T09:33:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.813+0000 7fa9c5e00700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c010a980 0x7fa9c01a13a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400e970 tx=0x7fa9b400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.814+0000 7fa9baa00700 1 -- 172.21.15.44:0/573031894 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4004a30 con 0x7fa9c010a980 2024-05-27T09:33:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.814+0000 7fa9baa00700 1 -- 172.21.15.44:0/573031894 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9b400cc90 con 0x7fa9c010a980 2024-05-27T09:33:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.814+0000 7fa9c8800700 1 -- 172.21.15.44:0/573031894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c01a6560 con 0x7fa9c010a980 2024-05-27T09:33:51.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.814+0000 7fa9baa00700 1 -- 172.21.15.44:0/573031894 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4005610 con 0x7fa9c010a980 2024-05-27T09:33:51.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.814+0000 7fa9c8800700 1 -- 172.21.15.44:0/573031894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c01ade80 con 0x7fa9c010a980 2024-05-27T09:33:51.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.815+0000 7fa9c8800700 1 -- 172.21.15.44:0/573031894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9c00512b0 con 0x7fa9c010a980 2024-05-27T09:33:51.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.817+0000 7fa9baa00700 1 -- 172.21.15.44:0/573031894 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fa9b400c7e0 con 0x7fa9c010a980 2024-05-27T09:33:51.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.819+0000 7fa9baa00700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa9a80b2900 0x7fa9a80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:51.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.819+0000 7fa9baa00700 1 -- 172.21.15.44:0/573031894 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fa9b40e2480 con 0x7fa9c010a980 2024-05-27T09:33:51.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.819+0000 7fa9c5400700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa9a80b2900 0x7fa9a80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:51.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.820+0000 7fa9c5400700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa9a80b2900 0x7fa9a80b4e10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0109720 tx=0x7fa9b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:51.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:51.821+0000 7fa9baa00700 1 -- 172.21.15.44:0/573031894 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa9b40aa150 con 0x7fa9c010a980 2024-05-27T09:33:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:51 smithi050 ceph-mon[36371]: pgmap v398: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:33:52.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:51 smithi189 ceph-mon[36671]: pgmap v398: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:33:52.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.116+0000 7fa9c8800700 1 -- 172.21.15.44:0/573031894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9c01a2700 con 0x7fa9c010a980 2024-05-27T09:33:52.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.118+0000 7fa9baa00700 1 -- 172.21.15.44:0/573031894 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7fa9b40a9a50 con 0x7fa9c010a980 2024-05-27T09:33:52.119 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:33:52.119 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:31:50.423700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:33:52.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.121+0000 7fa9c8800700 1 -- 172.21.15.44:0/573031894 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa9a80b2900 msgr2=0x7fa9a80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:52.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.121+0000 7fa9c8800700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa9a80b2900 0x7fa9a80b4e10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0109720 tx=0x7fa9b0009260 comp rx=0 tx=0).stop 2024-05-27T09:33:52.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.121+0000 7fa9c8800700 1 -- 172.21.15.44:0/573031894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c010a980 msgr2=0x7fa9c01a13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:52.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.121+0000 7fa9c8800700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c010a980 0x7fa9c01a13a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400e970 tx=0x7fa9b400ec80 comp rx=0 tx=0).stop 2024-05-27T09:33:52.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.121+0000 7fa9c8800700 1 -- 172.21.15.44:0/573031894 shutdown_connections 2024-05-27T09:33:52.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.121+0000 7fa9c8800700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fa9a80b2900 0x7fa9a80b4e10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:52.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.121+0000 7fa9c8800700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9c0110960 0x7fa9c01a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:52.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.121+0000 7fa9c8800700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c010a980 0x7fa9c01a13a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:52.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.121+0000 7fa9c8800700 1 --2- 172.21.15.44:0/573031894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c010b2d0 0x7fa9c01a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:52.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.121+0000 7fa9c8800700 1 -- 172.21.15.44:0/573031894 >> 172.21.15.44:0/573031894 conn(0x7fa9c0077c70 msgr2=0x7fa9c007b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:52.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.121+0000 7fa9c8800700 1 -- 172.21.15.44:0/573031894 shutdown_connections 2024-05-27T09:33:52.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:52.122+0000 7fa9c8800700 1 -- 172.21.15.44:0/573031894 wait complete. 2024-05-27T09:33:52.124 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-05-27T09:33:52.940 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:33:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:52 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/573031894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:33:53.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:52 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/573031894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:33:53.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:52 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/573031894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:33:53.441 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:53.886 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:53 smithi044 ceph-mon[31618]: pgmap v399: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:54.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:53 smithi050 ceph-mon[36371]: pgmap v399: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:54.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:53 smithi189 ceph-mon[36671]: pgmap v399: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:55.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.752+0000 7f0b17200700 1 -- 172.21.15.44:0/1591353255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b18076670 msgr2=0x7f0b18110460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:55.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.752+0000 7f0b17200700 1 --2- 172.21.15.44:0/1591353255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b18076670 0x7f0b18110460 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0b00009990 tx=0x7f0b00009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:55.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.752+0000 7f0b17200700 1 -- 172.21.15.44:0/1591353255 shutdown_connections 2024-05-27T09:33:55.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.752+0000 7f0b17200700 1 --2- 172.21.15.44:0/1591353255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b181109a0 0x7f0b18112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:55.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.752+0000 7f0b17200700 1 --2- 172.21.15.44:0/1591353255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b18075d20 0x7f0b18076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:55.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.752+0000 7f0b17200700 1 --2- 172.21.15.44:0/1591353255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b18076670 0x7f0b18110460 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:55.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.752+0000 7f0b17200700 1 -- 172.21.15.44:0/1591353255 >> 172.21.15.44:0/1591353255 conn(0x7f0b18079d00 msgr2=0x7f0b1807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:55.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.752+0000 7f0b17200700 1 -- 172.21.15.44:0/1591353255 shutdown_connections 2024-05-27T09:33:55.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.752+0000 7f0b17200700 1 -- 172.21.15.44:0/1591353255 wait complete. 2024-05-27T09:33:55.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.754+0000 7f0b17200700 1 Processor -- start 2024-05-27T09:33:55.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.754+0000 7f0b17200700 1 -- start start 2024-05-27T09:33:55.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.754+0000 7f0b17200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b18075d20 0x7f0b1819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:55.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.754+0000 7f0b17200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b18076670 0x7f0b1819d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:55.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.754+0000 7f0b17200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b181109a0 0x7f0b181a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:55.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.754+0000 7f0b17200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b181164a0 con 0x7f0b181109a0 2024-05-27T09:33:55.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.754+0000 7f0b17200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0b18116320 con 0x7f0b18076670 2024-05-27T09:33:55.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.754+0000 7f0b17200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0b18116620 con 0x7f0b18075d20 2024-05-27T09:33:55.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.755+0000 7f0b0fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b18076670 0x7f0b1819d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:55.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.755+0000 7f0b16800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b181109a0 0x7f0b181a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:55.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.755+0000 7f0b0fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b18076670 0x7f0b1819d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47396/0 (socket says 172.21.15.44:47396) 2024-05-27T09:33:55.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.755+0000 7f0b0fe00700 1 -- 172.21.15.44:0/986079886 learned_addr learned my addr 172.21.15.44:0/986079886 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:55.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.755+0000 7f0b15e00700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b18075d20 0x7f0b1819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:55.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.755+0000 7f0b0fe00700 1 -- 172.21.15.44:0/986079886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b18075d20 msgr2=0x7f0b1819ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:55.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.755+0000 7f0b0fe00700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b18075d20 0x7f0b1819ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:55.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.755+0000 7f0b0fe00700 1 -- 172.21.15.44:0/986079886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b181109a0 msgr2=0x7f0b181a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:55.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.755+0000 7f0b0fe00700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b181109a0 0x7f0b181a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:55.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.755+0000 7f0b0fe00700 1 -- 172.21.15.44:0/986079886 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b181a1d00 con 0x7f0b18076670 2024-05-27T09:33:55.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.756+0000 7f0b0fe00700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b18076670 0x7f0b1819d3d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0b000057f0 tx=0x7f0b0000bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:55.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.756+0000 7f0b0ea00700 1 -- 172.21.15.44:0/986079886 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00026070 con 0x7f0b18076670 2024-05-27T09:33:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.756+0000 7f0b0ea00700 1 -- 172.21.15.44:0/986079886 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b00021410 con 0x7f0b18076670 2024-05-27T09:33:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.756+0000 7f0b17200700 1 -- 172.21.15.44:0/986079886 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b181a1f90 con 0x7f0b18076670 2024-05-27T09:33:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.756+0000 7f0b0ea00700 1 -- 172.21.15.44:0/986079886 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00017650 con 0x7f0b18076670 2024-05-27T09:33:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.756+0000 7f0b17200700 1 -- 172.21.15.44:0/986079886 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b181a9910 con 0x7f0b18076670 2024-05-27T09:33:55.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.758+0000 7f0b17200700 1 -- 172.21.15.44:0/986079886 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b18077290 con 0x7f0b18076670 2024-05-27T09:33:55.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.760+0000 7f0b0ea00700 1 -- 172.21.15.44:0/986079886 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f0b00029070 con 0x7f0b18076670 2024-05-27T09:33:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.762+0000 7f0b0ea00700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0afc0b2900 0x7f0afc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:55.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.762+0000 7f0b0ea00700 1 -- 172.21.15.44:0/986079886 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f0b000e3170 con 0x7f0b18076670 2024-05-27T09:33:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.762+0000 7f0b15e00700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0afc0b2900 0x7f0afc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:55.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.763+0000 7f0b15e00700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0afc0b2900 0x7f0afc0b4e10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009c60 tx=0x7f0b08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:55.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:55.764+0000 7f0b0ea00700 1 -- 172.21.15.44:0/986079886 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0b00013070 con 0x7f0b18076670 2024-05-27T09:33:56.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:55 smithi050 ceph-mon[36371]: pgmap v400: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.059+0000 7f0b17200700 1 -- 172.21.15.44:0/986079886 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b1819e190 con 0x7f0b18076670 2024-05-27T09:33:56.060 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:55 smithi044 ceph-mon[31618]: pgmap v400: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:56.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.060+0000 7f0b0ea00700 1 -- 172.21.15.44:0/986079886 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7f0b000aaed0 con 0x7f0b18076670 2024-05-27T09:33:56.061 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:33:56.061 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:31:50.423700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:33:56.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 -- 172.21.15.44:0/986079886 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0afc0b2900 msgr2=0x7f0afc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0afc0b2900 0x7f0afc0b4e10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009c60 tx=0x7f0b08009260 comp rx=0 tx=0).stop 2024-05-27T09:33:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 -- 172.21.15.44:0/986079886 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b18076670 msgr2=0x7f0b1819d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b18076670 0x7f0b1819d3d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0b000057f0 tx=0x7f0b0000bfd0 comp rx=0 tx=0).stop 2024-05-27T09:33:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 -- 172.21.15.44:0/986079886 shutdown_connections 2024-05-27T09:33:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0afc0b2900 0x7f0afc0b4e10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b181109a0 0x7f0b181a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b18075d20 0x7f0b1819ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 --2- 172.21.15.44:0/986079886 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b18076670 0x7f0b1819d3d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 -- 172.21.15.44:0/986079886 >> 172.21.15.44:0/986079886 conn(0x7f0b18079d00 msgr2=0x7f0b18100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 -- 172.21.15.44:0/986079886 shutdown_connections 2024-05-27T09:33:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:56.064+0000 7f0b17200700 1 -- 172.21.15.44:0/986079886 wait complete. 2024-05-27T09:33:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-05-27T09:33:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:55 smithi189 ceph-mon[36671]: pgmap v400: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:33:57.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:56 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/986079886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:33:57.089 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:56 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/986079886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:33:57.093 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T09:33:57.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:56 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/986079886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:33:57.552 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:33:57.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:57 smithi044 ceph-mon[31618]: pgmap v401: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:58.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:57 smithi189 ceph-mon[36671]: pgmap v401: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:57 smithi050 ceph-mon[36371]: pgmap v401: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:33:59.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.733+0000 7f2931e00700 1 -- 172.21.15.44:0/2890042154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f292c109120 msgr2=0x7f292c109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:59.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.733+0000 7f2931e00700 1 --2- 172.21.15.44:0/2890042154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f292c109120 0x7f292c109530 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f291c009990 tx=0x7f291c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:33:59.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.733+0000 7f2931e00700 1 -- 172.21.15.44:0/2890042154 shutdown_connections 2024-05-27T09:33:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.733+0000 7f2931e00700 1 --2- 172.21.15.44:0/2890042154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f292c102f00 0x7f292c110f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.733+0000 7f2931e00700 1 --2- 172.21.15.44:0/2890042154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f292c109120 0x7f292c109530 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.733+0000 7f2931e00700 1 --2- 172.21.15.44:0/2890042154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f292c1025b0 0x7f292c1029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.733+0000 7f2931e00700 1 -- 172.21.15.44:0/2890042154 >> 172.21.15.44:0/2890042154 conn(0x7f292c078320 msgr2=0x7f292c078780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.733+0000 7f2931e00700 1 -- 172.21.15.44:0/2890042154 shutdown_connections 2024-05-27T09:33:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.733+0000 7f2931e00700 1 -- 172.21.15.44:0/2890042154 wait complete. 2024-05-27T09:33:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.735+0000 7f2931e00700 1 Processor -- start 2024-05-27T09:33:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.735+0000 7f2931e00700 1 -- start start 2024-05-27T09:33:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.735+0000 7f2931e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f292c1025b0 0x7f292c19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.735+0000 7f2931e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f292c102f00 0x7f292c19d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.735+0000 7f2931e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f292c109120 0x7f292c1a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.735+0000 7f2931e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f292c114400 con 0x7f292c1025b0 2024-05-27T09:33:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.735+0000 7f2931e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f292c114280 con 0x7f292c102f00 2024-05-27T09:33:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.735+0000 7f2931e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f292c114580 con 0x7f292c109120 2024-05-27T09:33:59.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.736+0000 7f2930a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f292c1025b0 0x7f292c19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:59.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.736+0000 7f2923e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f292c102f00 0x7f292c19d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:59.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.736+0000 7f2930a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f292c1025b0 0x7f292c19cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49006/0 (socket says 172.21.15.44:49006) 2024-05-27T09:33:59.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.736+0000 7f2930a00700 1 -- 172.21.15.44:0/3524365516 learned_addr learned my addr 172.21.15.44:0/3524365516 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:33:59.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.736+0000 7f2931400700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f292c109120 0x7f292c1a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:59.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.736+0000 7f2930a00700 1 -- 172.21.15.44:0/3524365516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f292c109120 msgr2=0x7f292c1a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:59.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.736+0000 7f2930a00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f292c109120 0x7f292c1a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:59.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.736+0000 7f2930a00700 1 -- 172.21.15.44:0/3524365516 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f292c102f00 msgr2=0x7f292c19d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:59.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.736+0000 7f2930a00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f292c102f00 0x7f292c19d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:59.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.736+0000 7f2930a00700 1 -- 172.21.15.44:0/3524365516 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292c1a1e80 con 0x7f292c1025b0 2024-05-27T09:33:59.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.736+0000 7f2923e00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f292c102f00 0x7f292c19d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:33:59.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.737+0000 7f2930a00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f292c1025b0 0x7f292c19cfb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f291c0098e0 tx=0x7f291c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:59.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.737+0000 7f292a000700 1 -- 172.21.15.44:0/3524365516 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291c027070 con 0x7f292c1025b0 2024-05-27T09:33:59.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.737+0000 7f292a000700 1 -- 172.21.15.44:0/3524365516 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f291c022470 con 0x7f292c1025b0 2024-05-27T09:33:59.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.737+0000 7f2931e00700 1 -- 172.21.15.44:0/3524365516 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f292c1a2110 con 0x7f292c1025b0 2024-05-27T09:33:59.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.737+0000 7f292a000700 1 -- 172.21.15.44:0/3524365516 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291c017650 con 0x7f292c1025b0 2024-05-27T09:33:59.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.737+0000 7f2931e00700 1 -- 172.21.15.44:0/3524365516 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f292c1a25f0 con 0x7f292c1025b0 2024-05-27T09:33:59.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.738+0000 7f2931e00700 1 -- 172.21.15.44:0/3524365516 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f292c0512b0 con 0x7f292c1025b0 2024-05-27T09:33:59.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.741+0000 7f292a000700 1 -- 172.21.15.44:0/3524365516 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f291c02a070 con 0x7f292c1025b0 2024-05-27T09:33:59.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.742+0000 7f292a000700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f29100b2880 0x7f29100b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:33:59.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.743+0000 7f292a000700 1 -- 172.21.15.44:0/3524365516 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f291c0e32a0 con 0x7f292c1025b0 2024-05-27T09:33:59.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.743+0000 7f2923e00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f29100b2880 0x7f29100b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:33:59.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.744+0000 7f2923e00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f29100b2880 0x7f29100b4d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f292c0ff3c0 tx=0x7f2914009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:33:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.746+0000 7f292a000700 1 -- 172.21.15.44:0/3524365516 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f291c013070 con 0x7f292c1025b0 2024-05-27T09:33:59.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.986+0000 7f2931e00700 1 -- 172.21.15.44:0/3524365516 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f292c1a9b50 con 0x7f292c1025b0 2024-05-27T09:33:59.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:33:59 smithi044 ceph-mon[31618]: pgmap v402: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:33:59.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.988+0000 7f292a000700 1 -- 172.21.15.44:0/3524365516 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22611 (secure 0 0 0) 0x7f291c0ab000 con 0x7f292c1025b0 2024-05-27T09:33:59.990 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:33:59.990 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":81,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:31:57.855852+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:33:59.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.992+0000 7f2931e00700 1 -- 172.21.15.44:0/3524365516 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f29100b2880 msgr2=0x7f29100b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:59.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.992+0000 7f2931e00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f29100b2880 0x7f29100b4d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f292c0ff3c0 tx=0x7f2914009260 comp rx=0 tx=0).stop 2024-05-27T09:33:59.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.992+0000 7f2931e00700 1 -- 172.21.15.44:0/3524365516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f292c1025b0 msgr2=0x7f292c19cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:33:59.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.992+0000 7f2931e00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f292c1025b0 0x7f292c19cfb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f291c0098e0 tx=0x7f291c0057c0 comp rx=0 tx=0).stop 2024-05-27T09:33:59.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.992+0000 7f2931e00700 1 -- 172.21.15.44:0/3524365516 shutdown_connections 2024-05-27T09:33:59.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.992+0000 7f2931e00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f29100b2880 0x7f29100b4d90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:59.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.992+0000 7f2931e00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f292c109120 0x7f292c1a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:59.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.992+0000 7f2931e00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f292c1025b0 0x7f292c19cfb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:59.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.992+0000 7f2931e00700 1 --2- 172.21.15.44:0/3524365516 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f292c102f00 0x7f292c19d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:33:59.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.992+0000 7f2931e00700 1 -- 172.21.15.44:0/3524365516 >> 172.21.15.44:0/3524365516 conn(0x7f292c078320 msgr2=0x7f292c0ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:33:59.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.993+0000 7f2931e00700 1 -- 172.21.15.44:0/3524365516 shutdown_connections 2024-05-27T09:33:59.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:33:59.993+0000 7f2931e00700 1 -- 172.21.15.44:0/3524365516 wait complete. 2024-05-27T09:34:00.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:33:59 smithi189 ceph-mon[36671]: pgmap v402: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:33:59 smithi050 ceph-mon[36371]: pgmap v402: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:00.844 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:34:01.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:00 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3524365516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:34:01.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:00 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3524365516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:34:01.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:00 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3524365516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:34:01.313 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:02.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:01 smithi189 ceph-mon[36671]: pgmap v403: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:02.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:01 smithi044 ceph-mon[31618]: pgmap v403: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:01 smithi050 ceph-mon[36371]: pgmap v403: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:03.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.632+0000 7fb9a8800700 1 -- 172.21.15.44:0/2281070009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9a0106ee0 msgr2=0x7fb9a0113370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:03.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.632+0000 7fb9a8800700 1 --2- 172.21.15.44:0/2281070009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9a0106ee0 0x7fb9a0113370 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb99c00b0a0 tx=0x7fb99c00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:34:03.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.632+0000 7fb9a8800700 1 -- 172.21.15.44:0/2281070009 shutdown_connections 2024-05-27T09:34:03.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.632+0000 7fb9a8800700 1 --2- 172.21.15.44:0/2281070009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9a0106ee0 0x7fb9a0113370 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:03.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.632+0000 7fb9a8800700 1 --2- 172.21.15.44:0/2281070009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9a010c5d0 0x7fb9a010c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:03.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.632+0000 7fb9a8800700 1 --2- 172.21.15.44:0/2281070009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a0106590 0x7fb9a01069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:03.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.632+0000 7fb9a8800700 1 -- 172.21.15.44:0/2281070009 >> 172.21.15.44:0/2281070009 conn(0x7fb9a01020b0 msgr2=0x7fb9a0104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:03.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.632+0000 7fb9a8800700 1 -- 172.21.15.44:0/2281070009 shutdown_connections 2024-05-27T09:34:03.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.632+0000 7fb9a8800700 1 -- 172.21.15.44:0/2281070009 wait complete. 2024-05-27T09:34:03.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.633+0000 7fb9a8800700 1 Processor -- start 2024-05-27T09:34:03.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.633+0000 7fb9a8800700 1 -- start start 2024-05-27T09:34:03.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.634+0000 7fb9a8800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9a0106590 0x7fb9a019cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.634+0000 7fb9a8800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a0106ee0 0x7fb9a019d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.634+0000 7fb9a8800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9a010c5d0 0x7fb9a01a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.634+0000 7fb9a8800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a0116750 con 0x7fb9a010c5d0 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.634+0000 7fb9a8800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a01165d0 con 0x7fb9a0106590 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.634+0000 7fb9a8800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a01168d0 con 0x7fb9a0106ee0 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.634+0000 7fb9a5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a0106ee0 0x7fb9a019d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.634+0000 7fb9a5e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9a0106590 0x7fb9a019cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.634+0000 7fb9a5e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9a0106590 0x7fb9a019cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50504/0 (socket says 172.21.15.44:50504) 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.634+0000 7fb9a5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a0106ee0 0x7fb9a019d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50746/0 (socket says 172.21.15.44:50746) 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.634+0000 7fb9a5e00700 1 -- 172.21.15.44:0/1215992031 learned_addr learned my addr 172.21.15.44:0/1215992031 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.635+0000 7fb9a5e00700 1 -- 172.21.15.44:0/1215992031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a0106ee0 msgr2=0x7fb9a019d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.635+0000 7fb9a6800700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9a010c5d0 0x7fb9a01a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.635+0000 7fb9a5e00700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a0106ee0 0x7fb9a019d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:03.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.635+0000 7fb9a5e00700 1 -- 172.21.15.44:0/1215992031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9a010c5d0 msgr2=0x7fb9a01a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:03.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.635+0000 7fb9a5e00700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9a010c5d0 0x7fb9a01a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:03.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.635+0000 7fb9a5e00700 1 -- 172.21.15.44:0/1215992031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a01a1eb0 con 0x7fb9a0106590 2024-05-27T09:34:03.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.635+0000 7fb9a5e00700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9a0106590 0x7fb9a019cfe0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb99400e970 tx=0x7fb99400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:03.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.636+0000 7fb99aa00700 1 -- 172.21.15.44:0/1215992031 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99400cbd0 con 0x7fb9a0106590 2024-05-27T09:34:03.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.636+0000 7fb99aa00700 1 -- 172.21.15.44:0/1215992031 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb99400cd70 con 0x7fb9a0106590 2024-05-27T09:34:03.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.636+0000 7fb9a8800700 1 -- 172.21.15.44:0/1215992031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a01a21a0 con 0x7fb9a0106590 2024-05-27T09:34:03.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.636+0000 7fb99aa00700 1 -- 172.21.15.44:0/1215992031 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb994005680 con 0x7fb9a0106590 2024-05-27T09:34:03.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.636+0000 7fb9a8800700 1 -- 172.21.15.44:0/1215992031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a01a9ac0 con 0x7fb9a0106590 2024-05-27T09:34:03.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.637+0000 7fb9a8800700 1 -- 172.21.15.44:0/1215992031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a00512b0 con 0x7fb9a0106590 2024-05-27T09:34:03.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.640+0000 7fb99aa00700 1 -- 172.21.15.44:0/1215992031 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fb9940468d0 con 0x7fb9a0106590 2024-05-27T09:34:03.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.641+0000 7fb99aa00700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb9880b2880 0x7fb9880b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:03.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.641+0000 7fb99aa00700 1 -- 172.21.15.44:0/1215992031 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fb9940e25f0 con 0x7fb9a0106590 2024-05-27T09:34:03.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.641+0000 7fb9a5400700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb9880b2880 0x7fb9880b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:03.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.642+0000 7fb9a5400700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb9880b2880 0x7fb9880b4d90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0103bc0 tx=0x7fb99000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:03.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.644+0000 7fb99aa00700 1 -- 172.21.15.44:0/1215992031 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb9940aa2c0 con 0x7fb9a0106590 2024-05-27T09:34:03.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.929+0000 7fb9a8800700 1 -- 172.21.15.44:0/1215992031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9a019e340 con 0x7fb9a0106590 2024-05-27T09:34:03.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:03 smithi044 ceph-mon[31618]: pgmap v404: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:03.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.930+0000 7fb99aa00700 1 -- 172.21.15.44:0/1215992031 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7fb9940a9bc0 con 0x7fb9a0106590 2024-05-27T09:34:03.932 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:34:03.932 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:31:50.423700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:34:03.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.933+0000 7fb9a8800700 1 -- 172.21.15.44:0/1215992031 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb9880b2880 msgr2=0x7fb9880b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:03.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.933+0000 7fb9a8800700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb9880b2880 0x7fb9880b4d90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0103bc0 tx=0x7fb99000b410 comp rx=0 tx=0).stop 2024-05-27T09:34:03.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.933+0000 7fb9a8800700 1 -- 172.21.15.44:0/1215992031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9a0106590 msgr2=0x7fb9a019cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:03.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.933+0000 7fb9a8800700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9a0106590 0x7fb9a019cfe0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb99400e970 tx=0x7fb99400ec80 comp rx=0 tx=0).stop 2024-05-27T09:34:03.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.934+0000 7fb9a8800700 1 -- 172.21.15.44:0/1215992031 shutdown_connections 2024-05-27T09:34:03.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.934+0000 7fb9a8800700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb9880b2880 0x7fb9880b4d90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:03.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.934+0000 7fb9a8800700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9a010c5d0 0x7fb9a01a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:03.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.934+0000 7fb9a8800700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9a0106590 0x7fb9a019cfe0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:03.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.934+0000 7fb9a8800700 1 --2- 172.21.15.44:0/1215992031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a0106ee0 0x7fb9a019d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:03.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.934+0000 7fb9a8800700 1 -- 172.21.15.44:0/1215992031 >> 172.21.15.44:0/1215992031 conn(0x7fb9a01020b0 msgr2=0x7fb9a0104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:03.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.934+0000 7fb9a8800700 1 -- 172.21.15.44:0/1215992031 shutdown_connections 2024-05-27T09:34:03.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:03.934+0000 7fb9a8800700 1 -- 172.21.15.44:0/1215992031 wait complete. 2024-05-27T09:34:03.936 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-05-27T09:34:04.095 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:03 smithi050 ceph-mon[36371]: pgmap v404: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:04.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:03 smithi189 ceph-mon[36671]: pgmap v404: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:04.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-27T09:31:27.747151+0000', 'modified': '2024-05-27T09:31:50.423700+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24542, 'mds_1': 24535, 'mds_2': 24520}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24491': {'gid': 24491, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.50:6835/1092513236', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6834', 'nonce': 1092513236}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 1092513236}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.50:6837/3540954143', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6836', 'nonce': 3540954143}, {'type': 'v1', 'addr': '172.21.15.50:6837', 'nonce': 3540954143}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6835/2673083839', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 2673083839}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 2673083839}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.189:6837/2265427173', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 2265427173}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 2265427173}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.189:6839/4109192600', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 4109192600}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 4109192600}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24542': {'gid': 24542, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.50:6839/1124366265', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6838', 'nonce': 1124366265}, {'type': 'v1', 'addr': '172.21.15.50:6839', 'nonce': 1124366265}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24542, 'qdb_cluster': [24520, 24542, 24535]} 2024-05-27T09:34:04.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-27T09:34:04.614 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.c status 2024-05-27T09:34:05.089 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:05.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:04 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1215992031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:34:05.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:04 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1215992031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:34:05.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:04 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1215992031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:34:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:05 smithi044 ceph-mon[31618]: pgmap v405: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:05 smithi050 ceph-mon[36371]: pgmap v405: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:05 smithi189 ceph-mon[36671]: pgmap v405: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:07.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.509+0000 7f5293600700 1 -- 172.21.15.44:0/452912918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f528c109840 msgr2=0x7f528c1111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:07.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.509+0000 7f5293600700 1 --2- 172.21.15.44:0/452912918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f528c109840 0x7f528c1111a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5284009f90 tx=0x7f528400b330 comp rx=0 tx=0).stop 2024-05-27T09:34:07.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.510+0000 7f5293600700 1 -- 172.21.15.44:0/452912918 shutdown_connections 2024-05-27T09:34:07.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.510+0000 7f5293600700 1 --2- 172.21.15.44:0/452912918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f528c109840 0x7f528c1111a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.510+0000 7f5293600700 1 --2- 172.21.15.44:0/452912918 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f528c1045c0 0x7f528c1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.510+0000 7f5293600700 1 --2- 172.21.15.44:0/452912918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f528c108ef0 0x7f528c109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.510+0000 7f5293600700 1 -- 172.21.15.44:0/452912918 >> 172.21.15.44:0/452912918 conn(0x7f528c0ffe90 msgr2=0x7f528c1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:07.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.510+0000 7f5293600700 1 -- 172.21.15.44:0/452912918 shutdown_connections 2024-05-27T09:34:07.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.510+0000 7f5293600700 1 -- 172.21.15.44:0/452912918 wait complete. 2024-05-27T09:34:07.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.511+0000 7f5293600700 1 Processor -- start 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.511+0000 7f5293600700 1 -- start start 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5293600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f528c1045c0 0x7f528c19d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5293600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f528c108ef0 0x7f528c19d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5293600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f528c109840 0x7f528c1a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5293600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f528c1144c0 con 0x7f528c108ef0 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5293600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f528c114340 con 0x7f528c1045c0 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5293600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f528c114640 con 0x7f528c109840 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5290c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f528c1045c0 0x7f528c19d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5283e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f528c108ef0 0x7f528c19d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5290c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f528c1045c0 0x7f528c19d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50526/0 (socket says 172.21.15.44:50526) 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5290c00700 1 -- 172.21.15.44:0/97946421 learned_addr learned my addr 172.21.15.44:0/97946421 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:07.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5291600700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f528c109840 0x7f528c1a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5290c00700 1 -- 172.21.15.44:0/97946421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f528c109840 msgr2=0x7f528c1a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:07.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.512+0000 7f5290c00700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f528c109840 0x7f528c1a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.513+0000 7f5290c00700 1 -- 172.21.15.44:0/97946421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f528c108ef0 msgr2=0x7f528c19d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:07.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.513+0000 7f5290c00700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f528c108ef0 0x7f528c19d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.513+0000 7f5290c00700 1 -- 172.21.15.44:0/97946421 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528c1a1f40 con 0x7f528c1045c0 2024-05-27T09:34:07.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.513+0000 7f5290c00700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f528c1045c0 0x7f528c19d070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f527400e9c0 tx=0x7f527400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:07.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.513+0000 7f528a000700 1 -- 172.21.15.44:0/97946421 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274004a30 con 0x7f528c1045c0 2024-05-27T09:34:07.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.513+0000 7f528a000700 1 -- 172.21.15.44:0/97946421 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f527400cc70 con 0x7f528c1045c0 2024-05-27T09:34:07.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.513+0000 7f5293600700 1 -- 172.21.15.44:0/97946421 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f528c1a2230 con 0x7f528c1045c0 2024-05-27T09:34:07.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.513+0000 7f528a000700 1 -- 172.21.15.44:0/97946421 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274005490 con 0x7f528c1045c0 2024-05-27T09:34:07.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.514+0000 7f5293600700 1 -- 172.21.15.44:0/97946421 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f528c1a9b50 con 0x7f528c1045c0 2024-05-27T09:34:07.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.517+0000 7f528a000700 1 -- 172.21.15.44:0/97946421 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f5274004bd0 con 0x7f528c1045c0 2024-05-27T09:34:07.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.519+0000 7f52a2233b80 1 Processor -- start 2024-05-27T09:34:07.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.519+0000 7f52a2233b80 1 -- start start 2024-05-27T09:34:07.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.519+0000 7f528a000700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f526c0b2900 0x7f526c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.519+0000 7f528a000700 1 -- 172.21.15.44:0/97946421 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f52740e2310 con 0x7f528c1045c0 2024-05-27T09:34:07.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.519+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x5571335e32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.519+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 0x5571335e3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.519+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335e3e20 0x5571335f00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.519+0000 7f52a2233b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557133511df0 con 0x557133464df0 2024-05-27T09:34:07.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.519+0000 7f52a2233b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557133501990 con 0x5571335e3e20 2024-05-27T09:34:07.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.519+0000 7f52a2233b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5571335059d0 con 0x5571335509a0 2024-05-27T09:34:07.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.519+0000 7f5291600700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f526c0b2900 0x7f526c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.520+0000 7f5290c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x5571335e32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.520+0000 7f5283e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 0x5571335e3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.520+0000 7f5290c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x5571335e32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50766/0 (socket says 172.21.15.44:50766) 2024-05-27T09:34:07.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.520+0000 7f5290c00700 1 -- 172.21.15.44:0/3803124642 learned_addr learned my addr 172.21.15.44:0/3803124642 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:07.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.520+0000 7f5291600700 1 --2- 172.21.15.44:0/3803124642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335e3e20 0x5571335f00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.520+0000 7f5283e00700 1 -- 172.21.15.44:0/3803124642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 msgr2=0x5571335e32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:07.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.520+0000 7f5283e00700 1 --2- 172.21.15.44:0/3803124642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x5571335e32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.520+0000 7f5283e00700 1 -- 172.21.15.44:0/3803124642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335e3e20 msgr2=0x5571335f00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:07.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.520+0000 7f5283e00700 1 --2- 172.21.15.44:0/3803124642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335e3e20 0x5571335f00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.521+0000 7f5283e00700 1 -- 172.21.15.44:0/3803124642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571334ca3d0 con 0x557133464df0 2024-05-27T09:34:07.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.521+0000 7f5291600700 1 --2- 172.21.15.44:0/3803124642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335e3e20 0x5571335f00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:34:07.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.521+0000 7f5291600700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f526c0b2900 0x7f526c0b4e10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5284009da0 tx=0x7f5284009420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:07.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.521+0000 7f5283e00700 1 --2- 172.21.15.44:0/3803124642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 0x5571335e3820 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f528c06c160 tx=0x7f527c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a620e928dcd5d6cd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:07.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.521+0000 7f5282000700 1 -- 172.21.15.44:0/3803124642 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c011c80 con 0x557133464df0 2024-05-27T09:34:07.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.521+0000 7f5282000700 1 -- 172.21.15.44:0/3803124642 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f527c011e20 con 0x557133464df0 2024-05-27T09:34:07.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.521+0000 7f5282000700 1 -- 172.21.15.44:0/3803124642 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c005280 con 0x557133464df0 2024-05-27T09:34:07.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.522+0000 7f52a2233b80 1 -- 172.21.15.44:0/3803124642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 msgr2=0x5571335e3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:07.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.522+0000 7f52a2233b80 1 --2- 172.21.15.44:0/3803124642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 0x5571335e3820 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f528c06c160 tx=0x7f527c009f60 comp rx=0 tx=0).stop 2024-05-27T09:34:07.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.522+0000 7f52a2233b80 1 -- 172.21.15.44:0/3803124642 shutdown_connections 2024-05-27T09:34:07.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.522+0000 7f52a2233b80 1 --2- 172.21.15.44:0/3803124642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335e3e20 0x5571335f00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.522+0000 7f52a2233b80 1 --2- 172.21.15.44:0/3803124642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x5571335e32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.522+0000 7f52a2233b80 1 --2- 172.21.15.44:0/3803124642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 0x5571335e3820 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.522+0000 7f52a2233b80 1 -- 172.21.15.44:0/3803124642 >> 172.21.15.44:0/3803124642 conn(0x5571334fe340 msgr2=0x5571335df470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:07.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.523+0000 7f52a2233b80 1 -- 172.21.15.44:0/3803124642 shutdown_connections 2024-05-27T09:34:07.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.523+0000 7f52a2233b80 1 -- 172.21.15.44:0/3803124642 wait complete. 2024-05-27T09:34:07.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.523+0000 7f52a2233b80 1 -- start start 2024-05-27T09:34:07.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.523+0000 7f52a2233b80 1 Processor -- start 2024-05-27T09:34:07.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 0x5571335ed500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x5571335eda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 0x5571335edf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f52a2233b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557133407f50 con 0x5571335e3e20 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f52a2233b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5571334a2710 con 0x557133464df0 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f52a2233b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5571335f0b30 con 0x5571335509a0 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f52a2233b80 10 client.0 _ll_register_callbacks cb 0x5571334abc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f5291600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 0x5571335edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f5283e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x5571335eda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f5291600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 0x5571335edf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54406/0 (socket says 172.21.15.44:54406) 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f5291600700 1 -- 172.21.15.44:0/2575830250 learned_addr learned my addr 172.21.15.44:0/2575830250 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.524+0000 7f5290c00700 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 0x5571335ed500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.525+0000 7f5291600700 1 -- 172.21.15.44:0/2575830250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 msgr2=0x5571335eda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:07.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.525+0000 7f5291600700 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x5571335eda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.525+0000 7f5291600700 1 -- 172.21.15.44:0/2575830250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 msgr2=0x5571335ed500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.525+0000 7f5291600700 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 0x5571335ed500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.525+0000 7f5291600700 1 -- 172.21.15.44:0/2575830250 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557133451880 con 0x5571335e3e20 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.525+0000 7f5283e00700 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x5571335eda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.525+0000 7f5291600700 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 0x5571335edf80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5284015000 tx=0x7f528401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817d8dff2df6d914 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.525+0000 7f527aa00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.526+0000 7f527aa00700 1 -- 172.21.15.44:0/2575830250 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528402d070 con 0x5571335e3e20 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.526+0000 7f527aa00700 1 -- 172.21.15.44:0/2575830250 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f5284007b90 con 0x5571335e3e20 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.526+0000 7f52a2233b80 1 -- 172.21.15.44:0/2575830250 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f527c011e20 con 0x5571335e3e20 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.526+0000 7f527aa00700 1 -- 172.21.15.44:0/2575830250 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5284007d90 con 0x5571335e3e20 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.526+0000 7f52a2233b80 10 client.14754 fetch_fsmap learned FSMap version 25 2024-05-27T09:34:07.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.526+0000 7f52a2233b80 1 -- 172.21.15.44:0/2575830250 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5571334d2010 con 0x5571335e3e20 2024-05-27T09:34:07.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.526+0000 7f527aa00700 1 -- 172.21.15.44:0/2575830250 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f527c011e20 con 0x5571335e3e20 2024-05-27T09:34:07.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.527+0000 7f527aa00700 1 -- 172.21.15.44:0/2575830250 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f5284011070 con 0x5571335e3e20 2024-05-27T09:34:07.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.527+0000 7f52a2233b80 10 client.14754 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:34:07.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.527+0000 7f52a2233b80 10 client.14754 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T09:34:07.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.527+0000 7f52a2233b80 20 client.14754 populate_metadata read hostname 'smithi044' 2024-05-27T09:34:07.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.527+0000 7f52a2233b80 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5571335fda60 0x5571335feed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.527+0000 7f52a2233b80 4 client.14754 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.527+0000 7f52a2233b80 1 -- 172.21.15.44:0/2575830250 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557133513d40 con 0x5571335fda60 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.527+0000 7f5290c00700 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5571335fda60 0x5571335feed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.528+0000 7f527aa00700 10 client.14754 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.528+0000 7f5290c00700 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5571335fda60 0x5571335feed0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=34710a1300d55b95 server_cookie=96b76f196ee29003 in_seq=0 out_seq=0 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f527aa00700 1 -- 172.21.15.44:0/2575830250 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557133513d40 con 0x5571335fda60 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f527aa00700 10 client.14754 handle_command_reply: tid=0 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f52a2233b80 1 client.14754 shutdown 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f52a2233b80 1 -- 172.21.15.44:0/2575830250 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 msgr2=0x5571335edf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f52a2233b80 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 0x5571335edf80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5284015000 tx=0x7f528401f000 comp rx=0 tx=0).stop 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f52a2233b80 1 -- 172.21.15.44:0/2575830250 shutdown_connections 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f52a2233b80 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5571335fda60 0x5571335feed0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f52a2233b80 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 0x5571335edf80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f52a2233b80 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 0x5571335ed500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f52a2233b80 1 --2- 172.21.15.44:0/2575830250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x5571335eda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f527aa00700 0 client.14754 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:34:07.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.529+0000 7f52a2233b80 1 -- 172.21.15.44:0/2575830250 >> 172.21.15.44:0/2575830250 conn(0x5571334fe340 msgr2=0x5571335e32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:07.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.530+0000 7f52a2233b80 1 -- 172.21.15.44:0/2575830250 shutdown_connections 2024-05-27T09:34:07.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.530+0000 7f52a2233b80 1 -- 172.21.15.44:0/2575830250 wait complete. 2024-05-27T09:34:07.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.530+0000 7f52a2233b80 20 client.14754 trim_cache size 0 max 16384 2024-05-27T09:34:07.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.553+0000 7f52a2233b80 1 Processor -- start 2024-05-27T09:34:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.553+0000 7f52a2233b80 1 -- start start 2024-05-27T09:34:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.554+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 0x5571335df610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.554+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335509a0 0x5571335e8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.554+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335e3e20 0x5571335e6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.554+0000 7f52a2233b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5571335fd210 con 0x557133464df0 2024-05-27T09:34:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.554+0000 7f52a2233b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5571335fd8e0 con 0x5571335509a0 2024-05-27T09:34:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.554+0000 7f52a2233b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5571335fdfb0 con 0x5571335e3e20 2024-05-27T09:34:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.554+0000 7f5283e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335509a0 0x5571335e8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.554+0000 7f5291600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335e3e20 0x5571335e6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.554+0000 7f5283e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335509a0 0x5571335e8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50560/0 (socket says 172.21.15.44:50560) 2024-05-27T09:34:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.554+0000 7f5283e00700 1 -- 172.21.15.44:0/987310646 learned_addr learned my addr 172.21.15.44:0/987310646 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:07.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.554+0000 7f5290c00700 1 --2- 172.21.15.44:0/987310646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 0x5571335df610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.555+0000 7f5283e00700 1 -- 172.21.15.44:0/987310646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335e3e20 msgr2=0x5571335e6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:07.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.555+0000 7f5283e00700 1 --2- 172.21.15.44:0/987310646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335e3e20 0x5571335e6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.555+0000 7f5283e00700 1 -- 172.21.15.44:0/987310646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 msgr2=0x5571335df610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:07.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.555+0000 7f5283e00700 1 --2- 172.21.15.44:0/987310646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 0x5571335df610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.555+0000 7f5283e00700 1 -- 172.21.15.44:0/987310646 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571335ee2f0 con 0x5571335509a0 2024-05-27T09:34:07.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.555+0000 7f5290c00700 1 --2- 172.21.15.44:0/987310646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 0x5571335df610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:34:07.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.555+0000 7f5283e00700 1 --2- 172.21.15.44:0/987310646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335509a0 0x5571335e8d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f527c000c00 tx=0x7f527c0037e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc05cb5c50c0f2e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:07.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.555+0000 7f5281600700 1 -- 172.21.15.44:0/987310646 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c028070 con 0x5571335509a0 2024-05-27T09:34:07.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.556+0000 7f5281600700 1 -- 172.21.15.44:0/987310646 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f527c003e40 con 0x5571335509a0 2024-05-27T09:34:07.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.556+0000 7f5281600700 1 -- 172.21.15.44:0/987310646 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c024440 con 0x5571335509a0 2024-05-27T09:34:07.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.556+0000 7f52a2233b80 1 -- 172.21.15.44:0/987310646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335509a0 msgr2=0x5571335e8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:07.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.556+0000 7f52a2233b80 1 --2- 172.21.15.44:0/987310646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335509a0 0x5571335e8d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f527c000c00 tx=0x7f527c0037e0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.557+0000 7f52a2233b80 1 -- 172.21.15.44:0/987310646 shutdown_connections 2024-05-27T09:34:07.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.557+0000 7f52a2233b80 1 --2- 172.21.15.44:0/987310646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335e3e20 0x5571335e6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.557+0000 7f52a2233b80 1 --2- 172.21.15.44:0/987310646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557133464df0 0x5571335df610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.557+0000 7f52a2233b80 1 --2- 172.21.15.44:0/987310646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5571335509a0 0x5571335e8d00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.557+0000 7f52a2233b80 1 -- 172.21.15.44:0/987310646 >> 172.21.15.44:0/987310646 conn(0x5571334fe340 msgr2=0x5571335def00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.557+0000 7f52a2233b80 1 -- 172.21.15.44:0/987310646 shutdown_connections 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.557+0000 7f52a2233b80 1 -- 172.21.15.44:0/987310646 wait complete. 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.557+0000 7f52a2233b80 1 -- start start 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.557+0000 7f52a2233b80 1 Processor -- start 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 0x557133608350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x557133608890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f52a2233b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 0x5571336014f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f52a2233b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5571335fed50 con 0x5571335e3e20 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f52a2233b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5571335fe680 con 0x557133464df0 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f52a2233b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5571335ff420 con 0x5571335509a0 2024-05-27T09:34:07.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f52a2233b80 10 client.0 _ll_register_callbacks cb 0x5571334abc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f5291600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 0x5571336014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f5290c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 0x557133608350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f5290c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 0x557133608350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50572/0 (socket says 172.21.15.44:50572) 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f5283e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x557133608890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f5290c00700 1 -- 172.21.15.44:0/475373805 learned_addr learned my addr 172.21.15.44:0/475373805 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.558+0000 7f5291600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 0x5571336014f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54420/0 (socket says 172.21.15.44:54420) 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.559+0000 7f5290c00700 1 -- 172.21.15.44:0/475373805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 msgr2=0x557133608890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.559+0000 7f5290c00700 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x557133608890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.559+0000 7f5290c00700 1 -- 172.21.15.44:0/475373805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 msgr2=0x5571336014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.559+0000 7f5290c00700 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 0x5571336014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.559+0000 7f5290c00700 1 -- 172.21.15.44:0/475373805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557133601ae0 con 0x557133464df0 2024-05-27T09:34:07.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.559+0000 7f5283e00700 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x557133608890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:07.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.559+0000 7f5290c00700 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 0x557133608350 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f527400f010 tx=0x7f5274091040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32d870c1f4449588 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:07.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.559+0000 7f527aa00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.559+0000 7f527aa00700 1 -- 172.21.15.44:0/475373805 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52740a98e0 con 0x557133464df0 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.559+0000 7f52a2233b80 1 -- 172.21.15.44:0/475373805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55713350d520 con 0x557133464df0 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.559+0000 7f527aa00700 1 -- 172.21.15.44:0/475373805 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f52740a5ed0 con 0x557133464df0 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.560+0000 7f527aa00700 1 -- 172.21.15.44:0/475373805 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f527408a050 con 0x557133464df0 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.560+0000 7f52a2233b80 10 client.24893 fetch_fsmap learned FSMap version 25 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.560+0000 7f52a2233b80 1 -- 172.21.15.44:0/475373805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5571336724a0 con 0x557133464df0 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.560+0000 7f527aa00700 1 -- 172.21.15.44:0/475373805 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55713350d520 con 0x557133464df0 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.560+0000 7f527aa00700 1 -- 172.21.15.44:0/475373805 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f52740b22b0 con 0x557133464df0 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.560+0000 7f52a2233b80 10 client.24893 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.560+0000 7f52a2233b80 10 client.24893 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.560+0000 7f52a2233b80 20 client.24893 populate_metadata read hostname 'smithi044' 2024-05-27T09:34:07.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.560+0000 7f52a2233b80 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557133674970 0x557133675de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:07.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.560+0000 7f52a2233b80 4 client.24893 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T09:34:07.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.560+0000 7f52a2233b80 1 -- 172.21.15.44:0/475373805 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "status"}) -- 0x557133384200 con 0x557133674970 2024-05-27T09:34:07.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.561+0000 7f5283e00700 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557133674970 0x557133675de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:07.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.561+0000 7f5283e00700 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557133674970 0x557133675de0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=87331c12bc25eb12 server_cookie=8b0ffb04e01478a2 in_seq=0 out_seq=0 2024-05-27T09:34:07.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.561+0000 7f527aa00700 10 client.24893 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:34:07.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f527aa00700 1 -- 172.21.15.44:0/475373805 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557133384200 con 0x557133674970 2024-05-27T09:34:07.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f527aa00700 10 client.24893 handle_command_reply: tid=0 2024-05-27T09:34:07.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 1 client.24893 shutdown 2024-05-27T09:34:07.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 1 -- 172.21.15.44:0/475373805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 msgr2=0x557133608350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:07.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 0x557133608350 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f527400f010 tx=0x7f5274091040 comp rx=0 tx=0).stop 2024-05-27T09:34:07.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 1 -- 172.21.15.44:0/475373805 shutdown_connections 2024-05-27T09:34:07.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557133674970 0x557133675de0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5571335e3e20 0x5571336014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557133464df0 0x557133608350 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 1 --2- 172.21.15.44:0/475373805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5571335509a0 0x557133608890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f527aa00700 0 client.24893 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:34:07.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 1 -- 172.21.15.44:0/475373805 >> 172.21.15.44:0/475373805 conn(0x5571334fe340 msgr2=0x5571335e3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:07.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 1 -- 172.21.15.44:0/475373805 shutdown_connections 2024-05-27T09:34:07.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 1 -- 172.21.15.44:0/475373805 wait complete. 2024-05-27T09:34:07.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.562+0000 7f52a2233b80 20 client.24893 trim_cache size 0 max 16384 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 2, 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24520, 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 143.819430009, 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 25, 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 81, 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 79, 2024-05-27T09:34:07.567 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 199.06154050500001 2024-05-27T09:34:07.568 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:34:07.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 -- 172.21.15.44:0/97946421 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f526c0b2900 msgr2=0x7f526c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:07.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f526c0b2900 0x7f526c0b4e10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5284009da0 tx=0x7f5284009420 comp rx=0 tx=0).stop 2024-05-27T09:34:07.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 -- 172.21.15.44:0/97946421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f528c1045c0 msgr2=0x7f528c19d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:07.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f528c1045c0 0x7f528c19d070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f527400e9c0 tx=0x7f527400ecd0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 -- 172.21.15.44:0/97946421 shutdown_connections 2024-05-27T09:34:07.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f526c0b2900 0x7f526c0b4e10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f528c109840 0x7f528c1a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f528c1045c0 0x7f528c19d070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 --2- 172.21.15.44:0/97946421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f528c108ef0 0x7f528c19d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:07.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 -- 172.21.15.44:0/97946421 >> 172.21.15.44:0/97946421 conn(0x7f528c0ffe90 msgr2=0x7f528c110e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:07.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 -- 172.21.15.44:0/97946421 shutdown_connections 2024-05-27T09:34:07.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:07.563+0000 7f5293600700 1 -- 172.21.15.44:0/97946421 wait complete. 2024-05-27T09:34:08.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:07 smithi044 ceph-mon[31618]: pgmap v406: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:08.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:07 smithi050 ceph-mon[36371]: pgmap v406: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:08.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:07 smithi189 ceph-mon[36671]: pgmap v406: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:08.366 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-05-27T09:34:08.860 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:10.064 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:09 smithi044 ceph-mon[31618]: pgmap v407: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:10.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:09 smithi050 ceph-mon[36371]: pgmap v407: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:10.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:09 smithi189 ceph-mon[36671]: pgmap v407: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.120+0000 7f4b11e00700 1 -- 172.21.15.44:0/3610355053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b0c1045e0 msgr2=0x7f4b0c1049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.120+0000 7f4b11e00700 1 --2- 172.21.15.44:0/3610355053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b0c1045e0 0x7f4b0c1049f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0099c0 tx=0x7f4afc009cd0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.120+0000 7f4b11e00700 1 -- 172.21.15.44:0/3610355053 shutdown_connections 2024-05-27T09:34:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.120+0000 7f4b11e00700 1 --2- 172.21.15.44:0/3610355053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c109860 0x7f4b0c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.120+0000 7f4b11e00700 1 --2- 172.21.15.44:0/3610355053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b0c1045e0 0x7f4b0c1049f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.120+0000 7f4b11e00700 1 --2- 172.21.15.44:0/3610355053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0c108f10 0x7f4b0c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.120+0000 7f4b11e00700 1 -- 172.21.15.44:0/3610355053 >> 172.21.15.44:0/3610355053 conn(0x7f4b0c100030 msgr2=0x7f4b0c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:11.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.120+0000 7f4b11e00700 1 -- 172.21.15.44:0/3610355053 shutdown_connections 2024-05-27T09:34:11.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.120+0000 7f4b11e00700 1 -- 172.21.15.44:0/3610355053 wait complete. 2024-05-27T09:34:11.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.121+0000 7f4b11e00700 1 Processor -- start 2024-05-27T09:34:11.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.122+0000 7f4b11e00700 1 -- start start 2024-05-27T09:34:11.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.122+0000 7f4b11e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c1045e0 0x7f4b0c19d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.122+0000 7f4b11e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0c108f10 0x7f4b0c19d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.122+0000 7f4b11e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b0c109860 0x7f4b0c1a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.122+0000 7f4b11e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c114500 con 0x7f4b0c1045e0 2024-05-27T09:34:11.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.122+0000 7f4b11e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c114380 con 0x7f4b0c108f10 2024-05-27T09:34:11.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.122+0000 7f4b11e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c114680 con 0x7f4b0c109860 2024-05-27T09:34:11.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.122+0000 7f4b0aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c1045e0 0x7f4b0c19d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.122+0000 7f4b0aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c1045e0 0x7f4b0c19d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54440/0 (socket says 172.21.15.44:54440) 2024-05-27T09:34:11.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.122+0000 7f4b0aa00700 1 -- 172.21.15.44:0/3230842003 learned_addr learned my addr 172.21.15.44:0/3230842003 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:11.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.122+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b0c109860 0x7f4b0c1a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.123+0000 7f4b02000700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0c108f10 0x7f4b0c19d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.123+0000 7f4b0b400700 1 -- 172.21.15.44:0/3230842003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0c108f10 msgr2=0x7f4b0c19d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.123+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0c108f10 0x7f4b0c19d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.123+0000 7f4b0b400700 1 -- 172.21.15.44:0/3230842003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c1045e0 msgr2=0x7f4b0c19d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.123+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c1045e0 0x7f4b0c19d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.123+0000 7f4b0b400700 1 -- 172.21.15.44:0/3230842003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0c1a1f10 con 0x7f4b0c109860 2024-05-27T09:34:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.123+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b0c109860 0x7f4b0c1a1870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400e970 tx=0x7f4b0400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.124+0000 7f4b10a00700 1 -- 172.21.15.44:0/3230842003 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b04004220 con 0x7f4b0c109860 2024-05-27T09:34:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.124+0000 7f4b10a00700 1 -- 172.21.15.44:0/3230842003 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b040043c0 con 0x7f4b0c109860 2024-05-27T09:34:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.124+0000 7f4b10a00700 1 -- 172.21.15.44:0/3230842003 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b04010700 con 0x7f4b0c109860 2024-05-27T09:34:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.124+0000 7f4b11e00700 1 -- 172.21.15.44:0/3230842003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0c1a2200 con 0x7f4b0c109860 2024-05-27T09:34:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.124+0000 7f4b11e00700 1 -- 172.21.15.44:0/3230842003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b0c1a9b20 con 0x7f4b0c109860 2024-05-27T09:34:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.128+0000 7f4b10a00700 1 -- 172.21.15.44:0/3230842003 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f4b04004750 con 0x7f4b0c109860 2024-05-27T09:34:11.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.129+0000 7f4b2096cb80 1 Processor -- start 2024-05-27T09:34:11.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.129+0000 7f4b2096cb80 1 -- start start 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.129+0000 7f4b10a00700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4af00b2900 0x7f4af00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.129+0000 7f4b10a00700 1 -- 172.21.15.44:0/3230842003 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f4b040e1f60 con 0x7f4b0c109860 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.129+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555989ade9a0 0x555989b712e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.129+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559899f2df0 0x555989b71820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.129+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b7e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555989a9fdf0 con 0x555989ade9a0 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x555989a8f990 con 0x555989b71e20 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555989a939d0 con 0x5559899f2df0 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b02000700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4af00b2900 0x7f4af00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b0aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555989ade9a0 0x555989b712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b0aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555989ade9a0 0x555989b712e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54454/0 (socket says 172.21.15.44:54454) 2024-05-27T09:34:11.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b0b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b7e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b0aa00700 1 -- 172.21.15.44:0/3119564120 learned_addr learned my addr 172.21.15.44:0/3119564120 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:11.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b02000700 1 --2- 172.21.15.44:0/3119564120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559899f2df0 0x555989b71820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b0b400700 1 -- 172.21.15.44:0/3119564120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559899f2df0 msgr2=0x555989b71820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:11.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3119564120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559899f2df0 0x555989b71820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b0b400700 1 -- 172.21.15.44:0/3119564120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555989ade9a0 msgr2=0x555989b712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:11.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3119564120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555989ade9a0 0x555989b712e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.130+0000 7f4b0b400700 1 -- 172.21.15.44:0/3119564120 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555989a583d0 con 0x555989b71e20 2024-05-27T09:34:11.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.131+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3119564120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b7e0c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400efd0 tx=0x7f4b040a90c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d82c073a047032fe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:11.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.131+0000 7f4b02000700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4af00b2900 0x7f4af00b4e10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x555989b6e740 tx=0x7f4af4009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:11.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.131+0000 7f4b00c00700 1 -- 172.21.15.44:0/3119564120 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0401c040 con 0x555989b71e20 2024-05-27T09:34:11.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.131+0000 7f4b00c00700 1 -- 172.21.15.44:0/3119564120 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b040ad8e0 con 0x555989b71e20 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.131+0000 7f4b00c00700 1 -- 172.21.15.44:0/3119564120 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b040b1f00 con 0x555989b71e20 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.132+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3119564120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 msgr2=0x555989b7e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.132+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3119564120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b7e0c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400efd0 tx=0x7f4b040a90c0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.132+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3119564120 shutdown_connections 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.132+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3119564120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b7e0c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.132+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3119564120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555989ade9a0 0x555989b712e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.132+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3119564120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559899f2df0 0x555989b71820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.132+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3119564120 >> 172.21.15.44:0/3119564120 conn(0x555989a8c340 msgr2=0x555989b6d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.132+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3119564120 shutdown_connections 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.132+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3119564120 wait complete. 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.133+0000 7f4b2096cb80 1 -- start start 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.133+0000 7f4b2096cb80 1 Processor -- start 2024-05-27T09:34:11.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.134+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559899f2df0 0x555989b7b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.134+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555989ade9a0 0x555989b7bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.134+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b7c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.134+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555989995f50 con 0x555989ade9a0 2024-05-27T09:34:11.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.134+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x555989a30710 con 0x555989b71e20 2024-05-27T09:34:11.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.134+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555989b7eb30 con 0x5559899f2df0 2024-05-27T09:34:11.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.134+0000 7f4b2096cb80 10 client.0 _ll_register_callbacks cb 0x555989a39c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:34:11.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.134+0000 7f4b02000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555989ade9a0 0x555989b7bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.134+0000 7f4b0b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b7c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.134+0000 7f4b0b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b7c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50600/0 (socket says 172.21.15.44:50600) 2024-05-27T09:34:11.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.135+0000 7f4b0b400700 1 -- 172.21.15.44:0/3723413092 learned_addr learned my addr 172.21.15.44:0/3723413092 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:11.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.135+0000 7f4b0aa00700 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559899f2df0 0x555989b7b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.135+0000 7f4b0b400700 1 -- 172.21.15.44:0/3723413092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559899f2df0 msgr2=0x555989b7b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.135+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559899f2df0 0x555989b7b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.135+0000 7f4b0b400700 1 -- 172.21.15.44:0/3723413092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555989ade9a0 msgr2=0x555989b7bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.135+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555989ade9a0 0x555989b7bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.135+0000 7f4b0b400700 1 -- 172.21.15.44:0/3723413092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559899df880 con 0x555989b71e20 2024-05-27T09:34:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.135+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b7c0d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4b04004620 tx=0x7f4b040b2410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f60845875a0f383 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:11.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.135+0000 7f4af9600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:34:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.135+0000 7f4af9600700 1 -- 172.21.15.44:0/3723413092 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b040c2070 con 0x555989b71e20 2024-05-27T09:34:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.135+0000 7f4af9600700 1 -- 172.21.15.44:0/3723413092 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f4b040b2100 con 0x555989b71e20 2024-05-27T09:34:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3723413092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b040ad8e0 con 0x555989b71e20 2024-05-27T09:34:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4af9600700 1 -- 172.21.15.44:0/3723413092 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4b0401c070 con 0x555989b71e20 2024-05-27T09:34:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4b2096cb80 10 client.24905 fetch_fsmap learned FSMap version 25 2024-05-27T09:34:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3723413092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555989a60010 con 0x555989b71e20 2024-05-27T09:34:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4af9600700 1 -- 172.21.15.44:0/3723413092 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b040ad8e0 con 0x555989b71e20 2024-05-27T09:34:11.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4af9600700 1 -- 172.21.15.44:0/3723413092 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f4b040a50a0 con 0x555989b71e20 2024-05-27T09:34:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4b2096cb80 10 client.24905 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:34:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4b2096cb80 10 client.24905 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T09:34:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4b2096cb80 20 client.24905 populate_metadata read hostname 'smithi044' 2024-05-27T09:34:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x555989b8b600 0x555989b8ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4b2096cb80 4 client.24905 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:34:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.136+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3723413092 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555989aa1d40 con 0x555989b8b600 2024-05-27T09:34:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.137+0000 7f4b0aa00700 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x555989b8b600 0x555989b8ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.137+0000 7f4b0aa00700 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x555989b8b600 0x555989b8ca70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a866355cfaeca5 server_cookie=611e3851898fa7ec in_seq=0 out_seq=0 2024-05-27T09:34:11.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.137+0000 7f4af9600700 10 client.24905 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:34:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.138+0000 7f4af9600700 1 -- 172.21.15.44:0/3723413092 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555989aa1d40 con 0x555989b8b600 2024-05-27T09:34:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.138+0000 7f4af9600700 10 client.24905 handle_command_reply: tid=0 2024-05-27T09:34:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.138+0000 7f4b2096cb80 1 client.24905 shutdown 2024-05-27T09:34:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3723413092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 msgr2=0x555989b7c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b7c0d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4b04004620 tx=0x7f4b040b2410 comp rx=0 tx=0).stop 2024-05-27T09:34:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3723413092 shutdown_connections 2024-05-27T09:34:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b7c0d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559899f2df0 0x555989b7b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x555989b8b600 0x555989b8ca70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3723413092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555989ade9a0 0x555989b7bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3723413092 >> 172.21.15.44:0/3723413092 conn(0x555989a8c340 msgr2=0x555989b71300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:11.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4af9600700 0 client.24905 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:34:11.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3723413092 shutdown_connections 2024-05-27T09:34:11.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3723413092 wait complete. 2024-05-27T09:34:11.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.139+0000 7f4b2096cb80 20 client.24905 trim_cache size 0 max 16384 2024-05-27T09:34:11.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.162+0000 7f4b2096cb80 1 Processor -- start 2024-05-27T09:34:11.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b2096cb80 1 -- start start 2024-05-27T09:34:11.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 0x555989b76e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989ade9a0 0x555989b7a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b87dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555989b8adb0 con 0x5559899f2df0 2024-05-27T09:34:11.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x555989b8b480 con 0x555989b71e20 2024-05-27T09:34:11.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555989b8bb50 con 0x555989ade9a0 2024-05-27T09:34:11.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b0aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 0x555989b76e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b02000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989ade9a0 0x555989b7a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b0aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 0x555989b76e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54470/0 (socket says 172.21.15.44:54470) 2024-05-27T09:34:11.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b0aa00700 1 -- 172.21.15.44:0/85410611 learned_addr learned my addr 172.21.15.44:0/85410611 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:11.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b02000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989ade9a0 0x555989b7a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50826/0 (socket says 172.21.15.44:50826) 2024-05-27T09:34:11.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.163+0000 7f4b0b400700 1 --2- 172.21.15.44:0/85410611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b87dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.164+0000 7f4b0b400700 1 -- 172.21.15.44:0/85410611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989ade9a0 msgr2=0x555989b7a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:11.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.164+0000 7f4b0b400700 1 --2- 172.21.15.44:0/85410611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989ade9a0 0x555989b7a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.164+0000 7f4b0b400700 1 -- 172.21.15.44:0/85410611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 msgr2=0x555989b76e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:11.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.164+0000 7f4b0b400700 1 --2- 172.21.15.44:0/85410611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 0x555989b76e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.164+0000 7f4b0b400700 1 -- 172.21.15.44:0/85410611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555989b88310 con 0x555989b71e20 2024-05-27T09:34:11.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.164+0000 7f4b0b400700 1 --2- 172.21.15.44:0/85410611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b87dd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400efd0 tx=0x7f4b040a90c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c717ebeecab1119 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:11.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.164+0000 7f4afbe00700 1 -- 172.21.15.44:0/85410611 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b040c2030 con 0x555989b71e20 2024-05-27T09:34:11.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.164+0000 7f4afbe00700 1 -- 172.21.15.44:0/85410611 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b040a8570 con 0x555989b71e20 2024-05-27T09:34:11.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.164+0000 7f4afbe00700 1 -- 172.21.15.44:0/85410611 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b040ca280 con 0x555989b71e20 2024-05-27T09:34:11.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.165+0000 7f4b2096cb80 1 -- 172.21.15.44:0/85410611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 msgr2=0x555989b87dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:11.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.165+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/85410611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b87dd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400efd0 tx=0x7f4b040a90c0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.165+0000 7f4b2096cb80 1 -- 172.21.15.44:0/85410611 shutdown_connections 2024-05-27T09:34:11.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.165+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/85410611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989b71e20 0x555989b87dd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.165+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/85410611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 0x555989b76e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.165+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/85410611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989ade9a0 0x555989b7a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.165+0000 7f4b2096cb80 1 -- 172.21.15.44:0/85410611 >> 172.21.15.44:0/85410611 conn(0x555989a8c340 msgr2=0x555989b716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:11.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.165+0000 7f4b2096cb80 1 -- 172.21.15.44:0/85410611 shutdown_connections 2024-05-27T09:34:11.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.165+0000 7f4b2096cb80 1 -- 172.21.15.44:0/85410611 wait complete. 2024-05-27T09:34:11.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.166+0000 7f4b2096cb80 1 -- start start 2024-05-27T09:34:11.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.166+0000 7f4b2096cb80 1 Processor -- start 2024-05-27T09:34:11.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 0x555989b96560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989ade9a0 0x555989b94210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b2096cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989b71e20 0x555989b94750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555989b8c8f0 con 0x5559899f2df0 2024-05-27T09:34:11.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x555989b8c220 con 0x555989ade9a0 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b2096cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555989b8cfc0 con 0x555989b71e20 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b2096cb80 10 client.0 _ll_register_callbacks cb 0x555989a39c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b0aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 0x555989b96560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b0aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 0x555989b96560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54482/0 (socket says 172.21.15.44:54482) 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b0aa00700 1 -- 172.21.15.44:0/3720126746 learned_addr learned my addr 172.21.15.44:0/3720126746 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989b71e20 0x555989b94750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.167+0000 7f4b02000700 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989ade9a0 0x555989b94210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4b0aa00700 1 -- 172.21.15.44:0/3720126746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989b71e20 msgr2=0x555989b94750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4b0aa00700 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989b71e20 0x555989b94750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4b0aa00700 1 -- 172.21.15.44:0/3720126746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989ade9a0 msgr2=0x555989b94210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4b0aa00700 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989ade9a0 0x555989b94210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4b0aa00700 1 -- 172.21.15.44:0/3720126746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555989b94e00 con 0x5559899f2df0 2024-05-27T09:34:11.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4b0b400700 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989b71e20 0x555989b94750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:11.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4b02000700 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989ade9a0 0x555989b94210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:11.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4b0aa00700 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 0x555989b96560 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009990 tx=0x7f4afc005310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425b46246347fd11 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:11.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4af9600700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:34:11.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4af9600700 1 -- 172.21.15.44:0/3720126746 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc0039c0 con 0x5559899f2df0 2024-05-27T09:34:11.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3720126746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b040a8570 con 0x5559899f2df0 2024-05-27T09:34:11.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.168+0000 7f4af9600700 1 -- 172.21.15.44:0/3720126746 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f4afc003b60 con 0x5559899f2df0 2024-05-27T09:34:11.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.169+0000 7f4af9600700 1 -- 172.21.15.44:0/3720126746 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4afc003d60 con 0x5559899f2df0 2024-05-27T09:34:11.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.169+0000 7f4b2096cb80 10 client.14778 fetch_fsmap learned FSMap version 25 2024-05-27T09:34:11.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.169+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3720126746 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555989b93020 con 0x5559899f2df0 2024-05-27T09:34:11.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.169+0000 7f4af9600700 1 -- 172.21.15.44:0/3720126746 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b040a8570 con 0x5559899f2df0 2024-05-27T09:34:11.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.169+0000 7f4af9600700 1 -- 172.21.15.44:0/3720126746 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f4afc00f580 con 0x5559899f2df0 2024-05-27T09:34:11.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.169+0000 7f4b2096cb80 10 client.14778 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:34:11.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.169+0000 7f4b2096cb80 10 client.14778 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T09:34:11.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.169+0000 7f4b2096cb80 20 client.14778 populate_metadata read hostname 'smithi044' 2024-05-27T09:34:11.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.169+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x555989bf9ff0 0x555989bfb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:11.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.170+0000 7f4b2096cb80 4 client.14778 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T09:34:11.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.170+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3720126746 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "status"}) -- 0x555989a9b520 con 0x555989bf9ff0 2024-05-27T09:34:11.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.170+0000 7f4b02000700 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x555989bf9ff0 0x555989bfb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:11.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b02000700 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x555989bf9ff0 0x555989bfb460 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a99016023eec1db2 server_cookie=ede435c60b7fa363 in_seq=0 out_seq=0 2024-05-27T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4af9600700 10 client.14778 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4af9600700 1 -- 172.21.15.44:0/3720126746 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x555989a9b520 con 0x555989bf9ff0 2024-05-27T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4af9600700 10 client.14778 handle_command_reply: tid=0 2024-05-27T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b2096cb80 1 client.14778 shutdown 2024-05-27T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3720126746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 msgr2=0x555989b96560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 0x555989b96560 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009990 tx=0x7f4afc005310 comp rx=0 tx=0).stop 2024-05-27T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3720126746 shutdown_connections 2024-05-27T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555989b71e20 0x555989b94750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x555989bf9ff0 0x555989bfb460 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559899f2df0 0x555989b96560 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b2096cb80 1 --2- 172.21.15.44:0/3720126746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555989ade9a0 0x555989b94210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3720126746 >> 172.21.15.44:0/3720126746 conn(0x555989a8c340 msgr2=0x555989b716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:11.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4af9600700 0 client.14778 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:34:11.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3720126746 shutdown_connections 2024-05-27T09:34:11.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.171+0000 7f4b2096cb80 1 -- 172.21.15.44:0/3720126746 wait complete. 2024-05-27T09:34:11.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.172+0000 7f4b2096cb80 20 client.14778 trim_cache size 0 max 16384 2024-05-27T09:34:11.183 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:34:11.183 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:34:11.183 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 1, 2024-05-27T09:34:11.183 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24535, 2024-05-27T09:34:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-05-27T09:34:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-05-27T09:34:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-05-27T09:34:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 148.44074951299999, 2024-05-27T09:34:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 25, 2024-05-27T09:34:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 81, 2024-05-27T09:34:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 79, 2024-05-27T09:34:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 197.16503442000001 2024-05-27T09:34:11.184 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:34:11.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.172+0000 7f4b11e00700 1 -- 172.21.15.44:0/3230842003 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4af00b2900 msgr2=0x7f4af00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:11.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.172+0000 7f4b11e00700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4af00b2900 0x7f4af00b4e10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x555989b6e740 tx=0x7f4af4009920 comp rx=0 tx=0).stop 2024-05-27T09:34:11.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.172+0000 7f4b11e00700 1 -- 172.21.15.44:0/3230842003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b0c109860 msgr2=0x7f4b0c1a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:11.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.172+0000 7f4b11e00700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b0c109860 0x7f4b0c1a1870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400e970 tx=0x7f4b0400ec80 comp rx=0 tx=0).stop 2024-05-27T09:34:11.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.172+0000 7f4b11e00700 1 -- 172.21.15.44:0/3230842003 shutdown_connections 2024-05-27T09:34:11.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.172+0000 7f4b11e00700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f4af00b2900 0x7f4af00b4e10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.172+0000 7f4b11e00700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b0c109860 0x7f4b0c1a1870 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.172+0000 7f4b11e00700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b0c1045e0 0x7f4b0c19d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.172+0000 7f4b11e00700 1 --2- 172.21.15.44:0/3230842003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0c108f10 0x7f4b0c19d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:11.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.172+0000 7f4b11e00700 1 -- 172.21.15.44:0/3230842003 >> 172.21.15.44:0/3230842003 conn(0x7f4b0c100030 msgr2=0x7f4b0c110490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:11.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.173+0000 7f4b11e00700 1 -- 172.21.15.44:0/3230842003 shutdown_connections 2024-05-27T09:34:11.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:11.173+0000 7f4b11e00700 1 -- 172.21.15.44:0/3230842003 wait complete. 2024-05-27T09:34:11.978 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T09:34:12.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:11 smithi044 ceph-mon[31618]: pgmap v408: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:12.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:11 smithi050 ceph-mon[36371]: pgmap v408: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:12.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:11 smithi189 ceph-mon[36671]: pgmap v408: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:14.140 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:13 smithi189 ceph-mon[36671]: pgmap v409: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:14.148 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:13 smithi044 ceph-mon[31618]: pgmap v409: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:14.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:13 smithi050 ceph-mon[36371]: pgmap v409: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:14.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.759+0000 7f43c1e00700 1 -- 172.21.15.44:0/3299194386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43bc076e30 msgr2=0x7f43bc077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:14.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.759+0000 7f43c1e00700 1 --2- 172.21.15.44:0/3299194386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43bc076e30 0x7f43bc077240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f43ac009990 tx=0x7f43ac009ca0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.760+0000 7f43c1e00700 1 -- 172.21.15.44:0/3299194386 shutdown_connections 2024-05-27T09:34:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.760+0000 7f43c1e00700 1 --2- 172.21.15.44:0/3299194386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43bc110b70 0x7f43bc112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.760+0000 7f43c1e00700 1 --2- 172.21.15.44:0/3299194386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43bc076e30 0x7f43bc077240 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.760+0000 7f43c1e00700 1 --2- 172.21.15.44:0/3299194386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43bc077780 0x7f43bc110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.760+0000 7f43c1e00700 1 -- 172.21.15.44:0/3299194386 >> 172.21.15.44:0/3299194386 conn(0x7f43bc0ffe90 msgr2=0x7f43bc1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.760+0000 7f43c1e00700 1 -- 172.21.15.44:0/3299194386 shutdown_connections 2024-05-27T09:34:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.760+0000 7f43c1e00700 1 -- 172.21.15.44:0/3299194386 wait complete. 2024-05-27T09:34:14.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.762+0000 7f43c1e00700 1 Processor -- start 2024-05-27T09:34:14.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.762+0000 7f43c1e00700 1 -- start start 2024-05-27T09:34:14.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.762+0000 7f43c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43bc076e30 0x7f43bc19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.762+0000 7f43c1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43bc077780 0x7f43bc19d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.762+0000 7f43c1e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43bc110b70 0x7f43bc1a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.762+0000 7f43c1e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc116660 con 0x7f43bc076e30 2024-05-27T09:34:14.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.762+0000 7f43c1e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc1164e0 con 0x7f43bc110b70 2024-05-27T09:34:14.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.762+0000 7f43c1e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc1167e0 con 0x7f43bc077780 2024-05-27T09:34:14.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.763+0000 7f43baa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43bc076e30 0x7f43bc19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.763+0000 7f43ba000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43bc077780 0x7f43bc19d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.763+0000 7f43baa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43bc076e30 0x7f43bc19cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47996/0 (socket says 172.21.15.44:47996) 2024-05-27T09:34:14.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.763+0000 7f43baa00700 1 -- 172.21.15.44:0/2979172908 learned_addr learned my addr 172.21.15.44:0/2979172908 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:14.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.763+0000 7f43bb400700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43bc110b70 0x7f43bc1a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.763+0000 7f43ba000700 1 -- 172.21.15.44:0/2979172908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43bc110b70 msgr2=0x7f43bc1a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.763+0000 7f43ba000700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43bc110b70 0x7f43bc1a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.763+0000 7f43ba000700 1 -- 172.21.15.44:0/2979172908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43bc076e30 msgr2=0x7f43bc19cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.763+0000 7f43ba000700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43bc076e30 0x7f43bc19cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.763+0000 7f43ba000700 1 -- 172.21.15.44:0/2979172908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43bc1a1ec0 con 0x7f43bc077780 2024-05-27T09:34:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.764+0000 7f43ba000700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43bc077780 0x7f43bc19d590 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f43a800e9f0 tx=0x7f43a800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.764+0000 7f43c0a00700 1 -- 172.21.15.44:0/2979172908 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a800c9c0 con 0x7f43bc077780 2024-05-27T09:34:14.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.764+0000 7f43c1e00700 1 -- 172.21.15.44:0/2979172908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43bc1a21b0 con 0x7f43bc077780 2024-05-27T09:34:14.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.764+0000 7f43c0a00700 1 -- 172.21.15.44:0/2979172908 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43a800cb60 con 0x7f43bc077780 2024-05-27T09:34:14.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.764+0000 7f43c0a00700 1 -- 172.21.15.44:0/2979172908 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a8018680 con 0x7f43bc077780 2024-05-27T09:34:14.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.764+0000 7f43c1e00700 1 -- 172.21.15.44:0/2979172908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43bc1a9ad0 con 0x7f43bc077780 2024-05-27T09:34:14.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.768+0000 7f43c0a00700 1 -- 172.21.15.44:0/2979172908 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f43a8018a30 con 0x7f43bc077780 2024-05-27T09:34:14.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.770+0000 7f43d0a28b80 1 Processor -- start 2024-05-27T09:34:14.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.770+0000 7f43d0a28b80 1 -- start start 2024-05-27T09:34:14.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.770+0000 7f43c0a00700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f43a00b2880 0x7f43a00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b7504820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43c0a00700 1 -- 172.21.15.44:0/2979172908 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f43a80e25b0 con 0x7f43bc077780 2024-05-27T09:34:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b7505170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5580b7432df0 con 0x5580b750ace0 2024-05-27T09:34:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5580b7422990 con 0x5580b7504d60 2024-05-27T09:34:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5580b74269d0 con 0x5580b7385df0 2024-05-27T09:34:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43bb400700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f43a00b2880 0x7f43a00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43baa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43baa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45966/0 (socket says 172.21.15.44:45966) 2024-05-27T09:34:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43baa00700 1 -- 172.21.15.44:0/1679797715 learned_addr learned my addr 172.21.15.44:0/1679797715 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:14.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43ba000700 1 --2- 172.21.15.44:0/1679797715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b7504820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.771+0000 7f43bb400700 1 --2- 172.21.15.44:0/1679797715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b7505170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.772+0000 7f43baa00700 1 -- 172.21.15.44:0/1679797715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 msgr2=0x5580b7505170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.772+0000 7f43baa00700 1 --2- 172.21.15.44:0/1679797715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b7505170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.772+0000 7f43baa00700 1 -- 172.21.15.44:0/1679797715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 msgr2=0x5580b7504820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.772+0000 7f43baa00700 1 --2- 172.21.15.44:0/1679797715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b7504820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.772+0000 7f43baa00700 1 -- 172.21.15.44:0/1679797715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b73eb3d0 con 0x5580b7385df0 2024-05-27T09:34:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.772+0000 7f43ba000700 1 --2- 172.21.15.44:0/1679797715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b7504820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:34:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.772+0000 7f43bb400700 1 --2- 172.21.15.44:0/1679797715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b7505170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:34:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.772+0000 7f43bb400700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f43a00b2880 0x7f43a00b4d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f43b4009fd0 tx=0x7f43b4009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.772+0000 7f43baa00700 1 --2- 172.21.15.44:0/1679797715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750a7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f43ac009740 tx=0x7f43ac005050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b234b09e79a2c61 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.772+0000 7f43b0c00700 1 -- 172.21.15.44:0/1679797715 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac027070 con 0x5580b7385df0 2024-05-27T09:34:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.773+0000 7f43b0c00700 1 -- 172.21.15.44:0/1679797715 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43ac00bbd0 con 0x5580b7385df0 2024-05-27T09:34:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.773+0000 7f43b0c00700 1 -- 172.21.15.44:0/1679797715 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac003e10 con 0x5580b7385df0 2024-05-27T09:34:14.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.773+0000 7f43d0a28b80 1 -- 172.21.15.44:0/1679797715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 msgr2=0x5580b750a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:14.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.773+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/1679797715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750a7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f43ac009740 tx=0x7f43ac005050 comp rx=0 tx=0).stop 2024-05-27T09:34:14.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.774+0000 7f43d0a28b80 1 -- 172.21.15.44:0/1679797715 shutdown_connections 2024-05-27T09:34:14.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.774+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/1679797715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b7505170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.774+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/1679797715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750a7a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.774+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/1679797715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b7504820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.774+0000 7f43d0a28b80 1 -- 172.21.15.44:0/1679797715 >> 172.21.15.44:0/1679797715 conn(0x5580b73e5f70 msgr2=0x5580b74ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:14.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.774+0000 7f43d0a28b80 1 -- 172.21.15.44:0/1679797715 shutdown_connections 2024-05-27T09:34:14.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.774+0000 7f43d0a28b80 1 -- 172.21.15.44:0/1679797715 wait complete. 2024-05-27T09:34:14.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.774+0000 7f43d0a28b80 1 -- start start 2024-05-27T09:34:14.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.775+0000 7f43d0a28b80 1 Processor -- start 2024-05-27T09:34:14.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.775+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.775+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b750e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.775+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b750eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.775+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5580b7328f50 con 0x5580b750ace0 2024-05-27T09:34:14.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.775+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5580b73c3710 con 0x5580b7504d60 2024-05-27T09:34:14.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.775+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5580b7511750 con 0x5580b7385df0 2024-05-27T09:34:14.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.775+0000 7f43d0a28b80 10 client.0 _ll_register_callbacks cb 0x5580b73ccc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:34:14.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.776+0000 7f43baa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.776+0000 7f43baa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45976/0 (socket says 172.21.15.44:45976) 2024-05-27T09:34:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.776+0000 7f43baa00700 1 -- 172.21.15.44:0/855213102 learned_addr learned my addr 172.21.15.44:0/855213102 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.776+0000 7f43bb400700 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b750eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.776+0000 7f43baa00700 1 -- 172.21.15.44:0/855213102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 msgr2=0x5580b750e980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T09:34:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.776+0000 7f43baa00700 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b750e980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.776+0000 7f43baa00700 1 -- 172.21.15.44:0/855213102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 msgr2=0x5580b750eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.776+0000 7f43baa00700 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b750eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.776+0000 7f43baa00700 1 -- 172.21.15.44:0/855213102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b7372880 con 0x5580b7385df0 2024-05-27T09:34:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.776+0000 7f43baa00700 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750e440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f43ac00b430 tx=0x7f43ac0038b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4789891c0f24b0c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.776+0000 7f43a5600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:34:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.777+0000 7f43a5600700 1 -- 172.21.15.44:0/855213102 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac030020 con 0x5580b7385df0 2024-05-27T09:34:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.777+0000 7f43d0a28b80 1 -- 172.21.15.44:0/855213102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43ac00bbd0 con 0x5580b7385df0 2024-05-27T09:34:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.777+0000 7f43a5600700 1 -- 172.21.15.44:0/855213102 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f43ac00b720 con 0x5580b7385df0 2024-05-27T09:34:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.777+0000 7f43a5600700 1 -- 172.21.15.44:0/855213102 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f43ac027040 con 0x5580b7385df0 2024-05-27T09:34:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.777+0000 7f43d0a28b80 10 client.24940 fetch_fsmap learned FSMap version 25 2024-05-27T09:34:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.777+0000 7f43d0a28b80 1 -- 172.21.15.44:0/855213102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5580b73f3010 con 0x5580b7385df0 2024-05-27T09:34:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.777+0000 7f43a5600700 1 -- 172.21.15.44:0/855213102 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43ac00bbd0 con 0x5580b7385df0 2024-05-27T09:34:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.778+0000 7f43a5600700 1 -- 172.21.15.44:0/855213102 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f43ac022850 con 0x5580b7385df0 2024-05-27T09:34:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.778+0000 7f43d0a28b80 10 client.24940 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:34:14.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.778+0000 7f43d0a28b80 10 client.24940 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T09:34:14.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.778+0000 7f43d0a28b80 20 client.24940 populate_metadata read hostname 'smithi044' 2024-05-27T09:34:14.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.778+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5580b751e1e0 0x5580b751f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.778+0000 7f43d0a28b80 4 client.24940 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:34:14.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.778+0000 7f43d0a28b80 1 -- 172.21.15.44:0/855213102 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580b7434d40 con 0x5580b751e1e0 2024-05-27T09:34:14.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.778+0000 7f43ba000700 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5580b751e1e0 0x5580b751f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.779+0000 7f43ba000700 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5580b751e1e0 0x5580b751f650 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f741ba2e1b4b79f server_cookie=337ba05f24cdb091 in_seq=0 out_seq=0 2024-05-27T09:34:14.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.779+0000 7f43a5600700 10 client.24940 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:34:14.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43a5600700 1 -- 172.21.15.44:0/855213102 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5580b7434d40 con 0x5580b751e1e0 2024-05-27T09:34:14.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43a5600700 10 client.24940 handle_command_reply: tid=0 2024-05-27T09:34:14.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43d0a28b80 1 client.24940 shutdown 2024-05-27T09:34:14.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43d0a28b80 1 -- 172.21.15.44:0/855213102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 msgr2=0x5580b750e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:14.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750e440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f43ac00b430 tx=0x7f43ac0038b0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43d0a28b80 1 -- 172.21.15.44:0/855213102 shutdown_connections 2024-05-27T09:34:14.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5580b751e1e0 0x5580b751f650 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b750eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b750e440 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/855213102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b750e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43a5600700 0 client.24940 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:34:14.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.780+0000 7f43d0a28b80 1 -- 172.21.15.44:0/855213102 >> 172.21.15.44:0/855213102 conn(0x5580b73e5f70 msgr2=0x5580b74ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:14.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.781+0000 7f43d0a28b80 1 -- 172.21.15.44:0/855213102 shutdown_connections 2024-05-27T09:34:14.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.781+0000 7f43d0a28b80 1 -- 172.21.15.44:0/855213102 wait complete. 2024-05-27T09:34:14.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.781+0000 7f43d0a28b80 20 client.24940 trim_cache size 0 max 16384 2024-05-27T09:34:14.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.804+0000 7f43d0a28b80 1 Processor -- start 2024-05-27T09:34:14.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43d0a28b80 1 -- start start 2024-05-27T09:34:14.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b7504820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b7508bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b75108e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5580b751d990 con 0x5580b750ace0 2024-05-27T09:34:14.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5580b751e060 con 0x5580b7504d60 2024-05-27T09:34:14.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5580b751e730 con 0x5580b7385df0 2024-05-27T09:34:14.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43bb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b75108e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43baa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b7504820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43bb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b75108e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48024/0 (socket says 172.21.15.44:48024) 2024-05-27T09:34:14.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43bb400700 1 -- 172.21.15.44:0/462370093 learned_addr learned my addr 172.21.15.44:0/462370093 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:14.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.805+0000 7f43ba000700 1 --2- 172.21.15.44:0/462370093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b7508bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.806+0000 7f43baa00700 1 -- 172.21.15.44:0/462370093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 msgr2=0x5580b7508bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:14.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.806+0000 7f43baa00700 1 --2- 172.21.15.44:0/462370093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b7508bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.806+0000 7f43baa00700 1 -- 172.21.15.44:0/462370093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 msgr2=0x5580b75108e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:14.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.806+0000 7f43baa00700 1 --2- 172.21.15.44:0/462370093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b75108e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.806+0000 7f43baa00700 1 -- 172.21.15.44:0/462370093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b7511430 con 0x5580b7385df0 2024-05-27T09:34:14.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.806+0000 7f43baa00700 1 --2- 172.21.15.44:0/462370093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b7504820 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f43ac0098e0 tx=0x7f43ac009290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39b869818d4c8270 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.806+0000 7f43a7e00700 1 -- 172.21.15.44:0/462370093 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac030070 con 0x5580b7385df0 2024-05-27T09:34:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.807+0000 7f43a7e00700 1 -- 172.21.15.44:0/462370093 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43ac0055c0 con 0x5580b7385df0 2024-05-27T09:34:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.807+0000 7f43a7e00700 1 -- 172.21.15.44:0/462370093 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac0226e0 con 0x5580b7385df0 2024-05-27T09:34:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.807+0000 7f43d0a28b80 1 -- 172.21.15.44:0/462370093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 msgr2=0x5580b7504820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.807+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/462370093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b7504820 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f43ac0098e0 tx=0x7f43ac009290 comp rx=0 tx=0).stop 2024-05-27T09:34:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.808+0000 7f43d0a28b80 1 -- 172.21.15.44:0/462370093 shutdown_connections 2024-05-27T09:34:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.808+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/462370093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b750ace0 0x5580b75108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.808+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/462370093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7385df0 0x5580b7504820 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.808+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/462370093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7504d60 0x5580b7508bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.808+0000 7f43d0a28b80 1 -- 172.21.15.44:0/462370093 >> 172.21.15.44:0/462370093 conn(0x5580b73e5f70 msgr2=0x5580b7509c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.808+0000 7f43d0a28b80 1 -- 172.21.15.44:0/462370093 shutdown_connections 2024-05-27T09:34:14.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.808+0000 7f43d0a28b80 1 -- 172.21.15.44:0/462370093 wait complete. 2024-05-27T09:34:14.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.808+0000 7f43d0a28b80 1 -- start start 2024-05-27T09:34:14.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.808+0000 7f43d0a28b80 1 Processor -- start 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.809+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7385df0 0x5580b7528ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.809+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b7504d60 0x5580b7529010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.809+0000 7f43d0a28b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b750ace0 0x5580b7523a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.809+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5580b751f4d0 con 0x5580b7504d60 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.809+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5580b751ee00 con 0x5580b7385df0 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.809+0000 7f43d0a28b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5580b751fba0 con 0x5580b750ace0 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.809+0000 7f43d0a28b80 10 client.0 _ll_register_callbacks cb 0x5580b73ccc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.809+0000 7f43ba000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b7504d60 0x5580b7529010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.809+0000 7f43ba000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b7504d60 0x5580b7529010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48028/0 (socket says 172.21.15.44:48028) 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.809+0000 7f43ba000700 1 -- 172.21.15.44:0/631713622 learned_addr learned my addr 172.21.15.44:0/631713622 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.809+0000 7f43baa00700 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7385df0 0x5580b7528ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.810+0000 7f43bb400700 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b750ace0 0x5580b7523a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.810+0000 7f43baa00700 1 -- 172.21.15.44:0/631713622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b750ace0 msgr2=0x5580b7523a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.810+0000 7f43baa00700 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b750ace0 0x5580b7523a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.810+0000 7f43baa00700 1 -- 172.21.15.44:0/631713622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b7504d60 msgr2=0x5580b7529010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.810+0000 7f43baa00700 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b7504d60 0x5580b7529010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.810+0000 7f43baa00700 1 -- 172.21.15.44:0/631713622 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b750f230 con 0x5580b7385df0 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.810+0000 7f43ba000700 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b7504d60 0x5580b7529010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.810+0000 7f43baa00700 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7385df0 0x5580b7528ad0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x5580b751fe30 tx=0x7f43ac005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad9dfd50884e2a0c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.810+0000 7f43a5600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.811+0000 7f43a5600700 1 -- 172.21.15.44:0/631713622 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac044060 con 0x5580b7385df0 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.811+0000 7f43d0a28b80 1 -- 172.21.15.44:0/631713622 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43ac0055c0 con 0x5580b7385df0 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.811+0000 7f43a5600700 1 -- 172.21.15.44:0/631713622 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f43ac003950 con 0x5580b7385df0 2024-05-27T09:34:14.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.811+0000 7f43a5600700 1 -- 172.21.15.44:0/631713622 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f43ac030040 con 0x5580b7385df0 2024-05-27T09:34:14.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.811+0000 7f43d0a28b80 10 client.24926 fetch_fsmap learned FSMap version 25 2024-05-27T09:34:14.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.811+0000 7f43d0a28b80 1 -- 172.21.15.44:0/631713622 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5580b7589e70 con 0x5580b7385df0 2024-05-27T09:34:14.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.811+0000 7f43a5600700 1 -- 172.21.15.44:0/631713622 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43ac0055c0 con 0x5580b7385df0 2024-05-27T09:34:14.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.811+0000 7f43a5600700 1 -- 172.21.15.44:0/631713622 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f43ac004050 con 0x5580b7385df0 2024-05-27T09:34:14.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.812+0000 7f43d0a28b80 10 client.24926 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:34:14.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.812+0000 7f43d0a28b80 10 client.24926 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T09:34:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.812+0000 7f43d0a28b80 20 client.24926 populate_metadata read hostname 'smithi044' 2024-05-27T09:34:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.812+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5580b758bd70 0x5580b758d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.812+0000 7f43d0a28b80 4 client.24926 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T09:34:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.812+0000 7f43d0a28b80 1 -- 172.21.15.44:0/631713622 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "status"}) -- 0x5580b751b400 con 0x5580b758bd70 2024-05-27T09:34:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.812+0000 7f43ba000700 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5580b758bd70 0x5580b758d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43ba000700 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5580b758bd70 0x5580b758d1e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90281ef93374be63 server_cookie=7f1209e9764b2495 in_seq=0 out_seq=0 2024-05-27T09:34:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43a5600700 10 client.24926 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:34:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43a5600700 1 -- 172.21.15.44:0/631713622 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5580b751b400 con 0x5580b758bd70 2024-05-27T09:34:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43a5600700 10 client.24926 handle_command_reply: tid=0 2024-05-27T09:34:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43d0a28b80 1 client.24926 shutdown 2024-05-27T09:34:14.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43d0a28b80 1 -- 172.21.15.44:0/631713622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7385df0 msgr2=0x5580b7528ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:14.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7385df0 0x5580b7528ad0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x5580b751fe30 tx=0x7f43ac005770 comp rx=0 tx=0).stop 2024-05-27T09:34:14.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43d0a28b80 1 -- 172.21.15.44:0/631713622 shutdown_connections 2024-05-27T09:34:14.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5580b758bd70 0x5580b758d1e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b750ace0 0x5580b7523a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5580b7385df0 0x5580b7528ad0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43d0a28b80 1 --2- 172.21.15.44:0/631713622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5580b7504d60 0x5580b7529010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43a5600700 0 client.24926 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:34:14.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43d0a28b80 1 -- 172.21.15.44:0/631713622 >> 172.21.15.44:0/631713622 conn(0x5580b73e5f70 msgr2=0x5580b750e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:14.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43d0a28b80 1 -- 172.21.15.44:0/631713622 shutdown_connections 2024-05-27T09:34:14.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.813+0000 7f43d0a28b80 1 -- 172.21.15.44:0/631713622 wait complete. 2024-05-27T09:34:14.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.814+0000 7f43d0a28b80 20 client.24926 trim_cache size 0 max 16384 2024-05-27T09:34:14.819 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:34:14.819 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:34:14.819 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-05-27T09:34:14.820 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24542, 2024-05-27T09:34:14.820 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-05-27T09:34:14.820 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-05-27T09:34:14.820 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-05-27T09:34:14.820 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 166.88260093700001, 2024-05-27T09:34:14.820 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 25, 2024-05-27T09:34:14.820 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 81, 2024-05-27T09:34:14.820 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 79, 2024-05-27T09:34:14.820 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 189.15279384900001 2024-05-27T09:34:14.820 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:34:14.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.814+0000 7f43c1e00700 1 -- 172.21.15.44:0/2979172908 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f43a00b2880 msgr2=0x7f43a00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:14.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.814+0000 7f43c1e00700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f43a00b2880 0x7f43a00b4d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f43b4009fd0 tx=0x7f43b4009920 comp rx=0 tx=0).stop 2024-05-27T09:34:14.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.814+0000 7f43c1e00700 1 -- 172.21.15.44:0/2979172908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43bc077780 msgr2=0x7f43bc19d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:14.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.814+0000 7f43c1e00700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43bc077780 0x7f43bc19d590 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f43a800e9f0 tx=0x7f43a800ed00 comp rx=0 tx=0).stop 2024-05-27T09:34:14.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.814+0000 7f43c1e00700 1 -- 172.21.15.44:0/2979172908 shutdown_connections 2024-05-27T09:34:14.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.814+0000 7f43c1e00700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f43a00b2880 0x7f43a00b4d90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.814+0000 7f43c1e00700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43bc110b70 0x7f43bc1a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.814+0000 7f43c1e00700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43bc076e30 0x7f43bc19cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.814+0000 7f43c1e00700 1 --2- 172.21.15.44:0/2979172908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43bc077780 0x7f43bc19d590 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:14.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.814+0000 7f43c1e00700 1 -- 172.21.15.44:0/2979172908 >> 172.21.15.44:0/2979172908 conn(0x7f43bc0ffe90 msgr2=0x7f43bc100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:14.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.815+0000 7f43c1e00700 1 -- 172.21.15.44:0/2979172908 shutdown_connections 2024-05-27T09:34:14.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:14.815+0000 7f43c1e00700 1 -- 172.21.15.44:0/2979172908 wait complete. 2024-05-27T09:34:15.772 INFO:tasks.cephadm:Setup complete, yielding 2024-05-27T09:34:15.772 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-27T09:34:15.782 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-05-27T09:34:15.783 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch status' 2024-05-27T09:34:15.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:15 smithi044 ceph-mon[31618]: pgmap v410: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:16.253 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:15 smithi050 ceph-mon[36371]: pgmap v410: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:16.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:15 smithi189 ceph-mon[36671]: pgmap v410: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:18.142 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:17 smithi044 ceph-mon[31618]: pgmap v411: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:18.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:17 smithi050 ceph-mon[36371]: pgmap v411: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:18.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:17 smithi189 ceph-mon[36671]: pgmap v411: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:18.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.497+0000 7f2fa7000700 1 -- 172.21.15.44:0/1472717041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa0108010 msgr2=0x7f2fa010e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:18.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.497+0000 7f2fa7000700 1 --2- 172.21.15.44:0/1472717041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa0108010 0x7f2fa010e8c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800b0a0 tx=0x7f2f9800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:34:18.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.497+0000 7f2fa7000700 1 -- 172.21.15.44:0/1472717041 shutdown_connections 2024-05-27T09:34:18.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.497+0000 7f2fa7000700 1 --2- 172.21.15.44:0/1472717041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa0108010 0x7f2fa010e8c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:18.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.497+0000 7f2fa7000700 1 --2- 172.21.15.44:0/1472717041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa0106540 0x7f2fa0106950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:18.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.497+0000 7f2fa7000700 1 --2- 172.21.15.44:0/1472717041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fa0107670 0x7f2fa0107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:18.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.497+0000 7f2fa7000700 1 -- 172.21.15.44:0/1472717041 >> 172.21.15.44:0/1472717041 conn(0x7f2fa0101b80 msgr2=0x7f2fa0104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:18.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.497+0000 7f2fa7000700 1 -- 172.21.15.44:0/1472717041 shutdown_connections 2024-05-27T09:34:18.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.497+0000 7f2fa7000700 1 -- 172.21.15.44:0/1472717041 wait complete. 2024-05-27T09:34:18.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.498+0000 7f2fa7000700 1 Processor -- start 2024-05-27T09:34:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.499+0000 7f2fa7000700 1 -- start start 2024-05-27T09:34:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.499+0000 7f2fa7000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa0106540 0x7f2fa019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.499+0000 7f2fa7000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fa0107670 0x7f2fa019cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.499+0000 7f2fa7000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa0108010 0x7f2fa01a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.499+0000 7f2fa7000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa0111fb0 con 0x7f2fa0108010 2024-05-27T09:34:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.499+0000 7f2fa7000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa0111e30 con 0x7f2fa0106540 2024-05-27T09:34:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.499+0000 7f2fa7000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa0112130 con 0x7f2fa0107670 2024-05-27T09:34:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.500+0000 7f2f9fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa0106540 0x7f2fa019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.500+0000 7f2f9fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa0106540 0x7f2fa019ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45610/0 (socket says 172.21.15.44:45610) 2024-05-27T09:34:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.500+0000 7f2f9fe00700 1 -- 172.21.15.44:0/3215685536 learned_addr learned my addr 172.21.15.44:0/3215685536 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:18.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.500+0000 7f2f9f400700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fa0107670 0x7f2fa019cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.500+0000 7f2fa5000700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa0108010 0x7f2fa01a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.500+0000 7f2f9fe00700 1 -- 172.21.15.44:0/3215685536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fa0107670 msgr2=0x7f2fa019cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.500+0000 7f2f9fe00700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fa0107670 0x7f2fa019cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.500+0000 7f2f9fe00700 1 -- 172.21.15.44:0/3215685536 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa0108010 msgr2=0x7f2fa01a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.500+0000 7f2f9fe00700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa0108010 0x7f2fa01a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:18.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.500+0000 7f2f9fe00700 1 -- 172.21.15.44:0/3215685536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa01a45d0 con 0x7f2fa0106540 2024-05-27T09:34:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.500+0000 7f2f9fe00700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa0106540 0x7f2fa019ca70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000d950 tx=0x7f2f9000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.501+0000 7f2f9cc00700 1 -- 172.21.15.44:0/3215685536 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90017070 con 0x7f2fa0106540 2024-05-27T09:34:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.501+0000 7f2f9cc00700 1 -- 172.21.15.44:0/3215685536 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f9000f690 con 0x7f2fa0106540 2024-05-27T09:34:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.501+0000 7f2fa7000700 1 -- 172.21.15.44:0/3215685536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa01a48c0 con 0x7f2fa0106540 2024-05-27T09:34:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.501+0000 7f2f9cc00700 1 -- 172.21.15.44:0/3215685536 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90003aa0 con 0x7f2fa0106540 2024-05-27T09:34:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.501+0000 7f2fa7000700 1 -- 172.21.15.44:0/3215685536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa01a4e00 con 0x7f2fa0106540 2024-05-27T09:34:18.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.502+0000 7f2fa7000700 1 -- 172.21.15.44:0/3215685536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa00512b0 con 0x7f2fa0106540 2024-05-27T09:34:18.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.505+0000 7f2f9cc00700 1 -- 172.21.15.44:0/3215685536 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f2f9000f840 con 0x7f2fa0106540 2024-05-27T09:34:18.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.506+0000 7f2f9cc00700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f840b2880 0x7f2f840b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:18.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.506+0000 7f2f9cc00700 1 -- 172.21.15.44:0/3215685536 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f2f900e24b0 con 0x7f2fa0106540 2024-05-27T09:34:18.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.507+0000 7f2f9f400700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f840b2880 0x7f2f840b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:18.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.507+0000 7f2f9f400700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f840b2880 0x7f2f840b4d90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2fa019db20 tx=0x7f2f8c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:18.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.508+0000 7f2f9cc00700 1 -- 172.21.15.44:0/3215685536 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2f90020030 con 0x7f2fa0106540 2024-05-27T09:34:18.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.767+0000 7f2fa7000700 1 -- 172.21.15.44:0/3215685536 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f2fa0063bf0 con 0x7f2f840b2880 2024-05-27T09:34:18.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.769+0000 7f2f9cc00700 1 -- 172.21.15.44:0/3215685536 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f2fa0063bf0 con 0x7f2f840b2880 2024-05-27T09:34:18.771 INFO:teuthology.orchestra.run.smithi044.stdout:Backend: cephadm 2024-05-27T09:34:18.772 INFO:teuthology.orchestra.run.smithi044.stdout:Available: Yes 2024-05-27T09:34:18.772 INFO:teuthology.orchestra.run.smithi044.stdout:Paused: No 2024-05-27T09:34:18.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.772+0000 7f2fa7000700 1 -- 172.21.15.44:0/3215685536 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f840b2880 msgr2=0x7f2f840b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:18.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.772+0000 7f2fa7000700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f840b2880 0x7f2f840b4d90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2fa019db20 tx=0x7f2f8c00b410 comp rx=0 tx=0).stop 2024-05-27T09:34:18.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.772+0000 7f2fa7000700 1 -- 172.21.15.44:0/3215685536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa0106540 msgr2=0x7f2fa019ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:18.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.772+0000 7f2fa7000700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa0106540 0x7f2fa019ca70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000d950 tx=0x7f2f9000dc60 comp rx=0 tx=0).stop 2024-05-27T09:34:18.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.772+0000 7f2fa7000700 1 -- 172.21.15.44:0/3215685536 shutdown_connections 2024-05-27T09:34:18.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.772+0000 7f2fa7000700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f2f840b2880 0x7f2f840b4d90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:18.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.772+0000 7f2fa7000700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fa0108010 0x7f2fa01a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:18.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.772+0000 7f2fa7000700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa0106540 0x7f2fa019ca70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:18.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.772+0000 7f2fa7000700 1 --2- 172.21.15.44:0/3215685536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fa0107670 0x7f2fa019cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:18.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.772+0000 7f2fa7000700 1 -- 172.21.15.44:0/3215685536 >> 172.21.15.44:0/3215685536 conn(0x7f2fa0101b80 msgr2=0x7f2fa010a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:18.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.773+0000 7f2fa7000700 1 -- 172.21.15.44:0/3215685536 shutdown_connections 2024-05-27T09:34:18.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:18.773+0000 7f2fa7000700 1 -- 172.21.15.44:0/3215685536 wait complete. 2024-05-27T09:34:19.649 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-27T09:34:20.134 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:20.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:19 smithi044 ceph-mon[31618]: pgmap v412: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:20.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:19 smithi044 ceph-mon[31618]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:19 smithi050 ceph-mon[36371]: pgmap v412: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:19 smithi050 ceph-mon[36371]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:20.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:19 smithi189 ceph-mon[36671]: pgmap v412: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:20.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:19 smithi189 ceph-mon[36671]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:22.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:21 smithi044 ceph-mon[31618]: pgmap v413: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:21 smithi050 ceph-mon[36371]: pgmap v413: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.314+0000 7f6bdf400700 1 -- 172.21.15.44:0/2115136523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bd8103230 msgr2=0x7f6bd8103640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.314+0000 7f6bdf400700 1 --2- 172.21.15.44:0/2115136523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bd8103230 0x7f6bd8103640 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009990 tx=0x7f6bcc009ca0 comp rx=0 tx=0).stop 2024-05-27T09:34:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.314+0000 7f6bdf400700 1 -- 172.21.15.44:0/2115136523 shutdown_connections 2024-05-27T09:34:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.314+0000 7f6bdf400700 1 --2- 172.21.15.44:0/2115136523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd8104d00 0x7f6bd810b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.314+0000 7f6bdf400700 1 --2- 172.21.15.44:0/2115136523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bd8103230 0x7f6bd8103640 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.314+0000 7f6bdf400700 1 --2- 172.21.15.44:0/2115136523 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bd8104360 0x7f6bd8104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.314+0000 7f6bdf400700 1 -- 172.21.15.44:0/2115136523 >> 172.21.15.44:0/2115136523 conn(0x7f6bd80fe870 msgr2=0x7f6bd8100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.314+0000 7f6bdf400700 1 -- 172.21.15.44:0/2115136523 shutdown_connections 2024-05-27T09:34:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.315+0000 7f6bdf400700 1 -- 172.21.15.44:0/2115136523 wait complete. 2024-05-27T09:34:22.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.316+0000 7f6bdf400700 1 Processor -- start 2024-05-27T09:34:22.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.316+0000 7f6bdf400700 1 -- start start 2024-05-27T09:34:22.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.317+0000 7f6bdf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd8103230 0x7f6bd8197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.317+0000 7f6bdf400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bd8104360 0x7f6bd81980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.317+0000 7f6bdf400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bd8104d00 0x7f6bd8194710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.317+0000 7f6bdf400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd810ec80 con 0x7f6bd8104360 2024-05-27T09:34:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.317+0000 7f6bdf400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd810eb00 con 0x7f6bd8104d00 2024-05-27T09:34:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.317+0000 7f6bdf400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd810ee00 con 0x7f6bd8103230 2024-05-27T09:34:22.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.317+0000 7f6bdca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd8103230 0x7f6bd8197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:22.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.317+0000 7f6bdca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd8103230 0x7f6bd8197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:46036/0 (socket says 172.21.15.44:46036) 2024-05-27T09:34:22.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.317+0000 7f6bdca00700 1 -- 172.21.15.44:0/2940813070 learned_addr learned my addr 172.21.15.44:0/2940813070 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:22.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.317+0000 7f6bd7e00700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bd8104360 0x7f6bd81980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:22.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.318+0000 7f6bdd400700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bd8104d00 0x7f6bd8194710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:22.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.318+0000 7f6bdca00700 1 -- 172.21.15.44:0/2940813070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bd8104d00 msgr2=0x7f6bd8194710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:22.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.318+0000 7f6bdca00700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bd8104d00 0x7f6bd8194710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:22.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.318+0000 7f6bdca00700 1 -- 172.21.15.44:0/2940813070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bd8104360 msgr2=0x7f6bd81980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:22.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.318+0000 7f6bdca00700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bd8104360 0x7f6bd81980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:22.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.318+0000 7f6bdca00700 1 -- 172.21.15.44:0/2940813070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd8194f40 con 0x7f6bd8103230 2024-05-27T09:34:22.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.318+0000 7f6bdca00700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd8103230 0x7f6bd8197b90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00b6c0 tx=0x7f6bcc009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:22.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.319+0000 7f6bd5600700 1 -- 172.21.15.44:0/2940813070 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc026070 con 0x7f6bd8103230 2024-05-27T09:34:22.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.319+0000 7f6bdf400700 1 -- 172.21.15.44:0/2940813070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd81951d0 con 0x7f6bd8103230 2024-05-27T09:34:22.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.319+0000 7f6bdf400700 1 -- 172.21.15.44:0/2940813070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd81a9230 con 0x7f6bd8103230 2024-05-27T09:34:22.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.319+0000 7f6bd5600700 1 -- 172.21.15.44:0/2940813070 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bcc0173f0 con 0x7f6bd8103230 2024-05-27T09:34:22.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.319+0000 7f6bd5600700 1 -- 172.21.15.44:0/2940813070 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc030060 con 0x7f6bd8103230 2024-05-27T09:34:22.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.320+0000 7f6bdf400700 1 -- 172.21.15.44:0/2940813070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bb8005330 con 0x7f6bd8103230 2024-05-27T09:34:22.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.323+0000 7f6bd5600700 1 -- 172.21.15.44:0/2940813070 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f6bcc029070 con 0x7f6bd8103230 2024-05-27T09:34:22.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.324+0000 7f6bd5600700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6bc00b2880 0x7f6bc00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:22.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.324+0000 7f6bd5600700 1 -- 172.21.15.44:0/2940813070 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f6bcc0e3150 con 0x7f6bd8103230 2024-05-27T09:34:22.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.325+0000 7f6bd7e00700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6bc00b2880 0x7f6bc00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:22.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.325+0000 7f6bd7e00700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6bc00b2880 0x7f6bc00b4d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8195780 tx=0x7f6bc4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:22.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.327+0000 7f6bd5600700 1 -- 172.21.15.44:0/2940813070 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6bcc013070 con 0x7f6bd8103230 2024-05-27T09:34:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:21 smithi189 ceph-mon[36671]: pgmap v413: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:22.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.573+0000 7f6bdf400700 1 -- 172.21.15.44:0/2940813070 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f6bb8000bc0 con 0x7f6bc00b2880 2024-05-27T09:34:22.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.615+0000 7f6bd5600700 1 -- 172.21.15.44:0/2940813070 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f6bb8000bc0 con 0x7f6bc00b2880 2024-05-27T09:34:22.617 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-27T09:34:22.617 INFO:teuthology.orchestra.run.smithi044.stdout:mds.a smithi044 running (3m) 44s ago 3m 80.4M - 19.0.0-3868-ga6507388 850a06a84d28 7658c9f94714 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mds.b smithi050 running (3m) 59s ago 3m 93.9M - 19.0.0-3868-ga6507388 850a06a84d28 8b7b22588b73 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mds.c smithi189 running (3m) 59s ago 3m 89.1M - 19.0.0-3868-ga6507388 850a06a84d28 9889fd83bbe8 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mds.d smithi044 running (3m) 44s ago 3m 80.6M - 19.0.0-3868-ga6507388 850a06a84d28 dd646d323f4b 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mds.e smithi050 running (3m) 59s ago 3m 89.1M - 19.0.0-3868-ga6507388 850a06a84d28 9e88f59da4be 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mds.f smithi189 running (3m) 59s ago 3m 87.2M - 19.0.0-3868-ga6507388 850a06a84d28 4d1bcbfcf9a8 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mds.g smithi044 running (3m) 44s ago 3m 82.7M - 19.0.0-3868-ga6507388 850a06a84d28 35e81d080ca2 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mds.h smithi050 running (3m) 59s ago 3m 86.0M - 19.0.0-3868-ga6507388 850a06a84d28 9d1fd8d5f05a 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mds.i smithi189 running (3m) 59s ago 3m 89.7M - 19.0.0-3868-ga6507388 850a06a84d28 05084cdaaf8e 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mds.j smithi044 running (3m) 44s ago 3m 80.7M - 19.0.0-3868-ga6507388 850a06a84d28 b861e9014d39 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mds.k smithi050 running (3m) 59s ago 3m 84.2M - 19.0.0-3868-ga6507388 850a06a84d28 166e6764c810 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mds.l smithi189 running (3m) 59s ago 3m 84.7M - 19.0.0-3868-ga6507388 850a06a84d28 2f8c7b3cbe98 2024-05-27T09:34:22.618 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi044 *:9283,8765 running (12m) 44s ago 12m 600M - 19.0.0-3868-ga6507388 850a06a84d28 310da0308fc5 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi050 *:8443,8765 running (10m) 59s ago 10m 535M - 19.0.0-3868-ga6507388 850a06a84d28 5211fc522376 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.z smithi189 *:8443,8765 running (10m) 59s ago 10m 537M - 19.0.0-3868-ga6507388 850a06a84d28 672bcb45c166 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (12m) 44s ago 12m 117M 2048M 19.0.0-3868-ga6507388 850a06a84d28 b84ccfae169a 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi050 running (10m) 59s ago 10m 108M 2048M 19.0.0-3868-ga6507388 850a06a84d28 c51b05fd7b42 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi189 running (10m) 59s ago 10m 110M 2048M 19.0.0-3868-ga6507388 850a06a84d28 f21d35a4925a 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (9m) 44s ago 9m 115M 4273M 19.0.0-3868-ga6507388 850a06a84d28 97e508c50010 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi050 running (9m) 59s ago 9m 116M 4273M 19.0.0-3868-ga6507388 850a06a84d28 52651e44b02c 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi189 running (8m) 59s ago 8m 128M 4272M 19.0.0-3868-ga6507388 850a06a84d28 a162344dbf31 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (8m) 44s ago 8m 114M 4273M 19.0.0-3868-ga6507388 850a06a84d28 0831d46132e9 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi050 running (7m) 59s ago 7m 112M 4273M 19.0.0-3868-ga6507388 850a06a84d28 a872924d2ef5 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi189 running (7m) 59s ago 7m 121M 4272M 19.0.0-3868-ga6507388 850a06a84d28 03afd35c8e2d 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi044 running (6m) 44s ago 6m 119M 4273M 19.0.0-3868-ga6507388 850a06a84d28 ef24bd158be4 2024-05-27T09:34:22.619 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi050 running (6m) 59s ago 6m 113M 4273M 19.0.0-3868-ga6507388 850a06a84d28 412cb68def25 2024-05-27T09:34:22.620 INFO:teuthology.orchestra.run.smithi044.stdout:osd.8 smithi189 running (5m) 59s ago 5m 113M 4272M 19.0.0-3868-ga6507388 850a06a84d28 240f01128c5a 2024-05-27T09:34:22.620 INFO:teuthology.orchestra.run.smithi044.stdout:osd.9 smithi044 running (5m) 44s ago 5m 113M 4273M 19.0.0-3868-ga6507388 850a06a84d28 011df4158fc9 2024-05-27T09:34:22.620 INFO:teuthology.orchestra.run.smithi044.stdout:osd.10 smithi050 running (4m) 59s ago 4m 122M 4273M 19.0.0-3868-ga6507388 850a06a84d28 5406196eefce 2024-05-27T09:34:22.620 INFO:teuthology.orchestra.run.smithi044.stdout:osd.11 smithi189 running (4m) 59s ago 4m 119M 4272M 19.0.0-3868-ga6507388 850a06a84d28 5b38665e141f 2024-05-27T09:34:22.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.619+0000 7f6bdf400700 1 -- 172.21.15.44:0/2940813070 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6bc00b2880 msgr2=0x7f6bc00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.619+0000 7f6bdf400700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6bc00b2880 0x7f6bc00b4d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8195780 tx=0x7f6bc4009260 comp rx=0 tx=0).stop 2024-05-27T09:34:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.619+0000 7f6bdf400700 1 -- 172.21.15.44:0/2940813070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd8103230 msgr2=0x7f6bd8197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.619+0000 7f6bdf400700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd8103230 0x7f6bd8197b90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00b6c0 tx=0x7f6bcc009580 comp rx=0 tx=0).stop 2024-05-27T09:34:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.619+0000 7f6bdf400700 1 -- 172.21.15.44:0/2940813070 shutdown_connections 2024-05-27T09:34:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.619+0000 7f6bdf400700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f6bc00b2880 0x7f6bc00b4d90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.619+0000 7f6bdf400700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bd8104d00 0x7f6bd8194710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:22.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.619+0000 7f6bdf400700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd8103230 0x7f6bd8197b90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:22.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.619+0000 7f6bdf400700 1 --2- 172.21.15.44:0/2940813070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bd8104360 0x7f6bd81980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:22.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.619+0000 7f6bdf400700 1 -- 172.21.15.44:0/2940813070 >> 172.21.15.44:0/2940813070 conn(0x7f6bd80fe870 msgr2=0x7f6bd8100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:22.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.620+0000 7f6bdf400700 1 -- 172.21.15.44:0/2940813070 shutdown_connections 2024-05-27T09:34:22.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:22.620+0000 7f6bdf400700 1 -- 172.21.15.44:0/2940813070 wait complete. 2024-05-27T09:34:23.459 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-27T09:34:23.930 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:24.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:23 smithi044 ceph-mon[31618]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:24.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:23 smithi044 ceph-mon[31618]: pgmap v414: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:23 smithi050 ceph-mon[36371]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:24.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:23 smithi050 ceph-mon[36371]: pgmap v414: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:23 smithi189 ceph-mon[36671]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:23 smithi189 ceph-mon[36671]: pgmap v414: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:25.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.968+0000 7f8fcd800700 1 -- 172.21.15.44:0/3488864608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fc8106540 msgr2=0x7f8fc8106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:25.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.968+0000 7f8fcd800700 1 --2- 172.21.15.44:0/3488864608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fc8106540 0x7f8fc8106950 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009f90 tx=0x7f8fb800b330 comp rx=0 tx=0).stop 2024-05-27T09:34:25.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:25 smithi044 ceph-mon[31618]: pgmap v415: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:25.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.968+0000 7f8fcd800700 1 -- 172.21.15.44:0/3488864608 shutdown_connections 2024-05-27T09:34:25.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.968+0000 7f8fcd800700 1 --2- 172.21.15.44:0/3488864608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fc8108010 0x7f8fc810e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:25.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.968+0000 7f8fcd800700 1 --2- 172.21.15.44:0/3488864608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fc8106540 0x7f8fc8106950 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.968+0000 7f8fcd800700 1 --2- 172.21.15.44:0/3488864608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fc8107670 0x7f8fc8107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.968+0000 7f8fcd800700 1 -- 172.21.15.44:0/3488864608 >> 172.21.15.44:0/3488864608 conn(0x7f8fc8101b80 msgr2=0x7f8fc8104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.968+0000 7f8fcd800700 1 -- 172.21.15.44:0/3488864608 shutdown_connections 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.968+0000 7f8fcd800700 1 -- 172.21.15.44:0/3488864608 wait complete. 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.970+0000 7f8fcd800700 1 Processor -- start 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.970+0000 7f8fcd800700 1 -- start start 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.970+0000 7f8fcd800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fc8106540 0x7f8fc819cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.970+0000 7f8fcd800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fc8107670 0x7f8fc819d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.970+0000 7f8fcd800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fc8108010 0x7f8fc81a4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.970+0000 7f8fcd800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc8111fe0 con 0x7f8fc8106540 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.970+0000 7f8fcd800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc8111e60 con 0x7f8fc8107670 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.970+0000 7f8fcd800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc8112160 con 0x7f8fc8108010 2024-05-27T09:34:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.971+0000 7f8fc6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fc8106540 0x7f8fc819cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.971+0000 7f8fbe000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fc8107670 0x7f8fc819d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.971+0000 7f8fc6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fc8106540 0x7f8fc819cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46276/0 (socket says 172.21.15.44:46276) 2024-05-27T09:34:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.971+0000 7f8fc6a00700 1 -- 172.21.15.44:0/104821022 learned_addr learned my addr 172.21.15.44:0/104821022 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.971+0000 7f8fc7400700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fc8108010 0x7f8fc81a4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:25.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.971+0000 7f8fbe000700 1 -- 172.21.15.44:0/104821022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fc8108010 msgr2=0x7f8fc81a4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:25.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.971+0000 7f8fbe000700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fc8108010 0x7f8fc81a4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:25.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.971+0000 7f8fbe000700 1 -- 172.21.15.44:0/104821022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fc8106540 msgr2=0x7f8fc819cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:25.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.971+0000 7f8fbe000700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fc8106540 0x7f8fc819cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:25.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.971+0000 7f8fbe000700 1 -- 172.21.15.44:0/104821022 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc81a46b0 con 0x7f8fc8107670 2024-05-27T09:34:25.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.971+0000 7f8fc7400700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fc8108010 0x7f8fc81a4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:34:25.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.972+0000 7f8fbe000700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fc8107670 0x7f8fc819d090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8fb000e9c0 tx=0x7f8fb000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:25.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.972+0000 7f8fbfe00700 1 -- 172.21.15.44:0/104821022 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb000c9c0 con 0x7f8fc8107670 2024-05-27T09:34:25.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.972+0000 7f8fbfe00700 1 -- 172.21.15.44:0/104821022 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fb000cb60 con 0x7f8fc8107670 2024-05-27T09:34:25.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.972+0000 7f8fbfe00700 1 -- 172.21.15.44:0/104821022 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0018680 con 0x7f8fc8107670 2024-05-27T09:34:25.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.972+0000 7f8fcd800700 1 -- 172.21.15.44:0/104821022 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc81a49a0 con 0x7f8fc8107670 2024-05-27T09:34:25.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.972+0000 7f8fcd800700 1 -- 172.21.15.44:0/104821022 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fc81a4e90 con 0x7f8fc8107670 2024-05-27T09:34:25.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.973+0000 7f8fcd800700 1 -- 172.21.15.44:0/104821022 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fc80512b0 con 0x7f8fc8107670 2024-05-27T09:34:25.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.976+0000 7f8fbfe00700 1 -- 172.21.15.44:0/104821022 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f8fb0010430 con 0x7f8fc8107670 2024-05-27T09:34:25.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.977+0000 7f8fbfe00700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8fac0b2880 0x7f8fac0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:25.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.978+0000 7f8fbfe00700 1 -- 172.21.15.44:0/104821022 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f8fb00e27a0 con 0x7f8fc8107670 2024-05-27T09:34:25.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.978+0000 7f8fc6a00700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8fac0b2880 0x7f8fac0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:25.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.978+0000 7f8fc6a00700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8fac0b2880 0x7f8fac0b4d90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009550 tx=0x7f8fb80096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:25.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:25.980+0000 7f8fbfe00700 1 -- 172.21.15.44:0/104821022 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8fb00aa470 con 0x7f8fc8107670 2024-05-27T09:34:26.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.229+0000 7f8fcd800700 1 -- 172.21.15.44:0/104821022 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f8fc8063bf0 con 0x7f8fac0b2880 2024-05-27T09:34:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.239+0000 7f8fbfe00700 1 -- 172.21.15.44:0/104821022 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f8fc8063bf0 con 0x7f8fac0b2880 2024-05-27T09:34:26.241 INFO:teuthology.orchestra.run.smithi044.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-27T09:34:26.241 INFO:teuthology.orchestra.run.smithi044.stdout:mds.all 12/12 63s ago 4m smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-05-27T09:34:26.241 INFO:teuthology.orchestra.run.smithi044.stdout:mgr 3/3 63s ago 10m smithi044=x;smithi050=y;smithi189=z;count:3 2024-05-27T09:34:26.241 INFO:teuthology.orchestra.run.smithi044.stdout:mon 3/3 63s ago 10m smithi044:172.21.15.44=a;smithi050:172.21.15.50=b;smithi189:172.21.15.189=c;count:3 2024-05-27T09:34:26.242 INFO:teuthology.orchestra.run.smithi044.stdout:osd 12 63s ago - 2024-05-27T09:34:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.242+0000 7f8fcd800700 1 -- 172.21.15.44:0/104821022 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8fac0b2880 msgr2=0x7f8fac0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.243+0000 7f8fcd800700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8fac0b2880 0x7f8fac0b4d90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009550 tx=0x7f8fb80096c0 comp rx=0 tx=0).stop 2024-05-27T09:34:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.243+0000 7f8fcd800700 1 -- 172.21.15.44:0/104821022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fc8107670 msgr2=0x7f8fc819d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.243+0000 7f8fcd800700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fc8107670 0x7f8fc819d090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8fb000e9c0 tx=0x7f8fb000ecd0 comp rx=0 tx=0).stop 2024-05-27T09:34:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.243+0000 7f8fcd800700 1 -- 172.21.15.44:0/104821022 shutdown_connections 2024-05-27T09:34:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.243+0000 7f8fcd800700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f8fac0b2880 0x7f8fac0b4d90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.243+0000 7f8fcd800700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fc8108010 0x7f8fc81a4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.243+0000 7f8fcd800700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fc8106540 0x7f8fc819cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.243+0000 7f8fcd800700 1 --2- 172.21.15.44:0/104821022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fc8107670 0x7f8fc819d090 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.243+0000 7f8fcd800700 1 -- 172.21.15.44:0/104821022 >> 172.21.15.44:0/104821022 conn(0x7f8fc8101b80 msgr2=0x7f8fc810a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.243+0000 7f8fcd800700 1 -- 172.21.15.44:0/104821022 shutdown_connections 2024-05-27T09:34:26.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:26.243+0000 7f8fcd800700 1 -- 172.21.15.44:0/104821022 wait complete. 2024-05-27T09:34:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:25 smithi050 ceph-mon[36371]: pgmap v415: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:26.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:25 smithi189 ceph-mon[36671]: pgmap v415: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:26.965 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-27T09:34:27.442 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:27 smithi044 ceph-mon[31618]: from='client.24944 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:27 smithi044 ceph-mon[31618]: pgmap v416: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:27 smithi050 ceph-mon[36371]: from='client.24944 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:27 smithi050 ceph-mon[36371]: pgmap v416: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:28.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:27 smithi189 ceph-mon[36671]: from='client.24944 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:28.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:27 smithi189 ceph-mon[36671]: pgmap v416: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:29.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.540+0000 7fae72c00700 1 -- 172.21.15.44:0/231984630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae6c103220 msgr2=0x7fae6c103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:29.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.540+0000 7fae72c00700 1 --2- 172.21.15.44:0/231984630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae6c103220 0x7fae6c103630 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fae5c009990 tx=0x7fae5c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:34:29.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.540+0000 7fae72c00700 1 -- 172.21.15.44:0/231984630 shutdown_connections 2024-05-27T09:34:29.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.540+0000 7fae72c00700 1 --2- 172.21.15.44:0/231984630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae6c104cf0 0x7fae6c10b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:29.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.540+0000 7fae72c00700 1 --2- 172.21.15.44:0/231984630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae6c103220 0x7fae6c103630 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:29.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.540+0000 7fae72c00700 1 --2- 172.21.15.44:0/231984630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae6c104350 0x7fae6c104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:29.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.540+0000 7fae72c00700 1 -- 172.21.15.44:0/231984630 >> 172.21.15.44:0/231984630 conn(0x7fae6c0fe860 msgr2=0x7fae6c100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:29.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.540+0000 7fae72c00700 1 -- 172.21.15.44:0/231984630 shutdown_connections 2024-05-27T09:34:29.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.540+0000 7fae72c00700 1 -- 172.21.15.44:0/231984630 wait complete. 2024-05-27T09:34:29.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.542+0000 7fae72c00700 1 Processor -- start 2024-05-27T09:34:29.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.542+0000 7fae72c00700 1 -- start start 2024-05-27T09:34:29.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.543+0000 7fae72c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae6c103220 0x7fae6c106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:29.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.543+0000 7fae72c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae6c104350 0x7fae6c105690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:29.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.543+0000 7fae72c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae6c104cf0 0x7fae6c105bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:29.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.543+0000 7fae72c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c10ecb0 con 0x7fae6c104350 2024-05-27T09:34:29.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.543+0000 7fae72c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c10eb30 con 0x7fae6c103220 2024-05-27T09:34:29.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.543+0000 7fae72c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c10ee30 con 0x7fae6c104cf0 2024-05-27T09:34:29.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.543+0000 7fae6be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae6c103220 0x7fae6c106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:29.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.543+0000 7fae6be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae6c103220 0x7fae6c106fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34838/0 (socket says 172.21.15.44:34838) 2024-05-27T09:34:29.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.543+0000 7fae6be00700 1 -- 172.21.15.44:0/1254634952 learned_addr learned my addr 172.21.15.44:0/1254634952 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:29.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.543+0000 7fae63e00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae6c104350 0x7fae6c105690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:29.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.543+0000 7fae6be00700 1 -- 172.21.15.44:0/1254634952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae6c104cf0 msgr2=0x7fae6c105bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:34:29.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.544+0000 7fae6be00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae6c104cf0 0x7fae6c105bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:29.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.544+0000 7fae6be00700 1 -- 172.21.15.44:0/1254634952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae6c104350 msgr2=0x7fae6c105690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:29.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.544+0000 7fae6be00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae6c104350 0x7fae6c105690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:29.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.544+0000 7fae6be00700 1 -- 172.21.15.44:0/1254634952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6c106330 con 0x7fae6c103220 2024-05-27T09:34:29.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.544+0000 7fae63e00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae6c104350 0x7fae6c105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:34:29.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.544+0000 7fae6be00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae6c103220 0x7fae6c106fe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00b6c0 tx=0x7fae5c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:29.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.544+0000 7fae69600700 1 -- 172.21.15.44:0/1254634952 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c026070 con 0x7fae6c103220 2024-05-27T09:34:29.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.545+0000 7fae72c00700 1 -- 172.21.15.44:0/1254634952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae6c1a4a70 con 0x7fae6c103220 2024-05-27T09:34:29.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.545+0000 7fae69600700 1 -- 172.21.15.44:0/1254634952 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae5c021410 con 0x7fae6c103220 2024-05-27T09:34:29.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.545+0000 7fae69600700 1 -- 172.21.15.44:0/1254634952 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c017650 con 0x7fae6c103220 2024-05-27T09:34:29.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.545+0000 7fae72c00700 1 -- 172.21.15.44:0/1254634952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae6c1a4e60 con 0x7fae6c103220 2024-05-27T09:34:29.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.546+0000 7fae72c00700 1 -- 172.21.15.44:0/1254634952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae6c0512b0 con 0x7fae6c103220 2024-05-27T09:34:29.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.549+0000 7fae69600700 1 -- 172.21.15.44:0/1254634952 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fae5c029070 con 0x7fae6c103220 2024-05-27T09:34:29.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.550+0000 7fae69600700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fae500b2900 0x7fae500b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:29.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.550+0000 7fae69600700 1 -- 172.21.15.44:0/1254634952 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fae5c0e3520 con 0x7fae6c103220 2024-05-27T09:34:29.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.550+0000 7fae63e00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fae500b2900 0x7fae500b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:29.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.551+0000 7fae63e00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fae500b2900 0x7fae500b4e10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fae6c106990 tx=0x7fae54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:29.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.552+0000 7fae69600700 1 -- 172.21.15.44:0/1254634952 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fae5c030080 con 0x7fae6c103220 2024-05-27T09:34:29.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.799+0000 7fae72c00700 1 -- 172.21.15.44:0/1254634952 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fae6c100910 con 0x7fae500b2900 2024-05-27T09:34:29.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.801+0000 7fae69600700 1 -- 172.21.15.44:0/1254634952 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fae6c100910 con 0x7fae500b2900 2024-05-27T09:34:29.802 INFO:teuthology.orchestra.run.smithi044.stdout:HOST ADDR LABELS STATUS 2024-05-27T09:34:29.802 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044 172.21.15.44 2024-05-27T09:34:29.802 INFO:teuthology.orchestra.run.smithi044.stdout:smithi050 172.21.15.50 2024-05-27T09:34:29.802 INFO:teuthology.orchestra.run.smithi044.stdout:smithi189 172.21.15.189 2024-05-27T09:34:29.802 INFO:teuthology.orchestra.run.smithi044.stdout:3 hosts in cluster 2024-05-27T09:34:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.804+0000 7fae72c00700 1 -- 172.21.15.44:0/1254634952 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fae500b2900 msgr2=0x7fae500b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.804+0000 7fae72c00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fae500b2900 0x7fae500b4e10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fae6c106990 tx=0x7fae54009260 comp rx=0 tx=0).stop 2024-05-27T09:34:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.804+0000 7fae72c00700 1 -- 172.21.15.44:0/1254634952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae6c103220 msgr2=0x7fae6c106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.804+0000 7fae72c00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae6c103220 0x7fae6c106fe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00b6c0 tx=0x7fae5c009580 comp rx=0 tx=0).stop 2024-05-27T09:34:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.804+0000 7fae72c00700 1 -- 172.21.15.44:0/1254634952 shutdown_connections 2024-05-27T09:34:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.804+0000 7fae72c00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fae500b2900 0x7fae500b4e10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.804+0000 7fae72c00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae6c104cf0 0x7fae6c105bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:29.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.804+0000 7fae72c00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae6c103220 0x7fae6c106fe0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:29.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.804+0000 7fae72c00700 1 --2- 172.21.15.44:0/1254634952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae6c104350 0x7fae6c105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:29.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.804+0000 7fae72c00700 1 -- 172.21.15.44:0/1254634952 >> 172.21.15.44:0/1254634952 conn(0x7fae6c0fe860 msgr2=0x7fae6c100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:29.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.805+0000 7fae72c00700 1 -- 172.21.15.44:0/1254634952 shutdown_connections 2024-05-27T09:34:29.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:29.805+0000 7fae72c00700 1 -- 172.21.15.44:0/1254634952 wait complete. 2024-05-27T09:34:30.110 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:29 smithi044 ceph-mon[31618]: pgmap v417: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:29 smithi050 ceph-mon[36371]: pgmap v417: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:29 smithi189 ceph-mon[36671]: pgmap v417: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:30.533 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-27T09:34:31.009 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:31.179 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:30 smithi044 ceph-mon[31618]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:31.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:30 smithi050 ceph-mon[36371]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:31.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:30 smithi189 ceph-mon[36671]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:32.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:31 smithi044 ceph-mon[31618]: pgmap v418: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:32.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:31 smithi050 ceph-mon[36371]: pgmap v418: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:31 smithi189 ceph-mon[36671]: pgmap v418: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:33.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.136+0000 7fb47b200700 1 -- 172.21.15.44:0/2934301968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb47410a100 msgr2=0x7fb47410c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:33.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.136+0000 7fb47b200700 1 --2- 172.21.15.44:0/2934301968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb47410a100 0x7fb47410c520 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00b0a0 tx=0x7fb46c00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:34:33.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.136+0000 7fb47b200700 1 -- 172.21.15.44:0/2934301968 shutdown_connections 2024-05-27T09:34:33.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.136+0000 7fb47b200700 1 --2- 172.21.15.44:0/2934301968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb47410a100 0x7fb47410c520 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:33.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.136+0000 7fb47b200700 1 --2- 172.21.15.44:0/2934301968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb474104e40 0x7fb474107260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:33.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.136+0000 7fb47b200700 1 --2- 172.21.15.44:0/2934301968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4741077a0 0x7fb474109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:33.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.136+0000 7fb47b200700 1 -- 172.21.15.44:0/2934301968 >> 172.21.15.44:0/2934301968 conn(0x7fb4740fe880 msgr2=0x7fb474100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.136+0000 7fb47b200700 1 -- 172.21.15.44:0/2934301968 shutdown_connections 2024-05-27T09:34:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.137+0000 7fb47b200700 1 -- 172.21.15.44:0/2934301968 wait complete. 2024-05-27T09:34:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.138+0000 7fb47b200700 1 Processor -- start 2024-05-27T09:34:33.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.138+0000 7fb47b200700 1 -- start start 2024-05-27T09:34:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.139+0000 7fb47b200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb474104e40 0x7fb47419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.139+0000 7fb47b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4741077a0 0x7fb47419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.139+0000 7fb47b200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb47410a100 0x7fb4741a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.139+0000 7fb47b200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb47410fde0 con 0x7fb474104e40 2024-05-27T09:34:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.139+0000 7fb47b200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb47410fc60 con 0x7fb47410a100 2024-05-27T09:34:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.139+0000 7fb47b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb47410ff60 con 0x7fb4741077a0 2024-05-27T09:34:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.139+0000 7fb473e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb474104e40 0x7fb47419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.139+0000 7fb473400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4741077a0 0x7fb47419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.139+0000 7fb473400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4741077a0 0x7fb47419ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43298/0 (socket says 172.21.15.44:43298) 2024-05-27T09:34:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.139+0000 7fb473400700 1 -- 172.21.15.44:0/3562509648 learned_addr learned my addr 172.21.15.44:0/3562509648 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.139+0000 7fb479200700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb47410a100 0x7fb4741a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.140+0000 7fb473e00700 1 -- 172.21.15.44:0/3562509648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4741077a0 msgr2=0x7fb47419ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.140+0000 7fb473e00700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4741077a0 0x7fb47419ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.140+0000 7fb473e00700 1 -- 172.21.15.44:0/3562509648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb47410a100 msgr2=0x7fb4741a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.140+0000 7fb473e00700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb47410a100 0x7fb4741a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:33.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.140+0000 7fb473e00700 1 -- 172.21.15.44:0/3562509648 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4741a2400 con 0x7fb474104e40 2024-05-27T09:34:33.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.140+0000 7fb473e00700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb474104e40 0x7fb47419a8a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb46400e9c0 tx=0x7fb46400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:33.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.140+0000 7fb470c00700 1 -- 172.21.15.44:0/3562509648 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46400cb70 con 0x7fb474104e40 2024-05-27T09:34:33.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.140+0000 7fb470c00700 1 -- 172.21.15.44:0/3562509648 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb46400ee60 con 0x7fb474104e40 2024-05-27T09:34:33.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.140+0000 7fb47b200700 1 -- 172.21.15.44:0/3562509648 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4741a26f0 con 0x7fb474104e40 2024-05-27T09:34:33.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.140+0000 7fb470c00700 1 -- 172.21.15.44:0/3562509648 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464003d60 con 0x7fb474104e40 2024-05-27T09:34:33.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.140+0000 7fb47b200700 1 -- 172.21.15.44:0/3562509648 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4741a2c80 con 0x7fb474104e40 2024-05-27T09:34:33.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.142+0000 7fb47b200700 1 -- 172.21.15.44:0/3562509648 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4740512b0 con 0x7fb474104e40 2024-05-27T09:34:33.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.146+0000 7fb470c00700 1 -- 172.21.15.44:0/3562509648 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fb46400cd10 con 0x7fb474104e40 2024-05-27T09:34:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.148+0000 7fb470c00700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb4580b2900 0x7fb4580b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.149+0000 7fb470c00700 1 -- 172.21.15.44:0/3562509648 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fb4640e23b0 con 0x7fb474104e40 2024-05-27T09:34:33.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.149+0000 7fb473400700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb4580b2900 0x7fb4580b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:33.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.150+0000 7fb473400700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb4580b2900 0x7fb4580b4e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb47419b950 tx=0x7fb460009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:33.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.151+0000 7fb470c00700 1 -- 172.21.15.44:0/3562509648 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb4640aa110 con 0x7fb474104e40 2024-05-27T09:34:33.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.397+0000 7fb47b200700 1 -- 172.21.15.44:0/3562509648 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fb474063bf0 con 0x7fb4580b2900 2024-05-27T09:34:33.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.401+0000 7fb470c00700 1 -- 172.21.15.44:0/3562509648 <== mgr.14152 v2:172.21.15.44:6800/4077502151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fb474063bf0 con 0x7fb4580b2900 2024-05-27T09:34:33.402 INFO:teuthology.orchestra.run.smithi044.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-27T09:34:33.402 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001K400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-05-27T09:34:33.402 INFO:teuthology.orchestra.run.smithi044.stdout:smithi050 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513001F480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-05-27T09:34:33.402 INFO:teuthology.orchestra.run.smithi044.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 -- 172.21.15.44:0/3562509648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb4580b2900 msgr2=0x7fb4580b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb4580b2900 0x7fb4580b4e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb47419b950 tx=0x7fb460009260 comp rx=0 tx=0).stop 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 -- 172.21.15.44:0/3562509648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb474104e40 msgr2=0x7fb47419a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb474104e40 0x7fb47419a8a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb46400e9c0 tx=0x7fb46400ecd0 comp rx=0 tx=0).stop 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 -- 172.21.15.44:0/3562509648 shutdown_connections 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fb4580b2900 0x7fb4580b4e10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb47410a100 0x7fb4741a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb474104e40 0x7fb47419a8a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 --2- 172.21.15.44:0/3562509648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4741077a0 0x7fb47419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 -- 172.21.15.44:0/3562509648 >> 172.21.15.44:0/3562509648 conn(0x7fb4740fe880 msgr2=0x7fb474105930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 -- 172.21.15.44:0/3562509648 shutdown_connections 2024-05-27T09:34:33.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:33.404+0000 7fb47b200700 1 -- 172.21.15.44:0/3562509648 wait complete. 2024-05-27T09:34:34.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:33 smithi044 ceph-mon[31618]: pgmap v419: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:34.154 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-27T09:34:34.163 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-05-27T09:34:34.163 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-27T09:34:34.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:33 smithi050 ceph-mon[36371]: pgmap v419: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:34.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:33 smithi189 ceph-mon[36671]: pgmap v419: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:34.644 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:35.113 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:34 smithi044 ceph-mon[31618]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:35.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:34 smithi050 ceph-mon[36371]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:35.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:34 smithi189 ceph-mon[36671]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:34:36.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:35 smithi044 ceph-mon[31618]: pgmap v420: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:36.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:35 smithi050 ceph-mon[36371]: pgmap v420: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:35 smithi189 ceph-mon[36671]: pgmap v420: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:36.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.751+0000 7f592a800700 1 -- 172.21.15.44:0/168362641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59241074a0 msgr2=0x7f59241078b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:36.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.751+0000 7f592a800700 1 --2- 172.21.15.44:0/168362641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59241074a0 0x7f59241078b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5910009990 tx=0x7f5910009ca0 comp rx=0 tx=0).stop 2024-05-27T09:34:36.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.751+0000 7f592a800700 1 -- 172.21.15.44:0/168362641 shutdown_connections 2024-05-27T09:34:36.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.751+0000 7f592a800700 1 --2- 172.21.15.44:0/168362641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5924107e40 0x7f592410e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:36.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.751+0000 7f592a800700 1 --2- 172.21.15.44:0/168362641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5924106370 0x7f5924106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:36.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.751+0000 7f592a800700 1 --2- 172.21.15.44:0/168362641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59241074a0 0x7f59241078b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:36.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.751+0000 7f592a800700 1 -- 172.21.15.44:0/168362641 >> 172.21.15.44:0/168362641 conn(0x7f59241019b0 msgr2=0x7f5924103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:36.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.751+0000 7f592a800700 1 -- 172.21.15.44:0/168362641 shutdown_connections 2024-05-27T09:34:36.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.752+0000 7f592a800700 1 -- 172.21.15.44:0/168362641 wait complete. 2024-05-27T09:34:36.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.753+0000 7f592a800700 1 Processor -- start 2024-05-27T09:34:36.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.753+0000 7f592a800700 1 -- start start 2024-05-27T09:34:36.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.753+0000 7f592a800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5924106370 0x7f592419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:36.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.753+0000 7f592a800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59241074a0 0x7f592419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:36.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.753+0000 7f592a800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5924107e40 0x7f59241a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:36.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.753+0000 7f592a800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5924111e30 con 0x7f5924106370 2024-05-27T09:34:36.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.753+0000 7f592a800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5924111cb0 con 0x7f59241074a0 2024-05-27T09:34:36.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.753+0000 7f592a800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5924111fb0 con 0x7f5924107e40 2024-05-27T09:34:36.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.754+0000 7f5929400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5924106370 0x7f592419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:36.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.754+0000 7f5929400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5924106370 0x7f592419c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49228/0 (socket says 172.21.15.44:49228) 2024-05-27T09:34:36.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.754+0000 7f5929400700 1 -- 172.21.15.44:0/2190298473 learned_addr learned my addr 172.21.15.44:0/2190298473 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:36.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.754+0000 7f5928a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59241074a0 0x7f592419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:36.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.754+0000 7f5929e00700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5924107e40 0x7f59241a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:36.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.754+0000 7f5929400700 1 -- 172.21.15.44:0/2190298473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5924107e40 msgr2=0x7f59241a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.754+0000 7f5929400700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5924107e40 0x7f59241a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.754+0000 7f5929400700 1 -- 172.21.15.44:0/2190298473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59241074a0 msgr2=0x7f592419cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.754+0000 7f5929400700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59241074a0 0x7f592419cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.754+0000 7f5929400700 1 -- 172.21.15.44:0/2190298473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59241a4500 con 0x7f5924106370 2024-05-27T09:34:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.754+0000 7f5928a00700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59241074a0 0x7f592419cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.755+0000 7f5929400700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5924106370 0x7f592419c9a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f591800e940 tx=0x7f591800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.755+0000 7f5922000700 1 -- 172.21.15.44:0/2190298473 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918004220 con 0x7f5924106370 2024-05-27T09:34:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.755+0000 7f5922000700 1 -- 172.21.15.44:0/2190298473 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59180043c0 con 0x7f5924106370 2024-05-27T09:34:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.755+0000 7f5922000700 1 -- 172.21.15.44:0/2190298473 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918003c60 con 0x7f5924106370 2024-05-27T09:34:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.755+0000 7f592a800700 1 -- 172.21.15.44:0/2190298473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59241a47f0 con 0x7f5924106370 2024-05-27T09:34:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.755+0000 7f592a800700 1 -- 172.21.15.44:0/2190298473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59241a4ce0 con 0x7f5924106370 2024-05-27T09:34:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.757+0000 7f592a800700 1 -- 172.21.15.44:0/2190298473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59240512b0 con 0x7f5924106370 2024-05-27T09:34:36.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.761+0000 7f5922000700 1 -- 172.21.15.44:0/2190298473 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f591800c7e0 con 0x7f5924106370 2024-05-27T09:34:36.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.763+0000 7f5922000700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f590c0b2880 0x7f590c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:36.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.763+0000 7f5922000700 1 -- 172.21.15.44:0/2190298473 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f59180e24d0 con 0x7f5924106370 2024-05-27T09:34:36.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.763+0000 7f5928a00700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f590c0b2880 0x7f590c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:36.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.764+0000 7f5928a00700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f590c0b2880 0x7f590c0b4d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f592419da50 tx=0x7f59100046f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:36.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:36.765+0000 7f5922000700 1 -- 172.21.15.44:0/2190298473 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f59180aa230 con 0x7f5924106370 2024-05-27T09:34:37.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.056+0000 7f592a800700 1 -- 172.21.15.44:0/2190298473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f592402e5e0 con 0x7f5924106370 2024-05-27T09:34:37.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.058+0000 7f5922000700 1 -- 172.21.15.44:0/2190298473 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3046 (secure 0 0 0) 0x7f59180a9b30 con 0x7f5924106370 2024-05-27T09:34:37.059 INFO:teuthology.orchestra.run.smithi044.stdout:e25 2024-05-27T09:34:37.059 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T09:34:37.059 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 25 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-05-27T09:31:27.747151+0000 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-05-27T09:31:50.423700+0000 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 300 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-05-27T09:34:37.060 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 3 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:in 0,1,2 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=24542,1=24535,2=24520} 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-05-27T09:34:37.061 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 0 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 24542 members: 24520,24542,24535 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.b{0:24542} state up:active seq 7 addr [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.f{0:24508} state up:standby-replay seq 1 addr [v2:172.21.15.189:6834/2673083839,v1:172.21.15.189:6835/2673083839] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.i{1:24535} state up:active seq 14 addr [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.h{1:24503} state up:standby-replay seq 1 addr [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.c{2:24520} state up:active seq 15 addr [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.e{2:24491} state up:standby-replay seq 1 addr [v2:172.21.15.50:6834/1092513236,v1:172.21.15.50:6835/1092513236] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.062 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:34:37.063 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:34:37.063 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-05-27T09:34:37.063 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:34:37.063 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.j{-1:14367} state up:standby seq 1 addr [v2:172.21.15.44:6834/907929206,v1:172.21.15.44:6835/907929206] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.063 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.d{-1:14373} state up:standby seq 1 addr [v2:172.21.15.44:6836/3467964892,v1:172.21.15.44:6837/3467964892] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.063 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.g{-1:14397} state up:standby seq 1 addr [v2:172.21.15.44:6838/285214695,v1:172.21.15.44:6839/285214695] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.063 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.a{-1:14409} state up:standby seq 1 addr [v2:172.21.15.44:6840/3128909996,v1:172.21.15.44:6841/3128909996] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.063 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.k{-1:24461} state up:standby seq 1 addr [v2:172.21.15.50:6832/4273379740,v1:172.21.15.50:6833/4273379740] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.063 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.l{-1:24472} state up:standby seq 1 addr [v2:172.21.15.189:6832/742901973,v1:172.21.15.189:6833/742901973] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T09:34:37.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 -- 172.21.15.44:0/2190298473 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f590c0b2880 msgr2=0x7f590c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f590c0b2880 0x7f590c0b4d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f592419da50 tx=0x7f59100046f0 comp rx=0 tx=0).stop 2024-05-27T09:34:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 -- 172.21.15.44:0/2190298473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5924106370 msgr2=0x7f592419c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5924106370 0x7f592419c9a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f591800e940 tx=0x7f591800ec50 comp rx=0 tx=0).stop 2024-05-27T09:34:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 -- 172.21.15.44:0/2190298473 shutdown_connections 2024-05-27T09:34:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f590c0b2880 0x7f590c0b4d90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5924107e40 0x7f59241a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5924106370 0x7f592419c9a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 --2- 172.21.15.44:0/2190298473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59241074a0 0x7f592419cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 -- 172.21.15.44:0/2190298473 >> 172.21.15.44:0/2190298473 conn(0x7f59241019b0 msgr2=0x7f592410a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 -- 172.21.15.44:0/2190298473 shutdown_connections 2024-05-27T09:34:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:37.061+0000 7f592a800700 1 -- 172.21.15.44:0/2190298473 wait complete. 2024-05-27T09:34:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-05-27T09:34:37.867 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-27T09:34:38.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:37 smithi044 ceph-mon[31618]: pgmap v421: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:38.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:37 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2190298473' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T09:34:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:37 smithi050 ceph-mon[36371]: pgmap v421: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:37 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2190298473' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T09:34:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:37 smithi189 ceph-mon[36671]: pgmap v421: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:38.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:37 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2190298473' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T09:34:38.348 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:40.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:39 smithi044 ceph-mon[31618]: pgmap v422: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:40.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:39 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:34:40.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:39 smithi050 ceph-mon[36371]: pgmap v422: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:40.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:39 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:34:40.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:39 smithi189 ceph-mon[36671]: pgmap v422: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:40.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:39 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:34:40.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.503+0000 7fc1a8600700 1 -- 172.21.15.44:0/2508933320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a0104e30 msgr2=0x7fc1a0107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:40.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.503+0000 7fc1a8600700 1 --2- 172.21.15.44:0/2508933320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a0104e30 0x7fc1a0107250 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc190009990 tx=0x7fc190009ca0 comp rx=0 tx=0).stop 2024-05-27T09:34:40.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.503+0000 7fc1a8600700 1 -- 172.21.15.44:0/2508933320 shutdown_connections 2024-05-27T09:34:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.503+0000 7fc1a8600700 1 --2- 172.21.15.44:0/2508933320 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1a010a0f0 0x7fc1a010c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.503+0000 7fc1a8600700 1 --2- 172.21.15.44:0/2508933320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a0104e30 0x7fc1a0107250 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.503+0000 7fc1a8600700 1 --2- 172.21.15.44:0/2508933320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a0107790 0x7fc1a0109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.503+0000 7fc1a8600700 1 -- 172.21.15.44:0/2508933320 >> 172.21.15.44:0/2508933320 conn(0x7fc1a00fe870 msgr2=0x7fc1a0100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.503+0000 7fc1a8600700 1 -- 172.21.15.44:0/2508933320 shutdown_connections 2024-05-27T09:34:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.503+0000 7fc1a8600700 1 -- 172.21.15.44:0/2508933320 wait complete. 2024-05-27T09:34:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.505+0000 7fc1a8600700 1 Processor -- start 2024-05-27T09:34:40.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.505+0000 7fc1a8600700 1 -- start start 2024-05-27T09:34:40.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.505+0000 7fc1a8600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1a0104e30 0x7fc1a019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:40.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.505+0000 7fc1a8600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a0107790 0x7fc1a019d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:40.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.505+0000 7fc1a8600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a010a0f0 0x7fc1a01a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:40.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.505+0000 7fc1a8600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a010fd70 con 0x7fc1a0107790 2024-05-27T09:34:40.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.505+0000 7fc1a8600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a010fbf0 con 0x7fc1a0104e30 2024-05-27T09:34:40.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.505+0000 7fc1a8600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a010fef0 con 0x7fc1a010a0f0 2024-05-27T09:34:40.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.506+0000 7fc1a5200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a0107790 0x7fc1a019d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:40.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.506+0000 7fc1a5c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1a0104e30 0x7fc1a019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:40.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.506+0000 7fc1a5c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1a0104e30 0x7fc1a019cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45108/0 (socket says 172.21.15.44:45108) 2024-05-27T09:34:40.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.506+0000 7fc1a5c00700 1 -- 172.21.15.44:0/2819207644 learned_addr learned my addr 172.21.15.44:0/2819207644 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.506+0000 7fc1a5200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a0107790 0x7fc1a019d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49246/0 (socket says 172.21.15.44:49246) 2024-05-27T09:34:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.506+0000 7fc1a5c00700 1 -- 172.21.15.44:0/2819207644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a010a0f0 msgr2=0x7fc1a01a4010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:34:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.506+0000 7fc1a5c00700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a010a0f0 0x7fc1a01a4010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.506+0000 7fc1a5c00700 1 -- 172.21.15.44:0/2819207644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a0107790 msgr2=0x7fc1a019d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.506+0000 7fc1a5c00700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a0107790 0x7fc1a019d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.506+0000 7fc1a5c00700 1 -- 172.21.15.44:0/2819207644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a01a4690 con 0x7fc1a0104e30 2024-05-27T09:34:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.507+0000 7fc1a5c00700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1a0104e30 0x7fc1a019cb30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc19000b760 tx=0x7fc190009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.507+0000 7fc19e000700 1 -- 172.21.15.44:0/2819207644 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc190026070 con 0x7fc1a0104e30 2024-05-27T09:34:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.507+0000 7fc19e000700 1 -- 172.21.15.44:0/2819207644 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc190021410 con 0x7fc1a0104e30 2024-05-27T09:34:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.507+0000 7fc19e000700 1 -- 172.21.15.44:0/2819207644 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc190017650 con 0x7fc1a0104e30 2024-05-27T09:34:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.507+0000 7fc1a8600700 1 -- 172.21.15.44:0/2819207644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1a01a4920 con 0x7fc1a0104e30 2024-05-27T09:34:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.507+0000 7fc1a8600700 1 -- 172.21.15.44:0/2819207644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1a01a4d20 con 0x7fc1a0104e30 2024-05-27T09:34:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.509+0000 7fc1a8600700 1 -- 172.21.15.44:0/2819207644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1a00512b0 con 0x7fc1a0104e30 2024-05-27T09:34:40.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.511+0000 7fc19e000700 1 -- 172.21.15.44:0/2819207644 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fc190029070 con 0x7fc1a0104e30 2024-05-27T09:34:40.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.512+0000 7fc19e000700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc1840b2900 0x7fc1840b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:40.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.513+0000 7fc19e000700 1 -- 172.21.15.44:0/2819207644 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fc1900e2ff0 con 0x7fc1a0104e30 2024-05-27T09:34:40.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.513+0000 7fc1a5200700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc1840b2900 0x7fc1840b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:40.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.513+0000 7fc1a5200700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc1840b2900 0x7fc1840b4e10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc1a019dbe0 tx=0x7fc18c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:40.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.517+0000 7fc19e000700 1 -- 172.21.15.44:0/2819207644 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc190013070 con 0x7fc1a0104e30 2024-05-27T09:34:40.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.755+0000 7fc1a8600700 1 -- 172.21.15.44:0/2819207644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fc1a01a5590 con 0x7fc1a0104e30 2024-05-27T09:34:40.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.756+0000 7fc19e000700 1 -- 172.21.15.44:0/2819207644 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v81) ==== 56+0+5268 (secure 0 0 0) 0x7fc1900aad50 con 0x7fc1a0104e30 2024-05-27T09:34:40.757 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 81 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-05-27T09:21:36.258511+0000 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-05-27T09:31:57.855852+0000 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:crush_version 26 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:full_ratio 0.95 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:backfillfull_ratio 0.9 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:nearfull_ratio 0.85 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:require_min_compat_client luminous 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:min_compat_client luminous 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:require_osd_release squid 2024-05-27T09:34:40.758 INFO:teuthology.orchestra.run.smithi044.stdout:stretch_mode_enabled false 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:max_osd 12 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 up in weight 1 up_from 8 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6802/3103307534,v1:172.21.15.44:6803/3103307534] [v2:172.21.15.44:6804/3103307534,v1:172.21.15.44:6805/3103307534] exists,up 8959d30f-ab21-41e8-b900-1f5ef3426892 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 up in weight 1 up_from 13 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6800/1520188354,v1:172.21.15.50:6801/1520188354] [v2:172.21.15.50:6802/1520188354,v1:172.21.15.50:6803/1520188354] exists,up 55f7adef-a968-40d3-9722-d17f5d4f9670 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 up in weight 1 up_from 18 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6800/1488616374,v1:172.21.15.189:6801/1488616374] [v2:172.21.15.189:6802/1488616374,v1:172.21.15.189:6803/1488616374] exists,up 6b3e59b2-65cf-40df-ab3c-eee28b4157d6 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 up in weight 1 up_from 25 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6810/2748877697,v1:172.21.15.44:6811/2748877697] [v2:172.21.15.44:6812/2748877697,v1:172.21.15.44:6813/2748877697] exists,up 4c9928a4-b7d8-436e-893b-5146bbbd7f4a 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 up in weight 1 up_from 31 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6808/299847660,v1:172.21.15.50:6809/299847660] [v2:172.21.15.50:6810/299847660,v1:172.21.15.50:6811/299847660] exists,up 88aa8a99-e01b-45d4-bac3-df51aaf8b94e 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 up in weight 1 up_from 36 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6808/2756976881,v1:172.21.15.189:6809/2756976881] [v2:172.21.15.189:6810/2756976881,v1:172.21.15.189:6811/2756976881] exists,up b075f197-da86-4cca-b088-d6912121fbad 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 up in weight 1 up_from 42 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6818/44684643,v1:172.21.15.44:6819/44684643] [v2:172.21.15.44:6820/44684643,v1:172.21.15.44:6821/44684643] exists,up 24e70958-59a8-415b-95f9-1b52a7dd8263 2024-05-27T09:34:40.759 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 up in weight 1 up_from 47 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6816/1616906446,v1:172.21.15.50:6817/1616906446] [v2:172.21.15.50:6818/1616906446,v1:172.21.15.50:6819/1616906446] exists,up cd1d6765-ace5-45f8-a07b-6cd583f32f7d 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:osd.8 up in weight 1 up_from 53 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6816/2379603694,v1:172.21.15.189:6817/2379603694] [v2:172.21.15.189:6818/2379603694,v1:172.21.15.189:6819/2379603694] exists,up 2602577e-c4b3-433e-a443-a71e4aa793a4 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:osd.9 up in weight 1 up_from 59 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6826/503459032,v1:172.21.15.44:6827/503459032] [v2:172.21.15.44:6828/503459032,v1:172.21.15.44:6829/503459032] exists,up 0307f939-76d2-4b3d-89ae-31581649f5e5 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:osd.10 up in weight 1 up_from 65 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6824/1371655028,v1:172.21.15.50:6825/1371655028] [v2:172.21.15.50:6826/1371655028,v1:172.21.15.50:6827/1371655028] exists,up f094a1ca-1b0a-47a4-974e-5620cd898ea4 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:osd.11 up in weight 1 up_from 71 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6824/2712606633,v1:172.21.15.189:6825/2712606633] [v2:172.21.15.189:6826/2712606633,v1:172.21.15.189:6827/2712606633] exists,up 4b9794fa-856f-4343-945e-4be444c0cd9b 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.0 [7,11] 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.b [2,1] 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.c [2,4] 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.1c [8,11] 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.1b [4,8] 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.34 [4,9] 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.38 [3,9,5,6] 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.3c [4,9] 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/1366147831 expires 2024-05-28T09:22:56.451569+0000 2024-05-27T09:34:40.760 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/3788109140 expires 2024-05-28T09:22:56.451569+0000 2024-05-27T09:34:40.761 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/2234109234 expires 2024-05-28T09:22:56.451569+0000 2024-05-27T09:34:40.761 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6801/4254617371 expires 2024-05-28T09:22:56.451569+0000 2024-05-27T09:34:40.761 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6800/2049000735 expires 2024-05-28T09:22:20.128618+0000 2024-05-27T09:34:40.761 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/2194243272 expires 2024-05-28T09:22:20.128618+0000 2024-05-27T09:34:40.761 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/340631364 expires 2024-05-28T09:22:20.128618+0000 2024-05-27T09:34:40.761 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6801/2049000735 expires 2024-05-28T09:22:20.128618+0000 2024-05-27T09:34:40.761 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/3022591733 expires 2024-05-28T09:22:20.128618+0000 2024-05-27T09:34:40.761 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6800/4254617371 expires 2024-05-28T09:22:56.451569+0000 2024-05-27T09:34:40.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.759+0000 7fc1a8600700 1 -- 172.21.15.44:0/2819207644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc1840b2900 msgr2=0x7fc1840b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:40.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.759+0000 7fc1a8600700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc1840b2900 0x7fc1840b4e10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc1a019dbe0 tx=0x7fc18c009260 comp rx=0 tx=0).stop 2024-05-27T09:34:40.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.759+0000 7fc1a8600700 1 -- 172.21.15.44:0/2819207644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1a0104e30 msgr2=0x7fc1a019cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:40.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.759+0000 7fc1a8600700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1a0104e30 0x7fc1a019cb30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc19000b760 tx=0x7fc190009580 comp rx=0 tx=0).stop 2024-05-27T09:34:40.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.760+0000 7fc1a8600700 1 -- 172.21.15.44:0/2819207644 shutdown_connections 2024-05-27T09:34:40.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.760+0000 7fc1a8600700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fc1840b2900 0x7fc1840b4e10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:40.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.760+0000 7fc1a8600700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a010a0f0 0x7fc1a01a4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.760+0000 7fc1a8600700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1a0104e30 0x7fc1a019cb30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.760+0000 7fc1a8600700 1 --2- 172.21.15.44:0/2819207644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a0107790 0x7fc1a019d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.760+0000 7fc1a8600700 1 -- 172.21.15.44:0/2819207644 >> 172.21.15.44:0/2819207644 conn(0x7fc1a00fe870 msgr2=0x7fc1a0102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.760+0000 7fc1a8600700 1 -- 172.21.15.44:0/2819207644 shutdown_connections 2024-05-27T09:34:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:40.760+0000 7fc1a8600700 1 -- 172.21.15.44:0/2819207644 wait complete. 2024-05-27T09:34:41.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:41 smithi044 ceph-mon[31618]: pgmap v423: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:41.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:41 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2819207644' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T09:34:41.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:34:41.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:34:41.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:41 smithi044 ceph-mon[31618]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.529 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-27T09:34:41.545 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:34:41.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:41 smithi189 ceph-mon[36671]: pgmap v423: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:41.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:41 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2819207644' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T09:34:41.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:34:41.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:34:41.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:41 smithi189 ceph-mon[36671]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:41 smithi050 ceph-mon[36371]: pgmap v423: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:41.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:41 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2819207644' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T09:34:41.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:34:41.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:34:41.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:41 smithi050 ceph-mon[36371]: from='mgr.14152 172.21.15.44:0/2236613152' entity='mgr.x' 2024-05-27T09:34:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:43.878 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:43 smithi044 ceph-mon[31618]: pgmap v424: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:44.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:43 smithi050 ceph-mon[36371]: pgmap v424: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:44.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:43 smithi189 ceph-mon[36671]: pgmap v424: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:44.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.127+0000 7f0e7cc00700 1 -- 172.21.15.44:0/1208730150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e78110960 msgr2=0x7f0e78110d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:44.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.127+0000 7f0e7cc00700 1 --2- 172.21.15.44:0/1208730150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e78110960 0x7f0e78110d70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0e68009990 tx=0x7f0e68009ca0 comp rx=0 tx=0).stop 2024-05-27T09:34:44.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.127+0000 7f0e7cc00700 1 -- 172.21.15.44:0/1208730150 shutdown_connections 2024-05-27T09:34:44.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.127+0000 7f0e7cc00700 1 --2- 172.21.15.44:0/1208730150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e7810b2d0 0x7f0e78117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:44.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.127+0000 7f0e7cc00700 1 --2- 172.21.15.44:0/1208730150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e78110960 0x7f0e78110d70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:44.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.127+0000 7f0e7cc00700 1 --2- 172.21.15.44:0/1208730150 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e7810a980 0x7f0e7810ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:44.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.127+0000 7f0e7cc00700 1 -- 172.21.15.44:0/1208730150 >> 172.21.15.44:0/1208730150 conn(0x7f0e78077c70 msgr2=0x7f0e780780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:44.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.127+0000 7f0e7cc00700 1 -- 172.21.15.44:0/1208730150 shutdown_connections 2024-05-27T09:34:44.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.127+0000 7f0e7cc00700 1 -- 172.21.15.44:0/1208730150 wait complete. 2024-05-27T09:34:44.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.129+0000 7f0e7cc00700 1 Processor -- start 2024-05-27T09:34:44.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.129+0000 7f0e7cc00700 1 -- start start 2024-05-27T09:34:44.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.130+0000 7f0e7cc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e7810a980 0x7f0e781a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:44.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.130+0000 7f0e7cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e7810b2d0 0x7f0e781a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:44.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.130+0000 7f0e7cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e78110960 0x7f0e781a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:44.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.130+0000 7f0e7cc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7811aab0 con 0x7f0e78110960 2024-05-27T09:34:44.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.130+0000 7f0e7cc00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7811a930 con 0x7f0e7810a980 2024-05-27T09:34:44.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.130+0000 7f0e7cc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7811ac30 con 0x7f0e7810b2d0 2024-05-27T09:34:44.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.130+0000 7f0e75400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e7810b2d0 0x7f0e781a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:44.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.130+0000 7f0e75400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e7810b2d0 0x7f0e781a1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:56024/0 (socket says 172.21.15.44:56024) 2024-05-27T09:34:44.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.130+0000 7f0e75400700 1 -- 172.21.15.44:0/3992443401 learned_addr learned my addr 172.21.15.44:0/3992443401 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:44.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.130+0000 7f0e75e00700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e7810a980 0x7f0e781a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:44.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.130+0000 7f0e76800700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e78110960 0x7f0e781a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e75400700 1 -- 172.21.15.44:0/3992443401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e7810a980 msgr2=0x7f0e781a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:44.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e75400700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e7810a980 0x7f0e781a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e75400700 1 -- 172.21.15.44:0/3992443401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e78110960 msgr2=0x7f0e781a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e75400700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e78110960 0x7f0e781a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e75400700 1 -- 172.21.15.44:0/3992443401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e781a6350 con 0x7f0e7810b2d0 2024-05-27T09:34:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e75e00700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e7810a980 0x7f0e781a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e76800700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e78110960 0x7f0e781a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e75400700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e7810b2d0 0x7f0e781a1a20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0e6400e970 tx=0x7f0e6400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e6ea00700 1 -- 172.21.15.44:0/3992443401 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64004a30 con 0x7f0e7810b2d0 2024-05-27T09:34:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e7cc00700 1 -- 172.21.15.44:0/3992443401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e781a6640 con 0x7f0e7810b2d0 2024-05-27T09:34:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e6ea00700 1 -- 172.21.15.44:0/3992443401 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e6400cc90 con 0x7f0e7810b2d0 2024-05-27T09:34:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e6ea00700 1 -- 172.21.15.44:0/3992443401 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64005610 con 0x7f0e7810b2d0 2024-05-27T09:34:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.131+0000 7f0e7cc00700 1 -- 172.21.15.44:0/3992443401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e781adf60 con 0x7f0e7810b2d0 2024-05-27T09:34:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.133+0000 7f0e7cc00700 1 -- 172.21.15.44:0/3992443401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e780512b0 con 0x7f0e7810b2d0 2024-05-27T09:34:44.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.136+0000 7f0e6ea00700 1 -- 172.21.15.44:0/3992443401 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f0e6400c7e0 con 0x7f0e7810b2d0 2024-05-27T09:34:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.137+0000 7f0e6ea00700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0e5c0b2880 0x7f0e5c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.137+0000 7f0e6ea00700 1 -- 172.21.15.44:0/3992443401 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f0e640e25e0 con 0x7f0e7810b2d0 2024-05-27T09:34:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.138+0000 7f0e75e00700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0e5c0b2880 0x7f0e5c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.138+0000 7f0e75e00700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0e5c0b2880 0x7f0e5c0b4d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800b2e0 tx=0x7f0e68005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:44.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.139+0000 7f0e6ea00700 1 -- 172.21.15.44:0/3992443401 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0e64054df0 con 0x7f0e7810b2d0 2024-05-27T09:34:44.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.427+0000 7f0e7cc00700 1 -- 172.21.15.44:0/3992443401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e7802e5e0 con 0x7f0e7810b2d0 2024-05-27T09:34:44.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.428+0000 7f0e6ea00700 1 -- 172.21.15.44:0/3992443401 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7f0e640aa2b0 con 0x7f0e7810b2d0 2024-05-27T09:34:44.430 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:34:44.430 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:31:50.423700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:34:44.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.432+0000 7f0e7cc00700 1 -- 172.21.15.44:0/3992443401 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0e5c0b2880 msgr2=0x7f0e5c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:44.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.432+0000 7f0e7cc00700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0e5c0b2880 0x7f0e5c0b4d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800b2e0 tx=0x7f0e68005fd0 comp rx=0 tx=0).stop 2024-05-27T09:34:44.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.432+0000 7f0e7cc00700 1 -- 172.21.15.44:0/3992443401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e7810b2d0 msgr2=0x7f0e781a1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:44.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.432+0000 7f0e7cc00700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e7810b2d0 0x7f0e781a1a20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0e6400e970 tx=0x7f0e6400ec80 comp rx=0 tx=0).stop 2024-05-27T09:34:44.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.433+0000 7f0e7cc00700 1 -- 172.21.15.44:0/3992443401 shutdown_connections 2024-05-27T09:34:44.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.433+0000 7f0e7cc00700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f0e5c0b2880 0x7f0e5c0b4d90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:44.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.433+0000 7f0e7cc00700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e78110960 0x7f0e781a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:44.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.433+0000 7f0e7cc00700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e7810a980 0x7f0e781a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:44.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.433+0000 7f0e7cc00700 1 --2- 172.21.15.44:0/3992443401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e7810b2d0 0x7f0e781a1a20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:44.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.433+0000 7f0e7cc00700 1 -- 172.21.15.44:0/3992443401 >> 172.21.15.44:0/3992443401 conn(0x7f0e78077c70 msgr2=0x7f0e7807bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:44.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.433+0000 7f0e7cc00700 1 -- 172.21.15.44:0/3992443401 shutdown_connections 2024-05-27T09:34:44.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:44.433+0000 7f0e7cc00700 1 -- 172.21.15.44:0/3992443401 wait complete. 2024-05-27T09:34:44.435 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-05-27T09:34:45.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:44 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3992443401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:34:45.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:44 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3992443401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:34:45.113 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:44 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3992443401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:34:45.173 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:34:45.643 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:46.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:45 smithi050 ceph-mon[36371]: pgmap v425: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:46.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:45 smithi189 ceph-mon[36671]: pgmap v425: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:46.093 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:45 smithi044 ceph-mon[31618]: pgmap v425: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:47.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.707+0000 7f61f2600700 1 -- 172.21.15.44:0/756925989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61ec076670 msgr2=0x7f61ec110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:47.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.707+0000 7f61f2600700 1 --2- 172.21.15.44:0/756925989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61ec076670 0x7f61ec110650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f61d80098f0 tx=0x7f61d8009c00 comp rx=0 tx=0).stop 2024-05-27T09:34:47.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.707+0000 7f61f2600700 1 -- 172.21.15.44:0/756925989 shutdown_connections 2024-05-27T09:34:47.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.707+0000 7f61f2600700 1 --2- 172.21.15.44:0/756925989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ec110b90 0x7f61ec112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:47.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.707+0000 7f61f2600700 1 --2- 172.21.15.44:0/756925989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61ec075d20 0x7f61ec076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:47.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.707+0000 7f61f2600700 1 --2- 172.21.15.44:0/756925989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61ec076670 0x7f61ec110650 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:47.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.707+0000 7f61f2600700 1 -- 172.21.15.44:0/756925989 >> 172.21.15.44:0/756925989 conn(0x7f61ec100030 msgr2=0x7f61ec102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:47.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.707+0000 7f61f2600700 1 -- 172.21.15.44:0/756925989 shutdown_connections 2024-05-27T09:34:47.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.707+0000 7f61f2600700 1 -- 172.21.15.44:0/756925989 wait complete. 2024-05-27T09:34:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.708+0000 7f61f2600700 1 Processor -- start 2024-05-27T09:34:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.709+0000 7f61f2600700 1 -- start start 2024-05-27T09:34:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.709+0000 7f61f2600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61ec075d20 0x7f61ec19d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.709+0000 7f61f2600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ec076670 0x7f61ec19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.709+0000 7f61f2600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61ec110b90 0x7f61ec1a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.709+0000 7f61f2600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f61ec1166e0 con 0x7f61ec075d20 2024-05-27T09:34:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.709+0000 7f61f2600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f61ec116560 con 0x7f61ec110b90 2024-05-27T09:34:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.709+0000 7f61f2600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61ec116860 con 0x7f61ec076670 2024-05-27T09:34:47.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.709+0000 7f61eb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61ec075d20 0x7f61ec19d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.710+0000 7f61eb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61ec075d20 0x7f61ec19d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53188/0 (socket says 172.21.15.44:53188) 2024-05-27T09:34:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.710+0000 7f61eb400700 1 -- 172.21.15.44:0/2759363020 learned_addr learned my addr 172.21.15.44:0/2759363020 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.710+0000 7f61ebe00700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61ec110b90 0x7f61ec1a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.710+0000 7f61eaa00700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ec076670 0x7f61ec19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.710+0000 7f61eb400700 1 -- 172.21.15.44:0/2759363020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ec076670 msgr2=0x7f61ec19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.710+0000 7f61eb400700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ec076670 0x7f61ec19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.710+0000 7f61eb400700 1 -- 172.21.15.44:0/2759363020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61ec110b90 msgr2=0x7f61ec1a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.710+0000 7f61eb400700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61ec110b90 0x7f61ec1a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.710+0000 7f61eb400700 1 -- 172.21.15.44:0/2759363020 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61ec1a1f80 con 0x7f61ec075d20 2024-05-27T09:34:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.710+0000 7f61ebe00700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61ec110b90 0x7f61ec1a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:47.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.710+0000 7f61eb400700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61ec075d20 0x7f61ec19d0b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00e9c0 tx=0x7f61dc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:47.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.711+0000 7f61e3e00700 1 -- 172.21.15.44:0/2759363020 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc004a30 con 0x7f61ec075d20 2024-05-27T09:34:47.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.711+0000 7f61e3e00700 1 -- 172.21.15.44:0/2759363020 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61dc00cc70 con 0x7f61ec075d20 2024-05-27T09:34:47.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.711+0000 7f61e3e00700 1 -- 172.21.15.44:0/2759363020 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc005490 con 0x7f61ec075d20 2024-05-27T09:34:47.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.711+0000 7f61f2600700 1 -- 172.21.15.44:0/2759363020 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61ec1a2270 con 0x7f61ec075d20 2024-05-27T09:34:47.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.711+0000 7f61f2600700 1 -- 172.21.15.44:0/2759363020 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61ec1a9b90 con 0x7f61ec075d20 2024-05-27T09:34:47.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.712+0000 7f61f2600700 1 -- 172.21.15.44:0/2759363020 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61ec0512b0 con 0x7f61ec075d20 2024-05-27T09:34:47.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.715+0000 7f61e3e00700 1 -- 172.21.15.44:0/2759363020 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f61dc004bd0 con 0x7f61ec075d20 2024-05-27T09:34:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.716+0000 7f61e3e00700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f61d00b2900 0x7f61d00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:47.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.716+0000 7f61e3e00700 1 -- 172.21.15.44:0/2759363020 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f61dc0e2510 con 0x7f61ec075d20 2024-05-27T09:34:47.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.717+0000 7f61eaa00700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f61d00b2900 0x7f61d00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:47.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.717+0000 7f61eaa00700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f61d00b2900 0x7f61d00b4e10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f61d80098c0 tx=0x7f61d8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:47.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:47.723+0000 7f61e3e00700 1 -- 172.21.15.44:0/2759363020 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f61dc054df0 con 0x7f61ec075d20 2024-05-27T09:34:48.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.012+0000 7f61f2600700 1 -- 172.21.15.44:0/2759363020 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61ec19e410 con 0x7f61ec075d20 2024-05-27T09:34:48.014 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:47 smithi044 ceph-mon[31618]: pgmap v426: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:48.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.015+0000 7f61e3e00700 1 -- 172.21.15.44:0/2759363020 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7f61dc0aa1e0 con 0x7f61ec075d20 2024-05-27T09:34:48.016 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:34:48.016 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:31:50.423700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:34:48.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 -- 172.21.15.44:0/2759363020 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f61d00b2900 msgr2=0x7f61d00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:48.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f61d00b2900 0x7f61d00b4e10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f61d80098c0 tx=0x7f61d8019040 comp rx=0 tx=0).stop 2024-05-27T09:34:48.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 -- 172.21.15.44:0/2759363020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61ec075d20 msgr2=0x7f61ec19d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:48.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61ec075d20 0x7f61ec19d0b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00e9c0 tx=0x7f61dc00ecd0 comp rx=0 tx=0).stop 2024-05-27T09:34:48.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 -- 172.21.15.44:0/2759363020 shutdown_connections 2024-05-27T09:34:48.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f61d00b2900 0x7f61d00b4e10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:48.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61ec110b90 0x7f61ec1a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:48.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61ec075d20 0x7f61ec19d0b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:48.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 --2- 172.21.15.44:0/2759363020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ec076670 0x7f61ec19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:48.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 -- 172.21.15.44:0/2759363020 >> 172.21.15.44:0/2759363020 conn(0x7f61ec100030 msgr2=0x7f61ec100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:48.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 -- 172.21.15.44:0/2759363020 shutdown_connections 2024-05-27T09:34:48.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:48.018+0000 7f61f2600700 1 -- 172.21.15.44:0/2759363020 wait complete. 2024-05-27T09:34:48.020 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-05-27T09:34:48.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:47 smithi050 ceph-mon[36371]: pgmap v426: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:48.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:47 smithi189 ceph-mon[36671]: pgmap v426: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:48.818 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T09:34:49.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:48 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2759363020' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:34:49.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:48 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2759363020' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:34:49.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:48 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2759363020' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:34:49.302 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:50.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:49 smithi189 ceph-mon[36671]: pgmap v427: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:50.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:49 smithi044 ceph-mon[31618]: pgmap v427: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:50.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:49 smithi050 ceph-mon[36371]: pgmap v427: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:51.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.472+0000 7fe7ad600700 1 -- 172.21.15.44:0/591203359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7a8103a20 msgr2=0x7fe7a8110740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:51.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.472+0000 7fe7ad600700 1 --2- 172.21.15.44:0/591203359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7a8103a20 0x7fe7a8110740 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe794009990 tx=0x7fe794009ca0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.472+0000 7fe7ad600700 1 -- 172.21.15.44:0/591203359 shutdown_connections 2024-05-27T09:34:51.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.472+0000 7fe7ad600700 1 --2- 172.21.15.44:0/591203359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7a8110c80 0x7fe7a81130a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.472+0000 7fe7ad600700 1 --2- 172.21.15.44:0/591203359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7a81030d0 0x7fe7a81034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.472+0000 7fe7ad600700 1 --2- 172.21.15.44:0/591203359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7a8103a20 0x7fe7a8110740 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.472+0000 7fe7ad600700 1 -- 172.21.15.44:0/591203359 >> 172.21.15.44:0/591203359 conn(0x7fe7a80fede0 msgr2=0x7fe7a8101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:51.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.472+0000 7fe7ad600700 1 -- 172.21.15.44:0/591203359 shutdown_connections 2024-05-27T09:34:51.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.472+0000 7fe7ad600700 1 -- 172.21.15.44:0/591203359 wait complete. 2024-05-27T09:34:51.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.474+0000 7fe7ad600700 1 Processor -- start 2024-05-27T09:34:51.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.474+0000 7fe7ad600700 1 -- start start 2024-05-27T09:34:51.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.474+0000 7fe7ad600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7a81030d0 0x7fe7a819d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:51.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.474+0000 7fe7ad600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7a8103a20 0x7fe7a819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:51.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.474+0000 7fe7ad600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7a8110c80 0x7fe7a81a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:51.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.474+0000 7fe7ad600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a8116690 con 0x7fe7a8110c80 2024-05-27T09:34:51.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7ad600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a8116510 con 0x7fe7a81030d0 2024-05-27T09:34:51.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7ad600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a8116810 con 0x7fe7a8103a20 2024-05-27T09:34:51.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7a6a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7a81030d0 0x7fe7a819d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:51.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7a6a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7a81030d0 0x7fe7a819d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60320/0 (socket says 172.21.15.44:60320) 2024-05-27T09:34:51.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7a6a00700 1 -- 172.21.15.44:0/1134558077 learned_addr learned my addr 172.21.15.44:0/1134558077 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:51.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7a7400700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7a8110c80 0x7fe7a81a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:51.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7a6000700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7a8103a20 0x7fe7a819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:51.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7a6a00700 1 -- 172.21.15.44:0/1134558077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7a8103a20 msgr2=0x7fe7a819d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:51.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7a6a00700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7a8103a20 0x7fe7a819d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7a6a00700 1 -- 172.21.15.44:0/1134558077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7a8110c80 msgr2=0x7fe7a81a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:51.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7a6a00700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7a8110c80 0x7fe7a81a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.475+0000 7fe7a6a00700 1 -- 172.21.15.44:0/1134558077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a81a1f10 con 0x7fe7a81030d0 2024-05-27T09:34:51.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.476+0000 7fe7a6a00700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7a81030d0 0x7fe7a819d040 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe79800e9a0 tx=0x7fe79800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:51.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.476+0000 7fe79f400700 1 -- 172.21.15.44:0/1134558077 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79800cbd0 con 0x7fe7a81030d0 2024-05-27T09:34:51.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.476+0000 7fe7ad600700 1 -- 172.21.15.44:0/1134558077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a81a2200 con 0x7fe7a81030d0 2024-05-27T09:34:51.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.476+0000 7fe79f400700 1 -- 172.21.15.44:0/1134558077 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe79800cd70 con 0x7fe7a81030d0 2024-05-27T09:34:51.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.476+0000 7fe79f400700 1 -- 172.21.15.44:0/1134558077 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe798018680 con 0x7fe7a81030d0 2024-05-27T09:34:51.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.476+0000 7fe7ad600700 1 -- 172.21.15.44:0/1134558077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a81a9b20 con 0x7fe7a81030d0 2024-05-27T09:34:51.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.478+0000 7fe7ad600700 1 -- 172.21.15.44:0/1134558077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7a80512b0 con 0x7fe7a81030d0 2024-05-27T09:34:51.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.480+0000 7fe79f400700 1 -- 172.21.15.44:0/1134558077 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fe798010430 con 0x7fe7a81030d0 2024-05-27T09:34:51.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.482+0000 7fe79f400700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe78c0b2880 0x7fe78c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:51.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.482+0000 7fe79f400700 1 -- 172.21.15.44:0/1134558077 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7fe7980e24e0 con 0x7fe7a81030d0 2024-05-27T09:34:51.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.482+0000 7fe7a6000700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe78c0b2880 0x7fe78c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:51.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.483+0000 7fe7a6000700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe78c0b2880 0x7fe78c0b4d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe79400b2e0 tx=0x7fe794005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:51.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.486+0000 7fe79f400700 1 -- 172.21.15.44:0/1134558077 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe7980aa1b0 con 0x7fe7a81030d0 2024-05-27T09:34:51.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.725+0000 7fe7ad600700 1 -- 172.21.15.44:0/1134558077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe7a802e5e0 con 0x7fe7a81030d0 2024-05-27T09:34:51.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.728+0000 7fe79f400700 1 -- 172.21.15.44:0/1134558077 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22611 (secure 0 0 0) 0x7fe7980b2230 con 0x7fe7a81030d0 2024-05-27T09:34:51.729 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:34:51.729 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":81,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:31:57.855852+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:34:51.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 -- 172.21.15.44:0/1134558077 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe78c0b2880 msgr2=0x7fe78c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:51.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe78c0b2880 0x7fe78c0b4d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe79400b2e0 tx=0x7fe794005fb0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 -- 172.21.15.44:0/1134558077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7a81030d0 msgr2=0x7fe7a819d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:51.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7a81030d0 0x7fe7a819d040 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe79800e9a0 tx=0x7fe79800ecb0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 -- 172.21.15.44:0/1134558077 shutdown_connections 2024-05-27T09:34:51.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7fe78c0b2880 0x7fe78c0b4d90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7a8110c80 0x7fe7a81a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7a81030d0 0x7fe7a819d040 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 --2- 172.21.15.44:0/1134558077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7a8103a20 0x7fe7a819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:51.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 -- 172.21.15.44:0/1134558077 >> 172.21.15.44:0/1134558077 conn(0x7fe7a80fede0 msgr2=0x7fe7a8112750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:51.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 -- 172.21.15.44:0/1134558077 shutdown_connections 2024-05-27T09:34:51.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:51.731+0000 7fe7ad600700 1 -- 172.21.15.44:0/1134558077 wait complete. 2024-05-27T09:34:52.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:51 smithi044 ceph-mon[31618]: pgmap v428: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:52.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:51 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1134558077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:34:52.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:51 smithi189 ceph-mon[36671]: pgmap v428: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:52.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:51 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1134558077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:34:52.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:51 smithi050 ceph-mon[36371]: pgmap v428: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:34:52.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:51 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1134558077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:34:52.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-27T09:31:27.747151+0000', 'modified': '2024-05-27T09:31:50.423700+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24542, 'mds_1': 24535, 'mds_2': 24520}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24491': {'gid': 24491, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.50:6835/1092513236', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6834', 'nonce': 1092513236}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 1092513236}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.50:6837/3540954143', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6836', 'nonce': 3540954143}, {'type': 'v1', 'addr': '172.21.15.50:6837', 'nonce': 3540954143}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6835/2673083839', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 2673083839}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 2673083839}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.189:6837/2265427173', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 2265427173}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 2265427173}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.189:6839/4109192600', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 4109192600}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 4109192600}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24542': {'gid': 24542, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.50:6839/1124366265', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6838', 'nonce': 1124366265}, {'type': 'v1', 'addr': '172.21.15.50:6839', 'nonce': 1124366265}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24542, 'qdb_cluster': [24520, 24542, 24535]} 2024-05-27T09:34:52.505 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-27T09:34:52.506 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.c status 2024-05-27T09:34:52.965 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:54.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:53 smithi189 ceph-mon[36671]: pgmap v429: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:54.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:53 smithi044 ceph-mon[31618]: pgmap v429: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:54.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:53 smithi050 ceph-mon[36371]: pgmap v429: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:34:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.070+0000 7f9063400700 1 -- 172.21.15.44:0/4001982241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f905c10a980 msgr2=0x7f905c10ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.070+0000 7f9063400700 1 --2- 172.21.15.44:0/4001982241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f905c10a980 0x7f905c10ad90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f904c009990 tx=0x7f904c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.070+0000 7f9063400700 1 -- 172.21.15.44:0/4001982241 shutdown_connections 2024-05-27T09:34:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.070+0000 7f9063400700 1 --2- 172.21.15.44:0/4001982241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f905c10b2d0 0x7f905c117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.070+0000 7f9063400700 1 --2- 172.21.15.44:0/4001982241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f905c110960 0x7f905c110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.070+0000 7f9063400700 1 --2- 172.21.15.44:0/4001982241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f905c10a980 0x7f905c10ad90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.070+0000 7f9063400700 1 -- 172.21.15.44:0/4001982241 >> 172.21.15.44:0/4001982241 conn(0x7f905c077c70 msgr2=0x7f905c0780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:55.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.070+0000 7f9063400700 1 -- 172.21.15.44:0/4001982241 shutdown_connections 2024-05-27T09:34:55.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.070+0000 7f9063400700 1 -- 172.21.15.44:0/4001982241 wait complete. 2024-05-27T09:34:55.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.071+0000 7f9063400700 1 Processor -- start 2024-05-27T09:34:55.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.072+0000 7f9063400700 1 -- start start 2024-05-27T09:34:55.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.072+0000 7f9063400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f905c10a980 0x7f905c1a1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.072+0000 7f9063400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f905c10b2d0 0x7f905c1a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.072+0000 7f9063400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f905c110960 0x7f905c1a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.072+0000 7f9063400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f905c11aa60 con 0x7f905c110960 2024-05-27T09:34:55.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.072+0000 7f9063400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f905c11a8e0 con 0x7f905c10b2d0 2024-05-27T09:34:55.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.072+0000 7f9063400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f905c11abe0 con 0x7f905c10a980 2024-05-27T09:34:55.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.072+0000 7f905be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f905c10b2d0 0x7f905c1a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.072+0000 7f9061400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f905c110960 0x7f905c1a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.073+0000 7f9061400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f905c110960 0x7f905c1a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39200/0 (socket says 172.21.15.44:39200) 2024-05-27T09:34:55.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.073+0000 7f905be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f905c10b2d0 0x7f905c1a19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45570/0 (socket says 172.21.15.44:45570) 2024-05-27T09:34:55.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.073+0000 7f9061400700 1 -- 172.21.15.44:0/1255770133 learned_addr learned my addr 172.21.15.44:0/1255770133 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:55.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.073+0000 7f9060a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f905c10a980 0x7f905c1a1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.073+0000 7f9061400700 1 -- 172.21.15.44:0/1255770133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f905c10a980 msgr2=0x7f905c1a1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:55.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.073+0000 7f9061400700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f905c10a980 0x7f905c1a1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.073+0000 7f9061400700 1 -- 172.21.15.44:0/1255770133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f905c10b2d0 msgr2=0x7f905c1a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:55.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.073+0000 7f9061400700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f905c10b2d0 0x7f905c1a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.073+0000 7f9061400700 1 -- 172.21.15.44:0/1255770133 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c1a6300 con 0x7f905c110960 2024-05-27T09:34:55.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.073+0000 7f905be00700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f905c10b2d0 0x7f905c1a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:55.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.074+0000 7f9061400700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f905c110960 0x7f905c1a5c60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f905400ef30 tx=0x7f905400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:55.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.074+0000 7f9059600700 1 -- 172.21.15.44:0/1255770133 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90540092e0 con 0x7f905c110960 2024-05-27T09:34:55.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.074+0000 7f9059600700 1 -- 172.21.15.44:0/1255770133 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9054004bb0 con 0x7f905c110960 2024-05-27T09:34:55.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.074+0000 7f9063400700 1 -- 172.21.15.44:0/1255770133 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905c1a6590 con 0x7f905c110960 2024-05-27T09:34:55.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.074+0000 7f9059600700 1 -- 172.21.15.44:0/1255770133 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054004020 con 0x7f905c110960 2024-05-27T09:34:55.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.074+0000 7f9063400700 1 -- 172.21.15.44:0/1255770133 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f905c1a6aa0 con 0x7f905c110960 2024-05-27T09:34:55.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.078+0000 7f9059600700 1 -- 172.21.15.44:0/1255770133 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f905401b030 con 0x7f905c110960 2024-05-27T09:34:55.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.079+0000 7f9071f2ab80 1 Processor -- start 2024-05-27T09:34:55.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.079+0000 7f9071f2ab80 1 -- start start 2024-05-27T09:34:55.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9059600700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f90440b2900 0x7f90440b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fb982e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fa19df0 0x56034fb98820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fba50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56034fac6df0 con 0x56034fb059a0 2024-05-27T09:34:55.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56034fab6990 con 0x56034fa19df0 2024-05-27T09:34:55.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56034faba9d0 con 0x56034fb98e20 2024-05-27T09:34:55.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9059600700 1 -- 172.21.15.44:0/1255770133 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f90540e1cd0 con 0x7f905c110960 2024-05-27T09:34:55.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9060a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fb982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9060a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fb982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39204/0 (socket says 172.21.15.44:39204) 2024-05-27T09:34:55.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f905be00700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f90440b2900 0x7f90440b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9060a00700 1 -- 172.21.15.44:0/2799032513 learned_addr learned my addr 172.21.15.44:0/2799032513 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:55.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f9061400700 1 --2- 172.21.15.44:0/2799032513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fba50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.080+0000 7f905be00700 1 --2- 172.21.15.44:0/2799032513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fa19df0 0x56034fb98820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.081+0000 7f9060a00700 1 -- 172.21.15.44:0/2799032513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 msgr2=0x56034fba50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:55.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.081+0000 7f9060a00700 1 --2- 172.21.15.44:0/2799032513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fba50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.081+0000 7f9060a00700 1 -- 172.21.15.44:0/2799032513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fa19df0 msgr2=0x56034fb98820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:55.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.081+0000 7f9060a00700 1 --2- 172.21.15.44:0/2799032513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fa19df0 0x56034fb98820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.081+0000 7f9060a00700 1 -- 172.21.15.44:0/2799032513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56034fa7f3d0 con 0x56034fb059a0 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.081+0000 7f9060a00700 1 --2- 172.21.15.44:0/2799032513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fb982e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9050009ca0 tx=0x7f9050011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c3ac9a9fe5ad6f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.081+0000 7f905be00700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f90440b2900 0x7f90440b4e10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f904c006010 tx=0x7f904c009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.081+0000 7f904a000700 1 -- 172.21.15.44:0/2799032513 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905001e070 con 0x56034fb059a0 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.081+0000 7f904a000700 1 -- 172.21.15.44:0/2799032513 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9050011d80 con 0x56034fb059a0 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.081+0000 7f904a000700 1 -- 172.21.15.44:0/2799032513 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905000e5f0 con 0x56034fb059a0 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.083+0000 7f9071f2ab80 1 -- 172.21.15.44:0/2799032513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 msgr2=0x56034fb982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.083+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/2799032513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fb982e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9050009ca0 tx=0x7f9050011440 comp rx=0 tx=0).stop 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.083+0000 7f9071f2ab80 1 -- 172.21.15.44:0/2799032513 shutdown_connections 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.083+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/2799032513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fba50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.083+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/2799032513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fb982e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.083+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/2799032513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fa19df0 0x56034fb98820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.083+0000 7f9071f2ab80 1 -- 172.21.15.44:0/2799032513 >> 172.21.15.44:0/2799032513 conn(0x56034fab3340 msgr2=0x56034fb94470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:55.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.083+0000 7f9071f2ab80 1 -- 172.21.15.44:0/2799032513 shutdown_connections 2024-05-27T09:34:55.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.083+0000 7f9071f2ab80 1 -- 172.21.15.44:0/2799032513 wait complete. 2024-05-27T09:34:55.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.084+0000 7f9071f2ab80 1 -- start start 2024-05-27T09:34:55.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.084+0000 7f9071f2ab80 1 Processor -- start 2024-05-27T09:34:55.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fa19df0 0x56034fba2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fb059a0 0x56034fba2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fba2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56034f9bcf50 con 0x56034fa19df0 2024-05-27T09:34:55.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56034fa57710 con 0x56034fb059a0 2024-05-27T09:34:55.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56034fba5b30 con 0x56034fb98e20 2024-05-27T09:34:55.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f9071f2ab80 10 client.0 _ll_register_callbacks cb 0x56034fa60c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:34:55.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f9060a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fa19df0 0x56034fba2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f9060a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fa19df0 0x56034fba2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39206/0 (socket says 172.21.15.44:39206) 2024-05-27T09:34:55.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f9060a00700 1 -- 172.21.15.44:0/3286726466 learned_addr learned my addr 172.21.15.44:0/3286726466 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:55.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f9061400700 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fba2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.085+0000 7f905be00700 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fb059a0 0x56034fba2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.086+0000 7f9060a00700 1 -- 172.21.15.44:0/3286726466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 msgr2=0x56034fba2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.086+0000 7f9060a00700 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fba2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.086+0000 7f9060a00700 1 -- 172.21.15.44:0/3286726466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fb059a0 msgr2=0x56034fba2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.086+0000 7f9060a00700 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fb059a0 0x56034fba2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.086+0000 7f9060a00700 1 -- 172.21.15.44:0/3286726466 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56034fa06880 con 0x56034fa19df0 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.086+0000 7f9061400700 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fba2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.086+0000 7f9060a00700 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fa19df0 0x56034fba2540 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9050005950 tx=0x7f90500056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78509e7f086a915e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.086+0000 7f903ea00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.086+0000 7f903ea00700 1 -- 172.21.15.44:0/3286726466 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905001e050 con 0x56034fa19df0 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.087+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3286726466 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9050011d80 con 0x56034fa19df0 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.087+0000 7f903ea00700 1 -- 172.21.15.44:0/3286726466 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f9050011b80 con 0x56034fa19df0 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.087+0000 7f903ea00700 1 -- 172.21.15.44:0/3286726466 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9050004960 con 0x56034fa19df0 2024-05-27T09:34:55.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.087+0000 7f9071f2ab80 10 client.14868 fetch_fsmap learned FSMap version 25 2024-05-27T09:34:55.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.087+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3286726466 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56034fa87010 con 0x56034fa19df0 2024-05-27T09:34:55.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.087+0000 7f903ea00700 1 -- 172.21.15.44:0/3286726466 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9050011d80 con 0x56034fa19df0 2024-05-27T09:34:55.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.087+0000 7f903ea00700 1 -- 172.21.15.44:0/3286726466 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f9050023400 con 0x56034fa19df0 2024-05-27T09:34:55.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.088+0000 7f9071f2ab80 10 client.14868 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:34:55.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.088+0000 7f9071f2ab80 10 client.14868 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T09:34:55.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.088+0000 7f9071f2ab80 20 client.14868 populate_metadata read hostname 'smithi044' 2024-05-27T09:34:55.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.088+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56034fbb27f0 0x56034fbb3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.088+0000 7f9071f2ab80 4 client.14868 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:34:55.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.088+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3286726466 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56034fac8d40 con 0x56034fbb27f0 2024-05-27T09:34:55.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.088+0000 7f905be00700 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56034fbb27f0 0x56034fbb3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.089+0000 7f905be00700 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56034fbb27f0 0x56034fbb3c60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=48c0b339236b4ab3 server_cookie=d36fcfb9d5a1a323 in_seq=0 out_seq=0 2024-05-27T09:34:55.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.089+0000 7f903ea00700 10 client.14868 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f903ea00700 1 -- 172.21.15.44:0/3286726466 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56034fac8d40 con 0x56034fbb27f0 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f903ea00700 10 client.14868 handle_command_reply: tid=0 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 1 client.14868 shutdown 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3286726466 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fa19df0 msgr2=0x56034fba2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fa19df0 0x56034fba2540 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9050005950 tx=0x7f90500056a0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3286726466 shutdown_connections 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fba2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56034fbb27f0 0x56034fbb3c60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fa19df0 0x56034fba2540 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/3286726466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fb059a0 0x56034fba2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f903ea00700 0 client.14868 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3286726466 >> 172.21.15.44:0/3286726466 conn(0x56034fab3340 msgr2=0x56034fb982c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:55.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3286726466 shutdown_connections 2024-05-27T09:34:55.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3286726466 wait complete. 2024-05-27T09:34:55.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.090+0000 7f9071f2ab80 20 client.14868 trim_cache size 0 max 16384 2024-05-27T09:34:55.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f9071f2ab80 1 Processor -- start 2024-05-27T09:34:55.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f9071f2ab80 1 -- start start 2024-05-27T09:34:55.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fa19df0 0x56034fbaee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fba14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fb9dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56034fbb1fa0 con 0x56034fb059a0 2024-05-27T09:34:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56034fbb2670 con 0x56034fa19df0 2024-05-27T09:34:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56034fbb2d40 con 0x56034fb98e20 2024-05-27T09:34:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f905be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fba14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f905be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fba14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39218/0 (socket says 172.21.15.44:39218) 2024-05-27T09:34:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f905be00700 1 -- 172.21.15.44:0/3631204294 learned_addr learned my addr 172.21.15.44:0/3631204294 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f9060a00700 1 --2- 172.21.15.44:0/3631204294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fa19df0 0x56034fbaee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.115+0000 7f9061400700 1 --2- 172.21.15.44:0/3631204294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fb9dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.116+0000 7f9060a00700 1 -- 172.21.15.44:0/3631204294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 msgr2=0x56034fb9dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.116+0000 7f9060a00700 1 --2- 172.21.15.44:0/3631204294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fb9dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.116+0000 7f9060a00700 1 -- 172.21.15.44:0/3631204294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 msgr2=0x56034fba14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.116+0000 7f9060a00700 1 --2- 172.21.15.44:0/3631204294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fba14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.116+0000 7f9060a00700 1 -- 172.21.15.44:0/3631204294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56034fba3330 con 0x56034fa19df0 2024-05-27T09:34:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.116+0000 7f9061400700 1 --2- 172.21.15.44:0/3631204294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fb9dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.116+0000 7f905be00700 1 --2- 172.21.15.44:0/3631204294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fba14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:55.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.116+0000 7f9060a00700 1 --2- 172.21.15.44:0/3631204294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fa19df0 0x56034fbaee30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9050023820 tx=0x7f9050011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9214c617db61ec53 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:55.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.116+0000 7f9049600700 1 -- 172.21.15.44:0/3631204294 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905001e030 con 0x56034fa19df0 2024-05-27T09:34:55.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.117+0000 7f9049600700 1 -- 172.21.15.44:0/3631204294 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f905001f040 con 0x56034fa19df0 2024-05-27T09:34:55.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.117+0000 7f9049600700 1 -- 172.21.15.44:0/3631204294 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905000fe50 con 0x56034fa19df0 2024-05-27T09:34:55.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.117+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3631204294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fa19df0 msgr2=0x56034fbaee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:55.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.117+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/3631204294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fa19df0 0x56034fbaee30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9050023820 tx=0x7f9050011440 comp rx=0 tx=0).stop 2024-05-27T09:34:55.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.118+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3631204294 shutdown_connections 2024-05-27T09:34:55.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.118+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/3631204294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fb98e20 0x56034fb9dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.118+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/3631204294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fa19df0 0x56034fbaee30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.118+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/3631204294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb059a0 0x56034fba14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.118+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3631204294 >> 172.21.15.44:0/3631204294 conn(0x56034fab3340 msgr2=0x56034fb98060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:55.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.118+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3631204294 shutdown_connections 2024-05-27T09:34:55.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.118+0000 7f9071f2ab80 1 -- 172.21.15.44:0/3631204294 wait complete. 2024-05-27T09:34:55.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.118+0000 7f9071f2ab80 1 -- start start 2024-05-27T09:34:55.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.118+0000 7f9071f2ab80 1 Processor -- start 2024-05-27T09:34:55.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fa19df0 0x56034fbbd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fb059a0 0x56034fbb6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9071f2ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb98e20 0x56034fbb6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56034fbb3ae0 con 0x56034fb98e20 2024-05-27T09:34:55.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56034fbb3410 con 0x56034fb059a0 2024-05-27T09:34:55.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9071f2ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56034fbb41b0 con 0x56034fa19df0 2024-05-27T09:34:55.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9071f2ab80 10 client.0 _ll_register_callbacks cb 0x56034fa60c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:34:55.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9061400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb98e20 0x56034fbb6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9060a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fa19df0 0x56034fbbd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9061400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb98e20 0x56034fbb6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39230/0 (socket says 172.21.15.44:39230) 2024-05-27T09:34:55.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9061400700 1 -- 172.21.15.44:0/27160334 learned_addr learned my addr 172.21.15.44:0/27160334 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:55.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f9060a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fa19df0 0x56034fbbd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:37766/0 (socket says 172.21.15.44:37766) 2024-05-27T09:34:55.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.119+0000 7f905be00700 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fb059a0 0x56034fbb6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.120+0000 7f9060a00700 1 -- 172.21.15.44:0/27160334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fb059a0 msgr2=0x56034fbb6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:55.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.120+0000 7f9060a00700 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fb059a0 0x56034fbb6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.120+0000 7f9060a00700 1 -- 172.21.15.44:0/27160334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb98e20 msgr2=0x56034fbb6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:55.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.120+0000 7f9060a00700 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb98e20 0x56034fbb6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.120+0000 7f9060a00700 1 -- 172.21.15.44:0/27160334 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56034fbb7490 con 0x56034fa19df0 2024-05-27T09:34:55.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.120+0000 7f9061400700 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb98e20 0x56034fbb6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:55.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.120+0000 7f9060a00700 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fa19df0 0x56034fbbd620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x56034fb99aa0 tx=0x7f90500236b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=578f8e78456838bf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:55.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.120+0000 7f903ea00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:34:55.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.120+0000 7f903ea00700 1 -- 172.21.15.44:0/27160334 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905001e040 con 0x56034fa19df0 2024-05-27T09:34:55.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.120+0000 7f9071f2ab80 1 -- 172.21.15.44:0/27160334 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f905001f040 con 0x56034fa19df0 2024-05-27T09:34:55.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f903ea00700 1 -- 172.21.15.44:0/27160334 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f9050004b00 con 0x56034fa19df0 2024-05-27T09:34:55.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f903ea00700 1 -- 172.21.15.44:0/27160334 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9050004d20 con 0x56034fa19df0 2024-05-27T09:34:55.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f9071f2ab80 10 client.25018 fetch_fsmap learned FSMap version 25 2024-05-27T09:34:55.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f9071f2ab80 1 -- 172.21.15.44:0/27160334 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56034fbbbe90 con 0x56034fa19df0 2024-05-27T09:34:55.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f903ea00700 1 -- 172.21.15.44:0/27160334 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f905001f040 con 0x56034fa19df0 2024-05-27T09:34:55.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f903ea00700 1 -- 172.21.15.44:0/27160334 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f9050038030 con 0x56034fa19df0 2024-05-27T09:34:55.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f9071f2ab80 10 client.25018 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:34:55.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f9071f2ab80 10 client.25018 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T09:34:55.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f9071f2ab80 20 client.25018 populate_metadata read hostname 'smithi044' 2024-05-27T09:34:55.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56034fc1fdf0 0x56034fc21260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:55.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f9071f2ab80 4 client.25018 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T09:34:55.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.121+0000 7f9071f2ab80 1 -- 172.21.15.44:0/27160334 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "status"}) -- 0x56034fac2520 con 0x56034fc1fdf0 2024-05-27T09:34:55.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.122+0000 7f905be00700 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56034fc1fdf0 0x56034fc21260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:55.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.122+0000 7f905be00700 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56034fc1fdf0 0x56034fc21260 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc36e8d9010af39 server_cookie=5aa7b0148394c703 in_seq=0 out_seq=0 2024-05-27T09:34:55.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.122+0000 7f903ea00700 10 client.25018 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:34:55.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f903ea00700 1 -- 172.21.15.44:0/27160334 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56034fac2520 con 0x56034fc1fdf0 2024-05-27T09:34:55.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f903ea00700 10 client.25018 handle_command_reply: tid=0 2024-05-27T09:34:55.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 1 client.25018 shutdown 2024-05-27T09:34:55.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 1 -- 172.21.15.44:0/27160334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fa19df0 msgr2=0x56034fbbd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:55.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fa19df0 0x56034fbbd620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x56034fb99aa0 tx=0x7f90500236b0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 1 -- 172.21.15.44:0/27160334 shutdown_connections 2024-05-27T09:34:55.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56034fc1fdf0 0x56034fc21260 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56034fb98e20 0x56034fbb6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56034fa19df0 0x56034fbbd620 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 1 --2- 172.21.15.44:0/27160334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56034fb059a0 0x56034fbb6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f903ea00700 0 client.25018 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:34:55.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 1 -- 172.21.15.44:0/27160334 >> 172.21.15.44:0/27160334 conn(0x56034fab3340 msgr2=0x56034fba2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:55.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 1 -- 172.21.15.44:0/27160334 shutdown_connections 2024-05-27T09:34:55.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 1 -- 172.21.15.44:0/27160334 wait complete. 2024-05-27T09:34:55.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.123+0000 7f9071f2ab80 20 client.25018 trim_cache size 0 max 16384 2024-05-27T09:34:55.129 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:34:55.129 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:34:55.130 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 2, 2024-05-27T09:34:55.130 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24520, 2024-05-27T09:34:55.130 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-05-27T09:34:55.130 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-05-27T09:34:55.130 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-05-27T09:34:55.130 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 191.38045649200001, 2024-05-27T09:34:55.130 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 25, 2024-05-27T09:34:55.130 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 81, 2024-05-27T09:34:55.130 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 79, 2024-05-27T09:34:55.130 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 246.62256704500001 2024-05-27T09:34:55.130 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:34:55.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 -- 172.21.15.44:0/1255770133 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f90440b2900 msgr2=0x7f90440b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:55.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f90440b2900 0x7f90440b4e10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f904c006010 tx=0x7f904c009510 comp rx=0 tx=0).stop 2024-05-27T09:34:55.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 -- 172.21.15.44:0/1255770133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f905c110960 msgr2=0x7f905c1a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:55.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f905c110960 0x7f905c1a5c60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f905400ef30 tx=0x7f905400c3a0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 -- 172.21.15.44:0/1255770133 shutdown_connections 2024-05-27T09:34:55.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f90440b2900 0x7f90440b4e10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f905c110960 0x7f905c1a5c60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f905c10a980 0x7f905c1a1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 --2- 172.21.15.44:0/1255770133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f905c10b2d0 0x7f905c1a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:55.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 -- 172.21.15.44:0/1255770133 >> 172.21.15.44:0/1255770133 conn(0x7f905c077c70 msgr2=0x7f905c07b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:55.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 -- 172.21.15.44:0/1255770133 shutdown_connections 2024-05-27T09:34:55.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:55.124+0000 7f9063400700 1 -- 172.21.15.44:0/1255770133 wait complete. 2024-05-27T09:34:55.997 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-05-27T09:34:56.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:55 smithi044 ceph-mon[31618]: pgmap v430: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:55 smithi189 ceph-mon[36671]: pgmap v430: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:56.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:55 smithi050 ceph-mon[36371]: pgmap v430: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T09:34:56.464 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:34:57.913 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:57 smithi044 ceph-mon[31618]: pgmap v431: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:58.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:57 smithi189 ceph-mon[36671]: pgmap v431: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:57 smithi050 ceph-mon[36371]: pgmap v431: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:34:58.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.671+0000 7f66d1200700 1 -- 172.21.15.44:0/3430132025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66cc110bf0 msgr2=0x7f66cc113010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:58.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.671+0000 7f66d1200700 1 --2- 172.21.15.44:0/3430132025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66cc110bf0 0x7f66cc113010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f66c400b0a0 tx=0x7f66c400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.671+0000 7f66d1200700 1 -- 172.21.15.44:0/3430132025 shutdown_connections 2024-05-27T09:34:58.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.671+0000 7f66d1200700 1 --2- 172.21.15.44:0/3430132025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66cc110bf0 0x7f66cc113010 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.671+0000 7f66d1200700 1 --2- 172.21.15.44:0/3430132025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66cc076be0 0x7f66cc076ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.671+0000 7f66d1200700 1 --2- 172.21.15.44:0/3430132025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66cc077530 0x7f66cc1106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.671+0000 7f66d1200700 1 -- 172.21.15.44:0/3430132025 >> 172.21.15.44:0/3430132025 conn(0x7f66cc0ffe90 msgr2=0x7f66cc1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:58.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.672+0000 7f66d1200700 1 -- 172.21.15.44:0/3430132025 shutdown_connections 2024-05-27T09:34:58.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.672+0000 7f66d1200700 1 -- 172.21.15.44:0/3430132025 wait complete. 2024-05-27T09:34:58.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.673+0000 7f66d1200700 1 Processor -- start 2024-05-27T09:34:58.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.674+0000 7f66d1200700 1 -- start start 2024-05-27T09:34:58.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.674+0000 7f66d1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66cc076be0 0x7f66cc19cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.674+0000 7f66d1200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66cc077530 0x7f66cc19d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.674+0000 7f66d1200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66cc110bf0 0x7f66cc1a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.674+0000 7f66d1200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f66cc116680 con 0x7f66cc077530 2024-05-27T09:34:58.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.674+0000 7f66d1200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f66cc116500 con 0x7f66cc110bf0 2024-05-27T09:34:58.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.674+0000 7f66d1200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66cc116800 con 0x7f66cc076be0 2024-05-27T09:34:58.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.675+0000 7f66c9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66cc076be0 0x7f66cc19cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.675+0000 7f66c9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66cc077530 0x7f66cc19d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.675+0000 7f66c9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66cc076be0 0x7f66cc19cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:37792/0 (socket says 172.21.15.44:37792) 2024-05-27T09:34:58.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.675+0000 7f66c9e00700 1 -- 172.21.15.44:0/3505529799 learned_addr learned my addr 172.21.15.44:0/3505529799 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:58.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.675+0000 7f66ca800700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66cc110bf0 0x7f66cc1a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.675+0000 7f66c9e00700 1 -- 172.21.15.44:0/3505529799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66cc110bf0 msgr2=0x7f66cc1a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:58.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.675+0000 7f66c9e00700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66cc110bf0 0x7f66cc1a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.675+0000 7f66c9e00700 1 -- 172.21.15.44:0/3505529799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66cc077530 msgr2=0x7f66cc19d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:58.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.675+0000 7f66c9e00700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66cc077530 0x7f66cc19d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.675+0000 7f66c9e00700 1 -- 172.21.15.44:0/3505529799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66cc1a1e60 con 0x7f66cc076be0 2024-05-27T09:34:58.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.676+0000 7f66c9e00700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66cc076be0 0x7f66cc19cf90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00e9a0 tx=0x7f66bc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:58.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.676+0000 7f66c2a00700 1 -- 172.21.15.44:0/3505529799 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc00ca10 con 0x7f66cc076be0 2024-05-27T09:34:58.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.676+0000 7f66c2a00700 1 -- 172.21.15.44:0/3505529799 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66bc00cbb0 con 0x7f66cc076be0 2024-05-27T09:34:58.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.676+0000 7f66c2a00700 1 -- 172.21.15.44:0/3505529799 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc018680 con 0x7f66cc076be0 2024-05-27T09:34:58.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.676+0000 7f66d1200700 1 -- 172.21.15.44:0/3505529799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66cc1a2150 con 0x7f66cc076be0 2024-05-27T09:34:58.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.676+0000 7f66d1200700 1 -- 172.21.15.44:0/3505529799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66cc1a9a70 con 0x7f66cc076be0 2024-05-27T09:34:58.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.680+0000 7f66c2a00700 1 -- 172.21.15.44:0/3505529799 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f66bc010430 con 0x7f66cc076be0 2024-05-27T09:34:58.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.681+0000 7f66dfdb8b80 1 Processor -- start 2024-05-27T09:34:58.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.681+0000 7f66dfdb8b80 1 -- start start 2024-05-27T09:34:58.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.681+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x5588812f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x5588812f6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x5588813030c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558881224df0 con 0x5588812639a0 2024-05-27T09:34:58.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558881214990 con 0x558881177df0 2024-05-27T09:34:58.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5588812189d0 con 0x5588812f6e20 2024-05-27T09:34:58.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66c2a00700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f66b00b2880 0x7f66b00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66c9e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x5588812f6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66c9400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x5588813030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66ca800700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f66b00b2880 0x7f66b00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66c9400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x5588813030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:37802/0 (socket says 172.21.15.44:37802) 2024-05-27T09:34:58.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66c9400700 1 -- 172.21.15.44:0/489336172 learned_addr learned my addr 172.21.15.44:0/489336172 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:58.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66c9e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x5588812f6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45634/0 (socket says 172.21.15.44:45634) 2024-05-27T09:34:58.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66c2a00700 1 -- 172.21.15.44:0/3505529799 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f66bc0e23b0 con 0x7f66cc076be0 2024-05-27T09:34:58.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.682+0000 7f66ca800700 1 --2- 172.21.15.44:0/489336172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x5588812f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.683+0000 7f66c9400700 1 -- 172.21.15.44:0/489336172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 msgr2=0x5588812f6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:58.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.683+0000 7f66c9400700 1 --2- 172.21.15.44:0/489336172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x5588812f6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.683+0000 7f66c9400700 1 -- 172.21.15.44:0/489336172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 msgr2=0x5588812f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:58.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.683+0000 7f66c9400700 1 --2- 172.21.15.44:0/489336172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x5588812f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.683+0000 7f66c9400700 1 -- 172.21.15.44:0/489336172 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588811dd3d0 con 0x5588812f6e20 2024-05-27T09:34:58.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.683+0000 7f66c9e00700 1 --2- 172.21.15.44:0/489336172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x5588812f6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:58.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.683+0000 7f66c9400700 1 --2- 172.21.15.44:0/489336172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x5588813030c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009cb0 tx=0x7f66b8011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae0b11507911f27c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:58.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.683+0000 7f66ca800700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f66b00b2880 0x7f66b00b4d90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f66c4009fd0 tx=0x7f66c4009b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:58.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.683+0000 7f66b7400700 1 -- 172.21.15.44:0/489336172 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8011d20 con 0x5588812f6e20 2024-05-27T09:34:58.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.683+0000 7f66b7400700 1 -- 172.21.15.44:0/489336172 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66b8004500 con 0x5588812f6e20 2024-05-27T09:34:58.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.683+0000 7f66b7400700 1 -- 172.21.15.44:0/489336172 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8005260 con 0x5588812f6e20 2024-05-27T09:34:58.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.684+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/489336172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 msgr2=0x5588813030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:58.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.684+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/489336172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x5588813030c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009cb0 tx=0x7f66b8011440 comp rx=0 tx=0).stop 2024-05-27T09:34:58.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.684+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/489336172 shutdown_connections 2024-05-27T09:34:58.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.684+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/489336172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x5588813030c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.684+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/489336172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x5588812f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.684+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/489336172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x5588812f6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.684+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/489336172 >> 172.21.15.44:0/489336172 conn(0x558881211340 msgr2=0x5588812f2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:58.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.685+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/489336172 shutdown_connections 2024-05-27T09:34:58.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.685+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/489336172 wait complete. 2024-05-27T09:34:58.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.685+0000 7f66dfdb8b80 1 -- start start 2024-05-27T09:34:58.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.686+0000 7f66dfdb8b80 1 Processor -- start 2024-05-27T09:34:58.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.686+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x558881300630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.686+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x558881300b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.686+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x5588813010b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.686+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55888111af50 con 0x5588812639a0 2024-05-27T09:34:58.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.686+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5588811b5710 con 0x558881177df0 2024-05-27T09:34:58.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.686+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558881303b30 con 0x5588812f6e20 2024-05-27T09:34:58.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.686+0000 7f66dfdb8b80 10 client.0 _ll_register_callbacks cb 0x5588811bec20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:34:58.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66c9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x558881300b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66c9e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x558881300630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66c9e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x558881300630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45638/0 (socket says 172.21.15.44:45638) 2024-05-27T09:34:58.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66c9e00700 1 -- 172.21.15.44:0/2469904429 learned_addr learned my addr 172.21.15.44:0/2469904429 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:58.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66ca800700 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x5588813010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66c9e00700 1 -- 172.21.15.44:0/2469904429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 msgr2=0x5588813010b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:58.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66c9e00700 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x5588813010b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66c9e00700 1 -- 172.21.15.44:0/2469904429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 msgr2=0x558881300b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:58.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66c9e00700 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x558881300b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66c9e00700 1 -- 172.21.15.44:0/2469904429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558881164880 con 0x558881177df0 2024-05-27T09:34:58.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66c9e00700 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x558881300630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00e970 tx=0x7f66bc014ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c0614c29663e2e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:58.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.687+0000 7f66abe00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:34:58.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.688+0000 7f66abe00700 1 -- 172.21.15.44:0/2469904429 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc0a9980 con 0x558881177df0 2024-05-27T09:34:58.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.688+0000 7f66abe00700 1 -- 172.21.15.44:0/2469904429 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f66bc013dc0 con 0x558881177df0 2024-05-27T09:34:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.688+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2469904429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66b8004500 con 0x558881177df0 2024-05-27T09:34:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.688+0000 7f66abe00700 1 -- 172.21.15.44:0/2469904429 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f66bc0add30 con 0x558881177df0 2024-05-27T09:34:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.688+0000 7f66dfdb8b80 10 client.25016 fetch_fsmap learned FSMap version 25 2024-05-27T09:34:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.688+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2469904429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5588811e5010 con 0x558881177df0 2024-05-27T09:34:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.688+0000 7f66abe00700 1 -- 172.21.15.44:0/2469904429 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66b8004500 con 0x558881177df0 2024-05-27T09:34:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.688+0000 7f66abe00700 1 -- 172.21.15.44:0/2469904429 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f66bc0a8be0 con 0x558881177df0 2024-05-27T09:34:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.689+0000 7f66dfdb8b80 10 client.25016 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:34:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.689+0000 7f66dfdb8b80 10 client.25016 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T09:34:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.689+0000 7f66dfdb8b80 20 client.25016 populate_metadata read hostname 'smithi044' 2024-05-27T09:34:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.689+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x558881310ad0 0x558881311f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.689+0000 7f66dfdb8b80 4 client.25016 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:34:58.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.689+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2469904429 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558881226d40 con 0x558881310ad0 2024-05-27T09:34:58.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.689+0000 7f66c9400700 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x558881310ad0 0x558881311f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.690+0000 7f66c9400700 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x558881310ad0 0x558881311f40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6c54875c91b52d9a server_cookie=908ecd62dc642c7 in_seq=0 out_seq=0 2024-05-27T09:34:58.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.690+0000 7f66abe00700 10 client.25016 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:34:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66abe00700 1 -- 172.21.15.44:0/2469904429 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558881226d40 con 0x558881310ad0 2024-05-27T09:34:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66abe00700 10 client.25016 handle_command_reply: tid=0 2024-05-27T09:34:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66dfdb8b80 1 client.25016 shutdown 2024-05-27T09:34:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2469904429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 msgr2=0x558881300630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x558881300630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00e970 tx=0x7f66bc014ed0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2469904429 shutdown_connections 2024-05-27T09:34:58.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x5588813010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x558881300630 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x558881310ad0 0x558881311f40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2469904429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x558881300b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2469904429 >> 172.21.15.44:0/2469904429 conn(0x558881211340 msgr2=0x5588812f62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:58.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66abe00700 0 client.25016 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:34:58.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.691+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2469904429 shutdown_connections 2024-05-27T09:34:58.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.692+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2469904429 wait complete. 2024-05-27T09:34:58.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.692+0000 7f66dfdb8b80 20 client.25016 trim_cache size 0 max 16384 2024-05-27T09:34:58.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.715+0000 7f66dfdb8b80 1 Processor -- start 2024-05-27T09:34:58.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.715+0000 7f66dfdb8b80 1 -- start start 2024-05-27T09:34:58.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.716+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558881177df0 0x5588812f2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.716+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588812639a0 0x5588812fbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.716+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812f6e20 0x5588812f6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.716+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558881310280 con 0x5588812f6e20 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.716+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558881310950 con 0x5588812639a0 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.716+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558881311020 con 0x558881177df0 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.716+0000 7f66ca800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812f6e20 0x5588812f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.716+0000 7f66ca800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812f6e20 0x5588812f6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39254/0 (socket says 172.21.15.44:39254) 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.716+0000 7f66ca800700 1 -- 172.21.15.44:0/3957948264 learned_addr learned my addr 172.21.15.44:0/3957948264 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.716+0000 7f66c9400700 1 --2- 172.21.15.44:0/3957948264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588812639a0 0x5588812fbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.716+0000 7f66c9e00700 1 --2- 172.21.15.44:0/3957948264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558881177df0 0x5588812f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.717+0000 7f66ca800700 1 -- 172.21.15.44:0/3957948264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558881177df0 msgr2=0x5588812f2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.717+0000 7f66ca800700 1 --2- 172.21.15.44:0/3957948264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558881177df0 0x5588812f2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.717+0000 7f66ca800700 1 -- 172.21.15.44:0/3957948264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588812639a0 msgr2=0x5588812fbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.717+0000 7f66ca800700 1 --2- 172.21.15.44:0/3957948264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588812639a0 0x5588812fbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.717+0000 7f66ca800700 1 -- 172.21.15.44:0/3957948264 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558881301420 con 0x5588812f6e20 2024-05-27T09:34:58.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.717+0000 7f66c9400700 1 --2- 172.21.15.44:0/3957948264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588812639a0 0x5588812fbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:34:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.717+0000 7f66ca800700 1 --2- 172.21.15.44:0/3957948264 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812f6e20 0x5588812f6060 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f66c400b070 tx=0x7f66c40099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d053d4511e0b333 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.717+0000 7f66b6a00700 1 -- 172.21.15.44:0/3957948264 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c4031070 con 0x5588812f6e20 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.718+0000 7f66b6a00700 1 -- 172.21.15.44:0/3957948264 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66c4007b30 con 0x5588812f6e20 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.718+0000 7f66b6a00700 1 -- 172.21.15.44:0/3957948264 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c4020c20 con 0x5588812f6e20 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.718+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/3957948264 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812f6e20 msgr2=0x5588812f6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.718+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/3957948264 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812f6e20 0x5588812f6060 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f66c400b070 tx=0x7f66c40099e0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.718+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/3957948264 shutdown_connections 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.718+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/3957948264 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812f6e20 0x5588812f6060 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.718+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/3957948264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558881177df0 0x5588812f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.718+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/3957948264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588812639a0 0x5588812fbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.718+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/3957948264 >> 172.21.15.44:0/3957948264 conn(0x558881211340 msgr2=0x5588812f1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.719+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/3957948264 shutdown_connections 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.719+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/3957948264 wait complete. 2024-05-27T09:34:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.719+0000 7f66dfdb8b80 1 -- start start 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.719+0000 7f66dfdb8b80 1 Processor -- start 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x55888131b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x55888131b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66dfdb8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x558881314360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558881311dc0 con 0x5588812639a0 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5588813116f0 con 0x558881177df0 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66dfdb8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558881312490 con 0x5588812f6e20 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66dfdb8b80 10 client.0 _ll_register_callbacks cb 0x5588811bec20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66c9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x55888131b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66c9e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x55888131b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66c9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x55888131b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39256/0 (socket says 172.21.15.44:39256) 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66c9400700 1 -- 172.21.15.44:0/2590187089 learned_addr learned my addr 172.21.15.44:0/2590187089 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:34:58.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.720+0000 7f66ca800700 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x558881314360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.721+0000 7f66c9e00700 1 -- 172.21.15.44:0/2590187089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 msgr2=0x558881314360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.721+0000 7f66c9e00700 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x558881314360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.721+0000 7f66c9e00700 1 -- 172.21.15.44:0/2590187089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 msgr2=0x55888131b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:34:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.721+0000 7f66c9e00700 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x55888131b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.721+0000 7f66c9e00700 1 -- 172.21.15.44:0/2590187089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588813148a0 con 0x558881177df0 2024-05-27T09:34:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.721+0000 7f66c9e00700 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x55888131b3c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f66bc0a6bf0 tx=0x7f66bc0a95f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e77e701a9e13aac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:34:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.721+0000 7f66abe00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:34:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.721+0000 7f66abe00700 1 -- 172.21.15.44:0/2590187089 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc0cc2e0 con 0x558881177df0 2024-05-27T09:34:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.721+0000 7f66abe00700 1 -- 172.21.15.44:0/2590187089 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f66bc0b2350 con 0x558881177df0 2024-05-27T09:34:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.721+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2590187089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66c4007b30 con 0x558881177df0 2024-05-27T09:34:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.722+0000 7f66abe00700 1 -- 172.21.15.44:0/2590187089 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f66bc0b27e0 con 0x558881177df0 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.722+0000 7f66dfdb8b80 10 client.25022 fetch_fsmap learned FSMap version 25 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.722+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2590187089 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55888137c9c0 con 0x558881177df0 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.722+0000 7f66abe00700 1 -- 172.21.15.44:0/2590187089 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66c4007b30 con 0x558881177df0 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.722+0000 7f66abe00700 1 -- 172.21.15.44:0/2590187089 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f66bc0a54f0 con 0x558881177df0 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.722+0000 7f66dfdb8b80 10 client.25022 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.722+0000 7f66dfdb8b80 10 client.25022 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.722+0000 7f66dfdb8b80 20 client.25022 populate_metadata read hostname 'smithi044' 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.722+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55888137ec00 0x558881380070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.722+0000 7f66dfdb8b80 4 client.25022 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.722+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2590187089 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "status"}) -- 0x558881220520 con 0x55888137ec00 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.723+0000 7f66c9400700 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55888137ec00 0x558881380070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.723+0000 7f66c9400700 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55888137ec00 0x558881380070 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=11bd7a4996594998 server_cookie=ebda08623f545663 in_seq=0 out_seq=0 2024-05-27T09:34:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.723+0000 7f66abe00700 10 client.25022 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.723+0000 7f66abe00700 1 -- 172.21.15.44:0/2590187089 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558881220520 con 0x55888137ec00 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.723+0000 7f66abe00700 10 client.25022 handle_command_reply: tid=0 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.723+0000 7f66dfdb8b80 1 client.25022 shutdown 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2590187089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 msgr2=0x55888131b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x55888131b3c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f66bc0a6bf0 tx=0x7f66bc0a95f0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2590187089 shutdown_connections 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588812f6e20 0x558881314360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558881177df0 0x55888131b3c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55888137ec00 0x558881380070 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66dfdb8b80 1 --2- 172.21.15.44:0/2590187089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5588812639a0 0x55888131b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2590187089 >> 172.21.15.44:0/2590187089 conn(0x558881211340 msgr2=0x5588812f6060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:58.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66abe00700 0 client.25022 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:34:58.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2590187089 shutdown_connections 2024-05-27T09:34:58.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66dfdb8b80 1 -- 172.21.15.44:0/2590187089 wait complete. 2024-05-27T09:34:58.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.724+0000 7f66dfdb8b80 20 client.25022 trim_cache size 0 max 16384 2024-05-27T09:34:58.730 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:34:58.730 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:34:58.730 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 1, 2024-05-27T09:34:58.730 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24535, 2024-05-27T09:34:58.730 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-05-27T09:34:58.730 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-05-27T09:34:58.731 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-05-27T09:34:58.731 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 195.99322848899999, 2024-05-27T09:34:58.731 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 25, 2024-05-27T09:34:58.731 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 81, 2024-05-27T09:34:58.731 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 79, 2024-05-27T09:34:58.731 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 244.71751161099999 2024-05-27T09:34:58.731 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:34:58.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 -- 172.21.15.44:0/3505529799 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f66b00b2880 msgr2=0x7f66b00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f66b00b2880 0x7f66b00b4d90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f66c4009fd0 tx=0x7f66c4009b80 comp rx=0 tx=0).stop 2024-05-27T09:34:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 -- 172.21.15.44:0/3505529799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66cc076be0 msgr2=0x7f66cc19cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:34:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66cc076be0 0x7f66cc19cf90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00e9a0 tx=0x7f66bc00ecb0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 -- 172.21.15.44:0/3505529799 shutdown_connections 2024-05-27T09:34:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f66b00b2880 0x7f66b00b4d90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66cc110bf0 0x7f66cc1a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66cc076be0 0x7f66cc19cf90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 --2- 172.21.15.44:0/3505529799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66cc077530 0x7f66cc19d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:34:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 -- 172.21.15.44:0/3505529799 >> 172.21.15.44:0/3505529799 conn(0x7f66cc0ffe90 msgr2=0x7f66cc100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:34:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 -- 172.21.15.44:0/3505529799 shutdown_connections 2024-05-27T09:34:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:34:58.725+0000 7f66d1200700 1 -- 172.21.15.44:0/3505529799 wait complete. 2024-05-27T09:34:59.599 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T09:34:59.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:34:59 smithi044 ceph-mon[31618]: pgmap v432: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:35:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:00.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:34:59 smithi189 ceph-mon[36671]: pgmap v432: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:35:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:34:59 smithi050 ceph-mon[36371]: pgmap v432: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:35:02.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.041+0000 7f9409e00700 1 -- 172.21.15.44:0/963151053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9404105c30 msgr2=0x7f940410e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:02.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.041+0000 7f9409e00700 1 --2- 172.21.15.44:0/963151053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9404105c30 0x7f940410e180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f93f00098f0 tx=0x7f93f0009c00 comp rx=0 tx=0).stop 2024-05-27T09:35:02.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.042+0000 7f9409e00700 1 -- 172.21.15.44:0/963151053 shutdown_connections 2024-05-27T09:35:02.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.042+0000 7f9409e00700 1 --2- 172.21.15.44:0/963151053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f940410e710 0x7f9404110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.042+0000 7f9409e00700 1 --2- 172.21.15.44:0/963151053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94041052e0 0x7f94041056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.042+0000 7f9409e00700 1 --2- 172.21.15.44:0/963151053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9404105c30 0x7f940410e180 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.042+0000 7f9409e00700 1 -- 172.21.15.44:0/963151053 >> 172.21.15.44:0/963151053 conn(0x7f9404078af0 msgr2=0x7f9404078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:02.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.042+0000 7f9409e00700 1 -- 172.21.15.44:0/963151053 shutdown_connections 2024-05-27T09:35:02.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.042+0000 7f9409e00700 1 -- 172.21.15.44:0/963151053 wait complete. 2024-05-27T09:35:02.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:01 smithi044 ceph-mon[31618]: pgmap v433: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:35:02.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.043+0000 7f9409e00700 1 Processor -- start 2024-05-27T09:35:02.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.043+0000 7f9409e00700 1 -- start start 2024-05-27T09:35:02.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.043+0000 7f9409e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94041052e0 0x7f940419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9409e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9404105c30 0x7f940419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9409e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f940410e710 0x7f94041a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9409e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f94041142c0 con 0x7f94041052e0 2024-05-27T09:35:02.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9409e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9404114140 con 0x7f9404105c30 2024-05-27T09:35:02.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9409e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9404114440 con 0x7f940410e710 2024-05-27T09:35:02.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9408a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94041052e0 0x7f940419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9408a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94041052e0 0x7f940419ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39270/0 (socket says 172.21.15.44:39270) 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9408a00700 1 -- 172.21.15.44:0/886678788 learned_addr learned my addr 172.21.15.44:0/886678788 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9403e00700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9404105c30 0x7f940419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9409400700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f940410e710 0x7f94041a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9408a00700 1 -- 172.21.15.44:0/886678788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f940410e710 msgr2=0x7f94041a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9408a00700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f940410e710 0x7f94041a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9408a00700 1 -- 172.21.15.44:0/886678788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9404105c30 msgr2=0x7f940419d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9408a00700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9404105c30 0x7f940419d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.044+0000 7f9408a00700 1 -- 172.21.15.44:0/886678788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94041a1d80 con 0x7f94041052e0 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.045+0000 7f9408a00700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94041052e0 0x7f940419ceb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f93f800e9c0 tx=0x7f93f800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.045+0000 7f9401600700 1 -- 172.21.15.44:0/886678788 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f8004a30 con 0x7f94041052e0 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.045+0000 7f9409e00700 1 -- 172.21.15.44:0/886678788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94041a2070 con 0x7f94041052e0 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.045+0000 7f9401600700 1 -- 172.21.15.44:0/886678788 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93f800cc70 con 0x7f94041052e0 2024-05-27T09:35:02.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.045+0000 7f9401600700 1 -- 172.21.15.44:0/886678788 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f8005580 con 0x7f94041052e0 2024-05-27T09:35:02.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.045+0000 7f9409e00700 1 -- 172.21.15.44:0/886678788 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94041a9990 con 0x7f94041052e0 2024-05-27T09:35:02.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.049+0000 7f9401600700 1 -- 172.21.15.44:0/886678788 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f93f800c7c0 con 0x7f94041052e0 2024-05-27T09:35:02.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.050+0000 7f9419fbcb80 1 Processor -- start 2024-05-27T09:35:02.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.050+0000 7f9419fbcb80 1 -- start start 2024-05-27T09:35:02.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.050+0000 7f9401600700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f93ec0b2880 0x7f93ec0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb7599700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb7599c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb75a5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562bb74c6df0 con 0x562bb75059a0 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562bb74b6990 con 0x562bb75a3270 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9401600700 1 -- 172.21.15.44:0/886678788 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f93f80e2320 con 0x7f94041052e0 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562bb74ba9d0 con 0x562bb7419df0 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9409400700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f93ec0b2880 0x7f93ec0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9409400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb75a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9408a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb7599700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9403e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb7599c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9409400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb75a5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45666/0 (socket says 172.21.15.44:45666) 2024-05-27T09:35:02.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9409400700 1 -- 172.21.15.44:0/2285475939 learned_addr learned my addr 172.21.15.44:0/2285475939 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:02.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.051+0000 7f9408a00700 1 -- 172.21.15.44:0/2285475939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 msgr2=0x562bb7599c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:02.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.052+0000 7f9408a00700 1 --2- 172.21.15.44:0/2285475939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb7599c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.052+0000 7f9408a00700 1 -- 172.21.15.44:0/2285475939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 msgr2=0x562bb75a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:02.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.052+0000 7f9408a00700 1 --2- 172.21.15.44:0/2285475939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb75a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.052+0000 7f9408a00700 1 -- 172.21.15.44:0/2285475939 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bb747f3d0 con 0x562bb75059a0 2024-05-27T09:35:02.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.052+0000 7f9409400700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f93ec0b2880 0x7f93ec0b4d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f93fc004a40 tx=0x7f93fc013540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:02.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.052+0000 7f9408a00700 1 --2- 172.21.15.44:0/2285475939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb7599700 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f93f80dfdd0 tx=0x7f93f80a90e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade5f426f06dd86 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:02.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.052+0000 7f93f6000700 1 -- 172.21.15.44:0/2285475939 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f808fe60 con 0x562bb75059a0 2024-05-27T09:35:02.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.052+0000 7f93f6000700 1 -- 172.21.15.44:0/2285475939 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93f80b20a0 con 0x562bb75059a0 2024-05-27T09:35:02.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.052+0000 7f93f6000700 1 -- 172.21.15.44:0/2285475939 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f80a98f0 con 0x562bb75059a0 2024-05-27T09:35:02.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.053+0000 7f9419fbcb80 1 -- 172.21.15.44:0/2285475939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 msgr2=0x562bb7599700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:02.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.053+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/2285475939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb7599700 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f93f80dfdd0 tx=0x7f93f80a90e0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.053+0000 7f9419fbcb80 1 -- 172.21.15.44:0/2285475939 shutdown_connections 2024-05-27T09:35:02.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.053+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/2285475939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb75a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.053+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/2285475939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb7599700 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.053+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/2285475939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb7599c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.053+0000 7f9419fbcb80 1 -- 172.21.15.44:0/2285475939 >> 172.21.15.44:0/2285475939 conn(0x562bb74b3340 msgr2=0x562bb75946c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:02.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.053+0000 7f9419fbcb80 1 -- 172.21.15.44:0/2285475939 shutdown_connections 2024-05-27T09:35:02.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.053+0000 7f9419fbcb80 1 -- 172.21.15.44:0/2285475939 wait complete. 2024-05-27T09:35:02.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.054+0000 7f9419fbcb80 1 -- start start 2024-05-27T09:35:02.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.054+0000 7f9419fbcb80 1 Processor -- start 2024-05-27T09:35:02.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb7419df0 0x562bb75a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb75059a0 0x562bb75a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75a3270 0x562bb75a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562bb73bcf50 con 0x562bb75a3270 2024-05-27T09:35:02.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562bb7457710 con 0x562bb7419df0 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562bb75a5db0 con 0x562bb75059a0 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9419fbcb80 10 client.0 _ll_register_callbacks cb 0x562bb7460c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9409400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75a3270 0x562bb75a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9409400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75a3270 0x562bb75a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39282/0 (socket says 172.21.15.44:39282) 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9409400700 1 -- 172.21.15.44:0/1505467398 learned_addr learned my addr 172.21.15.44:0/1505467398 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9408a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb7419df0 0x562bb75a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.055+0000 7f9403e00700 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb75059a0 0x562bb75a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f9408a00700 1 -- 172.21.15.44:0/1505467398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb75059a0 msgr2=0x562bb75a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f9408a00700 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb75059a0 0x562bb75a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f9408a00700 1 -- 172.21.15.44:0/1505467398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75a3270 msgr2=0x562bb75a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f9408a00700 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75a3270 0x562bb75a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f9408a00700 1 -- 172.21.15.44:0/1505467398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bb7406880 con 0x562bb7419df0 2024-05-27T09:35:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f9409400700 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75a3270 0x562bb75a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f9408a00700 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb7419df0 0x562bb75a02a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f93f80a9880 tx=0x7f93f8004e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cccb42db3600ee7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f93eaa00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:35:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f93eaa00700 1 -- 172.21.15.44:0/1505467398 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f808fe60 con 0x562bb7419df0 2024-05-27T09:35:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f93eaa00700 1 -- 172.21.15.44:0/1505467398 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f93f80a9d60 con 0x562bb7419df0 2024-05-27T09:35:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f9419fbcb80 1 -- 172.21.15.44:0/1505467398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93f80b20a0 con 0x562bb7419df0 2024-05-27T09:35:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f93eaa00700 1 -- 172.21.15.44:0/1505467398 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f93f8005720 con 0x562bb7419df0 2024-05-27T09:35:02.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f9419fbcb80 10 client.25034 fetch_fsmap learned FSMap version 25 2024-05-27T09:35:02.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.056+0000 7f9419fbcb80 1 -- 172.21.15.44:0/1505467398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562bb7487010 con 0x562bb7419df0 2024-05-27T09:35:02.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.057+0000 7f93eaa00700 1 -- 172.21.15.44:0/1505467398 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93f80b20a0 con 0x562bb7419df0 2024-05-27T09:35:02.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.057+0000 7f93eaa00700 1 -- 172.21.15.44:0/1505467398 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f93f80c9030 con 0x562bb7419df0 2024-05-27T09:35:02.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.057+0000 7f9419fbcb80 10 client.25034 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:35:02.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.057+0000 7f9419fbcb80 10 client.25034 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T09:35:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.057+0000 7f9419fbcb80 20 client.25034 populate_metadata read hostname 'smithi044' 2024-05-27T09:35:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.057+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562bb75b1a50 0x562bb75b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.057+0000 7f9419fbcb80 4 client.25034 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:35:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.057+0000 7f9419fbcb80 1 -- 172.21.15.44:0/1505467398 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bb74c8d40 con 0x562bb75b1a50 2024-05-27T09:35:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.058+0000 7f9403e00700 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562bb75b1a50 0x562bb75b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.058+0000 7f9403e00700 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562bb75b1a50 0x562bb75b3c00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ee0566b3913657e server_cookie=4cacd7dd1c6dd221 in_seq=0 out_seq=0 2024-05-27T09:35:02.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.058+0000 7f93eaa00700 10 client.25034 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:02.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f93eaa00700 1 -- 172.21.15.44:0/1505467398 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562bb74c8d40 con 0x562bb75b1a50 2024-05-27T09:35:02.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f93eaa00700 10 client.25034 handle_command_reply: tid=0 2024-05-27T09:35:02.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f9419fbcb80 1 client.25034 shutdown 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f9419fbcb80 1 -- 172.21.15.44:0/1505467398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb7419df0 msgr2=0x562bb75a02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb7419df0 0x562bb75a02a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f93f80a9880 tx=0x7f93f8004e00 comp rx=0 tx=0).stop 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f9419fbcb80 1 -- 172.21.15.44:0/1505467398 shutdown_connections 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562bb75b1a50 0x562bb75b3c00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75a3270 0x562bb75a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb7419df0 0x562bb75a02a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/1505467398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb75059a0 0x562bb75a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f93eaa00700 0 client.25034 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.059+0000 7f9419fbcb80 1 -- 172.21.15.44:0/1505467398 >> 172.21.15.44:0/1505467398 conn(0x562bb74b3340 msgr2=0x562bb75997a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.060+0000 7f9419fbcb80 1 -- 172.21.15.44:0/1505467398 shutdown_connections 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.060+0000 7f9419fbcb80 1 -- 172.21.15.44:0/1505467398 wait complete. 2024-05-27T09:35:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.060+0000 7f9419fbcb80 20 client.25034 trim_cache size 0 max 16384 2024-05-27T09:35:02.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.082+0000 7f9419fbcb80 1 Processor -- start 2024-05-27T09:35:02.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.082+0000 7f9419fbcb80 1 -- start start 2024-05-27T09:35:02.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.082+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb759f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.082+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75ae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.082+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb759d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.082+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562bb75b1f40 con 0x562bb75059a0 2024-05-27T09:35:02.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.082+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562bb75b2610 con 0x562bb75a3270 2024-05-27T09:35:02.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.082+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562bb75b2ce0 con 0x562bb7419df0 2024-05-27T09:35:02.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.083+0000 7f9403e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.083+0000 7f9403e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75ae750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39292/0 (socket says 172.21.15.44:39292) 2024-05-27T09:35:02.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.083+0000 7f9403e00700 1 -- 172.21.15.44:0/966245725 learned_addr learned my addr 172.21.15.44:0/966245725 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:02.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.083+0000 7f9409400700 1 --2- 172.21.15.44:0/966245725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb759d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.083+0000 7f9408a00700 1 --2- 172.21.15.44:0/966245725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb759f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.083+0000 7f9403e00700 1 -- 172.21.15.44:0/966245725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 msgr2=0x562bb759f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:02.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.083+0000 7f9403e00700 1 --2- 172.21.15.44:0/966245725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb759f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.083+0000 7f9403e00700 1 -- 172.21.15.44:0/966245725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 msgr2=0x562bb759d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:02.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.083+0000 7f9403e00700 1 --2- 172.21.15.44:0/966245725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb759d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.083+0000 7f9403e00700 1 -- 172.21.15.44:0/966245725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bb75a1090 con 0x562bb75059a0 2024-05-27T09:35:02.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.083+0000 7f9403e00700 1 --2- 172.21.15.44:0/966245725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75ae750 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009230 tx=0x7f93f000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be33ec1c8cd7e7ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:02.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.084+0000 7f93f5600700 1 -- 172.21.15.44:0/966245725 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f000f7a0 con 0x562bb75059a0 2024-05-27T09:35:02.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.084+0000 7f93f5600700 1 -- 172.21.15.44:0/966245725 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93f000f940 con 0x562bb75059a0 2024-05-27T09:35:02.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.084+0000 7f93f5600700 1 -- 172.21.15.44:0/966245725 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f0003b90 con 0x562bb75059a0 2024-05-27T09:35:02.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.085+0000 7f9419fbcb80 1 -- 172.21.15.44:0/966245725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 msgr2=0x562bb75ae750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:02.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.085+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/966245725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75ae750 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009230 tx=0x7f93f000bf60 comp rx=0 tx=0).stop 2024-05-27T09:35:02.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.085+0000 7f9419fbcb80 1 -- 172.21.15.44:0/966245725 shutdown_connections 2024-05-27T09:35:02.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.085+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/966245725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb759d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.085+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/966245725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb759f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.085+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/966245725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75ae750 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.085+0000 7f9419fbcb80 1 -- 172.21.15.44:0/966245725 >> 172.21.15.44:0/966245725 conn(0x562bb74b3340 msgr2=0x562bb75ae040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:02.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.085+0000 7f9419fbcb80 1 -- 172.21.15.44:0/966245725 shutdown_connections 2024-05-27T09:35:02.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.085+0000 7f9419fbcb80 1 -- 172.21.15.44:0/966245725 wait complete. 2024-05-27T09:35:02.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.086+0000 7f9419fbcb80 1 -- start start 2024-05-27T09:35:02.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.086+0000 7f9419fbcb80 1 Processor -- start 2024-05-27T09:35:02.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.086+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb75b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.086+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75b7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.086+0000 7f9419fbcb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb75b8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.086+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562bb75b3a80 con 0x562bb75059a0 2024-05-27T09:35:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.086+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562bb75b33b0 con 0x562bb75a3270 2024-05-27T09:35:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9419fbcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562bb75b4150 con 0x562bb7419df0 2024-05-27T09:35:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9419fbcb80 10 client.0 _ll_register_callbacks cb 0x562bb7460c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:35:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9403e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75b7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9403e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75b7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39300/0 (socket says 172.21.15.44:39300) 2024-05-27T09:35:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9403e00700 1 -- 172.21.15.44:0/3918226817 learned_addr learned my addr 172.21.15.44:0/3918226817 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9409400700 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb75b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9408a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb75b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9403e00700 1 -- 172.21.15.44:0/3918226817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 msgr2=0x562bb75b7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9403e00700 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb75b7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9403e00700 1 -- 172.21.15.44:0/3918226817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 msgr2=0x562bb75b8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9403e00700 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb75b8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9403e00700 1 -- 172.21.15.44:0/3918226817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bb75bb760 con 0x562bb75059a0 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.087+0000 7f9409400700 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb75b8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.088+0000 7f9403e00700 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75b7b40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009c40 tx=0x7f93f000bae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e101ef57081fa8c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.088+0000 7f93eaa00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.088+0000 7f93eaa00700 1 -- 172.21.15.44:0/3918226817 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f0036070 con 0x562bb75059a0 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.088+0000 7f9419fbcb80 1 -- 172.21.15.44:0/3918226817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93f000f940 con 0x562bb75059a0 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.088+0000 7f93eaa00700 1 -- 172.21.15.44:0/3918226817 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f93f0005270 con 0x562bb75059a0 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.088+0000 7f93eaa00700 1 -- 172.21.15.44:0/3918226817 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f93f0042050 con 0x562bb75059a0 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.088+0000 7f9419fbcb80 10 client.14910 fetch_fsmap learned FSMap version 25 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.088+0000 7f9419fbcb80 1 -- 172.21.15.44:0/3918226817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562bb761e260 con 0x562bb75059a0 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.089+0000 7f93eaa00700 1 -- 172.21.15.44:0/3918226817 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93f000f940 con 0x562bb75059a0 2024-05-27T09:35:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.089+0000 7f93eaa00700 1 -- 172.21.15.44:0/3918226817 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f93f80cc0d0 con 0x562bb75059a0 2024-05-27T09:35:02.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.089+0000 7f9419fbcb80 10 client.14910 fetch_fsmap finished waiting for FSMap version 25 2024-05-27T09:35:02.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.089+0000 7f9419fbcb80 10 client.14910 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T09:35:02.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.089+0000 7f9419fbcb80 20 client.14910 populate_metadata read hostname 'smithi044' 2024-05-27T09:35:02.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.089+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562bb7620310 0x562bb7621780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:02.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.089+0000 7f9419fbcb80 4 client.14910 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T09:35:02.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.089+0000 7f9419fbcb80 1 -- 172.21.15.44:0/3918226817 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "status"}) -- 0x562bb74c2520 con 0x562bb7620310 2024-05-27T09:35:02.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.089+0000 7f9408a00700 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562bb7620310 0x562bb7621780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:02.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f9408a00700 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562bb7620310 0x562bb7621780 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a428ae844518db5 server_cookie=9e57895490965df in_seq=0 out_seq=0 2024-05-27T09:35:02.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f93eaa00700 10 client.14910 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:02.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f93eaa00700 1 -- 172.21.15.44:0/3918226817 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562bb74c2520 con 0x562bb7620310 2024-05-27T09:35:02.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f93eaa00700 10 client.14910 handle_command_reply: tid=0 2024-05-27T09:35:02.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f9419fbcb80 1 client.14910 shutdown 2024-05-27T09:35:02.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f9419fbcb80 1 -- 172.21.15.44:0/3918226817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 msgr2=0x562bb75b7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:02.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75b7b40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009c40 tx=0x7f93f000bae0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f9419fbcb80 1 -- 172.21.15.44:0/3918226817 shutdown_connections 2024-05-27T09:35:02.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562bb7620310 0x562bb7621780 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562bb75a3270 0x562bb75b8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562bb7419df0 0x562bb75b7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f9419fbcb80 1 --2- 172.21.15.44:0/3918226817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562bb75059a0 0x562bb75b7b40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f9419fbcb80 1 -- 172.21.15.44:0/3918226817 >> 172.21.15.44:0/3918226817 conn(0x562bb74b3340 msgr2=0x562bb759f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:02.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.090+0000 7f93eaa00700 0 client.14910 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:02.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.091+0000 7f9419fbcb80 1 -- 172.21.15.44:0/3918226817 shutdown_connections 2024-05-27T09:35:02.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.091+0000 7f9419fbcb80 1 -- 172.21.15.44:0/3918226817 wait complete. 2024-05-27T09:35:02.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.091+0000 7f9419fbcb80 20 client.14910 trim_cache size 0 max 16384 2024-05-27T09:35:02.095 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:35:02.095 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:35:02.095 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-05-27T09:35:02.095 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24542, 2024-05-27T09:35:02.095 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-05-27T09:35:02.095 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-05-27T09:35:02.095 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-05-27T09:35:02.095 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 214.15878456499999, 2024-05-27T09:35:02.095 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 25, 2024-05-27T09:35:02.095 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 81, 2024-05-27T09:35:02.095 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 79, 2024-05-27T09:35:02.096 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 236.42897701699999 2024-05-27T09:35:02.096 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:35:02.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.091+0000 7f9409e00700 1 -- 172.21.15.44:0/886678788 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f93ec0b2880 msgr2=0x7f93ec0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:02.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.091+0000 7f9409e00700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f93ec0b2880 0x7f93ec0b4d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f93fc004a40 tx=0x7f93fc013540 comp rx=0 tx=0).stop 2024-05-27T09:35:02.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.091+0000 7f9409e00700 1 -- 172.21.15.44:0/886678788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94041052e0 msgr2=0x7f940419ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:02.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.091+0000 7f9409e00700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94041052e0 0x7f940419ceb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f93f800e9c0 tx=0x7f93f800ecd0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.092+0000 7f9409e00700 1 -- 172.21.15.44:0/886678788 shutdown_connections 2024-05-27T09:35:02.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.092+0000 7f9409e00700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f93ec0b2880 0x7f93ec0b4d90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.092+0000 7f9409e00700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f940410e710 0x7f94041a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.092+0000 7f9409e00700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94041052e0 0x7f940419ceb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.092+0000 7f9409e00700 1 --2- 172.21.15.44:0/886678788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9404105c30 0x7f940419d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:02.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.092+0000 7f9409e00700 1 -- 172.21.15.44:0/886678788 >> 172.21.15.44:0/886678788 conn(0x7f9404078af0 msgr2=0x7f9404104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:02.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.092+0000 7f9409e00700 1 -- 172.21.15.44:0/886678788 shutdown_connections 2024-05-27T09:35:02.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:02.092+0000 7f9409e00700 1 -- 172.21.15.44:0/886678788 wait complete. 2024-05-27T09:35:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:01 smithi050 ceph-mon[36371]: pgmap v433: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:35:02.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:01 smithi189 ceph-mon[36671]: pgmap v433: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:35:02.888 INFO:teuthology.run_tasks:Running task sequential... 2024-05-27T09:35:03.064 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-27T09:35:03.064 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-27T09:35:03.064 INFO:teuthology.task.print:Enabling mgr modules 2024-05-27T09:35:03.064 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-27T09:35:03.065 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T09:35:03.066 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-05-27T09:35:03.066 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-27T09:35:03.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.363+0000 7f22ffe00640 1 -- 172.21.15.44:0/469474741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23001099b0 msgr2=0x7f230010bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:03.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.363+0000 7f22ffe00640 1 --2- 172.21.15.44:0/469474741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23001099b0 0x7f230010bdd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009ca0 tx=0x7f22f802a8c0 comp rx=0 tx=0).stop 2024-05-27T09:35:03.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.363+0000 7f22ffe00640 1 -- 172.21.15.44:0/469474741 shutdown_connections 2024-05-27T09:35:03.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.363+0000 7f22ffe00640 1 --2- 172.21.15.44:0/469474741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23001099b0 0x7f230010bdd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:03.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.363+0000 7f22ffe00640 1 --2- 172.21.15.44:0/469474741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2300104cb0 0x7f23001050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:03.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.363+0000 7f22ffe00640 1 --2- 172.21.15.44:0/469474741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2300103a70 0x7f2300104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:03.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.363+0000 7f22ffe00640 1 -- 172.21.15.44:0/469474741 >> 172.21.15.44:0/469474741 conn(0x7f23000fec60 msgr2=0x7f2300101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:03.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.364+0000 7f22ffe00640 1 -- 172.21.15.44:0/469474741 shutdown_connections 2024-05-27T09:35:03.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.364+0000 7f22ffe00640 1 -- 172.21.15.44:0/469474741 wait complete. 2024-05-27T09:35:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.365+0000 7f22ffe00640 1 Processor -- start 2024-05-27T09:35:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.365+0000 7f22ffe00640 1 -- start start 2024-05-27T09:35:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.365+0000 7f22ffe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2300103a70 0x7f230019c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.365+0000 7f22ffe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2300104cb0 0x7f230019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.365+0000 7f22ffe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23001099b0 0x7f23001a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.365+0000 7f22ffe00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f230010f5a0 con 0x7f23001099b0 2024-05-27T09:35:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.365+0000 7f22ffe00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f230010f420 con 0x7f2300104cb0 2024-05-27T09:35:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.365+0000 7f22ffe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f230010f720 con 0x7f2300103a70 2024-05-27T09:35:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.365+0000 7f22ff400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23001099b0 0x7f23001a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.366+0000 7f22fe000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2300104cb0 0x7f230019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.366+0000 7f22fe000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2300104cb0 0x7f230019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37956/0 (socket says 172.21.15.44:37956) 2024-05-27T09:35:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.366+0000 7f22fea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2300103a70 0x7f230019c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.366+0000 7f22fe000640 1 -- 172.21.15.44:0/290526367 learned_addr learned my addr 172.21.15.44:0/290526367 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.366+0000 7f22ff400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23001099b0 0x7f23001a3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52250/0 (socket says 172.21.15.44:52250) 2024-05-27T09:35:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.366+0000 7f22fe000640 1 -- 172.21.15.44:0/290526367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2300103a70 msgr2=0x7f230019c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:03.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.366+0000 7f22fe000640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2300103a70 0x7f230019c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.366+0000 7f22fe000640 1 -- 172.21.15.44:0/290526367 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23001099b0 msgr2=0x7f23001a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.366+0000 7f22fe000640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23001099b0 0x7f23001a3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.366+0000 7f22fe000640 1 -- 172.21.15.44:0/290526367 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23001a3ed0 con 0x7f2300104cb0 2024-05-27T09:35:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.366+0000 7f22fe000640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2300104cb0 0x7f230019c8d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f22e800e9f0 tx=0x7f22e800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.367+0000 7f22f7400640 1 -- 172.21.15.44:0/290526367 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e800cd80 con 0x7f2300104cb0 2024-05-27T09:35:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.367+0000 7f22f7400640 1 -- 172.21.15.44:0/290526367 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22e80054d0 con 0x7f2300104cb0 2024-05-27T09:35:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.367+0000 7f22ffe00640 1 -- 172.21.15.44:0/290526367 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23001a41c0 con 0x7f2300104cb0 2024-05-27T09:35:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.367+0000 7f22f7400640 1 -- 172.21.15.44:0/290526367 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e8010690 con 0x7f2300104cb0 2024-05-27T09:35:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.367+0000 7f22ffe00640 1 -- 172.21.15.44:0/290526367 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23001a46c0 con 0x7f2300104cb0 2024-05-27T09:35:03.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.368+0000 7f22ffe00640 1 -- 172.21.15.44:0/290526367 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23001a4980 con 0x7f2300104cb0 2024-05-27T09:35:03.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.371+0000 7f22f7400640 1 -- 172.21.15.44:0/290526367 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f22e8004e30 con 0x7f2300104cb0 2024-05-27T09:35:03.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.373+0000 7f22f7400640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f22e40b27e0 0x7f22e40b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:03.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.373+0000 7f22f7400640 1 -- 172.21.15.44:0/290526367 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f22e80e27d0 con 0x7f2300104cb0 2024-05-27T09:35:03.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.373+0000 7f22fea00640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f22e40b27e0 0x7f22e40b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:03.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.374+0000 7f22fea00640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f22e40b27e0 0x7f22e40b4cf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f22f00095f0 tx=0x7f22f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:03.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.374+0000 7f22f7400640 1 -- 172.21.15.44:0/290526367 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f22e8054df0 con 0x7f2300104cb0 2024-05-27T09:35:03.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.642+0000 7f22ffe00640 1 -- 172.21.15.44:0/290526367 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f23001a4b50 con 0x7f2300104cb0 2024-05-27T09:35:03.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.877+0000 7f22f7400640 1 -- 172.21.15.44:0/290526367 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f22e80a6030 con 0x7f2300104cb0 2024-05-27T09:35:03.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.878+0000 7f22f7400640 1 -- 172.21.15.44:0/290526367 <== mon.2 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f22e80e3e70 con 0x7f2300104cb0 2024-05-27T09:35:03.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.884+0000 7f22ffe00640 1 -- 172.21.15.44:0/290526367 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f22e40b27e0 msgr2=0x7f22e40b4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:03.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.884+0000 7f22ffe00640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f22e40b27e0 0x7f22e40b4cf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f22f00095f0 tx=0x7f22f0005f90 comp rx=0 tx=0).stop 2024-05-27T09:35:03.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.884+0000 7f22ffe00640 1 -- 172.21.15.44:0/290526367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2300104cb0 msgr2=0x7f230019c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:03.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.884+0000 7f22ffe00640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2300104cb0 0x7f230019c8d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f22e800e9f0 tx=0x7f22e800edf0 comp rx=0 tx=0).stop 2024-05-27T09:35:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.884+0000 7f22ffe00640 1 -- 172.21.15.44:0/290526367 shutdown_connections 2024-05-27T09:35:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.884+0000 7f22ffe00640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f22e40b27e0 0x7f22e40b4cf0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.884+0000 7f22ffe00640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23001099b0 0x7f23001a3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.884+0000 7f22ffe00640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2300104cb0 0x7f230019c8d0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f22e800e9f0 tx=0x7f22e800edf0 comp rx=0 tx=0).stop 2024-05-27T09:35:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.884+0000 7f22ffe00640 1 --2- 172.21.15.44:0/290526367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2300103a70 0x7f230019c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.884+0000 7f22ffe00640 1 -- 172.21.15.44:0/290526367 >> 172.21.15.44:0/290526367 conn(0x7f23000fec60 msgr2=0x7f23000ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:03.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.885+0000 7f22ffe00640 1 -- 172.21.15.44:0/290526367 shutdown_connections 2024-05-27T09:35:03.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:03.885+0000 7f22ffe00640 1 -- 172.21.15.44:0/290526367 wait complete. 2024-05-27T09:35:03.921 INFO:teuthology.run_tasks:Running task kclient... 2024-05-27T09:35:03.947 INFO:tasks.kclient:Mounting kernel clients... 2024-05-27T09:35:03.947 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-27T09:35:03.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:35:03.947 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-05-27T09:35:04.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:35:04.005 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link delete ceph-brx 2024-05-27T09:35:04.081 INFO:teuthology.orchestra.run.smithi044.stderr:Cannot find device "ceph-brx" 2024-05-27T09:35:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:35:04.085 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-05-27T09:35:04.085 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-27T09:35:04.085 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T09:35:04.085 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-05-27T09:35:04.095 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:03 smithi050 ceph-mon[36371]: pgmap v434: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:35:04.095 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:03 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/290526367' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T09:35:04.095 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:03 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T09:35:04.095 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:03 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: ignoring --setuser ceph since I am not root 2024-05-27T09:35:04.096 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:03 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: ignoring --setgroup ceph since I am not root 2024-05-27T09:35:04.096 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:03 smithi050 ceph-mgr[37748]: -- 172.21.15.50:0/2637826201 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562efe4ba5a0 con 0x562f03244480 2024-05-27T09:35:04.096 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:04.080+0000 7fb143c29200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T09:35:04.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:03 smithi044 ceph-mon[31618]: pgmap v434: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:35:04.099 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:03 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/290526367' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T09:35:04.099 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:03 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T09:35:04.262 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:03 smithi189 ceph-mon[36671]: pgmap v434: 97 pgs: 97 active+clean; 640 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:35:04.262 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:03 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/290526367' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T09:35:04.262 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:03 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T09:35:04.262 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:03 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: ignoring --setuser ceph since I am not root 2024-05-27T09:35:04.262 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:03 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: ignoring --setgroup ceph since I am not root 2024-05-27T09:35:04.262 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:03 smithi189 ceph-mgr[38533]: -- 172.21.15.189:0/3409132282 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556e148ac5a0 con 0x556e19640000 2024-05-27T09:35:04.262 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:04.074+0000 7f80665af200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T09:35:04.262 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:04.261+0000 7f80665af200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:35:04.387 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:04.277+0000 7fb143c29200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:35:04.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:04 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: ignoring --setuser ceph since I am not root 2024-05-27T09:35:04.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:04 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: ignoring --setgroup ceph since I am not root 2024-05-27T09:35:04.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:04 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:04.291+0000 7f4a6f901200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T09:35:04.524 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:04.587 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:04.364+0000 7f80665af200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T09:35:04.857 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:04 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:04.578+0000 7f4a6f901200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:35:04.857 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:04 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:04.728+0000 7f4a6f901200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T09:35:05.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:04 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T09:35:05.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:04 smithi050 ceph-mon[36371]: mgrmap e19: x(active, since 12m), standbys: z, y 2024-05-27T09:35:05.044 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:04.385+0000 7fb143c29200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T09:35:05.044 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:04.711+0000 7fb143c29200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T09:35:05.044 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:04.802+0000 7fb143c29200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T09:35:05.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:04 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T09:35:05.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:04 smithi189 ceph-mon[36671]: mgrmap e19: x(active, since 12m), standbys: z, y 2024-05-27T09:35:05.088 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:04.677+0000 7f80665af200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T09:35:05.088 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:04.764+0000 7f80665af200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T09:35:05.124 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:04 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T09:35:05.124 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:04 smithi044 ceph-mon[31618]: mgrmap e19: x(active, since 12m), standbys: z, y 2024-05-27T09:35:05.452 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:05 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:05.155+0000 7f4a6f901200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T09:35:05.452 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:05 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:05.275+0000 7f4a6f901200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T09:35:06.308 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:05 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:05.961+0000 7f80665af200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T09:35:06.308 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:06.133+0000 7f80665af200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T09:35:06.308 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:06.307+0000 7f80665af200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T09:35:06.428 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:06.065+0000 7fb143c29200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T09:35:06.428 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:06.248+0000 7fb143c29200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T09:35:06.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.630+0000 7f1249800700 1 -- 172.21.15.44:0/1691250115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12441045e0 msgr2=0x7f12441049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.630+0000 7f1249800700 1 --2- 172.21.15.44:0/1691250115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12441045e0 0x7f12441049f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1234009990 tx=0x7f1234009ca0 comp rx=0 tx=0).stop 2024-05-27T09:35:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.631+0000 7f1249800700 1 -- 172.21.15.44:0/1691250115 shutdown_connections 2024-05-27T09:35:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.631+0000 7f1249800700 1 --2- 172.21.15.44:0/1691250115 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1244109860 0x7f12441111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.631+0000 7f1249800700 1 --2- 172.21.15.44:0/1691250115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12441045e0 0x7f12441049f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.631+0000 7f1249800700 1 --2- 172.21.15.44:0/1691250115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1244108f10 0x7f1244109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.631+0000 7f1249800700 1 -- 172.21.15.44:0/1691250115 >> 172.21.15.44:0/1691250115 conn(0x7f1244100030 msgr2=0x7f1244102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.631+0000 7f1249800700 1 -- 172.21.15.44:0/1691250115 shutdown_connections 2024-05-27T09:35:06.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.631+0000 7f1249800700 1 -- 172.21.15.44:0/1691250115 wait complete. 2024-05-27T09:35:06.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.632+0000 7f1249800700 1 Processor -- start 2024-05-27T09:35:06.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.632+0000 7f1249800700 1 -- start start 2024-05-27T09:35:06.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.633+0000 7f1249800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12441045e0 0x7f124419d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:06.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.633+0000 7f1249800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1244108f10 0x7f124419d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:06.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.633+0000 7f1249800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1244109860 0x7f12441a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:06.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.633+0000 7f1249800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1244114490 con 0x7f12441045e0 2024-05-27T09:35:06.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.633+0000 7f1249800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1244114310 con 0x7f1244108f10 2024-05-27T09:35:06.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.633+0000 7f1249800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1244114610 con 0x7f1244109860 2024-05-27T09:35:06.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.633+0000 7f1242000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1244108f10 0x7f124419d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:06.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.633+0000 7f1242000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1244108f10 0x7f124419d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37974/0 (socket says 172.21.15.44:37974) 2024-05-27T09:35:06.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.633+0000 7f1242000700 1 -- 172.21.15.44:0/3844309331 learned_addr learned my addr 172.21.15.44:0/3844309331 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:06.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.633+0000 7f1242a00700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12441045e0 0x7f124419d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:06.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.633+0000 7f1243400700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1244109860 0x7f12441a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:06.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.634+0000 7f1242000700 1 -- 172.21.15.44:0/3844309331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1244109860 msgr2=0x7f12441a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:06.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.634+0000 7f1242000700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1244109860 0x7f12441a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:06.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.634+0000 7f1242000700 1 -- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12441045e0 msgr2=0x7f124419d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:06.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.634+0000 7f1242000700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12441045e0 0x7f124419d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.634+0000 7f1242000700 1 -- 172.21.15.44:0/3844309331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12441a1f90 con 0x7f1244108f10 2024-05-27T09:35:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.634+0000 7f1242000700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1244108f10 0x7f124419d660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f123000e970 tx=0x7f123000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.634+0000 7f123b400700 1 -- 172.21.15.44:0/3844309331 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1230004a30 con 0x7f1244108f10 2024-05-27T09:35:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.634+0000 7f1249800700 1 -- 172.21.15.44:0/3844309331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12441a2280 con 0x7f1244108f10 2024-05-27T09:35:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.634+0000 7f123b400700 1 -- 172.21.15.44:0/3844309331 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f123000cc90 con 0x7f1244108f10 2024-05-27T09:35:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.634+0000 7f123b400700 1 -- 172.21.15.44:0/3844309331 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1230005610 con 0x7f1244108f10 2024-05-27T09:35:06.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.635+0000 7f1249800700 1 -- 172.21.15.44:0/3844309331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12441a9ba0 con 0x7f1244108f10 2024-05-27T09:35:06.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.636+0000 7f1249800700 1 -- 172.21.15.44:0/3844309331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12440512b0 con 0x7f1244108f10 2024-05-27T09:35:06.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.639+0000 7f123b400700 1 -- 172.21.15.44:0/3844309331 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f123000c7e0 con 0x7f1244108f10 2024-05-27T09:35:06.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.640+0000 7f123b400700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12280b28d0 0x7f12280b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:06.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.640+0000 7f1242a00700 1 -- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12280b28d0 msgr2=0x7f12280b4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4077502151 2024-05-27T09:35:06.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.641+0000 7f123b400700 1 -- 172.21.15.44:0/3844309331 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f12300e25d0 con 0x7f1244108f10 2024-05-27T09:35:06.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.641+0000 7f1242a00700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12280b28d0 0x7f12280b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T09:35:06.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.649+0000 7f123b400700 1 -- 172.21.15.44:0/3844309331 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f12300aa2a0 con 0x7f1244108f10 2024-05-27T09:35:06.741 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:06.427+0000 7fb143c29200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T09:35:06.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.842+0000 7f1242a00700 1 -- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12280b28d0 msgr2=0x7f12280b4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4077502151 2024-05-27T09:35:06.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.842+0000 7f1242a00700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12280b28d0 0x7f12280b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T09:35:06.902 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:06.600+0000 7f80665af200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T09:35:06.902 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:06.694+0000 7f80665af200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T09:35:06.902 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:06.901+0000 7f80665af200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T09:35:06.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.934+0000 7f1249800700 1 -- 172.21.15.44:0/3844309331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f12441a9d70 con 0x7f1244108f10 2024-05-27T09:35:06.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.935+0000 7f123b400700 1 -- 172.21.15.44:0/3844309331 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f12300a9ba0 con 0x7f1244108f10 2024-05-27T09:35:06.936 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-27T09:35:06.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.938+0000 7f1249800700 1 -- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12280b28d0 msgr2=0x7f12280b4de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:35:06.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.938+0000 7f1249800700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12280b28d0 0x7f12280b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:06.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.938+0000 7f1249800700 1 -- 172.21.15.44:0/3844309331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1244108f10 msgr2=0x7f124419d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:06.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.938+0000 7f1249800700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1244108f10 0x7f124419d660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f123000e970 tx=0x7f123000ec80 comp rx=0 tx=0).stop 2024-05-27T09:35:06.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.938+0000 7f1249800700 1 -- 172.21.15.44:0/3844309331 shutdown_connections 2024-05-27T09:35:06.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.938+0000 7f1249800700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f12280b28d0 0x7f12280b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:06.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.938+0000 7f1249800700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1244109860 0x7f12441a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:06.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.938+0000 7f1249800700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12441045e0 0x7f124419d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:06.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.938+0000 7f1249800700 1 --2- 172.21.15.44:0/3844309331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1244108f10 0x7f124419d660 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:06.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.938+0000 7f1249800700 1 -- 172.21.15.44:0/3844309331 >> 172.21.15.44:0/3844309331 conn(0x7f1244100030 msgr2=0x7f12441086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:06.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.939+0000 7f1249800700 1 -- 172.21.15.44:0/3844309331 shutdown_connections 2024-05-27T09:35:06.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:06.939+0000 7f1249800700 1 -- 172.21.15.44:0/3844309331 wait complete. 2024-05-27T09:35:07.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:06 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3844309331' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T09:35:07.044 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:06.739+0000 7fb143c29200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T09:35:07.044 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:06.838+0000 7fb143c29200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T09:35:07.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:06 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3844309331' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T09:35:07.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:06 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:06.936+0000 7f4a6f901200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T09:35:07.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:07 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:07.181+0000 7f4a6f901200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T09:35:07.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:06 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3844309331' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T09:35:07.338 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:07.098+0000 7f80665af200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:35:07.544 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:07.054+0000 7fb143c29200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T09:35:07.544 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:07.260+0000 7fb143c29200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:35:07.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:07 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:07.410+0000 7f4a6f901200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T09:35:07.831 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-27T09:35:07.831 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-27T09:35:07.831 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-05-27T09:35:07.832 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-27T09:35:07.832 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T09:35:07.832 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-27T09:35:07.832 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-27T09:35:07.832 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-27T09:35:07.832 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-27T09:35:07.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:35:07.832 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-05-27T09:35:07.922 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-27T09:35:07.922 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-27T09:35:07.922 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-05-27T09:35:07.922 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-05-27T09:35:07.922 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-05-27T09:35:07.922 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 41780sec preferred_lft 41780sec 2024-05-27T09:35:07.923 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-05-27T09:35:07.924 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-05-27T09:35:07.924 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T09:35:07.924 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-05-27T09:35:07.924 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-05-27T09:35:07.925 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-27T09:35:07.925 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T09:35:07.925 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-05-27T09:35:07.925 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add name ceph-brx type bridge 2024-05-27T09:35:07.925 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr flush dev ceph-brx 2024-05-27T09:35:07.925 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set ceph-brx up 2024-05-27T09:35:07.925 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-27T09:35:07.925 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-05-27T09:35:08.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T09:35:08.051 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:07 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:07.820+0000 7f4a6f901200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T09:35:08.051 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:07 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:07.955+0000 7f4a6f901200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T09:35:08.087 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:07.707+0000 7f80665af200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T09:35:08.087 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:07.931+0000 7f80665af200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T09:35:08.293 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:07.898+0000 7fb143c29200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T09:35:08.294 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:08.133+0000 7fb143c29200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T09:35:08.307 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:08 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:08.225+0000 7f4a6f901200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T09:35:08.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T09:35:08.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:35:08.411 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-27T09:35:08.477 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-05-27T09:35:08.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:35:08.481 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-05-27T09:35:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-05-27T09:35:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-27T09:35:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-27T09:35:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-27T09:35:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-27T09:35:08.538 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T09:35:08.538 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-05-27T09:35:08.538 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-27T09:35:08.538 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-27T09:35:08.539 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-27T09:35:08.539 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-05-27T09:35:08.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T09:35:08.665 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:08 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:08.510+0000 7f4a6f901200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T09:35:08.837 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:08.360+0000 7f80665af200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T09:35:08.837 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:08.458+0000 7f80665af200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T09:35:09.044 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:08.583+0000 7fb143c29200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T09:35:09.044 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:08.686+0000 7fb143c29200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T09:35:09.087 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:08.836+0000 7f80665af200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T09:35:09.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T09:35:09.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:35:09.291 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-05-27T09:35:09.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:35:09.337 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list-id 2024-05-27T09:35:09.431 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T09:35:09.431 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-05-27T09:35:09.431 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T09:35:09.431 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-27T09:35:09.431 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-05-27T09:35:09.459 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:09.078+0000 7fb143c29200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T09:35:09.459 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:09.458+0000 7fb143c29200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T09:35:09.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T09:35:09.541 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:09.199+0000 7f80665af200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T09:35:09.542 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:09.292+0000 7f80665af200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T09:35:09.563 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:09 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:09.390+0000 7f4a6f901200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T09:35:09.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T09:35:09.626 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-27T09:35:09.626 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T09:35:09.626 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-05-27T09:35:09.627 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-27T09:35:09.627 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-27T09:35:09.627 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-27T09:35:09.627 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-27T09:35:09.627 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-27T09:35:09.627 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-05-27T09:35:09.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T09:35:09.794 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:09.556+0000 7fb143c29200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T09:35:09.815 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:09 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:09.712+0000 7f4a6f901200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T09:35:09.837 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:09.540+0000 7f80665af200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T09:35:09.837 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:09.676+0000 7f80665af200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T09:35:09.837 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:09.773+0000 7f80665af200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T09:35:09.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T09:35:09.920 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T09:35:09.920 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-05-27T09:35:09.920 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-05-27T09:35:09.920 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set dev brx.0 master ceph-brx 2024-05-27T09:35:09.920 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-05-27T09:35:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T09:35:10.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T09:35:10.116 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T09:35:10.116 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-27T09:35:10.155 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-27T09:35:10.155 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T09:35:10.155 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-27T09:35:10.243 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-27T09:35:10.244 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-27T09:35:10.244 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/11c31bb5-2e8d-444b-b829-f9cacdbbd223 2024-05-27T09:35:10.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:35:10.244 DEBUG:teuthology.orchestra.run.smithi044:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/11c31bb5-2e8d-444b-b829-f9cacdbbd223 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-27T09:35:10.294 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:09.816+0000 7fb143c29200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T09:35:10.295 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:09.959+0000 7fb143c29200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T09:35:10.295 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:10.061+0000 7fb143c29200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T09:35:10.445 INFO:teuthology.orchestra.run.smithi044.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-27T09:35:10.445 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-05-27T09:35:10.445 INFO:teuthology.orchestra.run.smithi044.stdout:invalid new device string format 2024-05-27T09:35:10.587 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:10.107+0000 7f80665af200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T09:35:10.588 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:10.202+0000 7f80665af200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T09:35:10.696 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:10 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:10.327+0000 7f4a6f901200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T09:35:10.696 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:10 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:10.469+0000 7f4a6f901200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T09:35:10.794 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:10.405+0000 7fb143c29200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T09:35:10.794 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:10.503+0000 7fb143c29200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T09:35:10.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:10 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:10.962+0000 7f4a6f901200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T09:35:11.087 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:10.808+0000 7f80665af200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T09:35:11.087 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:35:11 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z[38529]: 2024-05-27T09:35:11.028+0000 7f80665af200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T09:35:11.205 INFO:teuthology.orchestra.run.smithi044.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-27T09:35:11.206 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-05-27T09:35:11.206 INFO:teuthology.orchestra.run.smithi044.stdout:invalid new device string format 2024-05-27T09:35:11.206 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: resolved to: "172.21.15.44:6789,172.21.15.50:6789,172.21.15.189:6789" 2024-05-27T09:35:11.206 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.44:6789,172.21.15.50:6789,172.21.15.189:6789:/volumes/qa/sv_1/11c31bb5-2e8d-444b-b829-f9cacdbbd223 2024-05-27T09:35:11.206 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,key=0,fsid=604b015a-1c0a-11ef-bc9b-c7b262605968" will pass to kernel 2024-05-27T09:35:11.209 INFO:teuthology.orchestra.run.smithi044.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-27T09:35:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: You just mounted a file system that supports labels which does not 2024-05-27T09:35:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-27T09:35:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: applications will generate AVC messages and not be allowed access to 2024-05-27T09:35:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-27T09:35:11.213 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-27T09:35:11.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:35:11.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-27T09:35:11.291 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T09:35:11.380 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:11 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:11.147+0000 7fb143c29200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T09:35:11.381 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:11 smithi050 ceph-mon[36371]: Standby manager daemon z restarted 2024-05-27T09:35:11.381 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:11 smithi050 ceph-mon[36371]: Standby manager daemon z started 2024-05-27T09:35:11.381 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:11 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T09:35:11.381 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:11 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:35:11.381 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:11 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T09:35:11.381 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:11 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:35:11.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:11 smithi044 ceph-mon[31618]: Standby manager daemon z restarted 2024-05-27T09:35:11.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:11 smithi044 ceph-mon[31618]: Standby manager daemon z started 2024-05-27T09:35:11.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:11 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T09:35:11.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:11 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:35:11.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:11 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T09:35:11.396 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:11 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:35:11.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T09:35:11.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T09:35:11.411 INFO:teuthology.orchestra.run.smithi044.stdout:client25070 2024-05-27T09:35:11.412 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:35:11.412 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T09:35:11.460 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T09:35:11.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:11 smithi189 ceph-mon[36671]: Standby manager daemon z restarted 2024-05-27T09:35:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:11 smithi189 ceph-mon[36671]: Standby manager daemon z started 2024-05-27T09:35:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:11 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T09:35:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:11 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:35:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:11 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T09:35:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:11 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.189:0/2221443697' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:35:11.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T09:35:11.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T09:35:11.619 INFO:teuthology.orchestra.run.smithi044.stdout:client25070 2024-05-27T09:35:11.620 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T09:35:11.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T09:35:11.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:11 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:11.466+0000 7f4a6f901200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T09:35:11.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:11 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:11.599+0000 7f4a6f901200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T09:35:11.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T09:35:11.735 INFO:teuthology.orchestra.run.smithi044.stdout:client25070 2024-05-27T09:35:11.735 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968.client25070/status') 2024-05-27T09:35:11.794 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:35:11 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y[37744]: 2024-05-27T09:35:11.380+0000 7fb143c29200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T09:35:11.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T09:35:11.861 INFO:teuthology.orchestra.run.smithi044.stdout:instance: client.25070 (3)172.21.15.44:0/1227851225 2024-05-27T09:35:11.862 INFO:teuthology.orchestra.run.smithi044.stdout:blocklisted: false 2024-05-27T09:35:11.862 INFO:teuthology.orchestra.run.smithi044.stderr:0+1 records in 2024-05-27T09:35:11.862 INFO:teuthology.orchestra.run.smithi044.stderr:0+1 records out 2024-05-27T09:35:11.862 INFO:teuthology.orchestra.run.smithi044.stderr:71 bytes copied, 7.7465e-05 s, 917 kB/s 2024-05-27T09:35:11.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T09:35:11.878 INFO:teuthology.run_tasks:Running task exec... 2024-05-27T09:35:11.887 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T09:35:11.887 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-05-27T09:35:11.887 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-05-27T09:35:12.137 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:11 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:11.964+0000 7f4a6f901200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T09:35:12.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.158+0000 7f76ade00640 1 -- 172.21.15.44:0/984510060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76a8104550 msgr2=0x7f76a8106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:12.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.158+0000 7f76ade00640 1 --2- 172.21.15.44:0/984510060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76a8104550 0x7f76a8106970 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f769c009900 tx=0x7f769c02a410 comp rx=0 tx=0).stop 2024-05-27T09:35:12.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.158+0000 7f76ade00640 1 -- 172.21.15.44:0/984510060 shutdown_connections 2024-05-27T09:35:12.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.158+0000 7f76ade00640 1 --2- 172.21.15.44:0/984510060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76a8109810 0x7f76a810bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:12.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.158+0000 7f76ade00640 1 --2- 172.21.15.44:0/984510060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a8106eb0 0x7f76a81092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:12.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.158+0000 7f76ade00640 1 --2- 172.21.15.44:0/984510060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76a8104550 0x7f76a8106970 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:12.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.158+0000 7f76ade00640 1 -- 172.21.15.44:0/984510060 >> 172.21.15.44:0/984510060 conn(0x7f76a80fdb50 msgr2=0x7f76a8100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:12.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.158+0000 7f76ade00640 1 -- 172.21.15.44:0/984510060 shutdown_connections 2024-05-27T09:35:12.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.158+0000 7f76ade00640 1 -- 172.21.15.44:0/984510060 wait complete. 2024-05-27T09:35:12.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.159+0000 7f76ade00640 1 Processor -- start 2024-05-27T09:35:12.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76ade00640 1 -- start start 2024-05-27T09:35:12.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76ade00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76a8104550 0x7f76a819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:12.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76ade00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76a8106eb0 0x7f76a819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76ade00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a8109810 0x7f76a81a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76ade00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f76a810f5b0 con 0x7f76a8106eb0 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76ade00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f76a810f430 con 0x7f76a8104550 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76ade00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f76a810f730 con 0x7f76a8109810 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76ad400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a8109810 0x7f76a81a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76a7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76a8106eb0 0x7f76a819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76ad400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a8109810 0x7f76a81a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49464/0 (socket says 172.21.15.44:49464) 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76ad400640 1 -- 172.21.15.44:0/3247160597 learned_addr learned my addr 172.21.15.44:0/3247160597 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.160+0000 7f76aca00640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76a8104550 0x7f76a819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.161+0000 7f76ad400640 1 -- 172.21.15.44:0/3247160597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76a8104550 msgr2=0x7f76a819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.161+0000 7f76ad400640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76a8104550 0x7f76a819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.161+0000 7f76ad400640 1 -- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76a8106eb0 msgr2=0x7f76a819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:12.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.161+0000 7f76ad400640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76a8106eb0 0x7f76a819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:12.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.161+0000 7f76ad400640 1 -- 172.21.15.44:0/3247160597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a81a1cf0 con 0x7f76a8109810 2024-05-27T09:35:12.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.161+0000 7f76a7e00640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76a8106eb0 0x7f76a819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:12.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.161+0000 7f76ad400640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a8109810 0x7f76a81a1680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f76a000eb40 tx=0x7f76a000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:12.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.161+0000 7f76a5600640 1 -- 172.21.15.44:0/3247160597 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a0004b20 con 0x7f76a8109810 2024-05-27T09:35:12.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.161+0000 7f76a5600640 1 -- 172.21.15.44:0/3247160597 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76a0004cc0 con 0x7f76a8109810 2024-05-27T09:35:12.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.162+0000 7f76ade00640 1 -- 172.21.15.44:0/3247160597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a81a1fe0 con 0x7f76a8109810 2024-05-27T09:35:12.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.162+0000 7f76a5600640 1 -- 172.21.15.44:0/3247160597 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a0010720 con 0x7f76a8109810 2024-05-27T09:35:12.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.162+0000 7f76ade00640 1 -- 172.21.15.44:0/3247160597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a81a2440 con 0x7f76a8109810 2024-05-27T09:35:12.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.163+0000 7f76ade00640 1 -- 172.21.15.44:0/3247160597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a806a230 con 0x7f76a8109810 2024-05-27T09:35:12.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.165+0000 7f76a5600640 1 -- 172.21.15.44:0/3247160597 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 21) ==== 149669+0+0 (secure 0 0 0) 0x7f76a0005020 con 0x7f76a8109810 2024-05-27T09:35:12.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.167+0000 7f76a5600640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f76900b28b0 0x7f76900b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:12.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.167+0000 7f76aca00640 1 -- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f76900b28b0 msgr2=0x7f76900b4dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4077502151 2024-05-27T09:35:12.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.167+0000 7f76a5600640 1 -- 172.21.15.44:0/3247160597 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f76a00e29a0 con 0x7f76a8109810 2024-05-27T09:35:12.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.167+0000 7f76aca00640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f76900b28b0 0x7f76900b4dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T09:35:12.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.171+0000 7f76a5600640 1 -- 172.21.15.44:0/3247160597 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f76a00aa670 con 0x7f76a8109810 2024-05-27T09:35:12.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.368+0000 7f76aca00640 1 -- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f76900b28b0 msgr2=0x7f76900b4dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4077502151 2024-05-27T09:35:12.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.368+0000 7f76aca00640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f76900b28b0 0x7f76900b4dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T09:35:12.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.424+0000 7f76ade00640 1 -- 172.21.15.44:0/3247160597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f76a80638c0 con 0x7f76a8109810 2024-05-27T09:35:12.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:12 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:12.184+0000 7f4a6f901200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T09:35:12.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:12 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:12.326+0000 7f4a6f901200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T09:35:12.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:12 smithi044 ceph-mon[31618]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-05-27T09:35:12.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:12 smithi044 ceph-mon[31618]: Standby manager daemon y restarted 2024-05-27T09:35:12.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:12 smithi044 ceph-mon[31618]: Standby manager daemon y started 2024-05-27T09:35:12.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:12 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T09:35:12.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:12 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:35:12.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:12 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T09:35:12.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:12 smithi044 ceph-mon[31618]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:35:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:12 smithi050 ceph-mon[36371]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-05-27T09:35:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:12 smithi050 ceph-mon[36371]: Standby manager daemon y restarted 2024-05-27T09:35:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:12 smithi050 ceph-mon[36371]: Standby manager daemon y started 2024-05-27T09:35:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:12 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T09:35:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:12 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:35:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:12 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T09:35:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:12 smithi050 ceph-mon[36371]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:35:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:12 smithi189 ceph-mon[36671]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-05-27T09:35:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:12 smithi189 ceph-mon[36671]: Standby manager daemon y restarted 2024-05-27T09:35:12.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:12 smithi189 ceph-mon[36671]: Standby manager daemon y started 2024-05-27T09:35:12.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:12 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T09:35:12.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:12 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T09:35:12.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:12 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T09:35:12.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:12 smithi189 ceph-mon[36671]: from='mgr.? 172.21.15.50:0/2029484270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T09:35:12.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.769+0000 7f76aca00640 1 -- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f76900b28b0 msgr2=0x7f76900b4dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4077502151 2024-05-27T09:35:12.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:12.769+0000 7f76aca00640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f76900b28b0 0x7f76900b4dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T09:35:13.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.143+0000 7f76a5600640 1 -- 172.21.15.44:0/3247160597 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7f76a00a9f70 con 0x7f76a8109810 2024-05-27T09:35:13.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 -- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f76900b28b0 msgr2=0x7f76900b4dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:35:13.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f76900b28b0 0x7f76900b4dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:13.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 -- 172.21.15.44:0/3247160597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a8109810 msgr2=0x7f76a81a1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:13.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a8109810 0x7f76a81a1680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f76a000eb40 tx=0x7f76a000c2d0 comp rx=0 tx=0).stop 2024-05-27T09:35:13.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 -- 172.21.15.44:0/3247160597 shutdown_connections 2024-05-27T09:35:13.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f76900b28b0 0x7f76900b4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:13.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a8109810 0x7f76a81a1680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:13.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76a8106eb0 0x7f76a819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:13.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 --2- 172.21.15.44:0/3247160597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76a8104550 0x7f76a819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:13.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 -- 172.21.15.44:0/3247160597 >> 172.21.15.44:0/3247160597 conn(0x7f76a80fdb50 msgr2=0x7f76a8105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:13.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 -- 172.21.15.44:0/3247160597 shutdown_connections 2024-05-27T09:35:13.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.146+0000 7f76ade00640 1 -- 172.21.15.44:0/3247160597 wait complete. 2024-05-27T09:35:13.149 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:12 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:12.744+0000 7f4a6f901200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T09:35:13.149 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:12 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:12.880+0000 7f4a6f901200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T09:35:13.171 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-05-27T09:35:13.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.420+0000 7f943e800640 1 -- 172.21.15.44:0/2846041392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438102b40 msgr2=0x7f9438103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:13.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.420+0000 7f943e800640 1 --2- 172.21.15.44:0/2846041392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438102b40 0x7f9438103840 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9428009900 tx=0x7f942802a410 comp rx=0 tx=0).stop 2024-05-27T09:35:13.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.420+0000 7f943e800640 1 -- 172.21.15.44:0/2846041392 shutdown_connections 2024-05-27T09:35:13.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.420+0000 7f943e800640 1 --2- 172.21.15.44:0/2846041392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9438108a80 0x7f943810aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:13.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.420+0000 7f943e800640 1 --2- 172.21.15.44:0/2846041392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438103d80 0x7f9438104190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:13.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.420+0000 7f943e800640 1 --2- 172.21.15.44:0/2846041392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438102b40 0x7f9438103840 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:13.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.420+0000 7f943e800640 1 -- 172.21.15.44:0/2846041392 >> 172.21.15.44:0/2846041392 conn(0x7f94380fdd30 msgr2=0x7f94381001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:13.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.420+0000 7f943e800640 1 -- 172.21.15.44:0/2846041392 shutdown_connections 2024-05-27T09:35:13.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.420+0000 7f943e800640 1 -- 172.21.15.44:0/2846041392 wait complete. 2024-05-27T09:35:13.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.421+0000 7f943e800640 1 Processor -- start 2024-05-27T09:35:13.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.421+0000 7f943e800640 1 -- start start 2024-05-27T09:35:13.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.422+0000 7f943e800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9438102b40 0x7f943819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:13.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.422+0000 7f943e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438103d80 0x7f943819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:13.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.422+0000 7f943e800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438108a80 0x7f94381a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:13.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.422+0000 7f943e800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f943810e640 con 0x7f9438108a80 2024-05-27T09:35:13.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.422+0000 7f943e800640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f943810e4c0 con 0x7f9438102b40 2024-05-27T09:35:13.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.422+0000 7f943e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f943810e7c0 con 0x7f9438103d80 2024-05-27T09:35:13.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.422+0000 7f9436a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438103d80 0x7f943819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:13.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.422+0000 7f9437e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438108a80 0x7f94381a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:13.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.422+0000 7f9436a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438103d80 0x7f943819cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55718/0 (socket says 172.21.15.44:55718) 2024-05-27T09:35:13.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.422+0000 7f9436a00640 1 -- 172.21.15.44:0/3015252489 learned_addr learned my addr 172.21.15.44:0/3015252489 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:13.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.422+0000 7f9437400640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9438102b40 0x7f943819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:13.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.423+0000 7f9436a00640 1 -- 172.21.15.44:0/3015252489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9438102b40 msgr2=0x7f943819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:13.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.423+0000 7f9436a00640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9438102b40 0x7f943819c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:13.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.423+0000 7f9436a00640 1 -- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438108a80 msgr2=0x7f94381a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:13.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.423+0000 7f9436a00640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438108a80 0x7f94381a3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:13.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.423+0000 7f9436a00640 1 -- 172.21.15.44:0/3015252489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94381a4100 con 0x7f9438103d80 2024-05-27T09:35:13.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.423+0000 7f9436a00640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438103d80 0x7f943819cb70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f942400e9f0 tx=0x7f942400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:13.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.423+0000 7f942fe00640 1 -- 172.21.15.44:0/3015252489 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942400cd80 con 0x7f9438103d80 2024-05-27T09:35:13.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.423+0000 7f943e800640 1 -- 172.21.15.44:0/3015252489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94381a43f0 con 0x7f9438103d80 2024-05-27T09:35:13.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.423+0000 7f942fe00640 1 -- 172.21.15.44:0/3015252489 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94240054d0 con 0x7f9438103d80 2024-05-27T09:35:13.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.423+0000 7f942fe00640 1 -- 172.21.15.44:0/3015252489 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424010690 con 0x7f9438103d80 2024-05-27T09:35:13.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.423+0000 7f943e800640 1 -- 172.21.15.44:0/3015252489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94381a48f0 con 0x7f9438103d80 2024-05-27T09:35:13.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.424+0000 7f943e800640 1 -- 172.21.15.44:0/3015252489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94381a4bb0 con 0x7f9438103d80 2024-05-27T09:35:13.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.427+0000 7f942fe00640 1 -- 172.21.15.44:0/3015252489 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 21) ==== 149669+0+0 (secure 0 0 0) 0x7f9424004e90 con 0x7f9438103d80 2024-05-27T09:35:13.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.428+0000 7f942fe00640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f941c0b2830 0x7f941c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:13.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.428+0000 7f9437400640 1 -- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f941c0b2830 msgr2=0x7f941c0b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4077502151 2024-05-27T09:35:13.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.428+0000 7f942fe00640 1 -- 172.21.15.44:0/3015252489 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8219+0+0 (secure 0 0 0) 0x7f94240e2710 con 0x7f9438103d80 2024-05-27T09:35:13.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.428+0000 7f9437400640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f941c0b2830 0x7f941c0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T09:35:13.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.431+0000 7f942fe00640 1 -- 172.21.15.44:0/3015252489 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f94240aa3e0 con 0x7f9438103d80 2024-05-27T09:35:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:13 smithi044 ceph-mon[31618]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-05-27T09:35:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:13 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3247160597' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-27T09:35:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:13 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-27T09:35:13.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:13 smithi050 ceph-mon[36371]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-05-27T09:35:13.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:13 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3247160597' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-27T09:35:13.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:13 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-27T09:35:13.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:13 smithi189 ceph-mon[36671]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-05-27T09:35:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:13 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3247160597' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-27T09:35:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:13 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-27T09:35:13.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.629+0000 7f9437400640 1 -- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f941c0b2830 msgr2=0x7f941c0b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4077502151 2024-05-27T09:35:13.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.629+0000 7f9437400640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f941c0b2830 0x7f941c0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T09:35:13.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:13.668+0000 7f943e800640 1 -- 172.21.15.44:0/3015252489 --> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f943810a6c0 con 0x7f941c0b2830 2024-05-27T09:35:13.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:13 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:13.713+0000 7f4a6f901200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T09:35:14.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:14.030+0000 7f9437400640 1 -- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f941c0b2830 msgr2=0x7f941c0b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4077502151 2024-05-27T09:35:14.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:14.030+0000 7f9437400640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f941c0b2830 0x7f941c0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T09:35:14.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:14.052+0000 7f942fe00640 1 -- 172.21.15.44:0/3015252489 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 22) ==== 149337+0+0 (secure 0 0 0) 0x7f94240a5f70 con 0x7f9438103d80 2024-05-27T09:35:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:14.054+0000 7f942fe00640 1 -- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f941c0b2830 msgr2=0x7f941c0b4d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:35:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:14.054+0000 7f942fe00640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/4077502151,v1:172.21.15.44:6801/4077502151] conn(0x7f941c0b2830 0x7f941c0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:14.292 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:14.023+0000 7f4a6f901200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T09:35:14.293 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-27T09:35:14.293 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:35:14.293 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: Active manager daemon x restarted 2024-05-27T09:35:14.293 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: Activating manager daemon x 2024-05-27T09:35:14.293 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T09:35:14.293 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: mgrmap e22: x(active, starting, since 0.0191231s), standbys: y, z 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T09:35:14.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:35:14.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:35:14.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:35:14.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:35:14.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:35:14.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:35:14.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:35:14.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T09:35:14.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T09:35:14.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T09:35:14.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: Manager daemon x is now available 2024-05-27T09:35:14.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:14 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:14.358 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-27T09:35:14.358 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:35:14.358 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: Active manager daemon x restarted 2024-05-27T09:35:14.358 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: Activating manager daemon x 2024-05-27T09:35:14.358 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T09:35:14.358 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: mgrmap e22: x(active, starting, since 0.0191231s), standbys: y, z 2024-05-27T09:35:14.358 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:35:14.359 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:35:14.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:35:14.361 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:35:14.361 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:35:14.361 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:35:14.361 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T09:35:14.361 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T09:35:14.361 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T09:35:14.361 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: Manager daemon x is now available 2024-05-27T09:35:14.361 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:14 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:14.375 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-27T09:35:14.375 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:35:14.375 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: Active manager daemon x restarted 2024-05-27T09:35:14.375 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: Activating manager daemon x 2024-05-27T09:35:14.375 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: mgrmap e22: x(active, starting, since 0.0191231s), standbys: y, z 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T09:35:14.376 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T09:35:14.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T09:35:14.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T09:35:14.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T09:35:14.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T09:35:14.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T09:35:14.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T09:35:14.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T09:35:14.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T09:35:14.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T09:35:14.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T09:35:14.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: Manager daemon x is now available 2024-05-27T09:35:14.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:14 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:14.611 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:14.365+0000 7f49bf800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:14.611 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:14.365+0000 7f49bf800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:14.611 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:14.365+0000 7f49bf800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:14.611 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:14.365+0000 7f49bf800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:14.611 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:14.365+0000 7f49bf800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:14.611 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:14.366+0000 7f49c4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:14.611 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:14.366+0000 7f49c4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:14.612 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:14.366+0000 7f49c4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:14.612 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:14.366+0000 7f49c4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:14.612 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:14.366+0000 7f49c4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:15.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.065+0000 7f942fe00640 1 -- 172.21.15.44:0/3015252489 <== mon.1 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 23) ==== 149464+0+0 (secure 0 0 0) 0x7f94240e0240 con 0x7f9438103d80 2024-05-27T09:35:15.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.066+0000 7f942fe00640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f941c075070 0x7f941c075480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:15.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.066+0000 7f942fe00640 1 -- 172.21.15.44:0/3015252489 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f941c042310 con 0x7f941c075070 2024-05-27T09:35:15.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.074+0000 7f9437400640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f941c075070 0x7f941c075480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:15.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.090+0000 7f9437400640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f941c075070 0x7f941c075480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9428005ab0 tx=0x7f9428031400 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:15.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.117+0000 7f942fe00640 1 -- 172.21.15.44:0/3015252489 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f941c042310 con 0x7f941c075070 2024-05-27T09:35:15.118 INFO:teuthology.orchestra.run.smithi044.stdout:{} 2024-05-27T09:35:15.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.120+0000 7f943e800640 1 -- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f941c075070 msgr2=0x7f941c075480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:15.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.120+0000 7f943e800640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f941c075070 0x7f941c075480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9428005ab0 tx=0x7f9428031400 comp rx=0 tx=0).stop 2024-05-27T09:35:15.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.120+0000 7f943e800640 1 -- 172.21.15.44:0/3015252489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438103d80 msgr2=0x7f943819cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:15.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.120+0000 7f943e800640 1 --2- 172.21.15.44:0/3015252489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438103d80 0x7f943819cb70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f942400e9f0 tx=0x7f942400edf0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.120+0000 7f9437e00640 1 -- 172.21.15.44:0/3015252489 reap_dead start 2024-05-27T09:35:15.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.120+0000 7f943e800640 1 -- 172.21.15.44:0/3015252489 shutdown_connections 2024-05-27T09:35:15.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.120+0000 7f943e800640 1 -- 172.21.15.44:0/3015252489 >> 172.21.15.44:0/3015252489 conn(0x7f94380fdd30 msgr2=0x7f9438109fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:15.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.120+0000 7f943e800640 1 -- 172.21.15.44:0/3015252489 shutdown_connections 2024-05-27T09:35:15.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.120+0000 7f943e800640 1 -- 172.21.15.44:0/3015252489 wait complete. 2024-05-27T09:35:15.144 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-27T09:35:15.155 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-27T09:35:15.166 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T09:35:15.434 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:15 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T09:35:15.434 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:15 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T09:35:15.434 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:15 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T09:35:15.434 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:15 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T09:35:15.434 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:15 smithi050 ceph-mon[36371]: [27/May/2024:09:35:14] ENGINE Bus STARTING 2024-05-27T09:35:15.434 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:15 smithi050 ceph-mon[36371]: [27/May/2024:09:35:14] ENGINE Serving on http://172.21.15.44:8765 2024-05-27T09:35:15.434 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:15 smithi050 ceph-mon[36371]: [27/May/2024:09:35:14] ENGINE Serving on https://172.21.15.44:7150 2024-05-27T09:35:15.434 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:15 smithi050 ceph-mon[36371]: [27/May/2024:09:35:14] ENGINE Bus STARTED 2024-05-27T09:35:15.435 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:15 smithi050 ceph-mon[36371]: mgrmap e23: x(active, since 1.03089s), standbys: y, z 2024-05-27T09:35:15.435 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:15 smithi050 ceph-mon[36371]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:15.435 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:15 smithi050 ceph-mon[36371]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:15.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.435+0000 7f81faa00640 1 -- 172.21.15.44:0/1935333876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81f41028f0 msgr2=0x7f81f41035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:15.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.435+0000 7f81faa00640 1 --2- 172.21.15.44:0/1935333876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81f41028f0 0x7f81f41035f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009ca0 tx=0x7f81e802a8c0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.435+0000 7f81faa00640 1 -- 172.21.15.44:0/1935333876 shutdown_connections 2024-05-27T09:35:15.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.435+0000 7f81faa00640 1 --2- 172.21.15.44:0/1935333876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f4108830 0x7f81f410ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.435+0000 7f81faa00640 1 --2- 172.21.15.44:0/1935333876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81f4103b30 0x7f81f4103f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.435+0000 7f81faa00640 1 --2- 172.21.15.44:0/1935333876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81f41028f0 0x7f81f41035f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.435+0000 7f81faa00640 1 -- 172.21.15.44:0/1935333876 >> 172.21.15.44:0/1935333876 conn(0x7f81f40fdae0 msgr2=0x7f81f40fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:15.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.435+0000 7f81faa00640 1 -- 172.21.15.44:0/1935333876 shutdown_connections 2024-05-27T09:35:15.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.435+0000 7f81faa00640 1 -- 172.21.15.44:0/1935333876 wait complete. 2024-05-27T09:35:15.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.437+0000 7f81faa00640 1 Processor -- start 2024-05-27T09:35:15.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.437+0000 7f81faa00640 1 -- start start 2024-05-27T09:35:15.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81faa00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81f41028f0 0x7f81f419c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:15.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81faa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f4103b30 0x7f81f419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:15.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81faa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81f4108830 0x7f81f41a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:15.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81faa00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f81f410d310 con 0x7f81f4108830 2024-05-27T09:35:15.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81faa00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f81f410d190 con 0x7f81f41028f0 2024-05-27T09:35:15.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81faa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f81f410d490 con 0x7f81f4103b30 2024-05-27T09:35:15.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81f9600640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81f41028f0 0x7f81f419c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:15.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81f9600640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81f41028f0 0x7f81f419c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42968/0 (socket says 172.21.15.44:42968) 2024-05-27T09:35:15.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81f9600640 1 -- 172.21.15.44:0/754771734 learned_addr learned my addr 172.21.15.44:0/754771734 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:15.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81f8c00640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f4103b30 0x7f81f419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:15.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81fa000640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81f4108830 0x7f81f41a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:15.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81f9600640 1 -- 172.21.15.44:0/754771734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f4103b30 msgr2=0x7f81f419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:15.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81f9600640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f4103b30 0x7f81f419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81f9600640 1 -- 172.21.15.44:0/754771734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81f4108830 msgr2=0x7f81f41a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:15.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81f9600640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81f4108830 0x7f81f41a3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81f9600640 1 -- 172.21.15.44:0/754771734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f41a3ea0 con 0x7f81f41028f0 2024-05-27T09:35:15.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.438+0000 7f81fa000640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81f4108830 0x7f81f41a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:35:15.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.439+0000 7f81f9600640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81f41028f0 0x7f81f419c360 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f81e80098d0 tx=0x7f81e8031700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:15.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.439+0000 7f81f2000640 1 -- 172.21.15.44:0/754771734 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e8003b70 con 0x7f81f41028f0 2024-05-27T09:35:15.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.439+0000 7f81f2000640 1 -- 172.21.15.44:0/754771734 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81e8003d10 con 0x7f81f41028f0 2024-05-27T09:35:15.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.439+0000 7f81f2000640 1 -- 172.21.15.44:0/754771734 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e802fa50 con 0x7f81f41028f0 2024-05-27T09:35:15.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.439+0000 7f81faa00640 1 -- 172.21.15.44:0/754771734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81f41a4130 con 0x7f81f41028f0 2024-05-27T09:35:15.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.439+0000 7f81faa00640 1 -- 172.21.15.44:0/754771734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81f41a4630 con 0x7f81f41028f0 2024-05-27T09:35:15.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.440+0000 7f81faa00640 1 -- 172.21.15.44:0/754771734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81f4107640 con 0x7f81f41028f0 2024-05-27T09:35:15.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.443+0000 7f81f2000640 1 -- 172.21.15.44:0/754771734 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 23) ==== 149464+0+0 (secure 0 0 0) 0x7f81e8004580 con 0x7f81f41028f0 2024-05-27T09:35:15.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.445+0000 7f81f2000640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f81dc0b2660 0x7f81dc0b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:15.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.445+0000 7f81f2000640 1 -- 172.21.15.44:0/754771734 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f81e81034b0 con 0x7f81f41028f0 2024-05-27T09:35:15.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.445+0000 7f81f8c00640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f81dc0b2660 0x7f81dc0b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:15.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.446+0000 7f81f8c00640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f81dc0b2660 0x7f81dc0b4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f81f419d400 tx=0x7f81e0005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:15.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.446+0000 7f81f2000640 1 -- 172.21.15.44:0/754771734 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f81e80cb390 con 0x7f81f41028f0 2024-05-27T09:35:15.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:15 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:15.106+0000 7f49bd000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:15.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:15 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:15.106+0000 7f49bd000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:15.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:15 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:15.106+0000 7f49bd000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:15.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:15 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:15.106+0000 7f49bd000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:15.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:35:15 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:35:15.106+0000 7f49bd000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:35:15.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:15 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T09:35:15.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:15 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T09:35:15.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:15 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T09:35:15.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:15 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T09:35:15.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:15 smithi044 ceph-mon[31618]: [27/May/2024:09:35:14] ENGINE Bus STARTING 2024-05-27T09:35:15.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:15 smithi044 ceph-mon[31618]: [27/May/2024:09:35:14] ENGINE Serving on http://172.21.15.44:8765 2024-05-27T09:35:15.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:15 smithi044 ceph-mon[31618]: [27/May/2024:09:35:14] ENGINE Serving on https://172.21.15.44:7150 2024-05-27T09:35:15.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:15 smithi044 ceph-mon[31618]: [27/May/2024:09:35:14] ENGINE Bus STARTED 2024-05-27T09:35:15.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:15 smithi044 ceph-mon[31618]: mgrmap e23: x(active, since 1.03089s), standbys: y, z 2024-05-27T09:35:15.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:15 smithi044 ceph-mon[31618]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:15.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:15 smithi044 ceph-mon[31618]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:15.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:15 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T09:35:15.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:15 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T09:35:15.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:15 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T09:35:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:15 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T09:35:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:15 smithi189 ceph-mon[36671]: [27/May/2024:09:35:14] ENGINE Bus STARTING 2024-05-27T09:35:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:15 smithi189 ceph-mon[36671]: [27/May/2024:09:35:14] ENGINE Serving on http://172.21.15.44:8765 2024-05-27T09:35:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:15 smithi189 ceph-mon[36671]: [27/May/2024:09:35:14] ENGINE Serving on https://172.21.15.44:7150 2024-05-27T09:35:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:15 smithi189 ceph-mon[36671]: [27/May/2024:09:35:14] ENGINE Bus STARTED 2024-05-27T09:35:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:15 smithi189 ceph-mon[36671]: mgrmap e23: x(active, since 1.03089s), standbys: y, z 2024-05-27T09:35:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:15 smithi189 ceph-mon[36671]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:15 smithi189 ceph-mon[36671]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:15.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.662+0000 7f81faa00640 1 -- 172.21.15.44:0/754771734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81f402ddf0 con 0x7f81f41028f0 2024-05-27T09:35:15.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.664+0000 7f81f2000640 1 -- 172.21.15.44:0/754771734 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+22988 (secure 0 0 0) 0x7f81e80cac90 con 0x7f81f41028f0 2024-05-27T09:35:15.665 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:15.666 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:35:14.029489+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6800/4077502151":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/2278564126":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/4102964244":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/408258823":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3716545176":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/4077502151":"2024-05-28T09:35:14.029409+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:35:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.667+0000 7f81faa00640 1 -- 172.21.15.44:0/754771734 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f81dc0b2660 msgr2=0x7f81dc0b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.667+0000 7f81faa00640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f81dc0b2660 0x7f81dc0b4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f81f419d400 tx=0x7f81e0005f90 comp rx=0 tx=0).stop 2024-05-27T09:35:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.667+0000 7f81faa00640 1 -- 172.21.15.44:0/754771734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81f41028f0 msgr2=0x7f81f419c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.667+0000 7f81faa00640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81f41028f0 0x7f81f419c360 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f81e80098d0 tx=0x7f81e8031700 comp rx=0 tx=0).stop 2024-05-27T09:35:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.667+0000 7f81faa00640 1 -- 172.21.15.44:0/754771734 shutdown_connections 2024-05-27T09:35:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.667+0000 7f81faa00640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f81dc0b2660 0x7f81dc0b4b70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.667+0000 7f81faa00640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81f4108830 0x7f81f41a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.667+0000 7f81faa00640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f4103b30 0x7f81f419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.667+0000 7f81faa00640 1 --2- 172.21.15.44:0/754771734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81f41028f0 0x7f81f419c360 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.667+0000 7f81faa00640 1 -- 172.21.15.44:0/754771734 >> 172.21.15.44:0/754771734 conn(0x7f81f40fdae0 msgr2=0x7f81f4104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.668+0000 7f81faa00640 1 -- 172.21.15.44:0/754771734 shutdown_connections 2024-05-27T09:35:15.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.668+0000 7f81faa00640 1 -- 172.21.15.44:0/754771734 wait complete. 2024-05-27T09:35:15.694 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T09:25:46.636320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T09:31:19.459994+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T09:31:23.749576+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T09:35:15.695 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T09:35:15.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.983+0000 7fa81d200640 1 -- 172.21.15.44:0/2052771444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa818106b70 msgr2=0x7fa818109050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:15.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.983+0000 7fa81d200640 1 --2- 172.21.15.44:0/2052771444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa818106b70 0x7fa818109050 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa804009900 tx=0x7fa80402a410 comp rx=0 tx=0).stop 2024-05-27T09:35:15.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.983+0000 7fa81d200640 1 -- 172.21.15.44:0/2052771444 shutdown_connections 2024-05-27T09:35:15.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.983+0000 7fa81d200640 1 --2- 172.21.15.44:0/2052771444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8181096e0 0x7fa81810bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.983+0000 7fa81d200640 1 --2- 172.21.15.44:0/2052771444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa818106b70 0x7fa818109050 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.983+0000 7fa81d200640 1 --2- 172.21.15.44:0/2052771444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa818104180 0x7fa8181065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.983+0000 7fa81d200640 1 -- 172.21.15.44:0/2052771444 >> 172.21.15.44:0/2052771444 conn(0x7fa8180fdb60 msgr2=0x7fa8180ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:15.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.983+0000 7fa81d200640 1 -- 172.21.15.44:0/2052771444 shutdown_connections 2024-05-27T09:35:15.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.983+0000 7fa81d200640 1 -- 172.21.15.44:0/2052771444 wait complete. 2024-05-27T09:35:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.984+0000 7fa81d200640 1 Processor -- start 2024-05-27T09:35:15.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa81d200640 1 -- start start 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa81d200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa818104180 0x7fa81819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa81d200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa818106b70 0x7fa81819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa81d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8181096e0 0x7fa8181a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa81d200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa81810f5a0 con 0x7fa818104180 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa81d200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa81810f420 con 0x7fa818106b70 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa81d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa81810f720 con 0x7fa8181096e0 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa817400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa818104180 0x7fa81819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa817e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8181096e0 0x7fa8181a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa817400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa818104180 0x7fa81819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44188/0 (socket says 172.21.15.44:44188) 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa817400640 1 -- 172.21.15.44:0/2248729677 learned_addr learned my addr 172.21.15.44:0/2248729677 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.985+0000 7fa816a00640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa818106b70 0x7fa81819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.986+0000 7fa817e00640 1 -- 172.21.15.44:0/2248729677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa818106b70 msgr2=0x7fa81819a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:15.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.986+0000 7fa817e00640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa818106b70 0x7fa81819a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.986+0000 7fa817e00640 1 -- 172.21.15.44:0/2248729677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa818104180 msgr2=0x7fa81819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.986+0000 7fa817e00640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa818104180 0x7fa81819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.986+0000 7fa817e00640 1 -- 172.21.15.44:0/2248729677 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8181a1c60 con 0x7fa8181096e0 2024-05-27T09:35:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.986+0000 7fa817400640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa818104180 0x7fa81819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:15.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.986+0000 7fa817e00640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8181096e0 0x7fa8181a15f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa810009fd0 tx=0x7fa81000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:15.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.986+0000 7fa80fe00640 1 -- 172.21.15.44:0/2248729677 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81000ce20 con 0x7fa8181096e0 2024-05-27T09:35:15.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.986+0000 7fa80fe00640 1 -- 172.21.15.44:0/2248729677 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa810005020 con 0x7fa8181096e0 2024-05-27T09:35:15.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.986+0000 7fa81d200640 1 -- 172.21.15.44:0/2248729677 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8181a1f50 con 0x7fa8181096e0 2024-05-27T09:35:15.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.987+0000 7fa80fe00640 1 -- 172.21.15.44:0/2248729677 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810010690 con 0x7fa8181096e0 2024-05-27T09:35:15.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.987+0000 7fa81d200640 1 -- 172.21.15.44:0/2248729677 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8181a23c0 con 0x7fa8181096e0 2024-05-27T09:35:15.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.988+0000 7fa81d200640 1 -- 172.21.15.44:0/2248729677 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8180ff680 con 0x7fa8181096e0 2024-05-27T09:35:15.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.990+0000 7fa80fe00640 1 -- 172.21.15.44:0/2248729677 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 23) ==== 149464+0+0 (secure 0 0 0) 0x7fa810004b20 con 0x7fa8181096e0 2024-05-27T09:35:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.991+0000 7fa80fe00640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7fc0b2660 0x7fa7fc0b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.992+0000 7fa80fe00640 1 -- 172.21.15.44:0/2248729677 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa8100e2550 con 0x7fa8181096e0 2024-05-27T09:35:15.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.992+0000 7fa817400640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7fc0b2660 0x7fa7fc0b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:15.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.993+0000 7fa817400640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7fc0b2660 0x7fa7fc0b4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa8080058c0 tx=0x7fa808005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:15.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:15.996+0000 7fa80fe00640 1 -- 172.21.15.44:0/2248729677 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa8100aa430 con 0x7fa8181096e0 2024-05-27T09:35:16.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.174+0000 7fa80fe00640 1 -- 172.21.15.44:0/2248729677 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 24) ==== 149823+0+0 (secure 0 0 0) 0x7fa8100a5fc0 con 0x7fa8181096e0 2024-05-27T09:35:16.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.212+0000 7fa81d200640 1 -- 172.21.15.44:0/2248729677 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa8180638c0 con 0x7fa8181096e0 2024-05-27T09:35:16.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.213+0000 7fa80fe00640 1 -- 172.21.15.44:0/2248729677 <== mon.1 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fa81004a800 con 0x7fa8181096e0 2024-05-27T09:35:16.214 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 -- 172.21.15.44:0/2248729677 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7fc0b2660 msgr2=0x7fa7fc0b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7fc0b2660 0x7fa7fc0b4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa8080058c0 tx=0x7fa808005f90 comp rx=0 tx=0).stop 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 -- 172.21.15.44:0/2248729677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8181096e0 msgr2=0x7fa8181a15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8181096e0 0x7fa8181a15f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa810009fd0 tx=0x7fa81000ef10 comp rx=0 tx=0).stop 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 -- 172.21.15.44:0/2248729677 shutdown_connections 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8181096e0 0x7fa8181a15f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa818106b70 0x7fa81819a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7fc0b2660 0x7fa7fc0b4b70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 --2- 172.21.15.44:0/2248729677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa818104180 0x7fa81819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 -- 172.21.15.44:0/2248729677 >> 172.21.15.44:0/2248729677 conn(0x7fa8180fdb60 msgr2=0x7fa818104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 -- 172.21.15.44:0/2248729677 shutdown_connections 2024-05-27T09:35:16.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.216+0000 7fa81d200640 1 -- 172.21.15.44:0/2248729677 wait complete. 2024-05-27T09:35:16.240 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T09:35:16.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:16 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/754771734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:35:16.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:16 smithi044 ceph-mon[31618]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:16.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.522+0000 7f2dbe800640 1 -- 172.21.15.44:0/1146356924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2db8103a30 msgr2=0x7f2db8104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:16.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.522+0000 7f2dbe800640 1 --2- 172.21.15.44:0/1146356924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2db8103a30 0x7f2db8104730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009900 tx=0x7f2dac02a410 comp rx=0 tx=0).stop 2024-05-27T09:35:16.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.522+0000 7f2dbe800640 1 -- 172.21.15.44:0/1146356924 shutdown_connections 2024-05-27T09:35:16.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.522+0000 7f2dbe800640 1 --2- 172.21.15.44:0/1146356924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db8109970 0x7f2db810bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.522+0000 7f2dbe800640 1 --2- 172.21.15.44:0/1146356924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2db8104c70 0x7f2db8105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.522+0000 7f2dbe800640 1 --2- 172.21.15.44:0/1146356924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2db8103a30 0x7f2db8104730 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.522+0000 7f2dbe800640 1 -- 172.21.15.44:0/1146356924 >> 172.21.15.44:0/1146356924 conn(0x7f2db80fec40 msgr2=0x7f2db81010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.522+0000 7f2dbe800640 1 -- 172.21.15.44:0/1146356924 shutdown_connections 2024-05-27T09:35:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.522+0000 7f2dbe800640 1 -- 172.21.15.44:0/1146356924 wait complete. 2024-05-27T09:35:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.523+0000 7f2dbe800640 1 Processor -- start 2024-05-27T09:35:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbe800640 1 -- start start 2024-05-27T09:35:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbe800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2db8103a30 0x7f2db819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbe800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db8104c70 0x7f2db819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbe800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2db8109970 0x7f2db81a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbe800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2db810f530 con 0x7f2db8103a30 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbe800640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2db810f3b0 con 0x7f2db8109970 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbe800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2db810f6b0 con 0x7f2db8104c70 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbde00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2db8109970 0x7f2db81a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbd400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2db8103a30 0x7f2db819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbde00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2db8109970 0x7f2db81a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43004/0 (socket says 172.21.15.44:43004) 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbde00640 1 -- 172.21.15.44:0/3213244282 learned_addr learned my addr 172.21.15.44:0/3213244282 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.524+0000 7f2dbca00640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db8104c70 0x7f2db819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.525+0000 7f2dbde00640 1 -- 172.21.15.44:0/3213244282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db8104c70 msgr2=0x7f2db819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.525+0000 7f2dbde00640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db8104c70 0x7f2db819a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.525+0000 7f2dbde00640 1 -- 172.21.15.44:0/3213244282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2db8103a30 msgr2=0x7f2db819a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.525+0000 7f2dbde00640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2db8103a30 0x7f2db819a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.525+0000 7f2dbde00640 1 -- 172.21.15.44:0/3213244282 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db81a1cb0 con 0x7f2db8109970 2024-05-27T09:35:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.525+0000 7f2dbde00640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2db8109970 0x7f2db81a1640 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2db000eb10 tx=0x7f2db000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.525+0000 7f2db6000640 1 -- 172.21.15.44:0/3213244282 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0004b20 con 0x7f2db8109970 2024-05-27T09:35:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.525+0000 7f2db6000640 1 -- 172.21.15.44:0/3213244282 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db0004cc0 con 0x7f2db8109970 2024-05-27T09:35:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.525+0000 7f2db6000640 1 -- 172.21.15.44:0/3213244282 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0010670 con 0x7f2db8109970 2024-05-27T09:35:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.525+0000 7f2dbe800640 1 -- 172.21.15.44:0/3213244282 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db81a1fa0 con 0x7f2db8109970 2024-05-27T09:35:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.526+0000 7f2dbe800640 1 -- 172.21.15.44:0/3213244282 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db81a23b0 con 0x7f2db8109970 2024-05-27T09:35:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.527+0000 7f2dbe800640 1 -- 172.21.15.44:0/3213244282 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db8108780 con 0x7f2db8109970 2024-05-27T09:35:16.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.529+0000 7f2db6000640 1 -- 172.21.15.44:0/3213244282 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 149823+0+0 (secure 0 0 0) 0x7f2db0047a00 con 0x7f2db8109970 2024-05-27T09:35:16.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.531+0000 7f2db6000640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2da00b29e0 0x7f2da00b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:16.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.531+0000 7f2db6000640 1 -- 172.21.15.44:0/3213244282 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f2db00e2cc0 con 0x7f2db8109970 2024-05-27T09:35:16.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.531+0000 7f2dbd400640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2da00b29e0 0x7f2da00b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:16.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.532+0000 7f2dbd400640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2da00b29e0 0x7f2da00b4ef0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2dac005ab0 tx=0x7f2dac031400 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:16.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.533+0000 7f2db6000640 1 -- 172.21.15.44:0/3213244282 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2db00aa500 con 0x7f2db8109970 2024-05-27T09:35:16.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:16 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/754771734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:35:16.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:16 smithi050 ceph-mon[36371]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:16.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:16 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/754771734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:35:16.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:16 smithi189 ceph-mon[36671]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:16.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.763+0000 7f2dbe800640 1 -- 172.21.15.44:0/3213244282 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2db802de90 con 0x7f2db8109970 2024-05-27T09:35:16.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.764+0000 7f2db6000640 1 -- 172.21.15.44:0/3213244282 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) ==== 104+0+11 (secure 0 0 0) 0x7f2db00a9e00 con 0x7f2db8109970 2024-05-27T09:35:16.764 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-05-27T09:35:16.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 -- 172.21.15.44:0/3213244282 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2da00b29e0 msgr2=0x7f2da00b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2da00b29e0 0x7f2da00b4ef0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2dac005ab0 tx=0x7f2dac031400 comp rx=0 tx=0).stop 2024-05-27T09:35:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 -- 172.21.15.44:0/3213244282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2db8109970 msgr2=0x7f2db81a1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2db8109970 0x7f2db81a1640 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2db000eb10 tx=0x7f2db000c2d0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 -- 172.21.15.44:0/3213244282 shutdown_connections 2024-05-27T09:35:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2db8109970 0x7f2db81a1640 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db8104c70 0x7f2db819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2da00b29e0 0x7f2da00b4ef0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 --2- 172.21.15.44:0/3213244282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2db8103a30 0x7f2db819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 -- 172.21.15.44:0/3213244282 >> 172.21.15.44:0/3213244282 conn(0x7f2db80fec40 msgr2=0x7f2db8105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 -- 172.21.15.44:0/3213244282 shutdown_connections 2024-05-27T09:35:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:16.767+0000 7f2dbe800640 1 -- 172.21.15.44:0/3213244282 wait complete. 2024-05-27T09:35:16.797 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T09:35:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.068+0000 7f2ed3400640 1 -- 172.21.15.44:0/1419208440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ecc109890 msgr2=0x7f2ecc10bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.068+0000 7f2ed3400640 1 --2- 172.21.15.44:0/1419208440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ecc109890 0x7f2ecc10bda0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4009ca0 tx=0x7f2ec402a8c0 comp rx=0 tx=0).stop 2024-05-27T09:35:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.069+0000 7f2ed3400640 1 -- 172.21.15.44:0/1419208440 shutdown_connections 2024-05-27T09:35:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.069+0000 7f2ed3400640 1 --2- 172.21.15.44:0/1419208440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ecc109890 0x7f2ecc10bda0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.069+0000 7f2ed3400640 1 --2- 172.21.15.44:0/1419208440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc106d60 0x7f2ecc109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.069+0000 7f2ed3400640 1 --2- 172.21.15.44:0/1419208440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ecc104370 0x7f2ecc106790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.069+0000 7f2ed3400640 1 -- 172.21.15.44:0/1419208440 >> 172.21.15.44:0/1419208440 conn(0x7f2ecc0fdcf0 msgr2=0x7f2ecc1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.069+0000 7f2ed3400640 1 -- 172.21.15.44:0/1419208440 shutdown_connections 2024-05-27T09:35:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.069+0000 7f2ed3400640 1 -- 172.21.15.44:0/1419208440 wait complete. 2024-05-27T09:35:17.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.070+0000 7f2ed3400640 1 Processor -- start 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.070+0000 7f2ed3400640 1 -- start start 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ed3400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ecc104370 0x7f2ecc103be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ed3400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc106d60 0x7f2ecc102280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ed3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ecc109890 0x7f2ecc1027c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ed3400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc10f770 con 0x7f2ecc106d60 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ed3400640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc10f5f0 con 0x7f2ecc104370 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ed3400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc10f8f0 con 0x7f2ecc109890 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ed0a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ecc104370 0x7f2ecc103be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ed1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ecc109890 0x7f2ecc1027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ecbe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc106d60 0x7f2ecc102280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ed0a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ecc104370 0x7f2ecc103be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43024/0 (socket says 172.21.15.44:43024) 2024-05-27T09:35:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ed0a00640 1 -- 172.21.15.44:0/2269675132 learned_addr learned my addr 172.21.15.44:0/2269675132 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:17.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.071+0000 7f2ed1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ecc109890 0x7f2ecc1027c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55908/0 (socket says 172.21.15.44:55908) 2024-05-27T09:35:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ed0a00640 1 -- 172.21.15.44:0/2269675132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ecc109890 msgr2=0x7f2ecc1027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ed0a00640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ecc109890 0x7f2ecc1027c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ed0a00640 1 -- 172.21.15.44:0/2269675132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc106d60 msgr2=0x7f2ecc102280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ed0a00640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc106d60 0x7f2ecc102280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ed0a00640 1 -- 172.21.15.44:0/2269675132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ecc102ef0 con 0x7f2ecc104370 2024-05-27T09:35:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ed1400640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ecc109890 0x7f2ecc1027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ed0a00640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ecc104370 0x7f2ecc103be0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00e9f0 tx=0x7f2ebc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ec9600640 1 -- 172.21.15.44:0/2269675132 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc00ccc0 con 0x7f2ecc104370 2024-05-27T09:35:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ec9600640 1 -- 172.21.15.44:0/2269675132 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ebc0054d0 con 0x7f2ecc104370 2024-05-27T09:35:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ec9600640 1 -- 172.21.15.44:0/2269675132 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc010430 con 0x7f2ecc104370 2024-05-27T09:35:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ed3400640 1 -- 172.21.15.44:0/2269675132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ecc1a43a0 con 0x7f2ecc104370 2024-05-27T09:35:17.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.072+0000 7f2ed3400640 1 -- 172.21.15.44:0/2269675132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ecc1a4860 con 0x7f2ecc104370 2024-05-27T09:35:17.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.074+0000 7f2ed3400640 1 -- 172.21.15.44:0/2269675132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ecc0ff810 con 0x7f2ecc104370 2024-05-27T09:35:17.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.076+0000 7f2ec9600640 1 -- 172.21.15.44:0/2269675132 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 149823+0+0 (secure 0 0 0) 0x7f2ebc0046c0 con 0x7f2ecc104370 2024-05-27T09:35:17.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.077+0000 7f2ec9600640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2eb40b2a70 0x7f2eb40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:17.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.078+0000 7f2ec9600640 1 -- 172.21.15.44:0/2269675132 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f2ebc0e2a30 con 0x7f2ecc104370 2024-05-27T09:35:17.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.078+0000 7f2ecbe00640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2eb40b2a70 0x7f2eb40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.079+0000 7f2ecbe00640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2eb40b2a70 0x7f2eb40b4f80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc103580 tx=0x7f2ec0005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.082+0000 7f2ec9600640 1 -- 172.21.15.44:0/2269675132 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2ebc0aa2d0 con 0x7f2ecc104370 2024-05-27T09:35:17.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.303+0000 7f2ed3400640 1 -- 172.21.15.44:0/2269675132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2ecc0638c0 con 0x7f2ecc104370 2024-05-27T09:35:17.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.304+0000 7f2ec9600640 1 -- 172.21.15.44:0/2269675132 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) ==== 100+0+11 (secure 0 0 0) 0x7f2ebc0a9bd0 con 0x7f2ecc104370 2024-05-27T09:35:17.304 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-05-27T09:35:17.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 -- 172.21.15.44:0/2269675132 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2eb40b2a70 msgr2=0x7f2eb40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:17.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2eb40b2a70 0x7f2eb40b4f80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc103580 tx=0x7f2ec0005f90 comp rx=0 tx=0).stop 2024-05-27T09:35:17.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 -- 172.21.15.44:0/2269675132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ecc104370 msgr2=0x7f2ecc103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:17.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ecc104370 0x7f2ecc103be0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00e9f0 tx=0x7f2ebc00eeb0 comp rx=0 tx=0).stop 2024-05-27T09:35:17.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 -- 172.21.15.44:0/2269675132 shutdown_connections 2024-05-27T09:35:17.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ecc109890 0x7f2ecc1027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:17.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2eb40b2a70 0x7f2eb40b4f80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:17.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ecc106d60 0x7f2ecc102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:17.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 --2- 172.21.15.44:0/2269675132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ecc104370 0x7f2ecc103be0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:17.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 -- 172.21.15.44:0/2269675132 >> 172.21.15.44:0/2269675132 conn(0x7f2ecc0fdcf0 msgr2=0x7f2ecc104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:17.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 -- 172.21.15.44:0/2269675132 shutdown_connections 2024-05-27T09:35:17.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:17.306+0000 7f2ed3400640 1 -- 172.21.15.44:0/2269675132 wait complete. 2024-05-27T09:35:17.332 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-27T09:35:17.332 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:35:17.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:17 smithi044 ceph-mon[31618]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-05-27T09:35:17.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:17 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2248729677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T09:35:17.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:17 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:17 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:17 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:17 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:17 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3213244282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T09:35:17.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:17 smithi050 ceph-mon[36371]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-05-27T09:35:17.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:17 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2248729677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T09:35:17.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:17 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:17 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:17 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:17 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:17 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3213244282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T09:35:17.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:17 smithi189 ceph-mon[36671]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-05-27T09:35:17.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:17 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2248729677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T09:35:17.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:17 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:17 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:17 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:17 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:17.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:17 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3213244282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T09:35:17.755 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:18.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2269675132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T09:35:18.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-05-27T09:35:18.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-05-27T09:35:18.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:18 smithi044 ceph-mon[31618]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2269675132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T09:35:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-05-27T09:35:18.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-05-27T09:35:18.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:18 smithi050 ceph-mon[36371]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:18.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2269675132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T09:35:18.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.590 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.590 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.590 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:18.590 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-05-27T09:35:18.590 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-05-27T09:35:18.590 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:18 smithi189 ceph-mon[36671]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:19.778 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:19 smithi044 ceph-mon[31618]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-05-27T09:35:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.031+0000 7f79d6600700 1 -- 172.21.15.44:0/3401091930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79d0110a20 msgr2=0x7f79d0112e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.031+0000 7f79d6600700 1 --2- 172.21.15.44:0/3401091930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79d0110a20 0x7f79d0112e40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f79c800b0a0 tx=0x7f79c800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:35:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.031+0000 7f79d6600700 1 -- 172.21.15.44:0/3401091930 shutdown_connections 2024-05-27T09:35:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.031+0000 7f79d6600700 1 --2- 172.21.15.44:0/3401091930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79d0110a20 0x7f79d0112e40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.031+0000 7f79d6600700 1 --2- 172.21.15.44:0/3401091930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79d0075ad0 0x7f79d0075ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:20.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.031+0000 7f79d6600700 1 --2- 172.21.15.44:0/3401091930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79d0076420 0x7f79d01104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:20.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.031+0000 7f79d6600700 1 -- 172.21.15.44:0/3401091930 >> 172.21.15.44:0/3401091930 conn(0x7f79d0079d00 msgr2=0x7f79d007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:20.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.031+0000 7f79d6600700 1 -- 172.21.15.44:0/3401091930 shutdown_connections 2024-05-27T09:35:20.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.031+0000 7f79d6600700 1 -- 172.21.15.44:0/3401091930 wait complete. 2024-05-27T09:35:20.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.033+0000 7f79d6600700 1 Processor -- start 2024-05-27T09:35:20.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.033+0000 7f79d6600700 1 -- start start 2024-05-27T09:35:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.034+0000 7f79d6600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79d0075ad0 0x7f79d019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.034+0000 7f79d6600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79d0076420 0x7f79d019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:20.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.034+0000 7f79d6600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79d0110a20 0x7f79d01a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:20.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.034+0000 7f79d6600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f79d0116500 con 0x7f79d0075ad0 2024-05-27T09:35:20.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.034+0000 7f79d6600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f79d0116380 con 0x7f79d0076420 2024-05-27T09:35:20.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.034+0000 7f79d6600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f79d0116680 con 0x7f79d0110a20 2024-05-27T09:35:20.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.034+0000 7f79d5c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79d0110a20 0x7f79d01a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.034+0000 7f79d5200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79d0075ad0 0x7f79d019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.034+0000 7f79cfe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79d0076420 0x7f79d019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.034+0000 7f79d5c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79d0110a20 0x7f79d01a16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55926/0 (socket says 172.21.15.44:55926) 2024-05-27T09:35:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.034+0000 7f79d5c00700 1 -- 172.21.15.44:0/2252593233 learned_addr learned my addr 172.21.15.44:0/2252593233 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.035+0000 7f79cfe00700 1 -- 172.21.15.44:0/2252593233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79d0110a20 msgr2=0x7f79d01a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.035+0000 7f79cfe00700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79d0110a20 0x7f79d01a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.035+0000 7f79cfe00700 1 -- 172.21.15.44:0/2252593233 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79d0075ad0 msgr2=0x7f79d019ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.035+0000 7f79cfe00700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79d0075ad0 0x7f79d019ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.035+0000 7f79cfe00700 1 -- 172.21.15.44:0/2252593233 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d01a1d40 con 0x7f79d0076420 2024-05-27T09:35:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.035+0000 7f79d5c00700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79d0110a20 0x7f79d01a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:20.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.035+0000 7f79cfe00700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79d0076420 0x7f79d019d410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00e9c0 tx=0x7f79bc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.036+0000 7f79cd600700 1 -- 172.21.15.44:0/2252593233 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc00cb70 con 0x7f79d0076420 2024-05-27T09:35:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.036+0000 7f79cd600700 1 -- 172.21.15.44:0/2252593233 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79bc00ee60 con 0x7f79d0076420 2024-05-27T09:35:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.036+0000 7f79d6600700 1 -- 172.21.15.44:0/2252593233 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d01a2030 con 0x7f79d0076420 2024-05-27T09:35:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.036+0000 7f79d6600700 1 -- 172.21.15.44:0/2252593233 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79d01a9950 con 0x7f79d0076420 2024-05-27T09:35:20.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.036+0000 7f79cd600700 1 -- 172.21.15.44:0/2252593233 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc003d60 con 0x7f79d0076420 2024-05-27T09:35:20.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.038+0000 7f79d6600700 1 -- 172.21.15.44:0/2252593233 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b0005330 con 0x7f79d0076420 2024-05-27T09:35:20.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.040+0000 7f79cd600700 1 -- 172.21.15.44:0/2252593233 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f79bc018430 con 0x7f79d0076420 2024-05-27T09:35:20.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.041+0000 7f79cd600700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79b80b2b90 0x7f79b80b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:20.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.041+0000 7f79cd600700 1 -- 172.21.15.44:0/2252593233 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f79bc0e2860 con 0x7f79d0076420 2024-05-27T09:35:20.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.042+0000 7f79d5200700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79b80b2b90 0x7f79b80b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:20.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:19 smithi050 ceph-mon[36371]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-05-27T09:35:20.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.042+0000 7f79d5200700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79b80b2b90 0x7f79b80b50a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009c60 tx=0x7f79c4009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:20.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.046+0000 7f79cd600700 1 -- 172.21.15.44:0/2252593233 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f79bc0a9fc0 con 0x7f79d0076420 2024-05-27T09:35:20.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:19 smithi189 ceph-mon[36671]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-05-27T09:35:20.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.358+0000 7f79d6600700 1 -- 172.21.15.44:0/2252593233 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79b00058f0 con 0x7f79d0076420 2024-05-27T09:35:20.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.360+0000 7f79cd600700 1 -- 172.21.15.44:0/2252593233 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7f79bc0b2040 con 0x7f79d0076420 2024-05-27T09:35:20.362 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:20.362 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:35:20.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.363+0000 7f79d6600700 1 -- 172.21.15.44:0/2252593233 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79b80b2b90 msgr2=0x7f79b80b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:20.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.363+0000 7f79d6600700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79b80b2b90 0x7f79b80b50a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009c60 tx=0x7f79c4009260 comp rx=0 tx=0).stop 2024-05-27T09:35:20.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.363+0000 7f79d6600700 1 -- 172.21.15.44:0/2252593233 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79d0076420 msgr2=0x7f79d019d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:20.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.363+0000 7f79d6600700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79d0076420 0x7f79d019d410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00e9c0 tx=0x7f79bc00ecd0 comp rx=0 tx=0).stop 2024-05-27T09:35:20.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.363+0000 7f79d6600700 1 -- 172.21.15.44:0/2252593233 shutdown_connections 2024-05-27T09:35:20.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.363+0000 7f79d6600700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79b80b2b90 0x7f79b80b50a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:20.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.363+0000 7f79d6600700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79d0110a20 0x7f79d01a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:20.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.363+0000 7f79d6600700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79d0075ad0 0x7f79d019ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:20.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.363+0000 7f79d6600700 1 --2- 172.21.15.44:0/2252593233 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79d0076420 0x7f79d019d410 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:20.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.363+0000 7f79d6600700 1 -- 172.21.15.44:0/2252593233 >> 172.21.15.44:0/2252593233 conn(0x7f79d0079d00 msgr2=0x7f79d0100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:20.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.364+0000 7f79d6600700 1 -- 172.21.15.44:0/2252593233 shutdown_connections 2024-05-27T09:35:20.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:20.364+0000 7f79d6600700 1 -- 172.21.15.44:0/2252593233 wait complete. 2024-05-27T09:35:20.366 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:35:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-05-27T09:35:21.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: Unable to set osd_memory_target on smithi044 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T09:35:21.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:21.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:21.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: Updating smithi044:/etc/ceph/ceph.conf 2024-05-27T09:35:21.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: Updating smithi050:/etc/ceph/ceph.conf 2024-05-27T09:35:21.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-27T09:35:21.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:21.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:20 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2252593233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:21.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-05-27T09:35:21.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: Unable to set osd_memory_target on smithi044 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T09:35:21.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:21.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:21.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: Updating smithi044:/etc/ceph/ceph.conf 2024-05-27T09:35:21.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: Updating smithi050:/etc/ceph/ceph.conf 2024-05-27T09:35:21.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-27T09:35:21.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:21.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:20 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2252593233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:21.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:21.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:21.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:35:21.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-05-27T09:35:21.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: Unable to set osd_memory_target on smithi044 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T09:35:21.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:21.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:21.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: Updating smithi044:/etc/ceph/ceph.conf 2024-05-27T09:35:21.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: Updating smithi050:/etc/ceph/ceph.conf 2024-05-27T09:35:21.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: Updating smithi189:/etc/ceph/ceph.conf 2024-05-27T09:35:21.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:35:21.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:20 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2252593233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:21.268 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-27T09:35:21.268 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-27T09:35:21.269 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T09:35:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.530+0000 7f3a7da00640 1 -- 172.21.15.44:0/1468010788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a78103640 msgr2=0x7f3a78103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.530+0000 7f3a7da00640 1 --2- 172.21.15.44:0/1468010788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a78103640 0x7f3a78103b40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009ca0 tx=0x7f3a7002a8b0 comp rx=0 tx=0).stop 2024-05-27T09:35:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.530+0000 7f3a7da00640 1 -- 172.21.15.44:0/1468010788 shutdown_connections 2024-05-27T09:35:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.530+0000 7f3a7da00640 1 --2- 172.21.15.44:0/1468010788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a78103640 0x7f3a78103b40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.530+0000 7f3a7da00640 1 --2- 172.21.15.44:0/1468010788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a78102ae0 0x7f3a78102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.530+0000 7f3a7da00640 1 --2- 172.21.15.44:0/1468010788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a78102100 0x7f3a78102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.530+0000 7f3a7da00640 1 -- 172.21.15.44:0/1468010788 >> 172.21.15.44:0/1468010788 conn(0x7f3a780fdcd0 msgr2=0x7f3a78100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.530+0000 7f3a7da00640 1 -- 172.21.15.44:0/1468010788 shutdown_connections 2024-05-27T09:35:21.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.530+0000 7f3a7da00640 1 -- 172.21.15.44:0/1468010788 wait complete. 2024-05-27T09:35:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.531+0000 7f3a7da00640 1 Processor -- start 2024-05-27T09:35:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.531+0000 7f3a7da00640 1 -- start start 2024-05-27T09:35:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a7da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a78102100 0x7f3a7819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a7da00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a78102ae0 0x7f3a7819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a7da00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a78103640 0x7f3a781a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a7da00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3a78111930 con 0x7f3a78102ae0 2024-05-27T09:35:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a7da00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3a781117b0 con 0x7f3a78103640 2024-05-27T09:35:21.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a7da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a78111ab0 con 0x7f3a78102100 2024-05-27T09:35:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a76a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a78102100 0x7f3a7819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a76000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a78102ae0 0x7f3a7819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a76a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a78102100 0x7f3a7819c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55944/0 (socket says 172.21.15.44:55944) 2024-05-27T09:35:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a76a00640 1 -- 172.21.15.44:0/302507603 learned_addr learned my addr 172.21.15.44:0/302507603 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a77400640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a78103640 0x7f3a781a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a76a00640 1 -- 172.21.15.44:0/302507603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a78103640 msgr2=0x7f3a781a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a76a00640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a78103640 0x7f3a781a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a76a00640 1 -- 172.21.15.44:0/302507603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a78102ae0 msgr2=0x7f3a7819c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a76a00640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a78102ae0 0x7f3a7819c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.532+0000 7f3a76a00640 1 -- 172.21.15.44:0/302507603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a781a3fb0 con 0x7f3a78102100 2024-05-27T09:35:21.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.533+0000 7f3a77400640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a78103640 0x7f3a781a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:21.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.533+0000 7f3a76a00640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a78102100 0x7f3a7819c470 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800e9c0 tx=0x7f3a6800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:21.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.533+0000 7f3a6f400640 1 -- 172.21.15.44:0/302507603 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6800cd80 con 0x7f3a78102100 2024-05-27T09:35:21.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.533+0000 7f3a6f400640 1 -- 172.21.15.44:0/302507603 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a680035f0 con 0x7f3a78102100 2024-05-27T09:35:21.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.533+0000 7f3a6f400640 1 -- 172.21.15.44:0/302507603 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68010690 con 0x7f3a78102100 2024-05-27T09:35:21.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.533+0000 7f3a7da00640 1 -- 172.21.15.44:0/302507603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a781a42a0 con 0x7f3a78102100 2024-05-27T09:35:21.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.533+0000 7f3a7da00640 1 -- 172.21.15.44:0/302507603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a781a4840 con 0x7f3a78102100 2024-05-27T09:35:21.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.535+0000 7f3a7da00640 1 -- 172.21.15.44:0/302507603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a7806b0c0 con 0x7f3a78102100 2024-05-27T09:35:21.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.537+0000 7f3a6f400640 1 -- 172.21.15.44:0/302507603 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f3a680037a0 con 0x7f3a78102100 2024-05-27T09:35:21.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.539+0000 7f3a6f400640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a5c0b2b00 0x7f3a5c0b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:21.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.539+0000 7f3a6f400640 1 -- 172.21.15.44:0/302507603 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f3a680e2c10 con 0x7f3a78102100 2024-05-27T09:35:21.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.539+0000 7f3a76000640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a5c0b2b00 0x7f3a5c0b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:21.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.540+0000 7f3a76000640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a5c0b2b00 0x7f3a5c0b5010 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3a7819d510 tx=0x7f3a64005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:21.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.543+0000 7f3a6f400640 1 -- 172.21.15.44:0/302507603 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3a680aa370 con 0x7f3a78102100 2024-05-27T09:35:21.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.769+0000 7f3a7da00640 1 -- 172.21.15.44:0/302507603 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3a78063100 con 0x7f3a5c0b2b00 2024-05-27T09:35:21.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.778+0000 7f3a6f400640 1 -- 172.21.15.44:0/302507603 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257750 (secure 0 0 0) 0x7f3a78063100 con 0x7f3a5c0b2b00 2024-05-27T09:35:21.783 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:21.785 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-05-27T09:35:21.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.784+0000 7f3a7da00640 1 -- 172.21.15.44:0/302507603 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a5c0b2b00 msgr2=0x7f3a5c0b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:21.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.784+0000 7f3a7da00640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a5c0b2b00 0x7f3a5c0b5010 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3a7819d510 tx=0x7f3a64005f90 comp rx=0 tx=0).stop 2024-05-27T09:35:21.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.784+0000 7f3a7da00640 1 -- 172.21.15.44:0/302507603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a78102100 msgr2=0x7f3a7819c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:21.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.784+0000 7f3a7da00640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a78102100 0x7f3a7819c470 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800e9c0 tx=0x7f3a6800ee80 comp rx=0 tx=0).stop 2024-05-27T09:35:21.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.784+0000 7f3a7da00640 1 -- 172.21.15.44:0/302507603 shutdown_connections 2024-05-27T09:35:21.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.784+0000 7f3a7da00640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a78103640 0x7f3a781a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:21.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.784+0000 7f3a7da00640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a5c0b2b00 0x7f3a5c0b5010 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:21.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.784+0000 7f3a7da00640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a78102ae0 0x7f3a7819c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:21.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.784+0000 7f3a7da00640 1 --2- 172.21.15.44:0/302507603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a78102100 0x7f3a7819c470 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:21.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.784+0000 7f3a7da00640 1 -- 172.21.15.44:0/302507603 >> 172.21.15.44:0/302507603 conn(0x7f3a780fdcd0 msgr2=0x7f3a7810a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:21.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.785+0000 7f3a7da00640 1 -- 172.21.15.44:0/302507603 shutdown_connections 2024-05-27T09:35:21.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:21.785+0000 7f3a7da00640 1 -- 172.21.15.44:0/302507603 wait complete. 2024-05-27T09:35:21.813 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-27T09:35:20.068606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188628,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1453,"num_read_kb":859,"num_write":422,"num_write_kb":3594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":827,"ondisk_log_size":827,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384088,"kb_used_data":14112,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557736,"statfs":{"total":1151940427776,"available":1151547121664,"internally_reserved":0,"allocated":14450688,"data_stored":8819754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.965797"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840251+0000","last_change":"2024-05-27T09:31:24.899109+0000","last_active":"2024-05-27T09:35:14.840251+0000","last_peered":"2024-05-27T09:35:14.840251+0000","last_clean":"2024-05-27T09:35:14.840251+0000","last_became_active":"2024-05-27T09:31:24.898783+0000","last_became_peered":"2024-05-27T09:31:24.898783+0000","last_unstale":"2024-05-27T09:35:14.840251+0000","last_undegraded":"2024-05-27T09:35:14.840251+0000","last_fullsized":"2024-05-27T09:35:14.840251+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:01:12.220389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042262+0000","last_change":"2024-05-27T09:31:24.898382+0000","last_active":"2024-05-27T09:35:14.042262+0000","last_peered":"2024-05-27T09:35:14.042262+0000","last_clean":"2024-05-27T09:35:14.042262+0000","last_became_active":"2024-05-27T09:31:24.897975+0000","last_became_peered":"2024-05-27T09:31:24.897975+0000","last_unstale":"2024-05-27T09:35:14.042262+0000","last_undegraded":"2024-05-27T09:35:14.042262+0000","last_fullsized":"2024-05-27T09:35:14.042262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:38.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041822+0000","last_change":"2024-05-27T09:31:24.896122+0000","last_active":"2024-05-27T09:35:14.041822+0000","last_peered":"2024-05-27T09:35:14.041822+0000","last_clean":"2024-05-27T09:35:14.041822+0000","last_became_active":"2024-05-27T09:31:24.890867+0000","last_became_peered":"2024-05-27T09:31:24.890867+0000","last_unstale":"2024-05-27T09:35:14.041822+0000","last_undegraded":"2024-05-27T09:35:14.041822+0000","last_fullsized":"2024-05-27T09:35:14.041822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:09.982177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046136+0000","last_change":"2024-05-27T09:31:57.868900+0000","last_active":"2024-05-27T09:35:14.046136+0000","last_peered":"2024-05-27T09:35:14.046136+0000","last_clean":"2024-05-27T09:35:14.046136+0000","last_became_active":"2024-05-27T09:31:57.868418+0000","last_became_peered":"2024-05-27T09:31:57.868418+0000","last_unstale":"2024-05-27T09:35:14.046136+0000","last_undegraded":"2024-05-27T09:35:14.046136+0000","last_fullsized":"2024-05-27T09:35:14.046136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:08.966704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169866+0000","last_change":"2024-05-27T09:31:24.910283+0000","last_active":"2024-05-27T09:35:14.169866+0000","last_peered":"2024-05-27T09:35:14.169866+0000","last_clean":"2024-05-27T09:35:14.169866+0000","last_became_active":"2024-05-27T09:31:24.909882+0000","last_became_peered":"2024-05-27T09:31:24.909882+0000","last_unstale":"2024-05-27T09:35:14.169866+0000","last_undegraded":"2024-05-27T09:35:14.169866+0000","last_fullsized":"2024-05-27T09:35:14.169866+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:43.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126676+0000","last_change":"2024-05-27T09:31:24.900934+0000","last_active":"2024-05-27T09:35:14.126676+0000","last_peered":"2024-05-27T09:35:14.126676+0000","last_clean":"2024-05-27T09:35:14.126676+0000","last_became_active":"2024-05-27T09:31:24.900534+0000","last_became_peered":"2024-05-27T09:31:24.900534+0000","last_unstale":"2024-05-27T09:35:14.126676+0000","last_undegraded":"2024-05-27T09:35:14.126676+0000","last_fullsized":"2024-05-27T09:35:14.126676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:53.318327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353192+0000","last_change":"2024-05-27T09:31:24.907080+0000","last_active":"2024-05-27T09:35:14.353192+0000","last_peered":"2024-05-27T09:35:14.353192+0000","last_clean":"2024-05-27T09:35:14.353192+0000","last_became_active":"2024-05-27T09:31:24.906715+0000","last_became_peered":"2024-05-27T09:31:24.906715+0000","last_unstale":"2024-05-27T09:35:14.353192+0000","last_undegraded":"2024-05-27T09:35:14.353192+0000","last_fullsized":"2024-05-27T09:35:14.353192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:36.214659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.044350+0000","last_change":"2024-05-27T09:31:58.200608+0000","last_active":"2024-05-27T09:35:14.044350+0000","last_peered":"2024-05-27T09:35:14.044350+0000","last_clean":"2024-05-27T09:35:14.044350+0000","last_became_active":"2024-05-27T09:31:58.199363+0000","last_became_peered":"2024-05-27T09:31:58.199363+0000","last_unstale":"2024-05-27T09:35:14.044350+0000","last_undegraded":"2024-05-27T09:35:14.044350+0000","last_fullsized":"2024-05-27T09:35:14.044350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:30:46.495930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373508+0000","last_change":"2024-05-27T09:31:24.892630+0000","last_active":"2024-05-27T09:35:14.373508+0000","last_peered":"2024-05-27T09:35:14.373508+0000","last_clean":"2024-05-27T09:35:14.373508+0000","last_became_active":"2024-05-27T09:31:24.892176+0000","last_became_peered":"2024-05-27T09:31:24.892176+0000","last_unstale":"2024-05-27T09:35:14.373508+0000","last_undegraded":"2024-05-27T09:35:14.373508+0000","last_fullsized":"2024-05-27T09:35:14.373508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:49:20.585601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.127155+0000","last_change":"2024-05-27T09:31:24.895281+0000","last_active":"2024-05-27T09:35:14.127155+0000","last_peered":"2024-05-27T09:35:14.127155+0000","last_clean":"2024-05-27T09:35:14.127155+0000","last_became_active":"2024-05-27T09:31:24.894886+0000","last_became_peered":"2024-05-27T09:31:24.894886+0000","last_unstale":"2024-05-27T09:35:14.127155+0000","last_undegraded":"2024-05-27T09:35:14.127155+0000","last_fullsized":"2024-05-27T09:35:14.127155+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:41.494434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353559+0000","last_change":"2024-05-27T09:31:24.900167+0000","last_active":"2024-05-27T09:35:14.353559+0000","last_peered":"2024-05-27T09:35:14.353559+0000","last_clean":"2024-05-27T09:35:14.353559+0000","last_became_active":"2024-05-27T09:31:24.899750+0000","last_became_peered":"2024-05-27T09:31:24.899750+0000","last_unstale":"2024-05-27T09:35:14.353559+0000","last_undegraded":"2024-05-27T09:35:14.353559+0000","last_fullsized":"2024-05-27T09:35:14.353559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:30.602246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.045840+0000","last_change":"2024-05-27T09:31:57.868818+0000","last_active":"2024-05-27T09:35:14.045840+0000","last_peered":"2024-05-27T09:35:14.045840+0000","last_clean":"2024-05-27T09:35:14.045840+0000","last_became_active":"2024-05-27T09:31:57.868322+0000","last_became_peered":"2024-05-27T09:31:57.868322+0000","last_unstale":"2024-05-27T09:35:14.045840+0000","last_undegraded":"2024-05-27T09:35:14.045840+0000","last_fullsized":"2024-05-27T09:35:14.045840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:35.304850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242036+0000","last_change":"2024-05-27T09:31:24.902537+0000","last_active":"2024-05-27T09:35:14.242036+0000","last_peered":"2024-05-27T09:35:14.242036+0000","last_clean":"2024-05-27T09:35:14.242036+0000","last_became_active":"2024-05-27T09:31:24.902132+0000","last_became_peered":"2024-05-27T09:31:24.902132+0000","last_unstale":"2024-05-27T09:35:14.242036+0000","last_undegraded":"2024-05-27T09:35:14.242036+0000","last_fullsized":"2024-05-27T09:35:14.242036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:20:50.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041926+0000","last_change":"2024-05-27T09:31:24.899182+0000","last_active":"2024-05-27T09:35:14.041926+0000","last_peered":"2024-05-27T09:35:14.041926+0000","last_clean":"2024-05-27T09:35:14.041926+0000","last_became_active":"2024-05-27T09:31:24.898749+0000","last_became_peered":"2024-05-27T09:31:24.898749+0000","last_unstale":"2024-05-27T09:35:14.041926+0000","last_undegraded":"2024-05-27T09:35:14.041926+0000","last_fullsized":"2024-05-27T09:35:14.041926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:44:44.844357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041660+0000","last_change":"2024-05-27T09:31:24.901115+0000","last_active":"2024-05-27T09:35:14.041660+0000","last_peered":"2024-05-27T09:35:14.041660+0000","last_clean":"2024-05-27T09:35:14.041660+0000","last_became_active":"2024-05-27T09:31:24.897011+0000","last_became_peered":"2024-05-27T09:31:24.897011+0000","last_unstale":"2024-05-27T09:35:14.041660+0000","last_undegraded":"2024-05-27T09:35:14.041660+0000","last_fullsized":"2024-05-27T09:35:14.041660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:01.130407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372690+0000","last_change":"2024-05-27T09:31:24.903063+0000","last_active":"2024-05-27T09:35:14.372690+0000","last_peered":"2024-05-27T09:35:14.372690+0000","last_clean":"2024-05-27T09:35:14.372690+0000","last_became_active":"2024-05-27T09:31:24.899766+0000","last_became_peered":"2024-05-27T09:31:24.899766+0000","last_unstale":"2024-05-27T09:35:14.372690+0000","last_undegraded":"2024-05-27T09:35:14.372690+0000","last_fullsized":"2024-05-27T09:35:14.372690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:45.984226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840388+0000","last_change":"2024-05-27T09:31:24.915922+0000","last_active":"2024-05-27T09:35:14.840388+0000","last_peered":"2024-05-27T09:35:14.840388+0000","last_clean":"2024-05-27T09:35:14.840388+0000","last_became_active":"2024-05-27T09:31:24.915623+0000","last_became_peered":"2024-05-27T09:31:24.915623+0000","last_unstale":"2024-05-27T09:35:14.840388+0000","last_undegraded":"2024-05-27T09:35:14.840388+0000","last_fullsized":"2024-05-27T09:35:14.840388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:02:01.006704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046094+0000","last_change":"2024-05-27T09:31:24.908397+0000","last_active":"2024-05-27T09:35:14.046094+0000","last_peered":"2024-05-27T09:35:14.046094+0000","last_clean":"2024-05-27T09:35:14.046094+0000","last_became_active":"2024-05-27T09:31:24.903644+0000","last_became_peered":"2024-05-27T09:31:24.903644+0000","last_unstale":"2024-05-27T09:35:14.046094+0000","last_undegraded":"2024-05-27T09:35:14.046094+0000","last_fullsized":"2024-05-27T09:35:14.046094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:09.281429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373078+0000","last_change":"2024-05-27T09:31:24.912706+0000","last_active":"2024-05-27T09:35:14.373078+0000","last_peered":"2024-05-27T09:35:14.373078+0000","last_clean":"2024-05-27T09:35:14.373078+0000","last_became_active":"2024-05-27T09:31:24.912357+0000","last_became_peered":"2024-05-27T09:31:24.912357+0000","last_unstale":"2024-05-27T09:35:14.373078+0000","last_undegraded":"2024-05-27T09:35:14.373078+0000","last_fullsized":"2024-05-27T09:35:14.373078+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:23.334041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.045022+0000","last_change":"2024-05-27T09:31:20.881652+0000","last_active":"2024-05-27T09:35:14.045022+0000","last_peered":"2024-05-27T09:35:14.045022+0000","last_clean":"2024-05-27T09:35:14.045022+0000","last_became_active":"2024-05-27T09:31:20.880757+0000","last_became_peered":"2024-05-27T09:31:20.880757+0000","last_unstale":"2024-05-27T09:35:14.045022+0000","last_undegraded":"2024-05-27T09:35:14.045022+0000","last_fullsized":"2024-05-27T09:35:14.045022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:01.433755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171024+0000","last_change":"2024-05-27T09:31:24.908010+0000","last_active":"2024-05-27T09:35:14.171024+0000","last_peered":"2024-05-27T09:35:14.171024+0000","last_clean":"2024-05-27T09:35:14.171024+0000","last_became_active":"2024-05-27T09:31:24.907638+0000","last_became_peered":"2024-05-27T09:31:24.907638+0000","last_unstale":"2024-05-27T09:35:14.171024+0000","last_undegraded":"2024-05-27T09:35:14.171024+0000","last_fullsized":"2024-05-27T09:35:14.171024+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:02:49.207709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171395+0000","last_change":"2024-05-27T09:31:20.860556+0000","last_active":"2024-05-27T09:35:14.171395+0000","last_peered":"2024-05-27T09:35:14.171395+0000","last_clean":"2024-05-27T09:35:14.171395+0000","last_became_active":"2024-05-27T09:31:20.860172+0000","last_became_peered":"2024-05-27T09:31:20.860172+0000","last_unstale":"2024-05-27T09:35:14.171395+0000","last_undegraded":"2024-05-27T09:35:14.171395+0000","last_fullsized":"2024-05-27T09:35:14.171395+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:08:21.545681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042957+0000","last_change":"2024-05-27T09:31:24.910111+0000","last_active":"2024-05-27T09:35:14.042957+0000","last_peered":"2024-05-27T09:35:14.042957+0000","last_clean":"2024-05-27T09:35:14.042957+0000","last_became_active":"2024-05-27T09:31:24.908985+0000","last_became_peered":"2024-05-27T09:31:24.908985+0000","last_unstale":"2024-05-27T09:35:14.042957+0000","last_undegraded":"2024-05-27T09:35:14.042957+0000","last_fullsized":"2024-05-27T09:35:14.042957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:53:29.645792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170222+0000","last_change":"2024-05-27T09:31:20.879993+0000","last_active":"2024-05-27T09:35:14.170222+0000","last_peered":"2024-05-27T09:35:14.170222+0000","last_clean":"2024-05-27T09:35:14.170222+0000","last_became_active":"2024-05-27T09:31:20.878802+0000","last_became_peered":"2024-05-27T09:31:20.878802+0000","last_unstale":"2024-05-27T09:35:14.170222+0000","last_undegraded":"2024-05-27T09:35:14.170222+0000","last_fullsized":"2024-05-27T09:35:14.170222+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:15.555612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372970+0000","last_change":"2024-05-27T09:31:24.903533+0000","last_active":"2024-05-27T09:35:14.372970+0000","last_peered":"2024-05-27T09:35:14.372970+0000","last_clean":"2024-05-27T09:35:14.372970+0000","last_became_active":"2024-05-27T09:31:24.903166+0000","last_became_peered":"2024-05-27T09:31:24.903166+0000","last_unstale":"2024-05-27T09:35:14.372970+0000","last_undegraded":"2024-05-27T09:35:14.372970+0000","last_fullsized":"2024-05-27T09:35:14.372970+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:52.539470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839887+0000","last_change":"2024-05-27T09:31:57.878426+0000","last_active":"2024-05-27T09:35:14.839887+0000","last_peered":"2024-05-27T09:35:14.839887+0000","last_clean":"2024-05-27T09:35:14.839887+0000","last_became_active":"2024-05-27T09:31:57.869146+0000","last_became_peered":"2024-05-27T09:31:57.869146+0000","last_unstale":"2024-05-27T09:35:14.839887+0000","last_undegraded":"2024-05-27T09:35:14.839887+0000","last_fullsized":"2024-05-27T09:35:14.839887+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:34:51.926092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129913+0000","last_change":"2024-05-27T09:31:24.897426+0000","last_active":"2024-05-27T09:35:14.129913+0000","last_peered":"2024-05-27T09:35:14.129913+0000","last_clean":"2024-05-27T09:35:14.129913+0000","last_became_active":"2024-05-27T09:31:24.897080+0000","last_became_peered":"2024-05-27T09:31:24.897080+0000","last_unstale":"2024-05-27T09:35:14.129913+0000","last_undegraded":"2024-05-27T09:35:14.129913+0000","last_fullsized":"2024-05-27T09:35:14.129913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:15:42.428391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'8","reported_seq":249,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.130006+0000","last_change":"2024-05-27T09:31:20.846681+0000","last_active":"2024-05-27T09:35:14.130006+0000","last_peered":"2024-05-27T09:35:14.130006+0000","last_clean":"2024-05-27T09:35:14.130006+0000","last_became_active":"2024-05-27T09:31:20.846302+0000","last_became_peered":"2024-05-27T09:31:20.846302+0000","last_unstale":"2024-05-27T09:35:14.130006+0000","last_undegraded":"2024-05-27T09:35:14.130006+0000","last_fullsized":"2024-05-27T09:35:14.130006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:19.609612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169711+0000","last_change":"2024-05-27T09:31:24.907209+0000","last_active":"2024-05-27T09:35:14.169711+0000","last_peered":"2024-05-27T09:35:14.169711+0000","last_clean":"2024-05-27T09:35:14.169711+0000","last_became_active":"2024-05-27T09:31:24.906752+0000","last_became_peered":"2024-05-27T09:31:24.906752+0000","last_unstale":"2024-05-27T09:35:14.169711+0000","last_undegraded":"2024-05-27T09:35:14.169711+0000","last_fullsized":"2024-05-27T09:35:14.169711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:53.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126569+0000","last_change":"2024-05-27T09:31:20.879778+0000","last_active":"2024-05-27T09:35:14.126569+0000","last_peered":"2024-05-27T09:35:14.126569+0000","last_clean":"2024-05-27T09:35:14.126569+0000","last_became_active":"2024-05-27T09:31:20.879259+0000","last_became_peered":"2024-05-27T09:31:20.879259+0000","last_unstale":"2024-05-27T09:35:14.126569+0000","last_undegraded":"2024-05-27T09:35:14.126569+0000","last_fullsized":"2024-05-27T09:35:14.126569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:20:07.972653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129702+0000","last_change":"2024-05-27T09:31:58.195516+0000","last_active":"2024-05-27T09:35:14.129702+0000","last_peered":"2024-05-27T09:35:14.129702+0000","last_clean":"2024-05-27T09:35:14.129702+0000","last_became_active":"2024-05-27T09:31:58.195118+0000","last_became_peered":"2024-05-27T09:31:58.195118+0000","last_unstale":"2024-05-27T09:35:14.129702+0000","last_undegraded":"2024-05-27T09:35:14.129702+0000","last_fullsized":"2024-05-27T09:35:14.129702+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:11:23.547605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241846+0000","last_change":"2024-05-27T09:31:20.863152+0000","last_active":"2024-05-27T09:35:14.241846+0000","last_peered":"2024-05-27T09:35:14.241846+0000","last_clean":"2024-05-27T09:35:14.241846+0000","last_became_active":"2024-05-27T09:31:20.862708+0000","last_became_peered":"2024-05-27T09:31:20.862708+0000","last_unstale":"2024-05-27T09:35:14.241846+0000","last_undegraded":"2024-05-27T09:35:14.241846+0000","last_fullsized":"2024-05-27T09:35:14.241846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:21.645773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170544+0000","last_change":"2024-05-27T09:31:24.893988+0000","last_active":"2024-05-27T09:35:14.170544+0000","last_peered":"2024-05-27T09:35:14.170544+0000","last_clean":"2024-05-27T09:35:14.170544+0000","last_became_active":"2024-05-27T09:31:24.893504+0000","last_became_peered":"2024-05-27T09:31:24.893504+0000","last_unstale":"2024-05-27T09:35:14.170544+0000","last_undegraded":"2024-05-27T09:35:14.170544+0000","last_fullsized":"2024-05-27T09:35:14.170544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:47.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129578+0000","last_change":"2024-05-27T09:31:20.845555+0000","last_active":"2024-05-27T09:35:14.129578+0000","last_peered":"2024-05-27T09:35:14.129578+0000","last_clean":"2024-05-27T09:35:14.129578+0000","last_became_active":"2024-05-27T09:31:20.845139+0000","last_became_peered":"2024-05-27T09:31:20.845139+0000","last_unstale":"2024-05-27T09:35:14.129578+0000","last_undegraded":"2024-05-27T09:35:14.129578+0000","last_fullsized":"2024-05-27T09:35:14.129578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:25.917845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.049161+0000","last_change":"2024-05-27T09:31:24.916142+0000","last_active":"2024-05-27T09:35:14.049161+0000","last_peered":"2024-05-27T09:35:14.049161+0000","last_clean":"2024-05-27T09:35:14.049161+0000","last_became_active":"2024-05-27T09:31:24.915709+0000","last_became_peered":"2024-05-27T09:31:24.915709+0000","last_unstale":"2024-05-27T09:35:14.049161+0000","last_undegraded":"2024-05-27T09:35:14.049161+0000","last_fullsized":"2024-05-27T09:35:14.049161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:38.029648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'5","reported_seq":242,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.127284+0000","last_change":"2024-05-27T09:31:20.859240+0000","last_active":"2024-05-27T09:35:14.127284+0000","last_peered":"2024-05-27T09:35:14.127284+0000","last_clean":"2024-05-27T09:35:14.127284+0000","last_became_active":"2024-05-27T09:31:20.858798+0000","last_became_peered":"2024-05-27T09:31:20.858798+0000","last_unstale":"2024-05-27T09:35:14.127284+0000","last_undegraded":"2024-05-27T09:35:14.127284+0000","last_fullsized":"2024-05-27T09:35:14.127284+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:42.280165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373425+0000","last_change":"2024-05-27T09:31:24.908818+0000","last_active":"2024-05-27T09:35:14.373425+0000","last_peered":"2024-05-27T09:35:14.373425+0000","last_clean":"2024-05-27T09:35:14.373425+0000","last_became_active":"2024-05-27T09:31:24.908410+0000","last_became_peered":"2024-05-27T09:31:24.908410+0000","last_unstale":"2024-05-27T09:35:14.373425+0000","last_undegraded":"2024-05-27T09:35:14.373425+0000","last_fullsized":"2024-05-27T09:35:14.373425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:05:40.326232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353642+0000","last_change":"2024-05-27T09:31:20.848632+0000","last_active":"2024-05-27T09:35:14.353642+0000","last_peered":"2024-05-27T09:35:14.353642+0000","last_clean":"2024-05-27T09:35:14.353642+0000","last_became_active":"2024-05-27T09:31:20.847558+0000","last_became_peered":"2024-05-27T09:31:20.847558+0000","last_unstale":"2024-05-27T09:35:14.353642+0000","last_undegraded":"2024-05-27T09:35:14.353642+0000","last_fullsized":"2024-05-27T09:35:14.353642+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:53.930565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043056+0000","last_change":"2024-05-27T09:31:24.901513+0000","last_active":"2024-05-27T09:35:14.043056+0000","last_peered":"2024-05-27T09:35:14.043056+0000","last_clean":"2024-05-27T09:35:14.043056+0000","last_became_active":"2024-05-27T09:31:24.895924+0000","last_became_peered":"2024-05-27T09:31:24.895924+0000","last_unstale":"2024-05-27T09:35:14.043056+0000","last_undegraded":"2024-05-27T09:35:14.043056+0000","last_fullsized":"2024-05-27T09:35:14.043056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:27:39.939098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.130061+0000","last_change":"2024-05-27T09:31:20.855195+0000","last_active":"2024-05-27T09:35:14.130061+0000","last_peered":"2024-05-27T09:35:14.130061+0000","last_clean":"2024-05-27T09:35:14.130061+0000","last_became_active":"2024-05-27T09:31:20.854910+0000","last_became_peered":"2024-05-27T09:31:20.854910+0000","last_unstale":"2024-05-27T09:35:14.130061+0000","last_undegraded":"2024-05-27T09:35:14.130061+0000","last_fullsized":"2024-05-27T09:35:14.130061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:02.795103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373077+0000","last_change":"2024-05-27T09:31:24.895887+0000","last_active":"2024-05-27T09:35:14.373077+0000","last_peered":"2024-05-27T09:35:14.373077+0000","last_clean":"2024-05-27T09:35:14.373077+0000","last_became_active":"2024-05-27T09:31:24.894622+0000","last_became_peered":"2024-05-27T09:31:24.894622+0000","last_unstale":"2024-05-27T09:35:14.373077+0000","last_undegraded":"2024-05-27T09:35:14.373077+0000","last_fullsized":"2024-05-27T09:35:14.373077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:32.833946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'36","reported_seq":491,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372952+0000","last_change":"2024-05-27T09:31:20.846063+0000","last_active":"2024-05-27T09:35:14.372952+0000","last_peered":"2024-05-27T09:35:14.372952+0000","last_clean":"2024-05-27T09:35:14.372952+0000","last_became_active":"2024-05-27T09:31:20.845401+0000","last_became_peered":"2024-05-27T09:31:20.845401+0000","last_unstale":"2024-05-27T09:35:14.372952+0000","last_undegraded":"2024-05-27T09:35:14.372952+0000","last_fullsized":"2024-05-27T09:35:14.372952+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:21:38.460183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67057,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":426,"num_read_kb":273,"num_write":37,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170004+0000","last_change":"2024-05-27T09:31:24.892748+0000","last_active":"2024-05-27T09:35:14.170004+0000","last_peered":"2024-05-27T09:35:14.170004+0000","last_clean":"2024-05-27T09:35:14.170004+0000","last_became_active":"2024-05-27T09:31:24.892394+0000","last_became_peered":"2024-05-27T09:31:24.892394+0000","last_unstale":"2024-05-27T09:35:14.170004+0000","last_undegraded":"2024-05-27T09:35:14.170004+0000","last_fullsized":"2024-05-27T09:35:14.170004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:39:31.381072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372830+0000","last_change":"2024-05-27T09:31:20.860789+0000","last_active":"2024-05-27T09:35:14.372830+0000","last_peered":"2024-05-27T09:35:14.372830+0000","last_clean":"2024-05-27T09:35:14.372830+0000","last_became_active":"2024-05-27T09:31:20.860369+0000","last_became_peered":"2024-05-27T09:31:20.860369+0000","last_unstale":"2024-05-27T09:35:14.372830+0000","last_undegraded":"2024-05-27T09:35:14.372830+0000","last_fullsized":"2024-05-27T09:35:14.372830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:05:35.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241584+0000","last_change":"2024-05-27T09:31:24.918239+0000","last_active":"2024-05-27T09:35:14.241584+0000","last_peered":"2024-05-27T09:35:14.241584+0000","last_clean":"2024-05-27T09:35:14.241584+0000","last_became_active":"2024-05-27T09:31:24.917750+0000","last_became_peered":"2024-05-27T09:31:24.917750+0000","last_unstale":"2024-05-27T09:35:14.241584+0000","last_undegraded":"2024-05-27T09:35:14.241584+0000","last_fullsized":"2024-05-27T09:35:14.241584+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:07.162994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129596+0000","last_change":"2024-05-27T09:31:20.865308+0000","last_active":"2024-05-27T09:35:14.129596+0000","last_peered":"2024-05-27T09:35:14.129596+0000","last_clean":"2024-05-27T09:35:14.129596+0000","last_became_active":"2024-05-27T09:31:20.864920+0000","last_became_peered":"2024-05-27T09:31:20.864920+0000","last_unstale":"2024-05-27T09:35:14.129596+0000","last_undegraded":"2024-05-27T09:35:14.129596+0000","last_fullsized":"2024-05-27T09:35:14.129596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:36.773322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.902533+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.902136+0000","last_became_peered":"2024-05-27T09:31:24.902136+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:49:32.203577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'8","reported_seq":250,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.375584+0000","last_change":"2024-05-27T09:31:20.846060+0000","last_active":"2024-05-27T09:35:14.375584+0000","last_peered":"2024-05-27T09:35:14.375584+0000","last_clean":"2024-05-27T09:35:14.375584+0000","last_became_active":"2024-05-27T09:31:20.845632+0000","last_became_peered":"2024-05-27T09:31:20.845632+0000","last_unstale":"2024-05-27T09:35:14.375584+0000","last_undegraded":"2024-05-27T09:35:14.375584+0000","last_fullsized":"2024-05-27T09:35:14.375584+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:55.166579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2208,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":214,"num_read_kb":6,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353158+0000","last_change":"2024-05-27T09:31:24.895866+0000","last_active":"2024-05-27T09:35:14.353158+0000","last_peered":"2024-05-27T09:35:14.353158+0000","last_clean":"2024-05-27T09:35:14.353158+0000","last_became_active":"2024-05-27T09:31:24.895413+0000","last_became_peered":"2024-05-27T09:31:24.895413+0000","last_unstale":"2024-05-27T09:35:14.353158+0000","last_undegraded":"2024-05-27T09:35:14.353158+0000","last_fullsized":"2024-05-27T09:35:14.353158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:01:30.075710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":45,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169708+0000","last_change":"2024-05-27T09:31:20.879086+0000","last_active":"2024-05-27T09:35:14.169708+0000","last_peered":"2024-05-27T09:35:14.169708+0000","last_clean":"2024-05-27T09:35:14.169708+0000","last_became_active":"2024-05-27T09:31:20.878331+0000","last_became_peered":"2024-05-27T09:31:20.878331+0000","last_unstale":"2024-05-27T09:35:14.169708+0000","last_undegraded":"2024-05-27T09:35:14.169708+0000","last_fullsized":"2024-05-27T09:35:14.169708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:31.169812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129641+0000","last_change":"2024-05-27T09:31:24.895069+0000","last_active":"2024-05-27T09:35:14.129641+0000","last_peered":"2024-05-27T09:35:14.129641+0000","last_clean":"2024-05-27T09:35:14.129641+0000","last_became_active":"2024-05-27T09:31:24.894626+0000","last_became_peered":"2024-05-27T09:31:24.894626+0000","last_unstale":"2024-05-27T09:35:14.129641+0000","last_undegraded":"2024-05-27T09:35:14.129641+0000","last_fullsized":"2024-05-27T09:35:14.129641+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:21:22.725268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170168+0000","last_change":"2024-05-27T09:31:20.877416+0000","last_active":"2024-05-27T09:35:14.170168+0000","last_peered":"2024-05-27T09:35:14.170168+0000","last_clean":"2024-05-27T09:35:14.170168+0000","last_became_active":"2024-05-27T09:31:20.876093+0000","last_became_peered":"2024-05-27T09:31:20.876093+0000","last_unstale":"2024-05-27T09:35:14.170168+0000","last_undegraded":"2024-05-27T09:35:14.170168+0000","last_fullsized":"2024-05-27T09:35:14.170168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:34.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840141+0000","last_change":"2024-05-27T09:31:24.898430+0000","last_active":"2024-05-27T09:35:14.840141+0000","last_peered":"2024-05-27T09:35:14.840141+0000","last_clean":"2024-05-27T09:35:14.840141+0000","last_became_active":"2024-05-27T09:31:24.897643+0000","last_became_peered":"2024-05-27T09:31:24.897643+0000","last_unstale":"2024-05-27T09:35:14.840141+0000","last_undegraded":"2024-05-27T09:35:14.840141+0000","last_fullsized":"2024-05-27T09:35:14.840141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:44.485561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046011+0000","last_change":"2024-05-27T09:31:20.858253+0000","last_active":"2024-05-27T09:35:14.046011+0000","last_peered":"2024-05-27T09:35:14.046011+0000","last_clean":"2024-05-27T09:35:14.046011+0000","last_became_active":"2024-05-27T09:31:20.857736+0000","last_became_peered":"2024-05-27T09:31:20.857736+0000","last_unstale":"2024-05-27T09:35:14.046011+0000","last_undegraded":"2024-05-27T09:35:14.046011+0000","last_fullsized":"2024-05-27T09:35:14.046011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:53:48.301763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126918+0000","last_change":"2024-05-27T09:31:24.912788+0000","last_active":"2024-05-27T09:35:14.126918+0000","last_peered":"2024-05-27T09:35:14.126918+0000","last_clean":"2024-05-27T09:35:14.126918+0000","last_became_active":"2024-05-27T09:31:24.912366+0000","last_became_peered":"2024-05-27T09:31:24.912366+0000","last_unstale":"2024-05-27T09:35:14.126918+0000","last_undegraded":"2024-05-27T09:35:14.126918+0000","last_fullsized":"2024-05-27T09:35:14.126918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:55:03.397578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353364+0000","last_change":"2024-05-27T09:31:20.862188+0000","last_active":"2024-05-27T09:35:14.353364+0000","last_peered":"2024-05-27T09:35:14.353364+0000","last_clean":"2024-05-27T09:35:14.353364+0000","last_became_active":"2024-05-27T09:31:20.861791+0000","last_became_peered":"2024-05-27T09:31:20.861791+0000","last_unstale":"2024-05-27T09:35:14.353364+0000","last_undegraded":"2024-05-27T09:35:14.353364+0000","last_fullsized":"2024-05-27T09:35:14.353364+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:16:09.127223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126652+0000","last_change":"2024-05-27T09:31:24.898460+0000","last_active":"2024-05-27T09:35:14.126652+0000","last_peered":"2024-05-27T09:35:14.126652+0000","last_clean":"2024-05-27T09:35:14.126652+0000","last_became_active":"2024-05-27T09:31:24.896787+0000","last_became_peered":"2024-05-27T09:31:24.896787+0000","last_unstale":"2024-05-27T09:35:14.126652+0000","last_undegraded":"2024-05-27T09:35:14.126652+0000","last_fullsized":"2024-05-27T09:35:14.126652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:10.199227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129938+0000","last_change":"2024-05-27T09:31:24.896323+0000","last_active":"2024-05-27T09:35:14.129938+0000","last_peered":"2024-05-27T09:35:14.129938+0000","last_clean":"2024-05-27T09:35:14.129938+0000","last_became_active":"2024-05-27T09:31:24.895919+0000","last_became_peered":"2024-05-27T09:31:24.895919+0000","last_unstale":"2024-05-27T09:35:14.129938+0000","last_undegraded":"2024-05-27T09:35:14.129938+0000","last_fullsized":"2024-05-27T09:35:14.129938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:04.883166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":243,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241731+0000","last_change":"2024-05-27T09:31:20.862126+0000","last_active":"2024-05-27T09:35:14.241731+0000","last_peered":"2024-05-27T09:35:14.241731+0000","last_clean":"2024-05-27T09:35:14.241731+0000","last_became_active":"2024-05-27T09:31:20.858964+0000","last_became_peered":"2024-05-27T09:31:20.858964+0000","last_unstale":"2024-05-27T09:35:14.241731+0000","last_undegraded":"2024-05-27T09:35:14.241731+0000","last_fullsized":"2024-05-27T09:35:14.241731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:51:26.838559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171127+0000","last_change":"2024-05-27T09:31:24.905549+0000","last_active":"2024-05-27T09:35:14.171127+0000","last_peered":"2024-05-27T09:35:14.171127+0000","last_clean":"2024-05-27T09:35:14.171127+0000","last_became_active":"2024-05-27T09:31:24.905212+0000","last_became_peered":"2024-05-27T09:31:24.905212+0000","last_unstale":"2024-05-27T09:35:14.171127+0000","last_undegraded":"2024-05-27T09:35:14.171127+0000","last_fullsized":"2024-05-27T09:35:14.171127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:18.267807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043259+0000","last_change":"2024-05-27T09:31:20.848527+0000","last_active":"2024-05-27T09:35:14.043259+0000","last_peered":"2024-05-27T09:35:14.043259+0000","last_clean":"2024-05-27T09:35:14.043259+0000","last_became_active":"2024-05-27T09:31:20.847779+0000","last_became_peered":"2024-05-27T09:31:20.847779+0000","last_unstale":"2024-05-27T09:35:14.043259+0000","last_undegraded":"2024-05-27T09:35:14.043259+0000","last_fullsized":"2024-05-27T09:35:14.043259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:15:01.131306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":105,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353135+0000","last_change":"2024-05-27T09:29:46.120380+0000","last_active":"2024-05-27T09:35:14.353135+0000","last_peered":"2024-05-27T09:35:14.353135+0000","last_clean":"2024-05-27T09:35:14.353135+0000","last_became_active":"2024-05-27T09:29:46.113658+0000","last_became_peered":"2024-05-27T09:29:46.113658+0000","last_unstale":"2024-05-27T09:35:14.353135+0000","last_undegraded":"2024-05-27T09:35:14.353135+0000","last_fullsized":"2024-05-27T09:35:14.353135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_clean_scrub_stamp":"2024-05-27T09:25:47.244521+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:28:40.424824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241501+0000","last_change":"2024-05-27T09:31:24.900736+0000","last_active":"2024-05-27T09:35:14.241501+0000","last_peered":"2024-05-27T09:35:14.241501+0000","last_clean":"2024-05-27T09:35:14.241501+0000","last_became_active":"2024-05-27T09:31:24.900354+0000","last_became_peered":"2024-05-27T09:31:24.900354+0000","last_unstale":"2024-05-27T09:35:14.241501+0000","last_undegraded":"2024-05-27T09:35:14.241501+0000","last_fullsized":"2024-05-27T09:35:14.241501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:39:33.171977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839843+0000","last_change":"2024-05-27T09:31:57.873603+0000","last_active":"2024-05-27T09:35:14.839843+0000","last_peered":"2024-05-27T09:35:14.839843+0000","last_clean":"2024-05-27T09:35:14.839843+0000","last_became_active":"2024-05-27T09:31:57.868616+0000","last_became_peered":"2024-05-27T09:31:57.868616+0000","last_unstale":"2024-05-27T09:35:14.839843+0000","last_undegraded":"2024-05-27T09:35:14.839843+0000","last_fullsized":"2024-05-27T09:35:14.839843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:02:20.112667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170091+0000","last_change":"2024-05-27T09:31:20.878557+0000","last_active":"2024-05-27T09:35:14.170091+0000","last_peered":"2024-05-27T09:35:14.170091+0000","last_clean":"2024-05-27T09:35:14.170091+0000","last_became_active":"2024-05-27T09:31:20.877969+0000","last_became_peered":"2024-05-27T09:31:20.877969+0000","last_unstale":"2024-05-27T09:35:14.170091+0000","last_undegraded":"2024-05-27T09:35:14.170091+0000","last_fullsized":"2024-05-27T09:35:14.170091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:46.859645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242211+0000","last_change":"2024-05-27T09:31:24.907906+0000","last_active":"2024-05-27T09:35:14.242211+0000","last_peered":"2024-05-27T09:35:14.242211+0000","last_clean":"2024-05-27T09:35:14.242211+0000","last_became_active":"2024-05-27T09:31:24.907473+0000","last_became_peered":"2024-05-27T09:31:24.907473+0000","last_unstale":"2024-05-27T09:35:14.242211+0000","last_undegraded":"2024-05-27T09:35:14.242211+0000","last_fullsized":"2024-05-27T09:35:14.242211+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:11:38.536919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'48","reported_seq":100,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.321061+0000","last_change":"2024-05-27T09:31:20.881250+0000","last_active":"2024-05-27T09:35:14.321061+0000","last_peered":"2024-05-27T09:35:14.321061+0000","last_clean":"2024-05-27T09:35:14.321061+0000","last_became_active":"2024-05-27T09:31:20.880550+0000","last_became_peered":"2024-05-27T09:31:20.880550+0000","last_unstale":"2024-05-27T09:35:14.321061+0000","last_undegraded":"2024-05-27T09:35:14.321061+0000","last_fullsized":"2024-05-27T09:35:14.321061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:14.783260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":86,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171395+0000","last_change":"2024-05-27T09:31:24.894554+0000","last_active":"2024-05-27T09:35:14.171395+0000","last_peered":"2024-05-27T09:35:14.171395+0000","last_clean":"2024-05-27T09:35:14.171395+0000","last_became_active":"2024-05-27T09:31:24.893800+0000","last_became_peered":"2024-05-27T09:31:24.893800+0000","last_unstale":"2024-05-27T09:35:14.171395+0000","last_undegraded":"2024-05-27T09:35:14.171395+0000","last_fullsized":"2024-05-27T09:35:14.171395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:13.368892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043584+0000","last_change":"2024-05-27T09:31:21.054476+0000","last_active":"2024-05-27T09:35:14.043584+0000","last_peered":"2024-05-27T09:35:14.043584+0000","last_clean":"2024-05-27T09:35:14.043584+0000","last_became_active":"2024-05-27T09:31:21.054091+0000","last_became_peered":"2024-05-27T09:31:21.054091+0000","last_unstale":"2024-05-27T09:35:14.043584+0000","last_undegraded":"2024-05-27T09:35:14.043584+0000","last_fullsized":"2024-05-27T09:35:14.043584+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:15.909230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043020+0000","last_change":"2024-05-27T09:31:24.913157+0000","last_active":"2024-05-27T09:35:14.043020+0000","last_peered":"2024-05-27T09:35:14.043020+0000","last_clean":"2024-05-27T09:35:14.043020+0000","last_became_active":"2024-05-27T09:31:24.912793+0000","last_became_peered":"2024-05-27T09:31:24.912793+0000","last_unstale":"2024-05-27T09:35:14.043020+0000","last_undegraded":"2024-05-27T09:35:14.043020+0000","last_fullsized":"2024-05-27T09:35:14.043020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:17:36.451069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372678+0000","last_change":"2024-05-27T09:31:20.861349+0000","last_active":"2024-05-27T09:35:14.372678+0000","last_peered":"2024-05-27T09:35:14.372678+0000","last_clean":"2024-05-27T09:35:14.372678+0000","last_became_active":"2024-05-27T09:31:20.860896+0000","last_became_peered":"2024-05-27T09:31:20.860896+0000","last_unstale":"2024-05-27T09:35:14.372678+0000","last_undegraded":"2024-05-27T09:35:14.372678+0000","last_fullsized":"2024-05-27T09:35:14.372678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:00.772910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169685+0000","last_change":"2024-05-27T09:31:24.893240+0000","last_active":"2024-05-27T09:35:14.169685+0000","last_peered":"2024-05-27T09:35:14.169685+0000","last_clean":"2024-05-27T09:35:14.169685+0000","last_became_active":"2024-05-27T09:31:24.892169+0000","last_became_peered":"2024-05-27T09:31:24.892169+0000","last_unstale":"2024-05-27T09:35:14.169685+0000","last_undegraded":"2024-05-27T09:35:14.169685+0000","last_fullsized":"2024-05-27T09:35:14.169685+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:26:59.660725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170715+0000","last_change":"2024-05-27T09:31:20.864307+0000","last_active":"2024-05-27T09:35:14.170715+0000","last_peered":"2024-05-27T09:35:14.170715+0000","last_clean":"2024-05-27T09:35:14.170715+0000","last_became_active":"2024-05-27T09:31:20.863913+0000","last_became_peered":"2024-05-27T09:31:20.863913+0000","last_unstale":"2024-05-27T09:35:14.170715+0000","last_undegraded":"2024-05-27T09:35:14.170715+0000","last_fullsized":"2024-05-27T09:35:14.170715+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:58:57.656731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839950+0000","last_change":"2024-05-27T09:31:24.905728+0000","last_active":"2024-05-27T09:35:14.839950+0000","last_peered":"2024-05-27T09:35:14.839950+0000","last_clean":"2024-05-27T09:35:14.839950+0000","last_became_active":"2024-05-27T09:31:24.905365+0000","last_became_peered":"2024-05-27T09:31:24.905365+0000","last_unstale":"2024-05-27T09:35:14.839950+0000","last_undegraded":"2024-05-27T09:35:14.839950+0000","last_fullsized":"2024-05-27T09:35:14.839950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:05.013037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373037+0000","last_change":"2024-05-27T09:31:20.878478+0000","last_active":"2024-05-27T09:35:14.373037+0000","last_peered":"2024-05-27T09:35:14.373037+0000","last_clean":"2024-05-27T09:35:14.373037+0000","last_became_active":"2024-05-27T09:31:20.877952+0000","last_became_peered":"2024-05-27T09:31:20.877952+0000","last_unstale":"2024-05-27T09:35:14.373037+0000","last_undegraded":"2024-05-27T09:35:14.373037+0000","last_fullsized":"2024-05-27T09:35:14.373037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:16:27.580641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171301+0000","last_change":"2024-05-27T09:31:24.904967+0000","last_active":"2024-05-27T09:35:14.171301+0000","last_peered":"2024-05-27T09:35:14.171301+0000","last_clean":"2024-05-27T09:35:14.171301+0000","last_became_active":"2024-05-27T09:31:24.904542+0000","last_became_peered":"2024-05-27T09:31:24.904542+0000","last_unstale":"2024-05-27T09:35:14.171301+0000","last_undegraded":"2024-05-27T09:35:14.171301+0000","last_fullsized":"2024-05-27T09:35:14.171301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:01:05.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169841+0000","last_change":"2024-05-27T09:31:57.875713+0000","last_active":"2024-05-27T09:35:14.169841+0000","last_peered":"2024-05-27T09:35:14.169841+0000","last_clean":"2024-05-27T09:35:14.169841+0000","last_became_active":"2024-05-27T09:31:57.870244+0000","last_became_peered":"2024-05-27T09:31:57.870244+0000","last_unstale":"2024-05-27T09:35:14.169841+0000","last_undegraded":"2024-05-27T09:35:14.169841+0000","last_fullsized":"2024-05-27T09:35:14.169841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:49.272418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353395+0000","last_change":"2024-05-27T09:31:24.892655+0000","last_active":"2024-05-27T09:35:14.353395+0000","last_peered":"2024-05-27T09:35:14.353395+0000","last_clean":"2024-05-27T09:35:14.353395+0000","last_became_active":"2024-05-27T09:31:24.892112+0000","last_became_peered":"2024-05-27T09:31:24.892112+0000","last_unstale":"2024-05-27T09:35:14.353395+0000","last_undegraded":"2024-05-27T09:35:14.353395+0000","last_fullsized":"2024-05-27T09:35:14.353395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.604118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043417+0000","last_change":"2024-05-27T09:31:57.875213+0000","last_active":"2024-05-27T09:35:14.043417+0000","last_peered":"2024-05-27T09:35:14.043417+0000","last_clean":"2024-05-27T09:35:14.043417+0000","last_became_active":"2024-05-27T09:31:57.869025+0000","last_became_peered":"2024-05-27T09:31:57.869025+0000","last_unstale":"2024-05-27T09:35:14.043417+0000","last_undegraded":"2024-05-27T09:35:14.043417+0000","last_fullsized":"2024-05-27T09:35:14.043417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:51.433347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353725+0000","last_change":"2024-05-27T09:31:24.899130+0000","last_active":"2024-05-27T09:35:14.353725+0000","last_peered":"2024-05-27T09:35:14.353725+0000","last_clean":"2024-05-27T09:35:14.353725+0000","last_became_active":"2024-05-27T09:31:24.898722+0000","last_became_peered":"2024-05-27T09:31:24.898722+0000","last_unstale":"2024-05-27T09:35:14.353725+0000","last_undegraded":"2024-05-27T09:35:14.353725+0000","last_fullsized":"2024-05-27T09:35:14.353725+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:51.247275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'39","reported_seq":76,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.314140+0000","last_change":"2024-05-27T09:31:20.863757+0000","last_active":"2024-05-27T09:35:14.314140+0000","last_peered":"2024-05-27T09:35:14.314140+0000","last_clean":"2024-05-27T09:35:14.314140+0000","last_became_active":"2024-05-27T09:31:20.863465+0000","last_became_peered":"2024-05-27T09:31:20.863465+0000","last_unstale":"2024-05-27T09:35:14.314140+0000","last_undegraded":"2024-05-27T09:35:14.314140+0000","last_fullsized":"2024-05-27T09:35:14.314140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:21:45.741822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.913749+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.913384+0000","last_became_peered":"2024-05-27T09:31:24.913384+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:24:29.826096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242122+0000","last_change":"2024-05-27T09:31:20.865844+0000","last_active":"2024-05-27T09:35:14.242122+0000","last_peered":"2024-05-27T09:35:14.242122+0000","last_clean":"2024-05-27T09:35:14.242122+0000","last_became_active":"2024-05-27T09:31:20.865364+0000","last_became_peered":"2024-05-27T09:31:20.865364+0000","last_unstale":"2024-05-27T09:35:14.242122+0000","last_undegraded":"2024-05-27T09:35:14.242122+0000","last_fullsized":"2024-05-27T09:35:14.242122+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:11:54.451294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353821+0000","last_change":"2024-05-27T09:31:24.890326+0000","last_active":"2024-05-27T09:35:14.353821+0000","last_peered":"2024-05-27T09:35:14.353821+0000","last_clean":"2024-05-27T09:35:14.353821+0000","last_became_active":"2024-05-27T09:31:24.889789+0000","last_became_peered":"2024-05-27T09:31:24.889789+0000","last_unstale":"2024-05-27T09:35:14.353821+0000","last_undegraded":"2024-05-27T09:35:14.353821+0000","last_fullsized":"2024-05-27T09:35:14.353821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:34:04.252231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840005+0000","last_change":"2024-05-27T09:31:24.896730+0000","last_active":"2024-05-27T09:35:14.840005+0000","last_peered":"2024-05-27T09:35:14.840005+0000","last_clean":"2024-05-27T09:35:14.840005+0000","last_became_active":"2024-05-27T09:31:24.896365+0000","last_became_peered":"2024-05-27T09:31:24.896365+0000","last_unstale":"2024-05-27T09:35:14.840005+0000","last_undegraded":"2024-05-27T09:35:14.840005+0000","last_fullsized":"2024-05-27T09:35:14.840005+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:37.443788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372837+0000","last_change":"2024-05-27T09:31:24.899215+0000","last_active":"2024-05-27T09:35:14.372837+0000","last_peered":"2024-05-27T09:35:14.372837+0000","last_clean":"2024-05-27T09:35:14.372837+0000","last_became_active":"2024-05-27T09:31:24.898733+0000","last_became_peered":"2024-05-27T09:31:24.898733+0000","last_unstale":"2024-05-27T09:35:14.372837+0000","last_undegraded":"2024-05-27T09:35:14.372837+0000","last_fullsized":"2024-05-27T09:35:14.372837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:10.085989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372709+0000","last_change":"2024-05-27T09:31:24.914922+0000","last_active":"2024-05-27T09:35:14.372709+0000","last_peered":"2024-05-27T09:35:14.372709+0000","last_clean":"2024-05-27T09:35:14.372709+0000","last_became_active":"2024-05-27T09:31:24.914541+0000","last_became_peered":"2024-05-27T09:31:24.914541+0000","last_unstale":"2024-05-27T09:35:14.372709+0000","last_undegraded":"2024-05-27T09:35:14.372709+0000","last_fullsized":"2024-05-27T09:35:14.372709+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:30:38.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042498+0000","last_change":"2024-05-27T09:31:24.909628+0000","last_active":"2024-05-27T09:35:14.042498+0000","last_peered":"2024-05-27T09:35:14.042498+0000","last_clean":"2024-05-27T09:35:14.042498+0000","last_became_active":"2024-05-27T09:31:24.902931+0000","last_became_peered":"2024-05-27T09:31:24.902931+0000","last_unstale":"2024-05-27T09:35:14.042498+0000","last_undegraded":"2024-05-27T09:35:14.042498+0000","last_fullsized":"2024-05-27T09:35:14.042498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:06.711316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242398+0000","last_change":"2024-05-27T09:31:24.912099+0000","last_active":"2024-05-27T09:35:14.242398+0000","last_peered":"2024-05-27T09:35:14.242398+0000","last_clean":"2024-05-27T09:35:14.242398+0000","last_became_active":"2024-05-27T09:31:24.911669+0000","last_became_peered":"2024-05-27T09:31:24.911669+0000","last_unstale":"2024-05-27T09:35:14.242398+0000","last_undegraded":"2024-05-27T09:35:14.242398+0000","last_fullsized":"2024-05-27T09:35:14.242398+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:58.068589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840085+0000","last_change":"2024-05-27T09:31:24.891426+0000","last_active":"2024-05-27T09:35:14.840085+0000","last_peered":"2024-05-27T09:35:14.840085+0000","last_clean":"2024-05-27T09:35:14.840085+0000","last_became_active":"2024-05-27T09:31:24.890877+0000","last_became_peered":"2024-05-27T09:31:24.890877+0000","last_unstale":"2024-05-27T09:35:14.840085+0000","last_undegraded":"2024-05-27T09:35:14.840085+0000","last_fullsized":"2024-05-27T09:35:14.840085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:15.338188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373079+0000","last_change":"2024-05-27T09:31:24.892021+0000","last_active":"2024-05-27T09:35:14.373079+0000","last_peered":"2024-05-27T09:35:14.373079+0000","last_clean":"2024-05-27T09:35:14.373079+0000","last_became_active":"2024-05-27T09:31:24.891607+0000","last_became_peered":"2024-05-27T09:31:24.891607+0000","last_unstale":"2024-05-27T09:35:14.373079+0000","last_undegraded":"2024-05-27T09:35:14.373079+0000","last_fullsized":"2024-05-27T09:35:14.373079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:05.623199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171321+0000","last_change":"2024-05-27T09:31:24.912166+0000","last_active":"2024-05-27T09:35:14.171321+0000","last_peered":"2024-05-27T09:35:14.171321+0000","last_clean":"2024-05-27T09:35:14.171321+0000","last_became_active":"2024-05-27T09:31:24.907555+0000","last_became_peered":"2024-05-27T09:31:24.907555+0000","last_unstale":"2024-05-27T09:35:14.171321+0000","last_undegraded":"2024-05-27T09:35:14.171321+0000","last_fullsized":"2024-05-27T09:35:14.171321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:55:33.772321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129575+0000","last_change":"2024-05-27T09:31:24.910525+0000","last_active":"2024-05-27T09:35:14.129575+0000","last_peered":"2024-05-27T09:35:14.129575+0000","last_clean":"2024-05-27T09:35:14.129575+0000","last_became_active":"2024-05-27T09:31:24.910089+0000","last_became_peered":"2024-05-27T09:31:24.910089+0000","last_unstale":"2024-05-27T09:35:14.129575+0000","last_undegraded":"2024-05-27T09:35:14.129575+0000","last_fullsized":"2024-05-27T09:35:14.129575+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:19:02.274781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170634+0000","last_change":"2024-05-27T09:31:24.901125+0000","last_active":"2024-05-27T09:35:14.170634+0000","last_peered":"2024-05-27T09:35:14.170634+0000","last_clean":"2024-05-27T09:35:14.170634+0000","last_became_active":"2024-05-27T09:31:24.900361+0000","last_became_peered":"2024-05-27T09:31:24.900361+0000","last_unstale":"2024-05-27T09:35:14.170634+0000","last_undegraded":"2024-05-27T09:35:14.170634+0000","last_fullsized":"2024-05-27T09:35:14.170634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:30.116349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372869+0000","last_change":"2024-05-27T09:31:24.917612+0000","last_active":"2024-05-27T09:35:14.372869+0000","last_peered":"2024-05-27T09:35:14.372869+0000","last_clean":"2024-05-27T09:35:14.372869+0000","last_became_active":"2024-05-27T09:31:24.917210+0000","last_became_peered":"2024-05-27T09:31:24.917210+0000","last_unstale":"2024-05-27T09:35:14.372869+0000","last_undegraded":"2024-05-27T09:35:14.372869+0000","last_fullsized":"2024-05-27T09:35:14.372869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:19:00.769114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241872+0000","last_change":"2024-05-27T09:31:24.909654+0000","last_active":"2024-05-27T09:35:14.241872+0000","last_peered":"2024-05-27T09:35:14.241872+0000","last_clean":"2024-05-27T09:35:14.241872+0000","last_became_active":"2024-05-27T09:31:24.909143+0000","last_became_peered":"2024-05-27T09:31:24.909143+0000","last_unstale":"2024-05-27T09:35:14.241872+0000","last_undegraded":"2024-05-27T09:35:14.241872+0000","last_fullsized":"2024-05-27T09:35:14.241872+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:42.844614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840108+0000","last_change":"2024-05-27T09:31:24.899031+0000","last_active":"2024-05-27T09:35:14.840108+0000","last_peered":"2024-05-27T09:35:14.840108+0000","last_clean":"2024-05-27T09:35:14.840108+0000","last_became_active":"2024-05-27T09:31:24.898664+0000","last_became_peered":"2024-05-27T09:31:24.898664+0000","last_unstale":"2024-05-27T09:35:14.840108+0000","last_undegraded":"2024-05-27T09:35:14.840108+0000","last_fullsized":"2024-05-27T09:35:14.840108+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:09.659646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598026,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1372,"num_read_kb":790,"num_write":297,"num_write_kb":1442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1794078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678079,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36192,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708960,"statfs":{"total":95995035648,"available":95957975040,"internally_reserved":0,"allocated":876544,"data_stored":431801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874307,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41188,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703964,"statfs":{"total":95995035648,"available":95952859136,"internally_reserved":0,"allocated":1732608,"data_stored":1232494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070537,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":834188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266767,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":834108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462997,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1486848,"data_stored":1024866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626522,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36196,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708956,"statfs":{"total":95995035648,"available":95957970944,"internally_reserved":0,"allocated":880640,"data_stored":431426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":507211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":441632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411444,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":508902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182507,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28424,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716728,"statfs":{"total":95995035648,"available":95965929472,"internally_reserved":0,"allocated":1441792,"data_stored":965488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986279,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":443000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822754,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36948,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708204,"statfs":{"total":95995035648,"available":95957200896,"internally_reserved":0,"allocated":1650688,"data_stored":1164638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":201855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T09:35:21.818 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T09:35:22.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.118+0000 7f4069200640 1 -- 172.21.15.44:0/1664346866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4064109ef0 msgr2=0x7f4064103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:22.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.118+0000 7f4069200640 1 --2- 172.21.15.44:0/1664346866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4064109ef0 0x7f4064103b40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4050009900 tx=0x7f405002a3f0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.118+0000 7f4069200640 1 -- 172.21.15.44:0/1664346866 shutdown_connections 2024-05-27T09:35:22.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.118+0000 7f4069200640 1 --2- 172.21.15.44:0/1664346866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40641041b0 0x7f40641104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.118+0000 7f4069200640 1 --2- 172.21.15.44:0/1664346866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4064109ef0 0x7f4064103b40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.118+0000 7f4069200640 1 --2- 172.21.15.44:0/1664346866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4064109a50 0x7f4064103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.118+0000 7f4069200640 1 -- 172.21.15.44:0/1664346866 >> 172.21.15.44:0/1664346866 conn(0x7f4064079100 msgr2=0x7f4064079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:22.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.119+0000 7f4069200640 1 -- 172.21.15.44:0/1664346866 shutdown_connections 2024-05-27T09:35:22.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.119+0000 7f4069200640 1 -- 172.21.15.44:0/1664346866 wait complete. 2024-05-27T09:35:22.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.120+0000 7f4069200640 1 Processor -- start 2024-05-27T09:35:22.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.120+0000 7f4069200640 1 -- start start 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.120+0000 7f4069200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40641041b0 0x7f406419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.120+0000 7f4069200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4064109a50 0x7f406419abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.120+0000 7f4069200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4064109ef0 0x7f406419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.120+0000 7f4069200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4064113cc0 con 0x7f4064109a50 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.120+0000 7f4069200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4064113b40 con 0x7f4064109ef0 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.120+0000 7f4069200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4064113e40 con 0x7f40641041b0 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.121+0000 7f4062a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4064109a50 0x7f406419abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.121+0000 7f4063e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4064109ef0 0x7f406419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.121+0000 7f4062a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4064109a50 0x7f406419abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44302/0 (socket says 172.21.15.44:44302) 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.121+0000 7f4062a00640 1 -- 172.21.15.44:0/3043012570 learned_addr learned my addr 172.21.15.44:0/3043012570 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.121+0000 7f4063400640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40641041b0 0x7f406419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:22.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.121+0000 7f4063e00640 1 -- 172.21.15.44:0/3043012570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40641041b0 msgr2=0x7f406419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:22.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.121+0000 7f4063e00640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40641041b0 0x7f406419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.121+0000 7f4063e00640 1 -- 172.21.15.44:0/3043012570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4064109a50 msgr2=0x7f406419abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:22.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.121+0000 7f4063e00640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4064109a50 0x7f406419abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.121+0000 7f4063e00640 1 -- 172.21.15.44:0/3043012570 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406419f4e0 con 0x7f4064109ef0 2024-05-27T09:35:22.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.122+0000 7f4063e00640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4064109ef0 0x7f406419eea0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f405c00ec50 tx=0x7f405c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:22.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.122+0000 7f405be00640 1 -- 172.21.15.44:0/3043012570 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c003b70 con 0x7f4064109ef0 2024-05-27T09:35:22.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.122+0000 7f405be00640 1 -- 172.21.15.44:0/3043012570 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f405c003d10 con 0x7f4064109ef0 2024-05-27T09:35:22.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.122+0000 7f405be00640 1 -- 172.21.15.44:0/3043012570 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c010830 con 0x7f4064109ef0 2024-05-27T09:35:22.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.122+0000 7f4069200640 1 -- 172.21.15.44:0/3043012570 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406419f7d0 con 0x7f4064109ef0 2024-05-27T09:35:22.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.122+0000 7f4069200640 1 -- 172.21.15.44:0/3043012570 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f406419fca0 con 0x7f4064109ef0 2024-05-27T09:35:22.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.123+0000 7f4069200640 1 -- 172.21.15.44:0/3043012570 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4064069d00 con 0x7f4064109ef0 2024-05-27T09:35:22.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.125+0000 7f405be00640 1 -- 172.21.15.44:0/3043012570 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f405c003ec0 con 0x7f4064109ef0 2024-05-27T09:35:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.127+0000 7f405be00640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40480b2af0 0x7f40480b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.127+0000 7f405be00640 1 -- 172.21.15.44:0/3043012570 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f405c0e2dc0 con 0x7f4064109ef0 2024-05-27T09:35:22.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.127+0000 7f4063400640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40480b2af0 0x7f40480b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:22.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.128+0000 7f4063400640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40480b2af0 0x7f40480b5000 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f406406c350 tx=0x7f4054005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:22.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.132+0000 7f405be00640 1 -- 172.21.15.44:0/3043012570 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f405c0aa520 con 0x7f4064109ef0 2024-05-27T09:35:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:35:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:35:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:35:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:35:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: Updating smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:35:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:35:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:35:22.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:22 smithi050 ceph-mon[36371]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:35:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:35:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:35:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:35:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:35:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: Updating smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:35:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:35:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:35:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:22 smithi189 ceph-mon[36671]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:35:22.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.344+0000 7f4069200640 1 -- 172.21.15.44:0/3043012570 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4064063100 con 0x7f40480b2af0 2024-05-27T09:35:22.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.356+0000 7f405be00640 1 -- 172.21.15.44:0/3043012570 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257738 (secure 0 0 0) 0x7f4064063100 con 0x7f40480b2af0 2024-05-27T09:35:22.361 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:22.363 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-05-27T09:35:22.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 -- 172.21.15.44:0/3043012570 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40480b2af0 msgr2=0x7f40480b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:22.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40480b2af0 0x7f40480b5000 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f406406c350 tx=0x7f4054005f90 comp rx=0 tx=0).stop 2024-05-27T09:35:22.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 -- 172.21.15.44:0/3043012570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4064109ef0 msgr2=0x7f406419eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:22.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4064109ef0 0x7f406419eea0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f405c00ec50 tx=0x7f405c00c6a0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 -- 172.21.15.44:0/3043012570 shutdown_connections 2024-05-27T09:35:22.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4064109ef0 0x7f406419eea0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40480b2af0 0x7f40480b5000 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4064109a50 0x7f406419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 --2- 172.21.15.44:0/3043012570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40641041b0 0x7f406419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 -- 172.21.15.44:0/3043012570 >> 172.21.15.44:0/3043012570 conn(0x7f4064079100 msgr2=0x7f4064105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:22.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 -- 172.21.15.44:0/3043012570 shutdown_connections 2024-05-27T09:35:22.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.361+0000 7f4069200640 1 -- 172.21.15.44:0/3043012570 wait complete. 2024-05-27T09:35:22.388 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:35:22.388 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:35:22.388 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:35:22.388 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:35:22.388 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: Updating smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T09:35:22.388 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: Updating smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:35:22.389 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: Updating smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:35:22.389 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.389 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.389 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.389 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:22.389 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:22 smithi044 ceph-mon[31618]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-05-27T09:35:22.390 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-05-27T09:35:22.069143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191539,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1550,"num_read_kb":1065,"num_write":438,"num_write_kb":3599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":837,"ondisk_log_size":837,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384144,"kb_used_data":14160,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557680,"statfs":{"total":1151940427776,"available":1151547064320,"internally_reserved":0,"allocated":14499840,"data_stored":8856465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2911,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":97,"num_read_kb":206,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.966333"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840251+0000","last_change":"2024-05-27T09:31:24.899109+0000","last_active":"2024-05-27T09:35:14.840251+0000","last_peered":"2024-05-27T09:35:14.840251+0000","last_clean":"2024-05-27T09:35:14.840251+0000","last_became_active":"2024-05-27T09:31:24.898783+0000","last_became_peered":"2024-05-27T09:31:24.898783+0000","last_unstale":"2024-05-27T09:35:14.840251+0000","last_undegraded":"2024-05-27T09:35:14.840251+0000","last_fullsized":"2024-05-27T09:35:14.840251+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:01:12.220389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042262+0000","last_change":"2024-05-27T09:31:24.898382+0000","last_active":"2024-05-27T09:35:14.042262+0000","last_peered":"2024-05-27T09:35:14.042262+0000","last_clean":"2024-05-27T09:35:14.042262+0000","last_became_active":"2024-05-27T09:31:24.897975+0000","last_became_peered":"2024-05-27T09:31:24.897975+0000","last_unstale":"2024-05-27T09:35:14.042262+0000","last_undegraded":"2024-05-27T09:35:14.042262+0000","last_fullsized":"2024-05-27T09:35:14.042262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:38.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041822+0000","last_change":"2024-05-27T09:31:24.896122+0000","last_active":"2024-05-27T09:35:14.041822+0000","last_peered":"2024-05-27T09:35:14.041822+0000","last_clean":"2024-05-27T09:35:14.041822+0000","last_became_active":"2024-05-27T09:31:24.890867+0000","last_became_peered":"2024-05-27T09:31:24.890867+0000","last_unstale":"2024-05-27T09:35:14.041822+0000","last_undegraded":"2024-05-27T09:35:14.041822+0000","last_fullsized":"2024-05-27T09:35:14.041822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:09.982177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046136+0000","last_change":"2024-05-27T09:31:57.868900+0000","last_active":"2024-05-27T09:35:14.046136+0000","last_peered":"2024-05-27T09:35:14.046136+0000","last_clean":"2024-05-27T09:35:14.046136+0000","last_became_active":"2024-05-27T09:31:57.868418+0000","last_became_peered":"2024-05-27T09:31:57.868418+0000","last_unstale":"2024-05-27T09:35:14.046136+0000","last_undegraded":"2024-05-27T09:35:14.046136+0000","last_fullsized":"2024-05-27T09:35:14.046136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:08.966704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169866+0000","last_change":"2024-05-27T09:31:24.910283+0000","last_active":"2024-05-27T09:35:14.169866+0000","last_peered":"2024-05-27T09:35:14.169866+0000","last_clean":"2024-05-27T09:35:14.169866+0000","last_became_active":"2024-05-27T09:31:24.909882+0000","last_became_peered":"2024-05-27T09:31:24.909882+0000","last_unstale":"2024-05-27T09:35:14.169866+0000","last_undegraded":"2024-05-27T09:35:14.169866+0000","last_fullsized":"2024-05-27T09:35:14.169866+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:43.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126676+0000","last_change":"2024-05-27T09:31:24.900934+0000","last_active":"2024-05-27T09:35:14.126676+0000","last_peered":"2024-05-27T09:35:14.126676+0000","last_clean":"2024-05-27T09:35:14.126676+0000","last_became_active":"2024-05-27T09:31:24.900534+0000","last_became_peered":"2024-05-27T09:31:24.900534+0000","last_unstale":"2024-05-27T09:35:14.126676+0000","last_undegraded":"2024-05-27T09:35:14.126676+0000","last_fullsized":"2024-05-27T09:35:14.126676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:53.318327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353192+0000","last_change":"2024-05-27T09:31:24.907080+0000","last_active":"2024-05-27T09:35:14.353192+0000","last_peered":"2024-05-27T09:35:14.353192+0000","last_clean":"2024-05-27T09:35:14.353192+0000","last_became_active":"2024-05-27T09:31:24.906715+0000","last_became_peered":"2024-05-27T09:31:24.906715+0000","last_unstale":"2024-05-27T09:35:14.353192+0000","last_undegraded":"2024-05-27T09:35:14.353192+0000","last_fullsized":"2024-05-27T09:35:14.353192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:36.214659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.044350+0000","last_change":"2024-05-27T09:31:58.200608+0000","last_active":"2024-05-27T09:35:14.044350+0000","last_peered":"2024-05-27T09:35:14.044350+0000","last_clean":"2024-05-27T09:35:14.044350+0000","last_became_active":"2024-05-27T09:31:58.199363+0000","last_became_peered":"2024-05-27T09:31:58.199363+0000","last_unstale":"2024-05-27T09:35:14.044350+0000","last_undegraded":"2024-05-27T09:35:14.044350+0000","last_fullsized":"2024-05-27T09:35:14.044350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:30:46.495930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373508+0000","last_change":"2024-05-27T09:31:24.892630+0000","last_active":"2024-05-27T09:35:14.373508+0000","last_peered":"2024-05-27T09:35:14.373508+0000","last_clean":"2024-05-27T09:35:14.373508+0000","last_became_active":"2024-05-27T09:31:24.892176+0000","last_became_peered":"2024-05-27T09:31:24.892176+0000","last_unstale":"2024-05-27T09:35:14.373508+0000","last_undegraded":"2024-05-27T09:35:14.373508+0000","last_fullsized":"2024-05-27T09:35:14.373508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:49:20.585601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.127155+0000","last_change":"2024-05-27T09:31:24.895281+0000","last_active":"2024-05-27T09:35:14.127155+0000","last_peered":"2024-05-27T09:35:14.127155+0000","last_clean":"2024-05-27T09:35:14.127155+0000","last_became_active":"2024-05-27T09:31:24.894886+0000","last_became_peered":"2024-05-27T09:31:24.894886+0000","last_unstale":"2024-05-27T09:35:14.127155+0000","last_undegraded":"2024-05-27T09:35:14.127155+0000","last_fullsized":"2024-05-27T09:35:14.127155+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:41.494434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353559+0000","last_change":"2024-05-27T09:31:24.900167+0000","last_active":"2024-05-27T09:35:14.353559+0000","last_peered":"2024-05-27T09:35:14.353559+0000","last_clean":"2024-05-27T09:35:14.353559+0000","last_became_active":"2024-05-27T09:31:24.899750+0000","last_became_peered":"2024-05-27T09:31:24.899750+0000","last_unstale":"2024-05-27T09:35:14.353559+0000","last_undegraded":"2024-05-27T09:35:14.353559+0000","last_fullsized":"2024-05-27T09:35:14.353559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:30.602246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.045840+0000","last_change":"2024-05-27T09:31:57.868818+0000","last_active":"2024-05-27T09:35:14.045840+0000","last_peered":"2024-05-27T09:35:14.045840+0000","last_clean":"2024-05-27T09:35:14.045840+0000","last_became_active":"2024-05-27T09:31:57.868322+0000","last_became_peered":"2024-05-27T09:31:57.868322+0000","last_unstale":"2024-05-27T09:35:14.045840+0000","last_undegraded":"2024-05-27T09:35:14.045840+0000","last_fullsized":"2024-05-27T09:35:14.045840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:35.304850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242036+0000","last_change":"2024-05-27T09:31:24.902537+0000","last_active":"2024-05-27T09:35:14.242036+0000","last_peered":"2024-05-27T09:35:14.242036+0000","last_clean":"2024-05-27T09:35:14.242036+0000","last_became_active":"2024-05-27T09:31:24.902132+0000","last_became_peered":"2024-05-27T09:31:24.902132+0000","last_unstale":"2024-05-27T09:35:14.242036+0000","last_undegraded":"2024-05-27T09:35:14.242036+0000","last_fullsized":"2024-05-27T09:35:14.242036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:20:50.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041926+0000","last_change":"2024-05-27T09:31:24.899182+0000","last_active":"2024-05-27T09:35:14.041926+0000","last_peered":"2024-05-27T09:35:14.041926+0000","last_clean":"2024-05-27T09:35:14.041926+0000","last_became_active":"2024-05-27T09:31:24.898749+0000","last_became_peered":"2024-05-27T09:31:24.898749+0000","last_unstale":"2024-05-27T09:35:14.041926+0000","last_undegraded":"2024-05-27T09:35:14.041926+0000","last_fullsized":"2024-05-27T09:35:14.041926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:44:44.844357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041660+0000","last_change":"2024-05-27T09:31:24.901115+0000","last_active":"2024-05-27T09:35:14.041660+0000","last_peered":"2024-05-27T09:35:14.041660+0000","last_clean":"2024-05-27T09:35:14.041660+0000","last_became_active":"2024-05-27T09:31:24.897011+0000","last_became_peered":"2024-05-27T09:31:24.897011+0000","last_unstale":"2024-05-27T09:35:14.041660+0000","last_undegraded":"2024-05-27T09:35:14.041660+0000","last_fullsized":"2024-05-27T09:35:14.041660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:01.130407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372690+0000","last_change":"2024-05-27T09:31:24.903063+0000","last_active":"2024-05-27T09:35:14.372690+0000","last_peered":"2024-05-27T09:35:14.372690+0000","last_clean":"2024-05-27T09:35:14.372690+0000","last_became_active":"2024-05-27T09:31:24.899766+0000","last_became_peered":"2024-05-27T09:31:24.899766+0000","last_unstale":"2024-05-27T09:35:14.372690+0000","last_undegraded":"2024-05-27T09:35:14.372690+0000","last_fullsized":"2024-05-27T09:35:14.372690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:45.984226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840388+0000","last_change":"2024-05-27T09:31:24.915922+0000","last_active":"2024-05-27T09:35:14.840388+0000","last_peered":"2024-05-27T09:35:14.840388+0000","last_clean":"2024-05-27T09:35:14.840388+0000","last_became_active":"2024-05-27T09:31:24.915623+0000","last_became_peered":"2024-05-27T09:31:24.915623+0000","last_unstale":"2024-05-27T09:35:14.840388+0000","last_undegraded":"2024-05-27T09:35:14.840388+0000","last_fullsized":"2024-05-27T09:35:14.840388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:02:01.006704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046094+0000","last_change":"2024-05-27T09:31:24.908397+0000","last_active":"2024-05-27T09:35:14.046094+0000","last_peered":"2024-05-27T09:35:14.046094+0000","last_clean":"2024-05-27T09:35:14.046094+0000","last_became_active":"2024-05-27T09:31:24.903644+0000","last_became_peered":"2024-05-27T09:31:24.903644+0000","last_unstale":"2024-05-27T09:35:14.046094+0000","last_undegraded":"2024-05-27T09:35:14.046094+0000","last_fullsized":"2024-05-27T09:35:14.046094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:09.281429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373078+0000","last_change":"2024-05-27T09:31:24.912706+0000","last_active":"2024-05-27T09:35:14.373078+0000","last_peered":"2024-05-27T09:35:14.373078+0000","last_clean":"2024-05-27T09:35:14.373078+0000","last_became_active":"2024-05-27T09:31:24.912357+0000","last_became_peered":"2024-05-27T09:31:24.912357+0000","last_unstale":"2024-05-27T09:35:14.373078+0000","last_undegraded":"2024-05-27T09:35:14.373078+0000","last_fullsized":"2024-05-27T09:35:14.373078+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:23.334041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.045022+0000","last_change":"2024-05-27T09:31:20.881652+0000","last_active":"2024-05-27T09:35:14.045022+0000","last_peered":"2024-05-27T09:35:14.045022+0000","last_clean":"2024-05-27T09:35:14.045022+0000","last_became_active":"2024-05-27T09:31:20.880757+0000","last_became_peered":"2024-05-27T09:31:20.880757+0000","last_unstale":"2024-05-27T09:35:14.045022+0000","last_undegraded":"2024-05-27T09:35:14.045022+0000","last_fullsized":"2024-05-27T09:35:14.045022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:01.433755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171024+0000","last_change":"2024-05-27T09:31:24.908010+0000","last_active":"2024-05-27T09:35:14.171024+0000","last_peered":"2024-05-27T09:35:14.171024+0000","last_clean":"2024-05-27T09:35:14.171024+0000","last_became_active":"2024-05-27T09:31:24.907638+0000","last_became_peered":"2024-05-27T09:31:24.907638+0000","last_unstale":"2024-05-27T09:35:14.171024+0000","last_undegraded":"2024-05-27T09:35:14.171024+0000","last_fullsized":"2024-05-27T09:35:14.171024+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:02:49.207709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171395+0000","last_change":"2024-05-27T09:31:20.860556+0000","last_active":"2024-05-27T09:35:14.171395+0000","last_peered":"2024-05-27T09:35:14.171395+0000","last_clean":"2024-05-27T09:35:14.171395+0000","last_became_active":"2024-05-27T09:31:20.860172+0000","last_became_peered":"2024-05-27T09:31:20.860172+0000","last_unstale":"2024-05-27T09:35:14.171395+0000","last_undegraded":"2024-05-27T09:35:14.171395+0000","last_fullsized":"2024-05-27T09:35:14.171395+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:08:21.545681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042957+0000","last_change":"2024-05-27T09:31:24.910111+0000","last_active":"2024-05-27T09:35:14.042957+0000","last_peered":"2024-05-27T09:35:14.042957+0000","last_clean":"2024-05-27T09:35:14.042957+0000","last_became_active":"2024-05-27T09:31:24.908985+0000","last_became_peered":"2024-05-27T09:31:24.908985+0000","last_unstale":"2024-05-27T09:35:14.042957+0000","last_undegraded":"2024-05-27T09:35:14.042957+0000","last_fullsized":"2024-05-27T09:35:14.042957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:53:29.645792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170222+0000","last_change":"2024-05-27T09:31:20.879993+0000","last_active":"2024-05-27T09:35:14.170222+0000","last_peered":"2024-05-27T09:35:14.170222+0000","last_clean":"2024-05-27T09:35:14.170222+0000","last_became_active":"2024-05-27T09:31:20.878802+0000","last_became_peered":"2024-05-27T09:31:20.878802+0000","last_unstale":"2024-05-27T09:35:14.170222+0000","last_undegraded":"2024-05-27T09:35:14.170222+0000","last_fullsized":"2024-05-27T09:35:14.170222+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:15.555612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372970+0000","last_change":"2024-05-27T09:31:24.903533+0000","last_active":"2024-05-27T09:35:14.372970+0000","last_peered":"2024-05-27T09:35:14.372970+0000","last_clean":"2024-05-27T09:35:14.372970+0000","last_became_active":"2024-05-27T09:31:24.903166+0000","last_became_peered":"2024-05-27T09:31:24.903166+0000","last_unstale":"2024-05-27T09:35:14.372970+0000","last_undegraded":"2024-05-27T09:35:14.372970+0000","last_fullsized":"2024-05-27T09:35:14.372970+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:52.539470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839887+0000","last_change":"2024-05-27T09:31:57.878426+0000","last_active":"2024-05-27T09:35:14.839887+0000","last_peered":"2024-05-27T09:35:14.839887+0000","last_clean":"2024-05-27T09:35:14.839887+0000","last_became_active":"2024-05-27T09:31:57.869146+0000","last_became_peered":"2024-05-27T09:31:57.869146+0000","last_unstale":"2024-05-27T09:35:14.839887+0000","last_undegraded":"2024-05-27T09:35:14.839887+0000","last_fullsized":"2024-05-27T09:35:14.839887+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:34:51.926092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129913+0000","last_change":"2024-05-27T09:31:24.897426+0000","last_active":"2024-05-27T09:35:14.129913+0000","last_peered":"2024-05-27T09:35:14.129913+0000","last_clean":"2024-05-27T09:35:14.129913+0000","last_became_active":"2024-05-27T09:31:24.897080+0000","last_became_peered":"2024-05-27T09:31:24.897080+0000","last_unstale":"2024-05-27T09:35:14.129913+0000","last_undegraded":"2024-05-27T09:35:14.129913+0000","last_fullsized":"2024-05-27T09:35:14.129913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:15:42.428391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'8","reported_seq":255,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:18.136294+0000","last_change":"2024-05-27T09:31:20.846681+0000","last_active":"2024-05-27T09:35:18.136294+0000","last_peered":"2024-05-27T09:35:18.136294+0000","last_clean":"2024-05-27T09:35:18.136294+0000","last_became_active":"2024-05-27T09:31:20.846302+0000","last_became_peered":"2024-05-27T09:31:20.846302+0000","last_unstale":"2024-05-27T09:35:18.136294+0000","last_undegraded":"2024-05-27T09:35:18.136294+0000","last_fullsized":"2024-05-27T09:35:18.136294+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:19.609612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":218,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169711+0000","last_change":"2024-05-27T09:31:24.907209+0000","last_active":"2024-05-27T09:35:14.169711+0000","last_peered":"2024-05-27T09:35:14.169711+0000","last_clean":"2024-05-27T09:35:14.169711+0000","last_became_active":"2024-05-27T09:31:24.906752+0000","last_became_peered":"2024-05-27T09:31:24.906752+0000","last_unstale":"2024-05-27T09:35:14.169711+0000","last_undegraded":"2024-05-27T09:35:14.169711+0000","last_fullsized":"2024-05-27T09:35:14.169711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:53.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126569+0000","last_change":"2024-05-27T09:31:20.879778+0000","last_active":"2024-05-27T09:35:14.126569+0000","last_peered":"2024-05-27T09:35:14.126569+0000","last_clean":"2024-05-27T09:35:14.126569+0000","last_became_active":"2024-05-27T09:31:20.879259+0000","last_became_peered":"2024-05-27T09:31:20.879259+0000","last_unstale":"2024-05-27T09:35:14.126569+0000","last_undegraded":"2024-05-27T09:35:14.126569+0000","last_fullsized":"2024-05-27T09:35:14.126569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:20:07.972653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129702+0000","last_change":"2024-05-27T09:31:58.195516+0000","last_active":"2024-05-27T09:35:14.129702+0000","last_peered":"2024-05-27T09:35:14.129702+0000","last_clean":"2024-05-27T09:35:14.129702+0000","last_became_active":"2024-05-27T09:31:58.195118+0000","last_became_peered":"2024-05-27T09:31:58.195118+0000","last_unstale":"2024-05-27T09:35:14.129702+0000","last_undegraded":"2024-05-27T09:35:14.129702+0000","last_fullsized":"2024-05-27T09:35:14.129702+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:11:23.547605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241846+0000","last_change":"2024-05-27T09:31:20.863152+0000","last_active":"2024-05-27T09:35:14.241846+0000","last_peered":"2024-05-27T09:35:14.241846+0000","last_clean":"2024-05-27T09:35:14.241846+0000","last_became_active":"2024-05-27T09:31:20.862708+0000","last_became_peered":"2024-05-27T09:31:20.862708+0000","last_unstale":"2024-05-27T09:35:14.241846+0000","last_undegraded":"2024-05-27T09:35:14.241846+0000","last_fullsized":"2024-05-27T09:35:14.241846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:21.645773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170544+0000","last_change":"2024-05-27T09:31:24.893988+0000","last_active":"2024-05-27T09:35:14.170544+0000","last_peered":"2024-05-27T09:35:14.170544+0000","last_clean":"2024-05-27T09:35:14.170544+0000","last_became_active":"2024-05-27T09:31:24.893504+0000","last_became_peered":"2024-05-27T09:31:24.893504+0000","last_unstale":"2024-05-27T09:35:14.170544+0000","last_undegraded":"2024-05-27T09:35:14.170544+0000","last_fullsized":"2024-05-27T09:35:14.170544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:47.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129578+0000","last_change":"2024-05-27T09:31:20.845555+0000","last_active":"2024-05-27T09:35:14.129578+0000","last_peered":"2024-05-27T09:35:14.129578+0000","last_clean":"2024-05-27T09:35:14.129578+0000","last_became_active":"2024-05-27T09:31:20.845139+0000","last_became_peered":"2024-05-27T09:31:20.845139+0000","last_unstale":"2024-05-27T09:35:14.129578+0000","last_undegraded":"2024-05-27T09:35:14.129578+0000","last_fullsized":"2024-05-27T09:35:14.129578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:25.917845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.049161+0000","last_change":"2024-05-27T09:31:24.916142+0000","last_active":"2024-05-27T09:35:14.049161+0000","last_peered":"2024-05-27T09:35:14.049161+0000","last_clean":"2024-05-27T09:35:14.049161+0000","last_became_active":"2024-05-27T09:31:24.915709+0000","last_became_peered":"2024-05-27T09:31:24.915709+0000","last_unstale":"2024-05-27T09:35:14.049161+0000","last_undegraded":"2024-05-27T09:35:14.049161+0000","last_fullsized":"2024-05-27T09:35:14.049161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:38.029648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'5","reported_seq":247,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:18.135754+0000","last_change":"2024-05-27T09:31:20.859240+0000","last_active":"2024-05-27T09:35:18.135754+0000","last_peered":"2024-05-27T09:35:18.135754+0000","last_clean":"2024-05-27T09:35:18.135754+0000","last_became_active":"2024-05-27T09:31:20.858798+0000","last_became_peered":"2024-05-27T09:31:20.858798+0000","last_unstale":"2024-05-27T09:35:18.135754+0000","last_undegraded":"2024-05-27T09:35:18.135754+0000","last_fullsized":"2024-05-27T09:35:18.135754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:42.280165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":213,"num_read_kb":213,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373425+0000","last_change":"2024-05-27T09:31:24.908818+0000","last_active":"2024-05-27T09:35:14.373425+0000","last_peered":"2024-05-27T09:35:14.373425+0000","last_clean":"2024-05-27T09:35:14.373425+0000","last_became_active":"2024-05-27T09:31:24.908410+0000","last_became_peered":"2024-05-27T09:31:24.908410+0000","last_unstale":"2024-05-27T09:35:14.373425+0000","last_undegraded":"2024-05-27T09:35:14.373425+0000","last_fullsized":"2024-05-27T09:35:14.373425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:05:40.326232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353642+0000","last_change":"2024-05-27T09:31:20.848632+0000","last_active":"2024-05-27T09:35:14.353642+0000","last_peered":"2024-05-27T09:35:14.353642+0000","last_clean":"2024-05-27T09:35:14.353642+0000","last_became_active":"2024-05-27T09:31:20.847558+0000","last_became_peered":"2024-05-27T09:31:20.847558+0000","last_unstale":"2024-05-27T09:35:14.353642+0000","last_undegraded":"2024-05-27T09:35:14.353642+0000","last_fullsized":"2024-05-27T09:35:14.353642+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:53.930565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043056+0000","last_change":"2024-05-27T09:31:24.901513+0000","last_active":"2024-05-27T09:35:14.043056+0000","last_peered":"2024-05-27T09:35:14.043056+0000","last_clean":"2024-05-27T09:35:14.043056+0000","last_became_active":"2024-05-27T09:31:24.895924+0000","last_became_peered":"2024-05-27T09:31:24.895924+0000","last_unstale":"2024-05-27T09:35:14.043056+0000","last_undegraded":"2024-05-27T09:35:14.043056+0000","last_fullsized":"2024-05-27T09:35:14.043056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:27:39.939098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.130061+0000","last_change":"2024-05-27T09:31:20.855195+0000","last_active":"2024-05-27T09:35:14.130061+0000","last_peered":"2024-05-27T09:35:14.130061+0000","last_clean":"2024-05-27T09:35:14.130061+0000","last_became_active":"2024-05-27T09:31:20.854910+0000","last_became_peered":"2024-05-27T09:31:20.854910+0000","last_unstale":"2024-05-27T09:35:14.130061+0000","last_undegraded":"2024-05-27T09:35:14.130061+0000","last_fullsized":"2024-05-27T09:35:14.130061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:02.795103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373077+0000","last_change":"2024-05-27T09:31:24.895887+0000","last_active":"2024-05-27T09:35:14.373077+0000","last_peered":"2024-05-27T09:35:14.373077+0000","last_clean":"2024-05-27T09:35:14.373077+0000","last_became_active":"2024-05-27T09:31:24.894622+0000","last_became_peered":"2024-05-27T09:31:24.894622+0000","last_unstale":"2024-05-27T09:35:14.373077+0000","last_undegraded":"2024-05-27T09:35:14.373077+0000","last_fullsized":"2024-05-27T09:35:14.373077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:32.833946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'38","reported_seq":505,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:19.137271+0000","last_change":"2024-05-27T09:31:20.846063+0000","last_active":"2024-05-27T09:35:19.137271+0000","last_peered":"2024-05-27T09:35:19.137271+0000","last_clean":"2024-05-27T09:35:19.137271+0000","last_became_active":"2024-05-27T09:31:20.845401+0000","last_became_peered":"2024-05-27T09:31:20.845401+0000","last_unstale":"2024-05-27T09:35:19.137271+0000","last_undegraded":"2024-05-27T09:35:19.137271+0000","last_fullsized":"2024-05-27T09:35:19.137271+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:21:38.460183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69968,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":438,"num_read_kb":288,"num_write":39,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170004+0000","last_change":"2024-05-27T09:31:24.892748+0000","last_active":"2024-05-27T09:35:14.170004+0000","last_peered":"2024-05-27T09:35:14.170004+0000","last_clean":"2024-05-27T09:35:14.170004+0000","last_became_active":"2024-05-27T09:31:24.892394+0000","last_became_peered":"2024-05-27T09:31:24.892394+0000","last_unstale":"2024-05-27T09:35:14.170004+0000","last_undegraded":"2024-05-27T09:35:14.170004+0000","last_fullsized":"2024-05-27T09:35:14.170004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:39:31.381072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372830+0000","last_change":"2024-05-27T09:31:20.860789+0000","last_active":"2024-05-27T09:35:14.372830+0000","last_peered":"2024-05-27T09:35:14.372830+0000","last_clean":"2024-05-27T09:35:14.372830+0000","last_became_active":"2024-05-27T09:31:20.860369+0000","last_became_peered":"2024-05-27T09:31:20.860369+0000","last_unstale":"2024-05-27T09:35:14.372830+0000","last_undegraded":"2024-05-27T09:35:14.372830+0000","last_fullsized":"2024-05-27T09:35:14.372830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:05:35.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241584+0000","last_change":"2024-05-27T09:31:24.918239+0000","last_active":"2024-05-27T09:35:14.241584+0000","last_peered":"2024-05-27T09:35:14.241584+0000","last_clean":"2024-05-27T09:35:14.241584+0000","last_became_active":"2024-05-27T09:31:24.917750+0000","last_became_peered":"2024-05-27T09:31:24.917750+0000","last_unstale":"2024-05-27T09:35:14.241584+0000","last_undegraded":"2024-05-27T09:35:14.241584+0000","last_fullsized":"2024-05-27T09:35:14.241584+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:07.162994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129596+0000","last_change":"2024-05-27T09:31:20.865308+0000","last_active":"2024-05-27T09:35:14.129596+0000","last_peered":"2024-05-27T09:35:14.129596+0000","last_clean":"2024-05-27T09:35:14.129596+0000","last_became_active":"2024-05-27T09:31:20.864920+0000","last_became_peered":"2024-05-27T09:31:20.864920+0000","last_unstale":"2024-05-27T09:35:14.129596+0000","last_undegraded":"2024-05-27T09:35:14.129596+0000","last_fullsized":"2024-05-27T09:35:14.129596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:36.773322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.902533+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.902136+0000","last_became_peered":"2024-05-27T09:31:24.902136+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:49:32.203577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'8","reported_seq":256,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:18.250624+0000","last_change":"2024-05-27T09:31:20.846060+0000","last_active":"2024-05-27T09:35:18.250624+0000","last_peered":"2024-05-27T09:35:18.250624+0000","last_clean":"2024-05-27T09:35:18.250624+0000","last_became_active":"2024-05-27T09:31:20.845632+0000","last_became_peered":"2024-05-27T09:31:20.845632+0000","last_unstale":"2024-05-27T09:35:18.250624+0000","last_undegraded":"2024-05-27T09:35:18.250624+0000","last_fullsized":"2024-05-27T09:35:18.250624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:55.166579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2208,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":220,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353158+0000","last_change":"2024-05-27T09:31:24.895866+0000","last_active":"2024-05-27T09:35:14.353158+0000","last_peered":"2024-05-27T09:35:14.353158+0000","last_clean":"2024-05-27T09:35:14.353158+0000","last_became_active":"2024-05-27T09:31:24.895413+0000","last_became_peered":"2024-05-27T09:31:24.895413+0000","last_unstale":"2024-05-27T09:35:14.353158+0000","last_undegraded":"2024-05-27T09:35:14.353158+0000","last_fullsized":"2024-05-27T09:35:14.353158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:01:30.075710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":45,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169708+0000","last_change":"2024-05-27T09:31:20.879086+0000","last_active":"2024-05-27T09:35:14.169708+0000","last_peered":"2024-05-27T09:35:14.169708+0000","last_clean":"2024-05-27T09:35:14.169708+0000","last_became_active":"2024-05-27T09:31:20.878331+0000","last_became_peered":"2024-05-27T09:31:20.878331+0000","last_unstale":"2024-05-27T09:35:14.169708+0000","last_undegraded":"2024-05-27T09:35:14.169708+0000","last_fullsized":"2024-05-27T09:35:14.169708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:31.169812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129641+0000","last_change":"2024-05-27T09:31:24.895069+0000","last_active":"2024-05-27T09:35:14.129641+0000","last_peered":"2024-05-27T09:35:14.129641+0000","last_clean":"2024-05-27T09:35:14.129641+0000","last_became_active":"2024-05-27T09:31:24.894626+0000","last_became_peered":"2024-05-27T09:31:24.894626+0000","last_unstale":"2024-05-27T09:35:14.129641+0000","last_undegraded":"2024-05-27T09:35:14.129641+0000","last_fullsized":"2024-05-27T09:35:14.129641+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:21:22.725268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170168+0000","last_change":"2024-05-27T09:31:20.877416+0000","last_active":"2024-05-27T09:35:14.170168+0000","last_peered":"2024-05-27T09:35:14.170168+0000","last_clean":"2024-05-27T09:35:14.170168+0000","last_became_active":"2024-05-27T09:31:20.876093+0000","last_became_peered":"2024-05-27T09:31:20.876093+0000","last_unstale":"2024-05-27T09:35:14.170168+0000","last_undegraded":"2024-05-27T09:35:14.170168+0000","last_fullsized":"2024-05-27T09:35:14.170168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:34.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840141+0000","last_change":"2024-05-27T09:31:24.898430+0000","last_active":"2024-05-27T09:35:14.840141+0000","last_peered":"2024-05-27T09:35:14.840141+0000","last_clean":"2024-05-27T09:35:14.840141+0000","last_became_active":"2024-05-27T09:31:24.897643+0000","last_became_peered":"2024-05-27T09:31:24.897643+0000","last_unstale":"2024-05-27T09:35:14.840141+0000","last_undegraded":"2024-05-27T09:35:14.840141+0000","last_fullsized":"2024-05-27T09:35:14.840141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:44.485561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046011+0000","last_change":"2024-05-27T09:31:20.858253+0000","last_active":"2024-05-27T09:35:14.046011+0000","last_peered":"2024-05-27T09:35:14.046011+0000","last_clean":"2024-05-27T09:35:14.046011+0000","last_became_active":"2024-05-27T09:31:20.857736+0000","last_became_peered":"2024-05-27T09:31:20.857736+0000","last_unstale":"2024-05-27T09:35:14.046011+0000","last_undegraded":"2024-05-27T09:35:14.046011+0000","last_fullsized":"2024-05-27T09:35:14.046011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:53:48.301763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126918+0000","last_change":"2024-05-27T09:31:24.912788+0000","last_active":"2024-05-27T09:35:14.126918+0000","last_peered":"2024-05-27T09:35:14.126918+0000","last_clean":"2024-05-27T09:35:14.126918+0000","last_became_active":"2024-05-27T09:31:24.912366+0000","last_became_peered":"2024-05-27T09:31:24.912366+0000","last_unstale":"2024-05-27T09:35:14.126918+0000","last_undegraded":"2024-05-27T09:35:14.126918+0000","last_fullsized":"2024-05-27T09:35:14.126918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:55:03.397578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353364+0000","last_change":"2024-05-27T09:31:20.862188+0000","last_active":"2024-05-27T09:35:14.353364+0000","last_peered":"2024-05-27T09:35:14.353364+0000","last_clean":"2024-05-27T09:35:14.353364+0000","last_became_active":"2024-05-27T09:31:20.861791+0000","last_became_peered":"2024-05-27T09:31:20.861791+0000","last_unstale":"2024-05-27T09:35:14.353364+0000","last_undegraded":"2024-05-27T09:35:14.353364+0000","last_fullsized":"2024-05-27T09:35:14.353364+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:16:09.127223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126652+0000","last_change":"2024-05-27T09:31:24.898460+0000","last_active":"2024-05-27T09:35:14.126652+0000","last_peered":"2024-05-27T09:35:14.126652+0000","last_clean":"2024-05-27T09:35:14.126652+0000","last_became_active":"2024-05-27T09:31:24.896787+0000","last_became_peered":"2024-05-27T09:31:24.896787+0000","last_unstale":"2024-05-27T09:35:14.126652+0000","last_undegraded":"2024-05-27T09:35:14.126652+0000","last_fullsized":"2024-05-27T09:35:14.126652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:10.199227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129938+0000","last_change":"2024-05-27T09:31:24.896323+0000","last_active":"2024-05-27T09:35:14.129938+0000","last_peered":"2024-05-27T09:35:14.129938+0000","last_clean":"2024-05-27T09:35:14.129938+0000","last_became_active":"2024-05-27T09:31:24.895919+0000","last_became_peered":"2024-05-27T09:31:24.895919+0000","last_unstale":"2024-05-27T09:35:14.129938+0000","last_undegraded":"2024-05-27T09:35:14.129938+0000","last_fullsized":"2024-05-27T09:35:14.129938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:04.883166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":248,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:18.249913+0000","last_change":"2024-05-27T09:31:20.862126+0000","last_active":"2024-05-27T09:35:18.249913+0000","last_peered":"2024-05-27T09:35:18.249913+0000","last_clean":"2024-05-27T09:35:18.249913+0000","last_became_active":"2024-05-27T09:31:20.858964+0000","last_became_peered":"2024-05-27T09:31:20.858964+0000","last_unstale":"2024-05-27T09:35:18.249913+0000","last_undegraded":"2024-05-27T09:35:18.249913+0000","last_fullsized":"2024-05-27T09:35:18.249913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:51:26.838559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":213,"num_read_kb":213,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171127+0000","last_change":"2024-05-27T09:31:24.905549+0000","last_active":"2024-05-27T09:35:14.171127+0000","last_peered":"2024-05-27T09:35:14.171127+0000","last_clean":"2024-05-27T09:35:14.171127+0000","last_became_active":"2024-05-27T09:31:24.905212+0000","last_became_peered":"2024-05-27T09:31:24.905212+0000","last_unstale":"2024-05-27T09:35:14.171127+0000","last_undegraded":"2024-05-27T09:35:14.171127+0000","last_fullsized":"2024-05-27T09:35:14.171127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:18.267807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043259+0000","last_change":"2024-05-27T09:31:20.848527+0000","last_active":"2024-05-27T09:35:14.043259+0000","last_peered":"2024-05-27T09:35:14.043259+0000","last_clean":"2024-05-27T09:35:14.043259+0000","last_became_active":"2024-05-27T09:31:20.847779+0000","last_became_peered":"2024-05-27T09:31:20.847779+0000","last_unstale":"2024-05-27T09:35:14.043259+0000","last_undegraded":"2024-05-27T09:35:14.043259+0000","last_fullsized":"2024-05-27T09:35:14.043259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:15:01.131306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"82'99","reported_seq":133,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:16.190224+0000","last_change":"2024-05-27T09:29:46.120380+0000","last_active":"2024-05-27T09:35:16.190224+0000","last_peered":"2024-05-27T09:35:16.190224+0000","last_clean":"2024-05-27T09:35:16.190224+0000","last_became_active":"2024-05-27T09:29:46.113658+0000","last_became_peered":"2024-05-27T09:29:46.113658+0000","last_unstale":"2024-05-27T09:35:16.190224+0000","last_undegraded":"2024-05-27T09:35:16.190224+0000","last_fullsized":"2024-05-27T09:35:16.190224+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_clean_scrub_stamp":"2024-05-27T09:25:47.244521+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:28:40.424824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241501+0000","last_change":"2024-05-27T09:31:24.900736+0000","last_active":"2024-05-27T09:35:14.241501+0000","last_peered":"2024-05-27T09:35:14.241501+0000","last_clean":"2024-05-27T09:35:14.241501+0000","last_became_active":"2024-05-27T09:31:24.900354+0000","last_became_peered":"2024-05-27T09:31:24.900354+0000","last_unstale":"2024-05-27T09:35:14.241501+0000","last_undegraded":"2024-05-27T09:35:14.241501+0000","last_fullsized":"2024-05-27T09:35:14.241501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:39:33.171977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839843+0000","last_change":"2024-05-27T09:31:57.873603+0000","last_active":"2024-05-27T09:35:14.839843+0000","last_peered":"2024-05-27T09:35:14.839843+0000","last_clean":"2024-05-27T09:35:14.839843+0000","last_became_active":"2024-05-27T09:31:57.868616+0000","last_became_peered":"2024-05-27T09:31:57.868616+0000","last_unstale":"2024-05-27T09:35:14.839843+0000","last_undegraded":"2024-05-27T09:35:14.839843+0000","last_fullsized":"2024-05-27T09:35:14.839843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:02:20.112667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170091+0000","last_change":"2024-05-27T09:31:20.878557+0000","last_active":"2024-05-27T09:35:14.170091+0000","last_peered":"2024-05-27T09:35:14.170091+0000","last_clean":"2024-05-27T09:35:14.170091+0000","last_became_active":"2024-05-27T09:31:20.877969+0000","last_became_peered":"2024-05-27T09:31:20.877969+0000","last_unstale":"2024-05-27T09:35:14.170091+0000","last_undegraded":"2024-05-27T09:35:14.170091+0000","last_fullsized":"2024-05-27T09:35:14.170091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:46.859645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242211+0000","last_change":"2024-05-27T09:31:24.907906+0000","last_active":"2024-05-27T09:35:14.242211+0000","last_peered":"2024-05-27T09:35:14.242211+0000","last_clean":"2024-05-27T09:35:14.242211+0000","last_became_active":"2024-05-27T09:31:24.907473+0000","last_became_peered":"2024-05-27T09:31:24.907473+0000","last_unstale":"2024-05-27T09:35:14.242211+0000","last_undegraded":"2024-05-27T09:35:14.242211+0000","last_fullsized":"2024-05-27T09:35:14.242211+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:11:38.536919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'49","reported_seq":101,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.979146+0000","last_change":"2024-05-27T09:31:20.881250+0000","last_active":"2024-05-27T09:35:14.979146+0000","last_peered":"2024-05-27T09:35:14.979146+0000","last_clean":"2024-05-27T09:35:14.979146+0000","last_became_active":"2024-05-27T09:31:20.880550+0000","last_became_peered":"2024-05-27T09:31:20.880550+0000","last_unstale":"2024-05-27T09:35:14.979146+0000","last_undegraded":"2024-05-27T09:35:14.979146+0000","last_fullsized":"2024-05-27T09:35:14.979146+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:14.783260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":88,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171395+0000","last_change":"2024-05-27T09:31:24.894554+0000","last_active":"2024-05-27T09:35:14.171395+0000","last_peered":"2024-05-27T09:35:14.171395+0000","last_clean":"2024-05-27T09:35:14.171395+0000","last_became_active":"2024-05-27T09:31:24.893800+0000","last_became_peered":"2024-05-27T09:31:24.893800+0000","last_unstale":"2024-05-27T09:35:14.171395+0000","last_undegraded":"2024-05-27T09:35:14.171395+0000","last_fullsized":"2024-05-27T09:35:14.171395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:13.368892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043584+0000","last_change":"2024-05-27T09:31:21.054476+0000","last_active":"2024-05-27T09:35:14.043584+0000","last_peered":"2024-05-27T09:35:14.043584+0000","last_clean":"2024-05-27T09:35:14.043584+0000","last_became_active":"2024-05-27T09:31:21.054091+0000","last_became_peered":"2024-05-27T09:31:21.054091+0000","last_unstale":"2024-05-27T09:35:14.043584+0000","last_undegraded":"2024-05-27T09:35:14.043584+0000","last_fullsized":"2024-05-27T09:35:14.043584+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:15.909230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043020+0000","last_change":"2024-05-27T09:31:24.913157+0000","last_active":"2024-05-27T09:35:14.043020+0000","last_peered":"2024-05-27T09:35:14.043020+0000","last_clean":"2024-05-27T09:35:14.043020+0000","last_became_active":"2024-05-27T09:31:24.912793+0000","last_became_peered":"2024-05-27T09:31:24.912793+0000","last_unstale":"2024-05-27T09:35:14.043020+0000","last_undegraded":"2024-05-27T09:35:14.043020+0000","last_fullsized":"2024-05-27T09:35:14.043020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:17:36.451069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372678+0000","last_change":"2024-05-27T09:31:20.861349+0000","last_active":"2024-05-27T09:35:14.372678+0000","last_peered":"2024-05-27T09:35:14.372678+0000","last_clean":"2024-05-27T09:35:14.372678+0000","last_became_active":"2024-05-27T09:31:20.860896+0000","last_became_peered":"2024-05-27T09:31:20.860896+0000","last_unstale":"2024-05-27T09:35:14.372678+0000","last_undegraded":"2024-05-27T09:35:14.372678+0000","last_fullsized":"2024-05-27T09:35:14.372678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:00.772910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169685+0000","last_change":"2024-05-27T09:31:24.893240+0000","last_active":"2024-05-27T09:35:14.169685+0000","last_peered":"2024-05-27T09:35:14.169685+0000","last_clean":"2024-05-27T09:35:14.169685+0000","last_became_active":"2024-05-27T09:31:24.892169+0000","last_became_peered":"2024-05-27T09:31:24.892169+0000","last_unstale":"2024-05-27T09:35:14.169685+0000","last_undegraded":"2024-05-27T09:35:14.169685+0000","last_fullsized":"2024-05-27T09:35:14.169685+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:26:59.660725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170715+0000","last_change":"2024-05-27T09:31:20.864307+0000","last_active":"2024-05-27T09:35:14.170715+0000","last_peered":"2024-05-27T09:35:14.170715+0000","last_clean":"2024-05-27T09:35:14.170715+0000","last_became_active":"2024-05-27T09:31:20.863913+0000","last_became_peered":"2024-05-27T09:31:20.863913+0000","last_unstale":"2024-05-27T09:35:14.170715+0000","last_undegraded":"2024-05-27T09:35:14.170715+0000","last_fullsized":"2024-05-27T09:35:14.170715+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:58:57.656731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839950+0000","last_change":"2024-05-27T09:31:24.905728+0000","last_active":"2024-05-27T09:35:14.839950+0000","last_peered":"2024-05-27T09:35:14.839950+0000","last_clean":"2024-05-27T09:35:14.839950+0000","last_became_active":"2024-05-27T09:31:24.905365+0000","last_became_peered":"2024-05-27T09:31:24.905365+0000","last_unstale":"2024-05-27T09:35:14.839950+0000","last_undegraded":"2024-05-27T09:35:14.839950+0000","last_fullsized":"2024-05-27T09:35:14.839950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:05.013037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373037+0000","last_change":"2024-05-27T09:31:20.878478+0000","last_active":"2024-05-27T09:35:14.373037+0000","last_peered":"2024-05-27T09:35:14.373037+0000","last_clean":"2024-05-27T09:35:14.373037+0000","last_became_active":"2024-05-27T09:31:20.877952+0000","last_became_peered":"2024-05-27T09:31:20.877952+0000","last_unstale":"2024-05-27T09:35:14.373037+0000","last_undegraded":"2024-05-27T09:35:14.373037+0000","last_fullsized":"2024-05-27T09:35:14.373037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:16:27.580641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171301+0000","last_change":"2024-05-27T09:31:24.904967+0000","last_active":"2024-05-27T09:35:14.171301+0000","last_peered":"2024-05-27T09:35:14.171301+0000","last_clean":"2024-05-27T09:35:14.171301+0000","last_became_active":"2024-05-27T09:31:24.904542+0000","last_became_peered":"2024-05-27T09:31:24.904542+0000","last_unstale":"2024-05-27T09:35:14.171301+0000","last_undegraded":"2024-05-27T09:35:14.171301+0000","last_fullsized":"2024-05-27T09:35:14.171301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:01:05.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169841+0000","last_change":"2024-05-27T09:31:57.875713+0000","last_active":"2024-05-27T09:35:14.169841+0000","last_peered":"2024-05-27T09:35:14.169841+0000","last_clean":"2024-05-27T09:35:14.169841+0000","last_became_active":"2024-05-27T09:31:57.870244+0000","last_became_peered":"2024-05-27T09:31:57.870244+0000","last_unstale":"2024-05-27T09:35:14.169841+0000","last_undegraded":"2024-05-27T09:35:14.169841+0000","last_fullsized":"2024-05-27T09:35:14.169841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:49.272418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353395+0000","last_change":"2024-05-27T09:31:24.892655+0000","last_active":"2024-05-27T09:35:14.353395+0000","last_peered":"2024-05-27T09:35:14.353395+0000","last_clean":"2024-05-27T09:35:14.353395+0000","last_became_active":"2024-05-27T09:31:24.892112+0000","last_became_peered":"2024-05-27T09:31:24.892112+0000","last_unstale":"2024-05-27T09:35:14.353395+0000","last_undegraded":"2024-05-27T09:35:14.353395+0000","last_fullsized":"2024-05-27T09:35:14.353395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.604118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043417+0000","last_change":"2024-05-27T09:31:57.875213+0000","last_active":"2024-05-27T09:35:14.043417+0000","last_peered":"2024-05-27T09:35:14.043417+0000","last_clean":"2024-05-27T09:35:14.043417+0000","last_became_active":"2024-05-27T09:31:57.869025+0000","last_became_peered":"2024-05-27T09:31:57.869025+0000","last_unstale":"2024-05-27T09:35:14.043417+0000","last_undegraded":"2024-05-27T09:35:14.043417+0000","last_fullsized":"2024-05-27T09:35:14.043417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:51.433347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353725+0000","last_change":"2024-05-27T09:31:24.899130+0000","last_active":"2024-05-27T09:35:14.353725+0000","last_peered":"2024-05-27T09:35:14.353725+0000","last_clean":"2024-05-27T09:35:14.353725+0000","last_became_active":"2024-05-27T09:31:24.898722+0000","last_became_peered":"2024-05-27T09:31:24.898722+0000","last_unstale":"2024-05-27T09:35:14.353725+0000","last_undegraded":"2024-05-27T09:35:14.353725+0000","last_fullsized":"2024-05-27T09:35:14.353725+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:51.247275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'39","reported_seq":76,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.314140+0000","last_change":"2024-05-27T09:31:20.863757+0000","last_active":"2024-05-27T09:35:14.314140+0000","last_peered":"2024-05-27T09:35:14.314140+0000","last_clean":"2024-05-27T09:35:14.314140+0000","last_became_active":"2024-05-27T09:31:20.863465+0000","last_became_peered":"2024-05-27T09:31:20.863465+0000","last_unstale":"2024-05-27T09:35:14.314140+0000","last_undegraded":"2024-05-27T09:35:14.314140+0000","last_fullsized":"2024-05-27T09:35:14.314140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:21:45.741822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.913749+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.913384+0000","last_became_peered":"2024-05-27T09:31:24.913384+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:24:29.826096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242122+0000","last_change":"2024-05-27T09:31:20.865844+0000","last_active":"2024-05-27T09:35:14.242122+0000","last_peered":"2024-05-27T09:35:14.242122+0000","last_clean":"2024-05-27T09:35:14.242122+0000","last_became_active":"2024-05-27T09:31:20.865364+0000","last_became_peered":"2024-05-27T09:31:20.865364+0000","last_unstale":"2024-05-27T09:35:14.242122+0000","last_undegraded":"2024-05-27T09:35:14.242122+0000","last_fullsized":"2024-05-27T09:35:14.242122+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:11:54.451294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353821+0000","last_change":"2024-05-27T09:31:24.890326+0000","last_active":"2024-05-27T09:35:14.353821+0000","last_peered":"2024-05-27T09:35:14.353821+0000","last_clean":"2024-05-27T09:35:14.353821+0000","last_became_active":"2024-05-27T09:31:24.889789+0000","last_became_peered":"2024-05-27T09:31:24.889789+0000","last_unstale":"2024-05-27T09:35:14.353821+0000","last_undegraded":"2024-05-27T09:35:14.353821+0000","last_fullsized":"2024-05-27T09:35:14.353821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:34:04.252231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840005+0000","last_change":"2024-05-27T09:31:24.896730+0000","last_active":"2024-05-27T09:35:14.840005+0000","last_peered":"2024-05-27T09:35:14.840005+0000","last_clean":"2024-05-27T09:35:14.840005+0000","last_became_active":"2024-05-27T09:31:24.896365+0000","last_became_peered":"2024-05-27T09:31:24.896365+0000","last_unstale":"2024-05-27T09:35:14.840005+0000","last_undegraded":"2024-05-27T09:35:14.840005+0000","last_fullsized":"2024-05-27T09:35:14.840005+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:37.443788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372837+0000","last_change":"2024-05-27T09:31:24.899215+0000","last_active":"2024-05-27T09:35:14.372837+0000","last_peered":"2024-05-27T09:35:14.372837+0000","last_clean":"2024-05-27T09:35:14.372837+0000","last_became_active":"2024-05-27T09:31:24.898733+0000","last_became_peered":"2024-05-27T09:31:24.898733+0000","last_unstale":"2024-05-27T09:35:14.372837+0000","last_undegraded":"2024-05-27T09:35:14.372837+0000","last_fullsized":"2024-05-27T09:35:14.372837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:10.085989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372709+0000","last_change":"2024-05-27T09:31:24.914922+0000","last_active":"2024-05-27T09:35:14.372709+0000","last_peered":"2024-05-27T09:35:14.372709+0000","last_clean":"2024-05-27T09:35:14.372709+0000","last_became_active":"2024-05-27T09:31:24.914541+0000","last_became_peered":"2024-05-27T09:31:24.914541+0000","last_unstale":"2024-05-27T09:35:14.372709+0000","last_undegraded":"2024-05-27T09:35:14.372709+0000","last_fullsized":"2024-05-27T09:35:14.372709+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:30:38.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042498+0000","last_change":"2024-05-27T09:31:24.909628+0000","last_active":"2024-05-27T09:35:14.042498+0000","last_peered":"2024-05-27T09:35:14.042498+0000","last_clean":"2024-05-27T09:35:14.042498+0000","last_became_active":"2024-05-27T09:31:24.902931+0000","last_became_peered":"2024-05-27T09:31:24.902931+0000","last_unstale":"2024-05-27T09:35:14.042498+0000","last_undegraded":"2024-05-27T09:35:14.042498+0000","last_fullsized":"2024-05-27T09:35:14.042498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:06.711316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242398+0000","last_change":"2024-05-27T09:31:24.912099+0000","last_active":"2024-05-27T09:35:14.242398+0000","last_peered":"2024-05-27T09:35:14.242398+0000","last_clean":"2024-05-27T09:35:14.242398+0000","last_became_active":"2024-05-27T09:31:24.911669+0000","last_became_peered":"2024-05-27T09:31:24.911669+0000","last_unstale":"2024-05-27T09:35:14.242398+0000","last_undegraded":"2024-05-27T09:35:14.242398+0000","last_fullsized":"2024-05-27T09:35:14.242398+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:58.068589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840085+0000","last_change":"2024-05-27T09:31:24.891426+0000","last_active":"2024-05-27T09:35:14.840085+0000","last_peered":"2024-05-27T09:35:14.840085+0000","last_clean":"2024-05-27T09:35:14.840085+0000","last_became_active":"2024-05-27T09:31:24.890877+0000","last_became_peered":"2024-05-27T09:31:24.890877+0000","last_unstale":"2024-05-27T09:35:14.840085+0000","last_undegraded":"2024-05-27T09:35:14.840085+0000","last_fullsized":"2024-05-27T09:35:14.840085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:15.338188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373079+0000","last_change":"2024-05-27T09:31:24.892021+0000","last_active":"2024-05-27T09:35:14.373079+0000","last_peered":"2024-05-27T09:35:14.373079+0000","last_clean":"2024-05-27T09:35:14.373079+0000","last_became_active":"2024-05-27T09:31:24.891607+0000","last_became_peered":"2024-05-27T09:31:24.891607+0000","last_unstale":"2024-05-27T09:35:14.373079+0000","last_undegraded":"2024-05-27T09:35:14.373079+0000","last_fullsized":"2024-05-27T09:35:14.373079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:05.623199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171321+0000","last_change":"2024-05-27T09:31:24.912166+0000","last_active":"2024-05-27T09:35:14.171321+0000","last_peered":"2024-05-27T09:35:14.171321+0000","last_clean":"2024-05-27T09:35:14.171321+0000","last_became_active":"2024-05-27T09:31:24.907555+0000","last_became_peered":"2024-05-27T09:31:24.907555+0000","last_unstale":"2024-05-27T09:35:14.171321+0000","last_undegraded":"2024-05-27T09:35:14.171321+0000","last_fullsized":"2024-05-27T09:35:14.171321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:55:33.772321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129575+0000","last_change":"2024-05-27T09:31:24.910525+0000","last_active":"2024-05-27T09:35:14.129575+0000","last_peered":"2024-05-27T09:35:14.129575+0000","last_clean":"2024-05-27T09:35:14.129575+0000","last_became_active":"2024-05-27T09:31:24.910089+0000","last_became_peered":"2024-05-27T09:31:24.910089+0000","last_unstale":"2024-05-27T09:35:14.129575+0000","last_undegraded":"2024-05-27T09:35:14.129575+0000","last_fullsized":"2024-05-27T09:35:14.129575+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:19:02.274781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170634+0000","last_change":"2024-05-27T09:31:24.901125+0000","last_active":"2024-05-27T09:35:14.170634+0000","last_peered":"2024-05-27T09:35:14.170634+0000","last_clean":"2024-05-27T09:35:14.170634+0000","last_became_active":"2024-05-27T09:31:24.900361+0000","last_became_peered":"2024-05-27T09:31:24.900361+0000","last_unstale":"2024-05-27T09:35:14.170634+0000","last_undegraded":"2024-05-27T09:35:14.170634+0000","last_fullsized":"2024-05-27T09:35:14.170634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:30.116349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372869+0000","last_change":"2024-05-27T09:31:24.917612+0000","last_active":"2024-05-27T09:35:14.372869+0000","last_peered":"2024-05-27T09:35:14.372869+0000","last_clean":"2024-05-27T09:35:14.372869+0000","last_became_active":"2024-05-27T09:31:24.917210+0000","last_became_peered":"2024-05-27T09:31:24.917210+0000","last_unstale":"2024-05-27T09:35:14.372869+0000","last_undegraded":"2024-05-27T09:35:14.372869+0000","last_fullsized":"2024-05-27T09:35:14.372869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:19:00.769114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241872+0000","last_change":"2024-05-27T09:31:24.909654+0000","last_active":"2024-05-27T09:35:14.241872+0000","last_peered":"2024-05-27T09:35:14.241872+0000","last_clean":"2024-05-27T09:35:14.241872+0000","last_became_active":"2024-05-27T09:31:24.909143+0000","last_became_peered":"2024-05-27T09:31:24.909143+0000","last_unstale":"2024-05-27T09:35:14.241872+0000","last_undegraded":"2024-05-27T09:35:14.241872+0000","last_fullsized":"2024-05-27T09:35:14.241872+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:42.844614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840108+0000","last_change":"2024-05-27T09:31:24.899031+0000","last_active":"2024-05-27T09:35:14.840108+0000","last_peered":"2024-05-27T09:35:14.840108+0000","last_clean":"2024-05-27T09:35:14.840108+0000","last_became_active":"2024-05-27T09:31:24.898664+0000","last_became_peered":"2024-05-27T09:31:24.898664+0000","last_unstale":"2024-05-27T09:35:14.840108+0000","last_undegraded":"2024-05-27T09:35:14.840108+0000","last_fullsized":"2024-05-27T09:35:14.840108+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:09.659646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600937,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1406,"num_read_kb":817,"num_write":301,"num_write_kb":1447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1802811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678080,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":892928,"data_stored":440736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874308,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41188,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703964,"statfs":{"total":95995035648,"available":95952859136,"internally_reserved":0,"allocated":1732608,"data_stored":1235405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070538,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":834188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266768,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":834108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462998,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28484,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716668,"statfs":{"total":95995035648,"available":95965868032,"internally_reserved":0,"allocated":1503232,"data_stored":1034974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626523,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":897024,"data_stored":440361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574970,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":510122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738493,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":441632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411445,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":511813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182508,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28424,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716728,"statfs":{"total":95995035648,"available":95965929472,"internally_reserved":0,"allocated":1441792,"data_stored":965488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986280,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":443000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822755,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36948,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708204,"statfs":{"total":95995035648,"available":95957200896,"internally_reserved":0,"allocated":1650688,"data_stored":1164638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":204766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T09:35:22.394 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-27T09:35:22.394 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T09:35:22.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.715+0000 7fd358800640 1 -- 172.21.15.44:0/731920920 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd350109c20 msgr2=0x7fd3501037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:22.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.715+0000 7fd358800640 1 --2- 172.21.15.44:0/731920920 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd350109c20 0x7fd3501037c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd340009860 tx=0x7fd34002a340 comp rx=0 tx=0).stop 2024-05-27T09:35:22.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.715+0000 7fd358800640 1 -- 172.21.15.44:0/731920920 shutdown_connections 2024-05-27T09:35:22.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.715+0000 7fd358800640 1 --2- 172.21.15.44:0/731920920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd350104370 0x7fd350110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.715+0000 7fd358800640 1 --2- 172.21.15.44:0/731920920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35010a0c0 0x7fd350103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.715+0000 7fd358800640 1 --2- 172.21.15.44:0/731920920 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd350109c20 0x7fd3501037c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.715+0000 7fd358800640 1 -- 172.21.15.44:0/731920920 >> 172.21.15.44:0/731920920 conn(0x7fd350079360 msgr2=0x7fd3500797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:22.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.715+0000 7fd358800640 1 -- 172.21.15.44:0/731920920 shutdown_connections 2024-05-27T09:35:22.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.715+0000 7fd358800640 1 -- 172.21.15.44:0/731920920 wait complete. 2024-05-27T09:35:22.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.716+0000 7fd358800640 1 Processor -- start 2024-05-27T09:35:22.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd358800640 1 -- start start 2024-05-27T09:35:22.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd358800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd350104370 0x7fd35019d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:22.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd358800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd350109c20 0x7fd35019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:22.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd358800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010a0c0 0x7fd3501a8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:22.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd358800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd350113e10 con 0x7fd35010a0c0 2024-05-27T09:35:22.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd358800640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd350113c90 con 0x7fd350109c20 2024-05-27T09:35:22.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd358800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd350113f90 con 0x7fd350104370 2024-05-27T09:35:22.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd355400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd350109c20 0x7fd35019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:22.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd355e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd350104370 0x7fd35019d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:22.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd355400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd350109c20 0x7fd35019ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43094/0 (socket says 172.21.15.44:43094) 2024-05-27T09:35:22.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd356800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010a0c0 0x7fd3501a8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:22.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.717+0000 7fd355400640 1 -- 172.21.15.44:0/3865166759 learned_addr learned my addr 172.21.15.44:0/3865166759 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:22.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.718+0000 7fd355e00640 1 -- 172.21.15.44:0/3865166759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd350109c20 msgr2=0x7fd35019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:22.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.718+0000 7fd355e00640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd350109c20 0x7fd35019ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.718+0000 7fd355e00640 1 -- 172.21.15.44:0/3865166759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010a0c0 msgr2=0x7fd3501a8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:22.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.718+0000 7fd355e00640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010a0c0 0x7fd3501a8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.718+0000 7fd355e00640 1 -- 172.21.15.44:0/3865166759 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3501a8c90 con 0x7fd350104370 2024-05-27T09:35:22.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.718+0000 7fd356800640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010a0c0 0x7fd3501a8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:22.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.718+0000 7fd355e00640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd350104370 0x7fd35019d8b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd340035040 tx=0x7fd340032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:22.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.719+0000 7fd34aa00640 1 -- 172.21.15.44:0/3865166759 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340040020 con 0x7fd350104370 2024-05-27T09:35:22.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.719+0000 7fd34aa00640 1 -- 172.21.15.44:0/3865166759 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd340030e10 con 0x7fd350104370 2024-05-27T09:35:22.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.719+0000 7fd34aa00640 1 -- 172.21.15.44:0/3865166759 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340037040 con 0x7fd350104370 2024-05-27T09:35:22.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.719+0000 7fd358800640 1 -- 172.21.15.44:0/3865166759 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3501a8ec0 con 0x7fd350104370 2024-05-27T09:35:22.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.719+0000 7fd358800640 1 -- 172.21.15.44:0/3865166759 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3501a92d0 con 0x7fd350104370 2024-05-27T09:35:22.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.720+0000 7fd358800640 1 -- 172.21.15.44:0/3865166759 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd35006a230 con 0x7fd350104370 2024-05-27T09:35:22.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.722+0000 7fd34aa00640 1 -- 172.21.15.44:0/3865166759 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7fd340043020 con 0x7fd350104370 2024-05-27T09:35:22.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.724+0000 7fd34aa00640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3380b2b00 0x7fd3380b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:22.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.724+0000 7fd34aa00640 1 -- 172.21.15.44:0/3865166759 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fd340102ab0 con 0x7fd350104370 2024-05-27T09:35:22.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.724+0000 7fd355400640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3380b2b00 0x7fd3380b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:22.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.725+0000 7fd355400640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3380b2b00 0x7fd3380b5010 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd350076c90 tx=0x7fd344005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:22.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.726+0000 7fd34aa00640 1 -- 172.21.15.44:0/3865166759 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd34004a020 con 0x7fd350104370 2024-05-27T09:35:22.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.928+0000 7fd358800640 1 -- 172.21.15.44:0/3865166759 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd3501029f0 con 0x7fd3380b2b00 2024-05-27T09:35:22.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.940+0000 7fd34aa00640 1 -- 172.21.15.44:0/3865166759 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257738 (secure 0 0 0) 0x7fd3501029f0 con 0x7fd3380b2b00 2024-05-27T09:35:22.946 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:22.947 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-05-27T09:35:22.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.946+0000 7fd358800640 1 -- 172.21.15.44:0/3865166759 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3380b2b00 msgr2=0x7fd3380b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:22.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.946+0000 7fd358800640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3380b2b00 0x7fd3380b5010 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd350076c90 tx=0x7fd344005f90 comp rx=0 tx=0).stop 2024-05-27T09:35:22.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.946+0000 7fd358800640 1 -- 172.21.15.44:0/3865166759 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd350104370 msgr2=0x7fd35019d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:22.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.946+0000 7fd358800640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd350104370 0x7fd35019d8b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd340035040 tx=0x7fd340032d50 comp rx=0 tx=0).stop 2024-05-27T09:35:22.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.946+0000 7fd358800640 1 -- 172.21.15.44:0/3865166759 shutdown_connections 2024-05-27T09:35:22.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.946+0000 7fd358800640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3380b2b00 0x7fd3380b5010 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.946+0000 7fd358800640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010a0c0 0x7fd3501a8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.946+0000 7fd358800640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd350109c20 0x7fd35019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.946+0000 7fd358800640 1 --2- 172.21.15.44:0/3865166759 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd350104370 0x7fd35019d8b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:22.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.946+0000 7fd358800640 1 -- 172.21.15.44:0/3865166759 >> 172.21.15.44:0/3865166759 conn(0x7fd350079360 msgr2=0x7fd3501022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:22.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.947+0000 7fd358800640 1 -- 172.21.15.44:0/3865166759 shutdown_connections 2024-05-27T09:35:22.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:22.947+0000 7fd358800640 1 -- 172.21.15.44:0/3865166759 wait complete. 2024-05-27T09:35:22.974 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-05-27T09:35:22.069143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191539,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1550,"num_read_kb":1065,"num_write":438,"num_write_kb":3599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":837,"ondisk_log_size":837,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384144,"kb_used_data":14160,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557680,"statfs":{"total":1151940427776,"available":1151547064320,"internally_reserved":0,"allocated":14499840,"data_stored":8856465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2911,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":97,"num_read_kb":206,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.966333"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840251+0000","last_change":"2024-05-27T09:31:24.899109+0000","last_active":"2024-05-27T09:35:14.840251+0000","last_peered":"2024-05-27T09:35:14.840251+0000","last_clean":"2024-05-27T09:35:14.840251+0000","last_became_active":"2024-05-27T09:31:24.898783+0000","last_became_peered":"2024-05-27T09:31:24.898783+0000","last_unstale":"2024-05-27T09:35:14.840251+0000","last_undegraded":"2024-05-27T09:35:14.840251+0000","last_fullsized":"2024-05-27T09:35:14.840251+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:01:12.220389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042262+0000","last_change":"2024-05-27T09:31:24.898382+0000","last_active":"2024-05-27T09:35:14.042262+0000","last_peered":"2024-05-27T09:35:14.042262+0000","last_clean":"2024-05-27T09:35:14.042262+0000","last_became_active":"2024-05-27T09:31:24.897975+0000","last_became_peered":"2024-05-27T09:31:24.897975+0000","last_unstale":"2024-05-27T09:35:14.042262+0000","last_undegraded":"2024-05-27T09:35:14.042262+0000","last_fullsized":"2024-05-27T09:35:14.042262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:38.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041822+0000","last_change":"2024-05-27T09:31:24.896122+0000","last_active":"2024-05-27T09:35:14.041822+0000","last_peered":"2024-05-27T09:35:14.041822+0000","last_clean":"2024-05-27T09:35:14.041822+0000","last_became_active":"2024-05-27T09:31:24.890867+0000","last_became_peered":"2024-05-27T09:31:24.890867+0000","last_unstale":"2024-05-27T09:35:14.041822+0000","last_undegraded":"2024-05-27T09:35:14.041822+0000","last_fullsized":"2024-05-27T09:35:14.041822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:09.982177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046136+0000","last_change":"2024-05-27T09:31:57.868900+0000","last_active":"2024-05-27T09:35:14.046136+0000","last_peered":"2024-05-27T09:35:14.046136+0000","last_clean":"2024-05-27T09:35:14.046136+0000","last_became_active":"2024-05-27T09:31:57.868418+0000","last_became_peered":"2024-05-27T09:31:57.868418+0000","last_unstale":"2024-05-27T09:35:14.046136+0000","last_undegraded":"2024-05-27T09:35:14.046136+0000","last_fullsized":"2024-05-27T09:35:14.046136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:08.966704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169866+0000","last_change":"2024-05-27T09:31:24.910283+0000","last_active":"2024-05-27T09:35:14.169866+0000","last_peered":"2024-05-27T09:35:14.169866+0000","last_clean":"2024-05-27T09:35:14.169866+0000","last_became_active":"2024-05-27T09:31:24.909882+0000","last_became_peered":"2024-05-27T09:31:24.909882+0000","last_unstale":"2024-05-27T09:35:14.169866+0000","last_undegraded":"2024-05-27T09:35:14.169866+0000","last_fullsized":"2024-05-27T09:35:14.169866+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:43.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126676+0000","last_change":"2024-05-27T09:31:24.900934+0000","last_active":"2024-05-27T09:35:14.126676+0000","last_peered":"2024-05-27T09:35:14.126676+0000","last_clean":"2024-05-27T09:35:14.126676+0000","last_became_active":"2024-05-27T09:31:24.900534+0000","last_became_peered":"2024-05-27T09:31:24.900534+0000","last_unstale":"2024-05-27T09:35:14.126676+0000","last_undegraded":"2024-05-27T09:35:14.126676+0000","last_fullsized":"2024-05-27T09:35:14.126676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:53.318327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353192+0000","last_change":"2024-05-27T09:31:24.907080+0000","last_active":"2024-05-27T09:35:14.353192+0000","last_peered":"2024-05-27T09:35:14.353192+0000","last_clean":"2024-05-27T09:35:14.353192+0000","last_became_active":"2024-05-27T09:31:24.906715+0000","last_became_peered":"2024-05-27T09:31:24.906715+0000","last_unstale":"2024-05-27T09:35:14.353192+0000","last_undegraded":"2024-05-27T09:35:14.353192+0000","last_fullsized":"2024-05-27T09:35:14.353192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:36.214659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.044350+0000","last_change":"2024-05-27T09:31:58.200608+0000","last_active":"2024-05-27T09:35:14.044350+0000","last_peered":"2024-05-27T09:35:14.044350+0000","last_clean":"2024-05-27T09:35:14.044350+0000","last_became_active":"2024-05-27T09:31:58.199363+0000","last_became_peered":"2024-05-27T09:31:58.199363+0000","last_unstale":"2024-05-27T09:35:14.044350+0000","last_undegraded":"2024-05-27T09:35:14.044350+0000","last_fullsized":"2024-05-27T09:35:14.044350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:30:46.495930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373508+0000","last_change":"2024-05-27T09:31:24.892630+0000","last_active":"2024-05-27T09:35:14.373508+0000","last_peered":"2024-05-27T09:35:14.373508+0000","last_clean":"2024-05-27T09:35:14.373508+0000","last_became_active":"2024-05-27T09:31:24.892176+0000","last_became_peered":"2024-05-27T09:31:24.892176+0000","last_unstale":"2024-05-27T09:35:14.373508+0000","last_undegraded":"2024-05-27T09:35:14.373508+0000","last_fullsized":"2024-05-27T09:35:14.373508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:49:20.585601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.127155+0000","last_change":"2024-05-27T09:31:24.895281+0000","last_active":"2024-05-27T09:35:14.127155+0000","last_peered":"2024-05-27T09:35:14.127155+0000","last_clean":"2024-05-27T09:35:14.127155+0000","last_became_active":"2024-05-27T09:31:24.894886+0000","last_became_peered":"2024-05-27T09:31:24.894886+0000","last_unstale":"2024-05-27T09:35:14.127155+0000","last_undegraded":"2024-05-27T09:35:14.127155+0000","last_fullsized":"2024-05-27T09:35:14.127155+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:41.494434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353559+0000","last_change":"2024-05-27T09:31:24.900167+0000","last_active":"2024-05-27T09:35:14.353559+0000","last_peered":"2024-05-27T09:35:14.353559+0000","last_clean":"2024-05-27T09:35:14.353559+0000","last_became_active":"2024-05-27T09:31:24.899750+0000","last_became_peered":"2024-05-27T09:31:24.899750+0000","last_unstale":"2024-05-27T09:35:14.353559+0000","last_undegraded":"2024-05-27T09:35:14.353559+0000","last_fullsized":"2024-05-27T09:35:14.353559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:30.602246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.045840+0000","last_change":"2024-05-27T09:31:57.868818+0000","last_active":"2024-05-27T09:35:14.045840+0000","last_peered":"2024-05-27T09:35:14.045840+0000","last_clean":"2024-05-27T09:35:14.045840+0000","last_became_active":"2024-05-27T09:31:57.868322+0000","last_became_peered":"2024-05-27T09:31:57.868322+0000","last_unstale":"2024-05-27T09:35:14.045840+0000","last_undegraded":"2024-05-27T09:35:14.045840+0000","last_fullsized":"2024-05-27T09:35:14.045840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:35.304850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242036+0000","last_change":"2024-05-27T09:31:24.902537+0000","last_active":"2024-05-27T09:35:14.242036+0000","last_peered":"2024-05-27T09:35:14.242036+0000","last_clean":"2024-05-27T09:35:14.242036+0000","last_became_active":"2024-05-27T09:31:24.902132+0000","last_became_peered":"2024-05-27T09:31:24.902132+0000","last_unstale":"2024-05-27T09:35:14.242036+0000","last_undegraded":"2024-05-27T09:35:14.242036+0000","last_fullsized":"2024-05-27T09:35:14.242036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:20:50.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041926+0000","last_change":"2024-05-27T09:31:24.899182+0000","last_active":"2024-05-27T09:35:14.041926+0000","last_peered":"2024-05-27T09:35:14.041926+0000","last_clean":"2024-05-27T09:35:14.041926+0000","last_became_active":"2024-05-27T09:31:24.898749+0000","last_became_peered":"2024-05-27T09:31:24.898749+0000","last_unstale":"2024-05-27T09:35:14.041926+0000","last_undegraded":"2024-05-27T09:35:14.041926+0000","last_fullsized":"2024-05-27T09:35:14.041926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:44:44.844357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041660+0000","last_change":"2024-05-27T09:31:24.901115+0000","last_active":"2024-05-27T09:35:14.041660+0000","last_peered":"2024-05-27T09:35:14.041660+0000","last_clean":"2024-05-27T09:35:14.041660+0000","last_became_active":"2024-05-27T09:31:24.897011+0000","last_became_peered":"2024-05-27T09:31:24.897011+0000","last_unstale":"2024-05-27T09:35:14.041660+0000","last_undegraded":"2024-05-27T09:35:14.041660+0000","last_fullsized":"2024-05-27T09:35:14.041660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:01.130407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372690+0000","last_change":"2024-05-27T09:31:24.903063+0000","last_active":"2024-05-27T09:35:14.372690+0000","last_peered":"2024-05-27T09:35:14.372690+0000","last_clean":"2024-05-27T09:35:14.372690+0000","last_became_active":"2024-05-27T09:31:24.899766+0000","last_became_peered":"2024-05-27T09:31:24.899766+0000","last_unstale":"2024-05-27T09:35:14.372690+0000","last_undegraded":"2024-05-27T09:35:14.372690+0000","last_fullsized":"2024-05-27T09:35:14.372690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:45.984226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840388+0000","last_change":"2024-05-27T09:31:24.915922+0000","last_active":"2024-05-27T09:35:14.840388+0000","last_peered":"2024-05-27T09:35:14.840388+0000","last_clean":"2024-05-27T09:35:14.840388+0000","last_became_active":"2024-05-27T09:31:24.915623+0000","last_became_peered":"2024-05-27T09:31:24.915623+0000","last_unstale":"2024-05-27T09:35:14.840388+0000","last_undegraded":"2024-05-27T09:35:14.840388+0000","last_fullsized":"2024-05-27T09:35:14.840388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:02:01.006704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046094+0000","last_change":"2024-05-27T09:31:24.908397+0000","last_active":"2024-05-27T09:35:14.046094+0000","last_peered":"2024-05-27T09:35:14.046094+0000","last_clean":"2024-05-27T09:35:14.046094+0000","last_became_active":"2024-05-27T09:31:24.903644+0000","last_became_peered":"2024-05-27T09:31:24.903644+0000","last_unstale":"2024-05-27T09:35:14.046094+0000","last_undegraded":"2024-05-27T09:35:14.046094+0000","last_fullsized":"2024-05-27T09:35:14.046094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:09.281429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373078+0000","last_change":"2024-05-27T09:31:24.912706+0000","last_active":"2024-05-27T09:35:14.373078+0000","last_peered":"2024-05-27T09:35:14.373078+0000","last_clean":"2024-05-27T09:35:14.373078+0000","last_became_active":"2024-05-27T09:31:24.912357+0000","last_became_peered":"2024-05-27T09:31:24.912357+0000","last_unstale":"2024-05-27T09:35:14.373078+0000","last_undegraded":"2024-05-27T09:35:14.373078+0000","last_fullsized":"2024-05-27T09:35:14.373078+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:23.334041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.045022+0000","last_change":"2024-05-27T09:31:20.881652+0000","last_active":"2024-05-27T09:35:14.045022+0000","last_peered":"2024-05-27T09:35:14.045022+0000","last_clean":"2024-05-27T09:35:14.045022+0000","last_became_active":"2024-05-27T09:31:20.880757+0000","last_became_peered":"2024-05-27T09:31:20.880757+0000","last_unstale":"2024-05-27T09:35:14.045022+0000","last_undegraded":"2024-05-27T09:35:14.045022+0000","last_fullsized":"2024-05-27T09:35:14.045022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:01.433755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171024+0000","last_change":"2024-05-27T09:31:24.908010+0000","last_active":"2024-05-27T09:35:14.171024+0000","last_peered":"2024-05-27T09:35:14.171024+0000","last_clean":"2024-05-27T09:35:14.171024+0000","last_became_active":"2024-05-27T09:31:24.907638+0000","last_became_peered":"2024-05-27T09:31:24.907638+0000","last_unstale":"2024-05-27T09:35:14.171024+0000","last_undegraded":"2024-05-27T09:35:14.171024+0000","last_fullsized":"2024-05-27T09:35:14.171024+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:02:49.207709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171395+0000","last_change":"2024-05-27T09:31:20.860556+0000","last_active":"2024-05-27T09:35:14.171395+0000","last_peered":"2024-05-27T09:35:14.171395+0000","last_clean":"2024-05-27T09:35:14.171395+0000","last_became_active":"2024-05-27T09:31:20.860172+0000","last_became_peered":"2024-05-27T09:31:20.860172+0000","last_unstale":"2024-05-27T09:35:14.171395+0000","last_undegraded":"2024-05-27T09:35:14.171395+0000","last_fullsized":"2024-05-27T09:35:14.171395+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:08:21.545681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042957+0000","last_change":"2024-05-27T09:31:24.910111+0000","last_active":"2024-05-27T09:35:14.042957+0000","last_peered":"2024-05-27T09:35:14.042957+0000","last_clean":"2024-05-27T09:35:14.042957+0000","last_became_active":"2024-05-27T09:31:24.908985+0000","last_became_peered":"2024-05-27T09:31:24.908985+0000","last_unstale":"2024-05-27T09:35:14.042957+0000","last_undegraded":"2024-05-27T09:35:14.042957+0000","last_fullsized":"2024-05-27T09:35:14.042957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:53:29.645792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170222+0000","last_change":"2024-05-27T09:31:20.879993+0000","last_active":"2024-05-27T09:35:14.170222+0000","last_peered":"2024-05-27T09:35:14.170222+0000","last_clean":"2024-05-27T09:35:14.170222+0000","last_became_active":"2024-05-27T09:31:20.878802+0000","last_became_peered":"2024-05-27T09:31:20.878802+0000","last_unstale":"2024-05-27T09:35:14.170222+0000","last_undegraded":"2024-05-27T09:35:14.170222+0000","last_fullsized":"2024-05-27T09:35:14.170222+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:15.555612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372970+0000","last_change":"2024-05-27T09:31:24.903533+0000","last_active":"2024-05-27T09:35:14.372970+0000","last_peered":"2024-05-27T09:35:14.372970+0000","last_clean":"2024-05-27T09:35:14.372970+0000","last_became_active":"2024-05-27T09:31:24.903166+0000","last_became_peered":"2024-05-27T09:31:24.903166+0000","last_unstale":"2024-05-27T09:35:14.372970+0000","last_undegraded":"2024-05-27T09:35:14.372970+0000","last_fullsized":"2024-05-27T09:35:14.372970+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:52.539470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839887+0000","last_change":"2024-05-27T09:31:57.878426+0000","last_active":"2024-05-27T09:35:14.839887+0000","last_peered":"2024-05-27T09:35:14.839887+0000","last_clean":"2024-05-27T09:35:14.839887+0000","last_became_active":"2024-05-27T09:31:57.869146+0000","last_became_peered":"2024-05-27T09:31:57.869146+0000","last_unstale":"2024-05-27T09:35:14.839887+0000","last_undegraded":"2024-05-27T09:35:14.839887+0000","last_fullsized":"2024-05-27T09:35:14.839887+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:34:51.926092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129913+0000","last_change":"2024-05-27T09:31:24.897426+0000","last_active":"2024-05-27T09:35:14.129913+0000","last_peered":"2024-05-27T09:35:14.129913+0000","last_clean":"2024-05-27T09:35:14.129913+0000","last_became_active":"2024-05-27T09:31:24.897080+0000","last_became_peered":"2024-05-27T09:31:24.897080+0000","last_unstale":"2024-05-27T09:35:14.129913+0000","last_undegraded":"2024-05-27T09:35:14.129913+0000","last_fullsized":"2024-05-27T09:35:14.129913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:15:42.428391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'8","reported_seq":255,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:18.136294+0000","last_change":"2024-05-27T09:31:20.846681+0000","last_active":"2024-05-27T09:35:18.136294+0000","last_peered":"2024-05-27T09:35:18.136294+0000","last_clean":"2024-05-27T09:35:18.136294+0000","last_became_active":"2024-05-27T09:31:20.846302+0000","last_became_peered":"2024-05-27T09:31:20.846302+0000","last_unstale":"2024-05-27T09:35:18.136294+0000","last_undegraded":"2024-05-27T09:35:18.136294+0000","last_fullsized":"2024-05-27T09:35:18.136294+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:19.609612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":218,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169711+0000","last_change":"2024-05-27T09:31:24.907209+0000","last_active":"2024-05-27T09:35:14.169711+0000","last_peered":"2024-05-27T09:35:14.169711+0000","last_clean":"2024-05-27T09:35:14.169711+0000","last_became_active":"2024-05-27T09:31:24.906752+0000","last_became_peered":"2024-05-27T09:31:24.906752+0000","last_unstale":"2024-05-27T09:35:14.169711+0000","last_undegraded":"2024-05-27T09:35:14.169711+0000","last_fullsized":"2024-05-27T09:35:14.169711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:53.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126569+0000","last_change":"2024-05-27T09:31:20.879778+0000","last_active":"2024-05-27T09:35:14.126569+0000","last_peered":"2024-05-27T09:35:14.126569+0000","last_clean":"2024-05-27T09:35:14.126569+0000","last_became_active":"2024-05-27T09:31:20.879259+0000","last_became_peered":"2024-05-27T09:31:20.879259+0000","last_unstale":"2024-05-27T09:35:14.126569+0000","last_undegraded":"2024-05-27T09:35:14.126569+0000","last_fullsized":"2024-05-27T09:35:14.126569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:20:07.972653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129702+0000","last_change":"2024-05-27T09:31:58.195516+0000","last_active":"2024-05-27T09:35:14.129702+0000","last_peered":"2024-05-27T09:35:14.129702+0000","last_clean":"2024-05-27T09:35:14.129702+0000","last_became_active":"2024-05-27T09:31:58.195118+0000","last_became_peered":"2024-05-27T09:31:58.195118+0000","last_unstale":"2024-05-27T09:35:14.129702+0000","last_undegraded":"2024-05-27T09:35:14.129702+0000","last_fullsized":"2024-05-27T09:35:14.129702+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:11:23.547605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241846+0000","last_change":"2024-05-27T09:31:20.863152+0000","last_active":"2024-05-27T09:35:14.241846+0000","last_peered":"2024-05-27T09:35:14.241846+0000","last_clean":"2024-05-27T09:35:14.241846+0000","last_became_active":"2024-05-27T09:31:20.862708+0000","last_became_peered":"2024-05-27T09:31:20.862708+0000","last_unstale":"2024-05-27T09:35:14.241846+0000","last_undegraded":"2024-05-27T09:35:14.241846+0000","last_fullsized":"2024-05-27T09:35:14.241846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:21.645773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170544+0000","last_change":"2024-05-27T09:31:24.893988+0000","last_active":"2024-05-27T09:35:14.170544+0000","last_peered":"2024-05-27T09:35:14.170544+0000","last_clean":"2024-05-27T09:35:14.170544+0000","last_became_active":"2024-05-27T09:31:24.893504+0000","last_became_peered":"2024-05-27T09:31:24.893504+0000","last_unstale":"2024-05-27T09:35:14.170544+0000","last_undegraded":"2024-05-27T09:35:14.170544+0000","last_fullsized":"2024-05-27T09:35:14.170544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:47.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129578+0000","last_change":"2024-05-27T09:31:20.845555+0000","last_active":"2024-05-27T09:35:14.129578+0000","last_peered":"2024-05-27T09:35:14.129578+0000","last_clean":"2024-05-27T09:35:14.129578+0000","last_became_active":"2024-05-27T09:31:20.845139+0000","last_became_peered":"2024-05-27T09:31:20.845139+0000","last_unstale":"2024-05-27T09:35:14.129578+0000","last_undegraded":"2024-05-27T09:35:14.129578+0000","last_fullsized":"2024-05-27T09:35:14.129578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:25.917845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.049161+0000","last_change":"2024-05-27T09:31:24.916142+0000","last_active":"2024-05-27T09:35:14.049161+0000","last_peered":"2024-05-27T09:35:14.049161+0000","last_clean":"2024-05-27T09:35:14.049161+0000","last_became_active":"2024-05-27T09:31:24.915709+0000","last_became_peered":"2024-05-27T09:31:24.915709+0000","last_unstale":"2024-05-27T09:35:14.049161+0000","last_undegraded":"2024-05-27T09:35:14.049161+0000","last_fullsized":"2024-05-27T09:35:14.049161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:38.029648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'5","reported_seq":247,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:18.135754+0000","last_change":"2024-05-27T09:31:20.859240+0000","last_active":"2024-05-27T09:35:18.135754+0000","last_peered":"2024-05-27T09:35:18.135754+0000","last_clean":"2024-05-27T09:35:18.135754+0000","last_became_active":"2024-05-27T09:31:20.858798+0000","last_became_peered":"2024-05-27T09:31:20.858798+0000","last_unstale":"2024-05-27T09:35:18.135754+0000","last_undegraded":"2024-05-27T09:35:18.135754+0000","last_fullsized":"2024-05-27T09:35:18.135754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:42.280165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":213,"num_read_kb":213,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373425+0000","last_change":"2024-05-27T09:31:24.908818+0000","last_active":"2024-05-27T09:35:14.373425+0000","last_peered":"2024-05-27T09:35:14.373425+0000","last_clean":"2024-05-27T09:35:14.373425+0000","last_became_active":"2024-05-27T09:31:24.908410+0000","last_became_peered":"2024-05-27T09:31:24.908410+0000","last_unstale":"2024-05-27T09:35:14.373425+0000","last_undegraded":"2024-05-27T09:35:14.373425+0000","last_fullsized":"2024-05-27T09:35:14.373425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:05:40.326232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353642+0000","last_change":"2024-05-27T09:31:20.848632+0000","last_active":"2024-05-27T09:35:14.353642+0000","last_peered":"2024-05-27T09:35:14.353642+0000","last_clean":"2024-05-27T09:35:14.353642+0000","last_became_active":"2024-05-27T09:31:20.847558+0000","last_became_peered":"2024-05-27T09:31:20.847558+0000","last_unstale":"2024-05-27T09:35:14.353642+0000","last_undegraded":"2024-05-27T09:35:14.353642+0000","last_fullsized":"2024-05-27T09:35:14.353642+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:53.930565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043056+0000","last_change":"2024-05-27T09:31:24.901513+0000","last_active":"2024-05-27T09:35:14.043056+0000","last_peered":"2024-05-27T09:35:14.043056+0000","last_clean":"2024-05-27T09:35:14.043056+0000","last_became_active":"2024-05-27T09:31:24.895924+0000","last_became_peered":"2024-05-27T09:31:24.895924+0000","last_unstale":"2024-05-27T09:35:14.043056+0000","last_undegraded":"2024-05-27T09:35:14.043056+0000","last_fullsized":"2024-05-27T09:35:14.043056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:27:39.939098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.130061+0000","last_change":"2024-05-27T09:31:20.855195+0000","last_active":"2024-05-27T09:35:14.130061+0000","last_peered":"2024-05-27T09:35:14.130061+0000","last_clean":"2024-05-27T09:35:14.130061+0000","last_became_active":"2024-05-27T09:31:20.854910+0000","last_became_peered":"2024-05-27T09:31:20.854910+0000","last_unstale":"2024-05-27T09:35:14.130061+0000","last_undegraded":"2024-05-27T09:35:14.130061+0000","last_fullsized":"2024-05-27T09:35:14.130061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:02.795103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373077+0000","last_change":"2024-05-27T09:31:24.895887+0000","last_active":"2024-05-27T09:35:14.373077+0000","last_peered":"2024-05-27T09:35:14.373077+0000","last_clean":"2024-05-27T09:35:14.373077+0000","last_became_active":"2024-05-27T09:31:24.894622+0000","last_became_peered":"2024-05-27T09:31:24.894622+0000","last_unstale":"2024-05-27T09:35:14.373077+0000","last_undegraded":"2024-05-27T09:35:14.373077+0000","last_fullsized":"2024-05-27T09:35:14.373077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:32.833946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'38","reported_seq":505,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:19.137271+0000","last_change":"2024-05-27T09:31:20.846063+0000","last_active":"2024-05-27T09:35:19.137271+0000","last_peered":"2024-05-27T09:35:19.137271+0000","last_clean":"2024-05-27T09:35:19.137271+0000","last_became_active":"2024-05-27T09:31:20.845401+0000","last_became_peered":"2024-05-27T09:31:20.845401+0000","last_unstale":"2024-05-27T09:35:19.137271+0000","last_undegraded":"2024-05-27T09:35:19.137271+0000","last_fullsized":"2024-05-27T09:35:19.137271+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:21:38.460183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69968,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":438,"num_read_kb":288,"num_write":39,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170004+0000","last_change":"2024-05-27T09:31:24.892748+0000","last_active":"2024-05-27T09:35:14.170004+0000","last_peered":"2024-05-27T09:35:14.170004+0000","last_clean":"2024-05-27T09:35:14.170004+0000","last_became_active":"2024-05-27T09:31:24.892394+0000","last_became_peered":"2024-05-27T09:31:24.892394+0000","last_unstale":"2024-05-27T09:35:14.170004+0000","last_undegraded":"2024-05-27T09:35:14.170004+0000","last_fullsized":"2024-05-27T09:35:14.170004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:39:31.381072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372830+0000","last_change":"2024-05-27T09:31:20.860789+0000","last_active":"2024-05-27T09:35:14.372830+0000","last_peered":"2024-05-27T09:35:14.372830+0000","last_clean":"2024-05-27T09:35:14.372830+0000","last_became_active":"2024-05-27T09:31:20.860369+0000","last_became_peered":"2024-05-27T09:31:20.860369+0000","last_unstale":"2024-05-27T09:35:14.372830+0000","last_undegraded":"2024-05-27T09:35:14.372830+0000","last_fullsized":"2024-05-27T09:35:14.372830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:05:35.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241584+0000","last_change":"2024-05-27T09:31:24.918239+0000","last_active":"2024-05-27T09:35:14.241584+0000","last_peered":"2024-05-27T09:35:14.241584+0000","last_clean":"2024-05-27T09:35:14.241584+0000","last_became_active":"2024-05-27T09:31:24.917750+0000","last_became_peered":"2024-05-27T09:31:24.917750+0000","last_unstale":"2024-05-27T09:35:14.241584+0000","last_undegraded":"2024-05-27T09:35:14.241584+0000","last_fullsized":"2024-05-27T09:35:14.241584+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:07.162994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129596+0000","last_change":"2024-05-27T09:31:20.865308+0000","last_active":"2024-05-27T09:35:14.129596+0000","last_peered":"2024-05-27T09:35:14.129596+0000","last_clean":"2024-05-27T09:35:14.129596+0000","last_became_active":"2024-05-27T09:31:20.864920+0000","last_became_peered":"2024-05-27T09:31:20.864920+0000","last_unstale":"2024-05-27T09:35:14.129596+0000","last_undegraded":"2024-05-27T09:35:14.129596+0000","last_fullsized":"2024-05-27T09:35:14.129596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:36.773322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.902533+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.902136+0000","last_became_peered":"2024-05-27T09:31:24.902136+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:49:32.203577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'8","reported_seq":256,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:18.250624+0000","last_change":"2024-05-27T09:31:20.846060+0000","last_active":"2024-05-27T09:35:18.250624+0000","last_peered":"2024-05-27T09:35:18.250624+0000","last_clean":"2024-05-27T09:35:18.250624+0000","last_became_active":"2024-05-27T09:31:20.845632+0000","last_became_peered":"2024-05-27T09:31:20.845632+0000","last_unstale":"2024-05-27T09:35:18.250624+0000","last_undegraded":"2024-05-27T09:35:18.250624+0000","last_fullsized":"2024-05-27T09:35:18.250624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:55.166579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2208,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":220,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353158+0000","last_change":"2024-05-27T09:31:24.895866+0000","last_active":"2024-05-27T09:35:14.353158+0000","last_peered":"2024-05-27T09:35:14.353158+0000","last_clean":"2024-05-27T09:35:14.353158+0000","last_became_active":"2024-05-27T09:31:24.895413+0000","last_became_peered":"2024-05-27T09:31:24.895413+0000","last_unstale":"2024-05-27T09:35:14.353158+0000","last_undegraded":"2024-05-27T09:35:14.353158+0000","last_fullsized":"2024-05-27T09:35:14.353158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:01:30.075710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":45,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169708+0000","last_change":"2024-05-27T09:31:20.879086+0000","last_active":"2024-05-27T09:35:14.169708+0000","last_peered":"2024-05-27T09:35:14.169708+0000","last_clean":"2024-05-27T09:35:14.169708+0000","last_became_active":"2024-05-27T09:31:20.878331+0000","last_became_peered":"2024-05-27T09:31:20.878331+0000","last_unstale":"2024-05-27T09:35:14.169708+0000","last_undegraded":"2024-05-27T09:35:14.169708+0000","last_fullsized":"2024-05-27T09:35:14.169708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:31.169812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129641+0000","last_change":"2024-05-27T09:31:24.895069+0000","last_active":"2024-05-27T09:35:14.129641+0000","last_peered":"2024-05-27T09:35:14.129641+0000","last_clean":"2024-05-27T09:35:14.129641+0000","last_became_active":"2024-05-27T09:31:24.894626+0000","last_became_peered":"2024-05-27T09:31:24.894626+0000","last_unstale":"2024-05-27T09:35:14.129641+0000","last_undegraded":"2024-05-27T09:35:14.129641+0000","last_fullsized":"2024-05-27T09:35:14.129641+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:21:22.725268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170168+0000","last_change":"2024-05-27T09:31:20.877416+0000","last_active":"2024-05-27T09:35:14.170168+0000","last_peered":"2024-05-27T09:35:14.170168+0000","last_clean":"2024-05-27T09:35:14.170168+0000","last_became_active":"2024-05-27T09:31:20.876093+0000","last_became_peered":"2024-05-27T09:31:20.876093+0000","last_unstale":"2024-05-27T09:35:14.170168+0000","last_undegraded":"2024-05-27T09:35:14.170168+0000","last_fullsized":"2024-05-27T09:35:14.170168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:34.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840141+0000","last_change":"2024-05-27T09:31:24.898430+0000","last_active":"2024-05-27T09:35:14.840141+0000","last_peered":"2024-05-27T09:35:14.840141+0000","last_clean":"2024-05-27T09:35:14.840141+0000","last_became_active":"2024-05-27T09:31:24.897643+0000","last_became_peered":"2024-05-27T09:31:24.897643+0000","last_unstale":"2024-05-27T09:35:14.840141+0000","last_undegraded":"2024-05-27T09:35:14.840141+0000","last_fullsized":"2024-05-27T09:35:14.840141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:44.485561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046011+0000","last_change":"2024-05-27T09:31:20.858253+0000","last_active":"2024-05-27T09:35:14.046011+0000","last_peered":"2024-05-27T09:35:14.046011+0000","last_clean":"2024-05-27T09:35:14.046011+0000","last_became_active":"2024-05-27T09:31:20.857736+0000","last_became_peered":"2024-05-27T09:31:20.857736+0000","last_unstale":"2024-05-27T09:35:14.046011+0000","last_undegraded":"2024-05-27T09:35:14.046011+0000","last_fullsized":"2024-05-27T09:35:14.046011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:53:48.301763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126918+0000","last_change":"2024-05-27T09:31:24.912788+0000","last_active":"2024-05-27T09:35:14.126918+0000","last_peered":"2024-05-27T09:35:14.126918+0000","last_clean":"2024-05-27T09:35:14.126918+0000","last_became_active":"2024-05-27T09:31:24.912366+0000","last_became_peered":"2024-05-27T09:31:24.912366+0000","last_unstale":"2024-05-27T09:35:14.126918+0000","last_undegraded":"2024-05-27T09:35:14.126918+0000","last_fullsized":"2024-05-27T09:35:14.126918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:55:03.397578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353364+0000","last_change":"2024-05-27T09:31:20.862188+0000","last_active":"2024-05-27T09:35:14.353364+0000","last_peered":"2024-05-27T09:35:14.353364+0000","last_clean":"2024-05-27T09:35:14.353364+0000","last_became_active":"2024-05-27T09:31:20.861791+0000","last_became_peered":"2024-05-27T09:31:20.861791+0000","last_unstale":"2024-05-27T09:35:14.353364+0000","last_undegraded":"2024-05-27T09:35:14.353364+0000","last_fullsized":"2024-05-27T09:35:14.353364+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:16:09.127223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126652+0000","last_change":"2024-05-27T09:31:24.898460+0000","last_active":"2024-05-27T09:35:14.126652+0000","last_peered":"2024-05-27T09:35:14.126652+0000","last_clean":"2024-05-27T09:35:14.126652+0000","last_became_active":"2024-05-27T09:31:24.896787+0000","last_became_peered":"2024-05-27T09:31:24.896787+0000","last_unstale":"2024-05-27T09:35:14.126652+0000","last_undegraded":"2024-05-27T09:35:14.126652+0000","last_fullsized":"2024-05-27T09:35:14.126652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:10.199227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129938+0000","last_change":"2024-05-27T09:31:24.896323+0000","last_active":"2024-05-27T09:35:14.129938+0000","last_peered":"2024-05-27T09:35:14.129938+0000","last_clean":"2024-05-27T09:35:14.129938+0000","last_became_active":"2024-05-27T09:31:24.895919+0000","last_became_peered":"2024-05-27T09:31:24.895919+0000","last_unstale":"2024-05-27T09:35:14.129938+0000","last_undegraded":"2024-05-27T09:35:14.129938+0000","last_fullsized":"2024-05-27T09:35:14.129938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:04.883166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":248,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:18.249913+0000","last_change":"2024-05-27T09:31:20.862126+0000","last_active":"2024-05-27T09:35:18.249913+0000","last_peered":"2024-05-27T09:35:18.249913+0000","last_clean":"2024-05-27T09:35:18.249913+0000","last_became_active":"2024-05-27T09:31:20.858964+0000","last_became_peered":"2024-05-27T09:31:20.858964+0000","last_unstale":"2024-05-27T09:35:18.249913+0000","last_undegraded":"2024-05-27T09:35:18.249913+0000","last_fullsized":"2024-05-27T09:35:18.249913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:51:26.838559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":213,"num_read_kb":213,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171127+0000","last_change":"2024-05-27T09:31:24.905549+0000","last_active":"2024-05-27T09:35:14.171127+0000","last_peered":"2024-05-27T09:35:14.171127+0000","last_clean":"2024-05-27T09:35:14.171127+0000","last_became_active":"2024-05-27T09:31:24.905212+0000","last_became_peered":"2024-05-27T09:31:24.905212+0000","last_unstale":"2024-05-27T09:35:14.171127+0000","last_undegraded":"2024-05-27T09:35:14.171127+0000","last_fullsized":"2024-05-27T09:35:14.171127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:18.267807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043259+0000","last_change":"2024-05-27T09:31:20.848527+0000","last_active":"2024-05-27T09:35:14.043259+0000","last_peered":"2024-05-27T09:35:14.043259+0000","last_clean":"2024-05-27T09:35:14.043259+0000","last_became_active":"2024-05-27T09:31:20.847779+0000","last_became_peered":"2024-05-27T09:31:20.847779+0000","last_unstale":"2024-05-27T09:35:14.043259+0000","last_undegraded":"2024-05-27T09:35:14.043259+0000","last_fullsized":"2024-05-27T09:35:14.043259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:15:01.131306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"82'99","reported_seq":133,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:16.190224+0000","last_change":"2024-05-27T09:29:46.120380+0000","last_active":"2024-05-27T09:35:16.190224+0000","last_peered":"2024-05-27T09:35:16.190224+0000","last_clean":"2024-05-27T09:35:16.190224+0000","last_became_active":"2024-05-27T09:29:46.113658+0000","last_became_peered":"2024-05-27T09:29:46.113658+0000","last_unstale":"2024-05-27T09:35:16.190224+0000","last_undegraded":"2024-05-27T09:35:16.190224+0000","last_fullsized":"2024-05-27T09:35:16.190224+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_clean_scrub_stamp":"2024-05-27T09:25:47.244521+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:28:40.424824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241501+0000","last_change":"2024-05-27T09:31:24.900736+0000","last_active":"2024-05-27T09:35:14.241501+0000","last_peered":"2024-05-27T09:35:14.241501+0000","last_clean":"2024-05-27T09:35:14.241501+0000","last_became_active":"2024-05-27T09:31:24.900354+0000","last_became_peered":"2024-05-27T09:31:24.900354+0000","last_unstale":"2024-05-27T09:35:14.241501+0000","last_undegraded":"2024-05-27T09:35:14.241501+0000","last_fullsized":"2024-05-27T09:35:14.241501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:39:33.171977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839843+0000","last_change":"2024-05-27T09:31:57.873603+0000","last_active":"2024-05-27T09:35:14.839843+0000","last_peered":"2024-05-27T09:35:14.839843+0000","last_clean":"2024-05-27T09:35:14.839843+0000","last_became_active":"2024-05-27T09:31:57.868616+0000","last_became_peered":"2024-05-27T09:31:57.868616+0000","last_unstale":"2024-05-27T09:35:14.839843+0000","last_undegraded":"2024-05-27T09:35:14.839843+0000","last_fullsized":"2024-05-27T09:35:14.839843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:02:20.112667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170091+0000","last_change":"2024-05-27T09:31:20.878557+0000","last_active":"2024-05-27T09:35:14.170091+0000","last_peered":"2024-05-27T09:35:14.170091+0000","last_clean":"2024-05-27T09:35:14.170091+0000","last_became_active":"2024-05-27T09:31:20.877969+0000","last_became_peered":"2024-05-27T09:31:20.877969+0000","last_unstale":"2024-05-27T09:35:14.170091+0000","last_undegraded":"2024-05-27T09:35:14.170091+0000","last_fullsized":"2024-05-27T09:35:14.170091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:46.859645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242211+0000","last_change":"2024-05-27T09:31:24.907906+0000","last_active":"2024-05-27T09:35:14.242211+0000","last_peered":"2024-05-27T09:35:14.242211+0000","last_clean":"2024-05-27T09:35:14.242211+0000","last_became_active":"2024-05-27T09:31:24.907473+0000","last_became_peered":"2024-05-27T09:31:24.907473+0000","last_unstale":"2024-05-27T09:35:14.242211+0000","last_undegraded":"2024-05-27T09:35:14.242211+0000","last_fullsized":"2024-05-27T09:35:14.242211+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:11:38.536919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'49","reported_seq":101,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.979146+0000","last_change":"2024-05-27T09:31:20.881250+0000","last_active":"2024-05-27T09:35:14.979146+0000","last_peered":"2024-05-27T09:35:14.979146+0000","last_clean":"2024-05-27T09:35:14.979146+0000","last_became_active":"2024-05-27T09:31:20.880550+0000","last_became_peered":"2024-05-27T09:31:20.880550+0000","last_unstale":"2024-05-27T09:35:14.979146+0000","last_undegraded":"2024-05-27T09:35:14.979146+0000","last_fullsized":"2024-05-27T09:35:14.979146+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:14.783260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":88,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171395+0000","last_change":"2024-05-27T09:31:24.894554+0000","last_active":"2024-05-27T09:35:14.171395+0000","last_peered":"2024-05-27T09:35:14.171395+0000","last_clean":"2024-05-27T09:35:14.171395+0000","last_became_active":"2024-05-27T09:31:24.893800+0000","last_became_peered":"2024-05-27T09:31:24.893800+0000","last_unstale":"2024-05-27T09:35:14.171395+0000","last_undegraded":"2024-05-27T09:35:14.171395+0000","last_fullsized":"2024-05-27T09:35:14.171395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:13.368892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043584+0000","last_change":"2024-05-27T09:31:21.054476+0000","last_active":"2024-05-27T09:35:14.043584+0000","last_peered":"2024-05-27T09:35:14.043584+0000","last_clean":"2024-05-27T09:35:14.043584+0000","last_became_active":"2024-05-27T09:31:21.054091+0000","last_became_peered":"2024-05-27T09:31:21.054091+0000","last_unstale":"2024-05-27T09:35:14.043584+0000","last_undegraded":"2024-05-27T09:35:14.043584+0000","last_fullsized":"2024-05-27T09:35:14.043584+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:15.909230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043020+0000","last_change":"2024-05-27T09:31:24.913157+0000","last_active":"2024-05-27T09:35:14.043020+0000","last_peered":"2024-05-27T09:35:14.043020+0000","last_clean":"2024-05-27T09:35:14.043020+0000","last_became_active":"2024-05-27T09:31:24.912793+0000","last_became_peered":"2024-05-27T09:31:24.912793+0000","last_unstale":"2024-05-27T09:35:14.043020+0000","last_undegraded":"2024-05-27T09:35:14.043020+0000","last_fullsized":"2024-05-27T09:35:14.043020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:17:36.451069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372678+0000","last_change":"2024-05-27T09:31:20.861349+0000","last_active":"2024-05-27T09:35:14.372678+0000","last_peered":"2024-05-27T09:35:14.372678+0000","last_clean":"2024-05-27T09:35:14.372678+0000","last_became_active":"2024-05-27T09:31:20.860896+0000","last_became_peered":"2024-05-27T09:31:20.860896+0000","last_unstale":"2024-05-27T09:35:14.372678+0000","last_undegraded":"2024-05-27T09:35:14.372678+0000","last_fullsized":"2024-05-27T09:35:14.372678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:00.772910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169685+0000","last_change":"2024-05-27T09:31:24.893240+0000","last_active":"2024-05-27T09:35:14.169685+0000","last_peered":"2024-05-27T09:35:14.169685+0000","last_clean":"2024-05-27T09:35:14.169685+0000","last_became_active":"2024-05-27T09:31:24.892169+0000","last_became_peered":"2024-05-27T09:31:24.892169+0000","last_unstale":"2024-05-27T09:35:14.169685+0000","last_undegraded":"2024-05-27T09:35:14.169685+0000","last_fullsized":"2024-05-27T09:35:14.169685+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:26:59.660725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170715+0000","last_change":"2024-05-27T09:31:20.864307+0000","last_active":"2024-05-27T09:35:14.170715+0000","last_peered":"2024-05-27T09:35:14.170715+0000","last_clean":"2024-05-27T09:35:14.170715+0000","last_became_active":"2024-05-27T09:31:20.863913+0000","last_became_peered":"2024-05-27T09:31:20.863913+0000","last_unstale":"2024-05-27T09:35:14.170715+0000","last_undegraded":"2024-05-27T09:35:14.170715+0000","last_fullsized":"2024-05-27T09:35:14.170715+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:58:57.656731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839950+0000","last_change":"2024-05-27T09:31:24.905728+0000","last_active":"2024-05-27T09:35:14.839950+0000","last_peered":"2024-05-27T09:35:14.839950+0000","last_clean":"2024-05-27T09:35:14.839950+0000","last_became_active":"2024-05-27T09:31:24.905365+0000","last_became_peered":"2024-05-27T09:31:24.905365+0000","last_unstale":"2024-05-27T09:35:14.839950+0000","last_undegraded":"2024-05-27T09:35:14.839950+0000","last_fullsized":"2024-05-27T09:35:14.839950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:05.013037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373037+0000","last_change":"2024-05-27T09:31:20.878478+0000","last_active":"2024-05-27T09:35:14.373037+0000","last_peered":"2024-05-27T09:35:14.373037+0000","last_clean":"2024-05-27T09:35:14.373037+0000","last_became_active":"2024-05-27T09:31:20.877952+0000","last_became_peered":"2024-05-27T09:31:20.877952+0000","last_unstale":"2024-05-27T09:35:14.373037+0000","last_undegraded":"2024-05-27T09:35:14.373037+0000","last_fullsized":"2024-05-27T09:35:14.373037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:16:27.580641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171301+0000","last_change":"2024-05-27T09:31:24.904967+0000","last_active":"2024-05-27T09:35:14.171301+0000","last_peered":"2024-05-27T09:35:14.171301+0000","last_clean":"2024-05-27T09:35:14.171301+0000","last_became_active":"2024-05-27T09:31:24.904542+0000","last_became_peered":"2024-05-27T09:31:24.904542+0000","last_unstale":"2024-05-27T09:35:14.171301+0000","last_undegraded":"2024-05-27T09:35:14.171301+0000","last_fullsized":"2024-05-27T09:35:14.171301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:01:05.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169841+0000","last_change":"2024-05-27T09:31:57.875713+0000","last_active":"2024-05-27T09:35:14.169841+0000","last_peered":"2024-05-27T09:35:14.169841+0000","last_clean":"2024-05-27T09:35:14.169841+0000","last_became_active":"2024-05-27T09:31:57.870244+0000","last_became_peered":"2024-05-27T09:31:57.870244+0000","last_unstale":"2024-05-27T09:35:14.169841+0000","last_undegraded":"2024-05-27T09:35:14.169841+0000","last_fullsized":"2024-05-27T09:35:14.169841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:49.272418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353395+0000","last_change":"2024-05-27T09:31:24.892655+0000","last_active":"2024-05-27T09:35:14.353395+0000","last_peered":"2024-05-27T09:35:14.353395+0000","last_clean":"2024-05-27T09:35:14.353395+0000","last_became_active":"2024-05-27T09:31:24.892112+0000","last_became_peered":"2024-05-27T09:31:24.892112+0000","last_unstale":"2024-05-27T09:35:14.353395+0000","last_undegraded":"2024-05-27T09:35:14.353395+0000","last_fullsized":"2024-05-27T09:35:14.353395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.604118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043417+0000","last_change":"2024-05-27T09:31:57.875213+0000","last_active":"2024-05-27T09:35:14.043417+0000","last_peered":"2024-05-27T09:35:14.043417+0000","last_clean":"2024-05-27T09:35:14.043417+0000","last_became_active":"2024-05-27T09:31:57.869025+0000","last_became_peered":"2024-05-27T09:31:57.869025+0000","last_unstale":"2024-05-27T09:35:14.043417+0000","last_undegraded":"2024-05-27T09:35:14.043417+0000","last_fullsized":"2024-05-27T09:35:14.043417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:51.433347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353725+0000","last_change":"2024-05-27T09:31:24.899130+0000","last_active":"2024-05-27T09:35:14.353725+0000","last_peered":"2024-05-27T09:35:14.353725+0000","last_clean":"2024-05-27T09:35:14.353725+0000","last_became_active":"2024-05-27T09:31:24.898722+0000","last_became_peered":"2024-05-27T09:31:24.898722+0000","last_unstale":"2024-05-27T09:35:14.353725+0000","last_undegraded":"2024-05-27T09:35:14.353725+0000","last_fullsized":"2024-05-27T09:35:14.353725+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:51.247275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'39","reported_seq":76,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.314140+0000","last_change":"2024-05-27T09:31:20.863757+0000","last_active":"2024-05-27T09:35:14.314140+0000","last_peered":"2024-05-27T09:35:14.314140+0000","last_clean":"2024-05-27T09:35:14.314140+0000","last_became_active":"2024-05-27T09:31:20.863465+0000","last_became_peered":"2024-05-27T09:31:20.863465+0000","last_unstale":"2024-05-27T09:35:14.314140+0000","last_undegraded":"2024-05-27T09:35:14.314140+0000","last_fullsized":"2024-05-27T09:35:14.314140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:21:45.741822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.913749+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.913384+0000","last_became_peered":"2024-05-27T09:31:24.913384+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:24:29.826096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242122+0000","last_change":"2024-05-27T09:31:20.865844+0000","last_active":"2024-05-27T09:35:14.242122+0000","last_peered":"2024-05-27T09:35:14.242122+0000","last_clean":"2024-05-27T09:35:14.242122+0000","last_became_active":"2024-05-27T09:31:20.865364+0000","last_became_peered":"2024-05-27T09:31:20.865364+0000","last_unstale":"2024-05-27T09:35:14.242122+0000","last_undegraded":"2024-05-27T09:35:14.242122+0000","last_fullsized":"2024-05-27T09:35:14.242122+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:11:54.451294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353821+0000","last_change":"2024-05-27T09:31:24.890326+0000","last_active":"2024-05-27T09:35:14.353821+0000","last_peered":"2024-05-27T09:35:14.353821+0000","last_clean":"2024-05-27T09:35:14.353821+0000","last_became_active":"2024-05-27T09:31:24.889789+0000","last_became_peered":"2024-05-27T09:31:24.889789+0000","last_unstale":"2024-05-27T09:35:14.353821+0000","last_undegraded":"2024-05-27T09:35:14.353821+0000","last_fullsized":"2024-05-27T09:35:14.353821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:34:04.252231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840005+0000","last_change":"2024-05-27T09:31:24.896730+0000","last_active":"2024-05-27T09:35:14.840005+0000","last_peered":"2024-05-27T09:35:14.840005+0000","last_clean":"2024-05-27T09:35:14.840005+0000","last_became_active":"2024-05-27T09:31:24.896365+0000","last_became_peered":"2024-05-27T09:31:24.896365+0000","last_unstale":"2024-05-27T09:35:14.840005+0000","last_undegraded":"2024-05-27T09:35:14.840005+0000","last_fullsized":"2024-05-27T09:35:14.840005+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:37.443788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372837+0000","last_change":"2024-05-27T09:31:24.899215+0000","last_active":"2024-05-27T09:35:14.372837+0000","last_peered":"2024-05-27T09:35:14.372837+0000","last_clean":"2024-05-27T09:35:14.372837+0000","last_became_active":"2024-05-27T09:31:24.898733+0000","last_became_peered":"2024-05-27T09:31:24.898733+0000","last_unstale":"2024-05-27T09:35:14.372837+0000","last_undegraded":"2024-05-27T09:35:14.372837+0000","last_fullsized":"2024-05-27T09:35:14.372837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:10.085989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372709+0000","last_change":"2024-05-27T09:31:24.914922+0000","last_active":"2024-05-27T09:35:14.372709+0000","last_peered":"2024-05-27T09:35:14.372709+0000","last_clean":"2024-05-27T09:35:14.372709+0000","last_became_active":"2024-05-27T09:31:24.914541+0000","last_became_peered":"2024-05-27T09:31:24.914541+0000","last_unstale":"2024-05-27T09:35:14.372709+0000","last_undegraded":"2024-05-27T09:35:14.372709+0000","last_fullsized":"2024-05-27T09:35:14.372709+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:30:38.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042498+0000","last_change":"2024-05-27T09:31:24.909628+0000","last_active":"2024-05-27T09:35:14.042498+0000","last_peered":"2024-05-27T09:35:14.042498+0000","last_clean":"2024-05-27T09:35:14.042498+0000","last_became_active":"2024-05-27T09:31:24.902931+0000","last_became_peered":"2024-05-27T09:31:24.902931+0000","last_unstale":"2024-05-27T09:35:14.042498+0000","last_undegraded":"2024-05-27T09:35:14.042498+0000","last_fullsized":"2024-05-27T09:35:14.042498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:06.711316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242398+0000","last_change":"2024-05-27T09:31:24.912099+0000","last_active":"2024-05-27T09:35:14.242398+0000","last_peered":"2024-05-27T09:35:14.242398+0000","last_clean":"2024-05-27T09:35:14.242398+0000","last_became_active":"2024-05-27T09:31:24.911669+0000","last_became_peered":"2024-05-27T09:31:24.911669+0000","last_unstale":"2024-05-27T09:35:14.242398+0000","last_undegraded":"2024-05-27T09:35:14.242398+0000","last_fullsized":"2024-05-27T09:35:14.242398+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:58.068589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840085+0000","last_change":"2024-05-27T09:31:24.891426+0000","last_active":"2024-05-27T09:35:14.840085+0000","last_peered":"2024-05-27T09:35:14.840085+0000","last_clean":"2024-05-27T09:35:14.840085+0000","last_became_active":"2024-05-27T09:31:24.890877+0000","last_became_peered":"2024-05-27T09:31:24.890877+0000","last_unstale":"2024-05-27T09:35:14.840085+0000","last_undegraded":"2024-05-27T09:35:14.840085+0000","last_fullsized":"2024-05-27T09:35:14.840085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:15.338188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373079+0000","last_change":"2024-05-27T09:31:24.892021+0000","last_active":"2024-05-27T09:35:14.373079+0000","last_peered":"2024-05-27T09:35:14.373079+0000","last_clean":"2024-05-27T09:35:14.373079+0000","last_became_active":"2024-05-27T09:31:24.891607+0000","last_became_peered":"2024-05-27T09:31:24.891607+0000","last_unstale":"2024-05-27T09:35:14.373079+0000","last_undegraded":"2024-05-27T09:35:14.373079+0000","last_fullsized":"2024-05-27T09:35:14.373079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:05.623199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171321+0000","last_change":"2024-05-27T09:31:24.912166+0000","last_active":"2024-05-27T09:35:14.171321+0000","last_peered":"2024-05-27T09:35:14.171321+0000","last_clean":"2024-05-27T09:35:14.171321+0000","last_became_active":"2024-05-27T09:31:24.907555+0000","last_became_peered":"2024-05-27T09:31:24.907555+0000","last_unstale":"2024-05-27T09:35:14.171321+0000","last_undegraded":"2024-05-27T09:35:14.171321+0000","last_fullsized":"2024-05-27T09:35:14.171321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:55:33.772321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129575+0000","last_change":"2024-05-27T09:31:24.910525+0000","last_active":"2024-05-27T09:35:14.129575+0000","last_peered":"2024-05-27T09:35:14.129575+0000","last_clean":"2024-05-27T09:35:14.129575+0000","last_became_active":"2024-05-27T09:31:24.910089+0000","last_became_peered":"2024-05-27T09:31:24.910089+0000","last_unstale":"2024-05-27T09:35:14.129575+0000","last_undegraded":"2024-05-27T09:35:14.129575+0000","last_fullsized":"2024-05-27T09:35:14.129575+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:19:02.274781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170634+0000","last_change":"2024-05-27T09:31:24.901125+0000","last_active":"2024-05-27T09:35:14.170634+0000","last_peered":"2024-05-27T09:35:14.170634+0000","last_clean":"2024-05-27T09:35:14.170634+0000","last_became_active":"2024-05-27T09:31:24.900361+0000","last_became_peered":"2024-05-27T09:31:24.900361+0000","last_unstale":"2024-05-27T09:35:14.170634+0000","last_undegraded":"2024-05-27T09:35:14.170634+0000","last_fullsized":"2024-05-27T09:35:14.170634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:30.116349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372869+0000","last_change":"2024-05-27T09:31:24.917612+0000","last_active":"2024-05-27T09:35:14.372869+0000","last_peered":"2024-05-27T09:35:14.372869+0000","last_clean":"2024-05-27T09:35:14.372869+0000","last_became_active":"2024-05-27T09:31:24.917210+0000","last_became_peered":"2024-05-27T09:31:24.917210+0000","last_unstale":"2024-05-27T09:35:14.372869+0000","last_undegraded":"2024-05-27T09:35:14.372869+0000","last_fullsized":"2024-05-27T09:35:14.372869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:19:00.769114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241872+0000","last_change":"2024-05-27T09:31:24.909654+0000","last_active":"2024-05-27T09:35:14.241872+0000","last_peered":"2024-05-27T09:35:14.241872+0000","last_clean":"2024-05-27T09:35:14.241872+0000","last_became_active":"2024-05-27T09:31:24.909143+0000","last_became_peered":"2024-05-27T09:31:24.909143+0000","last_unstale":"2024-05-27T09:35:14.241872+0000","last_undegraded":"2024-05-27T09:35:14.241872+0000","last_fullsized":"2024-05-27T09:35:14.241872+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:42.844614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840108+0000","last_change":"2024-05-27T09:31:24.899031+0000","last_active":"2024-05-27T09:35:14.840108+0000","last_peered":"2024-05-27T09:35:14.840108+0000","last_clean":"2024-05-27T09:35:14.840108+0000","last_became_active":"2024-05-27T09:31:24.898664+0000","last_became_peered":"2024-05-27T09:31:24.898664+0000","last_unstale":"2024-05-27T09:35:14.840108+0000","last_undegraded":"2024-05-27T09:35:14.840108+0000","last_fullsized":"2024-05-27T09:35:14.840108+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:09.659646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600937,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1406,"num_read_kb":817,"num_write":301,"num_write_kb":1447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1802811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678080,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":892928,"data_stored":440736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874308,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41188,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703964,"statfs":{"total":95995035648,"available":95952859136,"internally_reserved":0,"allocated":1732608,"data_stored":1235405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070538,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":834188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266768,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":834108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462998,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28484,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716668,"statfs":{"total":95995035648,"available":95965868032,"internally_reserved":0,"allocated":1503232,"data_stored":1034974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626523,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":897024,"data_stored":440361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574970,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":510122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738493,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":441632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411445,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":511813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182508,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28424,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716728,"statfs":{"total":95995035648,"available":95965929472,"internally_reserved":0,"allocated":1441792,"data_stored":965488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986280,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":443000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822755,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36948,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708204,"statfs":{"total":95995035648,"available":95957200896,"internally_reserved":0,"allocated":1650688,"data_stored":1164638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":204766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T09:35:22.978 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:35:23.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:23 smithi044 ceph-mon[31618]: Updating smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:35:23.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:23 smithi044 ceph-mon[31618]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:23.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:23 smithi044 ceph-mon[31618]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 734 B/s wr, 16 op/s 2024-05-27T09:35:23.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:23 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:23 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:23 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:23 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:23.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:23 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:23.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:23 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:23.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:23 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:23 smithi050 ceph-mon[36371]: Updating smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:35:23.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:23 smithi050 ceph-mon[36371]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:23.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:23 smithi050 ceph-mon[36371]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 734 B/s wr, 16 op/s 2024-05-27T09:35:23.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:23 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:23 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:23 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:23 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:23.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:23 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:23.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:23 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:23.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:23 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:23 smithi189 ceph-mon[36671]: Updating smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T09:35:23.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:23 smithi189 ceph-mon[36671]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:23.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:23 smithi189 ceph-mon[36671]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 734 B/s wr, 16 op/s 2024-05-27T09:35:23.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:23 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:23 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:23 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:23 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:23.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:23 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:23.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:23 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:23.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:23 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:23.480 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:24.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:24 smithi044 ceph-mon[31618]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:24.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:24 smithi044 ceph-mon[31618]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:24 smithi050 ceph-mon[36371]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:24 smithi050 ceph-mon[36371]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:24 smithi189 ceph-mon[36671]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:24.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:24 smithi189 ceph-mon[36671]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:25.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:25 smithi189 ceph-mon[36671]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 570 B/s wr, 12 op/s 2024-05-27T09:35:25.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:25 smithi044 ceph-mon[31618]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 570 B/s wr, 12 op/s 2024-05-27T09:35:25.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:25 smithi050 ceph-mon[36371]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 570 B/s wr, 12 op/s 2024-05-27T09:35:25.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.934+0000 7fd536200700 1 -- 172.21.15.44:0/432019125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd530105e40 msgr2=0x7fd53010e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:25.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.934+0000 7fd536200700 1 --2- 172.21.15.44:0/432019125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd530105e40 0x7fd53010e390 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd518009990 tx=0x7fd518009ca0 comp rx=0 tx=0).stop 2024-05-27T09:35:25.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.935+0000 7fd536200700 1 -- 172.21.15.44:0/432019125 shutdown_connections 2024-05-27T09:35:25.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.935+0000 7fd536200700 1 --2- 172.21.15.44:0/432019125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53010e8d0 0x7fd530110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:25.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.935+0000 7fd536200700 1 --2- 172.21.15.44:0/432019125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5301054f0 0x7fd530105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:25.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.935+0000 7fd536200700 1 --2- 172.21.15.44:0/432019125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd530105e40 0x7fd53010e390 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:25.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.935+0000 7fd536200700 1 -- 172.21.15.44:0/432019125 >> 172.21.15.44:0/432019125 conn(0x7fd5300fedb0 msgr2=0x7fd5301011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:25.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.935+0000 7fd536200700 1 -- 172.21.15.44:0/432019125 shutdown_connections 2024-05-27T09:35:25.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.935+0000 7fd536200700 1 -- 172.21.15.44:0/432019125 wait complete. 2024-05-27T09:35:25.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.937+0000 7fd536200700 1 Processor -- start 2024-05-27T09:35:25.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.937+0000 7fd536200700 1 -- start start 2024-05-27T09:35:25.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.938+0000 7fd536200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5301054f0 0x7fd5301031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:25.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.938+0000 7fd536200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd530105e40 0x7fd530103720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:25.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.938+0000 7fd536200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd53010e8d0 0x7fd530103c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:25.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.938+0000 7fd536200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd530114430 con 0x7fd5301054f0 2024-05-27T09:35:25.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.938+0000 7fd536200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5301142b0 con 0x7fd53010e8d0 2024-05-27T09:35:25.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.938+0000 7fd536200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5301145b0 con 0x7fd530105e40 2024-05-27T09:35:25.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.938+0000 7fd526a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd530105e40 0x7fd530103720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:25.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.938+0000 7fd526a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd530105e40 0x7fd530103720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45342/0 (socket says 172.21.15.44:45342) 2024-05-27T09:35:25.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.938+0000 7fd526a00700 1 -- 172.21.15.44:0/1302254287 learned_addr learned my addr 172.21.15.44:0/1302254287 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:25.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.939+0000 7fd52f400700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5301054f0 0x7fd5301031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:25.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.939+0000 7fd52fe00700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd53010e8d0 0x7fd530103c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:25.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.939+0000 7fd526a00700 1 -- 172.21.15.44:0/1302254287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd53010e8d0 msgr2=0x7fd530103c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:25.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.939+0000 7fd526a00700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd53010e8d0 0x7fd530103c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:25.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.939+0000 7fd526a00700 1 -- 172.21.15.44:0/1302254287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5301054f0 msgr2=0x7fd5301031e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:25.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.939+0000 7fd526a00700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5301054f0 0x7fd5301031e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:25.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.939+0000 7fd526a00700 1 -- 172.21.15.44:0/1302254287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5301a8fa0 con 0x7fd530105e40 2024-05-27T09:35:25.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.939+0000 7fd52f400700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5301054f0 0x7fd5301031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:35:25.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.939+0000 7fd526a00700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd530105e40 0x7fd530103720 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd518005910 tx=0x7fd5180057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:25.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.940+0000 7fd52cc00700 1 -- 172.21.15.44:0/1302254287 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51800fb80 con 0x7fd530105e40 2024-05-27T09:35:25.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.940+0000 7fd52cc00700 1 -- 172.21.15.44:0/1302254287 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd51800fd20 con 0x7fd530105e40 2024-05-27T09:35:25.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.940+0000 7fd536200700 1 -- 172.21.15.44:0/1302254287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5301a9230 con 0x7fd530105e40 2024-05-27T09:35:25.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.940+0000 7fd52cc00700 1 -- 172.21.15.44:0/1302254287 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518017770 con 0x7fd530105e40 2024-05-27T09:35:25.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.940+0000 7fd536200700 1 -- 172.21.15.44:0/1302254287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5301a9700 con 0x7fd530105e40 2024-05-27T09:35:25.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.941+0000 7fd536200700 1 -- 172.21.15.44:0/1302254287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5300512b0 con 0x7fd530105e40 2024-05-27T09:35:25.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.944+0000 7fd52cc00700 1 -- 172.21.15.44:0/1302254287 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7fd518009d10 con 0x7fd530105e40 2024-05-27T09:35:25.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.945+0000 7fd52cc00700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5140b2b90 0x7fd5140b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:25.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.946+0000 7fd52cc00700 1 -- 172.21.15.44:0/1302254287 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fd5180e3cd0 con 0x7fd530105e40 2024-05-27T09:35:25.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.946+0000 7fd52f400700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5140b2b90 0x7fd5140b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:25.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.947+0000 7fd52f400700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5140b2b90 0x7fd5140b50a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd520009cb0 tx=0x7fd520009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:25.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:25.948+0000 7fd52cc00700 1 -- 172.21.15.44:0/1302254287 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd518013070 con 0x7fd530105e40 2024-05-27T09:35:26.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:26 smithi044 ceph-mon[31618]: pgmap v9: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 466 B/s wr, 13 op/s 2024-05-27T09:35:26.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.234+0000 7fd536200700 1 -- 172.21.15.44:0/1302254287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5301a9a20 con 0x7fd530105e40 2024-05-27T09:35:26.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.236+0000 7fd52cc00700 1 -- 172.21.15.44:0/1302254287 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7fd5180ab4c0 con 0x7fd530105e40 2024-05-27T09:35:26.238 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:26.238 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:35:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.239+0000 7fd536200700 1 -- 172.21.15.44:0/1302254287 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5140b2b90 msgr2=0x7fd5140b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.239+0000 7fd536200700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5140b2b90 0x7fd5140b50a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd520009cb0 tx=0x7fd520009260 comp rx=0 tx=0).stop 2024-05-27T09:35:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.239+0000 7fd536200700 1 -- 172.21.15.44:0/1302254287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd530105e40 msgr2=0x7fd530103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.239+0000 7fd536200700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd530105e40 0x7fd530103720 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd518005910 tx=0x7fd5180057a0 comp rx=0 tx=0).stop 2024-05-27T09:35:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.239+0000 7fd536200700 1 -- 172.21.15.44:0/1302254287 shutdown_connections 2024-05-27T09:35:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.239+0000 7fd536200700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5140b2b90 0x7fd5140b50a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.239+0000 7fd536200700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd53010e8d0 0x7fd530103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.239+0000 7fd536200700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5301054f0 0x7fd5301031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.239+0000 7fd536200700 1 --2- 172.21.15.44:0/1302254287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd530105e40 0x7fd530103720 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.239+0000 7fd536200700 1 -- 172.21.15.44:0/1302254287 >> 172.21.15.44:0/1302254287 conn(0x7fd5300fedb0 msgr2=0x7fd530106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.240+0000 7fd536200700 1 -- 172.21.15.44:0/1302254287 shutdown_connections 2024-05-27T09:35:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:26.240+0000 7fd536200700 1 -- 172.21.15.44:0/1302254287 wait complete. 2024-05-27T09:35:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:35:26.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:26 smithi050 ceph-mon[36371]: pgmap v9: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 466 B/s wr, 13 op/s 2024-05-27T09:35:26.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:26 smithi189 ceph-mon[36671]: pgmap v9: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 466 B/s wr, 13 op/s 2024-05-27T09:35:27.081 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T09:35:27.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:27 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1302254287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:27.528 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:27.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:27 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1302254287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:27.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:27 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1302254287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:28.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:28 smithi044 ceph-mon[31618]: pgmap v10: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-27T09:35:28.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:28 smithi050 ceph-mon[36371]: pgmap v10: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-27T09:35:28.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:28 smithi189 ceph-mon[36671]: pgmap v10: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-27T09:35:29.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.744+0000 7f75aa000700 1 -- 172.21.15.44:0/3979072889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a4108f10 msgr2=0x7f75a4109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:29.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.744+0000 7f75aa000700 1 --2- 172.21.15.44:0/3979072889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a4108f10 0x7f75a4109320 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f758c0099c0 tx=0x7f758c009cd0 comp rx=0 tx=0).stop 2024-05-27T09:35:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.744+0000 7f75aa000700 1 -- 172.21.15.44:0/3979072889 shutdown_connections 2024-05-27T09:35:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.744+0000 7f75aa000700 1 --2- 172.21.15.44:0/3979072889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75a4109860 0x7f75a41111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.744+0000 7f75aa000700 1 --2- 172.21.15.44:0/3979072889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a41045e0 0x7f75a41049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.744+0000 7f75aa000700 1 --2- 172.21.15.44:0/3979072889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a4108f10 0x7f75a4109320 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.744+0000 7f75aa000700 1 -- 172.21.15.44:0/3979072889 >> 172.21.15.44:0/3979072889 conn(0x7f75a4100030 msgr2=0x7f75a4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.744+0000 7f75aa000700 1 -- 172.21.15.44:0/3979072889 shutdown_connections 2024-05-27T09:35:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.744+0000 7f75aa000700 1 -- 172.21.15.44:0/3979072889 wait complete. 2024-05-27T09:35:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.746+0000 7f75aa000700 1 Processor -- start 2024-05-27T09:35:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.746+0000 7f75aa000700 1 -- start start 2024-05-27T09:35:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.746+0000 7f75aa000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75a41045e0 0x7f75a419d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.746+0000 7f75aa000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a4108f10 0x7f75a419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.746+0000 7f75aa000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4109860 0x7f75a41a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.746+0000 7f75aa000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75a41144c0 con 0x7f75a41045e0 2024-05-27T09:35:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.746+0000 7f75aa000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f75a4114340 con 0x7f75a4109860 2024-05-27T09:35:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.746+0000 7f75aa000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75a4114640 con 0x7f75a4108f10 2024-05-27T09:35:29.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.747+0000 7f75a3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75a41045e0 0x7f75a419d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.747+0000 7f75a3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75a41045e0 0x7f75a419d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56104/0 (socket says 172.21.15.44:56104) 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.747+0000 7f75a3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4109860 0x7f75a41a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.747+0000 7f759aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a4108f10 0x7f75a419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.747+0000 7f75a3400700 1 -- 172.21.15.44:0/2187659076 learned_addr learned my addr 172.21.15.44:0/2187659076 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.747+0000 7f75a3e00700 1 -- 172.21.15.44:0/2187659076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a4108f10 msgr2=0x7f75a419d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.747+0000 7f75a3e00700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a4108f10 0x7f75a419d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.747+0000 7f75a3e00700 1 -- 172.21.15.44:0/2187659076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75a41045e0 msgr2=0x7f75a419d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.747+0000 7f75a3e00700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75a41045e0 0x7f75a419d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.747+0000 7f75a3e00700 1 -- 172.21.15.44:0/2187659076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a41a1f80 con 0x7f75a4109860 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.748+0000 7f75a3e00700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4109860 0x7f75a41a18e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f759c00efc0 tx=0x7f759c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.748+0000 7f75a0c00700 1 -- 172.21.15.44:0/2187659076 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c0092e0 con 0x7f75a4109860 2024-05-27T09:35:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.748+0000 7f75aa000700 1 -- 172.21.15.44:0/2187659076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a41a2210 con 0x7f75a4109860 2024-05-27T09:35:29.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.748+0000 7f75a0c00700 1 -- 172.21.15.44:0/2187659076 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f759c004bb0 con 0x7f75a4109860 2024-05-27T09:35:29.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.748+0000 7f75a0c00700 1 -- 172.21.15.44:0/2187659076 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c004020 con 0x7f75a4109860 2024-05-27T09:35:29.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.748+0000 7f75aa000700 1 -- 172.21.15.44:0/2187659076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a41a2720 con 0x7f75a4109860 2024-05-27T09:35:29.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.749+0000 7f75aa000700 1 -- 172.21.15.44:0/2187659076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a40512b0 con 0x7f75a4109860 2024-05-27T09:35:29.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.752+0000 7f75a0c00700 1 -- 172.21.15.44:0/2187659076 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f759c01b030 con 0x7f75a4109860 2024-05-27T09:35:29.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.754+0000 7f75a0c00700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75880b2b10 0x7f75880b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:29.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.754+0000 7f75a0c00700 1 -- 172.21.15.44:0/2187659076 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f759c0e2360 con 0x7f75a4109860 2024-05-27T09:35:29.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.754+0000 7f75a3400700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75880b2b10 0x7f75880b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:29.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.755+0000 7f75a3400700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75880b2b10 0x7f75880b5020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7594009d20 tx=0x7f7594009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.759+0000 7f75a0c00700 1 -- 172.21.15.44:0/2187659076 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f759c022020 con 0x7f75a4109860 2024-05-27T09:35:29.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.997+0000 7f75aa000700 1 -- 172.21.15.44:0/2187659076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75a41a9c20 con 0x7f75a4109860 2024-05-27T09:35:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:29.999+0000 7f75a0c00700 1 -- 172.21.15.44:0/2187659076 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+22988 (secure 0 0 0) 0x7f759c010040 con 0x7f75a4109860 2024-05-27T09:35:30.002 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:30.002 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:35:14.029489+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6800/4077502151":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/2278564126":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/4102964244":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/408258823":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3716545176":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/4077502151":"2024-05-28T09:35:14.029409+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:35:30.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.003+0000 7f75aa000700 1 -- 172.21.15.44:0/2187659076 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75880b2b10 msgr2=0x7f75880b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:30.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.003+0000 7f75aa000700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75880b2b10 0x7f75880b5020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7594009d20 tx=0x7f7594009260 comp rx=0 tx=0).stop 2024-05-27T09:35:30.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.003+0000 7f75aa000700 1 -- 172.21.15.44:0/2187659076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4109860 msgr2=0x7f75a41a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:30.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.003+0000 7f75aa000700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4109860 0x7f75a41a18e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f759c00efc0 tx=0x7f759c00c3a0 comp rx=0 tx=0).stop 2024-05-27T09:35:30.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.003+0000 7f75aa000700 1 -- 172.21.15.44:0/2187659076 shutdown_connections 2024-05-27T09:35:30.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.003+0000 7f75aa000700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75880b2b10 0x7f75880b5020 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:30.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.003+0000 7f75aa000700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4109860 0x7f75a41a18e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:30.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.003+0000 7f75aa000700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75a41045e0 0x7f75a419d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:30.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.003+0000 7f75aa000700 1 --2- 172.21.15.44:0/2187659076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a4108f10 0x7f75a419d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:30.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.003+0000 7f75aa000700 1 -- 172.21.15.44:0/2187659076 >> 172.21.15.44:0/2187659076 conn(0x7f75a4100030 msgr2=0x7f75a4110e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:30.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.004+0000 7f75aa000700 1 -- 172.21.15.44:0/2187659076 shutdown_connections 2024-05-27T09:35:30.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:30.004+0000 7f75aa000700 1 -- 172.21.15.44:0/2187659076 wait complete. 2024-05-27T09:35:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:30 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2187659076' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:35:30.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:30 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2187659076' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:35:30.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:30 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2187659076' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:35:30.770 DEBUG:tasks.fwd_scrub:Yielding 2024-05-27T09:35:30.770 INFO:teuthology.run_tasks:Running task exec... 2024-05-27T09:35:30.782 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T09:35:30.783 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-05-27T09:35:30.783 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-05-27T09:35:30.783 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-27T09:35:30.783 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:35:30.783 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:35:31.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.066+0000 7f93bb400640 1 -- 172.21.15.44:0/2127734319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4075660 msgr2=0x7f93b4075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:31.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.066+0000 7f93bb400640 1 --2- 172.21.15.44:0/2127734319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4075660 0x7f93b4075a70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f93a8009900 tx=0x7f93a802a480 comp rx=0 tx=0).stop 2024-05-27T09:35:31.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.066+0000 7f93bb400640 1 -- 172.21.15.44:0/2127734319 shutdown_connections 2024-05-27T09:35:31.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.066+0000 7f93bb400640 1 --2- 172.21.15.44:0/2127734319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93b41104a0 0x7f93b4112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.066+0000 7f93bb400640 1 --2- 172.21.15.44:0/2127734319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b410db40 0x7f93b410ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.066+0000 7f93bb400640 1 --2- 172.21.15.44:0/2127734319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4075660 0x7f93b4075a70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.066+0000 7f93bb400640 1 -- 172.21.15.44:0/2127734319 >> 172.21.15.44:0/2127734319 conn(0x7f93b40ff480 msgr2=0x7f93b41018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:31.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.066+0000 7f93bb400640 1 -- 172.21.15.44:0/2127734319 shutdown_connections 2024-05-27T09:35:31.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.066+0000 7f93bb400640 1 -- 172.21.15.44:0/2127734319 wait complete. 2024-05-27T09:35:31.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.068+0000 7f93bb400640 1 Processor -- start 2024-05-27T09:35:31.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.068+0000 7f93bb400640 1 -- start start 2024-05-27T09:35:31.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.068+0000 7f93bb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4075660 0x7f93b41a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:31.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.068+0000 7f93bb400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b410db40 0x7f93b41a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:31.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.068+0000 7f93bb400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93b41104a0 0x7f93b419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:31.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.068+0000 7f93bb400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f93b41160b0 con 0x7f93b41104a0 2024-05-27T09:35:31.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.068+0000 7f93bb400640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f93b4115f30 con 0x7f93b410db40 2024-05-27T09:35:31.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.068+0000 7f93bb400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93b4116230 con 0x7f93b4075660 2024-05-27T09:35:31.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.069+0000 7f93b3e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b410db40 0x7f93b41a2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:31.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.069+0000 7f93b9400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93b41104a0 0x7f93b419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:31.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.069+0000 7f93b3e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b410db40 0x7f93b41a2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45050/0 (socket says 172.21.15.44:45050) 2024-05-27T09:35:31.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.069+0000 7f93b3e00640 1 -- 172.21.15.44:0/1335631491 learned_addr learned my addr 172.21.15.44:0/1335631491 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:31.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.069+0000 7f93b8a00640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4075660 0x7f93b41a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:31.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.069+0000 7f93b3e00640 1 -- 172.21.15.44:0/1335631491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4075660 msgr2=0x7f93b41a26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:31.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.069+0000 7f93b3e00640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4075660 0x7f93b41a26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.069+0000 7f93b3e00640 1 -- 172.21.15.44:0/1335631491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93b41104a0 msgr2=0x7f93b419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:31.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.069+0000 7f93b3e00640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93b41104a0 0x7f93b419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.069+0000 7f93b3e00640 1 -- 172.21.15.44:0/1335631491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b419cf50 con 0x7f93b410db40 2024-05-27T09:35:31.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.069+0000 7f93b8a00640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4075660 0x7f93b41a26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:31.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.070+0000 7f93b3e00640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b410db40 0x7f93b41a2be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f93a000e9f0 tx=0x7f93a000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:31.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.070+0000 7f93b1600640 1 -- 172.21.15.44:0/1335631491 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a000cd80 con 0x7f93b410db40 2024-05-27T09:35:31.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.070+0000 7f93b1600640 1 -- 172.21.15.44:0/1335631491 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93a00054d0 con 0x7f93b410db40 2024-05-27T09:35:31.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.070+0000 7f93b1600640 1 -- 172.21.15.44:0/1335631491 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a0010690 con 0x7f93b410db40 2024-05-27T09:35:31.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.070+0000 7f93bb400640 1 -- 172.21.15.44:0/1335631491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b419d240 con 0x7f93b410db40 2024-05-27T09:35:31.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.070+0000 7f93bb400640 1 -- 172.21.15.44:0/1335631491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b41a9520 con 0x7f93b410db40 2024-05-27T09:35:31.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.071+0000 7f93bb400640 1 -- 172.21.15.44:0/1335631491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b406a230 con 0x7f93b410db40 2024-05-27T09:35:31.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.077+0000 7f93b1600640 1 -- 172.21.15.44:0/1335631491 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f93a0004e90 con 0x7f93b410db40 2024-05-27T09:35:31.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.078+0000 7f93b1600640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f939c0b2af0 0x7f939c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:31.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.079+0000 7f93b1600640 1 -- 172.21.15.44:0/1335631491 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f93a00e2c70 con 0x7f93b410db40 2024-05-27T09:35:31.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.079+0000 7f93b8a00640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f939c0b2af0 0x7f939c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:31.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.080+0000 7f93b8a00640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f939c0b2af0 0x7f939c0b5000 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f93a8005ab0 tx=0x7f93a8005a20 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:31.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.083+0000 7f93b1600640 1 -- 172.21.15.44:0/1335631491 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f93a00aa3d0 con 0x7f93b410db40 2024-05-27T09:35:31.278 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:31.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:31 smithi189 ceph-mon[36671]: pgmap v11: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-27T09:35:31.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.375+0000 7f93bb400640 1 -- 172.21.15.44:0/1335631491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f93b41a96f0 con 0x7f93b410db40 2024-05-27T09:35:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.384+0000 7f93b1600640 1 -- 172.21.15.44:0/1335631491 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v28)=0 v28) ==== 129+0+0 (secure 0 0 0) 0x7f93a00a9cd0 con 0x7f93b410db40 2024-05-27T09:35:31.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 -- 172.21.15.44:0/1335631491 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f939c0b2af0 msgr2=0x7f939c0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:31.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f939c0b2af0 0x7f939c0b5000 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f93a8005ab0 tx=0x7f93a8005a20 comp rx=0 tx=0).stop 2024-05-27T09:35:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 -- 172.21.15.44:0/1335631491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b410db40 msgr2=0x7f93b41a2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b410db40 0x7f93b41a2be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f93a000e9f0 tx=0x7f93a000edf0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 -- 172.21.15.44:0/1335631491 shutdown_connections 2024-05-27T09:35:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f939c0b2af0 0x7f939c0b5000 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93b41104a0 0x7f93b419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b410db40 0x7f93b41a2be0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 --2- 172.21.15.44:0/1335631491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4075660 0x7f93b41a26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 -- 172.21.15.44:0/1335631491 >> 172.21.15.44:0/1335631491 conn(0x7f93b40ff480 msgr2=0x7f93b41108f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 -- 172.21.15.44:0/1335631491 shutdown_connections 2024-05-27T09:35:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.387+0000 7f93bb400640 1 -- 172.21.15.44:0/1335631491 wait complete. 2024-05-27T09:35:31.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:31 smithi044 ceph-mon[31618]: pgmap v11: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-27T09:35:31.420 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-27T09:35:31.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:31 smithi050 ceph-mon[36371]: pgmap v11: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-27T09:35:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.692+0000 7f342b200640 1 -- 172.21.15.44:0/975380571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c109ab0 msgr2=0x7f342c1036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.692+0000 7f342b200640 1 --2- 172.21.15.44:0/975380571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c109ab0 0x7f342c1036a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f341c009900 tx=0x7f341c02a3f0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.692+0000 7f342b200640 1 -- 172.21.15.44:0/975380571 shutdown_connections 2024-05-27T09:35:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.692+0000 7f342b200640 1 --2- 172.21.15.44:0/975380571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c104250 0x7f342c1105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.692+0000 7f342b200640 1 --2- 172.21.15.44:0/975380571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c109f50 0x7f342c103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.692+0000 7f342b200640 1 --2- 172.21.15.44:0/975380571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c109ab0 0x7f342c1036a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.692+0000 7f342b200640 1 -- 172.21.15.44:0/975380571 >> 172.21.15.44:0/975380571 conn(0x7f342c079240 msgr2=0x7f342c079680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.692+0000 7f342b200640 1 -- 172.21.15.44:0/975380571 shutdown_connections 2024-05-27T09:35:31.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.692+0000 7f342b200640 1 -- 172.21.15.44:0/975380571 wait complete. 2024-05-27T09:35:31.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.694+0000 7f342b200640 1 Processor -- start 2024-05-27T09:35:31.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.694+0000 7f342b200640 1 -- start start 2024-05-27T09:35:31.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.694+0000 7f342b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c104250 0x7f342c074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:31.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.694+0000 7f342b200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c109ab0 0x7f342c075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:31.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.694+0000 7f342b200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c109f50 0x7f342c075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:31.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f342b200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f342c113cf0 con 0x7f342c109f50 2024-05-27T09:35:31.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f342b200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f342c113b70 con 0x7f342c109ab0 2024-05-27T09:35:31.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f342b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f342c113e70 con 0x7f342c104250 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f3429e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c104250 0x7f342c074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f342a800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c109f50 0x7f342c075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f3429e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c104250 0x7f342c074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45400/0 (socket says 172.21.15.44:45400) 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f3429e00640 1 -- 172.21.15.44:0/2891403169 learned_addr learned my addr 172.21.15.44:0/2891403169 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f3429400640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c109ab0 0x7f342c075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f3429e00640 1 -- 172.21.15.44:0/2891403169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c109ab0 msgr2=0x7f342c075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f3429e00640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c109ab0 0x7f342c075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f3429e00640 1 -- 172.21.15.44:0/2891403169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c109f50 msgr2=0x7f342c075a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f3429e00640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c109f50 0x7f342c075a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f3429e00640 1 -- 172.21.15.44:0/2891403169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c1a8a50 con 0x7f342c104250 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.695+0000 7f3429400640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c109ab0 0x7f342c075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:31.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.696+0000 7f3429e00640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c104250 0x7f342c074fc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f341c005ab0 tx=0x7f341c003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:31.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.696+0000 7f3422a00640 1 -- 172.21.15.44:0/2891403169 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c03f070 con 0x7f342c104250 2024-05-27T09:35:31.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.696+0000 7f3422a00640 1 -- 172.21.15.44:0/2891403169 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f341c031b50 con 0x7f342c104250 2024-05-27T09:35:31.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.696+0000 7f342b200640 1 -- 172.21.15.44:0/2891403169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f342c1a8ce0 con 0x7f342c104250 2024-05-27T09:35:31.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.696+0000 7f3422a00640 1 -- 172.21.15.44:0/2891403169 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c035030 con 0x7f342c104250 2024-05-27T09:35:31.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.696+0000 7f342b200640 1 -- 172.21.15.44:0/2891403169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f342c1a91d0 con 0x7f342c104250 2024-05-27T09:35:31.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.697+0000 7f342b200640 1 -- 172.21.15.44:0/2891403169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f342c06a230 con 0x7f342c104250 2024-05-27T09:35:31.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.700+0000 7f3422a00640 1 -- 172.21.15.44:0/2891403169 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f341c042070 con 0x7f342c104250 2024-05-27T09:35:31.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.701+0000 7f3422a00640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f34100b2af0 0x7f34100b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:31.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.702+0000 7f3422a00640 1 -- 172.21.15.44:0/2891403169 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f341c103180 con 0x7f342c104250 2024-05-27T09:35:31.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.702+0000 7f3429400640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f34100b2af0 0x7f34100b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:31.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.703+0000 7f3429400640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f34100b2af0 0x7f34100b5000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f342c10f970 tx=0x7f3418005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:31.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.704+0000 7f3422a00640 1 -- 172.21.15.44:0/2891403169 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f341c0ca3c0 con 0x7f342c104250 2024-05-27T09:35:31.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.922+0000 7f342b200640 1 -- 172.21.15.44:0/2891403169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f342c107f80 con 0x7f342c104250 2024-05-27T09:35:31.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.931+0000 7f3422a00640 1 -- 172.21.15.44:0/2891403169 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f341c0ce6c0 con 0x7f342c104250 2024-05-27T09:35:31.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.934+0000 7f342b200640 1 -- 172.21.15.44:0/2891403169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f34100b2af0 msgr2=0x7f34100b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:31.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.934+0000 7f342b200640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f34100b2af0 0x7f34100b5000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f342c10f970 tx=0x7f3418005f90 comp rx=0 tx=0).stop 2024-05-27T09:35:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.935+0000 7f342b200640 1 -- 172.21.15.44:0/2891403169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c104250 msgr2=0x7f342c074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.935+0000 7f342b200640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c104250 0x7f342c074fc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f341c005ab0 tx=0x7f341c003b90 comp rx=0 tx=0).stop 2024-05-27T09:35:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.935+0000 7f342b200640 1 -- 172.21.15.44:0/2891403169 shutdown_connections 2024-05-27T09:35:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.935+0000 7f342b200640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f34100b2af0 0x7f34100b5000 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.935+0000 7f342b200640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c109f50 0x7f342c075a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.935+0000 7f342b200640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c109ab0 0x7f342c075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.935+0000 7f342b200640 1 --2- 172.21.15.44:0/2891403169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c104250 0x7f342c074fc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.935+0000 7f342b200640 1 -- 172.21.15.44:0/2891403169 >> 172.21.15.44:0/2891403169 conn(0x7f342c079240 msgr2=0x7f342c105150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.935+0000 7f342b200640 1 -- 172.21.15.44:0/2891403169 shutdown_connections 2024-05-27T09:35:31.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:31.935+0000 7f342b200640 1 -- 172.21.15.44:0/2891403169 wait complete. 2024-05-27T09:35:31.966 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-27T09:35:32.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.239+0000 7f5dcda00640 1 -- 172.21.15.44:0/277405513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dc810a130 msgr2=0x7f5dc8103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:32.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.239+0000 7f5dcda00640 1 --2- 172.21.15.44:0/277405513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dc810a130 0x7f5dc8103d70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009900 tx=0x7f5db402a3f0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.239+0000 7f5dcda00640 1 -- 172.21.15.44:0/277405513 shutdown_connections 2024-05-27T09:35:32.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.239+0000 7f5dcda00640 1 --2- 172.21.15.44:0/277405513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5dc81043e0 0x7f5dc81107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.239+0000 7f5dcda00640 1 --2- 172.21.15.44:0/277405513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dc810a130 0x7f5dc8103d70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.239+0000 7f5dcda00640 1 --2- 172.21.15.44:0/277405513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8109c90 0x7f5dc8103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.239+0000 7f5dcda00640 1 -- 172.21.15.44:0/277405513 >> 172.21.15.44:0/277405513 conn(0x7f5dc80ff330 msgr2=0x7f5dc8101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:32.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.239+0000 7f5dcda00640 1 -- 172.21.15.44:0/277405513 shutdown_connections 2024-05-27T09:35:32.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.239+0000 7f5dcda00640 1 -- 172.21.15.44:0/277405513 wait complete. 2024-05-27T09:35:32.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.240+0000 7f5dcda00640 1 Processor -- start 2024-05-27T09:35:32.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.240+0000 7f5dcda00640 1 -- start start 2024-05-27T09:35:32.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.240+0000 7f5dcda00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5dc81043e0 0x7f5dc819d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.240+0000 7f5dcda00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dc8109c90 0x7f5dc819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.240+0000 7f5dcda00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc810a130 0x7f5dc81a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.240+0000 7f5dcda00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc8113e80 con 0x7f5dc8109c90 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.240+0000 7f5dcda00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc8113d00 con 0x7f5dc81043e0 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.240+0000 7f5dcda00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc8114000 con 0x7f5dc810a130 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.241+0000 7f5dc6a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5dc81043e0 0x7f5dc819d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.241+0000 7f5dc6000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dc8109c90 0x7f5dc819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.241+0000 7f5dc6a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5dc81043e0 0x7f5dc819d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45088/0 (socket says 172.21.15.44:45088) 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.241+0000 7f5dc6a00640 1 -- 172.21.15.44:0/2686538275 learned_addr learned my addr 172.21.15.44:0/2686538275 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.241+0000 7f5dc7400640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc810a130 0x7f5dc81a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.241+0000 7f5dc6a00640 1 -- 172.21.15.44:0/2686538275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc810a130 msgr2=0x7f5dc81a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.241+0000 7f5dc6a00640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc810a130 0x7f5dc81a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.241+0000 7f5dc6a00640 1 -- 172.21.15.44:0/2686538275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dc8109c90 msgr2=0x7f5dc819de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.241+0000 7f5dc6a00640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dc8109c90 0x7f5dc819de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.241+0000 7f5dc6a00640 1 -- 172.21.15.44:0/2686538275 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc81a8d20 con 0x7f5dc81043e0 2024-05-27T09:35:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.242+0000 7f5dc6a00640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5dc81043e0 0x7f5dc819d940 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5db800e9f0 tx=0x7f5db800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.242+0000 7f5dbf400640 1 -- 172.21.15.44:0/2686538275 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db800ccc0 con 0x7f5dc81043e0 2024-05-27T09:35:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.242+0000 7f5dbf400640 1 -- 172.21.15.44:0/2686538275 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5db80054d0 con 0x7f5dc81043e0 2024-05-27T09:35:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.242+0000 7f5dcda00640 1 -- 172.21.15.44:0/2686538275 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc81a8fb0 con 0x7f5dc81043e0 2024-05-27T09:35:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.242+0000 7f5dbf400640 1 -- 172.21.15.44:0/2686538275 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db8010430 con 0x7f5dc81043e0 2024-05-27T09:35:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.242+0000 7f5dcda00640 1 -- 172.21.15.44:0/2686538275 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc81a93c0 con 0x7f5dc81043e0 2024-05-27T09:35:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.243+0000 7f5dcda00640 1 -- 172.21.15.44:0/2686538275 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc8050860 con 0x7f5dc81043e0 2024-05-27T09:35:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.246+0000 7f5dbf400640 1 -- 172.21.15.44:0/2686538275 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f5db80046c0 con 0x7f5dc81043e0 2024-05-27T09:35:32.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.247+0000 7f5dbf400640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5dac0b2a70 0x7f5dac0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:32.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.247+0000 7f5dbf400640 1 -- 172.21.15.44:0/2686538275 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f5db80e2dd0 con 0x7f5dc81043e0 2024-05-27T09:35:32.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.248+0000 7f5dc6000640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5dac0b2a70 0x7f5dac0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:32.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.248+0000 7f5dc6000640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5dac0b2a70 0x7f5dac0b4f80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009230 tx=0x7f5db4031400 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:32.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.250+0000 7f5dbf400640 1 -- 172.21.15.44:0/2686538275 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5db8054df0 con 0x7f5dc81043e0 2024-05-27T09:35:32.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.470+0000 7f5dcda00640 1 -- 172.21.15.44:0/2686538275 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f5dc81a9750 con 0x7f5dc81043e0 2024-05-27T09:35:32.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.478+0000 7f5dbf400640 1 -- 172.21.15.44:0/2686538275 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f5db80aa530 con 0x7f5dc81043e0 2024-05-27T09:35:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 -- 172.21.15.44:0/2686538275 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5dac0b2a70 msgr2=0x7f5dac0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5dac0b2a70 0x7f5dac0b4f80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009230 tx=0x7f5db4031400 comp rx=0 tx=0).stop 2024-05-27T09:35:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 -- 172.21.15.44:0/2686538275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5dc81043e0 msgr2=0x7f5dc819d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5dc81043e0 0x7f5dc819d940 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5db800e9f0 tx=0x7f5db800eeb0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 -- 172.21.15.44:0/2686538275 shutdown_connections 2024-05-27T09:35:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc810a130 0x7f5dc81a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5dac0b2a70 0x7f5dac0b4f80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dc8109c90 0x7f5dc819de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 --2- 172.21.15.44:0/2686538275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5dc81043e0 0x7f5dc819d940 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 -- 172.21.15.44:0/2686538275 >> 172.21.15.44:0/2686538275 conn(0x7f5dc80ff330 msgr2=0x7f5dc8076000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:32.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 -- 172.21.15.44:0/2686538275 shutdown_connections 2024-05-27T09:35:32.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.481+0000 7f5dcda00640 1 -- 172.21.15.44:0/2686538275 wait complete. 2024-05-27T09:35:32.511 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-27T09:35:32.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:32 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' 2024-05-27T09:35:32.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:32 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:32.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:32 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:32.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:32 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:32.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:32 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:32.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:32 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' 2024-05-27T09:35:32.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:32 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:32.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:32 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:32.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:32 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:32.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:32 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:32.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:32 smithi044 ceph-mon[31618]: pgmap v12: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 426 B/s wr, 14 op/s 2024-05-27T09:35:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.781+0000 7f4d05a00640 1 -- 172.21.15.44:0/2936637968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d00104bb0 msgr2=0x7f4d00104fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.781+0000 7f4d05a00640 1 --2- 172.21.15.44:0/2936637968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d00104bb0 0x7f4d00104fc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009900 tx=0x7f4cf002a3f0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.781+0000 7f4d05a00640 1 -- 172.21.15.44:0/2936637968 shutdown_connections 2024-05-27T09:35:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.781+0000 7f4d05a00640 1 --2- 172.21.15.44:0/2936637968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d0010e000 0x7f4d001104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.781+0000 7f4d05a00640 1 --2- 172.21.15.44:0/2936637968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0010b610 0x7f4d0010da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.781+0000 7f4d05a00640 1 --2- 172.21.15.44:0/2936637968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d00104bb0 0x7f4d00104fc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.781+0000 7f4d05a00640 1 -- 172.21.15.44:0/2936637968 >> 172.21.15.44:0/2936637968 conn(0x7f4d000fe210 msgr2=0x7f4d00100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:32.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.781+0000 7f4d05a00640 1 -- 172.21.15.44:0/2936637968 shutdown_connections 2024-05-27T09:35:32.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.781+0000 7f4d05a00640 1 -- 172.21.15.44:0/2936637968 wait complete. 2024-05-27T09:35:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.782+0000 7f4d05a00640 1 Processor -- start 2024-05-27T09:35:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.783+0000 7f4d05a00640 1 -- start start 2024-05-27T09:35:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.783+0000 7f4d05a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00104bb0 0x7f4d001a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.783+0000 7f4d05a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d0010b610 0x7f4d001a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.783+0000 7f4d05a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d0010e000 0x7f4d0019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.783+0000 7f4d05a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4d00113c60 con 0x7f4d0010b610 2024-05-27T09:35:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.783+0000 7f4d05a00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4d00113ae0 con 0x7f4d00104bb0 2024-05-27T09:35:32.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.783+0000 7f4d05a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4d00113de0 con 0x7f4d0010e000 2024-05-27T09:35:32.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.783+0000 7f4cff400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d0010b610 0x7f4d001a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:32.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.783+0000 7f4cff400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d0010b610 0x7f4d001a2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56172/0 (socket says 172.21.15.44:56172) 2024-05-27T09:35:32.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.783+0000 7f4cff400640 1 -- 172.21.15.44:0/1896616104 learned_addr learned my addr 172.21.15.44:0/1896616104 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:32.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.783+0000 7f4d05000640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d0010e000 0x7f4d0019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:32.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.784+0000 7f4cffe00640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00104bb0 0x7f4d001a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.784+0000 7f4d05000640 1 -- 172.21.15.44:0/1896616104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00104bb0 msgr2=0x7f4d001a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.784+0000 7f4d05000640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00104bb0 0x7f4d001a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.784+0000 7f4d05000640 1 -- 172.21.15.44:0/1896616104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d0010b610 msgr2=0x7f4d001a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.784+0000 7f4d05000640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d0010b610 0x7f4d001a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.784+0000 7f4d05000640 1 -- 172.21.15.44:0/1896616104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0019cdf0 con 0x7f4d0010e000 2024-05-27T09:35:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.784+0000 7f4cff400640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d0010b610 0x7f4d001a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.784+0000 7f4d05000640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d0010e000 0x7f4d0019c6c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009fd0 tx=0x7f4cf800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:32.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.785+0000 7f4cfcc00640 1 -- 172.21.15.44:0/1896616104 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf800ce20 con 0x7f4d0010e000 2024-05-27T09:35:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.785+0000 7f4cfcc00640 1 -- 172.21.15.44:0/1896616104 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cf8005020 con 0x7f4d0010e000 2024-05-27T09:35:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.785+0000 7f4d05a00640 1 -- 172.21.15.44:0/1896616104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d0019d080 con 0x7f4d0010e000 2024-05-27T09:35:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.785+0000 7f4cfcc00640 1 -- 172.21.15.44:0/1896616104 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8010690 con 0x7f4d0010e000 2024-05-27T09:35:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.785+0000 7f4d05a00640 1 -- 172.21.15.44:0/1896616104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d001a92e0 con 0x7f4d0010e000 2024-05-27T09:35:32.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.786+0000 7f4d05a00640 1 -- 172.21.15.44:0/1896616104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d0006adc0 con 0x7f4d0010e000 2024-05-27T09:35:32.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.789+0000 7f4cfcc00640 1 -- 172.21.15.44:0/1896616104 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f4cf80054d0 con 0x7f4d0010e000 2024-05-27T09:35:32.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.790+0000 7f4cfcc00640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4ce40b2a70 0x7f4ce40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:32.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.790+0000 7f4cfcc00640 1 -- 172.21.15.44:0/1896616104 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f4cf80e2f60 con 0x7f4d0010e000 2024-05-27T09:35:32.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.790+0000 7f4cffe00640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4ce40b2a70 0x7f4ce40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:32.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.791+0000 7f4cffe00640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4ce40b2a70 0x7f4ce40b4f80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0031440 tx=0x7f4cf0009560 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:32.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:32.792+0000 7f4cfcc00640 1 -- 172.21.15.44:0/1896616104 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4cf80aa6c0 con 0x7f4d0010e000 2024-05-27T09:35:32.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:32 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' 2024-05-27T09:35:32.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:32 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:32.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:32 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:32.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:32 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:32.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:32 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:32.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:32 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' 2024-05-27T09:35:32.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:32 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:32.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:32 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:32.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:32 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:32.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:32 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:32.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:32 smithi050 ceph-mon[36371]: pgmap v12: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 426 B/s wr, 14 op/s 2024-05-27T09:35:32.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:32 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' 2024-05-27T09:35:32.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:32 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:32.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:32 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:32.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:32 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:32.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:32 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:32.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:32 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' 2024-05-27T09:35:32.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:32 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:32.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:32 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:32.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:32 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:32.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:32 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:32.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:32 smithi189 ceph-mon[36671]: pgmap v12: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 426 B/s wr, 14 op/s 2024-05-27T09:35:33.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.064+0000 7f4d05a00640 1 -- 172.21.15.44:0/1896616104 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4d00103790 con 0x7f4ce40b2a70 2024-05-27T09:35:33.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.211+0000 7f4cfcc00640 1 -- 172.21.15.44:0/1896616104 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4d00103790 con 0x7f4ce40b2a70 2024-05-27T09:35:33.213 INFO:teuthology.orchestra.run.smithi044.stdout:Schedule set for path / 2024-05-27T09:35:33.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.214+0000 7f4d05a00640 1 -- 172.21.15.44:0/1896616104 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4ce40b2a70 msgr2=0x7f4ce40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.214+0000 7f4d05a00640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4ce40b2a70 0x7f4ce40b4f80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0031440 tx=0x7f4cf0009560 comp rx=0 tx=0).stop 2024-05-27T09:35:33.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.214+0000 7f4d05a00640 1 -- 172.21.15.44:0/1896616104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d0010e000 msgr2=0x7f4d0019c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.214+0000 7f4d05a00640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d0010e000 0x7f4d0019c6c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009fd0 tx=0x7f4cf800ef10 comp rx=0 tx=0).stop 2024-05-27T09:35:33.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.214+0000 7f4d05a00640 1 -- 172.21.15.44:0/1896616104 shutdown_connections 2024-05-27T09:35:33.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.214+0000 7f4d05a00640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d0010e000 0x7f4d0019c6c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.214+0000 7f4d05a00640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4ce40b2a70 0x7f4ce40b4f80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.214+0000 7f4d05a00640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d0010b610 0x7f4d001a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.214+0000 7f4d05a00640 1 --2- 172.21.15.44:0/1896616104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00104bb0 0x7f4d001a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.214+0000 7f4d05a00640 1 -- 172.21.15.44:0/1896616104 >> 172.21.15.44:0/1896616104 conn(0x7f4d000fe210 msgr2=0x7f4d00109c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:33.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.215+0000 7f4d05a00640 1 -- 172.21.15.44:0/1896616104 shutdown_connections 2024-05-27T09:35:33.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.215+0000 7f4d05a00640 1 -- 172.21.15.44:0/1896616104 wait complete. 2024-05-27T09:35:33.245 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-27T09:35:33.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.517+0000 7fad4a400640 1 -- 172.21.15.44:0/4118410566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad44106600 msgr2=0x7fad441129d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.517+0000 7fad4a400640 1 --2- 172.21.15.44:0/4118410566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad44106600 0x7fad441129d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fad3c009ca0 tx=0x7fad3c02a8c0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.517+0000 7fad4a400640 1 -- 172.21.15.44:0/4118410566 shutdown_connections 2024-05-27T09:35:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.517+0000 7fad4a400640 1 --2- 172.21.15.44:0/4118410566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad44106600 0x7fad441129d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.517+0000 7fad4a400640 1 --2- 172.21.15.44:0/4118410566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4410c350 0x7fad44105f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.517+0000 7fad4a400640 1 --2- 172.21.15.44:0/4118410566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad4410beb0 0x7fad44105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.517+0000 7fad4a400640 1 -- 172.21.15.44:0/4118410566 >> 172.21.15.44:0/4118410566 conn(0x7fad441016b0 msgr2=0x7fad44103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.517+0000 7fad4a400640 1 -- 172.21.15.44:0/4118410566 shutdown_connections 2024-05-27T09:35:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.518+0000 7fad4a400640 1 -- 172.21.15.44:0/4118410566 wait complete. 2024-05-27T09:35:33.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.519+0000 7fad4a400640 1 Processor -- start 2024-05-27T09:35:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.519+0000 7fad4a400640 1 -- start start 2024-05-27T09:35:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.519+0000 7fad4a400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad44106600 0x7fad441a2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.519+0000 7fad4a400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4410beb0 0x7fad441a2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.519+0000 7fad4a400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad4410c350 0x7fad4419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.519+0000 7fad4a400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fad441160a0 con 0x7fad44106600 2024-05-27T09:35:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.519+0000 7fad4a400640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fad44115f20 con 0x7fad4410c350 2024-05-27T09:35:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.519+0000 7fad4a400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad44116220 con 0x7fad4410beb0 2024-05-27T09:35:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.520+0000 7fad43e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad4410c350 0x7fad4419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.520+0000 7fad43e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad4410c350 0x7fad4419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53290/0 (socket says 172.21.15.44:53290) 2024-05-27T09:35:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.520+0000 7fad43e00640 1 -- 172.21.15.44:0/686720879 learned_addr learned my addr 172.21.15.44:0/686720879 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.520+0000 7fad43400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad44106600 0x7fad441a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.520+0000 7fad3be00640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4410beb0 0x7fad441a2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.520+0000 7fad43e00640 1 -- 172.21.15.44:0/686720879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4410beb0 msgr2=0x7fad441a2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.520+0000 7fad43e00640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4410beb0 0x7fad441a2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.520+0000 7fad43e00640 1 -- 172.21.15.44:0/686720879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad44106600 msgr2=0x7fad441a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.520+0000 7fad43e00640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad44106600 0x7fad441a2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.520+0000 7fad43e00640 1 -- 172.21.15.44:0/686720879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4419cf20 con 0x7fad4410c350 2024-05-27T09:35:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.521+0000 7fad43e00640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad4410c350 0x7fad4419c880 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fad3c005cc0 tx=0x7fad3c031700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.521+0000 7fad40c00640 1 -- 172.21.15.44:0/686720879 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c003b70 con 0x7fad4410c350 2024-05-27T09:35:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.521+0000 7fad40c00640 1 -- 172.21.15.44:0/686720879 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad3c0318f0 con 0x7fad4410c350 2024-05-27T09:35:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.521+0000 7fad40c00640 1 -- 172.21.15.44:0/686720879 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c02f950 con 0x7fad4410c350 2024-05-27T09:35:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.521+0000 7fad4a400640 1 -- 172.21.15.44:0/686720879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad4419d1b0 con 0x7fad4410c350 2024-05-27T09:35:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.521+0000 7fad4a400640 1 -- 172.21.15.44:0/686720879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad4419d6b0 con 0x7fad4410c350 2024-05-27T09:35:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.522+0000 7fad4a400640 1 -- 172.21.15.44:0/686720879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad4419ddb0 con 0x7fad4410c350 2024-05-27T09:35:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.525+0000 7fad40c00640 1 -- 172.21.15.44:0/686720879 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7fad3c02fb10 con 0x7fad4410c350 2024-05-27T09:35:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.526+0000 7fad40c00640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad280b2af0 0x7fad280b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.526+0000 7fad40c00640 1 -- 172.21.15.44:0/686720879 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fad3c103c40 con 0x7fad4410c350 2024-05-27T09:35:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.527+0000 7fad43400640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad280b2af0 0x7fad280b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.527+0000 7fad43400640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad280b2af0 0x7fad280b5000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fad4419d9c0 tx=0x7fad34005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:33.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.529+0000 7fad40c00640 1 -- 172.21.15.44:0/686720879 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fad4419d6b0 con 0x7fad4410c350 2024-05-27T09:35:33.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.639+0000 7f0b46c00700 1 -- 172.21.15.44:0/2258550496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b40109850 msgr2=0x7f0b401111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.639+0000 7f0b46c00700 1 --2- 172.21.15.44:0/2258550496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b40109850 0x7f0b401111b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009f90 tx=0x7f0b3800b330 comp rx=0 tx=0).stop 2024-05-27T09:35:33.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.639+0000 7f0b46c00700 1 -- 172.21.15.44:0/2258550496 shutdown_connections 2024-05-27T09:35:33.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.639+0000 7f0b46c00700 1 --2- 172.21.15.44:0/2258550496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b40109850 0x7f0b401111b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.639+0000 7f0b46c00700 1 --2- 172.21.15.44:0/2258550496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b401045d0 0x7f0b401049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.639+0000 7f0b46c00700 1 --2- 172.21.15.44:0/2258550496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b40108f00 0x7f0b40109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.639+0000 7f0b46c00700 1 -- 172.21.15.44:0/2258550496 >> 172.21.15.44:0/2258550496 conn(0x7f0b400ffe60 msgr2=0x7f0b401022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:33.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.639+0000 7f0b46c00700 1 -- 172.21.15.44:0/2258550496 shutdown_connections 2024-05-27T09:35:33.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.639+0000 7f0b46c00700 1 -- 172.21.15.44:0/2258550496 wait complete. 2024-05-27T09:35:33.642 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:33 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' 2024-05-27T09:35:33.642 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:33 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:33.642 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:33 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:33.642 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:33 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:33.642 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:33 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:33.642 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:33 smithi044 ceph-mon[31618]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:33.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.641+0000 7f0b46c00700 1 Processor -- start 2024-05-27T09:35:33.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.641+0000 7f0b46c00700 1 -- start start 2024-05-27T09:35:33.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.642+0000 7f0b46c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b401045d0 0x7f0b4019d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.642+0000 7f0b46c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b40108f00 0x7f0b4019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.642+0000 7f0b46c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b40109850 0x7f0b401a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.642+0000 7f0b46c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401144d0 con 0x7f0b40108f00 2024-05-27T09:35:33.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.642+0000 7f0b46c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0b40114350 con 0x7f0b40109850 2024-05-27T09:35:33.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.642+0000 7f0b46c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0b40114650 con 0x7f0b401045d0 2024-05-27T09:35:33.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.642+0000 7f0b3fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b401045d0 0x7f0b4019d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.642+0000 7f0b3fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b401045d0 0x7f0b4019d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41920/0 (socket says 172.21.15.44:41920) 2024-05-27T09:35:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.642+0000 7f0b3fe00700 1 -- 172.21.15.44:0/2109112788 learned_addr learned my addr 172.21.15.44:0/2109112788 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.642+0000 7f0b3f400700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b40108f00 0x7f0b4019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.642+0000 7f0b44c00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b40109850 0x7f0b401a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.643+0000 7f0b3fe00700 1 -- 172.21.15.44:0/2109112788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b40109850 msgr2=0x7f0b401a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.643+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b40109850 0x7f0b401a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.643+0000 7f0b3fe00700 1 -- 172.21.15.44:0/2109112788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b40108f00 msgr2=0x7f0b4019d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.643+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b40108f00 0x7f0b4019d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.643+0000 7f0b3fe00700 1 -- 172.21.15.44:0/2109112788 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b401a1f10 con 0x7f0b401045d0 2024-05-27T09:35:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.643+0000 7f0b3f400700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b40108f00 0x7f0b4019d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:35:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.643+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b401045d0 0x7f0b4019d040 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000e9c0 tx=0x7f0b3000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.643+0000 7f0b3cc00700 1 -- 172.21.15.44:0/2109112788 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3000cbd0 con 0x7f0b401045d0 2024-05-27T09:35:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.643+0000 7f0b3cc00700 1 -- 172.21.15.44:0/2109112788 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b3000cd70 con 0x7f0b401045d0 2024-05-27T09:35:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.643+0000 7f0b3cc00700 1 -- 172.21.15.44:0/2109112788 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30005580 con 0x7f0b401045d0 2024-05-27T09:35:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.644+0000 7f0b46c00700 1 -- 172.21.15.44:0/2109112788 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b401a2200 con 0x7f0b401045d0 2024-05-27T09:35:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.644+0000 7f0b46c00700 1 -- 172.21.15.44:0/2109112788 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b401a9b20 con 0x7f0b401045d0 2024-05-27T09:35:33.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.648+0000 7f0b3cc00700 1 -- 172.21.15.44:0/2109112788 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f0b30004750 con 0x7f0b401045d0 2024-05-27T09:35:33.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b55751b80 1 Processor -- start 2024-05-27T09:35:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b55751b80 1 -- start start 2024-05-27T09:35:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b3cc00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0b240b2b10 0x7f0b240b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b3cc00700 1 -- 172.21.15.44:0/2109112788 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f0b300e2b50 con 0x7f0b401045d0 2024-05-27T09:35:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bb43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bb48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bc1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b55751b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b7b1ae2ec0 con 0x55b7b1b21a70 2024-05-27T09:35:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b55751b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b7b1ad2a60 con 0x55b7b1a35ec0 2024-05-27T09:35:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b55751b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b7b1ad6aa0 con 0x55b7b1bb4ef0 2024-05-27T09:35:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b44c00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0b240b2b10 0x7f0b240b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b3fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bb43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b3fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bb43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49748/0 (socket says 172.21.15.44:49748) 2024-05-27T09:35:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b3fe00700 1 -- 172.21.15.44:0/746154044 learned_addr learned my addr 172.21.15.44:0/746154044 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b3f400700 1 --2- 172.21.15.44:0/746154044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bb48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.650+0000 7f0b44c00700 1 --2- 172.21.15.44:0/746154044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bc1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.651+0000 7f0b3fe00700 1 -- 172.21.15.44:0/746154044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1bb4ef0 msgr2=0x55b7b1bc1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.651+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/746154044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bc1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.651+0000 7f0b3fe00700 1 -- 172.21.15.44:0/746154044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 msgr2=0x55b7b1bb48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.651+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/746154044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bb48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.651+0000 7f0b3fe00700 1 -- 172.21.15.44:0/746154044 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7b1a9b4a0 con 0x55b7b1b21a70 2024-05-27T09:35:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.651+0000 7f0b3f400700 1 --2- 172.21.15.44:0/746154044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bb48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.651+0000 7f0b44c00700 1 --2- 172.21.15.44:0/746154044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bc1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:35:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.651+0000 7f0b44c00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0b240b2b10 0x7f0b240b5020 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009840 tx=0x7f0b38003b60 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.651+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/746154044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bb43b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0b30000c00 tx=0x7f0b300a9320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590eddf8d9e3f6ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.652+0000 7f0b35600700 1 -- 172.21.15.44:0/746154044 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b300c3070 con 0x55b7b1b21a70 2024-05-27T09:35:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.652+0000 7f0b35600700 1 -- 172.21.15.44:0/746154044 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b300b2360 con 0x55b7b1b21a70 2024-05-27T09:35:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.652+0000 7f0b35600700 1 -- 172.21.15.44:0/746154044 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b300cae60 con 0x55b7b1b21a70 2024-05-27T09:35:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.653+0000 7f0b55751b80 1 -- 172.21.15.44:0/746154044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 msgr2=0x55b7b1bb43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.653+0000 7f0b55751b80 1 --2- 172.21.15.44:0/746154044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bb43b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0b30000c00 tx=0x7f0b300a9320 comp rx=0 tx=0).stop 2024-05-27T09:35:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.653+0000 7f0b55751b80 1 -- 172.21.15.44:0/746154044 shutdown_connections 2024-05-27T09:35:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.653+0000 7f0b55751b80 1 --2- 172.21.15.44:0/746154044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bc1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.653+0000 7f0b55751b80 1 --2- 172.21.15.44:0/746154044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bb43b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.653+0000 7f0b55751b80 1 --2- 172.21.15.44:0/746154044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bb48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.653+0000 7f0b55751b80 1 -- 172.21.15.44:0/746154044 >> 172.21.15.44:0/746154044 conn(0x55b7b1acf410 msgr2=0x55b7b1bb0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.653+0000 7f0b55751b80 1 -- 172.21.15.44:0/746154044 shutdown_connections 2024-05-27T09:35:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.653+0000 7f0b55751b80 1 -- 172.21.15.44:0/746154044 wait complete. 2024-05-27T09:35:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.654+0000 7f0b55751b80 1 -- start start 2024-05-27T09:35:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.654+0000 7f0b55751b80 1 Processor -- start 2024-05-27T09:35:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bbe6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bbebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bbf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b55751b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b7b19d9020 con 0x55b7b1b21a70 2024-05-27T09:35:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b55751b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b7b1a737e0 con 0x55b7b1a35ec0 2024-05-27T09:35:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b55751b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b7b1bc1c50 con 0x55b7b1bb4ef0 2024-05-27T09:35:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b55751b80 10 client.0 _ll_register_callbacks cb 0x55b7b1a7ccf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:35:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b3fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bbe6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b3f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bbebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b44c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bbf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b3f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bbebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49756/0 (socket says 172.21.15.44:49756) 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.655+0000 7f0b3f400700 1 -- 172.21.15.44:0/812018948 learned_addr learned my addr 172.21.15.44:0/812018948 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.656+0000 7f0b3fe00700 1 -- 172.21.15.44:0/812018948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1bb4ef0 msgr2=0x55b7b1bbf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.656+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/812018948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bbf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.656+0000 7f0b3fe00700 1 -- 172.21.15.44:0/812018948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 msgr2=0x55b7b1bbebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.656+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/812018948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bbebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.656+0000 7f0b3fe00700 1 -- 172.21.15.44:0/812018948 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7b1a22950 con 0x55b7b1a35ec0 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.656+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/812018948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bbe6b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0b300a5c30 tx=0x7f0b30005870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8072f8a5f6b6e16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.656+0000 7f0b2a000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.656+0000 7f0b2a000700 1 -- 172.21.15.44:0/812018948 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3006d020 con 0x55b7b1a35ec0 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.656+0000 7f0b2a000700 1 -- 172.21.15.44:0/812018948 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f0b300b2790 con 0x55b7b1a35ec0 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.656+0000 7f0b55751b80 1 -- 172.21.15.44:0/812018948 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b300b2360 con 0x55b7b1a35ec0 2024-05-27T09:35:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.657+0000 7f0b2a000700 1 -- 172.21.15.44:0/812018948 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0b300c3070 con 0x55b7b1a35ec0 2024-05-27T09:35:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.657+0000 7f0b55751b80 10 client.25229 fetch_fsmap learned FSMap version 26 2024-05-27T09:35:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.657+0000 7f0b55751b80 1 -- 172.21.15.44:0/812018948 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b7b1aa30e0 con 0x55b7b1a35ec0 2024-05-27T09:35:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.657+0000 7f0b2a000700 1 -- 172.21.15.44:0/812018948 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b300b2360 con 0x55b7b1a35ec0 2024-05-27T09:35:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.657+0000 7f0b2a000700 1 -- 172.21.15.44:0/812018948 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f0b300e03d0 con 0x55b7b1a35ec0 2024-05-27T09:35:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.657+0000 7f0b55751b80 10 client.25229 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:35:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.657+0000 7f0b55751b80 10 client.25229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:35:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.657+0000 7f0b55751b80 20 client.25229 populate_metadata read hostname 'smithi044' 2024-05-27T09:35:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.657+0000 7f0b55751b80 1 --2- 172.21.15.44:0/812018948 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b7b1bcea20 0x55b7b1bd0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.657+0000 7f0b55751b80 4 client.25229 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:35:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.657+0000 7f0b55751b80 1 -- 172.21.15.44:0/812018948 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7b1ae4e10 con 0x55b7b1bcea20 2024-05-27T09:35:33.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.658+0000 7f0b3f400700 1 --2- 172.21.15.44:0/812018948 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b7b1bcea20 0x55b7b1bd0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.658+0000 7f0b3f400700 1 --2- 172.21.15.44:0/812018948 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b7b1bcea20 0x55b7b1bd0e40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f21850c8ec61ff91 server_cookie=d334b191841c31b1 in_seq=0 out_seq=0 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.658+0000 7f0b2a000700 10 client.25229 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.659+0000 7f0b2a000700 1 -- 172.21.15.44:0/812018948 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b7b1ae4e10 con 0x55b7b1bcea20 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.659+0000 7f0b2a000700 10 client.25229 handle_command_reply: tid=0 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.659+0000 7f0b55751b80 1 client.25229 shutdown 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.659+0000 7f0b55751b80 1 -- 172.21.15.44:0/812018948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 msgr2=0x55b7b1bbe6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.659+0000 7f0b55751b80 1 --2- 172.21.15.44:0/812018948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bbe6b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0b300a5c30 tx=0x7f0b30005870 comp rx=0 tx=0).stop 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.660+0000 7f0b55751b80 1 -- 172.21.15.44:0/812018948 shutdown_connections 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.660+0000 7f0b55751b80 1 --2- 172.21.15.44:0/812018948 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b7b1bcea20 0x55b7b1bd0e40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.660+0000 7f0b55751b80 1 --2- 172.21.15.44:0/812018948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bbf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.660+0000 7f0b55751b80 1 --2- 172.21.15.44:0/812018948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bbe6b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.660+0000 7f0b55751b80 1 --2- 172.21.15.44:0/812018948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bbebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.660+0000 7f0b2a000700 0 client.25229 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.660+0000 7f0b55751b80 1 -- 172.21.15.44:0/812018948 >> 172.21.15.44:0/812018948 conn(0x55b7b1acf410 msgr2=0x55b7b1bb4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.660+0000 7f0b55751b80 1 -- 172.21.15.44:0/812018948 shutdown_connections 2024-05-27T09:35:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.660+0000 7f0b55751b80 1 -- 172.21.15.44:0/812018948 wait complete. 2024-05-27T09:35:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.660+0000 7f0b55751b80 20 client.25229 trim_cache size 0 max 16384 2024-05-27T09:35:33.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.682+0000 7f0b55751b80 1 Processor -- start 2024-05-27T09:35:33.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.682+0000 7f0b55751b80 1 -- start start 2024-05-27T09:35:33.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.683+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bbd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.683+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1b21a70 0x55b7b1bcaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.683+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bc0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.683+0000 7f0b55751b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401b3910 con 0x55b7b1bb4ef0 2024-05-27T09:35:33.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.683+0000 7f0b55751b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401b11a0 con 0x55b7b1b21a70 2024-05-27T09:35:33.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.683+0000 7f0b55751b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401aeff0 con 0x55b7b1a35ec0 2024-05-27T09:35:33.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.683+0000 7f0b3fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bbd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.683+0000 7f0b44c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bc0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.683+0000 7f0b3fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bbd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41932/0 (socket says 172.21.15.44:41932) 2024-05-27T09:35:33.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.683+0000 7f0b3fe00700 1 -- 172.21.15.44:0/67783252 learned_addr learned my addr 172.21.15.44:0/67783252 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:33.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.683+0000 7f0b3f400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1b21a70 0x55b7b1bcaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.684+0000 7f0b3f400700 1 -- 172.21.15.44:0/67783252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 msgr2=0x55b7b1bbd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.684+0000 7f0b3f400700 1 --2- 172.21.15.44:0/67783252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bbd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.684+0000 7f0b3f400700 1 -- 172.21.15.44:0/67783252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1bb4ef0 msgr2=0x55b7b1bc0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.684+0000 7f0b3f400700 1 --2- 172.21.15.44:0/67783252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bc0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.684+0000 7f0b3f400700 1 -- 172.21.15.44:0/67783252 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7b1bc1ae0 con 0x55b7b1b21a70 2024-05-27T09:35:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.684+0000 7f0b3f400700 1 --2- 172.21.15.44:0/67783252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1b21a70 0x55b7b1bcaef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55b7b1ad2bb0 tx=0x7f0b2c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16a65a872c69da8e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.685+0000 7f0b34c00700 1 -- 172.21.15.44:0/67783252 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c003a60 con 0x55b7b1b21a70 2024-05-27T09:35:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.685+0000 7f0b34c00700 1 -- 172.21.15.44:0/67783252 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b2c003c00 con 0x55b7b1b21a70 2024-05-27T09:35:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.685+0000 7f0b34c00700 1 -- 172.21.15.44:0/67783252 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c01d3f0 con 0x55b7b1b21a70 2024-05-27T09:35:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.686+0000 7f0b55751b80 1 -- 172.21.15.44:0/67783252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1b21a70 msgr2=0x55b7b1bcaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.686+0000 7f0b55751b80 1 --2- 172.21.15.44:0/67783252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1b21a70 0x55b7b1bcaef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55b7b1ad2bb0 tx=0x7f0b2c009260 comp rx=0 tx=0).stop 2024-05-27T09:35:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.686+0000 7f0b55751b80 1 -- 172.21.15.44:0/67783252 shutdown_connections 2024-05-27T09:35:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.686+0000 7f0b55751b80 1 --2- 172.21.15.44:0/67783252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1bc0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.686+0000 7f0b55751b80 1 --2- 172.21.15.44:0/67783252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bbd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.686+0000 7f0b55751b80 1 --2- 172.21.15.44:0/67783252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1b21a70 0x55b7b1bcaef0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.686+0000 7f0b55751b80 1 -- 172.21.15.44:0/67783252 >> 172.21.15.44:0/67783252 conn(0x55b7b1acf410 msgr2=0x55b7b1bbfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.686+0000 7f0b55751b80 1 -- 172.21.15.44:0/67783252 shutdown_connections 2024-05-27T09:35:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.686+0000 7f0b55751b80 1 -- 172.21.15.44:0/67783252 wait complete. 2024-05-27T09:35:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.686+0000 7f0b55751b80 1 -- start start 2024-05-27T09:35:33.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.687+0000 7f0b55751b80 1 Processor -- start 2024-05-27T09:35:33.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.687+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bd2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.687+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bd2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.687+0000 7f0b55751b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1c38290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.687+0000 7f0b55751b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401af660 con 0x55b7b1b21a70 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.687+0000 7f0b55751b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401b0ad0 con 0x55b7b1bb4ef0 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.687+0000 7f0b55751b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401afd30 con 0x55b7b1a35ec0 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.687+0000 7f0b55751b80 10 client.0 _ll_register_callbacks cb 0x55b7b1a7ccf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.687+0000 7f0b3f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bd2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b3f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bd2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49776/0 (socket says 172.21.15.44:49776) 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b3f400700 1 -- 172.21.15.44:0/2504909187 learned_addr learned my addr 172.21.15.44:0/2504909187 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bd2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b44c00700 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1c38290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b3f400700 1 -- 172.21.15.44:0/2504909187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 msgr2=0x55b7b1bd2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b3f400700 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bd2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b3f400700 1 -- 172.21.15.44:0/2504909187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1bb4ef0 msgr2=0x55b7b1c38290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:33.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b3f400700 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1c38290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b3f400700 1 -- 172.21.15.44:0/2504909187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7b1bbf4a0 con 0x55b7b1b21a70 2024-05-27T09:35:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bd2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b3f400700 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bd2770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0099d0 tx=0x7f0b2c003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d44e7239833509e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.688+0000 7f0b2a000700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:35:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.689+0000 7f0b2a000700 1 -- 172.21.15.44:0/2504909187 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c011c80 con 0x55b7b1b21a70 2024-05-27T09:35:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.689+0000 7f0b2a000700 1 -- 172.21.15.44:0/2504909187 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f0b2c011440 con 0x55b7b1b21a70 2024-05-27T09:35:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.689+0000 7f0b55751b80 1 -- 172.21.15.44:0/2504909187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b2c003c00 con 0x55b7b1b21a70 2024-05-27T09:35:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.689+0000 7f0b2a000700 1 -- 172.21.15.44:0/2504909187 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0b2c011e20 con 0x55b7b1b21a70 2024-05-27T09:35:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.689+0000 7f0b55751b80 10 client.15096 fetch_fsmap learned FSMap version 26 2024-05-27T09:35:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.689+0000 7f0b55751b80 1 -- 172.21.15.44:0/2504909187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b7b1bac2c0 con 0x55b7b1b21a70 2024-05-27T09:35:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.689+0000 7f0b2a000700 1 -- 172.21.15.44:0/2504909187 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b2c003c00 con 0x55b7b1b21a70 2024-05-27T09:35:33.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.690+0000 7f0b2a000700 1 -- 172.21.15.44:0/2504909187 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f0b2c02e070 con 0x55b7b1b21a70 2024-05-27T09:35:33.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.690+0000 7f0b55751b80 10 client.15096 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:35:33.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.690+0000 7f0b55751b80 10 client.15096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:35:33.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.690+0000 7f0b55751b80 20 client.15096 populate_metadata read hostname 'smithi044' 2024-05-27T09:35:33.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.690+0000 7f0b55751b80 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b7b1c39ae0 0x55b7b1c3bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:33.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.690+0000 7f0b55751b80 4 client.15096 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:35:33.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.690+0000 7f0b55751b80 1 -- 172.21.15.44:0/2504909187 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b7b1ade5f0 con 0x55b7b1c39ae0 2024-05-27T09:35:33.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.690+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b7b1c39ae0 0x55b7b1c3bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:33.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.691+0000 7f0b3fe00700 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b7b1c39ae0 0x55b7b1c3bf00 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0046a0b038dbe8 server_cookie=d4094b370c304ce2 in_seq=0 out_seq=0 2024-05-27T09:35:33.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.691+0000 7f0b2a000700 10 client.15096 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:33.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.692+0000 7f0b2a000700 1 -- 172.21.15.44:0/2504909187 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b7b1ade5f0 con 0x55b7b1c39ae0 2024-05-27T09:35:33.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.692+0000 7f0b2a000700 10 client.15096 handle_command_reply: tid=0 2024-05-27T09:35:33.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.692+0000 7f0b55751b80 1 client.15096 shutdown 2024-05-27T09:35:33.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.692+0000 7f0b55751b80 1 -- 172.21.15.44:0/2504909187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 msgr2=0x55b7b1bd2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.692+0000 7f0b55751b80 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bd2770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0099d0 tx=0x7f0b2c003af0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.693+0000 7f0b55751b80 1 -- 172.21.15.44:0/2504909187 shutdown_connections 2024-05-27T09:35:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.693+0000 7f0b55751b80 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b7b1c39ae0 0x55b7b1c3bf00 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.693+0000 7f0b55751b80 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7b1bb4ef0 0x55b7b1c38290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.693+0000 7f0b55751b80 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b7b1a35ec0 0x55b7b1bd2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.693+0000 7f0b55751b80 1 --2- 172.21.15.44:0/2504909187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7b1b21a70 0x55b7b1bd2770 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.693+0000 7f0b55751b80 1 -- 172.21.15.44:0/2504909187 >> 172.21.15.44:0/2504909187 conn(0x55b7b1acf410 msgr2=0x55b7b1bbd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.693+0000 7f0b2a000700 0 client.15096 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.693+0000 7f0b55751b80 1 -- 172.21.15.44:0/2504909187 shutdown_connections 2024-05-27T09:35:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.693+0000 7f0b55751b80 1 -- 172.21.15.44:0/2504909187 wait complete. 2024-05-27T09:35:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.693+0000 7f0b55751b80 20 client.15096 trim_cache size 0 max 16384 2024-05-27T09:35:33.697 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:35:33.698 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:35:33.698 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "bb6e9c27-6db5-4339-aa03-d1f54f0f55e0", 2024-05-27T09:35:33.698 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:35:33.698 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:35:33.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 -- 172.21.15.44:0/2109112788 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0b240b2b10 msgr2=0x7f0b240b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0b240b2b10 0x7f0b240b5020 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009840 tx=0x7f0b38003b60 comp rx=0 tx=0).stop 2024-05-27T09:35:33.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 -- 172.21.15.44:0/2109112788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b401045d0 msgr2=0x7f0b4019d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b401045d0 0x7f0b4019d040 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000e9c0 tx=0x7f0b3000ecd0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 -- 172.21.15.44:0/2109112788 shutdown_connections 2024-05-27T09:35:33.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0b240b2b10 0x7f0b240b5020 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b40109850 0x7f0b401a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b401045d0 0x7f0b4019d040 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 --2- 172.21.15.44:0/2109112788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b40108f00 0x7f0b4019d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 -- 172.21.15.44:0/2109112788 >> 172.21.15.44:0/2109112788 conn(0x7f0b400ffe60 msgr2=0x7f0b4010fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:33.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 -- 172.21.15.44:0/2109112788 shutdown_connections 2024-05-27T09:35:33.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.694+0000 7f0b46c00700 1 -- 172.21.15.44:0/2109112788 wait complete. 2024-05-27T09:35:33.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.780+0000 7fad4a400640 1 -- 172.21.15.44:0/686720879 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fad44063100 con 0x7fad280b2af0 2024-05-27T09:35:33.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:33 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' 2024-05-27T09:35:33.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:33 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:33.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:33 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:33.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:33 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:33.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:33 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:33.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:33 smithi050 ceph-mon[36371]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:33.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:33 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' 2024-05-27T09:35:33.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:33 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:35:33.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:33 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:35:33.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:33 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:35:33.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:33 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:35:33.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:33 smithi189 ceph-mon[36671]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:33.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.896+0000 7fad40c00640 1 -- 172.21.15.44:0/686720879 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fad44063100 con 0x7fad280b2af0 2024-05-27T09:35:33.898 INFO:teuthology.orchestra.run.smithi044.stdout:Retention added to path / 2024-05-27T09:35:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 -- 172.21.15.44:0/686720879 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad280b2af0 msgr2=0x7fad280b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad280b2af0 0x7fad280b5000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fad4419d9c0 tx=0x7fad34005f90 comp rx=0 tx=0).stop 2024-05-27T09:35:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 -- 172.21.15.44:0/686720879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad4410c350 msgr2=0x7fad4419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad4410c350 0x7fad4419c880 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fad3c005cc0 tx=0x7fad3c031700 comp rx=0 tx=0).stop 2024-05-27T09:35:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 -- 172.21.15.44:0/686720879 shutdown_connections 2024-05-27T09:35:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad4410c350 0x7fad4419c880 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4410beb0 0x7fad441a2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad280b2af0 0x7fad280b5000 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 --2- 172.21.15.44:0/686720879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad44106600 0x7fad441a2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:33.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 -- 172.21.15.44:0/686720879 >> 172.21.15.44:0/686720879 conn(0x7fad441016b0 msgr2=0x7fad441087d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:33.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 -- 172.21.15.44:0/686720879 shutdown_connections 2024-05-27T09:35:33.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:33.900+0000 7fad4a400640 1 -- 172.21.15.44:0/686720879 wait complete. 2024-05-27T09:35:33.925 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-27T09:35:34.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.205+0000 7f91e7200640 1 -- 172.21.15.44:0/404059310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91e810bce0 msgr2=0x7f91e8105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:34.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.205+0000 7f91e7200640 1 --2- 172.21.15.44:0/404059310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91e810bce0 0x7f91e8105880 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009860 tx=0x7f91d802a340 comp rx=0 tx=0).stop 2024-05-27T09:35:34.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.205+0000 7f91e7200640 1 -- 172.21.15.44:0/404059310 shutdown_connections 2024-05-27T09:35:34.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.205+0000 7f91e7200640 1 --2- 172.21.15.44:0/404059310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91e8106430 0x7f91e8112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.205+0000 7f91e7200640 1 --2- 172.21.15.44:0/404059310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91e810c180 0x7f91e8105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.205+0000 7f91e7200640 1 --2- 172.21.15.44:0/404059310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91e810bce0 0x7f91e8105880 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.205+0000 7f91e7200640 1 -- 172.21.15.44:0/404059310 >> 172.21.15.44:0/404059310 conn(0x7f91e8101380 msgr2=0x7f91e81037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:34.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.205+0000 7f91e7200640 1 -- 172.21.15.44:0/404059310 shutdown_connections 2024-05-27T09:35:34.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.206+0000 7f91e7200640 1 -- 172.21.15.44:0/404059310 wait complete. 2024-05-27T09:35:34.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.207+0000 7f91e7200640 1 Processor -- start 2024-05-27T09:35:34.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.207+0000 7f91e7200640 1 -- start start 2024-05-27T09:35:34.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.208+0000 7f91e7200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91e8106430 0x7f91e81a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:34.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.208+0000 7f91e7200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91e810bce0 0x7f91e81a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:34.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.208+0000 7f91e7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91e810c180 0x7f91e819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.208+0000 7f91e7200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f91e8115ed0 con 0x7f91e8106430 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.208+0000 7f91e7200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f91e8115d50 con 0x7f91e810bce0 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.208+0000 7f91e7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f91e8116050 con 0x7f91e810c180 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.208+0000 7f91e5e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91e8106430 0x7f91e81a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.208+0000 7f91e5400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91e810bce0 0x7f91e81a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.208+0000 7f91e5400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91e810bce0 0x7f91e81a2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53362/0 (socket says 172.21.15.44:53362) 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.208+0000 7f91e5400640 1 -- 172.21.15.44:0/3307078229 learned_addr learned my addr 172.21.15.44:0/3307078229 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.208+0000 7f91e6800640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91e810c180 0x7f91e819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.209+0000 7f91e5400640 1 -- 172.21.15.44:0/3307078229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91e810c180 msgr2=0x7f91e819c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.209+0000 7f91e5400640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91e810c180 0x7f91e819c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.209+0000 7f91e5400640 1 -- 172.21.15.44:0/3307078229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91e8106430 msgr2=0x7f91e81a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:34.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.209+0000 7f91e5400640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91e8106430 0x7f91e81a25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.209+0000 7f91e5400640 1 -- 172.21.15.44:0/3307078229 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e819ce60 con 0x7f91e810bce0 2024-05-27T09:35:34.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.209+0000 7f91e5400640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91e810bce0 0x7f91e81a2af0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f91d400e980 tx=0x7f91d400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:34.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.210+0000 7f91dea00640 1 -- 172.21.15.44:0/3307078229 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d400cd80 con 0x7f91e810bce0 2024-05-27T09:35:34.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.210+0000 7f91e7200640 1 -- 172.21.15.44:0/3307078229 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e819d0f0 con 0x7f91e810bce0 2024-05-27T09:35:34.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.210+0000 7f91dea00640 1 -- 172.21.15.44:0/3307078229 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91d40108e0 con 0x7f91e810bce0 2024-05-27T09:35:34.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.210+0000 7f91dea00640 1 -- 172.21.15.44:0/3307078229 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d40188f0 con 0x7f91e810bce0 2024-05-27T09:35:34.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.210+0000 7f91e7200640 1 -- 172.21.15.44:0/3307078229 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e81a9350 con 0x7f91e810bce0 2024-05-27T09:35:34.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.211+0000 7f91e7200640 1 -- 172.21.15.44:0/3307078229 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e8106fc0 con 0x7f91e810bce0 2024-05-27T09:35:34.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.213+0000 7f91dea00640 1 -- 172.21.15.44:0/3307078229 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f91d401f020 con 0x7f91e810bce0 2024-05-27T09:35:34.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.215+0000 7f91dea00640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91cc0b2a70 0x7f91cc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:34.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.215+0000 7f91dea00640 1 -- 172.21.15.44:0/3307078229 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f91d40e2fb0 con 0x7f91e810bce0 2024-05-27T09:35:34.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.215+0000 7f91e5e00640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91cc0b2a70 0x7f91cc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:34.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.216+0000 7f91e5e00640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91cc0b2a70 0x7f91cc0b4f80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009830 tx=0x7f91d80058d0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:34.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.217+0000 7f91dea00640 1 -- 172.21.15.44:0/3307078229 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f91d40aa710 con 0x7f91e810bce0 2024-05-27T09:35:34.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.453+0000 7f91e7200640 1 -- 172.21.15.44:0/3307078229 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f91e8101ea0 con 0x7f91cc0b2a70 2024-05-27T09:35:34.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.469+0000 7f91dea00640 1 -- 172.21.15.44:0/3307078229 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f91e8101ea0 con 0x7f91cc0b2a70 2024-05-27T09:35:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-27T00:00:00", "created": "2024-05-27T09:35:33", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-27T09:35:34.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 -- 172.21.15.44:0/3307078229 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91cc0b2a70 msgr2=0x7f91cc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:34.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91cc0b2a70 0x7f91cc0b4f80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009830 tx=0x7f91d80058d0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 -- 172.21.15.44:0/3307078229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91e810bce0 msgr2=0x7f91e81a2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:34.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91e810bce0 0x7f91e81a2af0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f91d400e980 tx=0x7f91d400ee40 comp rx=0 tx=0).stop 2024-05-27T09:35:34.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 -- 172.21.15.44:0/3307078229 shutdown_connections 2024-05-27T09:35:34.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91e810c180 0x7f91e819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91e810bce0 0x7f91e81a2af0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91cc0b2a70 0x7f91cc0b4f80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 --2- 172.21.15.44:0/3307078229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91e8106430 0x7f91e81a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 -- 172.21.15.44:0/3307078229 >> 172.21.15.44:0/3307078229 conn(0x7f91e8101380 msgr2=0x7f91e8101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:34.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 -- 172.21.15.44:0/3307078229 shutdown_connections 2024-05-27T09:35:34.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.472+0000 7f91e7200640 1 -- 172.21.15.44:0/3307078229 wait complete. 2024-05-27T09:35:34.498 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-27T09:35:34.576 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:35:34.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.765+0000 7f23c5800640 1 -- 172.21.15.44:0/463558857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23c0075480 msgr2=0x7f23c0075890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:34.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.765+0000 7f23c5800640 1 --2- 172.21.15.44:0/463558857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23c0075480 0x7f23c0075890 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009900 tx=0x7f23b002a480 comp rx=0 tx=0).stop 2024-05-27T09:35:34.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.765+0000 7f23c5800640 1 -- 172.21.15.44:0/463558857 shutdown_connections 2024-05-27T09:35:34.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.765+0000 7f23c5800640 1 --2- 172.21.15.44:0/463558857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c0114a70 0x7f23c0116e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.765+0000 7f23c5800640 1 --2- 172.21.15.44:0/463558857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c0112110 0x7f23c0114530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.765+0000 7f23c5800640 1 --2- 172.21.15.44:0/463558857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23c0075480 0x7f23c0075890 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.765+0000 7f23c5800640 1 -- 172.21.15.44:0/463558857 >> 172.21.15.44:0/463558857 conn(0x7f23c0079140 msgr2=0x7f23c0079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:34.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.765+0000 7f23c5800640 1 -- 172.21.15.44:0/463558857 shutdown_connections 2024-05-27T09:35:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.765+0000 7f23c5800640 1 -- 172.21.15.44:0/463558857 wait complete. 2024-05-27T09:35:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.767+0000 7f23c5800640 1 Processor -- start 2024-05-27T09:35:34.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.767+0000 7f23c5800640 1 -- start start 2024-05-27T09:35:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.767+0000 7f23c5800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c0075480 0x7f23c01a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.767+0000 7f23c5800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c0112110 0x7f23c01a6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23c5800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23c0114a70 0x7f23c01a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23c5800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f23c011a260 con 0x7f23c0112110 2024-05-27T09:35:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23c5800640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f23c011a0e0 con 0x7f23c0114a70 2024-05-27T09:35:34.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23c5800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23c011a3e0 con 0x7f23c0075480 2024-05-27T09:35:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23bf400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c0112110 0x7f23c01a6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23bfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c0075480 0x7f23c01a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23c4e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23c0114a70 0x7f23c01a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23bf400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c0112110 0x7f23c01a6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49812/0 (socket says 172.21.15.44:49812) 2024-05-27T09:35:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23bfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c0075480 0x7f23c01a68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41972/0 (socket says 172.21.15.44:41972) 2024-05-27T09:35:34.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23bf400640 1 -- 172.21.15.44:0/4174263932 learned_addr learned my addr 172.21.15.44:0/4174263932 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:34.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23c4e00640 1 -- 172.21.15.44:0/4174263932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c0075480 msgr2=0x7f23c01a68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:34.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23c4e00640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c0075480 0x7f23c01a68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23c4e00640 1 -- 172.21.15.44:0/4174263932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c0112110 msgr2=0x7f23c01a6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:34.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23c4e00640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c0112110 0x7f23c01a6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:34.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.768+0000 7f23c4e00640 1 -- 172.21.15.44:0/4174263932 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c01a1160 con 0x7f23c0114a70 2024-05-27T09:35:34.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.769+0000 7f23c4e00640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23c0114a70 0x7f23c01a0ac0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009fd0 tx=0x7f23b800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:34.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.769+0000 7f23bcc00640 1 -- 172.21.15.44:0/4174263932 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b800ce20 con 0x7f23c0114a70 2024-05-27T09:35:34.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.769+0000 7f23bcc00640 1 -- 172.21.15.44:0/4174263932 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23b8005020 con 0x7f23c0114a70 2024-05-27T09:35:34.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.769+0000 7f23c5800640 1 -- 172.21.15.44:0/4174263932 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23c01a1450 con 0x7f23c0114a70 2024-05-27T09:35:34.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.769+0000 7f23bcc00640 1 -- 172.21.15.44:0/4174263932 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b8010690 con 0x7f23c0114a70 2024-05-27T09:35:34.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.769+0000 7f23c5800640 1 -- 172.21.15.44:0/4174263932 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23c01ad730 con 0x7f23c0114a70 2024-05-27T09:35:34.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.771+0000 7f23c5800640 1 -- 172.21.15.44:0/4174263932 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23c006a230 con 0x7f23c0114a70 2024-05-27T09:35:34.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.773+0000 7f23bcc00640 1 -- 172.21.15.44:0/4174263932 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f23b80054d0 con 0x7f23c0114a70 2024-05-27T09:35:34.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.774+0000 7f23bcc00640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23a40b2a70 0x7f23a40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:34.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.774+0000 7f23bcc00640 1 -- 172.21.15.44:0/4174263932 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f23b80e2f60 con 0x7f23c0114a70 2024-05-27T09:35:34.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.775+0000 7f23bfe00640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23a40b2a70 0x7f23a40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:34.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.776+0000 7f23bfe00640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23a40b2a70 0x7f23a40b4f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f23b0000c00 tx=0x7f23b0031400 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:34.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:34.779+0000 7f23bcc00640 1 -- 172.21.15.44:0/4174263932 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f23b80aa6c0 con 0x7f23c0114a70 2024-05-27T09:35:34.793 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:34 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:35:34.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:34 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:35:34.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:34 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:35:34.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:34 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:35:34.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:34 smithi050 ceph-mon[36371]: from='client.25217 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:34.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:34 smithi050 ceph-mon[36371]: pgmap v13: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:35:34.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:34 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:35:34.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:34 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:35:34.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:34 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:35:34.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:34 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:35:34.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:34 smithi189 ceph-mon[36671]: from='client.25217 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:34.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:34 smithi189 ceph-mon[36671]: pgmap v13: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:35:34.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:34 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:35:34.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:34 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:35:34.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:34 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:35:34.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:34 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:35:34.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:34 smithi044 ceph-mon[31618]: from='client.25217 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:34.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:34 smithi044 ceph-mon[31618]: pgmap v13: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:35:35.016 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:35.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.024+0000 7f23c5800640 1 -- 172.21.15.44:0/4174263932 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f23c0112c30 con 0x7f23a40b2a70 2024-05-27T09:35:35.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.035+0000 7f23bcc00640 1 -- 172.21.15.44:0/4174263932 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f23c0112c30 con 0x7f23a40b2a70 2024-05-27T09:35:35.036 INFO:teuthology.orchestra.run.smithi044.stdout:/ 1m 6m3h 2024-05-27T09:35:35.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 -- 172.21.15.44:0/4174263932 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23a40b2a70 msgr2=0x7f23a40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:35.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23a40b2a70 0x7f23a40b4f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f23b0000c00 tx=0x7f23b0031400 comp rx=0 tx=0).stop 2024-05-27T09:35:35.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 -- 172.21.15.44:0/4174263932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23c0114a70 msgr2=0x7f23c01a0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:35.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23c0114a70 0x7f23c01a0ac0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009fd0 tx=0x7f23b800ef10 comp rx=0 tx=0).stop 2024-05-27T09:35:35.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 -- 172.21.15.44:0/4174263932 shutdown_connections 2024-05-27T09:35:35.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23c0114a70 0x7f23c01a0ac0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:35.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23a40b2a70 0x7f23a40b4f80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:35.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23c0112110 0x7f23c01a6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:35.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 --2- 172.21.15.44:0/4174263932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c0075480 0x7f23c01a68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:35.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 -- 172.21.15.44:0/4174263932 >> 172.21.15.44:0/4174263932 conn(0x7f23c0079140 msgr2=0x7f23c0112520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:35.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 -- 172.21.15.44:0/4174263932 shutdown_connections 2024-05-27T09:35:35.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:35.038+0000 7f23c5800640 1 -- 172.21.15.44:0/4174263932 wait complete. 2024-05-27T09:35:35.065 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-27T09:35:35.137 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-27T09:35:35.148 INFO:teuthology.run_tasks:Running task background_exec... 2024-05-27T09:35:35.164 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-05-27T09:35:35.164 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-05-27T09:35:35.172 INFO:teuthology.run_tasks:Running task quiescer... 2024-05-27T09:35:35.199 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-27T09:35:35.199 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T09:35:35.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:35 smithi044 ceph-mon[31618]: from='client.25241 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:35.594 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:35 smithi044 ceph-mon[31618]: from='client.25247 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:35.793 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:35 smithi050 ceph-mon[36371]: from='client.25241 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:35.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:35 smithi050 ceph-mon[36371]: from='client.25247 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:35.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:35 smithi189 ceph-mon[36671]: from='client.25241 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:35.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:35 smithi189 ceph-mon[36671]: from='client.25247 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:35:36.409 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:36.566 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:36 smithi044 ceph-mon[31618]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T09:35:36.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:36 smithi050 ceph-mon[36371]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T09:35:36.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:36 smithi189 ceph-mon[36671]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T09:35:37.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.369+0000 7fa287600700 1 -- 172.21.15.44:0/1485596597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa280106ec0 msgr2=0x7fa280113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:37.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.369+0000 7fa287600700 1 --2- 172.21.15.44:0/1485596597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa280106ec0 0x7fa280113350 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa27800b0a0 tx=0x7fa27800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.369+0000 7fa287600700 1 -- 172.21.15.44:0/1485596597 shutdown_connections 2024-05-27T09:35:37.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.369+0000 7fa287600700 1 --2- 172.21.15.44:0/1485596597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa280106ec0 0x7fa280113350 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.369+0000 7fa287600700 1 --2- 172.21.15.44:0/1485596597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa28010c5b0 0x7fa28010c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.369+0000 7fa287600700 1 --2- 172.21.15.44:0/1485596597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa280106570 0x7fa280106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.369+0000 7fa287600700 1 -- 172.21.15.44:0/1485596597 >> 172.21.15.44:0/1485596597 conn(0x7fa280102070 msgr2=0x7fa2801044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:37.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.369+0000 7fa287600700 1 -- 172.21.15.44:0/1485596597 shutdown_connections 2024-05-27T09:35:37.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.370+0000 7fa287600700 1 -- 172.21.15.44:0/1485596597 wait complete. 2024-05-27T09:35:37.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.371+0000 7fa287600700 1 Processor -- start 2024-05-27T09:35:37.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.371+0000 7fa287600700 1 -- start start 2024-05-27T09:35:37.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.371+0000 7fa287600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa280106570 0x7fa28019d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.371+0000 7fa287600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa280106ec0 0x7fa28019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.371+0000 7fa287600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa28010c5b0 0x7fa2801a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.371+0000 7fa287600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa280116730 con 0x7fa280106ec0 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.371+0000 7fa287600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa2801165b0 con 0x7fa280106570 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.371+0000 7fa287600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2801168b0 con 0x7fa28010c5b0 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.372+0000 7fa27fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa280106ec0 0x7fa28019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.372+0000 7fa284c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa280106570 0x7fa28019d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.372+0000 7fa285600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa28010c5b0 0x7fa2801a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.372+0000 7fa27fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa280106ec0 0x7fa28019d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49824/0 (socket says 172.21.15.44:49824) 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.372+0000 7fa27fe00700 1 -- 172.21.15.44:0/2230790290 learned_addr learned my addr 172.21.15.44:0/2230790290 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.372+0000 7fa285600700 1 -- 172.21.15.44:0/2230790290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa280106570 msgr2=0x7fa28019d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.372+0000 7fa285600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa280106570 0x7fa28019d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.372+0000 7fa285600700 1 -- 172.21.15.44:0/2230790290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa280106ec0 msgr2=0x7fa28019d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.372+0000 7fa285600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa280106ec0 0x7fa28019d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.372+0000 7fa285600700 1 -- 172.21.15.44:0/2230790290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2801a1ed0 con 0x7fa28010c5b0 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.372+0000 7fa27fe00700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa280106ec0 0x7fa28019d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.373+0000 7fa285600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa28010c5b0 0x7fa2801a1830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa27800b070 tx=0x7fa278004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.373+0000 7fa27d600700 1 -- 172.21.15.44:0/2230790290 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa278021070 con 0x7fa28010c5b0 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.373+0000 7fa27d600700 1 -- 172.21.15.44:0/2230790290 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa278007e30 con 0x7fa28010c5b0 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.373+0000 7fa287600700 1 -- 172.21.15.44:0/2230790290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2801a2160 con 0x7fa28010c5b0 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.373+0000 7fa27d600700 1 -- 172.21.15.44:0/2230790290 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2780045f0 con 0x7fa28010c5b0 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.373+0000 7fa287600700 1 -- 172.21.15.44:0/2230790290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2801a9ae0 con 0x7fa28010c5b0 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.379+0000 7fa27d600700 1 -- 172.21.15.44:0/2230790290 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7fa278007980 con 0x7fa28010c5b0 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.379+0000 7fa2960beb80 1 Processor -- start 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.380+0000 7fa2960beb80 1 -- start start 2024-05-27T09:35:37.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.380+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109593e40 0x5611097187f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.380+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109718d30 0x561109712870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.380+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109712db0 0x5611097131c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.380+0000 7fa2960beb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561109640e40 con 0x561109712db0 2024-05-27T09:35:37.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.380+0000 7fa2960beb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5611096309e0 con 0x561109593e40 2024-05-27T09:35:37.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.380+0000 7fa2960beb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561109634a20 con 0x561109718d30 2024-05-27T09:35:37.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.381+0000 7fa27fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109593e40 0x5611097187f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.381+0000 7fa27fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109593e40 0x5611097187f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53414/0 (socket says 172.21.15.44:53414) 2024-05-27T09:35:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.381+0000 7fa27fe00700 1 -- 172.21.15.44:0/3502546979 learned_addr learned my addr 172.21.15.44:0/3502546979 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.381+0000 7fa284c00700 1 --2- 172.21.15.44:0/3502546979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109718d30 0x561109712870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.381+0000 7fa27fe00700 1 --2- 172.21.15.44:0/3502546979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109712db0 0x5611097131c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.381+0000 7fa27d600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2680b2b10 0x7fa2680b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.381+0000 7fa27fe00700 1 -- 172.21.15.44:0/3502546979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109718d30 msgr2=0x561109712870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.381+0000 7fa27fe00700 1 --2- 172.21.15.44:0/3502546979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109718d30 0x561109712870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.381+0000 7fa27fe00700 1 -- 172.21.15.44:0/3502546979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109712db0 msgr2=0x5611097131c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.381+0000 7fa27fe00700 1 --2- 172.21.15.44:0/3502546979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109712db0 0x5611097131c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.381+0000 7fa27fe00700 1 -- 172.21.15.44:0/3502546979 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611095f9420 con 0x561109593e40 2024-05-27T09:35:37.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.382+0000 7fa27d600700 1 -- 172.21.15.44:0/2230790290 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa27800f020 con 0x7fa28010c5b0 2024-05-27T09:35:37.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.382+0000 7fa285600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2680b2b10 0x7fa2680b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.382+0000 7fa27fe00700 1 --2- 172.21.15.44:0/3502546979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109593e40 0x5611097187f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa26c011cc0 tx=0x7fa26c009c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6bd966f96cbf77d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:37.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.382+0000 7fa272000700 1 -- 172.21.15.44:0/3502546979 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c00f6a0 con 0x561109593e40 2024-05-27T09:35:37.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.382+0000 7fa272000700 1 -- 172.21.15.44:0/3502546979 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa26c00fcf0 con 0x561109593e40 2024-05-27T09:35:37.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.382+0000 7fa272000700 1 -- 172.21.15.44:0/3502546979 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c00ebd0 con 0x561109593e40 2024-05-27T09:35:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.383+0000 7fa2960beb80 1 -- 172.21.15.44:0/3502546979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109593e40 msgr2=0x5611097187f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.383+0000 7fa2960beb80 1 --2- 172.21.15.44:0/3502546979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109593e40 0x5611097187f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa26c011cc0 tx=0x7fa26c009c90 comp rx=0 tx=0).stop 2024-05-27T09:35:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.383+0000 7fa285600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2680b2b10 0x7fa2680b5020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa2780e1210 tx=0x7fa278009040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.383+0000 7fa2960beb80 1 -- 172.21.15.44:0/3502546979 shutdown_connections 2024-05-27T09:35:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.383+0000 7fa2960beb80 1 --2- 172.21.15.44:0/3502546979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109712db0 0x5611097131c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.383+0000 7fa2960beb80 1 --2- 172.21.15.44:0/3502546979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109593e40 0x5611097187f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.383+0000 7fa2960beb80 1 --2- 172.21.15.44:0/3502546979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109718d30 0x561109712870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.383+0000 7fa2960beb80 1 -- 172.21.15.44:0/3502546979 >> 172.21.15.44:0/3502546979 conn(0x5611095f3fc0 msgr2=0x56110970d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.383+0000 7fa2960beb80 1 -- 172.21.15.44:0/3502546979 shutdown_connections 2024-05-27T09:35:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.383+0000 7fa2960beb80 1 -- 172.21.15.44:0/3502546979 wait complete. 2024-05-27T09:35:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.384+0000 7fa2960beb80 1 -- start start 2024-05-27T09:35:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.384+0000 7fa2960beb80 1 Processor -- start 2024-05-27T09:35:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 0x56110971c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x56110971ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109718d30 0x56110971cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa2960beb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561109536fa0 con 0x561109718d30 2024-05-27T09:35:37.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa2960beb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5611095d1760 con 0x561109712db0 2024-05-27T09:35:37.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa2960beb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56110971f800 con 0x561109593e40 2024-05-27T09:35:37.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa2960beb80 10 client.0 _ll_register_callbacks cb 0x5611095dac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:35:37.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa27fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109718d30 0x56110971cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa27fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109718d30 0x56110971cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49842/0 (socket says 172.21.15.44:49842) 2024-05-27T09:35:37.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa284c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x56110971ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa27fe00700 1 -- 172.21.15.44:0/2703417208 learned_addr learned my addr 172.21.15.44:0/2703417208 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.385+0000 7fa27fe00700 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 0x56110971c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.386+0000 7fa284c00700 1 -- 172.21.15.44:0/2703417208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 msgr2=0x56110971c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.386+0000 7fa284c00700 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 0x56110971c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.386+0000 7fa284c00700 1 -- 172.21.15.44:0/2703417208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109718d30 msgr2=0x56110971cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.386+0000 7fa284c00700 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109718d30 0x56110971cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.386+0000 7fa284c00700 1 -- 172.21.15.44:0/2703417208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611095808d0 con 0x561109712db0 2024-05-27T09:35:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.386+0000 7fa284c00700 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x56110971ca50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa274012740 tx=0x7fa274012a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94694319c3c6174b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.386+0000 7fa266a00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:35:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.386+0000 7fa266a00700 1 -- 172.21.15.44:0/2703417208 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27400f4e0 con 0x561109712db0 2024-05-27T09:35:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.387+0000 7fa2960beb80 1 -- 172.21.15.44:0/2703417208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa26c00fcf0 con 0x561109712db0 2024-05-27T09:35:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.387+0000 7fa266a00700 1 -- 172.21.15.44:0/2703417208 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa27400f680 con 0x561109712db0 2024-05-27T09:35:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.387+0000 7fa266a00700 1 -- 172.21.15.44:0/2703417208 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa27400f880 con 0x561109712db0 2024-05-27T09:35:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.387+0000 7fa2960beb80 10 client.25259 fetch_fsmap learned FSMap version 26 2024-05-27T09:35:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.387+0000 7fa2960beb80 1 -- 172.21.15.44:0/2703417208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561109601060 con 0x561109712db0 2024-05-27T09:35:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.387+0000 7fa266a00700 1 -- 172.21.15.44:0/2703417208 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa26c00fcf0 con 0x561109712db0 2024-05-27T09:35:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.387+0000 7fa266a00700 1 -- 172.21.15.44:0/2703417208 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa274010aa0 con 0x561109712db0 2024-05-27T09:35:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.388+0000 7fa2960beb80 10 client.25259 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:35:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.388+0000 7fa2960beb80 10 client.25259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:35:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.388+0000 7fa2960beb80 20 client.25259 populate_metadata read hostname 'smithi044' 2024-05-27T09:35:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.388+0000 7fa2960beb80 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56110972c860 0x56110972ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.388+0000 7fa2960beb80 4 client.25259 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:35:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.388+0000 7fa2960beb80 1 -- 172.21.15.44:0/2703417208 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561109642d90 con 0x56110972c860 2024-05-27T09:35:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.388+0000 7fa27fe00700 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56110972c860 0x56110972ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.389+0000 7fa27fe00700 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56110972c860 0x56110972ec80 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29fcb8d9f5a5ab62 server_cookie=98f363f74284f47a in_seq=0 out_seq=0 2024-05-27T09:35:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.389+0000 7fa266a00700 10 client.25259 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.389+0000 7fa266a00700 1 -- 172.21.15.44:0/2703417208 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561109642d90 con 0x56110972c860 2024-05-27T09:35:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.389+0000 7fa266a00700 10 client.25259 handle_command_reply: tid=0 2024-05-27T09:35:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 1 client.25259 shutdown 2024-05-27T09:35:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 1 -- 172.21.15.44:0/2703417208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 msgr2=0x56110971ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x56110971ca50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa274012740 tx=0x7fa274012a50 comp rx=0 tx=0).stop 2024-05-27T09:35:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 1 -- 172.21.15.44:0/2703417208 shutdown_connections 2024-05-27T09:35:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56110972c860 0x56110972ec80 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109718d30 0x56110971cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 0x56110971c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa266a00700 0 client.25259 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 1 --2- 172.21.15.44:0/2703417208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x56110971ca50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 1 -- 172.21.15.44:0/2703417208 >> 172.21.15.44:0/2703417208 conn(0x5611095f3fc0 msgr2=0x56110970d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 1 -- 172.21.15.44:0/2703417208 shutdown_connections 2024-05-27T09:35:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 1 -- 172.21.15.44:0/2703417208 wait complete. 2024-05-27T09:35:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.390+0000 7fa2960beb80 20 client.25259 trim_cache size 0 max 16384 2024-05-27T09:35:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.415+0000 7fa2960beb80 1 Processor -- start 2024-05-27T09:35:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.416+0000 7fa2960beb80 1 -- start start 2024-05-27T09:35:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.416+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109593e40 0x561109712870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.416+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x56110971b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.416+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109718d30 0x561109716c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.416+0000 7fa2960beb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2801b3710 con 0x561109593e40 2024-05-27T09:35:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.416+0000 7fa2960beb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa2801b0fa0 con 0x561109712db0 2024-05-27T09:35:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.416+0000 7fa2960beb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2801aee10 con 0x561109718d30 2024-05-27T09:35:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.416+0000 7fa284c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x56110971b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.416+0000 7fa284c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x56110971b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53426/0 (socket says 172.21.15.44:53426) 2024-05-27T09:35:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.416+0000 7fa27fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109593e40 0x561109712870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.417+0000 7fa284c00700 1 -- 172.21.15.44:0/406303707 learned_addr learned my addr 172.21.15.44:0/406303707 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.417+0000 7fa284c00700 1 -- 172.21.15.44:0/406303707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109718d30 msgr2=0x561109716c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.417+0000 7fa284c00700 1 --2- 172.21.15.44:0/406303707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109718d30 0x561109716c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.417+0000 7fa284c00700 1 -- 172.21.15.44:0/406303707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109593e40 msgr2=0x561109712870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.417+0000 7fa284c00700 1 --2- 172.21.15.44:0/406303707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109593e40 0x561109712870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.417+0000 7fa284c00700 1 -- 172.21.15.44:0/406303707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56110971aa80 con 0x561109712db0 2024-05-27T09:35:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.417+0000 7fa284c00700 1 --2- 172.21.15.44:0/406303707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x56110971b7e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa27401c470 tx=0x7fa27400ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82938750bce5a040 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.417+0000 7fa271600700 1 -- 172.21.15.44:0/406303707 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa274029070 con 0x561109712db0 2024-05-27T09:35:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.417+0000 7fa271600700 1 -- 172.21.15.44:0/406303707 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa27401c630 con 0x561109712db0 2024-05-27T09:35:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.418+0000 7fa271600700 1 -- 172.21.15.44:0/406303707 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27401cd80 con 0x561109712db0 2024-05-27T09:35:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.418+0000 7fa2960beb80 1 -- 172.21.15.44:0/406303707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 msgr2=0x56110971b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.418+0000 7fa2960beb80 1 --2- 172.21.15.44:0/406303707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x56110971b7e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa27401c470 tx=0x7fa27400ef80 comp rx=0 tx=0).stop 2024-05-27T09:35:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.418+0000 7fa2960beb80 1 -- 172.21.15.44:0/406303707 shutdown_connections 2024-05-27T09:35:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.418+0000 7fa2960beb80 1 --2- 172.21.15.44:0/406303707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109718d30 0x561109716c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.418+0000 7fa2960beb80 1 --2- 172.21.15.44:0/406303707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109593e40 0x561109712870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.419+0000 7fa2960beb80 1 --2- 172.21.15.44:0/406303707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x56110971b7e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.419+0000 7fa2960beb80 1 -- 172.21.15.44:0/406303707 >> 172.21.15.44:0/406303707 conn(0x5611095f3fc0 msgr2=0x561109719430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.419+0000 7fa2960beb80 1 -- 172.21.15.44:0/406303707 shutdown_connections 2024-05-27T09:35:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.419+0000 7fa2960beb80 1 -- 172.21.15.44:0/406303707 wait complete. 2024-05-27T09:35:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.419+0000 7fa2960beb80 1 -- start start 2024-05-27T09:35:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.420+0000 7fa2960beb80 1 Processor -- start 2024-05-27T09:35:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.420+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 0x561109725080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.420+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x561109733ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.420+0000 7fa2960beb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109718d30 0x561109734020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.420+0000 7fa2960beb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2801af460 con 0x561109718d30 2024-05-27T09:35:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.420+0000 7fa2960beb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa2801b08d0 con 0x561109712db0 2024-05-27T09:35:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.420+0000 7fa2960beb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2801afb30 con 0x561109593e40 2024-05-27T09:35:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.420+0000 7fa2960beb80 10 client.0 _ll_register_callbacks cb 0x5611095dac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:35:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.420+0000 7fa27fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 0x561109725080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.421+0000 7fa27fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 0x561109725080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42016/0 (socket says 172.21.15.44:42016) 2024-05-27T09:35:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.421+0000 7fa27fe00700 1 -- 172.21.15.44:0/1575841348 learned_addr learned my addr 172.21.15.44:0/1575841348 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.421+0000 7fa284c00700 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x561109733ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.421+0000 7fa27fe00700 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109718d30 0x561109734020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.421+0000 7fa27fe00700 1 -- 172.21.15.44:0/1575841348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 msgr2=0x561109733ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.421+0000 7fa27fe00700 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x561109733ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.421+0000 7fa27fe00700 1 -- 172.21.15.44:0/1575841348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109718d30 msgr2=0x561109734020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.421+0000 7fa27fe00700 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109718d30 0x561109734020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.421+0000 7fa27fe00700 1 -- 172.21.15.44:0/1575841348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561109734690 con 0x561109593e40 2024-05-27T09:35:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.421+0000 7fa27fe00700 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 0x561109725080 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x56110971bd90 tx=0x7fa26c0114f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=821fc4a15ecf8a8e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.421+0000 7fa266a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:35:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.422+0000 7fa266a00700 1 -- 172.21.15.44:0/1575841348 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c024040 con 0x561109593e40 2024-05-27T09:35:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.422+0000 7fa2960beb80 1 -- 172.21.15.44:0/1575841348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa27401c630 con 0x561109593e40 2024-05-27T09:35:37.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.422+0000 7fa266a00700 1 -- 172.21.15.44:0/1575841348 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa26c0051b0 con 0x561109593e40 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.422+0000 7fa266a00700 1 -- 172.21.15.44:0/1575841348 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa26c0053b0 con 0x561109593e40 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.422+0000 7fa2960beb80 10 client.25270 fetch_fsmap learned FSMap version 26 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.422+0000 7fa2960beb80 1 -- 172.21.15.44:0/1575841348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561109797140 con 0x561109593e40 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.423+0000 7fa266a00700 1 -- 172.21.15.44:0/1575841348 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa27401c630 con 0x561109593e40 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.423+0000 7fa266a00700 1 -- 172.21.15.44:0/1575841348 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa26c02a040 con 0x561109593e40 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.423+0000 7fa2960beb80 10 client.25270 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.423+0000 7fa2960beb80 10 client.25270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.423+0000 7fa2960beb80 20 client.25270 populate_metadata read hostname 'smithi044' 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.423+0000 7fa2960beb80 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561109797670 0x561109799a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.423+0000 7fa2960beb80 4 client.25270 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.423+0000 7fa2960beb80 1 -- 172.21.15.44:0/1575841348 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56110963c570 con 0x561109797670 2024-05-27T09:35:37.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.423+0000 7fa284c00700 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561109797670 0x561109799a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:37.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.424+0000 7fa284c00700 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561109797670 0x561109799a90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d0e8553546ca418 server_cookie=4898e8053aeef182 in_seq=0 out_seq=0 2024-05-27T09:35:37.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.424+0000 7fa266a00700 10 client.25270 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:37.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.424+0000 7fa266a00700 1 -- 172.21.15.44:0/1575841348 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56110963c570 con 0x561109797670 2024-05-27T09:35:37.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.424+0000 7fa266a00700 10 client.25270 handle_command_reply: tid=0 2024-05-27T09:35:37.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.424+0000 7fa2960beb80 1 client.25270 shutdown 2024-05-27T09:35:37.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa2960beb80 1 -- 172.21.15.44:0/1575841348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 msgr2=0x561109725080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:37.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa2960beb80 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 0x561109725080 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x56110971bd90 tx=0x7fa26c0114f0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa2960beb80 1 -- 172.21.15.44:0/1575841348 shutdown_connections 2024-05-27T09:35:37.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa2960beb80 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561109797670 0x561109799a90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa2960beb80 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561109718d30 0x561109734020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa2960beb80 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561109593e40 0x561109725080 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa266a00700 0 client.25270 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa2960beb80 1 --2- 172.21.15.44:0/1575841348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561109712db0 0x561109733ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa2960beb80 1 -- 172.21.15.44:0/1575841348 >> 172.21.15.44:0/1575841348 conn(0x5611095f3fc0 msgr2=0x5611097169a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa2960beb80 1 -- 172.21.15.44:0/1575841348 shutdown_connections 2024-05-27T09:35:37.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa2960beb80 1 -- 172.21.15.44:0/1575841348 wait complete. 2024-05-27T09:35:37.622 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:35:37.622 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:35:37.623 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:35:37.623 INFO:teuthology.orchestra.run.smithi044.stdout: "bb6e9c27-6db5-4339-aa03-d1f54f0f55e0": { 2024-05-27T09:35:37.623 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:35:37.623 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "bb6e9c27-6db5-4339-aa03-d1f54f0f55e0", 2024-05-27T09:35:37.623 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:35:37.623 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:35:37.623 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:35:37.623 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:35:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.425+0000 7fa2960beb80 20 client.25270 trim_cache size 0 max 16384 2024-05-27T09:35:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 -- 172.21.15.44:0/2230790290 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2680b2b10 msgr2=0x7fa2680b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2680b2b10 0x7fa2680b5020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa2780e1210 tx=0x7fa278009040 comp rx=0 tx=0).stop 2024-05-27T09:35:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 -- 172.21.15.44:0/2230790290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa28010c5b0 msgr2=0x7fa2801a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa28010c5b0 0x7fa2801a1830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa27800b070 tx=0x7fa278004c80 comp rx=0 tx=0).stop 2024-05-27T09:35:37.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 -- 172.21.15.44:0/2230790290 shutdown_connections 2024-05-27T09:35:37.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2680b2b10 0x7fa2680b5020 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa28010c5b0 0x7fa2801a1830 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa280106570 0x7fa28019d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 --2- 172.21.15.44:0/2230790290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa280106ec0 0x7fa28019d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:37.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 -- 172.21.15.44:0/2230790290 >> 172.21.15.44:0/2230790290 conn(0x7fa280102070 msgr2=0x7fa280104710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:37.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 -- 172.21.15.44:0/2230790290 shutdown_connections 2024-05-27T09:35:37.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:37.426+0000 7fa287600700 1 -- 172.21.15.44:0/2230790290 wait complete. 2024-05-27T09:35:38.192 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:38 smithi044 ceph-mon[31618]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-05-27T09:35:38.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:38 smithi050 ceph-mon[36371]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-05-27T09:35:38.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:38 smithi189 ceph-mon[36671]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-05-27T09:35:38.873 INFO:tasks.cephfs.filesystem:scrub status for tag:bb6e9c27-6db5-4339-aa03-d1f54f0f55e0 - {'path': '/', 'tag': 'bb6e9c27-6db5-4339-aa03-d1f54f0f55e0', 'options': 'recursive,force'} 2024-05-27T09:35:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.544+0000 7f147a400700 1 -- 172.21.15.44:0/2868012780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14741045e0 msgr2=0x7f14741049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.544+0000 7f147a400700 1 --2- 172.21.15.44:0/2868012780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14741045e0 0x7f14741049f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1464009920 tx=0x7f1464009c30 comp rx=0 tx=0).stop 2024-05-27T09:35:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.544+0000 7f147a400700 1 -- 172.21.15.44:0/2868012780 shutdown_connections 2024-05-27T09:35:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.544+0000 7f147a400700 1 --2- 172.21.15.44:0/2868012780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1474109860 0x7f14741111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.544+0000 7f147a400700 1 --2- 172.21.15.44:0/2868012780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14741045e0 0x7f14741049f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.544+0000 7f147a400700 1 --2- 172.21.15.44:0/2868012780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474108f10 0x7f1474109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.544+0000 7f147a400700 1 -- 172.21.15.44:0/2868012780 >> 172.21.15.44:0/2868012780 conn(0x7f1474100030 msgr2=0x7f1474102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.544+0000 7f147a400700 1 -- 172.21.15.44:0/2868012780 shutdown_connections 2024-05-27T09:35:39.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.544+0000 7f147a400700 1 -- 172.21.15.44:0/2868012780 wait complete. 2024-05-27T09:35:39.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.545+0000 7f147a400700 1 Processor -- start 2024-05-27T09:35:39.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.545+0000 7f147a400700 1 -- start start 2024-05-27T09:35:39.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.546+0000 7f147a400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14741045e0 0x7f1474075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:39.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.546+0000 7f147a400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1474108f10 0x7f1474075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:39.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.546+0000 7f147a400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474109860 0x7f1474076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:39.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.546+0000 7f147a400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1474114490 con 0x7f14741045e0 2024-05-27T09:35:39.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.546+0000 7f147a400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1474114310 con 0x7f1474109860 2024-05-27T09:35:39.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.546+0000 7f147a400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1474114610 con 0x7f1474108f10 2024-05-27T09:35:39.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.546+0000 7f1473400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14741045e0 0x7f1474075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:39.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.546+0000 7f146aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1474108f10 0x7f1474075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:39.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.546+0000 7f1473400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14741045e0 0x7f1474075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49880/0 (socket says 172.21.15.44:49880) 2024-05-27T09:35:39.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.546+0000 7f1473400700 1 -- 172.21.15.44:0/3487747709 learned_addr learned my addr 172.21.15.44:0/3487747709 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:39.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.546+0000 7f1473e00700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474109860 0x7f1474076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:39.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.547+0000 7f1473400700 1 -- 172.21.15.44:0/3487747709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1474108f10 msgr2=0x7f1474075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:39.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.547+0000 7f1473400700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1474108f10 0x7f1474075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:39.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.547+0000 7f1473400700 1 -- 172.21.15.44:0/3487747709 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474109860 msgr2=0x7f1474076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:39.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.547+0000 7f1473400700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474109860 0x7f1474076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:39.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.547+0000 7f1473400700 1 -- 172.21.15.44:0/3487747709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14741a9240 con 0x7f14741045e0 2024-05-27T09:35:39.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.547+0000 7f146aa00700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1474108f10 0x7f1474075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:39.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.547+0000 7f1473400700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14741045e0 0x7f1474075ab0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1464009230 tx=0x7f146400f930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:39.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.547+0000 7f1470c00700 1 -- 172.21.15.44:0/3487747709 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1464027070 con 0x7f14741045e0 2024-05-27T09:35:39.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.547+0000 7f1470c00700 1 -- 172.21.15.44:0/3487747709 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1464005570 con 0x7f14741045e0 2024-05-27T09:35:39.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.547+0000 7f147a400700 1 -- 172.21.15.44:0/3487747709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14741a94d0 con 0x7f14741045e0 2024-05-27T09:35:39.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.548+0000 7f1470c00700 1 -- 172.21.15.44:0/3487747709 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1464004930 con 0x7f14741045e0 2024-05-27T09:35:39.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.548+0000 7f147a400700 1 -- 172.21.15.44:0/3487747709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14741a9920 con 0x7f14741045e0 2024-05-27T09:35:39.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.549+0000 7f147a400700 1 -- 172.21.15.44:0/3487747709 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14740512b0 con 0x7f14741045e0 2024-05-27T09:35:39.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.551+0000 7f1470c00700 1 -- 172.21.15.44:0/3487747709 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f146402a070 con 0x7f14741045e0 2024-05-27T09:35:39.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.553+0000 7f1470c00700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14580b2b10 0x7f14580b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:39.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.553+0000 7f1470c00700 1 -- 172.21.15.44:0/3487747709 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f14640e33a0 con 0x7f14741045e0 2024-05-27T09:35:39.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.553+0000 7f146aa00700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14580b2b10 0x7f14580b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:39.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.554+0000 7f146aa00700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14580b2b10 0x7f14580b5020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1474107750 tx=0x7f145c009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:39.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.558+0000 7f1470c00700 1 -- 172.21.15.44:0/3487747709 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1464013030 con 0x7f14741045e0 2024-05-27T09:35:39.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.790+0000 7f147a400700 1 -- 172.21.15.44:0/3487747709 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1474063bf0 con 0x7f14580b2b10 2024-05-27T09:35:39.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.807+0000 7f1470c00700 1 -- 172.21.15.44:0/3487747709 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257756 (secure 0 0 0) 0x7f1474063bf0 con 0x7f14580b2b10 2024-05-27T09:35:39.810 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:39.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.816+0000 7f147a400700 1 -- 172.21.15.44:0/3487747709 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14580b2b10 msgr2=0x7f14580b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:39.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.816+0000 7f147a400700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14580b2b10 0x7f14580b5020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1474107750 tx=0x7f145c009260 comp rx=0 tx=0).stop 2024-05-27T09:35:39.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.817+0000 7f147a400700 1 -- 172.21.15.44:0/3487747709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14741045e0 msgr2=0x7f1474075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:39.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.817+0000 7f147a400700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14741045e0 0x7f1474075ab0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1464009230 tx=0x7f146400f930 comp rx=0 tx=0).stop 2024-05-27T09:35:39.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.817+0000 7f147a400700 1 -- 172.21.15.44:0/3487747709 shutdown_connections 2024-05-27T09:35:39.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.817+0000 7f147a400700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14580b2b10 0x7f14580b5020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:39.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.817+0000 7f147a400700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474109860 0x7f1474076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:39.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.817+0000 7f147a400700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14741045e0 0x7f1474075ab0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:39.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.817+0000 7f147a400700 1 --2- 172.21.15.44:0/3487747709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1474108f10 0x7f1474075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:39.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.817+0000 7f147a400700 1 -- 172.21.15.44:0/3487747709 >> 172.21.15.44:0/3487747709 conn(0x7f1474100030 msgr2=0x7f1474102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:39.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.817+0000 7f147a400700 1 -- 172.21.15.44:0/3487747709 shutdown_connections 2024-05-27T09:35:39.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:39.818+0000 7f147a400700 1 -- 172.21.15.44:0/3487747709 wait complete. 2024-05-27T09:35:39.820 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-05-27T09:35:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:40 smithi044 ceph-mon[31618]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:40 smithi044 ceph-mon[31618]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-05-27T09:35:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:40 smithi050 ceph-mon[36371]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:40 smithi050 ceph-mon[36371]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-05-27T09:35:40.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:40 smithi189 ceph-mon[36671]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:40.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:40 smithi189 ceph-mon[36671]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-05-27T09:35:40.591 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-05-27T09:35:38.075385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1388171,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1895,"num_read_kb":1342,"num_write":564,"num_write_kb":4513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":916,"ondisk_log_size":916,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385072,"kb_used_data":14736,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124556752,"statfs":{"total":1151940427776,"available":1151546114048,"internally_reserved":0,"allocated":15089664,"data_stored":9446361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":315,"num_read_kb":262,"num_write":126,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004671"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840281+0000","last_change":"2024-05-27T09:31:24.899109+0000","last_active":"2024-05-27T09:35:14.840281+0000","last_peered":"2024-05-27T09:35:14.840281+0000","last_clean":"2024-05-27T09:35:14.840281+0000","last_became_active":"2024-05-27T09:31:24.898783+0000","last_became_peered":"2024-05-27T09:31:24.898783+0000","last_unstale":"2024-05-27T09:35:14.840281+0000","last_undegraded":"2024-05-27T09:35:14.840281+0000","last_fullsized":"2024-05-27T09:35:14.840281+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:01:12.220389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042833+0000","last_change":"2024-05-27T09:31:24.898382+0000","last_active":"2024-05-27T09:35:14.042833+0000","last_peered":"2024-05-27T09:35:14.042833+0000","last_clean":"2024-05-27T09:35:14.042833+0000","last_became_active":"2024-05-27T09:31:24.897975+0000","last_became_peered":"2024-05-27T09:31:24.897975+0000","last_unstale":"2024-05-27T09:35:14.042833+0000","last_undegraded":"2024-05-27T09:35:14.042833+0000","last_fullsized":"2024-05-27T09:35:14.042833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:38.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041882+0000","last_change":"2024-05-27T09:31:24.896122+0000","last_active":"2024-05-27T09:35:14.041882+0000","last_peered":"2024-05-27T09:35:14.041882+0000","last_clean":"2024-05-27T09:35:14.041882+0000","last_became_active":"2024-05-27T09:31:24.890867+0000","last_became_peered":"2024-05-27T09:31:24.890867+0000","last_unstale":"2024-05-27T09:35:14.041882+0000","last_undegraded":"2024-05-27T09:35:14.041882+0000","last_fullsized":"2024-05-27T09:35:14.041882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:09.982177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046136+0000","last_change":"2024-05-27T09:31:57.868900+0000","last_active":"2024-05-27T09:35:14.046136+0000","last_peered":"2024-05-27T09:35:14.046136+0000","last_clean":"2024-05-27T09:35:14.046136+0000","last_became_active":"2024-05-27T09:31:57.868418+0000","last_became_peered":"2024-05-27T09:31:57.868418+0000","last_unstale":"2024-05-27T09:35:14.046136+0000","last_undegraded":"2024-05-27T09:35:14.046136+0000","last_fullsized":"2024-05-27T09:35:14.046136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:08.966704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169866+0000","last_change":"2024-05-27T09:31:24.910283+0000","last_active":"2024-05-27T09:35:14.169866+0000","last_peered":"2024-05-27T09:35:14.169866+0000","last_clean":"2024-05-27T09:35:14.169866+0000","last_became_active":"2024-05-27T09:31:24.909882+0000","last_became_peered":"2024-05-27T09:31:24.909882+0000","last_unstale":"2024-05-27T09:35:14.169866+0000","last_undegraded":"2024-05-27T09:35:14.169866+0000","last_fullsized":"2024-05-27T09:35:14.169866+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:43.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126676+0000","last_change":"2024-05-27T09:31:24.900934+0000","last_active":"2024-05-27T09:35:14.126676+0000","last_peered":"2024-05-27T09:35:14.126676+0000","last_clean":"2024-05-27T09:35:14.126676+0000","last_became_active":"2024-05-27T09:31:24.900534+0000","last_became_peered":"2024-05-27T09:31:24.900534+0000","last_unstale":"2024-05-27T09:35:14.126676+0000","last_undegraded":"2024-05-27T09:35:14.126676+0000","last_fullsized":"2024-05-27T09:35:14.126676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:53.318327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353192+0000","last_change":"2024-05-27T09:31:24.907080+0000","last_active":"2024-05-27T09:35:14.353192+0000","last_peered":"2024-05-27T09:35:14.353192+0000","last_clean":"2024-05-27T09:35:14.353192+0000","last_became_active":"2024-05-27T09:31:24.906715+0000","last_became_peered":"2024-05-27T09:31:24.906715+0000","last_unstale":"2024-05-27T09:35:14.353192+0000","last_undegraded":"2024-05-27T09:35:14.353192+0000","last_fullsized":"2024-05-27T09:35:14.353192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:36.214659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.044350+0000","last_change":"2024-05-27T09:31:58.200608+0000","last_active":"2024-05-27T09:35:14.044350+0000","last_peered":"2024-05-27T09:35:14.044350+0000","last_clean":"2024-05-27T09:35:14.044350+0000","last_became_active":"2024-05-27T09:31:58.199363+0000","last_became_peered":"2024-05-27T09:31:58.199363+0000","last_unstale":"2024-05-27T09:35:14.044350+0000","last_undegraded":"2024-05-27T09:35:14.044350+0000","last_fullsized":"2024-05-27T09:35:14.044350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:30:46.495930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373508+0000","last_change":"2024-05-27T09:31:24.892630+0000","last_active":"2024-05-27T09:35:14.373508+0000","last_peered":"2024-05-27T09:35:14.373508+0000","last_clean":"2024-05-27T09:35:14.373508+0000","last_became_active":"2024-05-27T09:31:24.892176+0000","last_became_peered":"2024-05-27T09:31:24.892176+0000","last_unstale":"2024-05-27T09:35:14.373508+0000","last_undegraded":"2024-05-27T09:35:14.373508+0000","last_fullsized":"2024-05-27T09:35:14.373508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:49:20.585601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.127155+0000","last_change":"2024-05-27T09:31:24.895281+0000","last_active":"2024-05-27T09:35:14.127155+0000","last_peered":"2024-05-27T09:35:14.127155+0000","last_clean":"2024-05-27T09:35:14.127155+0000","last_became_active":"2024-05-27T09:31:24.894886+0000","last_became_peered":"2024-05-27T09:31:24.894886+0000","last_unstale":"2024-05-27T09:35:14.127155+0000","last_undegraded":"2024-05-27T09:35:14.127155+0000","last_fullsized":"2024-05-27T09:35:14.127155+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:41.494434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353559+0000","last_change":"2024-05-27T09:31:24.900167+0000","last_active":"2024-05-27T09:35:14.353559+0000","last_peered":"2024-05-27T09:35:14.353559+0000","last_clean":"2024-05-27T09:35:14.353559+0000","last_became_active":"2024-05-27T09:31:24.899750+0000","last_became_peered":"2024-05-27T09:31:24.899750+0000","last_unstale":"2024-05-27T09:35:14.353559+0000","last_undegraded":"2024-05-27T09:35:14.353559+0000","last_fullsized":"2024-05-27T09:35:14.353559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:30.602246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.045840+0000","last_change":"2024-05-27T09:31:57.868818+0000","last_active":"2024-05-27T09:35:14.045840+0000","last_peered":"2024-05-27T09:35:14.045840+0000","last_clean":"2024-05-27T09:35:14.045840+0000","last_became_active":"2024-05-27T09:31:57.868322+0000","last_became_peered":"2024-05-27T09:31:57.868322+0000","last_unstale":"2024-05-27T09:35:14.045840+0000","last_undegraded":"2024-05-27T09:35:14.045840+0000","last_fullsized":"2024-05-27T09:35:14.045840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:35.304850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242036+0000","last_change":"2024-05-27T09:31:24.902537+0000","last_active":"2024-05-27T09:35:14.242036+0000","last_peered":"2024-05-27T09:35:14.242036+0000","last_clean":"2024-05-27T09:35:14.242036+0000","last_became_active":"2024-05-27T09:31:24.902132+0000","last_became_peered":"2024-05-27T09:31:24.902132+0000","last_unstale":"2024-05-27T09:35:14.242036+0000","last_undegraded":"2024-05-27T09:35:14.242036+0000","last_fullsized":"2024-05-27T09:35:14.242036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:20:50.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041926+0000","last_change":"2024-05-27T09:31:24.899182+0000","last_active":"2024-05-27T09:35:14.041926+0000","last_peered":"2024-05-27T09:35:14.041926+0000","last_clean":"2024-05-27T09:35:14.041926+0000","last_became_active":"2024-05-27T09:31:24.898749+0000","last_became_peered":"2024-05-27T09:31:24.898749+0000","last_unstale":"2024-05-27T09:35:14.041926+0000","last_undegraded":"2024-05-27T09:35:14.041926+0000","last_fullsized":"2024-05-27T09:35:14.041926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:44:44.844357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041660+0000","last_change":"2024-05-27T09:31:24.901115+0000","last_active":"2024-05-27T09:35:14.041660+0000","last_peered":"2024-05-27T09:35:14.041660+0000","last_clean":"2024-05-27T09:35:14.041660+0000","last_became_active":"2024-05-27T09:31:24.897011+0000","last_became_peered":"2024-05-27T09:31:24.897011+0000","last_unstale":"2024-05-27T09:35:14.041660+0000","last_undegraded":"2024-05-27T09:35:14.041660+0000","last_fullsized":"2024-05-27T09:35:14.041660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:01.130407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372690+0000","last_change":"2024-05-27T09:31:24.903063+0000","last_active":"2024-05-27T09:35:14.372690+0000","last_peered":"2024-05-27T09:35:14.372690+0000","last_clean":"2024-05-27T09:35:14.372690+0000","last_became_active":"2024-05-27T09:31:24.899766+0000","last_became_peered":"2024-05-27T09:31:24.899766+0000","last_unstale":"2024-05-27T09:35:14.372690+0000","last_undegraded":"2024-05-27T09:35:14.372690+0000","last_fullsized":"2024-05-27T09:35:14.372690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:45.984226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840388+0000","last_change":"2024-05-27T09:31:24.915922+0000","last_active":"2024-05-27T09:35:14.840388+0000","last_peered":"2024-05-27T09:35:14.840388+0000","last_clean":"2024-05-27T09:35:14.840388+0000","last_became_active":"2024-05-27T09:31:24.915623+0000","last_became_peered":"2024-05-27T09:31:24.915623+0000","last_unstale":"2024-05-27T09:35:14.840388+0000","last_undegraded":"2024-05-27T09:35:14.840388+0000","last_fullsized":"2024-05-27T09:35:14.840388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:02:01.006704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046094+0000","last_change":"2024-05-27T09:31:24.908397+0000","last_active":"2024-05-27T09:35:14.046094+0000","last_peered":"2024-05-27T09:35:14.046094+0000","last_clean":"2024-05-27T09:35:14.046094+0000","last_became_active":"2024-05-27T09:31:24.903644+0000","last_became_peered":"2024-05-27T09:31:24.903644+0000","last_unstale":"2024-05-27T09:35:14.046094+0000","last_undegraded":"2024-05-27T09:35:14.046094+0000","last_fullsized":"2024-05-27T09:35:14.046094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:09.281429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373078+0000","last_change":"2024-05-27T09:31:24.912706+0000","last_active":"2024-05-27T09:35:14.373078+0000","last_peered":"2024-05-27T09:35:14.373078+0000","last_clean":"2024-05-27T09:35:14.373078+0000","last_became_active":"2024-05-27T09:31:24.912357+0000","last_became_peered":"2024-05-27T09:31:24.912357+0000","last_unstale":"2024-05-27T09:35:14.373078+0000","last_undegraded":"2024-05-27T09:35:14.373078+0000","last_fullsized":"2024-05-27T09:35:14.373078+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:23.334041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:30.093913+0000","last_change":"2024-05-27T09:31:20.881652+0000","last_active":"2024-05-27T09:35:30.093913+0000","last_peered":"2024-05-27T09:35:30.093913+0000","last_clean":"2024-05-27T09:35:30.093913+0000","last_became_active":"2024-05-27T09:31:20.880757+0000","last_became_peered":"2024-05-27T09:31:20.880757+0000","last_unstale":"2024-05-27T09:35:30.093913+0000","last_undegraded":"2024-05-27T09:35:30.093913+0000","last_fullsized":"2024-05-27T09:35:30.093913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:01.433755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171081+0000","last_change":"2024-05-27T09:31:24.908010+0000","last_active":"2024-05-27T09:35:14.171081+0000","last_peered":"2024-05-27T09:35:14.171081+0000","last_clean":"2024-05-27T09:35:14.171081+0000","last_became_active":"2024-05-27T09:31:24.907638+0000","last_became_peered":"2024-05-27T09:31:24.907638+0000","last_unstale":"2024-05-27T09:35:14.171081+0000","last_undegraded":"2024-05-27T09:35:14.171081+0000","last_fullsized":"2024-05-27T09:35:14.171081+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:02:49.207709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171465+0000","last_change":"2024-05-27T09:31:20.860556+0000","last_active":"2024-05-27T09:35:14.171465+0000","last_peered":"2024-05-27T09:35:14.171465+0000","last_clean":"2024-05-27T09:35:14.171465+0000","last_became_active":"2024-05-27T09:31:20.860172+0000","last_became_peered":"2024-05-27T09:31:20.860172+0000","last_unstale":"2024-05-27T09:35:14.171465+0000","last_undegraded":"2024-05-27T09:35:14.171465+0000","last_fullsized":"2024-05-27T09:35:14.171465+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:08:21.545681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043114+0000","last_change":"2024-05-27T09:31:24.910111+0000","last_active":"2024-05-27T09:35:14.043114+0000","last_peered":"2024-05-27T09:35:14.043114+0000","last_clean":"2024-05-27T09:35:14.043114+0000","last_became_active":"2024-05-27T09:31:24.908985+0000","last_became_peered":"2024-05-27T09:31:24.908985+0000","last_unstale":"2024-05-27T09:35:14.043114+0000","last_undegraded":"2024-05-27T09:35:14.043114+0000","last_fullsized":"2024-05-27T09:35:14.043114+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:53:29.645792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170222+0000","last_change":"2024-05-27T09:31:20.879993+0000","last_active":"2024-05-27T09:35:14.170222+0000","last_peered":"2024-05-27T09:35:14.170222+0000","last_clean":"2024-05-27T09:35:14.170222+0000","last_became_active":"2024-05-27T09:31:20.878802+0000","last_became_peered":"2024-05-27T09:31:20.878802+0000","last_unstale":"2024-05-27T09:35:14.170222+0000","last_undegraded":"2024-05-27T09:35:14.170222+0000","last_fullsized":"2024-05-27T09:35:14.170222+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:15.555612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372970+0000","last_change":"2024-05-27T09:31:24.903533+0000","last_active":"2024-05-27T09:35:14.372970+0000","last_peered":"2024-05-27T09:35:14.372970+0000","last_clean":"2024-05-27T09:35:14.372970+0000","last_became_active":"2024-05-27T09:31:24.903166+0000","last_became_peered":"2024-05-27T09:31:24.903166+0000","last_unstale":"2024-05-27T09:35:14.372970+0000","last_undegraded":"2024-05-27T09:35:14.372970+0000","last_fullsized":"2024-05-27T09:35:14.372970+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:52.539470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839887+0000","last_change":"2024-05-27T09:31:57.878426+0000","last_active":"2024-05-27T09:35:14.839887+0000","last_peered":"2024-05-27T09:35:14.839887+0000","last_clean":"2024-05-27T09:35:14.839887+0000","last_became_active":"2024-05-27T09:31:57.869146+0000","last_became_peered":"2024-05-27T09:31:57.869146+0000","last_unstale":"2024-05-27T09:35:14.839887+0000","last_undegraded":"2024-05-27T09:35:14.839887+0000","last_fullsized":"2024-05-27T09:35:14.839887+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:34:51.926092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129913+0000","last_change":"2024-05-27T09:31:24.897426+0000","last_active":"2024-05-27T09:35:14.129913+0000","last_peered":"2024-05-27T09:35:14.129913+0000","last_clean":"2024-05-27T09:35:14.129913+0000","last_became_active":"2024-05-27T09:31:24.897080+0000","last_became_peered":"2024-05-27T09:31:24.897080+0000","last_unstale":"2024-05-27T09:35:14.129913+0000","last_undegraded":"2024-05-27T09:35:14.129913+0000","last_fullsized":"2024-05-27T09:35:14.129913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:15:42.428391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'8","reported_seq":270,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:33.160421+0000","last_change":"2024-05-27T09:31:20.846681+0000","last_active":"2024-05-27T09:35:33.160421+0000","last_peered":"2024-05-27T09:35:33.160421+0000","last_clean":"2024-05-27T09:35:33.160421+0000","last_became_active":"2024-05-27T09:31:20.846302+0000","last_became_peered":"2024-05-27T09:31:20.846302+0000","last_unstale":"2024-05-27T09:35:33.160421+0000","last_undegraded":"2024-05-27T09:35:33.160421+0000","last_fullsized":"2024-05-27T09:35:33.160421+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:19.609612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":233,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169711+0000","last_change":"2024-05-27T09:31:24.907209+0000","last_active":"2024-05-27T09:35:14.169711+0000","last_peered":"2024-05-27T09:35:14.169711+0000","last_clean":"2024-05-27T09:35:14.169711+0000","last_became_active":"2024-05-27T09:31:24.906752+0000","last_became_peered":"2024-05-27T09:31:24.906752+0000","last_unstale":"2024-05-27T09:35:14.169711+0000","last_undegraded":"2024-05-27T09:35:14.169711+0000","last_fullsized":"2024-05-27T09:35:14.169711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:53.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126569+0000","last_change":"2024-05-27T09:31:20.879778+0000","last_active":"2024-05-27T09:35:14.126569+0000","last_peered":"2024-05-27T09:35:14.126569+0000","last_clean":"2024-05-27T09:35:14.126569+0000","last_became_active":"2024-05-27T09:31:20.879259+0000","last_became_peered":"2024-05-27T09:31:20.879259+0000","last_unstale":"2024-05-27T09:35:14.126569+0000","last_undegraded":"2024-05-27T09:35:14.126569+0000","last_fullsized":"2024-05-27T09:35:14.126569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:20:07.972653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129702+0000","last_change":"2024-05-27T09:31:58.195516+0000","last_active":"2024-05-27T09:35:14.129702+0000","last_peered":"2024-05-27T09:35:14.129702+0000","last_clean":"2024-05-27T09:35:14.129702+0000","last_became_active":"2024-05-27T09:31:58.195118+0000","last_became_peered":"2024-05-27T09:31:58.195118+0000","last_unstale":"2024-05-27T09:35:14.129702+0000","last_undegraded":"2024-05-27T09:35:14.129702+0000","last_fullsized":"2024-05-27T09:35:14.129702+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:11:23.547605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241846+0000","last_change":"2024-05-27T09:31:20.863152+0000","last_active":"2024-05-27T09:35:14.241846+0000","last_peered":"2024-05-27T09:35:14.241846+0000","last_clean":"2024-05-27T09:35:14.241846+0000","last_became_active":"2024-05-27T09:31:20.862708+0000","last_became_peered":"2024-05-27T09:31:20.862708+0000","last_unstale":"2024-05-27T09:35:14.241846+0000","last_undegraded":"2024-05-27T09:35:14.241846+0000","last_fullsized":"2024-05-27T09:35:14.241846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:21.645773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170544+0000","last_change":"2024-05-27T09:31:24.893988+0000","last_active":"2024-05-27T09:35:14.170544+0000","last_peered":"2024-05-27T09:35:14.170544+0000","last_clean":"2024-05-27T09:35:14.170544+0000","last_became_active":"2024-05-27T09:31:24.893504+0000","last_became_peered":"2024-05-27T09:31:24.893504+0000","last_unstale":"2024-05-27T09:35:14.170544+0000","last_undegraded":"2024-05-27T09:35:14.170544+0000","last_fullsized":"2024-05-27T09:35:14.170544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:47.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129578+0000","last_change":"2024-05-27T09:31:20.845555+0000","last_active":"2024-05-27T09:35:14.129578+0000","last_peered":"2024-05-27T09:35:14.129578+0000","last_clean":"2024-05-27T09:35:14.129578+0000","last_became_active":"2024-05-27T09:31:20.845139+0000","last_became_peered":"2024-05-27T09:31:20.845139+0000","last_unstale":"2024-05-27T09:35:14.129578+0000","last_undegraded":"2024-05-27T09:35:14.129578+0000","last_fullsized":"2024-05-27T09:35:14.129578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:25.917845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.049161+0000","last_change":"2024-05-27T09:31:24.916142+0000","last_active":"2024-05-27T09:35:14.049161+0000","last_peered":"2024-05-27T09:35:14.049161+0000","last_clean":"2024-05-27T09:35:14.049161+0000","last_became_active":"2024-05-27T09:31:24.915709+0000","last_became_peered":"2024-05-27T09:31:24.915709+0000","last_unstale":"2024-05-27T09:35:14.049161+0000","last_undegraded":"2024-05-27T09:35:14.049161+0000","last_fullsized":"2024-05-27T09:35:14.049161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:38.029648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'5","reported_seq":262,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:33.159893+0000","last_change":"2024-05-27T09:31:20.859240+0000","last_active":"2024-05-27T09:35:33.159893+0000","last_peered":"2024-05-27T09:35:33.159893+0000","last_clean":"2024-05-27T09:35:33.159893+0000","last_became_active":"2024-05-27T09:31:20.858798+0000","last_became_peered":"2024-05-27T09:31:20.858798+0000","last_unstale":"2024-05-27T09:35:33.159893+0000","last_undegraded":"2024-05-27T09:35:33.159893+0000","last_fullsized":"2024-05-27T09:35:33.159893+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:42.280165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":228,"num_read_kb":228,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373425+0000","last_change":"2024-05-27T09:31:24.908818+0000","last_active":"2024-05-27T09:35:14.373425+0000","last_peered":"2024-05-27T09:35:14.373425+0000","last_clean":"2024-05-27T09:35:14.373425+0000","last_became_active":"2024-05-27T09:31:24.908410+0000","last_became_peered":"2024-05-27T09:31:24.908410+0000","last_unstale":"2024-05-27T09:35:14.373425+0000","last_undegraded":"2024-05-27T09:35:14.373425+0000","last_fullsized":"2024-05-27T09:35:14.373425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:05:40.326232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353642+0000","last_change":"2024-05-27T09:31:20.848632+0000","last_active":"2024-05-27T09:35:14.353642+0000","last_peered":"2024-05-27T09:35:14.353642+0000","last_clean":"2024-05-27T09:35:14.353642+0000","last_became_active":"2024-05-27T09:31:20.847558+0000","last_became_peered":"2024-05-27T09:31:20.847558+0000","last_unstale":"2024-05-27T09:35:14.353642+0000","last_undegraded":"2024-05-27T09:35:14.353642+0000","last_fullsized":"2024-05-27T09:35:14.353642+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:53.930565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043056+0000","last_change":"2024-05-27T09:31:24.901513+0000","last_active":"2024-05-27T09:35:14.043056+0000","last_peered":"2024-05-27T09:35:14.043056+0000","last_clean":"2024-05-27T09:35:14.043056+0000","last_became_active":"2024-05-27T09:31:24.895924+0000","last_became_peered":"2024-05-27T09:31:24.895924+0000","last_unstale":"2024-05-27T09:35:14.043056+0000","last_undegraded":"2024-05-27T09:35:14.043056+0000","last_fullsized":"2024-05-27T09:35:14.043056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:27:39.939098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.130061+0000","last_change":"2024-05-27T09:31:20.855195+0000","last_active":"2024-05-27T09:35:14.130061+0000","last_peered":"2024-05-27T09:35:14.130061+0000","last_clean":"2024-05-27T09:35:14.130061+0000","last_became_active":"2024-05-27T09:31:20.854910+0000","last_became_peered":"2024-05-27T09:31:20.854910+0000","last_unstale":"2024-05-27T09:35:14.130061+0000","last_undegraded":"2024-05-27T09:35:14.130061+0000","last_fullsized":"2024-05-27T09:35:14.130061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:02.795103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373077+0000","last_change":"2024-05-27T09:31:24.895887+0000","last_active":"2024-05-27T09:35:14.373077+0000","last_peered":"2024-05-27T09:35:14.373077+0000","last_clean":"2024-05-27T09:35:14.373077+0000","last_became_active":"2024-05-27T09:31:24.894622+0000","last_became_peered":"2024-05-27T09:31:24.894622+0000","last_unstale":"2024-05-27T09:35:14.373077+0000","last_undegraded":"2024-05-27T09:35:14.373077+0000","last_fullsized":"2024-05-27T09:35:14.373077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:32.833946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'38","reported_seq":536,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:34.161273+0000","last_change":"2024-05-27T09:31:20.846063+0000","last_active":"2024-05-27T09:35:34.161273+0000","last_peered":"2024-05-27T09:35:34.161273+0000","last_clean":"2024-05-27T09:35:34.161273+0000","last_became_active":"2024-05-27T09:31:20.845401+0000","last_became_peered":"2024-05-27T09:31:20.845401+0000","last_unstale":"2024-05-27T09:35:34.161273+0000","last_undegraded":"2024-05-27T09:35:34.161273+0000","last_fullsized":"2024-05-27T09:35:34.161273+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:21:38.460183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69968,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":469,"num_read_kb":304,"num_write":39,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170004+0000","last_change":"2024-05-27T09:31:24.892748+0000","last_active":"2024-05-27T09:35:14.170004+0000","last_peered":"2024-05-27T09:35:14.170004+0000","last_clean":"2024-05-27T09:35:14.170004+0000","last_became_active":"2024-05-27T09:31:24.892394+0000","last_became_peered":"2024-05-27T09:31:24.892394+0000","last_unstale":"2024-05-27T09:35:14.170004+0000","last_undegraded":"2024-05-27T09:35:14.170004+0000","last_fullsized":"2024-05-27T09:35:14.170004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:39:31.381072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372830+0000","last_change":"2024-05-27T09:31:20.860789+0000","last_active":"2024-05-27T09:35:14.372830+0000","last_peered":"2024-05-27T09:35:14.372830+0000","last_clean":"2024-05-27T09:35:14.372830+0000","last_became_active":"2024-05-27T09:31:20.860369+0000","last_became_peered":"2024-05-27T09:31:20.860369+0000","last_unstale":"2024-05-27T09:35:14.372830+0000","last_undegraded":"2024-05-27T09:35:14.372830+0000","last_fullsized":"2024-05-27T09:35:14.372830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:05:35.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241584+0000","last_change":"2024-05-27T09:31:24.918239+0000","last_active":"2024-05-27T09:35:14.241584+0000","last_peered":"2024-05-27T09:35:14.241584+0000","last_clean":"2024-05-27T09:35:14.241584+0000","last_became_active":"2024-05-27T09:31:24.917750+0000","last_became_peered":"2024-05-27T09:31:24.917750+0000","last_unstale":"2024-05-27T09:35:14.241584+0000","last_undegraded":"2024-05-27T09:35:14.241584+0000","last_fullsized":"2024-05-27T09:35:14.241584+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:07.162994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129596+0000","last_change":"2024-05-27T09:31:20.865308+0000","last_active":"2024-05-27T09:35:14.129596+0000","last_peered":"2024-05-27T09:35:14.129596+0000","last_clean":"2024-05-27T09:35:14.129596+0000","last_became_active":"2024-05-27T09:31:20.864920+0000","last_became_peered":"2024-05-27T09:31:20.864920+0000","last_unstale":"2024-05-27T09:35:14.129596+0000","last_undegraded":"2024-05-27T09:35:14.129596+0000","last_fullsized":"2024-05-27T09:35:14.129596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:36.773322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.902533+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.902136+0000","last_became_peered":"2024-05-27T09:31:24.902136+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:49:32.203577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'8","reported_seq":271,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:33.284422+0000","last_change":"2024-05-27T09:31:20.846060+0000","last_active":"2024-05-27T09:35:33.284422+0000","last_peered":"2024-05-27T09:35:33.284422+0000","last_clean":"2024-05-27T09:35:33.284422+0000","last_became_active":"2024-05-27T09:31:20.845632+0000","last_became_peered":"2024-05-27T09:31:20.845632+0000","last_unstale":"2024-05-27T09:35:33.284422+0000","last_undegraded":"2024-05-27T09:35:33.284422+0000","last_fullsized":"2024-05-27T09:35:33.284422+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:55.166579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2208,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":235,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353158+0000","last_change":"2024-05-27T09:31:24.895866+0000","last_active":"2024-05-27T09:35:14.353158+0000","last_peered":"2024-05-27T09:35:14.353158+0000","last_clean":"2024-05-27T09:35:14.353158+0000","last_became_active":"2024-05-27T09:31:24.895413+0000","last_became_peered":"2024-05-27T09:31:24.895413+0000","last_unstale":"2024-05-27T09:35:14.353158+0000","last_undegraded":"2024-05-27T09:35:14.353158+0000","last_fullsized":"2024-05-27T09:35:14.353158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:01:30.075710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":45,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169708+0000","last_change":"2024-05-27T09:31:20.879086+0000","last_active":"2024-05-27T09:35:14.169708+0000","last_peered":"2024-05-27T09:35:14.169708+0000","last_clean":"2024-05-27T09:35:14.169708+0000","last_became_active":"2024-05-27T09:31:20.878331+0000","last_became_peered":"2024-05-27T09:31:20.878331+0000","last_unstale":"2024-05-27T09:35:14.169708+0000","last_undegraded":"2024-05-27T09:35:14.169708+0000","last_fullsized":"2024-05-27T09:35:14.169708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:31.169812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129641+0000","last_change":"2024-05-27T09:31:24.895069+0000","last_active":"2024-05-27T09:35:14.129641+0000","last_peered":"2024-05-27T09:35:14.129641+0000","last_clean":"2024-05-27T09:35:14.129641+0000","last_became_active":"2024-05-27T09:31:24.894626+0000","last_became_peered":"2024-05-27T09:31:24.894626+0000","last_unstale":"2024-05-27T09:35:14.129641+0000","last_undegraded":"2024-05-27T09:35:14.129641+0000","last_fullsized":"2024-05-27T09:35:14.129641+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:21:22.725268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170168+0000","last_change":"2024-05-27T09:31:20.877416+0000","last_active":"2024-05-27T09:35:14.170168+0000","last_peered":"2024-05-27T09:35:14.170168+0000","last_clean":"2024-05-27T09:35:14.170168+0000","last_became_active":"2024-05-27T09:31:20.876093+0000","last_became_peered":"2024-05-27T09:31:20.876093+0000","last_unstale":"2024-05-27T09:35:14.170168+0000","last_undegraded":"2024-05-27T09:35:14.170168+0000","last_fullsized":"2024-05-27T09:35:14.170168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:34.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840141+0000","last_change":"2024-05-27T09:31:24.898430+0000","last_active":"2024-05-27T09:35:14.840141+0000","last_peered":"2024-05-27T09:35:14.840141+0000","last_clean":"2024-05-27T09:35:14.840141+0000","last_became_active":"2024-05-27T09:31:24.897643+0000","last_became_peered":"2024-05-27T09:31:24.897643+0000","last_unstale":"2024-05-27T09:35:14.840141+0000","last_undegraded":"2024-05-27T09:35:14.840141+0000","last_fullsized":"2024-05-27T09:35:14.840141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:44.485561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046071+0000","last_change":"2024-05-27T09:31:20.858253+0000","last_active":"2024-05-27T09:35:14.046071+0000","last_peered":"2024-05-27T09:35:14.046071+0000","last_clean":"2024-05-27T09:35:14.046071+0000","last_became_active":"2024-05-27T09:31:20.857736+0000","last_became_peered":"2024-05-27T09:31:20.857736+0000","last_unstale":"2024-05-27T09:35:14.046071+0000","last_undegraded":"2024-05-27T09:35:14.046071+0000","last_fullsized":"2024-05-27T09:35:14.046071+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:53:48.301763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126982+0000","last_change":"2024-05-27T09:31:24.912788+0000","last_active":"2024-05-27T09:35:14.126982+0000","last_peered":"2024-05-27T09:35:14.126982+0000","last_clean":"2024-05-27T09:35:14.126982+0000","last_became_active":"2024-05-27T09:31:24.912366+0000","last_became_peered":"2024-05-27T09:31:24.912366+0000","last_unstale":"2024-05-27T09:35:14.126982+0000","last_undegraded":"2024-05-27T09:35:14.126982+0000","last_fullsized":"2024-05-27T09:35:14.126982+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:55:03.397578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353426+0000","last_change":"2024-05-27T09:31:20.862188+0000","last_active":"2024-05-27T09:35:14.353426+0000","last_peered":"2024-05-27T09:35:14.353426+0000","last_clean":"2024-05-27T09:35:14.353426+0000","last_became_active":"2024-05-27T09:31:20.861791+0000","last_became_peered":"2024-05-27T09:31:20.861791+0000","last_unstale":"2024-05-27T09:35:14.353426+0000","last_undegraded":"2024-05-27T09:35:14.353426+0000","last_fullsized":"2024-05-27T09:35:14.353426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:16:09.127223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126717+0000","last_change":"2024-05-27T09:31:24.898460+0000","last_active":"2024-05-27T09:35:14.126717+0000","last_peered":"2024-05-27T09:35:14.126717+0000","last_clean":"2024-05-27T09:35:14.126717+0000","last_became_active":"2024-05-27T09:31:24.896787+0000","last_became_peered":"2024-05-27T09:31:24.896787+0000","last_unstale":"2024-05-27T09:35:14.126717+0000","last_undegraded":"2024-05-27T09:35:14.126717+0000","last_fullsized":"2024-05-27T09:35:14.126717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:10.199227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.130015+0000","last_change":"2024-05-27T09:31:24.896323+0000","last_active":"2024-05-27T09:35:14.130015+0000","last_peered":"2024-05-27T09:35:14.130015+0000","last_clean":"2024-05-27T09:35:14.130015+0000","last_became_active":"2024-05-27T09:31:24.895919+0000","last_became_peered":"2024-05-27T09:31:24.895919+0000","last_unstale":"2024-05-27T09:35:14.130015+0000","last_undegraded":"2024-05-27T09:35:14.130015+0000","last_fullsized":"2024-05-27T09:35:14.130015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:04.883166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":264,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:33.283731+0000","last_change":"2024-05-27T09:31:20.862126+0000","last_active":"2024-05-27T09:35:33.283731+0000","last_peered":"2024-05-27T09:35:33.283731+0000","last_clean":"2024-05-27T09:35:33.283731+0000","last_became_active":"2024-05-27T09:31:20.858964+0000","last_became_peered":"2024-05-27T09:31:20.858964+0000","last_unstale":"2024-05-27T09:35:33.283731+0000","last_undegraded":"2024-05-27T09:35:33.283731+0000","last_fullsized":"2024-05-27T09:35:33.283731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:51:26.838559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":228,"num_read_kb":228,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171170+0000","last_change":"2024-05-27T09:31:24.905549+0000","last_active":"2024-05-27T09:35:14.171170+0000","last_peered":"2024-05-27T09:35:14.171170+0000","last_clean":"2024-05-27T09:35:14.171170+0000","last_became_active":"2024-05-27T09:31:24.905212+0000","last_became_peered":"2024-05-27T09:31:24.905212+0000","last_unstale":"2024-05-27T09:35:14.171170+0000","last_undegraded":"2024-05-27T09:35:14.171170+0000","last_fullsized":"2024-05-27T09:35:14.171170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:18.267807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043354+0000","last_change":"2024-05-27T09:31:20.848527+0000","last_active":"2024-05-27T09:35:14.043354+0000","last_peered":"2024-05-27T09:35:14.043354+0000","last_clean":"2024-05-27T09:35:14.043354+0000","last_became_active":"2024-05-27T09:31:20.847779+0000","last_became_peered":"2024-05-27T09:31:20.847779+0000","last_unstale":"2024-05-27T09:35:14.043354+0000","last_undegraded":"2024-05-27T09:35:14.043354+0000","last_fullsized":"2024-05-27T09:35:14.043354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:15:01.131306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"82'99","reported_seq":133,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:16.190224+0000","last_change":"2024-05-27T09:29:46.120380+0000","last_active":"2024-05-27T09:35:16.190224+0000","last_peered":"2024-05-27T09:35:16.190224+0000","last_clean":"2024-05-27T09:35:16.190224+0000","last_became_active":"2024-05-27T09:29:46.113658+0000","last_became_peered":"2024-05-27T09:29:46.113658+0000","last_unstale":"2024-05-27T09:35:16.190224+0000","last_undegraded":"2024-05-27T09:35:16.190224+0000","last_fullsized":"2024-05-27T09:35:16.190224+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_clean_scrub_stamp":"2024-05-27T09:25:47.244521+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:28:40.424824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241564+0000","last_change":"2024-05-27T09:31:24.900736+0000","last_active":"2024-05-27T09:35:14.241564+0000","last_peered":"2024-05-27T09:35:14.241564+0000","last_clean":"2024-05-27T09:35:14.241564+0000","last_became_active":"2024-05-27T09:31:24.900354+0000","last_became_peered":"2024-05-27T09:31:24.900354+0000","last_unstale":"2024-05-27T09:35:14.241564+0000","last_undegraded":"2024-05-27T09:35:14.241564+0000","last_fullsized":"2024-05-27T09:35:14.241564+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:39:33.171977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839843+0000","last_change":"2024-05-27T09:31:57.873603+0000","last_active":"2024-05-27T09:35:14.839843+0000","last_peered":"2024-05-27T09:35:14.839843+0000","last_clean":"2024-05-27T09:35:14.839843+0000","last_became_active":"2024-05-27T09:31:57.868616+0000","last_became_peered":"2024-05-27T09:31:57.868616+0000","last_unstale":"2024-05-27T09:35:14.839843+0000","last_undegraded":"2024-05-27T09:35:14.839843+0000","last_fullsized":"2024-05-27T09:35:14.839843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:02:20.112667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170149+0000","last_change":"2024-05-27T09:31:20.878557+0000","last_active":"2024-05-27T09:35:14.170149+0000","last_peered":"2024-05-27T09:35:14.170149+0000","last_clean":"2024-05-27T09:35:14.170149+0000","last_became_active":"2024-05-27T09:31:20.877969+0000","last_became_peered":"2024-05-27T09:31:20.877969+0000","last_unstale":"2024-05-27T09:35:14.170149+0000","last_undegraded":"2024-05-27T09:35:14.170149+0000","last_fullsized":"2024-05-27T09:35:14.170149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:46.859645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242259+0000","last_change":"2024-05-27T09:31:24.907906+0000","last_active":"2024-05-27T09:35:14.242259+0000","last_peered":"2024-05-27T09:35:14.242259+0000","last_clean":"2024-05-27T09:35:14.242259+0000","last_became_active":"2024-05-27T09:31:24.907473+0000","last_became_peered":"2024-05-27T09:31:24.907473+0000","last_unstale":"2024-05-27T09:35:14.242259+0000","last_undegraded":"2024-05-27T09:35:14.242259+0000","last_fullsized":"2024-05-27T09:35:14.242259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:11:38.536919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'102","reported_seq":223,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:35.032870+0000","last_change":"2024-05-27T09:31:20.881250+0000","last_active":"2024-05-27T09:35:35.032870+0000","last_peered":"2024-05-27T09:35:35.032870+0000","last_clean":"2024-05-27T09:35:35.032870+0000","last_became_active":"2024-05-27T09:31:20.880550+0000","last_became_peered":"2024-05-27T09:31:20.880550+0000","last_unstale":"2024-05-27T09:35:35.032870+0000","last_undegraded":"2024-05-27T09:35:35.032870+0000","last_fullsized":"2024-05-27T09:35:35.032870+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:14.783260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":187,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171395+0000","last_change":"2024-05-27T09:31:24.894554+0000","last_active":"2024-05-27T09:35:14.171395+0000","last_peered":"2024-05-27T09:35:14.171395+0000","last_clean":"2024-05-27T09:35:14.171395+0000","last_became_active":"2024-05-27T09:31:24.893800+0000","last_became_peered":"2024-05-27T09:31:24.893800+0000","last_unstale":"2024-05-27T09:35:14.171395+0000","last_undegraded":"2024-05-27T09:35:14.171395+0000","last_fullsized":"2024-05-27T09:35:14.171395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:13.368892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043584+0000","last_change":"2024-05-27T09:31:21.054476+0000","last_active":"2024-05-27T09:35:14.043584+0000","last_peered":"2024-05-27T09:35:14.043584+0000","last_clean":"2024-05-27T09:35:14.043584+0000","last_became_active":"2024-05-27T09:31:21.054091+0000","last_became_peered":"2024-05-27T09:31:21.054091+0000","last_unstale":"2024-05-27T09:35:14.043584+0000","last_undegraded":"2024-05-27T09:35:14.043584+0000","last_fullsized":"2024-05-27T09:35:14.043584+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:15.909230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043020+0000","last_change":"2024-05-27T09:31:24.913157+0000","last_active":"2024-05-27T09:35:14.043020+0000","last_peered":"2024-05-27T09:35:14.043020+0000","last_clean":"2024-05-27T09:35:14.043020+0000","last_became_active":"2024-05-27T09:31:24.912793+0000","last_became_peered":"2024-05-27T09:31:24.912793+0000","last_unstale":"2024-05-27T09:35:14.043020+0000","last_undegraded":"2024-05-27T09:35:14.043020+0000","last_fullsized":"2024-05-27T09:35:14.043020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:17:36.451069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372678+0000","last_change":"2024-05-27T09:31:20.861349+0000","last_active":"2024-05-27T09:35:14.372678+0000","last_peered":"2024-05-27T09:35:14.372678+0000","last_clean":"2024-05-27T09:35:14.372678+0000","last_became_active":"2024-05-27T09:31:20.860896+0000","last_became_peered":"2024-05-27T09:31:20.860896+0000","last_unstale":"2024-05-27T09:35:14.372678+0000","last_undegraded":"2024-05-27T09:35:14.372678+0000","last_fullsized":"2024-05-27T09:35:14.372678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:00.772910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169685+0000","last_change":"2024-05-27T09:31:24.893240+0000","last_active":"2024-05-27T09:35:14.169685+0000","last_peered":"2024-05-27T09:35:14.169685+0000","last_clean":"2024-05-27T09:35:14.169685+0000","last_became_active":"2024-05-27T09:31:24.892169+0000","last_became_peered":"2024-05-27T09:31:24.892169+0000","last_unstale":"2024-05-27T09:35:14.169685+0000","last_undegraded":"2024-05-27T09:35:14.169685+0000","last_fullsized":"2024-05-27T09:35:14.169685+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:26:59.660725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170715+0000","last_change":"2024-05-27T09:31:20.864307+0000","last_active":"2024-05-27T09:35:14.170715+0000","last_peered":"2024-05-27T09:35:14.170715+0000","last_clean":"2024-05-27T09:35:14.170715+0000","last_became_active":"2024-05-27T09:31:20.863913+0000","last_became_peered":"2024-05-27T09:31:20.863913+0000","last_unstale":"2024-05-27T09:35:14.170715+0000","last_undegraded":"2024-05-27T09:35:14.170715+0000","last_fullsized":"2024-05-27T09:35:14.170715+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:58:57.656731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839950+0000","last_change":"2024-05-27T09:31:24.905728+0000","last_active":"2024-05-27T09:35:14.839950+0000","last_peered":"2024-05-27T09:35:14.839950+0000","last_clean":"2024-05-27T09:35:14.839950+0000","last_became_active":"2024-05-27T09:31:24.905365+0000","last_became_peered":"2024-05-27T09:31:24.905365+0000","last_unstale":"2024-05-27T09:35:14.839950+0000","last_undegraded":"2024-05-27T09:35:14.839950+0000","last_fullsized":"2024-05-27T09:35:14.839950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:05.013037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373037+0000","last_change":"2024-05-27T09:31:20.878478+0000","last_active":"2024-05-27T09:35:14.373037+0000","last_peered":"2024-05-27T09:35:14.373037+0000","last_clean":"2024-05-27T09:35:14.373037+0000","last_became_active":"2024-05-27T09:31:20.877952+0000","last_became_peered":"2024-05-27T09:31:20.877952+0000","last_unstale":"2024-05-27T09:35:14.373037+0000","last_undegraded":"2024-05-27T09:35:14.373037+0000","last_fullsized":"2024-05-27T09:35:14.373037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:16:27.580641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171301+0000","last_change":"2024-05-27T09:31:24.904967+0000","last_active":"2024-05-27T09:35:14.171301+0000","last_peered":"2024-05-27T09:35:14.171301+0000","last_clean":"2024-05-27T09:35:14.171301+0000","last_became_active":"2024-05-27T09:31:24.904542+0000","last_became_peered":"2024-05-27T09:31:24.904542+0000","last_unstale":"2024-05-27T09:35:14.171301+0000","last_undegraded":"2024-05-27T09:35:14.171301+0000","last_fullsized":"2024-05-27T09:35:14.171301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:01:05.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169841+0000","last_change":"2024-05-27T09:31:57.875713+0000","last_active":"2024-05-27T09:35:14.169841+0000","last_peered":"2024-05-27T09:35:14.169841+0000","last_clean":"2024-05-27T09:35:14.169841+0000","last_became_active":"2024-05-27T09:31:57.870244+0000","last_became_peered":"2024-05-27T09:31:57.870244+0000","last_unstale":"2024-05-27T09:35:14.169841+0000","last_undegraded":"2024-05-27T09:35:14.169841+0000","last_fullsized":"2024-05-27T09:35:14.169841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:49.272418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353395+0000","last_change":"2024-05-27T09:31:24.892655+0000","last_active":"2024-05-27T09:35:14.353395+0000","last_peered":"2024-05-27T09:35:14.353395+0000","last_clean":"2024-05-27T09:35:14.353395+0000","last_became_active":"2024-05-27T09:31:24.892112+0000","last_became_peered":"2024-05-27T09:31:24.892112+0000","last_unstale":"2024-05-27T09:35:14.353395+0000","last_undegraded":"2024-05-27T09:35:14.353395+0000","last_fullsized":"2024-05-27T09:35:14.353395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.604118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043417+0000","last_change":"2024-05-27T09:31:57.875213+0000","last_active":"2024-05-27T09:35:14.043417+0000","last_peered":"2024-05-27T09:35:14.043417+0000","last_clean":"2024-05-27T09:35:14.043417+0000","last_became_active":"2024-05-27T09:31:57.869025+0000","last_became_peered":"2024-05-27T09:31:57.869025+0000","last_unstale":"2024-05-27T09:35:14.043417+0000","last_undegraded":"2024-05-27T09:35:14.043417+0000","last_fullsized":"2024-05-27T09:35:14.043417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:51.433347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353725+0000","last_change":"2024-05-27T09:31:24.899130+0000","last_active":"2024-05-27T09:35:14.353725+0000","last_peered":"2024-05-27T09:35:14.353725+0000","last_clean":"2024-05-27T09:35:14.353725+0000","last_became_active":"2024-05-27T09:31:24.898722+0000","last_became_peered":"2024-05-27T09:31:24.898722+0000","last_unstale":"2024-05-27T09:35:14.353725+0000","last_undegraded":"2024-05-27T09:35:14.353725+0000","last_fullsized":"2024-05-27T09:35:14.353725+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:51.247275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'65","reported_seq":125,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:34.462413+0000","last_change":"2024-05-27T09:31:20.863757+0000","last_active":"2024-05-27T09:35:34.462413+0000","last_peered":"2024-05-27T09:35:34.462413+0000","last_clean":"2024-05-27T09:35:34.462413+0000","last_became_active":"2024-05-27T09:31:20.863465+0000","last_became_peered":"2024-05-27T09:31:20.863465+0000","last_unstale":"2024-05-27T09:35:34.462413+0000","last_undegraded":"2024-05-27T09:35:34.462413+0000","last_fullsized":"2024-05-27T09:35:34.462413+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:21:45.741822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.913749+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.913384+0000","last_became_peered":"2024-05-27T09:31:24.913384+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:24:29.826096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242122+0000","last_change":"2024-05-27T09:31:20.865844+0000","last_active":"2024-05-27T09:35:14.242122+0000","last_peered":"2024-05-27T09:35:14.242122+0000","last_clean":"2024-05-27T09:35:14.242122+0000","last_became_active":"2024-05-27T09:31:20.865364+0000","last_became_peered":"2024-05-27T09:31:20.865364+0000","last_unstale":"2024-05-27T09:35:14.242122+0000","last_undegraded":"2024-05-27T09:35:14.242122+0000","last_fullsized":"2024-05-27T09:35:14.242122+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:11:54.451294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353821+0000","last_change":"2024-05-27T09:31:24.890326+0000","last_active":"2024-05-27T09:35:14.353821+0000","last_peered":"2024-05-27T09:35:14.353821+0000","last_clean":"2024-05-27T09:35:14.353821+0000","last_became_active":"2024-05-27T09:31:24.889789+0000","last_became_peered":"2024-05-27T09:31:24.889789+0000","last_unstale":"2024-05-27T09:35:14.353821+0000","last_undegraded":"2024-05-27T09:35:14.353821+0000","last_fullsized":"2024-05-27T09:35:14.353821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:34:04.252231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840061+0000","last_change":"2024-05-27T09:31:24.896730+0000","last_active":"2024-05-27T09:35:14.840061+0000","last_peered":"2024-05-27T09:35:14.840061+0000","last_clean":"2024-05-27T09:35:14.840061+0000","last_became_active":"2024-05-27T09:31:24.896365+0000","last_became_peered":"2024-05-27T09:31:24.896365+0000","last_unstale":"2024-05-27T09:35:14.840061+0000","last_undegraded":"2024-05-27T09:35:14.840061+0000","last_fullsized":"2024-05-27T09:35:14.840061+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:37.443788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372900+0000","last_change":"2024-05-27T09:31:24.899215+0000","last_active":"2024-05-27T09:35:14.372900+0000","last_peered":"2024-05-27T09:35:14.372900+0000","last_clean":"2024-05-27T09:35:14.372900+0000","last_became_active":"2024-05-27T09:31:24.898733+0000","last_became_peered":"2024-05-27T09:31:24.898733+0000","last_unstale":"2024-05-27T09:35:14.372900+0000","last_undegraded":"2024-05-27T09:35:14.372900+0000","last_fullsized":"2024-05-27T09:35:14.372900+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:10.085989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372709+0000","last_change":"2024-05-27T09:31:24.914922+0000","last_active":"2024-05-27T09:35:14.372709+0000","last_peered":"2024-05-27T09:35:14.372709+0000","last_clean":"2024-05-27T09:35:14.372709+0000","last_became_active":"2024-05-27T09:31:24.914541+0000","last_became_peered":"2024-05-27T09:31:24.914541+0000","last_unstale":"2024-05-27T09:35:14.372709+0000","last_undegraded":"2024-05-27T09:35:14.372709+0000","last_fullsized":"2024-05-27T09:35:14.372709+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:30:38.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042498+0000","last_change":"2024-05-27T09:31:24.909628+0000","last_active":"2024-05-27T09:35:14.042498+0000","last_peered":"2024-05-27T09:35:14.042498+0000","last_clean":"2024-05-27T09:35:14.042498+0000","last_became_active":"2024-05-27T09:31:24.902931+0000","last_became_peered":"2024-05-27T09:31:24.902931+0000","last_unstale":"2024-05-27T09:35:14.042498+0000","last_undegraded":"2024-05-27T09:35:14.042498+0000","last_fullsized":"2024-05-27T09:35:14.042498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:06.711316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242398+0000","last_change":"2024-05-27T09:31:24.912099+0000","last_active":"2024-05-27T09:35:14.242398+0000","last_peered":"2024-05-27T09:35:14.242398+0000","last_clean":"2024-05-27T09:35:14.242398+0000","last_became_active":"2024-05-27T09:31:24.911669+0000","last_became_peered":"2024-05-27T09:31:24.911669+0000","last_unstale":"2024-05-27T09:35:14.242398+0000","last_undegraded":"2024-05-27T09:35:14.242398+0000","last_fullsized":"2024-05-27T09:35:14.242398+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:58.068589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840085+0000","last_change":"2024-05-27T09:31:24.891426+0000","last_active":"2024-05-27T09:35:14.840085+0000","last_peered":"2024-05-27T09:35:14.840085+0000","last_clean":"2024-05-27T09:35:14.840085+0000","last_became_active":"2024-05-27T09:31:24.890877+0000","last_became_peered":"2024-05-27T09:31:24.890877+0000","last_unstale":"2024-05-27T09:35:14.840085+0000","last_undegraded":"2024-05-27T09:35:14.840085+0000","last_fullsized":"2024-05-27T09:35:14.840085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:15.338188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373079+0000","last_change":"2024-05-27T09:31:24.892021+0000","last_active":"2024-05-27T09:35:14.373079+0000","last_peered":"2024-05-27T09:35:14.373079+0000","last_clean":"2024-05-27T09:35:14.373079+0000","last_became_active":"2024-05-27T09:31:24.891607+0000","last_became_peered":"2024-05-27T09:31:24.891607+0000","last_unstale":"2024-05-27T09:35:14.373079+0000","last_undegraded":"2024-05-27T09:35:14.373079+0000","last_fullsized":"2024-05-27T09:35:14.373079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:05.623199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171321+0000","last_change":"2024-05-27T09:31:24.912166+0000","last_active":"2024-05-27T09:35:14.171321+0000","last_peered":"2024-05-27T09:35:14.171321+0000","last_clean":"2024-05-27T09:35:14.171321+0000","last_became_active":"2024-05-27T09:31:24.907555+0000","last_became_peered":"2024-05-27T09:31:24.907555+0000","last_unstale":"2024-05-27T09:35:14.171321+0000","last_undegraded":"2024-05-27T09:35:14.171321+0000","last_fullsized":"2024-05-27T09:35:14.171321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:55:33.772321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129575+0000","last_change":"2024-05-27T09:31:24.910525+0000","last_active":"2024-05-27T09:35:14.129575+0000","last_peered":"2024-05-27T09:35:14.129575+0000","last_clean":"2024-05-27T09:35:14.129575+0000","last_became_active":"2024-05-27T09:31:24.910089+0000","last_became_peered":"2024-05-27T09:31:24.910089+0000","last_unstale":"2024-05-27T09:35:14.129575+0000","last_undegraded":"2024-05-27T09:35:14.129575+0000","last_fullsized":"2024-05-27T09:35:14.129575+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:19:02.274781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170634+0000","last_change":"2024-05-27T09:31:24.901125+0000","last_active":"2024-05-27T09:35:14.170634+0000","last_peered":"2024-05-27T09:35:14.170634+0000","last_clean":"2024-05-27T09:35:14.170634+0000","last_became_active":"2024-05-27T09:31:24.900361+0000","last_became_peered":"2024-05-27T09:31:24.900361+0000","last_unstale":"2024-05-27T09:35:14.170634+0000","last_undegraded":"2024-05-27T09:35:14.170634+0000","last_fullsized":"2024-05-27T09:35:14.170634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:30.116349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372869+0000","last_change":"2024-05-27T09:31:24.917612+0000","last_active":"2024-05-27T09:35:14.372869+0000","last_peered":"2024-05-27T09:35:14.372869+0000","last_clean":"2024-05-27T09:35:14.372869+0000","last_became_active":"2024-05-27T09:31:24.917210+0000","last_became_peered":"2024-05-27T09:31:24.917210+0000","last_unstale":"2024-05-27T09:35:14.372869+0000","last_undegraded":"2024-05-27T09:35:14.372869+0000","last_fullsized":"2024-05-27T09:35:14.372869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:19:00.769114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241872+0000","last_change":"2024-05-27T09:31:24.909654+0000","last_active":"2024-05-27T09:35:14.241872+0000","last_peered":"2024-05-27T09:35:14.241872+0000","last_clean":"2024-05-27T09:35:14.241872+0000","last_became_active":"2024-05-27T09:31:24.909143+0000","last_became_peered":"2024-05-27T09:31:24.909143+0000","last_unstale":"2024-05-27T09:35:14.241872+0000","last_undegraded":"2024-05-27T09:35:14.241872+0000","last_fullsized":"2024-05-27T09:35:14.241872+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:42.844614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840108+0000","last_change":"2024-05-27T09:31:24.899031+0000","last_active":"2024-05-27T09:35:14.840108+0000","last_peered":"2024-05-27T09:35:14.840108+0000","last_clean":"2024-05-27T09:35:14.840108+0000","last_became_active":"2024-05-27T09:31:24.898664+0000","last_became_peered":"2024-05-27T09:31:24.898664+0000","last_unstale":"2024-05-27T09:35:14.840108+0000","last_undegraded":"2024-05-27T09:35:14.840108+0000","last_fullsized":"2024-05-27T09:35:14.840108+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:09.659646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":797569,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1751,"num_read_kb":1094,"num_write":427,"num_write_kb":2361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2617344,"data_stored":2392707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678083,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":892928,"data_stored":440736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874311,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41500,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703652,"statfs":{"total":95995035648,"available":95952539648,"internally_reserved":0,"allocated":1929216,"data_stored":1432037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070541,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1302528,"data_stored":834188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266771,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1302528,"data_stored":834108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863463001,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28484,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716668,"statfs":{"total":95995035648,"available":95965868032,"internally_reserved":0,"allocated":1503232,"data_stored":1034974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626526,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":897024,"data_stored":440361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":510122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":441632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411448,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":511813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182511,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28736,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716416,"statfs":{"total":95995035648,"available":95965609984,"internally_reserved":0,"allocated":1638400,"data_stored":1162120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986283,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":443000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822758,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37260,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707892,"statfs":{"total":95995035648,"available":95956881408,"internally_reserved":0,"allocated":1847296,"data_stored":1361270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":721732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":330631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":401398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T09:35:40.595 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T09:35:41.112 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:42.300 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:42 smithi044 ceph-mon[31618]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T09:35:42.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:42 smithi050 ceph-mon[36371]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T09:35:42.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:42 smithi189 ceph-mon[36671]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T09:35:43.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.371+0000 7ff641800700 1 -- 172.21.15.44:0/3288990119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff63c10b2d0 msgr2=0x7ff63c117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:43.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.371+0000 7ff641800700 1 --2- 172.21.15.44:0/3288990119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff63c10b2d0 0x7ff63c117800 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff634009f90 tx=0x7ff63400b330 comp rx=0 tx=0).stop 2024-05-27T09:35:43.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.371+0000 7ff641800700 1 -- 172.21.15.44:0/3288990119 shutdown_connections 2024-05-27T09:35:43.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.371+0000 7ff641800700 1 --2- 172.21.15.44:0/3288990119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff63c10b2d0 0x7ff63c117800 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:43.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.371+0000 7ff641800700 1 --2- 172.21.15.44:0/3288990119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c110960 0x7ff63c110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:43.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.371+0000 7ff641800700 1 --2- 172.21.15.44:0/3288990119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff63c10a980 0x7ff63c10ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:43.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.371+0000 7ff641800700 1 -- 172.21.15.44:0/3288990119 >> 172.21.15.44:0/3288990119 conn(0x7ff63c077c70 msgr2=0x7ff63c0780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:43.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.371+0000 7ff641800700 1 -- 172.21.15.44:0/3288990119 shutdown_connections 2024-05-27T09:35:43.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.372+0000 7ff641800700 1 -- 172.21.15.44:0/3288990119 wait complete. 2024-05-27T09:35:43.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.373+0000 7ff641800700 1 Processor -- start 2024-05-27T09:35:43.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.373+0000 7ff641800700 1 -- start start 2024-05-27T09:35:43.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.374+0000 7ff641800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff63c10a980 0x7ff63c1a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:43.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.374+0000 7ff641800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c10b2d0 0x7ff63c1a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:43.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.374+0000 7ff641800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff63c110960 0x7ff63c1a5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:43.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.374+0000 7ff641800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c11aad0 con 0x7ff63c10a980 2024-05-27T09:35:43.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.374+0000 7ff641800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c11a950 con 0x7ff63c110960 2024-05-27T09:35:43.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.374+0000 7ff641800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c11ac50 con 0x7ff63c10b2d0 2024-05-27T09:35:43.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.374+0000 7ff633e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c10b2d0 0x7ff63c1a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:43.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.374+0000 7ff633e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c10b2d0 0x7ff63c1a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:54872/0 (socket says 172.21.15.44:54872) 2024-05-27T09:35:43.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.374+0000 7ff633e00700 1 -- 172.21.15.44:0/3308959612 learned_addr learned my addr 172.21.15.44:0/3308959612 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:43.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.374+0000 7ff63aa00700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff63c10a980 0x7ff63c1a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:43.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.374+0000 7ff63b400700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff63c110960 0x7ff63c1a5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:43.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.375+0000 7ff633e00700 1 -- 172.21.15.44:0/3308959612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff63c110960 msgr2=0x7ff63c1a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:43.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.375+0000 7ff633e00700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff63c110960 0x7ff63c1a5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:43.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.375+0000 7ff633e00700 1 -- 172.21.15.44:0/3308959612 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff63c10a980 msgr2=0x7ff63c1a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:43.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.375+0000 7ff633e00700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff63c10a980 0x7ff63c1a1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:43.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.375+0000 7ff633e00700 1 -- 172.21.15.44:0/3308959612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff63c1a62f0 con 0x7ff63c10b2d0 2024-05-27T09:35:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.375+0000 7ff633e00700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c10b2d0 0x7ff63c1a19c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff62400ec70 tx=0x7ff62400ef80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.376+0000 7ff633400700 1 -- 172.21.15.44:0/3308959612 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62400cc90 con 0x7ff63c10b2d0 2024-05-27T09:35:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.376+0000 7ff641800700 1 -- 172.21.15.44:0/3308959612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63c1a65e0 con 0x7ff63c10b2d0 2024-05-27T09:35:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.376+0000 7ff633400700 1 -- 172.21.15.44:0/3308959612 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff62400ce30 con 0x7ff63c10b2d0 2024-05-27T09:35:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.376+0000 7ff633400700 1 -- 172.21.15.44:0/3308959612 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624010430 con 0x7ff63c10b2d0 2024-05-27T09:35:43.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.376+0000 7ff641800700 1 -- 172.21.15.44:0/3308959612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff63c1adf00 con 0x7ff63c10b2d0 2024-05-27T09:35:43.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.377+0000 7ff641800700 1 -- 172.21.15.44:0/3308959612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff63c0512b0 con 0x7ff63c10b2d0 2024-05-27T09:35:43.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.380+0000 7ff633400700 1 -- 172.21.15.44:0/3308959612 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7ff624003680 con 0x7ff63c10b2d0 2024-05-27T09:35:43.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.381+0000 7ff633400700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff61c0b2b10 0x7ff61c0b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:43.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.382+0000 7ff633400700 1 -- 172.21.15.44:0/3308959612 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff6240e2cc0 con 0x7ff63c10b2d0 2024-05-27T09:35:43.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.382+0000 7ff63aa00700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff61c0b2b10 0x7ff61c0b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:43.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.383+0000 7ff63aa00700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff61c0b2b10 0x7ff61c0b5020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff63c07d0e0 tx=0x7ff62c009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:43.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.383+0000 7ff633400700 1 -- 172.21.15.44:0/3308959612 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff6240aa420 con 0x7ff63c10b2d0 2024-05-27T09:35:43.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.617+0000 7ff641800700 1 -- 172.21.15.44:0/3308959612 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff63c063bf0 con 0x7ff61c0b2b10 2024-05-27T09:35:43.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.627+0000 7ff633400700 1 -- 172.21.15.44:0/3308959612 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257756 (secure 0 0 0) 0x7ff63c063bf0 con 0x7ff61c0b2b10 2024-05-27T09:35:43.632 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:43.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.632+0000 7ff641800700 1 -- 172.21.15.44:0/3308959612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff61c0b2b10 msgr2=0x7ff61c0b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:43.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.632+0000 7ff641800700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff61c0b2b10 0x7ff61c0b5020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff63c07d0e0 tx=0x7ff62c009260 comp rx=0 tx=0).stop 2024-05-27T09:35:43.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.632+0000 7ff641800700 1 -- 172.21.15.44:0/3308959612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c10b2d0 msgr2=0x7ff63c1a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:43.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.632+0000 7ff641800700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c10b2d0 0x7ff63c1a19c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff62400ec70 tx=0x7ff62400ef80 comp rx=0 tx=0).stop 2024-05-27T09:35:43.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.632+0000 7ff641800700 1 -- 172.21.15.44:0/3308959612 shutdown_connections 2024-05-27T09:35:43.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.632+0000 7ff641800700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff61c0b2b10 0x7ff61c0b5020 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:43.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.632+0000 7ff641800700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff63c110960 0x7ff63c1a5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:43.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.632+0000 7ff641800700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff63c10a980 0x7ff63c1a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:43.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.633+0000 7ff641800700 1 --2- 172.21.15.44:0/3308959612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c10b2d0 0x7ff63c1a19c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:43.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.633+0000 7ff641800700 1 -- 172.21.15.44:0/3308959612 >> 172.21.15.44:0/3308959612 conn(0x7ff63c077c70 msgr2=0x7ff63c07b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:43.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.633+0000 7ff641800700 1 -- 172.21.15.44:0/3308959612 shutdown_connections 2024-05-27T09:35:43.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:43.633+0000 7ff641800700 1 -- 172.21.15.44:0/3308959612 wait complete. 2024-05-27T09:35:43.635 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-05-27T09:35:44.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:44 smithi044 ceph-mon[31618]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:44.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:44 smithi044 ceph-mon[31618]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-05-27T09:35:44.460 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-05-27T09:35:42.077085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1388171,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1925,"num_read_kb":1357,"num_write":564,"num_write_kb":4513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":916,"ondisk_log_size":916,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385072,"kb_used_data":14736,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124556752,"statfs":{"total":1151940427776,"available":1151546114048,"internally_reserved":0,"allocated":15089664,"data_stored":9446361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":345,"num_read_kb":277,"num_write":126,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004690"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840281+0000","last_change":"2024-05-27T09:31:24.899109+0000","last_active":"2024-05-27T09:35:14.840281+0000","last_peered":"2024-05-27T09:35:14.840281+0000","last_clean":"2024-05-27T09:35:14.840281+0000","last_became_active":"2024-05-27T09:31:24.898783+0000","last_became_peered":"2024-05-27T09:31:24.898783+0000","last_unstale":"2024-05-27T09:35:14.840281+0000","last_undegraded":"2024-05-27T09:35:14.840281+0000","last_fullsized":"2024-05-27T09:35:14.840281+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:01:12.220389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042833+0000","last_change":"2024-05-27T09:31:24.898382+0000","last_active":"2024-05-27T09:35:14.042833+0000","last_peered":"2024-05-27T09:35:14.042833+0000","last_clean":"2024-05-27T09:35:14.042833+0000","last_became_active":"2024-05-27T09:31:24.897975+0000","last_became_peered":"2024-05-27T09:31:24.897975+0000","last_unstale":"2024-05-27T09:35:14.042833+0000","last_undegraded":"2024-05-27T09:35:14.042833+0000","last_fullsized":"2024-05-27T09:35:14.042833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:38.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041882+0000","last_change":"2024-05-27T09:31:24.896122+0000","last_active":"2024-05-27T09:35:14.041882+0000","last_peered":"2024-05-27T09:35:14.041882+0000","last_clean":"2024-05-27T09:35:14.041882+0000","last_became_active":"2024-05-27T09:31:24.890867+0000","last_became_peered":"2024-05-27T09:31:24.890867+0000","last_unstale":"2024-05-27T09:35:14.041882+0000","last_undegraded":"2024-05-27T09:35:14.041882+0000","last_fullsized":"2024-05-27T09:35:14.041882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:09.982177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046136+0000","last_change":"2024-05-27T09:31:57.868900+0000","last_active":"2024-05-27T09:35:14.046136+0000","last_peered":"2024-05-27T09:35:14.046136+0000","last_clean":"2024-05-27T09:35:14.046136+0000","last_became_active":"2024-05-27T09:31:57.868418+0000","last_became_peered":"2024-05-27T09:31:57.868418+0000","last_unstale":"2024-05-27T09:35:14.046136+0000","last_undegraded":"2024-05-27T09:35:14.046136+0000","last_fullsized":"2024-05-27T09:35:14.046136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:08.966704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169866+0000","last_change":"2024-05-27T09:31:24.910283+0000","last_active":"2024-05-27T09:35:14.169866+0000","last_peered":"2024-05-27T09:35:14.169866+0000","last_clean":"2024-05-27T09:35:14.169866+0000","last_became_active":"2024-05-27T09:31:24.909882+0000","last_became_peered":"2024-05-27T09:31:24.909882+0000","last_unstale":"2024-05-27T09:35:14.169866+0000","last_undegraded":"2024-05-27T09:35:14.169866+0000","last_fullsized":"2024-05-27T09:35:14.169866+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:43.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126676+0000","last_change":"2024-05-27T09:31:24.900934+0000","last_active":"2024-05-27T09:35:14.126676+0000","last_peered":"2024-05-27T09:35:14.126676+0000","last_clean":"2024-05-27T09:35:14.126676+0000","last_became_active":"2024-05-27T09:31:24.900534+0000","last_became_peered":"2024-05-27T09:31:24.900534+0000","last_unstale":"2024-05-27T09:35:14.126676+0000","last_undegraded":"2024-05-27T09:35:14.126676+0000","last_fullsized":"2024-05-27T09:35:14.126676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:53.318327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353192+0000","last_change":"2024-05-27T09:31:24.907080+0000","last_active":"2024-05-27T09:35:14.353192+0000","last_peered":"2024-05-27T09:35:14.353192+0000","last_clean":"2024-05-27T09:35:14.353192+0000","last_became_active":"2024-05-27T09:31:24.906715+0000","last_became_peered":"2024-05-27T09:31:24.906715+0000","last_unstale":"2024-05-27T09:35:14.353192+0000","last_undegraded":"2024-05-27T09:35:14.353192+0000","last_fullsized":"2024-05-27T09:35:14.353192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:36.214659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.044350+0000","last_change":"2024-05-27T09:31:58.200608+0000","last_active":"2024-05-27T09:35:14.044350+0000","last_peered":"2024-05-27T09:35:14.044350+0000","last_clean":"2024-05-27T09:35:14.044350+0000","last_became_active":"2024-05-27T09:31:58.199363+0000","last_became_peered":"2024-05-27T09:31:58.199363+0000","last_unstale":"2024-05-27T09:35:14.044350+0000","last_undegraded":"2024-05-27T09:35:14.044350+0000","last_fullsized":"2024-05-27T09:35:14.044350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:30:46.495930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373508+0000","last_change":"2024-05-27T09:31:24.892630+0000","last_active":"2024-05-27T09:35:14.373508+0000","last_peered":"2024-05-27T09:35:14.373508+0000","last_clean":"2024-05-27T09:35:14.373508+0000","last_became_active":"2024-05-27T09:31:24.892176+0000","last_became_peered":"2024-05-27T09:31:24.892176+0000","last_unstale":"2024-05-27T09:35:14.373508+0000","last_undegraded":"2024-05-27T09:35:14.373508+0000","last_fullsized":"2024-05-27T09:35:14.373508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:49:20.585601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.127155+0000","last_change":"2024-05-27T09:31:24.895281+0000","last_active":"2024-05-27T09:35:14.127155+0000","last_peered":"2024-05-27T09:35:14.127155+0000","last_clean":"2024-05-27T09:35:14.127155+0000","last_became_active":"2024-05-27T09:31:24.894886+0000","last_became_peered":"2024-05-27T09:31:24.894886+0000","last_unstale":"2024-05-27T09:35:14.127155+0000","last_undegraded":"2024-05-27T09:35:14.127155+0000","last_fullsized":"2024-05-27T09:35:14.127155+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:41.494434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353559+0000","last_change":"2024-05-27T09:31:24.900167+0000","last_active":"2024-05-27T09:35:14.353559+0000","last_peered":"2024-05-27T09:35:14.353559+0000","last_clean":"2024-05-27T09:35:14.353559+0000","last_became_active":"2024-05-27T09:31:24.899750+0000","last_became_peered":"2024-05-27T09:31:24.899750+0000","last_unstale":"2024-05-27T09:35:14.353559+0000","last_undegraded":"2024-05-27T09:35:14.353559+0000","last_fullsized":"2024-05-27T09:35:14.353559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:30.602246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.045840+0000","last_change":"2024-05-27T09:31:57.868818+0000","last_active":"2024-05-27T09:35:14.045840+0000","last_peered":"2024-05-27T09:35:14.045840+0000","last_clean":"2024-05-27T09:35:14.045840+0000","last_became_active":"2024-05-27T09:31:57.868322+0000","last_became_peered":"2024-05-27T09:31:57.868322+0000","last_unstale":"2024-05-27T09:35:14.045840+0000","last_undegraded":"2024-05-27T09:35:14.045840+0000","last_fullsized":"2024-05-27T09:35:14.045840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:35.304850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242036+0000","last_change":"2024-05-27T09:31:24.902537+0000","last_active":"2024-05-27T09:35:14.242036+0000","last_peered":"2024-05-27T09:35:14.242036+0000","last_clean":"2024-05-27T09:35:14.242036+0000","last_became_active":"2024-05-27T09:31:24.902132+0000","last_became_peered":"2024-05-27T09:31:24.902132+0000","last_unstale":"2024-05-27T09:35:14.242036+0000","last_undegraded":"2024-05-27T09:35:14.242036+0000","last_fullsized":"2024-05-27T09:35:14.242036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:20:50.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041926+0000","last_change":"2024-05-27T09:31:24.899182+0000","last_active":"2024-05-27T09:35:14.041926+0000","last_peered":"2024-05-27T09:35:14.041926+0000","last_clean":"2024-05-27T09:35:14.041926+0000","last_became_active":"2024-05-27T09:31:24.898749+0000","last_became_peered":"2024-05-27T09:31:24.898749+0000","last_unstale":"2024-05-27T09:35:14.041926+0000","last_undegraded":"2024-05-27T09:35:14.041926+0000","last_fullsized":"2024-05-27T09:35:14.041926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:44:44.844357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041660+0000","last_change":"2024-05-27T09:31:24.901115+0000","last_active":"2024-05-27T09:35:14.041660+0000","last_peered":"2024-05-27T09:35:14.041660+0000","last_clean":"2024-05-27T09:35:14.041660+0000","last_became_active":"2024-05-27T09:31:24.897011+0000","last_became_peered":"2024-05-27T09:31:24.897011+0000","last_unstale":"2024-05-27T09:35:14.041660+0000","last_undegraded":"2024-05-27T09:35:14.041660+0000","last_fullsized":"2024-05-27T09:35:14.041660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:01.130407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372690+0000","last_change":"2024-05-27T09:31:24.903063+0000","last_active":"2024-05-27T09:35:14.372690+0000","last_peered":"2024-05-27T09:35:14.372690+0000","last_clean":"2024-05-27T09:35:14.372690+0000","last_became_active":"2024-05-27T09:31:24.899766+0000","last_became_peered":"2024-05-27T09:31:24.899766+0000","last_unstale":"2024-05-27T09:35:14.372690+0000","last_undegraded":"2024-05-27T09:35:14.372690+0000","last_fullsized":"2024-05-27T09:35:14.372690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:45.984226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840388+0000","last_change":"2024-05-27T09:31:24.915922+0000","last_active":"2024-05-27T09:35:14.840388+0000","last_peered":"2024-05-27T09:35:14.840388+0000","last_clean":"2024-05-27T09:35:14.840388+0000","last_became_active":"2024-05-27T09:31:24.915623+0000","last_became_peered":"2024-05-27T09:31:24.915623+0000","last_unstale":"2024-05-27T09:35:14.840388+0000","last_undegraded":"2024-05-27T09:35:14.840388+0000","last_fullsized":"2024-05-27T09:35:14.840388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:02:01.006704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046094+0000","last_change":"2024-05-27T09:31:24.908397+0000","last_active":"2024-05-27T09:35:14.046094+0000","last_peered":"2024-05-27T09:35:14.046094+0000","last_clean":"2024-05-27T09:35:14.046094+0000","last_became_active":"2024-05-27T09:31:24.903644+0000","last_became_peered":"2024-05-27T09:31:24.903644+0000","last_unstale":"2024-05-27T09:35:14.046094+0000","last_undegraded":"2024-05-27T09:35:14.046094+0000","last_fullsized":"2024-05-27T09:35:14.046094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:09.281429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373078+0000","last_change":"2024-05-27T09:31:24.912706+0000","last_active":"2024-05-27T09:35:14.373078+0000","last_peered":"2024-05-27T09:35:14.373078+0000","last_clean":"2024-05-27T09:35:14.373078+0000","last_became_active":"2024-05-27T09:31:24.912357+0000","last_became_peered":"2024-05-27T09:31:24.912357+0000","last_unstale":"2024-05-27T09:35:14.373078+0000","last_undegraded":"2024-05-27T09:35:14.373078+0000","last_fullsized":"2024-05-27T09:35:14.373078+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:23.334041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:30.093913+0000","last_change":"2024-05-27T09:31:20.881652+0000","last_active":"2024-05-27T09:35:30.093913+0000","last_peered":"2024-05-27T09:35:30.093913+0000","last_clean":"2024-05-27T09:35:30.093913+0000","last_became_active":"2024-05-27T09:31:20.880757+0000","last_became_peered":"2024-05-27T09:31:20.880757+0000","last_unstale":"2024-05-27T09:35:30.093913+0000","last_undegraded":"2024-05-27T09:35:30.093913+0000","last_fullsized":"2024-05-27T09:35:30.093913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:01.433755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171081+0000","last_change":"2024-05-27T09:31:24.908010+0000","last_active":"2024-05-27T09:35:14.171081+0000","last_peered":"2024-05-27T09:35:14.171081+0000","last_clean":"2024-05-27T09:35:14.171081+0000","last_became_active":"2024-05-27T09:31:24.907638+0000","last_became_peered":"2024-05-27T09:31:24.907638+0000","last_unstale":"2024-05-27T09:35:14.171081+0000","last_undegraded":"2024-05-27T09:35:14.171081+0000","last_fullsized":"2024-05-27T09:35:14.171081+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:02:49.207709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171465+0000","last_change":"2024-05-27T09:31:20.860556+0000","last_active":"2024-05-27T09:35:14.171465+0000","last_peered":"2024-05-27T09:35:14.171465+0000","last_clean":"2024-05-27T09:35:14.171465+0000","last_became_active":"2024-05-27T09:31:20.860172+0000","last_became_peered":"2024-05-27T09:31:20.860172+0000","last_unstale":"2024-05-27T09:35:14.171465+0000","last_undegraded":"2024-05-27T09:35:14.171465+0000","last_fullsized":"2024-05-27T09:35:14.171465+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:08:21.545681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043114+0000","last_change":"2024-05-27T09:31:24.910111+0000","last_active":"2024-05-27T09:35:14.043114+0000","last_peered":"2024-05-27T09:35:14.043114+0000","last_clean":"2024-05-27T09:35:14.043114+0000","last_became_active":"2024-05-27T09:31:24.908985+0000","last_became_peered":"2024-05-27T09:31:24.908985+0000","last_unstale":"2024-05-27T09:35:14.043114+0000","last_undegraded":"2024-05-27T09:35:14.043114+0000","last_fullsized":"2024-05-27T09:35:14.043114+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:53:29.645792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170222+0000","last_change":"2024-05-27T09:31:20.879993+0000","last_active":"2024-05-27T09:35:14.170222+0000","last_peered":"2024-05-27T09:35:14.170222+0000","last_clean":"2024-05-27T09:35:14.170222+0000","last_became_active":"2024-05-27T09:31:20.878802+0000","last_became_peered":"2024-05-27T09:31:20.878802+0000","last_unstale":"2024-05-27T09:35:14.170222+0000","last_undegraded":"2024-05-27T09:35:14.170222+0000","last_fullsized":"2024-05-27T09:35:14.170222+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:15.555612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372970+0000","last_change":"2024-05-27T09:31:24.903533+0000","last_active":"2024-05-27T09:35:14.372970+0000","last_peered":"2024-05-27T09:35:14.372970+0000","last_clean":"2024-05-27T09:35:14.372970+0000","last_became_active":"2024-05-27T09:31:24.903166+0000","last_became_peered":"2024-05-27T09:31:24.903166+0000","last_unstale":"2024-05-27T09:35:14.372970+0000","last_undegraded":"2024-05-27T09:35:14.372970+0000","last_fullsized":"2024-05-27T09:35:14.372970+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:52.539470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839887+0000","last_change":"2024-05-27T09:31:57.878426+0000","last_active":"2024-05-27T09:35:14.839887+0000","last_peered":"2024-05-27T09:35:14.839887+0000","last_clean":"2024-05-27T09:35:14.839887+0000","last_became_active":"2024-05-27T09:31:57.869146+0000","last_became_peered":"2024-05-27T09:31:57.869146+0000","last_unstale":"2024-05-27T09:35:14.839887+0000","last_undegraded":"2024-05-27T09:35:14.839887+0000","last_fullsized":"2024-05-27T09:35:14.839887+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:34:51.926092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129913+0000","last_change":"2024-05-27T09:31:24.897426+0000","last_active":"2024-05-27T09:35:14.129913+0000","last_peered":"2024-05-27T09:35:14.129913+0000","last_clean":"2024-05-27T09:35:14.129913+0000","last_became_active":"2024-05-27T09:31:24.897080+0000","last_became_peered":"2024-05-27T09:31:24.897080+0000","last_unstale":"2024-05-27T09:35:14.129913+0000","last_undegraded":"2024-05-27T09:35:14.129913+0000","last_fullsized":"2024-05-27T09:35:14.129913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:15:42.428391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'8","reported_seq":275,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:38.167769+0000","last_change":"2024-05-27T09:31:20.846681+0000","last_active":"2024-05-27T09:35:38.167769+0000","last_peered":"2024-05-27T09:35:38.167769+0000","last_clean":"2024-05-27T09:35:38.167769+0000","last_became_active":"2024-05-27T09:31:20.846302+0000","last_became_peered":"2024-05-27T09:31:20.846302+0000","last_unstale":"2024-05-27T09:35:38.167769+0000","last_undegraded":"2024-05-27T09:35:38.167769+0000","last_fullsized":"2024-05-27T09:35:38.167769+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:19.609612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":238,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169711+0000","last_change":"2024-05-27T09:31:24.907209+0000","last_active":"2024-05-27T09:35:14.169711+0000","last_peered":"2024-05-27T09:35:14.169711+0000","last_clean":"2024-05-27T09:35:14.169711+0000","last_became_active":"2024-05-27T09:31:24.906752+0000","last_became_peered":"2024-05-27T09:31:24.906752+0000","last_unstale":"2024-05-27T09:35:14.169711+0000","last_undegraded":"2024-05-27T09:35:14.169711+0000","last_fullsized":"2024-05-27T09:35:14.169711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:53.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126569+0000","last_change":"2024-05-27T09:31:20.879778+0000","last_active":"2024-05-27T09:35:14.126569+0000","last_peered":"2024-05-27T09:35:14.126569+0000","last_clean":"2024-05-27T09:35:14.126569+0000","last_became_active":"2024-05-27T09:31:20.879259+0000","last_became_peered":"2024-05-27T09:31:20.879259+0000","last_unstale":"2024-05-27T09:35:14.126569+0000","last_undegraded":"2024-05-27T09:35:14.126569+0000","last_fullsized":"2024-05-27T09:35:14.126569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:20:07.972653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129702+0000","last_change":"2024-05-27T09:31:58.195516+0000","last_active":"2024-05-27T09:35:14.129702+0000","last_peered":"2024-05-27T09:35:14.129702+0000","last_clean":"2024-05-27T09:35:14.129702+0000","last_became_active":"2024-05-27T09:31:58.195118+0000","last_became_peered":"2024-05-27T09:31:58.195118+0000","last_unstale":"2024-05-27T09:35:14.129702+0000","last_undegraded":"2024-05-27T09:35:14.129702+0000","last_fullsized":"2024-05-27T09:35:14.129702+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:11:23.547605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241846+0000","last_change":"2024-05-27T09:31:20.863152+0000","last_active":"2024-05-27T09:35:14.241846+0000","last_peered":"2024-05-27T09:35:14.241846+0000","last_clean":"2024-05-27T09:35:14.241846+0000","last_became_active":"2024-05-27T09:31:20.862708+0000","last_became_peered":"2024-05-27T09:31:20.862708+0000","last_unstale":"2024-05-27T09:35:14.241846+0000","last_undegraded":"2024-05-27T09:35:14.241846+0000","last_fullsized":"2024-05-27T09:35:14.241846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:21.645773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170544+0000","last_change":"2024-05-27T09:31:24.893988+0000","last_active":"2024-05-27T09:35:14.170544+0000","last_peered":"2024-05-27T09:35:14.170544+0000","last_clean":"2024-05-27T09:35:14.170544+0000","last_became_active":"2024-05-27T09:31:24.893504+0000","last_became_peered":"2024-05-27T09:31:24.893504+0000","last_unstale":"2024-05-27T09:35:14.170544+0000","last_undegraded":"2024-05-27T09:35:14.170544+0000","last_fullsized":"2024-05-27T09:35:14.170544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:47.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129578+0000","last_change":"2024-05-27T09:31:20.845555+0000","last_active":"2024-05-27T09:35:14.129578+0000","last_peered":"2024-05-27T09:35:14.129578+0000","last_clean":"2024-05-27T09:35:14.129578+0000","last_became_active":"2024-05-27T09:31:20.845139+0000","last_became_peered":"2024-05-27T09:31:20.845139+0000","last_unstale":"2024-05-27T09:35:14.129578+0000","last_undegraded":"2024-05-27T09:35:14.129578+0000","last_fullsized":"2024-05-27T09:35:14.129578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:25.917845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.049161+0000","last_change":"2024-05-27T09:31:24.916142+0000","last_active":"2024-05-27T09:35:14.049161+0000","last_peered":"2024-05-27T09:35:14.049161+0000","last_clean":"2024-05-27T09:35:14.049161+0000","last_became_active":"2024-05-27T09:31:24.915709+0000","last_became_peered":"2024-05-27T09:31:24.915709+0000","last_unstale":"2024-05-27T09:35:14.049161+0000","last_undegraded":"2024-05-27T09:35:14.049161+0000","last_fullsized":"2024-05-27T09:35:14.049161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:38.029648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'5","reported_seq":267,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:38.167164+0000","last_change":"2024-05-27T09:31:20.859240+0000","last_active":"2024-05-27T09:35:38.167164+0000","last_peered":"2024-05-27T09:35:38.167164+0000","last_clean":"2024-05-27T09:35:38.167164+0000","last_became_active":"2024-05-27T09:31:20.858798+0000","last_became_peered":"2024-05-27T09:31:20.858798+0000","last_unstale":"2024-05-27T09:35:38.167164+0000","last_undegraded":"2024-05-27T09:35:38.167164+0000","last_fullsized":"2024-05-27T09:35:38.167164+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:42.280165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":233,"num_read_kb":233,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373425+0000","last_change":"2024-05-27T09:31:24.908818+0000","last_active":"2024-05-27T09:35:14.373425+0000","last_peered":"2024-05-27T09:35:14.373425+0000","last_clean":"2024-05-27T09:35:14.373425+0000","last_became_active":"2024-05-27T09:31:24.908410+0000","last_became_peered":"2024-05-27T09:31:24.908410+0000","last_unstale":"2024-05-27T09:35:14.373425+0000","last_undegraded":"2024-05-27T09:35:14.373425+0000","last_fullsized":"2024-05-27T09:35:14.373425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:05:40.326232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353642+0000","last_change":"2024-05-27T09:31:20.848632+0000","last_active":"2024-05-27T09:35:14.353642+0000","last_peered":"2024-05-27T09:35:14.353642+0000","last_clean":"2024-05-27T09:35:14.353642+0000","last_became_active":"2024-05-27T09:31:20.847558+0000","last_became_peered":"2024-05-27T09:31:20.847558+0000","last_unstale":"2024-05-27T09:35:14.353642+0000","last_undegraded":"2024-05-27T09:35:14.353642+0000","last_fullsized":"2024-05-27T09:35:14.353642+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:53.930565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043056+0000","last_change":"2024-05-27T09:31:24.901513+0000","last_active":"2024-05-27T09:35:14.043056+0000","last_peered":"2024-05-27T09:35:14.043056+0000","last_clean":"2024-05-27T09:35:14.043056+0000","last_became_active":"2024-05-27T09:31:24.895924+0000","last_became_peered":"2024-05-27T09:31:24.895924+0000","last_unstale":"2024-05-27T09:35:14.043056+0000","last_undegraded":"2024-05-27T09:35:14.043056+0000","last_fullsized":"2024-05-27T09:35:14.043056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:27:39.939098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.130061+0000","last_change":"2024-05-27T09:31:20.855195+0000","last_active":"2024-05-27T09:35:14.130061+0000","last_peered":"2024-05-27T09:35:14.130061+0000","last_clean":"2024-05-27T09:35:14.130061+0000","last_became_active":"2024-05-27T09:31:20.854910+0000","last_became_peered":"2024-05-27T09:31:20.854910+0000","last_unstale":"2024-05-27T09:35:14.130061+0000","last_undegraded":"2024-05-27T09:35:14.130061+0000","last_fullsized":"2024-05-27T09:35:14.130061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:02.795103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373077+0000","last_change":"2024-05-27T09:31:24.895887+0000","last_active":"2024-05-27T09:35:14.373077+0000","last_peered":"2024-05-27T09:35:14.373077+0000","last_clean":"2024-05-27T09:35:14.373077+0000","last_became_active":"2024-05-27T09:31:24.894622+0000","last_became_peered":"2024-05-27T09:31:24.894622+0000","last_unstale":"2024-05-27T09:35:14.373077+0000","last_undegraded":"2024-05-27T09:35:14.373077+0000","last_fullsized":"2024-05-27T09:35:14.373077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:32.833946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'38","reported_seq":546,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:39.168478+0000","last_change":"2024-05-27T09:31:20.846063+0000","last_active":"2024-05-27T09:35:39.168478+0000","last_peered":"2024-05-27T09:35:39.168478+0000","last_clean":"2024-05-27T09:35:39.168478+0000","last_became_active":"2024-05-27T09:31:20.845401+0000","last_became_peered":"2024-05-27T09:31:20.845401+0000","last_unstale":"2024-05-27T09:35:39.168478+0000","last_undegraded":"2024-05-27T09:35:39.168478+0000","last_fullsized":"2024-05-27T09:35:39.168478+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:21:38.460183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69968,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":479,"num_read_kb":309,"num_write":39,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170004+0000","last_change":"2024-05-27T09:31:24.892748+0000","last_active":"2024-05-27T09:35:14.170004+0000","last_peered":"2024-05-27T09:35:14.170004+0000","last_clean":"2024-05-27T09:35:14.170004+0000","last_became_active":"2024-05-27T09:31:24.892394+0000","last_became_peered":"2024-05-27T09:31:24.892394+0000","last_unstale":"2024-05-27T09:35:14.170004+0000","last_undegraded":"2024-05-27T09:35:14.170004+0000","last_fullsized":"2024-05-27T09:35:14.170004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:39:31.381072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372830+0000","last_change":"2024-05-27T09:31:20.860789+0000","last_active":"2024-05-27T09:35:14.372830+0000","last_peered":"2024-05-27T09:35:14.372830+0000","last_clean":"2024-05-27T09:35:14.372830+0000","last_became_active":"2024-05-27T09:31:20.860369+0000","last_became_peered":"2024-05-27T09:31:20.860369+0000","last_unstale":"2024-05-27T09:35:14.372830+0000","last_undegraded":"2024-05-27T09:35:14.372830+0000","last_fullsized":"2024-05-27T09:35:14.372830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:05:35.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241584+0000","last_change":"2024-05-27T09:31:24.918239+0000","last_active":"2024-05-27T09:35:14.241584+0000","last_peered":"2024-05-27T09:35:14.241584+0000","last_clean":"2024-05-27T09:35:14.241584+0000","last_became_active":"2024-05-27T09:31:24.917750+0000","last_became_peered":"2024-05-27T09:31:24.917750+0000","last_unstale":"2024-05-27T09:35:14.241584+0000","last_undegraded":"2024-05-27T09:35:14.241584+0000","last_fullsized":"2024-05-27T09:35:14.241584+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:07.162994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129596+0000","last_change":"2024-05-27T09:31:20.865308+0000","last_active":"2024-05-27T09:35:14.129596+0000","last_peered":"2024-05-27T09:35:14.129596+0000","last_clean":"2024-05-27T09:35:14.129596+0000","last_became_active":"2024-05-27T09:31:20.864920+0000","last_became_peered":"2024-05-27T09:31:20.864920+0000","last_unstale":"2024-05-27T09:35:14.129596+0000","last_undegraded":"2024-05-27T09:35:14.129596+0000","last_fullsized":"2024-05-27T09:35:14.129596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:36.773322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.902533+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.902136+0000","last_became_peered":"2024-05-27T09:31:24.902136+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:49:32.203577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'8","reported_seq":276,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:38.296345+0000","last_change":"2024-05-27T09:31:20.846060+0000","last_active":"2024-05-27T09:35:38.296345+0000","last_peered":"2024-05-27T09:35:38.296345+0000","last_clean":"2024-05-27T09:35:38.296345+0000","last_became_active":"2024-05-27T09:31:20.845632+0000","last_became_peered":"2024-05-27T09:31:20.845632+0000","last_unstale":"2024-05-27T09:35:38.296345+0000","last_undegraded":"2024-05-27T09:35:38.296345+0000","last_fullsized":"2024-05-27T09:35:38.296345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:55.166579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2208,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":240,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353158+0000","last_change":"2024-05-27T09:31:24.895866+0000","last_active":"2024-05-27T09:35:14.353158+0000","last_peered":"2024-05-27T09:35:14.353158+0000","last_clean":"2024-05-27T09:35:14.353158+0000","last_became_active":"2024-05-27T09:31:24.895413+0000","last_became_peered":"2024-05-27T09:31:24.895413+0000","last_unstale":"2024-05-27T09:35:14.353158+0000","last_undegraded":"2024-05-27T09:35:14.353158+0000","last_fullsized":"2024-05-27T09:35:14.353158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:01:30.075710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":45,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169708+0000","last_change":"2024-05-27T09:31:20.879086+0000","last_active":"2024-05-27T09:35:14.169708+0000","last_peered":"2024-05-27T09:35:14.169708+0000","last_clean":"2024-05-27T09:35:14.169708+0000","last_became_active":"2024-05-27T09:31:20.878331+0000","last_became_peered":"2024-05-27T09:31:20.878331+0000","last_unstale":"2024-05-27T09:35:14.169708+0000","last_undegraded":"2024-05-27T09:35:14.169708+0000","last_fullsized":"2024-05-27T09:35:14.169708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:31.169812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129641+0000","last_change":"2024-05-27T09:31:24.895069+0000","last_active":"2024-05-27T09:35:14.129641+0000","last_peered":"2024-05-27T09:35:14.129641+0000","last_clean":"2024-05-27T09:35:14.129641+0000","last_became_active":"2024-05-27T09:31:24.894626+0000","last_became_peered":"2024-05-27T09:31:24.894626+0000","last_unstale":"2024-05-27T09:35:14.129641+0000","last_undegraded":"2024-05-27T09:35:14.129641+0000","last_fullsized":"2024-05-27T09:35:14.129641+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:21:22.725268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170168+0000","last_change":"2024-05-27T09:31:20.877416+0000","last_active":"2024-05-27T09:35:14.170168+0000","last_peered":"2024-05-27T09:35:14.170168+0000","last_clean":"2024-05-27T09:35:14.170168+0000","last_became_active":"2024-05-27T09:31:20.876093+0000","last_became_peered":"2024-05-27T09:31:20.876093+0000","last_unstale":"2024-05-27T09:35:14.170168+0000","last_undegraded":"2024-05-27T09:35:14.170168+0000","last_fullsized":"2024-05-27T09:35:14.170168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:34.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840141+0000","last_change":"2024-05-27T09:31:24.898430+0000","last_active":"2024-05-27T09:35:14.840141+0000","last_peered":"2024-05-27T09:35:14.840141+0000","last_clean":"2024-05-27T09:35:14.840141+0000","last_became_active":"2024-05-27T09:31:24.897643+0000","last_became_peered":"2024-05-27T09:31:24.897643+0000","last_unstale":"2024-05-27T09:35:14.840141+0000","last_undegraded":"2024-05-27T09:35:14.840141+0000","last_fullsized":"2024-05-27T09:35:14.840141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:44.485561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046071+0000","last_change":"2024-05-27T09:31:20.858253+0000","last_active":"2024-05-27T09:35:14.046071+0000","last_peered":"2024-05-27T09:35:14.046071+0000","last_clean":"2024-05-27T09:35:14.046071+0000","last_became_active":"2024-05-27T09:31:20.857736+0000","last_became_peered":"2024-05-27T09:31:20.857736+0000","last_unstale":"2024-05-27T09:35:14.046071+0000","last_undegraded":"2024-05-27T09:35:14.046071+0000","last_fullsized":"2024-05-27T09:35:14.046071+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:53:48.301763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126982+0000","last_change":"2024-05-27T09:31:24.912788+0000","last_active":"2024-05-27T09:35:14.126982+0000","last_peered":"2024-05-27T09:35:14.126982+0000","last_clean":"2024-05-27T09:35:14.126982+0000","last_became_active":"2024-05-27T09:31:24.912366+0000","last_became_peered":"2024-05-27T09:31:24.912366+0000","last_unstale":"2024-05-27T09:35:14.126982+0000","last_undegraded":"2024-05-27T09:35:14.126982+0000","last_fullsized":"2024-05-27T09:35:14.126982+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:55:03.397578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353426+0000","last_change":"2024-05-27T09:31:20.862188+0000","last_active":"2024-05-27T09:35:14.353426+0000","last_peered":"2024-05-27T09:35:14.353426+0000","last_clean":"2024-05-27T09:35:14.353426+0000","last_became_active":"2024-05-27T09:31:20.861791+0000","last_became_peered":"2024-05-27T09:31:20.861791+0000","last_unstale":"2024-05-27T09:35:14.353426+0000","last_undegraded":"2024-05-27T09:35:14.353426+0000","last_fullsized":"2024-05-27T09:35:14.353426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:16:09.127223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126717+0000","last_change":"2024-05-27T09:31:24.898460+0000","last_active":"2024-05-27T09:35:14.126717+0000","last_peered":"2024-05-27T09:35:14.126717+0000","last_clean":"2024-05-27T09:35:14.126717+0000","last_became_active":"2024-05-27T09:31:24.896787+0000","last_became_peered":"2024-05-27T09:31:24.896787+0000","last_unstale":"2024-05-27T09:35:14.126717+0000","last_undegraded":"2024-05-27T09:35:14.126717+0000","last_fullsized":"2024-05-27T09:35:14.126717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:10.199227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.130015+0000","last_change":"2024-05-27T09:31:24.896323+0000","last_active":"2024-05-27T09:35:14.130015+0000","last_peered":"2024-05-27T09:35:14.130015+0000","last_clean":"2024-05-27T09:35:14.130015+0000","last_became_active":"2024-05-27T09:31:24.895919+0000","last_became_peered":"2024-05-27T09:31:24.895919+0000","last_unstale":"2024-05-27T09:35:14.130015+0000","last_undegraded":"2024-05-27T09:35:14.130015+0000","last_fullsized":"2024-05-27T09:35:14.130015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:04.883166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":269,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:38.295551+0000","last_change":"2024-05-27T09:31:20.862126+0000","last_active":"2024-05-27T09:35:38.295551+0000","last_peered":"2024-05-27T09:35:38.295551+0000","last_clean":"2024-05-27T09:35:38.295551+0000","last_became_active":"2024-05-27T09:31:20.858964+0000","last_became_peered":"2024-05-27T09:31:20.858964+0000","last_unstale":"2024-05-27T09:35:38.295551+0000","last_undegraded":"2024-05-27T09:35:38.295551+0000","last_fullsized":"2024-05-27T09:35:38.295551+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:51:26.838559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":233,"num_read_kb":233,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171170+0000","last_change":"2024-05-27T09:31:24.905549+0000","last_active":"2024-05-27T09:35:14.171170+0000","last_peered":"2024-05-27T09:35:14.171170+0000","last_clean":"2024-05-27T09:35:14.171170+0000","last_became_active":"2024-05-27T09:31:24.905212+0000","last_became_peered":"2024-05-27T09:31:24.905212+0000","last_unstale":"2024-05-27T09:35:14.171170+0000","last_undegraded":"2024-05-27T09:35:14.171170+0000","last_fullsized":"2024-05-27T09:35:14.171170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:18.267807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043354+0000","last_change":"2024-05-27T09:31:20.848527+0000","last_active":"2024-05-27T09:35:14.043354+0000","last_peered":"2024-05-27T09:35:14.043354+0000","last_clean":"2024-05-27T09:35:14.043354+0000","last_became_active":"2024-05-27T09:31:20.847779+0000","last_became_peered":"2024-05-27T09:31:20.847779+0000","last_unstale":"2024-05-27T09:35:14.043354+0000","last_undegraded":"2024-05-27T09:35:14.043354+0000","last_fullsized":"2024-05-27T09:35:14.043354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:15:01.131306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"82'99","reported_seq":133,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:16.190224+0000","last_change":"2024-05-27T09:29:46.120380+0000","last_active":"2024-05-27T09:35:16.190224+0000","last_peered":"2024-05-27T09:35:16.190224+0000","last_clean":"2024-05-27T09:35:16.190224+0000","last_became_active":"2024-05-27T09:29:46.113658+0000","last_became_peered":"2024-05-27T09:29:46.113658+0000","last_unstale":"2024-05-27T09:35:16.190224+0000","last_undegraded":"2024-05-27T09:35:16.190224+0000","last_fullsized":"2024-05-27T09:35:16.190224+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_clean_scrub_stamp":"2024-05-27T09:25:47.244521+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:28:40.424824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241564+0000","last_change":"2024-05-27T09:31:24.900736+0000","last_active":"2024-05-27T09:35:14.241564+0000","last_peered":"2024-05-27T09:35:14.241564+0000","last_clean":"2024-05-27T09:35:14.241564+0000","last_became_active":"2024-05-27T09:31:24.900354+0000","last_became_peered":"2024-05-27T09:31:24.900354+0000","last_unstale":"2024-05-27T09:35:14.241564+0000","last_undegraded":"2024-05-27T09:35:14.241564+0000","last_fullsized":"2024-05-27T09:35:14.241564+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:39:33.171977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839843+0000","last_change":"2024-05-27T09:31:57.873603+0000","last_active":"2024-05-27T09:35:14.839843+0000","last_peered":"2024-05-27T09:35:14.839843+0000","last_clean":"2024-05-27T09:35:14.839843+0000","last_became_active":"2024-05-27T09:31:57.868616+0000","last_became_peered":"2024-05-27T09:31:57.868616+0000","last_unstale":"2024-05-27T09:35:14.839843+0000","last_undegraded":"2024-05-27T09:35:14.839843+0000","last_fullsized":"2024-05-27T09:35:14.839843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:02:20.112667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170149+0000","last_change":"2024-05-27T09:31:20.878557+0000","last_active":"2024-05-27T09:35:14.170149+0000","last_peered":"2024-05-27T09:35:14.170149+0000","last_clean":"2024-05-27T09:35:14.170149+0000","last_became_active":"2024-05-27T09:31:20.877969+0000","last_became_peered":"2024-05-27T09:31:20.877969+0000","last_unstale":"2024-05-27T09:35:14.170149+0000","last_undegraded":"2024-05-27T09:35:14.170149+0000","last_fullsized":"2024-05-27T09:35:14.170149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:46.859645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242259+0000","last_change":"2024-05-27T09:31:24.907906+0000","last_active":"2024-05-27T09:35:14.242259+0000","last_peered":"2024-05-27T09:35:14.242259+0000","last_clean":"2024-05-27T09:35:14.242259+0000","last_became_active":"2024-05-27T09:31:24.907473+0000","last_became_peered":"2024-05-27T09:31:24.907473+0000","last_unstale":"2024-05-27T09:35:14.242259+0000","last_undegraded":"2024-05-27T09:35:14.242259+0000","last_fullsized":"2024-05-27T09:35:14.242259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:11:38.536919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'102","reported_seq":224,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:35.035590+0000","last_change":"2024-05-27T09:31:20.881250+0000","last_active":"2024-05-27T09:35:35.035590+0000","last_peered":"2024-05-27T09:35:35.035590+0000","last_clean":"2024-05-27T09:35:35.035590+0000","last_became_active":"2024-05-27T09:31:20.880550+0000","last_became_peered":"2024-05-27T09:31:20.880550+0000","last_unstale":"2024-05-27T09:35:35.035590+0000","last_undegraded":"2024-05-27T09:35:35.035590+0000","last_fullsized":"2024-05-27T09:35:35.035590+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:14.783260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":187,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171395+0000","last_change":"2024-05-27T09:31:24.894554+0000","last_active":"2024-05-27T09:35:14.171395+0000","last_peered":"2024-05-27T09:35:14.171395+0000","last_clean":"2024-05-27T09:35:14.171395+0000","last_became_active":"2024-05-27T09:31:24.893800+0000","last_became_peered":"2024-05-27T09:31:24.893800+0000","last_unstale":"2024-05-27T09:35:14.171395+0000","last_undegraded":"2024-05-27T09:35:14.171395+0000","last_fullsized":"2024-05-27T09:35:14.171395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:13.368892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043584+0000","last_change":"2024-05-27T09:31:21.054476+0000","last_active":"2024-05-27T09:35:14.043584+0000","last_peered":"2024-05-27T09:35:14.043584+0000","last_clean":"2024-05-27T09:35:14.043584+0000","last_became_active":"2024-05-27T09:31:21.054091+0000","last_became_peered":"2024-05-27T09:31:21.054091+0000","last_unstale":"2024-05-27T09:35:14.043584+0000","last_undegraded":"2024-05-27T09:35:14.043584+0000","last_fullsized":"2024-05-27T09:35:14.043584+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:15.909230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043020+0000","last_change":"2024-05-27T09:31:24.913157+0000","last_active":"2024-05-27T09:35:14.043020+0000","last_peered":"2024-05-27T09:35:14.043020+0000","last_clean":"2024-05-27T09:35:14.043020+0000","last_became_active":"2024-05-27T09:31:24.912793+0000","last_became_peered":"2024-05-27T09:31:24.912793+0000","last_unstale":"2024-05-27T09:35:14.043020+0000","last_undegraded":"2024-05-27T09:35:14.043020+0000","last_fullsized":"2024-05-27T09:35:14.043020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:17:36.451069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372678+0000","last_change":"2024-05-27T09:31:20.861349+0000","last_active":"2024-05-27T09:35:14.372678+0000","last_peered":"2024-05-27T09:35:14.372678+0000","last_clean":"2024-05-27T09:35:14.372678+0000","last_became_active":"2024-05-27T09:31:20.860896+0000","last_became_peered":"2024-05-27T09:31:20.860896+0000","last_unstale":"2024-05-27T09:35:14.372678+0000","last_undegraded":"2024-05-27T09:35:14.372678+0000","last_fullsized":"2024-05-27T09:35:14.372678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:00.772910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169685+0000","last_change":"2024-05-27T09:31:24.893240+0000","last_active":"2024-05-27T09:35:14.169685+0000","last_peered":"2024-05-27T09:35:14.169685+0000","last_clean":"2024-05-27T09:35:14.169685+0000","last_became_active":"2024-05-27T09:31:24.892169+0000","last_became_peered":"2024-05-27T09:31:24.892169+0000","last_unstale":"2024-05-27T09:35:14.169685+0000","last_undegraded":"2024-05-27T09:35:14.169685+0000","last_fullsized":"2024-05-27T09:35:14.169685+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:26:59.660725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170750+0000","last_change":"2024-05-27T09:31:20.864307+0000","last_active":"2024-05-27T09:35:14.170750+0000","last_peered":"2024-05-27T09:35:14.170750+0000","last_clean":"2024-05-27T09:35:14.170750+0000","last_became_active":"2024-05-27T09:31:20.863913+0000","last_became_peered":"2024-05-27T09:31:20.863913+0000","last_unstale":"2024-05-27T09:35:14.170750+0000","last_undegraded":"2024-05-27T09:35:14.170750+0000","last_fullsized":"2024-05-27T09:35:14.170750+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:58:57.656731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839950+0000","last_change":"2024-05-27T09:31:24.905728+0000","last_active":"2024-05-27T09:35:14.839950+0000","last_peered":"2024-05-27T09:35:14.839950+0000","last_clean":"2024-05-27T09:35:14.839950+0000","last_became_active":"2024-05-27T09:31:24.905365+0000","last_became_peered":"2024-05-27T09:31:24.905365+0000","last_unstale":"2024-05-27T09:35:14.839950+0000","last_undegraded":"2024-05-27T09:35:14.839950+0000","last_fullsized":"2024-05-27T09:35:14.839950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:05.013037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373037+0000","last_change":"2024-05-27T09:31:20.878478+0000","last_active":"2024-05-27T09:35:14.373037+0000","last_peered":"2024-05-27T09:35:14.373037+0000","last_clean":"2024-05-27T09:35:14.373037+0000","last_became_active":"2024-05-27T09:31:20.877952+0000","last_became_peered":"2024-05-27T09:31:20.877952+0000","last_unstale":"2024-05-27T09:35:14.373037+0000","last_undegraded":"2024-05-27T09:35:14.373037+0000","last_fullsized":"2024-05-27T09:35:14.373037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:16:27.580641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171301+0000","last_change":"2024-05-27T09:31:24.904967+0000","last_active":"2024-05-27T09:35:14.171301+0000","last_peered":"2024-05-27T09:35:14.171301+0000","last_clean":"2024-05-27T09:35:14.171301+0000","last_became_active":"2024-05-27T09:31:24.904542+0000","last_became_peered":"2024-05-27T09:31:24.904542+0000","last_unstale":"2024-05-27T09:35:14.171301+0000","last_undegraded":"2024-05-27T09:35:14.171301+0000","last_fullsized":"2024-05-27T09:35:14.171301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:01:05.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169841+0000","last_change":"2024-05-27T09:31:57.875713+0000","last_active":"2024-05-27T09:35:14.169841+0000","last_peered":"2024-05-27T09:35:14.169841+0000","last_clean":"2024-05-27T09:35:14.169841+0000","last_became_active":"2024-05-27T09:31:57.870244+0000","last_became_peered":"2024-05-27T09:31:57.870244+0000","last_unstale":"2024-05-27T09:35:14.169841+0000","last_undegraded":"2024-05-27T09:35:14.169841+0000","last_fullsized":"2024-05-27T09:35:14.169841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:49.272418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353395+0000","last_change":"2024-05-27T09:31:24.892655+0000","last_active":"2024-05-27T09:35:14.353395+0000","last_peered":"2024-05-27T09:35:14.353395+0000","last_clean":"2024-05-27T09:35:14.353395+0000","last_became_active":"2024-05-27T09:31:24.892112+0000","last_became_peered":"2024-05-27T09:31:24.892112+0000","last_unstale":"2024-05-27T09:35:14.353395+0000","last_undegraded":"2024-05-27T09:35:14.353395+0000","last_fullsized":"2024-05-27T09:35:14.353395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.604118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043417+0000","last_change":"2024-05-27T09:31:57.875213+0000","last_active":"2024-05-27T09:35:14.043417+0000","last_peered":"2024-05-27T09:35:14.043417+0000","last_clean":"2024-05-27T09:35:14.043417+0000","last_became_active":"2024-05-27T09:31:57.869025+0000","last_became_peered":"2024-05-27T09:31:57.869025+0000","last_unstale":"2024-05-27T09:35:14.043417+0000","last_undegraded":"2024-05-27T09:35:14.043417+0000","last_fullsized":"2024-05-27T09:35:14.043417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:51.433347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353725+0000","last_change":"2024-05-27T09:31:24.899130+0000","last_active":"2024-05-27T09:35:14.353725+0000","last_peered":"2024-05-27T09:35:14.353725+0000","last_clean":"2024-05-27T09:35:14.353725+0000","last_became_active":"2024-05-27T09:31:24.898722+0000","last_became_peered":"2024-05-27T09:31:24.898722+0000","last_unstale":"2024-05-27T09:35:14.353725+0000","last_undegraded":"2024-05-27T09:35:14.353725+0000","last_fullsized":"2024-05-27T09:35:14.353725+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:51.247275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'65","reported_seq":125,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:34.462413+0000","last_change":"2024-05-27T09:31:20.863757+0000","last_active":"2024-05-27T09:35:34.462413+0000","last_peered":"2024-05-27T09:35:34.462413+0000","last_clean":"2024-05-27T09:35:34.462413+0000","last_became_active":"2024-05-27T09:31:20.863465+0000","last_became_peered":"2024-05-27T09:31:20.863465+0000","last_unstale":"2024-05-27T09:35:34.462413+0000","last_undegraded":"2024-05-27T09:35:34.462413+0000","last_fullsized":"2024-05-27T09:35:34.462413+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:21:45.741822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.913749+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.913384+0000","last_became_peered":"2024-05-27T09:31:24.913384+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:24:29.826096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242122+0000","last_change":"2024-05-27T09:31:20.865844+0000","last_active":"2024-05-27T09:35:14.242122+0000","last_peered":"2024-05-27T09:35:14.242122+0000","last_clean":"2024-05-27T09:35:14.242122+0000","last_became_active":"2024-05-27T09:31:20.865364+0000","last_became_peered":"2024-05-27T09:31:20.865364+0000","last_unstale":"2024-05-27T09:35:14.242122+0000","last_undegraded":"2024-05-27T09:35:14.242122+0000","last_fullsized":"2024-05-27T09:35:14.242122+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:11:54.451294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353821+0000","last_change":"2024-05-27T09:31:24.890326+0000","last_active":"2024-05-27T09:35:14.353821+0000","last_peered":"2024-05-27T09:35:14.353821+0000","last_clean":"2024-05-27T09:35:14.353821+0000","last_became_active":"2024-05-27T09:31:24.889789+0000","last_became_peered":"2024-05-27T09:31:24.889789+0000","last_unstale":"2024-05-27T09:35:14.353821+0000","last_undegraded":"2024-05-27T09:35:14.353821+0000","last_fullsized":"2024-05-27T09:35:14.353821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:34:04.252231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840061+0000","last_change":"2024-05-27T09:31:24.896730+0000","last_active":"2024-05-27T09:35:14.840061+0000","last_peered":"2024-05-27T09:35:14.840061+0000","last_clean":"2024-05-27T09:35:14.840061+0000","last_became_active":"2024-05-27T09:31:24.896365+0000","last_became_peered":"2024-05-27T09:31:24.896365+0000","last_unstale":"2024-05-27T09:35:14.840061+0000","last_undegraded":"2024-05-27T09:35:14.840061+0000","last_fullsized":"2024-05-27T09:35:14.840061+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:37.443788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372900+0000","last_change":"2024-05-27T09:31:24.899215+0000","last_active":"2024-05-27T09:35:14.372900+0000","last_peered":"2024-05-27T09:35:14.372900+0000","last_clean":"2024-05-27T09:35:14.372900+0000","last_became_active":"2024-05-27T09:31:24.898733+0000","last_became_peered":"2024-05-27T09:31:24.898733+0000","last_unstale":"2024-05-27T09:35:14.372900+0000","last_undegraded":"2024-05-27T09:35:14.372900+0000","last_fullsized":"2024-05-27T09:35:14.372900+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:10.085989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372769+0000","last_change":"2024-05-27T09:31:24.914922+0000","last_active":"2024-05-27T09:35:14.372769+0000","last_peered":"2024-05-27T09:35:14.372769+0000","last_clean":"2024-05-27T09:35:14.372769+0000","last_became_active":"2024-05-27T09:31:24.914541+0000","last_became_peered":"2024-05-27T09:31:24.914541+0000","last_unstale":"2024-05-27T09:35:14.372769+0000","last_undegraded":"2024-05-27T09:35:14.372769+0000","last_fullsized":"2024-05-27T09:35:14.372769+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:30:38.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042624+0000","last_change":"2024-05-27T09:31:24.909628+0000","last_active":"2024-05-27T09:35:14.042624+0000","last_peered":"2024-05-27T09:35:14.042624+0000","last_clean":"2024-05-27T09:35:14.042624+0000","last_became_active":"2024-05-27T09:31:24.902931+0000","last_became_peered":"2024-05-27T09:31:24.902931+0000","last_unstale":"2024-05-27T09:35:14.042624+0000","last_undegraded":"2024-05-27T09:35:14.042624+0000","last_fullsized":"2024-05-27T09:35:14.042624+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:06.711316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242398+0000","last_change":"2024-05-27T09:31:24.912099+0000","last_active":"2024-05-27T09:35:14.242398+0000","last_peered":"2024-05-27T09:35:14.242398+0000","last_clean":"2024-05-27T09:35:14.242398+0000","last_became_active":"2024-05-27T09:31:24.911669+0000","last_became_peered":"2024-05-27T09:31:24.911669+0000","last_unstale":"2024-05-27T09:35:14.242398+0000","last_undegraded":"2024-05-27T09:35:14.242398+0000","last_fullsized":"2024-05-27T09:35:14.242398+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:58.068589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840085+0000","last_change":"2024-05-27T09:31:24.891426+0000","last_active":"2024-05-27T09:35:14.840085+0000","last_peered":"2024-05-27T09:35:14.840085+0000","last_clean":"2024-05-27T09:35:14.840085+0000","last_became_active":"2024-05-27T09:31:24.890877+0000","last_became_peered":"2024-05-27T09:31:24.890877+0000","last_unstale":"2024-05-27T09:35:14.840085+0000","last_undegraded":"2024-05-27T09:35:14.840085+0000","last_fullsized":"2024-05-27T09:35:14.840085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:15.338188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373079+0000","last_change":"2024-05-27T09:31:24.892021+0000","last_active":"2024-05-27T09:35:14.373079+0000","last_peered":"2024-05-27T09:35:14.373079+0000","last_clean":"2024-05-27T09:35:14.373079+0000","last_became_active":"2024-05-27T09:31:24.891607+0000","last_became_peered":"2024-05-27T09:31:24.891607+0000","last_unstale":"2024-05-27T09:35:14.373079+0000","last_undegraded":"2024-05-27T09:35:14.373079+0000","last_fullsized":"2024-05-27T09:35:14.373079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:05.623199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171321+0000","last_change":"2024-05-27T09:31:24.912166+0000","last_active":"2024-05-27T09:35:14.171321+0000","last_peered":"2024-05-27T09:35:14.171321+0000","last_clean":"2024-05-27T09:35:14.171321+0000","last_became_active":"2024-05-27T09:31:24.907555+0000","last_became_peered":"2024-05-27T09:31:24.907555+0000","last_unstale":"2024-05-27T09:35:14.171321+0000","last_undegraded":"2024-05-27T09:35:14.171321+0000","last_fullsized":"2024-05-27T09:35:14.171321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:55:33.772321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129575+0000","last_change":"2024-05-27T09:31:24.910525+0000","last_active":"2024-05-27T09:35:14.129575+0000","last_peered":"2024-05-27T09:35:14.129575+0000","last_clean":"2024-05-27T09:35:14.129575+0000","last_became_active":"2024-05-27T09:31:24.910089+0000","last_became_peered":"2024-05-27T09:31:24.910089+0000","last_unstale":"2024-05-27T09:35:14.129575+0000","last_undegraded":"2024-05-27T09:35:14.129575+0000","last_fullsized":"2024-05-27T09:35:14.129575+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:19:02.274781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170634+0000","last_change":"2024-05-27T09:31:24.901125+0000","last_active":"2024-05-27T09:35:14.170634+0000","last_peered":"2024-05-27T09:35:14.170634+0000","last_clean":"2024-05-27T09:35:14.170634+0000","last_became_active":"2024-05-27T09:31:24.900361+0000","last_became_peered":"2024-05-27T09:31:24.900361+0000","last_unstale":"2024-05-27T09:35:14.170634+0000","last_undegraded":"2024-05-27T09:35:14.170634+0000","last_fullsized":"2024-05-27T09:35:14.170634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:30.116349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372869+0000","last_change":"2024-05-27T09:31:24.917612+0000","last_active":"2024-05-27T09:35:14.372869+0000","last_peered":"2024-05-27T09:35:14.372869+0000","last_clean":"2024-05-27T09:35:14.372869+0000","last_became_active":"2024-05-27T09:31:24.917210+0000","last_became_peered":"2024-05-27T09:31:24.917210+0000","last_unstale":"2024-05-27T09:35:14.372869+0000","last_undegraded":"2024-05-27T09:35:14.372869+0000","last_fullsized":"2024-05-27T09:35:14.372869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:19:00.769114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241872+0000","last_change":"2024-05-27T09:31:24.909654+0000","last_active":"2024-05-27T09:35:14.241872+0000","last_peered":"2024-05-27T09:35:14.241872+0000","last_clean":"2024-05-27T09:35:14.241872+0000","last_became_active":"2024-05-27T09:31:24.909143+0000","last_became_peered":"2024-05-27T09:31:24.909143+0000","last_unstale":"2024-05-27T09:35:14.241872+0000","last_undegraded":"2024-05-27T09:35:14.241872+0000","last_fullsized":"2024-05-27T09:35:14.241872+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:42.844614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840108+0000","last_change":"2024-05-27T09:31:24.899031+0000","last_active":"2024-05-27T09:35:14.840108+0000","last_peered":"2024-05-27T09:35:14.840108+0000","last_clean":"2024-05-27T09:35:14.840108+0000","last_became_active":"2024-05-27T09:31:24.898664+0000","last_became_peered":"2024-05-27T09:31:24.898664+0000","last_unstale":"2024-05-27T09:35:14.840108+0000","last_undegraded":"2024-05-27T09:35:14.840108+0000","last_fullsized":"2024-05-27T09:35:14.840108+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:09.659646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":797569,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1781,"num_read_kb":1109,"num_write":427,"num_write_kb":2361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2617344,"data_stored":2392707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678084,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":892928,"data_stored":440736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874312,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41500,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703652,"statfs":{"total":95995035648,"available":95952539648,"internally_reserved":0,"allocated":1929216,"data_stored":1432037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070542,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1302528,"data_stored":834188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266772,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1302528,"data_stored":834108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863463002,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28484,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716668,"statfs":{"total":95995035648,"available":95965868032,"internally_reserved":0,"allocated":1503232,"data_stored":1034974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626527,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":897024,"data_stored":440361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574974,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":510122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738497,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":441632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411449,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":511813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182512,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28736,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716416,"statfs":{"total":95995035648,"available":95965609984,"internally_reserved":0,"allocated":1638400,"data_stored":1162120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986284,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":443000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822759,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37260,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707892,"statfs":{"total":95995035648,"available":95956881408,"internally_reserved":0,"allocated":1847296,"data_stored":1361270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":721732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":330631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":401398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T09:35:44.464 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-27T09:35:44.465 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T09:35:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:44 smithi050 ceph-mon[36371]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:44 smithi050 ceph-mon[36371]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-05-27T09:35:44.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:44 smithi189 ceph-mon[36671]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:44.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:44 smithi189 ceph-mon[36671]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-05-27T09:35:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:46.028 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:45 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:35:46.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:45 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:35:46.208 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:45 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:35:46.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:46 smithi044 ceph-mon[31618]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T09:35:47.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:46 smithi050 ceph-mon[36371]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T09:35:47.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:46 smithi189 ceph-mon[36671]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-05-27T09:35:47.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.293+0000 7f9c23800700 1 -- 172.21.15.44:0/3636721368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c106eb0 msgr2=0x7f9c1c113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:47.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.293+0000 7f9c23800700 1 --2- 172.21.15.44:0/3636721368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c106eb0 0x7f9c1c113340 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400b0a0 tx=0x7f9c1400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:35:47.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.293+0000 7f9c23800700 1 -- 172.21.15.44:0/3636721368 shutdown_connections 2024-05-27T09:35:47.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.293+0000 7f9c23800700 1 --2- 172.21.15.44:0/3636721368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c106eb0 0x7f9c1c113340 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:47.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.293+0000 7f9c23800700 1 --2- 172.21.15.44:0/3636721368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1c10c5a0 0x7f9c1c10c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:47.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.293+0000 7f9c23800700 1 --2- 172.21.15.44:0/3636721368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c1c106560 0x7f9c1c106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:47.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.293+0000 7f9c23800700 1 -- 172.21.15.44:0/3636721368 >> 172.21.15.44:0/3636721368 conn(0x7f9c1c102040 msgr2=0x7f9c1c104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:47.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.293+0000 7f9c23800700 1 -- 172.21.15.44:0/3636721368 shutdown_connections 2024-05-27T09:35:47.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.293+0000 7f9c23800700 1 -- 172.21.15.44:0/3636721368 wait complete. 2024-05-27T09:35:47.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.294+0000 7f9c23800700 1 Processor -- start 2024-05-27T09:35:47.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.295+0000 7f9c23800700 1 -- start start 2024-05-27T09:35:47.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.295+0000 7f9c23800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c1c106560 0x7f9c1c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:47.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.295+0000 7f9c23800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1c106eb0 0x7f9c1c19d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:47.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.295+0000 7f9c23800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c10c5a0 0x7f9c1c1a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:47.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.295+0000 7f9c23800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c116720 con 0x7f9c1c10c5a0 2024-05-27T09:35:47.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.295+0000 7f9c23800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c1165a0 con 0x7f9c1c106560 2024-05-27T09:35:47.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.295+0000 7f9c23800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c1168a0 con 0x7f9c1c106eb0 2024-05-27T09:35:47.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c20e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c1c106560 0x7f9c1c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:47.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c21800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c10c5a0 0x7f9c1c1a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:47.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c20e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c1c106560 0x7f9c1c19cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39828/0 (socket says 172.21.15.44:39828) 2024-05-27T09:35:47.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c20e00700 1 -- 172.21.15.44:0/2053766228 learned_addr learned my addr 172.21.15.44:0/2053766228 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:47.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c1be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1c106eb0 0x7f9c1c19d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:47.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c1be00700 1 -- 172.21.15.44:0/2053766228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c1c106560 msgr2=0x7f9c1c19cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:47.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c1be00700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c1c106560 0x7f9c1c19cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:47.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c1be00700 1 -- 172.21.15.44:0/2053766228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c10c5a0 msgr2=0x7f9c1c1a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:47.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c1be00700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c10c5a0 0x7f9c1c1a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:47.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c1be00700 1 -- 172.21.15.44:0/2053766228 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1c1a1ea0 con 0x7f9c1c106eb0 2024-05-27T09:35:47.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c21800700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c10c5a0 0x7f9c1c1a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:47.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.296+0000 7f9c20e00700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c1c106560 0x7f9c1c19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:47.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.297+0000 7f9c1be00700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1c106eb0 0x7f9c1c19d570 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c075b70 tx=0x7f9c0c00eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:47.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.297+0000 7f9c19600700 1 -- 172.21.15.44:0/2053766228 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c010900 con 0x7f9c1c106eb0 2024-05-27T09:35:47.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.297+0000 7f9c23800700 1 -- 172.21.15.44:0/2053766228 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c1c1a2190 con 0x7f9c1c106eb0 2024-05-27T09:35:47.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.297+0000 7f9c19600700 1 -- 172.21.15.44:0/2053766228 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c0c00c430 con 0x7f9c1c106eb0 2024-05-27T09:35:47.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.297+0000 7f9c19600700 1 -- 172.21.15.44:0/2053766228 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c018680 con 0x7f9c1c106eb0 2024-05-27T09:35:47.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.297+0000 7f9c23800700 1 -- 172.21.15.44:0/2053766228 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c1c1a9ab0 con 0x7f9c1c106eb0 2024-05-27T09:35:47.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.298+0000 7f9c23800700 1 -- 172.21.15.44:0/2053766228 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c1c0512b0 con 0x7f9c1c106eb0 2024-05-27T09:35:47.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.301+0000 7f9c19600700 1 -- 172.21.15.44:0/2053766228 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f9c0c04ab90 con 0x7f9c1c106eb0 2024-05-27T09:35:47.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.303+0000 7f9c19600700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9c040b2b90 0x7f9c040b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:47.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.303+0000 7f9c19600700 1 -- 172.21.15.44:0/2053766228 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f9c0c0e29c0 con 0x7f9c1c106eb0 2024-05-27T09:35:47.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.304+0000 7f9c20e00700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9c040b2b90 0x7f9c040b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:47.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.304+0000 7f9c20e00700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9c040b2b90 0x7f9c040b50a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c10005950 tx=0x7f9c10009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:47.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.305+0000 7f9c19600700 1 -- 172.21.15.44:0/2053766228 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9c0c0aa120 con 0x7f9c1c106eb0 2024-05-27T09:35:47.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.537+0000 7f9c23800700 1 -- 172.21.15.44:0/2053766228 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9c1c19e330 con 0x7f9c040b2b90 2024-05-27T09:35:47.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.559+0000 7f9c19600700 1 -- 172.21.15.44:0/2053766228 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257756 (secure 0 0 0) 0x7f9c1c19e330 con 0x7f9c040b2b90 2024-05-27T09:35:47.562 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:47.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 -- 172.21.15.44:0/2053766228 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9c040b2b90 msgr2=0x7f9c040b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:47.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9c040b2b90 0x7f9c040b50a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c10005950 tx=0x7f9c10009260 comp rx=0 tx=0).stop 2024-05-27T09:35:47.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 -- 172.21.15.44:0/2053766228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1c106eb0 msgr2=0x7f9c1c19d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:47.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1c106eb0 0x7f9c1c19d570 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c075b70 tx=0x7f9c0c00eca0 comp rx=0 tx=0).stop 2024-05-27T09:35:47.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 -- 172.21.15.44:0/2053766228 shutdown_connections 2024-05-27T09:35:47.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9c040b2b90 0x7f9c040b50a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:47.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c1c10c5a0 0x7f9c1c1a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:47.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c1c106560 0x7f9c1c19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:47.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 --2- 172.21.15.44:0/2053766228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c1c106eb0 0x7f9c1c19d570 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:47.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 -- 172.21.15.44:0/2053766228 >> 172.21.15.44:0/2053766228 conn(0x7f9c1c102040 msgr2=0x7f9c1c103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:47.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 -- 172.21.15.44:0/2053766228 shutdown_connections 2024-05-27T09:35:47.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:47.567+0000 7f9c23800700 1 -- 172.21.15.44:0/2053766228 wait complete. 2024-05-27T09:35:47.569 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-05-27T09:35:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:48 smithi044 ceph-mon[31618]: from='client.25288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:48 smithi044 ceph-mon[31618]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:35:48.474 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-05-27T09:35:46.079447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1388171,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1955,"num_read_kb":1372,"num_write":564,"num_write_kb":4513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2952422,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":916,"ondisk_log_size":916,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385072,"kb_used_data":14736,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124556752,"statfs":{"total":1151940427776,"available":1151546114048,"internally_reserved":0,"allocated":15089664,"data_stored":9446361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":345,"num_read_kb":277,"num_write":126,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005969"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840281+0000","last_change":"2024-05-27T09:31:24.899109+0000","last_active":"2024-05-27T09:35:14.840281+0000","last_peered":"2024-05-27T09:35:14.840281+0000","last_clean":"2024-05-27T09:35:14.840281+0000","last_became_active":"2024-05-27T09:31:24.898783+0000","last_became_peered":"2024-05-27T09:31:24.898783+0000","last_unstale":"2024-05-27T09:35:14.840281+0000","last_undegraded":"2024-05-27T09:35:14.840281+0000","last_fullsized":"2024-05-27T09:35:14.840281+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:01:12.220389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042833+0000","last_change":"2024-05-27T09:31:24.898382+0000","last_active":"2024-05-27T09:35:14.042833+0000","last_peered":"2024-05-27T09:35:14.042833+0000","last_clean":"2024-05-27T09:35:14.042833+0000","last_became_active":"2024-05-27T09:31:24.897975+0000","last_became_peered":"2024-05-27T09:31:24.897975+0000","last_unstale":"2024-05-27T09:35:14.042833+0000","last_undegraded":"2024-05-27T09:35:14.042833+0000","last_fullsized":"2024-05-27T09:35:14.042833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:24:38.866687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041882+0000","last_change":"2024-05-27T09:31:24.896122+0000","last_active":"2024-05-27T09:35:14.041882+0000","last_peered":"2024-05-27T09:35:14.041882+0000","last_clean":"2024-05-27T09:35:14.041882+0000","last_became_active":"2024-05-27T09:31:24.890867+0000","last_became_peered":"2024-05-27T09:31:24.890867+0000","last_unstale":"2024-05-27T09:35:14.041882+0000","last_undegraded":"2024-05-27T09:35:14.041882+0000","last_fullsized":"2024-05-27T09:35:14.041882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:09.982177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046136+0000","last_change":"2024-05-27T09:31:57.868900+0000","last_active":"2024-05-27T09:35:14.046136+0000","last_peered":"2024-05-27T09:35:14.046136+0000","last_clean":"2024-05-27T09:35:14.046136+0000","last_became_active":"2024-05-27T09:31:57.868418+0000","last_became_peered":"2024-05-27T09:31:57.868418+0000","last_unstale":"2024-05-27T09:35:14.046136+0000","last_undegraded":"2024-05-27T09:35:14.046136+0000","last_fullsized":"2024-05-27T09:35:14.046136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:08.966704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169866+0000","last_change":"2024-05-27T09:31:24.910283+0000","last_active":"2024-05-27T09:35:14.169866+0000","last_peered":"2024-05-27T09:35:14.169866+0000","last_clean":"2024-05-27T09:35:14.169866+0000","last_became_active":"2024-05-27T09:31:24.909882+0000","last_became_peered":"2024-05-27T09:31:24.909882+0000","last_unstale":"2024-05-27T09:35:14.169866+0000","last_undegraded":"2024-05-27T09:35:14.169866+0000","last_fullsized":"2024-05-27T09:35:14.169866+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:43.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126676+0000","last_change":"2024-05-27T09:31:24.900934+0000","last_active":"2024-05-27T09:35:14.126676+0000","last_peered":"2024-05-27T09:35:14.126676+0000","last_clean":"2024-05-27T09:35:14.126676+0000","last_became_active":"2024-05-27T09:31:24.900534+0000","last_became_peered":"2024-05-27T09:31:24.900534+0000","last_unstale":"2024-05-27T09:35:14.126676+0000","last_undegraded":"2024-05-27T09:35:14.126676+0000","last_fullsized":"2024-05-27T09:35:14.126676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:53.318327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353192+0000","last_change":"2024-05-27T09:31:24.907080+0000","last_active":"2024-05-27T09:35:14.353192+0000","last_peered":"2024-05-27T09:35:14.353192+0000","last_clean":"2024-05-27T09:35:14.353192+0000","last_became_active":"2024-05-27T09:31:24.906715+0000","last_became_peered":"2024-05-27T09:31:24.906715+0000","last_unstale":"2024-05-27T09:35:14.353192+0000","last_undegraded":"2024-05-27T09:35:14.353192+0000","last_fullsized":"2024-05-27T09:35:14.353192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:36.214659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.044350+0000","last_change":"2024-05-27T09:31:58.200608+0000","last_active":"2024-05-27T09:35:14.044350+0000","last_peered":"2024-05-27T09:35:14.044350+0000","last_clean":"2024-05-27T09:35:14.044350+0000","last_became_active":"2024-05-27T09:31:58.199363+0000","last_became_peered":"2024-05-27T09:31:58.199363+0000","last_unstale":"2024-05-27T09:35:14.044350+0000","last_undegraded":"2024-05-27T09:35:14.044350+0000","last_fullsized":"2024-05-27T09:35:14.044350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:30:46.495930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373508+0000","last_change":"2024-05-27T09:31:24.892630+0000","last_active":"2024-05-27T09:35:14.373508+0000","last_peered":"2024-05-27T09:35:14.373508+0000","last_clean":"2024-05-27T09:35:14.373508+0000","last_became_active":"2024-05-27T09:31:24.892176+0000","last_became_peered":"2024-05-27T09:31:24.892176+0000","last_unstale":"2024-05-27T09:35:14.373508+0000","last_undegraded":"2024-05-27T09:35:14.373508+0000","last_fullsized":"2024-05-27T09:35:14.373508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:49:20.585601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.127155+0000","last_change":"2024-05-27T09:31:24.895281+0000","last_active":"2024-05-27T09:35:14.127155+0000","last_peered":"2024-05-27T09:35:14.127155+0000","last_clean":"2024-05-27T09:35:14.127155+0000","last_became_active":"2024-05-27T09:31:24.894886+0000","last_became_peered":"2024-05-27T09:31:24.894886+0000","last_unstale":"2024-05-27T09:35:14.127155+0000","last_undegraded":"2024-05-27T09:35:14.127155+0000","last_fullsized":"2024-05-27T09:35:14.127155+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:41.494434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353559+0000","last_change":"2024-05-27T09:31:24.900167+0000","last_active":"2024-05-27T09:35:14.353559+0000","last_peered":"2024-05-27T09:35:14.353559+0000","last_clean":"2024-05-27T09:35:14.353559+0000","last_became_active":"2024-05-27T09:31:24.899750+0000","last_became_peered":"2024-05-27T09:31:24.899750+0000","last_unstale":"2024-05-27T09:35:14.353559+0000","last_undegraded":"2024-05-27T09:35:14.353559+0000","last_fullsized":"2024-05-27T09:35:14.353559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:30.602246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.045840+0000","last_change":"2024-05-27T09:31:57.868818+0000","last_active":"2024-05-27T09:35:14.045840+0000","last_peered":"2024-05-27T09:35:14.045840+0000","last_clean":"2024-05-27T09:35:14.045840+0000","last_became_active":"2024-05-27T09:31:57.868322+0000","last_became_peered":"2024-05-27T09:31:57.868322+0000","last_unstale":"2024-05-27T09:35:14.045840+0000","last_undegraded":"2024-05-27T09:35:14.045840+0000","last_fullsized":"2024-05-27T09:35:14.045840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:35.304850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242036+0000","last_change":"2024-05-27T09:31:24.902537+0000","last_active":"2024-05-27T09:35:14.242036+0000","last_peered":"2024-05-27T09:35:14.242036+0000","last_clean":"2024-05-27T09:35:14.242036+0000","last_became_active":"2024-05-27T09:31:24.902132+0000","last_became_peered":"2024-05-27T09:31:24.902132+0000","last_unstale":"2024-05-27T09:35:14.242036+0000","last_undegraded":"2024-05-27T09:35:14.242036+0000","last_fullsized":"2024-05-27T09:35:14.242036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:20:50.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041926+0000","last_change":"2024-05-27T09:31:24.899182+0000","last_active":"2024-05-27T09:35:14.041926+0000","last_peered":"2024-05-27T09:35:14.041926+0000","last_clean":"2024-05-27T09:35:14.041926+0000","last_became_active":"2024-05-27T09:31:24.898749+0000","last_became_peered":"2024-05-27T09:31:24.898749+0000","last_unstale":"2024-05-27T09:35:14.041926+0000","last_undegraded":"2024-05-27T09:35:14.041926+0000","last_fullsized":"2024-05-27T09:35:14.041926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:44:44.844357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.041660+0000","last_change":"2024-05-27T09:31:24.901115+0000","last_active":"2024-05-27T09:35:14.041660+0000","last_peered":"2024-05-27T09:35:14.041660+0000","last_clean":"2024-05-27T09:35:14.041660+0000","last_became_active":"2024-05-27T09:31:24.897011+0000","last_became_peered":"2024-05-27T09:31:24.897011+0000","last_unstale":"2024-05-27T09:35:14.041660+0000","last_undegraded":"2024-05-27T09:35:14.041660+0000","last_fullsized":"2024-05-27T09:35:14.041660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:01.130407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372690+0000","last_change":"2024-05-27T09:31:24.903063+0000","last_active":"2024-05-27T09:35:14.372690+0000","last_peered":"2024-05-27T09:35:14.372690+0000","last_clean":"2024-05-27T09:35:14.372690+0000","last_became_active":"2024-05-27T09:31:24.899766+0000","last_became_peered":"2024-05-27T09:31:24.899766+0000","last_unstale":"2024-05-27T09:35:14.372690+0000","last_undegraded":"2024-05-27T09:35:14.372690+0000","last_fullsized":"2024-05-27T09:35:14.372690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:45.984226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840388+0000","last_change":"2024-05-27T09:31:24.915922+0000","last_active":"2024-05-27T09:35:14.840388+0000","last_peered":"2024-05-27T09:35:14.840388+0000","last_clean":"2024-05-27T09:35:14.840388+0000","last_became_active":"2024-05-27T09:31:24.915623+0000","last_became_peered":"2024-05-27T09:31:24.915623+0000","last_unstale":"2024-05-27T09:35:14.840388+0000","last_undegraded":"2024-05-27T09:35:14.840388+0000","last_fullsized":"2024-05-27T09:35:14.840388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:02:01.006704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046094+0000","last_change":"2024-05-27T09:31:24.908397+0000","last_active":"2024-05-27T09:35:14.046094+0000","last_peered":"2024-05-27T09:35:14.046094+0000","last_clean":"2024-05-27T09:35:14.046094+0000","last_became_active":"2024-05-27T09:31:24.903644+0000","last_became_peered":"2024-05-27T09:31:24.903644+0000","last_unstale":"2024-05-27T09:35:14.046094+0000","last_undegraded":"2024-05-27T09:35:14.046094+0000","last_fullsized":"2024-05-27T09:35:14.046094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:09.281429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373078+0000","last_change":"2024-05-27T09:31:24.912706+0000","last_active":"2024-05-27T09:35:14.373078+0000","last_peered":"2024-05-27T09:35:14.373078+0000","last_clean":"2024-05-27T09:35:14.373078+0000","last_became_active":"2024-05-27T09:31:24.912357+0000","last_became_peered":"2024-05-27T09:31:24.912357+0000","last_unstale":"2024-05-27T09:35:14.373078+0000","last_undegraded":"2024-05-27T09:35:14.373078+0000","last_fullsized":"2024-05-27T09:35:14.373078+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:27:23.334041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:30.093913+0000","last_change":"2024-05-27T09:31:20.881652+0000","last_active":"2024-05-27T09:35:30.093913+0000","last_peered":"2024-05-27T09:35:30.093913+0000","last_clean":"2024-05-27T09:35:30.093913+0000","last_became_active":"2024-05-27T09:31:20.880757+0000","last_became_peered":"2024-05-27T09:31:20.880757+0000","last_unstale":"2024-05-27T09:35:30.093913+0000","last_undegraded":"2024-05-27T09:35:30.093913+0000","last_fullsized":"2024-05-27T09:35:30.093913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:01.433755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171081+0000","last_change":"2024-05-27T09:31:24.908010+0000","last_active":"2024-05-27T09:35:14.171081+0000","last_peered":"2024-05-27T09:35:14.171081+0000","last_clean":"2024-05-27T09:35:14.171081+0000","last_became_active":"2024-05-27T09:31:24.907638+0000","last_became_peered":"2024-05-27T09:31:24.907638+0000","last_unstale":"2024-05-27T09:35:14.171081+0000","last_undegraded":"2024-05-27T09:35:14.171081+0000","last_fullsized":"2024-05-27T09:35:14.171081+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:02:49.207709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171465+0000","last_change":"2024-05-27T09:31:20.860556+0000","last_active":"2024-05-27T09:35:14.171465+0000","last_peered":"2024-05-27T09:35:14.171465+0000","last_clean":"2024-05-27T09:35:14.171465+0000","last_became_active":"2024-05-27T09:31:20.860172+0000","last_became_peered":"2024-05-27T09:31:20.860172+0000","last_unstale":"2024-05-27T09:35:14.171465+0000","last_undegraded":"2024-05-27T09:35:14.171465+0000","last_fullsized":"2024-05-27T09:35:14.171465+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:08:21.545681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043114+0000","last_change":"2024-05-27T09:31:24.910111+0000","last_active":"2024-05-27T09:35:14.043114+0000","last_peered":"2024-05-27T09:35:14.043114+0000","last_clean":"2024-05-27T09:35:14.043114+0000","last_became_active":"2024-05-27T09:31:24.908985+0000","last_became_peered":"2024-05-27T09:31:24.908985+0000","last_unstale":"2024-05-27T09:35:14.043114+0000","last_undegraded":"2024-05-27T09:35:14.043114+0000","last_fullsized":"2024-05-27T09:35:14.043114+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:53:29.645792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170222+0000","last_change":"2024-05-27T09:31:20.879993+0000","last_active":"2024-05-27T09:35:14.170222+0000","last_peered":"2024-05-27T09:35:14.170222+0000","last_clean":"2024-05-27T09:35:14.170222+0000","last_became_active":"2024-05-27T09:31:20.878802+0000","last_became_peered":"2024-05-27T09:31:20.878802+0000","last_unstale":"2024-05-27T09:35:14.170222+0000","last_undegraded":"2024-05-27T09:35:14.170222+0000","last_fullsized":"2024-05-27T09:35:14.170222+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:28:15.555612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372970+0000","last_change":"2024-05-27T09:31:24.903533+0000","last_active":"2024-05-27T09:35:14.372970+0000","last_peered":"2024-05-27T09:35:14.372970+0000","last_clean":"2024-05-27T09:35:14.372970+0000","last_became_active":"2024-05-27T09:31:24.903166+0000","last_became_peered":"2024-05-27T09:31:24.903166+0000","last_unstale":"2024-05-27T09:35:14.372970+0000","last_undegraded":"2024-05-27T09:35:14.372970+0000","last_fullsized":"2024-05-27T09:35:14.372970+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:53:52.539470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839887+0000","last_change":"2024-05-27T09:31:57.878426+0000","last_active":"2024-05-27T09:35:14.839887+0000","last_peered":"2024-05-27T09:35:14.839887+0000","last_clean":"2024-05-27T09:35:14.839887+0000","last_became_active":"2024-05-27T09:31:57.869146+0000","last_became_peered":"2024-05-27T09:31:57.869146+0000","last_unstale":"2024-05-27T09:35:14.839887+0000","last_undegraded":"2024-05-27T09:35:14.839887+0000","last_fullsized":"2024-05-27T09:35:14.839887+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:34:51.926092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129913+0000","last_change":"2024-05-27T09:31:24.897426+0000","last_active":"2024-05-27T09:35:14.129913+0000","last_peered":"2024-05-27T09:35:14.129913+0000","last_clean":"2024-05-27T09:35:14.129913+0000","last_became_active":"2024-05-27T09:31:24.897080+0000","last_became_peered":"2024-05-27T09:31:24.897080+0000","last_unstale":"2024-05-27T09:35:14.129913+0000","last_undegraded":"2024-05-27T09:35:14.129913+0000","last_fullsized":"2024-05-27T09:35:14.129913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:15:42.428391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'8","reported_seq":280,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:43.175247+0000","last_change":"2024-05-27T09:31:20.846681+0000","last_active":"2024-05-27T09:35:43.175247+0000","last_peered":"2024-05-27T09:35:43.175247+0000","last_clean":"2024-05-27T09:35:43.175247+0000","last_became_active":"2024-05-27T09:31:20.846302+0000","last_became_peered":"2024-05-27T09:31:20.846302+0000","last_unstale":"2024-05-27T09:35:43.175247+0000","last_undegraded":"2024-05-27T09:35:43.175247+0000","last_fullsized":"2024-05-27T09:35:43.175247+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:19.609612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":243,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169711+0000","last_change":"2024-05-27T09:31:24.907209+0000","last_active":"2024-05-27T09:35:14.169711+0000","last_peered":"2024-05-27T09:35:14.169711+0000","last_clean":"2024-05-27T09:35:14.169711+0000","last_became_active":"2024-05-27T09:31:24.906752+0000","last_became_peered":"2024-05-27T09:31:24.906752+0000","last_unstale":"2024-05-27T09:35:14.169711+0000","last_undegraded":"2024-05-27T09:35:14.169711+0000","last_fullsized":"2024-05-27T09:35:14.169711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:53.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126569+0000","last_change":"2024-05-27T09:31:20.879778+0000","last_active":"2024-05-27T09:35:14.126569+0000","last_peered":"2024-05-27T09:35:14.126569+0000","last_clean":"2024-05-27T09:35:14.126569+0000","last_became_active":"2024-05-27T09:31:20.879259+0000","last_became_peered":"2024-05-27T09:31:20.879259+0000","last_unstale":"2024-05-27T09:35:14.126569+0000","last_undegraded":"2024-05-27T09:35:14.126569+0000","last_fullsized":"2024-05-27T09:35:14.126569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:20:07.972653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129702+0000","last_change":"2024-05-27T09:31:58.195516+0000","last_active":"2024-05-27T09:35:14.129702+0000","last_peered":"2024-05-27T09:35:14.129702+0000","last_clean":"2024-05-27T09:35:14.129702+0000","last_became_active":"2024-05-27T09:31:58.195118+0000","last_became_peered":"2024-05-27T09:31:58.195118+0000","last_unstale":"2024-05-27T09:35:14.129702+0000","last_undegraded":"2024-05-27T09:35:14.129702+0000","last_fullsized":"2024-05-27T09:35:14.129702+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:11:23.547605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241846+0000","last_change":"2024-05-27T09:31:20.863152+0000","last_active":"2024-05-27T09:35:14.241846+0000","last_peered":"2024-05-27T09:35:14.241846+0000","last_clean":"2024-05-27T09:35:14.241846+0000","last_became_active":"2024-05-27T09:31:20.862708+0000","last_became_peered":"2024-05-27T09:31:20.862708+0000","last_unstale":"2024-05-27T09:35:14.241846+0000","last_undegraded":"2024-05-27T09:35:14.241846+0000","last_fullsized":"2024-05-27T09:35:14.241846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:21.645773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170544+0000","last_change":"2024-05-27T09:31:24.893988+0000","last_active":"2024-05-27T09:35:14.170544+0000","last_peered":"2024-05-27T09:35:14.170544+0000","last_clean":"2024-05-27T09:35:14.170544+0000","last_became_active":"2024-05-27T09:31:24.893504+0000","last_became_peered":"2024-05-27T09:31:24.893504+0000","last_unstale":"2024-05-27T09:35:14.170544+0000","last_undegraded":"2024-05-27T09:35:14.170544+0000","last_fullsized":"2024-05-27T09:35:14.170544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:47.439452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129578+0000","last_change":"2024-05-27T09:31:20.845555+0000","last_active":"2024-05-27T09:35:14.129578+0000","last_peered":"2024-05-27T09:35:14.129578+0000","last_clean":"2024-05-27T09:35:14.129578+0000","last_became_active":"2024-05-27T09:31:20.845139+0000","last_became_peered":"2024-05-27T09:31:20.845139+0000","last_unstale":"2024-05-27T09:35:14.129578+0000","last_undegraded":"2024-05-27T09:35:14.129578+0000","last_fullsized":"2024-05-27T09:35:14.129578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:03:25.917845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.049161+0000","last_change":"2024-05-27T09:31:24.916142+0000","last_active":"2024-05-27T09:35:14.049161+0000","last_peered":"2024-05-27T09:35:14.049161+0000","last_clean":"2024-05-27T09:35:14.049161+0000","last_became_active":"2024-05-27T09:31:24.915709+0000","last_became_peered":"2024-05-27T09:31:24.915709+0000","last_unstale":"2024-05-27T09:35:14.049161+0000","last_undegraded":"2024-05-27T09:35:14.049161+0000","last_fullsized":"2024-05-27T09:35:14.049161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:38.029648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'5","reported_seq":272,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:43.174727+0000","last_change":"2024-05-27T09:31:20.859240+0000","last_active":"2024-05-27T09:35:43.174727+0000","last_peered":"2024-05-27T09:35:43.174727+0000","last_clean":"2024-05-27T09:35:43.174727+0000","last_became_active":"2024-05-27T09:31:20.858798+0000","last_became_peered":"2024-05-27T09:31:20.858798+0000","last_unstale":"2024-05-27T09:35:43.174727+0000","last_undegraded":"2024-05-27T09:35:43.174727+0000","last_fullsized":"2024-05-27T09:35:43.174727+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:42.280165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":238,"num_read_kb":238,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373425+0000","last_change":"2024-05-27T09:31:24.908818+0000","last_active":"2024-05-27T09:35:14.373425+0000","last_peered":"2024-05-27T09:35:14.373425+0000","last_clean":"2024-05-27T09:35:14.373425+0000","last_became_active":"2024-05-27T09:31:24.908410+0000","last_became_peered":"2024-05-27T09:31:24.908410+0000","last_unstale":"2024-05-27T09:35:14.373425+0000","last_undegraded":"2024-05-27T09:35:14.373425+0000","last_fullsized":"2024-05-27T09:35:14.373425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:05:40.326232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353642+0000","last_change":"2024-05-27T09:31:20.848632+0000","last_active":"2024-05-27T09:35:14.353642+0000","last_peered":"2024-05-27T09:35:14.353642+0000","last_clean":"2024-05-27T09:35:14.353642+0000","last_became_active":"2024-05-27T09:31:20.847558+0000","last_became_peered":"2024-05-27T09:31:20.847558+0000","last_unstale":"2024-05-27T09:35:14.353642+0000","last_undegraded":"2024-05-27T09:35:14.353642+0000","last_fullsized":"2024-05-27T09:35:14.353642+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:35:53.930565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043056+0000","last_change":"2024-05-27T09:31:24.901513+0000","last_active":"2024-05-27T09:35:14.043056+0000","last_peered":"2024-05-27T09:35:14.043056+0000","last_clean":"2024-05-27T09:35:14.043056+0000","last_became_active":"2024-05-27T09:31:24.895924+0000","last_became_peered":"2024-05-27T09:31:24.895924+0000","last_unstale":"2024-05-27T09:35:14.043056+0000","last_undegraded":"2024-05-27T09:35:14.043056+0000","last_fullsized":"2024-05-27T09:35:14.043056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:27:39.939098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.130061+0000","last_change":"2024-05-27T09:31:20.855195+0000","last_active":"2024-05-27T09:35:14.130061+0000","last_peered":"2024-05-27T09:35:14.130061+0000","last_clean":"2024-05-27T09:35:14.130061+0000","last_became_active":"2024-05-27T09:31:20.854910+0000","last_became_peered":"2024-05-27T09:31:20.854910+0000","last_unstale":"2024-05-27T09:35:14.130061+0000","last_undegraded":"2024-05-27T09:35:14.130061+0000","last_fullsized":"2024-05-27T09:35:14.130061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:46:02.795103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373077+0000","last_change":"2024-05-27T09:31:24.895887+0000","last_active":"2024-05-27T09:35:14.373077+0000","last_peered":"2024-05-27T09:35:14.373077+0000","last_clean":"2024-05-27T09:35:14.373077+0000","last_became_active":"2024-05-27T09:31:24.894622+0000","last_became_peered":"2024-05-27T09:31:24.894622+0000","last_unstale":"2024-05-27T09:35:14.373077+0000","last_undegraded":"2024-05-27T09:35:14.373077+0000","last_fullsized":"2024-05-27T09:35:14.373077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:32.833946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'38","reported_seq":556,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:44.176262+0000","last_change":"2024-05-27T09:31:20.846063+0000","last_active":"2024-05-27T09:35:44.176262+0000","last_peered":"2024-05-27T09:35:44.176262+0000","last_clean":"2024-05-27T09:35:44.176262+0000","last_became_active":"2024-05-27T09:31:20.845401+0000","last_became_peered":"2024-05-27T09:31:20.845401+0000","last_unstale":"2024-05-27T09:35:44.176262+0000","last_undegraded":"2024-05-27T09:35:44.176262+0000","last_fullsized":"2024-05-27T09:35:44.176262+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:21:38.460183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69968,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":489,"num_read_kb":314,"num_write":39,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170004+0000","last_change":"2024-05-27T09:31:24.892748+0000","last_active":"2024-05-27T09:35:14.170004+0000","last_peered":"2024-05-27T09:35:14.170004+0000","last_clean":"2024-05-27T09:35:14.170004+0000","last_became_active":"2024-05-27T09:31:24.892394+0000","last_became_peered":"2024-05-27T09:31:24.892394+0000","last_unstale":"2024-05-27T09:35:14.170004+0000","last_undegraded":"2024-05-27T09:35:14.170004+0000","last_fullsized":"2024-05-27T09:35:14.170004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:39:31.381072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372830+0000","last_change":"2024-05-27T09:31:20.860789+0000","last_active":"2024-05-27T09:35:14.372830+0000","last_peered":"2024-05-27T09:35:14.372830+0000","last_clean":"2024-05-27T09:35:14.372830+0000","last_became_active":"2024-05-27T09:31:20.860369+0000","last_became_peered":"2024-05-27T09:31:20.860369+0000","last_unstale":"2024-05-27T09:35:14.372830+0000","last_undegraded":"2024-05-27T09:35:14.372830+0000","last_fullsized":"2024-05-27T09:35:14.372830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:05:35.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241584+0000","last_change":"2024-05-27T09:31:24.918239+0000","last_active":"2024-05-27T09:35:14.241584+0000","last_peered":"2024-05-27T09:35:14.241584+0000","last_clean":"2024-05-27T09:35:14.241584+0000","last_became_active":"2024-05-27T09:31:24.917750+0000","last_became_peered":"2024-05-27T09:31:24.917750+0000","last_unstale":"2024-05-27T09:35:14.241584+0000","last_undegraded":"2024-05-27T09:35:14.241584+0000","last_fullsized":"2024-05-27T09:35:14.241584+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:07.162994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129596+0000","last_change":"2024-05-27T09:31:20.865308+0000","last_active":"2024-05-27T09:35:14.129596+0000","last_peered":"2024-05-27T09:35:14.129596+0000","last_clean":"2024-05-27T09:35:14.129596+0000","last_became_active":"2024-05-27T09:31:20.864920+0000","last_became_peered":"2024-05-27T09:31:20.864920+0000","last_unstale":"2024-05-27T09:35:14.129596+0000","last_undegraded":"2024-05-27T09:35:14.129596+0000","last_fullsized":"2024-05-27T09:35:14.129596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:36.773322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.902533+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.902136+0000","last_became_peered":"2024-05-27T09:31:24.902136+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:49:32.203577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'8","reported_seq":281,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:43.309835+0000","last_change":"2024-05-27T09:31:20.846060+0000","last_active":"2024-05-27T09:35:43.309835+0000","last_peered":"2024-05-27T09:35:43.309835+0000","last_clean":"2024-05-27T09:35:43.309835+0000","last_became_active":"2024-05-27T09:31:20.845632+0000","last_became_peered":"2024-05-27T09:31:20.845632+0000","last_unstale":"2024-05-27T09:35:43.309835+0000","last_undegraded":"2024-05-27T09:35:43.309835+0000","last_fullsized":"2024-05-27T09:35:43.309835+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:18:55.166579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2208,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353158+0000","last_change":"2024-05-27T09:31:24.895866+0000","last_active":"2024-05-27T09:35:14.353158+0000","last_peered":"2024-05-27T09:35:14.353158+0000","last_clean":"2024-05-27T09:35:14.353158+0000","last_became_active":"2024-05-27T09:31:24.895413+0000","last_became_peered":"2024-05-27T09:31:24.895413+0000","last_unstale":"2024-05-27T09:35:14.353158+0000","last_undegraded":"2024-05-27T09:35:14.353158+0000","last_fullsized":"2024-05-27T09:35:14.353158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:01:30.075710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":45,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169708+0000","last_change":"2024-05-27T09:31:20.879086+0000","last_active":"2024-05-27T09:35:14.169708+0000","last_peered":"2024-05-27T09:35:14.169708+0000","last_clean":"2024-05-27T09:35:14.169708+0000","last_became_active":"2024-05-27T09:31:20.878331+0000","last_became_peered":"2024-05-27T09:31:20.878331+0000","last_unstale":"2024-05-27T09:35:14.169708+0000","last_undegraded":"2024-05-27T09:35:14.169708+0000","last_fullsized":"2024-05-27T09:35:14.169708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:31.169812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129641+0000","last_change":"2024-05-27T09:31:24.895069+0000","last_active":"2024-05-27T09:35:14.129641+0000","last_peered":"2024-05-27T09:35:14.129641+0000","last_clean":"2024-05-27T09:35:14.129641+0000","last_became_active":"2024-05-27T09:31:24.894626+0000","last_became_peered":"2024-05-27T09:31:24.894626+0000","last_unstale":"2024-05-27T09:35:14.129641+0000","last_undegraded":"2024-05-27T09:35:14.129641+0000","last_fullsized":"2024-05-27T09:35:14.129641+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:21:22.725268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170168+0000","last_change":"2024-05-27T09:31:20.877416+0000","last_active":"2024-05-27T09:35:14.170168+0000","last_peered":"2024-05-27T09:35:14.170168+0000","last_clean":"2024-05-27T09:35:14.170168+0000","last_became_active":"2024-05-27T09:31:20.876093+0000","last_became_peered":"2024-05-27T09:31:20.876093+0000","last_unstale":"2024-05-27T09:35:14.170168+0000","last_undegraded":"2024-05-27T09:35:14.170168+0000","last_fullsized":"2024-05-27T09:35:14.170168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:34.490362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840141+0000","last_change":"2024-05-27T09:31:24.898430+0000","last_active":"2024-05-27T09:35:14.840141+0000","last_peered":"2024-05-27T09:35:14.840141+0000","last_clean":"2024-05-27T09:35:14.840141+0000","last_became_active":"2024-05-27T09:31:24.897643+0000","last_became_peered":"2024-05-27T09:31:24.897643+0000","last_unstale":"2024-05-27T09:35:14.840141+0000","last_undegraded":"2024-05-27T09:35:14.840141+0000","last_fullsized":"2024-05-27T09:35:14.840141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:44.485561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.046071+0000","last_change":"2024-05-27T09:31:20.858253+0000","last_active":"2024-05-27T09:35:14.046071+0000","last_peered":"2024-05-27T09:35:14.046071+0000","last_clean":"2024-05-27T09:35:14.046071+0000","last_became_active":"2024-05-27T09:31:20.857736+0000","last_became_peered":"2024-05-27T09:31:20.857736+0000","last_unstale":"2024-05-27T09:35:14.046071+0000","last_undegraded":"2024-05-27T09:35:14.046071+0000","last_fullsized":"2024-05-27T09:35:14.046071+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:53:48.301763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126982+0000","last_change":"2024-05-27T09:31:24.912788+0000","last_active":"2024-05-27T09:35:14.126982+0000","last_peered":"2024-05-27T09:35:14.126982+0000","last_clean":"2024-05-27T09:35:14.126982+0000","last_became_active":"2024-05-27T09:31:24.912366+0000","last_became_peered":"2024-05-27T09:31:24.912366+0000","last_unstale":"2024-05-27T09:35:14.126982+0000","last_undegraded":"2024-05-27T09:35:14.126982+0000","last_fullsized":"2024-05-27T09:35:14.126982+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:55:03.397578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353426+0000","last_change":"2024-05-27T09:31:20.862188+0000","last_active":"2024-05-27T09:35:14.353426+0000","last_peered":"2024-05-27T09:35:14.353426+0000","last_clean":"2024-05-27T09:35:14.353426+0000","last_became_active":"2024-05-27T09:31:20.861791+0000","last_became_peered":"2024-05-27T09:31:20.861791+0000","last_unstale":"2024-05-27T09:35:14.353426+0000","last_undegraded":"2024-05-27T09:35:14.353426+0000","last_fullsized":"2024-05-27T09:35:14.353426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:16:09.127223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.126717+0000","last_change":"2024-05-27T09:31:24.898460+0000","last_active":"2024-05-27T09:35:14.126717+0000","last_peered":"2024-05-27T09:35:14.126717+0000","last_clean":"2024-05-27T09:35:14.126717+0000","last_became_active":"2024-05-27T09:31:24.896787+0000","last_became_peered":"2024-05-27T09:31:24.896787+0000","last_unstale":"2024-05-27T09:35:14.126717+0000","last_undegraded":"2024-05-27T09:35:14.126717+0000","last_fullsized":"2024-05-27T09:35:14.126717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:10.199227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.130015+0000","last_change":"2024-05-27T09:31:24.896323+0000","last_active":"2024-05-27T09:35:14.130015+0000","last_peered":"2024-05-27T09:35:14.130015+0000","last_clean":"2024-05-27T09:35:14.130015+0000","last_became_active":"2024-05-27T09:31:24.895919+0000","last_became_peered":"2024-05-27T09:31:24.895919+0000","last_unstale":"2024-05-27T09:35:14.130015+0000","last_undegraded":"2024-05-27T09:35:14.130015+0000","last_fullsized":"2024-05-27T09:35:14.130015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:04.883166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":274,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:43.309116+0000","last_change":"2024-05-27T09:31:20.862126+0000","last_active":"2024-05-27T09:35:43.309116+0000","last_peered":"2024-05-27T09:35:43.309116+0000","last_clean":"2024-05-27T09:35:43.309116+0000","last_became_active":"2024-05-27T09:31:20.858964+0000","last_became_peered":"2024-05-27T09:31:20.858964+0000","last_unstale":"2024-05-27T09:35:43.309116+0000","last_undegraded":"2024-05-27T09:35:43.309116+0000","last_fullsized":"2024-05-27T09:35:43.309116+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:51:26.838559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":238,"num_read_kb":238,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171170+0000","last_change":"2024-05-27T09:31:24.905549+0000","last_active":"2024-05-27T09:35:14.171170+0000","last_peered":"2024-05-27T09:35:14.171170+0000","last_clean":"2024-05-27T09:35:14.171170+0000","last_became_active":"2024-05-27T09:31:24.905212+0000","last_became_peered":"2024-05-27T09:31:24.905212+0000","last_unstale":"2024-05-27T09:35:14.171170+0000","last_undegraded":"2024-05-27T09:35:14.171170+0000","last_fullsized":"2024-05-27T09:35:14.171170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:18.267807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043354+0000","last_change":"2024-05-27T09:31:20.848527+0000","last_active":"2024-05-27T09:35:14.043354+0000","last_peered":"2024-05-27T09:35:14.043354+0000","last_clean":"2024-05-27T09:35:14.043354+0000","last_became_active":"2024-05-27T09:31:20.847779+0000","last_became_peered":"2024-05-27T09:31:20.847779+0000","last_unstale":"2024-05-27T09:35:14.043354+0000","last_undegraded":"2024-05-27T09:35:14.043354+0000","last_fullsized":"2024-05-27T09:35:14.043354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:15:01.131306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"82'99","reported_seq":133,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:16.190224+0000","last_change":"2024-05-27T09:29:46.120380+0000","last_active":"2024-05-27T09:35:16.190224+0000","last_peered":"2024-05-27T09:35:16.190224+0000","last_clean":"2024-05-27T09:35:16.190224+0000","last_became_active":"2024-05-27T09:29:46.113658+0000","last_became_peered":"2024-05-27T09:29:46.113658+0000","last_unstale":"2024-05-27T09:35:16.190224+0000","last_undegraded":"2024-05-27T09:35:16.190224+0000","last_fullsized":"2024-05-27T09:35:16.190224+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:25:47.244521+0000","last_clean_scrub_stamp":"2024-05-27T09:25:47.244521+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:28:40.424824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241564+0000","last_change":"2024-05-27T09:31:24.900736+0000","last_active":"2024-05-27T09:35:14.241564+0000","last_peered":"2024-05-27T09:35:14.241564+0000","last_clean":"2024-05-27T09:35:14.241564+0000","last_became_active":"2024-05-27T09:31:24.900354+0000","last_became_peered":"2024-05-27T09:31:24.900354+0000","last_unstale":"2024-05-27T09:35:14.241564+0000","last_undegraded":"2024-05-27T09:35:14.241564+0000","last_fullsized":"2024-05-27T09:35:14.241564+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:39:33.171977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.839843+0000","last_change":"2024-05-27T09:31:57.873603+0000","last_active":"2024-05-27T09:35:14.839843+0000","last_peered":"2024-05-27T09:35:14.839843+0000","last_clean":"2024-05-27T09:35:14.839843+0000","last_became_active":"2024-05-27T09:31:57.868616+0000","last_became_peered":"2024-05-27T09:31:57.868616+0000","last_unstale":"2024-05-27T09:35:14.839843+0000","last_undegraded":"2024-05-27T09:35:14.839843+0000","last_fullsized":"2024-05-27T09:35:14.839843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:02:20.112667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170149+0000","last_change":"2024-05-27T09:31:20.878557+0000","last_active":"2024-05-27T09:35:14.170149+0000","last_peered":"2024-05-27T09:35:14.170149+0000","last_clean":"2024-05-27T09:35:14.170149+0000","last_became_active":"2024-05-27T09:31:20.877969+0000","last_became_peered":"2024-05-27T09:31:20.877969+0000","last_unstale":"2024-05-27T09:35:14.170149+0000","last_undegraded":"2024-05-27T09:35:14.170149+0000","last_fullsized":"2024-05-27T09:35:14.170149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:46.859645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242259+0000","last_change":"2024-05-27T09:31:24.907906+0000","last_active":"2024-05-27T09:35:14.242259+0000","last_peered":"2024-05-27T09:35:14.242259+0000","last_clean":"2024-05-27T09:35:14.242259+0000","last_became_active":"2024-05-27T09:31:24.907473+0000","last_became_peered":"2024-05-27T09:31:24.907473+0000","last_unstale":"2024-05-27T09:35:14.242259+0000","last_undegraded":"2024-05-27T09:35:14.242259+0000","last_fullsized":"2024-05-27T09:35:14.242259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:11:38.536919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'102","reported_seq":224,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:35.035590+0000","last_change":"2024-05-27T09:31:20.881250+0000","last_active":"2024-05-27T09:35:35.035590+0000","last_peered":"2024-05-27T09:35:35.035590+0000","last_clean":"2024-05-27T09:35:35.035590+0000","last_became_active":"2024-05-27T09:31:20.880550+0000","last_became_peered":"2024-05-27T09:31:20.880550+0000","last_unstale":"2024-05-27T09:35:35.035590+0000","last_undegraded":"2024-05-27T09:35:35.035590+0000","last_fullsized":"2024-05-27T09:35:35.035590+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:14.783260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":187,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171464+0000","last_change":"2024-05-27T09:31:24.894554+0000","last_active":"2024-05-27T09:35:14.171464+0000","last_peered":"2024-05-27T09:35:14.171464+0000","last_clean":"2024-05-27T09:35:14.171464+0000","last_became_active":"2024-05-27T09:31:24.893800+0000","last_became_peered":"2024-05-27T09:31:24.893800+0000","last_unstale":"2024-05-27T09:35:14.171464+0000","last_undegraded":"2024-05-27T09:35:14.171464+0000","last_fullsized":"2024-05-27T09:35:14.171464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:28:13.368892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043654+0000","last_change":"2024-05-27T09:31:21.054476+0000","last_active":"2024-05-27T09:35:14.043654+0000","last_peered":"2024-05-27T09:35:14.043654+0000","last_clean":"2024-05-27T09:35:14.043654+0000","last_became_active":"2024-05-27T09:31:21.054091+0000","last_became_peered":"2024-05-27T09:31:21.054091+0000","last_unstale":"2024-05-27T09:35:14.043654+0000","last_undegraded":"2024-05-27T09:35:14.043654+0000","last_fullsized":"2024-05-27T09:35:14.043654+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:23:15.909230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043220+0000","last_change":"2024-05-27T09:31:24.913157+0000","last_active":"2024-05-27T09:35:14.043220+0000","last_peered":"2024-05-27T09:35:14.043220+0000","last_clean":"2024-05-27T09:35:14.043220+0000","last_became_active":"2024-05-27T09:31:24.912793+0000","last_became_peered":"2024-05-27T09:31:24.912793+0000","last_unstale":"2024-05-27T09:35:14.043220+0000","last_undegraded":"2024-05-27T09:35:14.043220+0000","last_fullsized":"2024-05-27T09:35:14.043220+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:17:36.451069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372758+0000","last_change":"2024-05-27T09:31:20.861349+0000","last_active":"2024-05-27T09:35:14.372758+0000","last_peered":"2024-05-27T09:35:14.372758+0000","last_clean":"2024-05-27T09:35:14.372758+0000","last_became_active":"2024-05-27T09:31:20.860896+0000","last_became_peered":"2024-05-27T09:31:20.860896+0000","last_unstale":"2024-05-27T09:35:14.372758+0000","last_undegraded":"2024-05-27T09:35:14.372758+0000","last_fullsized":"2024-05-27T09:35:14.372758+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:00.772910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169761+0000","last_change":"2024-05-27T09:31:24.893240+0000","last_active":"2024-05-27T09:35:14.169761+0000","last_peered":"2024-05-27T09:35:14.169761+0000","last_clean":"2024-05-27T09:35:14.169761+0000","last_became_active":"2024-05-27T09:31:24.892169+0000","last_became_peered":"2024-05-27T09:31:24.892169+0000","last_unstale":"2024-05-27T09:35:14.169761+0000","last_undegraded":"2024-05-27T09:35:14.169761+0000","last_fullsized":"2024-05-27T09:35:14.169761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:26:59.660725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170750+0000","last_change":"2024-05-27T09:31:20.864307+0000","last_active":"2024-05-27T09:35:14.170750+0000","last_peered":"2024-05-27T09:35:14.170750+0000","last_clean":"2024-05-27T09:35:14.170750+0000","last_became_active":"2024-05-27T09:31:20.863913+0000","last_became_peered":"2024-05-27T09:31:20.863913+0000","last_unstale":"2024-05-27T09:35:14.170750+0000","last_undegraded":"2024-05-27T09:35:14.170750+0000","last_fullsized":"2024-05-27T09:35:14.170750+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:58:57.656731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840040+0000","last_change":"2024-05-27T09:31:24.905728+0000","last_active":"2024-05-27T09:35:14.840040+0000","last_peered":"2024-05-27T09:35:14.840040+0000","last_clean":"2024-05-27T09:35:14.840040+0000","last_became_active":"2024-05-27T09:31:24.905365+0000","last_became_peered":"2024-05-27T09:31:24.905365+0000","last_unstale":"2024-05-27T09:35:14.840040+0000","last_undegraded":"2024-05-27T09:35:14.840040+0000","last_fullsized":"2024-05-27T09:35:14.840040+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:59:05.013037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'2","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.766497+0000","last_change":"2024-05-27T09:31:20.878478+0000","last_active":"2024-05-27T09:35:14.766497+0000","last_peered":"2024-05-27T09:35:14.766497+0000","last_clean":"2024-05-27T09:35:14.766497+0000","last_became_active":"2024-05-27T09:31:20.877952+0000","last_became_peered":"2024-05-27T09:31:20.877952+0000","last_unstale":"2024-05-27T09:35:14.766497+0000","last_undegraded":"2024-05-27T09:35:14.766497+0000","last_fullsized":"2024-05-27T09:35:14.766497+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:16:27.580641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171301+0000","last_change":"2024-05-27T09:31:24.904967+0000","last_active":"2024-05-27T09:35:14.171301+0000","last_peered":"2024-05-27T09:35:14.171301+0000","last_clean":"2024-05-27T09:35:14.171301+0000","last_became_active":"2024-05-27T09:31:24.904542+0000","last_became_peered":"2024-05-27T09:31:24.904542+0000","last_unstale":"2024-05-27T09:35:14.171301+0000","last_undegraded":"2024-05-27T09:35:14.171301+0000","last_fullsized":"2024-05-27T09:35:14.171301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:01:05.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.169841+0000","last_change":"2024-05-27T09:31:57.875713+0000","last_active":"2024-05-27T09:35:14.169841+0000","last_peered":"2024-05-27T09:35:14.169841+0000","last_clean":"2024-05-27T09:35:14.169841+0000","last_became_active":"2024-05-27T09:31:57.870244+0000","last_became_peered":"2024-05-27T09:31:57.870244+0000","last_unstale":"2024-05-27T09:35:14.169841+0000","last_undegraded":"2024-05-27T09:35:14.169841+0000","last_fullsized":"2024-05-27T09:35:14.169841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:49.272418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353520+0000","last_change":"2024-05-27T09:31:24.892655+0000","last_active":"2024-05-27T09:35:14.353520+0000","last_peered":"2024-05-27T09:35:14.353520+0000","last_clean":"2024-05-27T09:35:14.353520+0000","last_became_active":"2024-05-27T09:31:24.892112+0000","last_became_peered":"2024-05-27T09:31:24.892112+0000","last_unstale":"2024-05-27T09:35:14.353520+0000","last_undegraded":"2024-05-27T09:35:14.353520+0000","last_fullsized":"2024-05-27T09:35:14.353520+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.604118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.043417+0000","last_change":"2024-05-27T09:31:57.875213+0000","last_active":"2024-05-27T09:35:14.043417+0000","last_peered":"2024-05-27T09:35:14.043417+0000","last_clean":"2024-05-27T09:35:14.043417+0000","last_became_active":"2024-05-27T09:31:57.869025+0000","last_became_peered":"2024-05-27T09:31:57.869025+0000","last_unstale":"2024-05-27T09:35:14.043417+0000","last_undegraded":"2024-05-27T09:35:14.043417+0000","last_fullsized":"2024-05-27T09:35:14.043417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:51.433347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353725+0000","last_change":"2024-05-27T09:31:24.899130+0000","last_active":"2024-05-27T09:35:14.353725+0000","last_peered":"2024-05-27T09:35:14.353725+0000","last_clean":"2024-05-27T09:35:14.353725+0000","last_became_active":"2024-05-27T09:31:24.898722+0000","last_became_peered":"2024-05-27T09:31:24.898722+0000","last_unstale":"2024-05-27T09:35:14.353725+0000","last_undegraded":"2024-05-27T09:35:14.353725+0000","last_fullsized":"2024-05-27T09:35:14.353725+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:51.247275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'65","reported_seq":125,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:34.462413+0000","last_change":"2024-05-27T09:31:20.863757+0000","last_active":"2024-05-27T09:35:34.462413+0000","last_peered":"2024-05-27T09:35:34.462413+0000","last_clean":"2024-05-27T09:35:34.462413+0000","last_became_active":"2024-05-27T09:31:20.863465+0000","last_became_peered":"2024-05-27T09:31:20.863465+0000","last_unstale":"2024-05-27T09:35:34.462413+0000","last_undegraded":"2024-05-27T09:35:34.462413+0000","last_fullsized":"2024-05-27T09:35:34.462413+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T20:21:45.741822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353380+0000","last_change":"2024-05-27T09:31:24.913749+0000","last_active":"2024-05-27T09:35:14.353380+0000","last_peered":"2024-05-27T09:35:14.353380+0000","last_clean":"2024-05-27T09:35:14.353380+0000","last_became_active":"2024-05-27T09:31:24.913384+0000","last_became_peered":"2024-05-27T09:31:24.913384+0000","last_unstale":"2024-05-27T09:35:14.353380+0000","last_undegraded":"2024-05-27T09:35:14.353380+0000","last_fullsized":"2024-05-27T09:35:14.353380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:24:29.826096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242122+0000","last_change":"2024-05-27T09:31:20.865844+0000","last_active":"2024-05-27T09:35:14.242122+0000","last_peered":"2024-05-27T09:35:14.242122+0000","last_clean":"2024-05-27T09:35:14.242122+0000","last_became_active":"2024-05-27T09:31:20.865364+0000","last_became_peered":"2024-05-27T09:31:20.865364+0000","last_unstale":"2024-05-27T09:35:14.242122+0000","last_undegraded":"2024-05-27T09:35:14.242122+0000","last_fullsized":"2024-05-27T09:35:14.242122+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:19.818077+0000","last_clean_scrub_stamp":"2024-05-27T09:31:19.818077+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:11:54.451294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.353821+0000","last_change":"2024-05-27T09:31:24.890326+0000","last_active":"2024-05-27T09:35:14.353821+0000","last_peered":"2024-05-27T09:35:14.353821+0000","last_clean":"2024-05-27T09:35:14.353821+0000","last_became_active":"2024-05-27T09:31:24.889789+0000","last_became_peered":"2024-05-27T09:31:24.889789+0000","last_unstale":"2024-05-27T09:35:14.353821+0000","last_undegraded":"2024-05-27T09:35:14.353821+0000","last_fullsized":"2024-05-27T09:35:14.353821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:34:04.252231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840061+0000","last_change":"2024-05-27T09:31:24.896730+0000","last_active":"2024-05-27T09:35:14.840061+0000","last_peered":"2024-05-27T09:35:14.840061+0000","last_clean":"2024-05-27T09:35:14.840061+0000","last_became_active":"2024-05-27T09:31:24.896365+0000","last_became_peered":"2024-05-27T09:31:24.896365+0000","last_unstale":"2024-05-27T09:35:14.840061+0000","last_undegraded":"2024-05-27T09:35:14.840061+0000","last_fullsized":"2024-05-27T09:35:14.840061+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:37.443788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372900+0000","last_change":"2024-05-27T09:31:24.899215+0000","last_active":"2024-05-27T09:35:14.372900+0000","last_peered":"2024-05-27T09:35:14.372900+0000","last_clean":"2024-05-27T09:35:14.372900+0000","last_became_active":"2024-05-27T09:31:24.898733+0000","last_became_peered":"2024-05-27T09:31:24.898733+0000","last_unstale":"2024-05-27T09:35:14.372900+0000","last_undegraded":"2024-05-27T09:35:14.372900+0000","last_fullsized":"2024-05-27T09:35:14.372900+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:10.085989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372769+0000","last_change":"2024-05-27T09:31:24.914922+0000","last_active":"2024-05-27T09:35:14.372769+0000","last_peered":"2024-05-27T09:35:14.372769+0000","last_clean":"2024-05-27T09:35:14.372769+0000","last_became_active":"2024-05-27T09:31:24.914541+0000","last_became_peered":"2024-05-27T09:31:24.914541+0000","last_unstale":"2024-05-27T09:35:14.372769+0000","last_undegraded":"2024-05-27T09:35:14.372769+0000","last_fullsized":"2024-05-27T09:35:14.372769+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:30:38.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.042624+0000","last_change":"2024-05-27T09:31:24.909628+0000","last_active":"2024-05-27T09:35:14.042624+0000","last_peered":"2024-05-27T09:35:14.042624+0000","last_clean":"2024-05-27T09:35:14.042624+0000","last_became_active":"2024-05-27T09:31:24.902931+0000","last_became_peered":"2024-05-27T09:31:24.902931+0000","last_unstale":"2024-05-27T09:35:14.042624+0000","last_undegraded":"2024-05-27T09:35:14.042624+0000","last_fullsized":"2024-05-27T09:35:14.042624+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:17:06.711316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.242522+0000","last_change":"2024-05-27T09:31:24.912099+0000","last_active":"2024-05-27T09:35:14.242522+0000","last_peered":"2024-05-27T09:35:14.242522+0000","last_clean":"2024-05-27T09:35:14.242522+0000","last_became_active":"2024-05-27T09:31:24.911669+0000","last_became_peered":"2024-05-27T09:31:24.911669+0000","last_unstale":"2024-05-27T09:35:14.242522+0000","last_undegraded":"2024-05-27T09:35:14.242522+0000","last_fullsized":"2024-05-27T09:35:14.242522+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:58.068589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840140+0000","last_change":"2024-05-27T09:31:24.891426+0000","last_active":"2024-05-27T09:35:14.840140+0000","last_peered":"2024-05-27T09:35:14.840140+0000","last_clean":"2024-05-27T09:35:14.840140+0000","last_became_active":"2024-05-27T09:31:24.890877+0000","last_became_peered":"2024-05-27T09:31:24.890877+0000","last_unstale":"2024-05-27T09:35:14.840140+0000","last_undegraded":"2024-05-27T09:35:14.840140+0000","last_fullsized":"2024-05-27T09:35:14.840140+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:52:15.338188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.373135+0000","last_change":"2024-05-27T09:31:24.892021+0000","last_active":"2024-05-27T09:35:14.373135+0000","last_peered":"2024-05-27T09:35:14.373135+0000","last_clean":"2024-05-27T09:35:14.373135+0000","last_became_active":"2024-05-27T09:31:24.891607+0000","last_became_peered":"2024-05-27T09:31:24.891607+0000","last_unstale":"2024-05-27T09:35:14.373135+0000","last_undegraded":"2024-05-27T09:35:14.373135+0000","last_fullsized":"2024-05-27T09:35:14.373135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:05.623199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.171384+0000","last_change":"2024-05-27T09:31:24.912166+0000","last_active":"2024-05-27T09:35:14.171384+0000","last_peered":"2024-05-27T09:35:14.171384+0000","last_clean":"2024-05-27T09:35:14.171384+0000","last_became_active":"2024-05-27T09:31:24.907555+0000","last_became_peered":"2024-05-27T09:31:24.907555+0000","last_unstale":"2024-05-27T09:35:14.171384+0000","last_undegraded":"2024-05-27T09:35:14.171384+0000","last_fullsized":"2024-05-27T09:35:14.171384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:55:33.772321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.129575+0000","last_change":"2024-05-27T09:31:24.910525+0000","last_active":"2024-05-27T09:35:14.129575+0000","last_peered":"2024-05-27T09:35:14.129575+0000","last_clean":"2024-05-27T09:35:14.129575+0000","last_became_active":"2024-05-27T09:31:24.910089+0000","last_became_peered":"2024-05-27T09:31:24.910089+0000","last_unstale":"2024-05-27T09:35:14.129575+0000","last_undegraded":"2024-05-27T09:35:14.129575+0000","last_fullsized":"2024-05-27T09:35:14.129575+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:19:02.274781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.170634+0000","last_change":"2024-05-27T09:31:24.901125+0000","last_active":"2024-05-27T09:35:14.170634+0000","last_peered":"2024-05-27T09:35:14.170634+0000","last_clean":"2024-05-27T09:35:14.170634+0000","last_became_active":"2024-05-27T09:31:24.900361+0000","last_became_peered":"2024-05-27T09:31:24.900361+0000","last_unstale":"2024-05-27T09:35:14.170634+0000","last_undegraded":"2024-05-27T09:35:14.170634+0000","last_fullsized":"2024-05-27T09:35:14.170634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:30.116349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.372869+0000","last_change":"2024-05-27T09:31:24.917612+0000","last_active":"2024-05-27T09:35:14.372869+0000","last_peered":"2024-05-27T09:35:14.372869+0000","last_clean":"2024-05-27T09:35:14.372869+0000","last_became_active":"2024-05-27T09:31:24.917210+0000","last_became_peered":"2024-05-27T09:31:24.917210+0000","last_unstale":"2024-05-27T09:35:14.372869+0000","last_undegraded":"2024-05-27T09:35:14.372869+0000","last_fullsized":"2024-05-27T09:35:14.372869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T21:19:00.769114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.241872+0000","last_change":"2024-05-27T09:31:24.909654+0000","last_active":"2024-05-27T09:35:14.241872+0000","last_peered":"2024-05-27T09:35:14.241872+0000","last_clean":"2024-05-27T09:35:14.241872+0000","last_became_active":"2024-05-27T09:31:24.909143+0000","last_became_peered":"2024-05-27T09:31:24.909143+0000","last_unstale":"2024-05-27T09:35:14.241872+0000","last_undegraded":"2024-05-27T09:35:14.241872+0000","last_fullsized":"2024-05-27T09:35:14.241872+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:26:42.844614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-27T09:35:14.840108+0000","last_change":"2024-05-27T09:31:24.899031+0000","last_active":"2024-05-27T09:35:14.840108+0000","last_peered":"2024-05-27T09:35:14.840108+0000","last_clean":"2024-05-27T09:35:14.840108+0000","last_became_active":"2024-05-27T09:31:24.898664+0000","last_became_peered":"2024-05-27T09:31:24.898664+0000","last_unstale":"2024-05-27T09:35:14.840108+0000","last_undegraded":"2024-05-27T09:35:14.840108+0000","last_fullsized":"2024-05-27T09:35:14.840108+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T09:31:23.862590+0000","last_clean_scrub_stamp":"2024-05-27T09:31:23.862590+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:09.659646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":248,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":797569,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1811,"num_read_kb":1124,"num_write":427,"num_write_kb":2361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2617344,"data_stored":2392707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678085,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":892928,"data_stored":440736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874313,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41500,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703652,"statfs":{"total":95995035648,"available":95952539648,"internally_reserved":0,"allocated":1929216,"data_stored":1432037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070543,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1302528,"data_stored":834188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266773,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1302528,"data_stored":834108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863463003,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28484,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716668,"statfs":{"total":95995035648,"available":95965868032,"internally_reserved":0,"allocated":1503232,"data_stored":1034974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626528,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":897024,"data_stored":440361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574975,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":510122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":441632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411450,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":511813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182513,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28736,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716416,"statfs":{"total":95995035648,"available":95965609984,"internally_reserved":0,"allocated":1638400,"data_stored":1162120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986285,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":443000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822760,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37260,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707892,"statfs":{"total":95995035648,"available":95956881408,"internally_reserved":0,"allocated":1847296,"data_stored":1361270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":721732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":330631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":401398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T09:35:48.479 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:35:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.520+0000 7f4bd6c00640 1 -- 172.21.15.44:0/2584753052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd0110520 msgr2=0x7f4bd010a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.520+0000 7f4bd6c00640 1 --2- 172.21.15.44:0/2584753052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd0110520 0x7f4bd010a200 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009900 tx=0x7f4bbc02a480 comp rx=0 tx=0).stop 2024-05-27T09:35:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.520+0000 7f4bd6c00640 1 -- 172.21.15.44:0/2584753052 shutdown_connections 2024-05-27T09:35:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.520+0000 7f4bd6c00640 1 --2- 172.21.15.44:0/2584753052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bd010a890 0x7f4bd0116b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.520+0000 7f4bd6c00640 1 --2- 172.21.15.44:0/2584753052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd0110520 0x7f4bd010a200 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.520+0000 7f4bd6c00640 1 --2- 172.21.15.44:0/2584753052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd0110110 0x7f4bd0109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.520+0000 7f4bd6c00640 1 -- 172.21.15.44:0/2584753052 >> 172.21.15.44:0/2584753052 conn(0x7f4bd0076070 msgr2=0x7f4bd0078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.520+0000 7f4bd6c00640 1 -- 172.21.15.44:0/2584753052 shutdown_connections 2024-05-27T09:35:48.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.520+0000 7f4bd6c00640 1 -- 172.21.15.44:0/2584753052 wait complete. 2024-05-27T09:35:48.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.522+0000 7f4bd6c00640 1 Processor -- start 2024-05-27T09:35:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.522+0000 7f4bd6c00640 1 -- start start 2024-05-27T09:35:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.523+0000 7f4bd6c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bd0110110 0x7f4bd01a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.523+0000 7f4bd6c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd0110520 0x7f4bd01a6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.523+0000 7f4bd6c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd01a06d0 0x7f4bd01a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.523+0000 7f4bd6c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd011a2b0 con 0x7f4bd01a06d0 2024-05-27T09:35:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.523+0000 7f4bd6c00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd011a130 con 0x7f4bd0110110 2024-05-27T09:35:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.523+0000 7f4bd6c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd011a430 con 0x7f4bd0110520 2024-05-27T09:35:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.523+0000 7f4bd4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd0110520 0x7f4bd01a6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.523+0000 7f4bd6200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd01a06d0 0x7f4bd01a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.523+0000 7f4bd4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd0110520 0x7f4bd01a6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:54892/0 (socket says 172.21.15.44:54892) 2024-05-27T09:35:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.523+0000 7f4bd4e00640 1 -- 172.21.15.44:0/99371305 learned_addr learned my addr 172.21.15.44:0/99371305 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.523+0000 7f4bd5800640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bd0110110 0x7f4bd01a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.524+0000 7f4bd5800640 1 -- 172.21.15.44:0/99371305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd0110520 msgr2=0x7f4bd01a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.524+0000 7f4bd5800640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd0110520 0x7f4bd01a6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.524+0000 7f4bd5800640 1 -- 172.21.15.44:0/99371305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd01a06d0 msgr2=0x7f4bd01a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.524+0000 7f4bd5800640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd01a06d0 0x7f4bd01a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.524+0000 7f4bd5800640 1 -- 172.21.15.44:0/99371305 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd01a11c0 con 0x7f4bd0110110 2024-05-27T09:35:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.524+0000 7f4bd4e00640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd0110520 0x7f4bd01a6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.524+0000 7f4bd5800640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bd0110110 0x7f4bd01a6550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400e9c0 tx=0x7f4bc400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.525+0000 7f4bce000640 1 -- 172.21.15.44:0/99371305 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc400ccc0 con 0x7f4bd0110110 2024-05-27T09:35:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.525+0000 7f4bce000640 1 -- 172.21.15.44:0/99371305 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc40035f0 con 0x7f4bd0110110 2024-05-27T09:35:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.525+0000 7f4bce000640 1 -- 172.21.15.44:0/99371305 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc400ce60 con 0x7f4bd0110110 2024-05-27T09:35:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.525+0000 7f4bd6c00640 1 -- 172.21.15.44:0/99371305 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd01a14b0 con 0x7f4bd0110110 2024-05-27T09:35:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.525+0000 7f4bd6c00640 1 -- 172.21.15.44:0/99371305 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd01ad590 con 0x7f4bd0110110 2024-05-27T09:35:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.528+0000 7f4bdf170740 1 Processor -- start 2024-05-27T09:35:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.528+0000 7f4bdf170740 1 -- start start 2024-05-27T09:35:48.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.529+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 0x55c09281ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.529+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 0x55c09281bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.529+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924584d0 0x55c09281c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.529+0000 7f4bdf170740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c09278cb90 con 0x55c0924502f0 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.529+0000 7f4bdf170740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c09277c730 con 0x55c0924791b0 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.529+0000 7f4bdf170740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c092780770 con 0x55c0924584d0 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.529+0000 7f4bce000640 1 -- 172.21.15.44:0/99371305 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f4bc40037a0 con 0x7f4bd0110110 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.529+0000 7f4bd6200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 0x55c09281ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.529+0000 7f4bd6200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 0x55c09281ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39854/0 (socket says 172.21.15.44:39854) 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.529+0000 7f4bd6200640 1 -- 172.21.15.44:0/3578873902 learned_addr learned my addr 172.21.15.44:0/3578873902 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.529+0000 7f4bd4e00640 1 --2- 172.21.15.44:0/3578873902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924584d0 0x55c09281c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.530+0000 7f4bd5800640 1 --2- 172.21.15.44:0/3578873902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 0x55c09281bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.530+0000 7f4bd6200640 1 -- 172.21.15.44:0/3578873902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924584d0 msgr2=0x55c09281c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.530+0000 7f4bd6200640 1 --2- 172.21.15.44:0/3578873902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924584d0 0x55c09281c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.530+0000 7f4bd6200640 1 -- 172.21.15.44:0/3578873902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 msgr2=0x55c09281bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.530+0000 7f4bd6200640 1 --2- 172.21.15.44:0/3578873902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 0x55c09281bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.530+0000 7f4bd6200640 1 -- 172.21.15.44:0/3578873902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0927158b0 con 0x55c0924791b0 2024-05-27T09:35:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.530+0000 7f4bd6200640 1 --2- 172.21.15.44:0/3578873902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 0x55c09281ba60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8011440 tx=0x7f4bc8011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67e34ae557088800 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.530+0000 7f4bc2a00640 1 -- 172.21.15.44:0/3578873902 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8003c30 con 0x55c0924791b0 2024-05-27T09:35:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.531+0000 7f4bc2a00640 1 -- 172.21.15.44:0/3578873902 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc8003dd0 con 0x55c0924791b0 2024-05-27T09:35:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.531+0000 7f4bc2a00640 1 -- 172.21.15.44:0/3578873902 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800fae0 con 0x55c0924791b0 2024-05-27T09:35:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.531+0000 7f4bce000640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4bb80b2af0 0x7f4bb80b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.531+0000 7f4bce000640 1 -- 172.21.15.44:0/99371305 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f4bc40e2b70 con 0x7f4bd0110110 2024-05-27T09:35:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.531+0000 7f4bd5800640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4bb80b2af0 0x7f4bb80b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.532+0000 7f4bdf170740 1 -- 172.21.15.44:0/3578873902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 msgr2=0x55c09281ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.532+0000 7f4bdf170740 1 --2- 172.21.15.44:0/3578873902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 0x55c09281ba60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8011440 tx=0x7f4bc8011900 comp rx=0 tx=0).stop 2024-05-27T09:35:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.532+0000 7f4bdf170740 1 -- 172.21.15.44:0/3578873902 shutdown_connections 2024-05-27T09:35:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.532+0000 7f4bdf170740 1 --2- 172.21.15.44:0/3578873902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924584d0 0x55c09281c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.532+0000 7f4bdf170740 1 --2- 172.21.15.44:0/3578873902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 0x55c09281bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.532+0000 7f4bdf170740 1 --2- 172.21.15.44:0/3578873902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 0x55c09281ba60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.532+0000 7f4bdf170740 1 -- 172.21.15.44:0/3578873902 >> 172.21.15.44:0/3578873902 conn(0x55c09246d300 msgr2=0x55c092818210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.532+0000 7f4bdf170740 1 -- 172.21.15.44:0/3578873902 shutdown_connections 2024-05-27T09:35:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.532+0000 7f4bdf170740 1 -- 172.21.15.44:0/3578873902 wait complete. 2024-05-27T09:35:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.532+0000 7f4bd5800640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4bb80b2af0 0x7f4bb80b5000 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4bc40e0170 tx=0x7f4bc4003e80 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.533+0000 7f4bdf170740 1 -- start start 2024-05-27T09:35:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.533+0000 7f4bdf170740 1 Processor -- start 2024-05-27T09:35:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924502f0 0x55c092825ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924584d0 0x55c092826400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924791b0 0x55c092831ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bdf170740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c092462790 con 0x55c0924791b0 2024-05-27T09:35:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bdf170740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c092724bc0 con 0x55c0924584d0 2024-05-27T09:35:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bdf170740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c092828b90 con 0x55c0924502f0 2024-05-27T09:35:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bdf170740 10 client.0 _ll_register_callbacks cb 0x55c092702400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:35:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bd6200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924791b0 0x55c092831ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bd6200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924791b0 0x55c092831ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44518/0 (socket says 172.21.15.44:44518) 2024-05-27T09:35:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bd6200640 1 -- 172.21.15.44:0/1656275097 learned_addr learned my addr 172.21.15.44:0/1656275097 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bd4e00640 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924584d0 0x55c092826400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.534+0000 7f4bd6200640 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924502f0 0x55c092825ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bd4e00640 1 -- 172.21.15.44:0/1656275097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924502f0 msgr2=0x55c092825ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bd4e00640 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924502f0 0x55c092825ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bd4e00640 1 -- 172.21.15.44:0/1656275097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924791b0 msgr2=0x55c092831ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bd4e00640 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924791b0 0x55c092831ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bd4e00640 1 -- 172.21.15.44:0/1656275097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c092491210 con 0x55c0924584d0 2024-05-27T09:35:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bd4e00640 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924584d0 0x55c092826400 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc004770 tx=0x7f4bbc031740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28018a225a220029 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bb7400640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:35:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bb7400640 1 -- 172.21.15.44:0/1656275097 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc035040 con 0x55c0924584d0 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bdf170740 1 -- 172.21.15.44:0/1656275097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bc8003dd0 con 0x55c0924584d0 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bb7400640 1 -- 172.21.15.44:0/1656275097 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f4bbc004450 con 0x55c0924584d0 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bb7400640 1 -- 172.21.15.44:0/1656275097 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4bbc005020 con 0x55c0924584d0 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bdf170740 10 client.25295 fetch_fsmap learned FSMap version 26 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.535+0000 7f4bdf170740 1 -- 172.21.15.44:0/1656275097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c09255d620 con 0x55c0924584d0 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.536+0000 7f4bb7400640 1 -- 172.21.15.44:0/1656275097 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc8003dd0 con 0x55c0924584d0 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.536+0000 7f4bb7400640 1 -- 172.21.15.44:0/1656275097 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4bbc0035f0 con 0x55c0924584d0 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.536+0000 7f4bdf170740 10 client.25295 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.536+0000 7f4bdf170740 10 client.25295 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.536+0000 7f4bdf170740 20 client.25295 populate_metadata read hostname 'smithi044' 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.536+0000 7f4bdf170740 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55c092437d80 0x55c092837f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.536+0000 7f4bdf170740 4 client.25295 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:35:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.536+0000 7f4bdf170740 1 -- 172.21.15.44:0/1656275097 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c09278eae0 con 0x55c092437d80 2024-05-27T09:35:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.536+0000 7f4bd6200640 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55c092437d80 0x55c092837f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.537+0000 7f4bd6200640 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55c092437d80 0x55c092837f10 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=908db141caac62e6 server_cookie=f95d3c914af5c87b in_seq=0 out_seq=0 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.537+0000 7f4bb7400640 10 client.25295 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bb7400640 1 -- 172.21.15.44:0/1656275097 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c09278eae0 con 0x55c092437d80 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bb7400640 10 client.25295 handle_command_reply: tid=0 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bdf170740 1 client.25295 shutdown 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bdf170740 1 -- 172.21.15.44:0/1656275097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924584d0 msgr2=0x55c092826400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bdf170740 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924584d0 0x55c092826400 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc004770 tx=0x7f4bbc031740 comp rx=0 tx=0).stop 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bdf170740 1 -- 172.21.15.44:0/1656275097 shutdown_connections 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bdf170740 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55c092437d80 0x55c092837f10 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bdf170740 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924791b0 0x55c092831ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bdf170740 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924584d0 0x55c092826400 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bdf170740 1 --2- 172.21.15.44:0/1656275097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924502f0 0x55c092825ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bb7400640 0 client.25295 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:48.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bdf170740 1 -- 172.21.15.44:0/1656275097 >> 172.21.15.44:0/1656275097 conn(0x55c09246d300 msgr2=0x55c09281b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:48.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bdf170740 1 -- 172.21.15.44:0/1656275097 shutdown_connections 2024-05-27T09:35:48.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.538+0000 7f4bdf170740 1 -- 172.21.15.44:0/1656275097 wait complete. 2024-05-27T09:35:48.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.539+0000 7f4bdf170740 20 client.25295 trim_cache size 0 max 16384 2024-05-27T09:35:48.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:48 smithi050 ceph-mon[36371]: from='client.25288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:48.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:48 smithi050 ceph-mon[36371]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:35:48.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.556+0000 7f4bdf170740 1 Processor -- start 2024-05-27T09:35:48.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.556+0000 7f4bdf170740 1 -- start start 2024-05-27T09:35:48.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.556+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924502f0 0x55c092821440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.556+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924584d0 0x55c09281bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.556+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924791b0 0x55c092831c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.556+0000 7f4bdf170740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01b8730 con 0x55c0924584d0 2024-05-27T09:35:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.556+0000 7f4bdf170740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01b6000 con 0x55c0924502f0 2024-05-27T09:35:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.556+0000 7f4bdf170740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01b3e50 con 0x55c0924791b0 2024-05-27T09:35:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.556+0000 7f4bd4e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924584d0 0x55c09281bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.556+0000 7f4bd4e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924584d0 0x55c09281bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44532/0 (socket says 172.21.15.44:44532) 2024-05-27T09:35:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.556+0000 7f4bd4e00640 1 -- 172.21.15.44:0/3774692422 learned_addr learned my addr 172.21.15.44:0/3774692422 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.557+0000 7f4bd6200640 1 --2- 172.21.15.44:0/3774692422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924502f0 0x55c092821440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.557+0000 7f4bd6200640 1 --2- 172.21.15.44:0/3774692422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924791b0 0x55c092831c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.557+0000 7f4bd6200640 1 -- 172.21.15.44:0/3774692422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924502f0 msgr2=0x55c092821440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.557+0000 7f4bd6200640 1 --2- 172.21.15.44:0/3774692422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924502f0 0x55c092821440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.557+0000 7f4bd6200640 1 -- 172.21.15.44:0/3774692422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924584d0 msgr2=0x55c09281bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.557+0000 7f4bd6200640 1 --2- 172.21.15.44:0/3774692422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924584d0 0x55c09281bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.557+0000 7f4bd6200640 1 -- 172.21.15.44:0/3774692422 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0928201f0 con 0x55c0924791b0 2024-05-27T09:35:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.557+0000 7f4bd6200640 1 --2- 172.21.15.44:0/3774692422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924791b0 0x55c092831c30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55c092819280 tx=0x7f4bc800f680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=764b47cbb781e68e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.558+0000 7f4bc2000640 1 -- 172.21.15.44:0/3774692422 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800f9b0 con 0x55c0924791b0 2024-05-27T09:35:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.558+0000 7f4bc2000640 1 -- 172.21.15.44:0/3774692422 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc8004050 con 0x55c0924791b0 2024-05-27T09:35:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.558+0000 7f4bc2000640 1 -- 172.21.15.44:0/3774692422 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80286a0 con 0x55c0924791b0 2024-05-27T09:35:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.559+0000 7f4bdf170740 1 -- 172.21.15.44:0/3774692422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924791b0 msgr2=0x55c092831c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.559+0000 7f4bdf170740 1 --2- 172.21.15.44:0/3774692422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924791b0 0x55c092831c30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55c092819280 tx=0x7f4bc800f680 comp rx=0 tx=0).stop 2024-05-27T09:35:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.559+0000 7f4bdf170740 1 -- 172.21.15.44:0/3774692422 shutdown_connections 2024-05-27T09:35:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.559+0000 7f4bdf170740 1 --2- 172.21.15.44:0/3774692422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924791b0 0x55c092831c30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.559+0000 7f4bdf170740 1 --2- 172.21.15.44:0/3774692422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924584d0 0x55c09281bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.559+0000 7f4bdf170740 1 --2- 172.21.15.44:0/3774692422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924502f0 0x55c092821440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.559+0000 7f4bdf170740 1 -- 172.21.15.44:0/3774692422 >> 172.21.15.44:0/3774692422 conn(0x55c09246d300 msgr2=0x55c09281b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.559+0000 7f4bdf170740 1 -- 172.21.15.44:0/3774692422 shutdown_connections 2024-05-27T09:35:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.559+0000 7f4bdf170740 1 -- 172.21.15.44:0/3774692422 wait complete. 2024-05-27T09:35:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.559+0000 7f4bdf170740 1 -- start start 2024-05-27T09:35:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.560+0000 7f4bdf170740 1 Processor -- start 2024-05-27T09:35:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.560+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 0x55c0928291d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.560+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924584d0 0x55c092829710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.560+0000 7f4bdf170740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 0x55c09289e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.560+0000 7f4bdf170740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01b44c0 con 0x55c0924502f0 2024-05-27T09:35:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.560+0000 7f4bdf170740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01b5930 con 0x55c0924791b0 2024-05-27T09:35:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.560+0000 7f4bdf170740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01b4b90 con 0x55c0924584d0 2024-05-27T09:35:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.560+0000 7f4bdf170740 10 client.0 _ll_register_callbacks cb 0x7f4bbc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:35:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.560+0000 7f4bd6200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 0x55c0928291d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.560+0000 7f4bd6200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 0x55c0928291d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44544/0 (socket says 172.21.15.44:44544) 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.561+0000 7f4bd6200640 1 -- 172.21.15.44:0/908864635 learned_addr learned my addr 172.21.15.44:0/908864635 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.561+0000 7f4bd4e00640 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924584d0 0x55c092829710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.561+0000 7f4bd6200640 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 0x55c09289e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.561+0000 7f4bd6200640 1 -- 172.21.15.44:0/908864635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924584d0 msgr2=0x55c092829710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.561+0000 7f4bd6200640 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924584d0 0x55c092829710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.561+0000 7f4bd6200640 1 -- 172.21.15.44:0/908864635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 msgr2=0x55c09289e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.561+0000 7f4bd6200640 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 0x55c09289e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.561+0000 7f4bd6200640 1 -- 172.21.15.44:0/908864635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c09289ea80 con 0x55c0924502f0 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.561+0000 7f4bd4e00640 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924584d0 0x55c092829710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.561+0000 7f4bd6200640 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 0x55c0928291d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800f750 tx=0x7f4bc8004200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef1f20920cdf65d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.561+0000 7f4bb7400640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.562+0000 7f4bb7400640 1 -- 172.21.15.44:0/908864635 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8004270 con 0x55c0924502f0 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.562+0000 7f4bb7400640 1 -- 172.21.15.44:0/908864635 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f4bc8004410 con 0x55c0924502f0 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.562+0000 7f4bdf170740 1 -- 172.21.15.44:0/908864635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bc8004050 con 0x55c0924502f0 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.562+0000 7f4bb7400640 1 -- 172.21.15.44:0/908864635 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4bc8004610 con 0x55c0924502f0 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.562+0000 7f4bdf170740 10 client.15162 fetch_fsmap learned FSMap version 26 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.562+0000 7f4bdf170740 1 -- 172.21.15.44:0/908864635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c09289fbc0 con 0x55c0924502f0 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.562+0000 7f4bb7400640 1 -- 172.21.15.44:0/908864635 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc8004050 con 0x55c0924502f0 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.562+0000 7f4bb7400640 1 -- 172.21.15.44:0/908864635 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4bc8025040 con 0x55c0924502f0 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.563+0000 7f4bdf170740 10 client.15162 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.563+0000 7f4bdf170740 10 client.15162 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.563+0000 7f4bdf170740 20 client.15162 populate_metadata read hostname 'smithi044' 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.563+0000 7f4bdf170740 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55c09240d1c0 0x55c0928a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.563+0000 7f4bdf170740 4 client.15162 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:35:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.563+0000 7f4bdf170740 1 -- 172.21.15.44:0/908864635 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c0927882c0 con 0x55c09240d1c0 2024-05-27T09:35:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.563+0000 7f4bd4e00640 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55c09240d1c0 0x55c0928a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.564+0000 7f4bd4e00640 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55c09240d1c0 0x55c0928a20b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca835e6d434e72b9 server_cookie=711a9176ca1fada9 in_seq=0 out_seq=0 2024-05-27T09:35:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.564+0000 7f4bb7400640 10 client.15162 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bb7400640 1 -- 172.21.15.44:0/908864635 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c0927882c0 con 0x55c09240d1c0 2024-05-27T09:35:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bb7400640 10 client.15162 handle_command_reply: tid=0 2024-05-27T09:35:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 1 client.15162 shutdown 2024-05-27T09:35:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 1 -- 172.21.15.44:0/908864635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 msgr2=0x55c0928291d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 0x55c0928291d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800f750 tx=0x7f4bc8004200 comp rx=0 tx=0).stop 2024-05-27T09:35:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 1 -- 172.21.15.44:0/908864635 shutdown_connections 2024-05-27T09:35:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55c09240d1c0 0x55c0928a20b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c0924791b0 0x55c09289e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c0924584d0 0x55c092829710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 1 --2- 172.21.15.44:0/908864635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c0924502f0 0x55c0928291d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bb7400640 0 client.15162 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:35:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 1 -- 172.21.15.44:0/908864635 >> 172.21.15.44:0/908864635 conn(0x55c09246d300 msgr2=0x55c092827d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 1 -- 172.21.15.44:0/908864635 shutdown_connections 2024-05-27T09:35:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 1 -- 172.21.15.44:0/908864635 wait complete. 2024-05-27T09:35:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.578+0000 7f4bdf170740 20 client.15162 trim_cache size 0 max 16384 2024-05-27T09:35:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:35:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:35:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:35:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 -- 172.21.15.44:0/99371305 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4bb80b2af0 msgr2=0x7f4bb80b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4bb80b2af0 0x7f4bb80b5000 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4bc40e0170 tx=0x7f4bc4003e80 comp rx=0 tx=0).stop 2024-05-27T09:35:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 -- 172.21.15.44:0/99371305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bd0110110 msgr2=0x7f4bd01a6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bd0110110 0x7f4bd01a6550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400e9c0 tx=0x7f4bc400edc0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 -- 172.21.15.44:0/99371305 shutdown_connections 2024-05-27T09:35:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4bb80b2af0 0x7f4bb80b5000 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4bd01a06d0 0x7f4bd01a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.582 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd0110520 0x7f4bd01a6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.582 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 --2- 172.21.15.44:0/99371305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bd0110110 0x7f4bd01a6550 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:48.582 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 -- 172.21.15.44:0/99371305 >> 172.21.15.44:0/99371305 conn(0x7f4bd0076070 msgr2=0x7f4bd0076780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:48.582 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 -- 172.21.15.44:0/99371305 shutdown_connections 2024-05-27T09:35:48.582 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:35:48.579+0000 7f4bd6c00640 1 -- 172.21.15.44:0/99371305 wait complete. 2024-05-27T09:35:48.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:48 smithi189 ceph-mon[36671]: from='client.25288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T09:35:48.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:48 smithi189 ceph-mon[36671]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:35:48.949 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:50.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:50 smithi044 ceph-mon[31618]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:35:50.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:50 smithi050 ceph-mon[36371]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:35:50.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:50 smithi189 ceph-mon[36671]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:35:51.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.330+0000 7fc16a400700 1 -- 172.21.15.44:0/2760124655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc164108d20 msgr2=0x7fc164109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:51.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.330+0000 7fc16a400700 1 --2- 172.21.15.44:0/2760124655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc164108d20 0x7fc164109130 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc150009990 tx=0x7fc150009ca0 comp rx=0 tx=0).stop 2024-05-27T09:35:51.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.330+0000 7fc16a400700 1 -- 172.21.15.44:0/2760124655 shutdown_connections 2024-05-27T09:35:51.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.330+0000 7fc16a400700 1 --2- 172.21.15.44:0/2760124655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164109670 0x7fc164110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:51.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.330+0000 7fc16a400700 1 --2- 172.21.15.44:0/2760124655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1641043f0 0x7fc164104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:51.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.330+0000 7fc16a400700 1 --2- 172.21.15.44:0/2760124655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc164108d20 0x7fc164109130 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:51.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.330+0000 7fc16a400700 1 -- 172.21.15.44:0/2760124655 >> 172.21.15.44:0/2760124655 conn(0x7fc164079d00 msgr2=0x7fc16407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:51.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.330+0000 7fc16a400700 1 -- 172.21.15.44:0/2760124655 shutdown_connections 2024-05-27T09:35:51.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.330+0000 7fc16a400700 1 -- 172.21.15.44:0/2760124655 wait complete. 2024-05-27T09:35:51.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.332+0000 7fc16a400700 1 Processor -- start 2024-05-27T09:35:51.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.333+0000 7fc16a400700 1 -- start start 2024-05-27T09:35:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.333+0000 7fc16a400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1641043f0 0x7fc16419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.333+0000 7fc16a400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc164108d20 0x7fc16419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.333+0000 7fc16a400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164109670 0x7fc1641a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc16a400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc1641142d0 con 0x7fc164108d20 2024-05-27T09:35:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc16a400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc164114150 con 0x7fc1641043f0 2024-05-27T09:35:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc16a400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc164114450 con 0x7fc164109670 2024-05-27T09:35:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc163e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc164108d20 0x7fc16419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc163e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc164108d20 0x7fc16419d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44572/0 (socket says 172.21.15.44:44572) 2024-05-27T09:35:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc163e00700 1 -- 172.21.15.44:0/2041167480 learned_addr learned my addr 172.21.15.44:0/2041167480 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc169000700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1641043f0 0x7fc16419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc169a00700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164109670 0x7fc1641a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc163e00700 1 -- 172.21.15.44:0/2041167480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164109670 msgr2=0x7fc1641a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc163e00700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164109670 0x7fc1641a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:51.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc163e00700 1 -- 172.21.15.44:0/2041167480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1641043f0 msgr2=0x7fc16419ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:51.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc163e00700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1641043f0 0x7fc16419ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:51.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.334+0000 7fc163e00700 1 -- 172.21.15.44:0/2041167480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1641a1d50 con 0x7fc164108d20 2024-05-27T09:35:51.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.335+0000 7fc163e00700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc164108d20 0x7fc16419d420 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc150000c00 tx=0x7fc1500057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:51.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.335+0000 7fc161600700 1 -- 172.21.15.44:0/2041167480 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150027070 con 0x7fc164108d20 2024-05-27T09:35:51.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.335+0000 7fc16a400700 1 -- 172.21.15.44:0/2041167480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1641a1fe0 con 0x7fc164108d20 2024-05-27T09:35:51.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.335+0000 7fc161600700 1 -- 172.21.15.44:0/2041167480 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc150022470 con 0x7fc164108d20 2024-05-27T09:35:51.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.335+0000 7fc161600700 1 -- 172.21.15.44:0/2041167480 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150017650 con 0x7fc164108d20 2024-05-27T09:35:51.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.335+0000 7fc16a400700 1 -- 172.21.15.44:0/2041167480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1641a24c0 con 0x7fc164108d20 2024-05-27T09:35:51.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.337+0000 7fc16a400700 1 -- 172.21.15.44:0/2041167480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1641058a0 con 0x7fc164108d20 2024-05-27T09:35:51.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.339+0000 7fc161600700 1 -- 172.21.15.44:0/2041167480 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7fc15002a070 con 0x7fc164108d20 2024-05-27T09:35:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.341+0000 7fc161600700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc14c0b2b10 0x7fc14c0b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.341+0000 7fc161600700 1 -- 172.21.15.44:0/2041167480 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fc1500e3a20 con 0x7fc164108d20 2024-05-27T09:35:51.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.341+0000 7fc169000700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc14c0b2b10 0x7fc14c0b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:51.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.342+0000 7fc169000700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc14c0b2b10 0x7fc14c0b5020 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc15800f540 tx=0x7fc15800b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.344+0000 7fc161600700 1 -- 172.21.15.44:0/2041167480 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc150013070 con 0x7fc164108d20 2024-05-27T09:35:51.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.620+0000 7fc16a400700 1 -- 172.21.15.44:0/2041167480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1640512b0 con 0x7fc164108d20 2024-05-27T09:35:51.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.622+0000 7fc161600700 1 -- 172.21.15.44:0/2041167480 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7fc1500ab2a0 con 0x7fc164108d20 2024-05-27T09:35:51.623 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:51.623 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:35:51.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.625+0000 7fc16a400700 1 -- 172.21.15.44:0/2041167480 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc14c0b2b10 msgr2=0x7fc14c0b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:51.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.625+0000 7fc16a400700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc14c0b2b10 0x7fc14c0b5020 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc15800f540 tx=0x7fc15800b040 comp rx=0 tx=0).stop 2024-05-27T09:35:51.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.625+0000 7fc16a400700 1 -- 172.21.15.44:0/2041167480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc164108d20 msgr2=0x7fc16419d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:51.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.625+0000 7fc16a400700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc164108d20 0x7fc16419d420 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc150000c00 tx=0x7fc1500057c0 comp rx=0 tx=0).stop 2024-05-27T09:35:51.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.626+0000 7fc16a400700 1 -- 172.21.15.44:0/2041167480 shutdown_connections 2024-05-27T09:35:51.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.626+0000 7fc16a400700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc14c0b2b10 0x7fc14c0b5020 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:51.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.626+0000 7fc16a400700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164109670 0x7fc1641a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:51.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.626+0000 7fc16a400700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1641043f0 0x7fc16419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:51.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.626+0000 7fc16a400700 1 --2- 172.21.15.44:0/2041167480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc164108d20 0x7fc16419d420 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:51.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.626+0000 7fc16a400700 1 -- 172.21.15.44:0/2041167480 >> 172.21.15.44:0/2041167480 conn(0x7fc164079d00 msgr2=0x7fc16410f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:51.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.626+0000 7fc16a400700 1 -- 172.21.15.44:0/2041167480 shutdown_connections 2024-05-27T09:35:51.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:51.626+0000 7fc16a400700 1 -- 172.21.15.44:0/2041167480 wait complete. 2024-05-27T09:35:51.628 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:35:51.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:51 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2041167480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:51 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2041167480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:52.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:51 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2041167480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:52.434 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:35:52.945 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:52.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:52 smithi044 ceph-mon[31618]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:35:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:52 smithi050 ceph-mon[36371]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:35:53.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:52 smithi189 ceph-mon[36671]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:35:54.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:54 smithi044 ceph-mon[31618]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:35:54.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:54 smithi050 ceph-mon[36371]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:35:54.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:54 smithi189 ceph-mon[36671]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:35:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.208+0000 7f2f22800700 1 -- 172.21.15.44:0/1036819912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1c110960 msgr2=0x7f2f1c110d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.208+0000 7f2f22800700 1 --2- 172.21.15.44:0/1036819912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1c110960 0x7f2f1c110d70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c0098f0 tx=0x7f2f0c009c00 comp rx=0 tx=0).stop 2024-05-27T09:35:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.208+0000 7f2f22800700 1 -- 172.21.15.44:0/1036819912 shutdown_connections 2024-05-27T09:35:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.208+0000 7f2f22800700 1 --2- 172.21.15.44:0/1036819912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f1c10b2d0 0x7f2f1c117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.208+0000 7f2f22800700 1 --2- 172.21.15.44:0/1036819912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1c110960 0x7f2f1c110d70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.208+0000 7f2f22800700 1 --2- 172.21.15.44:0/1036819912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f1c10a980 0x7f2f1c10ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.208+0000 7f2f22800700 1 -- 172.21.15.44:0/1036819912 >> 172.21.15.44:0/1036819912 conn(0x7f2f1c077c70 msgr2=0x7f2f1c0780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.208+0000 7f2f22800700 1 -- 172.21.15.44:0/1036819912 shutdown_connections 2024-05-27T09:35:55.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.209+0000 7f2f22800700 1 -- 172.21.15.44:0/1036819912 wait complete. 2024-05-27T09:35:55.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.210+0000 7f2f22800700 1 Processor -- start 2024-05-27T09:35:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.210+0000 7f2f22800700 1 -- start start 2024-05-27T09:35:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.210+0000 7f2f22800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1c10a980 0x7f2f1c1a1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f22800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f1c10b2d0 0x7f2f1c1a1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f22800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f1c110960 0x7f2f1c1a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f22800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c11aa60 con 0x7f2f1c10b2d0 2024-05-27T09:35:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f22800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c11a8e0 con 0x7f2f1c110960 2024-05-27T09:35:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f22800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c11abe0 con 0x7f2f1c10a980 2024-05-27T09:35:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f1b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1c10a980 0x7f2f1c1a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f1b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1c10a980 0x7f2f1c1a1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43932/0 (socket says 172.21.15.44:43932) 2024-05-27T09:35:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f1b400700 1 -- 172.21.15.44:0/1173239505 learned_addr learned my addr 172.21.15.44:0/1173239505 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f1be00700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f1c110960 0x7f2f1c1a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:55.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f12a00700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f1c10b2d0 0x7f2f1c1a1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:55.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f1b400700 1 -- 172.21.15.44:0/1173239505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f1c110960 msgr2=0x7f2f1c1a5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f1b400700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f1c110960 0x7f2f1c1a5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f1b400700 1 -- 172.21.15.44:0/1173239505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f1c10b2d0 msgr2=0x7f2f1c1a1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f1b400700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f1c10b2d0 0x7f2f1c1a1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f1b400700 1 -- 172.21.15.44:0/1173239505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1c1a6360 con 0x7f2f1c10a980 2024-05-27T09:35:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.211+0000 7f2f12a00700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f1c10b2d0 0x7f2f1c1a1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:35:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.212+0000 7f2f1b400700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1c10a980 0x7f2f1c1a1490 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009230 tx=0x7f2f0c00f990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.212+0000 7f2f18c00700 1 -- 172.21.15.44:0/1173239505 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c005140 con 0x7f2f1c10a980 2024-05-27T09:35:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.212+0000 7f2f18c00700 1 -- 172.21.15.44:0/1173239505 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f0c00b560 con 0x7f2f1c10a980 2024-05-27T09:35:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.212+0000 7f2f18c00700 1 -- 172.21.15.44:0/1173239505 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c003aa0 con 0x7f2f1c10a980 2024-05-27T09:35:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.212+0000 7f2f22800700 1 -- 172.21.15.44:0/1173239505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1c1a65f0 con 0x7f2f1c10a980 2024-05-27T09:35:55.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.212+0000 7f2f22800700 1 -- 172.21.15.44:0/1173239505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1c1a6a80 con 0x7f2f1c10a980 2024-05-27T09:35:55.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.213+0000 7f2f22800700 1 -- 172.21.15.44:0/1173239505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f1c0512b0 con 0x7f2f1c10a980 2024-05-27T09:35:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.216+0000 7f2f18c00700 1 -- 172.21.15.44:0/1173239505 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f2f0c00ba50 con 0x7f2f1c10a980 2024-05-27T09:35:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.218+0000 7f2f18c00700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2f000b2b10 0x7f2f000b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.218+0000 7f2f18c00700 1 -- 172.21.15.44:0/1173239505 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f2f0c0e3ca0 con 0x7f2f1c10a980 2024-05-27T09:35:55.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.218+0000 7f2f12a00700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2f000b2b10 0x7f2f000b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:55.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.219+0000 7f2f12a00700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2f000b2b10 0x7f2f000b5020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c07b0c0 tx=0x7f2f0400b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:55.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.220+0000 7f2f18c00700 1 -- 172.21.15.44:0/1173239505 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2f0c013030 con 0x7f2f1c10a980 2024-05-27T09:35:55.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.513+0000 7f2f22800700 1 -- 172.21.15.44:0/1173239505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f1c02e5e0 con 0x7f2f1c10a980 2024-05-27T09:35:55.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.514+0000 7f2f18c00700 1 -- 172.21.15.44:0/1173239505 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7f2f0c0ab490 con 0x7f2f1c10a980 2024-05-27T09:35:55.515 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:55.515 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:35:55.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.517+0000 7f2f22800700 1 -- 172.21.15.44:0/1173239505 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2f000b2b10 msgr2=0x7f2f000b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:55.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.517+0000 7f2f22800700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2f000b2b10 0x7f2f000b5020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c07b0c0 tx=0x7f2f0400b040 comp rx=0 tx=0).stop 2024-05-27T09:35:55.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.517+0000 7f2f22800700 1 -- 172.21.15.44:0/1173239505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1c10a980 msgr2=0x7f2f1c1a1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.517+0000 7f2f22800700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1c10a980 0x7f2f1c1a1490 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009230 tx=0x7f2f0c00f990 comp rx=0 tx=0).stop 2024-05-27T09:35:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.517+0000 7f2f22800700 1 -- 172.21.15.44:0/1173239505 shutdown_connections 2024-05-27T09:35:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.517+0000 7f2f22800700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2f000b2b10 0x7f2f000b5020 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.517+0000 7f2f22800700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f1c110960 0x7f2f1c1a5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.517+0000 7f2f22800700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1c10a980 0x7f2f1c1a1490 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.517+0000 7f2f22800700 1 --2- 172.21.15.44:0/1173239505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2f1c10b2d0 0x7f2f1c1a1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.517+0000 7f2f22800700 1 -- 172.21.15.44:0/1173239505 >> 172.21.15.44:0/1173239505 conn(0x7f2f1c077c70 msgr2=0x7f2f1c07bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.518+0000 7f2f22800700 1 -- 172.21.15.44:0/1173239505 shutdown_connections 2024-05-27T09:35:55.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:55.518+0000 7f2f22800700 1 -- 172.21.15.44:0/1173239505 wait complete. 2024-05-27T09:35:55.520 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:35:55.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:55 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1173239505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:55.900 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:55 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1173239505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:55 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1173239505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:35:56.591 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T09:35:56.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:56 smithi189 ceph-mon[36671]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:35:56.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:56 smithi044 ceph-mon[31618]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:35:57.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:56 smithi050 ceph-mon[36371]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:35:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:35:58.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:58 smithi044 ceph-mon[31618]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:35:58.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:58 smithi050 ceph-mon[36371]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:35:58.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:58 smithi189 ceph-mon[36671]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:35:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.211+0000 7f4108a00700 1 -- 172.21.15.44:0/2007490648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41041045d0 msgr2=0x7f41041049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.211+0000 7f4108a00700 1 --2- 172.21.15.44:0/2007490648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41041045d0 0x7f41041049e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009990 tx=0x7f40f4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:35:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.211+0000 7f4108a00700 1 -- 172.21.15.44:0/2007490648 shutdown_connections 2024-05-27T09:35:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.211+0000 7f4108a00700 1 --2- 172.21.15.44:0/2007490648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4104109850 0x7f41041111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.211+0000 7f4108a00700 1 --2- 172.21.15.44:0/2007490648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41041045d0 0x7f41041049e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.211+0000 7f4108a00700 1 --2- 172.21.15.44:0/2007490648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4104108f00 0x7f4104109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.211+0000 7f4108a00700 1 -- 172.21.15.44:0/2007490648 >> 172.21.15.44:0/2007490648 conn(0x7f41040ffec0 msgr2=0x7f4104102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.211+0000 7f4108a00700 1 -- 172.21.15.44:0/2007490648 shutdown_connections 2024-05-27T09:35:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.211+0000 7f4108a00700 1 -- 172.21.15.44:0/2007490648 wait complete. 2024-05-27T09:35:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.212+0000 7f4108a00700 1 Processor -- start 2024-05-27T09:35:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.213+0000 7f4108a00700 1 -- start start 2024-05-27T09:35:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.213+0000 7f4108a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41041045d0 0x7f410419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.213+0000 7f4108a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4104108f00 0x7f410419b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.213+0000 7f4108a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4104109850 0x7f410419f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.213+0000 7f4108a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f41041144d0 con 0x7f4104108f00 2024-05-27T09:35:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.213+0000 7f4108a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4104114350 con 0x7f41041045d0 2024-05-27T09:35:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.213+0000 7f4108a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4104114650 con 0x7f4104109850 2024-05-27T09:35:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.214+0000 7f4101e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41041045d0 0x7f410419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.214+0000 7f4101400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4104108f00 0x7f410419b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.214+0000 7f4101e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41041045d0 0x7f410419ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39786/0 (socket says 172.21.15.44:39786) 2024-05-27T09:35:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.214+0000 7f4101e00700 1 -- 172.21.15.44:0/2903234916 learned_addr learned my addr 172.21.15.44:0/2903234916 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:35:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.214+0000 7f4102800700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4104109850 0x7f410419f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.214+0000 7f4101e00700 1 -- 172.21.15.44:0/2903234916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4104109850 msgr2=0x7f410419f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.214+0000 7f4101e00700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4104109850 0x7f410419f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.214+0000 7f4101e00700 1 -- 172.21.15.44:0/2903234916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4104108f00 msgr2=0x7f410419b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.214+0000 7f4101e00700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4104108f00 0x7f410419b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.214+0000 7f4101e00700 1 -- 172.21.15.44:0/2903234916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f410419fcf0 con 0x7f41041045d0 2024-05-27T09:35:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.214+0000 7f4101e00700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41041045d0 0x7f410419ae20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f40f4000c00 tx=0x7f40f40057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:59.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.215+0000 7f40faa00700 1 -- 172.21.15.44:0/2903234916 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4027070 con 0x7f41041045d0 2024-05-27T09:35:59.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.215+0000 7f4108a00700 1 -- 172.21.15.44:0/2903234916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f410419ff80 con 0x7f41041045d0 2024-05-27T09:35:59.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.215+0000 7f40faa00700 1 -- 172.21.15.44:0/2903234916 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40f400b610 con 0x7f41041045d0 2024-05-27T09:35:59.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.215+0000 7f40faa00700 1 -- 172.21.15.44:0/2903234916 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f40173f0 con 0x7f41041045d0 2024-05-27T09:35:59.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.215+0000 7f4108a00700 1 -- 172.21.15.44:0/2903234916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41041a7900 con 0x7f41041045d0 2024-05-27T09:35:59.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.216+0000 7f4108a00700 1 -- 172.21.15.44:0/2903234916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41040512b0 con 0x7f41041045d0 2024-05-27T09:35:59.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.219+0000 7f40faa00700 1 -- 172.21.15.44:0/2903234916 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7f40f402a070 con 0x7f41041045d0 2024-05-27T09:35:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.220+0000 7f40faa00700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40e80b2b10 0x7f40e80b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:35:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.221+0000 7f40faa00700 1 -- 172.21.15.44:0/2903234916 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f40f40e3aa0 con 0x7f41041045d0 2024-05-27T09:35:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.221+0000 7f4101400700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40e80b2b10 0x7f40e80b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:35:59.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.222+0000 7f4101400700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40e80b2b10 0x7f40e80b5020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4104107d60 tx=0x7f40f0009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:35:59.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.223+0000 7f40faa00700 1 -- 172.21.15.44:0/2903234916 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f40f405fdf0 con 0x7f41041045d0 2024-05-27T09:35:59.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.462+0000 7f4108a00700 1 -- 172.21.15.44:0/2903234916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f410402e5e0 con 0x7f41041045d0 2024-05-27T09:35:59.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.465+0000 7f40faa00700 1 -- 172.21.15.44:0/2903234916 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+22988 (secure 0 0 0) 0x7f40f4013070 con 0x7f41041045d0 2024-05-27T09:35:59.466 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:35:59.466 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:35:14.029489+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6800/4077502151":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/2278564126":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/4102964244":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/408258823":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3716545176":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/4077502151":"2024-05-28T09:35:14.029409+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:35:59.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.468+0000 7f4108a00700 1 -- 172.21.15.44:0/2903234916 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40e80b2b10 msgr2=0x7f40e80b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:59.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.468+0000 7f4108a00700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40e80b2b10 0x7f40e80b5020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4104107d60 tx=0x7f40f0009260 comp rx=0 tx=0).stop 2024-05-27T09:35:59.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.468+0000 7f4108a00700 1 -- 172.21.15.44:0/2903234916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41041045d0 msgr2=0x7f410419ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:35:59.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.468+0000 7f4108a00700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41041045d0 0x7f410419ae20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f40f4000c00 tx=0x7f40f40057c0 comp rx=0 tx=0).stop 2024-05-27T09:35:59.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.468+0000 7f4108a00700 1 -- 172.21.15.44:0/2903234916 shutdown_connections 2024-05-27T09:35:59.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.468+0000 7f4108a00700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40e80b2b10 0x7f40e80b5020 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:59.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.468+0000 7f4108a00700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4104109850 0x7f410419f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:59.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.468+0000 7f4108a00700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41041045d0 0x7f410419ae20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:59.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.468+0000 7f4108a00700 1 --2- 172.21.15.44:0/2903234916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4104108f00 0x7f410419b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:35:59.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.468+0000 7f4108a00700 1 -- 172.21.15.44:0/2903234916 >> 172.21.15.44:0/2903234916 conn(0x7f41040ffec0 msgr2=0x7f41041086e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:35:59.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.469+0000 7f4108a00700 1 -- 172.21.15.44:0/2903234916 shutdown_connections 2024-05-27T09:35:59.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:35:59.469+0000 7f4108a00700 1 -- 172.21.15.44:0/2903234916 wait complete. 2024-05-27T09:35:59.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:35:59 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2903234916' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:35:59.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:35:59 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2903234916' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:35:59.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:35:59 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2903234916' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:36:00.380 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-05-27T09:36:00.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:00 smithi044 ceph-mon[31618]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:36:00.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:00 smithi050 ceph-mon[36371]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:36:00.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:00 smithi189 ceph-mon[36671]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T09:36:00.855 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:01.223 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:36:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:36:00.913+0000 7f498f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:36:01.223 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:36:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:36:00.913+0000 7f498f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:36:01.223 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:36:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:36:00.913+0000 7f498f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:36:01.223 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:36:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:36:00.913+0000 7f498f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:36:01.224 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:36:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:36:00.913+0000 7f498f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T09:36:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.823+0000 7fbe6d600640 1 -- 172.21.15.44:0/3994648260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe68110520 msgr2=0x7fbe6810a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.823+0000 7fbe6d600640 1 --2- 172.21.15.44:0/3994648260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe68110520 0x7fbe6810a200 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbe50009860 tx=0x7fbe5002a3d0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.823+0000 7fbe6d600640 1 -- 172.21.15.44:0/3994648260 shutdown_connections 2024-05-27T09:36:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.823+0000 7fbe6d600640 1 --2- 172.21.15.44:0/3994648260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe6810a890 0x7fbe68116b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.823+0000 7fbe6d600640 1 --2- 172.21.15.44:0/3994648260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe68110520 0x7fbe6810a200 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.823+0000 7fbe6d600640 1 --2- 172.21.15.44:0/3994648260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe68110110 0x7fbe68109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.823+0000 7fbe6d600640 1 -- 172.21.15.44:0/3994648260 >> 172.21.15.44:0/3994648260 conn(0x7fbe68076070 msgr2=0x7fbe68078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.823+0000 7fbe6d600640 1 -- 172.21.15.44:0/3994648260 shutdown_connections 2024-05-27T09:36:01.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.823+0000 7fbe6d600640 1 -- 172.21.15.44:0/3994648260 wait complete. 2024-05-27T09:36:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.824+0000 7fbe6d600640 1 Processor -- start 2024-05-27T09:36:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.824+0000 7fbe6d600640 1 -- start start 2024-05-27T09:36:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.825+0000 7fbe6d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe68110110 0x7fbe681a6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.825+0000 7fbe6d600640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe681a6ba0 0x7fbe681a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.825+0000 7fbe6d600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe681a0d10 0x7fbe681a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.825+0000 7fbe6d600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6811a2e0 con 0x7fbe681a0d10 2024-05-27T09:36:01.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.825+0000 7fbe6d600640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6811a160 con 0x7fbe681a6ba0 2024-05-27T09:36:01.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.825+0000 7fbe6d600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6811a460 con 0x7fbe68110110 2024-05-27T09:36:01.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.825+0000 7fbe67e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe68110110 0x7fbe681a6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.825+0000 7fbe67e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe68110110 0x7fbe681a6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43974/0 (socket says 172.21.15.44:43974) 2024-05-27T09:36:01.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.825+0000 7fbe67e00640 1 -- 172.21.15.44:0/470404220 learned_addr learned my addr 172.21.15.44:0/470404220 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:01.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.825+0000 7fbe67400640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe681a6ba0 0x7fbe681a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.825+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe681a0d10 0x7fbe681a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.826+0000 7fbe67e00640 1 -- 172.21.15.44:0/470404220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe681a6ba0 msgr2=0x7fbe681a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:01.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.826+0000 7fbe67e00640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe681a6ba0 0x7fbe681a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.826+0000 7fbe67e00640 1 -- 172.21.15.44:0/470404220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe681a0d10 msgr2=0x7fbe681a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.826+0000 7fbe67e00640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe681a0d10 0x7fbe681a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.826+0000 7fbe67e00640 1 -- 172.21.15.44:0/470404220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe681ad3f0 con 0x7fbe68110110 2024-05-27T09:36:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.826+0000 7fbe67e00640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe68110110 0x7fbe681a6660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800e9f0 tx=0x7fbe5800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.826+0000 7fbe64c00640 1 -- 172.21.15.44:0/470404220 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800ccc0 con 0x7fbe68110110 2024-05-27T09:36:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.827+0000 7fbe64c00640 1 -- 172.21.15.44:0/470404220 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe580035f0 con 0x7fbe68110110 2024-05-27T09:36:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.827+0000 7fbe6d600640 1 -- 172.21.15.44:0/470404220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe681ad680 con 0x7fbe68110110 2024-05-27T09:36:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.827+0000 7fbe64c00640 1 -- 172.21.15.44:0/470404220 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800ce60 con 0x7fbe68110110 2024-05-27T09:36:01.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.827+0000 7fbe6d600640 1 -- 172.21.15.44:0/470404220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe681adba0 con 0x7fbe68110110 2024-05-27T09:36:01.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.831+0000 7fbe75ba5740 1 Processor -- start 2024-05-27T09:36:01.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.831+0000 7fbe75ba5740 1 -- start start 2024-05-27T09:36:01.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.832+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93318b1b0 0x55a93352da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.832+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9331622f0 0x55a93352dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.832+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93316a4d0 0x55a93352e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.832+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a93349eb90 con 0x55a93318b1b0 2024-05-27T09:36:01.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.832+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a93348e730 con 0x55a93316a4d0 2024-05-27T09:36:01.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.832+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a933492770 con 0x55a9331622f0 2024-05-27T09:36:01.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.832+0000 7fbe6cc00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93318b1b0 0x55a93352da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.832+0000 7fbe6cc00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93318b1b0 0x55a93352da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56818/0 (socket says 172.21.15.44:56818) 2024-05-27T09:36:01.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.832+0000 7fbe6cc00640 1 -- 172.21.15.44:0/3898798615 learned_addr learned my addr 172.21.15.44:0/3898798615 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:01.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.832+0000 7fbe67400640 1 --2- 172.21.15.44:0/3898798615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93316a4d0 0x55a93352e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.832+0000 7fbe64c00640 1 -- 172.21.15.44:0/470404220 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 149879+0+0 (secure 0 0 0) 0x7fbe58004000 con 0x7fbe68110110 2024-05-27T09:36:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.833+0000 7fbe6cc00640 1 -- 172.21.15.44:0/3898798615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9331622f0 msgr2=0x55a93352dfa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:36:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.833+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/3898798615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9331622f0 0x55a93352dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.833+0000 7fbe6cc00640 1 -- 172.21.15.44:0/3898798615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93316a4d0 msgr2=0x55a93352e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.833+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/3898798615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93316a4d0 0x55a93352e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.833+0000 7fbe6cc00640 1 -- 172.21.15.44:0/3898798615 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9334278b0 con 0x55a93318b1b0 2024-05-27T09:36:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.833+0000 7fbe67400640 1 --2- 172.21.15.44:0/3898798615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93316a4d0 0x55a93352e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.833+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/3898798615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93318b1b0 0x55a93352da60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbe60011440 tx=0x7fbe60011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59dfe1f58b595440 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:01.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.833+0000 7fbe5d200640 1 -- 172.21.15.44:0/3898798615 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60003c30 con 0x55a93318b1b0 2024-05-27T09:36:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.834+0000 7fbe5d200640 1 -- 172.21.15.44:0/3898798615 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe60003dd0 con 0x55a93318b1b0 2024-05-27T09:36:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.834+0000 7fbe5d200640 1 -- 172.21.15.44:0/3898798615 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6000fae0 con 0x55a93318b1b0 2024-05-27T09:36:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.834+0000 7fbe64c00640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe4c0b2a70 0x7fbe4c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe75ba5740 1 -- 172.21.15.44:0/3898798615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93318b1b0 msgr2=0x55a93352da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/3898798615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93318b1b0 0x55a93352da60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbe60011440 tx=0x7fbe60011900 comp rx=0 tx=0).stop 2024-05-27T09:36:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe67e00640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe4c0b2a70 0x7fbe4c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe64c00640 1 -- 172.21.15.44:0/470404220 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbe580e2ad0 con 0x7fbe68110110 2024-05-27T09:36:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe75ba5740 1 -- 172.21.15.44:0/3898798615 shutdown_connections 2024-05-27T09:36:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/3898798615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93316a4d0 0x55a93352e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/3898798615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9331622f0 0x55a93352dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/3898798615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93318b1b0 0x55a93352da60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe75ba5740 1 -- 172.21.15.44:0/3898798615 >> 172.21.15.44:0/3898798615 conn(0x55a93317f300 msgr2=0x55a93352a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe75ba5740 1 -- 172.21.15.44:0/3898798615 shutdown_connections 2024-05-27T09:36:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe75ba5740 1 -- 172.21.15.44:0/3898798615 wait complete. 2024-05-27T09:36:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.835+0000 7fbe67e00640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe4c0b2a70 0x7fbe4c0b4f80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbe580e0130 tx=0x7fbe58007cc0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.836+0000 7fbe75ba5740 1 -- start start 2024-05-27T09:36:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.836+0000 7fbe75ba5740 1 Processor -- start 2024-05-27T09:36:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a9331622f0 0x55a933537c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93316a4d0 0x55a9335381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93318b1b0 0x55a933543ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a933174790 con 0x55a93316a4d0 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a933436bc0 con 0x55a9331622f0 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a93353ab90 con 0x55a93318b1b0 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe75ba5740 10 client.0 _ll_register_callbacks cb 0x55a933414400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe6cc00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a9331622f0 0x55a933537c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe67400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93316a4d0 0x55a9335381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe6cc00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a9331622f0 0x55a933537c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39846/0 (socket says 172.21.15.44:39846) 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe67400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93316a4d0 0x55a9335381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56832/0 (socket says 172.21.15.44:56832) 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe6cc00640 1 -- 172.21.15.44:0/748254136 learned_addr learned my addr 172.21.15.44:0/748254136 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93318b1b0 0x55a933543ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.837+0000 7fbe67400640 1 -- 172.21.15.44:0/748254136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93318b1b0 msgr2=0x55a933543ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.838+0000 7fbe67400640 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93318b1b0 0x55a933543ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.838+0000 7fbe67400640 1 -- 172.21.15.44:0/748254136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a9331622f0 msgr2=0x55a933537c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:01.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.838+0000 7fbe67400640 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a9331622f0 0x55a933537c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.838+0000 7fbe67400640 1 -- 172.21.15.44:0/748254136 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9331a3210 con 0x55a93316a4d0 2024-05-27T09:36:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.838+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a9331622f0 0x55a933537c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.838+0000 7fbe67400640 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93316a4d0 0x55a9335381a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbe6810adc0 tx=0x7fbe50032fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a584d257d00a4ac3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:01.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.838+0000 7fbe55600640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:36:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.838+0000 7fbe55600640 1 -- 172.21.15.44:0/748254136 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe50005240 con 0x55a93316a4d0 2024-05-27T09:36:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.838+0000 7fbe55600640 1 -- 172.21.15.44:0/748254136 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbe500053e0 con 0x55a93316a4d0 2024-05-27T09:36:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.838+0000 7fbe75ba5740 1 -- 172.21.15.44:0/748254136 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe60003dd0 con 0x55a93316a4d0 2024-05-27T09:36:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.839+0000 7fbe55600640 1 -- 172.21.15.44:0/748254136 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbe500055e0 con 0x55a93316a4d0 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.839+0000 7fbe75ba5740 10 client.15198 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.839+0000 7fbe75ba5740 1 -- 172.21.15.44:0/748254136 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a93326f620 con 0x55a93316a4d0 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.839+0000 7fbe55600640 1 -- 172.21.15.44:0/748254136 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe60003dd0 con 0x55a93316a4d0 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.839+0000 7fbe55600640 1 -- 172.21.15.44:0/748254136 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbe5004f750 con 0x55a93316a4d0 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.839+0000 7fbe75ba5740 10 client.15198 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.839+0000 7fbe75ba5740 10 client.15198 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.839+0000 7fbe75ba5740 20 client.15198 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.839+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a933149d80 0x55a933549e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.839+0000 7fbe75ba5740 4 client.15198 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.839+0000 7fbe75ba5740 1 -- 172.21.15.44:0/748254136 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9334a0ae0 con 0x55a933149d80 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.840+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a933149d80 0x55a933549e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.840+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a933149d80 0x55a933549e70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb3b2b23465e06 server_cookie=d76d9f635e436aba in_seq=0 out_seq=0 2024-05-27T09:36:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.840+0000 7fbe55600640 10 client.15198 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe55600640 1 -- 172.21.15.44:0/748254136 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a9334a0ae0 con 0x55a933149d80 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe55600640 10 client.15198 handle_command_reply: tid=0 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe75ba5740 1 client.15198 shutdown 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe75ba5740 1 -- 172.21.15.44:0/748254136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93316a4d0 msgr2=0x55a9335381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93316a4d0 0x55a9335381a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbe6810adc0 tx=0x7fbe50032fb0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe75ba5740 1 -- 172.21.15.44:0/748254136 shutdown_connections 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a933149d80 0x55a933549e70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93318b1b0 0x55a933543ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a93316a4d0 0x55a9335381a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/748254136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a9331622f0 0x55a933537c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe55600640 0 client.15198 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.841+0000 7fbe75ba5740 1 -- 172.21.15.44:0/748254136 >> 172.21.15.44:0/748254136 conn(0x55a93317f300 msgr2=0x55a93352d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.842+0000 7fbe75ba5740 1 -- 172.21.15.44:0/748254136 shutdown_connections 2024-05-27T09:36:01.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.842+0000 7fbe75ba5740 1 -- 172.21.15.44:0/748254136 wait complete. 2024-05-27T09:36:01.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.842+0000 7fbe75ba5740 20 client.15198 trim_cache size 0 max 16384 2024-05-27T09:36:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.858+0000 7fbe75ba5740 1 Processor -- start 2024-05-27T09:36:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.858+0000 7fbe75ba5740 1 -- start start 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.859+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a933533440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.859+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93316a4d0 0x55a933529240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.859+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93318b1b0 0x55a93352df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.859+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681b8d30 con 0x55a9331622f0 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.859+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681b6600 con 0x55a93316a4d0 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.859+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681b4450 con 0x55a93318b1b0 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.859+0000 7fbe6cc00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a933533440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.859+0000 7fbe6cc00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a933533440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56844/0 (socket says 172.21.15.44:56844) 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.859+0000 7fbe6cc00640 1 -- 172.21.15.44:0/1536735684 learned_addr learned my addr 172.21.15.44:0/1536735684 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.859+0000 7fbe67400640 1 --2- 172.21.15.44:0/1536735684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93316a4d0 0x55a933529240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.859+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/1536735684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93318b1b0 0x55a93352df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.860+0000 7fbe6cc00640 1 -- 172.21.15.44:0/1536735684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93318b1b0 msgr2=0x55a93352df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:01.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.860+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/1536735684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93318b1b0 0x55a93352df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.860+0000 7fbe6cc00640 1 -- 172.21.15.44:0/1536735684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93316a4d0 msgr2=0x55a933529240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:01.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.860+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/1536735684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93316a4d0 0x55a933529240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.860+0000 7fbe6cc00640 1 -- 172.21.15.44:0/1536735684 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a93354a1e0 con 0x55a9331622f0 2024-05-27T09:36:01.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.860+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/1536735684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a933533440 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbe60011e00 tx=0x7fbe60009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d61c68526da234b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:01.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.860+0000 7fbe57e00640 1 -- 172.21.15.44:0/1536735684 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6000f940 con 0x55a9331622f0 2024-05-27T09:36:01.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.860+0000 7fbe57e00640 1 -- 172.21.15.44:0/1536735684 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe6002c020 con 0x55a9331622f0 2024-05-27T09:36:01.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.860+0000 7fbe57e00640 1 -- 172.21.15.44:0/1536735684 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60028450 con 0x55a9331622f0 2024-05-27T09:36:01.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.861+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1536735684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 msgr2=0x55a933533440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:01.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.861+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/1536735684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a933533440 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbe60011e00 tx=0x7fbe60009500 comp rx=0 tx=0).stop 2024-05-27T09:36:01.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.861+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1536735684 shutdown_connections 2024-05-27T09:36:01.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.861+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/1536735684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93318b1b0 0x55a93352df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.861+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/1536735684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93316a4d0 0x55a933529240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.861+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/1536735684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a933533440 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.861+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1536735684 >> 172.21.15.44:0/1536735684 conn(0x55a93317f300 msgr2=0x55a93352d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:01.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.861+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1536735684 shutdown_connections 2024-05-27T09:36:01.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.861+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1536735684 wait complete. 2024-05-27T09:36:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.862+0000 7fbe75ba5740 1 -- start start 2024-05-27T09:36:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.862+0000 7fbe75ba5740 1 Processor -- start 2024-05-27T09:36:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.862+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a93353ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.862+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93316a4d0 0x55a93353b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.862+0000 7fbe75ba5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93318b1b0 0x55a93353b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.862+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681b4ac0 con 0x55a9331622f0 2024-05-27T09:36:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.862+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681b5f30 con 0x55a93318b1b0 2024-05-27T09:36:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.862+0000 7fbe75ba5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681b5190 con 0x55a93316a4d0 2024-05-27T09:36:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.862+0000 7fbe75ba5740 10 client.0 _ll_register_callbacks cb 0x7fbe500096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.863+0000 7fbe6cc00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a93353ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.863+0000 7fbe6cc00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a93353ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56850/0 (socket says 172.21.15.44:56850) 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.863+0000 7fbe6cc00640 1 -- 172.21.15.44:0/1858230097 learned_addr learned my addr 172.21.15.44:0/1858230097 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.863+0000 7fbe67400640 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93316a4d0 0x55a93353b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.863+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93318b1b0 0x55a93353b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.863+0000 7fbe67400640 1 -- 172.21.15.44:0/1858230097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93318b1b0 msgr2=0x55a93353b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.863+0000 7fbe67400640 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93318b1b0 0x55a93353b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.863+0000 7fbe67400640 1 -- 172.21.15.44:0/1858230097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 msgr2=0x55a93353ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.863+0000 7fbe67400640 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a93353ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.863+0000 7fbe67400640 1 -- 172.21.15.44:0/1858230097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9335b06c0 con 0x55a93316a4d0 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.863+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a93353ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.864+0000 7fbe67400640 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93316a4d0 0x55a93353b360 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe5004fb40 tx=0x7fbe50030bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a9a4b0064fbaf0e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.864+0000 7fbe55600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:36:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.864+0000 7fbe55600640 1 -- 172.21.15.44:0/1858230097 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5004f860 con 0x55a93316a4d0 2024-05-27T09:36:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.864+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1858230097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe6002c020 con 0x55a93316a4d0 2024-05-27T09:36:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.864+0000 7fbe55600640 1 -- 172.21.15.44:0/1858230097 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbe50004030 con 0x55a93316a4d0 2024-05-27T09:36:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.864+0000 7fbe55600640 1 -- 172.21.15.44:0/1858230097 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbe5005f050 con 0x55a93316a4d0 2024-05-27T09:36:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.864+0000 7fbe75ba5740 10 client.25345 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.864+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1858230097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a9335b1910 con 0x55a93316a4d0 2024-05-27T09:36:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.865+0000 7fbe55600640 1 -- 172.21.15.44:0/1858230097 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe6002c020 con 0x55a93316a4d0 2024-05-27T09:36:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.865+0000 7fbe55600640 1 -- 172.21.15.44:0/1858230097 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbe50003ca0 con 0x55a93316a4d0 2024-05-27T09:36:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.865+0000 7fbe75ba5740 10 client.25345 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.865+0000 7fbe75ba5740 10 client.25345 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.865+0000 7fbe75ba5740 20 client.25345 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.865+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a93311f1c0 0x55a9335b4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.865+0000 7fbe75ba5740 4 client.25345 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:36:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.865+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1858230097 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a93349a2c0 con 0x55a93311f1c0 2024-05-27T09:36:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.865+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a93311f1c0 0x55a9335b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.866+0000 7fbe6cc00640 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a93311f1c0 0x55a9335b4150 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6faa94cb11749e77 server_cookie=9812ec5da4722a3 in_seq=0 out_seq=0 2024-05-27T09:36:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.866+0000 7fbe55600640 10 client.25345 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:01.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.875+0000 7fbe55600640 1 -- 172.21.15.44:0/1858230097 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a93349a2c0 con 0x55a93311f1c0 2024-05-27T09:36:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.875+0000 7fbe55600640 10 client.25345 handle_command_reply: tid=0 2024-05-27T09:36:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.875+0000 7fbe75ba5740 1 client.25345 shutdown 2024-05-27T09:36:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1858230097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93316a4d0 msgr2=0x55a93353b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93316a4d0 0x55a93353b360 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe5004fb40 tx=0x7fbe50030bb0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1858230097 shutdown_connections 2024-05-27T09:36:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a93311f1c0 0x55a9335b4150 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a93318b1b0 0x55a93353b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe55600640 0 client.25345 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a93316a4d0 0x55a93353b360 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe75ba5740 1 --2- 172.21.15.44:0/1858230097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a9331622f0 0x55a93353ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1858230097 >> 172.21.15.44:0/1858230097 conn(0x55a93317f300 msgr2=0x55a933528fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1858230097 shutdown_connections 2024-05-27T09:36:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe75ba5740 1 -- 172.21.15.44:0/1858230097 wait complete. 2024-05-27T09:36:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.876+0000 7fbe75ba5740 20 client.25345 trim_cache size 0 max 16384 2024-05-27T09:36:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:36:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:36:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:36:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 -- 172.21.15.44:0/470404220 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe4c0b2a70 msgr2=0x7fbe4c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe4c0b2a70 0x7fbe4c0b4f80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbe580e0130 tx=0x7fbe58007cc0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 -- 172.21.15.44:0/470404220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe68110110 msgr2=0x7fbe681a6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe68110110 0x7fbe681a6660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800e9f0 tx=0x7fbe5800edf0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 -- 172.21.15.44:0/470404220 shutdown_connections 2024-05-27T09:36:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe4c0b2a70 0x7fbe4c0b4f80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe681a0d10 0x7fbe681a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe681a6ba0 0x7fbe681a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 --2- 172.21.15.44:0/470404220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe68110110 0x7fbe681a6660 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 -- 172.21.15.44:0/470404220 >> 172.21.15.44:0/470404220 conn(0x7fbe68076070 msgr2=0x7fbe68115770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 -- 172.21.15.44:0/470404220 shutdown_connections 2024-05-27T09:36:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:01.877+0000 7fbe6d600640 1 -- 172.21.15.44:0/470404220 wait complete. 2024-05-27T09:36:03.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:03 smithi044 ceph-mon[31618]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:36:03.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:03 smithi044 ceph-mon[31618]: mgrmap e26: x(active, since 48s), standbys: y, z 2024-05-27T09:36:03.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:03 smithi050 ceph-mon[36371]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:36:03.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:03 smithi050 ceph-mon[36371]: mgrmap e26: x(active, since 48s), standbys: y, z 2024-05-27T09:36:03.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:03 smithi189 ceph-mon[36671]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-27T09:36:03.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:03 smithi189 ceph-mon[36671]: mgrmap e26: x(active, since 48s), standbys: y, z 2024-05-27T09:36:04.148 INFO:teuthology.orchestra.run.smithi044.stdout:{"run_dir":"/var/run/ceph"} 2024-05-27T09:36:04.149 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-05-27T09:36:04.149 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 2274 2024-05-27T09:36:04.150 DEBUG:tasks.quiescer:Yielding 2024-05-27T09:36:04.150 INFO:teuthology.run_tasks:Running task workunit... 2024-05-27T09:36:04.165 INFO:tasks.workunit:Pulling workunits from ref a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:36:04.166 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-27T09:36:04.166 INFO:tasks.workunit:timeout=3h 2024-05-27T09:36:04.166 INFO:tasks.workunit:cleanup=True 2024-05-27T09:36:04.166 DEBUG:teuthology.orchestra.run.smithi044:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-27T09:36:04.167 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:36:04.222 INFO:teuthology.orchestra.run.smithi044.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-27T09:36:04.222 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-27T09:36:04.222 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-05-27T09:36:04.222 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T09:36:04.222 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-27T09:36:04.223 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-05-27 09:32:05.460182274 +0000 2024-05-27T09:36:04.223 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-05-27 09:32:05.460182274 +0000 2024-05-27T09:36:04.223 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-05-27 09:35:11.283120420 +0000 2024-05-27T09:36:04.223 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-05-27 09:32:05.460182274 +0000 2024-05-27T09:36:04.223 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-27T09:36:04.223 DEBUG:teuthology.orchestra.run.smithi044:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-27T09:36:04.333 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:04 smithi044 ceph-mon[31618]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:36:04.361 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a6507388dd5057528934822c0163b0c347ef1d5d 2024-05-27T09:36:04.417 INFO:tasks.workunit.client.0.smithi044.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-27T09:36:04.445 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:04 smithi050 ceph-mon[36371]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:36:04.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:04 smithi189 ceph-mon[36671]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:36:04.614 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:06.459 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:06 smithi044 ceph-mon[31618]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-27T09:36:06.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:06 smithi050 ceph-mon[36371]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-27T09:36:06.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:06 smithi189 ceph-mon[36671]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-27T09:36:06.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.858+0000 7fbe2d600700 1 -- 172.21.15.44:0/4012328995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe2810b2d0 msgr2=0x7fbe28117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:06.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.858+0000 7fbe2d600700 1 --2- 172.21.15.44:0/4012328995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe2810b2d0 0x7fbe28117800 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000b0a0 tx=0x7fbe2000b3b0 comp rx=0 tx=0).stop 2024-05-27T09:36:06.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.858+0000 7fbe2d600700 1 -- 172.21.15.44:0/4012328995 shutdown_connections 2024-05-27T09:36:06.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.858+0000 7fbe2d600700 1 --2- 172.21.15.44:0/4012328995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe2810b2d0 0x7fbe28117800 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:06.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.858+0000 7fbe2d600700 1 --2- 172.21.15.44:0/4012328995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe28110960 0x7fbe28110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:06.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.858+0000 7fbe2d600700 1 --2- 172.21.15.44:0/4012328995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe2810a980 0x7fbe2810ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:06.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.858+0000 7fbe2d600700 1 -- 172.21.15.44:0/4012328995 >> 172.21.15.44:0/4012328995 conn(0x7fbe28077c70 msgr2=0x7fbe280780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:06.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.858+0000 7fbe2d600700 1 -- 172.21.15.44:0/4012328995 shutdown_connections 2024-05-27T09:36:06.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.858+0000 7fbe2d600700 1 -- 172.21.15.44:0/4012328995 wait complete. 2024-05-27T09:36:06.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.859+0000 7fbe2d600700 1 Processor -- start 2024-05-27T09:36:06.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.860+0000 7fbe2d600700 1 -- start start 2024-05-27T09:36:06.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.860+0000 7fbe2d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe2810a980 0x7fbe281a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:06.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.860+0000 7fbe2d600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe2810b2d0 0x7fbe281a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:06.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.860+0000 7fbe2d600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe28110960 0x7fbe281a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:06.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.860+0000 7fbe2d600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2811aa80 con 0x7fbe2810b2d0 2024-05-27T09:36:06.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.860+0000 7fbe2d600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2811a900 con 0x7fbe28110960 2024-05-27T09:36:06.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.860+0000 7fbe2d600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2811ac00 con 0x7fbe2810a980 2024-05-27T09:36:06.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.861+0000 7fbe27400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe28110960 0x7fbe281a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:06.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.861+0000 7fbe26000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe2810b2d0 0x7fbe281a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:06.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.861+0000 7fbe26a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe2810a980 0x7fbe281a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:06.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.861+0000 7fbe27400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe28110960 0x7fbe281a5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51734/0 (socket says 172.21.15.44:51734) 2024-05-27T09:36:06.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.861+0000 7fbe27400700 1 -- 172.21.15.44:0/3619302436 learned_addr learned my addr 172.21.15.44:0/3619302436 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.861+0000 7fbe27400700 1 -- 172.21.15.44:0/3619302436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe2810a980 msgr2=0x7fbe281a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.861+0000 7fbe27400700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe2810a980 0x7fbe281a1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.861+0000 7fbe27400700 1 -- 172.21.15.44:0/3619302436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe2810b2d0 msgr2=0x7fbe281a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.861+0000 7fbe27400700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe2810b2d0 0x7fbe281a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.861+0000 7fbe27400700 1 -- 172.21.15.44:0/3619302436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe281a62e0 con 0x7fbe28110960 2024-05-27T09:36:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.861+0000 7fbe26a00700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe2810a980 0x7fbe281a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.862+0000 7fbe27400700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe28110960 0x7fbe281a5c40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbe200062a0 tx=0x7fbe20004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.862+0000 7fbe1f400700 1 -- 172.21.15.44:0/3619302436 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20021070 con 0x7fbe28110960 2024-05-27T09:36:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.862+0000 7fbe1f400700 1 -- 172.21.15.44:0/3619302436 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe20007e30 con 0x7fbe28110960 2024-05-27T09:36:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.862+0000 7fbe2d600700 1 -- 172.21.15.44:0/3619302436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe281a6570 con 0x7fbe28110960 2024-05-27T09:36:06.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.862+0000 7fbe1f400700 1 -- 172.21.15.44:0/3619302436 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe200046d0 con 0x7fbe28110960 2024-05-27T09:36:06.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.862+0000 7fbe2d600700 1 -- 172.21.15.44:0/3619302436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe281adef0 con 0x7fbe28110960 2024-05-27T09:36:06.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.863+0000 7fbe2d600700 1 -- 172.21.15.44:0/3619302436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe2810c070 con 0x7fbe28110960 2024-05-27T09:36:06.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.866+0000 7fbe1f400700 1 -- 172.21.15.44:0/3619302436 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7fbe20007980 con 0x7fbe28110960 2024-05-27T09:36:06.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.867+0000 7fbe1f400700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe0c0b2ba0 0x7fbe0c0b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:06.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.868+0000 7fbe1f400700 1 -- 172.21.15.44:0/3619302436 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbe2000f020 con 0x7fbe28110960 2024-05-27T09:36:06.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.868+0000 7fbe26a00700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe0c0b2ba0 0x7fbe0c0b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:06.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.868+0000 7fbe26a00700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe0c0b2ba0 0x7fbe0c0b50b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbe2807d0e0 tx=0x7fbe18009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:06.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:06.870+0000 7fbe1f400700 1 -- 172.21.15.44:0/3619302436 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbe2002a020 con 0x7fbe28110960 2024-05-27T09:36:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.158+0000 7fbe2d600700 1 -- 172.21.15.44:0/3619302436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe280512b0 con 0x7fbe28110960 2024-05-27T09:36:07.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.159+0000 7fbe1f400700 1 -- 172.21.15.44:0/3619302436 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7fbe20004870 con 0x7fbe28110960 2024-05-27T09:36:07.160 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:36:07.161 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:36:07.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 -- 172.21.15.44:0/3619302436 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe0c0b2ba0 msgr2=0x7fbe0c0b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:07.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe0c0b2ba0 0x7fbe0c0b50b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbe2807d0e0 tx=0x7fbe18009260 comp rx=0 tx=0).stop 2024-05-27T09:36:07.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 -- 172.21.15.44:0/3619302436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe28110960 msgr2=0x7fbe281a5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:07.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe28110960 0x7fbe281a5c40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbe200062a0 tx=0x7fbe20004c80 comp rx=0 tx=0).stop 2024-05-27T09:36:07.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 -- 172.21.15.44:0/3619302436 shutdown_connections 2024-05-27T09:36:07.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbe0c0b2ba0 0x7fbe0c0b50b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:07.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe28110960 0x7fbe281a5c40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:07.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe2810a980 0x7fbe281a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:07.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 --2- 172.21.15.44:0/3619302436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe2810b2d0 0x7fbe281a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:07.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 -- 172.21.15.44:0/3619302436 >> 172.21.15.44:0/3619302436 conn(0x7fbe28077c70 msgr2=0x7fbe2807b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:07.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 -- 172.21.15.44:0/3619302436 shutdown_connections 2024-05-27T09:36:07.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:07.163+0000 7fbe2d600700 1 -- 172.21.15.44:0/3619302436 wait complete. 2024-05-27T09:36:07.165 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:36:07.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:07 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3619302436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:36:07.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:07 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3619302436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:36:07.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:07 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3619302436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:36:07.956 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-27T09:31:27.747151+0000', 'modified': '2024-05-27T09:35:12.426904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24542, 'mds_1': 24535, 'mds_2': 24520}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24491': {'gid': 24491, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.50:6835/1092513236', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6834', 'nonce': 1092513236}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 1092513236}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.50:6837/3540954143', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6836', 'nonce': 3540954143}, {'type': 'v1', 'addr': '172.21.15.50:6837', 'nonce': 3540954143}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6835/2673083839', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 2673083839}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 2673083839}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.189:6837/2265427173', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 2265427173}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 2265427173}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.189:6839/4109192600', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 4109192600}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 4109192600}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24542': {'gid': 24542, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.50:6839/1124366265', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6838', 'nonce': 1124366265}, {'type': 'v1', 'addr': '172.21.15.50:6839', 'nonce': 1124366265}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24542, 'qdb_cluster': [24520, 24535, 24542]} 2024-05-27T09:36:07.956 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-27T09:36:07.956 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.c status 2024-05-27T09:36:08.434 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:08.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:08 smithi044 ceph-mon[31618]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-27T09:36:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:08 smithi050 ceph-mon[36371]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-27T09:36:08.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:08 smithi189 ceph-mon[36671]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-27T09:36:08.875 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:36:09.780 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:10.204 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:10 smithi044 ceph-mon[31618]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-27T09:36:10.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:10 smithi050 ceph-mon[36371]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-27T09:36:10.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:10 smithi189 ceph-mon[36671]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-27T09:36:11.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.036+0000 7f514c600700 1 -- 172.21.15.44:0/1693923574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5144108f10 msgr2=0x7f5144109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:11.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.036+0000 7f514c600700 1 --2- 172.21.15.44:0/1693923574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5144108f10 0x7f5144109320 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5130009990 tx=0x7f5130009ca0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.036+0000 7f514c600700 1 -- 172.21.15.44:0/1693923574 shutdown_connections 2024-05-27T09:36:11.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.036+0000 7f514c600700 1 --2- 172.21.15.44:0/1693923574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5144109860 0x7f51441111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.036+0000 7f514c600700 1 --2- 172.21.15.44:0/1693923574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51441045e0 0x7f51441049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.036+0000 7f514c600700 1 --2- 172.21.15.44:0/1693923574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5144108f10 0x7f5144109320 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.036+0000 7f514c600700 1 -- 172.21.15.44:0/1693923574 >> 172.21.15.44:0/1693923574 conn(0x7f5144100030 msgr2=0x7f5144102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:11.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.036+0000 7f514c600700 1 -- 172.21.15.44:0/1693923574 shutdown_connections 2024-05-27T09:36:11.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.036+0000 7f514c600700 1 -- 172.21.15.44:0/1693923574 wait complete. 2024-05-27T09:36:11.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.037+0000 7f514c600700 1 Processor -- start 2024-05-27T09:36:11.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.038+0000 7f514c600700 1 -- start start 2024-05-27T09:36:11.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.038+0000 7f514c600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51441045e0 0x7f514419d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.038+0000 7f514c600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5144108f10 0x7f514419d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.038+0000 7f514c600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144109860 0x7f51441a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.038+0000 7f514c600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f51441144c0 con 0x7f5144109860 2024-05-27T09:36:11.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.038+0000 7f514c600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5144114340 con 0x7f51441045e0 2024-05-27T09:36:11.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.038+0000 7f514c600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5144114640 con 0x7f5144108f10 2024-05-27T09:36:11.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.038+0000 7f5149c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51441045e0 0x7f514419d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.038+0000 7f5149c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51441045e0 0x7f514419d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51762/0 (socket says 172.21.15.44:51762) 2024-05-27T09:36:11.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.038+0000 7f5149c00700 1 -- 172.21.15.44:0/2947207156 learned_addr learned my addr 172.21.15.44:0/2947207156 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:11.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.039+0000 7f5149200700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5144108f10 0x7f514419d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.039+0000 7f514a600700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144109860 0x7f51441a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.039+0000 7f5149c00700 1 -- 172.21.15.44:0/2947207156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5144108f10 msgr2=0x7f514419d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:11.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.039+0000 7f5149c00700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5144108f10 0x7f514419d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.039+0000 7f5149c00700 1 -- 172.21.15.44:0/2947207156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144109860 msgr2=0x7f51441a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:11.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.039+0000 7f5149c00700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144109860 0x7f51441a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.039+0000 7f5149c00700 1 -- 172.21.15.44:0/2947207156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51441a1fa0 con 0x7f51441045e0 2024-05-27T09:36:11.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.039+0000 7f514a600700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144109860 0x7f51441a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:36:11.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.039+0000 7f5149c00700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51441045e0 0x7f514419d0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f513800c370 tx=0x7f513800c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:11.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.039+0000 7f5142000700 1 -- 172.21.15.44:0/2947207156 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513800cea0 con 0x7f51441045e0 2024-05-27T09:36:11.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.040+0000 7f5142000700 1 -- 172.21.15.44:0/2947207156 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51380090d0 con 0x7f51441045e0 2024-05-27T09:36:11.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.040+0000 7f514c600700 1 -- 172.21.15.44:0/2947207156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51441a2230 con 0x7f51441045e0 2024-05-27T09:36:11.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.040+0000 7f5142000700 1 -- 172.21.15.44:0/2947207156 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138004770 con 0x7f51441045e0 2024-05-27T09:36:11.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.040+0000 7f514c600700 1 -- 172.21.15.44:0/2947207156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51441a9bb0 con 0x7f51441045e0 2024-05-27T09:36:11.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.044+0000 7f5142000700 1 -- 172.21.15.44:0/2947207156 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f513801a020 con 0x7f51441045e0 2024-05-27T09:36:11.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.045+0000 7f515b189b80 1 Processor -- start 2024-05-27T09:36:11.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.045+0000 7f515b189b80 1 -- start start 2024-05-27T09:36:11.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.045+0000 7f515b189b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 0x560c8888f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.045+0000 7f5142000700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f512c0b2ba0 0x7f512c0b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.045+0000 7f515b189b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c8888fce0 0x560c88889820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.045+0000 7f515b189b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c88889d60 0x560c8888a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.045+0000 7f515b189b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560c887b7df0 con 0x560c8870adf0 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.045+0000 7f515b189b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560c887a7990 con 0x560c88889d60 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.045+0000 7f515b189b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560c887ab9d0 con 0x560c8888fce0 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f5142000700 1 -- 172.21.15.44:0/2947207156 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f51380e2250 con 0x7f51441045e0 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f5149c00700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f512c0b2ba0 0x7f512c0b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f514a600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 0x560c8888f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f514a600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 0x560c8888f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44200/0 (socket says 172.21.15.44:44200) 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f514a600700 1 -- 172.21.15.44:0/271748601 learned_addr learned my addr 172.21.15.44:0/271748601 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f5149200700 1 --2- 172.21.15.44:0/271748601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c8888fce0 0x560c88889820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f514a600700 1 --2- 172.21.15.44:0/271748601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c88889d60 0x560c8888a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f514a600700 1 -- 172.21.15.44:0/271748601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c8888fce0 msgr2=0x560c88889820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:11.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f514a600700 1 --2- 172.21.15.44:0/271748601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c8888fce0 0x560c88889820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f514a600700 1 -- 172.21.15.44:0/271748601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c88889d60 msgr2=0x560c8888a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:11.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f514a600700 1 --2- 172.21.15.44:0/271748601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c88889d60 0x560c8888a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.046+0000 7f514a600700 1 -- 172.21.15.44:0/271748601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c887703d0 con 0x560c8870adf0 2024-05-27T09:36:11.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.047+0000 7f514a600700 1 --2- 172.21.15.44:0/271748601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 0x560c8888f7a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f513c009f00 tx=0x7f513c0097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67b3ee9d0d4b603 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:11.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.047+0000 7f5149c00700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f512c0b2ba0 0x7f512c0b50b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f51380df7f0 tx=0x7f5138004ad0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:11.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.047+0000 7f5136a00700 1 -- 172.21.15.44:0/271748601 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c010420 con 0x560c8870adf0 2024-05-27T09:36:11.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.047+0000 7f5136a00700 1 -- 172.21.15.44:0/271748601 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f513c010a70 con 0x560c8870adf0 2024-05-27T09:36:11.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.047+0000 7f5136a00700 1 -- 172.21.15.44:0/271748601 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c00f8c0 con 0x560c8870adf0 2024-05-27T09:36:11.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.048+0000 7f515b189b80 1 -- 172.21.15.44:0/271748601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 msgr2=0x560c8888f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:11.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.048+0000 7f515b189b80 1 --2- 172.21.15.44:0/271748601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 0x560c8888f7a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f513c009f00 tx=0x7f513c0097b0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.048+0000 7f515b189b80 1 -- 172.21.15.44:0/271748601 shutdown_connections 2024-05-27T09:36:11.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.048+0000 7f515b189b80 1 --2- 172.21.15.44:0/271748601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c88889d60 0x560c8888a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.048+0000 7f515b189b80 1 --2- 172.21.15.44:0/271748601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 0x560c8888f7a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.048+0000 7f515b189b80 1 --2- 172.21.15.44:0/271748601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c8888fce0 0x560c88889820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.048+0000 7f515b189b80 1 -- 172.21.15.44:0/271748601 >> 172.21.15.44:0/271748601 conn(0x560c8876af70 msgr2=0x560c88884880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:11.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.048+0000 7f515b189b80 1 -- 172.21.15.44:0/271748601 shutdown_connections 2024-05-27T09:36:11.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.048+0000 7f515b189b80 1 -- 172.21.15.44:0/271748601 wait complete. 2024-05-27T09:36:11.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.049+0000 7f515b189b80 1 -- start start 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.049+0000 7f515b189b80 1 Processor -- start 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.050+0000 7f515b189b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c8870adf0 0x560c88893490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.050+0000 7f515b189b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c88889d60 0x560c888939d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.050+0000 7f515b189b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8888fce0 0x560c88893f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.050+0000 7f515b189b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560c886adf50 con 0x560c88889d60 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.050+0000 7f515b189b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560c88748710 con 0x560c8888fce0 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.050+0000 7f515b189b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560c888967b0 con 0x560c8870adf0 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.050+0000 7f515b189b80 10 client.0 _ll_register_callbacks cb 0x560c88751c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.050+0000 7f5149200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c88889d60 0x560c888939d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.050+0000 7f514a600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c8870adf0 0x560c88893490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.050+0000 7f5149200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c88889d60 0x560c888939d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44206/0 (socket says 172.21.15.44:44206) 2024-05-27T09:36:11.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.050+0000 7f5149200700 1 -- 172.21.15.44:0/3085134966 learned_addr learned my addr 172.21.15.44:0/3085134966 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:11.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.051+0000 7f514a600700 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8888fce0 0x560c88893f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.051+0000 7f514a600700 1 -- 172.21.15.44:0/3085134966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8888fce0 msgr2=0x560c88893f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:11.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.051+0000 7f514a600700 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8888fce0 0x560c88893f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.051+0000 7f514a600700 1 -- 172.21.15.44:0/3085134966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c88889d60 msgr2=0x560c888939d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:11.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.051+0000 7f514a600700 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c88889d60 0x560c888939d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.051+0000 7f514a600700 1 -- 172.21.15.44:0/3085134966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c886f7880 con 0x560c8870adf0 2024-05-27T09:36:11.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.051+0000 7f514a600700 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c8870adf0 0x560c88893490 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f513c009ed0 tx=0x7f513c00f500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed68915b229f5af6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:11.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.051+0000 7f512b400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:36:11.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.051+0000 7f512b400700 1 -- 172.21.15.44:0/3085134966 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c005250 con 0x560c8870adf0 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.052+0000 7f515b189b80 1 -- 172.21.15.44:0/3085134966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513c010a70 con 0x560c8870adf0 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.052+0000 7f512b400700 1 -- 172.21.15.44:0/3085134966 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f513c0053f0 con 0x560c8870adf0 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.052+0000 7f512b400700 1 -- 172.21.15.44:0/3085134966 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f513c0108b0 con 0x560c8870adf0 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.052+0000 7f515b189b80 10 client.25363 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.052+0000 7f515b189b80 1 -- 172.21.15.44:0/3085134966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560c88778010 con 0x560c8870adf0 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.052+0000 7f512b400700 1 -- 172.21.15.44:0/3085134966 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f513c010a70 con 0x560c8870adf0 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.052+0000 7f512b400700 1 -- 172.21.15.44:0/3085134966 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f513c00f630 con 0x560c8870adf0 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.052+0000 7f515b189b80 10 client.25363 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.053+0000 7f515b189b80 10 client.25363 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.053+0000 7f515b189b80 20 client.25363 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.053+0000 7f515b189b80 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560c888a3440 0x560c888a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.053+0000 7f515b189b80 4 client.25363 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:11.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.053+0000 7f515b189b80 1 -- 172.21.15.44:0/3085134966 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c887b9d40 con 0x560c888a3440 2024-05-27T09:36:11.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.053+0000 7f5149200700 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560c888a3440 0x560c888a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.054+0000 7f5149200700 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560c888a3440 0x560c888a48b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d52852c940093ae server_cookie=e6a0dd8efbe3f70a in_seq=0 out_seq=0 2024-05-27T09:36:11.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.054+0000 7f512b400700 10 client.25363 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:36:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f512b400700 1 -- 172.21.15.44:0/3085134966 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560c887b9d40 con 0x560c888a3440 2024-05-27T09:36:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f512b400700 10 client.25363 handle_command_reply: tid=0 2024-05-27T09:36:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f515b189b80 1 client.25363 shutdown 2024-05-27T09:36:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f515b189b80 1 -- 172.21.15.44:0/3085134966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c8870adf0 msgr2=0x560c88893490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f515b189b80 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c8870adf0 0x560c88893490 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f513c009ed0 tx=0x7f513c00f500 comp rx=0 tx=0).stop 2024-05-27T09:36:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f515b189b80 1 -- 172.21.15.44:0/3085134966 shutdown_connections 2024-05-27T09:36:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f515b189b80 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8888fce0 0x560c88893f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f515b189b80 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c8870adf0 0x560c88893490 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f515b189b80 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560c888a3440 0x560c888a48b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f515b189b80 1 --2- 172.21.15.44:0/3085134966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c88889d60 0x560c888939d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f512b400700 0 client.25363 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:36:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f515b189b80 1 -- 172.21.15.44:0/3085134966 >> 172.21.15.44:0/3085134966 conn(0x560c8876af70 msgr2=0x560c88884530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f515b189b80 1 -- 172.21.15.44:0/3085134966 shutdown_connections 2024-05-27T09:36:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.055+0000 7f515b189b80 1 -- 172.21.15.44:0/3085134966 wait complete. 2024-05-27T09:36:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.056+0000 7f515b189b80 20 client.25363 trim_cache size 0 max 16384 2024-05-27T09:36:11.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.080+0000 7f515b189b80 1 Processor -- start 2024-05-27T09:36:11.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.080+0000 7f515b189b80 1 -- start start 2024-05-27T09:36:11.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.080+0000 7f515b189b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8870adf0 0x560c88889820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.080+0000 7f515b189b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 0x560c8889fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.080+0000 7f515b189b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8888fce0 0x560c8888ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.080+0000 7f515b189b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560c888a2bf0 con 0x560c8888fce0 2024-05-27T09:36:11.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.080+0000 7f515b189b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560c888a32c0 con 0x560c8870adf0 2024-05-27T09:36:11.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.080+0000 7f515b189b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560c888a3990 con 0x560c88889d60 2024-05-27T09:36:11.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.081+0000 7f5149200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 0x560c8889fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.081+0000 7f5149200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 0x560c8889fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:52492/0 (socket says 172.21.15.44:52492) 2024-05-27T09:36:11.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.081+0000 7f5149200700 1 -- 172.21.15.44:0/3148127169 learned_addr learned my addr 172.21.15.44:0/3148127169 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:11.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.081+0000 7f514a600700 1 --2- 172.21.15.44:0/3148127169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8870adf0 0x560c88889820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.081+0000 7f514a600700 1 --2- 172.21.15.44:0/3148127169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8888fce0 0x560c8888ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.081+0000 7f5149200700 1 -- 172.21.15.44:0/3148127169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8870adf0 msgr2=0x560c88889820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:11.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.081+0000 7f5149200700 1 --2- 172.21.15.44:0/3148127169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8870adf0 0x560c88889820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.081+0000 7f5149200700 1 -- 172.21.15.44:0/3148127169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8888fce0 msgr2=0x560c8888ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:11.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.081+0000 7f5149200700 1 --2- 172.21.15.44:0/3148127169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8888fce0 0x560c8888ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.081+0000 7f5149200700 1 -- 172.21.15.44:0/3148127169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c888927b0 con 0x560c88889d60 2024-05-27T09:36:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.082+0000 7f5149200700 1 --2- 172.21.15.44:0/3148127169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 0x560c8889fa80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f51300098e0 tx=0x7f5130005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6daffc5cc33be86 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.082+0000 7f5136000700 1 -- 172.21.15.44:0/3148127169 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513000b9f0 con 0x560c88889d60 2024-05-27T09:36:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.082+0000 7f5136000700 1 -- 172.21.15.44:0/3148127169 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5130003680 con 0x560c88889d60 2024-05-27T09:36:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.082+0000 7f5136000700 1 -- 172.21.15.44:0/3148127169 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513000f6c0 con 0x560c88889d60 2024-05-27T09:36:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.083+0000 7f515b189b80 1 -- 172.21.15.44:0/3148127169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 msgr2=0x560c8889fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.083+0000 7f515b189b80 1 --2- 172.21.15.44:0/3148127169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 0x560c8889fa80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f51300098e0 tx=0x7f5130005040 comp rx=0 tx=0).stop 2024-05-27T09:36:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.083+0000 7f515b189b80 1 -- 172.21.15.44:0/3148127169 shutdown_connections 2024-05-27T09:36:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.083+0000 7f515b189b80 1 --2- 172.21.15.44:0/3148127169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8888fce0 0x560c8888ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.083+0000 7f515b189b80 1 --2- 172.21.15.44:0/3148127169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8870adf0 0x560c88889820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.083+0000 7f515b189b80 1 --2- 172.21.15.44:0/3148127169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 0x560c8889fa80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.083+0000 7f515b189b80 1 -- 172.21.15.44:0/3148127169 >> 172.21.15.44:0/3148127169 conn(0x560c8876af70 msgr2=0x560c8888e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:11.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.083+0000 7f515b189b80 1 -- 172.21.15.44:0/3148127169 shutdown_connections 2024-05-27T09:36:11.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.083+0000 7f515b189b80 1 -- 172.21.15.44:0/3148127169 wait complete. 2024-05-27T09:36:11.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.084+0000 7f515b189b80 1 -- start start 2024-05-27T09:36:11.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.084+0000 7f515b189b80 1 Processor -- start 2024-05-27T09:36:11.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.084+0000 7f515b189b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 0x560c888ae510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.084+0000 7f515b189b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 0x560c888888d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.084+0000 7f515b189b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8888fce0 0x560c88888f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.084+0000 7f515b189b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560c888a4730 con 0x560c8870adf0 2024-05-27T09:36:11.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.084+0000 7f515b189b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560c888a4060 con 0x560c8888fce0 2024-05-27T09:36:11.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.084+0000 7f515b189b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560c888a4e00 con 0x560c88889d60 2024-05-27T09:36:11.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.084+0000 7f515b189b80 10 client.0 _ll_register_callbacks cb 0x560c88751c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:11.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.085+0000 7f514a600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 0x560c888ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.085+0000 7f514a600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 0x560c888ae510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44220/0 (socket says 172.21.15.44:44220) 2024-05-27T09:36:11.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.085+0000 7f514a600700 1 -- 172.21.15.44:0/3337072699 learned_addr learned my addr 172.21.15.44:0/3337072699 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:11.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.085+0000 7f5149200700 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 0x560c888888d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.085+0000 7f514a600700 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8888fce0 0x560c88888f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.085+0000 7f5149200700 1 -- 172.21.15.44:0/3337072699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8888fce0 msgr2=0x560c88888f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.085+0000 7f5149200700 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8888fce0 0x560c88888f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.085+0000 7f5149200700 1 -- 172.21.15.44:0/3337072699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 msgr2=0x560c888ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.085+0000 7f5149200700 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 0x560c888ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.085+0000 7f5149200700 1 -- 172.21.15.44:0/3337072699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c888896a0 con 0x560c88889d60 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.086+0000 7f5149200700 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 0x560c888888d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5130005010 tx=0x7f51300058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be38f960a417895a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.086+0000 7f512b400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.086+0000 7f512b400700 1 -- 172.21.15.44:0/3337072699 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513003e060 con 0x560c88889d60 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.086+0000 7f515b189b80 1 -- 172.21.15.44:0/3337072699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5130003680 con 0x560c88889d60 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.086+0000 7f512b400700 1 -- 172.21.15.44:0/3337072699 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f51300038b0 con 0x560c88889d60 2024-05-27T09:36:11.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.086+0000 7f512b400700 1 -- 172.21.15.44:0/3337072699 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5130003af0 con 0x560c88889d60 2024-05-27T09:36:11.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.086+0000 7f515b189b80 10 client.25369 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:11.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.086+0000 7f515b189b80 1 -- 172.21.15.44:0/3337072699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560c8890ebd0 con 0x560c88889d60 2024-05-27T09:36:11.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.087+0000 7f512b400700 1 -- 172.21.15.44:0/3337072699 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5130003680 con 0x560c88889d60 2024-05-27T09:36:11.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.087+0000 7f512b400700 1 -- 172.21.15.44:0/3337072699 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5130043680 con 0x560c88889d60 2024-05-27T09:36:11.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.087+0000 7f515b189b80 10 client.25369 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.087+0000 7f515b189b80 10 client.25369 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.087+0000 7f515b189b80 20 client.25369 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.087+0000 7f515b189b80 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560c88910a70 0x560c88911ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.087+0000 7f515b189b80 4 client.25369 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.087+0000 7f515b189b80 1 -- 172.21.15.44:0/3337072699 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "status"}) -- 0x560c887b3520 con 0x560c88910a70 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.087+0000 7f514a600700 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560c88910a70 0x560c88911ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.088+0000 7f514a600700 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560c88910a70 0x560c88911ee0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2fbc56e02ce67103 server_cookie=d6a6b076587f126c in_seq=0 out_seq=0 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.088+0000 7f512b400700 10 client.25369 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.088+0000 7f512b400700 1 -- 172.21.15.44:0/3337072699 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x560c887b3520 con 0x560c88910a70 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.088+0000 7f512b400700 10 client.25369 handle_command_reply: tid=0 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.088+0000 7f515b189b80 1 client.25369 shutdown 2024-05-27T09:36:11.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.088+0000 7f515b189b80 1 -- 172.21.15.44:0/3337072699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 msgr2=0x560c888888d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:11.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.088+0000 7f515b189b80 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 0x560c888888d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5130005010 tx=0x7f51300058e0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.089+0000 7f515b189b80 1 -- 172.21.15.44:0/3337072699 shutdown_connections 2024-05-27T09:36:11.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.089+0000 7f515b189b80 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560c8888fce0 0x560c88888f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.089+0000 7f515b189b80 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560c88910a70 0x560c88911ee0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.089+0000 7f515b189b80 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c8870adf0 0x560c888ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.089+0000 7f515b189b80 1 --2- 172.21.15.44:0/3337072699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c88889d60 0x560c888888d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.089+0000 7f512b400700 0 client.25369 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:36:11.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.089+0000 7f515b189b80 1 -- 172.21.15.44:0/3337072699 >> 172.21.15.44:0/3337072699 conn(0x560c8876af70 msgr2=0x560c8888d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:11.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.089+0000 7f515b189b80 1 -- 172.21.15.44:0/3337072699 shutdown_connections 2024-05-27T09:36:11.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.089+0000 7f515b189b80 1 -- 172.21.15.44:0/3337072699 wait complete. 2024-05-27T09:36:11.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.089+0000 7f515b189b80 20 client.25369 trim_cache size 0 max 16384 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 2, 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24520, 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 267.34610922100001, 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 26, 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 82, 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 81, 2024-05-27T09:36:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 322.588219853 2024-05-27T09:36:11.094 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:36:11.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 -- 172.21.15.44:0/2947207156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f512c0b2ba0 msgr2=0x7f512c0b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:11.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f512c0b2ba0 0x7f512c0b50b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f51380df7f0 tx=0x7f5138004ad0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 -- 172.21.15.44:0/2947207156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51441045e0 msgr2=0x7f514419d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:11.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51441045e0 0x7f514419d0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f513800c370 tx=0x7f513800c680 comp rx=0 tx=0).stop 2024-05-27T09:36:11.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 -- 172.21.15.44:0/2947207156 shutdown_connections 2024-05-27T09:36:11.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f512c0b2ba0 0x7f512c0b50b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5144109860 0x7f51441a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51441045e0 0x7f514419d0d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 --2- 172.21.15.44:0/2947207156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5144108f10 0x7f514419d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:11.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 -- 172.21.15.44:0/2947207156 >> 172.21.15.44:0/2947207156 conn(0x7f5144100030 msgr2=0x7f514410a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:11.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 -- 172.21.15.44:0/2947207156 shutdown_connections 2024-05-27T09:36:11.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:11.090+0000 7f514c600700 1 -- 172.21.15.44:0/2947207156 wait complete. 2024-05-27T09:36:12.297 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-05-27T09:36:12.426 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:12 smithi044 ceph-mon[31618]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-27T09:36:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:12 smithi050 ceph-mon[36371]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-27T09:36:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:12 smithi189 ceph-mon[36671]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-27T09:36:12.766 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:12.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.782+0000 7fcb1be00700 1 -- 172.21.15.44:0/590967895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb1c115170 msgr2=0x7fcb1c117590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:12.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.782+0000 7fcb1be00700 1 --2- 172.21.15.44:0/590967895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb1c115170 0x7fcb1c117590 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400b0a0 tx=0x7fcb1400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.782+0000 7fcb1be00700 1 -- 172.21.15.44:0/590967895 shutdown_connections 2024-05-27T09:36:12.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.782+0000 7fcb1be00700 1 --2- 172.21.15.44:0/590967895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb1c115170 0x7fcb1c117590 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.782+0000 7fcb1be00700 1 --2- 172.21.15.44:0/590967895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb1c075b50 0x7fcb1c075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.782+0000 7fcb1be00700 1 --2- 172.21.15.44:0/590967895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb1c0764a0 0x7fcb1c114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.782+0000 7fcb1be00700 1 -- 172.21.15.44:0/590967895 >> 172.21.15.44:0/590967895 conn(0x7fcb1c079d00 msgr2=0x7fcb1c07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:12.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.782+0000 7fcb1be00700 1 -- 172.21.15.44:0/590967895 shutdown_connections 2024-05-27T09:36:12.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.782+0000 7fcb1be00700 1 -- 172.21.15.44:0/590967895 wait complete. 2024-05-27T09:36:12.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.783+0000 7fcb1be00700 1 Processor -- start 2024-05-27T09:36:12.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.783+0000 7fcb1be00700 1 -- start start 2024-05-27T09:36:12.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.784+0000 7fcb1be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb1c075b50 0x7fcb1c1a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.784+0000 7fcb1be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb1c0764a0 0x7fcb1c1a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.784+0000 7fcb1be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb1c115170 0x7fcb1c1a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.784+0000 7fcb1be00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c11a870 con 0x7fcb1c0764a0 2024-05-27T09:36:12.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.784+0000 7fcb1be00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c11a6f0 con 0x7fcb1c115170 2024-05-27T09:36:12.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.784+0000 7fcb1be00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c11a9f0 con 0x7fcb1c075b50 2024-05-27T09:36:12.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.784+0000 7fcb1aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb1c075b50 0x7fcb1c1a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.784+0000 7fcb1a000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb1c0764a0 0x7fcb1c1a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.784+0000 7fcb1aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb1c075b50 0x7fcb1c1a1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:52516/0 (socket says 172.21.15.44:52516) 2024-05-27T09:36:12.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.784+0000 7fcb1aa00700 1 -- 172.21.15.44:0/423451399 learned_addr learned my addr 172.21.15.44:0/423451399 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:12.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.785+0000 7fcb1b400700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb1c115170 0x7fcb1c1a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.785+0000 7fcb1aa00700 1 -- 172.21.15.44:0/423451399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb1c115170 msgr2=0x7fcb1c1a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:12.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.785+0000 7fcb1aa00700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb1c115170 0x7fcb1c1a5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.785+0000 7fcb1aa00700 1 -- 172.21.15.44:0/423451399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb1c0764a0 msgr2=0x7fcb1c1a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:12.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.785+0000 7fcb1aa00700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb1c0764a0 0x7fcb1c1a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.785+0000 7fcb1aa00700 1 -- 172.21.15.44:0/423451399 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c1a6110 con 0x7fcb1c075b50 2024-05-27T09:36:12.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.785+0000 7fcb1aa00700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb1c075b50 0x7fcb1c1a1240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00e9a0 tx=0x7fcb0c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:12.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.786+0000 7fcb13400700 1 -- 172.21.15.44:0/423451399 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c00ca10 con 0x7fcb1c075b50 2024-05-27T09:36:12.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.786+0000 7fcb13400700 1 -- 172.21.15.44:0/423451399 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb0c00cbb0 con 0x7fcb1c075b50 2024-05-27T09:36:12.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.786+0000 7fcb1be00700 1 -- 172.21.15.44:0/423451399 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb1c1a6400 con 0x7fcb1c075b50 2024-05-27T09:36:12.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.786+0000 7fcb13400700 1 -- 172.21.15.44:0/423451399 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c018680 con 0x7fcb1c075b50 2024-05-27T09:36:12.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.786+0000 7fcb1be00700 1 -- 172.21.15.44:0/423451399 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb1c1add20 con 0x7fcb1c075b50 2024-05-27T09:36:12.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.790+0000 7fcb301dab80 1 Processor -- start 2024-05-27T09:36:12.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.791+0000 7fcb301dab80 1 -- start start 2024-05-27T09:36:12.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.791+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238a83e40 0x557238c01ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.791+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c0cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.791+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238c02820 0x557238c0f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.791+0000 7fcb13400700 1 -- 172.21.15.44:0/423451399 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7fcb0c010430 con 0x7fcb1c075b50 2024-05-27T09:36:12.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.791+0000 7fcb301dab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557238b30e40 con 0x557238a83e40 2024-05-27T09:36:12.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.791+0000 7fcb301dab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557238b209e0 con 0x557238c02410 2024-05-27T09:36:12.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.791+0000 7fcb301dab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557238b24a20 con 0x557238c02820 2024-05-27T09:36:12.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.792+0000 7fcb1a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238c02820 0x557238c0f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.792+0000 7fcb1aa00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c0cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.792+0000 7fcb1b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238a83e40 0x557238c01ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.792+0000 7fcb1a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238c02820 0x557238c0f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:52518/0 (socket says 172.21.15.44:52518) 2024-05-27T09:36:12.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.792+0000 7fcb1b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238a83e40 0x557238c01ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44246/0 (socket says 172.21.15.44:44246) 2024-05-27T09:36:12.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.792+0000 7fcb1a000700 1 -- 172.21.15.44:0/1635305185 learned_addr learned my addr 172.21.15.44:0/1635305185 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:12.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.792+0000 7fcb1a000700 1 -- 172.21.15.44:0/1635305185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 msgr2=0x557238c0cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:12.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.792+0000 7fcb1a000700 1 --2- 172.21.15.44:0/1635305185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c0cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.792+0000 7fcb1a000700 1 -- 172.21.15.44:0/1635305185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238a83e40 msgr2=0x557238c01ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:12.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.792+0000 7fcb1a000700 1 --2- 172.21.15.44:0/1635305185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238a83e40 0x557238c01ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.792+0000 7fcb1a000700 1 -- 172.21.15.44:0/1635305185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557238ae9420 con 0x557238c02820 2024-05-27T09:36:12.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.793+0000 7fcb1a000700 1 --2- 172.21.15.44:0/1635305185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238c02820 0x557238c0f020 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009ca0 tx=0x7fcb08011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc5d955ed6809f76 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:12.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.793+0000 7fcb07e00700 1 -- 172.21.15.44:0/1635305185 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08011d80 con 0x557238c02820 2024-05-27T09:36:12.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.793+0000 7fcb07e00700 1 -- 172.21.15.44:0/1635305185 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb08004ce0 con 0x557238c02820 2024-05-27T09:36:12.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.793+0000 7fcb07e00700 1 -- 172.21.15.44:0/1635305185 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08019690 con 0x557238c02820 2024-05-27T09:36:12.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.793+0000 7fcb13400700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb000b2c20 0x7fcb000b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.794+0000 7fcb13400700 1 -- 172.21.15.44:0/423451399 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fcb0c0e29b0 con 0x7fcb1c075b50 2024-05-27T09:36:12.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.794+0000 7fcb1b400700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb000b2c20 0x7fcb000b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.794+0000 7fcb301dab80 1 -- 172.21.15.44:0/1635305185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238c02820 msgr2=0x557238c0f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:12.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.794+0000 7fcb301dab80 1 --2- 172.21.15.44:0/1635305185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238c02820 0x557238c0f020 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009ca0 tx=0x7fcb08011440 comp rx=0 tx=0).stop 2024-05-27T09:36:12.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.795+0000 7fcb301dab80 1 -- 172.21.15.44:0/1635305185 shutdown_connections 2024-05-27T09:36:12.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.795+0000 7fcb301dab80 1 --2- 172.21.15.44:0/1635305185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238c02820 0x557238c0f020 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.795+0000 7fcb301dab80 1 --2- 172.21.15.44:0/1635305185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238a83e40 0x557238c01ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.795+0000 7fcb301dab80 1 --2- 172.21.15.44:0/1635305185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c0cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.795+0000 7fcb301dab80 1 -- 172.21.15.44:0/1635305185 >> 172.21.15.44:0/1635305185 conn(0x557238ae3fc0 msgr2=0x557238bfdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:12.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.795+0000 7fcb301dab80 1 -- 172.21.15.44:0/1635305185 shutdown_connections 2024-05-27T09:36:12.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.795+0000 7fcb301dab80 1 -- 172.21.15.44:0/1635305185 wait complete. 2024-05-27T09:36:12.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.795+0000 7fcb1b400700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb000b2c20 0x7fcb000b5130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x557238bfe690 tx=0x7fcb1400b720 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:12.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.796+0000 7fcb301dab80 1 -- start start 2024-05-27T09:36:12.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.796+0000 7fcb301dab80 1 Processor -- start 2024-05-27T09:36:12.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238a83e40 0x557238c0c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c0ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02820 0x557238c0cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb301dab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557238a26fa0 con 0x557238c02820 2024-05-27T09:36:12.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb301dab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557238ac1760 con 0x557238c02410 2024-05-27T09:36:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb301dab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557238c0f990 con 0x557238a83e40 2024-05-27T09:36:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb301dab80 10 client.0 _ll_register_callbacks cb 0x557238acac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb1b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02820 0x557238c0cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb1b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02820 0x557238c0cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44258/0 (socket says 172.21.15.44:44258) 2024-05-27T09:36:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb1b400700 1 -- 172.21.15.44:0/3306301341 learned_addr learned my addr 172.21.15.44:0/3306301341 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb1a000700 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c0ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.797+0000 7fcb1aa00700 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238a83e40 0x557238c0c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.798+0000 7fcb1aa00700 1 -- 172.21.15.44:0/3306301341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 msgr2=0x557238c0ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.798+0000 7fcb1aa00700 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c0ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.798+0000 7fcb1aa00700 1 -- 172.21.15.44:0/3306301341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02820 msgr2=0x557238c0cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:12.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.798+0000 7fcb1aa00700 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02820 0x557238c0cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.798+0000 7fcb1aa00700 1 -- 172.21.15.44:0/3306301341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557238a708d0 con 0x557238a83e40 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.798+0000 7fcb1b400700 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02820 0x557238c0cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.798+0000 7fcb1aa00700 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238a83e40 0x557238c0c540 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0dfef0 tx=0x7fcb0c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=604ffe49c640c8c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.798+0000 7fcb04c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.799+0000 7fcb04c00700 1 -- 172.21.15.44:0/3306301341 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c0a9850 con 0x557238a83e40 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.799+0000 7fcb04c00700 1 -- 172.21.15.44:0/3306301341 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fcb0c0a8ef0 con 0x557238a83e40 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.799+0000 7fcb301dab80 1 -- 172.21.15.44:0/3306301341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb08004ce0 con 0x557238a83e40 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.799+0000 7fcb04c00700 1 -- 172.21.15.44:0/3306301341 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcb0c0a5210 con 0x557238a83e40 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.799+0000 7fcb301dab80 10 client.25381 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.799+0000 7fcb301dab80 1 -- 172.21.15.44:0/3306301341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557238af1060 con 0x557238a83e40 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.800+0000 7fcb04c00700 1 -- 172.21.15.44:0/3306301341 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb08004ce0 con 0x557238a83e40 2024-05-27T09:36:12.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.800+0000 7fcb04c00700 1 -- 172.21.15.44:0/3306301341 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fcb0c010720 con 0x557238a83e40 2024-05-27T09:36:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.800+0000 7fcb301dab80 10 client.25381 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.800+0000 7fcb301dab80 10 client.25381 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.800+0000 7fcb301dab80 20 client.25381 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.800+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x557238c1c840 0x557238c1ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.800+0000 7fcb301dab80 4 client.25381 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:12.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.800+0000 7fcb301dab80 1 -- 172.21.15.44:0/3306301341 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557238b32d90 con 0x557238c1c840 2024-05-27T09:36:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.801+0000 7fcb1a000700 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x557238c1c840 0x557238c1ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.801+0000 7fcb1a000700 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x557238c1c840 0x557238c1ec60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ff147ea938e287a server_cookie=df1878340298d05c in_seq=0 out_seq=0 2024-05-27T09:36:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.801+0000 7fcb04c00700 10 client.25381 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.802+0000 7fcb04c00700 1 -- 172.21.15.44:0/3306301341 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557238b32d90 con 0x557238c1c840 2024-05-27T09:36:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.802+0000 7fcb04c00700 10 client.25381 handle_command_reply: tid=0 2024-05-27T09:36:12.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.802+0000 7fcb301dab80 1 client.25381 shutdown 2024-05-27T09:36:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.802+0000 7fcb301dab80 1 -- 172.21.15.44:0/3306301341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238a83e40 msgr2=0x557238c0c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.802+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238a83e40 0x557238c0c540 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0dfef0 tx=0x7fcb0c0057f0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.803+0000 7fcb301dab80 1 -- 172.21.15.44:0/3306301341 shutdown_connections 2024-05-27T09:36:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.803+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x557238c1c840 0x557238c1ec60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.803+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02820 0x557238c0cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.803+0000 7fcb04c00700 0 client.25381 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.803+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238a83e40 0x557238c0c540 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.803+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3306301341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c0ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.803+0000 7fcb301dab80 1 -- 172.21.15.44:0/3306301341 >> 172.21.15.44:0/3306301341 conn(0x557238ae3fc0 msgr2=0x557238bfcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:12.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.803+0000 7fcb301dab80 1 -- 172.21.15.44:0/3306301341 shutdown_connections 2024-05-27T09:36:12.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.803+0000 7fcb301dab80 1 -- 172.21.15.44:0/3306301341 wait complete. 2024-05-27T09:36:12.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.803+0000 7fcb301dab80 20 client.25381 trim_cache size 0 max 16384 2024-05-27T09:36:12.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.828+0000 7fcb301dab80 1 Processor -- start 2024-05-27T09:36:12.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.828+0000 7fcb301dab80 1 -- start start 2024-05-27T09:36:12.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.828+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 0x557238c0e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.828+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238a83e40 0x557238c18a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.828+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02410 0x557238c243d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.828+0000 7fcb301dab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c1b7bf0 con 0x557238c02410 2024-05-27T09:36:12.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.828+0000 7fcb301dab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c1b5480 con 0x557238a83e40 2024-05-27T09:36:12.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.828+0000 7fcb301dab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c1b32d0 con 0x557238ae3fc0 2024-05-27T09:36:12.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.829+0000 7fcb1a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238a83e40 0x557238c18a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.829+0000 7fcb1a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238a83e40 0x557238c18a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51828/0 (socket says 172.21.15.44:51828) 2024-05-27T09:36:12.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.829+0000 7fcb1a000700 1 -- 172.21.15.44:0/3235277510 learned_addr learned my addr 172.21.15.44:0/3235277510 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:12.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.829+0000 7fcb1b400700 1 --2- 172.21.15.44:0/3235277510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02410 0x557238c243d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.829+0000 7fcb1aa00700 1 --2- 172.21.15.44:0/3235277510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 0x557238c0e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.830+0000 7fcb1a000700 1 -- 172.21.15.44:0/3235277510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 msgr2=0x557238c0e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:12.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.830+0000 7fcb1a000700 1 --2- 172.21.15.44:0/3235277510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 0x557238c0e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.830+0000 7fcb1a000700 1 -- 172.21.15.44:0/3235277510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02410 msgr2=0x557238c243d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:12.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.830+0000 7fcb1a000700 1 --2- 172.21.15.44:0/3235277510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02410 0x557238c243d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.830+0000 7fcb1a000700 1 -- 172.21.15.44:0/3235277510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557238bfe910 con 0x557238a83e40 2024-05-27T09:36:12.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.830+0000 7fcb1a000700 1 --2- 172.21.15.44:0/3235277510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238a83e40 0x557238c18a30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x557238c02cb0 tx=0x7fcb080051e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82d95aff0c671c79 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:12.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.831+0000 7fcb07400700 1 -- 172.21.15.44:0/3235277510 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08005350 con 0x557238a83e40 2024-05-27T09:36:12.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.831+0000 7fcb07400700 1 -- 172.21.15.44:0/3235277510 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb08030070 con 0x557238a83e40 2024-05-27T09:36:12.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.831+0000 7fcb07400700 1 -- 172.21.15.44:0/3235277510 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08019430 con 0x557238a83e40 2024-05-27T09:36:12.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.832+0000 7fcb301dab80 1 -- 172.21.15.44:0/3235277510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238a83e40 msgr2=0x557238c18a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:12.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.832+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3235277510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238a83e40 0x557238c18a30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x557238c02cb0 tx=0x7fcb080051e0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.832+0000 7fcb301dab80 1 -- 172.21.15.44:0/3235277510 shutdown_connections 2024-05-27T09:36:12.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.832+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3235277510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238c02410 0x557238c243d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.832+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3235277510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 0x557238c0e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.832+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3235277510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238a83e40 0x557238c18a30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.832+0000 7fcb301dab80 1 -- 172.21.15.44:0/3235277510 >> 172.21.15.44:0/3235277510 conn(0x557238bf6f60 msgr2=0x557238c0b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:12.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.832+0000 7fcb301dab80 1 -- 172.21.15.44:0/3235277510 shutdown_connections 2024-05-27T09:36:12.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.832+0000 7fcb301dab80 1 -- 172.21.15.44:0/3235277510 wait complete. 2024-05-27T09:36:12.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.833+0000 7fcb301dab80 1 -- start start 2024-05-27T09:36:12.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.833+0000 7fcb301dab80 1 Processor -- start 2024-05-27T09:36:12.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.833+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238a83e40 0x557238c10d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.833+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 0x557238c112a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb301dab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c862a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb301dab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c1b3940 con 0x557238a83e40 2024-05-27T09:36:12.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb301dab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c1b4db0 con 0x557238c02410 2024-05-27T09:36:12.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb301dab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c1b4010 con 0x557238ae3fc0 2024-05-27T09:36:12.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb301dab80 10 client.0 _ll_register_callbacks cb 0x557238acac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:12.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb1a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 0x557238c112a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb1aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238a83e40 0x557238c10d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb1a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 0x557238c112a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:52554/0 (socket says 172.21.15.44:52554) 2024-05-27T09:36:12.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb1a000700 1 -- 172.21.15.44:0/3577965387 learned_addr learned my addr 172.21.15.44:0/3577965387 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:12.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb1b400700 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c862a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb1a000700 1 -- 172.21.15.44:0/3577965387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 msgr2=0x557238c862a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:12.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb1a000700 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c862a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb1a000700 1 -- 172.21.15.44:0/3577965387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238a83e40 msgr2=0x557238c10d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:12.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb1a000700 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238a83e40 0x557238c10d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.834+0000 7fcb1a000700 1 -- 172.21.15.44:0/3577965387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557238c0bbf0 con 0x557238ae3fc0 2024-05-27T09:36:12.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.835+0000 7fcb1a000700 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 0x557238c112a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcb08005c80 tx=0x7fcb08005580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50ec0f32b00495b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:12.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.835+0000 7fcb04c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:36:12.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.835+0000 7fcb04c00700 1 -- 172.21.15.44:0/3577965387 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08003890 con 0x557238ae3fc0 2024-05-27T09:36:12.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.835+0000 7fcb04c00700 1 -- 172.21.15.44:0/3577965387 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fcb08003a30 con 0x557238ae3fc0 2024-05-27T09:36:12.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.835+0000 7fcb301dab80 1 -- 172.21.15.44:0/3577965387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb08030070 con 0x557238ae3fc0 2024-05-27T09:36:12.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.836+0000 7fcb04c00700 1 -- 172.21.15.44:0/3577965387 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcb08003c50 con 0x557238ae3fc0 2024-05-27T09:36:12.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.836+0000 7fcb301dab80 10 client.25387 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:12.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.836+0000 7fcb301dab80 1 -- 172.21.15.44:0/3577965387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557238c876e0 con 0x557238ae3fc0 2024-05-27T09:36:12.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.836+0000 7fcb04c00700 1 -- 172.21.15.44:0/3577965387 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb08030070 con 0x557238ae3fc0 2024-05-27T09:36:12.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.836+0000 7fcb04c00700 1 -- 172.21.15.44:0/3577965387 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fcb0803d430 con 0x557238ae3fc0 2024-05-27T09:36:12.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.836+0000 7fcb301dab80 10 client.25387 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:12.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.836+0000 7fcb301dab80 10 client.25387 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:12.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.836+0000 7fcb301dab80 20 client.25387 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:12.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.836+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x557238c02820 0x557238c89af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:12.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.836+0000 7fcb301dab80 4 client.25387 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:36:12.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.836+0000 7fcb301dab80 1 -- 172.21.15.44:0/3577965387 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557238b2c570 con 0x557238c02820 2024-05-27T09:36:12.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.837+0000 7fcb1aa00700 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x557238c02820 0x557238c89af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:12.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.837+0000 7fcb1aa00700 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x557238c02820 0x557238c89af0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90303c275cbbf95d server_cookie=66227303a6d1ba85 in_seq=0 out_seq=0 2024-05-27T09:36:12.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.837+0000 7fcb04c00700 10 client.25387 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:12.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb04c00700 1 -- 172.21.15.44:0/3577965387 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557238b2c570 con 0x557238c02820 2024-05-27T09:36:12.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb04c00700 10 client.25387 handle_command_reply: tid=0 2024-05-27T09:36:12.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb301dab80 1 client.25387 shutdown 2024-05-27T09:36:12.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb301dab80 1 -- 172.21.15.44:0/3577965387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 msgr2=0x557238c112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:12.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 0x557238c112a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcb08005c80 tx=0x7fcb08005580 comp rx=0 tx=0).stop 2024-05-27T09:36:12.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb301dab80 1 -- 172.21.15.44:0/3577965387 shutdown_connections 2024-05-27T09:36:12.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x557238c02820 0x557238c89af0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557238c02410 0x557238c862a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557238a83e40 0x557238c10d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb301dab80 1 --2- 172.21.15.44:0/3577965387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557238ae3fc0 0x557238c112a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb04c00700 0 client.25387 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:12.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb301dab80 1 -- 172.21.15.44:0/3577965387 >> 172.21.15.44:0/3577965387 conn(0x557238bf6f60 msgr2=0x557238c0b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:12.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb301dab80 1 -- 172.21.15.44:0/3577965387 shutdown_connections 2024-05-27T09:36:12.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.838+0000 7fcb301dab80 1 -- 172.21.15.44:0/3577965387 wait complete. 2024-05-27T09:36:12.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.839+0000 7fcb301dab80 20 client.25387 trim_cache size 0 max 16384 2024-05-27T09:36:12.841 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:36:12.841 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:36:12.841 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:36:12.842 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:36:12.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.839+0000 7fcb1be00700 1 -- 172.21.15.44:0/423451399 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb000b2c20 msgr2=0x7fcb000b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:12.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.839+0000 7fcb1be00700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb000b2c20 0x7fcb000b5130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x557238bfe690 tx=0x7fcb1400b720 comp rx=0 tx=0).stop 2024-05-27T09:36:12.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.839+0000 7fcb1be00700 1 -- 172.21.15.44:0/423451399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb1c075b50 msgr2=0x7fcb1c1a1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:12.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.839+0000 7fcb1be00700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb1c075b50 0x7fcb1c1a1240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00e9a0 tx=0x7fcb0c00ecb0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.840+0000 7fcb1be00700 1 -- 172.21.15.44:0/423451399 shutdown_connections 2024-05-27T09:36:12.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.840+0000 7fcb1be00700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb000b2c20 0x7fcb000b5130 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.840+0000 7fcb1be00700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb1c115170 0x7fcb1c1a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.840+0000 7fcb1be00700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb1c075b50 0x7fcb1c1a1240 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.840+0000 7fcb1be00700 1 --2- 172.21.15.44:0/423451399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb1c0764a0 0x7fcb1c1a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:12.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.840+0000 7fcb1be00700 1 -- 172.21.15.44:0/423451399 >> 172.21.15.44:0/423451399 conn(0x7fcb1c079d00 msgr2=0x7fcb1c1021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:12.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.840+0000 7fcb1be00700 1 -- 172.21.15.44:0/423451399 shutdown_connections 2024-05-27T09:36:12.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:12.840+0000 7fcb1be00700 1 -- 172.21.15.44:0/423451399 wait complete. 2024-05-27T09:36:14.028 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:36:14.028 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:36:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:14 smithi044 ceph-mon[31618]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-27T09:36:14.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:14 smithi050 ceph-mon[36371]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-27T09:36:14.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:14 smithi189 ceph-mon[36671]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-27T09:36:14.890 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.114+0000 7fbb79e00640 1 -- 172.21.15.44:0/1078570203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb74106420 msgr2=0x7fbb741127f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.114+0000 7fbb79e00640 1 --2- 172.21.15.44:0/1078570203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb74106420 0x7fbb741127f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009be0 tx=0x7fbb6c02a6c0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.115+0000 7fbb79e00640 1 -- 172.21.15.44:0/1078570203 shutdown_connections 2024-05-27T09:36:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.115+0000 7fbb79e00640 1 --2- 172.21.15.44:0/1078570203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb74106420 0x7fbb741127f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.115+0000 7fbb79e00640 1 --2- 172.21.15.44:0/1078570203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb7410c170 0x7fbb74105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.115+0000 7fbb79e00640 1 --2- 172.21.15.44:0/1078570203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb7410bcd0 0x7fbb74105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.115+0000 7fbb79e00640 1 -- 172.21.15.44:0/1078570203 >> 172.21.15.44:0/1078570203 conn(0x7fbb741013d0 msgr2=0x7fbb74103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.115+0000 7fbb79e00640 1 -- 172.21.15.44:0/1078570203 shutdown_connections 2024-05-27T09:36:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.115+0000 7fbb79e00640 1 -- 172.21.15.44:0/1078570203 wait complete. 2024-05-27T09:36:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.116+0000 7fbb79e00640 1 Processor -- start 2024-05-27T09:36:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.116+0000 7fbb79e00640 1 -- start start 2024-05-27T09:36:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.116+0000 7fbb79e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb74106420 0x7fbb741a2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb79e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb7410bcd0 0x7fbb741a2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb79e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb7410c170 0x7fbb7419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb79e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbb74115ee0 con 0x7fbb7410bcd0 2024-05-27T09:36:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb79e00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbb74115d60 con 0x7fbb7410c170 2024-05-27T09:36:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb79e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbb74116060 con 0x7fbb74106420 2024-05-27T09:36:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb78a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb74106420 0x7fbb741a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb78a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb74106420 0x7fbb741a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43046/0 (socket says 172.21.15.44:43046) 2024-05-27T09:36:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb78a00640 1 -- 172.21.15.44:0/3743116871 learned_addr learned my addr 172.21.15.44:0/3743116871 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb73e00640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb7410bcd0 0x7fbb741a2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb79400640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb7410c170 0x7fbb7419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb78a00640 1 -- 172.21.15.44:0/3743116871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb7410c170 msgr2=0x7fbb7419c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb78a00640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb7410c170 0x7fbb7419c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb78a00640 1 -- 172.21.15.44:0/3743116871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb7410bcd0 msgr2=0x7fbb741a2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb78a00640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb7410bcd0 0x7fbb741a2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.117+0000 7fbb78a00640 1 -- 172.21.15.44:0/3743116871 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7419ce30 con 0x7fbb74106420 2024-05-27T09:36:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.118+0000 7fbb78a00640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb74106420 0x7fbb741a2530 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800e9f0 tx=0x7fbb6800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.118+0000 7fbb71600640 1 -- 172.21.15.44:0/3743116871 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6800ccc0 con 0x7fbb74106420 2024-05-27T09:36:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.118+0000 7fbb79e00640 1 -- 172.21.15.44:0/3743116871 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7419d0c0 con 0x7fbb74106420 2024-05-27T09:36:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.118+0000 7fbb71600640 1 -- 172.21.15.44:0/3743116871 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb680054d0 con 0x7fbb74106420 2024-05-27T09:36:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.118+0000 7fbb71600640 1 -- 172.21.15.44:0/3743116871 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68010430 con 0x7fbb74106420 2024-05-27T09:36:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.118+0000 7fbb79e00640 1 -- 172.21.15.44:0/3743116871 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb741a92d0 con 0x7fbb74106420 2024-05-27T09:36:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.122+0000 7fbb823d7740 1 Processor -- start 2024-05-27T09:36:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.122+0000 7fbb823d7740 1 -- start start 2024-05-27T09:36:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.122+0000 7fbb71600640 1 -- 172.21.15.44:0/3743116871 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7fbb680046c0 con 0x7fbb74106420 2024-05-27T09:36:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.123+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a25a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.123+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a25fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.123+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426624d0 0x55ae42a264e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.123+0000 7fbb823d7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ae42996b90 con 0x55ae426831b0 2024-05-27T09:36:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.123+0000 7fbb823d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ae42986730 con 0x55ae4265a2f0 2024-05-27T09:36:15.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.123+0000 7fbb823d7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ae4298a770 con 0x55ae426624d0 2024-05-27T09:36:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.123+0000 7fbb79400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a25a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.123+0000 7fbb79400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a25a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49224/0 (socket says 172.21.15.44:49224) 2024-05-27T09:36:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.123+0000 7fbb79400640 1 -- 172.21.15.44:0/2485335858 learned_addr learned my addr 172.21.15.44:0/2485335858 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.123+0000 7fbb73e00640 1 --2- 172.21.15.44:0/2485335858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426624d0 0x55ae42a264e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.123+0000 7fbb78a00640 1 --2- 172.21.15.44:0/2485335858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a25fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.124+0000 7fbb79400640 1 -- 172.21.15.44:0/2485335858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426624d0 msgr2=0x55ae42a264e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.124+0000 7fbb79400640 1 --2- 172.21.15.44:0/2485335858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426624d0 0x55ae42a264e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.124+0000 7fbb79400640 1 -- 172.21.15.44:0/2485335858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 msgr2=0x55ae42a25fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.124+0000 7fbb79400640 1 --2- 172.21.15.44:0/2485335858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a25fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.124+0000 7fbb79400640 1 -- 172.21.15.44:0/2485335858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae4291f8b0 con 0x55ae426831b0 2024-05-27T09:36:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.124+0000 7fbb78a00640 1 --2- 172.21.15.44:0/2485335858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a25fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:36:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.124+0000 7fbb79400640 1 --2- 172.21.15.44:0/2485335858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a25a60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009be0 tx=0x7fbb6c030430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a6afe2f832b8c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.124+0000 7fbb65c00640 1 -- 172.21.15.44:0/2485335858 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c041030 con 0x55ae426831b0 2024-05-27T09:36:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.125+0000 7fbb65c00640 1 -- 172.21.15.44:0/2485335858 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6c003e20 con 0x55ae426831b0 2024-05-27T09:36:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.125+0000 7fbb65c00640 1 -- 172.21.15.44:0/2485335858 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c036040 con 0x55ae426831b0 2024-05-27T09:36:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.125+0000 7fbb823d7740 1 -- 172.21.15.44:0/2485335858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 msgr2=0x55ae42a25a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.125+0000 7fbb823d7740 1 --2- 172.21.15.44:0/2485335858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a25a60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009be0 tx=0x7fbb6c030430 comp rx=0 tx=0).stop 2024-05-27T09:36:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.125+0000 7fbb71600640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbb5c0b2b00 0x7fbb5c0b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.125+0000 7fbb823d7740 1 -- 172.21.15.44:0/2485335858 shutdown_connections 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.125+0000 7fbb823d7740 1 --2- 172.21.15.44:0/2485335858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426624d0 0x55ae42a264e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.125+0000 7fbb823d7740 1 --2- 172.21.15.44:0/2485335858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a25fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.125+0000 7fbb823d7740 1 --2- 172.21.15.44:0/2485335858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a25a60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.126+0000 7fbb823d7740 1 -- 172.21.15.44:0/2485335858 >> 172.21.15.44:0/2485335858 conn(0x55ae42677300 msgr2=0x55ae42a22210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.126+0000 7fbb71600640 1 -- 172.21.15.44:0/3743116871 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbb680e2bc0 con 0x7fbb74106420 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.126+0000 7fbb823d7740 1 -- 172.21.15.44:0/2485335858 shutdown_connections 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.126+0000 7fbb79400640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbb5c0b2b00 0x7fbb5c0b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.126+0000 7fbb823d7740 1 -- 172.21.15.44:0/2485335858 wait complete. 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.127+0000 7fbb79400640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbb5c0b2b00 0x7fbb5c0b5010 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c004af0 tx=0x7fbb6c02a6c0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.127+0000 7fbb823d7740 1 -- start start 2024-05-27T09:36:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.127+0000 7fbb823d7740 1 Processor -- start 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.127+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a2fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.127+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426624d0 0x55ae42a300f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.127+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a3be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb823d7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ae4266c790 con 0x55ae426831b0 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb823d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ae4292ebc0 con 0x55ae4265a2f0 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb823d7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ae42a32b30 con 0x55ae426624d0 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb823d7740 10 client.0 _ll_register_callbacks cb 0x55ae4290c400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb78a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a2fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb73e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426624d0 0x55ae42a300f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb79400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a3be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb78a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a2fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60946/0 (socket says 172.21.15.44:60946) 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb78a00640 1 -- 172.21.15.44:0/1094704657 learned_addr learned my addr 172.21.15.44:0/1094704657 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb78a00640 1 -- 172.21.15.44:0/1094704657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426624d0 msgr2=0x55ae42a300f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb78a00640 1 --2- 172.21.15.44:0/1094704657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426624d0 0x55ae42a300f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb78a00640 1 -- 172.21.15.44:0/1094704657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 msgr2=0x55ae42a3be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb78a00640 1 --2- 172.21.15.44:0/1094704657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a3be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.128+0000 7fbb78a00640 1 -- 172.21.15.44:0/1094704657 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae4269b210 con 0x55ae4265a2f0 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.129+0000 7fbb78a00640 1 --2- 172.21.15.44:0/1094704657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a2fbb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbb680e01f0 tx=0x7fbb680b29e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d2681f441a324bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.129+0000 7fbb5a000640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.129+0000 7fbb5a000640 1 -- 172.21.15.44:0/1094704657 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb680a95e0 con 0x55ae4265a2f0 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.129+0000 7fbb5a000640 1 -- 172.21.15.44:0/1094704657 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbb68005210 con 0x55ae4265a2f0 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.129+0000 7fbb823d7740 1 -- 172.21.15.44:0/1094704657 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb6c003e20 con 0x55ae4265a2f0 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.129+0000 7fbb5a000640 1 -- 172.21.15.44:0/1094704657 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbb680a5420 con 0x55ae4265a2f0 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.129+0000 7fbb823d7740 10 client.25397 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.129+0000 7fbb823d7740 1 -- 172.21.15.44:0/1094704657 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ae42767620 con 0x55ae4265a2f0 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.129+0000 7fbb5a000640 1 -- 172.21.15.44:0/1094704657 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6c003e20 con 0x55ae4265a2f0 2024-05-27T09:36:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.130+0000 7fbb5a000640 1 -- 172.21.15.44:0/1094704657 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbb680a8960 con 0x55ae4265a2f0 2024-05-27T09:36:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.130+0000 7fbb823d7740 10 client.25397 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.130+0000 7fbb823d7740 10 client.25397 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.130+0000 7fbb823d7740 20 client.25397 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.130+0000 7fbb823d7740 1 --2- 172.21.15.44:0/1094704657 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae42641d80 0x55ae42a41e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.130+0000 7fbb823d7740 4 client.25397 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.130+0000 7fbb823d7740 1 -- 172.21.15.44:0/1094704657 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae42998ae0 con 0x55ae42641d80 2024-05-27T09:36:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.130+0000 7fbb73e00640 1 --2- 172.21.15.44:0/1094704657 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae42641d80 0x55ae42a41e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.131+0000 7fbb73e00640 1 --2- 172.21.15.44:0/1094704657 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae42641d80 0x55ae42a41e10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b0847956d088bdf server_cookie=3ef1ed857e7abcf4 in_seq=0 out_seq=0 2024-05-27T09:36:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.131+0000 7fbb5a000640 10 client.25397 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb5a000640 1 -- 172.21.15.44:0/1094704657 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ae42998ae0 con 0x55ae42641d80 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb5a000640 10 client.25397 handle_command_reply: tid=0 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb823d7740 1 client.25397 shutdown 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb823d7740 1 -- 172.21.15.44:0/1094704657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 msgr2=0x55ae42a2fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb823d7740 1 --2- 172.21.15.44:0/1094704657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a2fbb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbb680e01f0 tx=0x7fbb680b29e0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb823d7740 1 -- 172.21.15.44:0/1094704657 shutdown_connections 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb823d7740 1 --2- 172.21.15.44:0/1094704657 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae42641d80 0x55ae42a41e10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb823d7740 1 --2- 172.21.15.44:0/1094704657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a3be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb823d7740 1 --2- 172.21.15.44:0/1094704657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426624d0 0x55ae42a300f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb823d7740 1 --2- 172.21.15.44:0/1094704657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a2fbb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb5a000640 0 client.25397 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb823d7740 1 -- 172.21.15.44:0/1094704657 >> 172.21.15.44:0/1094704657 conn(0x55ae42677300 msgr2=0x55ae42a25890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb823d7740 1 -- 172.21.15.44:0/1094704657 shutdown_connections 2024-05-27T09:36:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.132+0000 7fbb823d7740 1 -- 172.21.15.44:0/1094704657 wait complete. 2024-05-27T09:36:15.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.133+0000 7fbb823d7740 20 client.25397 trim_cache size 0 max 16384 2024-05-27T09:36:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.150+0000 7fbb823d7740 1 Processor -- start 2024-05-27T09:36:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.150+0000 7fbb823d7740 1 -- start start 2024-05-27T09:36:15.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.150+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae4265a2f0 0x55ae42a3bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.150+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae426624d0 0x55ae42a3c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.150+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a47610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.150+0000 7fbb823d7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741b4370 con 0x55ae426831b0 2024-05-27T09:36:15.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.150+0000 7fbb823d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741b1c40 con 0x55ae426624d0 2024-05-27T09:36:15.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.150+0000 7fbb823d7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741afa90 con 0x55ae4265a2f0 2024-05-27T09:36:15.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.151+0000 7fbb79400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a47610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.151+0000 7fbb73e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae426624d0 0x55ae42a3c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.151+0000 7fbb73e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae426624d0 0x55ae42a3c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60952/0 (socket says 172.21.15.44:60952) 2024-05-27T09:36:15.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.151+0000 7fbb73e00640 1 -- 172.21.15.44:0/3279758166 learned_addr learned my addr 172.21.15.44:0/3279758166 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:15.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.151+0000 7fbb78a00640 1 --2- 172.21.15.44:0/3279758166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae4265a2f0 0x55ae42a3bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.151+0000 7fbb73e00640 1 -- 172.21.15.44:0/3279758166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae4265a2f0 msgr2=0x55ae42a3bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.151+0000 7fbb73e00640 1 --2- 172.21.15.44:0/3279758166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae4265a2f0 0x55ae42a3bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.151+0000 7fbb73e00640 1 -- 172.21.15.44:0/3279758166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 msgr2=0x55ae42a47610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.151+0000 7fbb73e00640 1 --2- 172.21.15.44:0/3279758166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a47610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.151+0000 7fbb73e00640 1 -- 172.21.15.44:0/3279758166 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae42a25fc0 con 0x55ae426624d0 2024-05-27T09:36:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.152+0000 7fbb73e00640 1 --2- 172.21.15.44:0/3279758166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae426624d0 0x55ae42a3c110 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb7410c5a0 tx=0x7fbb60009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82dad841cfee5933 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.152+0000 7fbb65200640 1 -- 172.21.15.44:0/3279758166 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60003d60 con 0x55ae426624d0 2024-05-27T09:36:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.152+0000 7fbb65200640 1 -- 172.21.15.44:0/3279758166 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6001e930 con 0x55ae426624d0 2024-05-27T09:36:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.152+0000 7fbb65200640 1 -- 172.21.15.44:0/3279758166 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60010960 con 0x55ae426624d0 2024-05-27T09:36:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.153+0000 7fbb823d7740 1 -- 172.21.15.44:0/3279758166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae426624d0 msgr2=0x55ae42a3c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.153+0000 7fbb823d7740 1 --2- 172.21.15.44:0/3279758166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae426624d0 0x55ae42a3c110 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb7410c5a0 tx=0x7fbb60009260 comp rx=0 tx=0).stop 2024-05-27T09:36:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.153+0000 7fbb823d7740 1 -- 172.21.15.44:0/3279758166 shutdown_connections 2024-05-27T09:36:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.153+0000 7fbb823d7740 1 --2- 172.21.15.44:0/3279758166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426831b0 0x55ae42a47610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.153+0000 7fbb823d7740 1 --2- 172.21.15.44:0/3279758166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae426624d0 0x55ae42a3c110 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.153+0000 7fbb823d7740 1 --2- 172.21.15.44:0/3279758166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae4265a2f0 0x55ae42a3bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.153+0000 7fbb823d7740 1 -- 172.21.15.44:0/3279758166 >> 172.21.15.44:0/3279758166 conn(0x55ae42677300 msgr2=0x55ae42a2fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.153+0000 7fbb823d7740 1 -- 172.21.15.44:0/3279758166 shutdown_connections 2024-05-27T09:36:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.153+0000 7fbb823d7740 1 -- 172.21.15.44:0/3279758166 wait complete. 2024-05-27T09:36:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.154+0000 7fbb823d7740 1 -- start start 2024-05-27T09:36:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.154+0000 7fbb823d7740 1 Processor -- start 2024-05-27T09:36:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.154+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a383f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.154+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426624d0 0x55ae42a35760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.154+0000 7fbb823d7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426831b0 0x55ae42a35ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.154+0000 7fbb823d7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741b0100 con 0x55ae426624d0 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.154+0000 7fbb823d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741b1570 con 0x55ae4265a2f0 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.154+0000 7fbb823d7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741b07d0 con 0x55ae426831b0 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.155+0000 7fbb823d7740 10 client.0 _ll_register_callbacks cb 0x7fbb6800cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.155+0000 7fbb73e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426624d0 0x55ae42a35760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.155+0000 7fbb78a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.155+0000 7fbb73e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426624d0 0x55ae42a35760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49248/0 (socket says 172.21.15.44:49248) 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.155+0000 7fbb73e00640 1 -- 172.21.15.44:0/3548794297 learned_addr learned my addr 172.21.15.44:0/3548794297 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.155+0000 7fbb79400640 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426831b0 0x55ae42a35ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.155+0000 7fbb73e00640 1 -- 172.21.15.44:0/3548794297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426831b0 msgr2=0x55ae42a35ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.155+0000 7fbb73e00640 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426831b0 0x55ae42a35ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.155+0000 7fbb73e00640 1 -- 172.21.15.44:0/3548794297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 msgr2=0x55ae42a383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.155+0000 7fbb73e00640 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a383f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.155+0000 7fbb73e00640 1 -- 172.21.15.44:0/3548794297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae42a36310 con 0x55ae426624d0 2024-05-27T09:36:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.156+0000 7fbb73e00640 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426624d0 0x55ae42a35760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbb60000c00 tx=0x7fbb60005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9168991e0e98fd5e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.156+0000 7fbb5a000640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:36:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.156+0000 7fbb5a000640 1 -- 172.21.15.44:0/3548794297 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60036060 con 0x55ae426624d0 2024-05-27T09:36:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.156+0000 7fbb5a000640 1 -- 172.21.15.44:0/3548794297 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbb6001e680 con 0x55ae426624d0 2024-05-27T09:36:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.156+0000 7fbb823d7740 1 -- 172.21.15.44:0/3548794297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb6001e930 con 0x55ae426624d0 2024-05-27T09:36:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.156+0000 7fbb5a000640 1 -- 172.21.15.44:0/3548794297 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbb60003df0 con 0x55ae426624d0 2024-05-27T09:36:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.156+0000 7fbb823d7740 10 client.15264 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.156+0000 7fbb823d7740 1 -- 172.21.15.44:0/3548794297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ae42aaa300 con 0x55ae426624d0 2024-05-27T09:36:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.157+0000 7fbb5a000640 1 -- 172.21.15.44:0/3548794297 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6001e930 con 0x55ae426624d0 2024-05-27T09:36:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.157+0000 7fbb5a000640 1 -- 172.21.15.44:0/3548794297 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbb6001b070 con 0x55ae426624d0 2024-05-27T09:36:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.157+0000 7fbb823d7740 10 client.15264 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.157+0000 7fbb823d7740 10 client.15264 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.157+0000 7fbb823d7740 20 client.15264 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.157+0000 7fbb823d7740 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae426171c0 0x55ae42aac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.157+0000 7fbb823d7740 4 client.15264 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:36:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.157+0000 7fbb823d7740 1 -- 172.21.15.44:0/3548794297 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ae429922c0 con 0x55ae426171c0 2024-05-27T09:36:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.157+0000 7fbb78a00640 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae426171c0 0x55ae42aac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.158+0000 7fbb78a00640 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae426171c0 0x55ae42aac9e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=191103bcf3d526ff server_cookie=515dfa069c1cf370 in_seq=0 out_seq=0 2024-05-27T09:36:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.158+0000 7fbb5a000640 10 client.15264 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb5a000640 1 -- 172.21.15.44:0/3548794297 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55ae429922c0 con 0x55ae426171c0 2024-05-27T09:36:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb5a000640 10 client.15264 handle_command_reply: tid=0 2024-05-27T09:36:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb823d7740 1 client.15264 shutdown 2024-05-27T09:36:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb823d7740 1 -- 172.21.15.44:0/3548794297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426624d0 msgr2=0x55ae42a35760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb823d7740 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426624d0 0x55ae42a35760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbb60000c00 tx=0x7fbb60005780 comp rx=0 tx=0).stop 2024-05-27T09:36:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb823d7740 1 -- 172.21.15.44:0/3548794297 shutdown_connections 2024-05-27T09:36:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb823d7740 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae426171c0 0x55ae42aac9e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb823d7740 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae426831b0 0x55ae42a35ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb823d7740 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae426624d0 0x55ae42a35760 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb823d7740 1 --2- 172.21.15.44:0/3548794297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae4265a2f0 0x55ae42a383f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb823d7740 1 -- 172.21.15.44:0/3548794297 >> 172.21.15.44:0/3548794297 conn(0x55ae42677300 msgr2=0x55ae42a2a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.167+0000 7fbb5a000640 0 client.15264 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.168+0000 7fbb823d7740 1 -- 172.21.15.44:0/3548794297 shutdown_connections 2024-05-27T09:36:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.168+0000 7fbb823d7740 1 -- 172.21.15.44:0/3548794297 wait complete. 2024-05-27T09:36:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.168+0000 7fbb823d7740 20 client.15264 trim_cache size 0 max 16384 2024-05-27T09:36:15.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:36:15.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:36:15.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:36:15.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 -- 172.21.15.44:0/3743116871 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbb5c0b2b00 msgr2=0x7fbb5c0b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:15.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbb5c0b2b00 0x7fbb5c0b5010 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c004af0 tx=0x7fbb6c02a6c0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 -- 172.21.15.44:0/3743116871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb74106420 msgr2=0x7fbb741a2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb74106420 0x7fbb741a2530 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800e9f0 tx=0x7fbb6800edf0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 -- 172.21.15.44:0/3743116871 shutdown_connections 2024-05-27T09:36:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb7410c170 0x7fbb7419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbb5c0b2b00 0x7fbb5c0b5010 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb7410bcd0 0x7fbb741a2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 --2- 172.21.15.44:0/3743116871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb74106420 0x7fbb741a2530 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 -- 172.21.15.44:0/3743116871 >> 172.21.15.44:0/3743116871 conn(0x7fbb741013d0 msgr2=0x7fbb741086e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 -- 172.21.15.44:0/3743116871 shutdown_connections 2024-05-27T09:36:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:15.169+0000 7fbb79e00640 1 -- 172.21.15.44:0/3743116871 wait complete. 2024-05-27T09:36:16.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.970+0000 7f8e80400700 1 -- 172.21.15.44:0/2878210155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78109860 msgr2=0x7f8e781111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:16.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.970+0000 7f8e80400700 1 --2- 172.21.15.44:0/2878210155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78109860 0x7f8e781111c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8e70009f90 tx=0x7f8e7000b330 comp rx=0 tx=0).stop 2024-05-27T09:36:16.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.970+0000 7f8e80400700 1 -- 172.21.15.44:0/2878210155 shutdown_connections 2024-05-27T09:36:16.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.970+0000 7f8e80400700 1 --2- 172.21.15.44:0/2878210155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78109860 0x7f8e781111c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.970+0000 7f8e80400700 1 --2- 172.21.15.44:0/2878210155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e781045e0 0x7f8e781049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.970+0000 7f8e80400700 1 --2- 172.21.15.44:0/2878210155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e78108f10 0x7f8e78109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.970+0000 7f8e80400700 1 -- 172.21.15.44:0/2878210155 >> 172.21.15.44:0/2878210155 conn(0x7f8e78100030 msgr2=0x7f8e78102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.970+0000 7f8e80400700 1 -- 172.21.15.44:0/2878210155 shutdown_connections 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.970+0000 7f8e80400700 1 -- 172.21.15.44:0/2878210155 wait complete. 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.972+0000 7f8e80400700 1 Processor -- start 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.972+0000 7f8e80400700 1 -- start start 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e80400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e781045e0 0x7f8e78075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e80400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e78108f10 0x7f8e78075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e80400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78109860 0x7f8e78076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e80400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8e78114490 con 0x7f8e781045e0 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e80400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8e78114310 con 0x7f8e78109860 2024-05-27T09:36:16.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e80400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e78114610 con 0x7f8e78108f10 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e7da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e781045e0 0x7f8e78075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e7d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e78108f10 0x7f8e78075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e7da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e781045e0 0x7f8e78075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49266/0 (socket says 172.21.15.44:49266) 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e7e400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78109860 0x7f8e78076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e7da00700 1 -- 172.21.15.44:0/4156061644 learned_addr learned my addr 172.21.15.44:0/4156061644 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e7e400700 1 -- 172.21.15.44:0/4156061644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e78108f10 msgr2=0x7f8e78075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e7e400700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e78108f10 0x7f8e78075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e7e400700 1 -- 172.21.15.44:0/4156061644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e781045e0 msgr2=0x7f8e78075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.973+0000 7f8e7e400700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e781045e0 0x7f8e78075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.974+0000 7f8e7e400700 1 -- 172.21.15.44:0/4156061644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e781a9160 con 0x7f8e78109860 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.974+0000 7f8e7d000700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e78108f10 0x7f8e78075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.974+0000 7f8e7e400700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78109860 0x7f8e78076530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8e70009790 tx=0x7f8e7000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:16.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.974+0000 7f8e76000700 1 -- 172.21.15.44:0/4156061644 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e70021070 con 0x7f8e78109860 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.974+0000 7f8e76000700 1 -- 172.21.15.44:0/4156061644 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e7001d450 con 0x7f8e78109860 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.974+0000 7f8e80400700 1 -- 172.21.15.44:0/4156061644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e781a93f0 con 0x7f8e78109860 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.974+0000 7f8e76000700 1 -- 172.21.15.44:0/4156061644 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e70005280 con 0x7f8e78109860 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.974+0000 7f8e80400700 1 -- 172.21.15.44:0/4156061644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e781a98a0 con 0x7f8e78109860 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.978+0000 7f8e76000700 1 -- 172.21.15.44:0/4156061644 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f8e70024070 con 0x7f8e78109860 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.979+0000 7f8e8efd8b80 1 Processor -- start 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.979+0000 7f8e8efd8b80 1 -- start start 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b3459a0 0x563d2b3d82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3d8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3e50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563d2b306df0 con 0x563d2b259df0 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563d2b2f6990 con 0x563d2b3d8e20 2024-05-27T09:36:16.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563d2b2fa9d0 con 0x563d2b3459a0 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e7da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3d8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e7da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3d8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49272/0 (socket says 172.21.15.44:49272) 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e7da00700 1 -- 172.21.15.44:0/766465608 learned_addr learned my addr 172.21.15.44:0/766465608 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e7d000700 1 --2- 172.21.15.44:0/766465608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b3459a0 0x563d2b3d82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e76000700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8e600b2ba0 0x7f8e600b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.980+0000 7f8e7d000700 1 --2- 172.21.15.44:0/766465608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3e50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.981+0000 7f8e7da00700 1 -- 172.21.15.44:0/766465608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b3459a0 msgr2=0x563d2b3d82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.981+0000 7f8e7da00700 1 --2- 172.21.15.44:0/766465608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b3459a0 0x563d2b3d82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.981+0000 7f8e7da00700 1 -- 172.21.15.44:0/766465608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 msgr2=0x563d2b3e50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.981+0000 7f8e7da00700 1 --2- 172.21.15.44:0/766465608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3e50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.981+0000 7f8e7da00700 1 -- 172.21.15.44:0/766465608 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d2b2bf3d0 con 0x563d2b259df0 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.981+0000 7f8e76000700 1 -- 172.21.15.44:0/4156061644 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f8e7000f020 con 0x7f8e78109860 2024-05-27T09:36:16.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.981+0000 7f8e7e400700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8e600b2ba0 0x7f8e600b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.981+0000 7f8e7da00700 1 --2- 172.21.15.44:0/766465608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3d8820 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009ea0 tx=0x7f8e6c011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90722c162d926011 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:16.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.981+0000 7f8e6aa00700 1 -- 172.21.15.44:0/766465608 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c01e070 con 0x563d2b259df0 2024-05-27T09:36:16.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.981+0000 7f8e6aa00700 1 -- 172.21.15.44:0/766465608 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e6c00fb30 con 0x563d2b259df0 2024-05-27T09:36:16.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.981+0000 7f8e7e400700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8e600b2ba0 0x7f8e600b50b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8e700e1110 tx=0x7f8e7001dce0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:16.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.982+0000 7f8e6aa00700 1 -- 172.21.15.44:0/766465608 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c019c20 con 0x563d2b259df0 2024-05-27T09:36:16.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.982+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/766465608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 msgr2=0x563d2b3d8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:16.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.982+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/766465608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3d8820 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009ea0 tx=0x7f8e6c011720 comp rx=0 tx=0).stop 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.983+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/766465608 shutdown_connections 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.983+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/766465608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3e50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.983+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/766465608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b3459a0 0x563d2b3d82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.983+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/766465608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3d8820 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.983+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/766465608 >> 172.21.15.44:0/766465608 conn(0x563d2b2f3340 msgr2=0x563d2b3d4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.983+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/766465608 shutdown_connections 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.983+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/766465608 wait complete. 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.983+0000 7f8e8efd8b80 1 -- start start 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.984+0000 7f8e8efd8b80 1 Processor -- start 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.984+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b259df0 0x563d2b3e2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.984+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 0x563d2b3e2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.984+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3e30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.984+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563d2b1fcf50 con 0x563d2b3459a0 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.984+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563d2b297710 con 0x563d2b3d8e20 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.984+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563d2b3e5b30 con 0x563d2b259df0 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.984+0000 7f8e8efd8b80 10 client.0 _ll_register_callbacks cb 0x563d2b2a0c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.984+0000 7f8e7da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 0x563d2b3e2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.985+0000 7f8e7d000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3e30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.985+0000 7f8e7d000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3e30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60990/0 (socket says 172.21.15.44:60990) 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.985+0000 7f8e7d000700 1 -- 172.21.15.44:0/751989088 learned_addr learned my addr 172.21.15.44:0/751989088 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.985+0000 7f8e7d000700 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b259df0 0x563d2b3e2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.985+0000 7f8e7da00700 1 -- 172.21.15.44:0/751989088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b259df0 msgr2=0x563d2b3e2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.985+0000 7f8e7da00700 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b259df0 0x563d2b3e2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.985+0000 7f8e7da00700 1 -- 172.21.15.44:0/751989088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 msgr2=0x563d2b3e30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:16.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.985+0000 7f8e7da00700 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3e30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.985+0000 7f8e7da00700 1 -- 172.21.15.44:0/751989088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d2b246880 con 0x563d2b3459a0 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.985+0000 7f8e7d000700 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3e30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.986+0000 7f8e7da00700 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 0x563d2b3e2b60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x563d2b3d5740 tx=0x7f8e6c019600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab353f0bbce69f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.986+0000 7f8e5f400700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.986+0000 7f8e5f400700 1 -- 172.21.15.44:0/751989088 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c028060 con 0x563d2b3459a0 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.986+0000 7f8e5f400700 1 -- 172.21.15.44:0/751989088 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f8e6c00f910 con 0x563d2b3459a0 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.986+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/751989088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e6c00fb30 con 0x563d2b3459a0 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.986+0000 7f8e5f400700 1 -- 172.21.15.44:0/751989088 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8e6c01e040 con 0x563d2b3459a0 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.986+0000 7f8e8efd8b80 10 client.15276 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.986+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/751989088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563d2b2c7010 con 0x563d2b3459a0 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.987+0000 7f8e5f400700 1 -- 172.21.15.44:0/751989088 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e6c00fb30 con 0x563d2b3459a0 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.987+0000 7f8e5f400700 1 -- 172.21.15.44:0/751989088 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8e6c0197c0 con 0x563d2b3459a0 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.987+0000 7f8e8efd8b80 10 client.15276 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:16.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.987+0000 7f8e8efd8b80 10 client.15276 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T09:36:16.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.987+0000 7f8e8efd8b80 20 client.15276 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:16.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.987+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x563d2b3f27f0 0x563d2b3f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.987+0000 7f8e8efd8b80 4 client.15276 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:16.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.987+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/751989088 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d2b308d40 con 0x563d2b3f27f0 2024-05-27T09:36:16.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.988+0000 7f8e7d000700 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x563d2b3f27f0 0x563d2b3f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.988+0000 7f8e7d000700 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x563d2b3f27f0 0x563d2b3f3c60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=af48d7aa716d57c6 server_cookie=5651fe5168d89733 in_seq=0 out_seq=0 2024-05-27T09:36:16.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.988+0000 7f8e5f400700 10 client.15276 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:36:16.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e5f400700 1 -- 172.21.15.44:0/751989088 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563d2b308d40 con 0x563d2b3f27f0 2024-05-27T09:36:16.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e5f400700 10 client.15276 handle_command_reply: tid=0 2024-05-27T09:36:16.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e8efd8b80 1 client.15276 shutdown 2024-05-27T09:36:16.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/751989088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 msgr2=0x563d2b3e2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:16.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 0x563d2b3e2b60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x563d2b3d5740 tx=0x7f8e6c019600 comp rx=0 tx=0).stop 2024-05-27T09:36:16.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/751989088 shutdown_connections 2024-05-27T09:36:16.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3e30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b259df0 0x563d2b3e2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x563d2b3f27f0 0x563d2b3f3c60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/751989088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 0x563d2b3e2b60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/751989088 >> 172.21.15.44:0/751989088 conn(0x563d2b2f3340 msgr2=0x563d2b3d82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:16.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e5f400700 0 client.15276 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:36:16.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.989+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/751989088 shutdown_connections 2024-05-27T09:36:16.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.990+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/751989088 wait complete. 2024-05-27T09:36:16.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:15.990+0000 7f8e8efd8b80 20 client.15276 trim_cache size 0 max 16384 2024-05-27T09:36:16.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.014+0000 7f8e8efd8b80 1 Processor -- start 2024-05-27T09:36:16.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.014+0000 7f8e8efd8b80 1 -- start start 2024-05-27T09:36:16.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.014+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b259df0 0x563d2b3d4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.014+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 0x563d2b3eee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.014+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3d80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.014+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563d2b3f1fa0 con 0x563d2b3459a0 2024-05-27T09:36:16.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.014+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563d2b3f2670 con 0x563d2b3d8e20 2024-05-27T09:36:16.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.014+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563d2b3f2d40 con 0x563d2b259df0 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.015+0000 7f8e7da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 0x563d2b3eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.015+0000 7f8e7da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 0x563d2b3eee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49288/0 (socket says 172.21.15.44:49288) 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.015+0000 7f8e7da00700 1 -- 172.21.15.44:0/1270789044 learned_addr learned my addr 172.21.15.44:0/1270789044 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.015+0000 7f8e7d000700 1 --2- 172.21.15.44:0/1270789044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b259df0 0x563d2b3d4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.015+0000 7f8e7d000700 1 --2- 172.21.15.44:0/1270789044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3d80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.015+0000 7f8e7da00700 1 -- 172.21.15.44:0/1270789044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b259df0 msgr2=0x563d2b3d4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.015+0000 7f8e7da00700 1 --2- 172.21.15.44:0/1270789044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b259df0 0x563d2b3d4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.015+0000 7f8e7da00700 1 -- 172.21.15.44:0/1270789044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 msgr2=0x563d2b3d80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.015+0000 7f8e7da00700 1 --2- 172.21.15.44:0/1270789044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3d80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.015+0000 7f8e7da00700 1 -- 172.21.15.44:0/1270789044 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d2b3dde40 con 0x563d2b3459a0 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.015+0000 7f8e7d000700 1 --2- 172.21.15.44:0/1270789044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3d80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:36:16.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.016+0000 7f8e7da00700 1 --2- 172.21.15.44:0/1270789044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 0x563d2b3eee00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c01c040 tx=0x7f8e6c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=764528d813e617ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:16.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.016+0000 7f8e6a000700 1 -- 172.21.15.44:0/1270789044 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c028040 con 0x563d2b3459a0 2024-05-27T09:36:16.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.016+0000 7f8e6a000700 1 -- 172.21.15.44:0/1270789044 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e6c00f710 con 0x563d2b3459a0 2024-05-27T09:36:16.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.016+0000 7f8e6a000700 1 -- 172.21.15.44:0/1270789044 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c02db20 con 0x563d2b3459a0 2024-05-27T09:36:16.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.017+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1270789044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 msgr2=0x563d2b3eee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:16.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.017+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/1270789044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 0x563d2b3eee00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c01c040 tx=0x7f8e6c001e50 comp rx=0 tx=0).stop 2024-05-27T09:36:16.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.017+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1270789044 shutdown_connections 2024-05-27T09:36:16.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.017+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/1270789044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3d8e20 0x563d2b3d80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.017+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/1270789044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b259df0 0x563d2b3d4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.017+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/1270789044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b3459a0 0x563d2b3eee00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.017+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1270789044 >> 172.21.15.44:0/1270789044 conn(0x563d2b2f3340 msgr2=0x563d2b3d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:16.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.017+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1270789044 shutdown_connections 2024-05-27T09:36:16.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.017+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1270789044 wait complete. 2024-05-27T09:36:16.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.018+0000 7f8e8efd8b80 1 -- start start 2024-05-27T09:36:16.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.018+0000 7f8e8efd8b80 1 Processor -- start 2024-05-27T09:36:16.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3fd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3459a0 0x563d2b3f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e8efd8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b3d8e20 0x563d2b3f7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563d2b3f3ae0 con 0x563d2b259df0 2024-05-27T09:36:16.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563d2b3f3410 con 0x563d2b3459a0 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e8efd8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563d2b3f41b0 con 0x563d2b3d8e20 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e8efd8b80 10 client.0 _ll_register_callbacks cb 0x563d2b2a0c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e7d000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3fd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e7d000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3fd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49304/0 (socket says 172.21.15.44:49304) 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e7d000700 1 -- 172.21.15.44:0/1121690015 learned_addr learned my addr 172.21.15.44:0/1121690015 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e7da00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3459a0 0x563d2b3f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e7d000700 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b3d8e20 0x563d2b3f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e7da00700 1 -- 172.21.15.44:0/1121690015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b3d8e20 msgr2=0x563d2b3f7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e7da00700 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b3d8e20 0x563d2b3f7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e7da00700 1 -- 172.21.15.44:0/1121690015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 msgr2=0x563d2b3fd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e7da00700 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3fd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e7da00700 1 -- 172.21.15.44:0/1121690015 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d2b3f7720 con 0x563d2b3459a0 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.019+0000 7f8e7d000700 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3fd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.020+0000 7f8e7da00700 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3459a0 0x563d2b3f6ae0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c013010 tx=0x7f8e6c011a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfd67202b0574fd6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:16.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.020+0000 7f8e5f400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:16.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.020+0000 7f8e5f400700 1 -- 172.21.15.44:0/1121690015 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c043060 con 0x563d2b3459a0 2024-05-27T09:36:16.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.020+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1121690015 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e6c00f710 con 0x563d2b3459a0 2024-05-27T09:36:16.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.020+0000 7f8e5f400700 1 -- 172.21.15.44:0/1121690015 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f8e6c02d7a0 con 0x563d2b3459a0 2024-05-27T09:36:16.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.020+0000 7f8e5f400700 1 -- 172.21.15.44:0/1121690015 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8e6c028040 con 0x563d2b3459a0 2024-05-27T09:36:16.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.021+0000 7f8e8efd8b80 10 client.25421 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:16.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.021+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1121690015 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563d2b45df80 con 0x563d2b3459a0 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.021+0000 7f8e5f400700 1 -- 172.21.15.44:0/1121690015 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e6c00f710 con 0x563d2b3459a0 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.021+0000 7f8e5f400700 1 -- 172.21.15.44:0/1121690015 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8e6c023a10 con 0x563d2b3459a0 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.021+0000 7f8e8efd8b80 10 client.25421 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.021+0000 7f8e8efd8b80 10 client.25421 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.021+0000 7f8e8efd8b80 20 client.25421 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.021+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x563d2b45ffa0 0x563d2b461410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.021+0000 7f8e8efd8b80 4 client.25421 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.022+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1121690015 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "status"}) -- 0x563d2b302520 con 0x563d2b45ffa0 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.022+0000 7f8e7d000700 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x563d2b45ffa0 0x563d2b461410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.022+0000 7f8e7d000700 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x563d2b45ffa0 0x563d2b461410 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6077fa8947e680b server_cookie=bc696d22695501ed in_seq=0 out_seq=0 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.022+0000 7f8e5f400700 10 client.25421 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:36:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e5f400700 1 -- 172.21.15.44:0/1121690015 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563d2b302520 con 0x563d2b45ffa0 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e5f400700 10 client.25421 handle_command_reply: tid=0 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e8efd8b80 1 client.25421 shutdown 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1121690015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3459a0 msgr2=0x563d2b3f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3459a0 0x563d2b3f6ae0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c013010 tx=0x7f8e6c011a70 comp rx=0 tx=0).stop 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1121690015 shutdown_connections 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d2b3d8e20 0x563d2b3f7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x563d2b45ffa0 0x563d2b461410 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563d2b259df0 0x563d2b3fd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e8efd8b80 1 --2- 172.21.15.44:0/1121690015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563d2b3459a0 0x563d2b3f6ae0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1121690015 >> 172.21.15.44:0/1121690015 conn(0x563d2b2f3340 msgr2=0x563d2b3d8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e5f400700 0 client.25421 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1121690015 shutdown_connections 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.023+0000 7f8e8efd8b80 1 -- 172.21.15.44:0/1121690015 wait complete. 2024-05-27T09:36:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.024+0000 7f8e8efd8b80 20 client.25421 trim_cache size 0 max 16384 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 1, 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24535, 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 273.29258869300003, 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 26, 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 82, 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 81, 2024-05-27T09:36:16.028 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 322.01687193800001 2024-05-27T09:36:16.029 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:36:16.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.024+0000 7f8e80400700 1 -- 172.21.15.44:0/4156061644 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8e600b2ba0 msgr2=0x7f8e600b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:16.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.024+0000 7f8e80400700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8e600b2ba0 0x7f8e600b50b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8e700e1110 tx=0x7f8e7001dce0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.024+0000 7f8e80400700 1 -- 172.21.15.44:0/4156061644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78109860 msgr2=0x7f8e78076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:16.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.024+0000 7f8e80400700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78109860 0x7f8e78076530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8e70009790 tx=0x7f8e7000ccb0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.024+0000 7f8e80400700 1 -- 172.21.15.44:0/4156061644 shutdown_connections 2024-05-27T09:36:16.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.024+0000 7f8e80400700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8e600b2ba0 0x7f8e600b50b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.024+0000 7f8e80400700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78109860 0x7f8e78076530 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.024+0000 7f8e80400700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e781045e0 0x7f8e78075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.024+0000 7f8e80400700 1 --2- 172.21.15.44:0/4156061644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e78108f10 0x7f8e78075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:16.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.024+0000 7f8e80400700 1 -- 172.21.15.44:0/4156061644 >> 172.21.15.44:0/4156061644 conn(0x7f8e78100030 msgr2=0x7f8e781085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:16.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.025+0000 7f8e80400700 1 -- 172.21.15.44:0/4156061644 shutdown_connections 2024-05-27T09:36:16.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:16.025+0000 7f8e80400700 1 -- 172.21.15.44:0/4156061644 wait complete. 2024-05-27T09:36:16.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:16 smithi044 ceph-mon[31618]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-05-27T09:36:16.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:16 smithi050 ceph-mon[36371]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-05-27T09:36:16.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:16 smithi189 ceph-mon[36671]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-05-27T09:36:16.995 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T09:36:17.546 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:18.257 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:18 smithi044 ceph-mon[31618]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-27T09:36:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:18 smithi050 ceph-mon[36371]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-27T09:36:18.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:18 smithi189 ceph-mon[36671]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-27T09:36:18.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.893+0000 7fccc8800700 1 -- 172.21.15.44:0/2949972877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccc01045c0 msgr2=0x7fccc01049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:18.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.893+0000 7fccc8800700 1 --2- 172.21.15.44:0/2949972877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccc01045c0 0x7fccc01049d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fccb40099c0 tx=0x7fccb4009cd0 comp rx=0 tx=0).stop 2024-05-27T09:36:18.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.893+0000 7fccc8800700 1 -- 172.21.15.44:0/2949972877 shutdown_connections 2024-05-27T09:36:18.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.893+0000 7fccc8800700 1 --2- 172.21.15.44:0/2949972877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccc0109840 0x7fccc01111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:18.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.893+0000 7fccc8800700 1 --2- 172.21.15.44:0/2949972877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccc01045c0 0x7fccc01049d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:18.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.893+0000 7fccc8800700 1 --2- 172.21.15.44:0/2949972877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccc0108ef0 0x7fccc0109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:18.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.893+0000 7fccc8800700 1 -- 172.21.15.44:0/2949972877 >> 172.21.15.44:0/2949972877 conn(0x7fccc00ffe90 msgr2=0x7fccc01022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:18.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.893+0000 7fccc8800700 1 -- 172.21.15.44:0/2949972877 shutdown_connections 2024-05-27T09:36:18.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.893+0000 7fccc8800700 1 -- 172.21.15.44:0/2949972877 wait complete. 2024-05-27T09:36:18.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.894+0000 7fccc8800700 1 Processor -- start 2024-05-27T09:36:18.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.895+0000 7fccc8800700 1 -- start start 2024-05-27T09:36:18.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.895+0000 7fccc8800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccc01045c0 0x7fccc0076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:18.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.895+0000 7fccc8800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccc0108ef0 0x7fccc0077100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:18.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.895+0000 7fccc8800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccc0109840 0x7fccc0077640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:18.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.895+0000 7fccc8800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fccc01144e0 con 0x7fccc0108ef0 2024-05-27T09:36:18.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.895+0000 7fccc8800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fccc0114360 con 0x7fccc0109840 2024-05-27T09:36:18.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.895+0000 7fccc8800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fccc0114660 con 0x7fccc01045c0 2024-05-27T09:36:18.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.895+0000 7fccc5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccc0108ef0 0x7fccc0077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:18.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.895+0000 7fccc5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccc0108ef0 0x7fccc0077100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49324/0 (socket says 172.21.15.44:49324) 2024-05-27T09:36:18.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.896+0000 7fccc5400700 1 -- 172.21.15.44:0/1987234472 learned_addr learned my addr 172.21.15.44:0/1987234472 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:18.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.896+0000 7fccc5e00700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccc01045c0 0x7fccc0076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:18.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.896+0000 7fccc6800700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccc0109840 0x7fccc0077640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:18.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.896+0000 7fccc5400700 1 -- 172.21.15.44:0/1987234472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccc01045c0 msgr2=0x7fccc0076bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:18.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.896+0000 7fccc5400700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccc01045c0 0x7fccc0076bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:18.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.896+0000 7fccc5400700 1 -- 172.21.15.44:0/1987234472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccc0109840 msgr2=0x7fccc0077640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:18.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.896+0000 7fccc5400700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccc0109840 0x7fccc0077640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:18.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.896+0000 7fccc5400700 1 -- 172.21.15.44:0/1987234472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc01a9210 con 0x7fccc0108ef0 2024-05-27T09:36:18.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.896+0000 7fccc5e00700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccc01045c0 0x7fccc0076bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:18.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.897+0000 7fccc5400700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccc0108ef0 0x7fccc0077100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fccb000e9f0 tx=0x7fccb000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:18.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.897+0000 7fccbaa00700 1 -- 172.21.15.44:0/1987234472 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0004a30 con 0x7fccc0108ef0 2024-05-27T09:36:18.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.897+0000 7fccbaa00700 1 -- 172.21.15.44:0/1987234472 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccb000cc70 con 0x7fccc0108ef0 2024-05-27T09:36:18.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.897+0000 7fccbaa00700 1 -- 172.21.15.44:0/1987234472 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0005530 con 0x7fccc0108ef0 2024-05-27T09:36:18.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.897+0000 7fccc8800700 1 -- 172.21.15.44:0/1987234472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc01a9500 con 0x7fccc0108ef0 2024-05-27T09:36:18.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.897+0000 7fccc8800700 1 -- 172.21.15.44:0/1987234472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc01a9a40 con 0x7fccc0108ef0 2024-05-27T09:36:18.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.898+0000 7fccc8800700 1 -- 172.21.15.44:0/1987234472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccc00512b0 con 0x7fccc0108ef0 2024-05-27T09:36:18.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.902+0000 7fccbaa00700 1 -- 172.21.15.44:0/1987234472 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7fccb0004bd0 con 0x7fccc0108ef0 2024-05-27T09:36:18.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.903+0000 7fccbaa00700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcca80b2c20 0x7fcca80b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:18.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.904+0000 7fccbaa00700 1 -- 172.21.15.44:0/1987234472 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fccb00e2c20 con 0x7fccc0108ef0 2024-05-27T09:36:18.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.904+0000 7fccc5e00700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcca80b2c20 0x7fcca80b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:18.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.905+0000 7fccc5e00700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcca80b2c20 0x7fcca80b5130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fccb4000c00 tx=0x7fccb401a040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:18.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:18.909+0000 7fccbaa00700 1 -- 172.21.15.44:0/1987234472 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fccb000ce20 con 0x7fccc0108ef0 2024-05-27T09:36:19.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.206+0000 7fccc8800700 1 -- 172.21.15.44:0/1987234472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fccc002e5e0 con 0x7fccc0108ef0 2024-05-27T09:36:19.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.207+0000 7fccbaa00700 1 -- 172.21.15.44:0/1987234472 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7fccb00aa3f0 con 0x7fccc0108ef0 2024-05-27T09:36:19.209 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:36:19.209 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:36:19.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.210+0000 7fccc8800700 1 -- 172.21.15.44:0/1987234472 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcca80b2c20 msgr2=0x7fcca80b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:19.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.210+0000 7fccc8800700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcca80b2c20 0x7fcca80b5130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fccb4000c00 tx=0x7fccb401a040 comp rx=0 tx=0).stop 2024-05-27T09:36:19.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.210+0000 7fccc8800700 1 -- 172.21.15.44:0/1987234472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccc0108ef0 msgr2=0x7fccc0077100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:19.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.210+0000 7fccc8800700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccc0108ef0 0x7fccc0077100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fccb000e9f0 tx=0x7fccb000ed00 comp rx=0 tx=0).stop 2024-05-27T09:36:19.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.211+0000 7fccc8800700 1 -- 172.21.15.44:0/1987234472 shutdown_connections 2024-05-27T09:36:19.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.211+0000 7fccc8800700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcca80b2c20 0x7fcca80b5130 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:19.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.211+0000 7fccc8800700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccc0109840 0x7fccc0077640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:19.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.211+0000 7fccc8800700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccc01045c0 0x7fccc0076bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:19.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.211+0000 7fccc8800700 1 --2- 172.21.15.44:0/1987234472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccc0108ef0 0x7fccc0077100 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:19.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.211+0000 7fccc8800700 1 -- 172.21.15.44:0/1987234472 >> 172.21.15.44:0/1987234472 conn(0x7fccc00ffe90 msgr2=0x7fccc01022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:19.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.211+0000 7fccc8800700 1 -- 172.21.15.44:0/1987234472 shutdown_connections 2024-05-27T09:36:19.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:19.211+0000 7fccc8800700 1 -- 172.21.15.44:0/1987234472 wait complete. 2024-05-27T09:36:19.213 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:36:19.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:19 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1987234472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:36:19.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:19 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1987234472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:36:19.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:19 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1987234472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:36:20.091 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:36:20.299 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:20 smithi044 ceph-mon[31618]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-27T09:36:20.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:20 smithi050 ceph-mon[36371]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-27T09:36:20.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:20 smithi189 ceph-mon[36671]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-27T09:36:20.621 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.686+0000 7f1062a00700 1 -- 172.21.15.44:0/1439667144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f105c075d20 msgr2=0x7f105c076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.686+0000 7f1062a00700 1 --2- 172.21.15.44:0/1439667144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f105c075d20 0x7f105c076130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f104c009990 tx=0x7f104c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.686+0000 7f1062a00700 1 -- 172.21.15.44:0/1439667144 shutdown_connections 2024-05-27T09:36:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.686+0000 7f1062a00700 1 --2- 172.21.15.44:0/1439667144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f105c110b90 0x7f105c112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.686+0000 7f1062a00700 1 --2- 172.21.15.44:0/1439667144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f105c075d20 0x7f105c076130 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.686+0000 7f1062a00700 1 --2- 172.21.15.44:0/1439667144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f105c076670 0x7f105c110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.686+0000 7f1062a00700 1 -- 172.21.15.44:0/1439667144 >> 172.21.15.44:0/1439667144 conn(0x7f105c100030 msgr2=0x7f105c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:20.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.687+0000 7f1062a00700 1 -- 172.21.15.44:0/1439667144 shutdown_connections 2024-05-27T09:36:20.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.687+0000 7f1062a00700 1 -- 172.21.15.44:0/1439667144 wait complete. 2024-05-27T09:36:20.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.688+0000 7f1062a00700 1 Processor -- start 2024-05-27T09:36:20.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.689+0000 7f1062a00700 1 -- start start 2024-05-27T09:36:20.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.689+0000 7f1062a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f105c075d20 0x7f105c19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.689+0000 7f1062a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f105c076670 0x7f105c19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.689+0000 7f1062a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f105c110b90 0x7f105c1a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.689+0000 7f1062a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f105c1166d0 con 0x7f105c075d20 2024-05-27T09:36:20.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.689+0000 7f1062a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f105c116550 con 0x7f105c076670 2024-05-27T09:36:20.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.689+0000 7f1062a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f105c116850 con 0x7f105c110b90 2024-05-27T09:36:20.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f105be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f105c075d20 0x7f105c19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f1060a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f105c110b90 0x7f105c1a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f105b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f105c076670 0x7f105c19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f105b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f105c076670 0x7f105c19d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:32822/0 (socket says 172.21.15.44:32822) 2024-05-27T09:36:20.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f105b400700 1 -- 172.21.15.44:0/1898514607 learned_addr learned my addr 172.21.15.44:0/1898514607 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:20.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f1060a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f105c110b90 0x7f105c1a1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43198/0 (socket says 172.21.15.44:43198) 2024-05-27T09:36:20.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f105b400700 1 -- 172.21.15.44:0/1898514607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f105c110b90 msgr2=0x7f105c1a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:20.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f105b400700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f105c110b90 0x7f105c1a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f105b400700 1 -- 172.21.15.44:0/1898514607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f105c075d20 msgr2=0x7f105c19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:20.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f105b400700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f105c075d20 0x7f105c19d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f105b400700 1 -- 172.21.15.44:0/1898514607 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105c1a1ef0 con 0x7f105c076670 2024-05-27T09:36:20.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.690+0000 7f105be00700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f105c075d20 0x7f105c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:36:20.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.691+0000 7f105b400700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f105c076670 0x7f105c19d5c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f104800eb30 tx=0x7f104800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:20.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.691+0000 7f1058c00700 1 -- 172.21.15.44:0/1898514607 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104800cc90 con 0x7f105c076670 2024-05-27T09:36:20.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.691+0000 7f1058c00700 1 -- 172.21.15.44:0/1898514607 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10480044d0 con 0x7f105c076670 2024-05-27T09:36:20.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.691+0000 7f1058c00700 1 -- 172.21.15.44:0/1898514607 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1048010430 con 0x7f105c076670 2024-05-27T09:36:20.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.691+0000 7f1062a00700 1 -- 172.21.15.44:0/1898514607 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f105c1a21e0 con 0x7f105c076670 2024-05-27T09:36:20.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.691+0000 7f1062a00700 1 -- 172.21.15.44:0/1898514607 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f105c1a9b00 con 0x7f105c076670 2024-05-27T09:36:20.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.698+0000 7f1071516b80 1 Processor -- start 2024-05-27T09:36:20.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.698+0000 7f1071516b80 1 -- start start 2024-05-27T09:36:20.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.698+0000 7f1058c00700 1 -- 172.21.15.44:0/1898514607 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f1048003ea0 con 0x7f105c076670 2024-05-27T09:36:20.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.699+0000 7f1071516b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7e229a0 0x55aab7eb52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.699+0000 7f1071516b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 0x55aab7eb5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.699+0000 7f1071516b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7eb5e20 0x55aab7ec20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.699+0000 7f1071516b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55aab7de3df0 con 0x55aab7e229a0 2024-05-27T09:36:20.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.699+0000 7f1071516b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55aab7dd3990 con 0x55aab7eb5e20 2024-05-27T09:36:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.699+0000 7f1071516b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55aab7dd79d0 con 0x55aab7d36df0 2024-05-27T09:36:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.699+0000 7f1060a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7e229a0 0x55aab7eb52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.699+0000 7f1060a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7e229a0 0x55aab7eb52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49340/0 (socket says 172.21.15.44:49340) 2024-05-27T09:36:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.699+0000 7f105b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7eb5e20 0x55aab7ec20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.699+0000 7f1060a00700 1 -- 172.21.15.44:0/1718687862 learned_addr learned my addr 172.21.15.44:0/1718687862 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.699+0000 7f105be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 0x55aab7eb5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.700+0000 7f105be00700 1 -- 172.21.15.44:0/1718687862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7eb5e20 msgr2=0x55aab7ec20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.700+0000 7f105be00700 1 --2- 172.21.15.44:0/1718687862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7eb5e20 0x55aab7ec20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.700+0000 7f105be00700 1 -- 172.21.15.44:0/1718687862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7e229a0 msgr2=0x55aab7eb52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.700+0000 7f105be00700 1 --2- 172.21.15.44:0/1718687862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7e229a0 0x55aab7eb52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.700+0000 7f105be00700 1 -- 172.21.15.44:0/1718687862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aab7d9c3d0 con 0x55aab7d36df0 2024-05-27T09:36:20.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.700+0000 7f1060a00700 1 --2- 172.21.15.44:0/1718687862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7e229a0 0x55aab7eb52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:20.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.700+0000 7f105be00700 1 --2- 172.21.15.44:0/1718687862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 0x55aab7eb5820 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f104c00b2e0 tx=0x7f104c0096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c07245081a7783c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:20.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.700+0000 7f1051600700 1 -- 172.21.15.44:0/1718687862 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c0054a0 con 0x55aab7d36df0 2024-05-27T09:36:20.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.700+0000 7f1051600700 1 -- 172.21.15.44:0/1718687862 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f104c005640 con 0x55aab7d36df0 2024-05-27T09:36:20.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.700+0000 7f1051600700 1 -- 172.21.15.44:0/1718687862 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c003cc0 con 0x55aab7d36df0 2024-05-27T09:36:20.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.701+0000 7f1058c00700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10400b2c20 0x7f10400b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.701+0000 7f1071516b80 1 -- 172.21.15.44:0/1718687862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 msgr2=0x55aab7eb5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:20.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.701+0000 7f1071516b80 1 --2- 172.21.15.44:0/1718687862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 0x55aab7eb5820 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f104c00b2e0 tx=0x7f104c0096c0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.702+0000 7f1058c00700 1 -- 172.21.15.44:0/1898514607 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f10480e2a80 con 0x7f105c076670 2024-05-27T09:36:20.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.702+0000 7f1071516b80 1 -- 172.21.15.44:0/1718687862 shutdown_connections 2024-05-27T09:36:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.702+0000 7f1071516b80 1 --2- 172.21.15.44:0/1718687862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7eb5e20 0x55aab7ec20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.702+0000 7f1071516b80 1 --2- 172.21.15.44:0/1718687862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7e229a0 0x55aab7eb52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.702+0000 7f1071516b80 1 --2- 172.21.15.44:0/1718687862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 0x55aab7eb5820 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.702+0000 7f1071516b80 1 -- 172.21.15.44:0/1718687862 >> 172.21.15.44:0/1718687862 conn(0x55aab7dd0340 msgr2=0x55aab7eb1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.702+0000 7f1060a00700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10400b2c20 0x7f10400b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.702+0000 7f1071516b80 1 -- 172.21.15.44:0/1718687862 shutdown_connections 2024-05-27T09:36:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.702+0000 7f1071516b80 1 -- 172.21.15.44:0/1718687862 wait complete. 2024-05-27T09:36:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.703+0000 7f1071516b80 1 -- start start 2024-05-27T09:36:20.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.703+0000 7f1060a00700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10400b2c20 0x7f10400b5130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55aab7eb7fa0 tx=0x7f105400b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:20.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.703+0000 7f1071516b80 1 Processor -- start 2024-05-27T09:36:20.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.704+0000 7f1071516b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 0x55aab7ebf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.704+0000 7f1071516b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 0x55aab7ebfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.704+0000 7f105be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 0x55aab7ebf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.704+0000 7f1071516b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7ebff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.704+0000 7f105be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 0x55aab7ebf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:32826/0 (socket says 172.21.15.44:32826) 2024-05-27T09:36:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.704+0000 7f105be00700 1 -- 172.21.15.44:0/350361065 learned_addr learned my addr 172.21.15.44:0/350361065 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.704+0000 7f1071516b80 1 -- 172.21.15.44:0/350361065 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55aab7cd9f50 con 0x55aab7eb5e20 2024-05-27T09:36:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.704+0000 7f1071516b80 1 -- 172.21.15.44:0/350361065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55aab7d74710 con 0x55aab7d36df0 2024-05-27T09:36:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.704+0000 7f1071516b80 1 -- 172.21.15.44:0/350361065 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55aab7ec2ad0 con 0x55aab7e229a0 2024-05-27T09:36:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.704+0000 7f1071516b80 10 client.0 _ll_register_callbacks cb 0x55aab7d7dc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.705+0000 7f1060a00700 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7ebff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.705+0000 7f105be00700 1 -- 172.21.15.44:0/350361065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 msgr2=0x55aab7ebfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.705+0000 7f105b400700 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 0x55aab7ebfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.705+0000 7f105be00700 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 0x55aab7ebfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.705+0000 7f105be00700 1 -- 172.21.15.44:0/350361065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 msgr2=0x55aab7ebff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.705+0000 7f105be00700 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7ebff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.705+0000 7f105be00700 1 -- 172.21.15.44:0/350361065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aab7d23880 con 0x55aab7d36df0 2024-05-27T09:36:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.705+0000 7f1060a00700 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7ebff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:36:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.705+0000 7f105b400700 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 0x55aab7ebfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:36:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.705+0000 7f105be00700 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 0x55aab7ebf510 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f104c005b40 tx=0x7f104c005500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ceb037ddeb9c19 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.705+0000 7f1046000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.706+0000 7f1046000700 1 -- 172.21.15.44:0/350361065 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c00b9f0 con 0x55aab7d36df0 2024-05-27T09:36:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.706+0000 7f1071516b80 1 -- 172.21.15.44:0/350361065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f104c005640 con 0x55aab7d36df0 2024-05-27T09:36:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.706+0000 7f1046000700 1 -- 172.21.15.44:0/350361065 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f104c027070 con 0x55aab7d36df0 2024-05-27T09:36:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.706+0000 7f1046000700 1 -- 172.21.15.44:0/350361065 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f104c00bb90 con 0x55aab7d36df0 2024-05-27T09:36:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.706+0000 7f1071516b80 10 client.25439 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.706+0000 7f1071516b80 1 -- 172.21.15.44:0/350361065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55aab7da4010 con 0x55aab7d36df0 2024-05-27T09:36:20.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.706+0000 7f1046000700 1 -- 172.21.15.44:0/350361065 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f104c005640 con 0x55aab7d36df0 2024-05-27T09:36:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.707+0000 7f1046000700 1 -- 172.21.15.44:0/350361065 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f104c00f460 con 0x55aab7d36df0 2024-05-27T09:36:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.707+0000 7f1071516b80 10 client.25439 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.707+0000 7f1071516b80 10 client.25439 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T09:36:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.707+0000 7f1071516b80 20 client.25439 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.707+0000 7f1071516b80 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55aab7ecf420 0x55aab7ed0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.707+0000 7f1071516b80 4 client.25439 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.707+0000 7f1071516b80 1 -- 172.21.15.44:0/350361065 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aab7de5d40 con 0x55aab7ecf420 2024-05-27T09:36:20.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.707+0000 7f105b400700 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55aab7ecf420 0x55aab7ed0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.708+0000 7f105b400700 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55aab7ecf420 0x55aab7ed0890 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af74c4bff871e0aa server_cookie=886f134246f8ae91 in_seq=0 out_seq=0 2024-05-27T09:36:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.708+0000 7f1046000700 10 client.25439 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.709+0000 7f1046000700 1 -- 172.21.15.44:0/350361065 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55aab7de5d40 con 0x55aab7ecf420 2024-05-27T09:36:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.709+0000 7f1046000700 10 client.25439 handle_command_reply: tid=0 2024-05-27T09:36:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.709+0000 7f1071516b80 1 client.25439 shutdown 2024-05-27T09:36:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.709+0000 7f1071516b80 1 -- 172.21.15.44:0/350361065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 msgr2=0x55aab7ebf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.709+0000 7f1071516b80 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 0x55aab7ebf510 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f104c005b40 tx=0x7f104c005500 comp rx=0 tx=0).stop 2024-05-27T09:36:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.710+0000 7f1071516b80 1 -- 172.21.15.44:0/350361065 shutdown_connections 2024-05-27T09:36:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.710+0000 7f1071516b80 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55aab7ecf420 0x55aab7ed0890 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.710+0000 7f1071516b80 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7ebff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.710+0000 7f1071516b80 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 0x55aab7ebf510 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.710+0000 7f1071516b80 1 --2- 172.21.15.44:0/350361065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 0x55aab7ebfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.710+0000 7f1071516b80 1 -- 172.21.15.44:0/350361065 >> 172.21.15.44:0/350361065 conn(0x55aab7dd0340 msgr2=0x55aab7eb52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.710+0000 7f1046000700 0 client.25439 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.710+0000 7f1071516b80 1 -- 172.21.15.44:0/350361065 shutdown_connections 2024-05-27T09:36:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.710+0000 7f1071516b80 1 -- 172.21.15.44:0/350361065 wait complete. 2024-05-27T09:36:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.710+0000 7f1071516b80 20 client.25439 trim_cache size 0 max 16384 2024-05-27T09:36:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.734+0000 7f1071516b80 1 Processor -- start 2024-05-27T09:36:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.734+0000 7f1071516b80 1 -- start start 2024-05-27T09:36:20.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f1071516b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 0x55aab7ebad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f1071516b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 0x55aab7ecbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f1071516b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7eb5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f1071516b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55aab7ecebd0 con 0x55aab7eb5e20 2024-05-27T09:36:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f1071516b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55aab7ecf2a0 con 0x55aab7d36df0 2024-05-27T09:36:20.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f1071516b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55aab7ecf970 con 0x55aab7e229a0 2024-05-27T09:36:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f105b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 0x55aab7ecbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f1060a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7eb5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f105b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 0x55aab7ecbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43224/0 (socket says 172.21.15.44:43224) 2024-05-27T09:36:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f105b400700 1 -- 172.21.15.44:0/3491579427 learned_addr learned my addr 172.21.15.44:0/3491579427 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f105be00700 1 --2- 172.21.15.44:0/3491579427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 0x55aab7ebad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f105b400700 1 -- 172.21.15.44:0/3491579427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 msgr2=0x55aab7ebad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:20.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f105b400700 1 --2- 172.21.15.44:0/3491579427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 0x55aab7ebad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f105b400700 1 -- 172.21.15.44:0/3491579427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 msgr2=0x55aab7eb5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:20.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f105b400700 1 --2- 172.21.15.44:0/3491579427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7eb5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f105b400700 1 -- 172.21.15.44:0/3491579427 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aab7ec0300 con 0x55aab7e229a0 2024-05-27T09:36:20.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.735+0000 7f105be00700 1 --2- 172.21.15.44:0/3491579427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 0x55aab7ebad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:36:20.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.736+0000 7f105b400700 1 --2- 172.21.15.44:0/3491579427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 0x55aab7ecbda0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f10480a5bd0 tx=0x7f10480e6530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3368746d4fc72d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:20.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.736+0000 7f1050c00700 1 -- 172.21.15.44:0/3491579427 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10480e9030 con 0x55aab7e229a0 2024-05-27T09:36:20.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.736+0000 7f1050c00700 1 -- 172.21.15.44:0/3491579427 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1048009500 con 0x55aab7e229a0 2024-05-27T09:36:20.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.736+0000 7f1050c00700 1 -- 172.21.15.44:0/3491579427 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10480ec040 con 0x55aab7e229a0 2024-05-27T09:36:20.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.737+0000 7f1071516b80 1 -- 172.21.15.44:0/3491579427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 msgr2=0x55aab7ecbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:20.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.737+0000 7f1071516b80 1 --2- 172.21.15.44:0/3491579427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 0x55aab7ecbda0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f10480a5bd0 tx=0x7f10480e6530 comp rx=0 tx=0).stop 2024-05-27T09:36:20.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.737+0000 7f1071516b80 1 -- 172.21.15.44:0/3491579427 shutdown_connections 2024-05-27T09:36:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.737+0000 7f1071516b80 1 --2- 172.21.15.44:0/3491579427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7eb5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.737+0000 7f1071516b80 1 --2- 172.21.15.44:0/3491579427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7d36df0 0x55aab7ebad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.737+0000 7f1071516b80 1 --2- 172.21.15.44:0/3491579427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7e229a0 0x55aab7ecbda0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.737+0000 7f1071516b80 1 -- 172.21.15.44:0/3491579427 >> 172.21.15.44:0/3491579427 conn(0x55aab7dd0340 msgr2=0x55aab7eb5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.737+0000 7f1071516b80 1 -- 172.21.15.44:0/3491579427 shutdown_connections 2024-05-27T09:36:20.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.737+0000 7f1071516b80 1 -- 172.21.15.44:0/3491579427 wait complete. 2024-05-27T09:36:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.738+0000 7f1071516b80 1 -- start start 2024-05-27T09:36:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.738+0000 7f1071516b80 1 Processor -- start 2024-05-27T09:36:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f1071516b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 0x55aab7eba290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f1071516b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7e229a0 0x55aab7eba7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f1071516b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7ed6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f1071516b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55aab7ed0710 con 0x55aab7eb5e20 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f1071516b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55aab7ed0040 con 0x55aab7e229a0 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f1071516b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55aab7ed0de0 con 0x55aab7d36df0 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f1071516b80 10 client.0 _ll_register_callbacks cb 0x55aab7d7dc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f105be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 0x55aab7eba290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f105be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 0x55aab7eba290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43228/0 (socket says 172.21.15.44:43228) 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f105be00700 1 -- 172.21.15.44:0/3476018485 learned_addr learned my addr 172.21.15.44:0/3476018485 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f105b400700 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7e229a0 0x55aab7eba7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.739+0000 7f1060a00700 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7ed6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.740+0000 7f105b400700 1 -- 172.21.15.44:0/3476018485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 msgr2=0x55aab7eba290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.740+0000 7f105b400700 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 0x55aab7eba290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.740+0000 7f105b400700 1 -- 172.21.15.44:0/3476018485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 msgr2=0x55aab7ed6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:20.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.740+0000 7f105b400700 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7ed6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.740+0000 7f105b400700 1 -- 172.21.15.44:0/3476018485 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aab7ebb000 con 0x55aab7e229a0 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.740+0000 7f105be00700 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 0x55aab7eba290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.740+0000 7f105b400700 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7e229a0 0x55aab7eba7d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55aab7eb5df0 tx=0x7f10480038b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a22a3f167e817389 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.740+0000 7f1046000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.740+0000 7f1046000700 1 -- 172.21.15.44:0/3476018485 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10480e9040 con 0x55aab7e229a0 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.741+0000 7f1071516b80 1 -- 172.21.15.44:0/3476018485 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1048009500 con 0x55aab7e229a0 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.741+0000 7f1046000700 1 -- 172.21.15.44:0/3476018485 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f10480cae00 con 0x55aab7e229a0 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.741+0000 7f1046000700 1 -- 172.21.15.44:0/3476018485 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f10480c3050 con 0x55aab7e229a0 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.741+0000 7f1071516b80 10 client.25445 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.741+0000 7f1071516b80 1 -- 172.21.15.44:0/3476018485 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55aab7f43820 con 0x55aab7e229a0 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.741+0000 7f1046000700 1 -- 172.21.15.44:0/3476018485 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1048009500 con 0x55aab7e229a0 2024-05-27T09:36:20.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.741+0000 7f1046000700 1 -- 172.21.15.44:0/3476018485 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f10480a9db0 con 0x55aab7e229a0 2024-05-27T09:36:20.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.742+0000 7f1071516b80 10 client.25445 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:20.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.742+0000 7f1071516b80 10 client.25445 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T09:36:20.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.742+0000 7f1071516b80 20 client.25445 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:20.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.742+0000 7f1071516b80 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55aab7f45710 0x55aab7f46b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:20.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.742+0000 7f1071516b80 4 client.25445 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T09:36:20.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.742+0000 7f1071516b80 1 -- 172.21.15.44:0/3476018485 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "status"}) -- 0x55aab7ddf520 con 0x55aab7f45710 2024-05-27T09:36:20.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.742+0000 7f105be00700 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55aab7f45710 0x55aab7f46b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:20.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f105be00700 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55aab7f45710 0x55aab7f46b80 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bff27e0312970998 server_cookie=84df9d64562ffebe in_seq=0 out_seq=0 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1046000700 10 client.25445 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1046000700 1 -- 172.21.15.44:0/3476018485 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55aab7ddf520 con 0x55aab7f45710 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1046000700 10 client.25445 handle_command_reply: tid=0 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1071516b80 1 client.25445 shutdown 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1071516b80 1 -- 172.21.15.44:0/3476018485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7e229a0 msgr2=0x55aab7eba7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1071516b80 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7e229a0 0x55aab7eba7d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55aab7eb5df0 tx=0x7f10480038b0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1071516b80 1 -- 172.21.15.44:0/3476018485 shutdown_connections 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1071516b80 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55aab7f45710 0x55aab7f46b80 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1071516b80 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aab7eb5e20 0x55aab7ed6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1071516b80 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aab7d36df0 0x55aab7eba290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1071516b80 1 --2- 172.21.15.44:0/3476018485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aab7e229a0 0x55aab7eba7d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1071516b80 1 -- 172.21.15.44:0/3476018485 >> 172.21.15.44:0/3476018485 conn(0x55aab7dd0340 msgr2=0x55aab7ebdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:20.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.743+0000 7f1046000700 0 client.25445 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.744+0000 7f1071516b80 1 -- 172.21.15.44:0/3476018485 shutdown_connections 2024-05-27T09:36:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.744+0000 7f1071516b80 1 -- 172.21.15.44:0/3476018485 wait complete. 2024-05-27T09:36:20.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.744+0000 7f1071516b80 20 client.25445 trim_cache size 0 max 16384 2024-05-27T09:36:20.748 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:36:20.748 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "604b015a-1c0a-11ef-bc9b-c7b262605968", 2024-05-27T09:36:20.748 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-05-27T09:36:20.748 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24542, 2024-05-27T09:36:20.748 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-05-27T09:36:20.748 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-05-27T09:36:20.748 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-05-27T09:36:20.749 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 292.81172038400001, 2024-05-27T09:36:20.749 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 26, 2024-05-27T09:36:20.749 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 82, 2024-05-27T09:36:20.749 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 81, 2024-05-27T09:36:20.749 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 315.08191312100001 2024-05-27T09:36:20.749 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:36:20.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 -- 172.21.15.44:0/1898514607 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10400b2c20 msgr2=0x7f10400b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10400b2c20 0x7f10400b5130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55aab7eb7fa0 tx=0x7f105400b040 comp rx=0 tx=0).stop 2024-05-27T09:36:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 -- 172.21.15.44:0/1898514607 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f105c076670 msgr2=0x7f105c19d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f105c076670 0x7f105c19d5c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f104800eb30 tx=0x7f104800ee40 comp rx=0 tx=0).stop 2024-05-27T09:36:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 -- 172.21.15.44:0/1898514607 shutdown_connections 2024-05-27T09:36:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10400b2c20 0x7f10400b5130 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f105c110b90 0x7f105c1a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f105c075d20 0x7f105c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 --2- 172.21.15.44:0/1898514607 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f105c076670 0x7f105c19d5c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 -- 172.21.15.44:0/1898514607 >> 172.21.15.44:0/1898514607 conn(0x7f105c100030 msgr2=0x7f105c101dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 -- 172.21.15.44:0/1898514607 shutdown_connections 2024-05-27T09:36:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:20.745+0000 7f1062a00700 1 -- 172.21.15.44:0/1898514607 wait complete. 2024-05-27T09:36:22.307 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-05-27T09:36:22.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:22 smithi044 ceph-mon[31618]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-05-27T09:36:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:22 smithi050 ceph-mon[36371]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-05-27T09:36:22.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:22 smithi189 ceph-mon[36671]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-05-27T09:36:23.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.703+0000 7fbed1800700 1 -- 172.21.15.44:0/4137044938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbecc10b890 msgr2=0x7fbecc1131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:23.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.703+0000 7fbed1800700 1 --2- 172.21.15.44:0/4137044938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbecc10b890 0x7fbecc1131f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbec400b0d0 tx=0x7fbec400b3e0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.704+0000 7fbed1800700 1 -- 172.21.15.44:0/4137044938 shutdown_connections 2024-05-27T09:36:23.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.704+0000 7fbed1800700 1 --2- 172.21.15.44:0/4137044938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbecc10b890 0x7fbecc1131f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.704+0000 7fbed1800700 1 --2- 172.21.15.44:0/4137044938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc106610 0x7fbecc106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.704+0000 7fbed1800700 1 --2- 172.21.15.44:0/4137044938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbecc10af40 0x7fbecc10b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.704+0000 7fbed1800700 1 -- 172.21.15.44:0/4137044938 >> 172.21.15.44:0/4137044938 conn(0x7fbecc102020 msgr2=0x7fbecc104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:23.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.704+0000 7fbed1800700 1 -- 172.21.15.44:0/4137044938 shutdown_connections 2024-05-27T09:36:23.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.704+0000 7fbed1800700 1 -- 172.21.15.44:0/4137044938 wait complete. 2024-05-27T09:36:23.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.705+0000 7fbed1800700 1 Processor -- start 2024-05-27T09:36:23.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.705+0000 7fbed1800700 1 -- start start 2024-05-27T09:36:23.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.706+0000 7fbed1800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc106610 0x7fbecc19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.706+0000 7fbed1800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbecc10af40 0x7fbecc19d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.706+0000 7fbed1800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbecc10b890 0x7fbecc1a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.706+0000 7fbed1800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1164c0 con 0x7fbecc10b890 2024-05-27T09:36:23.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.706+0000 7fbed1800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc116340 con 0x7fbecc106610 2024-05-27T09:36:23.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.706+0000 7fbed1800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc116640 con 0x7fbecc10af40 2024-05-27T09:36:23.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.706+0000 7fbed0e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbecc10b890 0x7fbecc1a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.706+0000 7fbecbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc106610 0x7fbecc19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.706+0000 7fbecbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc106610 0x7fbecc19cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60626/0 (socket says 172.21.15.44:60626) 2024-05-27T09:36:23.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.706+0000 7fbecbe00700 1 -- 172.21.15.44:0/1831377872 learned_addr learned my addr 172.21.15.44:0/1831377872 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:23.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.707+0000 7fbecb400700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbecc10af40 0x7fbecc19d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.707+0000 7fbecbe00700 1 -- 172.21.15.44:0/1831377872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbecc10af40 msgr2=0x7fbecc19d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:23.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.707+0000 7fbecbe00700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbecc10af40 0x7fbecc19d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.707+0000 7fbecbe00700 1 -- 172.21.15.44:0/1831377872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbecc10b890 msgr2=0x7fbecc1a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:23.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.707+0000 7fbecbe00700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbecc10b890 0x7fbecc1a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.707+0000 7fbecbe00700 1 -- 172.21.15.44:0/1831377872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc1a1ca0 con 0x7fbecc106610 2024-05-27T09:36:23.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.707+0000 7fbecbe00700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc106610 0x7fbecc19cdd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00eb30 tx=0x7fbebc00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:23.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.708+0000 7fbec8c00700 1 -- 172.21.15.44:0/1831377872 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc00cbd0 con 0x7fbecc106610 2024-05-27T09:36:23.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.708+0000 7fbed1800700 1 -- 172.21.15.44:0/1831377872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbecc1a1f90 con 0x7fbecc106610 2024-05-27T09:36:23.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.708+0000 7fbec8c00700 1 -- 172.21.15.44:0/1831377872 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbebc0044d0 con 0x7fbecc106610 2024-05-27T09:36:23.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.708+0000 7fbec8c00700 1 -- 172.21.15.44:0/1831377872 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc003e30 con 0x7fbecc106610 2024-05-27T09:36:23.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.708+0000 7fbed1800700 1 -- 172.21.15.44:0/1831377872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbecc1a98b0 con 0x7fbecc106610 2024-05-27T09:36:23.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.712+0000 7fbec8c00700 1 -- 172.21.15.44:0/1831377872 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7fbebc004000 con 0x7fbecc106610 2024-05-27T09:36:23.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.713+0000 7fbee1904b80 1 Processor -- start 2024-05-27T09:36:23.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.713+0000 7fbee1904b80 1 -- start start 2024-05-27T09:36:23.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.713+0000 7fbec8c00700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbeb00b2ba0 0x7fbeb00b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 0x55ba3fe07750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbec8c00700 1 -- 172.21.15.44:0/1831377872 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbebc0e2c00 con 0x7fbecc106610 2024-05-27T09:36:23.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe07c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fe112c0 0x55ba3fe136e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbee1904b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ba3fd34e40 con 0x55ba3fd739f0 2024-05-27T09:36:23.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbee1904b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ba3fd249e0 con 0x55ba3fe112c0 2024-05-27T09:36:23.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbee1904b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ba3fd28a20 con 0x55ba3fc87e40 2024-05-27T09:36:23.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbed0e00700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbeb00b2ba0 0x7fbeb00b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbecbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 0x55ba3fe07750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbecb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe07c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbecb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe07c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42910/0 (socket says 172.21.15.44:42910) 2024-05-27T09:36:23.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbecbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 0x55ba3fe07750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58362/0 (socket says 172.21.15.44:58362) 2024-05-27T09:36:23.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbecb400700 1 -- 172.21.15.44:0/1757674739 learned_addr learned my addr 172.21.15.44:0/1757674739 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:23.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.714+0000 7fbed0e00700 1 --2- 172.21.15.44:0/1757674739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fe112c0 0x55ba3fe136e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.715+0000 7fbecbe00700 1 -- 172.21.15.44:0/1757674739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 msgr2=0x55ba3fe07c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:23.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.715+0000 7fbecbe00700 1 --2- 172.21.15.44:0/1757674739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe07c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.715+0000 7fbecbe00700 1 -- 172.21.15.44:0/1757674739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fe112c0 msgr2=0x55ba3fe136e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:23.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.715+0000 7fbecbe00700 1 --2- 172.21.15.44:0/1757674739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fe112c0 0x55ba3fe136e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.715+0000 7fbecbe00700 1 -- 172.21.15.44:0/1757674739 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba3fced420 con 0x55ba3fd739f0 2024-05-27T09:36:23.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.715+0000 7fbecb400700 1 --2- 172.21.15.44:0/1757674739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe07c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:23.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.715+0000 7fbed0e00700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbeb00b2ba0 0x7fbeb00b50b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009fd0 tx=0x7fbec40093c0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:23.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.715+0000 7fbecbe00700 1 --2- 172.21.15.44:0/1757674739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 0x55ba3fe07750 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0e0140 tx=0x7fbebc0a9aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e75539017c19ba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:23.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.715+0000 7fbec1600700 1 -- 172.21.15.44:0/1757674739 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0c1030 con 0x55ba3fd739f0 2024-05-27T09:36:23.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.716+0000 7fbec1600700 1 -- 172.21.15.44:0/1757674739 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbebc0a97e0 con 0x55ba3fd739f0 2024-05-27T09:36:23.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.716+0000 7fbec1600700 1 -- 172.21.15.44:0/1757674739 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0b2b30 con 0x55ba3fd739f0 2024-05-27T09:36:23.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.717+0000 7fbee1904b80 1 -- 172.21.15.44:0/1757674739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 msgr2=0x55ba3fe07750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.717+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1757674739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 0x55ba3fe07750 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0e0140 tx=0x7fbebc0a9aa0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.717+0000 7fbee1904b80 1 -- 172.21.15.44:0/1757674739 shutdown_connections 2024-05-27T09:36:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.717+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1757674739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fe112c0 0x55ba3fe136e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.717+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1757674739 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 0x55ba3fe07750 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.717+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1757674739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe07c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.717+0000 7fbee1904b80 1 -- 172.21.15.44:0/1757674739 >> 172.21.15.44:0/1757674739 conn(0x55ba3fd21390 msgr2=0x55ba3fe02710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.717+0000 7fbee1904b80 1 -- 172.21.15.44:0/1757674739 shutdown_connections 2024-05-27T09:36:23.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.717+0000 7fbee1904b80 1 -- 172.21.15.44:0/1757674739 wait complete. 2024-05-27T09:36:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.718+0000 7fbee1904b80 1 -- start start 2024-05-27T09:36:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.718+0000 7fbee1904b80 1 Processor -- start 2024-05-27T09:36:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.718+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe0e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fd739f0 0x55ba3fe0e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fe112c0 0x55ba3fe0ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbee1904b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ba3fc2afa0 con 0x55ba3fe112c0 2024-05-27T09:36:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbee1904b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ba3fcc5760 con 0x55ba3fd739f0 2024-05-27T09:36:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbee1904b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ba3fe13e00 con 0x55ba3fc87e40 2024-05-27T09:36:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbee1904b80 10 client.0 _ll_register_callbacks cb 0x55ba3fccec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbed0e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fe112c0 0x55ba3fe0ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbecbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe0e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbecb400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fd739f0 0x55ba3fe0e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbecbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe0e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42916/0 (socket says 172.21.15.44:42916) 2024-05-27T09:36:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbecbe00700 1 -- 172.21.15.44:0/1927777658 learned_addr learned my addr 172.21.15.44:0/1927777658 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbecbe00700 1 -- 172.21.15.44:0/1927777658 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fd739f0 msgr2=0x55ba3fe0e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbecbe00700 1 --2- 172.21.15.44:0/1927777658 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fd739f0 0x55ba3fe0e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbecbe00700 1 -- 172.21.15.44:0/1927777658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fe112c0 msgr2=0x55ba3fe0ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbecbe00700 1 --2- 172.21.15.44:0/1927777658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fe112c0 0x55ba3fe0ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.719+0000 7fbecbe00700 1 -- 172.21.15.44:0/1927777658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba3fc748d0 con 0x55ba3fc87e40 2024-05-27T09:36:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.720+0000 7fbecbe00700 1 --2- 172.21.15.44:0/1927777658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe0e280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0a5d50 tx=0x7fbebc0a94d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff6591f1cd0ede10 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.720+0000 7fbeb6000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:36:23.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.720+0000 7fbeb6000700 1 -- 172.21.15.44:0/1927777658 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc06d070 con 0x55ba3fc87e40 2024-05-27T09:36:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.720+0000 7fbeb6000700 1 -- 172.21.15.44:0/1927777658 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbebc0a62c0 con 0x55ba3fc87e40 2024-05-27T09:36:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.720+0000 7fbee1904b80 1 -- 172.21.15.44:0/1927777658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbebc0a97e0 con 0x55ba3fc87e40 2024-05-27T09:36:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.720+0000 7fbeb6000700 1 -- 172.21.15.44:0/1927777658 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbebc0c1070 con 0x55ba3fc87e40 2024-05-27T09:36:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.720+0000 7fbee1904b80 10 client.25456 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:23.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.720+0000 7fbee1904b80 1 -- 172.21.15.44:0/1927777658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ba3fcf5060 con 0x55ba3fc87e40 2024-05-27T09:36:23.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.721+0000 7fbeb6000700 1 -- 172.21.15.44:0/1927777658 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbebc0a97e0 con 0x55ba3fc87e40 2024-05-27T09:36:23.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.721+0000 7fbeb6000700 1 -- 172.21.15.44:0/1927777658 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbebc0e0450 con 0x55ba3fc87e40 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.721+0000 7fbee1904b80 10 client.25456 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.721+0000 7fbee1904b80 10 client.25456 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.721+0000 7fbee1904b80 20 client.25456 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.721+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1927777658 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55ba3fe20ba0 0x55ba3fe22fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.721+0000 7fbee1904b80 4 client.25456 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.721+0000 7fbee1904b80 1 -- 172.21.15.44:0/1927777658 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba3fd36d90 con 0x55ba3fe20ba0 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.721+0000 7fbecb400700 1 --2- 172.21.15.44:0/1927777658 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55ba3fe20ba0 0x55ba3fe22fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.722+0000 7fbecb400700 1 --2- 172.21.15.44:0/1927777658 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55ba3fe20ba0 0x55ba3fe22fc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c11323669aac5b2 server_cookie=cc581b33276657d6 in_seq=0 out_seq=0 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.722+0000 7fbeb6000700 10 client.25456 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbeb6000700 1 -- 172.21.15.44:0/1927777658 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ba3fd36d90 con 0x55ba3fe20ba0 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbeb6000700 10 client.25456 handle_command_reply: tid=0 2024-05-27T09:36:23.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbee1904b80 1 client.25456 shutdown 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbee1904b80 1 -- 172.21.15.44:0/1927777658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 msgr2=0x55ba3fe0e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1927777658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe0e280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0a5d50 tx=0x7fbebc0a94d0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbee1904b80 1 -- 172.21.15.44:0/1927777658 shutdown_connections 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1927777658 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55ba3fe20ba0 0x55ba3fe22fc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1927777658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fe112c0 0x55ba3fe0ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1927777658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe0e280 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1927777658 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fd739f0 0x55ba3fe0e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbeb6000700 0 client.25456 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbee1904b80 1 -- 172.21.15.44:0/1927777658 >> 172.21.15.44:0/1927777658 conn(0x55ba3fd21390 msgr2=0x55ba3fe07820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.723+0000 7fbee1904b80 1 -- 172.21.15.44:0/1927777658 shutdown_connections 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.724+0000 7fbee1904b80 1 -- 172.21.15.44:0/1927777658 wait complete. 2024-05-27T09:36:23.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.724+0000 7fbee1904b80 20 client.25456 trim_cache size 0 max 16384 2024-05-27T09:36:23.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.741+0000 7fbee1904b80 1 Processor -- start 2024-05-27T09:36:23.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.741+0000 7fbee1904b80 1 -- start start 2024-05-27T09:36:23.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.741+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe1d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.741+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fd739f0 0x55ba3fe02740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.741+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fe112c0 0x55ba3fe07dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.741+0000 7fbee1904b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1b3690 con 0x55ba3fe112c0 2024-05-27T09:36:23.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.741+0000 7fbee1904b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1b0f20 con 0x55ba3fd739f0 2024-05-27T09:36:23.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.741+0000 7fbee1904b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1aed70 con 0x55ba3fc87e40 2024-05-27T09:36:23.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.741+0000 7fbed0e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fe112c0 0x55ba3fe07dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.741+0000 7fbed0e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fe112c0 0x55ba3fe07dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58380/0 (socket says 172.21.15.44:58380) 2024-05-27T09:36:23.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.741+0000 7fbed0e00700 1 -- 172.21.15.44:0/1321668741 learned_addr learned my addr 172.21.15.44:0/1321668741 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:23.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.742+0000 7fbecb400700 1 --2- 172.21.15.44:0/1321668741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fd739f0 0x55ba3fe02740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.742+0000 7fbecbe00700 1 --2- 172.21.15.44:0/1321668741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe1d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.742+0000 7fbecb400700 1 -- 172.21.15.44:0/1321668741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 msgr2=0x55ba3fe1d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:23.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.742+0000 7fbecb400700 1 --2- 172.21.15.44:0/1321668741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe1d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.742+0000 7fbecb400700 1 -- 172.21.15.44:0/1321668741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fe112c0 msgr2=0x55ba3fe07dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:23.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.742+0000 7fbecb400700 1 --2- 172.21.15.44:0/1321668741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fe112c0 0x55ba3fe07dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.742+0000 7fbecb400700 1 -- 172.21.15.44:0/1321668741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba3fe0f070 con 0x55ba3fd739f0 2024-05-27T09:36:23.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.742+0000 7fbecbe00700 1 --2- 172.21.15.44:0/1321668741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe1d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:36:23.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.742+0000 7fbecb400700 1 --2- 172.21.15.44:0/1321668741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fd739f0 0x55ba3fe02740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55ba3fcc57c0 tx=0x7fbeb8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28b2ac09b1af444a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:23.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.743+0000 7fbec0c00700 1 -- 172.21.15.44:0/1321668741 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb801e430 con 0x55ba3fd739f0 2024-05-27T09:36:23.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.743+0000 7fbec0c00700 1 -- 172.21.15.44:0/1321668741 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb801ea80 con 0x55ba3fd739f0 2024-05-27T09:36:23.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.743+0000 7fbec0c00700 1 -- 172.21.15.44:0/1321668741 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb801ccb0 con 0x55ba3fd739f0 2024-05-27T09:36:23.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.743+0000 7fbee1904b80 1 -- 172.21.15.44:0/1321668741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fd739f0 msgr2=0x55ba3fe02740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:23.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.743+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1321668741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fd739f0 0x55ba3fe02740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55ba3fcc57c0 tx=0x7fbeb8004750 comp rx=0 tx=0).stop 2024-05-27T09:36:23.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.744+0000 7fbee1904b80 1 -- 172.21.15.44:0/1321668741 shutdown_connections 2024-05-27T09:36:23.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.744+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1321668741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fe112c0 0x55ba3fe07dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.744+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1321668741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fc87e40 0x55ba3fe1d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.744+0000 7fbee1904b80 1 --2- 172.21.15.44:0/1321668741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fd739f0 0x55ba3fe02740 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.744+0000 7fbee1904b80 1 -- 172.21.15.44:0/1321668741 >> 172.21.15.44:0/1321668741 conn(0x55ba3fd21390 msgr2=0x55ba3fe0f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:23.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.744+0000 7fbee1904b80 1 -- 172.21.15.44:0/1321668741 shutdown_connections 2024-05-27T09:36:23.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.744+0000 7fbee1904b80 1 -- 172.21.15.44:0/1321668741 wait complete. 2024-05-27T09:36:23.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.744+0000 7fbee1904b80 1 -- start start 2024-05-27T09:36:23.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.745+0000 7fbee1904b80 1 Processor -- start 2024-05-27T09:36:23.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.745+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fc87e40 0x55ba3fe23430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.745+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 0x55ba3fe23970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.745+0000 7fbee1904b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fe112c0 0x55ba3fe89400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.745+0000 7fbee1904b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1af3e0 con 0x55ba3fd739f0 2024-05-27T09:36:23.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.745+0000 7fbee1904b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1b0850 con 0x55ba3fc87e40 2024-05-27T09:36:23.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.745+0000 7fbee1904b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1afab0 con 0x55ba3fe112c0 2024-05-27T09:36:23.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.745+0000 7fbee1904b80 10 client.0 _ll_register_callbacks cb 0x55ba3fccec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:23.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.746+0000 7fbecbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fc87e40 0x55ba3fe23430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.746+0000 7fbecbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fc87e40 0x55ba3fe23430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60650/0 (socket says 172.21.15.44:60650) 2024-05-27T09:36:23.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.746+0000 7fbecbe00700 1 -- 172.21.15.44:0/2603999820 learned_addr learned my addr 172.21.15.44:0/2603999820 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:23.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.746+0000 7fbecb400700 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 0x55ba3fe23970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.746+0000 7fbed0e00700 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fe112c0 0x55ba3fe89400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.746+0000 7fbecbe00700 1 -- 172.21.15.44:0/2603999820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fe112c0 msgr2=0x55ba3fe89400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:23.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.746+0000 7fbecbe00700 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fe112c0 0x55ba3fe89400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.746+0000 7fbecbe00700 1 -- 172.21.15.44:0/2603999820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 msgr2=0x55ba3fe23970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:23.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.746+0000 7fbecbe00700 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 0x55ba3fe23970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.746+0000 7fbecbe00700 1 -- 172.21.15.44:0/2603999820 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba3fe89940 con 0x55ba3fc87e40 2024-05-27T09:36:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.746+0000 7fbed0e00700 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fe112c0 0x55ba3fe89400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:36:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.747+0000 7fbecbe00700 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fc87e40 0x55ba3fe23430 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00c5b0 tx=0x7fbebc00cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b5fb193a188d0fc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.747+0000 7fbeb6000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.747+0000 7fbeb6000700 1 -- 172.21.15.44:0/2603999820 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc06d070 con 0x55ba3fc87e40 2024-05-27T09:36:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.747+0000 7fbeb6000700 1 -- 172.21.15.44:0/2603999820 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbebc0c4360 con 0x55ba3fc87e40 2024-05-27T09:36:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.747+0000 7fbee1904b80 1 -- 172.21.15.44:0/2603999820 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbeb801ea80 con 0x55ba3fc87e40 2024-05-27T09:36:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.747+0000 7fbeb6000700 1 -- 172.21.15.44:0/2603999820 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbebc0042f0 con 0x55ba3fc87e40 2024-05-27T09:36:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.747+0000 7fbee1904b80 10 client.25463 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:23.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.747+0000 7fbee1904b80 1 -- 172.21.15.44:0/2603999820 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ba3fe8aad0 con 0x55ba3fc87e40 2024-05-27T09:36:23.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.748+0000 7fbeb6000700 1 -- 172.21.15.44:0/2603999820 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb801ea80 con 0x55ba3fc87e40 2024-05-27T09:36:23.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.748+0000 7fbeb6000700 1 -- 172.21.15.44:0/2603999820 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbebc0e0ad0 con 0x55ba3fc87e40 2024-05-27T09:36:23.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.748+0000 7fbee1904b80 10 client.25463 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:23.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.748+0000 7fbee1904b80 10 client.25463 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:36:23.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.748+0000 7fbee1904b80 20 client.25463 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:23.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.748+0000 7fbee1904b80 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55ba3fe8aee0 0x55ba3fe8d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:23.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.748+0000 7fbee1904b80 4 client.25463 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:36:23.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.748+0000 7fbee1904b80 1 -- 172.21.15.44:0/2603999820 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ba3fd30570 con 0x55ba3fe8aee0 2024-05-27T09:36:23.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.748+0000 7fbecb400700 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55ba3fe8aee0 0x55ba3fe8d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:23.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.749+0000 7fbecb400700 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55ba3fe8aee0 0x55ba3fe8d300 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e93f9cbef99ccd9a server_cookie=f86800c3867813f3 in_seq=0 out_seq=0 2024-05-27T09:36:23.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.749+0000 7fbeb6000700 10 client.25463 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:36:23.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.749+0000 7fbeb6000700 1 -- 172.21.15.44:0/2603999820 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ba3fd30570 con 0x55ba3fe8aee0 2024-05-27T09:36:23.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.749+0000 7fbeb6000700 10 client.25463 handle_command_reply: tid=0 2024-05-27T09:36:23.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.749+0000 7fbee1904b80 1 client.25463 shutdown 2024-05-27T09:36:23.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.749+0000 7fbee1904b80 1 -- 172.21.15.44:0/2603999820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fc87e40 msgr2=0x55ba3fe23430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:23.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.749+0000 7fbee1904b80 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fc87e40 0x55ba3fe23430 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00c5b0 tx=0x7fbebc00cef0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.749+0000 7fbee1904b80 1 -- 172.21.15.44:0/2603999820 shutdown_connections 2024-05-27T09:36:23.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.749+0000 7fbee1904b80 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba3fe112c0 0x55ba3fe89400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.749+0000 7fbee1904b80 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ba3fc87e40 0x55ba3fe23430 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.749+0000 7fbee1904b80 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55ba3fe8aee0 0x55ba3fe8d300 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.750+0000 7fbee1904b80 1 --2- 172.21.15.44:0/2603999820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ba3fd739f0 0x55ba3fe23970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.750+0000 7fbee1904b80 1 -- 172.21.15.44:0/2603999820 >> 172.21.15.44:0/2603999820 conn(0x55ba3fd21390 msgr2=0x55ba3fe068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:23.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.750+0000 7fbeb6000700 0 client.25463 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:36:23.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.750+0000 7fbee1904b80 1 -- 172.21.15.44:0/2603999820 shutdown_connections 2024-05-27T09:36:23.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.750+0000 7fbee1904b80 1 -- 172.21.15.44:0/2603999820 wait complete. 2024-05-27T09:36:23.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.750+0000 7fbee1904b80 20 client.25463 trim_cache size 0 max 16384 2024-05-27T09:36:23.757 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:36:23.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 -- 172.21.15.44:0/1831377872 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbeb00b2ba0 msgr2=0x7fbeb00b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:23.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbeb00b2ba0 0x7fbeb00b50b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009fd0 tx=0x7fbec40093c0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 -- 172.21.15.44:0/1831377872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc106610 msgr2=0x7fbecc19cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:23.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc106610 0x7fbecc19cdd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00eb30 tx=0x7fbebc00ee40 comp rx=0 tx=0).stop 2024-05-27T09:36:23.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 -- 172.21.15.44:0/1831377872 shutdown_connections 2024-05-27T09:36:23.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbeb00b2ba0 0x7fbeb00b50b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbecc10b890 0x7fbecc1a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc106610 0x7fbecc19cdd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 --2- 172.21.15.44:0/1831377872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbecc10af40 0x7fbecc19d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:23.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 -- 172.21.15.44:0/1831377872 >> 172.21.15.44:0/1831377872 conn(0x7fbecc102020 msgr2=0x7fbecc10a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:23.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 -- 172.21.15.44:0/1831377872 shutdown_connections 2024-05-27T09:36:23.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:23.751+0000 7fbed1800700 1 -- 172.21.15.44:0/1831377872 wait complete. 2024-05-27T09:36:24.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:24 smithi044 ceph-mon[31618]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-27T09:36:24.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:24 smithi050 ceph-mon[36371]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-27T09:36:24.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:24 smithi189 ceph-mon[36671]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-27T09:36:24.631 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:36:25.115 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:26.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:26 smithi044 ceph-mon[31618]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-05-27T09:36:26.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:26 smithi050 ceph-mon[36371]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-05-27T09:36:26.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:26 smithi189 ceph-mon[36671]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-05-27T09:36:27.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.508+0000 7fbd56600700 1 -- 172.21.15.44:0/1284014406 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd50109840 msgr2=0x7fbd501111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:27.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.508+0000 7fbd56600700 1 --2- 172.21.15.44:0/1284014406 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd50109840 0x7fbd501111a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbd48009f90 tx=0x7fbd4800b330 comp rx=0 tx=0).stop 2024-05-27T09:36:27.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.508+0000 7fbd56600700 1 -- 172.21.15.44:0/1284014406 shutdown_connections 2024-05-27T09:36:27.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.508+0000 7fbd56600700 1 --2- 172.21.15.44:0/1284014406 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd50109840 0x7fbd501111a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.508+0000 7fbd56600700 1 --2- 172.21.15.44:0/1284014406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd501045c0 0x7fbd501049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.508+0000 7fbd56600700 1 --2- 172.21.15.44:0/1284014406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd50108ef0 0x7fbd50109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.508+0000 7fbd56600700 1 -- 172.21.15.44:0/1284014406 >> 172.21.15.44:0/1284014406 conn(0x7fbd500ffe90 msgr2=0x7fbd501022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:27.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.508+0000 7fbd56600700 1 -- 172.21.15.44:0/1284014406 shutdown_connections 2024-05-27T09:36:27.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.508+0000 7fbd56600700 1 -- 172.21.15.44:0/1284014406 wait complete. 2024-05-27T09:36:27.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.509+0000 7fbd56600700 1 Processor -- start 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.510+0000 7fbd56600700 1 -- start start 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.510+0000 7fbd56600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd501045c0 0x7fbd50076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.510+0000 7fbd56600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd50108ef0 0x7fbd50077100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.510+0000 7fbd56600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd50109840 0x7fbd50077640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.510+0000 7fbd56600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd501144c0 con 0x7fbd50108ef0 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.510+0000 7fbd56600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbd50114340 con 0x7fbd50109840 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.510+0000 7fbd56600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbd50114640 con 0x7fbd501045c0 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.511+0000 7fbd46a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd50108ef0 0x7fbd50077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.511+0000 7fbd46a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd50108ef0 0x7fbd50077100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58414/0 (socket says 172.21.15.44:58414) 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.511+0000 7fbd4f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd501045c0 0x7fbd50076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.511+0000 7fbd46a00700 1 -- 172.21.15.44:0/3563656673 learned_addr learned my addr 172.21.15.44:0/3563656673 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:27.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.511+0000 7fbd4fe00700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd50109840 0x7fbd50077640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.511+0000 7fbd46a00700 1 -- 172.21.15.44:0/3563656673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd501045c0 msgr2=0x7fbd50076bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:27.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.511+0000 7fbd46a00700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd501045c0 0x7fbd50076bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.511+0000 7fbd46a00700 1 -- 172.21.15.44:0/3563656673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd50109840 msgr2=0x7fbd50077640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:27.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.511+0000 7fbd46a00700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd50109840 0x7fbd50077640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.511+0000 7fbd46a00700 1 -- 172.21.15.44:0/3563656673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd501a9190 con 0x7fbd50108ef0 2024-05-27T09:36:27.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.512+0000 7fbd46a00700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd50108ef0 0x7fbd50077100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800e970 tx=0x7fbd3800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:27.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.512+0000 7fbd4cc00700 1 -- 172.21.15.44:0/3563656673 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38004a30 con 0x7fbd50108ef0 2024-05-27T09:36:27.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.512+0000 7fbd4cc00700 1 -- 172.21.15.44:0/3563656673 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd3800cc70 con 0x7fbd50108ef0 2024-05-27T09:36:27.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.512+0000 7fbd56600700 1 -- 172.21.15.44:0/3563656673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd501a9480 con 0x7fbd50108ef0 2024-05-27T09:36:27.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.512+0000 7fbd4cc00700 1 -- 172.21.15.44:0/3563656673 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38005580 con 0x7fbd50108ef0 2024-05-27T09:36:27.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.512+0000 7fbd56600700 1 -- 172.21.15.44:0/3563656673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd501a99c0 con 0x7fbd50108ef0 2024-05-27T09:36:27.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.516+0000 7fbd4cc00700 1 -- 172.21.15.44:0/3563656673 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7fbd3800c7c0 con 0x7fbd50108ef0 2024-05-27T09:36:27.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.517+0000 7fbd65144b80 1 Processor -- start 2024-05-27T09:36:27.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.517+0000 7fbd65144b80 1 -- start start 2024-05-27T09:36:27.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 0x55da9e082870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9e082e70 0x55da9e08f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd65144b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55da9dfb0e40 con 0x55da9df03e40 2024-05-27T09:36:27.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd65144b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55da9dfa09e0 con 0x55da9dfef9f0 2024-05-27T09:36:27.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd4cc00700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd340b2ba0 0x7fbd340b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd65144b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55da9dfa4a20 con 0x55da9e082e70 2024-05-27T09:36:27.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd4cc00700 1 -- 172.21.15.44:0/3563656673 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbd380e29a0 con 0x7fbd50108ef0 2024-05-27T09:36:27.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd4f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 0x55da9e082870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd4fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd4f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 0x55da9e082870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58424/0 (socket says 172.21.15.44:58424) 2024-05-27T09:36:27.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd4fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60674/0 (socket says 172.21.15.44:60674) 2024-05-27T09:36:27.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd4f400700 1 -- 172.21.15.44:0/470954718 learned_addr learned my addr 172.21.15.44:0/470954718 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd4fe00700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd340b2ba0 0x7fbd340b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.518+0000 7fbd46a00700 1 --2- 172.21.15.44:0/470954718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9e082e70 0x55da9e08f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.519+0000 7fbd4f400700 1 -- 172.21.15.44:0/470954718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9e082e70 msgr2=0x55da9e08f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.519+0000 7fbd4f400700 1 --2- 172.21.15.44:0/470954718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9e082e70 0x55da9e08f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.519+0000 7fbd4f400700 1 -- 172.21.15.44:0/470954718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 msgr2=0x55da9e082330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.519+0000 7fbd4f400700 1 --2- 172.21.15.44:0/470954718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.519+0000 7fbd4f400700 1 -- 172.21.15.44:0/470954718 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da9df69420 con 0x55da9df03e40 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.519+0000 7fbd46a00700 1 --2- 172.21.15.44:0/470954718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9e082e70 0x55da9e08f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.519+0000 7fbd4fe00700 1 --2- 172.21.15.44:0/470954718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.519+0000 7fbd4f400700 1 --2- 172.21.15.44:0/470954718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 0x55da9e082870 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009d60 tx=0x7fbd40011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e628bcd76085bc9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.519+0000 7fbd4fe00700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd340b2ba0 0x7fbd340b50b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55da9e084ff0 tx=0x7fbd48009700 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:27.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.520+0000 7fbd44c00700 1 -- 172.21.15.44:0/470954718 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4001e070 con 0x55da9df03e40 2024-05-27T09:36:27.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.520+0000 7fbd44c00700 1 -- 172.21.15.44:0/470954718 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd40019ae0 con 0x55da9df03e40 2024-05-27T09:36:27.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.520+0000 7fbd44c00700 1 -- 172.21.15.44:0/470954718 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4000e9b0 con 0x55da9df03e40 2024-05-27T09:36:27.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.521+0000 7fbd65144b80 1 -- 172.21.15.44:0/470954718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 msgr2=0x55da9e082870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:27.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.521+0000 7fbd65144b80 1 --2- 172.21.15.44:0/470954718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 0x55da9e082870 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009d60 tx=0x7fbd40011720 comp rx=0 tx=0).stop 2024-05-27T09:36:27.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.521+0000 7fbd65144b80 1 -- 172.21.15.44:0/470954718 shutdown_connections 2024-05-27T09:36:27.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.521+0000 7fbd65144b80 1 --2- 172.21.15.44:0/470954718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9e082e70 0x55da9e08f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.521+0000 7fbd65144b80 1 --2- 172.21.15.44:0/470954718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.521+0000 7fbd65144b80 1 --2- 172.21.15.44:0/470954718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 0x55da9e082870 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.521+0000 7fbd65144b80 1 -- 172.21.15.44:0/470954718 >> 172.21.15.44:0/470954718 conn(0x55da9df9d390 msgr2=0x55da9e07e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:27.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.521+0000 7fbd65144b80 1 -- 172.21.15.44:0/470954718 shutdown_connections 2024-05-27T09:36:27.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.521+0000 7fbd65144b80 1 -- 172.21.15.44:0/470954718 wait complete. 2024-05-27T09:36:27.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.522+0000 7fbd65144b80 1 -- start start 2024-05-27T09:36:27.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.522+0000 7fbd65144b80 1 Processor -- start 2024-05-27T09:36:27.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.522+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9df03e40 0x55da9e08c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.522+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9dfef9f0 0x55da9e08cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9e082e70 0x55da9e08d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd65144b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55da9dea6fa0 con 0x55da9e082e70 2024-05-27T09:36:27.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd65144b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55da9df41760 con 0x55da9df03e40 2024-05-27T09:36:27.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd65144b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55da9e08fb80 con 0x55da9dfef9f0 2024-05-27T09:36:27.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd65144b80 10 client.0 _ll_register_callbacks cb 0x55da9df4ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:27.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd4f400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9df03e40 0x55da9e08c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd4f400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9df03e40 0x55da9e08c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60690/0 (socket says 172.21.15.44:60690) 2024-05-27T09:36:27.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd4f400700 1 -- 172.21.15.44:0/1456113522 learned_addr learned my addr 172.21.15.44:0/1456113522 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd46a00700 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9dfef9f0 0x55da9e08cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd4fe00700 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9e082e70 0x55da9e08d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd4f400700 1 -- 172.21.15.44:0/1456113522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9dfef9f0 msgr2=0x55da9e08cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd4f400700 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9dfef9f0 0x55da9e08cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd4f400700 1 -- 172.21.15.44:0/1456113522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9e082e70 msgr2=0x55da9e08d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd4f400700 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9e082e70 0x55da9e08d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd4f400700 1 -- 172.21.15.44:0/1456113522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da9def08d0 con 0x55da9df03e40 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.523+0000 7fbd46a00700 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9dfef9f0 0x55da9e08cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.524+0000 7fbd4f400700 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9df03e40 0x55da9e08c660 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009960 tx=0x7fbd40019660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e38d6475214aa45e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.524+0000 7fbd3d600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.524+0000 7fbd3d600700 1 -- 172.21.15.44:0/1456113522 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4001e050 con 0x55da9df03e40 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.524+0000 7fbd3d600700 1 -- 172.21.15.44:0/1456113522 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbd40004230 con 0x55da9df03e40 2024-05-27T09:36:27.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.524+0000 7fbd65144b80 1 -- 172.21.15.44:0/1456113522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd40019ae0 con 0x55da9df03e40 2024-05-27T09:36:27.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.525+0000 7fbd3d600700 1 -- 172.21.15.44:0/1456113522 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbd40004430 con 0x55da9df03e40 2024-05-27T09:36:27.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.525+0000 7fbd65144b80 10 client.25475 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:27.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.525+0000 7fbd65144b80 1 -- 172.21.15.44:0/1456113522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55da9df71060 con 0x55da9df03e40 2024-05-27T09:36:27.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.525+0000 7fbd3d600700 1 -- 172.21.15.44:0/1456113522 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd40019ae0 con 0x55da9df03e40 2024-05-27T09:36:27.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.525+0000 7fbd3d600700 1 -- 172.21.15.44:0/1456113522 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbd40004af0 con 0x55da9df03e40 2024-05-27T09:36:27.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.525+0000 7fbd65144b80 10 client.25475 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:27.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.525+0000 7fbd65144b80 10 client.25475 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:36:27.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.525+0000 7fbd65144b80 20 client.25475 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:27.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.525+0000 7fbd65144b80 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55da9e09c980 0x55da9e09eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.525+0000 7fbd65144b80 4 client.25475 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:27.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.525+0000 7fbd65144b80 1 -- 172.21.15.44:0/1456113522 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da9dfb2d90 con 0x55da9e09c980 2024-05-27T09:36:27.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.526+0000 7fbd46a00700 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55da9e09c980 0x55da9e09eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.526+0000 7fbd46a00700 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55da9e09c980 0x55da9e09eda0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d944a4c36e2b1a5d server_cookie=f9652a6edbfaaea in_seq=0 out_seq=0 2024-05-27T09:36:27.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.526+0000 7fbd3d600700 10 client.25475 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:36:27.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.527+0000 7fbd3d600700 1 -- 172.21.15.44:0/1456113522 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55da9dfb2d90 con 0x55da9e09c980 2024-05-27T09:36:27.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.527+0000 7fbd3d600700 10 client.25475 handle_command_reply: tid=0 2024-05-27T09:36:27.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.527+0000 7fbd65144b80 1 client.25475 shutdown 2024-05-27T09:36:27.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.527+0000 7fbd65144b80 1 -- 172.21.15.44:0/1456113522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9df03e40 msgr2=0x55da9e08c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:27.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.527+0000 7fbd65144b80 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9df03e40 0x55da9e08c660 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009960 tx=0x7fbd40019660 comp rx=0 tx=0).stop 2024-05-27T09:36:27.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.528+0000 7fbd65144b80 1 -- 172.21.15.44:0/1456113522 shutdown_connections 2024-05-27T09:36:27.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.528+0000 7fbd65144b80 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55da9e09c980 0x55da9e09eda0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.528+0000 7fbd65144b80 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9e082e70 0x55da9e08d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.528+0000 7fbd65144b80 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9df03e40 0x55da9e08c660 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.528+0000 7fbd65144b80 1 --2- 172.21.15.44:0/1456113522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9dfef9f0 0x55da9e08cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.528+0000 7fbd3d600700 0 client.25475 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:36:27.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.528+0000 7fbd65144b80 1 -- 172.21.15.44:0/1456113522 >> 172.21.15.44:0/1456113522 conn(0x55da9df9d390 msgr2=0x55da9e082370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:27.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.528+0000 7fbd65144b80 1 -- 172.21.15.44:0/1456113522 shutdown_connections 2024-05-27T09:36:27.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.528+0000 7fbd65144b80 1 -- 172.21.15.44:0/1456113522 wait complete. 2024-05-27T09:36:27.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.528+0000 7fbd65144b80 20 client.25475 trim_cache size 0 max 16384 2024-05-27T09:36:27.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.545+0000 7fbd65144b80 1 Processor -- start 2024-05-27T09:36:27.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.545+0000 7fbd65144b80 1 -- start start 2024-05-27T09:36:27.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.545+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 0x55da9e087e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.545+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.545+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9e082e70 0x55da9e08e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.545+0000 7fbd65144b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd501b3970 con 0x55da9df03e40 2024-05-27T09:36:27.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.545+0000 7fbd65144b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbd501b1200 con 0x55da9dfef9f0 2024-05-27T09:36:27.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.545+0000 7fbd65144b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbd501af070 con 0x55da9e082e70 2024-05-27T09:36:27.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.546+0000 7fbd46a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.546+0000 7fbd4f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 0x55da9e087e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.546+0000 7fbd46a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60704/0 (socket says 172.21.15.44:60704) 2024-05-27T09:36:27.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.546+0000 7fbd46a00700 1 -- 172.21.15.44:0/3466047795 learned_addr learned my addr 172.21.15.44:0/3466047795 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:27.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.546+0000 7fbd4fe00700 1 --2- 172.21.15.44:0/3466047795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9e082e70 0x55da9e08e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.546+0000 7fbd46a00700 1 -- 172.21.15.44:0/3466047795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9e082e70 msgr2=0x55da9e08e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:27.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.546+0000 7fbd46a00700 1 --2- 172.21.15.44:0/3466047795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9e082e70 0x55da9e08e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.546+0000 7fbd46a00700 1 -- 172.21.15.44:0/3466047795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 msgr2=0x55da9e087e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:27.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.546+0000 7fbd46a00700 1 --2- 172.21.15.44:0/3466047795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 0x55da9e087e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.546+0000 7fbd46a00700 1 -- 172.21.15.44:0/3466047795 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da9e07a240 con 0x55da9dfef9f0 2024-05-27T09:36:27.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.546+0000 7fbd4f400700 1 --2- 172.21.15.44:0/3466047795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 0x55da9e087e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:27.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.547+0000 7fbd46a00700 1 --2- 172.21.15.44:0/3466047795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbd380a5af0 tx=0x7fbd38005000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91bfefe6d940eed2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:27.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.547+0000 7fbd3fe00700 1 -- 172.21.15.44:0/3466047795 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3800cab0 con 0x55da9dfef9f0 2024-05-27T09:36:27.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.547+0000 7fbd3fe00700 1 -- 172.21.15.44:0/3466047795 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd38005720 con 0x55da9dfef9f0 2024-05-27T09:36:27.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.547+0000 7fbd3fe00700 1 -- 172.21.15.44:0/3466047795 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd380ade20 con 0x55da9dfef9f0 2024-05-27T09:36:27.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.548+0000 7fbd65144b80 1 -- 172.21.15.44:0/3466047795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 msgr2=0x55da9e082790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:27.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.548+0000 7fbd65144b80 1 --2- 172.21.15.44:0/3466047795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbd380a5af0 tx=0x7fbd38005000 comp rx=0 tx=0).stop 2024-05-27T09:36:27.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.548+0000 7fbd65144b80 1 -- 172.21.15.44:0/3466047795 shutdown_connections 2024-05-27T09:36:27.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.548+0000 7fbd65144b80 1 --2- 172.21.15.44:0/3466047795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9e082e70 0x55da9e08e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.548+0000 7fbd65144b80 1 --2- 172.21.15.44:0/3466047795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9df03e40 0x55da9e087e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.548+0000 7fbd65144b80 1 --2- 172.21.15.44:0/3466047795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e082790 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.548+0000 7fbd65144b80 1 -- 172.21.15.44:0/3466047795 >> 172.21.15.44:0/3466047795 conn(0x55da9df9d390 msgr2=0x55da9e083ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:27.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.548+0000 7fbd65144b80 1 -- 172.21.15.44:0/3466047795 shutdown_connections 2024-05-27T09:36:27.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.548+0000 7fbd65144b80 1 -- 172.21.15.44:0/3466047795 wait complete. 2024-05-27T09:36:27.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.549+0000 7fbd65144b80 1 -- start start 2024-05-27T09:36:27.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.549+0000 7fbd65144b80 1 Processor -- start 2024-05-27T09:36:27.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.549+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9df03e40 0x55da9e087db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.549+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e086190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.549+0000 7fbd65144b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9e082e70 0x55da9e0867e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.549+0000 7fbd65144b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd501af6c0 con 0x55da9e082e70 2024-05-27T09:36:27.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.549+0000 7fbd65144b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbd501b0b30 con 0x55da9dfef9f0 2024-05-27T09:36:27.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.549+0000 7fbd65144b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbd501afd90 con 0x55da9df03e40 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.549+0000 7fbd65144b80 10 client.0 _ll_register_callbacks cb 0x55da9df4ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.550+0000 7fbd4fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9e082e70 0x55da9e0867e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.550+0000 7fbd4f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9df03e40 0x55da9e087db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.550+0000 7fbd4fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9e082e70 0x55da9e0867e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58468/0 (socket says 172.21.15.44:58468) 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.550+0000 7fbd4fe00700 1 -- 172.21.15.44:0/891541071 learned_addr learned my addr 172.21.15.44:0/891541071 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.550+0000 7fbd46a00700 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e086190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.550+0000 7fbd46a00700 1 -- 172.21.15.44:0/891541071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9df03e40 msgr2=0x55da9e087db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.550+0000 7fbd46a00700 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9df03e40 0x55da9e087db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.550+0000 7fbd46a00700 1 -- 172.21.15.44:0/891541071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9e082e70 msgr2=0x55da9e0867e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.550+0000 7fbd46a00700 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9e082e70 0x55da9e0867e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.550+0000 7fbd46a00700 1 -- 172.21.15.44:0/891541071 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da9e08d450 con 0x55da9dfef9f0 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.550+0000 7fbd4f400700 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9df03e40 0x55da9e087db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:27.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.551+0000 7fbd46a00700 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e086190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbd380a9770 tx=0x7fbd38004e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2958b1a056c3e12 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.551+0000 7fbd3d600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.551+0000 7fbd3d600700 1 -- 172.21.15.44:0/891541071 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd380b2210 con 0x55da9dfef9f0 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.551+0000 7fbd3d600700 1 -- 172.21.15.44:0/891541071 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fbd380a9cd0 con 0x55da9dfef9f0 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.551+0000 7fbd65144b80 1 -- 172.21.15.44:0/891541071 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55da9dfac570 con 0x55da9dfef9f0 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.551+0000 7fbd3d600700 1 -- 172.21.15.44:0/891541071 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbd380a9580 con 0x55da9dfef9f0 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.551+0000 7fbd65144b80 10 client.25481 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.551+0000 7fbd65144b80 1 -- 172.21.15.44:0/891541071 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55da9e094c80 con 0x55da9dfef9f0 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.551+0000 7fbd3d600700 1 -- 172.21.15.44:0/891541071 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55da9dfac570 con 0x55da9dfef9f0 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.551+0000 7fbd3d600700 1 -- 172.21.15.44:0/891541071 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbd380b2440 con 0x55da9dfef9f0 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.552+0000 7fbd65144b80 10 client.25481 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.552+0000 7fbd65144b80 10 client.25481 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:36:27.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.552+0000 7fbd65144b80 20 client.25481 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:27.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.552+0000 7fbd65144b80 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55da9e095300 0x55da9e111b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:27.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.552+0000 7fbd65144b80 4 client.25481 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:36:27.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.552+0000 7fbd65144b80 1 -- 172.21.15.44:0/891541071 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da9de23250 con 0x55da9e095300 2024-05-27T09:36:27.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.552+0000 7fbd4f400700 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55da9e095300 0x55da9e111b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:27.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.552+0000 7fbd4f400700 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55da9e095300 0x55da9e111b30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a54e9b7f824de81c server_cookie=677835ec6c594e3e in_seq=0 out_seq=0 2024-05-27T09:36:27.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.552+0000 7fbd3d600700 10 client.25481 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:36:27.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd3d600700 1 -- 172.21.15.44:0/891541071 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da9de23250 con 0x55da9e095300 2024-05-27T09:36:27.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd3d600700 10 client.25481 handle_command_reply: tid=0 2024-05-27T09:36:27.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 1 client.25481 shutdown 2024-05-27T09:36:27.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 1 -- 172.21.15.44:0/891541071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 msgr2=0x55da9e086190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:27.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e086190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbd380a9770 tx=0x7fbd38004e00 comp rx=0 tx=0).stop 2024-05-27T09:36:27.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 1 -- 172.21.15.44:0/891541071 shutdown_connections 2024-05-27T09:36:27.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55da9e095300 0x55da9e111b30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55da9e082e70 0x55da9e0867e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da9df03e40 0x55da9e087db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 1 --2- 172.21.15.44:0/891541071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55da9dfef9f0 0x55da9e086190 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd3d600700 0 client.25481 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:36:27.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 1 -- 172.21.15.44:0/891541071 >> 172.21.15.44:0/891541071 conn(0x55da9df9d390 msgr2=0x55da9e08cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:27.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 1 -- 172.21.15.44:0/891541071 shutdown_connections 2024-05-27T09:36:27.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 1 -- 172.21.15.44:0/891541071 wait complete. 2024-05-27T09:36:27.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.553+0000 7fbd65144b80 20 client.25481 trim_cache size 0 max 16384 2024-05-27T09:36:27.559 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:36:27.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.554+0000 7fbd56600700 1 -- 172.21.15.44:0/3563656673 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd340b2ba0 msgr2=0x7fbd340b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:27.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.554+0000 7fbd56600700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd340b2ba0 0x7fbd340b50b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55da9e084ff0 tx=0x7fbd48009700 comp rx=0 tx=0).stop 2024-05-27T09:36:27.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.554+0000 7fbd56600700 1 -- 172.21.15.44:0/3563656673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd50108ef0 msgr2=0x7fbd50077100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:27.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.554+0000 7fbd56600700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd50108ef0 0x7fbd50077100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800e970 tx=0x7fbd3800ec80 comp rx=0 tx=0).stop 2024-05-27T09:36:27.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.554+0000 7fbd56600700 1 -- 172.21.15.44:0/3563656673 shutdown_connections 2024-05-27T09:36:27.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.554+0000 7fbd56600700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd340b2ba0 0x7fbd340b50b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.554+0000 7fbd56600700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd50109840 0x7fbd50077640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.554+0000 7fbd56600700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd501045c0 0x7fbd50076bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.554+0000 7fbd56600700 1 --2- 172.21.15.44:0/3563656673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd50108ef0 0x7fbd50077100 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:27.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.554+0000 7fbd56600700 1 -- 172.21.15.44:0/3563656673 >> 172.21.15.44:0/3563656673 conn(0x7fbd500ffe90 msgr2=0x7fbd50108660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:27.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.555+0000 7fbd56600700 1 -- 172.21.15.44:0/3563656673 shutdown_connections 2024-05-27T09:36:27.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:27.555+0000 7fbd56600700 1 -- 172.21.15.44:0/3563656673 wait complete. 2024-05-27T09:36:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.401+0000 7f1c4ce00640 1 -- 172.21.15.44:0/2790052358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c48109ab0 msgr2=0x7f1c481036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.401+0000 7f1c4ce00640 1 --2- 172.21.15.44:0/2790052358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c48109ab0 0x7f1c481036a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009be0 tx=0x7f1c3802a6c0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.401+0000 7f1c4ce00640 1 -- 172.21.15.44:0/2790052358 shutdown_connections 2024-05-27T09:36:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.401+0000 7f1c4ce00640 1 --2- 172.21.15.44:0/2790052358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c48104250 0x7f1c481105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.401+0000 7f1c4ce00640 1 --2- 172.21.15.44:0/2790052358 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c48109f50 0x7f1c48103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.401+0000 7f1c4ce00640 1 --2- 172.21.15.44:0/2790052358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c48109ab0 0x7f1c481036a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.401+0000 7f1c4ce00640 1 -- 172.21.15.44:0/2790052358 >> 172.21.15.44:0/2790052358 conn(0x7f1c48079240 msgr2=0x7f1c48079680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.401+0000 7f1c4ce00640 1 -- 172.21.15.44:0/2790052358 shutdown_connections 2024-05-27T09:36:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.401+0000 7f1c4ce00640 1 -- 172.21.15.44:0/2790052358 wait complete. 2024-05-27T09:36:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.403+0000 7f1c4ce00640 1 Processor -- start 2024-05-27T09:36:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.403+0000 7f1c4ce00640 1 -- start start 2024-05-27T09:36:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.403+0000 7f1c4ce00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c48104250 0x7f1c4819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.403+0000 7f1c4ce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c48109ab0 0x7f1c4819acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.403+0000 7f1c4ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c48109f50 0x7f1c4819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c4ce00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1c48113ca0 con 0x7f1c48109ab0 2024-05-27T09:36:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c4ce00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1c48113b20 con 0x7f1c48104250 2024-05-27T09:36:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c4ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1c48113e20 con 0x7f1c48109f50 2024-05-27T09:36:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c47400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c48104250 0x7f1c4819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c46a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c48109ab0 0x7f1c4819acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c47400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c48104250 0x7f1c4819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60734/0 (socket says 172.21.15.44:60734) 2024-05-27T09:36:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c47400640 1 -- 172.21.15.44:0/803095028 learned_addr learned my addr 172.21.15.44:0/803095028 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c47e00640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c48109f50 0x7f1c4819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c47400640 1 -- 172.21.15.44:0/803095028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c48109f50 msgr2=0x7f1c4819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c47400640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c48109f50 0x7f1c4819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c47400640 1 -- 172.21.15.44:0/803095028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c48109ab0 msgr2=0x7f1c4819acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c47400640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c48109ab0 0x7f1c4819acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.404+0000 7f1c47400640 1 -- 172.21.15.44:0/803095028 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4819f5c0 con 0x7f1c48104250 2024-05-27T09:36:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.405+0000 7f1c47400640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c48104250 0x7f1c4819a760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1c38000f80 tx=0x7f1c38003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.405+0000 7f1c3fe00640 1 -- 172.21.15.44:0/803095028 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c38041020 con 0x7f1c48104250 2024-05-27T09:36:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.405+0000 7f1c3fe00640 1 -- 172.21.15.44:0/803095028 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c380318f0 con 0x7f1c48104250 2024-05-27T09:36:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.405+0000 7f1c3fe00640 1 -- 172.21.15.44:0/803095028 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c38036040 con 0x7f1c48104250 2024-05-27T09:36:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.405+0000 7f1c4ce00640 1 -- 172.21.15.44:0/803095028 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c4819f850 con 0x7f1c48104250 2024-05-27T09:36:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.405+0000 7f1c4ce00640 1 -- 172.21.15.44:0/803095028 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c4819fd90 con 0x7f1c48104250 2024-05-27T09:36:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.408+0000 7f1c553bb740 1 Processor -- start 2024-05-27T09:36:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.409+0000 7f1c553bb740 1 -- start start 2024-05-27T09:36:28.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.409+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fc7a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.409+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8af2f0 0x55612fc7ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.409+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc7b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.409+0000 7f1c553bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55612fbebb90 con 0x55612f8af2f0 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.409+0000 7f1c553bb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55612fbdb730 con 0x55612f8d81b0 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.409+0000 7f1c553bb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55612fbdf770 con 0x55612f8b74d0 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.409+0000 7f1c47e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fc7a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.409+0000 7f1c47e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fc7a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60750/0 (socket says 172.21.15.44:60750) 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.409+0000 7f1c47e00640 1 -- 172.21.15.44:0/2414201040 learned_addr learned my addr 172.21.15.44:0/2414201040 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.410+0000 7f1c46a00640 1 --2- 172.21.15.44:0/2414201040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc7b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.410+0000 7f1c47e00640 1 -- 172.21.15.44:0/2414201040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 msgr2=0x55612fc7b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.410+0000 7f1c47e00640 1 --2- 172.21.15.44:0/2414201040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc7b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.410+0000 7f1c47e00640 1 -- 172.21.15.44:0/2414201040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8af2f0 msgr2=0x55612fc7ae10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:36:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.410+0000 7f1c47e00640 1 --2- 172.21.15.44:0/2414201040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8af2f0 0x55612fc7ae10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.410+0000 7f1c47e00640 1 -- 172.21.15.44:0/2414201040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55612fb748b0 con 0x55612f8d81b0 2024-05-27T09:36:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.410+0000 7f1c3fe00640 1 -- 172.21.15.44:0/803095028 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f1c38044020 con 0x7f1c48104250 2024-05-27T09:36:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.410+0000 7f1c47e00640 1 --2- 172.21.15.44:0/2414201040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fc7a8d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009c60 tx=0x7f1c40011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f855b281a321f072 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.410+0000 7f1c3cc00640 1 -- 172.21.15.44:0/2414201040 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4001e070 con 0x55612f8d81b0 2024-05-27T09:36:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.411+0000 7f1c3cc00640 1 -- 172.21.15.44:0/2414201040 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c40019ae0 con 0x55612f8d81b0 2024-05-27T09:36:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.411+0000 7f1c3cc00640 1 -- 172.21.15.44:0/2414201040 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4000e940 con 0x55612f8d81b0 2024-05-27T09:36:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.411+0000 7f1c553bb740 1 -- 172.21.15.44:0/2414201040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 msgr2=0x55612fc7a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.411+0000 7f1c553bb740 1 --2- 172.21.15.44:0/2414201040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fc7a8d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009c60 tx=0x7f1c40011810 comp rx=0 tx=0).stop 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.412+0000 7f1c553bb740 1 -- 172.21.15.44:0/2414201040 shutdown_connections 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.412+0000 7f1c553bb740 1 --2- 172.21.15.44:0/2414201040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc7b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.412+0000 7f1c553bb740 1 --2- 172.21.15.44:0/2414201040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8af2f0 0x55612fc7ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.412+0000 7f1c553bb740 1 --2- 172.21.15.44:0/2414201040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fc7a8d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.412+0000 7f1c553bb740 1 -- 172.21.15.44:0/2414201040 >> 172.21.15.44:0/2414201040 conn(0x55612f8cc300 msgr2=0x55612fc77210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.412+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1c2c0b2b80 0x7f1c2c0b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.412+0000 7f1c553bb740 1 -- 172.21.15.44:0/2414201040 shutdown_connections 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.412+0000 7f1c553bb740 1 -- 172.21.15.44:0/2414201040 wait complete. 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.412+0000 7f1c3fe00640 1 -- 172.21.15.44:0/803095028 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1c381033b0 con 0x7f1c48104250 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.412+0000 7f1c46a00640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1c2c0b2b80 0x7f1c2c0b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.412+0000 7f1c553bb740 1 -- start start 2024-05-27T09:36:28.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.413+0000 7f1c553bb740 1 Processor -- start 2024-05-27T09:36:28.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.413+0000 7f1c46a00640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1c2c0b2b80 0x7f1c2c0b5090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1c48101780 tx=0x7f1c30005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:28.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.413+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8af2f0 0x55612fc89010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.413+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc89550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.413+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8d81b0 0x55612fc95300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.413+0000 7f1c553bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55612f8c1790 con 0x55612f8d81b0 2024-05-27T09:36:28.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.413+0000 7f1c553bb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55612fb83bc0 con 0x55612f8af2f0 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.413+0000 7f1c553bb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55612fc8bfa0 con 0x55612f8b74d0 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.413+0000 7f1c553bb740 10 client.0 _ll_register_callbacks cb 0x55612fb61400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c47400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8af2f0 0x55612fc89010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c47400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8af2f0 0x55612fc89010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60754/0 (socket says 172.21.15.44:60754) 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c47400640 1 -- 172.21.15.44:0/3019294010 learned_addr learned my addr 172.21.15.44:0/3019294010 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c47e00640 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8d81b0 0x55612fc95300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c46a00640 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc89550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c47400640 1 -- 172.21.15.44:0/3019294010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 msgr2=0x55612fc89550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c47400640 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc89550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c47400640 1 -- 172.21.15.44:0/3019294010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8d81b0 msgr2=0x55612fc95300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c47400640 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8d81b0 0x55612fc95300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c47400640 1 -- 172.21.15.44:0/3019294010 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55612f8f0210 con 0x55612f8af2f0 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c46a00640 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc89550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c47e00640 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8d81b0 0x55612fc95300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c47400640 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8af2f0 0x55612fc89010 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1c381009e0 tx=0x7f1c380caaa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bae1c6ef8e25950 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c35600640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.414+0000 7f1c35600640 1 -- 172.21.15.44:0/3019294010 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c380c9f90 con 0x55612f8af2f0 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c35600640 1 -- 172.21.15.44:0/3019294010 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1c380d1dc0 con 0x55612f8af2f0 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c553bb740 1 -- 172.21.15.44:0/3019294010 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c40019ae0 con 0x55612f8af2f0 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c35600640 1 -- 172.21.15.44:0/3019294010 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1c380cab80 con 0x55612f8af2f0 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c553bb740 10 client.25493 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c553bb740 1 -- 172.21.15.44:0/3019294010 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55612f9bc620 con 0x55612f8af2f0 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c35600640 1 -- 172.21.15.44:0/3019294010 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c40019ae0 con 0x55612f8af2f0 2024-05-27T09:36:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c35600640 1 -- 172.21.15.44:0/3019294010 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1c38004b40 con 0x55612f8af2f0 2024-05-27T09:36:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c553bb740 10 client.25493 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c553bb740 10 client.25493 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c553bb740 20 client.25493 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55612f896d80 0x55612fc9b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c553bb740 4 client.25493 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.415+0000 7f1c553bb740 1 -- 172.21.15.44:0/3019294010 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55612fbedae0 con 0x55612f896d80 2024-05-27T09:36:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.416+0000 7f1c46a00640 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55612f896d80 0x55612fc9b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.416+0000 7f1c46a00640 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55612f896d80 0x55612fc9b230 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3a45c4fe8c3191e server_cookie=98b0d7e5a19b82a7 in_seq=0 out_seq=0 2024-05-27T09:36:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.416+0000 7f1c35600640 10 client.25493 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.417+0000 7f1c35600640 1 -- 172.21.15.44:0/3019294010 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55612fbedae0 con 0x55612f896d80 2024-05-27T09:36:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.417+0000 7f1c35600640 10 client.25493 handle_command_reply: tid=0 2024-05-27T09:36:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.417+0000 7f1c553bb740 1 client.25493 shutdown 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c553bb740 1 -- 172.21.15.44:0/3019294010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8af2f0 msgr2=0x55612fc89010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8af2f0 0x55612fc89010 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1c381009e0 tx=0x7f1c380caaa0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c553bb740 1 -- 172.21.15.44:0/3019294010 shutdown_connections 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55612f896d80 0x55612fc9b230 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8d81b0 0x55612fc95300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc89550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3019294010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8af2f0 0x55612fc89010 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c553bb740 1 -- 172.21.15.44:0/3019294010 >> 172.21.15.44:0/3019294010 conn(0x55612f8cc300 msgr2=0x55612fc7c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c35600640 0 client.25493 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c553bb740 1 -- 172.21.15.44:0/3019294010 shutdown_connections 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c553bb740 1 -- 172.21.15.44:0/3019294010 wait complete. 2024-05-27T09:36:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.418+0000 7f1c553bb740 20 client.25493 trim_cache size 0 max 16384 2024-05-27T09:36:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.435+0000 7f1c553bb740 1 Processor -- start 2024-05-27T09:36:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.435+0000 7f1c553bb740 1 -- start start 2024-05-27T09:36:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.435+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8af2f0 0x55612fc8b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.435+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc84830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fc82ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c553bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1c481b0010 con 0x55612f8af2f0 2024-05-27T09:36:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c553bb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1c481ad8e0 con 0x55612f8d81b0 2024-05-27T09:36:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c553bb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1c481ab750 con 0x55612f8b74d0 2024-05-27T09:36:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c47400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8af2f0 0x55612fc8b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c46a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc84830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c47400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8af2f0 0x55612fc8b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58496/0 (socket says 172.21.15.44:58496) 2024-05-27T09:36:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c47400640 1 -- 172.21.15.44:0/3637734096 learned_addr learned my addr 172.21.15.44:0/3637734096 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c46a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc84830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43058/0 (socket says 172.21.15.44:43058) 2024-05-27T09:36:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c47e00640 1 --2- 172.21.15.44:0/3637734096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fc82ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c46a00640 1 -- 172.21.15.44:0/3637734096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 msgr2=0x55612fc82ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.436+0000 7f1c46a00640 1 --2- 172.21.15.44:0/3637734096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fc82ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.437+0000 7f1c46a00640 1 -- 172.21.15.44:0/3637734096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8af2f0 msgr2=0x55612fc8b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.437+0000 7f1c46a00640 1 --2- 172.21.15.44:0/3637734096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8af2f0 0x55612fc8b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.437+0000 7f1c46a00640 1 -- 172.21.15.44:0/3637734096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55612fc87560 con 0x55612f8b74d0 2024-05-27T09:36:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.437+0000 7f1c47400640 1 --2- 172.21.15.44:0/3637734096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8af2f0 0x55612fc8b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.437+0000 7f1c46a00640 1 --2- 172.21.15.44:0/3637734096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc84830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55612fc9b970 tx=0x7f1c3001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76dea9634d0ca370 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.437+0000 7f1c37e00640 1 -- 172.21.15.44:0/3637734096 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3001de00 con 0x55612f8b74d0 2024-05-27T09:36:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.437+0000 7f1c37e00640 1 -- 172.21.15.44:0/3637734096 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c300035f0 con 0x55612f8b74d0 2024-05-27T09:36:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.437+0000 7f1c37e00640 1 -- 172.21.15.44:0/3637734096 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c300044e0 con 0x55612f8b74d0 2024-05-27T09:36:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.438+0000 7f1c553bb740 1 -- 172.21.15.44:0/3637734096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 msgr2=0x55612fc84830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.438+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3637734096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc84830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55612fc9b970 tx=0x7f1c3001b000 comp rx=0 tx=0).stop 2024-05-27T09:36:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.438+0000 7f1c553bb740 1 -- 172.21.15.44:0/3637734096 shutdown_connections 2024-05-27T09:36:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.438+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3637734096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fc82ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.438+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3637734096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8b74d0 0x55612fc84830 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.438+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3637734096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8af2f0 0x55612fc8b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.438+0000 7f1c553bb740 1 -- 172.21.15.44:0/3637734096 >> 172.21.15.44:0/3637734096 conn(0x55612f8cc300 msgr2=0x55612fc7b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.438+0000 7f1c553bb740 1 -- 172.21.15.44:0/3637734096 shutdown_connections 2024-05-27T09:36:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.438+0000 7f1c553bb740 1 -- 172.21.15.44:0/3637734096 wait complete. 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.439+0000 7f1c553bb740 1 -- start start 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.439+0000 7f1c553bb740 1 Processor -- start 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.439+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8af2f0 0x55612fca0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.439+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8b74d0 0x55612fca0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.439+0000 7f1c553bb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fca11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.439+0000 7f1c553bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1c481abda0 con 0x55612f8b74d0 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.439+0000 7f1c553bb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1c481ad210 con 0x55612f8d81b0 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.439+0000 7f1c553bb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1c481ac470 con 0x55612f8af2f0 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.439+0000 7f1c553bb740 10 client.0 _ll_register_callbacks cb 0x7f1c380093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c46a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8b74d0 0x55612fca0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c47e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fca11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c47e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fca11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60772/0 (socket says 172.21.15.44:60772) 2024-05-27T09:36:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c47e00640 1 -- 172.21.15.44:0/3008886515 learned_addr learned my addr 172.21.15.44:0/3008886515 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c47400640 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8af2f0 0x55612fca0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c47e00640 1 -- 172.21.15.44:0/3008886515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8af2f0 msgr2=0x55612fca0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c47e00640 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8af2f0 0x55612fca0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c47e00640 1 -- 172.21.15.44:0/3008886515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8b74d0 msgr2=0x55612fca0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c47e00640 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8b74d0 0x55612fca0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c47e00640 1 -- 172.21.15.44:0/3008886515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55612fc8fa20 con 0x55612f8d81b0 2024-05-27T09:36:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c46a00640 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8b74d0 0x55612fca0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.440+0000 7f1c47400640 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8af2f0 0x55612fca0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.441+0000 7f1c47e00640 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fca11b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009c60 tx=0x7f1c400051a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e847c4c1de77492 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.441+0000 7f1c35600640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.441+0000 7f1c35600640 1 -- 172.21.15.44:0/3008886515 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4002a070 con 0x55612f8d81b0 2024-05-27T09:36:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.441+0000 7f1c35600640 1 -- 172.21.15.44:0/3008886515 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1c40019430 con 0x55612f8d81b0 2024-05-27T09:36:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.441+0000 7f1c553bb740 1 -- 172.21.15.44:0/3008886515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c300035f0 con 0x55612f8d81b0 2024-05-27T09:36:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.442+0000 7f1c35600640 1 -- 172.21.15.44:0/3008886515 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1c4001e040 con 0x55612f8d81b0 2024-05-27T09:36:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.442+0000 7f1c553bb740 10 client.25499 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.442+0000 7f1c553bb740 1 -- 172.21.15.44:0/3008886515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55612fd03c90 con 0x55612f8d81b0 2024-05-27T09:36:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.442+0000 7f1c35600640 1 -- 172.21.15.44:0/3008886515 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c300035f0 con 0x55612f8d81b0 2024-05-27T09:36:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.442+0000 7f1c35600640 1 -- 172.21.15.44:0/3008886515 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1c40004c20 con 0x55612f8d81b0 2024-05-27T09:36:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.442+0000 7f1c553bb740 10 client.25499 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.442+0000 7f1c553bb740 10 client.25499 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.443+0000 7f1c553bb740 20 client.25499 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.443+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55612f86c1c0 0x55612fd06290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.443+0000 7f1c553bb740 4 client.25499 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:36:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.443+0000 7f1c553bb740 1 -- 172.21.15.44:0/3008886515 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55612fbe72c0 con 0x55612f86c1c0 2024-05-27T09:36:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.443+0000 7f1c47400640 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55612f86c1c0 0x55612fd06290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.443+0000 7f1c47400640 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55612f86c1c0 0x55612fd06290 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e75f4274a0a8378 server_cookie=df30505b3d198260 in_seq=0 out_seq=0 2024-05-27T09:36:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.443+0000 7f1c35600640 10 client.25499 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c35600640 1 -- 172.21.15.44:0/3008886515 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55612fbe72c0 con 0x55612f86c1c0 2024-05-27T09:36:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c35600640 10 client.25499 handle_command_reply: tid=0 2024-05-27T09:36:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c553bb740 1 client.25499 shutdown 2024-05-27T09:36:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c553bb740 1 -- 172.21.15.44:0/3008886515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 msgr2=0x55612fca11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fca11b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009c60 tx=0x7f1c400051a0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c553bb740 1 -- 172.21.15.44:0/3008886515 shutdown_connections 2024-05-27T09:36:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55612f86c1c0 0x55612fd06290 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55612f8d81b0 0x55612fca11b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55612f8b74d0 0x55612fca0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c553bb740 1 --2- 172.21.15.44:0/3008886515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55612f8af2f0 0x55612fca0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c35600640 0 client.25499 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.450+0000 7f1c553bb740 1 -- 172.21.15.44:0/3008886515 >> 172.21.15.44:0/3008886515 conn(0x55612f8cc300 msgr2=0x55612fc7b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.451+0000 7f1c553bb740 1 -- 172.21.15.44:0/3008886515 shutdown_connections 2024-05-27T09:36:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.451+0000 7f1c553bb740 1 -- 172.21.15.44:0/3008886515 wait complete. 2024-05-27T09:36:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.451+0000 7f1c553bb740 20 client.25499 trim_cache size 0 max 16384 2024-05-27T09:36:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:36:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:36:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:36:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 -- 172.21.15.44:0/803095028 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1c2c0b2b80 msgr2=0x7f1c2c0b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1c2c0b2b80 0x7f1c2c0b5090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1c48101780 tx=0x7f1c30005f90 comp rx=0 tx=0).stop 2024-05-27T09:36:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 -- 172.21.15.44:0/803095028 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c48104250 msgr2=0x7f1c4819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c48104250 0x7f1c4819a760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1c38000f80 tx=0x7f1c38003ba0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 -- 172.21.15.44:0/803095028 shutdown_connections 2024-05-27T09:36:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c48109f50 0x7f1c4819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1c2c0b2b80 0x7f1c2c0b5090 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c48109ab0 0x7f1c4819acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 --2- 172.21.15.44:0/803095028 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c48104250 0x7f1c4819a760 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 -- 172.21.15.44:0/803095028 >> 172.21.15.44:0/803095028 conn(0x7f1c48079240 msgr2=0x7f1c48101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 -- 172.21.15.44:0/803095028 shutdown_connections 2024-05-27T09:36:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:28.452+0000 7f1c4ce00640 1 -- 172.21.15.44:0/803095028 wait complete. 2024-05-27T09:36:28.456 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:28 smithi044 ceph-mon[31618]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-27T09:36:28.497 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:36:28.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:28 smithi050 ceph-mon[36371]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-27T09:36:28.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:28 smithi189 ceph-mon[36671]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-27T09:36:28.966 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:30.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:30 smithi044 ceph-mon[31618]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-27T09:36:30.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:30 smithi050 ceph-mon[36371]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-27T09:36:30.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:30 smithi189 ceph-mon[36671]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-27T09:36:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.154+0000 7f85b1800700 1 -- 172.21.15.44:0/1406215050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85ac106ec0 msgr2=0x7f85ac113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.154+0000 7f85b1800700 1 --2- 172.21.15.44:0/1406215050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85ac106ec0 0x7f85ac113350 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f85a4009f90 tx=0x7f85a400b330 comp rx=0 tx=0).stop 2024-05-27T09:36:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.154+0000 7f85b1800700 1 -- 172.21.15.44:0/1406215050 shutdown_connections 2024-05-27T09:36:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.154+0000 7f85b1800700 1 --2- 172.21.15.44:0/1406215050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85ac106ec0 0x7f85ac113350 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.154+0000 7f85b1800700 1 --2- 172.21.15.44:0/1406215050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85ac10c5b0 0x7f85ac10c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.154+0000 7f85b1800700 1 --2- 172.21.15.44:0/1406215050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ac106570 0x7f85ac106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.154+0000 7f85b1800700 1 -- 172.21.15.44:0/1406215050 >> 172.21.15.44:0/1406215050 conn(0x7f85ac102070 msgr2=0x7f85ac1044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.154+0000 7f85b1800700 1 -- 172.21.15.44:0/1406215050 shutdown_connections 2024-05-27T09:36:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.154+0000 7f85b1800700 1 -- 172.21.15.44:0/1406215050 wait complete. 2024-05-27T09:36:31.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.156+0000 7f85b1800700 1 Processor -- start 2024-05-27T09:36:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.156+0000 7f85b1800700 1 -- start start 2024-05-27T09:36:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.156+0000 7f85b1800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85ac106570 0x7f85ac1106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.156+0000 7f85b1800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ac106ec0 0x7f85ac110c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.156+0000 7f85b1800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85ac10c5b0 0x7f85ac111160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85b1800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1166e0 con 0x7f85ac106ec0 2024-05-27T09:36:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85b1800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac116560 con 0x7f85ac10c5b0 2024-05-27T09:36:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85b1800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac116860 con 0x7f85ac106570 2024-05-27T09:36:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85aa000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ac106ec0 0x7f85ac110c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85aaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85ac106570 0x7f85ac1106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85aaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85ac106570 0x7f85ac1106e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43092/0 (socket says 172.21.15.44:43092) 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85aaa00700 1 -- 172.21.15.44:0/2856971312 learned_addr learned my addr 172.21.15.44:0/2856971312 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85ab400700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85ac10c5b0 0x7f85ac111160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85aaa00700 1 -- 172.21.15.44:0/2856971312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85ac10c5b0 msgr2=0x7f85ac111160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85aaa00700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85ac10c5b0 0x7f85ac111160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85aaa00700 1 -- 172.21.15.44:0/2856971312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ac106ec0 msgr2=0x7f85ac110c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85aaa00700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ac106ec0 0x7f85ac110c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.157+0000 7f85aaa00700 1 -- 172.21.15.44:0/2856971312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ac1ad4d0 con 0x7f85ac106570 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.158+0000 7f85aaa00700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85ac106570 0x7f85ac1106e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f859c00e970 tx=0x7f859c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.158+0000 7f85a3400700 1 -- 172.21.15.44:0/2856971312 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c004a30 con 0x7f85ac106570 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.158+0000 7f85a3400700 1 -- 172.21.15.44:0/2856971312 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f859c00cc90 con 0x7f85ac106570 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.158+0000 7f85b1800700 1 -- 172.21.15.44:0/2856971312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ac1ad7c0 con 0x7f85ac106570 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.158+0000 7f85a3400700 1 -- 172.21.15.44:0/2856971312 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c005610 con 0x7f85ac106570 2024-05-27T09:36:31.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.158+0000 7f85b1800700 1 -- 172.21.15.44:0/2856971312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ac1add50 con 0x7f85ac106570 2024-05-27T09:36:31.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.162+0000 7f85a3400700 1 -- 172.21.15.44:0/2856971312 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f859c00c7e0 con 0x7f85ac106570 2024-05-27T09:36:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.163+0000 7f85c03cdb80 1 Processor -- start 2024-05-27T09:36:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.163+0000 7f85c03cdb80 1 -- start start 2024-05-27T09:36:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.164+0000 7f85a3400700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f85900b2c20 0x7f85900b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.164+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd5e49f0 0x55bcfd677330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.164+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd677870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.164+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 0x55bcfd684110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.164+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bcfd5a5e40 con 0x55bcfd4f8e40 2024-05-27T09:36:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.164+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bcfd5959e0 con 0x55bcfd677e70 2024-05-27T09:36:31.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.164+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bcfd599a20 con 0x55bcfd5e49f0 2024-05-27T09:36:31.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.164+0000 7f85a3400700 1 -- 172.21.15.44:0/2856971312 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f859c0e2980 con 0x7f85ac106570 2024-05-27T09:36:31.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.164+0000 7f85aa000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd677870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.164+0000 7f85ab400700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f85900b2c20 0x7f85900b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.165+0000 7f85aaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd5e49f0 0x55bcfd677330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.165+0000 7f85aaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd5e49f0 0x55bcfd677330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43108/0 (socket says 172.21.15.44:43108) 2024-05-27T09:36:31.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.165+0000 7f85aaa00700 1 -- 172.21.15.44:0/4088248476 learned_addr learned my addr 172.21.15.44:0/4088248476 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:31.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.165+0000 7f85ab400700 1 --2- 172.21.15.44:0/4088248476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 0x55bcfd684110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.165+0000 7f85aaa00700 1 -- 172.21.15.44:0/4088248476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 msgr2=0x55bcfd684110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.165+0000 7f85aaa00700 1 --2- 172.21.15.44:0/4088248476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 0x55bcfd684110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.165+0000 7f85aaa00700 1 -- 172.21.15.44:0/4088248476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 msgr2=0x55bcfd677870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.165+0000 7f85aaa00700 1 --2- 172.21.15.44:0/4088248476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd677870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.165+0000 7f85aaa00700 1 -- 172.21.15.44:0/4088248476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcfd55e420 con 0x55bcfd5e49f0 2024-05-27T09:36:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.165+0000 7f85aaa00700 1 --2- 172.21.15.44:0/4088248476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd5e49f0 0x55bcfd677330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f859c0a9750 tx=0x7f859c0a59e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f42736a4a45a079 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.166+0000 7f85ab400700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f85900b2c20 0x7f85900b5130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f85a4009550 tx=0x7f85a4009c20 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.166+0000 7f8597e00700 1 -- 172.21.15.44:0/4088248476 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c0a5040 con 0x55bcfd5e49f0 2024-05-27T09:36:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.166+0000 7f8597e00700 1 -- 172.21.15.44:0/4088248476 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f859c0b2050 con 0x55bcfd5e49f0 2024-05-27T09:36:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.166+0000 7f8597e00700 1 -- 172.21.15.44:0/4088248476 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c00cad0 con 0x55bcfd5e49f0 2024-05-27T09:36:31.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.167+0000 7f85c03cdb80 1 -- 172.21.15.44:0/4088248476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd5e49f0 msgr2=0x55bcfd677330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:31.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.167+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/4088248476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd5e49f0 0x55bcfd677330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f859c0a9750 tx=0x7f859c0a59e0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.167+0000 7f85c03cdb80 1 -- 172.21.15.44:0/4088248476 shutdown_connections 2024-05-27T09:36:31.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.167+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/4088248476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 0x55bcfd684110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.167+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/4088248476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd5e49f0 0x55bcfd677330 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.167+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/4088248476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd677870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.167+0000 7f85c03cdb80 1 -- 172.21.15.44:0/4088248476 >> 172.21.15.44:0/4088248476 conn(0x55bcfd592390 msgr2=0x55bcfd6734c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:31.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.167+0000 7f85c03cdb80 1 -- 172.21.15.44:0/4088248476 shutdown_connections 2024-05-27T09:36:31.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.167+0000 7f85c03cdb80 1 -- 172.21.15.44:0/4088248476 wait complete. 2024-05-27T09:36:31.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.167+0000 7f85c03cdb80 1 -- start start 2024-05-27T09:36:31.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.168+0000 7f85c03cdb80 1 Processor -- start 2024-05-27T09:36:31.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.168+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd681640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.168+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd5e49f0 0x55bcfd681b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.168+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 0x55bcfd6820c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.168+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bcfd49bfa0 con 0x55bcfd4f8e40 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.168+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bcfd536760 con 0x55bcfd5e49f0 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.168+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bcfd684b80 con 0x55bcfd677e70 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.168+0000 7f85c03cdb80 10 client.0 _ll_register_callbacks cb 0x55bcfd53fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.169+0000 7f85aa000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd5e49f0 0x55bcfd681b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.169+0000 7f85aaa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd681640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.169+0000 7f85aa000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd5e49f0 0x55bcfd681b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60814/0 (socket says 172.21.15.44:60814) 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.169+0000 7f85aa000700 1 -- 172.21.15.44:0/2752992542 learned_addr learned my addr 172.21.15.44:0/2752992542 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.169+0000 7f85ab400700 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 0x55bcfd6820c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.169+0000 7f85aa000700 1 -- 172.21.15.44:0/2752992542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 msgr2=0x55bcfd6820c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.169+0000 7f85aa000700 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 0x55bcfd6820c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.169+0000 7f85aa000700 1 -- 172.21.15.44:0/2752992542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 msgr2=0x55bcfd681640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:31.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.169+0000 7f85aa000700 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd681640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.169+0000 7f85aa000700 1 -- 172.21.15.44:0/2752992542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcfd4e58d0 con 0x55bcfd5e49f0 2024-05-27T09:36:31.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.169+0000 7f85aaa00700 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd681640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:31.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.170+0000 7f85aa000700 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd5e49f0 0x55bcfd681b80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8598005fd0 tx=0x7f8598012aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29697e60f42ad8d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:31.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.170+0000 7f8594c00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:31.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.170+0000 7f8594c00700 1 -- 172.21.15.44:0/2752992542 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598004030 con 0x55bcfd5e49f0 2024-05-27T09:36:31.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.170+0000 7f85c03cdb80 1 -- 172.21.15.44:0/2752992542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f859c0b2050 con 0x55bcfd5e49f0 2024-05-27T09:36:31.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.170+0000 7f8594c00700 1 -- 172.21.15.44:0/2752992542 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f85980041d0 con 0x55bcfd5e49f0 2024-05-27T09:36:31.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.170+0000 7f8594c00700 1 -- 172.21.15.44:0/2752992542 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f85980043d0 con 0x55bcfd5e49f0 2024-05-27T09:36:31.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.170+0000 7f85c03cdb80 10 client.25511 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:31.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.170+0000 7f85c03cdb80 1 -- 172.21.15.44:0/2752992542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bcfd566060 con 0x55bcfd5e49f0 2024-05-27T09:36:31.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.171+0000 7f8594c00700 1 -- 172.21.15.44:0/2752992542 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f859c0b2050 con 0x55bcfd5e49f0 2024-05-27T09:36:31.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.171+0000 7f8594c00700 1 -- 172.21.15.44:0/2752992542 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f859801c580 con 0x55bcfd5e49f0 2024-05-27T09:36:31.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.171+0000 7f85c03cdb80 10 client.25511 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:31.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.171+0000 7f85c03cdb80 10 client.25511 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:31.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.171+0000 7f85c03cdb80 20 client.25511 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:31.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.171+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bcfd691b20 0x55bcfd693f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.171+0000 7f85c03cdb80 4 client.25511 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:31.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.171+0000 7f85c03cdb80 1 -- 172.21.15.44:0/2752992542 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcfd5a7d90 con 0x55bcfd691b20 2024-05-27T09:36:31.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.172+0000 7f85aaa00700 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bcfd691b20 0x55bcfd693f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.172+0000 7f85aaa00700 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bcfd691b20 0x55bcfd693f40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c692a372700a32c6 server_cookie=b394f887243fb16b in_seq=0 out_seq=0 2024-05-27T09:36:31.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.172+0000 7f8594c00700 10 client.25511 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:31.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.173+0000 7f8594c00700 1 -- 172.21.15.44:0/2752992542 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bcfd5a7d90 con 0x55bcfd691b20 2024-05-27T09:36:31.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.173+0000 7f8594c00700 10 client.25511 handle_command_reply: tid=0 2024-05-27T09:36:31.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.173+0000 7f85c03cdb80 1 client.25511 shutdown 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f85c03cdb80 1 -- 172.21.15.44:0/2752992542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd5e49f0 msgr2=0x55bcfd681b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd5e49f0 0x55bcfd681b80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8598005fd0 tx=0x7f8598012aa0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f85c03cdb80 1 -- 172.21.15.44:0/2752992542 shutdown_connections 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bcfd691b20 0x55bcfd693f40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 0x55bcfd6820c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd681640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f8594c00700 0 client.25511 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/2752992542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd5e49f0 0x55bcfd681b80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f85c03cdb80 1 -- 172.21.15.44:0/2752992542 >> 172.21.15.44:0/2752992542 conn(0x55bcfd592390 msgr2=0x55bcfd677310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f85c03cdb80 1 -- 172.21.15.44:0/2752992542 shutdown_connections 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f85c03cdb80 1 -- 172.21.15.44:0/2752992542 wait complete. 2024-05-27T09:36:31.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.174+0000 7f85c03cdb80 20 client.25511 trim_cache size 0 max 16384 2024-05-27T09:36:31.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.191+0000 7f85c03cdb80 1 Processor -- start 2024-05-27T09:36:31.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.191+0000 7f85c03cdb80 1 -- start start 2024-05-27T09:36:31.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.191+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd4f8e40 0x55bcfd68dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.191+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd5e49f0 0x55bcfd67b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.191+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 0x55bcfd67b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.191+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b7d00 con 0x55bcfd5e49f0 2024-05-27T09:36:31.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.191+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b5590 con 0x55bcfd677e70 2024-05-27T09:36:31.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.191+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b33e0 con 0x55bcfd4f8e40 2024-05-27T09:36:31.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.192+0000 7f85ab400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 0x55bcfd67b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.192+0000 7f85aa000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd5e49f0 0x55bcfd67b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.192+0000 7f85ab400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 0x55bcfd67b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60828/0 (socket says 172.21.15.44:60828) 2024-05-27T09:36:31.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.192+0000 7f85ab400700 1 -- 172.21.15.44:0/3477513757 learned_addr learned my addr 172.21.15.44:0/3477513757 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:31.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.192+0000 7f85aaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd4f8e40 0x55bcfd68dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.192+0000 7f85aa000700 1 -- 172.21.15.44:0/3477513757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd4f8e40 msgr2=0x55bcfd68dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:31.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.192+0000 7f85aa000700 1 --2- 172.21.15.44:0/3477513757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd4f8e40 0x55bcfd68dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.192+0000 7f85aa000700 1 -- 172.21.15.44:0/3477513757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 msgr2=0x55bcfd67b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:31.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.192+0000 7f85aa000700 1 --2- 172.21.15.44:0/3477513757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 0x55bcfd67b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.192+0000 7f85aa000700 1 -- 172.21.15.44:0/3477513757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcfd682430 con 0x55bcfd5e49f0 2024-05-27T09:36:31.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.192+0000 7f85ab400700 1 --2- 172.21.15.44:0/3477513757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 0x55bcfd67b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:31.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.193+0000 7f85aa000700 1 --2- 172.21.15.44:0/3477513757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd5e49f0 0x55bcfd67b030 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f859801cb60 tx=0x7f859801ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa08565b6ace02c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:31.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.193+0000 7f8597400700 1 -- 172.21.15.44:0/3477513757 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598029040 con 0x55bcfd5e49f0 2024-05-27T09:36:31.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.193+0000 7f8597400700 1 -- 172.21.15.44:0/3477513757 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8598005030 con 0x55bcfd5e49f0 2024-05-27T09:36:31.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.193+0000 7f8597400700 1 -- 172.21.15.44:0/3477513757 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85980048a0 con 0x55bcfd5e49f0 2024-05-27T09:36:31.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.194+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3477513757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd5e49f0 msgr2=0x55bcfd67b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:31.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.194+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/3477513757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd5e49f0 0x55bcfd67b030 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f859801cb60 tx=0x7f859801ce70 comp rx=0 tx=0).stop 2024-05-27T09:36:31.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.194+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3477513757 shutdown_connections 2024-05-27T09:36:31.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.194+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/3477513757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd677e70 0x55bcfd67b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.194+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/3477513757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd4f8e40 0x55bcfd68dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.194+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/3477513757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd5e49f0 0x55bcfd67b030 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.194+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3477513757 >> 172.21.15.44:0/3477513757 conn(0x55bcfd592390 msgr2=0x55bcfd67ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:31.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.194+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3477513757 shutdown_connections 2024-05-27T09:36:31.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.194+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3477513757 wait complete. 2024-05-27T09:36:31.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.195+0000 7f85c03cdb80 1 -- start start 2024-05-27T09:36:31.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.195+0000 7f85c03cdb80 1 Processor -- start 2024-05-27T09:36:31.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd67b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 0x55bcfd67b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85c03cdb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd67bb50 0x55bcfd688e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b3a50 con 0x55bcfd4f8e40 2024-05-27T09:36:31.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b4ec0 con 0x55bcfd67bb50 2024-05-27T09:36:31.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85c03cdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b4120 con 0x55bcfd677e70 2024-05-27T09:36:31.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85c03cdb80 10 client.0 _ll_register_callbacks cb 0x55bcfd53fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:31.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85aa000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 0x55bcfd67b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85aaa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd67b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85aa000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 0x55bcfd67b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43134/0 (socket says 172.21.15.44:43134) 2024-05-27T09:36:31.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85aa000700 1 -- 172.21.15.44:0/3825531310 learned_addr learned my addr 172.21.15.44:0/3825531310 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:31.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85ab400700 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd67bb50 0x55bcfd688e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85aa000700 1 -- 172.21.15.44:0/3825531310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd67bb50 msgr2=0x55bcfd688e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:31.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85aa000700 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd67bb50 0x55bcfd688e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85aa000700 1 -- 172.21.15.44:0/3825531310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 msgr2=0x55bcfd67b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:31.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85aa000700 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd67b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.196+0000 7f85aa000700 1 -- 172.21.15.44:0/3825531310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcfd689350 con 0x55bcfd677e70 2024-05-27T09:36:31.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.197+0000 7f85aa000700 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 0x55bcfd67b610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f859801c970 tx=0x7f8598009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1e1e33a82956210 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:31.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.197+0000 7f8594c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:36:31.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.197+0000 7f8594c00700 1 -- 172.21.15.44:0/3825531310 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598029070 con 0x55bcfd677e70 2024-05-27T09:36:31.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.197+0000 7f8594c00700 1 -- 172.21.15.44:0/3825531310 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f85980051e0 con 0x55bcfd677e70 2024-05-27T09:36:31.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.197+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3825531310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8598005030 con 0x55bcfd677e70 2024-05-27T09:36:31.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.197+0000 7f8594c00700 1 -- 172.21.15.44:0/3825531310 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f85980053e0 con 0x55bcfd677e70 2024-05-27T09:36:31.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.198+0000 7f85c03cdb80 10 client.25513 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:31.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.198+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3825531310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bcfd68a5b0 con 0x55bcfd677e70 2024-05-27T09:36:31.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.198+0000 7f8594c00700 1 -- 172.21.15.44:0/3825531310 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8598005030 con 0x55bcfd677e70 2024-05-27T09:36:31.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.198+0000 7f8594c00700 1 -- 172.21.15.44:0/3825531310 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f859802c020 con 0x55bcfd677e70 2024-05-27T09:36:31.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.198+0000 7f85c03cdb80 10 client.25513 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:31.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.198+0000 7f85c03cdb80 10 client.25513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:31.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.198+0000 7f85c03cdb80 20 client.25513 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:31.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.198+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bcfd6fc610 0x55bcfd6fea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:31.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.198+0000 7f85c03cdb80 4 client.25513 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:36:31.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.198+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3825531310 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bcfd5a1570 con 0x55bcfd6fc610 2024-05-27T09:36:31.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.199+0000 7f85aaa00700 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bcfd6fc610 0x55bcfd6fea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:31.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.199+0000 7f85aaa00700 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bcfd6fc610 0x55bcfd6fea30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ab6024c1bcb95bb server_cookie=a3527db36c2da375 in_seq=0 out_seq=0 2024-05-27T09:36:31.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.199+0000 7f8594c00700 10 client.25513 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.199+0000 7f8594c00700 1 -- 172.21.15.44:0/3825531310 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bcfd5a1570 con 0x55bcfd6fc610 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.199+0000 7f8594c00700 10 client.25513 handle_command_reply: tid=0 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.199+0000 7f85c03cdb80 1 client.25513 shutdown 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3825531310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 msgr2=0x55bcfd67b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 0x55bcfd67b610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f859801c970 tx=0x7f8598009310 comp rx=0 tx=0).stop 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3825531310 shutdown_connections 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bcfd6fc610 0x55bcfd6fea30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bcfd67bb50 0x55bcfd688e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcfd4f8e40 0x55bcfd67b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f85c03cdb80 1 --2- 172.21.15.44:0/3825531310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcfd677e70 0x55bcfd67b610 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3825531310 >> 172.21.15.44:0/3825531310 conn(0x55bcfd592390 msgr2=0x55bcfd6770b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f8594c00700 0 client.25513 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3825531310 shutdown_connections 2024-05-27T09:36:31.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f85c03cdb80 1 -- 172.21.15.44:0/3825531310 wait complete. 2024-05-27T09:36:31.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.200+0000 7f85c03cdb80 20 client.25513 trim_cache size 0 max 16384 2024-05-27T09:36:31.205 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:36:31.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 -- 172.21.15.44:0/2856971312 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f85900b2c20 msgr2=0x7f85900b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f85900b2c20 0x7f85900b5130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f85a4009550 tx=0x7f85a4009c20 comp rx=0 tx=0).stop 2024-05-27T09:36:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 -- 172.21.15.44:0/2856971312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85ac106570 msgr2=0x7f85ac1106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85ac106570 0x7f85ac1106e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f859c00e970 tx=0x7f859c00ec80 comp rx=0 tx=0).stop 2024-05-27T09:36:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 -- 172.21.15.44:0/2856971312 shutdown_connections 2024-05-27T09:36:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f85900b2c20 0x7f85900b5130 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85ac10c5b0 0x7f85ac111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85ac106570 0x7f85ac1106e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 --2- 172.21.15.44:0/2856971312 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f85ac106ec0 0x7f85ac110c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 -- 172.21.15.44:0/2856971312 >> 172.21.15.44:0/2856971312 conn(0x7f85ac102070 msgr2=0x7f85ac104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 -- 172.21.15.44:0/2856971312 shutdown_connections 2024-05-27T09:36:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:31.201+0000 7f85b1800700 1 -- 172.21.15.44:0/2856971312 wait complete. 2024-05-27T09:36:32.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:32 smithi044 ceph-mon[31618]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-27T09:36:32.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:32 smithi050 ceph-mon[36371]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-27T09:36:32.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:32 smithi189 ceph-mon[36671]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-27T09:36:33.092 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:36:33.092 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:36:33.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:33 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:36:33.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:33 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:36:33.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:33 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:36:33.588 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:34.445 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:34 smithi050 ceph-mon[36371]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-27T09:36:34.445 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:34 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:36:34.445 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:34 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:36:34.446 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:34 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:36:34.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:34 smithi044 ceph-mon[31618]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-27T09:36:34.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:34 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:36:34.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:34 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:36:34.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:34 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:36:34.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:34 smithi189 ceph-mon[36671]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-27T09:36:34.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:34 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:36:34.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:34 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:36:34.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:34 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:36:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.950+0000 7fd8b6800700 1 -- 172.21.15.44:0/2174872695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b010a7f0 msgr2=0x7fd8b010ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.950+0000 7fd8b6800700 1 --2- 172.21.15.44:0/2174872695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b010a7f0 0x7fd8b010ac00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009990 tx=0x7fd89c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.951+0000 7fd8b6800700 1 -- 172.21.15.44:0/2174872695 shutdown_connections 2024-05-27T09:36:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.951+0000 7fd8b6800700 1 --2- 172.21.15.44:0/2174872695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8b010b140 0x7fd8b0117670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.951+0000 7fd8b6800700 1 --2- 172.21.15.44:0/2174872695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b01107d0 0x7fd8b0110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.951+0000 7fd8b6800700 1 --2- 172.21.15.44:0/2174872695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b010a7f0 0x7fd8b010ac00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.951+0000 7fd8b6800700 1 -- 172.21.15.44:0/2174872695 >> 172.21.15.44:0/2174872695 conn(0x7fd8b0076af0 msgr2=0x7fd8b0076f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:35.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.951+0000 7fd8b6800700 1 -- 172.21.15.44:0/2174872695 shutdown_connections 2024-05-27T09:36:35.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.951+0000 7fd8b6800700 1 -- 172.21.15.44:0/2174872695 wait complete. 2024-05-27T09:36:35.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.952+0000 7fd8b6800700 1 Processor -- start 2024-05-27T09:36:35.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.952+0000 7fd8b6800700 1 -- start start 2024-05-27T09:36:35.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.953+0000 7fd8b6800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b010a7f0 0x7fd8b01a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.953+0000 7fd8b6800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b010b140 0x7fd8b01a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.953+0000 7fd8b6800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8b01107d0 0x7fd8b01a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.953+0000 7fd8b6800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b011a8e0 con 0x7fd8b010b140 2024-05-27T09:36:35.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.953+0000 7fd8b6800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b011a760 con 0x7fd8b01107d0 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.953+0000 7fd8b6800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b011aa60 con 0x7fd8b010a7f0 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.953+0000 7fd8b4a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b010b140 0x7fd8b01a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.953+0000 7fd8b5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b010a7f0 0x7fd8b01a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.953+0000 7fd8b4a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b010b140 0x7fd8b01a1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36372/0 (socket says 172.21.15.44:36372) 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.953+0000 7fd8b4a00700 1 -- 172.21.15.44:0/1895373390 learned_addr learned my addr 172.21.15.44:0/1895373390 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.953+0000 7fd8b5e00700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8b01107d0 0x7fd8b01a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.954+0000 7fd8b5400700 1 -- 172.21.15.44:0/1895373390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8b01107d0 msgr2=0x7fd8b01a5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.954+0000 7fd8b5400700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8b01107d0 0x7fd8b01a5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.954+0000 7fd8b5400700 1 -- 172.21.15.44:0/1895373390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b010b140 msgr2=0x7fd8b01a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.954+0000 7fd8b5400700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b010b140 0x7fd8b01a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.954+0000 7fd8b5400700 1 -- 172.21.15.44:0/1895373390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b01a61a0 con 0x7fd8b010a7f0 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.954+0000 7fd8b4a00700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b010b140 0x7fd8b01a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:35.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.954+0000 7fd8b5400700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b010a7f0 0x7fd8b01a12d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400e9c0 tx=0x7fd8a400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:35.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.954+0000 7fd8ae000700 1 -- 172.21.15.44:0/1895373390 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a400c9c0 con 0x7fd8b010a7f0 2024-05-27T09:36:35.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.954+0000 7fd8b6800700 1 -- 172.21.15.44:0/1895373390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b01a6490 con 0x7fd8b010a7f0 2024-05-27T09:36:35.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.954+0000 7fd8ae000700 1 -- 172.21.15.44:0/1895373390 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8a400cb60 con 0x7fd8b010a7f0 2024-05-27T09:36:35.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.954+0000 7fd8ae000700 1 -- 172.21.15.44:0/1895373390 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4018680 con 0x7fd8b010a7f0 2024-05-27T09:36:35.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.955+0000 7fd8b6800700 1 -- 172.21.15.44:0/1895373390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8b01addb0 con 0x7fd8b010a7f0 2024-05-27T09:36:35.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.958+0000 7fd8ae000700 1 -- 172.21.15.44:0/1895373390 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7fd8a4010430 con 0x7fd8b010a7f0 2024-05-27T09:36:35.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.959+0000 7fd8c68ffb80 1 Processor -- start 2024-05-27T09:36:35.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.959+0000 7fd8c68ffb80 1 -- start start 2024-05-27T09:36:35.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351d6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351e1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8ae000700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd8980b2ba0 0x7fd8980b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fa35105ec0 con 0x55fa351d78a0 2024-05-27T09:36:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55fa350f5a60 con 0x55fa35058ec0 2024-05-27T09:36:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fa350f9aa0 con 0x55fa351d7490 2024-05-27T09:36:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8b5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8ae000700 1 -- 172.21.15.44:0/1895373390 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fd8a40e2990 con 0x7fd8b010a7f0 2024-05-27T09:36:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8b5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351e1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8b5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36386/0 (socket says 172.21.15.44:36386) 2024-05-27T09:36:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.960+0000 7fd8b5e00700 1 -- 172.21.15.44:0/746987370 learned_addr learned my addr 172.21.15.44:0/746987370 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.961+0000 7fd8b5e00700 1 --2- 172.21.15.44:0/746987370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351d6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.961+0000 7fd8b4a00700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd8980b2ba0 0x7fd8980b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.961+0000 7fd8b5e00700 1 -- 172.21.15.44:0/746987370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 msgr2=0x55fa351e1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:35.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.961+0000 7fd8b5e00700 1 --2- 172.21.15.44:0/746987370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351e1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.961+0000 7fd8b5e00700 1 -- 172.21.15.44:0/746987370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 msgr2=0x55fa351d6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:35.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.961+0000 7fd8b5e00700 1 --2- 172.21.15.44:0/746987370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351d6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.961+0000 7fd8b5e00700 1 -- 172.21.15.44:0/746987370 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa350be4a0 con 0x55fa351d78a0 2024-05-27T09:36:35.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.961+0000 7fd8b5e00700 1 --2- 172.21.15.44:0/746987370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e40a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009fd0 tx=0x7fd8a8009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb36b07f1a4c5a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:35.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.962+0000 7fd8a2a00700 1 -- 172.21.15.44:0/746987370 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8003a00 con 0x55fa351d78a0 2024-05-27T09:36:35.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.962+0000 7fd8b4a00700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd8980b2ba0 0x7fd8980b50b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0077f90 tx=0x7fd89c005960 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:35.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.962+0000 7fd8a2a00700 1 -- 172.21.15.44:0/746987370 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8a8018d30 con 0x55fa351d78a0 2024-05-27T09:36:35.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.962+0000 7fd8a2a00700 1 -- 172.21.15.44:0/746987370 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a80041a0 con 0x55fa351d78a0 2024-05-27T09:36:35.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.963+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/746987370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 msgr2=0x55fa351e40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:35.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.963+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/746987370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e40a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009fd0 tx=0x7fd8a8009920 comp rx=0 tx=0).stop 2024-05-27T09:36:35.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.963+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/746987370 shutdown_connections 2024-05-27T09:36:35.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.963+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/746987370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e40a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.963+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/746987370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351d6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.963+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/746987370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351e1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.963+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/746987370 >> 172.21.15.44:0/746987370 conn(0x55fa350b9040 msgr2=0x55fa351d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:35.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.963+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/746987370 shutdown_connections 2024-05-27T09:36:35.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.963+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/746987370 wait complete. 2024-05-27T09:36:35.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.964+0000 7fd8c68ffb80 1 -- start start 2024-05-27T09:36:35.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.964+0000 7fd8c68ffb80 1 Processor -- start 2024-05-27T09:36:35.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa35058ec0 0x55fa351e1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa351d7490 0x55fa351e1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fa34ffc020 con 0x55fa351d78a0 2024-05-27T09:36:35.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55fa350967e0 con 0x55fa351d7490 2024-05-27T09:36:35.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fa351e4a10 con 0x55fa35058ec0 2024-05-27T09:36:35.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8c68ffb80 10 client.0 _ll_register_callbacks cb 0x55fa3509fcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:35.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8b5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa35058ec0 0x55fa351e1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8b5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa35058ec0 0x55fa351e1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39970/0 (socket says 172.21.15.44:39970) 2024-05-27T09:36:35.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8b5400700 1 -- 172.21.15.44:0/3401662633 learned_addr learned my addr 172.21.15.44:0/3401662633 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:35.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8b5e00700 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.965+0000 7fd8b4a00700 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa351d7490 0x55fa351e1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.966+0000 7fd8b5400700 1 -- 172.21.15.44:0/3401662633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa351d7490 msgr2=0x55fa351e1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.966+0000 7fd8b5400700 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa351d7490 0x55fa351e1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.966+0000 7fd8b5400700 1 -- 172.21.15.44:0/3401662633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 msgr2=0x55fa351e2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.966+0000 7fd8b5400700 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.966+0000 7fd8b5400700 1 -- 172.21.15.44:0/3401662633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa35045950 con 0x55fa35058ec0 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.966+0000 7fd8b4a00700 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa351d7490 0x55fa351e1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.966+0000 7fd8b5400700 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa35058ec0 0x55fa351e1690 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd8a40a6cd0 tx=0x7fd8a40b2710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30d309b29af7da7a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.966+0000 7fd897400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.966+0000 7fd897400700 1 -- 172.21.15.44:0/3401662633 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a40a9570 con 0x55fa35058ec0 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.966+0000 7fd897400700 1 -- 172.21.15.44:0/3401662633 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fd8a40a9ef0 con 0x55fa35058ec0 2024-05-27T09:36:35.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.966+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/3401662633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8a8018d30 con 0x55fa35058ec0 2024-05-27T09:36:35.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.967+0000 7fd897400700 1 -- 172.21.15.44:0/3401662633 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd8a40aa160 con 0x55fa35058ec0 2024-05-27T09:36:35.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.967+0000 7fd8c68ffb80 10 client.25525 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:35.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.967+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/3401662633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fa350c60e0 con 0x55fa35058ec0 2024-05-27T09:36:35.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.967+0000 7fd897400700 1 -- 172.21.15.44:0/3401662633 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8a8018d30 con 0x55fa35058ec0 2024-05-27T09:36:35.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.967+0000 7fd897400700 1 -- 172.21.15.44:0/3401662633 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd8a40a53d0 con 0x55fa35058ec0 2024-05-27T09:36:35.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.967+0000 7fd8c68ffb80 10 client.25525 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:35.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.967+0000 7fd8c68ffb80 10 client.25525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:35.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.967+0000 7fd8c68ffb80 20 client.25525 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:35.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.967+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55fa351f1980 0x55fa351f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.967+0000 7fd8c68ffb80 4 client.25525 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:35.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.967+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/3401662633 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa35107e10 con 0x55fa351f1980 2024-05-27T09:36:35.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.968+0000 7fd8b4a00700 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55fa351f1980 0x55fa351f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.968+0000 7fd8b4a00700 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55fa351f1980 0x55fa351f3de0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f02392e99e7f60c server_cookie=569cbee1befe32c5 in_seq=0 out_seq=0 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.968+0000 7fd897400700 10 client.25525 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.969+0000 7fd897400700 1 -- 172.21.15.44:0/3401662633 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fa35107e10 con 0x55fa351f1980 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.969+0000 7fd897400700 10 client.25525 handle_command_reply: tid=0 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.969+0000 7fd8c68ffb80 1 client.25525 shutdown 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.969+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/3401662633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa35058ec0 msgr2=0x55fa351e1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.969+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa35058ec0 0x55fa351e1690 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd8a40a6cd0 tx=0x7fd8a40b2710 comp rx=0 tx=0).stop 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.970+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/3401662633 shutdown_connections 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.970+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55fa351f1980 0x55fa351f3de0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.970+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.970+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa35058ec0 0x55fa351e1690 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.970+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/3401662633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa351d7490 0x55fa351e1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.970+0000 7fd897400700 0 client.25525 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.970+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/3401662633 >> 172.21.15.44:0/3401662633 conn(0x55fa350b9040 msgr2=0x55fa351d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:35.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.970+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/3401662633 shutdown_connections 2024-05-27T09:36:35.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.970+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/3401662633 wait complete. 2024-05-27T09:36:35.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.970+0000 7fd8c68ffb80 20 client.25525 trim_cache size 0 max 16384 2024-05-27T09:36:35.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.994+0000 7fd8c68ffb80 1 Processor -- start 2024-05-27T09:36:35.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.994+0000 7fd8c68ffb80 1 -- start start 2024-05-27T09:36:35.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.994+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351dceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.994+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351db820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.995+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:35.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.995+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b01b7ad0 con 0x55fa351d78a0 2024-05-27T09:36:35.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.995+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b01b5360 con 0x55fa35058ec0 2024-05-27T09:36:35.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.995+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b01b31b0 con 0x55fa351d7490 2024-05-27T09:36:35.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.995+0000 7fd8b5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.995+0000 7fd8b5400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351dceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.995+0000 7fd8b4a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351db820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.995+0000 7fd8b5400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351dceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50210/0 (socket says 172.21.15.44:50210) 2024-05-27T09:36:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.995+0000 7fd8b4a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351db820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39972/0 (socket says 172.21.15.44:39972) 2024-05-27T09:36:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.995+0000 7fd8b5400700 1 -- 172.21.15.44:0/1636226838 learned_addr learned my addr 172.21.15.44:0/1636226838 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.996+0000 7fd8b4a00700 1 -- 172.21.15.44:0/1636226838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 msgr2=0x55fa351dceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.996+0000 7fd8b4a00700 1 --2- 172.21.15.44:0/1636226838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351dceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.996+0000 7fd8b4a00700 1 -- 172.21.15.44:0/1636226838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 msgr2=0x55fa351e0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.996+0000 7fd8b4a00700 1 --2- 172.21.15.44:0/1636226838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.996+0000 7fd8b4a00700 1 -- 172.21.15.44:0/1636226838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa351d43d0 con 0x55fa351d7490 2024-05-27T09:36:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.996+0000 7fd8b5400700 1 --2- 172.21.15.44:0/1636226838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351dceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.996+0000 7fd8b5e00700 1 --2- 172.21.15.44:0/1636226838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:35.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.996+0000 7fd8b4a00700 1 --2- 172.21.15.44:0/1636226838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351db820 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009750 tx=0x7fd89c01e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89c1e36e896fa24a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:36.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.996+0000 7fd8a2000700 1 -- 172.21.15.44:0/1636226838 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c00f660 con 0x55fa351d7490 2024-05-27T09:36:36.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.996+0000 7fd8a2000700 1 -- 172.21.15.44:0/1636226838 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd89c00fcb0 con 0x55fa351d7490 2024-05-27T09:36:36.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.996+0000 7fd8a2000700 1 -- 172.21.15.44:0/1636226838 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c031a30 con 0x55fa351d7490 2024-05-27T09:36:36.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.997+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1636226838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 msgr2=0x55fa351db820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:36.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.997+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/1636226838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351db820 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009750 tx=0x7fd89c01e000 comp rx=0 tx=0).stop 2024-05-27T09:36:36.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.997+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1636226838 shutdown_connections 2024-05-27T09:36:36.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.997+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/1636226838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.997+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/1636226838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351dceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.997+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/1636226838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351db820 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.997+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1636226838 >> 172.21.15.44:0/1636226838 conn(0x55fa350b9040 msgr2=0x55fa351ecaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.998+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1636226838 shutdown_connections 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.998+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1636226838 wait complete. 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.998+0000 7fd8c68ffb80 1 -- start start 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.998+0000 7fd8c68ffb80 1 Processor -- start 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351e9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351ea3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8c68ffb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b01b3820 con 0x55fa351d78a0 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b01b4c90 con 0x55fa35058ec0 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8c68ffb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b01b3ef0 con 0x55fa351d7490 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8c68ffb80 10 client.0 _ll_register_callbacks cb 0x55fa351f42c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:36.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8b5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8b5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36416/0 (socket says 172.21.15.44:36416) 2024-05-27T09:36:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8b5e00700 1 -- 172.21.15.44:0/1722641863 learned_addr learned my addr 172.21.15.44:0/1722641863 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8b4a00700 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351ea3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:35.999+0000 7fd8b5400700 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351e9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.000+0000 7fd8b5e00700 1 -- 172.21.15.44:0/1722641863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 msgr2=0x55fa351ea3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.000+0000 7fd8b5e00700 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351ea3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.000+0000 7fd8b5e00700 1 -- 172.21.15.44:0/1722641863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 msgr2=0x55fa351e9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.000+0000 7fd8b5e00700 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351e9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.000+0000 7fd8b5e00700 1 -- 172.21.15.44:0/1722641863 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa351e73a0 con 0x55fa351d78a0 2024-05-27T09:36:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.000+0000 7fd8b5e00700 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e6c70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8003f00 tx=0x7fd8a8003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25175d79adc0a7b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.000+0000 7fd897400700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:36:36.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.000+0000 7fd897400700 1 -- 172.21.15.44:0/1722641863 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8018a90 con 0x55fa351d78a0 2024-05-27T09:36:36.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.001+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1722641863 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd89c00fcb0 con 0x55fa351d78a0 2024-05-27T09:36:36.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.001+0000 7fd897400700 1 -- 172.21.15.44:0/1722641863 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fd8a8018c30 con 0x55fa351d78a0 2024-05-27T09:36:36.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.001+0000 7fd897400700 1 -- 172.21.15.44:0/1722641863 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd8a8035050 con 0x55fa351d78a0 2024-05-27T09:36:36.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.001+0000 7fd8c68ffb80 10 client.15390 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:36.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.001+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1722641863 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fa3525ce40 con 0x55fa351d78a0 2024-05-27T09:36:36.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.001+0000 7fd897400700 1 -- 172.21.15.44:0/1722641863 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd89c00fcb0 con 0x55fa351d78a0 2024-05-27T09:36:36.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.001+0000 7fd897400700 1 -- 172.21.15.44:0/1722641863 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd8a801b450 con 0x55fa351d78a0 2024-05-27T09:36:36.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.002+0000 7fd8c68ffb80 10 client.15390 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:36.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.002+0000 7fd8c68ffb80 10 client.15390 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:36.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.002+0000 7fd8c68ffb80 20 client.15390 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:36.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.002+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55fa3525d110 0x55fa3525f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:36.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.002+0000 7fd8c68ffb80 4 client.15390 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:36:36.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.002+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1722641863 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fa351015f0 con 0x55fa3525d110 2024-05-27T09:36:36.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.002+0000 7fd8b5400700 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55fa3525d110 0x55fa3525f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:36.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.002+0000 7fd8b5400700 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55fa3525d110 0x55fa3525f530 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4eb6bec2c472030f server_cookie=f18cd77664fbb5a9 in_seq=0 out_seq=0 2024-05-27T09:36:36.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.002+0000 7fd897400700 10 client.15390 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:36.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.004+0000 7fd897400700 1 -- 172.21.15.44:0/1722641863 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fa351015f0 con 0x55fa3525d110 2024-05-27T09:36:36.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.004+0000 7fd897400700 10 client.15390 handle_command_reply: tid=0 2024-05-27T09:36:36.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.004+0000 7fd8c68ffb80 1 client.15390 shutdown 2024-05-27T09:36:36.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.004+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1722641863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 msgr2=0x55fa351e6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:36.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.004+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e6c70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8003f00 tx=0x7fd8a8003c30 comp rx=0 tx=0).stop 2024-05-27T09:36:36.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.005+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1722641863 shutdown_connections 2024-05-27T09:36:36.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.005+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55fa3525d110 0x55fa3525f530 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.005+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fa351d78a0 0x55fa351e6c70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.005+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa35058ec0 0x55fa351e9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.005+0000 7fd8c68ffb80 1 --2- 172.21.15.44:0/1722641863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa351d7490 0x55fa351ea3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.005+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1722641863 >> 172.21.15.44:0/1722641863 conn(0x55fa350b9040 msgr2=0x55fa351e05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:36.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.005+0000 7fd897400700 0 client.15390 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:36.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.005+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1722641863 shutdown_connections 2024-05-27T09:36:36.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.005+0000 7fd8c68ffb80 1 -- 172.21.15.44:0/1722641863 wait complete. 2024-05-27T09:36:36.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.006+0000 7fd8c68ffb80 20 client.15390 trim_cache size 0 max 16384 2024-05-27T09:36:36.012 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:36:36.013 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:36:36.013 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "10edc55e-9b6f-4945-a36e-50c3feaf8134", 2024-05-27T09:36:36.013 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:36:36.013 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:36:36.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.006+0000 7fd8b6800700 1 -- 172.21.15.44:0/1895373390 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd8980b2ba0 msgr2=0x7fd8980b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:36.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.006+0000 7fd8b6800700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd8980b2ba0 0x7fd8980b50b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0077f90 tx=0x7fd89c005960 comp rx=0 tx=0).stop 2024-05-27T09:36:36.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.006+0000 7fd8b6800700 1 -- 172.21.15.44:0/1895373390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b010a7f0 msgr2=0x7fd8b01a12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:36.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.006+0000 7fd8b6800700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b010a7f0 0x7fd8b01a12d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400e9c0 tx=0x7fd8a400ecd0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.007+0000 7fd8b6800700 1 -- 172.21.15.44:0/1895373390 shutdown_connections 2024-05-27T09:36:36.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.007+0000 7fd8b6800700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd8980b2ba0 0x7fd8980b50b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.007+0000 7fd8b6800700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8b01107d0 0x7fd8b01a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.007+0000 7fd8b6800700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b010a7f0 0x7fd8b01a12d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.007+0000 7fd8b6800700 1 --2- 172.21.15.44:0/1895373390 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8b010b140 0x7fd8b01a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:36.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.007+0000 7fd8b6800700 1 -- 172.21.15.44:0/1895373390 >> 172.21.15.44:0/1895373390 conn(0x7fd8b0076af0 msgr2=0x7fd8b0108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:36.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.007+0000 7fd8b6800700 1 -- 172.21.15.44:0/1895373390 shutdown_connections 2024-05-27T09:36:36.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:36.007+0000 7fd8b6800700 1 -- 172.21.15.44:0/1895373390 wait complete. 2024-05-27T09:36:36.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:36 smithi044 ceph-mon[31618]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-05-27T09:36:36.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:36 smithi050 ceph-mon[36371]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-05-27T09:36:36.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:36 smithi189 ceph-mon[36671]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-05-27T09:36:36.884 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:36:37.309 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.4 2024-05-27T09:36:37.309 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:36:37.348 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:37 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:36:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:37 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:36:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:37 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:36:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:37 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:36:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:37 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:36:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:37 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:36:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:37 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:36:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:37 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:36:37.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:37 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:36:37.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:37 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:36:37.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:37 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:36:37.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:37 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:36:38.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:38 smithi044 ceph-mon[31618]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-27T09:36:38.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:38 smithi050 ceph-mon[36371]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-27T09:36:38.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:38 smithi189 ceph-mon[36671]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-27T09:36:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:39.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.939+0000 7f3233800700 1 -- 172.21.15.44:0/845037287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c103a40 msgr2=0x7f322c110760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:39.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.939+0000 7f3233800700 1 --2- 172.21.15.44:0/845037287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c103a40 0x7f322c110760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3218009990 tx=0x7f3218009ca0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.939+0000 7f3233800700 1 -- 172.21.15.44:0/845037287 shutdown_connections 2024-05-27T09:36:39.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.939+0000 7f3233800700 1 --2- 172.21.15.44:0/845037287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f322c110ca0 0x7f322c1130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.939+0000 7f3233800700 1 --2- 172.21.15.44:0/845037287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f322c1030f0 0x7f322c103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.939+0000 7f3233800700 1 --2- 172.21.15.44:0/845037287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c103a40 0x7f322c110760 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.939+0000 7f3233800700 1 -- 172.21.15.44:0/845037287 >> 172.21.15.44:0/845037287 conn(0x7f322c0fedb0 msgr2=0x7f322c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:39.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.939+0000 7f3233800700 1 -- 172.21.15.44:0/845037287 shutdown_connections 2024-05-27T09:36:39.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.940+0000 7f3233800700 1 -- 172.21.15.44:0/845037287 wait complete. 2024-05-27T09:36:39.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.941+0000 7f3233800700 1 Processor -- start 2024-05-27T09:36:39.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.941+0000 7f3233800700 1 -- start start 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.941+0000 7f3233800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f322c1030f0 0x7f322c19d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.941+0000 7f3233800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f322c110ca0 0x7f322c19d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.941+0000 7f3233800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c1a19a0 0x7f322c1a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.941+0000 7f3233800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1166a0 con 0x7f322c1030f0 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.941+0000 7f3233800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f322c116520 con 0x7f322c1a19a0 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.941+0000 7f3233800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f322c116820 con 0x7f322c110ca0 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.942+0000 7f3231800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c1a19a0 0x7f322c1a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.942+0000 7f3230e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f322c1030f0 0x7f322c19d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.942+0000 7f3231800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c1a19a0 0x7f322c1a1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50248/0 (socket says 172.21.15.44:50248) 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.942+0000 7f3231800700 1 -- 172.21.15.44:0/2670687543 learned_addr learned my addr 172.21.15.44:0/2670687543 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.942+0000 7f322be00700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f322c110ca0 0x7f322c19d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.942+0000 7f3231800700 1 -- 172.21.15.44:0/2670687543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f322c110ca0 msgr2=0x7f322c19d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.942+0000 7f3231800700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f322c110ca0 0x7f322c19d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.942+0000 7f3231800700 1 -- 172.21.15.44:0/2670687543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f322c1030f0 msgr2=0x7f322c19d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:39.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.942+0000 7f3231800700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f322c1030f0 0x7f322c19d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.942+0000 7f3231800700 1 -- 172.21.15.44:0/2670687543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322c1a24f0 con 0x7f322c1a19a0 2024-05-27T09:36:39.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.943+0000 7f3231800700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c1a19a0 0x7f322c1a1db0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f322400ef30 tx=0x7f322400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:39.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.943+0000 7f3229600700 1 -- 172.21.15.44:0/2670687543 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32240092e0 con 0x7f322c1a19a0 2024-05-27T09:36:39.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.943+0000 7f3229600700 1 -- 172.21.15.44:0/2670687543 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3224004bb0 con 0x7f322c1a19a0 2024-05-27T09:36:39.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.943+0000 7f3229600700 1 -- 172.21.15.44:0/2670687543 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3224004020 con 0x7f322c1a19a0 2024-05-27T09:36:39.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.943+0000 7f3233800700 1 -- 172.21.15.44:0/2670687543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f322c1a2780 con 0x7f322c1a19a0 2024-05-27T09:36:39.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.943+0000 7f3233800700 1 -- 172.21.15.44:0/2670687543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f322c1a9ee0 con 0x7f322c1a19a0 2024-05-27T09:36:39.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.947+0000 7f3229600700 1 -- 172.21.15.44:0/2670687543 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f322401b030 con 0x7f322c1a19a0 2024-05-27T09:36:39.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.948+0000 7f3242386b80 1 Processor -- start 2024-05-27T09:36:39.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f3242386b80 1 -- start start 2024-05-27T09:36:39.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f3229600700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32140b2c20 0x7f32140b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f3242386b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0f0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f3242386b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 0x55d02d0f0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f3242386b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 0x55d02d0fd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f3242386b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d02d01ee40 con 0x55d02d05d9f0 2024-05-27T09:36:39.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f3242386b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d02d00e9e0 con 0x55d02cf71e40 2024-05-27T09:36:39.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f3242386b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d02d012a20 con 0x55d02d0f0e70 2024-05-27T09:36:39.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f3229600700 1 -- 172.21.15.44:0/2670687543 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f32240e2240 con 0x7f322c1a19a0 2024-05-27T09:36:39.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f322be00700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32140b2c20 0x7f32140b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f322be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 0x55d02d0f0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f3231800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 0x55d02d0fd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.949+0000 7f3230e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0f0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.950+0000 7f322be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 0x55d02d0f0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50256/0 (socket says 172.21.15.44:50256) 2024-05-27T09:36:39.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.950+0000 7f322be00700 1 -- 172.21.15.44:0/4010017873 learned_addr learned my addr 172.21.15.44:0/4010017873 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:39.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.950+0000 7f3231800700 1 -- 172.21.15.44:0/4010017873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 msgr2=0x55d02d0f0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:39.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.950+0000 7f3231800700 1 --2- 172.21.15.44:0/4010017873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 0x55d02d0f0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.950+0000 7f3231800700 1 -- 172.21.15.44:0/4010017873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 msgr2=0x55d02d0f0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:39.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.950+0000 7f3231800700 1 --2- 172.21.15.44:0/4010017873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0f0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.950+0000 7f3231800700 1 -- 172.21.15.44:0/4010017873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d02cfd7420 con 0x55d02d0f0e70 2024-05-27T09:36:39.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.950+0000 7f3231800700 1 --2- 172.21.15.44:0/4010017873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 0x55d02d0fd110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f32240df780 tx=0x7f32240a8eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6251e6be54f7b5e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:39.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.951+0000 7f321e000700 1 -- 172.21.15.44:0/4010017873 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32240160e0 con 0x55d02d0f0e70 2024-05-27T09:36:39.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.951+0000 7f322be00700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32140b2c20 0x7f32140b5130 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55d02d0eda20 tx=0x7f321801a040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:39.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.951+0000 7f321e000700 1 -- 172.21.15.44:0/4010017873 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3224013aa0 con 0x55d02d0f0e70 2024-05-27T09:36:39.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.951+0000 7f321e000700 1 -- 172.21.15.44:0/4010017873 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322401b640 con 0x55d02d0f0e70 2024-05-27T09:36:39.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.952+0000 7f3242386b80 1 -- 172.21.15.44:0/4010017873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 msgr2=0x55d02d0fd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:39.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.952+0000 7f3242386b80 1 --2- 172.21.15.44:0/4010017873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 0x55d02d0fd110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f32240df780 tx=0x7f32240a8eb0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.952+0000 7f3242386b80 1 -- 172.21.15.44:0/4010017873 shutdown_connections 2024-05-27T09:36:39.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.952+0000 7f3242386b80 1 --2- 172.21.15.44:0/4010017873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 0x55d02d0fd110 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.952+0000 7f3242386b80 1 --2- 172.21.15.44:0/4010017873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0f0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.952+0000 7f3242386b80 1 --2- 172.21.15.44:0/4010017873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 0x55d02d0f0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.952+0000 7f3242386b80 1 -- 172.21.15.44:0/4010017873 >> 172.21.15.44:0/4010017873 conn(0x55d02d00b390 msgr2=0x55d02d0ec4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:39.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.952+0000 7f3242386b80 1 -- 172.21.15.44:0/4010017873 shutdown_connections 2024-05-27T09:36:39.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.952+0000 7f3242386b80 1 -- 172.21.15.44:0/4010017873 wait complete. 2024-05-27T09:36:39.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.953+0000 7f3242386b80 1 -- start start 2024-05-27T09:36:39.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.953+0000 7f3242386b80 1 Processor -- start 2024-05-27T09:36:39.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3242386b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0fa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3242386b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0faba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3242386b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02d0f0e70 0x55d02d0fb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3242386b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d02cf14fa0 con 0x55d02d05d9f0 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3242386b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d02cfaf760 con 0x55d02d0f0e70 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3242386b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d02d0fdb80 con 0x55d02cf71e40 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3242386b80 10 client.0 _ll_register_callbacks cb 0x55d02cfb8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f322be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0faba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3230e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0fa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3230e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0fa660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:40030/0 (socket says 172.21.15.44:40030) 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3230e00700 1 -- 172.21.15.44:0/4061244315 learned_addr learned my addr 172.21.15.44:0/4061244315 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3231800700 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02d0f0e70 0x55d02d0fb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3230e00700 1 -- 172.21.15.44:0/4061244315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02d0f0e70 msgr2=0x55d02d0fb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3230e00700 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02d0f0e70 0x55d02d0fb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3230e00700 1 -- 172.21.15.44:0/4061244315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 msgr2=0x55d02d0faba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:39.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3230e00700 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0faba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.954+0000 7f3230e00700 1 -- 172.21.15.44:0/4061244315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d02cf5e8d0 con 0x55d02cf71e40 2024-05-27T09:36:39.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.955+0000 7f3230e00700 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0fa660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f32200127f0 tx=0x7f3220012b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43dc0bd437aa337c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:39.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.955+0000 7f3212a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:36:39.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.955+0000 7f3212a00700 1 -- 172.21.15.44:0/4061244315 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322000f4e0 con 0x55d02cf71e40 2024-05-27T09:36:39.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.955+0000 7f3212a00700 1 -- 172.21.15.44:0/4061244315 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f322000f680 con 0x55d02cf71e40 2024-05-27T09:36:39.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.955+0000 7f3242386b80 1 -- 172.21.15.44:0/4061244315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3224013aa0 con 0x55d02cf71e40 2024-05-27T09:36:39.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.956+0000 7f3212a00700 1 -- 172.21.15.44:0/4061244315 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f322000f880 con 0x55d02cf71e40 2024-05-27T09:36:39.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.956+0000 7f3242386b80 10 client.25543 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:39.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.956+0000 7f3242386b80 1 -- 172.21.15.44:0/4061244315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d02cfdf060 con 0x55d02cf71e40 2024-05-27T09:36:39.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.956+0000 7f3212a00700 1 -- 172.21.15.44:0/4061244315 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3224013aa0 con 0x55d02cf71e40 2024-05-27T09:36:39.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.956+0000 7f3212a00700 1 -- 172.21.15.44:0/4061244315 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3220010a10 con 0x55d02cf71e40 2024-05-27T09:36:39.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.956+0000 7f3242386b80 10 client.25543 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:39.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.956+0000 7f3242386b80 10 client.25543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:39.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.956+0000 7f3242386b80 20 client.25543 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:39.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.956+0000 7f3242386b80 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d02d10ab30 0x55d02d10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.956+0000 7f3242386b80 4 client.25543 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:39.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.956+0000 7f3242386b80 1 -- 172.21.15.44:0/4061244315 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d02d020d90 con 0x55d02d10ab30 2024-05-27T09:36:39.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.957+0000 7f322be00700 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d02d10ab30 0x55d02d10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.957+0000 7f322be00700 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d02d10ab30 0x55d02d10cfd0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2039bf5cde0106 server_cookie=61523742340c6b26 in_seq=0 out_seq=0 2024-05-27T09:36:39.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.957+0000 7f3212a00700 10 client.25543 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:39.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.958+0000 7f3212a00700 1 -- 172.21.15.44:0/4061244315 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d02d020d90 con 0x55d02d10ab30 2024-05-27T09:36:39.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.958+0000 7f3212a00700 10 client.25543 handle_command_reply: tid=0 2024-05-27T09:36:39.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.958+0000 7f3242386b80 1 client.25543 shutdown 2024-05-27T09:36:39.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3242386b80 1 -- 172.21.15.44:0/4061244315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 msgr2=0x55d02d0fa660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:39.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3242386b80 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0fa660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f32200127f0 tx=0x7f3220012b00 comp rx=0 tx=0).stop 2024-05-27T09:36:39.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3242386b80 1 -- 172.21.15.44:0/4061244315 shutdown_connections 2024-05-27T09:36:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3242386b80 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d02d10ab30 0x55d02d10cfd0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3242386b80 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02d0f0e70 0x55d02d0fb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3242386b80 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0fa660 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3242386b80 1 --2- 172.21.15.44:0/4061244315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0faba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3212a00700 0 client.25543 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3242386b80 1 -- 172.21.15.44:0/4061244315 >> 172.21.15.44:0/4061244315 conn(0x55d02d00b390 msgr2=0x55d02d0f0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3242386b80 1 -- 172.21.15.44:0/4061244315 shutdown_connections 2024-05-27T09:36:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3242386b80 1 -- 172.21.15.44:0/4061244315 wait complete. 2024-05-27T09:36:39.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.959+0000 7f3242386b80 20 client.25543 trim_cache size 0 max 16384 2024-05-27T09:36:39.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.982+0000 7f3242386b80 1 Processor -- start 2024-05-27T09:36:39.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.982+0000 7f3242386b80 1 -- start start 2024-05-27T09:36:39.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.982+0000 7f3242386b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.982+0000 7f3242386b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02d05d9f0 0x55d02d106c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.982+0000 7f3242386b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d0f0e70 0x55d02d0fc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.982+0000 7f3242386b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1b3f10 con 0x55d02d0f0e70 2024-05-27T09:36:39.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.982+0000 7f3242386b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1b17a0 con 0x55d02d05d9f0 2024-05-27T09:36:39.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.982+0000 7f3242386b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1af5f0 con 0x55d02cf71e40 2024-05-27T09:36:39.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f3230e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f3231800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d0f0e70 0x55d02d0fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f3230e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0f5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:40034/0 (socket says 172.21.15.44:40034) 2024-05-27T09:36:39.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f3230e00700 1 -- 172.21.15.44:0/1893019826 learned_addr learned my addr 172.21.15.44:0/1893019826 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:39.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f322be00700 1 --2- 172.21.15.44:0/1893019826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02d05d9f0 0x55d02d106c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f3230e00700 1 -- 172.21.15.44:0/1893019826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02d05d9f0 msgr2=0x55d02d106c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:39.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f3230e00700 1 --2- 172.21.15.44:0/1893019826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02d05d9f0 0x55d02d106c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f3230e00700 1 -- 172.21.15.44:0/1893019826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d0f0e70 msgr2=0x55d02d0fc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:39.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f3230e00700 1 --2- 172.21.15.44:0/1893019826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d0f0e70 0x55d02d0fc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f3230e00700 1 -- 172.21.15.44:0/1893019826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d02d0ed850 con 0x55d02cf71e40 2024-05-27T09:36:39.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f322be00700 1 --2- 172.21.15.44:0/1893019826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02d05d9f0 0x55d02d106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:36:39.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.983+0000 7f3230e00700 1 --2- 172.21.15.44:0/1893019826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0f5e90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3220025020 tx=0x7f322000f770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23dd401315bee694 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:39.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.984+0000 7f321d600700 1 -- 172.21.15.44:0/1893019826 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3220029070 con 0x55d02cf71e40 2024-05-27T09:36:39.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.984+0000 7f321d600700 1 -- 172.21.15.44:0/1893019826 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32200050b0 con 0x55d02cf71e40 2024-05-27T09:36:39.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.984+0000 7f321d600700 1 -- 172.21.15.44:0/1893019826 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32200038e0 con 0x55d02cf71e40 2024-05-27T09:36:39.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.985+0000 7f3242386b80 1 -- 172.21.15.44:0/1893019826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 msgr2=0x55d02d0f5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:39.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.985+0000 7f3242386b80 1 --2- 172.21.15.44:0/1893019826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0f5e90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3220025020 tx=0x7f322000f770 comp rx=0 tx=0).stop 2024-05-27T09:36:39.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.985+0000 7f3242386b80 1 -- 172.21.15.44:0/1893019826 shutdown_connections 2024-05-27T09:36:39.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.985+0000 7f3242386b80 1 --2- 172.21.15.44:0/1893019826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d0f0e70 0x55d02d0fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.985+0000 7f3242386b80 1 --2- 172.21.15.44:0/1893019826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02cf71e40 0x55d02d0f5e90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.985+0000 7f3242386b80 1 --2- 172.21.15.44:0/1893019826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02d05d9f0 0x55d02d106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.985+0000 7f3242386b80 1 -- 172.21.15.44:0/1893019826 >> 172.21.15.44:0/1893019826 conn(0x55d02d00b390 msgr2=0x55d02d0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:39.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.985+0000 7f3242386b80 1 -- 172.21.15.44:0/1893019826 shutdown_connections 2024-05-27T09:36:39.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.985+0000 7f3242386b80 1 -- 172.21.15.44:0/1893019826 wait complete. 2024-05-27T09:36:39.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.986+0000 7f3242386b80 1 -- start start 2024-05-27T09:36:39.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.986+0000 7f3242386b80 1 Processor -- start 2024-05-27T09:36:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f3242386b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 0x55d02d0fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f3242386b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0ff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f3242386b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 0x55d02d0ff810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f3242386b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1afc60 con 0x55d02d05d9f0 2024-05-27T09:36:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f3242386b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1b10d0 con 0x55d02cf71e40 2024-05-27T09:36:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f3242386b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1b0330 con 0x55d02d0f0e70 2024-05-27T09:36:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f3242386b80 10 client.0 _ll_register_callbacks cb 0x55d02cfb8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f3230e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 0x55d02d0fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f3230e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 0x55d02d0fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50286/0 (socket says 172.21.15.44:50286) 2024-05-27T09:36:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f3231800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 0x55d02d0ff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f322be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0ff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.987+0000 7f3230e00700 1 -- 172.21.15.44:0/75387623 learned_addr learned my addr 172.21.15.44:0/75387623 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.988+0000 7f3230e00700 1 -- 172.21.15.44:0/75387623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 msgr2=0x55d02d0ff810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.988+0000 7f3230e00700 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 0x55d02d0ff810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.988+0000 7f3230e00700 1 -- 172.21.15.44:0/75387623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 msgr2=0x55d02d0ff2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.988+0000 7f3230e00700 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0ff2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.988+0000 7f3230e00700 1 -- 172.21.15.44:0/75387623 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d02d1747d0 con 0x55d02cf71e40 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.988+0000 7f3231800700 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 0x55d02d0ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.988+0000 7f3230e00700 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 0x55d02d0fed90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3220000c00 tx=0x7f3220005510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8e23c3fb63cf556 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.988+0000 7f3212a00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.988+0000 7f3212a00700 1 -- 172.21.15.44:0/75387623 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3220029070 con 0x55d02cf71e40 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.988+0000 7f3212a00700 1 -- 172.21.15.44:0/75387623 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f322000fc30 con 0x55d02cf71e40 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3242386b80 1 -- 172.21.15.44:0/75387623 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32200050b0 con 0x55d02cf71e40 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3212a00700 1 -- 172.21.15.44:0/75387623 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f322003e050 con 0x55d02cf71e40 2024-05-27T09:36:39.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3242386b80 10 client.25553 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:39.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3242386b80 1 -- 172.21.15.44:0/75387623 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d02d175a40 con 0x55d02cf71e40 2024-05-27T09:36:39.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3212a00700 1 -- 172.21.15.44:0/75387623 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32200050b0 con 0x55d02cf71e40 2024-05-27T09:36:39.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3212a00700 1 -- 172.21.15.44:0/75387623 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3220010430 con 0x55d02cf71e40 2024-05-27T09:36:39.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3242386b80 10 client.25553 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:39.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3242386b80 10 client.25553 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:39.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3242386b80 20 client.25553 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:39.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3242386b80 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d02d176010 0x55d02d178430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:39.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3242386b80 4 client.25553 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:36:39.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.989+0000 7f3242386b80 1 -- 172.21.15.44:0/75387623 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d02d01a570 con 0x55d02d176010 2024-05-27T09:36:39.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.990+0000 7f322be00700 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d02d176010 0x55d02d178430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:39.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.990+0000 7f322be00700 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d02d176010 0x55d02d178430 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=602a79fc08b4e1bb server_cookie=3753af7950fca1ac in_seq=0 out_seq=0 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.990+0000 7f3212a00700 10 client.25553 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3212a00700 1 -- 172.21.15.44:0/75387623 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d02d01a570 con 0x55d02d176010 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3212a00700 10 client.25553 handle_command_reply: tid=0 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 1 client.25553 shutdown 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 1 -- 172.21.15.44:0/75387623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 msgr2=0x55d02d0fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 0x55d02d0fed90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3220000c00 tx=0x7f3220005510 comp rx=0 tx=0).stop 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 1 -- 172.21.15.44:0/75387623 shutdown_connections 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d02d176010 0x55d02d178430 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d02d0f0e70 0x55d02d0ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d02cf71e40 0x55d02d0fed90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3212a00700 0 client.25553 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:39.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 1 --2- 172.21.15.44:0/75387623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d02d05d9f0 0x55d02d0ff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 1 -- 172.21.15.44:0/75387623 >> 172.21.15.44:0/75387623 conn(0x55d02d00b390 msgr2=0x55d02d0fae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:39.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 1 -- 172.21.15.44:0/75387623 shutdown_connections 2024-05-27T09:36:39.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 1 -- 172.21.15.44:0/75387623 wait complete. 2024-05-27T09:36:39.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.991+0000 7f3242386b80 20 client.25553 trim_cache size 0 max 16384 2024-05-27T09:36:39.994 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:36:39.994 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:36:39.994 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:36:39.994 INFO:teuthology.orchestra.run.smithi044.stdout: "10edc55e-9b6f-4945-a36e-50c3feaf8134": { 2024-05-27T09:36:39.994 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:36:39.995 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "10edc55e-9b6f-4945-a36e-50c3feaf8134", 2024-05-27T09:36:39.995 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:36:39.995 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:36:39.995 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:36:39.995 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:36:39.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.992+0000 7f3233800700 1 -- 172.21.15.44:0/2670687543 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32140b2c20 msgr2=0x7f32140b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:39.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.992+0000 7f3233800700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32140b2c20 0x7f32140b5130 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55d02d0eda20 tx=0x7f321801a040 comp rx=0 tx=0).stop 2024-05-27T09:36:39.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.992+0000 7f3233800700 1 -- 172.21.15.44:0/2670687543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c1a19a0 msgr2=0x7f322c1a1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:39.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.992+0000 7f3233800700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c1a19a0 0x7f322c1a1db0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f322400ef30 tx=0x7f322400c3a0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.992+0000 7f3233800700 1 -- 172.21.15.44:0/2670687543 shutdown_connections 2024-05-27T09:36:39.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.992+0000 7f3233800700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32140b2c20 0x7f32140b5130 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.992+0000 7f3233800700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c1a19a0 0x7f322c1a1db0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.992+0000 7f3233800700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f322c1030f0 0x7f322c19d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.992+0000 7f3233800700 1 --2- 172.21.15.44:0/2670687543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f322c110ca0 0x7f322c19d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:39.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.992+0000 7f3233800700 1 -- 172.21.15.44:0/2670687543 >> 172.21.15.44:0/2670687543 conn(0x7f322c0fedb0 msgr2=0x7f322c101d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:39.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.992+0000 7f3233800700 1 -- 172.21.15.44:0/2670687543 shutdown_connections 2024-05-27T09:36:39.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:39.993+0000 7f3233800700 1 -- 172.21.15.44:0/2670687543 wait complete. 2024-05-27T09:36:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:40 smithi044 ceph-mon[31618]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-05-27T09:36:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:40 smithi050 ceph-mon[36371]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-05-27T09:36:40.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:40 smithi189 ceph-mon[36671]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-05-27T09:36:41.412 INFO:tasks.cephfs.filesystem:scrub status for tag:10edc55e-9b6f-4945-a36e-50c3feaf8134 - {'path': '/', 'tag': '10edc55e-9b6f-4945-a36e-50c3feaf8134', 'options': 'recursive,force'} 2024-05-27T09:36:41.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:41 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:36:41.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:41 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:36:41.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:41 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:36:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.693+0000 7f5480e00640 1 -- 172.21.15.44:0/822567172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f547c109f50 msgr2=0x7f547c103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.693+0000 7f5480e00640 1 --2- 172.21.15.44:0/822567172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f547c109f50 0x7f547c103b90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5468009860 tx=0x7f546802a340 comp rx=0 tx=0).stop 2024-05-27T09:36:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.693+0000 7f5480e00640 1 -- 172.21.15.44:0/822567172 shutdown_connections 2024-05-27T09:36:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.693+0000 7f5480e00640 1 --2- 172.21.15.44:0/822567172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f547c104200 0x7f547c1105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.693+0000 7f5480e00640 1 --2- 172.21.15.44:0/822567172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f547c109f50 0x7f547c103b90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.693+0000 7f5480e00640 1 --2- 172.21.15.44:0/822567172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547c109ab0 0x7f547c103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.693+0000 7f5480e00640 1 -- 172.21.15.44:0/822567172 >> 172.21.15.44:0/822567172 conn(0x7f547c079140 msgr2=0x7f547c079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.694+0000 7f5480e00640 1 -- 172.21.15.44:0/822567172 shutdown_connections 2024-05-27T09:36:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.694+0000 7f5480e00640 1 -- 172.21.15.44:0/822567172 wait complete. 2024-05-27T09:36:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.695+0000 7f5480e00640 1 Processor -- start 2024-05-27T09:36:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.695+0000 7f5480e00640 1 -- start start 2024-05-27T09:36:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.695+0000 7f5480e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f547c104200 0x7f547c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.695+0000 7f5480e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f547c109ab0 0x7f547c19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.695+0000 7f5480e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547c109f50 0x7f547c19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.695+0000 7f5480e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f547c113ca0 con 0x7f547c104200 2024-05-27T09:36:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.695+0000 7f5480e00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f547c113b20 con 0x7f547c109f50 2024-05-27T09:36:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.695+0000 7f5480e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f547c113e20 con 0x7f547c109ab0 2024-05-27T09:36:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.696+0000 7f547b400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f547c104200 0x7f547c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.696+0000 7f547b400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f547c104200 0x7f547c19a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36486/0 (socket says 172.21.15.44:36486) 2024-05-27T09:36:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.696+0000 7f547b400640 1 -- 172.21.15.44:0/1057604308 learned_addr learned my addr 172.21.15.44:0/1057604308 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.696+0000 7f547aa00640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f547c109ab0 0x7f547c19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.696+0000 7f547be00640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547c109f50 0x7f547c19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.696+0000 7f547b400640 1 -- 172.21.15.44:0/1057604308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f547c109ab0 msgr2=0x7f547c19ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.696+0000 7f547b400640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f547c109ab0 0x7f547c19ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.696+0000 7f547b400640 1 -- 172.21.15.44:0/1057604308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547c109f50 msgr2=0x7f547c19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.696+0000 7f547b400640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547c109f50 0x7f547c19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.696+0000 7f547b400640 1 -- 172.21.15.44:0/1057604308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547c19f580 con 0x7f547c104200 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.696+0000 7f547b400640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f547c104200 0x7f547c19a720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f546c00e9f0 tx=0x7f546c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.697+0000 7f5473e00640 1 -- 172.21.15.44:0/1057604308 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c00ccc0 con 0x7f547c104200 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.697+0000 7f5473e00640 1 -- 172.21.15.44:0/1057604308 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f546c0108e0 con 0x7f547c104200 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.697+0000 7f5480e00640 1 -- 172.21.15.44:0/1057604308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f547c19f870 con 0x7f547c104200 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.697+0000 7f5473e00640 1 -- 172.21.15.44:0/1057604308 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c018880 con 0x7f547c104200 2024-05-27T09:36:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.697+0000 7f5480e00640 1 -- 172.21.15.44:0/1057604308 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f547c1a71f0 con 0x7f547c104200 2024-05-27T09:36:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.701+0000 7f5489455740 1 Processor -- start 2024-05-27T09:36:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.702+0000 7f5489455740 1 -- start start 2024-05-27T09:36:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.702+0000 7f5489455740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b751b0 0x565488f17910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.702+0000 7f5489455740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b4c2f0 0x565488f17e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.702+0000 7f5489455740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b544d0 0x565488f18390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.702+0000 7f5489455740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x565488e88b90 con 0x565488b751b0 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.702+0000 7f5489455740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x565488e78730 con 0x565488b4c2f0 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.702+0000 7f5489455740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x565488e7c770 con 0x565488b544d0 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.703+0000 7f5473e00640 1 -- 172.21.15.44:0/1057604308 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7f546c010430 con 0x7f547c104200 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.703+0000 7f547be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b751b0 0x565488f17910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.703+0000 7f547be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b751b0 0x565488f17910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36498/0 (socket says 172.21.15.44:36498) 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.703+0000 7f547be00640 1 -- 172.21.15.44:0/4079378798 learned_addr learned my addr 172.21.15.44:0/4079378798 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.703+0000 7f547aa00640 1 --2- 172.21.15.44:0/4079378798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b544d0 0x565488f18390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.703+0000 7f547b400640 1 --2- 172.21.15.44:0/4079378798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b4c2f0 0x565488f17e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.703+0000 7f547be00640 1 -- 172.21.15.44:0/4079378798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b544d0 msgr2=0x565488f18390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:41.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.703+0000 7f547be00640 1 --2- 172.21.15.44:0/4079378798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b544d0 0x565488f18390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.703+0000 7f547be00640 1 -- 172.21.15.44:0/4079378798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b4c2f0 msgr2=0x565488f17e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:41.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.703+0000 7f547be00640 1 --2- 172.21.15.44:0/4079378798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b4c2f0 0x565488f17e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.705 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.703+0000 7f547be00640 1 -- 172.21.15.44:0/4079378798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565488e118b0 con 0x565488b751b0 2024-05-27T09:36:41.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.704+0000 7f547be00640 1 --2- 172.21.15.44:0/4079378798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b751b0 0x565488f17910 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5474009fd0 tx=0x7f54740118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd6fa2e362a3565c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:41.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.704+0000 7f5470c00640 1 -- 172.21.15.44:0/4079378798 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5474004d20 con 0x565488b751b0 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.704+0000 7f5470c00640 1 -- 172.21.15.44:0/4079378798 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5474004050 con 0x565488b751b0 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.704+0000 7f5470c00640 1 -- 172.21.15.44:0/4079378798 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5474005280 con 0x565488b751b0 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.704+0000 7f5473e00640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54600b2b80 0x7f54600b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.705+0000 7f5473e00640 1 -- 172.21.15.44:0/1057604308 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f546c0e2a00 con 0x7f547c104200 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.705+0000 7f547b400640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54600b2b80 0x7f54600b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.705+0000 7f5489455740 1 -- 172.21.15.44:0/4079378798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b751b0 msgr2=0x565488f17910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.705+0000 7f5489455740 1 --2- 172.21.15.44:0/4079378798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b751b0 0x565488f17910 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5474009fd0 tx=0x7f54740118d0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.705+0000 7f5489455740 1 -- 172.21.15.44:0/4079378798 shutdown_connections 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.705+0000 7f5489455740 1 --2- 172.21.15.44:0/4079378798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b544d0 0x565488f18390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.705+0000 7f5489455740 1 --2- 172.21.15.44:0/4079378798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b4c2f0 0x565488f17e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.705+0000 7f5489455740 1 --2- 172.21.15.44:0/4079378798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b751b0 0x565488f17910 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.705+0000 7f5489455740 1 -- 172.21.15.44:0/4079378798 >> 172.21.15.44:0/4079378798 conn(0x565488b69300 msgr2=0x565488f14210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.705+0000 7f5489455740 1 -- 172.21.15.44:0/4079378798 shutdown_connections 2024-05-27T09:36:41.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.705+0000 7f5489455740 1 -- 172.21.15.44:0/4079378798 wait complete. 2024-05-27T09:36:41.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.706+0000 7f547b400640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54600b2b80 0x7f54600b5090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f546c0dffa0 tx=0x7f546c00cef0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:41.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.706+0000 7f5489455740 1 -- start start 2024-05-27T09:36:41.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.706+0000 7f5489455740 1 Processor -- start 2024-05-27T09:36:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f5489455740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 0x565488f26060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f5489455740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b544d0 0x565488f265a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f5489455740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b751b0 0x565488f322d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f5489455740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x565488b5e790 con 0x565488b4c2f0 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f5489455740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x565488e20bc0 con 0x565488b751b0 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f5489455740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x565488f28fa0 con 0x565488b544d0 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f5489455740 10 client.0 _ll_register_callbacks cb 0x565488dfe400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f547be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 0x565488f26060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f547be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 0x565488f26060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36514/0 (socket says 172.21.15.44:36514) 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f547be00640 1 -- 172.21.15.44:0/2090650049 learned_addr learned my addr 172.21.15.44:0/2090650049 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f547aa00640 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b544d0 0x565488f265a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.707+0000 7f547be00640 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b751b0 0x565488f322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.708+0000 7f547be00640 1 -- 172.21.15.44:0/2090650049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b544d0 msgr2=0x565488f265a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.708+0000 7f547be00640 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b544d0 0x565488f265a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.708+0000 7f547be00640 1 -- 172.21.15.44:0/2090650049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b751b0 msgr2=0x565488f322d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.708+0000 7f547be00640 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b751b0 0x565488f322d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.708+0000 7f547be00640 1 -- 172.21.15.44:0/2090650049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565488b8d210 con 0x565488b4c2f0 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.708+0000 7f547be00640 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 0x565488f26060 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x565488f14e90 tx=0x7f5474003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e5ae109279268b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.708+0000 7f5465600640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.708+0000 7f5465600640 1 -- 172.21.15.44:0/2090650049 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5474004610 con 0x565488b4c2f0 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.708+0000 7f5465600640 1 -- 172.21.15.44:0/2090650049 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f5474005040 con 0x565488b4c2f0 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.708+0000 7f5489455740 1 -- 172.21.15.44:0/2090650049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5474004050 con 0x565488b4c2f0 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.709+0000 7f5465600640 1 -- 172.21.15.44:0/2090650049 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5474035050 con 0x565488b4c2f0 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.709+0000 7f5489455740 10 client.15420 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.709+0000 7f5489455740 1 -- 172.21.15.44:0/2090650049 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x565488c59620 con 0x565488b4c2f0 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.709+0000 7f5465600640 1 -- 172.21.15.44:0/2090650049 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5474004050 con 0x565488b4c2f0 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.709+0000 7f5465600640 1 -- 172.21.15.44:0/2090650049 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f547400f600 con 0x565488b4c2f0 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.709+0000 7f5489455740 10 client.15420 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.709+0000 7f5489455740 10 client.15420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.709+0000 7f5489455740 20 client.15420 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.709+0000 7f5489455740 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565488b33d80 0x565488f381c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.709+0000 7f5489455740 4 client.15420 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.709+0000 7f5489455740 1 -- 172.21.15.44:0/2090650049 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565488e8aae0 con 0x565488b33d80 2024-05-27T09:36:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.710+0000 7f547aa00640 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565488b33d80 0x565488f381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.710+0000 7f547aa00640 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565488b33d80 0x565488f381c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f84e2c11c729c40a server_cookie=ea35b4c6af0b9741 in_seq=0 out_seq=0 2024-05-27T09:36:41.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.710+0000 7f5465600640 10 client.15420 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:41.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.711+0000 7f5465600640 1 -- 172.21.15.44:0/2090650049 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x565488e8aae0 con 0x565488b33d80 2024-05-27T09:36:41.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.711+0000 7f5465600640 10 client.15420 handle_command_reply: tid=0 2024-05-27T09:36:41.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.711+0000 7f5489455740 1 client.15420 shutdown 2024-05-27T09:36:41.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5489455740 1 -- 172.21.15.44:0/2090650049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 msgr2=0x565488f26060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:41.714 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5489455740 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 0x565488f26060 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x565488f14e90 tx=0x7f5474003f50 comp rx=0 tx=0).stop 2024-05-27T09:36:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5489455740 1 -- 172.21.15.44:0/2090650049 shutdown_connections 2024-05-27T09:36:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5489455740 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565488b33d80 0x565488f381c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5489455740 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b751b0 0x565488f322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5489455740 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b544d0 0x565488f265a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5489455740 1 --2- 172.21.15.44:0/2090650049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 0x565488f26060 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5465600640 0 client.15420 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5489455740 1 -- 172.21.15.44:0/2090650049 >> 172.21.15.44:0/2090650049 conn(0x565488b69300 msgr2=0x565488f19710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5489455740 1 -- 172.21.15.44:0/2090650049 shutdown_connections 2024-05-27T09:36:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5489455740 1 -- 172.21.15.44:0/2090650049 wait complete. 2024-05-27T09:36:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.712+0000 7f5489455740 20 client.15420 trim_cache size 0 max 16384 2024-05-27T09:36:41.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.729+0000 7f5489455740 1 Processor -- start 2024-05-27T09:36:41.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.729+0000 7f5489455740 1 -- start start 2024-05-27T09:36:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.729+0000 7f5489455740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b4c2f0 0x565488f28790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.729+0000 7f5489455740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b544d0 0x565488f21890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.729+0000 7f5489455740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b751b0 0x565488f1fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.729+0000 7f5489455740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f547c1b0020 con 0x565488b544d0 2024-05-27T09:36:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.729+0000 7f5489455740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f547c1ad8f0 con 0x565488b751b0 2024-05-27T09:36:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.729+0000 7f5489455740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f547c1ab760 con 0x565488b4c2f0 2024-05-27T09:36:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b544d0 0x565488f21890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b544d0 0x565488f21890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36528/0 (socket says 172.21.15.44:36528) 2024-05-27T09:36:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547aa00640 1 -- 172.21.15.44:0/584063061 learned_addr learned my addr 172.21.15.44:0/584063061 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547be00640 1 --2- 172.21.15.44:0/584063061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b4c2f0 0x565488f28790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547aa00640 1 -- 172.21.15.44:0/584063061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b4c2f0 msgr2=0x565488f28790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:41.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547aa00640 1 --2- 172.21.15.44:0/584063061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b4c2f0 0x565488f28790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547be00640 1 --2- 172.21.15.44:0/584063061 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b751b0 0x565488f1fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547aa00640 1 -- 172.21.15.44:0/584063061 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b751b0 msgr2=0x565488f1fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:41.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547aa00640 1 --2- 172.21.15.44:0/584063061 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b751b0 0x565488f1fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547aa00640 1 -- 172.21.15.44:0/584063061 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565488f0f1f0 con 0x565488b544d0 2024-05-27T09:36:41.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547be00640 1 --2- 172.21.15.44:0/584063061 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b751b0 0x565488f1fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:36:41.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.730+0000 7f547aa00640 1 --2- 172.21.15.44:0/584063061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b544d0 0x565488f21890 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5468009830 tx=0x7f5468005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df89db0da8f4f1d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.731+0000 7f5467e00640 1 -- 172.21.15.44:0/584063061 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468037040 con 0x565488b544d0 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.731+0000 7f5467e00640 1 -- 172.21.15.44:0/584063061 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5468030900 con 0x565488b544d0 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.731+0000 7f5467e00640 1 -- 172.21.15.44:0/584063061 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468003af0 con 0x565488b544d0 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.732+0000 7f5489455740 1 -- 172.21.15.44:0/584063061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b544d0 msgr2=0x565488f21890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.732+0000 7f5489455740 1 --2- 172.21.15.44:0/584063061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b544d0 0x565488f21890 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5468009830 tx=0x7f5468005e70 comp rx=0 tx=0).stop 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.732+0000 7f5489455740 1 -- 172.21.15.44:0/584063061 shutdown_connections 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.732+0000 7f5489455740 1 --2- 172.21.15.44:0/584063061 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b751b0 0x565488f1fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.732+0000 7f5489455740 1 --2- 172.21.15.44:0/584063061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b544d0 0x565488f21890 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.732+0000 7f5489455740 1 --2- 172.21.15.44:0/584063061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b4c2f0 0x565488f28790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.732+0000 7f5489455740 1 -- 172.21.15.44:0/584063061 >> 172.21.15.44:0/584063061 conn(0x565488b69300 msgr2=0x565488f18220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.732+0000 7f5489455740 1 -- 172.21.15.44:0/584063061 shutdown_connections 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.732+0000 7f5489455740 1 -- 172.21.15.44:0/584063061 wait complete. 2024-05-27T09:36:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.732+0000 7f5489455740 1 -- start start 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.732+0000 7f5489455740 1 Processor -- start 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.733+0000 7f5489455740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 0x565488f1ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.733+0000 7f5489455740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b544d0 0x565488f204f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.733+0000 7f5489455740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b751b0 0x565488f20a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.733+0000 7f5489455740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f547c1abdb0 con 0x565488b4c2f0 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.733+0000 7f5489455740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f547c1ad220 con 0x565488b544d0 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.733+0000 7f5489455740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f547c1ac480 con 0x565488b751b0 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.733+0000 7f5489455740 10 client.0 _ll_register_callbacks cb 0x7f546c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.733+0000 7f547be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 0x565488f1ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.733+0000 7f547aa00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b544d0 0x565488f204f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.733+0000 7f547be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 0x565488f1ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36542/0 (socket says 172.21.15.44:36542) 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.733+0000 7f547be00640 1 -- 172.21.15.44:0/3936485974 learned_addr learned my addr 172.21.15.44:0/3936485974 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:41.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.734+0000 7f547be00640 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b751b0 0x565488f20a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.734+0000 7f547aa00640 1 -- 172.21.15.44:0/3936485974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b751b0 msgr2=0x565488f20a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:41.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.734+0000 7f547aa00640 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b751b0 0x565488f20a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.734+0000 7f547aa00640 1 -- 172.21.15.44:0/3936485974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 msgr2=0x565488f1ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:41.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.734+0000 7f547aa00640 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 0x565488f1ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.734+0000 7f547aa00640 1 -- 172.21.15.44:0/3936485974 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565488f2d3e0 con 0x565488b544d0 2024-05-27T09:36:41.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.734+0000 7f547be00640 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b751b0 0x565488f20a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:36:41.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.734+0000 7f547aa00640 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b544d0 0x565488f204f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5468009780 tx=0x7f5468003670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fa5754d70db4513 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:41.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.734+0000 7f5465600640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:36:41.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.734+0000 7f5465600640 1 -- 172.21.15.44:0/3936485974 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546804c070 con 0x565488b544d0 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.735+0000 7f5465600640 1 -- 172.21.15.44:0/3936485974 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f5468051460 con 0x565488b544d0 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.735+0000 7f5489455740 1 -- 172.21.15.44:0/3936485974 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5468030900 con 0x565488b544d0 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.735+0000 7f5465600640 1 -- 172.21.15.44:0/3936485974 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5468037040 con 0x565488b544d0 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.735+0000 7f5489455740 10 client.25568 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.735+0000 7f5489455740 1 -- 172.21.15.44:0/3936485974 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x565488f2e560 con 0x565488b544d0 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.735+0000 7f5465600640 1 -- 172.21.15.44:0/3936485974 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5468030900 con 0x565488b544d0 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.736+0000 7f5465600640 1 -- 172.21.15.44:0/3936485974 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5468044070 con 0x565488b544d0 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.736+0000 7f5489455740 10 client.25568 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.736+0000 7f5489455740 10 client.25568 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.736+0000 7f5489455740 20 client.25568 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.736+0000 7f5489455740 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565488b091c0 0x565488fa26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.736+0000 7f5489455740 4 client.25568 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.736+0000 7f5489455740 1 -- 172.21.15.44:0/3936485974 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565488e842c0 con 0x565488b091c0 2024-05-27T09:36:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.736+0000 7f547be00640 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565488b091c0 0x565488fa26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.736+0000 7f5465600640 10 client.25568 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.736+0000 7f547be00640 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565488b091c0 0x565488fa26d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3fa2b506a7e311b server_cookie=cbcbad8da2b2f4ef in_seq=0 out_seq=0 2024-05-27T09:36:41.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.744+0000 7f5465600640 1 -- 172.21.15.44:0/3936485974 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x565488e842c0 con 0x565488b091c0 2024-05-27T09:36:41.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5465600640 10 client.25568 handle_command_reply: tid=0 2024-05-27T09:36:41.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 1 client.25568 shutdown 2024-05-27T09:36:41.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 1 -- 172.21.15.44:0/3936485974 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b544d0 msgr2=0x565488f204f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:41.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b544d0 0x565488f204f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5468009780 tx=0x7f5468003670 comp rx=0 tx=0).stop 2024-05-27T09:36:41.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 1 -- 172.21.15.44:0/3936485974 shutdown_connections 2024-05-27T09:36:41.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565488b091c0 0x565488fa26d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565488b751b0 0x565488f20a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x565488b544d0 0x565488f204f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5465600640 0 client.25568 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:41.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 1 --2- 172.21.15.44:0/3936485974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565488b4c2f0 0x565488f1ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 1 -- 172.21.15.44:0/3936485974 >> 172.21.15.44:0/3936485974 conn(0x565488b69300 msgr2=0x565488f26320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:41.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 1 -- 172.21.15.44:0/3936485974 shutdown_connections 2024-05-27T09:36:41.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 1 -- 172.21.15.44:0/3936485974 wait complete. 2024-05-27T09:36:41.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.745+0000 7f5489455740 20 client.25568 trim_cache size 0 max 16384 2024-05-27T09:36:41.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:36:41.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:36:41.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:36:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 -- 172.21.15.44:0/1057604308 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54600b2b80 msgr2=0x7f54600b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54600b2b80 0x7f54600b5090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f546c0dffa0 tx=0x7f546c00cef0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 -- 172.21.15.44:0/1057604308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f547c104200 msgr2=0x7f547c19a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f547c104200 0x7f547c19a720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f546c00e9f0 tx=0x7f546c00eeb0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 -- 172.21.15.44:0/1057604308 shutdown_connections 2024-05-27T09:36:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547c109f50 0x7f547c19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f547c109ab0 0x7f547c19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54600b2b80 0x7f54600b5090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 --2- 172.21.15.44:0/1057604308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f547c104200 0x7f547c19a720 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 -- 172.21.15.44:0/1057604308 >> 172.21.15.44:0/1057604308 conn(0x7f547c079140 msgr2=0x7f547c102270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 -- 172.21.15.44:0/1057604308 shutdown_connections 2024-05-27T09:36:41.749 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:41.746+0000 7f5480e00640 1 -- 172.21.15.44:0/1057604308 wait complete. 2024-05-27T09:36:42.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.030+0000 7fc31bc00700 1 -- 172.21.15.44:0/3863054731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314106ec0 msgr2=0x7fc314113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:42.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.030+0000 7fc31bc00700 1 --2- 172.21.15.44:0/3863054731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314106ec0 0x7fc314113350 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00b0a0 tx=0x7fc30c00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:36:42.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.030+0000 7fc31bc00700 1 -- 172.21.15.44:0/3863054731 shutdown_connections 2024-05-27T09:36:42.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.030+0000 7fc31bc00700 1 --2- 172.21.15.44:0/3863054731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314106ec0 0x7fc314113350 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:42.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.030+0000 7fc31bc00700 1 --2- 172.21.15.44:0/3863054731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc31410c5b0 0x7fc31410c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:42.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.030+0000 7fc31bc00700 1 --2- 172.21.15.44:0/3863054731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314106570 0x7fc314106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:42.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.030+0000 7fc31bc00700 1 -- 172.21.15.44:0/3863054731 >> 172.21.15.44:0/3863054731 conn(0x7fc314102070 msgr2=0x7fc3141044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:42.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.030+0000 7fc31bc00700 1 -- 172.21.15.44:0/3863054731 shutdown_connections 2024-05-27T09:36:42.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.030+0000 7fc31bc00700 1 -- 172.21.15.44:0/3863054731 wait complete. 2024-05-27T09:36:42.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.032+0000 7fc31bc00700 1 Processor -- start 2024-05-27T09:36:42.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.032+0000 7fc31bc00700 1 -- start start 2024-05-27T09:36:42.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.032+0000 7fc31bc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314106570 0x7fc31419d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:42.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.032+0000 7fc31bc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314106ec0 0x7fc31419d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:42.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.032+0000 7fc31bc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc31410c5b0 0x7fc3141a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:42.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.032+0000 7fc31bc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc314116730 con 0x7fc31410c5b0 2024-05-27T09:36:42.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.032+0000 7fc31bc00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc3141165b0 con 0x7fc314106570 2024-05-27T09:36:42.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.032+0000 7fc31bc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc3141168b0 con 0x7fc314106ec0 2024-05-27T09:36:42.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.033+0000 7fc319200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314106570 0x7fc31419d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:42.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.033+0000 7fc319200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314106570 0x7fc31419d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50360/0 (socket says 172.21.15.44:50360) 2024-05-27T09:36:42.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.033+0000 7fc319200700 1 -- 172.21.15.44:0/826921142 learned_addr learned my addr 172.21.15.44:0/826921142 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:42.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.033+0000 7fc319c00700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc31410c5b0 0x7fc3141a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:42.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.033+0000 7fc313e00700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314106ec0 0x7fc31419d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:42.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.033+0000 7fc319200700 1 -- 172.21.15.44:0/826921142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314106ec0 msgr2=0x7fc31419d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:42.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.033+0000 7fc319200700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314106ec0 0x7fc31419d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:42.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.033+0000 7fc319200700 1 -- 172.21.15.44:0/826921142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc31410c5b0 msgr2=0x7fc3141a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:42.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.033+0000 7fc319200700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc31410c5b0 0x7fc3141a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:42.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.033+0000 7fc319200700 1 -- 172.21.15.44:0/826921142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3141a1ef0 con 0x7fc314106570 2024-05-27T09:36:42.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.034+0000 7fc319200700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314106570 0x7fc31419d020 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc30800e970 tx=0x7fc30800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:42.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.034+0000 7fc311600700 1 -- 172.21.15.44:0/826921142 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308004a30 con 0x7fc314106570 2024-05-27T09:36:42.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.034+0000 7fc311600700 1 -- 172.21.15.44:0/826921142 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30800cc90 con 0x7fc314106570 2024-05-27T09:36:42.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.034+0000 7fc311600700 1 -- 172.21.15.44:0/826921142 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308005610 con 0x7fc314106570 2024-05-27T09:36:42.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.034+0000 7fc31bc00700 1 -- 172.21.15.44:0/826921142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3141a21e0 con 0x7fc314106570 2024-05-27T09:36:42.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.034+0000 7fc31bc00700 1 -- 172.21.15.44:0/826921142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3141a9b00 con 0x7fc314106570 2024-05-27T09:36:42.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.035+0000 7fc31bc00700 1 -- 172.21.15.44:0/826921142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3140512b0 con 0x7fc314106570 2024-05-27T09:36:42.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.038+0000 7fc311600700 1 -- 172.21.15.44:0/826921142 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7fc30800c7e0 con 0x7fc314106570 2024-05-27T09:36:42.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.039+0000 7fc311600700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2fc0b2c20 0x7fc2fc0b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:42.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.040+0000 7fc311600700 1 -- 172.21.15.44:0/826921142 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fc3080e2bf0 con 0x7fc314106570 2024-05-27T09:36:42.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.040+0000 7fc313e00700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2fc0b2c20 0x7fc2fc0b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:42.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.041+0000 7fc313e00700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2fc0b2c20 0x7fc2fc0b5130 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc300009680 tx=0x7fc300009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:42.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.042+0000 7fc311600700 1 -- 172.21.15.44:0/826921142 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc308054df0 con 0x7fc314106570 2024-05-27T09:36:42.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.332+0000 7fc31bc00700 1 -- 172.21.15.44:0/826921142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc31402e5e0 con 0x7fc314106570 2024-05-27T09:36:42.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.333+0000 7fc311600700 1 -- 172.21.15.44:0/826921142 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7fc3080aa3c0 con 0x7fc314106570 2024-05-27T09:36:42.335 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:36:42.335 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:36:42.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 -- 172.21.15.44:0/826921142 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2fc0b2c20 msgr2=0x7fc2fc0b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:42.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2fc0b2c20 0x7fc2fc0b5130 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc300009680 tx=0x7fc300009260 comp rx=0 tx=0).stop 2024-05-27T09:36:42.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 -- 172.21.15.44:0/826921142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314106570 msgr2=0x7fc31419d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:42.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314106570 0x7fc31419d020 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc30800e970 tx=0x7fc30800ec80 comp rx=0 tx=0).stop 2024-05-27T09:36:42.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 -- 172.21.15.44:0/826921142 shutdown_connections 2024-05-27T09:36:42.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2fc0b2c20 0x7fc2fc0b5130 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:42.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc31410c5b0 0x7fc3141a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:42.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc314106570 0x7fc31419d020 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:42.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 --2- 172.21.15.44:0/826921142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc314106ec0 0x7fc31419d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:42.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 -- 172.21.15.44:0/826921142 >> 172.21.15.44:0/826921142 conn(0x7fc314102070 msgr2=0x7fc314104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:42.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 -- 172.21.15.44:0/826921142 shutdown_connections 2024-05-27T09:36:42.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:42.337+0000 7fc31bc00700 1 -- 172.21.15.44:0/826921142 wait complete. 2024-05-27T09:36:42.339 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:36:42.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:42 smithi044 ceph-mon[31618]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-27T09:36:42.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:42 smithi050 ceph-mon[36371]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-27T09:36:42.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:42 smithi189 ceph-mon[36671]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-27T09:36:43.263 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db / --timeout 90 --expiration 149.4 --await-for 45.0' 2024-05-27T09:36:43.263 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db / --timeout 90 --expiration 149.4 --await-for 45.0 2024-05-27T09:36:43.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:43 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/826921142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:36:43.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:43 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/826921142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:36:43.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:43 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/826921142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:36:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:36:44.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:44 smithi044 ceph-mon[31618]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-27T09:36:44.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:44 smithi189 ceph-mon[36671]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-27T09:36:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:44 smithi050 ceph-mon[36371]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-27T09:36:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.983+0000 7fd17f600700 1 -- 172.21.15.44:0/3117198164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd178106570 msgr2=0x7fd178106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.983+0000 7fd17f600700 1 --2- 172.21.15.44:0/3117198164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd178106570 0x7fd178106980 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd1640098f0 tx=0x7fd164009c00 comp rx=0 tx=0).stop 2024-05-27T09:36:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.984+0000 7fd17f600700 1 -- 172.21.15.44:0/3117198164 shutdown_connections 2024-05-27T09:36:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.984+0000 7fd17f600700 1 --2- 172.21.15.44:0/3117198164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd178106ec0 0x7fd178113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.984+0000 7fd17f600700 1 --2- 172.21.15.44:0/3117198164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17810c5b0 0x7fd17810c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.984+0000 7fd17f600700 1 --2- 172.21.15.44:0/3117198164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd178106570 0x7fd178106980 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.984+0000 7fd17f600700 1 -- 172.21.15.44:0/3117198164 >> 172.21.15.44:0/3117198164 conn(0x7fd178102070 msgr2=0x7fd1781044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.984+0000 7fd17f600700 1 -- 172.21.15.44:0/3117198164 shutdown_connections 2024-05-27T09:36:45.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.984+0000 7fd17f600700 1 -- 172.21.15.44:0/3117198164 wait complete. 2024-05-27T09:36:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.985+0000 7fd17f600700 1 Processor -- start 2024-05-27T09:36:45.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.986+0000 7fd17f600700 1 -- start start 2024-05-27T09:36:45.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.986+0000 7fd17f600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd178106570 0x7fd17819d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:45.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.986+0000 7fd17f600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd178106ec0 0x7fd17819d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:45.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.986+0000 7fd17f600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17810c5b0 0x7fd1781a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:45.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.986+0000 7fd17f600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd178116690 con 0x7fd178106ec0 2024-05-27T09:36:45.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.986+0000 7fd17f600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd178116510 con 0x7fd178106570 2024-05-27T09:36:45.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.986+0000 7fd17f600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd178116810 con 0x7fd17810c5b0 2024-05-27T09:36:45.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.987+0000 7fd17cc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd178106570 0x7fd17819d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:45.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.987+0000 7fd177e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd178106ec0 0x7fd17819d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:45.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.987+0000 7fd17d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17810c5b0 0x7fd1781a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:45.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.987+0000 7fd17cc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd178106570 0x7fd17819d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35936/0 (socket says 172.21.15.44:35936) 2024-05-27T09:36:45.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.987+0000 7fd17d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17810c5b0 0x7fd1781a1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43908/0 (socket says 172.21.15.44:43908) 2024-05-27T09:36:45.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.987+0000 7fd17cc00700 1 -- 172.21.15.44:0/3840065964 learned_addr learned my addr 172.21.15.44:0/3840065964 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:45.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.987+0000 7fd17cc00700 1 -- 172.21.15.44:0/3840065964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17810c5b0 msgr2=0x7fd1781a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:45.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.987+0000 7fd17cc00700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17810c5b0 0x7fd1781a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:45.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.987+0000 7fd17cc00700 1 -- 172.21.15.44:0/3840065964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd178106ec0 msgr2=0x7fd17819d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:45.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.987+0000 7fd17cc00700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd178106ec0 0x7fd17819d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:45.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.987+0000 7fd17cc00700 1 -- 172.21.15.44:0/3840065964 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1781a1f30 con 0x7fd178106570 2024-05-27T09:36:45.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.988+0000 7fd17cc00700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd178106570 0x7fd17819d060 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00ec70 tx=0x7fd16c00ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:45.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.988+0000 7fd175600700 1 -- 172.21.15.44:0/3840065964 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c00cc90 con 0x7fd178106570 2024-05-27T09:36:45.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.988+0000 7fd17f600700 1 -- 172.21.15.44:0/3840065964 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1781a2220 con 0x7fd178106570 2024-05-27T09:36:45.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.988+0000 7fd175600700 1 -- 172.21.15.44:0/3840065964 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd16c00ce30 con 0x7fd178106570 2024-05-27T09:36:45.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.988+0000 7fd175600700 1 -- 172.21.15.44:0/3840065964 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c010430 con 0x7fd178106570 2024-05-27T09:36:45.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.988+0000 7fd17f600700 1 -- 172.21.15.44:0/3840065964 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1781a9b40 con 0x7fd178106570 2024-05-27T09:36:45.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.992+0000 7fd175600700 1 -- 172.21.15.44:0/3840065964 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 149936+0+0 (secure 0 0 0) 0x7fd16c003680 con 0x7fd178106570 2024-05-27T09:36:45.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.993+0000 7fd18e102b80 1 Processor -- start 2024-05-27T09:36:45.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.993+0000 7fd18e102b80 1 -- start start 2024-05-27T09:36:45.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.993+0000 7fd175600700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1600b2ba0 0x7fd1600b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab13516b80 0x55ab135b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1342afd0 0x55ab135b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab135aa670 0x55ab135aaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd175600700 1 -- 172.21.15.44:0/3840065964 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fd16c0e2b60 con 0x7fd178106570 2024-05-27T09:36:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd18e102b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ab134d7fd0 con 0x55ab1342afd0 2024-05-27T09:36:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd18e102b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ab134c7b70 con 0x55ab13516b80 2024-05-27T09:36:45.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd18e102b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ab134cbbb0 con 0x55ab135aa670 2024-05-27T09:36:45.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd17d600700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1600b2ba0 0x7fd1600b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:45.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd177e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1342afd0 0x55ab135b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:45.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd17d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab135aa670 0x55ab135aaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:45.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd177e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1342afd0 0x55ab135b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59182/0 (socket says 172.21.15.44:59182) 2024-05-27T09:36:45.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd17cc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab13516b80 0x55ab135b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:45.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.994+0000 7fd177e00700 1 -- 172.21.15.44:0/2267972216 learned_addr learned my addr 172.21.15.44:0/2267972216 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.995+0000 7fd17cc00700 1 -- 172.21.15.44:0/2267972216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab135aa670 msgr2=0x55ab135aaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.995+0000 7fd17cc00700 1 --2- 172.21.15.44:0/2267972216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab135aa670 0x55ab135aaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.995+0000 7fd17cc00700 1 -- 172.21.15.44:0/2267972216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1342afd0 msgr2=0x55ab135b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.995+0000 7fd17cc00700 1 --2- 172.21.15.44:0/2267972216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1342afd0 0x55ab135b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.995+0000 7fd17cc00700 1 -- 172.21.15.44:0/2267972216 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab134905b0 con 0x55ab13516b80 2024-05-27T09:36:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.995+0000 7fd177e00700 1 --2- 172.21.15.44:0/2267972216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1342afd0 0x55ab135b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.995+0000 7fd17d600700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1600b2ba0 0x7fd1600b50b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd170009ec0 tx=0x7fd1700134a0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.995+0000 7fd17cc00700 1 --2- 172.21.15.44:0/2267972216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab13516b80 0x55ab135b05c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0e00a0 tx=0x7fd16c0a96b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e62e8b6c5ad04f76 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.995+0000 7fd16a000700 1 -- 172.21.15.44:0/2267972216 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c0a9740 con 0x55ab13516b80 2024-05-27T09:36:45.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.996+0000 7fd16a000700 1 -- 172.21.15.44:0/2267972216 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd16c0aa330 con 0x55ab13516b80 2024-05-27T09:36:45.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.996+0000 7fd16a000700 1 -- 172.21.15.44:0/2267972216 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c0aa0a0 con 0x55ab13516b80 2024-05-27T09:36:45.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.996+0000 7fd18e102b80 1 -- 172.21.15.44:0/2267972216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab13516b80 msgr2=0x55ab135b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:45.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.996+0000 7fd18e102b80 1 --2- 172.21.15.44:0/2267972216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab13516b80 0x55ab135b05c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0e00a0 tx=0x7fd16c0a96b0 comp rx=0 tx=0).stop 2024-05-27T09:36:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.997+0000 7fd18e102b80 1 -- 172.21.15.44:0/2267972216 shutdown_connections 2024-05-27T09:36:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.997+0000 7fd18e102b80 1 --2- 172.21.15.44:0/2267972216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab135aa670 0x55ab135aaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.997+0000 7fd18e102b80 1 --2- 172.21.15.44:0/2267972216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab13516b80 0x55ab135b05c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.997+0000 7fd18e102b80 1 --2- 172.21.15.44:0/2267972216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1342afd0 0x55ab135b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:45.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.997+0000 7fd18e102b80 1 -- 172.21.15.44:0/2267972216 >> 172.21.15.44:0/2267972216 conn(0x55ab134c4520 msgr2=0x55ab135a56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.997+0000 7fd18e102b80 1 -- 172.21.15.44:0/2267972216 shutdown_connections 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.997+0000 7fd18e102b80 1 -- 172.21.15.44:0/2267972216 wait complete. 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.997+0000 7fd18e102b80 1 -- start start 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.998+0000 7fd18e102b80 1 Processor -- start 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.998+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.998+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab13516b80 0x55ab135b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.998+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab135aa670 0x55ab135b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.998+0000 7fd18e102b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ab133ce130 con 0x55ab13516b80 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.998+0000 7fd18e102b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ab134688f0 con 0x55ab135aa670 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.998+0000 7fd18e102b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ab135ab460 con 0x55ab1342afd0 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.998+0000 7fd18e102b80 10 client.0 _ll_register_callbacks cb 0x55ab13471e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:46.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.999+0000 7fd17cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:46.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.999+0000 7fd17cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43920/0 (socket says 172.21.15.44:43920) 2024-05-27T09:36:46.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.999+0000 7fd17cc00700 1 -- 172.21.15.44:0/3673971374 learned_addr learned my addr 172.21.15.44:0/3673971374 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:46.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.999+0000 7fd177e00700 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab13516b80 0x55ab135b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:46.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.999+0000 7fd17d600700 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab135aa670 0x55ab135b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:46.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.999+0000 7fd17cc00700 1 -- 172.21.15.44:0/3673971374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab135aa670 msgr2=0x55ab135b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.999+0000 7fd17cc00700 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab135aa670 0x55ab135b4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.999+0000 7fd17cc00700 1 -- 172.21.15.44:0/3673971374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab13516b80 msgr2=0x55ab135b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.999+0000 7fd17cc00700 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab13516b80 0x55ab135b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.999+0000 7fd17cc00700 1 -- 172.21.15.44:0/3673971374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab13417a60 con 0x55ab1342afd0 2024-05-27T09:36:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:45.999+0000 7fd17d600700 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab135aa670 0x55ab135b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.000+0000 7fd17cc00700 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b43c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd16c000c00 tx=0x7fd16c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a489f02225c385cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.000+0000 7fd15ea00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:36:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.000+0000 7fd15ea00700 1 -- 172.21.15.44:0/3673971374 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c0c9060 con 0x55ab1342afd0 2024-05-27T09:36:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.000+0000 7fd15ea00700 1 -- 172.21.15.44:0/3673971374 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fd16c0c42c0 con 0x55ab1342afd0 2024-05-27T09:36:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.000+0000 7fd18e102b80 1 -- 172.21.15.44:0/3673971374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd16c0aa330 con 0x55ab1342afd0 2024-05-27T09:36:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.000+0000 7fd15ea00700 1 -- 172.21.15.44:0/3673971374 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd16c0a9740 con 0x55ab1342afd0 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.000+0000 7fd18e102b80 10 client.25582 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.000+0000 7fd18e102b80 1 -- 172.21.15.44:0/3673971374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ab134981f0 con 0x55ab1342afd0 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.000+0000 7fd15ea00700 1 -- 172.21.15.44:0/3673971374 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd16c0aa330 con 0x55ab1342afd0 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.001+0000 7fd15ea00700 1 -- 172.21.15.44:0/3673971374 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd16c085020 con 0x55ab1342afd0 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.001+0000 7fd18e102b80 10 client.25582 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.001+0000 7fd18e102b80 10 client.25582 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.001+0000 7fd18e102b80 20 client.25582 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.001+0000 7fd18e102b80 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab135c3fa0 0x55ab135c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.001+0000 7fd18e102b80 4 client.25582 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.001+0000 7fd18e102b80 1 -- 172.21.15.44:0/3673971374 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab134d9f20 con 0x55ab135c3fa0 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.001+0000 7fd177e00700 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab135c3fa0 0x55ab135c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.002+0000 7fd177e00700 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab135c3fa0 0x55ab135c63c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dab6bf583cc109eb server_cookie=b5fb41bd2d47c123 in_seq=0 out_seq=0 2024-05-27T09:36:46.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.002+0000 7fd15ea00700 10 client.25582 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:46.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd15ea00700 1 -- 172.21.15.44:0/3673971374 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ab134d9f20 con 0x55ab135c3fa0 2024-05-27T09:36:46.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd15ea00700 10 client.25582 handle_command_reply: tid=0 2024-05-27T09:36:46.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd18e102b80 1 client.25582 shutdown 2024-05-27T09:36:46.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd18e102b80 1 -- 172.21.15.44:0/3673971374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 msgr2=0x55ab135b43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:46.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd18e102b80 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b43c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd16c000c00 tx=0x7fd16c004750 comp rx=0 tx=0).stop 2024-05-27T09:36:46.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd18e102b80 1 -- 172.21.15.44:0/3673971374 shutdown_connections 2024-05-27T09:36:46.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd18e102b80 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab135c3fa0 0x55ab135c63c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd18e102b80 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab135aa670 0x55ab135b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd18e102b80 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b43c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd15ea00700 0 client.25582 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:46.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd18e102b80 1 --2- 172.21.15.44:0/3673971374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab13516b80 0x55ab135b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.003+0000 7fd18e102b80 1 -- 172.21.15.44:0/3673971374 >> 172.21.15.44:0/3673971374 conn(0x55ab134c4520 msgr2=0x55ab135a4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:46.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.004+0000 7fd18e102b80 1 -- 172.21.15.44:0/3673971374 shutdown_connections 2024-05-27T09:36:46.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.004+0000 7fd18e102b80 1 -- 172.21.15.44:0/3673971374 wait complete. 2024-05-27T09:36:46.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.004+0000 7fd18e102b80 20 client.25582 trim_cache size 0 max 16384 2024-05-27T09:36:46.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.026+0000 7fd18e102b80 1 Processor -- start 2024-05-27T09:36:46.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.026+0000 7fd18e102b80 1 -- start start 2024-05-27T09:36:46.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.027+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab134c4520 0x55ab135b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:46.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.027+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:46.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.027+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab13516b80 0x55ab135b1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:46.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.027+0000 7fd18e102b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd1781b38d0 con 0x55ab13516b80 2024-05-27T09:36:46.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.027+0000 7fd18e102b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd1781b1160 con 0x55ab134c4520 2024-05-27T09:36:46.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.027+0000 7fd18e102b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd1781aefb0 con 0x55ab1342afd0 2024-05-27T09:36:46.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.027+0000 7fd177e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:46.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.027+0000 7fd17d600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab13516b80 0x55ab135b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:46.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.027+0000 7fd177e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43924/0 (socket says 172.21.15.44:43924) 2024-05-27T09:36:46.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.027+0000 7fd177e00700 1 -- 172.21.15.44:0/1068076054 learned_addr learned my addr 172.21.15.44:0/1068076054 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:46.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.027+0000 7fd17cc00700 1 --2- 172.21.15.44:0/1068076054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab134c4520 0x55ab135b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:46.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.028+0000 7fd177e00700 1 -- 172.21.15.44:0/1068076054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab134c4520 msgr2=0x55ab135b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:46.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.028+0000 7fd177e00700 1 --2- 172.21.15.44:0/1068076054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab134c4520 0x55ab135b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.028+0000 7fd177e00700 1 -- 172.21.15.44:0/1068076054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab13516b80 msgr2=0x55ab135b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:46.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.028+0000 7fd177e00700 1 --2- 172.21.15.44:0/1068076054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab13516b80 0x55ab135b1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.028+0000 7fd177e00700 1 -- 172.21.15.44:0/1068076054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab135b0000 con 0x55ab1342afd0 2024-05-27T09:36:46.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.028+0000 7fd177e00700 1 --2- 172.21.15.44:0/1068076054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b4df0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd1640098f0 tx=0x7fd164005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a29ccd973eaed0c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:46.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.029+0000 7fd169600700 1 -- 172.21.15.44:0/1068076054 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16400f4e0 con 0x55ab1342afd0 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.029+0000 7fd169600700 1 -- 172.21.15.44:0/1068076054 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd164005040 con 0x55ab1342afd0 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.029+0000 7fd169600700 1 -- 172.21.15.44:0/1068076054 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164003a00 con 0x55ab1342afd0 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.029+0000 7fd18e102b80 1 -- 172.21.15.44:0/1068076054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 msgr2=0x55ab135b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.029+0000 7fd18e102b80 1 --2- 172.21.15.44:0/1068076054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b4df0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd1640098f0 tx=0x7fd164005dc0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.030+0000 7fd18e102b80 1 -- 172.21.15.44:0/1068076054 shutdown_connections 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.030+0000 7fd18e102b80 1 --2- 172.21.15.44:0/1068076054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab13516b80 0x55ab135b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.030+0000 7fd18e102b80 1 --2- 172.21.15.44:0/1068076054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab134c4520 0x55ab135b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.030+0000 7fd18e102b80 1 --2- 172.21.15.44:0/1068076054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab1342afd0 0x55ab135b4df0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.030+0000 7fd18e102b80 1 -- 172.21.15.44:0/1068076054 >> 172.21.15.44:0/1068076054 conn(0x55ab135aad50 msgr2=0x55ab135b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.030+0000 7fd18e102b80 1 -- 172.21.15.44:0/1068076054 shutdown_connections 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.030+0000 7fd18e102b80 1 -- 172.21.15.44:0/1068076054 wait complete. 2024-05-27T09:36:46.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.030+0000 7fd18e102b80 1 -- start start 2024-05-27T09:36:46.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.031+0000 7fd18e102b80 1 Processor -- start 2024-05-27T09:36:46.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.031+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab1342afd0 0x55ab135c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.031+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab134c4520 0x55ab135c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.031+0000 7fd18e102b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab13516b80 0x55ab135c8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.031+0000 7fd18e102b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd1781af620 con 0x55ab134c4520 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.031+0000 7fd18e102b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd1781b0a90 con 0x55ab1342afd0 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.031+0000 7fd18e102b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd1781afcf0 con 0x55ab13516b80 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.031+0000 7fd18e102b80 10 client.0 _ll_register_callbacks cb 0x55ab13471e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.031+0000 7fd177e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab134c4520 0x55ab135c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd177e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab134c4520 0x55ab135c7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59208/0 (socket says 172.21.15.44:59208) 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd17cc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab1342afd0 0x55ab135c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd177e00700 1 -- 172.21.15.44:0/2385901167 learned_addr learned my addr 172.21.15.44:0/2385901167 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd17d600700 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab13516b80 0x55ab135c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:46.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd177e00700 1 -- 172.21.15.44:0/2385901167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab13516b80 msgr2=0x55ab135c8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:46.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd177e00700 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab13516b80 0x55ab135c8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd177e00700 1 -- 172.21.15.44:0/2385901167 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab1342afd0 msgr2=0x55ab135c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:46.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd177e00700 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab1342afd0 0x55ab135c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd177e00700 1 -- 172.21.15.44:0/2385901167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab135c8640 con 0x55ab134c4520 2024-05-27T09:36:46.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd17cc00700 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab1342afd0 0x55ab135c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:46.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd177e00700 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab134c4520 0x55ab135c7bc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd164009890 tx=0x7fd1640053d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7279a7c15a3e93 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:46.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.032+0000 7fd15ea00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:36:46.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.033+0000 7fd15ea00700 1 -- 172.21.15.44:0/2385901167 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16400f6e0 con 0x55ab134c4520 2024-05-27T09:36:46.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.033+0000 7fd18e102b80 1 -- 172.21.15.44:0/2385901167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd164005040 con 0x55ab134c4520 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.033+0000 7fd15ea00700 1 -- 172.21.15.44:0/2385901167 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fd16400f880 con 0x55ab134c4520 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.033+0000 7fd15ea00700 1 -- 172.21.15.44:0/2385901167 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd164003680 con 0x55ab134c4520 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.033+0000 7fd18e102b80 10 client.15450 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.033+0000 7fd18e102b80 1 -- 172.21.15.44:0/2385901167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ab1362f450 con 0x55ab134c4520 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.033+0000 7fd15ea00700 1 -- 172.21.15.44:0/2385901167 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd164005040 con 0x55ab134c4520 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.034+0000 7fd15ea00700 1 -- 172.21.15.44:0/2385901167 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd16402d020 con 0x55ab134c4520 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.034+0000 7fd18e102b80 10 client.15450 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.034+0000 7fd18e102b80 10 client.15450 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.034+0000 7fd18e102b80 20 client.15450 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.034+0000 7fd18e102b80 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab135aa670 0x55ab13631a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.034+0000 7fd18e102b80 4 client.15450 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.4, "await_for": 45.0}] 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.034+0000 7fd18e102b80 1 -- 172.21.15.44:0/2385901167 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.4, "await_for": 45.0}) -- 0x55ab1359d8a0 con 0x55ab135aa670 2024-05-27T09:36:46.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.034+0000 7fd17cc00700 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab135aa670 0x55ab13631a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:46.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.035+0000 7fd17cc00700 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab135aa670 0x55ab13631a80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b081be656a15e49 server_cookie=acfcd3486135eb17 in_seq=0 out_seq=0 2024-05-27T09:36:46.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.035+0000 7fd15ea00700 10 client.15450 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:46.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.049+0000 7fd15ea00700 1 -- 172.21.15.44:0/2385901167 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55ab1359d8a0 con 0x55ab135aa670 2024-05-27T09:36:46.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.049+0000 7fd15ea00700 10 client.15450 handle_command_reply: tid=0 2024-05-27T09:36:46.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.049+0000 7fd18e102b80 1 client.15450 shutdown 2024-05-27T09:36:46.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd18e102b80 1 -- 172.21.15.44:0/2385901167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab134c4520 msgr2=0x55ab135c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:46.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd18e102b80 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab134c4520 0x55ab135c7bc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd164009890 tx=0x7fd1640053d0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd18e102b80 1 -- 172.21.15.44:0/2385901167 shutdown_connections 2024-05-27T09:36:46.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd18e102b80 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab135aa670 0x55ab13631a80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd18e102b80 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab13516b80 0x55ab135c8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd18e102b80 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab1342afd0 0x55ab135c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd18e102b80 1 --2- 172.21.15.44:0/2385901167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab134c4520 0x55ab135c7bc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd15ea00700 0 client.15450 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:46.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd18e102b80 1 -- 172.21.15.44:0/2385901167 >> 172.21.15.44:0/2385901167 conn(0x55ab135aad50 msgr2=0x55ab135b4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:46.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd18e102b80 1 -- 172.21.15.44:0/2385901167 shutdown_connections 2024-05-27T09:36:46.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd18e102b80 1 -- 172.21.15.44:0/2385901167 wait complete. 2024-05-27T09:36:46.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.050+0000 7fd18e102b80 20 client.15450 trim_cache size 0 max 16384 2024-05-27T09:36:46.052 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:36:46.052 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 26, 2024-05-27T09:36:46.052 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:36:46.052 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 2, 2024-05-27T09:36:46.052 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:36:46.052 INFO:teuthology.orchestra.run.smithi044.stdout: "4295cdb8": { 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 2, 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 149.4, 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:36:46.053 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:36:46.054 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:36:46.054 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:36:46.054 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:36:46.054 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:36:46.054 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:36:46.054 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:36:46.054 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:36:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:36:46.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 -- 172.21.15.44:0/3840065964 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1600b2ba0 msgr2=0x7fd1600b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:46.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1600b2ba0 0x7fd1600b50b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd170009ec0 tx=0x7fd1700134a0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 -- 172.21.15.44:0/3840065964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd178106570 msgr2=0x7fd17819d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:46.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd178106570 0x7fd17819d060 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00ec70 tx=0x7fd16c00ef80 comp rx=0 tx=0).stop 2024-05-27T09:36:46.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 -- 172.21.15.44:0/3840065964 shutdown_connections 2024-05-27T09:36:46.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1600b2ba0 0x7fd1600b50b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17810c5b0 0x7fd1781a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd178106570 0x7fd17819d060 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 --2- 172.21.15.44:0/3840065964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd178106ec0 0x7fd17819d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:46.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 -- 172.21.15.44:0/3840065964 >> 172.21.15.44:0/3840065964 conn(0x7fd178102070 msgr2=0x7fd178104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:46.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 -- 172.21.15.44:0/3840065964 shutdown_connections 2024-05-27T09:36:46.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:36:46.051+0000 7fd17f600700 1 -- 172.21.15.44:0/3840065964 wait complete. 2024-05-27T09:36:46.431 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:46 smithi044 ceph-mon[31618]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-05-27T09:36:46.431 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:46 smithi044 ceph-mon[31618]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-05-27T09:36:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:46 smithi050 ceph-mon[36371]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-05-27T09:36:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:46 smithi050 ceph-mon[36371]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-05-27T09:36:46.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:46 smithi189 ceph-mon[36671]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-05-27T09:36:46.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:46 smithi189 ceph-mon[36671]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-05-27T09:36:46.919 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '4295cdb8', quiesce took 9.6 seconds. Will release after: 19.799999999999997 2024-05-27T09:36:48.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:48 smithi044 ceph-mon[31618]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 3.1 KiB/s wr, 9 op/s 2024-05-27T09:36:48.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:48 smithi050 ceph-mon[36371]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 3.1 KiB/s wr, 9 op/s 2024-05-27T09:36:48.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:48 smithi189 ceph-mon[36671]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 3.1 KiB/s wr, 9 op/s 2024-05-27T09:36:50.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:50 smithi044 ceph-mon[31618]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-05-27T09:36:50.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:50 smithi050 ceph-mon[36371]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-05-27T09:36:50.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:50 smithi189 ceph-mon[36671]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-05-27T09:36:52.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:52 smithi044 ceph-mon[31618]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-05-27T09:36:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:52 smithi050 ceph-mon[36371]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-05-27T09:36:52.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:52 smithi189 ceph-mon[36671]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-05-27T09:36:54.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:54 smithi044 ceph-mon[31618]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-05-27T09:36:54.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:54 smithi050 ceph-mon[36371]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-05-27T09:36:54.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:54 smithi189 ceph-mon[36671]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-05-27T09:36:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.976+0000 7eff6ac00640 1 -- 172.21.15.44:0/591124237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff64104c50 msgr2=0x7eff64105060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.976+0000 7eff6ac00640 1 --2- 172.21.15.44:0/591124237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff64104c50 0x7eff64105060 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7eff58009860 tx=0x7eff5802a3d0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.976+0000 7eff6ac00640 1 -- 172.21.15.44:0/591124237 shutdown_connections 2024-05-27T09:36:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.976+0000 7eff6ac00640 1 --2- 172.21.15.44:0/591124237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff6410e010 0x7eff64110500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.976+0000 7eff6ac00640 1 --2- 172.21.15.44:0/591124237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff6410b6b0 0x7eff6410dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.976+0000 7eff6ac00640 1 --2- 172.21.15.44:0/591124237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff64104c50 0x7eff64105060 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.976+0000 7eff6ac00640 1 -- 172.21.15.44:0/591124237 >> 172.21.15.44:0/591124237 conn(0x7eff640fe210 msgr2=0x7eff64100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.976+0000 7eff6ac00640 1 -- 172.21.15.44:0/591124237 shutdown_connections 2024-05-27T09:36:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.977+0000 7eff6ac00640 1 -- 172.21.15.44:0/591124237 wait complete. 2024-05-27T09:36:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.978+0000 7eff6ac00640 1 Processor -- start 2024-05-27T09:36:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.978+0000 7eff6ac00640 1 -- start start 2024-05-27T09:36:54.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6ac00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff64104c50 0x7eff64102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6ac00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff6410b6b0 0x7eff64102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff6410e010 0x7eff64102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6ac00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7eff64113cc0 con 0x7eff64104c50 2024-05-27T09:36:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6ac00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7eff64113b40 con 0x7eff6410b6b0 2024-05-27T09:36:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7eff64113e40 con 0x7eff6410e010 2024-05-27T09:36:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff6410e010 0x7eff64102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff68e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff6410b6b0 0x7eff64102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff6410e010 0x7eff64102f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47202/0 (socket says 172.21.15.44:47202) 2024-05-27T09:36:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6a200640 1 -- 172.21.15.44:0/47343969 learned_addr learned my addr 172.21.15.44:0/47343969 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff69800640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff64104c50 0x7eff64102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6a200640 1 -- 172.21.15.44:0/47343969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff6410b6b0 msgr2=0x7eff64102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6a200640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff6410b6b0 0x7eff64102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6a200640 1 -- 172.21.15.44:0/47343969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff64104c50 msgr2=0x7eff64102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6a200640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff64104c50 0x7eff64102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.979+0000 7eff6a200640 1 -- 172.21.15.44:0/47343969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff641a8a20 con 0x7eff6410e010 2024-05-27T09:36:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.980+0000 7eff6a200640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff6410e010 0x7eff64102f90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009fd0 tx=0x7eff5c00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.980+0000 7eff62000640 1 -- 172.21.15.44:0/47343969 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c004b20 con 0x7eff6410e010 2024-05-27T09:36:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.980+0000 7eff62000640 1 -- 172.21.15.44:0/47343969 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff5c003e20 con 0x7eff6410e010 2024-05-27T09:36:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.980+0000 7eff6ac00640 1 -- 172.21.15.44:0/47343969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff641a8d10 con 0x7eff6410e010 2024-05-27T09:36:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.980+0000 7eff62000640 1 -- 172.21.15.44:0/47343969 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c004820 con 0x7eff6410e010 2024-05-27T09:36:54.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.980+0000 7eff6ac00640 1 -- 172.21.15.44:0/47343969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff641a9200 con 0x7eff6410e010 2024-05-27T09:36:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.984+0000 7eff731e8740 1 Processor -- start 2024-05-27T09:36:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.984+0000 7eff62000640 1 -- 172.21.15.44:0/47343969 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7eff5c01d020 con 0x7eff6410e010 2024-05-27T09:36:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.985+0000 7eff731e8740 1 -- start start 2024-05-27T09:36:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.985+0000 7eff731e8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283bca1b0 0x560283f6ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.985+0000 7eff731e8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f6cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.985+0000 7eff731e8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283ba94d0 0x560283f6d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.985+0000 7eff731e8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560283eddb90 con 0x560283ba12f0 2024-05-27T09:36:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.985+0000 7eff731e8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560283ecd730 con 0x560283bca1b0 2024-05-27T09:36:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.985+0000 7eff731e8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560283ed1770 con 0x560283ba94d0 2024-05-27T09:36:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.985+0000 7eff69800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f6cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.985+0000 7eff69800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f6cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49242/0 (socket says 172.21.15.44:49242) 2024-05-27T09:36:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.985+0000 7eff69800640 1 -- 172.21.15.44:0/3084411397 learned_addr learned my addr 172.21.15.44:0/3084411397 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.985+0000 7eff68e00640 1 --2- 172.21.15.44:0/3084411397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283bca1b0 0x560283f6ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.986+0000 7eff68e00640 1 --2- 172.21.15.44:0/3084411397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283ba94d0 0x560283f6d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.986+0000 7eff69800640 1 -- 172.21.15.44:0/3084411397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283ba94d0 msgr2=0x560283f6d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:54.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.986+0000 7eff69800640 1 --2- 172.21.15.44:0/3084411397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283ba94d0 0x560283f6d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.986+0000 7eff69800640 1 -- 172.21.15.44:0/3084411397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283bca1b0 msgr2=0x560283f6ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:54.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.986+0000 7eff69800640 1 --2- 172.21.15.44:0/3084411397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283bca1b0 0x560283f6ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.986+0000 7eff69800640 1 -- 172.21.15.44:0/3084411397 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560283e668b0 con 0x560283ba12f0 2024-05-27T09:36:54.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.986+0000 7eff68e00640 1 --2- 172.21.15.44:0/3084411397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283ba94d0 0x560283f6d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:36:54.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.986+0000 7eff69800640 1 --2- 172.21.15.44:0/3084411397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f6cfa0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7eff58034000 tx=0x7eff580303a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cd2ee596526ba9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:54.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.987+0000 7eff62000640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff4c0b29d0 0x7eff4c0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:54.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.987+0000 7eff56a00640 1 -- 172.21.15.44:0/3084411397 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58030790 con 0x560283ba12f0 2024-05-27T09:36:54.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.987+0000 7eff62000640 1 -- 172.21.15.44:0/47343969 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7eff5c0e2540 con 0x7eff6410e010 2024-05-27T09:36:54.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.987+0000 7eff56a00640 1 -- 172.21.15.44:0/3084411397 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff58041030 con 0x560283ba12f0 2024-05-27T09:36:54.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.987+0000 7eff56a00640 1 -- 172.21.15.44:0/3084411397 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58005340 con 0x560283ba12f0 2024-05-27T09:36:54.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.987+0000 7eff68e00640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff4c0b29d0 0x7eff4c0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:54.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.988+0000 7eff731e8740 1 -- 172.21.15.44:0/3084411397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 msgr2=0x560283f6cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.988+0000 7eff68e00640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff4c0b29d0 0x7eff4c0b4ee0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x560283f69bd0 tx=0x7eff50004730 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.988+0000 7eff731e8740 1 --2- 172.21.15.44:0/3084411397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f6cfa0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7eff58034000 tx=0x7eff580303a0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.988+0000 7eff731e8740 1 -- 172.21.15.44:0/3084411397 shutdown_connections 2024-05-27T09:36:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.988+0000 7eff731e8740 1 --2- 172.21.15.44:0/3084411397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283ba94d0 0x560283f6d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.988+0000 7eff731e8740 1 --2- 172.21.15.44:0/3084411397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f6cfa0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.988+0000 7eff731e8740 1 --2- 172.21.15.44:0/3084411397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283bca1b0 0x560283f6ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.988+0000 7eff731e8740 1 -- 172.21.15.44:0/3084411397 >> 172.21.15.44:0/3084411397 conn(0x560283bbe300 msgr2=0x560283f69210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.988+0000 7eff731e8740 1 -- 172.21.15.44:0/3084411397 shutdown_connections 2024-05-27T09:36:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.988+0000 7eff731e8740 1 -- 172.21.15.44:0/3084411397 wait complete. 2024-05-27T09:36:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.989+0000 7eff731e8740 1 -- start start 2024-05-27T09:36:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.989+0000 7eff731e8740 1 Processor -- start 2024-05-27T09:36:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff731e8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba12f0 0x560283f76eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff731e8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba94d0 0x560283f773f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff731e8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283bca1b0 0x560283f82e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:54.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff731e8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560283bb3790 con 0x560283ba94d0 2024-05-27T09:36:54.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff731e8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560283e75bc0 con 0x560283ba12f0 2024-05-27T09:36:54.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff731e8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560283f79b30 con 0x560283bca1b0 2024-05-27T09:36:54.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff731e8740 10 client.0 _ll_register_callbacks cb 0x560283e53400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:54.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff68e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba94d0 0x560283f773f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:54.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff68e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba94d0 0x560283f773f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49258/0 (socket says 172.21.15.44:49258) 2024-05-27T09:36:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff68e00640 1 -- 172.21.15.44:0/1934897583 learned_addr learned my addr 172.21.15.44:0/1934897583 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff69800640 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba12f0 0x560283f76eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.990+0000 7eff6a200640 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283bca1b0 0x560283f82e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.991+0000 7eff68e00640 1 -- 172.21.15.44:0/1934897583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283bca1b0 msgr2=0x560283f82e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.991+0000 7eff68e00640 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283bca1b0 0x560283f82e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.991+0000 7eff68e00640 1 -- 172.21.15.44:0/1934897583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba12f0 msgr2=0x560283f76eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.991+0000 7eff68e00640 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba12f0 0x560283f76eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.991+0000 7eff68e00640 1 -- 172.21.15.44:0/1934897583 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560283be2210 con 0x560283ba94d0 2024-05-27T09:36:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.991+0000 7eff68e00640 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba94d0 0x560283f773f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7eff50004590 tx=0x7eff50005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1ecd9b4e1eb308 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.991+0000 7eff4b400640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:36:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.992+0000 7eff4b400640 1 -- 172.21.15.44:0/1934897583 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff50022640 con 0x560283ba94d0 2024-05-27T09:36:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.992+0000 7eff4b400640 1 -- 172.21.15.44:0/1934897583 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7eff500227e0 con 0x560283ba94d0 2024-05-27T09:36:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.992+0000 7eff731e8740 1 -- 172.21.15.44:0/1934897583 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff58041030 con 0x560283ba94d0 2024-05-27T09:36:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.992+0000 7eff4b400640 1 -- 172.21.15.44:0/1934897583 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7eff500229e0 con 0x560283ba94d0 2024-05-27T09:36:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.992+0000 7eff731e8740 10 client.15462 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.992+0000 7eff731e8740 1 -- 172.21.15.44:0/1934897583 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560283cae620 con 0x560283ba94d0 2024-05-27T09:36:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.992+0000 7eff4b400640 1 -- 172.21.15.44:0/1934897583 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff58041030 con 0x560283ba94d0 2024-05-27T09:36:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.993+0000 7eff4b400640 1 -- 172.21.15.44:0/1934897583 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7eff5002bc50 con 0x560283ba94d0 2024-05-27T09:36:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.993+0000 7eff731e8740 10 client.15462 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.993+0000 7eff731e8740 10 client.15462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.993+0000 7eff731e8740 20 client.15462 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.993+0000 7eff731e8740 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560283b88d80 0x560283f88f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.993+0000 7eff731e8740 4 client.15462 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:36:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.993+0000 7eff731e8740 1 -- 172.21.15.44:0/1934897583 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560283edfae0 con 0x560283b88d80 2024-05-27T09:36:54.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.993+0000 7eff69800640 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560283b88d80 0x560283f88f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:54.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.994+0000 7eff69800640 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560283b88d80 0x560283f88f30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26d1828aee5d884c server_cookie=beabdb5c0ef08f03 in_seq=0 out_seq=0 2024-05-27T09:36:54.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.994+0000 7eff4b400640 10 client.15462 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:54.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff4b400640 1 -- 172.21.15.44:0/1934897583 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560283edfae0 con 0x560283b88d80 2024-05-27T09:36:54.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff4b400640 10 client.15462 handle_command_reply: tid=0 2024-05-27T09:36:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff731e8740 1 client.15462 shutdown 2024-05-27T09:36:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff731e8740 1 -- 172.21.15.44:0/1934897583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba94d0 msgr2=0x560283f773f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff731e8740 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba94d0 0x560283f773f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7eff50004590 tx=0x7eff50005f90 comp rx=0 tx=0).stop 2024-05-27T09:36:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff731e8740 1 -- 172.21.15.44:0/1934897583 shutdown_connections 2024-05-27T09:36:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff731e8740 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560283b88d80 0x560283f88f30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff731e8740 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283bca1b0 0x560283f82e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff731e8740 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba94d0 0x560283f773f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff731e8740 1 --2- 172.21.15.44:0/1934897583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba12f0 0x560283f76eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff4b400640 0 client.15462 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff731e8740 1 -- 172.21.15.44:0/1934897583 >> 172.21.15.44:0/1934897583 conn(0x560283bbe300 msgr2=0x560283f6e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:55.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff731e8740 1 -- 172.21.15.44:0/1934897583 shutdown_connections 2024-05-27T09:36:55.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.995+0000 7eff731e8740 1 -- 172.21.15.44:0/1934897583 wait complete. 2024-05-27T09:36:55.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:54.996+0000 7eff731e8740 20 client.15462 trim_cache size 0 max 16384 2024-05-27T09:36:55.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.013+0000 7eff731e8740 1 Processor -- start 2024-05-27T09:36:55.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.013+0000 7eff731e8740 1 -- start start 2024-05-27T09:36:55.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.013+0000 7eff731e8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba12f0 0x560283f724e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:55.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.013+0000 7eff731e8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283ba94d0 0x560283f6d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:55.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.013+0000 7eff731e8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283bca1b0 0x560283f707c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:55.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.013+0000 7eff731e8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7eff641b4360 con 0x560283bca1b0 2024-05-27T09:36:55.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.013+0000 7eff731e8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7eff641b1c30 con 0x560283ba12f0 2024-05-27T09:36:55.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.013+0000 7eff731e8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7eff641afa80 con 0x560283ba94d0 2024-05-27T09:36:55.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.013+0000 7eff6a200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283bca1b0 0x560283f707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:55.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.014+0000 7eff69800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba12f0 0x560283f724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:55.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.013+0000 7eff6a200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283bca1b0 0x560283f707c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49260/0 (socket says 172.21.15.44:49260) 2024-05-27T09:36:55.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.014+0000 7eff6a200640 1 -- 172.21.15.44:0/2899725866 learned_addr learned my addr 172.21.15.44:0/2899725866 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.014+0000 7eff68e00640 1 --2- 172.21.15.44:0/2899725866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283ba94d0 0x560283f6d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.014+0000 7eff69800640 1 -- 172.21.15.44:0/2899725866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283ba94d0 msgr2=0x560283f6d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.014+0000 7eff69800640 1 --2- 172.21.15.44:0/2899725866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283ba94d0 0x560283f6d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.014+0000 7eff69800640 1 -- 172.21.15.44:0/2899725866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283bca1b0 msgr2=0x560283f707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.014+0000 7eff69800640 1 --2- 172.21.15.44:0/2899725866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283bca1b0 0x560283f707c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.014+0000 7eff69800640 1 -- 172.21.15.44:0/2899725866 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560283f69e00 con 0x560283ba12f0 2024-05-27T09:36:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.014+0000 7eff69800640 1 --2- 172.21.15.44:0/2899725866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba12f0 0x560283f724e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x560283f6de10 tx=0x7eff5803f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b0cac772dec12fa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.015+0000 7eff56000640 1 -- 172.21.15.44:0/2899725866 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58030510 con 0x560283ba12f0 2024-05-27T09:36:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.015+0000 7eff56000640 1 -- 172.21.15.44:0/2899725866 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff58041070 con 0x560283ba12f0 2024-05-27T09:36:55.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.015+0000 7eff56000640 1 -- 172.21.15.44:0/2899725866 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58005320 con 0x560283ba12f0 2024-05-27T09:36:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.016+0000 7eff731e8740 1 -- 172.21.15.44:0/2899725866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba12f0 msgr2=0x560283f724e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.016+0000 7eff731e8740 1 --2- 172.21.15.44:0/2899725866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba12f0 0x560283f724e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x560283f6de10 tx=0x7eff5803f040 comp rx=0 tx=0).stop 2024-05-27T09:36:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.016+0000 7eff731e8740 1 -- 172.21.15.44:0/2899725866 shutdown_connections 2024-05-27T09:36:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.016+0000 7eff731e8740 1 --2- 172.21.15.44:0/2899725866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283bca1b0 0x560283f707c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.016+0000 7eff731e8740 1 --2- 172.21.15.44:0/2899725866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283ba94d0 0x560283f6d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.016+0000 7eff731e8740 1 --2- 172.21.15.44:0/2899725866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba12f0 0x560283f724e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.016+0000 7eff731e8740 1 -- 172.21.15.44:0/2899725866 >> 172.21.15.44:0/2899725866 conn(0x560283bbe300 msgr2=0x560283f6c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:55.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.016+0000 7eff731e8740 1 -- 172.21.15.44:0/2899725866 shutdown_connections 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.016+0000 7eff731e8740 1 -- 172.21.15.44:0/2899725866 wait complete. 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.016+0000 7eff731e8740 1 -- start start 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff731e8740 1 Processor -- start 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff731e8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f70d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff731e8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba94d0 0x560283f71250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff731e8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283bca1b0 0x560283f717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff731e8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7eff641b00f0 con 0x560283ba12f0 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff731e8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7eff641b1560 con 0x560283ba94d0 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff731e8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7eff641b07c0 con 0x560283bca1b0 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff731e8740 10 client.0 _ll_register_callbacks cb 0x7eff50009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff69800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f70d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:55.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff69800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f70d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49270/0 (socket says 172.21.15.44:49270) 2024-05-27T09:36:55.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff69800640 1 -- 172.21.15.44:0/3747339865 learned_addr learned my addr 172.21.15.44:0/3747339865 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:36:55.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.017+0000 7eff68e00640 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba94d0 0x560283f71250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:55.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.018+0000 7eff6a200640 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283bca1b0 0x560283f717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.018+0000 7eff69800640 1 -- 172.21.15.44:0/3747339865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283bca1b0 msgr2=0x560283f717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.018+0000 7eff69800640 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283bca1b0 0x560283f717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.018+0000 7eff69800640 1 -- 172.21.15.44:0/3747339865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba94d0 msgr2=0x560283f71250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:36:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.018+0000 7eff69800640 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba94d0 0x560283f71250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.018+0000 7eff69800640 1 -- 172.21.15.44:0/3747339865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560283f7e040 con 0x560283ba12f0 2024-05-27T09:36:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.018+0000 7eff68e00640 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba94d0 0x560283f71250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:36:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.018+0000 7eff69800640 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f70d10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7eff58005eb0 tx=0x7eff58030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d32f7568562aca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:36:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.018+0000 7eff4b400640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:36:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.019+0000 7eff4b400640 1 -- 172.21.15.44:0/3747339865 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58030980 con 0x560283ba12f0 2024-05-27T09:36:55.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.019+0000 7eff4b400640 1 -- 172.21.15.44:0/3747339865 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7eff58030b20 con 0x560283ba12f0 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.019+0000 7eff731e8740 1 -- 172.21.15.44:0/3747339865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff58041070 con 0x560283ba12f0 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.019+0000 7eff4b400640 1 -- 172.21.15.44:0/3747339865 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7eff58004040 con 0x560283ba12f0 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.019+0000 7eff731e8740 10 client.15468 fetch_fsmap learned FSMap version 26 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.019+0000 7eff731e8740 1 -- 172.21.15.44:0/3747339865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560283f7f2d0 con 0x560283ba12f0 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.019+0000 7eff4b400640 1 -- 172.21.15.44:0/3747339865 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff58041070 con 0x560283ba12f0 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.019+0000 7eff4b400640 1 -- 172.21.15.44:0/3747339865 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7eff5805c070 con 0x560283ba12f0 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.020+0000 7eff731e8740 10 client.15468 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.020+0000 7eff731e8740 10 client.15468 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.020+0000 7eff731e8740 20 client.15468 populate_metadata read hostname 'smithi044' 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.020+0000 7eff731e8740 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560283b5e1c0 0x560283f7f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.020+0000 7eff731e8740 4 client.15468 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.020+0000 7eff731e8740 1 -- 172.21.15.44:0/3747339865 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560283ed92c0 con 0x560283b5e1c0 2024-05-27T09:36:55.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.020+0000 7eff68e00640 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560283b5e1c0 0x560283f7f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:36:55.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.021+0000 7eff68e00640 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560283b5e1c0 0x560283f7f5f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c59fc2b160d2355d server_cookie=b2253e2d0d97c3ee in_seq=0 out_seq=0 2024-05-27T09:36:55.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.021+0000 7eff4b400640 10 client.15468 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:55.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.027+0000 7eff4b400640 1 -- 172.21.15.44:0/3747339865 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x560283ed92c0 con 0x560283b5e1c0 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.027+0000 7eff4b400640 10 client.15468 handle_command_reply: tid=0 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.027+0000 7eff731e8740 1 client.15468 shutdown 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff731e8740 1 -- 172.21.15.44:0/3747339865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 msgr2=0x560283f70d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff731e8740 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f70d10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7eff58005eb0 tx=0x7eff58030e90 comp rx=0 tx=0).stop 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff731e8740 1 -- 172.21.15.44:0/3747339865 shutdown_connections 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff731e8740 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560283b5e1c0 0x560283f7f5f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff731e8740 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560283bca1b0 0x560283f717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff731e8740 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560283ba94d0 0x560283f71250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff4b400640 0 client.15468 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff731e8740 1 --2- 172.21.15.44:0/3747339865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560283ba12f0 0x560283f70d10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff731e8740 1 -- 172.21.15.44:0/3747339865 >> 172.21.15.44:0/3747339865 conn(0x560283bbe300 msgr2=0x560283f6eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff731e8740 1 -- 172.21.15.44:0/3747339865 shutdown_connections 2024-05-27T09:36:55.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff731e8740 1 -- 172.21.15.44:0/3747339865 wait complete. 2024-05-27T09:36:55.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.028+0000 7eff731e8740 20 client.15468 trim_cache size 0 max 16384 2024-05-27T09:36:55.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:36:55.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:36:55.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 -- 172.21.15.44:0/47343969 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff4c0b29d0 msgr2=0x7eff4c0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff4c0b29d0 0x7eff4c0b4ee0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x560283f69bd0 tx=0x7eff50004730 comp rx=0 tx=0).stop 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 -- 172.21.15.44:0/47343969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff6410e010 msgr2=0x7eff64102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff6410e010 0x7eff64102f90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009fd0 tx=0x7eff5c00c800 comp rx=0 tx=0).stop 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 -- 172.21.15.44:0/47343969 shutdown_connections 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff6410e010 0x7eff64102f90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff6410b6b0 0x7eff64102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff4c0b29d0 0x7eff4c0b4ee0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 --2- 172.21.15.44:0/47343969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff64104c50 0x7eff64102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 -- 172.21.15.44:0/47343969 >> 172.21.15.44:0/47343969 conn(0x7eff640fe210 msgr2=0x7eff6410e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 -- 172.21.15.44:0/47343969 shutdown_connections 2024-05-27T09:36:55.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:36:55.029+0000 7eff6ac00640 1 -- 172.21.15.44:0/47343969 wait complete. 2024-05-27T09:36:56.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:56 smithi044 ceph-mon[31618]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-05-27T09:36:56.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:56 smithi050 ceph-mon[36371]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-05-27T09:36:56.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:56 smithi189 ceph-mon[36671]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-05-27T09:36:58.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:36:58 smithi044 ceph-mon[31618]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:36:58.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:36:58 smithi050 ceph-mon[36371]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:36:58.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:36:58 smithi189 ceph-mon[36671]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:37:00.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:00 smithi044 ceph-mon[31618]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-05-27T09:37:00.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:00 smithi050 ceph-mon[36371]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-05-27T09:37:00.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:00 smithi189 ceph-mon[36671]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-05-27T09:37:02.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:02 smithi044 ceph-mon[31618]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-05-27T09:37:02.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:02 smithi050 ceph-mon[36371]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-05-27T09:37:02.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:02 smithi189 ceph-mon[36671]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-05-27T09:37:04.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:04 smithi044 ceph-mon[31618]: pgmap v58: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-05-27T09:37:04.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:04 smithi050 ceph-mon[36371]: pgmap v58: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-05-27T09:37:04.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:04 smithi189 ceph-mon[36671]: pgmap v58: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-05-27T09:37:06.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:06 smithi044 ceph-mon[31618]: pgmap v59: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T09:37:06.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:06 smithi050 ceph-mon[36371]: pgmap v59: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T09:37:06.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:06 smithi189 ceph-mon[36671]: pgmap v59: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T09:37:06.720 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:37:07.181 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:37:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.302+0000 7f117ae00640 1 -- 172.21.15.44:0/2845091288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f117410d9c0 msgr2=0x7f117410fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.302+0000 7f117ae00640 1 --2- 172.21.15.44:0/2845091288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f117410d9c0 0x7f117410fde0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1160009900 tx=0x7f116002a480 comp rx=0 tx=0).stop 2024-05-27T09:37:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.302+0000 7f117ae00640 1 -- 172.21.15.44:0/2845091288 shutdown_connections 2024-05-27T09:37:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.302+0000 7f117ae00640 1 --2- 172.21.15.44:0/2845091288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1174110320 0x7f1174112740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.302+0000 7f117ae00640 1 --2- 172.21.15.44:0/2845091288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f117410d9c0 0x7f117410fde0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.302+0000 7f117ae00640 1 --2- 172.21.15.44:0/2845091288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1174075420 0x7f1174075830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.302+0000 7f117ae00640 1 -- 172.21.15.44:0/2845091288 >> 172.21.15.44:0/2845091288 conn(0x7f1174079100 msgr2=0x7f1174079540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.302+0000 7f117ae00640 1 -- 172.21.15.44:0/2845091288 shutdown_connections 2024-05-27T09:37:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.302+0000 7f117ae00640 1 -- 172.21.15.44:0/2845091288 wait complete. 2024-05-27T09:37:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.303+0000 7f117ae00640 1 Processor -- start 2024-05-27T09:37:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.304+0000 7f117ae00640 1 -- start start 2024-05-27T09:37:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.304+0000 7f117ae00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1174075420 0x7f11741a25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.304+0000 7f117ae00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f117410d9c0 0x7f11741a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.304+0000 7f117ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1174110320 0x7f117419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.304+0000 7f117ae00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1174115ee0 con 0x7f1174075420 2024-05-27T09:37:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.304+0000 7f117ae00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1174115d60 con 0x7f117410d9c0 2024-05-27T09:37:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.304+0000 7f117ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1174116060 con 0x7f1174110320 2024-05-27T09:37:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.305+0000 7f1179a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1174075420 0x7f11741a25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.305+0000 7f1179a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1174075420 0x7f11741a25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59232/0 (socket says 172.21.15.44:59232) 2024-05-27T09:37:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.305+0000 7f1179a00640 1 -- 172.21.15.44:0/545110628 learned_addr learned my addr 172.21.15.44:0/545110628 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.305+0000 7f1179000640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f117410d9c0 0x7f11741a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.305+0000 7f117a400640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1174110320 0x7f117419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.305+0000 7f1179a00640 1 -- 172.21.15.44:0/545110628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1174110320 msgr2=0x7f117419c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.305+0000 7f1179a00640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1174110320 0x7f117419c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.305+0000 7f1179a00640 1 -- 172.21.15.44:0/545110628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f117410d9c0 msgr2=0x7f11741a2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.305+0000 7f1179a00640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f117410d9c0 0x7f11741a2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.305+0000 7f1179a00640 1 -- 172.21.15.44:0/545110628 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117419ce50 con 0x7f1174075420 2024-05-27T09:37:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.306+0000 7f1179a00640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1174075420 0x7f11741a25a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f116800ea60 tx=0x7f116800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:08.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.306+0000 7f1172000640 1 -- 172.21.15.44:0/545110628 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116800cde0 con 0x7f1174075420 2024-05-27T09:37:08.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.306+0000 7f1172000640 1 -- 172.21.15.44:0/545110628 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11680035f0 con 0x7f1174075420 2024-05-27T09:37:08.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.306+0000 7f117ae00640 1 -- 172.21.15.44:0/545110628 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f117419d0e0 con 0x7f1174075420 2024-05-27T09:37:08.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.306+0000 7f1172000640 1 -- 172.21.15.44:0/545110628 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168010690 con 0x7f1174075420 2024-05-27T09:37:08.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.306+0000 7f117ae00640 1 -- 172.21.15.44:0/545110628 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11741a9340 con 0x7f1174075420 2024-05-27T09:37:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.311+0000 7f1183395740 1 Processor -- start 2024-05-27T09:37:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.311+0000 7f1183395740 1 -- start start 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.311+0000 7f1183395740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc51c1b0 0x55edcc8bea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.311+0000 7f1183395740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 0x55edcc8befa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.311+0000 7f1183395740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 0x55edcc8bf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.311+0000 7f1183395740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55edcc82fb90 con 0x55edcc51c1b0 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.311+0000 7f1183395740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55edcc81f730 con 0x55edcc4f32f0 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.311+0000 7f1183395740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55edcc823770 con 0x55edcc4fb4d0 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.312+0000 7f1172000640 1 -- 172.21.15.44:0/545110628 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f11680037a0 con 0x7f1174075420 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.312+0000 7f117a400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc51c1b0 0x55edcc8bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.312+0000 7f117a400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc51c1b0 0x55edcc8bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59236/0 (socket says 172.21.15.44:59236) 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.312+0000 7f117a400640 1 -- 172.21.15.44:0/3929176895 learned_addr learned my addr 172.21.15.44:0/3929176895 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.312+0000 7f1179000640 1 --2- 172.21.15.44:0/3929176895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 0x55edcc8bf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.312+0000 7f1179a00640 1 --2- 172.21.15.44:0/3929176895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 0x55edcc8befa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.312+0000 7f117a400640 1 -- 172.21.15.44:0/3929176895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 msgr2=0x55edcc8bf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.313+0000 7f117a400640 1 --2- 172.21.15.44:0/3929176895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 0x55edcc8bf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.313+0000 7f117a400640 1 -- 172.21.15.44:0/3929176895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 msgr2=0x55edcc8befa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.313+0000 7f117a400640 1 --2- 172.21.15.44:0/3929176895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 0x55edcc8befa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.313+0000 7f117a400640 1 -- 172.21.15.44:0/3929176895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edcc7b88b0 con 0x55edcc51c1b0 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.313+0000 7f1179a00640 1 --2- 172.21.15.44:0/3929176895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 0x55edcc8befa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.313+0000 7f1179000640 1 --2- 172.21.15.44:0/3929176895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 0x55edcc8bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.313+0000 7f117a400640 1 --2- 172.21.15.44:0/3929176895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc51c1b0 0x55edcc8bea60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f116c009f50 tx=0x7f116c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8786f02139e66eb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.314+0000 7f1166a00640 1 -- 172.21.15.44:0/3929176895 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c004b20 con 0x55edcc51c1b0 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.314+0000 7f1166a00640 1 -- 172.21.15.44:0/3929176895 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f116c004cc0 con 0x55edcc51c1b0 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.314+0000 7f1166a00640 1 -- 172.21.15.44:0/3929176895 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c005020 con 0x55edcc51c1b0 2024-05-27T09:37:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.314+0000 7f1172000640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f115c0b2950 0x7f115c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.314+0000 7f1172000640 1 -- 172.21.15.44:0/545110628 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f11680e28d0 con 0x7f1174075420 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.315+0000 7f1183395740 1 -- 172.21.15.44:0/3929176895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc51c1b0 msgr2=0x55edcc8bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.315+0000 7f1183395740 1 --2- 172.21.15.44:0/3929176895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc51c1b0 0x55edcc8bea60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f116c009f50 tx=0x7f116c0118d0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.315+0000 7f1179a00640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f115c0b2950 0x7f115c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.315+0000 7f1183395740 1 -- 172.21.15.44:0/3929176895 shutdown_connections 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.315+0000 7f1183395740 1 --2- 172.21.15.44:0/3929176895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 0x55edcc8bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.315+0000 7f1183395740 1 --2- 172.21.15.44:0/3929176895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 0x55edcc8befa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.315+0000 7f1183395740 1 --2- 172.21.15.44:0/3929176895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc51c1b0 0x55edcc8bea60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.315+0000 7f1183395740 1 -- 172.21.15.44:0/3929176895 >> 172.21.15.44:0/3929176895 conn(0x55edcc510300 msgr2=0x55edcc8bb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.315+0000 7f1183395740 1 -- 172.21.15.44:0/3929176895 shutdown_connections 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.315+0000 7f1183395740 1 -- 172.21.15.44:0/3929176895 wait complete. 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.316+0000 7f1179a00640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f115c0b2950 0x7f115c0b4e60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55edcc8bc550 tx=0x7f116800ef90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.316+0000 7f1183395740 1 -- start start 2024-05-27T09:37:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.316+0000 7f1183395740 1 Processor -- start 2024-05-27T09:37:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f1183395740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 0x55edcc8c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f1183395740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4fb4d0 0x55edcc8c9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f1183395740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc51c1b0 0x55edcc8d4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f1183395740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55edcc505790 con 0x55edcc4f32f0 2024-05-27T09:37:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f1183395740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55edcc7c7bc0 con 0x55edcc4fb4d0 2024-05-27T09:37:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f1183395740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55edcc8cbb90 con 0x55edcc51c1b0 2024-05-27T09:37:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f1183395740 10 client.0 _ll_register_callbacks cb 0x55edcc7a5400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f117a400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 0x55edcc8c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f117a400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 0x55edcc8c8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59248/0 (socket says 172.21.15.44:59248) 2024-05-27T09:37:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f117a400640 1 -- 172.21.15.44:0/1471743455 learned_addr learned my addr 172.21.15.44:0/1471743455 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f1179000640 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4fb4d0 0x55edcc8c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.317+0000 7f117a400640 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc51c1b0 0x55edcc8d4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.318+0000 7f1179000640 1 -- 172.21.15.44:0/1471743455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc51c1b0 msgr2=0x55edcc8d4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.318+0000 7f1179000640 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc51c1b0 0x55edcc8d4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.318+0000 7f1179000640 1 -- 172.21.15.44:0/1471743455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 msgr2=0x55edcc8c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.318+0000 7f1179000640 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 0x55edcc8c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.318+0000 7f1179000640 1 -- 172.21.15.44:0/1471743455 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edcc534210 con 0x55edcc4fb4d0 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.318+0000 7f1179000640 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4fb4d0 0x55edcc8c9180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f116002a9a0 tx=0x7f11600043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf9bc5b619365bfd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.318+0000 7f115b400640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.319+0000 7f115b400640 1 -- 172.21.15.44:0/1471743455 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1160035040 con 0x55edcc4fb4d0 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.319+0000 7f1183395740 1 -- 172.21.15.44:0/1471743455 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f116c004cc0 con 0x55edcc4fb4d0 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.319+0000 7f115b400640 1 -- 172.21.15.44:0/1471743455 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1160005450 con 0x55edcc4fb4d0 2024-05-27T09:37:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.319+0000 7f115b400640 1 -- 172.21.15.44:0/1471743455 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1160005650 con 0x55edcc4fb4d0 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.319+0000 7f1183395740 10 client.25622 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.319+0000 7f1183395740 1 -- 172.21.15.44:0/1471743455 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55edcc600620 con 0x55edcc4fb4d0 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.319+0000 7f115b400640 1 -- 172.21.15.44:0/1471743455 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f116c004cc0 con 0x55edcc4fb4d0 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.320+0000 7f115b400640 1 -- 172.21.15.44:0/1471743455 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f11600039b0 con 0x55edcc4fb4d0 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.320+0000 7f1183395740 10 client.25622 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.320+0000 7f1183395740 10 client.25622 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.320+0000 7f1183395740 20 client.25622 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.320+0000 7f1183395740 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55edcc4dad80 0x55edcc8dae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.320+0000 7f1183395740 4 client.25622 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.320+0000 7f1183395740 1 -- 172.21.15.44:0/1471743455 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edcc831ae0 con 0x55edcc4dad80 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.320+0000 7f117a400640 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55edcc4dad80 0x55edcc8dae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.321+0000 7f117a400640 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55edcc4dad80 0x55edcc8dae50 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2b11e4bc9b66a74 server_cookie=22fd33fcaecbe3ad in_seq=0 out_seq=0 2024-05-27T09:37:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.321+0000 7f115b400640 10 client.25622 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f115b400640 1 -- 172.21.15.44:0/1471743455 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55edcc831ae0 con 0x55edcc4dad80 2024-05-27T09:37:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f115b400640 10 client.25622 handle_command_reply: tid=0 2024-05-27T09:37:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f1183395740 1 client.25622 shutdown 2024-05-27T09:37:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f1183395740 1 -- 172.21.15.44:0/1471743455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4fb4d0 msgr2=0x55edcc8c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f1183395740 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4fb4d0 0x55edcc8c9180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f116002a9a0 tx=0x7f11600043b0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f1183395740 1 -- 172.21.15.44:0/1471743455 shutdown_connections 2024-05-27T09:37:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f1183395740 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55edcc4dad80 0x55edcc8dae50 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f1183395740 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc51c1b0 0x55edcc8d4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f1183395740 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4fb4d0 0x55edcc8c9180 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f1183395740 1 --2- 172.21.15.44:0/1471743455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 0x55edcc8c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f1183395740 1 -- 172.21.15.44:0/1471743455 >> 172.21.15.44:0/1471743455 conn(0x55edcc510300 msgr2=0x55edcc8be890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f115b400640 0 client.25622 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f1183395740 1 -- 172.21.15.44:0/1471743455 shutdown_connections 2024-05-27T09:37:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.322+0000 7f1183395740 1 -- 172.21.15.44:0/1471743455 wait complete. 2024-05-27T09:37:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.325+0000 7f1183395740 20 client.25622 trim_cache size 0 max 16384 2024-05-27T09:37:08.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.356+0000 7f1183395740 1 Processor -- start 2024-05-27T09:37:08.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.356+0000 7f1183395740 1 -- start start 2024-05-27T09:37:08.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.357+0000 7f1183395740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 0x55edcc8cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.357+0000 7f1183395740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 0x55edcc8d4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.357+0000 7f1183395740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc51c1b0 0x55edcc8d5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.357+0000 7f1183395740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f11741aff80 con 0x55edcc4f32f0 2024-05-27T09:37:08.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.357+0000 7f1183395740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f11741ad850 con 0x55edcc51c1b0 2024-05-27T09:37:08.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.357+0000 7f1183395740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f11741ab6e0 con 0x55edcc4fb4d0 2024-05-27T09:37:08.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.357+0000 7f117a400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 0x55edcc8cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.357+0000 7f117a400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 0x55edcc8cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59256/0 (socket says 172.21.15.44:59256) 2024-05-27T09:37:08.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.357+0000 7f117a400640 1 -- 172.21.15.44:0/2131368380 learned_addr learned my addr 172.21.15.44:0/2131368380 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:08.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.357+0000 7f1179000640 1 --2- 172.21.15.44:0/2131368380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 0x55edcc8d4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.358+0000 7f117a400640 1 --2- 172.21.15.44:0/2131368380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc51c1b0 0x55edcc8d5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.358+0000 7f117a400640 1 -- 172.21.15.44:0/2131368380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 msgr2=0x55edcc8d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.358+0000 7f117a400640 1 --2- 172.21.15.44:0/2131368380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 0x55edcc8d4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.358+0000 7f117a400640 1 -- 172.21.15.44:0/2131368380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc51c1b0 msgr2=0x55edcc8d5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.358+0000 7f117a400640 1 --2- 172.21.15.44:0/2131368380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc51c1b0 0x55edcc8d5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.358+0000 7f117a400640 1 -- 172.21.15.44:0/2131368380 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edcc8c7190 con 0x55edcc4f32f0 2024-05-27T09:37:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.358+0000 7f1179000640 1 --2- 172.21.15.44:0/2131368380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 0x55edcc8d4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.358+0000 7f117a400640 1 --2- 172.21.15.44:0/2131368380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 0x55edcc8cb370 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f116c009320 tx=0x7f116c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b301edb5bb38b14 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.359+0000 7f1166000640 1 -- 172.21.15.44:0/2131368380 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c004510 con 0x55edcc4f32f0 2024-05-27T09:37:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.359+0000 7f1166000640 1 -- 172.21.15.44:0/2131368380 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f116c003ee0 con 0x55edcc4f32f0 2024-05-27T09:37:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.359+0000 7f1166000640 1 -- 172.21.15.44:0/2131368380 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c0287b0 con 0x55edcc4f32f0 2024-05-27T09:37:08.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.360+0000 7f1183395740 1 -- 172.21.15.44:0/2131368380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 msgr2=0x55edcc8cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:08.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.360+0000 7f1183395740 1 --2- 172.21.15.44:0/2131368380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 0x55edcc8cb370 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f116c009320 tx=0x7f116c005fd0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.360+0000 7f1183395740 1 -- 172.21.15.44:0/2131368380 shutdown_connections 2024-05-27T09:37:08.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.360+0000 7f1183395740 1 --2- 172.21.15.44:0/2131368380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc51c1b0 0x55edcc8d5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.360+0000 7f1183395740 1 --2- 172.21.15.44:0/2131368380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc4fb4d0 0x55edcc8d4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.360+0000 7f1183395740 1 --2- 172.21.15.44:0/2131368380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4f32f0 0x55edcc8cb370 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.360+0000 7f1183395740 1 -- 172.21.15.44:0/2131368380 >> 172.21.15.44:0/2131368380 conn(0x55edcc510300 msgr2=0x55edcc8c8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.360+0000 7f1183395740 1 -- 172.21.15.44:0/2131368380 shutdown_connections 2024-05-27T09:37:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.360+0000 7f1183395740 1 -- 172.21.15.44:0/2131368380 wait complete. 2024-05-27T09:37:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.361+0000 7f1183395740 1 -- start start 2024-05-27T09:37:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.361+0000 7f1183395740 1 Processor -- start 2024-05-27T09:37:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.362+0000 7f1183395740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 0x55edcc8ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.362+0000 7f1183395740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4fb4d0 0x55edcc8ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.362+0000 7f1183395740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc51c1b0 0x55edcc941490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.362+0000 7f1183395740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f11741abd30 con 0x55edcc4fb4d0 2024-05-27T09:37:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.362+0000 7f1183395740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f11741ad180 con 0x55edcc4f32f0 2024-05-27T09:37:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.362+0000 7f1183395740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f11741ac3e0 con 0x55edcc51c1b0 2024-05-27T09:37:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.362+0000 7f1183395740 10 client.0 _ll_register_callbacks cb 0x7f1160009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.362+0000 7f117a400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 0x55edcc8ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.363+0000 7f117a400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 0x55edcc8ce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:59316/0 (socket says 172.21.15.44:59316) 2024-05-27T09:37:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.363+0000 7f117a400640 1 -- 172.21.15.44:0/357606128 learned_addr learned my addr 172.21.15.44:0/357606128 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.363+0000 7f1179000640 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4fb4d0 0x55edcc8ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.363+0000 7f117a400640 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc51c1b0 0x55edcc941490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.363+0000 7f117a400640 1 -- 172.21.15.44:0/357606128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc51c1b0 msgr2=0x55edcc941490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.363+0000 7f117a400640 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc51c1b0 0x55edcc941490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.363+0000 7f117a400640 1 -- 172.21.15.44:0/357606128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4fb4d0 msgr2=0x55edcc8ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.363+0000 7f117a400640 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4fb4d0 0x55edcc8ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.363+0000 7f117a400640 1 -- 172.21.15.44:0/357606128 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edcc9419d0 con 0x55edcc4f32f0 2024-05-27T09:37:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.363+0000 7f1179000640 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4fb4d0 0x55edcc8ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.364+0000 7f117a400640 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 0x55edcc8ce280 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f116c009d60 tx=0x7f116c009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe17580464cd9592 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.364+0000 7f115b400640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:37:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.364+0000 7f115b400640 1 -- 172.21.15.44:0/357606128 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c004090 con 0x55edcc4f32f0 2024-05-27T09:37:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.364+0000 7f115b400640 1 -- 172.21.15.44:0/357606128 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f116c003a30 con 0x55edcc4f32f0 2024-05-27T09:37:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.364+0000 7f1183395740 1 -- 172.21.15.44:0/357606128 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f116c003ee0 con 0x55edcc4f32f0 2024-05-27T09:37:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.365+0000 7f115b400640 1 -- 172.21.15.44:0/357606128 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f116c004230 con 0x55edcc4f32f0 2024-05-27T09:37:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.365+0000 7f1183395740 10 client.25628 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.365+0000 7f1183395740 1 -- 172.21.15.44:0/357606128 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55edcc942b70 con 0x55edcc4f32f0 2024-05-27T09:37:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.365+0000 7f115b400640 1 -- 172.21.15.44:0/357606128 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f116c003ee0 con 0x55edcc4f32f0 2024-05-27T09:37:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.366+0000 7f115b400640 1 -- 172.21.15.44:0/357606128 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f116c028450 con 0x55edcc4f32f0 2024-05-27T09:37:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.366+0000 7f1183395740 10 client.25628 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.366+0000 7f1183395740 10 client.25628 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.366+0000 7f1183395740 20 client.25628 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.366+0000 7f1183395740 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55edcc4b01c0 0x55edcc944f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.366+0000 7f1183395740 4 client.25628 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:37:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.366+0000 7f1183395740 1 -- 172.21.15.44:0/357606128 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55edcc82b2c0 con 0x55edcc4b01c0 2024-05-27T09:37:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.367+0000 7f1179000640 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55edcc4b01c0 0x55edcc944f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.367+0000 7f1179000640 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55edcc4b01c0 0x55edcc944f00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f69c20732dd5523 server_cookie=9c36001ac9ac4495 in_seq=0 out_seq=0 2024-05-27T09:37:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.367+0000 7f115b400640 10 client.25628 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:08.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.374+0000 7f115b400640 1 -- 172.21.15.44:0/357606128 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55edcc82b2c0 con 0x55edcc4b01c0 2024-05-27T09:37:08.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.374+0000 7f115b400640 10 client.25628 handle_command_reply: tid=0 2024-05-27T09:37:08.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.374+0000 7f1183395740 1 client.25628 shutdown 2024-05-27T09:37:08.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.374+0000 7f1183395740 1 -- 172.21.15.44:0/357606128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 msgr2=0x55edcc8ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.374+0000 7f1183395740 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 0x55edcc8ce280 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f116c009d60 tx=0x7f116c009bb0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.374+0000 7f1183395740 1 -- 172.21.15.44:0/357606128 shutdown_connections 2024-05-27T09:37:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.374+0000 7f1183395740 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55edcc4b01c0 0x55edcc944f00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.374+0000 7f1183395740 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edcc51c1b0 0x55edcc941490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.374+0000 7f1183395740 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55edcc4fb4d0 0x55edcc8ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.374+0000 7f1183395740 1 --2- 172.21.15.44:0/357606128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55edcc4f32f0 0x55edcc8ce280 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.375+0000 7f1183395740 1 -- 172.21.15.44:0/357606128 >> 172.21.15.44:0/357606128 conn(0x55edcc510300 msgr2=0x55edcc8c3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.375+0000 7f115b400640 0 client.25628 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.375+0000 7f1183395740 1 -- 172.21.15.44:0/357606128 shutdown_connections 2024-05-27T09:37:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.375+0000 7f1183395740 1 -- 172.21.15.44:0/357606128 wait complete. 2024-05-27T09:37:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.375+0000 7f1183395740 20 client.25628 trim_cache size 0 max 16384 2024-05-27T09:37:08.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:37:08.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:37:08.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:37:08.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.376+0000 7f117ae00640 1 -- 172.21.15.44:0/545110628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f115c0b2950 msgr2=0x7f115c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:08.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.376+0000 7f117ae00640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f115c0b2950 0x7f115c0b4e60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55edcc8bc550 tx=0x7f116800ef90 comp rx=0 tx=0).stop 2024-05-27T09:37:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.376+0000 7f117ae00640 1 -- 172.21.15.44:0/545110628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1174075420 msgr2=0x7f11741a25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.376+0000 7f117ae00640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1174075420 0x7f11741a25a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f116800ea60 tx=0x7f116800ef20 comp rx=0 tx=0).stop 2024-05-27T09:37:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.376+0000 7f117ae00640 1 -- 172.21.15.44:0/545110628 shutdown_connections 2024-05-27T09:37:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.376+0000 7f117ae00640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1174110320 0x7f117419c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.376+0000 7f117ae00640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f117410d9c0 0x7f11741a2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.376+0000 7f117ae00640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f115c0b2950 0x7f115c0b4e60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.376+0000 7f117ae00640 1 --2- 172.21.15.44:0/545110628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1174075420 0x7f11741a25a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.376+0000 7f117ae00640 1 -- 172.21.15.44:0/545110628 >> 172.21.15.44:0/545110628 conn(0x7f1174079100 msgr2=0x7f11741022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.376+0000 7f117ae00640 1 -- 172.21.15.44:0/545110628 shutdown_connections 2024-05-27T09:37:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:08.377+0000 7f117ae00640 1 -- 172.21.15.44:0/545110628 wait complete. 2024-05-27T09:37:08.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:08 smithi044 ceph-mon[31618]: pgmap v60: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T09:37:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:08 smithi050 ceph-mon[36371]: pgmap v60: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T09:37:08.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:08 smithi189 ceph-mon[36671]: pgmap v60: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T09:37:09.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.223+0000 7f7aba200700 1 -- 172.21.15.44:0/38631414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ab41043f0 msgr2=0x7f7ab4104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:09.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.223+0000 7f7aba200700 1 --2- 172.21.15.44:0/38631414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ab41043f0 0x7f7ab4104800 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80099c0 tx=0x7f7aa8009cd0 comp rx=0 tx=0).stop 2024-05-27T09:37:09.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.223+0000 7f7aba200700 1 -- 172.21.15.44:0/38631414 shutdown_connections 2024-05-27T09:37:09.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.223+0000 7f7aba200700 1 --2- 172.21.15.44:0/38631414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ab4109670 0x7f7ab4110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:09.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.223+0000 7f7aba200700 1 --2- 172.21.15.44:0/38631414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ab41043f0 0x7f7ab4104800 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:09.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.223+0000 7f7aba200700 1 --2- 172.21.15.44:0/38631414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ab4108d20 0x7f7ab4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:09.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.223+0000 7f7aba200700 1 -- 172.21.15.44:0/38631414 >> 172.21.15.44:0/38631414 conn(0x7f7ab4079d00 msgr2=0x7f7ab407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:09.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.223+0000 7f7aba200700 1 -- 172.21.15.44:0/38631414 shutdown_connections 2024-05-27T09:37:09.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.223+0000 7f7aba200700 1 -- 172.21.15.44:0/38631414 wait complete. 2024-05-27T09:37:09.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.224+0000 7f7aba200700 1 Processor -- start 2024-05-27T09:37:09.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.225+0000 7f7aba200700 1 -- start start 2024-05-27T09:37:09.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.225+0000 7f7aba200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ab41043f0 0x7f7ab419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.225+0000 7f7aba200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ab4108d20 0x7f7ab419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.225+0000 7f7aba200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ab4109670 0x7f7ab41a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.225+0000 7f7aba200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41142f0 con 0x7f7ab4109670 2024-05-27T09:37:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.225+0000 7f7aba200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab4114170 con 0x7f7ab4108d20 2024-05-27T09:37:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.225+0000 7f7aba200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab4114470 con 0x7f7ab41043f0 2024-05-27T09:37:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.226+0000 7f7ab8e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ab41043f0 0x7f7ab419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.226+0000 7f7ab8e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ab41043f0 0x7f7ab419cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:46600/0 (socket says 172.21.15.44:46600) 2024-05-27T09:37:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.226+0000 7f7ab8e00700 1 -- 172.21.15.44:0/2695874693 learned_addr learned my addr 172.21.15.44:0/2695874693 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.226+0000 7f7ab3e00700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ab4108d20 0x7f7ab419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.226+0000 7f7ab8e00700 1 -- 172.21.15.44:0/2695874693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ab4108d20 msgr2=0x7f7ab419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:09.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.226+0000 7f7ab8e00700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ab4108d20 0x7f7ab419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:09.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.226+0000 7f7ab8e00700 1 -- 172.21.15.44:0/2695874693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ab4109670 msgr2=0x7f7ab41a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:09.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.226+0000 7f7ab8e00700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ab4109670 0x7f7ab41a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:09.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.226+0000 7f7ab8e00700 1 -- 172.21.15.44:0/2695874693 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab41a1dd0 con 0x7f7ab41043f0 2024-05-27T09:37:09.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.227+0000 7f7ab8e00700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ab41043f0 0x7f7ab419cf00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80098e0 tx=0x7f7aa8005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:09.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.227+0000 7f7ab1600700 1 -- 172.21.15.44:0/2695874693 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8026070 con 0x7f7ab41043f0 2024-05-27T09:37:09.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.227+0000 7f7ab1600700 1 -- 172.21.15.44:0/2695874693 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7aa8003c70 con 0x7f7ab41043f0 2024-05-27T09:37:09.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.227+0000 7f7ab1600700 1 -- 172.21.15.44:0/2695874693 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa800fa60 con 0x7f7ab41043f0 2024-05-27T09:37:09.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.227+0000 7f7aba200700 1 -- 172.21.15.44:0/2695874693 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ab41a2060 con 0x7f7ab41043f0 2024-05-27T09:37:09.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.227+0000 7f7aba200700 1 -- 172.21.15.44:0/2695874693 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ab41a99e0 con 0x7f7ab41043f0 2024-05-27T09:37:09.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.229+0000 7f7aba200700 1 -- 172.21.15.44:0/2695874693 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ab40512b0 con 0x7f7ab41043f0 2024-05-27T09:37:09.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.231+0000 7f7ab1600700 1 -- 172.21.15.44:0/2695874693 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f7aa80037c0 con 0x7f7ab41043f0 2024-05-27T09:37:09.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.234+0000 7f7ab1600700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a9c0b2a70 0x7f7a9c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:09.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.234+0000 7f7ab1600700 1 -- 172.21.15.44:0/2695874693 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f7aa80e3830 con 0x7f7ab41043f0 2024-05-27T09:37:09.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.234+0000 7f7ab3e00700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a9c0b2a70 0x7f7a9c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:09.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.235+0000 7f7ab1600700 1 -- 172.21.15.44:0/2695874693 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7aa802f020 con 0x7f7ab41043f0 2024-05-27T09:37:09.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.235+0000 7f7ab3e00700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a9c0b2a70 0x7f7a9c0b4f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4101d20 tx=0x7f7aa0009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.635+0000 7f7aba200700 1 -- 172.21.15.44:0/2695874693 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ab41a9bb0 con 0x7f7ab41043f0 2024-05-27T09:37:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.636+0000 7f7ab1600700 1 -- 172.21.15.44:0/2695874693 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7f7aa8013070 con 0x7f7ab41043f0 2024-05-27T09:37:09.637 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:37:09.637 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:37:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.639+0000 7f7aba200700 1 -- 172.21.15.44:0/2695874693 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a9c0b2a70 msgr2=0x7f7a9c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.639+0000 7f7aba200700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a9c0b2a70 0x7f7a9c0b4f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4101d20 tx=0x7f7aa0009260 comp rx=0 tx=0).stop 2024-05-27T09:37:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.639+0000 7f7aba200700 1 -- 172.21.15.44:0/2695874693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ab41043f0 msgr2=0x7f7ab419cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.639+0000 7f7aba200700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ab41043f0 0x7f7ab419cf00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80098e0 tx=0x7f7aa8005280 comp rx=0 tx=0).stop 2024-05-27T09:37:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.639+0000 7f7aba200700 1 -- 172.21.15.44:0/2695874693 shutdown_connections 2024-05-27T09:37:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.639+0000 7f7aba200700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a9c0b2a70 0x7f7a9c0b4f80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:09.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.639+0000 7f7aba200700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ab4109670 0x7f7ab41a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:09.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.639+0000 7f7aba200700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ab41043f0 0x7f7ab419cf00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:09.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.639+0000 7f7aba200700 1 --2- 172.21.15.44:0/2695874693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ab4108d20 0x7f7ab419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:09.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.639+0000 7f7aba200700 1 -- 172.21.15.44:0/2695874693 >> 172.21.15.44:0/2695874693 conn(0x7f7ab4079d00 msgr2=0x7f7ab4102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:09.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.640+0000 7f7aba200700 1 -- 172.21.15.44:0/2695874693 shutdown_connections 2024-05-27T09:37:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:09.640+0000 7f7aba200700 1 -- 172.21.15.44:0/2695874693 wait complete. 2024-05-27T09:37:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:37:09.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:09 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2695874693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:37:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:09 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2695874693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:37:10.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:09 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2695874693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:37:10.503 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db --set-id 4295cdb8 --release --await' 2024-05-27T09:37:10.503 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db --set-id 4295cdb8 --release --await 2024-05-27T09:37:10.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:10 smithi044 ceph-mon[31618]: pgmap v61: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-27T09:37:11.027 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:37:11.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:10 smithi050 ceph-mon[36371]: pgmap v61: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-27T09:37:11.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:10 smithi189 ceph-mon[36671]: pgmap v61: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-27T09:37:11.413 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:37:12.395 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:37:12.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:12 smithi044 ceph-mon[31618]: pgmap v62: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 767 B/s wr, 15 op/s 2024-05-27T09:37:12.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:12 smithi050 ceph-mon[36371]: pgmap v62: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 767 B/s wr, 15 op/s 2024-05-27T09:37:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:12 smithi189 ceph-mon[36671]: pgmap v62: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 767 B/s wr, 15 op/s 2024-05-27T09:37:13.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.510+0000 7fe800800700 1 -- 172.21.15.44:0/3669450528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f8075af0 msgr2=0x7fe7f8075f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:13.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.510+0000 7fe800800700 1 --2- 172.21.15.44:0/3669450528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f8075af0 0x7fe7f8075f00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009990 tx=0x7fe7ec009ca0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.510+0000 7fe800800700 1 -- 172.21.15.44:0/3669450528 shutdown_connections 2024-05-27T09:37:13.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.510+0000 7fe800800700 1 --2- 172.21.15.44:0/3669450528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f8110c30 0x7fe7f8113050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.510+0000 7fe800800700 1 --2- 172.21.15.44:0/3669450528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f8075af0 0x7fe7f8075f00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.510+0000 7fe800800700 1 --2- 172.21.15.44:0/3669450528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7f8076440 0x7fe7f81106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.510+0000 7fe800800700 1 -- 172.21.15.44:0/3669450528 >> 172.21.15.44:0/3669450528 conn(0x7fe7f8100090 msgr2=0x7fe7f81024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:13.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.511+0000 7fe800800700 1 -- 172.21.15.44:0/3669450528 shutdown_connections 2024-05-27T09:37:13.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.511+0000 7fe800800700 1 -- 172.21.15.44:0/3669450528 wait complete. 2024-05-27T09:37:13.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.512+0000 7fe800800700 1 Processor -- start 2024-05-27T09:37:13.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.513+0000 7fe800800700 1 -- start start 2024-05-27T09:37:13.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.513+0000 7fe800800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f8075af0 0x7fe7f819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.513+0000 7fe800800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f8076440 0x7fe7f819d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.513+0000 7fe800800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7f8110c30 0x7fe7f81a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.513+0000 7fe800800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f8116640 con 0x7fe7f8076440 2024-05-27T09:37:13.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.513+0000 7fe800800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f81164c0 con 0x7fe7f8075af0 2024-05-27T09:37:13.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.513+0000 7fe800800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f81167c0 con 0x7fe7f8110c30 2024-05-27T09:37:13.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.514+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f8075af0 0x7fe7f819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.514+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f8075af0 0x7fe7f819cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50424/0 (socket says 172.21.15.44:50424) 2024-05-27T09:37:13.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.514+0000 7fe7fde00700 1 -- 172.21.15.44:0/855749993 learned_addr learned my addr 172.21.15.44:0/855749993 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:13.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.514+0000 7fe7fd400700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f8076440 0x7fe7f819d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.514+0000 7fe7fe800700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7f8110c30 0x7fe7f81a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.514+0000 7fe7fd400700 1 -- 172.21.15.44:0/855749993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7f8110c30 msgr2=0x7fe7f81a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.515+0000 7fe7fd400700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7f8110c30 0x7fe7f81a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.515+0000 7fe7fd400700 1 -- 172.21.15.44:0/855749993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f8075af0 msgr2=0x7fe7f819cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.515+0000 7fe7fd400700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f8075af0 0x7fe7f819cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.515+0000 7fe7fd400700 1 -- 172.21.15.44:0/855749993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f81a1ec0 con 0x7fe7f8076440 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.515+0000 7fe7fe800700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7f8110c30 0x7fe7f81a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.515+0000 7fe7fde00700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f8075af0 0x7fe7f819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.515+0000 7fe7fd400700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f8076440 0x7fe7f819d590 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800e9f0 tx=0x7fe7e800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.515+0000 7fe7f2a00700 1 -- 172.21.15.44:0/855749993 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e800c9c0 con 0x7fe7f8076440 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.516+0000 7fe7f2a00700 1 -- 172.21.15.44:0/855749993 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7e800cb60 con 0x7fe7f8076440 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.516+0000 7fe7f2a00700 1 -- 172.21.15.44:0/855749993 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8018680 con 0x7fe7f8076440 2024-05-27T09:37:13.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.516+0000 7fe800800700 1 -- 172.21.15.44:0/855749993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f81a21b0 con 0x7fe7f8076440 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.516+0000 7fe800800700 1 -- 172.21.15.44:0/855749993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f81a9ad0 con 0x7fe7f8076440 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.521+0000 7fe80f310b80 1 Processor -- start 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.521+0000 7fe7f2a00700 1 -- 172.21.15.44:0/855749993 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fe7e8010430 con 0x7fe7f8076440 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.521+0000 7fe80f310b80 1 -- start start 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.521+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 0x55ef92835530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.521+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 0x55ef92835a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.521+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef9282f5e0 0x55ef9282f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.522+0000 7fe80f310b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ef9275cf40 con 0x55ef926aff40 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.522+0000 7fe80f310b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ef9274cae0 con 0x55ef9279baf0 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.522+0000 7fe80f310b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ef92750b20 con 0x55ef9282f5e0 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.522+0000 7fe7fe800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 0x55ef92835530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.522+0000 7fe7fe800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 0x55ef92835530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50426/0 (socket says 172.21.15.44:50426) 2024-05-27T09:37:13.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.522+0000 7fe7fe800700 1 -- 172.21.15.44:0/644252924 learned_addr learned my addr 172.21.15.44:0/644252924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:13.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.522+0000 7fe7fd400700 1 --2- 172.21.15.44:0/644252924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef9282f5e0 0x55ef9282f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.522+0000 7fe7fde00700 1 --2- 172.21.15.44:0/644252924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 0x55ef92835a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.523+0000 7fe7fe800700 1 -- 172.21.15.44:0/644252924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef9282f5e0 msgr2=0x55ef9282f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:13.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.523+0000 7fe7fe800700 1 --2- 172.21.15.44:0/644252924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef9282f5e0 0x55ef9282f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.523+0000 7fe7fe800700 1 -- 172.21.15.44:0/644252924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 msgr2=0x55ef92835a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:13.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.523+0000 7fe7fe800700 1 --2- 172.21.15.44:0/644252924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 0x55ef92835a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.523+0000 7fe7fe800700 1 -- 172.21.15.44:0/644252924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef92715520 con 0x55ef9279baf0 2024-05-27T09:37:13.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.524+0000 7fe7f2a00700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe7e00b2a70 0x7fe7e00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.524+0000 7fe7f2a00700 1 -- 172.21.15.44:0/855749993 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fe7e80e2730 con 0x7fe7f8076440 2024-05-27T09:37:13.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.524+0000 7fe7fde00700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe7e00b2a70 0x7fe7e00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.524+0000 7fe7fe800700 1 --2- 172.21.15.44:0/644252924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 0x55ef92835530 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4003eb0 tx=0x7fe7f4009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b613b9be39c2f486 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:13.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.525+0000 7fe7e7400700 1 -- 172.21.15.44:0/644252924 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4019020 con 0x55ef9279baf0 2024-05-27T09:37:13.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.525+0000 7fe7e7400700 1 -- 172.21.15.44:0/644252924 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7f40092d0 con 0x55ef9279baf0 2024-05-27T09:37:13.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.525+0000 7fe7e7400700 1 -- 172.21.15.44:0/644252924 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f400e870 con 0x55ef9279baf0 2024-05-27T09:37:13.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.526+0000 7fe80f310b80 1 -- 172.21.15.44:0/644252924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 msgr2=0x55ef92835530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:13.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.526+0000 7fe7fde00700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe7e00b2a70 0x7fe7e00b4f80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009750 tx=0x7fe7ec005040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:13.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.526+0000 7fe80f310b80 1 --2- 172.21.15.44:0/644252924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 0x55ef92835530 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4003eb0 tx=0x7fe7f4009920 comp rx=0 tx=0).stop 2024-05-27T09:37:13.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.526+0000 7fe80f310b80 1 -- 172.21.15.44:0/644252924 shutdown_connections 2024-05-27T09:37:13.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.526+0000 7fe80f310b80 1 --2- 172.21.15.44:0/644252924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef9282f5e0 0x55ef9282f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.526+0000 7fe80f310b80 1 --2- 172.21.15.44:0/644252924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 0x55ef92835530 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.526+0000 7fe80f310b80 1 --2- 172.21.15.44:0/644252924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 0x55ef92835a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.526+0000 7fe80f310b80 1 -- 172.21.15.44:0/644252924 >> 172.21.15.44:0/644252924 conn(0x55ef92749490 msgr2=0x55ef9282a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:13.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.526+0000 7fe80f310b80 1 -- 172.21.15.44:0/644252924 shutdown_connections 2024-05-27T09:37:13.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.526+0000 7fe80f310b80 1 -- 172.21.15.44:0/644252924 wait complete. 2024-05-27T09:37:13.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.527+0000 7fe80f310b80 1 -- start start 2024-05-27T09:37:13.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.527+0000 7fe80f310b80 1 Processor -- start 2024-05-27T09:37:13.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.528+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92839360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.528+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 0x55ef928398a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.528+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9282f5e0 0x55ef92839de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.528+0000 7fe80f310b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ef926530a0 con 0x55ef9279baf0 2024-05-27T09:37:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.528+0000 7fe80f310b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ef926ed860 con 0x55ef9282f5e0 2024-05-27T09:37:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.528+0000 7fe80f310b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ef928303d0 con 0x55ef926aff40 2024-05-27T09:37:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.528+0000 7fe80f310b80 10 client.0 _ll_register_callbacks cb 0x55ef926f6d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.528+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92839360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.528+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92839360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49232/0 (socket says 172.21.15.44:49232) 2024-05-27T09:37:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.529+0000 7fe7fde00700 1 -- 172.21.15.44:0/3512505315 learned_addr learned my addr 172.21.15.44:0/3512505315 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.529+0000 7fe7fd400700 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 0x55ef928398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.529+0000 7fe7fe800700 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9282f5e0 0x55ef92839de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.529+0000 7fe7fde00700 1 -- 172.21.15.44:0/3512505315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9282f5e0 msgr2=0x55ef92839de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.529+0000 7fe7fde00700 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9282f5e0 0x55ef92839de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.529+0000 7fe7fde00700 1 -- 172.21.15.44:0/3512505315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 msgr2=0x55ef928398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.529+0000 7fe7fde00700 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 0x55ef928398a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.529+0000 7fe7fde00700 1 -- 172.21.15.44:0/3512505315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef9269c9d0 con 0x55ef926aff40 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.529+0000 7fe7fd400700 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 0x55ef928398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.530+0000 7fe7fde00700 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92839360 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec003900 tx=0x7fe7ec009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50ac795113f0ddc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.530+0000 7fe7dbe00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.530+0000 7fe7fe800700 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9282f5e0 0x55ef92839de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.530+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3512505315 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec035060 con 0x55ef926aff40 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.530+0000 7fe80f310b80 1 -- 172.21.15.44:0/3512505315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7f40092d0 con 0x55ef926aff40 2024-05-27T09:37:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.530+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3512505315 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fe7ec025a90 con 0x55ef926aff40 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.531+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3512505315 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe7ec025c90 con 0x55ef926aff40 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.531+0000 7fe80f310b80 10 client.25642 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.531+0000 7fe80f310b80 1 -- 172.21.15.44:0/3512505315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ef9271d160 con 0x55ef926aff40 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.531+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3512505315 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7f40092d0 con 0x55ef926aff40 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.531+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3512505315 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe7ec00f460 con 0x55ef926aff40 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.531+0000 7fe80f310b80 10 client.25642 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.532+0000 7fe80f310b80 10 client.25642 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.532+0000 7fe80f310b80 20 client.25642 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.532+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef92849060 0x55ef9284b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.532+0000 7fe80f310b80 4 client.25642 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.532+0000 7fe80f310b80 1 -- 172.21.15.44:0/3512505315 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef9275ee90 con 0x55ef92849060 2024-05-27T09:37:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.532+0000 7fe7fd400700 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef92849060 0x55ef9284b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.533+0000 7fe7fd400700 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef92849060 0x55ef9284b4c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=118e8fcd86cc0d90 server_cookie=adb0c012a763bc33 in_seq=0 out_seq=0 2024-05-27T09:37:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.533+0000 7fe7dbe00700 10 client.25642 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.533+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3512505315 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ef9275ee90 con 0x55ef92849060 2024-05-27T09:37:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.533+0000 7fe7dbe00700 10 client.25642 handle_command_reply: tid=0 2024-05-27T09:37:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 1 client.25642 shutdown 2024-05-27T09:37:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 1 -- 172.21.15.44:0/3512505315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 msgr2=0x55ef92839360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92839360 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec003900 tx=0x7fe7ec009310 comp rx=0 tx=0).stop 2024-05-27T09:37:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 1 -- 172.21.15.44:0/3512505315 shutdown_connections 2024-05-27T09:37:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef92849060 0x55ef9284b4c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9282f5e0 0x55ef92839de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92839360 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3512505315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 0x55ef928398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 1 -- 172.21.15.44:0/3512505315 >> 172.21.15.44:0/3512505315 conn(0x55ef92749490 msgr2=0x55ef928299d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:13.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe7dbe00700 0 client.25642 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:13.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 1 -- 172.21.15.44:0/3512505315 shutdown_connections 2024-05-27T09:37:13.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 1 -- 172.21.15.44:0/3512505315 wait complete. 2024-05-27T09:37:13.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.534+0000 7fe80f310b80 20 client.25642 trim_cache size 0 max 16384 2024-05-27T09:37:13.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.556+0000 7fe80f310b80 1 Processor -- start 2024-05-27T09:37:13.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.557+0000 7fe80f310b80 1 -- start start 2024-05-27T09:37:13.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.557+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92833630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.557+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 0x55ef92834b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.557+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9282f5e0 0x55ef92845210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.557+0000 7fe80f310b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f81b3960 con 0x55ef9279baf0 2024-05-27T09:37:13.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.557+0000 7fe80f310b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f81b11f0 con 0x55ef9282f5e0 2024-05-27T09:37:13.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.557+0000 7fe80f310b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f81af040 con 0x55ef926aff40 2024-05-27T09:37:13.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.557+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92833630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.558+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92833630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49248/0 (socket says 172.21.15.44:49248) 2024-05-27T09:37:13.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.558+0000 7fe7fde00700 1 -- 172.21.15.44:0/4215734646 learned_addr learned my addr 172.21.15.44:0/4215734646 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:13.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.558+0000 7fe7fe800700 1 --2- 172.21.15.44:0/4215734646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9282f5e0 0x55ef92845210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.558+0000 7fe7fd400700 1 --2- 172.21.15.44:0/4215734646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 0x55ef92834b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.558+0000 7fe7fde00700 1 -- 172.21.15.44:0/4215734646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9282f5e0 msgr2=0x55ef92845210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:13.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.558+0000 7fe7fde00700 1 --2- 172.21.15.44:0/4215734646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9282f5e0 0x55ef92845210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.558+0000 7fe7fde00700 1 -- 172.21.15.44:0/4215734646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 msgr2=0x55ef92834b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:13.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.558+0000 7fe7fde00700 1 --2- 172.21.15.44:0/4215734646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 0x55ef92834b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.558+0000 7fe7fde00700 1 -- 172.21.15.44:0/4215734646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef9282b120 con 0x55ef926aff40 2024-05-27T09:37:13.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.558+0000 7fe7fd400700 1 --2- 172.21.15.44:0/4215734646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 0x55ef92834b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:37:13.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.558+0000 7fe7fde00700 1 --2- 172.21.15.44:0/4215734646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92833630 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00f8b0 tx=0x7fe7ec0051b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3ab835493cbef98 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:13.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.559+0000 7fe7e6a00700 1 -- 172.21.15.44:0/4215734646 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec035040 con 0x55ef926aff40 2024-05-27T09:37:13.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.559+0000 7fe7e6a00700 1 -- 172.21.15.44:0/4215734646 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7ec025960 con 0x55ef926aff40 2024-05-27T09:37:13.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.559+0000 7fe7e6a00700 1 -- 172.21.15.44:0/4215734646 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec03ac70 con 0x55ef926aff40 2024-05-27T09:37:13.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.560+0000 7fe80f310b80 1 -- 172.21.15.44:0/4215734646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 msgr2=0x55ef92833630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:13.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.560+0000 7fe80f310b80 1 --2- 172.21.15.44:0/4215734646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92833630 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00f8b0 tx=0x7fe7ec0051b0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.560+0000 7fe80f310b80 1 -- 172.21.15.44:0/4215734646 shutdown_connections 2024-05-27T09:37:13.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.560+0000 7fe80f310b80 1 --2- 172.21.15.44:0/4215734646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9282f5e0 0x55ef92845210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.560+0000 7fe80f310b80 1 --2- 172.21.15.44:0/4215734646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef926aff40 0x55ef92833630 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.560+0000 7fe80f310b80 1 --2- 172.21.15.44:0/4215734646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef9279baf0 0x55ef92834b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.560+0000 7fe80f310b80 1 -- 172.21.15.44:0/4215734646 >> 172.21.15.44:0/4215734646 conn(0x55ef92749490 msgr2=0x55ef92839b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:13.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.560+0000 7fe80f310b80 1 -- 172.21.15.44:0/4215734646 shutdown_connections 2024-05-27T09:37:13.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.560+0000 7fe80f310b80 1 -- 172.21.15.44:0/4215734646 wait complete. 2024-05-27T09:37:13.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.561+0000 7fe80f310b80 1 -- start start 2024-05-27T09:37:13.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.561+0000 7fe80f310b80 1 Processor -- start 2024-05-27T09:37:13.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.561+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 0x55ef9283e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.561+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 0x55ef9283e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.562+0000 7fe80f310b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef9282f5e0 0x55ef9283ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.562+0000 7fe80f310b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f81af6b0 con 0x55ef926aff40 2024-05-27T09:37:13.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.562+0000 7fe80f310b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f81b0b20 con 0x55ef9279baf0 2024-05-27T09:37:13.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.562+0000 7fe80f310b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f81afd80 con 0x55ef9282f5e0 2024-05-27T09:37:13.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.562+0000 7fe80f310b80 10 client.0 _ll_register_callbacks cb 0x55ef926f6d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:13.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.562+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 0x55ef9283e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.562+0000 7fe7fde00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 0x55ef9283e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36236/0 (socket says 172.21.15.44:36236) 2024-05-27T09:37:13.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.562+0000 7fe7fde00700 1 -- 172.21.15.44:0/3036300192 learned_addr learned my addr 172.21.15.44:0/3036300192 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:13.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.562+0000 7fe7fd400700 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 0x55ef9283e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.562+0000 7fe7fe800700 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef9282f5e0 0x55ef9283ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.563+0000 7fe7fde00700 1 -- 172.21.15.44:0/3036300192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef9282f5e0 msgr2=0x55ef9283ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:13.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.563+0000 7fe7fde00700 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef9282f5e0 0x55ef9283ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.563+0000 7fe7fde00700 1 -- 172.21.15.44:0/3036300192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 msgr2=0x55ef9283e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:13.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.563+0000 7fe7fde00700 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 0x55ef9283e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.563+0000 7fe7fde00700 1 -- 172.21.15.44:0/3036300192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef9283c160 con 0x55ef926aff40 2024-05-27T09:37:13.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.563+0000 7fe7fde00700 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 0x55ef9283e3d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec03a890 tx=0x7fe7ec03a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1db1c2628c0ec52 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:13.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.563+0000 7fe7dbe00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:37:13.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.564+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3036300192 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec00fc30 con 0x55ef926aff40 2024-05-27T09:37:13.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.564+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3036300192 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fe7ec00fdd0 con 0x55ef926aff40 2024-05-27T09:37:13.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.564+0000 7fe80f310b80 1 -- 172.21.15.44:0/3036300192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7ec025960 con 0x55ef926aff40 2024-05-27T09:37:13.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.564+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3036300192 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe7ec053050 con 0x55ef926aff40 2024-05-27T09:37:13.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.564+0000 7fe80f310b80 10 client.15504 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:13.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.564+0000 7fe80f310b80 1 -- 172.21.15.44:0/3036300192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ef928bd280 con 0x55ef926aff40 2024-05-27T09:37:13.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.565+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3036300192 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7ec025960 con 0x55ef926aff40 2024-05-27T09:37:13.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.565+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3036300192 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe7ec043a50 con 0x55ef926aff40 2024-05-27T09:37:13.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.565+0000 7fe80f310b80 10 client.15504 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:13.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.565+0000 7fe80f310b80 10 client.15504 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:37:13.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.565+0000 7fe80f310b80 20 client.15504 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:13.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.565+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef928bd8e0 0x55ef928bfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:13.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.565+0000 7fe80f310b80 4 client.15504 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "4295cdb8", "release": true, "await": true}] 2024-05-27T09:37:13.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.565+0000 7fe80f310b80 1 -- 172.21.15.44:0/3036300192 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "set_id": "4295cdb8", "release": true, "await": true}) -- 0x55ef92822810 con 0x55ef928bd8e0 2024-05-27T09:37:13.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.565+0000 7fe7fd400700 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef928bd8e0 0x55ef928bfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:13.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.566+0000 7fe7fd400700 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef928bd8e0 0x55ef928bfd00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8881e5116dc68320 server_cookie=bf353e5d0450b0da in_seq=0 out_seq=0 2024-05-27T09:37:13.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.566+0000 7fe7dbe00700 10 client.15504 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:13.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe7dbe00700 1 -- 172.21.15.44:0/3036300192 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55ef92822810 con 0x55ef928bd8e0 2024-05-27T09:37:13.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe7dbe00700 10 client.15504 handle_command_reply: tid=0 2024-05-27T09:37:13.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe80f310b80 1 client.15504 shutdown 2024-05-27T09:37:13.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe80f310b80 1 -- 172.21.15.44:0/3036300192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 msgr2=0x55ef9283e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:13.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 0x55ef9283e3d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec03a890 tx=0x7fe7ec03a730 comp rx=0 tx=0).stop 2024-05-27T09:37:13.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe80f310b80 1 -- 172.21.15.44:0/3036300192 shutdown_connections 2024-05-27T09:37:13.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef928bd8e0 0x55ef928bfd00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef9282f5e0 0x55ef9283ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef926aff40 0x55ef9283e3d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe80f310b80 1 --2- 172.21.15.44:0/3036300192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef9279baf0 0x55ef9283e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe7dbe00700 0 client.15504 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.568+0000 7fe80f310b80 1 -- 172.21.15.44:0/3036300192 >> 172.21.15.44:0/3036300192 conn(0x55ef92749490 msgr2=0x55ef92834b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.569+0000 7fe80f310b80 1 -- 172.21.15.44:0/3036300192 shutdown_connections 2024-05-27T09:37:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.569+0000 7fe80f310b80 1 -- 172.21.15.44:0/3036300192 wait complete. 2024-05-27T09:37:13.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.569+0000 7fe80f310b80 20 client.15504 trim_cache size 0 max 16384 2024-05-27T09:37:13.596 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:37:13.596 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 26, 2024-05-27T09:37:13.596 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:37:13.596 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 4, 2024-05-27T09:37:13.596 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:37:13.596 INFO:teuthology.orchestra.run.smithi044.stdout: "4295cdb8": { 2024-05-27T09:37:13.596 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 4, 2024-05-27T09:37:13.596 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:37:13.596 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:37:13.596 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 149.4, 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:37:13.597 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:37:13.598 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:37:13.598 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:37:13.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.570+0000 7fe800800700 1 -- 172.21.15.44:0/855749993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe7e00b2a70 msgr2=0x7fe7e00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:13.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.570+0000 7fe800800700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe7e00b2a70 0x7fe7e00b4f80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009750 tx=0x7fe7ec005040 comp rx=0 tx=0).stop 2024-05-27T09:37:13.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.570+0000 7fe800800700 1 -- 172.21.15.44:0/855749993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f8076440 msgr2=0x7fe7f819d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.570+0000 7fe800800700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f8076440 0x7fe7f819d590 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800e9f0 tx=0x7fe7e800ed00 comp rx=0 tx=0).stop 2024-05-27T09:37:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.570+0000 7fe800800700 1 -- 172.21.15.44:0/855749993 shutdown_connections 2024-05-27T09:37:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.570+0000 7fe800800700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe7e00b2a70 0x7fe7e00b4f80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.570+0000 7fe800800700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7f8110c30 0x7fe7f81a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.570+0000 7fe800800700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f8075af0 0x7fe7f819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.570+0000 7fe800800700 1 --2- 172.21.15.44:0/855749993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7f8076440 0x7fe7f819d590 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.570+0000 7fe800800700 1 -- 172.21.15.44:0/855749993 >> 172.21.15.44:0/855749993 conn(0x7fe7f8100090 msgr2=0x7fe7f8100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.570+0000 7fe800800700 1 -- 172.21.15.44:0/855749993 shutdown_connections 2024-05-27T09:37:13.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:13.571+0000 7fe800800700 1 -- 172.21.15.44:0/855749993 wait complete. 2024-05-27T09:37:14.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:14 smithi044 ceph-mon[31618]: pgmap v63: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 767 B/s wr, 14 op/s 2024-05-27T09:37:14.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:14 smithi050 ceph-mon[36371]: pgmap v63: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 767 B/s wr, 14 op/s 2024-05-27T09:37:14.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:14 smithi189 ceph-mon[36671]: pgmap v63: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 767 B/s wr, 14 op/s 2024-05-27T09:37:14.860 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '4295cdb8', total seconds elapsed: 37.6 2024-05-27T09:37:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.511+0000 7f7c9e000700 1 -- 172.21.15.44:0/794857379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c98105280 msgr2=0x7f7c98105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.511+0000 7f7c9e000700 1 --2- 172.21.15.44:0/794857379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c98105280 0x7f7c98105690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0098f0 tx=0x7f7c8c009c00 comp rx=0 tx=0).stop 2024-05-27T09:37:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.511+0000 7f7c9e000700 1 -- 172.21.15.44:0/794857379 shutdown_connections 2024-05-27T09:37:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.511+0000 7f7c9e000700 1 --2- 172.21.15.44:0/794857379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c9810e6f0 0x7f7c98110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.511+0000 7f7c9e000700 1 --2- 172.21.15.44:0/794857379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c98105280 0x7f7c98105690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.511+0000 7f7c9e000700 1 --2- 172.21.15.44:0/794857379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c98105bd0 0x7f7c9810e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.511+0000 7f7c9e000700 1 -- 172.21.15.44:0/794857379 >> 172.21.15.44:0/794857379 conn(0x7f7c98078ab0 msgr2=0x7f7c98078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:15.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.511+0000 7f7c9e000700 1 -- 172.21.15.44:0/794857379 shutdown_connections 2024-05-27T09:37:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.511+0000 7f7c9e000700 1 -- 172.21.15.44:0/794857379 wait complete. 2024-05-27T09:37:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.513+0000 7f7c9e000700 1 Processor -- start 2024-05-27T09:37:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.513+0000 7f7c9e000700 1 -- start start 2024-05-27T09:37:15.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.513+0000 7f7c9e000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c98105280 0x7f7c9819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.513+0000 7f7c9e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c98105bd0 0x7f7c9819d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.513+0000 7f7c9e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c9810e6f0 0x7f7c981a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.514+0000 7f7c9e000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7c98114290 con 0x7f7c98105bd0 2024-05-27T09:37:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.514+0000 7f7c9e000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7c98114110 con 0x7f7c98105280 2024-05-27T09:37:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.514+0000 7f7c9e000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c98114410 con 0x7f7c9810e6f0 2024-05-27T09:37:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.514+0000 7f7c9d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c9810e6f0 0x7f7c981a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.514+0000 7f7c9cc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c98105280 0x7f7c9819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.514+0000 7f7c9cc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c98105280 0x7f7c9819cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50480/0 (socket says 172.21.15.44:50480) 2024-05-27T09:37:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.514+0000 7f7c9cc00700 1 -- 172.21.15.44:0/2534821045 learned_addr learned my addr 172.21.15.44:0/2534821045 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:15.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.514+0000 7f7c97e00700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c98105bd0 0x7f7c9819d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.515+0000 7f7c9cc00700 1 -- 172.21.15.44:0/2534821045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c9810e6f0 msgr2=0x7f7c981a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.515+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c9810e6f0 0x7f7c981a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.515+0000 7f7c9cc00700 1 -- 172.21.15.44:0/2534821045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c98105bd0 msgr2=0x7f7c9819d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.515+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c98105bd0 0x7f7c9819d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.515+0000 7f7c9cc00700 1 -- 172.21.15.44:0/2534821045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c981a1eb0 con 0x7f7c98105280 2024-05-27T09:37:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.515+0000 7f7c97e00700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c98105bd0 0x7f7c9819d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:15.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.515+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c98105280 0x7f7c9819cfe0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009230 tx=0x7f7c8c00fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.516+0000 7f7c95600700 1 -- 172.21.15.44:0/2534821045 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c005140 con 0x7f7c98105280 2024-05-27T09:37:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.516+0000 7f7c9e000700 1 -- 172.21.15.44:0/2534821045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c981a2140 con 0x7f7c98105280 2024-05-27T09:37:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.516+0000 7f7c95600700 1 -- 172.21.15.44:0/2534821045 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c8c00b560 con 0x7f7c98105280 2024-05-27T09:37:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.516+0000 7f7c95600700 1 -- 172.21.15.44:0/2534821045 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c003ae0 con 0x7f7c98105280 2024-05-27T09:37:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.516+0000 7f7c9e000700 1 -- 172.21.15.44:0/2534821045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c981a9ac0 con 0x7f7c98105280 2024-05-27T09:37:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.520+0000 7f7c95600700 1 -- 172.21.15.44:0/2534821045 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f7c8c003c80 con 0x7f7c98105280 2024-05-27T09:37:15.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.522+0000 7f7c95600700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c800b2af0 0x7f7c800b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.522+0000 7f7c95600700 1 -- 172.21.15.44:0/2534821045 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f7c8c0e35c0 con 0x7f7c98105280 2024-05-27T09:37:15.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.522+0000 7f7c97e00700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c800b2af0 0x7f7c800b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.523+0000 7f7cae128b80 1 Processor -- start 2024-05-27T09:37:15.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.523+0000 7f7cae128b80 1 -- start start 2024-05-27T09:37:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.523+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174bc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.524+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56301733ce40 0x5630174bcc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.524+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.524+0000 7f7cae128b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5630173e9e40 con 0x5630174289f0 2024-05-27T09:37:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.524+0000 7f7cae128b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5630173d99e0 con 0x56301733ce40 2024-05-27T09:37:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.524+0000 7f7c97e00700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c800b2af0 0x7f7c800b5000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7c981026f0 tx=0x7f7c84009310 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.524+0000 7f7cae128b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5630173dda20 con 0x5630174c62c0 2024-05-27T09:37:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.524+0000 7f7c9d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.524+0000 7f7c9d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174c86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49284/0 (socket says 172.21.15.44:49284) 2024-05-27T09:37:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.524+0000 7f7c9d600700 1 -- 172.21.15.44:0/2982432807 learned_addr learned my addr 172.21.15.44:0/2982432807 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:15.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.524+0000 7f7c9cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174bc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.524+0000 7f7c97e00700 1 --2- 172.21.15.44:0/2982432807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56301733ce40 0x5630174bcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.525+0000 7f7c9cc00700 1 -- 172.21.15.44:0/2982432807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 msgr2=0x5630174c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.525+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/2982432807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174c86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.525+0000 7f7c9cc00700 1 -- 172.21.15.44:0/2982432807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56301733ce40 msgr2=0x5630174bcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.525+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/2982432807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56301733ce40 0x5630174bcc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.525+0000 7f7c9cc00700 1 -- 172.21.15.44:0/2982432807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630173a2420 con 0x5630174289f0 2024-05-27T09:37:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.525+0000 7f7c97e00700 1 --2- 172.21.15.44:0/2982432807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56301733ce40 0x5630174bcc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.526+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/2982432807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174bc750 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0e0ce0 tx=0x7f7c8c009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2309a6ee9136f3ae server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.526+0000 7f7c8a000700 1 -- 172.21.15.44:0/2982432807 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c0aecb0 con 0x5630174289f0 2024-05-27T09:37:15.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.526+0000 7f7c8a000700 1 -- 172.21.15.44:0/2982432807 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c8c0b36d0 con 0x5630174289f0 2024-05-27T09:37:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.526+0000 7f7c8a000700 1 -- 172.21.15.44:0/2982432807 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c0e0f60 con 0x5630174289f0 2024-05-27T09:37:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.528+0000 7f7cae128b80 1 -- 172.21.15.44:0/2982432807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 msgr2=0x5630174bc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.528+0000 7f7cae128b80 1 --2- 172.21.15.44:0/2982432807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174bc750 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0e0ce0 tx=0x7f7c8c009610 comp rx=0 tx=0).stop 2024-05-27T09:37:15.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.528+0000 7f7cae128b80 1 -- 172.21.15.44:0/2982432807 shutdown_connections 2024-05-27T09:37:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.528+0000 7f7cae128b80 1 --2- 172.21.15.44:0/2982432807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.528+0000 7f7cae128b80 1 --2- 172.21.15.44:0/2982432807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174bc750 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.528+0000 7f7cae128b80 1 --2- 172.21.15.44:0/2982432807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56301733ce40 0x5630174bcc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.528+0000 7f7cae128b80 1 -- 172.21.15.44:0/2982432807 >> 172.21.15.44:0/2982432807 conn(0x5630173d6390 msgr2=0x5630174b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:15.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.528+0000 7f7cae128b80 1 -- 172.21.15.44:0/2982432807 shutdown_connections 2024-05-27T09:37:15.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.528+0000 7f7cae128b80 1 -- 172.21.15.44:0/2982432807 wait complete. 2024-05-27T09:37:15.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.529+0000 7f7cae128b80 1 -- start start 2024-05-27T09:37:15.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.529+0000 7f7cae128b80 1 Processor -- start 2024-05-27T09:37:15.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56301733ce40 0x5630174c3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174c3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174c3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7cae128b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5630172dffa0 con 0x5630174289f0 2024-05-27T09:37:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7cae128b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56301737a760 con 0x56301733ce40 2024-05-27T09:37:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7cae128b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5630174c8c20 con 0x5630174c62c0 2024-05-27T09:37:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7cae128b80 10 client.0 _ll_register_callbacks cb 0x563017383c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:15.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7c9d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174c3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7c9d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174c3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49288/0 (socket says 172.21.15.44:49288) 2024-05-27T09:37:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7c9d600700 1 -- 172.21.15.44:0/3274140368 learned_addr learned my addr 172.21.15.44:0/3274140368 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56301733ce40 0x5630174c3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.530+0000 7f7c97e00700 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.531+0000 7f7c9d600700 1 -- 172.21.15.44:0/3274140368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56301733ce40 msgr2=0x5630174c3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.531+0000 7f7c9d600700 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56301733ce40 0x5630174c3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.531+0000 7f7c9d600700 1 -- 172.21.15.44:0/3274140368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 msgr2=0x5630174c3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.531+0000 7f7c9d600700 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174c3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.531+0000 7f7c9d600700 1 -- 172.21.15.44:0/3274140368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630173298d0 con 0x5630174c62c0 2024-05-27T09:37:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.531+0000 7f7c97e00700 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174c3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:37:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.531+0000 7f7c9d600700 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174c3da0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7c900047d0 tx=0x7f7c90014600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f67c311d4d0c19ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.531+0000 7f7c7ea00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:37:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.532+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3274140368 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c90014b30 con 0x5630174c62c0 2024-05-27T09:37:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.532+0000 7f7cae128b80 1 -- 172.21.15.44:0/3274140368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c8c0b36d0 con 0x5630174c62c0 2024-05-27T09:37:15.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.533+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3274140368 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f7c90014cd0 con 0x5630174c62c0 2024-05-27T09:37:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.533+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3274140368 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7c90007500 con 0x5630174c62c0 2024-05-27T09:37:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.533+0000 7f7cae128b80 10 client.25660 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.533+0000 7f7cae128b80 1 -- 172.21.15.44:0/3274140368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5630173aa060 con 0x5630174c62c0 2024-05-27T09:37:15.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.534+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3274140368 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c8c0b36d0 con 0x5630174c62c0 2024-05-27T09:37:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.534+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3274140368 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7c9001e4b0 con 0x5630174c62c0 2024-05-27T09:37:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.534+0000 7f7cae128b80 10 client.25660 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.534+0000 7f7cae128b80 10 client.25660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.534+0000 7f7cae128b80 20 client.25660 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.534+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5630174d5bd0 0x5630174d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.534+0000 7f7cae128b80 4 client.25660 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:37:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.534+0000 7f7cae128b80 1 -- 172.21.15.44:0/3274140368 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630173ebd90 con 0x5630174d5bd0 2024-05-27T09:37:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.535+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5630174d5bd0 0x5630174d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.535+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5630174d5bd0 0x5630174d7ff0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b78bb4978bd800c server_cookie=d23831bc05b4fc77 in_seq=0 out_seq=0 2024-05-27T09:37:15.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.535+0000 7f7c7ea00700 10 client.25660 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.536+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3274140368 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5630173ebd90 con 0x5630174d5bd0 2024-05-27T09:37:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.536+0000 7f7c7ea00700 10 client.25660 handle_command_reply: tid=0 2024-05-27T09:37:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.536+0000 7f7cae128b80 1 client.25660 shutdown 2024-05-27T09:37:15.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.537+0000 7f7cae128b80 1 -- 172.21.15.44:0/3274140368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 msgr2=0x5630174c3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:15.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.537+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174c3da0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7c900047d0 tx=0x7f7c90014600 comp rx=0 tx=0).stop 2024-05-27T09:37:15.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.537+0000 7f7cae128b80 1 -- 172.21.15.44:0/3274140368 shutdown_connections 2024-05-27T09:37:15.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.537+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5630174d5bd0 0x5630174d7ff0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.537+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174c3da0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.537+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56301733ce40 0x5630174c3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.537+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3274140368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174c3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.537+0000 7f7cae128b80 1 -- 172.21.15.44:0/3274140368 >> 172.21.15.44:0/3274140368 conn(0x5630173d6390 msgr2=0x5630174bc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:15.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.537+0000 7f7c7ea00700 0 client.25660 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:15.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.537+0000 7f7cae128b80 1 -- 172.21.15.44:0/3274140368 shutdown_connections 2024-05-27T09:37:15.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.537+0000 7f7cae128b80 1 -- 172.21.15.44:0/3274140368 wait complete. 2024-05-27T09:37:15.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.538+0000 7f7cae128b80 20 client.25660 trim_cache size 0 max 16384 2024-05-27T09:37:15.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.559+0000 7f7cae128b80 1 Processor -- start 2024-05-27T09:37:15.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.560+0000 7f7cae128b80 1 -- start start 2024-05-27T09:37:15.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.560+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301733ce40 0x5630174bbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.560+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5630174289f0 0x5630174b7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.560+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174be780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.560+0000 7f7cae128b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7c981b3710 con 0x56301733ce40 2024-05-27T09:37:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.560+0000 7f7cae128b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7c981b0fa0 con 0x5630174289f0 2024-05-27T09:37:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.560+0000 7f7cae128b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c981aee10 con 0x5630174c62c0 2024-05-27T09:37:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.561+0000 7f7c9cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301733ce40 0x5630174bbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.561+0000 7f7c9cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301733ce40 0x5630174bbb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36270/0 (socket says 172.21.15.44:36270) 2024-05-27T09:37:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.561+0000 7f7c9cc00700 1 -- 172.21.15.44:0/2211161783 learned_addr learned my addr 172.21.15.44:0/2211161783 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.561+0000 7f7c9d600700 1 --2- 172.21.15.44:0/2211161783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174be780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.561+0000 7f7c9cc00700 1 -- 172.21.15.44:0/2211161783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 msgr2=0x5630174be780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.561+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/2211161783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174be780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.561+0000 7f7c9cc00700 1 -- 172.21.15.44:0/2211161783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5630174289f0 msgr2=0x5630174b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:15.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.561+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/2211161783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5630174289f0 0x5630174b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.561+0000 7f7c9cc00700 1 -- 172.21.15.44:0/2211161783 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630174c6060 con 0x56301733ce40 2024-05-27T09:37:15.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.561+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/2211161783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301733ce40 0x5630174bbb50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009680 tx=0x7f7c8c0a70d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c945b7aadb2c186 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:15.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.562+0000 7f7c89600700 1 -- 172.21.15.44:0/2211161783 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c00ba30 con 0x56301733ce40 2024-05-27T09:37:15.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.562+0000 7f7c89600700 1 -- 172.21.15.44:0/2211161783 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c8c0a7ad0 con 0x56301733ce40 2024-05-27T09:37:15.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.562+0000 7f7c89600700 1 -- 172.21.15.44:0/2211161783 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c0052e0 con 0x56301733ce40 2024-05-27T09:37:15.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.563+0000 7f7cae128b80 1 -- 172.21.15.44:0/2211161783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301733ce40 msgr2=0x5630174bbb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:15.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.563+0000 7f7cae128b80 1 --2- 172.21.15.44:0/2211161783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301733ce40 0x5630174bbb50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009680 tx=0x7f7c8c0a70d0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.563+0000 7f7cae128b80 1 -- 172.21.15.44:0/2211161783 shutdown_connections 2024-05-27T09:37:15.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.563+0000 7f7cae128b80 1 --2- 172.21.15.44:0/2211161783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630174c62c0 0x5630174be780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.563+0000 7f7cae128b80 1 --2- 172.21.15.44:0/2211161783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301733ce40 0x5630174bbb50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.563+0000 7f7cae128b80 1 --2- 172.21.15.44:0/2211161783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5630174289f0 0x5630174b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.563+0000 7f7cae128b80 1 -- 172.21.15.44:0/2211161783 >> 172.21.15.44:0/2211161783 conn(0x5630173d6390 msgr2=0x5630174be070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:15.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.563+0000 7f7cae128b80 1 -- 172.21.15.44:0/2211161783 shutdown_connections 2024-05-27T09:37:15.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.563+0000 7f7cae128b80 1 -- 172.21.15.44:0/2211161783 wait complete. 2024-05-27T09:37:15.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.564+0000 7f7cae128b80 1 -- start start 2024-05-27T09:37:15.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.564+0000 7f7cae128b80 1 Processor -- start 2024-05-27T09:37:15.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.565+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56301733ce40 0x5630174ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.565+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.565+0000 7f7cae128b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5630174c62c0 0x5630174c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.565+0000 7f7cae128b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7c981af460 con 0x5630174289f0 2024-05-27T09:37:15.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.565+0000 7f7cae128b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7c981b08d0 con 0x5630174c62c0 2024-05-27T09:37:15.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.565+0000 7f7cae128b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c981afb30 con 0x56301733ce40 2024-05-27T09:37:15.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.565+0000 7f7cae128b80 10 client.0 _ll_register_callbacks cb 0x563017383c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:15.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.565+0000 7f7c9cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56301733ce40 0x5630174ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.566+0000 7f7c9cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56301733ce40 0x5630174ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49310/0 (socket says 172.21.15.44:49310) 2024-05-27T09:37:15.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.566+0000 7f7c9cc00700 1 -- 172.21.15.44:0/3736271062 learned_addr learned my addr 172.21.15.44:0/3736271062 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:15.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.566+0000 7f7c97e00700 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.566+0000 7f7c9d600700 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5630174c62c0 0x5630174c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.566+0000 7f7c9cc00700 1 -- 172.21.15.44:0/3736271062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5630174c62c0 msgr2=0x5630174c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:15.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.566+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5630174c62c0 0x5630174c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.566+0000 7f7c9cc00700 1 -- 172.21.15.44:0/3736271062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 msgr2=0x5630174c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:15.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.566+0000 7f7c9d600700 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5630174c62c0 0x5630174c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:37:15.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.566+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.566+0000 7f7c9cc00700 1 -- 172.21.15.44:0/3736271062 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630174d35a0 con 0x56301733ce40 2024-05-27T09:37:15.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.567+0000 7f7c97e00700 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:15.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.567+0000 7f7c9cc00700 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56301733ce40 0x5630174ce340 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c005950 tx=0x7f7c8c0cd300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=784a6d235b5e7a65 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:15.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.567+0000 7f7c7ea00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:37:15.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.567+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3736271062 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c0b36d0 con 0x56301733ce40 2024-05-27T09:37:15.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.567+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3736271062 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f7c8c00ba30 con 0x56301733ce40 2024-05-27T09:37:15.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.567+0000 7f7cae128b80 1 -- 172.21.15.44:0/3736271062 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c8c0a7ad0 con 0x56301733ce40 2024-05-27T09:37:15.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.567+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3736271062 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7c8c0aac60 con 0x56301733ce40 2024-05-27T09:37:15.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.567+0000 7f7cae128b80 10 client.25666 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:15.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.568+0000 7f7cae128b80 1 -- 172.21.15.44:0/3736271062 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563017548ae0 con 0x56301733ce40 2024-05-27T09:37:15.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.568+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3736271062 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c8c0a7ad0 con 0x56301733ce40 2024-05-27T09:37:15.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.569+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3736271062 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7c8c0b3240 con 0x56301733ce40 2024-05-27T09:37:15.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.569+0000 7f7cae128b80 10 client.25666 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:15.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.569+0000 7f7cae128b80 10 client.25666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:15.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.569+0000 7f7cae128b80 20 client.25666 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:15.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.569+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563017548d10 0x56301754b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:15.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.569+0000 7f7cae128b80 4 client.25666 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:37:15.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.569+0000 7f7cae128b80 1 -- 172.21.15.44:0/3736271062 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5630173e5570 con 0x563017548d10 2024-05-27T09:37:15.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.570+0000 7f7c97e00700 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563017548d10 0x56301754b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:15.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.570+0000 7f7c97e00700 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563017548d10 0x56301754b130 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=271cce6a8bf4d5b4 server_cookie=82dea61dc35f6ecf in_seq=0 out_seq=0 2024-05-27T09:37:15.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.570+0000 7f7c7ea00700 10 client.25666 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:15.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.571+0000 7f7c7ea00700 1 -- 172.21.15.44:0/3736271062 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5630173e5570 con 0x563017548d10 2024-05-27T09:37:15.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.571+0000 7f7c7ea00700 10 client.25666 handle_command_reply: tid=0 2024-05-27T09:37:15.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.571+0000 7f7cae128b80 1 client.25666 shutdown 2024-05-27T09:37:15.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.571+0000 7f7cae128b80 1 -- 172.21.15.44:0/3736271062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56301733ce40 msgr2=0x5630174ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:15.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.571+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56301733ce40 0x5630174ce340 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c005950 tx=0x7f7c8c0cd300 comp rx=0 tx=0).stop 2024-05-27T09:37:15.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.571+0000 7f7cae128b80 1 -- 172.21.15.44:0/3736271062 shutdown_connections 2024-05-27T09:37:15.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.571+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563017548d10 0x56301754b130 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.572+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5630174c62c0 0x5630174c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.572+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56301733ce40 0x5630174ce340 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.572+0000 7f7cae128b80 1 --2- 172.21.15.44:0/3736271062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5630174289f0 0x5630174c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.572+0000 7f7c7ea00700 0 client.25666 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:15.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.572+0000 7f7cae128b80 1 -- 172.21.15.44:0/3736271062 >> 172.21.15.44:0/3736271062 conn(0x5630173d6390 msgr2=0x5630174bb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:15.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.572+0000 7f7cae128b80 1 -- 172.21.15.44:0/3736271062 shutdown_connections 2024-05-27T09:37:15.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.572+0000 7f7cae128b80 1 -- 172.21.15.44:0/3736271062 wait complete. 2024-05-27T09:37:15.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.572+0000 7f7cae128b80 20 client.25666 trim_cache size 0 max 16384 2024-05-27T09:37:15.574 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:37:15.575 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:37:15.575 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:37:15.575 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:37:15.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.573+0000 7f7c9e000700 1 -- 172.21.15.44:0/2534821045 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c800b2af0 msgr2=0x7f7c800b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:15.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.573+0000 7f7c9e000700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c800b2af0 0x7f7c800b5000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7c981026f0 tx=0x7f7c84009310 comp rx=0 tx=0).stop 2024-05-27T09:37:15.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.573+0000 7f7c9e000700 1 -- 172.21.15.44:0/2534821045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c98105280 msgr2=0x7f7c9819cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:15.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.573+0000 7f7c9e000700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c98105280 0x7f7c9819cfe0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009230 tx=0x7f7c8c00fa70 comp rx=0 tx=0).stop 2024-05-27T09:37:15.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.573+0000 7f7c9e000700 1 -- 172.21.15.44:0/2534821045 shutdown_connections 2024-05-27T09:37:15.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.573+0000 7f7c9e000700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c800b2af0 0x7f7c800b5000 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.573+0000 7f7c9e000700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c9810e6f0 0x7f7c981a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.573+0000 7f7c9e000700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c98105280 0x7f7c9819cfe0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.573+0000 7f7c9e000700 1 --2- 172.21.15.44:0/2534821045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c98105bd0 0x7f7c9819d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:15.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.573+0000 7f7c9e000700 1 -- 172.21.15.44:0/2534821045 >> 172.21.15.44:0/2534821045 conn(0x7f7c98078ab0 msgr2=0x7f7c980ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:15.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.574+0000 7f7c9e000700 1 -- 172.21.15.44:0/2534821045 shutdown_connections 2024-05-27T09:37:15.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:15.574+0000 7f7c9e000700 1 -- 172.21.15.44:0/2534821045 wait complete. 2024-05-27T09:37:16.183 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:16 smithi044 ceph-mon[31618]: pgmap v64: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 767 B/s wr, 17 op/s 2024-05-27T09:37:16.427 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:37:16.427 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:37:16.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:16 smithi050 ceph-mon[36371]: pgmap v64: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 767 B/s wr, 17 op/s 2024-05-27T09:37:16.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:16 smithi189 ceph-mon[36671]: pgmap v64: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 767 B/s wr, 17 op/s 2024-05-27T09:37:16.900 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:37:18.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:18 smithi044 ceph-mon[31618]: pgmap v65: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-27T09:37:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:18 smithi050 ceph-mon[36371]: pgmap v65: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-27T09:37:18.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:18 smithi189 ceph-mon[36671]: pgmap v65: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-27T09:37:19.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.044+0000 7fe3bb200700 1 -- 172.21.15.44:0/3640637077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3b4105bd0 msgr2=0x7fe3b410e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:19.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.044+0000 7fe3bb200700 1 --2- 172.21.15.44:0/3640637077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3b4105bd0 0x7fe3b410e120 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40098f0 tx=0x7fe3a4009c00 comp rx=0 tx=0).stop 2024-05-27T09:37:19.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.044+0000 7fe3bb200700 1 -- 172.21.15.44:0/3640637077 shutdown_connections 2024-05-27T09:37:19.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.044+0000 7fe3bb200700 1 --2- 172.21.15.44:0/3640637077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3b410e6f0 0x7fe3b4110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:19.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.044+0000 7fe3bb200700 1 --2- 172.21.15.44:0/3640637077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b4105280 0x7fe3b4105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:19.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.044+0000 7fe3bb200700 1 --2- 172.21.15.44:0/3640637077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3b4105bd0 0x7fe3b410e120 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:19.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.044+0000 7fe3bb200700 1 -- 172.21.15.44:0/3640637077 >> 172.21.15.44:0/3640637077 conn(0x7fe3b4078ab0 msgr2=0x7fe3b4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:19.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.044+0000 7fe3bb200700 1 -- 172.21.15.44:0/3640637077 shutdown_connections 2024-05-27T09:37:19.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.044+0000 7fe3bb200700 1 -- 172.21.15.44:0/3640637077 wait complete. 2024-05-27T09:37:19.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.046+0000 7fe3bb200700 1 Processor -- start 2024-05-27T09:37:19.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.046+0000 7fe3bb200700 1 -- start start 2024-05-27T09:37:19.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.046+0000 7fe3bb200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3b4105280 0x7fe3b41a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:19.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.046+0000 7fe3bb200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b4105bd0 0x7fe3b41a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:19.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.046+0000 7fe3bb200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3b410e6f0 0x7fe3b419aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:19.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.046+0000 7fe3bb200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b4114250 con 0x7fe3b4105280 2024-05-27T09:37:19.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.046+0000 7fe3bb200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b41140d0 con 0x7fe3b410e6f0 2024-05-27T09:37:19.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.046+0000 7fe3bb200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b41143d0 con 0x7fe3b4105bd0 2024-05-27T09:37:19.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.047+0000 7fe3b9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3b4105280 0x7fe3b41a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:19.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.047+0000 7fe3b9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3b4105280 0x7fe3b41a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36294/0 (socket says 172.21.15.44:36294) 2024-05-27T09:37:19.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.047+0000 7fe3b9e00700 1 -- 172.21.15.44:0/633406805 learned_addr learned my addr 172.21.15.44:0/633406805 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:19.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.047+0000 7fe3b9400700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b4105bd0 0x7fe3b41a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:19.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.047+0000 7fe3ba800700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3b410e6f0 0x7fe3b419aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:19.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.047+0000 7fe3b9e00700 1 -- 172.21.15.44:0/633406805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b4105bd0 msgr2=0x7fe3b41a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:19.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.047+0000 7fe3b9e00700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b4105bd0 0x7fe3b41a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:19.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.047+0000 7fe3b9e00700 1 -- 172.21.15.44:0/633406805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3b410e6f0 msgr2=0x7fe3b419aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:19.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.047+0000 7fe3b9e00700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3b410e6f0 0x7fe3b419aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:19.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.047+0000 7fe3b9e00700 1 -- 172.21.15.44:0/633406805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b419b120 con 0x7fe3b4105280 2024-05-27T09:37:19.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.048+0000 7fe3b9e00700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3b4105280 0x7fe3b41a0850 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800e9c0 tx=0x7fe3a800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.048+0000 7fe3aea00700 1 -- 172.21.15.44:0/633406805 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8004a30 con 0x7fe3b4105280 2024-05-27T09:37:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.048+0000 7fe3bb200700 1 -- 172.21.15.44:0/633406805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3b419b410 con 0x7fe3b4105280 2024-05-27T09:37:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.049+0000 7fe3bb200700 1 -- 172.21.15.44:0/633406805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3b4102ea0 con 0x7fe3b4105280 2024-05-27T09:37:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.049+0000 7fe3aea00700 1 -- 172.21.15.44:0/633406805 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3a800cc70 con 0x7fe3b4105280 2024-05-27T09:37:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.049+0000 7fe3aea00700 1 -- 172.21.15.44:0/633406805 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a80054a0 con 0x7fe3b4105280 2024-05-27T09:37:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.051+0000 7fe3bb200700 1 -- 172.21.15.44:0/633406805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe398005330 con 0x7fe3b4105280 2024-05-27T09:37:19.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.053+0000 7fe3aea00700 1 -- 172.21.15.44:0/633406805 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fe3a8004bd0 con 0x7fe3b4105280 2024-05-27T09:37:19.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.055+0000 7fe3aea00700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe39c0b2a70 0x7fe39c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:19.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.055+0000 7fe3aea00700 1 -- 172.21.15.44:0/633406805 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fe3a80e2640 con 0x7fe3b4105280 2024-05-27T09:37:19.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.056+0000 7fe3b9400700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe39c0b2a70 0x7fe39c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:19.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.057+0000 7fe3b9400700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe39c0b2a70 0x7fe39c0b4f80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4009fd0 tx=0x7fe3a401d040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:19.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.065+0000 7fe3aea00700 1 -- 172.21.15.44:0/633406805 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe3a800ce20 con 0x7fe3b4105280 2024-05-27T09:37:19.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.361+0000 7fe3bb200700 1 -- 172.21.15.44:0/633406805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3980058f0 con 0x7fe3b4105280 2024-05-27T09:37:19.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.362+0000 7fe3aea00700 1 -- 172.21.15.44:0/633406805 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7fe3a80a9ff0 con 0x7fe3b4105280 2024-05-27T09:37:19.363 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:37:19.363 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:37:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.365+0000 7fe3bb200700 1 -- 172.21.15.44:0/633406805 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe39c0b2a70 msgr2=0x7fe39c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.365+0000 7fe3bb200700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe39c0b2a70 0x7fe39c0b4f80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4009fd0 tx=0x7fe3a401d040 comp rx=0 tx=0).stop 2024-05-27T09:37:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.365+0000 7fe3bb200700 1 -- 172.21.15.44:0/633406805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3b4105280 msgr2=0x7fe3b41a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.365+0000 7fe3bb200700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3b4105280 0x7fe3b41a0850 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800e9c0 tx=0x7fe3a800ecd0 comp rx=0 tx=0).stop 2024-05-27T09:37:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.365+0000 7fe3bb200700 1 -- 172.21.15.44:0/633406805 shutdown_connections 2024-05-27T09:37:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.365+0000 7fe3bb200700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe39c0b2a70 0x7fe39c0b4f80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.365+0000 7fe3bb200700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3b410e6f0 0x7fe3b419aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:19.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.365+0000 7fe3bb200700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe3b4105280 0x7fe3b41a0850 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:19.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.365+0000 7fe3bb200700 1 --2- 172.21.15.44:0/633406805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b4105bd0 0x7fe3b41a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:19.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.365+0000 7fe3bb200700 1 -- 172.21.15.44:0/633406805 >> 172.21.15.44:0/633406805 conn(0x7fe3b4078ab0 msgr2=0x7fe3b40fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:19.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.366+0000 7fe3bb200700 1 -- 172.21.15.44:0/633406805 shutdown_connections 2024-05-27T09:37:19.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:19.366+0000 7fe3bb200700 1 -- 172.21.15.44:0/633406805 wait complete. 2024-05-27T09:37:19.368 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:37:19.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:19 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/633406805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:37:19.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:19 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/633406805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:37:19.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:19 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/633406805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:37:20.154 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:37:20.589 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:37:20.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:20 smithi044 ceph-mon[31618]: pgmap v66: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 8 op/s 2024-05-27T09:37:20.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:20 smithi050 ceph-mon[36371]: pgmap v66: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 8 op/s 2024-05-27T09:37:20.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:20 smithi189 ceph-mon[36671]: pgmap v66: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 8 op/s 2024-05-27T09:37:21.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.663+0000 7fa0ed400640 1 -- 172.21.15.44:0/3872891822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0e8104220 msgr2=0x7fa0e81105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.663+0000 7fa0ed400640 1 --2- 172.21.15.44:0/3872891822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0e8104220 0x7fa0e81105f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009be0 tx=0x7fa0e002a6c0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.664+0000 7fa0ed400640 1 -- 172.21.15.44:0/3872891822 shutdown_connections 2024-05-27T09:37:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.664+0000 7fa0ed400640 1 --2- 172.21.15.44:0/3872891822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0e8104220 0x7fa0e81105f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.664+0000 7fa0ed400640 1 --2- 172.21.15.44:0/3872891822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0e8109f70 0x7fa0e8103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.664+0000 7fa0ed400640 1 --2- 172.21.15.44:0/3872891822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0e8109ad0 0x7fa0e8103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.664+0000 7fa0ed400640 1 -- 172.21.15.44:0/3872891822 >> 172.21.15.44:0/3872891822 conn(0x7fa0e8079100 msgr2=0x7fa0e8079540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.664+0000 7fa0ed400640 1 -- 172.21.15.44:0/3872891822 shutdown_connections 2024-05-27T09:37:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.664+0000 7fa0ed400640 1 -- 172.21.15.44:0/3872891822 wait complete. 2024-05-27T09:37:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.666+0000 7fa0ed400640 1 Processor -- start 2024-05-27T09:37:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.666+0000 7fa0ed400640 1 -- start start 2024-05-27T09:37:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.666+0000 7fa0ed400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0e8104220 0x7fa0e819a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.666+0000 7fa0ed400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0e8109ad0 0x7fa0e819aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.666+0000 7fa0ed400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0e8109f70 0x7fa0e819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.666+0000 7fa0ed400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e8113ce0 con 0x7fa0e8109f70 2024-05-27T09:37:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.666+0000 7fa0ed400640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e8113b60 con 0x7fa0e8109ad0 2024-05-27T09:37:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.666+0000 7fa0ed400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e8113e60 con 0x7fa0e8104220 2024-05-27T09:37:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.667+0000 7fa0eca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0e8109f70 0x7fa0e819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.667+0000 7fa0eca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0e8109f70 0x7fa0e819ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36316/0 (socket says 172.21.15.44:36316) 2024-05-27T09:37:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.667+0000 7fa0eca00640 1 -- 172.21.15.44:0/3124991029 learned_addr learned my addr 172.21.15.44:0/3124991029 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.667+0000 7fa0e7400640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0e8109ad0 0x7fa0e819aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.667+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0e8104220 0x7fa0e819a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.667+0000 7fa0e7400640 1 -- 172.21.15.44:0/3124991029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0e8104220 msgr2=0x7fa0e819a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.667+0000 7fa0e7400640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0e8104220 0x7fa0e819a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.667+0000 7fa0e7400640 1 -- 172.21.15.44:0/3124991029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0e8109f70 msgr2=0x7fa0e819ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.667+0000 7fa0e7400640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0e8109f70 0x7fa0e819ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.668+0000 7fa0e7400640 1 -- 172.21.15.44:0/3124991029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e819f4a0 con 0x7fa0e8109ad0 2024-05-27T09:37:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.668+0000 7fa0eca00640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0e8109f70 0x7fa0e819ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.668+0000 7fa0e7400640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0e8109ad0 0x7fa0e819aba0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa0e806c3a0 tx=0x7fa0d400b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.668+0000 7fa0e4c00640 1 -- 172.21.15.44:0/3124991029 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4009cb0 con 0x7fa0e8109ad0 2024-05-27T09:37:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.668+0000 7fa0e4c00640 1 -- 172.21.15.44:0/3124991029 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0d4009e50 con 0x7fa0e8109ad0 2024-05-27T09:37:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.668+0000 7fa0e4c00640 1 -- 172.21.15.44:0/3124991029 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4019410 con 0x7fa0e8109ad0 2024-05-27T09:37:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.669+0000 7fa0ed400640 1 -- 172.21.15.44:0/3124991029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e819f790 con 0x7fa0e8109ad0 2024-05-27T09:37:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.669+0000 7fa0ed400640 1 -- 172.21.15.44:0/3124991029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e819fc60 con 0x7fa0e8109ad0 2024-05-27T09:37:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.674+0000 7fa0f59a0740 1 Processor -- start 2024-05-27T09:37:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.674+0000 7fa0f59a0740 1 -- start start 2024-05-27T09:37:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.675+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc268a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.675+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc268fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.675+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc2694e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.675+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e5dc1d9b90 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.675+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e5dc1c9730 con 0x55e5dbea54d0 2024-05-27T09:37:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.675+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e5dc1cd770 con 0x55e5dbec61b0 2024-05-27T09:37:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.675+0000 7fa0e7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc268fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.675+0000 7fa0e7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc268fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36328/0 (socket says 172.21.15.44:36328) 2024-05-27T09:37:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.675+0000 7fa0eca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc268a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.675+0000 7fa0e7e00640 1 -- 172.21.15.44:0/196637040 learned_addr learned my addr 172.21.15.44:0/196637040 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.676+0000 7fa0e4c00640 1 -- 172.21.15.44:0/3124991029 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa0d4020020 con 0x7fa0e8109ad0 2024-05-27T09:37:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.676+0000 7fa0e7e00640 1 -- 172.21.15.44:0/196637040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 msgr2=0x55e5dc268a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.676+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/196637040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc268a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.676+0000 7fa0e7e00640 1 -- 172.21.15.44:0/196637040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 msgr2=0x55e5dc2694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.676+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/196637040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc2694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.676+0000 7fa0e7e00640 1 -- 172.21.15.44:0/196637040 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5dc1628b0 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.677+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/196637040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc268fa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009ca0 tx=0x7fa0d8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82844e40ee4bf190 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.677+0000 7fa0dd200640 1 -- 172.21.15.44:0/196637040 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d801e070 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.677+0000 7fa0dd200640 1 -- 172.21.15.44:0/196637040 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0d8019ae0 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.677+0000 7fa0dd200640 1 -- 172.21.15.44:0/196637040 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d800e950 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.678+0000 7fa0f59a0740 1 -- 172.21.15.44:0/196637040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 msgr2=0x55e5dc268fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.678+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/196637040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc268fa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009ca0 tx=0x7fa0d8011810 comp rx=0 tx=0).stop 2024-05-27T09:37:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.678+0000 7fa0e4c00640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa0cc0b29d0 0x7fa0cc0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.679+0000 7fa0e4c00640 1 -- 172.21.15.44:0/3124991029 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa0d40e5bf0 con 0x7fa0e8109ad0 2024-05-27T09:37:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.679+0000 7fa0f59a0740 1 -- 172.21.15.44:0/196637040 shutdown_connections 2024-05-27T09:37:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.679+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/196637040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc2694e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.679+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/196637040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc268fa0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.679+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/196637040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc268a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.679+0000 7fa0f59a0740 1 -- 172.21.15.44:0/196637040 >> 172.21.15.44:0/196637040 conn(0x55e5dbeba300 msgr2=0x55e5dc265210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.679+0000 7fa0eca00640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa0cc0b29d0 0x7fa0cc0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.679+0000 7fa0f59a0740 1 -- 172.21.15.44:0/196637040 shutdown_connections 2024-05-27T09:37:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.679+0000 7fa0f59a0740 1 -- 172.21.15.44:0/196637040 wait complete. 2024-05-27T09:37:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.680+0000 7fa0f59a0740 1 -- start start 2024-05-27T09:37:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.680+0000 7fa0eca00640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa0cc0b29d0 0x7fa0cc0b4ee0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55e5dc265bd0 tx=0x7fa0e00093a0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.680+0000 7fa0f59a0740 1 Processor -- start 2024-05-27T09:37:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.681+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc281690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.681+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc281bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.681+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc27ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.681+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e5dbeaf790 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.681+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e5dc171bc0 con 0x55e5dbea54d0 2024-05-27T09:37:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.681+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e5dc275b90 con 0x55e5dbec61b0 2024-05-27T09:37:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.681+0000 7fa0f59a0740 10 client.0 _ll_register_callbacks cb 0x55e5dc14f400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.681+0000 7fa0e7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc281690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0e7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc281690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36338/0 (socket says 172.21.15.44:36338) 2024-05-27T09:37:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0e7400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc281bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0e7e00640 1 -- 172.21.15.44:0/3024997910 learned_addr learned my addr 172.21.15.44:0/3024997910 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0eca00640 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc27ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0e7400640 1 -- 172.21.15.44:0/3024997910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 msgr2=0x55e5dc27ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0e7400640 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc27ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0e7400640 1 -- 172.21.15.44:0/3024997910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 msgr2=0x55e5dc281690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0e7400640 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc281690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0e7400640 1 -- 172.21.15.44:0/3024997910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5dbede210 con 0x55e5dbea54d0 2024-05-27T09:37:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0eca00640 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc27ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0e7400640 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc281bd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40e3310 tx=0x7fa0d4003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b92982d5c3f932d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.682+0000 7fa0d1600640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:37:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.683+0000 7fa0d1600640 1 -- 172.21.15.44:0/3024997910 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d40b1250 con 0x55e5dbea54d0 2024-05-27T09:37:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.683+0000 7fa0d1600640 1 -- 172.21.15.44:0/3024997910 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa0d40a9490 con 0x55e5dbea54d0 2024-05-27T09:37:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.683+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3024997910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0d8019ae0 con 0x55e5dbea54d0 2024-05-27T09:37:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.683+0000 7fa0d1600640 1 -- 172.21.15.44:0/3024997910 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa0d40b5d00 con 0x55e5dbea54d0 2024-05-27T09:37:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.683+0000 7fa0f59a0740 10 client.25676 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.683+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3024997910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e5dbfaa620 con 0x55e5dbea54d0 2024-05-27T09:37:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.684+0000 7fa0d1600640 1 -- 172.21.15.44:0/3024997910 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0d8019ae0 con 0x55e5dbea54d0 2024-05-27T09:37:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.684+0000 7fa0d1600640 1 -- 172.21.15.44:0/3024997910 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa0d40c3070 con 0x55e5dbea54d0 2024-05-27T09:37:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.684+0000 7fa0f59a0740 10 client.25676 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.684+0000 7fa0f59a0740 10 client.25676 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.684+0000 7fa0f59a0740 20 client.25676 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.684+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5dbe84d80 0x55e5dc28d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.684+0000 7fa0f59a0740 4 client.25676 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:37:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.684+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3024997910 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5dc1dbae0 con 0x55e5dbe84d80 2024-05-27T09:37:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.685+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5dbe84d80 0x55e5dc28d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.685+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5dbe84d80 0x55e5dc28d790 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c42efb81c04b51a8 server_cookie=8ad2c5987c0c8186 in_seq=0 out_seq=0 2024-05-27T09:37:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.685+0000 7fa0d1600640 10 client.25676 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.686+0000 7fa0d1600640 1 -- 172.21.15.44:0/3024997910 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e5dc1dbae0 con 0x55e5dbe84d80 2024-05-27T09:37:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.686+0000 7fa0d1600640 10 client.25676 handle_command_reply: tid=0 2024-05-27T09:37:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.686+0000 7fa0f59a0740 1 client.25676 shutdown 2024-05-27T09:37:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.689+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3024997910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 msgr2=0x55e5dc281bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.689+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc281bd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40e3310 tx=0x7fa0d4003820 comp rx=0 tx=0).stop 2024-05-27T09:37:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.689+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3024997910 shutdown_connections 2024-05-27T09:37:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.689+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5dbe84d80 0x55e5dc28d790 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.689+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc27ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.689+0000 7fa0d1600640 0 client.25676 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.689+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc281bd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.689+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3024997910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc281690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.689+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3024997910 >> 172.21.15.44:0/3024997910 conn(0x55e5dbeba300 msgr2=0x55e5dc26a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.689+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3024997910 shutdown_connections 2024-05-27T09:37:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.689+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3024997910 wait complete. 2024-05-27T09:37:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.690+0000 7fa0f59a0740 20 client.25676 trim_cache size 0 max 16384 2024-05-27T09:37:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.722+0000 7fa0f59a0740 1 Processor -- start 2024-05-27T09:37:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.722+0000 7fa0f59a0740 1 -- start start 2024-05-27T09:37:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.722+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc2698b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.722+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc276e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.722+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbec61b0 0x55e5dc263f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.722+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e81afea0 con 0x55e5dbec61b0 2024-05-27T09:37:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.722+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e81ad770 con 0x55e5dbea54d0 2024-05-27T09:37:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.722+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e81ab5a0 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.723+0000 7fa0e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc2698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.723+0000 7fa0eca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbec61b0 0x55e5dc263f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.723+0000 7fa0e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc2698b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49358/0 (socket says 172.21.15.44:49358) 2024-05-27T09:37:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.723+0000 7fa0e7e00640 1 -- 172.21.15.44:0/289806691 learned_addr learned my addr 172.21.15.44:0/289806691 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.723+0000 7fa0e7400640 1 --2- 172.21.15.44:0/289806691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc276e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.723+0000 7fa0e7e00640 1 -- 172.21.15.44:0/289806691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 msgr2=0x55e5dc276e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.723+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/289806691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc276e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.723+0000 7fa0e7e00640 1 -- 172.21.15.44:0/289806691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbec61b0 msgr2=0x55e5dc263f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.723+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/289806691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbec61b0 0x55e5dc263f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.723+0000 7fa0e7e00640 1 -- 172.21.15.44:0/289806691 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5dc265c00 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.724+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/289806691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc2698b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800e5a0 tx=0x7fa0d800e5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0f4664ad7010772 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.724+0000 7fa0d3e00640 1 -- 172.21.15.44:0/289806691 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d801e040 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.724+0000 7fa0d3e00640 1 -- 172.21.15.44:0/289806691 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0d8019880 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.724+0000 7fa0d3e00640 1 -- 172.21.15.44:0/289806691 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d80035f0 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.725+0000 7fa0f59a0740 1 -- 172.21.15.44:0/289806691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbe9d2f0 msgr2=0x55e5dc2698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.725+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/289806691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc2698b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800e5a0 tx=0x7fa0d800e5d0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.726+0000 7fa0f59a0740 1 -- 172.21.15.44:0/289806691 shutdown_connections 2024-05-27T09:37:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.726+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/289806691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbec61b0 0x55e5dc263f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.726+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/289806691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc276e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.726+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/289806691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc2698b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.726+0000 7fa0f59a0740 1 -- 172.21.15.44:0/289806691 >> 172.21.15.44:0/289806691 conn(0x55e5dbeba300 msgr2=0x55e5dc263810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.726+0000 7fa0f59a0740 1 -- 172.21.15.44:0/289806691 shutdown_connections 2024-05-27T09:37:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.726+0000 7fa0f59a0740 1 -- 172.21.15.44:0/289806691 wait complete. 2024-05-27T09:37:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.726+0000 7fa0f59a0740 1 -- start start 2024-05-27T09:37:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.727+0000 7fa0f59a0740 1 Processor -- start 2024-05-27T09:37:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.727+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc282a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.727+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc282fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0f59a0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc2f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e81abc30 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e81ad0a0 con 0x55e5dbea54d0 2024-05-27T09:37:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0f59a0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e81ac300 con 0x55e5dbec61b0 2024-05-27T09:37:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0f59a0740 10 client.0 _ll_register_callbacks cb 0x7fa0d40038c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0e7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc282a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0e7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc282a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36350/0 (socket says 172.21.15.44:36350) 2024-05-27T09:37:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0e7e00640 1 -- 172.21.15.44:0/3374294274 learned_addr learned my addr 172.21.15.44:0/3374294274 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0eca00640 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc2f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0e7400640 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc282fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0e7e00640 1 -- 172.21.15.44:0/3374294274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 msgr2=0x55e5dc2f3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.728+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc2f3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.729+0000 7fa0e7e00640 1 -- 172.21.15.44:0/3374294274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 msgr2=0x55e5dc282fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.729+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc282fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.729+0000 7fa0e7e00640 1 -- 172.21.15.44:0/3374294274 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5dc2837f0 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.729+0000 7fa0e7400640 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc282fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:37:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.729+0000 7fa0e7e00640 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc282a60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009a30 tx=0x7fa0d8005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202c6cbf087dd083 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.729+0000 7fa0d1600640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:37:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.729+0000 7fa0d1600640 1 -- 172.21.15.44:0/3374294274 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d801e040 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.730+0000 7fa0d1600640 1 -- 172.21.15.44:0/3374294274 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa0d80052f0 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.730+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3374294274 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0d8019880 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.730+0000 7fa0d1600640 1 -- 172.21.15.44:0/3374294274 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa0d8005c50 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.730+0000 7fa0f59a0740 10 client.15543 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.730+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3374294274 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e5dc2f5300 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.731+0000 7fa0d1600640 1 -- 172.21.15.44:0/3374294274 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0d8019880 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.731+0000 7fa0d1600640 1 -- 172.21.15.44:0/3374294274 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa0d8003800 con 0x55e5dbe9d2f0 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.731+0000 7fa0f59a0740 10 client.15543 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.731+0000 7fa0f59a0740 10 client.15543 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.731+0000 7fa0f59a0740 20 client.15543 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.731+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5dbe5a1c0 0x55e5dc2f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.731+0000 7fa0f59a0740 4 client.15543 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:37:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.731+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3374294274 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e5dc1d52c0 con 0x55e5dbe5a1c0 2024-05-27T09:37:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.732+0000 7fa0e7400640 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5dbe5a1c0 0x55e5dc2f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.733+0000 7fa0e7400640 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5dbe5a1c0 0x55e5dc2f77e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfcdcf08c2724589 server_cookie=16072c0a899c72c8 in_seq=0 out_seq=0 2024-05-27T09:37:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.733+0000 7fa0d1600640 10 client.15543 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.737+0000 7fa0d1600640 1 -- 172.21.15.44:0/3374294274 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e5dc1d52c0 con 0x55e5dbe5a1c0 2024-05-27T09:37:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.737+0000 7fa0d1600640 10 client.15543 handle_command_reply: tid=0 2024-05-27T09:37:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.737+0000 7fa0f59a0740 1 client.15543 shutdown 2024-05-27T09:37:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3374294274 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 msgr2=0x55e5dc282a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc282a60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009a30 tx=0x7fa0d8005f10 comp rx=0 tx=0).stop 2024-05-27T09:37:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3374294274 shutdown_connections 2024-05-27T09:37:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5dbe5a1c0 0x55e5dc2f77e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5dbec61b0 0x55e5dc2f3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0d1600640 0 client.15543 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5dbea54d0 0x55e5dc282fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0f59a0740 1 --2- 172.21.15.44:0/3374294274 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5dbe9d2f0 0x55e5dc282a60 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3374294274 >> 172.21.15.44:0/3374294274 conn(0x55e5dbeba300 msgr2=0x55e5dc263810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3374294274 shutdown_connections 2024-05-27T09:37:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0f59a0740 1 -- 172.21.15.44:0/3374294274 wait complete. 2024-05-27T09:37:21.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.738+0000 7fa0f59a0740 20 client.15543 trim_cache size 0 max 16384 2024-05-27T09:37:21.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:37:21.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:37:21.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:37:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.739+0000 7fa0ed400640 1 -- 172.21.15.44:0/3124991029 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa0cc0b29d0 msgr2=0x7fa0cc0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.739+0000 7fa0ed400640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa0cc0b29d0 0x7fa0cc0b4ee0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55e5dc265bd0 tx=0x7fa0e00093a0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.739+0000 7fa0ed400640 1 -- 172.21.15.44:0/3124991029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0e8109ad0 msgr2=0x7fa0e819aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.739+0000 7fa0ed400640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0e8109ad0 0x7fa0e819aba0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa0e806c3a0 tx=0x7fa0d400b690 comp rx=0 tx=0).stop 2024-05-27T09:37:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.740+0000 7fa0ed400640 1 -- 172.21.15.44:0/3124991029 shutdown_connections 2024-05-27T09:37:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.740+0000 7fa0ed400640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa0cc0b29d0 0x7fa0cc0b4ee0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.740+0000 7fa0ed400640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0e8109f70 0x7fa0e819ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.740+0000 7fa0ed400640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0e8109ad0 0x7fa0e819aba0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.740+0000 7fa0ed400640 1 --2- 172.21.15.44:0/3124991029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0e8104220 0x7fa0e819a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:21.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.740+0000 7fa0ed400640 1 -- 172.21.15.44:0/3124991029 >> 172.21.15.44:0/3124991029 conn(0x7fa0e8079100 msgr2=0x7fa0e8100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:21.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.740+0000 7fa0ed400640 1 -- 172.21.15.44:0/3124991029 shutdown_connections 2024-05-27T09:37:21.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:21.740+0000 7fa0ed400640 1 -- 172.21.15.44:0/3124991029 wait complete. 2024-05-27T09:37:22.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:22 smithi044 ceph-mon[31618]: pgmap v67: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:37:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:22 smithi050 ceph-mon[36371]: pgmap v67: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:37:22.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:22 smithi189 ceph-mon[36671]: pgmap v67: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:37:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.734+0000 7f9bd6200700 1 -- 172.21.15.44:0/2211011410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bd0105bd0 msgr2=0x7f9bd010e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.734+0000 7f9bd6200700 1 --2- 172.21.15.44:0/2211011410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bd0105bd0 0x7f9bd010e120 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc0098f0 tx=0x7f9bbc009c00 comp rx=0 tx=0).stop 2024-05-27T09:37:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.734+0000 7f9bd6200700 1 -- 172.21.15.44:0/2211011410 shutdown_connections 2024-05-27T09:37:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.734+0000 7f9bd6200700 1 --2- 172.21.15.44:0/2211011410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd010e6f0 0x7f9bd0110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.734+0000 7f9bd6200700 1 --2- 172.21.15.44:0/2211011410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9bd0105280 0x7f9bd0105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.734+0000 7f9bd6200700 1 --2- 172.21.15.44:0/2211011410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bd0105bd0 0x7f9bd010e120 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.734+0000 7f9bd6200700 1 -- 172.21.15.44:0/2211011410 >> 172.21.15.44:0/2211011410 conn(0x7f9bd0078ab0 msgr2=0x7f9bd0078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:22.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.734+0000 7f9bd6200700 1 -- 172.21.15.44:0/2211011410 shutdown_connections 2024-05-27T09:37:22.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.734+0000 7f9bd6200700 1 -- 172.21.15.44:0/2211011410 wait complete. 2024-05-27T09:37:22.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.736+0000 7f9bd6200700 1 Processor -- start 2024-05-27T09:37:22.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.736+0000 7f9bd6200700 1 -- start start 2024-05-27T09:37:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.736+0000 7f9bd6200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd0105280 0x7f9bd01a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.736+0000 7f9bd6200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9bd0105bd0 0x7f9bd01a2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.736+0000 7f9bd6200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bd010e6f0 0x7f9bd01acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.736+0000 7f9bd6200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd0114250 con 0x7f9bd0105bd0 2024-05-27T09:37:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.736+0000 7f9bd6200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd01140d0 con 0x7f9bd010e6f0 2024-05-27T09:37:22.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.736+0000 7f9bd6200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd01143d0 con 0x7f9bd0105280 2024-05-27T09:37:22.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.737+0000 7f9bd4e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd0105280 0x7f9bd01a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.737+0000 7f9bd5800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bd010e6f0 0x7f9bd01acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.737+0000 7f9bcfe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9bd0105bd0 0x7f9bd01a2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.737+0000 7f9bd5800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bd010e6f0 0x7f9bd01acfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50594/0 (socket says 172.21.15.44:50594) 2024-05-27T09:37:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.737+0000 7f9bd5800700 1 -- 172.21.15.44:0/3260494726 learned_addr learned my addr 172.21.15.44:0/3260494726 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.738+0000 7f9bd4e00700 1 -- 172.21.15.44:0/3260494726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bd010e6f0 msgr2=0x7f9bd01acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.738+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bd010e6f0 0x7f9bd01acfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.738+0000 7f9bd4e00700 1 -- 172.21.15.44:0/3260494726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9bd0105bd0 msgr2=0x7f9bd01a2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:22.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.738+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9bd0105bd0 0x7f9bd01a2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.738+0000 7f9bd4e00700 1 -- 172.21.15.44:0/3260494726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd01ad4f0 con 0x7f9bd0105280 2024-05-27T09:37:22.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.738+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd0105280 0x7f9bd01a1f30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400eb90 tx=0x7f9bc400eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:22.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.739+0000 7f9bcd600700 1 -- 172.21.15.44:0/3260494726 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc400cc90 con 0x7f9bd0105280 2024-05-27T09:37:22.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.739+0000 7f9bcd600700 1 -- 172.21.15.44:0/3260494726 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bc400ce30 con 0x7f9bd0105280 2024-05-27T09:37:22.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.739+0000 7f9bcd600700 1 -- 172.21.15.44:0/3260494726 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4010430 con 0x7f9bd0105280 2024-05-27T09:37:22.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.739+0000 7f9bd6200700 1 -- 172.21.15.44:0/3260494726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd01ad720 con 0x7f9bd0105280 2024-05-27T09:37:22.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.739+0000 7f9bd6200700 1 -- 172.21.15.44:0/3260494726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd01adbe0 con 0x7f9bd0105280 2024-05-27T09:37:22.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.744+0000 7f9bcd600700 1 -- 172.21.15.44:0/3260494726 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f9bc4004020 con 0x7f9bd0105280 2024-05-27T09:37:22.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.745+0000 7f9be62c7b80 1 Processor -- start 2024-05-27T09:37:22.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.745+0000 7f9be62c7b80 1 -- start start 2024-05-27T09:37:22.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.746+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b8544f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.746+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b852cfe40 0x563b8544fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.746+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 0x563b8545b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.746+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563b8537ce40 con 0x563b853bb9f0 2024-05-27T09:37:22.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.746+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563b8536c9e0 con 0x563b852cfe40 2024-05-27T09:37:22.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.746+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563b85370a20 con 0x563b854592c0 2024-05-27T09:37:22.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.747+0000 7f9bd5800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b8544f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.747+0000 7f9bcd600700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9bb80b2b40 0x7f9bb80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.747+0000 7f9bd5800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b8544f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36370/0 (socket says 172.21.15.44:36370) 2024-05-27T09:37:22.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.747+0000 7f9bd5800700 1 -- 172.21.15.44:0/2049506125 learned_addr learned my addr 172.21.15.44:0/2049506125 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:22.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.747+0000 7f9bcd600700 1 -- 172.21.15.44:0/3260494726 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f9bc40e2880 con 0x7f9bd0105280 2024-05-27T09:37:22.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.747+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/2049506125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b852cfe40 0x563b8544fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.747+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/2049506125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 0x563b8545b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.748+0000 7f9bd5800700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9bb80b2b40 0x7f9bb80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.748+0000 7f9bcfe00700 1 -- 172.21.15.44:0/2049506125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b852cfe40 msgr2=0x563b8544fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:22.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.748+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/2049506125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b852cfe40 0x563b8544fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.748+0000 7f9bcfe00700 1 -- 172.21.15.44:0/2049506125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 msgr2=0x563b8544f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:22.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.748+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/2049506125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b8544f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.748+0000 7f9bcfe00700 1 -- 172.21.15.44:0/2049506125 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b85335420 con 0x563b854592c0 2024-05-27T09:37:22.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.748+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/2049506125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 0x563b8545b6e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9bd010e630 tx=0x7f9bbc005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9a8c87fe96736d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:22.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.749+0000 7f9bc2000700 1 -- 172.21.15.44:0/2049506125 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc00f460 con 0x563b854592c0 2024-05-27T09:37:22.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.749+0000 7f9bc2000700 1 -- 172.21.15.44:0/2049506125 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bbc005040 con 0x563b854592c0 2024-05-27T09:37:22.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.749+0000 7f9bc2000700 1 -- 172.21.15.44:0/2049506125 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc0039b0 con 0x563b854592c0 2024-05-27T09:37:22.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.750+0000 7f9be62c7b80 1 -- 172.21.15.44:0/2049506125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 msgr2=0x563b8545b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:22.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.750+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/2049506125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 0x563b8545b6e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9bd010e630 tx=0x7f9bbc005dc0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.750+0000 7f9bd5800700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9bb80b2b40 0x7f9bb80b5050 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x563b8544ba90 tx=0x7f9bc8004b40 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:22.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.750+0000 7f9be62c7b80 1 -- 172.21.15.44:0/2049506125 shutdown_connections 2024-05-27T09:37:22.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.750+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/2049506125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 0x563b8545b6e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.750+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/2049506125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b8544f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.750+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/2049506125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b852cfe40 0x563b8544fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.750+0000 7f9be62c7b80 1 -- 172.21.15.44:0/2049506125 >> 172.21.15.44:0/2049506125 conn(0x563b85369390 msgr2=0x563b8544a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:22.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.751+0000 7f9be62c7b80 1 -- 172.21.15.44:0/2049506125 shutdown_connections 2024-05-27T09:37:22.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.751+0000 7f9be62c7b80 1 -- 172.21.15.44:0/2049506125 wait complete. 2024-05-27T09:37:22.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.751+0000 7f9be62c7b80 1 -- start start 2024-05-27T09:37:22.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.752+0000 7f9be62c7b80 1 Processor -- start 2024-05-27T09:37:22.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.752+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b852cfe40 0x563b85456260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.752+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b853bb9f0 0x563b854567a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.752+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 0x563b85456ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.752+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563b85272fa0 con 0x563b852cfe40 2024-05-27T09:37:22.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.752+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563b8530d760 con 0x563b853bb9f0 2024-05-27T09:37:22.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.752+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563b8545bda0 con 0x563b854592c0 2024-05-27T09:37:22.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.752+0000 7f9be62c7b80 10 client.0 _ll_register_callbacks cb 0x563b85316c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:22.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.753+0000 7f9bd5800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 0x563b85456ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.753+0000 7f9bd5800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 0x563b85456ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49398/0 (socket says 172.21.15.44:49398) 2024-05-27T09:37:22.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.753+0000 7f9bd4e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b852cfe40 0x563b85456260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.753+0000 7f9bd5800700 1 -- 172.21.15.44:0/3028184738 learned_addr learned my addr 172.21.15.44:0/3028184738 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:22.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.753+0000 7f9bcfe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b853bb9f0 0x563b854567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.753+0000 7f9bcfe00700 1 -- 172.21.15.44:0/3028184738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 msgr2=0x563b85456ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:22.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.753+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/3028184738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 0x563b85456ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.753+0000 7f9bcfe00700 1 -- 172.21.15.44:0/3028184738 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b852cfe40 msgr2=0x563b85456260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:22.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.753+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/3028184738 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b852cfe40 0x563b85456260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.753+0000 7f9bcfe00700 1 -- 172.21.15.44:0/3028184738 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b852bc8d0 con 0x563b853bb9f0 2024-05-27T09:37:22.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.755+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/3028184738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b853bb9f0 0x563b854567a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009c40 tx=0x7f9bbc00b2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86e8c66f1cd67f03 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:22.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.755+0000 7f9bb6a00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:37:22.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.755+0000 7f9bb6a00700 1 -- 172.21.15.44:0/3028184738 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc02f070 con 0x563b853bb9f0 2024-05-27T09:37:22.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.755+0000 7f9be62c7b80 1 -- 172.21.15.44:0/3028184738 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bbc005040 con 0x563b853bb9f0 2024-05-27T09:37:22.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.755+0000 7f9bb6a00700 1 -- 172.21.15.44:0/3028184738 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f9bbc00f8f0 con 0x563b853bb9f0 2024-05-27T09:37:22.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.756+0000 7f9bb6a00700 1 -- 172.21.15.44:0/3028184738 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9bbc03a050 con 0x563b853bb9f0 2024-05-27T09:37:22.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.756+0000 7f9be62c7b80 10 client.25691 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:22.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.756+0000 7f9be62c7b80 1 -- 172.21.15.44:0/3028184738 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563b8533d060 con 0x563b853bb9f0 2024-05-27T09:37:22.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.756+0000 7f9bb6a00700 1 -- 172.21.15.44:0/3028184738 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bbc005040 con 0x563b853bb9f0 2024-05-27T09:37:22.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.756+0000 7f9bb6a00700 1 -- 172.21.15.44:0/3028184738 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9bbc00b840 con 0x563b853bb9f0 2024-05-27T09:37:22.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.756+0000 7f9be62c7b80 10 client.25691 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:22.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.756+0000 7f9be62c7b80 10 client.25691 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:37:22.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.756+0000 7f9be62c7b80 20 client.25691 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:22.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.757+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/3028184738 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x563b85468b20 0x563b8546af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.757+0000 7f9be62c7b80 4 client.25691 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:37:22.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.757+0000 7f9be62c7b80 1 -- 172.21.15.44:0/3028184738 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b8537ed90 con 0x563b85468b20 2024-05-27T09:37:22.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.757+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/3028184738 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x563b85468b20 0x563b8546af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.758+0000 7f9bb6a00700 10 client.25691 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:37:22.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.758+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/3028184738 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x563b85468b20 0x563b8546af40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a59ba66b8529d4a3 server_cookie=1c112b1a6605639a in_seq=0 out_seq=0 2024-05-27T09:37:22.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.758+0000 7f9bb6a00700 1 -- 172.21.15.44:0/3028184738 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563b8537ed90 con 0x563b85468b20 2024-05-27T09:37:22.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.758+0000 7f9bb6a00700 10 client.25691 handle_command_reply: tid=0 2024-05-27T09:37:22.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.758+0000 7f9be62c7b80 1 client.25691 shutdown 2024-05-27T09:37:22.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.759+0000 7f9be62c7b80 1 -- 172.21.15.44:0/3028184738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b853bb9f0 msgr2=0x563b854567a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:22.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.759+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/3028184738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b853bb9f0 0x563b854567a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009c40 tx=0x7f9bbc00b2e0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.759+0000 7f9be62c7b80 1 -- 172.21.15.44:0/3028184738 shutdown_connections 2024-05-27T09:37:22.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.759+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/3028184738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b854592c0 0x563b85456ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.759+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/3028184738 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x563b85468b20 0x563b8546af40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.759+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/3028184738 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b852cfe40 0x563b85456260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.759+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/3028184738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b853bb9f0 0x563b854567a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.759+0000 7f9be62c7b80 1 -- 172.21.15.44:0/3028184738 >> 172.21.15.44:0/3028184738 conn(0x563b85369390 msgr2=0x563b8544a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:22.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.759+0000 7f9bb6a00700 0 client.25691 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:37:22.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.759+0000 7f9be62c7b80 1 -- 172.21.15.44:0/3028184738 shutdown_connections 2024-05-27T09:37:22.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.759+0000 7f9be62c7b80 1 -- 172.21.15.44:0/3028184738 wait complete. 2024-05-27T09:37:22.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.760+0000 7f9be62c7b80 20 client.25691 trim_cache size 0 max 16384 2024-05-27T09:37:22.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.778+0000 7f9be62c7b80 1 Processor -- start 2024-05-27T09:37:22.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.778+0000 7f9be62c7b80 1 -- start start 2024-05-27T09:37:22.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.778+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8544a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.778+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b85465040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.778+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b854592c0 0x563b85455260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.778+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd01b7990 con 0x563b853bb9f0 2024-05-27T09:37:22.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.778+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd01b5220 con 0x563b854592c0 2024-05-27T09:37:22.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.778+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd01b3070 con 0x563b852cfe40 2024-05-27T09:37:22.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.779+0000 7f9bd4e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8544a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.779+0000 7f9bd4e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8544a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49410/0 (socket says 172.21.15.44:49410) 2024-05-27T09:37:22.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.779+0000 7f9bd4e00700 1 -- 172.21.15.44:0/142848586 learned_addr learned my addr 172.21.15.44:0/142848586 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:22.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.779+0000 7f9bd5800700 1 --2- 172.21.15.44:0/142848586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b854592c0 0x563b85455260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.780+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/142848586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b85465040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.780+0000 7f9bd4e00700 1 -- 172.21.15.44:0/142848586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b854592c0 msgr2=0x563b85455260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:22.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.780+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/142848586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b854592c0 0x563b85455260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.780+0000 7f9bd4e00700 1 -- 172.21.15.44:0/142848586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 msgr2=0x563b85465040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:22.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.780+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/142848586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b85465040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.780+0000 7f9bd4e00700 1 -- 172.21.15.44:0/142848586 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b85457050 con 0x563b852cfe40 2024-05-27T09:37:22.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.780+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/142848586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8544a740 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400c5b0 tx=0x7f9bc40bff90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea34e4d94468f419 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:22.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.780+0000 7f9bc1600700 1 -- 172.21.15.44:0/142848586 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40c8030 con 0x563b852cfe40 2024-05-27T09:37:22.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.781+0000 7f9bc1600700 1 -- 172.21.15.44:0/142848586 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bc40b22d0 con 0x563b852cfe40 2024-05-27T09:37:22.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.781+0000 7f9bc1600700 1 -- 172.21.15.44:0/142848586 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40b2990 con 0x563b852cfe40 2024-05-27T09:37:22.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.782+0000 7f9be62c7b80 1 -- 172.21.15.44:0/142848586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 msgr2=0x563b8544a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:22.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.782+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/142848586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8544a740 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400c5b0 tx=0x7f9bc40bff90 comp rx=0 tx=0).stop 2024-05-27T09:37:22.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.782+0000 7f9be62c7b80 1 -- 172.21.15.44:0/142848586 shutdown_connections 2024-05-27T09:37:22.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.782+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/142848586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b854592c0 0x563b85455260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.782+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/142848586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8544a740 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.782+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/142848586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b85465040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.782+0000 7f9be62c7b80 1 -- 172.21.15.44:0/142848586 >> 172.21.15.44:0/142848586 conn(0x563b85369390 msgr2=0x563b85454b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:22.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.782+0000 7f9be62c7b80 1 -- 172.21.15.44:0/142848586 shutdown_connections 2024-05-27T09:37:22.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.783+0000 7f9be62c7b80 1 -- 172.21.15.44:0/142848586 wait complete. 2024-05-27T09:37:22.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.783+0000 7f9be62c7b80 1 -- start start 2024-05-27T09:37:22.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.783+0000 7f9be62c7b80 1 Processor -- start 2024-05-27T09:37:22.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.784+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8546ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.784+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b8546b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.784+0000 7f9be62c7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b854592c0 0x563b854d1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.784+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd01b36e0 con 0x563b853bb9f0 2024-05-27T09:37:22.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.784+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd01b4b50 con 0x563b854592c0 2024-05-27T09:37:22.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.784+0000 7f9be62c7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd01b3db0 con 0x563b852cfe40 2024-05-27T09:37:22.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.784+0000 7f9be62c7b80 10 client.0 _ll_register_callbacks cb 0x563b85316c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:22.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.785+0000 7f9bd4e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8546ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.785+0000 7f9bd4e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8546ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49418/0 (socket says 172.21.15.44:49418) 2024-05-27T09:37:22.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.785+0000 7f9bd4e00700 1 -- 172.21.15.44:0/1842771610 learned_addr learned my addr 172.21.15.44:0/1842771610 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:22.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.785+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b8546b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.786+0000 7f9bd5800700 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b854592c0 0x563b854d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.786+0000 7f9bd4e00700 1 -- 172.21.15.44:0/1842771610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b854592c0 msgr2=0x563b854d1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:22.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.786+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b854592c0 0x563b854d1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.786+0000 7f9bd4e00700 1 -- 172.21.15.44:0/1842771610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 msgr2=0x563b8546b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.786+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b8546b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.786+0000 7f9bd4e00700 1 -- 172.21.15.44:0/1842771610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b8546b9f0 con 0x563b852cfe40 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.786+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b8546b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.786+0000 7f9bd4e00700 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8546ac20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40a5bb0 tx=0x7f9bc4009380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aab763aab105cb16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.786+0000 7f9bb6a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.787+0000 7f9bb6a00700 1 -- 172.21.15.44:0/1842771610 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40aae60 con 0x563b852cfe40 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.787+0000 7f9bb6a00700 1 -- 172.21.15.44:0/1842771610 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f9bc4004310 con 0x563b852cfe40 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.787+0000 7f9be62c7b80 1 -- 172.21.15.44:0/1842771610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563b85378570 con 0x563b852cfe40 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.787+0000 7f9bb6a00700 1 -- 172.21.15.44:0/1842771610 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9bc40c8070 con 0x563b852cfe40 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.787+0000 7f9be62c7b80 10 client.25708 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.787+0000 7f9be62c7b80 1 -- 172.21.15.44:0/1842771610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563b854d28e0 con 0x563b852cfe40 2024-05-27T09:37:22.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.788+0000 7f9bb6a00700 1 -- 172.21.15.44:0/1842771610 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563b85378570 con 0x563b852cfe40 2024-05-27T09:37:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.789+0000 7f9bb6a00700 1 -- 172.21.15.44:0/1842771610 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9bc40b2be0 con 0x563b852cfe40 2024-05-27T09:37:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.789+0000 7f9be62c7b80 10 client.25708 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:22.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.789+0000 7f9be62c7b80 10 client.25708 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:37:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.789+0000 7f9be62c7b80 20 client.25708 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.789+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x563b854d2eb0 0x563b854d52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.789+0000 7f9be62c7b80 4 client.25708 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:37:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.789+0000 7f9be62c7b80 1 -- 172.21.15.44:0/1842771610 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563b851ef250 con 0x563b854d2eb0 2024-05-27T09:37:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.790+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x563b854d2eb0 0x563b854d52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.791+0000 7f9bcfe00700 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x563b854d2eb0 0x563b854d52d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bcaf2a802d64d961 server_cookie=1905d4a8b7824f2c in_seq=0 out_seq=0 2024-05-27T09:37:22.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.791+0000 7f9bb6a00700 10 client.25708 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:37:22.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.791+0000 7f9bb6a00700 1 -- 172.21.15.44:0/1842771610 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563b851ef250 con 0x563b854d2eb0 2024-05-27T09:37:22.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.791+0000 7f9bb6a00700 10 client.25708 handle_command_reply: tid=0 2024-05-27T09:37:22.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.791+0000 7f9be62c7b80 1 client.25708 shutdown 2024-05-27T09:37:22.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.792+0000 7f9be62c7b80 1 -- 172.21.15.44:0/1842771610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 msgr2=0x563b8546ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:22.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.792+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8546ac20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40a5bb0 tx=0x7f9bc4009380 comp rx=0 tx=0).stop 2024-05-27T09:37:22.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.792+0000 7f9be62c7b80 1 -- 172.21.15.44:0/1842771610 shutdown_connections 2024-05-27T09:37:22.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.792+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b854592c0 0x563b854d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.792+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b852cfe40 0x563b8546ac20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.792+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x563b854d2eb0 0x563b854d52d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.792+0000 7f9be62c7b80 1 --2- 172.21.15.44:0/1842771610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b853bb9f0 0x563b8546b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.792+0000 7f9be62c7b80 1 -- 172.21.15.44:0/1842771610 >> 172.21.15.44:0/1842771610 conn(0x563b85369390 msgr2=0x563b8544e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:22.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.792+0000 7f9bb6a00700 0 client.25708 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:37:22.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.793+0000 7f9be62c7b80 1 -- 172.21.15.44:0/1842771610 shutdown_connections 2024-05-27T09:37:22.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.793+0000 7f9be62c7b80 1 -- 172.21.15.44:0/1842771610 wait complete. 2024-05-27T09:37:22.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.793+0000 7f9be62c7b80 20 client.25708 trim_cache size 0 max 16384 2024-05-27T09:37:22.794 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:37:22.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.794+0000 7f9bd6200700 1 -- 172.21.15.44:0/3260494726 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9bb80b2b40 msgr2=0x7f9bb80b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:22.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.794+0000 7f9bd6200700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9bb80b2b40 0x7f9bb80b5050 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x563b8544ba90 tx=0x7f9bc8004b40 comp rx=0 tx=0).stop 2024-05-27T09:37:22.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.794+0000 7f9bd6200700 1 -- 172.21.15.44:0/3260494726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd0105280 msgr2=0x7f9bd01a1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:22.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.794+0000 7f9bd6200700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd0105280 0x7f9bd01a1f30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400eb90 tx=0x7f9bc400eea0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.794+0000 7f9bd6200700 1 -- 172.21.15.44:0/3260494726 shutdown_connections 2024-05-27T09:37:22.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.794+0000 7f9bd6200700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9bb80b2b40 0x7f9bb80b5050 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.794+0000 7f9bd6200700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bd010e6f0 0x7f9bd01acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.794+0000 7f9bd6200700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd0105280 0x7f9bd01a1f30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.794+0000 7f9bd6200700 1 --2- 172.21.15.44:0/3260494726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9bd0105bd0 0x7f9bd01a2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:22.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.794+0000 7f9bd6200700 1 -- 172.21.15.44:0/3260494726 >> 172.21.15.44:0/3260494726 conn(0x7f9bd0078ab0 msgr2=0x7f9bd0101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:22.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.794+0000 7f9bd6200700 1 -- 172.21.15.44:0/3260494726 shutdown_connections 2024-05-27T09:37:22.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:22.795+0000 7f9bd6200700 1 -- 172.21.15.44:0/3260494726 wait complete. 2024-05-27T09:37:23.709 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:37:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:37:24.454 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:24 smithi044 ceph-mon[31618]: pgmap v68: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:37:24.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:24 smithi050 ceph-mon[36371]: pgmap v68: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:37:24.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:24 smithi189 ceph-mon[36671]: pgmap v68: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:37:26.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.366+0000 7f6cb4600700 1 -- 172.21.15.44:0/138367132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac1034b0 msgr2=0x7f6cac1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:26.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.366+0000 7f6cb4600700 1 --2- 172.21.15.44:0/138367132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac1034b0 0x7f6cac1038c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6ca00099c0 tx=0x7f6ca0009cd0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.367+0000 7f6cb4600700 1 -- 172.21.15.44:0/138367132 shutdown_connections 2024-05-27T09:37:26.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.367+0000 7f6cb4600700 1 --2- 172.21.15.44:0/138367132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cac108730 0x7f6cac110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.367+0000 7f6cb4600700 1 --2- 172.21.15.44:0/138367132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac1034b0 0x7f6cac1038c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.367+0000 7f6cb4600700 1 --2- 172.21.15.44:0/138367132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cac107de0 0x7f6cac1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.367+0000 7f6cb4600700 1 -- 172.21.15.44:0/138367132 >> 172.21.15.44:0/138367132 conn(0x7f6cac0fedb0 msgr2=0x7f6cac1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:26.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.367+0000 7f6cb4600700 1 -- 172.21.15.44:0/138367132 shutdown_connections 2024-05-27T09:37:26.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.367+0000 7f6cb4600700 1 -- 172.21.15.44:0/138367132 wait complete. 2024-05-27T09:37:26.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.368+0000 7f6cb4600700 1 Processor -- start 2024-05-27T09:37:26.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.368+0000 7f6cb4600700 1 -- start start 2024-05-27T09:37:26.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.369+0000 7f6cb4600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cac1034b0 0x7f6cac0ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.369+0000 7f6cb4600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cac107de0 0x7f6cac0ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.369+0000 7f6cb4600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac108730 0x7f6cac0ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.369+0000 7f6cb4600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac113360 con 0x7f6cac1034b0 2024-05-27T09:37:26.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.369+0000 7f6cb4600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac1131e0 con 0x7f6cac107de0 2024-05-27T09:37:26.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.369+0000 7f6cb4600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac1134e0 con 0x7f6cac108730 2024-05-27T09:37:26.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.369+0000 7f6cb1c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cac1034b0 0x7f6cac0ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.369+0000 7f6cb1c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cac1034b0 0x7f6cac0ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35110/0 (socket says 172.21.15.44:35110) 2024-05-27T09:37:26.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.369+0000 7f6cb1c00700 1 -- 172.21.15.44:0/3961789612 learned_addr learned my addr 172.21.15.44:0/3961789612 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:26.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.370+0000 7f6cb2600700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac108730 0x7f6cac0ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.370+0000 7f6cb1200700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cac107de0 0x7f6cac0ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.370+0000 7f6cb1c00700 1 -- 172.21.15.44:0/3961789612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac108730 msgr2=0x7f6cac0ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:26.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.370+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac108730 0x7f6cac0ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.370+0000 7f6cb1c00700 1 -- 172.21.15.44:0/3961789612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cac107de0 msgr2=0x7f6cac0ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:26.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.370+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cac107de0 0x7f6cac0ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.370+0000 7f6cb1c00700 1 -- 172.21.15.44:0/3961789612 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cac1a91c0 con 0x7f6cac1034b0 2024-05-27T09:37:26.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.370+0000 7f6cb1200700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cac107de0 0x7f6cac0ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:37:26.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.371+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cac1034b0 0x7f6cac0ff290 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0006010 tx=0x7f6ca0004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:26.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.371+0000 7f6caa000700 1 -- 172.21.15.44:0/3961789612 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0026070 con 0x7f6cac1034b0 2024-05-27T09:37:26.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.371+0000 7f6cb4600700 1 -- 172.21.15.44:0/3961789612 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cac1a9450 con 0x7f6cac1034b0 2024-05-27T09:37:26.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.371+0000 7f6caa000700 1 -- 172.21.15.44:0/3961789612 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ca0003c70 con 0x7f6cac1034b0 2024-05-27T09:37:26.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.371+0000 7f6caa000700 1 -- 172.21.15.44:0/3961789612 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca000fbe0 con 0x7f6cac1034b0 2024-05-27T09:37:26.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.371+0000 7f6cb4600700 1 -- 172.21.15.44:0/3961789612 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cac1a9900 con 0x7f6cac1034b0 2024-05-27T09:37:26.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.375+0000 7f6caa000700 1 -- 172.21.15.44:0/3961789612 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6ca0029070 con 0x7f6cac1034b0 2024-05-27T09:37:26.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.377+0000 7f6cc31d1b80 1 Processor -- start 2024-05-27T09:37:26.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.377+0000 7f6cc31d1b80 1 -- start start 2024-05-27T09:37:26.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.377+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffb0047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.377+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffb004d30 0x557ffaffe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.377+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffafff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.377+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557ffaf2ce40 con 0x557ffb004d30 2024-05-27T09:37:26.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.377+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557ffaf1c9e0 con 0x557ffae7fe40 2024-05-27T09:37:26.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.377+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557ffaf20a20 con 0x557ffaffedb0 2024-05-27T09:37:26.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.378+0000 7f6cb1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffafff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.378+0000 7f6caa000700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c940b2a70 0x7f6c940b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.378+0000 7f6cb1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffafff1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35114/0 (socket says 172.21.15.44:35114) 2024-05-27T09:37:26.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.378+0000 7f6cb1200700 1 -- 172.21.15.44:0/1090842306 learned_addr learned my addr 172.21.15.44:0/1090842306 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:26.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.378+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/1090842306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffb004d30 0x557ffaffe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.379+0000 7f6cb2600700 1 --2- 172.21.15.44:0/1090842306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffb0047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:26 smithi044 ceph-mon[31618]: pgmap v69: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-27T09:37:26.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.379+0000 7f6cb2600700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c940b2a70 0x7f6c940b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.379+0000 7f6cb1200700 1 -- 172.21.15.44:0/1090842306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 msgr2=0x557ffb0047f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:26.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.380+0000 7f6caa000700 1 -- 172.21.15.44:0/3961789612 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f6ca00e2e70 con 0x7f6cac1034b0 2024-05-27T09:37:26.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.380+0000 7f6cb1200700 1 --2- 172.21.15.44:0/1090842306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffb0047f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.380+0000 7f6cb1200700 1 -- 172.21.15.44:0/1090842306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffb004d30 msgr2=0x557ffaffe870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:26.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.380+0000 7f6cb1200700 1 --2- 172.21.15.44:0/1090842306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffb004d30 0x557ffaffe870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.380+0000 7f6cb1200700 1 -- 172.21.15.44:0/1090842306 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ffaee5420 con 0x557ffaffedb0 2024-05-27T09:37:26.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.380+0000 7f6cb2600700 1 --2- 172.21.15.44:0/1090842306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffb0047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:37:26.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.381+0000 7f6cb1200700 1 --2- 172.21.15.44:0/1090842306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffafff1c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009d30 tx=0x7f6c98010650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfc77afd239c69c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:26.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.382+0000 7f6cb2600700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c940b2a70 0x7f6c940b4f80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x557ffafffb30 tx=0x7f6ca4004b40 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:26.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.382+0000 7f6c9ea00700 1 -- 172.21.15.44:0/1090842306 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98010db0 con 0x557ffaffedb0 2024-05-27T09:37:26.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.382+0000 7f6c9ea00700 1 -- 172.21.15.44:0/1090842306 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c980044e0 con 0x557ffaffedb0 2024-05-27T09:37:26.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.382+0000 7f6c9ea00700 1 -- 172.21.15.44:0/1090842306 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98012450 con 0x557ffaffedb0 2024-05-27T09:37:26.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.383+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/1090842306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 msgr2=0x557ffafff1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:26.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.383+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/1090842306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffafff1c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009d30 tx=0x7f6c98010650 comp rx=0 tx=0).stop 2024-05-27T09:37:26.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.383+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/1090842306 shutdown_connections 2024-05-27T09:37:26.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.383+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/1090842306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffafff1c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.383+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/1090842306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffb0047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.383+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/1090842306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffb004d30 0x557ffaffe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.383+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/1090842306 >> 172.21.15.44:0/1090842306 conn(0x557ffaedffc0 msgr2=0x557ffaff98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:26.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.384+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/1090842306 shutdown_connections 2024-05-27T09:37:26.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.384+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/1090842306 wait complete. 2024-05-27T09:37:26.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.385+0000 7f6cc31d1b80 1 -- start start 2024-05-27T09:37:26.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.385+0000 7f6cc31d1b80 1 Processor -- start 2024-05-27T09:37:26.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.386+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffb0084e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.386+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffaffedb0 0x557ffb008a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.386+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffb004d30 0x557ffb008f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.386+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557ffae22fa0 con 0x557ffaffedb0 2024-05-27T09:37:26.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.386+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557ffaebd760 con 0x557ffae7fe40 2024-05-27T09:37:26.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.386+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557ffb00b800 con 0x557ffb004d30 2024-05-27T09:37:26.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.387+0000 7f6cc31d1b80 10 client.0 _ll_register_callbacks cb 0x557ffaec6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:26.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.387+0000 7f6cb1c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffb0084e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.387+0000 7f6cb1c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffb0084e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47544/0 (socket says 172.21.15.44:47544) 2024-05-27T09:37:26.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.387+0000 7f6cb1c00700 1 -- 172.21.15.44:0/3340348741 learned_addr learned my addr 172.21.15.44:0/3340348741 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:26.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.388+0000 7f6cb1200700 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffaffedb0 0x557ffb008a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.387+0000 7f6cb2600700 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffb004d30 0x557ffb008f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.388+0000 7f6cb1c00700 1 -- 172.21.15.44:0/3340348741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffb004d30 msgr2=0x557ffb008f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:26.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.388+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffb004d30 0x557ffb008f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.388+0000 7f6cb1c00700 1 -- 172.21.15.44:0/3340348741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffaffedb0 msgr2=0x557ffb008a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:26.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.388+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffaffedb0 0x557ffb008a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.388+0000 7f6cb1c00700 1 -- 172.21.15.44:0/3340348741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ffae6c8d0 con 0x557ffae7fe40 2024-05-27T09:37:26.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.388+0000 7f6cb1200700 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffaffedb0 0x557ffb008a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:26.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.389+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffb0084e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009990 tx=0x7f6ca00e0af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca42954889a6a06a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:26.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.389+0000 7f6c93400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:37:26.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.389+0000 7f6c93400700 1 -- 172.21.15.44:0/3340348741 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca00a9a00 con 0x557ffae7fe40 2024-05-27T09:37:26.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.390+0000 7f6c93400700 1 -- 172.21.15.44:0/3340348741 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f6ca00037c0 con 0x557ffae7fe40 2024-05-27T09:37:26.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.390+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/3340348741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c980044e0 con 0x557ffae7fe40 2024-05-27T09:37:26.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.390+0000 7f6c93400700 1 -- 172.21.15.44:0/3340348741 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6ca00095c0 con 0x557ffae7fe40 2024-05-27T09:37:26.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.390+0000 7f6cc31d1b80 10 client.25700 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:26.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.390+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/3340348741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557ffaeed060 con 0x557ffae7fe40 2024-05-27T09:37:26.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.391+0000 7f6c93400700 1 -- 172.21.15.44:0/3340348741 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c980044e0 con 0x557ffae7fe40 2024-05-27T09:37:26.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.391+0000 7f6c93400700 1 -- 172.21.15.44:0/3340348741 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6ca00e0ce0 con 0x557ffae7fe40 2024-05-27T09:37:26.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.391+0000 7f6cc31d1b80 10 client.25700 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:26.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.391+0000 7f6cc31d1b80 10 client.25700 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:37:26.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.391+0000 7f6cc31d1b80 20 client.25700 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:26.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.391+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x557ffb031e70 0x557ffb034320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.391+0000 7f6cc31d1b80 4 client.25700 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:37:26.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.391+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/3340348741 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ffaf2ed90 con 0x557ffb031e70 2024-05-27T09:37:26.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.393+0000 7f6cb1200700 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x557ffb031e70 0x557ffb034320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.394+0000 7f6cb1200700 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x557ffb031e70 0x557ffb034320 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b22a60b5afbf98e8 server_cookie=7f78495d6651d886 in_seq=0 out_seq=0 2024-05-27T09:37:26.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.394+0000 7f6c93400700 10 client.25700 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:37:26.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6c93400700 1 -- 172.21.15.44:0/3340348741 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557ffaf2ed90 con 0x557ffb031e70 2024-05-27T09:37:26.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6c93400700 10 client.25700 handle_command_reply: tid=0 2024-05-27T09:37:26.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6cc31d1b80 1 client.25700 shutdown 2024-05-27T09:37:26.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/3340348741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 msgr2=0x557ffb0084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:26.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffb0084e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009990 tx=0x7f6ca00e0af0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/3340348741 shutdown_connections 2024-05-27T09:37:26.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffb004d30 0x557ffb008f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffb0084e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x557ffb031e70 0x557ffb034320 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/3340348741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffaffedb0 0x557ffb008a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6c93400700 0 client.25700 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:37:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.395+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/3340348741 >> 172.21.15.44:0/3340348741 conn(0x557ffaedffc0 msgr2=0x557ffaff9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.396+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/3340348741 shutdown_connections 2024-05-27T09:37:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.396+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/3340348741 wait complete. 2024-05-27T09:37:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.396+0000 7f6cc31d1b80 20 client.25700 trim_cache size 0 max 16384 2024-05-27T09:37:26.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.428+0000 7f6cc31d1b80 1 Processor -- start 2024-05-27T09:37:26.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.428+0000 7f6cc31d1b80 1 -- start start 2024-05-27T09:37:26.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.429+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffaffe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.429+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffb002850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.429+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffb004d30 0x557ffb01c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.429+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c00fdd0 con 0x557ffb004d30 2024-05-27T09:37:26.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.429+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c00d660 con 0x557ffae7fe40 2024-05-27T09:37:26.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.429+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c00b4b0 con 0x557ffaffedb0 2024-05-27T09:37:26.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.429+0000 7f6cb1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffb002850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.429+0000 7f6cb1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffb002850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35134/0 (socket says 172.21.15.44:35134) 2024-05-27T09:37:26.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.429+0000 7f6cb1200700 1 -- 172.21.15.44:0/2442732247 learned_addr learned my addr 172.21.15.44:0/2442732247 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:26.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.429+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/2442732247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffaffe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.430+0000 7f6cb2600700 1 --2- 172.21.15.44:0/2442732247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffb004d30 0x557ffb01c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.430+0000 7f6cb1200700 1 -- 172.21.15.44:0/2442732247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 msgr2=0x557ffaffe870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:26.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.430+0000 7f6cb1200700 1 --2- 172.21.15.44:0/2442732247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffaffe870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.430+0000 7f6cb1200700 1 -- 172.21.15.44:0/2442732247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffb004d30 msgr2=0x557ffb01c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:26.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.430+0000 7f6cb1200700 1 --2- 172.21.15.44:0/2442732247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffb004d30 0x557ffb01c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.430+0000 7f6cb1200700 1 -- 172.21.15.44:0/2442732247 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ffb0092d0 con 0x557ffaffedb0 2024-05-27T09:37:26.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.430+0000 7f6cb1200700 1 --2- 172.21.15.44:0/2442732247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffb002850 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c98000c00 tx=0x7f6c98009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcd55790417e16c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:26.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.431+0000 7f6c9e000700 1 -- 172.21.15.44:0/2442732247 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98010db0 con 0x557ffaffedb0 2024-05-27T09:37:26.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.431+0000 7f6c9e000700 1 -- 172.21.15.44:0/2442732247 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c98029030 con 0x557ffaffedb0 2024-05-27T09:37:26.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.431+0000 7f6c9e000700 1 -- 172.21.15.44:0/2442732247 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c980126b0 con 0x557ffaffedb0 2024-05-27T09:37:26.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.432+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/2442732247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 msgr2=0x557ffb002850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:26.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.432+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/2442732247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffb002850 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c98000c00 tx=0x7f6c98009c30 comp rx=0 tx=0).stop 2024-05-27T09:37:26.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.432+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/2442732247 shutdown_connections 2024-05-27T09:37:26.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.432+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/2442732247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffb004d30 0x557ffb01c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.432+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/2442732247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffae7fe40 0x557ffaffe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.432+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/2442732247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffaffedb0 0x557ffb002850 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.432+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/2442732247 >> 172.21.15.44:0/2442732247 conn(0x557ffaedffc0 msgr2=0x557ffb002140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:26.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.433+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/2442732247 shutdown_connections 2024-05-27T09:37:26.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.433+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/2442732247 wait complete. 2024-05-27T09:37:26.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.434+0000 7f6cc31d1b80 1 -- start start 2024-05-27T09:37:26.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.434+0000 7f6cc31d1b80 1 Processor -- start 2024-05-27T09:37:26.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.435+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffae7fe40 0x557ffb019380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.435+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffaffedb0 0x557ffb0198c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.435+0000 7f6cc31d1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffb004d30 0x557ffb038b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.435+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c00bb20 con 0x557ffae7fe40 2024-05-27T09:37:26.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.435+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c00cf90 con 0x557ffaffedb0 2024-05-27T09:37:26.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.435+0000 7f6cc31d1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c00c1f0 con 0x557ffb004d30 2024-05-27T09:37:26.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.435+0000 7f6cc31d1b80 10 client.0 _ll_register_callbacks cb 0x557ffaec6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:26.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.435+0000 7f6cb1200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffaffedb0 0x557ffb0198c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.436+0000 7f6cb1200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffaffedb0 0x557ffb0198c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47560/0 (socket says 172.21.15.44:47560) 2024-05-27T09:37:26.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.436+0000 7f6cb1200700 1 -- 172.21.15.44:0/566541673 learned_addr learned my addr 172.21.15.44:0/566541673 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:26.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.436+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffae7fe40 0x557ffb019380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.436+0000 7f6cb1200700 1 -- 172.21.15.44:0/566541673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffb004d30 msgr2=0x557ffb038b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:26.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.436+0000 7f6cb1200700 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffb004d30 0x557ffb038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.436+0000 7f6cb1200700 1 -- 172.21.15.44:0/566541673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffae7fe40 msgr2=0x557ffb019380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:26.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.436+0000 7f6cb1200700 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffae7fe40 0x557ffb019380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.436+0000 7f6cb1200700 1 -- 172.21.15.44:0/566541673 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ffb039050 con 0x557ffaffedb0 2024-05-27T09:37:26.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.436+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffae7fe40 0x557ffb019380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:26.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.437+0000 7f6cb1200700 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffaffedb0 0x557ffb0198c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6c980039f0 tx=0x7f6c980053b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20225e7ee04ab053 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:26.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.437+0000 7f6c93400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:37:26.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.437+0000 7f6c93400700 1 -- 172.21.15.44:0/566541673 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98005580 con 0x557ffaffedb0 2024-05-27T09:37:26.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.437+0000 7f6c93400700 1 -- 172.21.15.44:0/566541673 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f6c98003a90 con 0x557ffaffedb0 2024-05-27T09:37:26.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.437+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/566541673 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c98029030 con 0x557ffaffedb0 2024-05-27T09:37:26.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.438+0000 7f6c93400700 1 -- 172.21.15.44:0/566541673 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6c98040050 con 0x557ffaffedb0 2024-05-27T09:37:26.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.438+0000 7f6cc31d1b80 10 client.25706 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:26.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.438+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/566541673 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557ffb09c0f0 con 0x557ffaffedb0 2024-05-27T09:37:26.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.439+0000 7f6c93400700 1 -- 172.21.15.44:0/566541673 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c98029030 con 0x557ffaffedb0 2024-05-27T09:37:26.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.439+0000 7f6c93400700 1 -- 172.21.15.44:0/566541673 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6c9800f040 con 0x557ffaffedb0 2024-05-27T09:37:26.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.439+0000 7f6cc31d1b80 10 client.25706 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:26.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.439+0000 7f6cc31d1b80 10 client.25706 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:37:26.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.439+0000 7f6cc31d1b80 20 client.25706 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:26.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.439+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x557ffb09c590 0x557ffb09e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:26.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.439+0000 7f6cc31d1b80 4 client.25706 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:37:26.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.439+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/566541673 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557ffaf28570 con 0x557ffb09c590 2024-05-27T09:37:26.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.440+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x557ffb09c590 0x557ffb09e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:26.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.440+0000 7f6cb1c00700 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x557ffb09c590 0x557ffb09e9b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2421a6a1a9ff778b server_cookie=f233d37c0837606b in_seq=0 out_seq=0 2024-05-27T09:37:26.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.440+0000 7f6c93400700 10 client.25706 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:37:26.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6c93400700 1 -- 172.21.15.44:0/566541673 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557ffaf28570 con 0x557ffb09c590 2024-05-27T09:37:26.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6c93400700 10 client.25706 handle_command_reply: tid=0 2024-05-27T09:37:26.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6cc31d1b80 1 client.25706 shutdown 2024-05-27T09:37:26.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/566541673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffaffedb0 msgr2=0x557ffb0198c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:26.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffaffedb0 0x557ffb0198c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6c980039f0 tx=0x7f6c980053b0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/566541673 shutdown_connections 2024-05-27T09:37:26.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ffb004d30 0x557ffb038b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x557ffb09c590 0x557ffb09e9b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557ffae7fe40 0x557ffb019380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6c93400700 0 client.25706 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:37:26.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6cc31d1b80 1 --2- 172.21.15.44:0/566541673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ffaffedb0 0x557ffb0198c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.441+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/566541673 >> 172.21.15.44:0/566541673 conn(0x557ffaedffc0 msgr2=0x557ffb003d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:26.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.442+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/566541673 shutdown_connections 2024-05-27T09:37:26.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.442+0000 7f6cc31d1b80 1 -- 172.21.15.44:0/566541673 wait complete. 2024-05-27T09:37:26.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.442+0000 7f6cc31d1b80 20 client.25706 trim_cache size 0 max 16384 2024-05-27T09:37:26.444 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:37:26.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.443+0000 7f6cb4600700 1 -- 172.21.15.44:0/3961789612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c940b2a70 msgr2=0x7f6c940b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:26.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.443+0000 7f6cb4600700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c940b2a70 0x7f6c940b4f80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x557ffafffb30 tx=0x7f6ca4004b40 comp rx=0 tx=0).stop 2024-05-27T09:37:26.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.443+0000 7f6cb4600700 1 -- 172.21.15.44:0/3961789612 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cac1034b0 msgr2=0x7f6cac0ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:26.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.443+0000 7f6cb4600700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cac1034b0 0x7f6cac0ff290 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0006010 tx=0x7f6ca0004320 comp rx=0 tx=0).stop 2024-05-27T09:37:26.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.444+0000 7f6cb4600700 1 -- 172.21.15.44:0/3961789612 shutdown_connections 2024-05-27T09:37:26.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.444+0000 7f6cb4600700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c940b2a70 0x7f6c940b4f80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.444+0000 7f6cb4600700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac108730 0x7f6cac0ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.444+0000 7f6cb4600700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cac1034b0 0x7f6cac0ff290 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.444+0000 7f6cb4600700 1 --2- 172.21.15.44:0/3961789612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cac107de0 0x7f6cac0ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:26.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.444+0000 7f6cb4600700 1 -- 172.21.15.44:0/3961789612 >> 172.21.15.44:0/3961789612 conn(0x7f6cac0fedb0 msgr2=0x7f6cac106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:26.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.444+0000 7f6cb4600700 1 -- 172.21.15.44:0/3961789612 shutdown_connections 2024-05-27T09:37:26.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:26.444+0000 7f6cb4600700 1 -- 172.21.15.44:0/3961789612 wait complete. 2024-05-27T09:37:26.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:26 smithi050 ceph-mon[36371]: pgmap v69: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-27T09:37:26.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:26 smithi189 ceph-mon[36671]: pgmap v69: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-27T09:37:27.275 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:37:27.902 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:37:28.361 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:28 smithi044 ceph-mon[31618]: pgmap v70: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:37:28.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:28 smithi050 ceph-mon[36371]: pgmap v70: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:37:28.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:28 smithi189 ceph-mon[36671]: pgmap v70: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:37:30.448 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:30 smithi044 ceph-mon[31618]: pgmap v71: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-27T09:37:30.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:30 smithi050 ceph-mon[36371]: pgmap v71: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-27T09:37:30.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:30 smithi189 ceph-mon[36671]: pgmap v71: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-27T09:37:31.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.363+0000 7fcb71600700 1 -- 172.21.15.44:0/1370831813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c108d20 msgr2=0x7fcb6c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:31.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.363+0000 7fcb71600700 1 --2- 172.21.15.44:0/1370831813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c108d20 0x7fcb6c109130 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009990 tx=0x7fcb58009ca0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.364+0000 7fcb71600700 1 -- 172.21.15.44:0/1370831813 shutdown_connections 2024-05-27T09:37:31.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.364+0000 7fcb71600700 1 --2- 172.21.15.44:0/1370831813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb6c109670 0x7fcb6c110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.364+0000 7fcb71600700 1 --2- 172.21.15.44:0/1370831813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb6c1043f0 0x7fcb6c104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.364+0000 7fcb71600700 1 --2- 172.21.15.44:0/1370831813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c108d20 0x7fcb6c109130 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.364+0000 7fcb71600700 1 -- 172.21.15.44:0/1370831813 >> 172.21.15.44:0/1370831813 conn(0x7fcb6c079d00 msgr2=0x7fcb6c07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:31.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.365+0000 7fcb71600700 1 -- 172.21.15.44:0/1370831813 shutdown_connections 2024-05-27T09:37:31.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.365+0000 7fcb71600700 1 -- 172.21.15.44:0/1370831813 wait complete. 2024-05-27T09:37:31.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.366+0000 7fcb71600700 1 Processor -- start 2024-05-27T09:37:31.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.367+0000 7fcb71600700 1 -- start start 2024-05-27T09:37:31.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.367+0000 7fcb71600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1043f0 0x7fcb6c1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.367+0000 7fcb71600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb6c108d20 0x7fcb6c1a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.367+0000 7fcb71600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb6c109670 0x7fcb6c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.367+0000 7fcb71600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c1142d0 con 0x7fcb6c109670 2024-05-27T09:37:31.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.367+0000 7fcb71600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c114150 con 0x7fcb6c1043f0 2024-05-27T09:37:31.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.367+0000 7fcb71600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c114450 con 0x7fcb6c108d20 2024-05-27T09:37:31.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.368+0000 7fcb6b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb6c108d20 0x7fcb6c1a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.368+0000 7fcb6be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1043f0 0x7fcb6c1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.368+0000 7fcb6be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1043f0 0x7fcb6c1a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47588/0 (socket says 172.21.15.44:47588) 2024-05-27T09:37:31.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.368+0000 7fcb6be00700 1 -- 172.21.15.44:0/2632505496 learned_addr learned my addr 172.21.15.44:0/2632505496 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:31.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.368+0000 7fcb70c00700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb6c109670 0x7fcb6c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.369+0000 7fcb6be00700 1 -- 172.21.15.44:0/2632505496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb6c108d20 msgr2=0x7fcb6c1a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:31.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.369+0000 7fcb6be00700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb6c108d20 0x7fcb6c1a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.369+0000 7fcb6be00700 1 -- 172.21.15.44:0/2632505496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb6c109670 msgr2=0x7fcb6c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:31.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.369+0000 7fcb6be00700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb6c109670 0x7fcb6c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.369+0000 7fcb6be00700 1 -- 172.21.15.44:0/2632505496 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c19b1b0 con 0x7fcb6c1043f0 2024-05-27T09:37:31.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.369+0000 7fcb70c00700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb6c109670 0x7fcb6c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:31.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.370+0000 7fcb6be00700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1043f0 0x7fcb6c1a0890 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00d980 tx=0x7fcb5c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:31.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.370+0000 7fcb68c00700 1 -- 172.21.15.44:0/2632505496 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c016070 con 0x7fcb6c1043f0 2024-05-27T09:37:31.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.370+0000 7fcb71600700 1 -- 172.21.15.44:0/2632505496 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6c19b4a0 con 0x7fcb6c1043f0 2024-05-27T09:37:31.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.370+0000 7fcb68c00700 1 -- 172.21.15.44:0/2632505496 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb5c004830 con 0x7fcb6c1043f0 2024-05-27T09:37:31.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.370+0000 7fcb68c00700 1 -- 172.21.15.44:0/2632505496 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c005540 con 0x7fcb6c1043f0 2024-05-27T09:37:31.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.370+0000 7fcb71600700 1 -- 172.21.15.44:0/2632505496 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6c1a7740 con 0x7fcb6c1043f0 2024-05-27T09:37:31.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.377+0000 7fcb68c00700 1 -- 172.21.15.44:0/2632505496 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fcb5c00f690 con 0x7fcb6c1043f0 2024-05-27T09:37:31.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.377+0000 7fcb817cdb80 1 Processor -- start 2024-05-27T09:37:31.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.378+0000 7fcb817cdb80 1 -- start start 2024-05-27T09:37:31.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.378+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e08bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.378+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 0x55864e096ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.378+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e099020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.378+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55864dfbae40 con 0x55864e08c820 2024-05-27T09:37:31.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.378+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55864dfaa9e0 con 0x55864e08c410 2024-05-27T09:37:31.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.378+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55864dfaea20 con 0x55864df0de40 2024-05-27T09:37:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.379+0000 7fcb70c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e08bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.379+0000 7fcb70c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e08bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35166/0 (socket says 172.21.15.44:35166) 2024-05-27T09:37:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.379+0000 7fcb70c00700 1 -- 172.21.15.44:0/114576112 learned_addr learned my addr 172.21.15.44:0/114576112 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.379+0000 7fcb6be00700 1 --2- 172.21.15.44:0/114576112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 0x55864e096ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.379+0000 7fcb6b400700 1 --2- 172.21.15.44:0/114576112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e099020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.380+0000 7fcb70c00700 1 -- 172.21.15.44:0/114576112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 msgr2=0x55864e096ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.380+0000 7fcb70c00700 1 --2- 172.21.15.44:0/114576112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 0x55864e096ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.380+0000 7fcb70c00700 1 -- 172.21.15.44:0/114576112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 msgr2=0x55864e099020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.380+0000 7fcb70c00700 1 --2- 172.21.15.44:0/114576112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e099020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.380+0000 7fcb70c00700 1 -- 172.21.15.44:0/114576112 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55864df73420 con 0x55864df0de40 2024-05-27T09:37:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.380+0000 7fcb6be00700 1 --2- 172.21.15.44:0/114576112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 0x55864e096ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.380+0000 7fcb6b400700 1 --2- 172.21.15.44:0/114576112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e099020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.380+0000 7fcb70c00700 1 --2- 172.21.15.44:0/114576112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e08bed0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009550 tx=0x7fcb640079f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9aa3c088fede2cc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.381+0000 7fcb68c00700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb500b2af0 0x7fcb500b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.381+0000 7fcb61600700 1 -- 172.21.15.44:0/114576112 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb64019020 con 0x55864df0de40 2024-05-27T09:37:31.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.381+0000 7fcb68c00700 1 -- 172.21.15.44:0/2632505496 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fcb5c0e2390 con 0x7fcb6c1043f0 2024-05-27T09:37:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.382+0000 7fcb61600700 1 -- 172.21.15.44:0/114576112 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb64015d00 con 0x55864df0de40 2024-05-27T09:37:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.382+0000 7fcb61600700 1 -- 172.21.15.44:0/114576112 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb64014cd0 con 0x55864df0de40 2024-05-27T09:37:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.382+0000 7fcb6be00700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb500b2af0 0x7fcb500b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.383+0000 7fcb817cdb80 1 -- 172.21.15.44:0/114576112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 msgr2=0x55864e08bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.383+0000 7fcb6be00700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb500b2af0 0x7fcb500b5000 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55864e089050 tx=0x7fcb5c000bc0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.383+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/114576112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e08bed0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009550 tx=0x7fcb640079f0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.383+0000 7fcb817cdb80 1 -- 172.21.15.44:0/114576112 shutdown_connections 2024-05-27T09:37:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.383+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/114576112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e099020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.383+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/114576112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e08bed0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.383+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/114576112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 0x55864e096ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.383+0000 7fcb817cdb80 1 -- 172.21.15.44:0/114576112 >> 172.21.15.44:0/114576112 conn(0x55864df6dfc0 msgr2=0x55864e087d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.383+0000 7fcb817cdb80 1 -- 172.21.15.44:0/114576112 shutdown_connections 2024-05-27T09:37:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.383+0000 7fcb817cdb80 1 -- 172.21.15.44:0/114576112 wait complete. 2024-05-27T09:37:31.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.384+0000 7fcb817cdb80 1 -- start start 2024-05-27T09:37:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.384+0000 7fcb817cdb80 1 Processor -- start 2024-05-27T09:37:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.385+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e0a5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.385+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 0x55864e0a5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.385+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e09f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.385+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55864deb0fa0 con 0x55864e08c820 2024-05-27T09:37:31.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.385+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55864df4b760 con 0x55864e08c410 2024-05-27T09:37:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.385+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55864e09dda0 con 0x55864df0de40 2024-05-27T09:37:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.385+0000 7fcb817cdb80 10 client.0 _ll_register_callbacks cb 0x55864df54c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.386+0000 7fcb6b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 0x55864e0a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.386+0000 7fcb70c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e0a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.386+0000 7fcb70c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e0a5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35170/0 (socket says 172.21.15.44:35170) 2024-05-27T09:37:31.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.386+0000 7fcb70c00700 1 -- 172.21.15.44:0/1100378286 learned_addr learned my addr 172.21.15.44:0/1100378286 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.386+0000 7fcb6b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 0x55864e0a5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47606/0 (socket says 172.21.15.44:47606) 2024-05-27T09:37:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.386+0000 7fcb70c00700 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e09f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.386+0000 7fcb6b400700 1 -- 172.21.15.44:0/1100378286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 msgr2=0x55864e0a5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.386+0000 7fcb6b400700 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e0a5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.386+0000 7fcb6b400700 1 -- 172.21.15.44:0/1100378286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 msgr2=0x55864e09f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.386+0000 7fcb6b400700 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e09f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.387+0000 7fcb6b400700 1 -- 172.21.15.44:0/1100378286 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55864defa8d0 con 0x55864e08c410 2024-05-27T09:37:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.387+0000 7fcb6b400700 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 0x55864e0a5570 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcb580098e0 tx=0x7fcb58009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67960758b7221b94 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.387+0000 7fcb56000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:37:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.388+0000 7fcb56000700 1 -- 172.21.15.44:0/1100378286 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb580052b0 con 0x55864e08c410 2024-05-27T09:37:31.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.388+0000 7fcb817cdb80 1 -- 172.21.15.44:0/1100378286 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb64015d00 con 0x55864e08c410 2024-05-27T09:37:31.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.388+0000 7fcb56000700 1 -- 172.21.15.44:0/1100378286 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fcb58005450 con 0x55864e08c410 2024-05-27T09:37:31.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.388+0000 7fcb56000700 1 -- 172.21.15.44:0/1100378286 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcb58041050 con 0x55864e08c410 2024-05-27T09:37:31.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.388+0000 7fcb817cdb80 10 client.25718 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:31.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.388+0000 7fcb817cdb80 1 -- 172.21.15.44:0/1100378286 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55864df7b060 con 0x55864e08c410 2024-05-27T09:37:31.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.389+0000 7fcb56000700 1 -- 172.21.15.44:0/1100378286 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb64015d00 con 0x55864e08c410 2024-05-27T09:37:31.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.390+0000 7fcb56000700 1 -- 172.21.15.44:0/1100378286 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fcb5800f760 con 0x55864e08c410 2024-05-27T09:37:31.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.390+0000 7fcb817cdb80 10 client.25718 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:31.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.390+0000 7fcb817cdb80 10 client.25718 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:31.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.390+0000 7fcb817cdb80 20 client.25718 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:31.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.390+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55864e090e90 0x55864e0932e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.390+0000 7fcb817cdb80 4 client.25718 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:37:31.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.390+0000 7fcb817cdb80 1 -- 172.21.15.44:0/1100378286 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55864dfbcd90 con 0x55864e090e90 2024-05-27T09:37:31.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.390+0000 7fcb70c00700 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55864e090e90 0x55864e0932e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.391+0000 7fcb70c00700 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55864e090e90 0x55864e0932e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89e7364aada02a08 server_cookie=aa2db6561587729c in_seq=0 out_seq=0 2024-05-27T09:37:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.391+0000 7fcb56000700 10 client.25718 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.392+0000 7fcb56000700 1 -- 172.21.15.44:0/1100378286 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55864dfbcd90 con 0x55864e090e90 2024-05-27T09:37:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.392+0000 7fcb56000700 10 client.25718 handle_command_reply: tid=0 2024-05-27T09:37:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.392+0000 7fcb817cdb80 1 client.25718 shutdown 2024-05-27T09:37:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.392+0000 7fcb817cdb80 1 -- 172.21.15.44:0/1100378286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 msgr2=0x55864e0a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.392+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 0x55864e0a5570 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcb580098e0 tx=0x7fcb58009ee0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.393+0000 7fcb817cdb80 1 -- 172.21.15.44:0/1100378286 shutdown_connections 2024-05-27T09:37:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.393+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55864e090e90 0x55864e0932e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.393+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e09f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.393+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e0a5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.393+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/1100378286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c410 0x55864e0a5570 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.393+0000 7fcb817cdb80 1 -- 172.21.15.44:0/1100378286 >> 172.21.15.44:0/1100378286 conn(0x55864df6dfc0 msgr2=0x55864e086ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.393+0000 7fcb56000700 0 client.25718 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.393+0000 7fcb817cdb80 1 -- 172.21.15.44:0/1100378286 shutdown_connections 2024-05-27T09:37:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.393+0000 7fcb817cdb80 1 -- 172.21.15.44:0/1100378286 wait complete. 2024-05-27T09:37:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.393+0000 7fcb817cdb80 20 client.25718 trim_cache size 0 max 16384 2024-05-27T09:37:31.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.425+0000 7fcb817cdb80 1 Processor -- start 2024-05-27T09:37:31.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.425+0000 7fcb817cdb80 1 -- start start 2024-05-27T09:37:31.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.426+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864df0de40 0x55864e0a48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.426+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864e08c410 0x55864e098a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.426+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e098f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.426+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c1af320 con 0x55864e08c820 2024-05-27T09:37:31.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.426+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c1acbb0 con 0x55864df0de40 2024-05-27T09:37:31.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.426+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c1aaa00 con 0x55864e08c410 2024-05-27T09:37:31.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.426+0000 7fcb6b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864e08c410 0x55864e098a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.426+0000 7fcb6b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864e08c410 0x55864e098a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35186/0 (socket says 172.21.15.44:35186) 2024-05-27T09:37:31.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.427+0000 7fcb6b400700 1 -- 172.21.15.44:0/3904635902 learned_addr learned my addr 172.21.15.44:0/3904635902 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:31.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.427+0000 7fcb70c00700 1 --2- 172.21.15.44:0/3904635902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e098f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.427+0000 7fcb6b400700 1 -- 172.21.15.44:0/3904635902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864df0de40 msgr2=0x55864e0a48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:31.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.427+0000 7fcb70c00700 1 --2- 172.21.15.44:0/3904635902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864df0de40 0x55864e0a48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.427+0000 7fcb6b400700 1 --2- 172.21.15.44:0/3904635902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864df0de40 0x55864e0a48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.427+0000 7fcb6b400700 1 -- 172.21.15.44:0/3904635902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 msgr2=0x55864e098f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:31.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.427+0000 7fcb6b400700 1 --2- 172.21.15.44:0/3904635902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e098f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.427+0000 7fcb6b400700 1 -- 172.21.15.44:0/3904635902 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55864e09f210 con 0x55864e08c410 2024-05-27T09:37:31.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.427+0000 7fcb70c00700 1 --2- 172.21.15.44:0/3904635902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e098f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:37:31.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.428+0000 7fcb6b400700 1 --2- 172.21.15.44:0/3904635902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864e08c410 0x55864e098a40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800fc90 tx=0x7fcb58003700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6499ac4bb7c020f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:31.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.428+0000 7fcb60c00700 1 -- 172.21.15.44:0/3904635902 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5800f660 con 0x55864e08c410 2024-05-27T09:37:31.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.428+0000 7fcb60c00700 1 -- 172.21.15.44:0/3904635902 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb58038070 con 0x55864e08c410 2024-05-27T09:37:31.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.428+0000 7fcb60c00700 1 -- 172.21.15.44:0/3904635902 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5800bae0 con 0x55864e08c410 2024-05-27T09:37:31.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.430+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3904635902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864e08c410 msgr2=0x55864e098a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:31.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.430+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/3904635902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864e08c410 0x55864e098a40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800fc90 tx=0x7fcb58003700 comp rx=0 tx=0).stop 2024-05-27T09:37:31.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.430+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3904635902 shutdown_connections 2024-05-27T09:37:31.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.430+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/3904635902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c820 0x55864e098f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.430+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/3904635902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864df0de40 0x55864e0a48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.430+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/3904635902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864e08c410 0x55864e098a40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.430+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3904635902 >> 172.21.15.44:0/3904635902 conn(0x55864df6dfc0 msgr2=0x55864e0a5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:31.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.430+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3904635902 shutdown_connections 2024-05-27T09:37:31.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.431+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3904635902 wait complete. 2024-05-27T09:37:31.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.431+0000 7fcb817cdb80 1 -- start start 2024-05-27T09:37:31.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.432+0000 7fcb817cdb80 1 Processor -- start 2024-05-27T09:37:31.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.432+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e0b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.433+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c410 0x55864e0af810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.433+0000 7fcb817cdb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c820 0x55864e0afe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.433+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c1ab070 con 0x55864e08c410 2024-05-27T09:37:31.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.433+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c1ac4e0 con 0x55864e08c820 2024-05-27T09:37:31.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.433+0000 7fcb817cdb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c1ab740 con 0x55864df0de40 2024-05-27T09:37:31.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.433+0000 7fcb817cdb80 10 client.0 _ll_register_callbacks cb 0x55864df54c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:31.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.433+0000 7fcb70c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c820 0x55864e0afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.433+0000 7fcb6b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c410 0x55864e0af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.433+0000 7fcb70c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e0b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.433+0000 7fcb70c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e0b2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35196/0 (socket says 172.21.15.44:35196) 2024-05-27T09:37:31.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.433+0000 7fcb70c00700 1 -- 172.21.15.44:0/3688819258 learned_addr learned my addr 172.21.15.44:0/3688819258 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:31.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.434+0000 7fcb70c00700 1 -- 172.21.15.44:0/3688819258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 msgr2=0x55864e0b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:31.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.434+0000 7fcb70c00700 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e0b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.434+0000 7fcb70c00700 1 -- 172.21.15.44:0/3688819258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c410 msgr2=0x55864e0af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:31.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.434+0000 7fcb70c00700 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c410 0x55864e0af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.434+0000 7fcb70c00700 1 -- 172.21.15.44:0/3688819258 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55864e0b0600 con 0x55864e08c820 2024-05-27T09:37:31.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.434+0000 7fcb6b400700 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c410 0x55864e0af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:31.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.434+0000 7fcb70c00700 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c820 0x55864e0afe60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55864e0939b0 tx=0x7fcb64000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b71d69387b5b1cda server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:31.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.434+0000 7fcb56000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:37:31.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.435+0000 7fcb56000700 1 -- 172.21.15.44:0/3688819258 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb64029040 con 0x55864e08c820 2024-05-27T09:37:31.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.435+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3688819258 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb58038070 con 0x55864e08c820 2024-05-27T09:37:31.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.435+0000 7fcb56000700 1 -- 172.21.15.44:0/3688819258 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fcb64014880 con 0x55864e08c820 2024-05-27T09:37:31.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.436+0000 7fcb56000700 1 -- 172.21.15.44:0/3688819258 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcb64019040 con 0x55864e08c820 2024-05-27T09:37:31.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.436+0000 7fcb817cdb80 10 client.25721 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:31.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.436+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3688819258 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55864e110af0 con 0x55864e08c820 2024-05-27T09:37:31.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.436+0000 7fcb56000700 1 -- 172.21.15.44:0/3688819258 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb58038070 con 0x55864e08c820 2024-05-27T09:37:31.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.437+0000 7fcb56000700 1 -- 172.21.15.44:0/3688819258 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fcb6402c270 con 0x55864e08c820 2024-05-27T09:37:31.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.437+0000 7fcb817cdb80 10 client.25721 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:31.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.437+0000 7fcb817cdb80 10 client.25721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:31.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.437+0000 7fcb817cdb80 20 client.25721 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:31.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.437+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55864e111070 0x55864e113490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:31.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.437+0000 7fcb817cdb80 4 client.25721 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:37:31.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.437+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3688819258 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55864dfb6570 con 0x55864e111070 2024-05-27T09:37:31.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.437+0000 7fcb6b400700 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55864e111070 0x55864e113490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:31.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.438+0000 7fcb6b400700 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55864e111070 0x55864e113490 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94b62013af11e5de server_cookie=c27552083ee72644 in_seq=0 out_seq=0 2024-05-27T09:37:31.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.438+0000 7fcb56000700 10 client.25721 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:31.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.438+0000 7fcb56000700 1 -- 172.21.15.44:0/3688819258 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55864dfb6570 con 0x55864e111070 2024-05-27T09:37:31.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.438+0000 7fcb56000700 10 client.25721 handle_command_reply: tid=0 2024-05-27T09:37:31.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.438+0000 7fcb817cdb80 1 client.25721 shutdown 2024-05-27T09:37:31.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.439+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3688819258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c820 msgr2=0x55864e0afe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:31.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.439+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c820 0x55864e0afe60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55864e0939b0 tx=0x7fcb64000f40 comp rx=0 tx=0).stop 2024-05-27T09:37:31.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.439+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3688819258 shutdown_connections 2024-05-27T09:37:31.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.439+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55864e111070 0x55864e113490 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.439+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55864e08c820 0x55864e0afe60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.439+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55864df0de40 0x55864e0b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.439+0000 7fcb817cdb80 1 --2- 172.21.15.44:0/3688819258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864e08c410 0x55864e0af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.439+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3688819258 >> 172.21.15.44:0/3688819258 conn(0x55864df6dfc0 msgr2=0x55864e08d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:31.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.439+0000 7fcb56000700 0 client.25721 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:31.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.440+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3688819258 shutdown_connections 2024-05-27T09:37:31.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.440+0000 7fcb817cdb80 1 -- 172.21.15.44:0/3688819258 wait complete. 2024-05-27T09:37:31.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.440+0000 7fcb817cdb80 20 client.25721 trim_cache size 0 max 16384 2024-05-27T09:37:31.443 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:37:31.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.441+0000 7fcb71600700 1 -- 172.21.15.44:0/2632505496 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb500b2af0 msgr2=0x7fcb500b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:31.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.441+0000 7fcb71600700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb500b2af0 0x7fcb500b5000 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55864e089050 tx=0x7fcb5c000bc0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.441+0000 7fcb71600700 1 -- 172.21.15.44:0/2632505496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1043f0 msgr2=0x7fcb6c1a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:31.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.441+0000 7fcb71600700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1043f0 0x7fcb6c1a0890 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00d980 tx=0x7fcb5c00dc90 comp rx=0 tx=0).stop 2024-05-27T09:37:31.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.441+0000 7fcb71600700 1 -- 172.21.15.44:0/2632505496 shutdown_connections 2024-05-27T09:37:31.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.441+0000 7fcb71600700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcb500b2af0 0x7fcb500b5000 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.441+0000 7fcb71600700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb6c109670 0x7fcb6c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.441+0000 7fcb71600700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1043f0 0x7fcb6c1a0890 secure :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00d980 tx=0x7fcb5c00dc90 comp rx=0 tx=0).stop 2024-05-27T09:37:31.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.441+0000 7fcb71600700 1 --2- 172.21.15.44:0/2632505496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb6c108d20 0x7fcb6c1a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:31.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.441+0000 7fcb71600700 1 -- 172.21.15.44:0/2632505496 >> 172.21.15.44:0/2632505496 conn(0x7fcb6c079d00 msgr2=0x7fcb6c101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:31.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.441+0000 7fcb71600700 1 -- 172.21.15.44:0/2632505496 shutdown_connections 2024-05-27T09:37:31.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:31.442+0000 7fcb71600700 1 -- 172.21.15.44:0/2632505496 wait complete. 2024-05-27T09:37:32.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:32 smithi044 ceph-mon[31618]: pgmap v72: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:37:32.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:32 smithi050 ceph-mon[36371]: pgmap v72: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:37:32.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:32 smithi189 ceph-mon[36671]: pgmap v72: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:37:33.221 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:37:33.221 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:37:33.728 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:37:34.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:34 smithi044 ceph-mon[31618]: pgmap v73: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:37:34.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:34 smithi050 ceph-mon[36371]: pgmap v73: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:37:34.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:34 smithi189 ceph-mon[36671]: pgmap v73: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-27T09:37:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.076+0000 7f54d1600640 1 -- 172.21.15.44:0/3831110730 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c40318a0 con 0x7f54d4075660 2024-05-27T09:37:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.077+0000 7f54d9a00640 1 -- 172.21.15.44:0/3831110730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54d4075660 msgr2=0x7f54d4075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.077+0000 7f54d9a00640 1 --2- 172.21.15.44:0/3831110730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54d4075660 0x7f54d4075a70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f54c4009900 tx=0x7f54c402a480 comp rx=0 tx=0).stop 2024-05-27T09:37:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.077+0000 7f54d9a00640 1 -- 172.21.15.44:0/3831110730 shutdown_connections 2024-05-27T09:37:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.077+0000 7f54d9a00640 1 --2- 172.21.15.44:0/3831110730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d41104a0 0x7f54d4112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.077+0000 7f54d9a00640 1 --2- 172.21.15.44:0/3831110730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54d410db40 0x7f54d410ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.077+0000 7f54d9a00640 1 --2- 172.21.15.44:0/3831110730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54d4075660 0x7f54d4075a70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.077+0000 7f54d9a00640 1 -- 172.21.15.44:0/3831110730 >> 172.21.15.44:0/3831110730 conn(0x7f54d40ff480 msgr2=0x7f54d41018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.077+0000 7f54d9a00640 1 -- 172.21.15.44:0/3831110730 shutdown_connections 2024-05-27T09:37:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.077+0000 7f54d9a00640 1 -- 172.21.15.44:0/3831110730 wait complete. 2024-05-27T09:37:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.079+0000 7f54d9a00640 1 Processor -- start 2024-05-27T09:37:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.079+0000 7f54d9a00640 1 -- start start 2024-05-27T09:37:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.080+0000 7f54d9a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54d4075660 0x7f54d410c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.080+0000 7f54d9a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d410db40 0x7f54d410c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.080+0000 7f54d9a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54d41104a0 0x7f54d410cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.080+0000 7f54d9a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f54d4104790 con 0x7f54d4075660 2024-05-27T09:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.080+0000 7f54d9a00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f54d4104610 con 0x7f54d410db40 2024-05-27T09:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.080+0000 7f54d9a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f54d4104910 con 0x7f54d41104a0 2024-05-27T09:37:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.081+0000 7f54d2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54d4075660 0x7f54d410c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.081+0000 7f54d3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54d41104a0 0x7f54d410cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.081+0000 7f54d2000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d410db40 0x7f54d410c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.081+0000 7f54d3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54d41104a0 0x7f54d410cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55186/0 (socket says 172.21.15.44:55186) 2024-05-27T09:37:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.081+0000 7f54d3400640 1 -- 172.21.15.44:0/514715295 learned_addr learned my addr 172.21.15.44:0/514715295 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.082+0000 7f54d3400640 1 -- 172.21.15.44:0/514715295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d410db40 msgr2=0x7f54d410c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.082+0000 7f54d3400640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d410db40 0x7f54d410c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.082+0000 7f54d3400640 1 -- 172.21.15.44:0/514715295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54d4075660 msgr2=0x7f54d410c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.082+0000 7f54d3400640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54d4075660 0x7f54d410c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.082+0000 7f54d3400640 1 -- 172.21.15.44:0/514715295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d41b57b0 con 0x7f54d41104a0 2024-05-27T09:37:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.082+0000 7f54d3400640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54d41104a0 0x7f54d410cd70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00efd0 tx=0x7f54cc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.083+0000 7f54cb400640 1 -- 172.21.15.44:0/514715295 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc003b70 con 0x7f54d41104a0 2024-05-27T09:37:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.083+0000 7f54d9a00640 1 -- 172.21.15.44:0/514715295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d41b5aa0 con 0x7f54d41104a0 2024-05-27T09:37:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.083+0000 7f54cb400640 1 -- 172.21.15.44:0/514715295 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54cc003d10 con 0x7f54d41104a0 2024-05-27T09:37:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.083+0000 7f54d9a00640 1 -- 172.21.15.44:0/514715295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d41b5ff0 con 0x7f54d41104a0 2024-05-27T09:37:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.083+0000 7f54cb400640 1 -- 172.21.15.44:0/514715295 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc0108a0 con 0x7f54d41104a0 2024-05-27T09:37:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.087+0000 7f54e0a10740 1 Processor -- start 2024-05-27T09:37:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.089+0000 7f54cb400640 1 -- 172.21.15.44:0/514715295 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f54cc004020 con 0x7f54d41104a0 2024-05-27T09:37:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.090+0000 7f54e0a10740 1 -- start start 2024-05-27T09:37:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.090+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 0x562e498f4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.090+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 0x562e498f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.090+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495314d0 0x562e498f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.090+0000 7f54e0a10740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562e49865b90 con 0x562e495521b0 2024-05-27T09:37:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.090+0000 7f54e0a10740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562e49855730 con 0x562e495292f0 2024-05-27T09:37:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.090+0000 7f54e0a10740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562e49859770 con 0x562e495314d0 2024-05-27T09:37:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.092+0000 7f54d2a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 0x562e498f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.092+0000 7f54d2a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 0x562e498f4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43998/0 (socket says 172.21.15.44:43998) 2024-05-27T09:37:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.092+0000 7f54d2a00640 1 -- 172.21.15.44:0/664004830 learned_addr learned my addr 172.21.15.44:0/664004830 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.092+0000 7f54d2a00640 1 -- 172.21.15.44:0/664004830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495314d0 msgr2=0x562e498f52e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:37:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.092+0000 7f54d2a00640 1 --2- 172.21.15.44:0/664004830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495314d0 0x562e498f52e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.092+0000 7f54d2000640 1 --2- 172.21.15.44:0/664004830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 0x562e498f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.092+0000 7f54d2a00640 1 -- 172.21.15.44:0/664004830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 msgr2=0x562e498f4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.092+0000 7f54d2a00640 1 --2- 172.21.15.44:0/664004830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 0x562e498f4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.092+0000 7f54d2a00640 1 -- 172.21.15.44:0/664004830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e497ee8b0 con 0x562e495292f0 2024-05-27T09:37:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.092+0000 7f54d2000640 1 --2- 172.21.15.44:0/664004830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 0x562e498f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:37:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.093+0000 7f54cb400640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54b80b29d0 0x7f54b80b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.093+0000 7f54d2000640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54b80b29d0 0x7f54b80b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.094+0000 7f54cb400640 1 -- 172.21.15.44:0/514715295 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f54cc0e2920 con 0x7f54d41104a0 2024-05-27T09:37:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.094+0000 7f54d2a00640 1 --2- 172.21.15.44:0/664004830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 0x562e498f4da0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f54c4009900 tx=0x7f54c4032b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49ff3e6b00bba86 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.095+0000 7f54bfe00640 1 -- 172.21.15.44:0/664004830 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4040070 con 0x562e495292f0 2024-05-27T09:37:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.095+0000 7f54bfe00640 1 -- 172.21.15.44:0/664004830 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54c4004030 con 0x562e495292f0 2024-05-27T09:37:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.095+0000 7f54bfe00640 1 -- 172.21.15.44:0/664004830 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4035040 con 0x562e495292f0 2024-05-27T09:37:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.096+0000 7f54e0a10740 1 -- 172.21.15.44:0/664004830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 msgr2=0x562e498f4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.096+0000 7f54e0a10740 1 --2- 172.21.15.44:0/664004830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 0x562e498f4da0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f54c4009900 tx=0x7f54c4032b20 comp rx=0 tx=0).stop 2024-05-27T09:37:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.096+0000 7f54d2000640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54b80b29d0 0x7f54b80b4ee0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x562e498f19d0 tx=0x7f54c0005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.099+0000 7f54e0a10740 1 -- 172.21.15.44:0/664004830 shutdown_connections 2024-05-27T09:37:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.099+0000 7f54e0a10740 1 --2- 172.21.15.44:0/664004830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495314d0 0x562e498f52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.099+0000 7f54e0a10740 1 --2- 172.21.15.44:0/664004830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 0x562e498f4da0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.099+0000 7f54e0a10740 1 --2- 172.21.15.44:0/664004830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 0x562e498f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.099+0000 7f54e0a10740 1 -- 172.21.15.44:0/664004830 >> 172.21.15.44:0/664004830 conn(0x562e49546300 msgr2=0x562e498f1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.099+0000 7f54e0a10740 1 -- 172.21.15.44:0/664004830 shutdown_connections 2024-05-27T09:37:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.099+0000 7f54e0a10740 1 -- 172.21.15.44:0/664004830 wait complete. 2024-05-27T09:37:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.100+0000 7f54e0a10740 1 -- start start 2024-05-27T09:37:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.100+0000 7f54e0a10740 1 Processor -- start 2024-05-27T09:37:35.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.101+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e498fe990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.101+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495314d0 0x562e498feed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.101+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 0x562e4990acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.101+0000 7f54e0a10740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562e4953b790 con 0x562e495521b0 2024-05-27T09:37:35.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.101+0000 7f54e0a10740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562e497fdbc0 con 0x562e495314d0 2024-05-27T09:37:35.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.101+0000 7f54e0a10740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562e49901930 con 0x562e495292f0 2024-05-27T09:37:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.101+0000 7f54e0a10740 10 client.0 _ll_register_callbacks cb 0x562e497db400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.102+0000 7f54d2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e498fe990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.102+0000 7f54d2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e498fe990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55194/0 (socket says 172.21.15.44:55194) 2024-05-27T09:37:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.102+0000 7f54d2a00640 1 -- 172.21.15.44:0/2003030498 learned_addr learned my addr 172.21.15.44:0/2003030498 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.102+0000 7f54d2000640 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495314d0 0x562e498feed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.102+0000 7f54d3400640 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 0x562e4990acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.103+0000 7f54d2000640 1 -- 172.21.15.44:0/2003030498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 msgr2=0x562e498fe990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.103+0000 7f54d2000640 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e498fe990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.103+0000 7f54d2000640 1 -- 172.21.15.44:0/2003030498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 msgr2=0x562e4990acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.103+0000 7f54d2000640 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 0x562e4990acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.103+0000 7f54d2000640 1 -- 172.21.15.44:0/2003030498 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e4956a210 con 0x562e495314d0 2024-05-27T09:37:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.103+0000 7f54d2a00640 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e498fe990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.103+0000 7f54d3400640 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 0x562e4990acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:35.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.104+0000 7f54d2000640 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495314d0 0x562e498feed0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f54c00044e0 tx=0x7f54c00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a6b63bbd715ce29 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:35.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.104+0000 7f54bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:37:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.104+0000 7f54bcc00640 1 -- 172.21.15.44:0/2003030498 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0005220 con 0x562e495314d0 2024-05-27T09:37:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.104+0000 7f54bcc00640 1 -- 172.21.15.44:0/2003030498 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f54c00053c0 con 0x562e495314d0 2024-05-27T09:37:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.105+0000 7f54e0a10740 1 -- 172.21.15.44:0/2003030498 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54c4004030 con 0x562e495314d0 2024-05-27T09:37:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.105+0000 7f54bcc00640 1 -- 172.21.15.44:0/2003030498 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f54c00055c0 con 0x562e495314d0 2024-05-27T09:37:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.105+0000 7f54e0a10740 10 client.25733 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.105+0000 7f54e0a10740 1 -- 172.21.15.44:0/2003030498 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562e49636620 con 0x562e495314d0 2024-05-27T09:37:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.106+0000 7f54bcc00640 1 -- 172.21.15.44:0/2003030498 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54c4004030 con 0x562e495314d0 2024-05-27T09:37:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.106+0000 7f54bcc00640 1 -- 172.21.15.44:0/2003030498 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f54c0011b30 con 0x562e495314d0 2024-05-27T09:37:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.107+0000 7f54e0a10740 10 client.25733 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.107+0000 7f54e0a10740 10 client.25733 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.107+0000 7f54e0a10740 20 client.25733 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.107+0000 7f54e0a10740 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562e49510d80 0x562e49910c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.107+0000 7f54e0a10740 4 client.25733 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:37:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.107+0000 7f54e0a10740 1 -- 172.21.15.44:0/2003030498 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e49867ae0 con 0x562e49510d80 2024-05-27T09:37:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.108+0000 7f54d2a00640 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562e49510d80 0x562e49910c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.109+0000 7f54d2a00640 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562e49510d80 0x562e49910c70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=410ddc95e5e38d0a server_cookie=82cd50de5fbd4be2 in_seq=0 out_seq=0 2024-05-27T09:37:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.109+0000 7f54bcc00640 10 client.25733 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54bcc00640 1 -- 172.21.15.44:0/2003030498 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562e49867ae0 con 0x562e49510d80 2024-05-27T09:37:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54bcc00640 10 client.25733 handle_command_reply: tid=0 2024-05-27T09:37:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54e0a10740 1 client.25733 shutdown 2024-05-27T09:37:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54e0a10740 1 -- 172.21.15.44:0/2003030498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495314d0 msgr2=0x562e498feed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54e0a10740 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495314d0 0x562e498feed0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f54c00044e0 tx=0x7f54c00046c0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54e0a10740 1 -- 172.21.15.44:0/2003030498 shutdown_connections 2024-05-27T09:37:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54e0a10740 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562e49510d80 0x562e49910c70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54e0a10740 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495521b0 0x562e4990acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54e0a10740 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495314d0 0x562e498feed0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54e0a10740 1 --2- 172.21.15.44:0/2003030498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e498fe990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54e0a10740 1 -- 172.21.15.44:0/2003030498 >> 172.21.15.44:0/2003030498 conn(0x562e49546300 msgr2=0x562e498f4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.110+0000 7f54bcc00640 0 client.25733 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.111+0000 7f54e0a10740 1 -- 172.21.15.44:0/2003030498 shutdown_connections 2024-05-27T09:37:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.111+0000 7f54e0a10740 1 -- 172.21.15.44:0/2003030498 wait complete. 2024-05-27T09:37:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.111+0000 7f54e0a10740 20 client.25733 trim_cache size 0 max 16384 2024-05-27T09:37:35.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.143+0000 7f54e0a10740 1 Processor -- start 2024-05-27T09:37:35.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.143+0000 7f54e0a10740 1 -- start start 2024-05-27T09:37:35.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.144+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e499007c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.144+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495314d0 0x562e4990aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.144+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495521b0 0x562e4990af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.144+0000 7f54e0a10740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41c11f0 con 0x562e495314d0 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.144+0000 7f54e0a10740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41beac0 con 0x562e495521b0 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.144+0000 7f54e0a10740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41bc910 con 0x562e495292f0 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.144+0000 7f54d2000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495314d0 0x562e4990aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.144+0000 7f54d2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e499007c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.144+0000 7f54d2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e499007c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55208/0 (socket says 172.21.15.44:55208) 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.144+0000 7f54d2a00640 1 -- 172.21.15.44:0/492403724 learned_addr learned my addr 172.21.15.44:0/492403724 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.144+0000 7f54d3400640 1 --2- 172.21.15.44:0/492403724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495521b0 0x562e4990af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.145+0000 7f54d2a00640 1 -- 172.21.15.44:0/492403724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495521b0 msgr2=0x562e4990af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.145+0000 7f54d2a00640 1 --2- 172.21.15.44:0/492403724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495521b0 0x562e4990af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.145+0000 7f54d2a00640 1 -- 172.21.15.44:0/492403724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495314d0 msgr2=0x562e4990aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.145+0000 7f54d2a00640 1 --2- 172.21.15.44:0/492403724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495314d0 0x562e4990aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.145+0000 7f54d2a00640 1 -- 172.21.15.44:0/492403724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e498ff230 con 0x562e495292f0 2024-05-27T09:37:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.145+0000 7f54d2a00640 1 --2- 172.21.15.44:0/492403724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e499007c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54c4033000 tx=0x7f54c40050e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c78dd73b442cecd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.145+0000 7f54bf400640 1 -- 172.21.15.44:0/492403724 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4035040 con 0x562e495292f0 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.145+0000 7f54bf400640 1 -- 172.21.15.44:0/492403724 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54c4004390 con 0x562e495292f0 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.145+0000 7f54bf400640 1 -- 172.21.15.44:0/492403724 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4040040 con 0x562e495292f0 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.146+0000 7f54e0a10740 1 -- 172.21.15.44:0/492403724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 msgr2=0x562e499007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.146+0000 7f54e0a10740 1 --2- 172.21.15.44:0/492403724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e499007c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54c4033000 tx=0x7f54c40050e0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.146+0000 7f54e0a10740 1 -- 172.21.15.44:0/492403724 shutdown_connections 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.146+0000 7f54e0a10740 1 --2- 172.21.15.44:0/492403724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495521b0 0x562e4990af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.146+0000 7f54e0a10740 1 --2- 172.21.15.44:0/492403724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495314d0 0x562e4990aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.147+0000 7f54e0a10740 1 --2- 172.21.15.44:0/492403724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495292f0 0x562e499007c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.147+0000 7f54e0a10740 1 -- 172.21.15.44:0/492403724 >> 172.21.15.44:0/492403724 conn(0x562e49546300 msgr2=0x562e499000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.147+0000 7f54e0a10740 1 -- 172.21.15.44:0/492403724 shutdown_connections 2024-05-27T09:37:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.147+0000 7f54e0a10740 1 -- 172.21.15.44:0/492403724 wait complete. 2024-05-27T09:37:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.147+0000 7f54e0a10740 1 -- start start 2024-05-27T09:37:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.148+0000 7f54e0a10740 1 Processor -- start 2024-05-27T09:37:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.148+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 0x562e499038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.149+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495314d0 0x562e49903e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.149+0000 7f54e0a10740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495521b0 0x562e49904360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.149+0000 7f54e0a10740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41bcf80 con 0x562e495314d0 2024-05-27T09:37:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.149+0000 7f54e0a10740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41be3f0 con 0x562e495292f0 2024-05-27T09:37:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.149+0000 7f54e0a10740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41bd650 con 0x562e495521b0 2024-05-27T09:37:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.149+0000 7f54e0a10740 10 client.0 _ll_register_callbacks cb 0x7f54c0009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.149+0000 7f54d3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495521b0 0x562e49904360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.149+0000 7f54d3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495521b0 0x562e49904360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55216/0 (socket says 172.21.15.44:55216) 2024-05-27T09:37:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.149+0000 7f54d3400640 1 -- 172.21.15.44:0/3004913476 learned_addr learned my addr 172.21.15.44:0/3004913476 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.149+0000 7f54d2000640 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495314d0 0x562e49903e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.149+0000 7f54d2a00640 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 0x562e499038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.150+0000 7f54d3400640 1 -- 172.21.15.44:0/3004913476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 msgr2=0x562e499038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.150+0000 7f54d3400640 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 0x562e499038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.150+0000 7f54d3400640 1 -- 172.21.15.44:0/3004913476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495314d0 msgr2=0x562e49903e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.150+0000 7f54d3400640 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495314d0 0x562e49903e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.150+0000 7f54d3400640 1 -- 172.21.15.44:0/3004913476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e49977300 con 0x562e495521b0 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.150+0000 7f54d2000640 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495314d0 0x562e49903e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.150+0000 7f54d3400640 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495521b0 0x562e49904360 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0a9930 tx=0x7f54cc0aa230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9726246eb5adbbac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.150+0000 7f54bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.150+0000 7f54bcc00640 1 -- 172.21.15.44:0/3004913476 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc0a53c0 con 0x562e495521b0 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.150+0000 7f54bcc00640 1 -- 172.21.15.44:0/3004913476 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f54cc0a6220 con 0x562e495521b0 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.150+0000 7f54e0a10740 1 -- 172.21.15.44:0/3004913476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54c4004390 con 0x562e495521b0 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.151+0000 7f54bcc00640 1 -- 172.21.15.44:0/3004913476 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f54cc009820 con 0x562e495521b0 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.151+0000 7f54e0a10740 10 client.25756 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.151+0000 7f54e0a10740 1 -- 172.21.15.44:0/3004913476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562e499784a0 con 0x562e495521b0 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.151+0000 7f54bcc00640 1 -- 172.21.15.44:0/3004913476 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54c4004390 con 0x562e495521b0 2024-05-27T09:37:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.152+0000 7f54bcc00640 1 -- 172.21.15.44:0/3004913476 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f54cc085030 con 0x562e495521b0 2024-05-27T09:37:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.152+0000 7f54e0a10740 10 client.25756 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.152+0000 7f54e0a10740 10 client.25756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.152+0000 7f54e0a10740 20 client.25756 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.152+0000 7f54e0a10740 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562e494e61c0 0x562e4997ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.152+0000 7f54e0a10740 4 client.25756 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:37:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.152+0000 7f54e0a10740 1 -- 172.21.15.44:0/3004913476 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e498612c0 con 0x562e494e61c0 2024-05-27T09:37:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.152+0000 7f54d2a00640 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562e494e61c0 0x562e4997ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.153+0000 7f54d2a00640 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562e494e61c0 0x562e4997ac50 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e37a0a1c3476fb90 server_cookie=d6693ff3d629c4f4 in_seq=0 out_seq=0 2024-05-27T09:37:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.153+0000 7f54bcc00640 10 client.25756 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.156+0000 7f54bcc00640 1 -- 172.21.15.44:0/3004913476 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562e498612c0 con 0x562e494e61c0 2024-05-27T09:37:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.156+0000 7f54bcc00640 10 client.25756 handle_command_reply: tid=0 2024-05-27T09:37:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.156+0000 7f54e0a10740 1 client.25756 shutdown 2024-05-27T09:37:35.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54e0a10740 1 -- 172.21.15.44:0/3004913476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495521b0 msgr2=0x562e49904360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:35.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54e0a10740 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495521b0 0x562e49904360 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0a9930 tx=0x7f54cc0aa230 comp rx=0 tx=0).stop 2024-05-27T09:37:35.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54e0a10740 1 -- 172.21.15.44:0/3004913476 shutdown_connections 2024-05-27T09:37:35.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54e0a10740 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562e494e61c0 0x562e4997ac50 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54e0a10740 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e495521b0 0x562e49904360 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54e0a10740 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562e495314d0 0x562e49903e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54e0a10740 1 --2- 172.21.15.44:0/3004913476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e495292f0 0x562e499038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54e0a10740 1 -- 172.21.15.44:0/3004913476 >> 172.21.15.44:0/3004913476 conn(0x562e49546300 msgr2=0x562e49900990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54bcc00640 0 client.25756 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54e0a10740 1 -- 172.21.15.44:0/3004913476 shutdown_connections 2024-05-27T09:37:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54e0a10740 1 -- 172.21.15.44:0/3004913476 wait complete. 2024-05-27T09:37:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.157+0000 7f54e0a10740 20 client.25756 trim_cache size 0 max 16384 2024-05-27T09:37:35.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:37:35.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:37:35.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:37:35.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 -- 172.21.15.44:0/514715295 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54b80b29d0 msgr2=0x7f54b80b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54b80b29d0 0x7f54b80b4ee0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x562e498f19d0 tx=0x7f54c0005f90 comp rx=0 tx=0).stop 2024-05-27T09:37:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 -- 172.21.15.44:0/514715295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54d41104a0 msgr2=0x7f54d410cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54d41104a0 0x7f54d410cd70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00efd0 tx=0x7f54cc00c6a0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 -- 172.21.15.44:0/514715295 shutdown_connections 2024-05-27T09:37:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54d41104a0 0x7f54d410cd70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d410db40 0x7f54d410c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f54b80b29d0 0x7f54b80b4ee0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 --2- 172.21.15.44:0/514715295 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54d4075660 0x7f54d410c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 -- 172.21.15.44:0/514715295 >> 172.21.15.44:0/514715295 conn(0x7f54d40ff480 msgr2=0x7f54d41108f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 -- 172.21.15.44:0/514715295 shutdown_connections 2024-05-27T09:37:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:35.158+0000 7f54d9a00640 1 -- 172.21.15.44:0/514715295 wait complete. 2024-05-27T09:37:35.470 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:35 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:37:35.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:35 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:37:35.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:35 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:37:36.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.076+0000 7f0903200700 1 -- 172.21.15.44:0/916495660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0904105280 msgr2=0x7f0904105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:36.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.076+0000 7f0903200700 1 --2- 172.21.15.44:0/916495660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0904105280 0x7f0904105690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009990 tx=0x7f08f4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.077+0000 7f0903200700 1 -- 172.21.15.44:0/916495660 shutdown_connections 2024-05-27T09:37:36.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.077+0000 7f0903200700 1 --2- 172.21.15.44:0/916495660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f090410e6f0 0x7f0904110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.077+0000 7f0903200700 1 --2- 172.21.15.44:0/916495660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0904105280 0x7f0904105690 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.077+0000 7f0903200700 1 --2- 172.21.15.44:0/916495660 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0904105bd0 0x7f090410e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.077+0000 7f0903200700 1 -- 172.21.15.44:0/916495660 >> 172.21.15.44:0/916495660 conn(0x7f0904078ab0 msgr2=0x7f0904078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:36.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.077+0000 7f0903200700 1 -- 172.21.15.44:0/916495660 shutdown_connections 2024-05-27T09:37:36.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.077+0000 7f0903200700 1 -- 172.21.15.44:0/916495660 wait complete. 2024-05-27T09:37:36.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.078+0000 7f0903200700 1 Processor -- start 2024-05-27T09:37:36.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.078+0000 7f0903200700 1 -- start start 2024-05-27T09:37:36.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.078+0000 7f0903200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0904105280 0x7f0904199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.078+0000 7f0903200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0904105bd0 0x7f0904199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.078+0000 7f0903200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f090410e6f0 0x7f09041a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.078+0000 7f0903200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0904114240 con 0x7f0904105bd0 2024-05-27T09:37:36.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.078+0000 7f0903200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f09041140c0 con 0x7f090410e6f0 2024-05-27T09:37:36.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.078+0000 7f0903200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09041143c0 con 0x7f0904105280 2024-05-27T09:37:36.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.079+0000 7f0901400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0904105bd0 0x7f0904199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.079+0000 7f0901400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0904105bd0 0x7f0904199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46736/0 (socket says 172.21.15.44:46736) 2024-05-27T09:37:36.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.079+0000 7f0901400700 1 -- 172.21.15.44:0/3573846008 learned_addr learned my addr 172.21.15.44:0/3573846008 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:36.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.079+0000 7f0901e00700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0904105280 0x7f0904199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.079+0000 7f0901400700 1 -- 172.21.15.44:0/3573846008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0904105280 msgr2=0x7f0904199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:36.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.079+0000 7f0901400700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0904105280 0x7f0904199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.079+0000 7f0901400700 1 -- 172.21.15.44:0/3573846008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f090410e6f0 msgr2=0x7f09041a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:36.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.079+0000 7f0901400700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f090410e6f0 0x7f09041a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.079+0000 7f0901400700 1 -- 172.21.15.44:0/3573846008 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09041a0cb0 con 0x7f0904105bd0 2024-05-27T09:37:36.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.080+0000 7f0901400700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0904105bd0 0x7f0904199c50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f08f000e9a0 tx=0x7f08f000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:36.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.080+0000 7f08faa00700 1 -- 172.21.15.44:0/3573846008 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f000cbd0 con 0x7f0904105bd0 2024-05-27T09:37:36.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.080+0000 7f08faa00700 1 -- 172.21.15.44:0/3573846008 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08f000cd70 con 0x7f0904105bd0 2024-05-27T09:37:36.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.080+0000 7f08faa00700 1 -- 172.21.15.44:0/3573846008 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f0018680 con 0x7f0904105bd0 2024-05-27T09:37:36.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.080+0000 7f0903200700 1 -- 172.21.15.44:0/3573846008 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09041a0f40 con 0x7f0904105bd0 2024-05-27T09:37:36.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.081+0000 7f0903200700 1 -- 172.21.15.44:0/3573846008 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09041a1430 con 0x7f0904105bd0 2024-05-27T09:37:36.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.086+0000 7f08faa00700 1 -- 172.21.15.44:0/3573846008 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f08f0010430 con 0x7f0904105bd0 2024-05-27T09:37:36.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.086+0000 7f091753ab80 1 Processor -- start 2024-05-27T09:37:36.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.087+0000 7f091753ab80 1 -- start start 2024-05-27T09:37:36.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.087+0000 7f091753ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 0x563c8e9007d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.087+0000 7f091753ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e780ec0 0x563c8e900d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.087+0000 7f091753ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e90a340 0x563c8e90c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.087+0000 7f091753ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563c8e82dec0 con 0x563c8e90a340 2024-05-27T09:37:36.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.087+0000 7f091753ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563c8e81da60 con 0x563c8e780ec0 2024-05-27T09:37:36.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.087+0000 7f091753ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563c8e821aa0 con 0x563c8e86ca70 2024-05-27T09:37:36.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.088+0000 7f0902800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 0x563c8e9007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.088+0000 7f0902800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 0x563c8e9007d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55250/0 (socket says 172.21.15.44:55250) 2024-05-27T09:37:36.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.088+0000 7f0902800700 1 -- 172.21.15.44:0/2743694585 learned_addr learned my addr 172.21.15.44:0/2743694585 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:36.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.088+0000 7f0901e00700 1 --2- 172.21.15.44:0/2743694585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e780ec0 0x563c8e900d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.088+0000 7f08faa00700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f08e80c3b50 0x7f08e80c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.088+0000 7f0901400700 1 --2- 172.21.15.44:0/2743694585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e90a340 0x563c8e90c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.089+0000 7f0902800700 1 -- 172.21.15.44:0/2743694585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e780ec0 msgr2=0x563c8e900d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:36.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.089+0000 7f0902800700 1 --2- 172.21.15.44:0/2743694585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e780ec0 0x563c8e900d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.089+0000 7f0902800700 1 -- 172.21.15.44:0/2743694585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e90a340 msgr2=0x563c8e90c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:36.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.089+0000 7f0902800700 1 --2- 172.21.15.44:0/2743694585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e90a340 0x563c8e90c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.089+0000 7f0902800700 1 -- 172.21.15.44:0/2743694585 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c8e7e64a0 con 0x563c8e86ca70 2024-05-27T09:37:36.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.089+0000 7f0902800700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f08e80c3b50 0x7f08e80c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.089+0000 7f08faa00700 1 -- 172.21.15.44:0/3573846008 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f08f00e29f0 con 0x7f0904105bd0 2024-05-27T09:37:36.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.089+0000 7f0902800700 1 --2- 172.21.15.44:0/2743694585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 0x563c8e9007d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00e4d0 tx=0x7f08fc00e7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78a342a0ea4b6987 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:36.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.090+0000 7f08ef400700 1 -- 172.21.15.44:0/2743694585 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc0092c0 con 0x563c8e86ca70 2024-05-27T09:37:36.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.090+0000 7f08ef400700 1 -- 172.21.15.44:0/2743694585 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08fc01c020 con 0x563c8e86ca70 2024-05-27T09:37:36.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.090+0000 7f08ef400700 1 -- 172.21.15.44:0/2743694585 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc01b040 con 0x563c8e86ca70 2024-05-27T09:37:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.091+0000 7f091753ab80 1 -- 172.21.15.44:0/2743694585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 msgr2=0x563c8e9007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.091+0000 7f0902800700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f08e80c3b50 0x7f08e80c6060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f08fc01d000 tx=0x7f08fc0166c0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.091+0000 7f091753ab80 1 --2- 172.21.15.44:0/2743694585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 0x563c8e9007d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00e4d0 tx=0x7f08fc00e7e0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.091+0000 7f091753ab80 1 -- 172.21.15.44:0/2743694585 shutdown_connections 2024-05-27T09:37:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.091+0000 7f091753ab80 1 --2- 172.21.15.44:0/2743694585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e90a340 0x563c8e90c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.091+0000 7f091753ab80 1 --2- 172.21.15.44:0/2743694585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 0x563c8e9007d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.091+0000 7f091753ab80 1 --2- 172.21.15.44:0/2743694585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e780ec0 0x563c8e900d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.091+0000 7f091753ab80 1 -- 172.21.15.44:0/2743694585 >> 172.21.15.44:0/2743694585 conn(0x563c8e81a410 msgr2=0x563c8e8fb790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.092+0000 7f091753ab80 1 -- 172.21.15.44:0/2743694585 shutdown_connections 2024-05-27T09:37:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.092+0000 7f091753ab80 1 -- 172.21.15.44:0/2743694585 wait complete. 2024-05-27T09:37:36.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.092+0000 7f091753ab80 1 -- start start 2024-05-27T09:37:36.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.093+0000 7f091753ab80 1 Processor -- start 2024-05-27T09:37:36.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.093+0000 7f091753ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 0x563c8e9072d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.093+0000 7f091753ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e86ca70 0x563c8e907810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.093+0000 7f091753ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e907d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.093+0000 7f091753ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563c8e724020 con 0x563c8e86ca70 2024-05-27T09:37:36.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.093+0000 7f091753ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563c8e7be7e0 con 0x563c8e90a340 2024-05-27T09:37:36.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.093+0000 7f091753ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563c8e90ce80 con 0x563c8e780ec0 2024-05-27T09:37:36.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.093+0000 7f091753ab80 10 client.0 _ll_register_callbacks cb 0x563c8e7c7cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:36.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.094+0000 7f0901e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 0x563c8e9072d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.094+0000 7f0901400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e86ca70 0x563c8e907810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.094+0000 7f0902800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e907d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.094+0000 7f0901e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 0x563c8e9072d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55260/0 (socket says 172.21.15.44:55260) 2024-05-27T09:37:36.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.094+0000 7f0901e00700 1 -- 172.21.15.44:0/2202256795 learned_addr learned my addr 172.21.15.44:0/2202256795 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:36.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.094+0000 7f0902800700 1 -- 172.21.15.44:0/2202256795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 msgr2=0x563c8e9072d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:36.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.095+0000 7f0902800700 1 --2- 172.21.15.44:0/2202256795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 0x563c8e9072d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.095+0000 7f0902800700 1 -- 172.21.15.44:0/2202256795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e86ca70 msgr2=0x563c8e907810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:36.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.095+0000 7f0902800700 1 --2- 172.21.15.44:0/2202256795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e86ca70 0x563c8e907810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.095+0000 7f0902800700 1 -- 172.21.15.44:0/2202256795 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c8e76d950 con 0x563c8e90a340 2024-05-27T09:37:36.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.095+0000 7f0902800700 1 --2- 172.21.15.44:0/2202256795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e907d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00e4a0 tx=0x7f08fc015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d28a7722b700c192 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:36.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.095+0000 7f08e3e00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:37:36.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.095+0000 7f08e3e00700 1 -- 172.21.15.44:0/2202256795 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc02e070 con 0x563c8e90a340 2024-05-27T09:37:36.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.096+0000 7f08e3e00700 1 -- 172.21.15.44:0/2202256795 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f08fc016bf0 con 0x563c8e90a340 2024-05-27T09:37:36.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.096+0000 7f091753ab80 1 -- 172.21.15.44:0/2202256795 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08fc01c020 con 0x563c8e90a340 2024-05-27T09:37:36.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.096+0000 7f08e3e00700 1 -- 172.21.15.44:0/2202256795 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f08fc038050 con 0x563c8e90a340 2024-05-27T09:37:36.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.096+0000 7f091753ab80 10 client.25748 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:36.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.096+0000 7f091753ab80 1 -- 172.21.15.44:0/2202256795 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563c8e7ee0e0 con 0x563c8e90a340 2024-05-27T09:37:36.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.097+0000 7f08e3e00700 1 -- 172.21.15.44:0/2202256795 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08fc01c020 con 0x563c8e90a340 2024-05-27T09:37:36.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.097+0000 7f08e3e00700 1 -- 172.21.15.44:0/2202256795 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f08fc007c70 con 0x563c8e90a340 2024-05-27T09:37:36.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.097+0000 7f091753ab80 10 client.25748 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:36.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.097+0000 7f091753ab80 10 client.25748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:36.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.097+0000 7f091753ab80 20 client.25748 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:36.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.097+0000 7f091753ab80 1 --2- 172.21.15.44:0/2202256795 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563c8e919d10 0x563c8e91c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.097+0000 7f091753ab80 4 client.25748 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:37:36.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.097+0000 7f091753ab80 1 -- 172.21.15.44:0/2202256795 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c8e82fe10 con 0x563c8e919d10 2024-05-27T09:37:36.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.098+0000 7f0901e00700 1 --2- 172.21.15.44:0/2202256795 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563c8e919d10 0x563c8e91c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.098+0000 7f0901e00700 1 --2- 172.21.15.44:0/2202256795 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563c8e919d10 0x563c8e91c130 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ff1e29742dfcb2f server_cookie=7334220bfa7fb9e2 in_seq=0 out_seq=0 2024-05-27T09:37:36.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.098+0000 7f08e3e00700 10 client.25748 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:36.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.099+0000 7f08e3e00700 1 -- 172.21.15.44:0/2202256795 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563c8e82fe10 con 0x563c8e919d10 2024-05-27T09:37:36.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.099+0000 7f08e3e00700 10 client.25748 handle_command_reply: tid=0 2024-05-27T09:37:36.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.099+0000 7f091753ab80 1 client.25748 shutdown 2024-05-27T09:37:36.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f091753ab80 1 -- 172.21.15.44:0/2202256795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 msgr2=0x563c8e907d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:36.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f091753ab80 1 --2- 172.21.15.44:0/2202256795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e907d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00e4a0 tx=0x7f08fc015000 comp rx=0 tx=0).stop 2024-05-27T09:37:36.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f091753ab80 1 -- 172.21.15.44:0/2202256795 shutdown_connections 2024-05-27T09:37:36.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f091753ab80 1 --2- 172.21.15.44:0/2202256795 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563c8e919d10 0x563c8e91c130 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f091753ab80 1 --2- 172.21.15.44:0/2202256795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e907d50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f091753ab80 1 --2- 172.21.15.44:0/2202256795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 0x563c8e9072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f091753ab80 1 --2- 172.21.15.44:0/2202256795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e86ca70 0x563c8e907810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f091753ab80 1 -- 172.21.15.44:0/2202256795 >> 172.21.15.44:0/2202256795 conn(0x563c8e81a410 msgr2=0x563c8e9008a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:36.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f08e3e00700 0 client.25748 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:36.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f091753ab80 1 -- 172.21.15.44:0/2202256795 shutdown_connections 2024-05-27T09:37:36.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f091753ab80 1 -- 172.21.15.44:0/2202256795 wait complete. 2024-05-27T09:37:36.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.100+0000 7f091753ab80 20 client.25748 trim_cache size 0 max 16384 2024-05-27T09:37:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.125+0000 7f091753ab80 1 Processor -- start 2024-05-27T09:37:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.125+0000 7f091753ab80 1 -- start start 2024-05-27T09:37:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.125+0000 7f091753ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e780ec0 0x563c8e915f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.125+0000 7f091753ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 0x563c8e902800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.126+0000 7f091753ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e921830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.126+0000 7f091753ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09041b7b40 con 0x563c8e780ec0 2024-05-27T09:37:36.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.126+0000 7f091753ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0904073e50 con 0x563c8e90a340 2024-05-27T09:37:36.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.126+0000 7f091753ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0904071ca0 con 0x563c8e86ca70 2024-05-27T09:37:36.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.126+0000 7f0901e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e780ec0 0x563c8e915f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.126+0000 7f0901e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e780ec0 0x563c8e915f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46768/0 (socket says 172.21.15.44:46768) 2024-05-27T09:37:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.126+0000 7f0901e00700 1 -- 172.21.15.44:0/3931006828 learned_addr learned my addr 172.21.15.44:0/3931006828 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.126+0000 7f0902800700 1 --2- 172.21.15.44:0/3931006828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e921830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.127+0000 7f0901400700 1 --2- 172.21.15.44:0/3931006828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 0x563c8e902800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.127+0000 7f0901e00700 1 -- 172.21.15.44:0/3931006828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 msgr2=0x563c8e902800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.127+0000 7f0901e00700 1 --2- 172.21.15.44:0/3931006828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 0x563c8e902800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.127+0000 7f0901e00700 1 -- 172.21.15.44:0/3931006828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 msgr2=0x563c8e921830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.127+0000 7f0901e00700 1 --2- 172.21.15.44:0/3931006828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e921830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.127+0000 7f0901e00700 1 -- 172.21.15.44:0/3931006828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c8e9080c0 con 0x563c8e780ec0 2024-05-27T09:37:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.127+0000 7f0901400700 1 --2- 172.21.15.44:0/3931006828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 0x563c8e902800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.127+0000 7f0901e00700 1 --2- 172.21.15.44:0/3931006828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e780ec0 0x563c8e915f20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f08f4006010 tx=0x7f08f4005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6080c3edc97cd662 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.128+0000 7f08eea00700 1 -- 172.21.15.44:0/3931006828 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f400b720 con 0x563c8e780ec0 2024-05-27T09:37:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.128+0000 7f08eea00700 1 -- 172.21.15.44:0/3931006828 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08f400bd70 con 0x563c8e780ec0 2024-05-27T09:37:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.128+0000 7f08eea00700 1 -- 172.21.15.44:0/3931006828 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f400f460 con 0x563c8e780ec0 2024-05-27T09:37:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.129+0000 7f091753ab80 1 -- 172.21.15.44:0/3931006828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e780ec0 msgr2=0x563c8e915f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.129+0000 7f091753ab80 1 --2- 172.21.15.44:0/3931006828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e780ec0 0x563c8e915f20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f08f4006010 tx=0x7f08f4005a90 comp rx=0 tx=0).stop 2024-05-27T09:37:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.129+0000 7f091753ab80 1 -- 172.21.15.44:0/3931006828 shutdown_connections 2024-05-27T09:37:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.129+0000 7f091753ab80 1 --2- 172.21.15.44:0/3931006828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e921830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.129+0000 7f091753ab80 1 --2- 172.21.15.44:0/3931006828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e780ec0 0x563c8e915f20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.129+0000 7f091753ab80 1 --2- 172.21.15.44:0/3931006828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e86ca70 0x563c8e902800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.129+0000 7f091753ab80 1 -- 172.21.15.44:0/3931006828 >> 172.21.15.44:0/3931006828 conn(0x563c8e81a410 msgr2=0x563c8e9020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.129+0000 7f091753ab80 1 -- 172.21.15.44:0/3931006828 shutdown_connections 2024-05-27T09:37:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.129+0000 7f091753ab80 1 -- 172.21.15.44:0/3931006828 wait complete. 2024-05-27T09:37:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.130+0000 7f091753ab80 1 -- start start 2024-05-27T09:37:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.130+0000 7f091753ab80 1 Processor -- start 2024-05-27T09:37:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.131+0000 7f091753ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 0x563c8e90e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.131+0000 7f091753ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e86ca70 0x563c8e90e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.131+0000 7f091753ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e90eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.131+0000 7f091753ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0904072310 con 0x563c8e86ca70 2024-05-27T09:37:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.131+0000 7f091753ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0904073780 con 0x563c8e90a340 2024-05-27T09:37:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.131+0000 7f091753ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09040729e0 con 0x563c8e780ec0 2024-05-27T09:37:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.131+0000 7f091753ab80 10 client.0 _ll_register_callbacks cb 0x563c8e7c7cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.131+0000 7f0901e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 0x563c8e90e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.131+0000 7f0901e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 0x563c8e90e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55276/0 (socket says 172.21.15.44:55276) 2024-05-27T09:37:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.131+0000 7f0901e00700 1 -- 172.21.15.44:0/2451341093 learned_addr learned my addr 172.21.15.44:0/2451341093 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.131+0000 7f0901400700 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e86ca70 0x563c8e90e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.132+0000 7f0901e00700 1 -- 172.21.15.44:0/2451341093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 msgr2=0x563c8e90eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.132+0000 7f0901e00700 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e90eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.132+0000 7f0901e00700 1 -- 172.21.15.44:0/2451341093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e86ca70 msgr2=0x563c8e90e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:36.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.132+0000 7f0901e00700 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e86ca70 0x563c8e90e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.132+0000 7f0901e00700 1 -- 172.21.15.44:0/2451341093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c8e983910 con 0x563c8e780ec0 2024-05-27T09:37:36.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.132+0000 7f0901e00700 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 0x563c8e90e0e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f08f400b4b0 tx=0x7f08f4005b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3834c956424a3375 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:36.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.132+0000 7f08e3e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:37:36.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.132+0000 7f08e3e00700 1 -- 172.21.15.44:0/2451341093 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f4038020 con 0x563c8e780ec0 2024-05-27T09:37:36.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.132+0000 7f08e3e00700 1 -- 172.21.15.44:0/2451341093 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f08f4003d30 con 0x563c8e780ec0 2024-05-27T09:37:36.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.133+0000 7f091753ab80 1 -- 172.21.15.44:0/2451341093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08f400bd70 con 0x563c8e780ec0 2024-05-27T09:37:36.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.133+0000 7f08e3e00700 1 -- 172.21.15.44:0/2451341093 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f08f4003950 con 0x563c8e780ec0 2024-05-27T09:37:36.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.133+0000 7f091753ab80 10 client.25771 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:36.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.133+0000 7f091753ab80 1 -- 172.21.15.44:0/2451341093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563c8e984bb0 con 0x563c8e780ec0 2024-05-27T09:37:36.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.133+0000 7f08e3e00700 1 -- 172.21.15.44:0/2451341093 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08f400bd70 con 0x563c8e780ec0 2024-05-27T09:37:36.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.133+0000 7f08e3e00700 1 -- 172.21.15.44:0/2451341093 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f08f400f9f0 con 0x563c8e780ec0 2024-05-27T09:37:36.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.133+0000 7f091753ab80 10 client.25771 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:36.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.133+0000 7f091753ab80 10 client.25771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:36.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.134+0000 7f091753ab80 20 client.25771 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:36.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.134+0000 7f091753ab80 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563c8e985230 0x563c8e987650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:36.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.134+0000 7f091753ab80 4 client.25771 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:37:36.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.134+0000 7f091753ab80 1 -- 172.21.15.44:0/2451341093 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563c8e8295f0 con 0x563c8e985230 2024-05-27T09:37:36.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.135+0000 7f0901400700 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563c8e985230 0x563c8e987650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:36.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.135+0000 7f0901400700 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563c8e985230 0x563c8e987650 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=805f8d788105dd8e server_cookie=cd7c85e99e889543 in_seq=0 out_seq=0 2024-05-27T09:37:36.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.135+0000 7f08e3e00700 10 client.25771 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:36.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.136+0000 7f08e3e00700 1 -- 172.21.15.44:0/2451341093 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563c8e8295f0 con 0x563c8e985230 2024-05-27T09:37:36.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.136+0000 7f08e3e00700 10 client.25771 handle_command_reply: tid=0 2024-05-27T09:37:36.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.136+0000 7f091753ab80 1 client.25771 shutdown 2024-05-27T09:37:36.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.136+0000 7f091753ab80 1 -- 172.21.15.44:0/2451341093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 msgr2=0x563c8e90e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.136+0000 7f091753ab80 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 0x563c8e90e0e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f08f400b4b0 tx=0x7f08f4005b00 comp rx=0 tx=0).stop 2024-05-27T09:37:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.137+0000 7f091753ab80 1 -- 172.21.15.44:0/2451341093 shutdown_connections 2024-05-27T09:37:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.137+0000 7f091753ab80 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563c8e985230 0x563c8e987650 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.137+0000 7f091753ab80 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563c8e90a340 0x563c8e90eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.137+0000 7f08e3e00700 0 client.25771 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.137+0000 7f091753ab80 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8e780ec0 0x563c8e90e0e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.137+0000 7f091753ab80 1 --2- 172.21.15.44:0/2451341093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563c8e86ca70 0x563c8e90e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.137+0000 7f091753ab80 1 -- 172.21.15.44:0/2451341093 >> 172.21.15.44:0/2451341093 conn(0x563c8e81a410 msgr2=0x563c8e902880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.137+0000 7f091753ab80 1 -- 172.21.15.44:0/2451341093 shutdown_connections 2024-05-27T09:37:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.137+0000 7f091753ab80 1 -- 172.21.15.44:0/2451341093 wait complete. 2024-05-27T09:37:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.138+0000 7f091753ab80 20 client.25771 trim_cache size 0 max 16384 2024-05-27T09:37:36.140 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:37:36.140 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:37:36.140 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "0a6e902d-b9cd-4334-9dac-26cea2c912f7", 2024-05-27T09:37:36.140 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:37:36.140 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:37:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 -- 172.21.15.44:0/3573846008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f08e80c3b50 msgr2=0x7f08e80c6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f08e80c3b50 0x7f08e80c6060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f08fc01d000 tx=0x7f08fc0166c0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 -- 172.21.15.44:0/3573846008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0904105bd0 msgr2=0x7f0904199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0904105bd0 0x7f0904199c50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f08f000e9a0 tx=0x7f08f000ed60 comp rx=0 tx=0).stop 2024-05-27T09:37:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 -- 172.21.15.44:0/3573846008 shutdown_connections 2024-05-27T09:37:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f08e80c3b50 0x7f08e80c6060 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f090410e6f0 0x7f09041a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0904105280 0x7f0904199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 --2- 172.21.15.44:0/3573846008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0904105bd0 0x7f0904199c50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 -- 172.21.15.44:0/3573846008 >> 172.21.15.44:0/3573846008 conn(0x7f0904078ab0 msgr2=0x7f0904102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 -- 172.21.15.44:0/3573846008 shutdown_connections 2024-05-27T09:37:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:36.139+0000 7f0903200700 1 -- 172.21.15.44:0/3573846008 wait complete. 2024-05-27T09:37:36.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:36 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:37:36.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:36 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:37:36.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:36 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:37:36.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:36 smithi044 ceph-mon[31618]: pgmap v74: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-27T09:37:36.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:36 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:37:36.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:36 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:37:36.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:36 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:37:36.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:36 smithi050 ceph-mon[36371]: pgmap v74: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-27T09:37:36.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:36 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:37:36.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:36 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:37:36.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:36 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:37:36.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:36 smithi189 ceph-mon[36671]: pgmap v74: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-27T09:37:37.012 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:37:37.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:37 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:37:37.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:37 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:37:37.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:37 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:37:37.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:37 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:37:37.494 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:37:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:37 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:37:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:37 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:37:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:37 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:37:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:37 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:37:37.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:37 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:37:37.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:37 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:37:37.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:37 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:37:37.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:37 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:37:38.948 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:38 smithi044 ceph-mon[31618]: pgmap v75: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:37:39.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:38 smithi050 ceph-mon[36371]: pgmap v75: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:37:39.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:38 smithi189 ceph-mon[36671]: pgmap v75: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:37:39.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.561+0000 7f5ae3a00700 1 -- 172.21.15.44:0/1348201784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5adc109840 msgr2=0x7f5adc1111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:39.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.561+0000 7f5ae3a00700 1 --2- 172.21.15.44:0/1348201784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5adc109840 0x7f5adc1111a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400b0d0 tx=0x7f5ad400b3e0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.562+0000 7f5ae3a00700 1 -- 172.21.15.44:0/1348201784 shutdown_connections 2024-05-27T09:37:39.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.562+0000 7f5ae3a00700 1 --2- 172.21.15.44:0/1348201784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5adc109840 0x7f5adc1111a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.562+0000 7f5ae3a00700 1 --2- 172.21.15.44:0/1348201784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5adc1045c0 0x7f5adc1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.562+0000 7f5ae3a00700 1 --2- 172.21.15.44:0/1348201784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5adc108ef0 0x7f5adc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.562+0000 7f5ae3a00700 1 -- 172.21.15.44:0/1348201784 >> 172.21.15.44:0/1348201784 conn(0x7f5adc0ffe90 msgr2=0x7f5adc1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:39.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.562+0000 7f5ae3a00700 1 -- 172.21.15.44:0/1348201784 shutdown_connections 2024-05-27T09:37:39.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.562+0000 7f5ae3a00700 1 -- 172.21.15.44:0/1348201784 wait complete. 2024-05-27T09:37:39.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.563+0000 7f5ae3a00700 1 Processor -- start 2024-05-27T09:37:39.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.563+0000 7f5ae3a00700 1 -- start start 2024-05-27T09:37:39.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.564+0000 7f5ae3a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5adc1045c0 0x7f5adc076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.564+0000 7f5ae3a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5adc108ef0 0x7f5adc077100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.564+0000 7f5ae3a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5adc109840 0x7f5adc077640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.564+0000 7f5ae3a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1144c0 con 0x7f5adc109840 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.564+0000 7f5ae3a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc114340 con 0x7f5adc1045c0 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.564+0000 7f5ae3a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc114640 con 0x7f5adc108ef0 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.564+0000 7f5adbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5adc108ef0 0x7f5adc077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.564+0000 7f5ae1a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5adc109840 0x7f5adc077640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.564+0000 7f5adbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5adc108ef0 0x7f5adc077100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55296/0 (socket says 172.21.15.44:55296) 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.564+0000 7f5adbe00700 1 -- 172.21.15.44:0/161916993 learned_addr learned my addr 172.21.15.44:0/161916993 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.564+0000 7f5ae1000700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5adc1045c0 0x7f5adc076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.565+0000 7f5adbe00700 1 -- 172.21.15.44:0/161916993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5adc1045c0 msgr2=0x7f5adc076bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.565+0000 7f5adbe00700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5adc1045c0 0x7f5adc076bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.565+0000 7f5adbe00700 1 -- 172.21.15.44:0/161916993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5adc109840 msgr2=0x7f5adc077640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:39.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.565+0000 7f5adbe00700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5adc109840 0x7f5adc077640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.565+0000 7f5adbe00700 1 -- 172.21.15.44:0/161916993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc1a9130 con 0x7f5adc108ef0 2024-05-27T09:37:39.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.565+0000 7f5ae1a00700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5adc109840 0x7f5adc077640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:39.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.565+0000 7f5ae1000700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5adc1045c0 0x7f5adc076bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:39.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.565+0000 7f5adbe00700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5adc108ef0 0x7f5adc077100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e970 tx=0x7f5ac800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:39.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.565+0000 7f5ad9600700 1 -- 172.21.15.44:0/161916993 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8004a30 con 0x7f5adc108ef0 2024-05-27T09:37:39.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.565+0000 7f5ad9600700 1 -- 172.21.15.44:0/161916993 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac800cc90 con 0x7f5adc108ef0 2024-05-27T09:37:39.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.565+0000 7f5ae3a00700 1 -- 172.21.15.44:0/161916993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5adc1a9420 con 0x7f5adc108ef0 2024-05-27T09:37:39.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.566+0000 7f5ae3a00700 1 -- 172.21.15.44:0/161916993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5adc1a9960 con 0x7f5adc108ef0 2024-05-27T09:37:39.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.566+0000 7f5ad9600700 1 -- 172.21.15.44:0/161916993 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8005610 con 0x7f5adc108ef0 2024-05-27T09:37:39.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.570+0000 7f5af24f4b80 1 Processor -- start 2024-05-27T09:37:39.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.570+0000 7f5af24f4b80 1 -- start start 2024-05-27T09:37:39.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.570+0000 7f5ad9600700 1 -- 172.21.15.44:0/161916993 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5ac8010430 con 0x7f5adc108ef0 2024-05-27T09:37:39.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.571+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.571+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113c0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.571+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113c11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.571+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e2112eee40 con 0x55e2113c0db0 2024-05-27T09:37:39.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.571+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e2112de9e0 con 0x55e2113c6d30 2024-05-27T09:37:39.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.571+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e2112e2a20 con 0x55e211241e40 2024-05-27T09:37:39.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.571+0000 7f5ae1000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113c0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.571+0000 7f5ae1000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113c0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:44112/0 (socket says 172.21.15.44:44112) 2024-05-27T09:37:39.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.571+0000 7f5ae1000700 1 -- 172.21.15.44:0/662904837 learned_addr learned my addr 172.21.15.44:0/662904837 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:39.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.571+0000 7f5ae1a00700 1 --2- 172.21.15.44:0/662904837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.571+0000 7f5adbe00700 1 --2- 172.21.15.44:0/662904837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113c11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.572+0000 7f5ae1a00700 1 -- 172.21.15.44:0/662904837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 msgr2=0x55e2113c0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:39.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.572+0000 7f5ae1a00700 1 --2- 172.21.15.44:0/662904837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113c0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.572+0000 7f5ae1a00700 1 -- 172.21.15.44:0/662904837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 msgr2=0x55e2113c11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:39.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.572+0000 7f5ae1a00700 1 --2- 172.21.15.44:0/662904837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113c11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.572+0000 7f5ae1a00700 1 -- 172.21.15.44:0/662904837 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2112a7420 con 0x55e211241e40 2024-05-27T09:37:39.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.572+0000 7f5ae1a00700 1 --2- 172.21.15.44:0/662904837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113c67f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400b770 tx=0x7f5ad4009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f5eab696316e79a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:39.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.572+0000 7f5ace000700 1 -- 172.21.15.44:0/662904837 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4004a30 con 0x55e211241e40 2024-05-27T09:37:39.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.573+0000 7f5ace000700 1 -- 172.21.15.44:0/662904837 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ad4004bd0 con 0x55e211241e40 2024-05-27T09:37:39.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.573+0000 7f5ace000700 1 -- 172.21.15.44:0/662904837 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad40052a0 con 0x55e211241e40 2024-05-27T09:37:39.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.573+0000 7f5ad9600700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ac40b29f0 0x7f5ac40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.573+0000 7f5ad9600700 1 -- 172.21.15.44:0/161916993 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f5ac80e2900 con 0x7f5adc108ef0 2024-05-27T09:37:39.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.573+0000 7f5ae1000700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ac40b29f0 0x7f5ac40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.574+0000 7f5af24f4b80 1 -- 172.21.15.44:0/662904837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 msgr2=0x55e2113c67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:39.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.574+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/662904837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113c67f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400b770 tx=0x7f5ad4009b80 comp rx=0 tx=0).stop 2024-05-27T09:37:39.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.574+0000 7f5af24f4b80 1 -- 172.21.15.44:0/662904837 shutdown_connections 2024-05-27T09:37:39.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.574+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/662904837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113c11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.574+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/662904837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113c67f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.574+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/662904837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113c0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.574+0000 7f5af24f4b80 1 -- 172.21.15.44:0/662904837 >> 172.21.15.44:0/662904837 conn(0x55e2112a1fc0 msgr2=0x55e2113bb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:39.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.574+0000 7f5ae1000700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ac40b29f0 0x7f5ac40b4f00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009fd0 tx=0x7f5ad0009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:39.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.574+0000 7f5af24f4b80 1 -- 172.21.15.44:0/662904837 shutdown_connections 2024-05-27T09:37:39.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.574+0000 7f5af24f4b80 1 -- 172.21.15.44:0/662904837 wait complete. 2024-05-27T09:37:39.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.575+0000 7f5af24f4b80 1 -- start start 2024-05-27T09:37:39.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.575+0000 7f5af24f4b80 1 Processor -- start 2024-05-27T09:37:39.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.576+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.576+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.576+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113caf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.576+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e2111e4fa0 con 0x55e2113c0db0 2024-05-27T09:37:39.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.576+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e21127f760 con 0x55e2113c6d30 2024-05-27T09:37:39.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.576+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e2113cd7a0 con 0x55e211241e40 2024-05-27T09:37:39.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.576+0000 7f5af24f4b80 10 client.0 _ll_register_callbacks cb 0x55e211288c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:39.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5adbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5ae1000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5adbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46816/0 (socket says 172.21.15.44:46816) 2024-05-27T09:37:39.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5adbe00700 1 -- 172.21.15.44:0/837470869 learned_addr learned my addr 172.21.15.44:0/837470869 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:39.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5ae1a00700 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5ae1000700 1 -- 172.21.15.44:0/837470869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 msgr2=0x55e2113caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:39.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5ae1000700 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113caf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5ae1000700 1 -- 172.21.15.44:0/837470869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 msgr2=0x55e2113ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:39.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5ae1000700 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5ae1000700 1 -- 172.21.15.44:0/837470869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e21122e8d0 con 0x55e211241e40 2024-05-27T09:37:39.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5adbe00700 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:39.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5ae1000700 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113ca4b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0004250 tx=0x7f5ad0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19cefb50606f32e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:39.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.577+0000 7f5ac2a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:37:39.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.578+0000 7f5ac2a00700 1 -- 172.21.15.44:0/837470869 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0003ca0 con 0x55e211241e40 2024-05-27T09:37:39.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.578+0000 7f5af24f4b80 1 -- 172.21.15.44:0/837470869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ad4004bd0 con 0x55e211241e40 2024-05-27T09:37:39.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.578+0000 7f5ac2a00700 1 -- 172.21.15.44:0/837470869 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f5ad0029470 con 0x55e211241e40 2024-05-27T09:37:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.578+0000 7f5ac2a00700 1 -- 172.21.15.44:0/837470869 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5ad0003e40 con 0x55e211241e40 2024-05-27T09:37:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.578+0000 7f5af24f4b80 10 client.25783 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.578+0000 7f5af24f4b80 1 -- 172.21.15.44:0/837470869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e2112af060 con 0x55e211241e40 2024-05-27T09:37:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.579+0000 7f5ac2a00700 1 -- 172.21.15.44:0/837470869 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ad4004bd0 con 0x55e211241e40 2024-05-27T09:37:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.579+0000 7f5ac2a00700 1 -- 172.21.15.44:0/837470869 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5ad000f840 con 0x55e211241e40 2024-05-27T09:37:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.579+0000 7f5af24f4b80 10 client.25783 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.579+0000 7f5af24f4b80 10 client.25783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.579+0000 7f5af24f4b80 20 client.25783 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.579+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e2113da6f0 0x55e2113dcb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.579+0000 7f5af24f4b80 4 client.25783 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:37:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.579+0000 7f5af24f4b80 1 -- 172.21.15.44:0/837470869 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2112f0d90 con 0x55e2113da6f0 2024-05-27T09:37:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.579+0000 7f5adbe00700 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e2113da6f0 0x55e2113dcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.580+0000 7f5adbe00700 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e2113da6f0 0x55e2113dcb50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2174cf77f1eec4a8 server_cookie=a14b871cc5fb5660 in_seq=0 out_seq=0 2024-05-27T09:37:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.580+0000 7f5ac2a00700 10 client.25783 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.580+0000 7f5ac2a00700 1 -- 172.21.15.44:0/837470869 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e2112f0d90 con 0x55e2113da6f0 2024-05-27T09:37:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.580+0000 7f5ac2a00700 10 client.25783 handle_command_reply: tid=0 2024-05-27T09:37:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 1 client.25783 shutdown 2024-05-27T09:37:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 1 -- 172.21.15.44:0/837470869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 msgr2=0x55e2113ca4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113ca4b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0004250 tx=0x7f5ad0004750 comp rx=0 tx=0).stop 2024-05-27T09:37:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 1 -- 172.21.15.44:0/837470869 shutdown_connections 2024-05-27T09:37:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e2113da6f0 0x55e2113dcb50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113ca4b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5ac2a00700 0 client.25783 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:39.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/837470869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 1 -- 172.21.15.44:0/837470869 >> 172.21.15.44:0/837470869 conn(0x55e2112a1fc0 msgr2=0x55e2113bb580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:39.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 1 -- 172.21.15.44:0/837470869 shutdown_connections 2024-05-27T09:37:39.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 1 -- 172.21.15.44:0/837470869 wait complete. 2024-05-27T09:37:39.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.581+0000 7f5af24f4b80 20 client.25783 trim_cache size 0 max 16384 2024-05-27T09:37:39.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.604+0000 7f5af24f4b80 1 Processor -- start 2024-05-27T09:37:39.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.604+0000 7f5af24f4b80 1 -- start start 2024-05-27T09:37:39.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.604+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113c0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.604+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113d6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.604+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.604+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1b3730 con 0x55e2113c0db0 2024-05-27T09:37:39.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.604+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1b0fc0 con 0x55e2113c6d30 2024-05-27T09:37:39.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.604+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1aee10 con 0x55e211241e40 2024-05-27T09:37:39.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.605+0000 7f5adbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113d6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.605+0000 7f5ae1a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.605+0000 7f5ae1a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113c48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:44128/0 (socket says 172.21.15.44:44128) 2024-05-27T09:37:39.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.605+0000 7f5ae1a00700 1 -- 172.21.15.44:0/1144358611 learned_addr learned my addr 172.21.15.44:0/1144358611 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:39.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.605+0000 7f5ae1000700 1 --2- 172.21.15.44:0/1144358611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113c0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.605+0000 7f5ae1a00700 1 -- 172.21.15.44:0/1144358611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 msgr2=0x55e2113c0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:39.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.605+0000 7f5ae1a00700 1 --2- 172.21.15.44:0/1144358611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113c0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.605+0000 7f5ae1a00700 1 -- 172.21.15.44:0/1144358611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 msgr2=0x55e2113d6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:39.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.605+0000 7f5ae1a00700 1 --2- 172.21.15.44:0/1144358611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113d6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.605+0000 7f5ae1a00700 1 -- 172.21.15.44:0/1144358611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2113d8110 con 0x55e2113c6d30 2024-05-27T09:37:39.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.605+0000 7f5ae1000700 1 --2- 172.21.15.44:0/1144358611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113c0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:39.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.606+0000 7f5ae1a00700 1 --2- 172.21.15.44:0/1144358611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113c48e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400b770 tx=0x7f5ad4009e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df1ade573d8e8122 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:39.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.606+0000 7f5acd600700 1 -- 172.21.15.44:0/1144358611 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4007d10 con 0x55e2113c6d30 2024-05-27T09:37:39.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.606+0000 7f5acd600700 1 -- 172.21.15.44:0/1144358611 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ad4033070 con 0x55e2113c6d30 2024-05-27T09:37:39.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.606+0000 7f5acd600700 1 -- 172.21.15.44:0/1144358611 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4005070 con 0x55e2113c6d30 2024-05-27T09:37:39.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.607+0000 7f5af24f4b80 1 -- 172.21.15.44:0/1144358611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 msgr2=0x55e2113c48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:39.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.607+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/1144358611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113c48e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400b770 tx=0x7f5ad4009e20 comp rx=0 tx=0).stop 2024-05-27T09:37:39.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.607+0000 7f5af24f4b80 1 -- 172.21.15.44:0/1144358611 shutdown_connections 2024-05-27T09:37:39.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.607+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/1144358611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113c48e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.607+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/1144358611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113c0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.607+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/1144358611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113d6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.607+0000 7f5af24f4b80 1 -- 172.21.15.44:0/1144358611 >> 172.21.15.44:0/1144358611 conn(0x55e2112a1fc0 msgr2=0x55e2113cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:39.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.607+0000 7f5af24f4b80 1 -- 172.21.15.44:0/1144358611 shutdown_connections 2024-05-27T09:37:39.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.607+0000 7f5af24f4b80 1 -- 172.21.15.44:0/1144358611 wait complete. 2024-05-27T09:37:39.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.608+0000 7f5af24f4b80 1 -- start start 2024-05-27T09:37:39.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.608+0000 7f5af24f4b80 1 Processor -- start 2024-05-27T09:37:39.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113e1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113e2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5af24f4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113e2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1af480 con 0x55e2113c0db0 2024-05-27T09:37:39.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1b08f0 con 0x55e2113c6d30 2024-05-27T09:37:39.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5af24f4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1afb50 con 0x55e211241e40 2024-05-27T09:37:39.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5af24f4b80 10 client.0 _ll_register_callbacks cb 0x55e211288c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:39.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5ae1000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113e1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5ae1000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113e1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:55322/0 (socket says 172.21.15.44:55322) 2024-05-27T09:37:39.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5ae1000700 1 -- 172.21.15.44:0/2861641964 learned_addr learned my addr 172.21.15.44:0/2861641964 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:39.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5adbe00700 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113e2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.609+0000 7f5ae1a00700 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113e2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.610+0000 7f5ae1000700 1 -- 172.21.15.44:0/2861641964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 msgr2=0x55e2113e2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:39.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.610+0000 7f5ae1000700 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113e2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.610+0000 7f5ae1000700 1 -- 172.21.15.44:0/2861641964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 msgr2=0x55e2113e2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:39.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.610+0000 7f5ae1000700 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113e2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.610+0000 7f5ae1000700 1 -- 172.21.15.44:0/2861641964 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2114442b0 con 0x55e211241e40 2024-05-27T09:37:39.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.610+0000 7f5ae1000700 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113e1ff0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0000c00 tx=0x7f5ad000faf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac4105f290d98d7a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:39.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.610+0000 7f5ac2a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:37:39.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.610+0000 7f5ac2a00700 1 -- 172.21.15.44:0/2861641964 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0036040 con 0x55e211241e40 2024-05-27T09:37:39.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.610+0000 7f5af24f4b80 1 -- 172.21.15.44:0/2861641964 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ad4033070 con 0x55e211241e40 2024-05-27T09:37:39.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.610+0000 7f5ac2a00700 1 -- 172.21.15.44:0/2861641964 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f5ad0005240 con 0x55e211241e40 2024-05-27T09:37:39.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.611+0000 7f5ac2a00700 1 -- 172.21.15.44:0/2861641964 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5ad0049050 con 0x55e211241e40 2024-05-27T09:37:39.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.611+0000 7f5af24f4b80 10 client.25789 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:39.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.611+0000 7f5af24f4b80 1 -- 172.21.15.44:0/2861641964 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e211445520 con 0x55e211241e40 2024-05-27T09:37:39.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.611+0000 7f5ac2a00700 1 -- 172.21.15.44:0/2861641964 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ad4033070 con 0x55e211241e40 2024-05-27T09:37:39.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.611+0000 7f5ac2a00700 1 -- 172.21.15.44:0/2861641964 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5ad003b470 con 0x55e211241e40 2024-05-27T09:37:39.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.611+0000 7f5af24f4b80 10 client.25789 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:39.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.611+0000 7f5af24f4b80 10 client.25789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:39.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.611+0000 7f5af24f4b80 20 client.25789 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:39.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.611+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e211445c20 0x55e211448040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:39.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.611+0000 7f5af24f4b80 4 client.25789 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:37:39.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.611+0000 7f5af24f4b80 1 -- 172.21.15.44:0/2861641964 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e2112ea570 con 0x55e211445c20 2024-05-27T09:37:39.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.612+0000 7f5adbe00700 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e211445c20 0x55e211448040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:39.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.612+0000 7f5adbe00700 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e211445c20 0x55e211448040 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea3a5ae711663e2b server_cookie=39eaf9ae7bb136a6 in_seq=0 out_seq=0 2024-05-27T09:37:39.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.612+0000 7f5ac2a00700 10 client.25789 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:39.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.612+0000 7f5ac2a00700 1 -- 172.21.15.44:0/2861641964 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e2112ea570 con 0x55e211445c20 2024-05-27T09:37:39.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.612+0000 7f5ac2a00700 10 client.25789 handle_command_reply: tid=0 2024-05-27T09:37:39.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.612+0000 7f5af24f4b80 1 client.25789 shutdown 2024-05-27T09:37:39.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.612+0000 7f5af24f4b80 1 -- 172.21.15.44:0/2861641964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 msgr2=0x55e2113e1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:39.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.612+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113e1ff0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0000c00 tx=0x7f5ad000faf0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.613+0000 7f5af24f4b80 1 -- 172.21.15.44:0/2861641964 shutdown_connections 2024-05-27T09:37:39.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.613+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e211445c20 0x55e211448040 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.613+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e2113c6d30 0x55e2113e2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.613+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e211241e40 0x55e2113e1ff0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.613+0000 7f5af24f4b80 1 --2- 172.21.15.44:0/2861641964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e2113c0db0 0x55e2113e2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.613+0000 7f5ac2a00700 0 client.25789 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:39.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.613+0000 7f5af24f4b80 1 -- 172.21.15.44:0/2861641964 >> 172.21.15.44:0/2861641964 conn(0x55e2112a1fc0 msgr2=0x55e2113cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:39.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.613+0000 7f5af24f4b80 1 -- 172.21.15.44:0/2861641964 shutdown_connections 2024-05-27T09:37:39.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.613+0000 7f5af24f4b80 1 -- 172.21.15.44:0/2861641964 wait complete. 2024-05-27T09:37:39.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.613+0000 7f5af24f4b80 20 client.25789 trim_cache size 0 max 16384 2024-05-27T09:37:39.622 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:37:39.622 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:37:39.622 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:37:39.622 INFO:teuthology.orchestra.run.smithi044.stdout: "0a6e902d-b9cd-4334-9dac-26cea2c912f7": { 2024-05-27T09:37:39.622 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:37:39.622 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "0a6e902d-b9cd-4334-9dac-26cea2c912f7", 2024-05-27T09:37:39.622 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:37:39.623 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:37:39.623 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:37:39.623 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:37:39.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 -- 172.21.15.44:0/161916993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ac40b29f0 msgr2=0x7f5ac40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ac40b29f0 0x7f5ac40b4f00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009fd0 tx=0x7f5ad0009260 comp rx=0 tx=0).stop 2024-05-27T09:37:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 -- 172.21.15.44:0/161916993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5adc108ef0 msgr2=0x7f5adc077100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5adc108ef0 0x7f5adc077100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e970 tx=0x7f5ac800ec80 comp rx=0 tx=0).stop 2024-05-27T09:37:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 -- 172.21.15.44:0/161916993 shutdown_connections 2024-05-27T09:37:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ac40b29f0 0x7f5ac40b4f00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5adc109840 0x7f5adc077640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5adc1045c0 0x7f5adc076bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 --2- 172.21.15.44:0/161916993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5adc108ef0 0x7f5adc077100 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 -- 172.21.15.44:0/161916993 >> 172.21.15.44:0/161916993 conn(0x7f5adc0ffe90 msgr2=0x7f5adc108660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 -- 172.21.15.44:0/161916993 shutdown_connections 2024-05-27T09:37:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:37:39.614+0000 7f5ae3a00700 1 -- 172.21.15.44:0/161916993 wait complete. 2024-05-27T09:37:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:40 smithi044 ceph-mon[31618]: pgmap v76: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T09:37:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:40 smithi050 ceph-mon[36371]: pgmap v76: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T09:37:40.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:40 smithi189 ceph-mon[36671]: pgmap v76: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T09:37:40.931 INFO:tasks.cephfs.filesystem:scrub status for tag:0a6e902d-b9cd-4334-9dac-26cea2c912f7 - {'path': '/', 'tag': '0a6e902d-b9cd-4334-9dac-26cea2c912f7', 'options': 'recursive,force'} 2024-05-27T09:37:41.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:41 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:37:41.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:41 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:37:41.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:41 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:37:42.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:42 smithi044 ceph-mon[31618]: pgmap v77: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-27T09:37:42.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:42 smithi050 ceph-mon[36371]: pgmap v77: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-27T09:37:42.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:42 smithi189 ceph-mon[36671]: pgmap v77: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-27T09:37:43.216 INFO:tasks.workunit.client.0.smithi044.stderr:Updating files: 59% (7242/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 95% (11762/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-05-27T09:37:43.951 INFO:tasks.workunit.client.0.smithi044.stderr:Note: switching to 'a6507388dd5057528934822c0163b0c347ef1d5d'. 2024-05-27T09:37:43.951 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-05-27T09:37:43.951 INFO:tasks.workunit.client.0.smithi044.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-27T09:37:43.951 INFO:tasks.workunit.client.0.smithi044.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-27T09:37:43.951 INFO:tasks.workunit.client.0.smithi044.stderr:state without impacting any branches by switching back to a branch. 2024-05-27T09:37:43.951 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-05-27T09:37:43.951 INFO:tasks.workunit.client.0.smithi044.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-27T09:37:43.951 INFO:tasks.workunit.client.0.smithi044.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-27T09:37:43.952 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-05-27T09:37:43.952 INFO:tasks.workunit.client.0.smithi044.stderr: git switch -c 2024-05-27T09:37:43.952 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-05-27T09:37:43.952 INFO:tasks.workunit.client.0.smithi044.stderr:Or undo this operation with: 2024-05-27T09:37:43.952 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-05-27T09:37:43.952 INFO:tasks.workunit.client.0.smithi044.stderr: git switch - 2024-05-27T09:37:43.952 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-05-27T09:37:43.952 INFO:tasks.workunit.client.0.smithi044.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-27T09:37:43.952 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-05-27T09:37:43.952 INFO:tasks.workunit.client.0.smithi044.stderr:HEAD is now at a6507388dd5 Merge PR #53755 into wip-vshankar-testing-20240521.105740-debug 2024-05-27T09:37:43.966 DEBUG:teuthology.orchestra.run.smithi044:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-27T09:37:44.023 INFO:tasks.workunit.client.0.smithi044.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-27T09:37:44.030 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T09:37:44.030 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-27T09:37:44.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:44 smithi044 ceph-mon[31618]: pgmap v78: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-27T09:37:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:44 smithi050 ceph-mon[36371]: pgmap v78: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-27T09:37:44.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:44 smithi189 ceph-mon[36671]: pgmap v78: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-27T09:37:44.943 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-27T09:37:45.052 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-27T09:37:45.144 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T09:37:45.149 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T09:37:45.150 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-27T09:37:45.235 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T09:37:45.248 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:37:45.248 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-27T09:37:45.301 INFO:tasks.workunit:Running workunits matching fs/misc on client.0... 2024-05-27T09:37:45.302 INFO:tasks.workunit:Running workunit fs/misc/acl.sh... 2024-05-27T09:37:45.303 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/acl.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6507388dd5057528934822c0163b0c347ef1d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh 2024-05-27T09:37:45.375 INFO:tasks.workunit.client.0.smithi044.stderr:+ set -e 2024-05-27T09:37:45.375 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir -p testdir 2024-05-27T09:37:45.382 INFO:tasks.workunit.client.0.smithi044.stderr:+ cd testdir 2024-05-27T09:37:45.382 INFO:tasks.workunit.client.0.smithi044.stderr:+ set +e 2024-05-27T09:37:45.382 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfacl -d -m u:nobody:rw . 2024-05-27T09:37:45.394 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2024-05-27T09:37:45.394 INFO:tasks.workunit.client.0.smithi044.stderr:+ set -e 2024-05-27T09:37:45.394 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=0 2024-05-27T09:37:45.395 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 0 -lt 100 ']' 2024-05-27T09:37:45.395 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 0 + 1 2024-05-27T09:37:45.398 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=1 2024-05-27T09:37:45.398 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:45.404 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:45.405 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:45.412 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:45.412 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:45.413 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:46.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:46 smithi044 ceph-mon[31618]: pgmap v79: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T09:37:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:46 smithi050 ceph-mon[36371]: pgmap v79: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T09:37:46.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:46 smithi189 ceph-mon[36671]: pgmap v79: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T09:37:48.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:48 smithi044 ceph-mon[31618]: pgmap v80: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-27T09:37:48.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:48 smithi050 ceph-mon[36371]: pgmap v80: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-27T09:37:48.558 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:48.559 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:48.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:48 smithi189 ceph-mon[36671]: pgmap v80: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-27T09:37:48.699 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:48.699 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:48.712 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:48.712 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:48.713 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 1 -lt 100 ']' 2024-05-27T09:37:48.714 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 1 + 1 2024-05-27T09:37:48.758 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=2 2024-05-27T09:37:48.759 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:48.805 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:48.806 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:48.811 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:48.812 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:48.812 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:49.725 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:49.726 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:49.932 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:49.932 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:49.945 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:49.945 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:49.945 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -lt 100 ']' 2024-05-27T09:37:49.947 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 2 + 1 2024-05-27T09:37:49.978 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=3 2024-05-27T09:37:49.978 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:50.002 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:50.003 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:50.007 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:50.009 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:50.010 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:50.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:50 smithi044 ceph-mon[31618]: pgmap v81: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-05-27T09:37:50.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:50 smithi050 ceph-mon[36371]: pgmap v81: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-05-27T09:37:50.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:50 smithi189 ceph-mon[36671]: pgmap v81: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-05-27T09:37:50.842 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:50.842 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:51.165 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:51.165 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:51.186 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:51.186 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:51.186 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 3 -lt 100 ']' 2024-05-27T09:37:51.188 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 3 + 1 2024-05-27T09:37:51.241 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=4 2024-05-27T09:37:51.242 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.250+0000 7f802a200640 1 -- 172.21.15.44:0/172568223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f801c0a0f50 msgr2=0x7f801c0a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.250+0000 7f802a200640 1 --2- 172.21.15.44:0/172568223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f801c0a0f50 0x7f801c0a3440 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8024069170 tx=0x7f8024097ab0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.250+0000 7f802a200640 1 -- 172.21.15.44:0/172568223 shutdown_connections 2024-05-27T09:37:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.250+0000 7f802a200640 1 --2- 172.21.15.44:0/172568223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f801c0a0f50 0x7f801c0a3440 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.250+0000 7f802a200640 1 --2- 172.21.15.44:0/172568223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c09e5f0 0x7f801c0a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.250+0000 7f802a200640 1 --2- 172.21.15.44:0/172568223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f801c097b90 0x7f801c097fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.250+0000 7f802a200640 1 -- 172.21.15.44:0/172568223 >> 172.21.15.44:0/172568223 conn(0x7f801c00b000 msgr2=0x7f801c00b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.251+0000 7f802a200640 1 -- 172.21.15.44:0/172568223 shutdown_connections 2024-05-27T09:37:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.251+0000 7f802a200640 1 -- 172.21.15.44:0/172568223 wait complete. 2024-05-27T09:37:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.276+0000 7f802a200640 1 Processor -- start 2024-05-27T09:37:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.276+0000 7f802a200640 1 -- start start 2024-05-27T09:37:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.278+0000 7f802a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c097b90 0x7f801c12d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.278+0000 7f802a200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f801c09e5f0 0x7f801c12da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.278+0000 7f802a200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f801c0a0f50 0x7f801c131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.278+0000 7f802a200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f801c0a6b30 con 0x7f801c09e5f0 2024-05-27T09:37:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.278+0000 7f802a200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f801c0a69b0 con 0x7f801c0a0f50 2024-05-27T09:37:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.278+0000 7f802a200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f801c0a6cb0 con 0x7f801c097b90 2024-05-27T09:37:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.278+0000 7f8028e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c097b90 0x7f801c12d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.278+0000 7f8028e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c097b90 0x7f801c12d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:33444/0 (socket says 172.21.15.44:33444) 2024-05-27T09:37:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.278+0000 7f8028e00640 1 -- 172.21.15.44:0/2543664578 learned_addr learned my addr 172.21.15.44:0/2543664578 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.278+0000 7f8029800640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f801c0a0f50 0x7f801c131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.278+0000 7f8023e00640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f801c09e5f0 0x7f801c12da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.279+0000 7f8028e00640 1 -- 172.21.15.44:0/2543664578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f801c0a0f50 msgr2=0x7f801c131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.279+0000 7f8028e00640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f801c0a0f50 0x7f801c131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.279+0000 7f8028e00640 1 -- 172.21.15.44:0/2543664578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f801c09e5f0 msgr2=0x7f801c12da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.279+0000 7f8028e00640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f801c09e5f0 0x7f801c12da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.279+0000 7f8028e00640 1 -- 172.21.15.44:0/2543664578 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f801c132370 con 0x7f801c097b90 2024-05-27T09:37:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.279+0000 7f8029800640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f801c0a0f50 0x7f801c131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.279+0000 7f8028e00640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c097b90 0x7f801c12d510 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f801800e9c0 tx=0x7f801800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.280+0000 7f8021600640 1 -- 172.21.15.44:0/2543664578 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801800cde0 con 0x7f801c097b90 2024-05-27T09:37:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.280+0000 7f8021600640 1 -- 172.21.15.44:0/2543664578 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80180035f0 con 0x7f801c097b90 2024-05-27T09:37:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.280+0000 7f802a200640 1 -- 172.21.15.44:0/2543664578 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f801c132660 con 0x7f801c097b90 2024-05-27T09:37:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.280+0000 7f8021600640 1 -- 172.21.15.44:0/2543664578 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018010690 con 0x7f801c097b90 2024-05-27T09:37:51.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.281+0000 7f802a200640 1 -- 172.21.15.44:0/2543664578 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f801c139fe0 con 0x7f801c097b90 2024-05-27T09:37:51.287 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:51.287 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.287+0000 7f8021600640 1 -- 172.21.15.44:0/2543664578 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f8018004000 con 0x7f801c097b90 2024-05-27T09:37:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.289+0000 7f8021600640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f800c0b2ab0 0x7f800c0b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.289+0000 7f8021600640 1 -- 172.21.15.44:0/2543664578 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f80180e2960 con 0x7f801c097b90 2024-05-27T09:37:51.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.289+0000 7f8023e00640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f800c0b2ab0 0x7f800c0b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.290 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:51.291 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.291+0000 7f8023e00640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f800c0b2ab0 0x7f800c0b4fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f801c0a17b0 tx=0x7f8010005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:51.292 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.357+0000 7f8032817740 1 Processor -- start 2024-05-27T09:37:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.357+0000 7f8032817740 1 -- start start 2024-05-27T09:37:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.358+0000 7f8032817740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 0x559773e30900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.358+0000 7f8032817740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 0x559773e30e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.358+0000 7f8032817740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e31380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.358+0000 7f8032817740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559773da1b90 con 0x559773a652f0 2024-05-27T09:37:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.359+0000 7f8032817740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559773d91730 con 0x559773a8e1b0 2024-05-27T09:37:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.359+0000 7f8032817740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559773d95770 con 0x559773a6d4d0 2024-05-27T09:37:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.359+0000 7f8028e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 0x559773e30900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.359+0000 7f8023e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 0x559773e30e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.359+0000 7f8028e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 0x559773e30900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37018/0 (socket says 172.21.15.44:37018) 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.359+0000 7f8023e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 0x559773e30e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53034/0 (socket says 172.21.15.44:53034) 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.359+0000 7f8028e00640 1 -- 172.21.15.44:0/3953157847 learned_addr learned my addr 172.21.15.44:0/3953157847 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.359+0000 7f8029800640 1 --2- 172.21.15.44:0/3953157847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e31380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.359+0000 7f8028e00640 1 -- 172.21.15.44:0/3953157847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 msgr2=0x559773e31380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.360+0000 7f8028e00640 1 --2- 172.21.15.44:0/3953157847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e31380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.360+0000 7f8028e00640 1 -- 172.21.15.44:0/3953157847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 msgr2=0x559773e30e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.360+0000 7f8028e00640 1 --2- 172.21.15.44:0/3953157847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 0x559773e30e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.360+0000 7f8028e00640 1 -- 172.21.15.44:0/3953157847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559773d2a8b0 con 0x559773a8e1b0 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.360+0000 7f8023e00640 1 --2- 172.21.15.44:0/3953157847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 0x559773e30e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.360+0000 7f8029800640 1 --2- 172.21.15.44:0/3953157847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e31380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.360+0000 7f8028e00640 1 --2- 172.21.15.44:0/3953157847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 0x559773e30900 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f801800c8e0 tx=0x7f80180a53a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fab761e85037b1b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.360+0000 7f8016000640 1 -- 172.21.15.44:0/3953157847 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80180b23c0 con 0x559773a8e1b0 2024-05-27T09:37:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.361+0000 7f8016000640 1 -- 172.21.15.44:0/3953157847 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80180aa310 con 0x559773a8e1b0 2024-05-27T09:37:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.361+0000 7f8016000640 1 -- 172.21.15.44:0/3953157847 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80180037c0 con 0x559773a8e1b0 2024-05-27T09:37:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.362+0000 7f8032817740 1 -- 172.21.15.44:0/3953157847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 msgr2=0x559773e30900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.362+0000 7f8032817740 1 --2- 172.21.15.44:0/3953157847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 0x559773e30900 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f801800c8e0 tx=0x7f80180a53a0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.362+0000 7f8032817740 1 -- 172.21.15.44:0/3953157847 shutdown_connections 2024-05-27T09:37:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.362+0000 7f8032817740 1 --2- 172.21.15.44:0/3953157847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e31380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.363+0000 7f8032817740 1 --2- 172.21.15.44:0/3953157847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 0x559773e30e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.363+0000 7f8032817740 1 --2- 172.21.15.44:0/3953157847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 0x559773e30900 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.363+0000 7f8032817740 1 -- 172.21.15.44:0/3953157847 >> 172.21.15.44:0/3953157847 conn(0x559773a82300 msgr2=0x559773e2d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.363+0000 7f8032817740 1 -- 172.21.15.44:0/3953157847 shutdown_connections 2024-05-27T09:37:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.363+0000 7f8032817740 1 -- 172.21.15.44:0/3953157847 wait complete. 2024-05-27T09:37:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.390+0000 7f8032817740 1 -- start start 2024-05-27T09:37:51.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.390+0000 7f8032817740 1 Processor -- start 2024-05-27T09:37:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.392+0000 7f8032817740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 0x559773e3ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.392+0000 7f8032817740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e3f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.392+0000 7f8032817740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a8e1b0 0x559773e3fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.392+0000 7f8032817740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559773a77790 con 0x559773a8e1b0 2024-05-27T09:37:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.392+0000 7f8032817740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559773d39bc0 con 0x559773a652f0 2024-05-27T09:37:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.392+0000 7f8032817740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559773e41fd0 con 0x559773a6d4d0 2024-05-27T09:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.392+0000 7f8023e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e3f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.392+0000 7f8028e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 0x559773e3ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.392+0000 7f8023e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e3f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:33454/0 (socket says 172.21.15.44:33454) 2024-05-27T09:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.392+0000 7f8023e00640 1 -- 172.21.15.44:0/2869422627 learned_addr learned my addr 172.21.15.44:0/2869422627 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.392+0000 7f8029800640 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a8e1b0 0x559773e3fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.393+0000 7f8032817740 10 client.0 _ll_register_callbacks cb 0x559773d17400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.393+0000 7f8023e00640 1 -- 172.21.15.44:0/2869422627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 msgr2=0x559773e3ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.393+0000 7f8023e00640 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 0x559773e3ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.393+0000 7f8023e00640 1 -- 172.21.15.44:0/2869422627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a8e1b0 msgr2=0x559773e3fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.393+0000 7f8023e00640 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a8e1b0 0x559773e3fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.393+0000 7f8023e00640 1 -- 172.21.15.44:0/2869422627 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559773aa6210 con 0x559773a6d4d0 2024-05-27T09:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.393+0000 7f8028e00640 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 0x559773e3ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.393+0000 7f8032817740 1 -- 172.21.15.44:0/2869422627 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80180aa310 con 0x559773a6d4d0 2024-05-27T09:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.393+0000 7f8023e00640 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e3f4c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8010012c40 tx=0x7f8010004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f8ad3c908826b4a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.393+0000 7f800aa00640 10 client.25801 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.394+0000 7f800aa00640 1 -- 172.21.15.44:0/2869422627 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8010004050 con 0x559773a6d4d0 2024-05-27T09:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.394+0000 7f800aa00640 1 -- 172.21.15.44:0/2869422627 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f80100041f0 con 0x559773a6d4d0 2024-05-27T09:37:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.394+0000 7f800aa00640 1 -- 172.21.15.44:0/2869422627 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8010004410 con 0x559773a6d4d0 2024-05-27T09:37:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.394+0000 7f8032817740 10 client.25801 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.394+0000 7f8032817740 1 -- 172.21.15.44:0/2869422627 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=26}) -- 0x559773b72620 con 0x559773a6d4d0 2024-05-27T09:37:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.395+0000 7f800aa00640 1 -- 172.21.15.44:0/2869422627 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8010027020 con 0x559773a6d4d0 2024-05-27T09:37:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.396+0000 7f8032817740 10 client.25801 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.396+0000 7f8032817740 10 client.25801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.396+0000 7f8032817740 20 client.25801 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.396+0000 7f8032817740 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559773a4cd80 0x559773e51140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.396+0000 7f8032817740 4 client.25801 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:37:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.396+0000 7f8032817740 1 -- 172.21.15.44:0/2869422627 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559773da3ae0 con 0x559773a4cd80 2024-05-27T09:37:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.396+0000 7f8028e00640 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559773a4cd80 0x559773e51140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.397+0000 7f8028e00640 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559773a4cd80 0x559773e51140 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9c5e081f5183753 server_cookie=e9421339bcf3bc85 in_seq=0 out_seq=0 2024-05-27T09:37:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.397+0000 7f800aa00640 10 client.25801 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.398+0000 7f800aa00640 1 -- 172.21.15.44:0/2869422627 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559773da3ae0 con 0x559773a4cd80 2024-05-27T09:37:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.398+0000 7f800aa00640 10 client.25801 handle_command_reply: tid=0 2024-05-27T09:37:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.398+0000 7f8032817740 1 client.25801 shutdown 2024-05-27T09:37:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.399+0000 7f8032817740 1 -- 172.21.15.44:0/2869422627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 msgr2=0x559773e3f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.399+0000 7f8032817740 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e3f4c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8010012c40 tx=0x7f8010004730 comp rx=0 tx=0).stop 2024-05-27T09:37:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.399+0000 7f8032817740 1 -- 172.21.15.44:0/2869422627 shutdown_connections 2024-05-27T09:37:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.399+0000 7f8032817740 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559773a4cd80 0x559773e51140 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.400+0000 7f8032817740 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a8e1b0 0x559773e3fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.400+0000 7f8032817740 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e3f4c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.400+0000 7f8032817740 1 --2- 172.21.15.44:0/2869422627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 0x559773e3ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.400+0000 7f800aa00640 0 client.25801 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.400+0000 7f8032817740 1 -- 172.21.15.44:0/2869422627 >> 172.21.15.44:0/2869422627 conn(0x559773a82300 msgr2=0x559773e326a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.400+0000 7f8032817740 1 -- 172.21.15.44:0/2869422627 shutdown_connections 2024-05-27T09:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.400+0000 7f8032817740 1 -- 172.21.15.44:0/2869422627 wait complete. 2024-05-27T09:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.400+0000 7f8032817740 20 client.25801 trim_cache size 0 max 16384 2024-05-27T09:37:51.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.431+0000 7f8032817740 1 Processor -- start 2024-05-27T09:37:51.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.431+0000 7f8032817740 1 -- start start 2024-05-27T09:37:51.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.431+0000 7f8032817740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 0x559773e3a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.432+0000 7f8032817740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e2b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.432+0000 7f8032817740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 0x559773e3f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.432+0000 7f8032817740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f801c142dc0 con 0x559773a652f0 2024-05-27T09:37:51.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.432+0000 7f8032817740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f801c140690 con 0x559773a8e1b0 2024-05-27T09:37:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.432+0000 7f8032817740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f801c13e4e0 con 0x559773a6d4d0 2024-05-27T09:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.432+0000 7f8023e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e2b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.432+0000 7f8028e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 0x559773e3a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.432+0000 7f8023e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e2b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:33462/0 (socket says 172.21.15.44:33462) 2024-05-27T09:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.432+0000 7f8023e00640 1 -- 172.21.15.44:0/689852680 learned_addr learned my addr 172.21.15.44:0/689852680 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.432+0000 7f8029800640 1 --2- 172.21.15.44:0/689852680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 0x559773e3f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.433+0000 7f8023e00640 1 -- 172.21.15.44:0/689852680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 msgr2=0x559773e3f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.433+0000 7f8023e00640 1 --2- 172.21.15.44:0/689852680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 0x559773e3f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.433+0000 7f8023e00640 1 -- 172.21.15.44:0/689852680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 msgr2=0x559773e3a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.433+0000 7f8023e00640 1 --2- 172.21.15.44:0/689852680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 0x559773e3a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.433+0000 7f8023e00640 1 -- 172.21.15.44:0/689852680 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559773e336b0 con 0x559773a6d4d0 2024-05-27T09:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.433+0000 7f8028e00640 1 --2- 172.21.15.44:0/689852680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 0x559773e3a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:37:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.433+0000 7f8023e00640 1 --2- 172.21.15.44:0/689852680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e2b810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8010011850 tx=0x7f8010022040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18fcf5622ab0b142 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.434+0000 7f8015600640 1 -- 172.21.15.44:0/689852680 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8010005040 con 0x559773a6d4d0 2024-05-27T09:37:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.434+0000 7f8015600640 1 -- 172.21.15.44:0/689852680 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80180aa310 con 0x559773a6d4d0 2024-05-27T09:37:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.434+0000 7f8015600640 1 -- 172.21.15.44:0/689852680 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8010004de0 con 0x559773a6d4d0 2024-05-27T09:37:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.435+0000 7f8032817740 1 -- 172.21.15.44:0/689852680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 msgr2=0x559773e2b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.435+0000 7f8032817740 1 --2- 172.21.15.44:0/689852680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e2b810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8010011850 tx=0x7f8010022040 comp rx=0 tx=0).stop 2024-05-27T09:37:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.435+0000 7f8032817740 1 -- 172.21.15.44:0/689852680 shutdown_connections 2024-05-27T09:37:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.435+0000 7f8032817740 1 --2- 172.21.15.44:0/689852680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a8e1b0 0x559773e3f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.436+0000 7f8032817740 1 --2- 172.21.15.44:0/689852680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e2b810 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.436+0000 7f8032817740 1 --2- 172.21.15.44:0/689852680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a652f0 0x559773e3a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.436+0000 7f8032817740 1 -- 172.21.15.44:0/689852680 >> 172.21.15.44:0/689852680 conn(0x559773a82300 msgr2=0x559773e32fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.436+0000 7f8032817740 1 -- 172.21.15.44:0/689852680 shutdown_connections 2024-05-27T09:37:51.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.436+0000 7f8032817740 1 -- 172.21.15.44:0/689852680 wait complete. 2024-05-27T09:37:51.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.437+0000 7f8032817740 1 -- start start 2024-05-27T09:37:51.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.437+0000 7f8032817740 1 Processor -- start 2024-05-27T09:37:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.438+0000 7f8032817740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 0x559773e38e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.438+0000 7f8032817740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.438+0000 7f8032817740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a8e1b0 0x559773e39900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.438+0000 7f8032817740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f801c13eb50 con 0x559773a8e1b0 2024-05-27T09:37:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.438+0000 7f8032817740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f801c13ffc0 con 0x559773a652f0 2024-05-27T09:37:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.438+0000 7f8032817740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f801c13f220 con 0x559773a6d4d0 2024-05-27T09:37:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.438+0000 7f8032817740 10 client.0 _ll_register_callbacks cb 0x7f80100128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:37:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.439+0000 7f8023e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.439+0000 7f8023e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e393c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:33464/0 (socket says 172.21.15.44:33464) 2024-05-27T09:37:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.439+0000 7f8023e00640 1 -- 172.21.15.44:0/4257287405 learned_addr learned my addr 172.21.15.44:0/4257287405 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:37:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.439+0000 7f8028e00640 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 0x559773e38e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.439+0000 7f8029800640 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a8e1b0 0x559773e39900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.439+0000 7f8023e00640 1 -- 172.21.15.44:0/4257287405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 msgr2=0x559773e38e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.439+0000 7f8023e00640 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 0x559773e38e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.439+0000 7f8023e00640 1 -- 172.21.15.44:0/4257287405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a8e1b0 msgr2=0x559773e39900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.439+0000 7f8023e00640 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a8e1b0 0x559773e39900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.439+0000 7f8023e00640 1 -- 172.21.15.44:0/4257287405 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559773e46470 con 0x559773a6d4d0 2024-05-27T09:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.439+0000 7f8028e00640 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 0x559773e38e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.440+0000 7f8023e00640 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e393c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8010020d00 tx=0x7f8010003710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4860af9465cd0f44 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.440+0000 7f800aa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.440+0000 7f800aa00640 1 -- 172.21.15.44:0/4257287405 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801004a060 con 0x559773a6d4d0 2024-05-27T09:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.440+0000 7f800aa00640 1 -- 172.21.15.44:0/4257287405 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f8010005250 con 0x559773a6d4d0 2024-05-27T09:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.440+0000 7f8032817740 1 -- 172.21.15.44:0/4257287405 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80180aa310 con 0x559773a6d4d0 2024-05-27T09:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.441+0000 7f800aa00640 1 -- 172.21.15.44:0/4257287405 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8010005450 con 0x559773a6d4d0 2024-05-27T09:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.441+0000 7f8032817740 10 client.25807 fetch_fsmap learned FSMap version 26 2024-05-27T09:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.441+0000 7f8032817740 1 -- 172.21.15.44:0/4257287405 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559773e476e0 con 0x559773a6d4d0 2024-05-27T09:37:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.441+0000 7f800aa00640 1 -- 172.21.15.44:0/4257287405 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80180aa310 con 0x559773a6d4d0 2024-05-27T09:37:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.441+0000 7f800aa00640 1 -- 172.21.15.44:0/4257287405 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8010027040 con 0x559773a6d4d0 2024-05-27T09:37:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.441+0000 7f8032817740 10 client.25807 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:37:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.441+0000 7f8032817740 10 client.25807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:37:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.441+0000 7f8032817740 20 client.25807 populate_metadata read hostname 'smithi044' 2024-05-27T09:37:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.441+0000 7f8032817740 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559773a221c0 0x559773ebb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:37:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.442+0000 7f8032817740 4 client.25807 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:37:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.442+0000 7f8032817740 1 -- 172.21.15.44:0/4257287405 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559773d9d2c0 con 0x559773a221c0 2024-05-27T09:37:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.442+0000 7f8028e00640 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559773a221c0 0x559773ebb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:37:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.443+0000 7f8028e00640 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559773a221c0 0x559773ebb820 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6134d85023f92314 server_cookie=d6a482e63a8d36ca in_seq=0 out_seq=0 2024-05-27T09:37:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.443+0000 7f800aa00640 10 client.25807 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.461+0000 7f800aa00640 1 -- 172.21.15.44:0/4257287405 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x559773d9d2c0 con 0x559773a221c0 2024-05-27T09:37:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.461+0000 7f800aa00640 10 client.25807 handle_command_reply: tid=0 2024-05-27T09:37:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.461+0000 7f8032817740 1 client.25807 shutdown 2024-05-27T09:37:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f8032817740 1 -- 172.21.15.44:0/4257287405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 msgr2=0x559773e393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f8032817740 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e393c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8010020d00 tx=0x7f8010003710 comp rx=0 tx=0).stop 2024-05-27T09:37:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f8032817740 1 -- 172.21.15.44:0/4257287405 shutdown_connections 2024-05-27T09:37:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f8032817740 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559773a221c0 0x559773ebb820 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f8032817740 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559773a8e1b0 0x559773e39900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f8032817740 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559773a6d4d0 0x559773e393c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f8032817740 1 --2- 172.21.15.44:0/4257287405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559773a652f0 0x559773e38e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f800aa00640 0 client.25807 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:37:51.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f8032817740 1 -- 172.21.15.44:0/4257287405 >> 172.21.15.44:0/4257287405 conn(0x559773a82300 msgr2=0x559773e4aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f8032817740 1 -- 172.21.15.44:0/4257287405 shutdown_connections 2024-05-27T09:37:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f8032817740 1 -- 172.21.15.44:0/4257287405 wait complete. 2024-05-27T09:37:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.462+0000 7f8032817740 20 client.25807 trim_cache size 0 max 16384 2024-05-27T09:37:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:37:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:37:51.464 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:37:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.465+0000 7f802a200640 1 -- 172.21.15.44:0/2543664578 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f800c0b2ab0 msgr2=0x7f800c0b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.465+0000 7f802a200640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f800c0b2ab0 0x7f800c0b4fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f801c0a17b0 tx=0x7f8010005f90 comp rx=0 tx=0).stop 2024-05-27T09:37:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.465+0000 7f802a200640 1 -- 172.21.15.44:0/2543664578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c097b90 msgr2=0x7f801c12d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:37:51.465 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.465+0000 7f802a200640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c097b90 0x7f801c12d510 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f801800e9c0 tx=0x7f801800ee80 comp rx=0 tx=0).stop 2024-05-27T09:37:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.465+0000 7f802a200640 1 -- 172.21.15.44:0/2543664578 shutdown_connections 2024-05-27T09:37:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.465+0000 7f802a200640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f801c0a0f50 0x7f801c131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.465+0000 7f802a200640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f800c0b2ab0 0x7f800c0b4fc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.465+0000 7f802a200640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f801c09e5f0 0x7f801c12da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.465+0000 7f802a200640 1 --2- 172.21.15.44:0/2543664578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c097b90 0x7f801c12d510 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:37:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.465+0000 7f802a200640 1 -- 172.21.15.44:0/2543664578 >> 172.21.15.44:0/2543664578 conn(0x7f801c00b000 msgr2=0x7f801c094090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:37:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.465+0000 7f802a200640 1 -- 172.21.15.44:0/2543664578 shutdown_connections 2024-05-27T09:37:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:37:51.466+0000 7f802a200640 1 -- 172.21.15.44:0/2543664578 wait complete. 2024-05-27T09:37:51.983 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:51.983 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:52.116 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:52.116 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:52.123 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:52.123 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:52.123 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 4 -lt 100 ']' 2024-05-27T09:37:52.125 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 4 + 1 2024-05-27T09:37:52.153 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=5 2024-05-27T09:37:52.153 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:52.186 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:52.187 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:52.192 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:52.193 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:52.193 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:52.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:52 smithi044 ceph-mon[31618]: pgmap v82: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-27T09:37:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:52 smithi050 ceph-mon[36371]: pgmap v82: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-27T09:37:52.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:52 smithi189 ceph-mon[36671]: pgmap v82: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-27T09:37:52.752 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:52.753 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:52.890 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:52.890 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:52.903 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:52.903 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:52.903 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 5 -lt 100 ']' 2024-05-27T09:37:52.905 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 5 + 1 2024-05-27T09:37:52.927 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=6 2024-05-27T09:37:52.928 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:52.951 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:52.952 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:52.957 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:52.958 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:52.959 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:53.444 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:53.444 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:53.590 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:53.590 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:53.602 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:53.602 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:53.602 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 6 -lt 100 ']' 2024-05-27T09:37:53.603 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 6 + 1 2024-05-27T09:37:53.627 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=7 2024-05-27T09:37:53.628 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:53.661 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:53.661 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:53.666 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:53.667 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:53.667 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:54.150 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:54.150 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:54.273 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:54.273 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:54.286 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:54.286 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:54.286 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 7 -lt 100 ']' 2024-05-27T09:37:54.288 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 7 + 1 2024-05-27T09:37:54.311 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=8 2024-05-27T09:37:54.311 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:54.334 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:54.335 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:54.340 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:54.342 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:54.342 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:54.960 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:54.961 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:55.099 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:55.099 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:55.111 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:55.111 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:55.111 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 8 -lt 100 ']' 2024-05-27T09:37:55.112 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 8 + 1 2024-05-27T09:37:55.136 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=9 2024-05-27T09:37:55.136 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:55.158 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:55.159 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:55.163 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:55.164 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:55.165 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:55.465 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:55 smithi044 ceph-mon[31618]: pgmap v83: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-27T09:37:55.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:55 smithi050 ceph-mon[36371]: pgmap v83: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-27T09:37:55.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:55 smithi189 ceph-mon[36671]: pgmap v83: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-27T09:37:55.678 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:55.678 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:55.814 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:55.815 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:55.827 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:55.827 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:55.828 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 9 -lt 100 ']' 2024-05-27T09:37:55.829 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 9 + 1 2024-05-27T09:37:55.852 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=10 2024-05-27T09:37:55.852 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:55.894 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:55.894 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:55.899 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:55.900 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:55.900 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:56.474 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:56.474 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:56.597 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:56.598 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:56.619 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:56.619 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:56.619 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 10 -lt 100 ']' 2024-05-27T09:37:56.620 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 10 + 1 2024-05-27T09:37:56.726 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=11 2024-05-27T09:37:56.726 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:56.760 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:56.760 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:56.765 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:56.766 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:56.766 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:57.301 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:57.302 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:57.422 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:57.422 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:57.435 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:57.436 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:57.436 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 11 -lt 100 ']' 2024-05-27T09:37:57.437 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 11 + 1 2024-05-27T09:37:57.460 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=12 2024-05-27T09:37:57.460 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:57.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:57 smithi044 ceph-mon[31618]: pgmap v84: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-05-27T09:37:57.484 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:57.485 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:57.489 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:57.491 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:57.491 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:57.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:57 smithi050 ceph-mon[36371]: pgmap v84: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-05-27T09:37:57.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:57 smithi189 ceph-mon[36671]: pgmap v84: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-05-27T09:37:58.002 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:58.002 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:58.125 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:58.125 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:58.135 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:58.135 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:58.135 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 12 -lt 100 ']' 2024-05-27T09:37:58.136 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 12 + 1 2024-05-27T09:37:58.160 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=13 2024-05-27T09:37:58.160 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:58.182 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:58.183 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:58.187 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:58.188 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:58.189 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:58.682 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:58.683 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:58.873 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:58.873 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:58.886 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:58.886 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:58.886 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 13 -lt 100 ']' 2024-05-27T09:37:58.887 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 13 + 1 2024-05-27T09:37:58.910 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=14 2024-05-27T09:37:58.911 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:58.936 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:58.937 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:58.942 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:58.943 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:58.943 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:37:59.418 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:59.419 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:59.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:37:59 smithi044 ceph-mon[31618]: pgmap v85: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-05-27T09:37:59.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:37:59 smithi050 ceph-mon[36371]: pgmap v85: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-05-27T09:37:59.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:37:59 smithi189 ceph-mon[36671]: pgmap v85: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-05-27T09:37:59.622 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:37:59.623 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:37:59.643 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:59.643 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:37:59.643 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 14 -lt 100 ']' 2024-05-27T09:37:59.644 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 14 + 1 2024-05-27T09:37:59.676 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=15 2024-05-27T09:37:59.676 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:37:59.700 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:37:59.700 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:37:59.705 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:37:59.706 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:37:59.706 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:00.202 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:00.203 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:00.323 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:00.323 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:00.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:00.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:00.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 15 -lt 100 ']' 2024-05-27T09:38:00.337 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 15 + 1 2024-05-27T09:38:00.360 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=16 2024-05-27T09:38:00.360 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:00.383 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:00.383 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:00.388 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:00.390 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:00.390 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:00.941 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:00.941 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:01.097 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:01.097 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:01.109 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:01.109 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:01.109 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 16 -lt 100 ']' 2024-05-27T09:38:01.110 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 16 + 1 2024-05-27T09:38:01.134 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=17 2024-05-27T09:38:01.134 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:01.167 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:01.168 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:01.173 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:01.174 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:01.174 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:01.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:01 smithi044 ceph-mon[31618]: pgmap v86: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 14 op/s 2024-05-27T09:38:01.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:01 smithi050 ceph-mon[36371]: pgmap v86: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 14 op/s 2024-05-27T09:38:01.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:01 smithi189 ceph-mon[36671]: pgmap v86: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 14 op/s 2024-05-27T09:38:01.706 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:01.707 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:01.930 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:01.931 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:01.944 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:01.944 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:01.944 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 17 -lt 100 ']' 2024-05-27T09:38:01.945 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 17 + 1 2024-05-27T09:38:01.968 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=18 2024-05-27T09:38:01.968 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:01.991 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:01.992 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:01.996 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:01.998 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:01.999 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:02.624 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:02.625 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:02.822 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:02.822 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:02.834 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:02.834 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:02.834 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 18 -lt 100 ']' 2024-05-27T09:38:02.835 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 18 + 1 2024-05-27T09:38:02.859 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=19 2024-05-27T09:38:02.860 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:02.882 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:02.882 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:02.887 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:02.888 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:02.889 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:03.393 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:03.393 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:03.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:03 smithi044 ceph-mon[31618]: pgmap v87: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-05-27T09:38:03.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:03 smithi050 ceph-mon[36371]: pgmap v87: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-05-27T09:38:03.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:03 smithi189 ceph-mon[36671]: pgmap v87: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-05-27T09:38:03.597 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:03.597 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:03.610 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:03.610 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:03.610 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 19 -lt 100 ']' 2024-05-27T09:38:03.612 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 19 + 1 2024-05-27T09:38:03.634 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=20 2024-05-27T09:38:03.635 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:03.666 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:03.667 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:03.671 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:03.672 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:03.673 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:04.173 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:04.174 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:04.305 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:04.305 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:04.317 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:04.318 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:04.318 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 20 -lt 100 ']' 2024-05-27T09:38:04.319 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 20 + 1 2024-05-27T09:38:04.343 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=21 2024-05-27T09:38:04.343 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:04.366 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:04.366 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:04.371 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:04.372 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:04.373 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:05.072 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:05.073 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:05.338 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:05.338 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:05.360 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:05.360 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:05.360 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 21 -lt 100 ']' 2024-05-27T09:38:05.361 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 21 + 1 2024-05-27T09:38:05.385 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=22 2024-05-27T09:38:05.385 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:05.416 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:05.416 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:05.422 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:05.423 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:05.424 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:05.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:05 smithi050 ceph-mon[36371]: pgmap v88: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-05-27T09:38:05.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:05 smithi189 ceph-mon[36671]: pgmap v88: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-05-27T09:38:05.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:05 smithi044 ceph-mon[31618]: pgmap v88: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-05-27T09:38:06.282 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:06.282 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:06.413 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:06.413 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:06.426 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:06.426 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:06.426 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 22 -lt 100 ']' 2024-05-27T09:38:06.427 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 22 + 1 2024-05-27T09:38:06.592 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=23 2024-05-27T09:38:06.592 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:06.624 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:06.625 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:06.629 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:06.630 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:06.631 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:07.416 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:07 smithi044 ceph-mon[31618]: pgmap v89: 97 pgs: 97 active+clean; 1.6 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 46 KiB/s wr, 30 op/s 2024-05-27T09:38:07.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:07 smithi050 ceph-mon[36371]: pgmap v89: 97 pgs: 97 active+clean; 1.6 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 46 KiB/s wr, 30 op/s 2024-05-27T09:38:07.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:07 smithi189 ceph-mon[36671]: pgmap v89: 97 pgs: 97 active+clean; 1.6 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 46 KiB/s wr, 30 op/s 2024-05-27T09:38:07.588 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:07.589 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:07.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.640+0000 7fa225a00640 1 -- 172.21.15.44:0/4147748679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa220109c60 msgr2=0x7fa220103800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:07.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.640+0000 7fa225a00640 1 --2- 172.21.15.44:0/4147748679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa220109c60 0x7fa220103800 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa210009860 tx=0x7fa21002a340 comp rx=0 tx=0).stop 2024-05-27T09:38:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.640+0000 7fa225a00640 1 -- 172.21.15.44:0/4147748679 shutdown_connections 2024-05-27T09:38:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.640+0000 7fa225a00640 1 --2- 172.21.15.44:0/4147748679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2201043b0 0x7fa220110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.640+0000 7fa225a00640 1 --2- 172.21.15.44:0/4147748679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa22010a100 0x7fa220103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.640+0000 7fa225a00640 1 --2- 172.21.15.44:0/4147748679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa220109c60 0x7fa220103800 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.641+0000 7fa225a00640 1 -- 172.21.15.44:0/4147748679 >> 172.21.15.44:0/4147748679 conn(0x7fa2200ff360 msgr2=0x7fa2201017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.641+0000 7fa225a00640 1 -- 172.21.15.44:0/4147748679 shutdown_connections 2024-05-27T09:38:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.641+0000 7fa225a00640 1 -- 172.21.15.44:0/4147748679 wait complete. 2024-05-27T09:38:07.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.658+0000 7fa225a00640 1 Processor -- start 2024-05-27T09:38:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.659+0000 7fa225a00640 1 -- start start 2024-05-27T09:38:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.660+0000 7fa225a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2201043b0 0x7fa2200760d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.660+0000 7fa225a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa220109c60 0x7fa220076610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.660+0000 7fa225a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa22010a100 0x7fa220076b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.660+0000 7fa225a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa220113e50 con 0x7fa2201043b0 2024-05-27T09:38:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.660+0000 7fa225a00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa220113cd0 con 0x7fa22010a100 2024-05-27T09:38:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.660+0000 7fa225a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa220113fd0 con 0x7fa220109c60 2024-05-27T09:38:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.660+0000 7fa217e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa220109c60 0x7fa220076610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.660+0000 7fa217e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa220109c60 0x7fa220076610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45264/0 (socket says 172.21.15.44:45264) 2024-05-27T09:38:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.660+0000 7fa217e00640 1 -- 172.21.15.44:0/1994295837 learned_addr learned my addr 172.21.15.44:0/1994295837 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.661+0000 7fa21f400640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa22010a100 0x7fa220076b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.661+0000 7fa21ea00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2201043b0 0x7fa2200760d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.661+0000 7fa217e00640 1 -- 172.21.15.44:0/1994295837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa22010a100 msgr2=0x7fa220076b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.661+0000 7fa217e00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa22010a100 0x7fa220076b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.661+0000 7fa217e00640 1 -- 172.21.15.44:0/1994295837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2201043b0 msgr2=0x7fa2200760d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.661+0000 7fa217e00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2201043b0 0x7fa2200760d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.661+0000 7fa217e00640 1 -- 172.21.15.44:0/1994295837 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2201a8bf0 con 0x7fa220109c60 2024-05-27T09:38:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.661+0000 7fa21ea00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2201043b0 0x7fa2200760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:38:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.662+0000 7fa217e00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa220109c60 0x7fa220076610 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa20800e980 tx=0x7fa20800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.663+0000 7fa217400640 1 -- 172.21.15.44:0/1994295837 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20800cd80 con 0x7fa220109c60 2024-05-27T09:38:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.663+0000 7fa217400640 1 -- 172.21.15.44:0/1994295837 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2080108e0 con 0x7fa220109c60 2024-05-27T09:38:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.663+0000 7fa217400640 1 -- 172.21.15.44:0/1994295837 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208018940 con 0x7fa220109c60 2024-05-27T09:38:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.663+0000 7fa225a00640 1 -- 172.21.15.44:0/1994295837 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2201a8ee0 con 0x7fa220109c60 2024-05-27T09:38:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.664+0000 7fa225a00640 1 -- 172.21.15.44:0/1994295837 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2201a92f0 con 0x7fa220109c60 2024-05-27T09:38:07.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.669+0000 7fa217400640 1 -- 172.21.15.44:0/1994295837 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa20801f020 con 0x7fa220109c60 2024-05-27T09:38:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.671+0000 7fa217400640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2040b2aa0 0x7fa2040b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.671+0000 7fa217400640 1 -- 172.21.15.44:0/1994295837 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa2080e26a0 con 0x7fa220109c60 2024-05-27T09:38:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.672+0000 7fa21ea00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2040b2aa0 0x7fa2040b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.673+0000 7fa21ea00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2040b2aa0 0x7fa2040b4fb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa21002a930 tx=0x7fa2100058b0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.777+0000 7fa22ca3a740 1 Processor -- start 2024-05-27T09:38:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.777+0000 7fa22ca3a740 1 -- start start 2024-05-27T09:38:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.778+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b60553f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.778+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051742f0 0x55b60553fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.779+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b60517c4d0 0x55b6055402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.779+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b6054b0b90 con 0x55b6051742f0 2024-05-27T09:38:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.779+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b6054a0730 con 0x55b60517c4d0 2024-05-27T09:38:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.779+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b6054a4770 con 0x55b60519d1b0 2024-05-27T09:38:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.779+0000 7fa217e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051742f0 0x55b60553fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.779+0000 7fa217e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051742f0 0x55b60553fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37548/0 (socket says 172.21.15.44:37548) 2024-05-27T09:38:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.779+0000 7fa217e00640 1 -- 172.21.15.44:0/4125031001 learned_addr learned my addr 172.21.15.44:0/4125031001 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.779+0000 7fa21ea00640 1 --2- 172.21.15.44:0/4125031001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b60553f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.779+0000 7fa21f400640 1 --2- 172.21.15.44:0/4125031001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b60517c4d0 0x55b6055402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.780+0000 7fa217e00640 1 -- 172.21.15.44:0/4125031001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 msgr2=0x55b60553f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.780+0000 7fa217e00640 1 --2- 172.21.15.44:0/4125031001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b60553f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.780+0000 7fa217e00640 1 -- 172.21.15.44:0/4125031001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b60517c4d0 msgr2=0x55b6055402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.780+0000 7fa217e00640 1 --2- 172.21.15.44:0/4125031001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b60517c4d0 0x55b6055402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.780+0000 7fa217e00640 1 -- 172.21.15.44:0/4125031001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6054398b0 con 0x55b6051742f0 2024-05-27T09:38:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.780+0000 7fa217e00640 1 --2- 172.21.15.44:0/4125031001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051742f0 0x55b60553fda0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa208009230 tx=0x7fa2080a9460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f21cac2aa8674e44 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.781+0000 7fa20fe00640 1 -- 172.21.15.44:0/4125031001 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2080add00 con 0x55b6051742f0 2024-05-27T09:38:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.781+0000 7fa20fe00640 1 -- 172.21.15.44:0/4125031001 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2080b27b0 con 0x55b6051742f0 2024-05-27T09:38:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.781+0000 7fa20fe00640 1 -- 172.21.15.44:0/4125031001 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208018b00 con 0x55b6051742f0 2024-05-27T09:38:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.782+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4125031001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051742f0 msgr2=0x55b60553fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.782+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4125031001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051742f0 0x55b60553fda0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa208009230 tx=0x7fa2080a9460 comp rx=0 tx=0).stop 2024-05-27T09:38:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.782+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4125031001 shutdown_connections 2024-05-27T09:38:07.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.782+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4125031001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b60517c4d0 0x55b6055402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.782+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4125031001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051742f0 0x55b60553fda0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.782+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4125031001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b60553f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.782+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4125031001 >> 172.21.15.44:0/4125031001 conn(0x55b605191300 msgr2=0x55b60553c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:07.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.782+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4125031001 shutdown_connections 2024-05-27T09:38:07.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.782+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4125031001 wait complete. 2024-05-27T09:38:07.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.804+0000 7fa22ca3a740 1 -- start start 2024-05-27T09:38:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.805+0000 7fa22ca3a740 1 Processor -- start 2024-05-27T09:38:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.806+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b6051742f0 0x55b605549d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.806+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b60517c4d0 0x55b60554a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.806+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b605555cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.806+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b605186790 con 0x55b60517c4d0 2024-05-27T09:38:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.806+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b605448bc0 con 0x55b6051742f0 2024-05-27T09:38:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.806+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b60554c990 con 0x55b60519d1b0 2024-05-27T09:38:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.807+0000 7fa217e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b60517c4d0 0x55b60554a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.807+0000 7fa21ea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b6051742f0 0x55b605549d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.807+0000 7fa217e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b60517c4d0 0x55b60554a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37562/0 (socket says 172.21.15.44:37562) 2024-05-27T09:38:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.807+0000 7fa217e00640 1 -- 172.21.15.44:0/3238065104 learned_addr learned my addr 172.21.15.44:0/3238065104 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.807+0000 7fa21f400640 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b605555cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.807+0000 7fa217e00640 1 -- 172.21.15.44:0/3238065104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 msgr2=0x55b605555cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.807+0000 7fa217e00640 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b605555cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.807+0000 7fa217e00640 1 -- 172.21.15.44:0/3238065104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b6051742f0 msgr2=0x55b605549d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.807+0000 7fa217e00640 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b6051742f0 0x55b605549d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.807+0000 7fa217e00640 1 -- 172.21.15.44:0/3238065104 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6051b5210 con 0x55b60517c4d0 2024-05-27T09:38:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.808+0000 7fa217e00640 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b60517c4d0 0x55b60554a260 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa2080a9650 tx=0x7fa208010f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465d6f6327032a64 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.808+0000 7fa20cc00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:38:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.808+0000 7fa20cc00640 1 -- 172.21.15.44:0/3238065104 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2080e0040 con 0x55b60517c4d0 2024-05-27T09:38:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.808+0000 7fa20cc00640 1 -- 172.21.15.44:0/3238065104 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa2080cc0d0 con 0x55b60517c4d0 2024-05-27T09:38:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.819+0000 7fa22ca3a740 10 client.0 _ll_register_callbacks cb 0x55b605426400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:38:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.819+0000 7fa22ca3a740 1 -- 172.21.15.44:0/3238065104 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2080b27b0 con 0x55b60517c4d0 2024-05-27T09:38:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.819+0000 7fa20cc00640 1 -- 172.21.15.44:0/3238065104 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa2080add00 con 0x55b60517c4d0 2024-05-27T09:38:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.820+0000 7fa22ca3a740 10 client.15675 fetch_fsmap learned FSMap version 26 2024-05-27T09:38:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.820+0000 7fa22ca3a740 1 -- 172.21.15.44:0/3238065104 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=26}) -- 0x55b605281620 con 0x55b60517c4d0 2024-05-27T09:38:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.821+0000 7fa20cc00640 1 -- 172.21.15.44:0/3238065104 <== mon.0 v2:172.21.15.44:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa2080b2320 con 0x55b60517c4d0 2024-05-27T09:38:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.821+0000 7fa22ca3a740 10 client.15675 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:38:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.821+0000 7fa22ca3a740 10 client.15675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:38:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.821+0000 7fa22ca3a740 20 client.15675 populate_metadata read hostname 'smithi044' 2024-05-27T09:38:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.821+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b60515bd80 0x55b60555ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.821+0000 7fa22ca3a740 4 client.15675 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:38:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.821+0000 7fa22ca3a740 1 -- 172.21.15.44:0/3238065104 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6054b2ae0 con 0x55b60515bd80 2024-05-27T09:38:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.822+0000 7fa21ea00640 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b60515bd80 0x55b60555ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.823+0000 7fa21ea00640 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b60515bd80 0x55b60555ba20 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc488c762cdc5e24 server_cookie=81f116d96e63f68e in_seq=0 out_seq=0 2024-05-27T09:38:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.823+0000 7fa20cc00640 10 client.15675 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.824+0000 7fa20cc00640 1 -- 172.21.15.44:0/3238065104 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b6054b2ae0 con 0x55b60515bd80 2024-05-27T09:38:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.824+0000 7fa20cc00640 10 client.15675 handle_command_reply: tid=0 2024-05-27T09:38:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.824+0000 7fa22ca3a740 1 client.15675 shutdown 2024-05-27T09:38:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.825+0000 7fa22ca3a740 1 -- 172.21.15.44:0/3238065104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b60517c4d0 msgr2=0x55b60554a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.825+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b60517c4d0 0x55b60554a260 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa2080a9650 tx=0x7fa208010f50 comp rx=0 tx=0).stop 2024-05-27T09:38:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.825+0000 7fa22ca3a740 1 -- 172.21.15.44:0/3238065104 shutdown_connections 2024-05-27T09:38:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.825+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b60515bd80 0x55b60555ba20 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.825+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b605555cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.825+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b60517c4d0 0x55b60554a260 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.826+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/3238065104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b6051742f0 0x55b605549d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.826+0000 7fa22ca3a740 1 -- 172.21.15.44:0/3238065104 >> 172.21.15.44:0/3238065104 conn(0x55b605191300 msgr2=0x55b60553f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.826+0000 7fa20cc00640 0 client.15675 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.826+0000 7fa22ca3a740 1 -- 172.21.15.44:0/3238065104 shutdown_connections 2024-05-27T09:38:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.826+0000 7fa22ca3a740 1 -- 172.21.15.44:0/3238065104 wait complete. 2024-05-27T09:38:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.826+0000 7fa22ca3a740 20 client.15675 trim_cache size 0 max 16384 2024-05-27T09:38:07.838 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:07.838 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:07.851 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:07.851 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:07.851 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 23 -lt 100 ']' 2024-05-27T09:38:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.851+0000 7fa22ca3a740 1 Processor -- start 2024-05-27T09:38:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.851+0000 7fa22ca3a740 1 -- start start 2024-05-27T09:38:07.852 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 23 + 1 2024-05-27T09:38:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.852+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b6051742f0 0x55b6055493f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.852+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b60555ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.852+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 0x55b60555ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.852+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2201b4540 con 0x55b6051311c0 2024-05-27T09:38:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.852+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa2201b1e10 con 0x55b6051742f0 2024-05-27T09:38:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.852+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2201afc60 con 0x55b60519d1b0 2024-05-27T09:38:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.852+0000 7fa21ea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b6051742f0 0x55b6055493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.852+0000 7fa217e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b60555ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.852+0000 7fa217e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b60555ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45290/0 (socket says 172.21.15.44:45290) 2024-05-27T09:38:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.852+0000 7fa217e00640 1 -- 172.21.15.44:0/4143779743 learned_addr learned my addr 172.21.15.44:0/4143779743 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.853+0000 7fa21f400640 1 --2- 172.21.15.44:0/4143779743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 0x55b60555ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.853+0000 7fa217e00640 1 -- 172.21.15.44:0/4143779743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b6051742f0 msgr2=0x55b6055493f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.853+0000 7fa217e00640 1 --2- 172.21.15.44:0/4143779743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b6051742f0 0x55b6055493f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.853+0000 7fa217e00640 1 -- 172.21.15.44:0/4143779743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 msgr2=0x55b60555ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.853+0000 7fa217e00640 1 --2- 172.21.15.44:0/4143779743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 0x55b60555ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.853+0000 7fa217e00640 1 -- 172.21.15.44:0/4143779743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b60554a9d0 con 0x55b60519d1b0 2024-05-27T09:38:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.853+0000 7fa21f400640 1 --2- 172.21.15.44:0/4143779743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 0x55b60555ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:38:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.853+0000 7fa217e00640 1 --2- 172.21.15.44:0/4143779743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b60555ea50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa2080adec0 tx=0x7fa2080c3000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41a13de9fb4bed8f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.854+0000 7fa20f400640 1 -- 172.21.15.44:0/4143779743 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2080e0040 con 0x55b60519d1b0 2024-05-27T09:38:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.854+0000 7fa20f400640 1 -- 172.21.15.44:0/4143779743 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2080b27b0 con 0x55b60519d1b0 2024-05-27T09:38:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.854+0000 7fa20f400640 1 -- 172.21.15.44:0/4143779743 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2080a8bb0 con 0x55b60519d1b0 2024-05-27T09:38:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.855+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4143779743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 msgr2=0x55b60555ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.855+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4143779743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b60555ea50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa2080adec0 tx=0x7fa2080c3000 comp rx=0 tx=0).stop 2024-05-27T09:38:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.855+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4143779743 shutdown_connections 2024-05-27T09:38:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.855+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4143779743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 0x55b60555ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.855+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4143779743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b60519d1b0 0x55b60555ea50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.855+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4143779743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b6051742f0 0x55b6055493f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.855+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4143779743 >> 172.21.15.44:0/4143779743 conn(0x55b605191300 msgr2=0x55b605544270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.856+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4143779743 shutdown_connections 2024-05-27T09:38:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.856+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4143779743 wait complete. 2024-05-27T09:38:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.856+0000 7fa22ca3a740 1 -- start start 2024-05-27T09:38:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.857+0000 7fa22ca3a740 1 Processor -- start 2024-05-27T09:38:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 0x55b60555dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6051742f0 0x55b60555e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa22ca3a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b60519d1b0 0x55b6055c1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2201b02d0 con 0x55b6051311c0 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa2201b1740 con 0x55b60519d1b0 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa22ca3a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2201b09a0 con 0x55b6051742f0 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa22ca3a740 10 client.0 _ll_register_callbacks cb 0x7fa218009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa21ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 0x55b60555dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa217e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6051742f0 0x55b60555e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa21ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 0x55b60555dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37590/0 (socket says 172.21.15.44:37590) 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa21ea00640 1 -- 172.21.15.44:0/4276820224 learned_addr learned my addr 172.21.15.44:0/4276820224 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.858+0000 7fa21f400640 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b60519d1b0 0x55b6055c1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.859+0000 7fa217e00640 1 -- 172.21.15.44:0/4276820224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b60519d1b0 msgr2=0x55b6055c1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.859+0000 7fa217e00640 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b60519d1b0 0x55b6055c1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.859+0000 7fa217e00640 1 -- 172.21.15.44:0/4276820224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 msgr2=0x55b60555dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.859+0000 7fa217e00640 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 0x55b60555dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.859+0000 7fa217e00640 1 -- 172.21.15.44:0/4276820224 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6055c23b0 con 0x55b6051742f0 2024-05-27T09:38:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.859+0000 7fa217e00640 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6051742f0 0x55b60555e400 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55b60553f350 tx=0x7fa2080cc380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=269a21a687c69886 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.859+0000 7fa20cc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:38:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.859+0000 7fa20cc00640 1 -- 172.21.15.44:0/4276820224 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208010430 con 0x55b6051742f0 2024-05-27T09:38:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.860+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4276820224 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2080b27b0 con 0x55b6051742f0 2024-05-27T09:38:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.860+0000 7fa20cc00640 1 -- 172.21.15.44:0/4276820224 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa2080105d0 con 0x55b6051742f0 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.860+0000 7fa20cc00640 1 -- 172.21.15.44:0/4276820224 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa2080a8bb0 con 0x55b6051742f0 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.860+0000 7fa22ca3a740 10 client.25825 fetch_fsmap learned FSMap version 26 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.860+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4276820224 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b6055c23b0 con 0x55b6051742f0 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.860+0000 7fa20cc00640 1 -- 172.21.15.44:0/4276820224 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2080b27b0 con 0x55b6051742f0 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.860+0000 7fa20cc00640 1 -- 172.21.15.44:0/4276820224 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa2080912a0 con 0x55b6051742f0 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.861+0000 7fa22ca3a740 10 client.25825 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.861+0000 7fa22ca3a740 10 client.25825 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.861+0000 7fa22ca3a740 20 client.25825 populate_metadata read hostname 'smithi044' 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.861+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b6055c3880 0x55b6055c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.861+0000 7fa22ca3a740 4 client.25825 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.861+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4276820224 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b6054ac2c0 con 0x55b6055c3880 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.861+0000 7fa21ea00640 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b6055c3880 0x55b6055c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.862+0000 7fa21ea00640 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b6055c3880 0x55b6055c5ca0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9348f1e3e6dffa87 server_cookie=ca874e2f482206d1 in_seq=0 out_seq=0 2024-05-27T09:38:07.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.862+0000 7fa20cc00640 10 client.25825 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:07.877 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=24 2024-05-27T09:38:07.877 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:07.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa20cc00640 1 -- 172.21.15.44:0/4276820224 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b6054ac2c0 con 0x55b6055c3880 2024-05-27T09:38:07.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa20cc00640 10 client.25825 handle_command_reply: tid=0 2024-05-27T09:38:07.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa22ca3a740 1 client.25825 shutdown 2024-05-27T09:38:07.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4276820224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6051742f0 msgr2=0x55b60555e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:07.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6051742f0 0x55b60555e400 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55b60553f350 tx=0x7fa2080cc380 comp rx=0 tx=0).stop 2024-05-27T09:38:07.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4276820224 shutdown_connections 2024-05-27T09:38:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b6055c3880 0x55b6055c5ca0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b60519d1b0 0x55b6055c1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6051742f0 0x55b60555e400 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa22ca3a740 1 --2- 172.21.15.44:0/4276820224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b6051311c0 0x55b60555dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa20cc00640 0 client.25825 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4276820224 >> 172.21.15.44:0/4276820224 conn(0x55b605191300 msgr2=0x55b605544c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.885+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4276820224 shutdown_connections 2024-05-27T09:38:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.886+0000 7fa22ca3a740 1 -- 172.21.15.44:0/4276820224 wait complete. 2024-05-27T09:38:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.886+0000 7fa22ca3a740 20 client.25825 trim_cache size 0 max 16384 2024-05-27T09:38:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:38:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:38:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:38:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 -- 172.21.15.44:0/1994295837 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2040b2aa0 msgr2=0x7fa2040b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2040b2aa0 0x7fa2040b4fb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa21002a930 tx=0x7fa2100058b0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 -- 172.21.15.44:0/1994295837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa220109c60 msgr2=0x7fa220076610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa220109c60 0x7fa220076610 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa20800e980 tx=0x7fa20800ee40 comp rx=0 tx=0).stop 2024-05-27T09:38:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 -- 172.21.15.44:0/1994295837 shutdown_connections 2024-05-27T09:38:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa22010a100 0x7fa220076b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa220109c60 0x7fa220076610 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2040b2aa0 0x7fa2040b4fb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 --2- 172.21.15.44:0/1994295837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2201043b0 0x7fa2200760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 -- 172.21.15.44:0/1994295837 >> 172.21.15.44:0/1994295837 conn(0x7fa2200ff360 msgr2=0x7fa2200ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 -- 172.21.15.44:0/1994295837 shutdown_connections 2024-05-27T09:38:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:07.887+0000 7fa225a00640 1 -- 172.21.15.44:0/1994295837 wait complete. 2024-05-27T09:38:07.895 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:07.896 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:07.899 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:07.899 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:07.900 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:08.548 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:08.548 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:08.704 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:08.705 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:08.716 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:08.716 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:08.716 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 24 -lt 100 ']' 2024-05-27T09:38:08.717 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 24 + 1 2024-05-27T09:38:08.742 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=25 2024-05-27T09:38:08.742 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:08.766 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:08.766 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:08.771 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:08.772 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:08.772 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:09.259 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:09.259 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:09.461 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-05-27T09:38:09.462 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:38:09.462 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:09.462 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:09.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:09 smithi044 ceph-mon[31618]: pgmap v90: 97 pgs: 97 active+clean; 1.6 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 KiB/s wr, 22 op/s 2024-05-27T09:38:09.475 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:09.476 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:09.476 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 25 -lt 100 ']' 2024-05-27T09:38:09.477 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 25 + 1 2024-05-27T09:38:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:09 smithi050 ceph-mon[36371]: pgmap v90: 97 pgs: 97 active+clean; 1.6 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 KiB/s wr, 22 op/s 2024-05-27T09:38:09.548 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=26 2024-05-27T09:38:09.548 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:09.573 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:09.573 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:09.578 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:09.578 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:09.578 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:09 smithi189 ceph-mon[36671]: pgmap v90: 97 pgs: 97 active+clean; 1.6 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 KiB/s wr, 22 op/s 2024-05-27T09:38:10.347 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:10.348 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:10.554 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:10.554 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:10.568 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:10.568 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:10.568 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 26 -lt 100 ']' 2024-05-27T09:38:10.569 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 26 + 1 2024-05-27T09:38:10.634 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=27 2024-05-27T09:38:10.635 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:10.666 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:10.667 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:10.671 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:10.672 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:10.673 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:10.932 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:38:11.323 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:11.323 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:11.446 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:11.446 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:11.454 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:11.454 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:11.454 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 27 -lt 100 ']' 2024-05-27T09:38:11.455 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 27 + 1 2024-05-27T09:38:11.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:11 smithi044 ceph-mon[31618]: pgmap v91: 97 pgs: 97 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-05-27T09:38:11.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:11 smithi050 ceph-mon[36371]: pgmap v91: 97 pgs: 97 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-05-27T09:38:11.549 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=28 2024-05-27T09:38:11.549 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:11.564 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:11.564 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:11.569 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:11.570 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:11.571 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:11.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:11 smithi189 ceph-mon[36671]: pgmap v91: 97 pgs: 97 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-05-27T09:38:12.598 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:12.598 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:13.454 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:13.454 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:13.468 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:13.468 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:13.468 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 28 -lt 100 ']' 2024-05-27T09:38:13.468 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 28 + 1 2024-05-27T09:38:13.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:13 smithi050 ceph-mon[36371]: pgmap v92: 97 pgs: 97 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 50 KiB/s wr, 29 op/s 2024-05-27T09:38:13.559 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=29 2024-05-27T09:38:13.559 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:13.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:13 smithi189 ceph-mon[36671]: pgmap v92: 97 pgs: 97 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 50 KiB/s wr, 29 op/s 2024-05-27T09:38:13.706 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:13.707 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:13.711 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:13.712 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:13.713 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:13.723 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:13 smithi044 ceph-mon[31618]: pgmap v92: 97 pgs: 97 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 50 KiB/s wr, 29 op/s 2024-05-27T09:38:15.280 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:15.284 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:15.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:15 smithi044 ceph-mon[31618]: pgmap v93: 97 pgs: 97 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 50 KiB/s wr, 29 op/s 2024-05-27T09:38:15.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:15 smithi050 ceph-mon[36371]: pgmap v93: 97 pgs: 97 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 50 KiB/s wr, 29 op/s 2024-05-27T09:38:15.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:15 smithi189 ceph-mon[36671]: pgmap v93: 97 pgs: 97 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 50 KiB/s wr, 29 op/s 2024-05-27T09:38:15.721 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:15.721 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:15.741 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:15.741 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:15.741 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 29 -lt 100 ']' 2024-05-27T09:38:15.742 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 29 + 1 2024-05-27T09:38:15.792 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=30 2024-05-27T09:38:15.792 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:15.858 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:15.859 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:15.864 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:15.865 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:15.866 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:16.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:16 smithi050 ceph-mon[36371]: pgmap v94: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-05-27T09:38:16.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:16 smithi189 ceph-mon[36671]: pgmap v94: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-05-27T09:38:16.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:16 smithi044 ceph-mon[31618]: pgmap v94: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-05-27T09:38:17.688 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:17.689 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:18.003 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:18.003 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:18.025 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:18.025 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:18.025 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 30 -lt 100 ']' 2024-05-27T09:38:18.027 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 30 + 1 2024-05-27T09:38:18.075 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=31 2024-05-27T09:38:18.076 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:18.150 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:18.151 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:18.156 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:18.157 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:18.158 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:19.213 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:19.213 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:19.453 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:19.453 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:19.467 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:19.467 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:19.467 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 31 -lt 100 ']' 2024-05-27T09:38:19.469 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 31 + 1 2024-05-27T09:38:19.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:19 smithi044 ceph-mon[31618]: pgmap v95: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 KiB/s wr, 13 op/s 2024-05-27T09:38:19.509 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=32 2024-05-27T09:38:19.509 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:19.543 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:19.543 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:19.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:19 smithi050 ceph-mon[36371]: pgmap v95: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 KiB/s wr, 13 op/s 2024-05-27T09:38:19.548 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:19.549 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:19.550 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:19.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:19 smithi189 ceph-mon[36671]: pgmap v95: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 KiB/s wr, 13 op/s 2024-05-27T09:38:20.348 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:20.349 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:20.653 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:20.653 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:20.666 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:20.666 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:20.666 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 32 -lt 100 ']' 2024-05-27T09:38:20.667 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 32 + 1 2024-05-27T09:38:20.716 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=33 2024-05-27T09:38:20.716 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:20.738 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:20.738 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:20.742 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:20.742 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:20.743 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:21.367 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:38:21.398 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:38:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:21 smithi044 ceph-mon[31618]: pgmap v96: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 17 op/s 2024-05-27T09:38:21.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:21 smithi050 ceph-mon[36371]: pgmap v96: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 17 op/s 2024-05-27T09:38:21.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:21 smithi189 ceph-mon[36671]: pgmap v96: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 17 op/s 2024-05-27T09:38:21.729 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:21.730 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:22.311 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:22.311 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:22.325 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:22.325 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:22.325 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 33 -lt 100 ']' 2024-05-27T09:38:22.326 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 33 + 1 2024-05-27T09:38:22.531 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=34 2024-05-27T09:38:22.531 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:22.644 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:22.645 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:22.649 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:22.651 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:22.652 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:23.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:23 smithi044 ceph-mon[31618]: pgmap v97: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-05-27T09:38:23.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:23 smithi050 ceph-mon[36371]: pgmap v97: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-05-27T09:38:23.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:23 smithi189 ceph-mon[36671]: pgmap v97: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-05-27T09:38:23.662 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:23.663 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:24.069 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:24.069 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:24.116 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:24.116 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:24.117 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 34 -lt 100 ']' 2024-05-27T09:38:24.118 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 34 + 1 2024-05-27T09:38:24.239 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=35 2024-05-27T09:38:24.240 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:24.357 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:24.358 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:24.362 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:24.363 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:24.364 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.429+0000 7fc253e00640 1 -- 172.21.15.44:0/2519862941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c109c80 msgr2=0x7fc24c103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.429+0000 7fc253e00640 1 --2- 172.21.15.44:0/2519862941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c109c80 0x7fc24c103820 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc240009860 tx=0x7fc24002a340 comp rx=0 tx=0).stop 2024-05-27T09:38:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.430+0000 7fc253e00640 1 -- 172.21.15.44:0/2519862941 shutdown_connections 2024-05-27T09:38:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.430+0000 7fc253e00640 1 --2- 172.21.15.44:0/2519862941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc24c1043d0 0x7fc24c1107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.430+0000 7fc253e00640 1 --2- 172.21.15.44:0/2519862941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c10a120 0x7fc24c103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.430+0000 7fc253e00640 1 --2- 172.21.15.44:0/2519862941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c109c80 0x7fc24c103820 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.430+0000 7fc253e00640 1 -- 172.21.15.44:0/2519862941 >> 172.21.15.44:0/2519862941 conn(0x7fc24c0ff340 msgr2=0x7fc24c101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.431+0000 7fc253e00640 1 -- 172.21.15.44:0/2519862941 shutdown_connections 2024-05-27T09:38:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.431+0000 7fc253e00640 1 -- 172.21.15.44:0/2519862941 wait complete. 2024-05-27T09:38:25.446 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:25.448 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:25.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:25 smithi044 ceph-mon[31618]: pgmap v98: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-05-27T09:38:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.502+0000 7fc253e00640 1 Processor -- start 2024-05-27T09:38:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.502+0000 7fc253e00640 1 -- start start 2024-05-27T09:38:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.503+0000 7fc253e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c1043d0 0x7fc24c19d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.504+0000 7fc253e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc24c109c80 0x7fc24c19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.504+0000 7fc253e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c10a120 0x7fc24c1a87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.504+0000 7fc253e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c113e70 con 0x7fc24c109c80 2024-05-27T09:38:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.504+0000 7fc253e00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c113cf0 con 0x7fc24c10a120 2024-05-27T09:38:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.504+0000 7fc253e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c113ff0 con 0x7fc24c1043d0 2024-05-27T09:38:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.504+0000 7fc251400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c1043d0 0x7fc24c19d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.504+0000 7fc251400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c1043d0 0x7fc24c19d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60374/0 (socket says 172.21.15.44:60374) 2024-05-27T09:38:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.504+0000 7fc251400640 1 -- 172.21.15.44:0/188010422 learned_addr learned my addr 172.21.15.44:0/188010422 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.504+0000 7fc250a00640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc24c109c80 0x7fc24c19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.504+0000 7fc251e00640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c10a120 0x7fc24c1a87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.505+0000 7fc251e00640 1 -- 172.21.15.44:0/188010422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c1043d0 msgr2=0x7fc24c19d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.505+0000 7fc251e00640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c1043d0 0x7fc24c19d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.505+0000 7fc251e00640 1 -- 172.21.15.44:0/188010422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc24c109c80 msgr2=0x7fc24c19de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.505+0000 7fc251e00640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc24c109c80 0x7fc24c19de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.505+0000 7fc251e00640 1 -- 172.21.15.44:0/188010422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c1a8d70 con 0x7fc24c10a120 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.505+0000 7fc251e00640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c10a120 0x7fc24c1a87b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc24400ec50 tx=0x7fc24400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.506+0000 7fc24a000640 1 -- 172.21.15.44:0/188010422 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244003b70 con 0x7fc24c10a120 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.506+0000 7fc24a000640 1 -- 172.21.15.44:0/188010422 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc244003d10 con 0x7fc24c10a120 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.506+0000 7fc24a000640 1 -- 172.21.15.44:0/188010422 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2440107f0 con 0x7fc24c10a120 2024-05-27T09:38:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.506+0000 7fc253e00640 1 -- 172.21.15.44:0/188010422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24c1a9060 con 0x7fc24c10a120 2024-05-27T09:38:25.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.506+0000 7fc253e00640 1 -- 172.21.15.44:0/188010422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24c1a9560 con 0x7fc24c10a120 2024-05-27T09:38:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.512+0000 7fc24a000640 1 -- 172.21.15.44:0/188010422 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc244003fc0 con 0x7fc24c10a120 2024-05-27T09:38:25.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.514+0000 7fc24a000640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2340b2aa0 0x7fc2340b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.514+0000 7fc24a000640 1 -- 172.21.15.44:0/188010422 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fc2440e2a60 con 0x7fc24c10a120 2024-05-27T09:38:25.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.515+0000 7fc251400640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2340b2aa0 0x7fc2340b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.516+0000 7fc251400640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2340b2aa0 0x7fc2340b4fb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc2400057c0 tx=0x7fc2400058b0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:25.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:25 smithi050 ceph-mon[36371]: pgmap v98: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-05-27T09:38:25.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:25 smithi189 ceph-mon[36671]: pgmap v98: 97 pgs: 97 active+clean; 1.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-05-27T09:38:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.763+0000 7fc25ad6d740 1 Processor -- start 2024-05-27T09:38:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.764+0000 7fc25ad6d740 1 -- start start 2024-05-27T09:38:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.765+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a1100860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.765+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d352f0 0x5625a1100da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.765+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 0x5625a11012e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.765+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5625a1071b90 con 0x5625a0d3d4d0 2024-05-27T09:38:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.765+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5625a1061730 con 0x5625a0d352f0 2024-05-27T09:38:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.765+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5625a1065770 con 0x5625a0d5e1b0 2024-05-27T09:38:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc251400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a1100860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc251400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a1100860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60378/0 (socket says 172.21.15.44:60378) 2024-05-27T09:38:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc251400640 1 -- 172.21.15.44:0/457890774 learned_addr learned my addr 172.21.15.44:0/457890774 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc250a00640 1 --2- 172.21.15.44:0/457890774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d352f0 0x5625a1100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc251e00640 1 --2- 172.21.15.44:0/457890774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 0x5625a11012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc251400640 1 -- 172.21.15.44:0/457890774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d352f0 msgr2=0x5625a1100da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc251400640 1 --2- 172.21.15.44:0/457890774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d352f0 0x5625a1100da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc251400640 1 -- 172.21.15.44:0/457890774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 msgr2=0x5625a11012e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:25.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc251400640 1 --2- 172.21.15.44:0/457890774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 0x5625a11012e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc251400640 1 -- 172.21.15.44:0/457890774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625a0ffa8b0 con 0x5625a0d5e1b0 2024-05-27T09:38:25.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc250a00640 1 --2- 172.21.15.44:0/457890774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d352f0 0x5625a1100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:38:25.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.766+0000 7fc251e00640 1 --2- 172.21.15.44:0/457890774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 0x5625a11012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:38:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.767+0000 7fc251400640 1 --2- 172.21.15.44:0/457890774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a1100860 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc240009780 tx=0x7fc24003c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81e9226f5a0511e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.767+0000 7fc23e600640 1 -- 172.21.15.44:0/457890774 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240037040 con 0x5625a0d5e1b0 2024-05-27T09:38:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.768+0000 7fc23e600640 1 -- 172.21.15.44:0/457890774 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc24004d070 con 0x5625a0d5e1b0 2024-05-27T09:38:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.768+0000 7fc23e600640 1 -- 172.21.15.44:0/457890774 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2400486c0 con 0x5625a0d5e1b0 2024-05-27T09:38:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.770+0000 7fc25ad6d740 1 -- 172.21.15.44:0/457890774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 msgr2=0x5625a1100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.770+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/457890774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a1100860 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc240009780 tx=0x7fc24003c000 comp rx=0 tx=0).stop 2024-05-27T09:38:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.770+0000 7fc25ad6d740 1 -- 172.21.15.44:0/457890774 shutdown_connections 2024-05-27T09:38:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.770+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/457890774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 0x5625a11012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.770+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/457890774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d352f0 0x5625a1100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.770+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/457890774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a1100860 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.770+0000 7fc25ad6d740 1 -- 172.21.15.44:0/457890774 >> 172.21.15.44:0/457890774 conn(0x5625a0d52300 msgr2=0x5625a10fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.770+0000 7fc25ad6d740 1 -- 172.21.15.44:0/457890774 shutdown_connections 2024-05-27T09:38:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.770+0000 7fc25ad6d740 1 -- 172.21.15.44:0/457890774 wait complete. 2024-05-27T09:38:25.819 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:25.819 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:25.848 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:25.848 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:25.848 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 35 -lt 100 ']' 2024-05-27T09:38:25.849 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 35 + 1 2024-05-27T09:38:25.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.853+0000 7fc25ad6d740 1 -- start start 2024-05-27T09:38:25.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.855+0000 7fc25ad6d740 1 Processor -- start 2024-05-27T09:38:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.856+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d352f0 0x5625a110aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.856+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d3d4d0 0x5625a110af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.856+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d5e1b0 0x5625a1116c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.856+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5625a0d47790 con 0x5625a0d5e1b0 2024-05-27T09:38:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5625a1009bc0 con 0x5625a0d3d4d0 2024-05-27T09:38:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5625a110d930 con 0x5625a0d352f0 2024-05-27T09:38:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc251400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d352f0 0x5625a110aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc251400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d352f0 0x5625a110aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60390/0 (socket says 172.21.15.44:60390) 2024-05-27T09:38:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc251400640 1 -- 172.21.15.44:0/3515204753 learned_addr learned my addr 172.21.15.44:0/3515204753 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc250a00640 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d3d4d0 0x5625a110af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc251e00640 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d5e1b0 0x5625a1116c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc251400640 1 -- 172.21.15.44:0/3515204753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d3d4d0 msgr2=0x5625a110af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc251400640 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d3d4d0 0x5625a110af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc251400640 1 -- 172.21.15.44:0/3515204753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d5e1b0 msgr2=0x5625a1116c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc251400640 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d5e1b0 0x5625a1116c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.857+0000 7fc251400640 1 -- 172.21.15.44:0/3515204753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625a0d76210 con 0x5625a0d352f0 2024-05-27T09:38:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.858+0000 7fc25ad6d740 10 client.0 _ll_register_callbacks cb 0x5625a0fe7400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:38:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.858+0000 7fc25ad6d740 1 -- 172.21.15.44:0/3515204753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc24004d070 con 0x5625a0d352f0 2024-05-27T09:38:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.858+0000 7fc251400640 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d352f0 0x5625a110aa50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc240009250 tx=0x7fc240009550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6995eef43edced1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.858+0000 7fc232a00640 10 client.25837 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:38:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.859+0000 7fc232a00640 1 -- 172.21.15.44:0/3515204753 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240062050 con 0x5625a0d352f0 2024-05-27T09:38:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.859+0000 7fc232a00640 1 -- 172.21.15.44:0/3515204753 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fc240030650 con 0x5625a0d352f0 2024-05-27T09:38:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.859+0000 7fc232a00640 1 -- 172.21.15.44:0/3515204753 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc240058070 con 0x5625a0d352f0 2024-05-27T09:38:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.859+0000 7fc25ad6d740 10 client.25837 fetch_fsmap learned FSMap version 26 2024-05-27T09:38:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.859+0000 7fc25ad6d740 1 -- 172.21.15.44:0/3515204753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=26}) -- 0x5625a0e42620 con 0x5625a0d352f0 2024-05-27T09:38:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.860+0000 7fc232a00640 1 -- 172.21.15.44:0/3515204753 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc240065020 con 0x5625a0d352f0 2024-05-27T09:38:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.860+0000 7fc25ad6d740 10 client.25837 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:38:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.860+0000 7fc25ad6d740 10 client.25837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:38:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.861+0000 7fc25ad6d740 20 client.25837 populate_metadata read hostname 'smithi044' 2024-05-27T09:38:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.861+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a0d1cd80 0x5625a111c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.861+0000 7fc25ad6d740 4 client.25837 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:38:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.861+0000 7fc25ad6d740 1 -- 172.21.15.44:0/3515204753 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625a1073ae0 con 0x5625a0d1cd80 2024-05-27T09:38:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.861+0000 7fc250a00640 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a0d1cd80 0x5625a111c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.862+0000 7fc250a00640 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a0d1cd80 0x5625a111c910 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11838513dd1e0766 server_cookie=157053aa2d48c66e in_seq=0 out_seq=0 2024-05-27T09:38:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.862+0000 7fc232a00640 10 client.25837 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.863+0000 7fc232a00640 1 -- 172.21.15.44:0/3515204753 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5625a1073ae0 con 0x5625a0d1cd80 2024-05-27T09:38:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.863+0000 7fc232a00640 10 client.25837 handle_command_reply: tid=0 2024-05-27T09:38:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.863+0000 7fc25ad6d740 1 client.25837 shutdown 2024-05-27T09:38:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.864+0000 7fc25ad6d740 1 -- 172.21.15.44:0/3515204753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d352f0 msgr2=0x5625a110aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.864+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d352f0 0x5625a110aa50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc240009250 tx=0x7fc240009550 comp rx=0 tx=0).stop 2024-05-27T09:38:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.865+0000 7fc25ad6d740 1 -- 172.21.15.44:0/3515204753 shutdown_connections 2024-05-27T09:38:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.865+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a0d1cd80 0x5625a111c910 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.865+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d5e1b0 0x5625a1116c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.865+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d3d4d0 0x5625a110af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.865+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/3515204753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d352f0 0x5625a110aa50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.865+0000 7fc232a00640 0 client.25837 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.865+0000 7fc25ad6d740 1 -- 172.21.15.44:0/3515204753 >> 172.21.15.44:0/3515204753 conn(0x5625a0d52300 msgr2=0x5625a1100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.865+0000 7fc25ad6d740 1 -- 172.21.15.44:0/3515204753 shutdown_connections 2024-05-27T09:38:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.865+0000 7fc25ad6d740 1 -- 172.21.15.44:0/3515204753 wait complete. 2024-05-27T09:38:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.865+0000 7fc25ad6d740 20 client.25837 trim_cache size 0 max 16384 2024-05-27T09:38:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.916+0000 7fc25ad6d740 1 Processor -- start 2024-05-27T09:38:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.916+0000 7fc25ad6d740 1 -- start start 2024-05-27T09:38:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.917+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d352f0 0x5625a110d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.917+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d3d4d0 0x5625a10fc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.917+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d5e1b0 0x5625a1116bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.917+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1b44d0 con 0x5625a0d352f0 2024-05-27T09:38:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.917+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1b1da0 con 0x5625a0d5e1b0 2024-05-27T09:38:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.917+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1afbf0 con 0x5625a0d3d4d0 2024-05-27T09:38:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.917+0000 7fc251400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d352f0 0x5625a110d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.917+0000 7fc250a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d3d4d0 0x5625a10fc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.917+0000 7fc251e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d5e1b0 0x5625a1116bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.917+0000 7fc251400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d352f0 0x5625a110d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48620/0 (socket says 172.21.15.44:48620) 2024-05-27T09:38:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.917+0000 7fc251400640 1 -- 172.21.15.44:0/1514536957 learned_addr learned my addr 172.21.15.44:0/1514536957 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.918+0000 7fc251e00640 1 -- 172.21.15.44:0/1514536957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d3d4d0 msgr2=0x5625a10fc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.918+0000 7fc251e00640 1 --2- 172.21.15.44:0/1514536957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d3d4d0 0x5625a10fc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.918+0000 7fc251e00640 1 -- 172.21.15.44:0/1514536957 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d352f0 msgr2=0x5625a110d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.918+0000 7fc251e00640 1 --2- 172.21.15.44:0/1514536957 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d352f0 0x5625a110d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.918+0000 7fc251e00640 1 -- 172.21.15.44:0/1514536957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625a1104c80 con 0x5625a0d5e1b0 2024-05-27T09:38:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.918+0000 7fc251e00640 1 --2- 172.21.15.44:0/1514536957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d5e1b0 0x5625a1116bd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc2440e0180 tx=0x7fc2440a9770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=402a3d1e297d4d16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.919+0000 7fc23dc00640 1 -- 172.21.15.44:0/1514536957 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2440c4400 con 0x5625a0d5e1b0 2024-05-27T09:38:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.919+0000 7fc23dc00640 1 -- 172.21.15.44:0/1514536957 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2440a9820 con 0x5625a0d5e1b0 2024-05-27T09:38:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.919+0000 7fc23dc00640 1 -- 172.21.15.44:0/1514536957 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2440a5fa0 con 0x5625a0d5e1b0 2024-05-27T09:38:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.920+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1514536957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d5e1b0 msgr2=0x5625a1116bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.920+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/1514536957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d5e1b0 0x5625a1116bd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc2440e0180 tx=0x7fc2440a9770 comp rx=0 tx=0).stop 2024-05-27T09:38:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.920+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1514536957 shutdown_connections 2024-05-27T09:38:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.920+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/1514536957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d5e1b0 0x5625a1116bd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.920+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/1514536957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d3d4d0 0x5625a10fc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.920+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/1514536957 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d352f0 0x5625a110d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.920+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1514536957 >> 172.21.15.44:0/1514536957 conn(0x5625a0d52300 msgr2=0x5625a110ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.920+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1514536957 shutdown_connections 2024-05-27T09:38:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.920+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1514536957 wait complete. 2024-05-27T09:38:25.922 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.921+0000 7fc25ad6d740 1 -- start start 2024-05-27T09:38:25.922 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.921+0000 7fc25ad6d740 1 Processor -- start 2024-05-27T09:38:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d352f0 0x5625a111f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 0x5625a111ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc25ad6d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a111ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1b0260 con 0x5625a0d3d4d0 2024-05-27T09:38:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1b16d0 con 0x5625a0d352f0 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc25ad6d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1b0930 con 0x5625a0d5e1b0 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc25ad6d740 10 client.0 _ll_register_callbacks cb 0x7fc238009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc250a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 0x5625a111ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc251e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a111ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc250a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 0x5625a111ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48630/0 (socket says 172.21.15.44:48630) 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc250a00640 1 -- 172.21.15.44:0/1796666969 learned_addr learned my addr 172.21.15.44:0/1796666969 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc251e00640 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a111ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60416/0 (socket says 172.21.15.44:60416) 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.922+0000 7fc251400640 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d352f0 0x5625a111f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.923+0000 7fc251e00640 1 -- 172.21.15.44:0/1796666969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d352f0 msgr2=0x5625a111f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.923+0000 7fc251e00640 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d352f0 0x5625a111f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.923+0000 7fc251e00640 1 -- 172.21.15.44:0/1796666969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 msgr2=0x5625a111ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.923+0000 7fc251e00640 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 0x5625a111ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.923+0000 7fc251e00640 1 -- 172.21.15.44:0/1796666969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625a111d550 con 0x5625a0d5e1b0 2024-05-27T09:38:25.925 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=36 2024-05-27T09:38:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.923+0000 7fc250a00640 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 0x5625a111ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:38:25.925 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.923+0000 7fc251e00640 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a111ce60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc244005cc0 tx=0x7fc2440c4400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95d9ae0134f97630 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.923+0000 7fc232a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:38:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.924+0000 7fc232a00640 1 -- 172.21.15.44:0/1796666969 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2440b7060 con 0x5625a0d5e1b0 2024-05-27T09:38:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.924+0000 7fc232a00640 1 -- 172.21.15.44:0/1796666969 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fc2440a6300 con 0x5625a0d5e1b0 2024-05-27T09:38:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.924+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1796666969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2440a9820 con 0x5625a0d5e1b0 2024-05-27T09:38:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.924+0000 7fc232a00640 1 -- 172.21.15.44:0/1796666969 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc2440aa410 con 0x5625a0d5e1b0 2024-05-27T09:38:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.924+0000 7fc25ad6d740 10 client.25843 fetch_fsmap learned FSMap version 26 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.924+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1796666969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5625a1184690 con 0x5625a0d5e1b0 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.924+0000 7fc232a00640 1 -- 172.21.15.44:0/1796666969 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2440a9820 con 0x5625a0d5e1b0 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.924+0000 7fc232a00640 1 -- 172.21.15.44:0/1796666969 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc2440a8ae0 con 0x5625a0d5e1b0 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.925+0000 7fc25ad6d740 10 client.25843 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.925+0000 7fc25ad6d740 10 client.25843 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.925+0000 7fc25ad6d740 20 client.25843 populate_metadata read hostname 'smithi044' 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.925+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a0cf21c0 0x5625a1186d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.925+0000 7fc25ad6d740 4 client.25843 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.925+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1796666969 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5625a106d2c0 con 0x5625a0cf21c0 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.925+0000 7fc251400640 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a0cf21c0 0x5625a1186d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.926+0000 7fc232a00640 10 client.25843 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.926+0000 7fc251400640 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a0cf21c0 0x5625a1186d40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a75520aaee5b4f9 server_cookie=23ddd23d60b4a4b8 in_seq=0 out_seq=0 2024-05-27T09:38:25.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc232a00640 1 -- 172.21.15.44:0/1796666969 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5625a106d2c0 con 0x5625a0cf21c0 2024-05-27T09:38:25.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc232a00640 10 client.25843 handle_command_reply: tid=0 2024-05-27T09:38:25.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc25ad6d740 1 client.25843 shutdown 2024-05-27T09:38:25.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1796666969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 msgr2=0x5625a111ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:25.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a111ce60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc244005cc0 tx=0x7fc2440c4400 comp rx=0 tx=0).stop 2024-05-27T09:38:25.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1796666969 shutdown_connections 2024-05-27T09:38:25.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a0cf21c0 0x5625a1186d40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a0d5e1b0 0x5625a111ce60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a0d3d4d0 0x5625a111ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc25ad6d740 1 --2- 172.21.15.44:0/1796666969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a0d352f0 0x5625a111f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc232a00640 0 client.25843 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:25.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.945+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1796666969 >> 172.21.15.44:0/1796666969 conn(0x5625a0d52300 msgr2=0x5625a1105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:25.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.946+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1796666969 shutdown_connections 2024-05-27T09:38:25.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.946+0000 7fc25ad6d740 1 -- 172.21.15.44:0/1796666969 wait complete. 2024-05-27T09:38:25.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.946+0000 7fc25ad6d740 20 client.25843 trim_cache size 0 max 16384 2024-05-27T09:38:25.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:38:25.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:38:25.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:38:25.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 -- 172.21.15.44:0/188010422 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2340b2aa0 msgr2=0x7fc2340b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:25.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2340b2aa0 0x7fc2340b4fb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc2400057c0 tx=0x7fc2400058b0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 -- 172.21.15.44:0/188010422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c10a120 msgr2=0x7fc24c1a87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c10a120 0x7fc24c1a87b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc24400ec50 tx=0x7fc24400c6a0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 -- 172.21.15.44:0/188010422 shutdown_connections 2024-05-27T09:38:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c10a120 0x7fc24c1a87b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2340b2aa0 0x7fc2340b4fb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc24c109c80 0x7fc24c19de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 --2- 172.21.15.44:0/188010422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c1043d0 0x7fc24c19d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 -- 172.21.15.44:0/188010422 >> 172.21.15.44:0/188010422 conn(0x7fc24c0ff340 msgr2=0x7fc24c076000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 -- 172.21.15.44:0/188010422 shutdown_connections 2024-05-27T09:38:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:25.947+0000 7fc253e00640 1 -- 172.21.15.44:0/188010422 wait complete. 2024-05-27T09:38:25.972 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:25.972 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:25.977 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:25.978 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:25.978 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:26.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:26 smithi050 ceph-mon[36371]: pgmap v99: 97 pgs: 97 active+clean; 2.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-05-27T09:38:26.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:26 smithi189 ceph-mon[36671]: pgmap v99: 97 pgs: 97 active+clean; 2.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-05-27T09:38:26.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:26 smithi044 ceph-mon[31618]: pgmap v99: 97 pgs: 97 active+clean; 2.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-05-27T09:38:27.196 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:27.196 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:27.427 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:27.427 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:27.473 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:27.473 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:27.473 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 36 -lt 100 ']' 2024-05-27T09:38:27.474 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 36 + 1 2024-05-27T09:38:27.557 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=37 2024-05-27T09:38:27.557 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:27.597 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:27.599 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:27.603 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:27.605 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:27.606 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:29.028 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:29.029 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:29.269 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:29.269 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:29.325 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:29.325 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:29.325 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 37 -lt 100 ']' 2024-05-27T09:38:29.326 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 37 + 1 2024-05-27T09:38:29.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:29 smithi044 ceph-mon[31618]: pgmap v100: 97 pgs: 97 active+clean; 2.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-27T09:38:29.496 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=38 2024-05-27T09:38:29.496 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:29.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:29 smithi050 ceph-mon[36371]: pgmap v100: 97 pgs: 97 active+clean; 2.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-27T09:38:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:29 smithi189 ceph-mon[36671]: pgmap v100: 97 pgs: 97 active+clean; 2.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-27T09:38:29.617 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:29.617 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:29.622 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:29.624 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:29.624 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:31.212 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:31.212 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:31.468 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:31.468 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:31.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:31 smithi044 ceph-mon[31618]: pgmap v101: 97 pgs: 97 active+clean; 2.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 15 op/s 2024-05-27T09:38:31.497 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:31.498 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:31.498 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 38 -lt 100 ']' 2024-05-27T09:38:31.499 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 38 + 1 2024-05-27T09:38:31.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:31 smithi050 ceph-mon[36371]: pgmap v101: 97 pgs: 97 active+clean; 2.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 15 op/s 2024-05-27T09:38:31.561 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=39 2024-05-27T09:38:31.561 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:31.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:31 smithi189 ceph-mon[36671]: pgmap v101: 97 pgs: 97 active+clean; 2.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 15 op/s 2024-05-27T09:38:31.603 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:31.604 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:31.609 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:31.610 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:31.611 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:32.646 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:32.646 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:33.085 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:33.085 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:33.151 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:33.151 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:33.151 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 39 -lt 100 ']' 2024-05-27T09:38:33.153 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 39 + 1 2024-05-27T09:38:33.320 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=40 2024-05-27T09:38:33.321 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:33.420 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:33.421 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:33.425 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:33.426 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:33.427 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:33.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:33 smithi044 ceph-mon[31618]: pgmap v102: 97 pgs: 97 active+clean; 2.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-05-27T09:38:33.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:33 smithi050 ceph-mon[36371]: pgmap v102: 97 pgs: 97 active+clean; 2.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-05-27T09:38:33.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:33 smithi189 ceph-mon[36671]: pgmap v102: 97 pgs: 97 active+clean; 2.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-05-27T09:38:34.490 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:34 smithi044 ceph-mon[31618]: pgmap v103: 97 pgs: 97 active+clean; 2.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-05-27T09:38:34.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:34 smithi050 ceph-mon[36371]: pgmap v103: 97 pgs: 97 active+clean; 2.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-05-27T09:38:34.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:34 smithi189 ceph-mon[36671]: pgmap v103: 97 pgs: 97 active+clean; 2.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-05-27T09:38:34.853 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:34.854 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:35.435 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:35.435 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:35.473 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:35.474 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:35.474 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 40 -lt 100 ']' 2024-05-27T09:38:35.475 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 40 + 1 2024-05-27T09:38:35.668 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=41 2024-05-27T09:38:35.668 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:35.815 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:35.815 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:35.820 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:35.821 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:35.821 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:36.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:36 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:38:36.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:36 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:38:36.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:36 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:38:37.344 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:37.344 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:37.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:37 smithi044 ceph-mon[31618]: pgmap v104: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 15 op/s 2024-05-27T09:38:37.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:37 smithi050 ceph-mon[36371]: pgmap v104: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 15 op/s 2024-05-27T09:38:37.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:37 smithi189 ceph-mon[36671]: pgmap v104: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 15 op/s 2024-05-27T09:38:37.793 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:37.793 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:37.850 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:37.850 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:37.850 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 41 -lt 100 ']' 2024-05-27T09:38:37.851 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 41 + 1 2024-05-27T09:38:38.029 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=42 2024-05-27T09:38:38.030 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:38.165 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:38.165 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:38.169 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:38.170 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:38.171 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:39.269 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:39.270 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:39.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:39 smithi044 ceph-mon[31618]: pgmap v105: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 11 op/s 2024-05-27T09:38:39.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:39 smithi050 ceph-mon[36371]: pgmap v105: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 11 op/s 2024-05-27T09:38:39.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:39 smithi189 ceph-mon[36671]: pgmap v105: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 11 op/s 2024-05-27T09:38:39.767 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:39.767 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:39.781 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:39.781 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:39.781 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 42 -lt 100 ']' 2024-05-27T09:38:39.782 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 42 + 1 2024-05-27T09:38:39.922 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=43 2024-05-27T09:38:39.923 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:39.968 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:39.968 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:39.972 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:39.973 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:39.974 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:41.394 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:41.395 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:41.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:41 smithi044 ceph-mon[31618]: pgmap v106: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 KiB/s wr, 15 op/s 2024-05-27T09:38:41.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:41 smithi050 ceph-mon[36371]: pgmap v106: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 KiB/s wr, 15 op/s 2024-05-27T09:38:41.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:41 smithi189 ceph-mon[36671]: pgmap v106: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 KiB/s wr, 15 op/s 2024-05-27T09:38:41.717 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:41.717 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:41.771 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:41.771 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:41.771 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 43 -lt 100 ']' 2024-05-27T09:38:41.771 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 43 + 1 2024-05-27T09:38:41.805 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=44 2024-05-27T09:38:41.805 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:41.880 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:41.881 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:41.884 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:41.885 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:41.886 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:43.160 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:43.160 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:43.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:43 smithi044 ceph-mon[31618]: pgmap v107: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-05-27T09:38:43.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:43 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:38:43.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:43 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:38:43.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:43 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:38:43.454 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:43.454 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:43.514 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:43.514 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:43.514 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 44 -lt 100 ']' 2024-05-27T09:38:43.515 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 44 + 1 2024-05-27T09:38:43.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:43 smithi050 ceph-mon[36371]: pgmap v107: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-05-27T09:38:43.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:43 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:38:43.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:43 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:38:43.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:43 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:38:43.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:43 smithi189 ceph-mon[36671]: pgmap v107: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-05-27T09:38:43.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:43 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:38:43.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:43 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:38:43.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:43 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:38:43.655 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=45 2024-05-27T09:38:43.655 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:43.735 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:43.735 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:43.739 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:43.740 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:43.741 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:44.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.146+0000 7f60f6000700 1 -- 172.21.15.44:0/3245208871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60f01027d0 msgr2=0x7f60f0102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.146+0000 7f60f6000700 1 --2- 172.21.15.44:0/3245208871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60f01027d0 0x7f60f0102be0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f60dc009990 tx=0x7f60dc009ca0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.146+0000 7f60f6000700 1 -- 172.21.15.44:0/3245208871 shutdown_connections 2024-05-27T09:38:44.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.146+0000 7f60f6000700 1 --2- 172.21.15.44:0/3245208871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60f0103120 0x7f60f0111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.146+0000 7f60f6000700 1 --2- 172.21.15.44:0/3245208871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f010a3d0 0x7f60f010a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.147+0000 7f60f6000700 1 --2- 172.21.15.44:0/3245208871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60f01027d0 0x7f60f0102be0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.147+0000 7f60f6000700 1 -- 172.21.15.44:0/3245208871 >> 172.21.15.44:0/3245208871 conn(0x7f60f00fe6e0 msgr2=0x7f60f0100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.147+0000 7f60f6000700 1 -- 172.21.15.44:0/3245208871 shutdown_connections 2024-05-27T09:38:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.147+0000 7f60f6000700 1 -- 172.21.15.44:0/3245208871 wait complete. 2024-05-27T09:38:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.146+0000 7f3431600700 1 -- 172.21.15.44:0/2256675120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c106eb0 msgr2=0x7f342c113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.146+0000 7f3431600700 1 --2- 172.21.15.44:0/2256675120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c106eb0 0x7f342c113340 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3424009f90 tx=0x7f342400b330 comp rx=0 tx=0).stop 2024-05-27T09:38:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.147+0000 7f3431600700 1 -- 172.21.15.44:0/2256675120 shutdown_connections 2024-05-27T09:38:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.147+0000 7f3431600700 1 --2- 172.21.15.44:0/2256675120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c106eb0 0x7f342c113340 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.147+0000 7f3431600700 1 --2- 172.21.15.44:0/2256675120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c10c5a0 0x7f342c10c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.147+0000 7f3431600700 1 --2- 172.21.15.44:0/2256675120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c106560 0x7f342c106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.147+0000 7f3431600700 1 -- 172.21.15.44:0/2256675120 >> 172.21.15.44:0/2256675120 conn(0x7f342c102040 msgr2=0x7f342c104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.147+0000 7f3431600700 1 -- 172.21.15.44:0/2256675120 shutdown_connections 2024-05-27T09:38:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.147+0000 7f3431600700 1 -- 172.21.15.44:0/2256675120 wait complete. 2024-05-27T09:38:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.148+0000 7f3431600700 1 Processor -- start 2024-05-27T09:38:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.149+0000 7f3431600700 1 -- start start 2024-05-27T09:38:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.149+0000 7f60f6000700 1 Processor -- start 2024-05-27T09:38:44.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.149+0000 7f3431600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c106560 0x7f342c19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.149+0000 7f3431600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c106eb0 0x7f342c19d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.149+0000 7f3431600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c10c5a0 0x7f342c1a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.149+0000 7f3431600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f342c116720 con 0x7f342c106560 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.149+0000 7f3431600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1165a0 con 0x7f342c106eb0 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.149+0000 7f3431600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1168a0 con 0x7f342c10c5a0 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f3422000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c106eb0 0x7f342c19d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f342aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c106560 0x7f342c19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f342aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c106560 0x7f342c19cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55140/0 (socket says 172.21.15.44:55140) 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f3422000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c106eb0 0x7f342c19d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34316/0 (socket says 172.21.15.44:34316) 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f342aa00700 1 -- 172.21.15.44:0/1344119961 learned_addr learned my addr 172.21.15.44:0/1344119961 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f3422000700 1 -- 172.21.15.44:0/1344119961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c10c5a0 msgr2=0x7f342c1a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f3422000700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c10c5a0 0x7f342c1a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f3422000700 1 -- 172.21.15.44:0/1344119961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c106560 msgr2=0x7f342c19cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f3422000700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c106560 0x7f342c19cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f3422000700 1 -- 172.21.15.44:0/1344119961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c1a1ec0 con 0x7f342c106eb0 2024-05-27T09:38:44.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.151+0000 7f3422000700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c106eb0 0x7f342c19d590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f342c075b70 tx=0x7f341400eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.149+0000 7f60f6000700 1 -- start start 2024-05-27T09:38:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f60f6000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60f01027d0 0x7f60f010b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f60f6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f0103120 0x7f60f010b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f60f6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60f010a3d0 0x7f60f010bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f60f6000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f60f0114450 con 0x7f60f0103120 2024-05-27T09:38:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.150+0000 7f60f6000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f60f01142d0 con 0x7f60f01027d0 2024-05-27T09:38:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.151+0000 7f60f6000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f60f01145d0 con 0x7f60f010a3d0 2024-05-27T09:38:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.151+0000 7f60efe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60f010a3d0 0x7f60f010bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.151+0000 7f60eea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f0103120 0x7f60f010b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.151+0000 7f60efe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60f010a3d0 0x7f60f010bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50942/0 (socket says 172.21.15.44:50942) 2024-05-27T09:38:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.151+0000 7f60efe00700 1 -- 172.21.15.44:0/3258360717 learned_addr learned my addr 172.21.15.44:0/3258360717 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.151+0000 7f60ef400700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60f01027d0 0x7f60f010b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.151+0000 7f3423e00700 1 -- 172.21.15.44:0/1344119961 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341400c990 con 0x7f342c106eb0 2024-05-27T09:38:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.151+0000 7f3423e00700 1 -- 172.21.15.44:0/1344119961 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3414010430 con 0x7f342c106eb0 2024-05-27T09:38:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.151+0000 7f3423e00700 1 -- 172.21.15.44:0/1344119961 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414018680 con 0x7f342c106eb0 2024-05-27T09:38:44.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.151+0000 7f3431600700 1 -- 172.21.15.44:0/1344119961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f342c1a21b0 con 0x7f342c106eb0 2024-05-27T09:38:44.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.152+0000 7f60efe00700 1 -- 172.21.15.44:0/3258360717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60f01027d0 msgr2=0x7f60f010b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.152+0000 7f60efe00700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60f01027d0 0x7f60f010b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.152+0000 7f60efe00700 1 -- 172.21.15.44:0/3258360717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f0103120 msgr2=0x7f60f010b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.152+0000 7f60efe00700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f0103120 0x7f60f010b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.152+0000 7f60efe00700 1 -- 172.21.15.44:0/3258360717 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f019cea0 con 0x7f60f010a3d0 2024-05-27T09:38:44.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.152+0000 7f3431600700 1 -- 172.21.15.44:0/1344119961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f342c1a9ad0 con 0x7f342c106eb0 2024-05-27T09:38:44.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.152+0000 7f60efe00700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60f010a3d0 0x7f60f010bc10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f60e800ef30 tx=0x7f60e800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:44.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.153+0000 7f60e7e00700 1 -- 172.21.15.44:0/3258360717 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e80092e0 con 0x7f60f010a3d0 2024-05-27T09:38:44.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.153+0000 7f60e7e00700 1 -- 172.21.15.44:0/3258360717 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60e8004bb0 con 0x7f60f010a3d0 2024-05-27T09:38:44.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.153+0000 7f60f6000700 1 -- 172.21.15.44:0/3258360717 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f019d130 con 0x7f60f010a3d0 2024-05-27T09:38:44.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.153+0000 7f60f6000700 1 -- 172.21.15.44:0/3258360717 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f019d640 con 0x7f60f010a3d0 2024-05-27T09:38:44.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.153+0000 7f60e7e00700 1 -- 172.21.15.44:0/3258360717 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8004280 con 0x7f60f010a3d0 2024-05-27T09:38:44.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.156+0000 7f60f6000700 1 -- 172.21.15.44:0/3258360717 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d0005330 con 0x7f60f010a3d0 2024-05-27T09:38:44.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.158+0000 7f3423e00700 1 -- 172.21.15.44:0/1344119961 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f341400cb30 con 0x7f342c106eb0 2024-05-27T09:38:44.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.158+0000 7f60e7e00700 1 -- 172.21.15.44:0/3258360717 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f60e801b030 con 0x7f60f010a3d0 2024-05-27T09:38:44.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.160+0000 7f3423e00700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f340c0b2a70 0x7f340c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.160+0000 7f3423e00700 1 -- 172.21.15.44:0/1344119961 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f34140e27d0 con 0x7f342c106eb0 2024-05-27T09:38:44.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.160+0000 7f342aa00700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f340c0b2a70 0x7f340c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.160+0000 7f60e7e00700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60d40b2a70 0x7f60d40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.161+0000 7f60e7e00700 1 -- 172.21.15.44:0/3258360717 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f60e80e2140 con 0x7f60f010a3d0 2024-05-27T09:38:44.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.161+0000 7f60ef400700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60d40b2a70 0x7f60d40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.161+0000 7f342aa00700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f340c0b2a70 0x7f340c0b4f80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f341c005c50 tx=0x7f341c007480 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:44.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.162+0000 7f60ef400700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60d40b2a70 0x7f60d40b4f80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f60e0009bd0 tx=0x7f60e0009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:44.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.165+0000 7f60e7e00700 1 -- 172.21.15.44:0/3258360717 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f60e8022020 con 0x7f60f010a3d0 2024-05-27T09:38:44.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.368+0000 7f3440172b80 1 Processor -- start 2024-05-27T09:38:44.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.368+0000 7f3440172b80 1 -- start start 2024-05-27T09:38:44.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.369+0000 7f3440172b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4061939f0 0x55f406226330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.369+0000 7f3440172b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4060a7e40 0x55f406226870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.369+0000 7f3440172b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f406233110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.369+0000 7f3440172b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f406154e40 con 0x55f4060a7e40 2024-05-27T09:38:44.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.369+0000 7f3440172b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f4061449e0 con 0x55f406226e70 2024-05-27T09:38:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.369+0000 7f3440172b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f406148a20 con 0x55f4061939f0 2024-05-27T09:38:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.369+0000 7f342aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4061939f0 0x55f406226330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.369+0000 7f3422000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4060a7e40 0x55f406226870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.369+0000 7f342aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4061939f0 0x55f406226330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50944/0 (socket says 172.21.15.44:50944) 2024-05-27T09:38:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.370+0000 7f342aa00700 1 -- 172.21.15.44:0/3353083795 learned_addr learned my addr 172.21.15.44:0/3353083795 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:44.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.370+0000 7f342b400700 1 --2- 172.21.15.44:0/3353083795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f406233110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.370+0000 7f342aa00700 1 -- 172.21.15.44:0/3353083795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 msgr2=0x55f406233110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.370+0000 7f342aa00700 1 --2- 172.21.15.44:0/3353083795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f406233110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.370+0000 7f342aa00700 1 -- 172.21.15.44:0/3353083795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4060a7e40 msgr2=0x55f406226870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.370+0000 7f342aa00700 1 --2- 172.21.15.44:0/3353083795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4060a7e40 0x55f406226870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.370+0000 7f342aa00700 1 -- 172.21.15.44:0/3353083795 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f40610d420 con 0x55f4061939f0 2024-05-27T09:38:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.371+0000 7f342aa00700 1 --2- 172.21.15.44:0/3353083795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4061939f0 0x55f406226330 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f341c010fa0 tx=0x7f341c013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53fc658c6eed9cdc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.371+0000 7f341be00700 1 -- 172.21.15.44:0/3353083795 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c011db0 con 0x55f4061939f0 2024-05-27T09:38:44.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.371+0000 7f341be00700 1 -- 172.21.15.44:0/3353083795 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f341c007e50 con 0x55f4061939f0 2024-05-27T09:38:44.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.371+0000 7f341be00700 1 -- 172.21.15.44:0/3353083795 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c01c740 con 0x55f4061939f0 2024-05-27T09:38:44.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.372+0000 7f3440172b80 1 -- 172.21.15.44:0/3353083795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4061939f0 msgr2=0x55f406226330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.372+0000 7f3440172b80 1 --2- 172.21.15.44:0/3353083795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4061939f0 0x55f406226330 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f341c010fa0 tx=0x7f341c013000 comp rx=0 tx=0).stop 2024-05-27T09:38:44.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.372+0000 7f3440172b80 1 -- 172.21.15.44:0/3353083795 shutdown_connections 2024-05-27T09:38:44.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.372+0000 7f3440172b80 1 --2- 172.21.15.44:0/3353083795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f406233110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.372+0000 7f3440172b80 1 --2- 172.21.15.44:0/3353083795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4061939f0 0x55f406226330 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.372+0000 7f3440172b80 1 --2- 172.21.15.44:0/3353083795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4060a7e40 0x55f406226870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.372+0000 7f3440172b80 1 -- 172.21.15.44:0/3353083795 >> 172.21.15.44:0/3353083795 conn(0x55f406141390 msgr2=0x55f4062224c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:44.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.373+0000 7f3440172b80 1 -- 172.21.15.44:0/3353083795 shutdown_connections 2024-05-27T09:38:44.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.373+0000 7f3440172b80 1 -- 172.21.15.44:0/3353083795 wait complete. 2024-05-27T09:38:44.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.373+0000 7f3440172b80 1 -- start start 2024-05-27T09:38:44.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.374+0000 7f3440172b80 1 Processor -- start 2024-05-27T09:38:44.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.374+0000 7f3440172b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406230710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.374+0000 7f3440172b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4061939f0 0x55f406230c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.374+0000 7f3440172b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f406231190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.374+0000 7f3440172b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f40604afa0 con 0x55f4061939f0 2024-05-27T09:38:44.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.374+0000 7f3440172b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f4060e5760 con 0x55f406226e70 2024-05-27T09:38:44.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.374+0000 7f3440172b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f406233b80 con 0x55f4060a7e40 2024-05-27T09:38:44.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.374+0000 7f3440172b80 10 client.0 _ll_register_callbacks cb 0x55f4060eec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:38:44.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.375+0000 7f342aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406230710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.375+0000 7f3422000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4061939f0 0x55f406230c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.375+0000 7f342aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406230710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50946/0 (socket says 172.21.15.44:50946) 2024-05-27T09:38:44.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.375+0000 7f342aa00700 1 -- 172.21.15.44:0/3628056013 learned_addr learned my addr 172.21.15.44:0/3628056013 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:44.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.375+0000 7f342b400700 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f406231190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.375+0000 7f3422000700 1 -- 172.21.15.44:0/3628056013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 msgr2=0x55f406230710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:44.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.375+0000 7f3422000700 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406230710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.375+0000 7f3422000700 1 -- 172.21.15.44:0/3628056013 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 msgr2=0x55f406231190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:44.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.375+0000 7f3422000700 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f406231190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.375+0000 7f3422000700 1 -- 172.21.15.44:0/3628056013 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4060948d0 con 0x55f4061939f0 2024-05-27T09:38:44.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.375+0000 7f342aa00700 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406230710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:38:44.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.376+0000 7f3422000700 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4061939f0 0x55f406230c50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f34140dfef0 tx=0x7f34140b2220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b24c345c88eb8bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:44.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.376+0000 7f3418c00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:38:44.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.376+0000 7f3418c00700 1 -- 172.21.15.44:0/3628056013 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34140a9590 con 0x55f4061939f0 2024-05-27T09:38:44.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.376+0000 7f3418c00700 1 -- 172.21.15.44:0/3628056013 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f34140b2660 con 0x55f4061939f0 2024-05-27T09:38:44.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.376+0000 7f3440172b80 1 -- 172.21.15.44:0/3628056013 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f341c007e50 con 0x55f4061939f0 2024-05-27T09:38:44.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.376+0000 7f3418c00700 1 -- 172.21.15.44:0/3628056013 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f341400ce20 con 0x55f4061939f0 2024-05-27T09:38:44.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.377+0000 7f3440172b80 10 client.15714 fetch_fsmap learned FSMap version 26 2024-05-27T09:38:44.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.377+0000 7f3440172b80 1 -- 172.21.15.44:0/3628056013 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f406115060 con 0x55f4061939f0 2024-05-27T09:38:44.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.377+0000 7f3418c00700 1 -- 172.21.15.44:0/3628056013 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f341c007e50 con 0x55f4061939f0 2024-05-27T09:38:44.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.377+0000 7f3418c00700 1 -- 172.21.15.44:0/3628056013 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f34140cc0e0 con 0x55f4061939f0 2024-05-27T09:38:44.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.377+0000 7f3440172b80 10 client.15714 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:38:44.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.377+0000 7f3440172b80 10 client.15714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:38:44.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.377+0000 7f3440172b80 20 client.15714 populate_metadata read hostname 'smithi044' 2024-05-27T09:38:44.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.377+0000 7f3440172b80 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f406240ad0 0x55f406242ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.377+0000 7f3440172b80 4 client.15714 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:38:44.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.377+0000 7f3440172b80 1 -- 172.21.15.44:0/3628056013 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f406156d90 con 0x55f406240ad0 2024-05-27T09:38:44.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.378+0000 7f342aa00700 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f406240ad0 0x55f406242ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.378+0000 7f342aa00700 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f406240ad0 0x55f406242ef0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=156565ec9277d376 server_cookie=d5b4b3c73e3e79de in_seq=0 out_seq=0 2024-05-27T09:38:44.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.378+0000 7f3418c00700 10 client.15714 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:44.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.379+0000 7f3418c00700 1 -- 172.21.15.44:0/3628056013 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f406156d90 con 0x55f406240ad0 2024-05-27T09:38:44.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.379+0000 7f3418c00700 10 client.15714 handle_command_reply: tid=0 2024-05-27T09:38:44.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.379+0000 7f3440172b80 1 client.15714 shutdown 2024-05-27T09:38:44.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3440172b80 1 -- 172.21.15.44:0/3628056013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4061939f0 msgr2=0x55f406230c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3440172b80 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4061939f0 0x55f406230c50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f34140dfef0 tx=0x7f34140b2220 comp rx=0 tx=0).stop 2024-05-27T09:38:44.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3440172b80 1 -- 172.21.15.44:0/3628056013 shutdown_connections 2024-05-27T09:38:44.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3440172b80 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f406240ad0 0x55f406242ef0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3440172b80 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f406231190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3440172b80 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406230710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3440172b80 1 --2- 172.21.15.44:0/3628056013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4061939f0 0x55f406230c50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3440172b80 1 -- 172.21.15.44:0/3628056013 >> 172.21.15.44:0/3628056013 conn(0x55f406141390 msgr2=0x55f406226370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:44.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3418c00700 0 client.15714 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:44.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3440172b80 1 -- 172.21.15.44:0/3628056013 shutdown_connections 2024-05-27T09:38:44.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3440172b80 1 -- 172.21.15.44:0/3628056013 wait complete. 2024-05-27T09:38:44.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.380+0000 7f3440172b80 20 client.15714 trim_cache size 0 max 16384 2024-05-27T09:38:44.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.409+0000 7f3440172b80 1 Processor -- start 2024-05-27T09:38:44.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.409+0000 7f3440172b80 1 -- start start 2024-05-27T09:38:44.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.410+0000 7f3440172b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4060a7e40 0x55f40622bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.410+0000 7f3440172b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4061939f0 0x55f406222590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.410+0000 7f3440172b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f406226e70 0x55f40623cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.410+0000 7f3440172b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1b3930 con 0x55f4060a7e40 2024-05-27T09:38:44.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.410+0000 7f3440172b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1b11c0 con 0x55f4061939f0 2024-05-27T09:38:44.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.410+0000 7f3440172b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1af010 con 0x55f406226e70 2024-05-27T09:38:44.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.410+0000 7f342aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4060a7e40 0x55f40622bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.410+0000 7f3422000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4061939f0 0x55f406222590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.410+0000 7f3422000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4061939f0 0x55f406222590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34348/0 (socket says 172.21.15.44:34348) 2024-05-27T09:38:44.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.410+0000 7f3422000700 1 -- 172.21.15.44:0/3425472308 learned_addr learned my addr 172.21.15.44:0/3425472308 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:44.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f342b400700 1 --2- 172.21.15.44:0/3425472308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f406226e70 0x55f40623cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f3422000700 1 -- 172.21.15.44:0/3425472308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f406226e70 msgr2=0x55f40623cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:44.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f3422000700 1 --2- 172.21.15.44:0/3425472308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f406226e70 0x55f40623cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f3422000700 1 -- 172.21.15.44:0/3425472308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4060a7e40 msgr2=0x55f40622bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:44.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f3422000700 1 --2- 172.21.15.44:0/3425472308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4060a7e40 0x55f40622bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f3422000700 1 -- 172.21.15.44:0/3425472308 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f40621e240 con 0x55f4061939f0 2024-05-27T09:38:44.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f342b400700 1 --2- 172.21.15.44:0/3425472308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f406226e70 0x55f40623cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:38:44.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f342aa00700 1 --2- 172.21.15.44:0/3425472308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4060a7e40 0x55f40622bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:38:44.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f3422000700 1 --2- 172.21.15.44:0/3425472308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4061939f0 0x55f406222590 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f34140a9a10 tx=0x7f34140a5a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=417bb4c53b4459b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:44.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f341b400700 1 -- 172.21.15.44:0/3425472308 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34140a9590 con 0x55f4061939f0 2024-05-27T09:38:44.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f341b400700 1 -- 172.21.15.44:0/3425472308 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34140aa180 con 0x55f4061939f0 2024-05-27T09:38:44.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.411+0000 7f341b400700 1 -- 172.21.15.44:0/3425472308 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34140cc1a0 con 0x55f4061939f0 2024-05-27T09:38:44.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.413+0000 7f3440172b80 1 -- 172.21.15.44:0/3425472308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4061939f0 msgr2=0x55f406222590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.413+0000 7f3440172b80 1 --2- 172.21.15.44:0/3425472308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4061939f0 0x55f406222590 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f34140a9a10 tx=0x7f34140a5a70 comp rx=0 tx=0).stop 2024-05-27T09:38:44.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.413+0000 7f3440172b80 1 -- 172.21.15.44:0/3425472308 shutdown_connections 2024-05-27T09:38:44.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.413+0000 7f3440172b80 1 --2- 172.21.15.44:0/3425472308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f406226e70 0x55f40623cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.413+0000 7f3440172b80 1 --2- 172.21.15.44:0/3425472308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4060a7e40 0x55f40622bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.413+0000 7f3440172b80 1 --2- 172.21.15.44:0/3425472308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4061939f0 0x55f406222590 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.413+0000 7f3440172b80 1 -- 172.21.15.44:0/3425472308 >> 172.21.15.44:0/3425472308 conn(0x55f406141390 msgr2=0x55f406221e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:44.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.413+0000 7f3440172b80 1 -- 172.21.15.44:0/3425472308 shutdown_connections 2024-05-27T09:38:44.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.413+0000 7f3440172b80 1 -- 172.21.15.44:0/3425472308 wait complete. 2024-05-27T09:38:44.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.414+0000 7f3440172b80 1 -- start start 2024-05-27T09:38:44.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.414+0000 7f3440172b80 1 Processor -- start 2024-05-27T09:38:44.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f3440172b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406233f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f3440172b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4061939f0 0x55f406234450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f3440172b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f4062a9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f3440172b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1af680 con 0x55f4061939f0 2024-05-27T09:38:44.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f3440172b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1b0af0 con 0x55f406226e70 2024-05-27T09:38:44.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f3440172b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1afd50 con 0x55f4060a7e40 2024-05-27T09:38:44.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f3440172b80 10 client.0 _ll_register_callbacks cb 0x55f406221220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:38:44.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f342aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406233f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f3422000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4061939f0 0x55f406234450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f342aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406233f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50964/0 (socket says 172.21.15.44:50964) 2024-05-27T09:38:44.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f342aa00700 1 -- 172.21.15.44:0/1336645930 learned_addr learned my addr 172.21.15.44:0/1336645930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:44.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f342b400700 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f4062a9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.415+0000 7f342aa00700 1 -- 172.21.15.44:0/1336645930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 msgr2=0x55f4062a9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:44.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.416+0000 7f342aa00700 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f4062a9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.416+0000 7f342aa00700 1 -- 172.21.15.44:0/1336645930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4061939f0 msgr2=0x55f406234450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:44.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.416+0000 7f342aa00700 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4061939f0 0x55f406234450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.416+0000 7f342aa00700 1 -- 172.21.15.44:0/1336645930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f406234d30 con 0x55f4060a7e40 2024-05-27T09:38:44.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.416+0000 7f342aa00700 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406233f10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f341c011010 tx=0x7f341c007f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3bff6e310d63bc6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:44.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.416+0000 7f3418c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.416+0000 7f3418c00700 1 -- 172.21.15.44:0/1336645930 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c005250 con 0x55f4060a7e40 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.416+0000 7f3418c00700 1 -- 172.21.15.44:0/1336645930 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f341c0053f0 con 0x55f4060a7e40 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.416+0000 7f3440172b80 1 -- 172.21.15.44:0/1336645930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34140aa180 con 0x55f4060a7e40 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.417+0000 7f3418c00700 1 -- 172.21.15.44:0/1336645930 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f341c0055f0 con 0x55f4060a7e40 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.417+0000 7f3440172b80 10 client.25861 fetch_fsmap learned FSMap version 26 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.417+0000 7f3440172b80 1 -- 172.21.15.44:0/1336645930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f4062aaa30 con 0x55f4060a7e40 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.417+0000 7f3418c00700 1 -- 172.21.15.44:0/1336645930 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34140aa180 con 0x55f4060a7e40 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.417+0000 7f3418c00700 1 -- 172.21.15.44:0/1336645930 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f341c01e6f0 con 0x55f4060a7e40 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.417+0000 7f3440172b80 10 client.25861 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.417+0000 7f3440172b80 10 client.25861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.417+0000 7f3440172b80 20 client.25861 populate_metadata read hostname 'smithi044' 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.417+0000 7f3440172b80 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4062aaf70 0x55f4062ad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:44.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.417+0000 7f3440172b80 4 client.25861 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:38:44.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.417+0000 7f3440172b80 1 -- 172.21.15.44:0/1336645930 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f406150570 con 0x55f4062aaf70 2024-05-27T09:38:44.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.418+0000 7f3422000700 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4062aaf70 0x55f4062ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:44.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.418+0000 7f3418c00700 10 client.25861 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:44.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.418+0000 7f3422000700 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4062aaf70 0x55f4062ad390 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9742bc68e011a2 server_cookie=67a37e1e1ecd8139 in_seq=0 out_seq=0 2024-05-27T09:38:44.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3418c00700 1 -- 172.21.15.44:0/1336645930 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f406150570 con 0x55f4062aaf70 2024-05-27T09:38:44.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3418c00700 10 client.25861 handle_command_reply: tid=0 2024-05-27T09:38:44.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3440172b80 1 client.25861 shutdown 2024-05-27T09:38:44.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3440172b80 1 -- 172.21.15.44:0/1336645930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 msgr2=0x55f406233f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3440172b80 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406233f10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f341c011010 tx=0x7f341c007f50 comp rx=0 tx=0).stop 2024-05-27T09:38:44.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3440172b80 1 -- 172.21.15.44:0/1336645930 shutdown_connections 2024-05-27T09:38:44.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3440172b80 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4062aaf70 0x55f4062ad390 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3440172b80 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f406226e70 0x55f4062a9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3440172b80 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4060a7e40 0x55f406233f10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3418c00700 0 client.25861 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:44.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3440172b80 1 --2- 172.21.15.44:0/1336645930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4061939f0 0x55f406234450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3440172b80 1 -- 172.21.15.44:0/1336645930 >> 172.21.15.44:0/1336645930 conn(0x55f406141390 msgr2=0x55f4062304f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:44.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3440172b80 1 -- 172.21.15.44:0/1336645930 shutdown_connections 2024-05-27T09:38:44.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.419+0000 7f3440172b80 1 -- 172.21.15.44:0/1336645930 wait complete. 2024-05-27T09:38:44.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.420+0000 7f3440172b80 20 client.25861 trim_cache size 0 max 16384 2024-05-27T09:38:44.423 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:38:44.424 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:38:44.424 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:38:44.424 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:38:44.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.420+0000 7f3431600700 1 -- 172.21.15.44:0/1344119961 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f340c0b2a70 msgr2=0x7f340c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.421+0000 7f3431600700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f340c0b2a70 0x7f340c0b4f80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f341c005c50 tx=0x7f341c007480 comp rx=0 tx=0).stop 2024-05-27T09:38:44.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.421+0000 7f3431600700 1 -- 172.21.15.44:0/1344119961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c106eb0 msgr2=0x7f342c19d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.421+0000 7f3431600700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c106eb0 0x7f342c19d590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f342c075b70 tx=0x7f341400eca0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.421+0000 7f3431600700 1 -- 172.21.15.44:0/1344119961 shutdown_connections 2024-05-27T09:38:44.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.421+0000 7f3431600700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f340c0b2a70 0x7f340c0b4f80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.421+0000 7f3431600700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342c10c5a0 0x7f342c1a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.421+0000 7f3431600700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f342c106560 0x7f342c19cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.421+0000 7f3431600700 1 --2- 172.21.15.44:0/1344119961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f342c106eb0 0x7f342c19d590 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.421+0000 7f3431600700 1 -- 172.21.15.44:0/1344119961 >> 172.21.15.44:0/1344119961 conn(0x7f342c102040 msgr2=0x7f342c10d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:44.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.421+0000 7f3431600700 1 -- 172.21.15.44:0/1344119961 shutdown_connections 2024-05-27T09:38:44.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.421+0000 7f3431600700 1 -- 172.21.15.44:0/1344119961 wait complete. 2024-05-27T09:38:44.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.495+0000 7f60f6000700 1 -- 172.21.15.44:0/3258360717 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60d00058f0 con 0x7f60f010a3d0 2024-05-27T09:38:44.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.496+0000 7f60e7e00700 1 -- 172.21.15.44:0/3258360717 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7f60e8010040 con 0x7f60f010a3d0 2024-05-27T09:38:44.499 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:38:44.499 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:38:44.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.499+0000 7f60f6000700 1 -- 172.21.15.44:0/3258360717 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60d40b2a70 msgr2=0x7f60d40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.499+0000 7f60f6000700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60d40b2a70 0x7f60d40b4f80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f60e0009bd0 tx=0x7f60e0009260 comp rx=0 tx=0).stop 2024-05-27T09:38:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.499+0000 7f60f6000700 1 -- 172.21.15.44:0/3258360717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60f010a3d0 msgr2=0x7f60f010bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.499+0000 7f60f6000700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60f010a3d0 0x7f60f010bc10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f60e800ef30 tx=0x7f60e800c3a0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.500+0000 7f60f6000700 1 -- 172.21.15.44:0/3258360717 shutdown_connections 2024-05-27T09:38:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.500+0000 7f60f6000700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60d40b2a70 0x7f60d40b4f80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.500+0000 7f60f6000700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60f010a3d0 0x7f60f010bc10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.500+0000 7f60f6000700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60f01027d0 0x7f60f010b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.500+0000 7f60f6000700 1 --2- 172.21.15.44:0/3258360717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f0103120 0x7f60f010b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.500+0000 7f60f6000700 1 -- 172.21.15.44:0/3258360717 >> 172.21.15.44:0/3258360717 conn(0x7f60f00fe6e0 msgr2=0x7f60f01038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.500+0000 7f60f6000700 1 -- 172.21.15.44:0/3258360717 shutdown_connections 2024-05-27T09:38:44.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:38:44.500+0000 7f60f6000700 1 -- 172.21.15.44:0/3258360717 wait complete. 2024-05-27T09:38:44.503 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:38:44.835 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:44.835 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:45.408 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:45.408 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:45.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:45 smithi044 ceph-mon[31618]: pgmap v108: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-05-27T09:38:45.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:45 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3258360717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:38:45.507 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:45.507 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:45.507 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 45 -lt 100 ']' 2024-05-27T09:38:45.509 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 45 + 1 2024-05-27T09:38:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:45 smithi050 ceph-mon[36371]: pgmap v108: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-05-27T09:38:45.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:45 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3258360717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:38:45.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:45 smithi189 ceph-mon[36671]: pgmap v108: 97 pgs: 97 active+clean; 2.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-05-27T09:38:45.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:45 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3258360717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:38:45.669 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=46 2024-05-27T09:38:45.669 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:45.800 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:45.801 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:45.806 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:45.808 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:45.808 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:46.959 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:46.959 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:47.375 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:47.375 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:47.417 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:47.417 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:47.417 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 46 -lt 100 ']' 2024-05-27T09:38:47.419 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 46 + 1 2024-05-27T09:38:47.572 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=47 2024-05-27T09:38:47.572 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:47.665 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:47.666 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.667+0000 7f1e21000640 1 -- 172.21.15.44:0/2679279913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10bec0 msgr2=0x7f1e1c105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.667+0000 7f1e21000640 1 --2- 172.21.15.44:0/2679279913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10bec0 0x7f1e1c105a60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009be0 tx=0x7f1e0c02a6d0 comp rx=0 tx=0).stop 2024-05-27T09:38:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.667+0000 7f1e21000640 1 -- 172.21.15.44:0/2679279913 shutdown_connections 2024-05-27T09:38:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.667+0000 7f1e21000640 1 --2- 172.21.15.44:0/2679279913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106610 0x7f1e1c1129e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.667+0000 7f1e21000640 1 --2- 172.21.15.44:0/2679279913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c10c360 0x7f1e1c105fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.667+0000 7f1e21000640 1 --2- 172.21.15.44:0/2679279913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10bec0 0x7f1e1c105a60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:47.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.667+0000 7f1e21000640 1 -- 172.21.15.44:0/2679279913 >> 172.21.15.44:0/2679279913 conn(0x7f1e1c1016b0 msgr2=0x7f1e1c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:47.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.667+0000 7f1e21000640 1 -- 172.21.15.44:0/2679279913 shutdown_connections 2024-05-27T09:38:47.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.667+0000 7f1e21000640 1 -- 172.21.15.44:0/2679279913 wait complete. 2024-05-27T09:38:47.670 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:47.671 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:47.672 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:47.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:47 smithi044 ceph-mon[31618]: pgmap v109: 97 pgs: 97 active+clean; 2.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-05-27T09:38:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.738+0000 7f1e21000640 1 Processor -- start 2024-05-27T09:38:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.738+0000 7f1e21000640 1 -- start start 2024-05-27T09:38:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.738+0000 7f1e21000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106610 0x7f1e1c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.738+0000 7f1e21000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c10bec0 0x7f1e1c19d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.738+0000 7f1e21000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10c360 0x7f1e1c1a1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.738+0000 7f1e21000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c1160a0 con 0x7f1e1c10c360 2024-05-27T09:38:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.738+0000 7f1e21000640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c115f20 con 0x7f1e1c106610 2024-05-27T09:38:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.738+0000 7f1e21000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c116220 con 0x7f1e1c10bec0 2024-05-27T09:38:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.739+0000 7f1e19400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c10bec0 0x7f1e1c19d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.739+0000 7f1e19e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106610 0x7f1e1c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.739+0000 7f1e1a800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10c360 0x7f1e1c1a1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.739+0000 7f1e19e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106610 0x7f1e1c19cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34384/0 (socket says 172.21.15.44:34384) 2024-05-27T09:38:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.739+0000 7f1e19400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c10bec0 0x7f1e1c19d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50990/0 (socket says 172.21.15.44:50990) 2024-05-27T09:38:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.739+0000 7f1e19e00640 1 -- 172.21.15.44:0/3915636732 learned_addr learned my addr 172.21.15.44:0/3915636732 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.740+0000 7f1e19400640 1 -- 172.21.15.44:0/3915636732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106610 msgr2=0x7f1e1c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.740+0000 7f1e19400640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106610 0x7f1e1c19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.740+0000 7f1e19400640 1 -- 172.21.15.44:0/3915636732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10c360 msgr2=0x7f1e1c1a1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.740+0000 7f1e19400640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10c360 0x7f1e1c1a1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.740+0000 7f1e19400640 1 -- 172.21.15.44:0/3915636732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1c1a1a40 con 0x7f1e1c10bec0 2024-05-27T09:38:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.740+0000 7f1e19400640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c10bec0 0x7f1e1c19d140 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c10c790 tx=0x7f1e0800ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:47.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.764+0000 7f1e12a00640 1 -- 172.21.15.44:0/3915636732 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e08004440 con 0x7f1e1c10bec0 2024-05-27T09:38:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.764+0000 7f1e12a00640 1 -- 172.21.15.44:0/3915636732 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e0800cd30 con 0x7f1e1c10bec0 2024-05-27T09:38:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.764+0000 7f1e12a00640 1 -- 172.21.15.44:0/3915636732 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e08019480 con 0x7f1e1c10bec0 2024-05-27T09:38:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.764+0000 7f1e21000640 1 -- 172.21.15.44:0/3915636732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e1c1a1d30 con 0x7f1e1c10bec0 2024-05-27T09:38:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.764+0000 7f1e21000640 1 -- 172.21.15.44:0/3915636732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e1c1a2230 con 0x7f1e1c10bec0 2024-05-27T09:38:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.802+0000 7f1e12a00640 1 -- 172.21.15.44:0/3915636732 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f1e08003680 con 0x7f1e1c10bec0 2024-05-27T09:38:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.805+0000 7f1e12a00640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b2aa0 0x7f1e000b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.806+0000 7f1e12a00640 1 -- 172.21.15.44:0/3915636732 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1e080e2830 con 0x7f1e1c10bec0 2024-05-27T09:38:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.806+0000 7f1e19e00640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b2aa0 0x7f1e000b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:47.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:47.808+0000 7f1e19e00640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b2aa0 0x7f1e000b4fb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c006010 tx=0x7f1e0c009280 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:48.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:47 smithi050 ceph-mon[36371]: pgmap v109: 97 pgs: 97 active+clean; 2.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-05-27T09:38:48.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:47 smithi189 ceph-mon[36671]: pgmap v109: 97 pgs: 97 active+clean; 2.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-05-27T09:38:48.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.145+0000 7f1e28008740 1 Processor -- start 2024-05-27T09:38:48.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.146+0000 7f1e28008740 1 -- start start 2024-05-27T09:38:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.147+0000 7f1e28008740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.147+0000 7f1e28008740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0868252f0 0x55f086bf0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.147+0000 7f1e28008740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f08682d4d0 0x55f086bf12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.147+0000 7f1e28008740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f086b61b90 con 0x55f0868252f0 2024-05-27T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.147+0000 7f1e28008740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f086b51730 con 0x55f08684e1b0 2024-05-27T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.147+0000 7f1e28008740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f086b55770 con 0x55f08682d4d0 2024-05-27T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.147+0000 7f1e19e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.148+0000 7f1e19e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34398/0 (socket says 172.21.15.44:34398) 2024-05-27T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.148+0000 7f1e19e00640 1 -- 172.21.15.44:0/2494165591 learned_addr learned my addr 172.21.15.44:0/2494165591 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.148+0000 7f1e19400640 1 --2- 172.21.15.44:0/2494165591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0868252f0 0x55f086bf0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.148+0000 7f1e1a800640 1 --2- 172.21.15.44:0/2494165591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f08682d4d0 0x55f086bf12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.148+0000 7f1e1a800640 1 -- 172.21.15.44:0/2494165591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 msgr2=0x55f086bf0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.148+0000 7f1e1a800640 1 --2- 172.21.15.44:0/2494165591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.148+0000 7f1e1a800640 1 -- 172.21.15.44:0/2494165591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0868252f0 msgr2=0x55f086bf0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.148+0000 7f1e1a800640 1 --2- 172.21.15.44:0/2494165591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0868252f0 0x55f086bf0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.148+0000 7f1e1a800640 1 -- 172.21.15.44:0/2494165591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f086aea8b0 con 0x55f08682d4d0 2024-05-27T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.148+0000 7f1e19e00640 1 --2- 172.21.15.44:0/2494165591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.149+0000 7f1e1a800640 1 --2- 172.21.15.44:0/2494165591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f08682d4d0 0x55f086bf12e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009b90 tx=0x7f1e14011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4002eaacfeef639e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.149+0000 7f1e07400640 1 -- 172.21.15.44:0/2494165591 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e14004030 con 0x55f08682d4d0 2024-05-27T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.149+0000 7f1e07400640 1 -- 172.21.15.44:0/2494165591 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e140035f0 con 0x55f08682d4d0 2024-05-27T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.149+0000 7f1e07400640 1 -- 172.21.15.44:0/2494165591 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e14019710 con 0x55f08682d4d0 2024-05-27T09:38:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.151+0000 7f1e28008740 1 -- 172.21.15.44:0/2494165591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f08682d4d0 msgr2=0x55f086bf12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.151+0000 7f1e28008740 1 --2- 172.21.15.44:0/2494165591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f08682d4d0 0x55f086bf12e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009b90 tx=0x7f1e14011810 comp rx=0 tx=0).stop 2024-05-27T09:38:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.152+0000 7f1e28008740 1 -- 172.21.15.44:0/2494165591 shutdown_connections 2024-05-27T09:38:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.152+0000 7f1e28008740 1 --2- 172.21.15.44:0/2494165591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f08682d4d0 0x55f086bf12e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.152+0000 7f1e28008740 1 --2- 172.21.15.44:0/2494165591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f0868252f0 0x55f086bf0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.152+0000 7f1e28008740 1 --2- 172.21.15.44:0/2494165591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.152+0000 7f1e28008740 1 -- 172.21.15.44:0/2494165591 >> 172.21.15.44:0/2494165591 conn(0x55f086842300 msgr2=0x55f086bed010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.152+0000 7f1e28008740 1 -- 172.21.15.44:0/2494165591 shutdown_connections 2024-05-27T09:38:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.152+0000 7f1e28008740 1 -- 172.21.15.44:0/2494165591 wait complete. 2024-05-27T09:38:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.257+0000 7f1e28008740 1 -- start start 2024-05-27T09:38:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.257+0000 7f1e28008740 1 Processor -- start 2024-05-27T09:38:48.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.259+0000 7f1e28008740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 0x55f086bfa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.259+0000 7f1e28008740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086bfaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.259+0000 7f1e28008740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086c06cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.259+0000 7f1e28008740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f086837790 con 0x55f08682d4d0 2024-05-27T09:38:48.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.259+0000 7f1e28008740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f086af9bc0 con 0x55f08684e1b0 2024-05-27T09:38:48.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.259+0000 7f1e28008740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f086bfd990 con 0x55f0868252f0 2024-05-27T09:38:48.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.259+0000 7f1e19400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086bfaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e19e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 0x55f086bfa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e1a800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086c06cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e19400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086bfaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55230/0 (socket says 172.21.15.44:55230) 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e1a800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086c06cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34412/0 (socket says 172.21.15.44:34412) 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e19400640 1 -- 172.21.15.44:0/3090802885 learned_addr learned my addr 172.21.15.44:0/3090802885 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e19e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 0x55f086bfa970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51012/0 (socket says 172.21.15.44:51012) 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e19400640 1 -- 172.21.15.44:0/3090802885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 msgr2=0x55f086bfa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e19400640 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 0x55f086bfa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e19400640 1 -- 172.21.15.44:0/3090802885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 msgr2=0x55f086c06cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e19400640 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086c06cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e19400640 1 -- 172.21.15.44:0/3090802885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f086866210 con 0x55f08682d4d0 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.260+0000 7f1e1a800640 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086c06cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:38:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.261+0000 7f1e19400640 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086bfaeb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1e080a6d50 tx=0x7f1e080a53a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83ca7556b2c7d19 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.261+0000 7f1dfbe00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:38:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.261+0000 7f1dfbe00640 1 -- 172.21.15.44:0/3090802885 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e080c2030 con 0x55f08682d4d0 2024-05-27T09:38:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.261+0000 7f1dfbe00640 1 -- 172.21.15.44:0/3090802885 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1e080a60d0 con 0x55f08682d4d0 2024-05-27T09:38:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.271+0000 7f1e28008740 10 client.0 _ll_register_callbacks cb 0x55f086ad7400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:38:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.271+0000 7f1e28008740 1 -- 172.21.15.44:0/3090802885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e140035f0 con 0x55f08682d4d0 2024-05-27T09:38:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.272+0000 7f1dfbe00640 1 -- 172.21.15.44:0/3090802885 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1e080aa1e0 con 0x55f08682d4d0 2024-05-27T09:38:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.272+0000 7f1e28008740 10 client.15732 fetch_fsmap learned FSMap version 26 2024-05-27T09:38:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.272+0000 7f1e28008740 1 -- 172.21.15.44:0/3090802885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=26}) -- 0x55f086932620 con 0x55f08682d4d0 2024-05-27T09:38:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.273+0000 7f1dfbe00640 1 -- 172.21.15.44:0/3090802885 <== mon.0 v2:172.21.15.44:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1e080b2b90 con 0x55f08682d4d0 2024-05-27T09:38:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.273+0000 7f1e28008740 10 client.15732 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:38:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.288+0000 7f1e28008740 10 client.15732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:38:48.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.288+0000 7f1e28008740 20 client.15732 populate_metadata read hostname 'smithi044' 2024-05-27T09:38:48.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.288+0000 7f1e28008740 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f08680cd80 0x55f086c0caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.288+0000 7f1e28008740 4 client.15732 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:38:48.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.288+0000 7f1e28008740 1 -- 172.21.15.44:0/3090802885 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f086b63ae0 con 0x55f08680cd80 2024-05-27T09:38:48.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.289+0000 7f1e19e00640 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f08680cd80 0x55f086c0caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.290+0000 7f1e19e00640 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f08680cd80 0x55f086c0caa0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d897a468351247b server_cookie=99de95d261046fb9 in_seq=0 out_seq=0 2024-05-27T09:38:48.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.290+0000 7f1dfbe00640 10 client.15732 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:48.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.291+0000 7f1dfbe00640 1 -- 172.21.15.44:0/3090802885 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f086b63ae0 con 0x55f08680cd80 2024-05-27T09:38:48.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.291+0000 7f1dfbe00640 10 client.15732 handle_command_reply: tid=0 2024-05-27T09:38:48.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.291+0000 7f1e28008740 1 client.15732 shutdown 2024-05-27T09:38:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.293+0000 7f1e28008740 1 -- 172.21.15.44:0/3090802885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 msgr2=0x55f086bfaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.293+0000 7f1e28008740 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086bfaeb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1e080a6d50 tx=0x7f1e080a53a0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.293+0000 7f1e28008740 1 -- 172.21.15.44:0/3090802885 shutdown_connections 2024-05-27T09:38:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.293+0000 7f1e28008740 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f08680cd80 0x55f086c0caa0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.294+0000 7f1e28008740 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086c06cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.294+0000 7f1e28008740 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086bfaeb0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.294+0000 7f1dfbe00640 0 client.15732 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.294+0000 7f1e28008740 1 --2- 172.21.15.44:0/3090802885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 0x55f086bfa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.294+0000 7f1e28008740 1 -- 172.21.15.44:0/3090802885 >> 172.21.15.44:0/3090802885 conn(0x55f086842300 msgr2=0x55f086bf2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.294+0000 7f1e28008740 1 -- 172.21.15.44:0/3090802885 shutdown_connections 2024-05-27T09:38:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.294+0000 7f1e28008740 1 -- 172.21.15.44:0/3090802885 wait complete. 2024-05-27T09:38:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.294+0000 7f1e28008740 20 client.15732 trim_cache size 0 max 16384 2024-05-27T09:38:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.340+0000 7f1e28008740 1 Processor -- start 2024-05-27T09:38:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.340+0000 7f1e28008740 1 -- start start 2024-05-27T09:38:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.341+0000 7f1e28008740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 0x55f086bf6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.341+0000 7f1e28008740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086bfcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.341+0000 7f1e28008740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.341+0000 7f1e28008740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c1b4560 con 0x55f08682d4d0 2024-05-27T09:38:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.341+0000 7f1e28008740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c1b1e30 con 0x55f08684e1b0 2024-05-27T09:38:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.341+0000 7f1e28008740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c1afc80 con 0x55f0868252f0 2024-05-27T09:38:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.342+0000 7f1e1a800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.342+0000 7f1e19e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 0x55f086bf6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.342+0000 7f1e19400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086bfcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.342+0000 7f1e1a800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34424/0 (socket says 172.21.15.44:34424) 2024-05-27T09:38:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.342+0000 7f1e1a800640 1 -- 172.21.15.44:0/4226830299 learned_addr learned my addr 172.21.15.44:0/4226830299 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.342+0000 7f1e19e00640 1 -- 172.21.15.44:0/4226830299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 msgr2=0x55f086bf0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.342+0000 7f1e19e00640 1 --2- 172.21.15.44:0/4226830299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.342+0000 7f1e19e00640 1 -- 172.21.15.44:0/4226830299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 msgr2=0x55f086bfcac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.342+0000 7f1e19e00640 1 --2- 172.21.15.44:0/4226830299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086bfcac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.342+0000 7f1e19e00640 1 -- 172.21.15.44:0/4226830299 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f086bedf30 con 0x55f0868252f0 2024-05-27T09:38:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.342+0000 7f1e1a800640 1 --2- 172.21.15.44:0/4226830299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:38:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.343+0000 7f1e19e00640 1 --2- 172.21.15.44:0/4226830299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 0x55f086bf6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c004af0 tx=0x7f1e0c03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d73ba36c3be13021 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.343+0000 7f1e06a00640 1 -- 172.21.15.44:0/4226830299 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c057070 con 0x55f0868252f0 2024-05-27T09:38:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.343+0000 7f1e06a00640 1 -- 172.21.15.44:0/4226830299 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e0c03dce0 con 0x55f0868252f0 2024-05-27T09:38:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.343+0000 7f1e06a00640 1 -- 172.21.15.44:0/4226830299 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c036040 con 0x55f0868252f0 2024-05-27T09:38:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.345+0000 7f1e28008740 1 -- 172.21.15.44:0/4226830299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 msgr2=0x55f086bf6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.345+0000 7f1e28008740 1 --2- 172.21.15.44:0/4226830299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 0x55f086bf6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c004af0 tx=0x7f1e0c03d000 comp rx=0 tx=0).stop 2024-05-27T09:38:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.345+0000 7f1e28008740 1 -- 172.21.15.44:0/4226830299 shutdown_connections 2024-05-27T09:38:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.345+0000 7f1e28008740 1 --2- 172.21.15.44:0/4226830299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f08684e1b0 0x55f086bf0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.345+0000 7f1e28008740 1 --2- 172.21.15.44:0/4226830299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086bfcac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.345+0000 7f1e28008740 1 --2- 172.21.15.44:0/4226830299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0868252f0 0x55f086bf6190 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.345+0000 7f1e28008740 1 -- 172.21.15.44:0/4226830299 >> 172.21.15.44:0/4226830299 conn(0x55f086842300 msgr2=0x55f086bfa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.345+0000 7f1e28008740 1 -- 172.21.15.44:0/4226830299 shutdown_connections 2024-05-27T09:38:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.345+0000 7f1e28008740 1 -- 172.21.15.44:0/4226830299 wait complete. 2024-05-27T09:38:48.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.346+0000 7f1e28008740 1 -- start start 2024-05-27T09:38:48.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.347+0000 7f1e28008740 1 Processor -- start 2024-05-27T09:38:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.348+0000 7f1e28008740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f0868252f0 0x55f086c03340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.348+0000 7f1e28008740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086c006b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.348+0000 7f1e28008740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f08684e1b0 0x55f086c00bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.348+0000 7f1e28008740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c1b02f0 con 0x55f08682d4d0 2024-05-27T09:38:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.348+0000 7f1e28008740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c1b1760 con 0x55f0868252f0 2024-05-27T09:38:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.348+0000 7f1e28008740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c1b09c0 con 0x55f08684e1b0 2024-05-27T09:38:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.348+0000 7f1e28008740 10 client.0 _ll_register_callbacks cb 0x7f1e080a99a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:38:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.349+0000 7f1e19400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086c006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.349+0000 7f1e19e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f0868252f0 0x55f086c03340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.349+0000 7f1e19e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f0868252f0 0x55f086c03340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34436/0 (socket says 172.21.15.44:34436) 2024-05-27T09:38:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.349+0000 7f1e19e00640 1 -- 172.21.15.44:0/2742796755 learned_addr learned my addr 172.21.15.44:0/2742796755 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:38:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.349+0000 7f1e1a800640 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f08684e1b0 0x55f086c00bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.349+0000 7f1e19e00640 1 -- 172.21.15.44:0/2742796755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f08684e1b0 msgr2=0x55f086c00bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.349+0000 7f1e19e00640 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f08684e1b0 0x55f086c00bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.349+0000 7f1e19e00640 1 -- 172.21.15.44:0/2742796755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 msgr2=0x55f086c006b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:38:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.349+0000 7f1e19e00640 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086c006b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.349+0000 7f1e19e00640 1 -- 172.21.15.44:0/2742796755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f086c01330 con 0x55f0868252f0 2024-05-27T09:38:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.350+0000 7f1e19e00640 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f0868252f0 0x55f086c03340 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c03dfa0 tx=0x7f1e0c030970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d8fdb6b88dec5bb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:38:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.350+0000 7f1dfbe00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:38:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.350+0000 7f1dfbe00640 1 -- 172.21.15.44:0/2742796755 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c060060 con 0x55f0868252f0 2024-05-27T09:38:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.350+0000 7f1dfbe00640 1 -- 172.21.15.44:0/2742796755 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1e0c03d830 con 0x55f0868252f0 2024-05-27T09:38:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.350+0000 7f1e28008740 1 -- 172.21.15.44:0/2742796755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e0c03dce0 con 0x55f0868252f0 2024-05-27T09:38:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.350+0000 7f1dfbe00640 1 -- 172.21.15.44:0/2742796755 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1e0c057030 con 0x55f0868252f0 2024-05-27T09:38:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.350+0000 7f1e28008740 10 client.25865 fetch_fsmap learned FSMap version 26 2024-05-27T09:38:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.350+0000 7f1e28008740 1 -- 172.21.15.44:0/2742796755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f086c750d0 con 0x55f0868252f0 2024-05-27T09:38:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.351+0000 7f1dfbe00640 1 -- 172.21.15.44:0/2742796755 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e0c03dce0 con 0x55f0868252f0 2024-05-27T09:38:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.351+0000 7f1dfbe00640 1 -- 172.21.15.44:0/2742796755 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1e0c05b410 con 0x55f0868252f0 2024-05-27T09:38:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.351+0000 7f1e28008740 10 client.25865 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:38:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.351+0000 7f1e28008740 10 client.25865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:38:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.351+0000 7f1e28008740 20 client.25865 populate_metadata read hostname 'smithi044' 2024-05-27T09:38:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.351+0000 7f1e28008740 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f0867e21c0 0x55f086c77750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:38:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.351+0000 7f1e28008740 4 client.25865 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:38:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.351+0000 7f1e28008740 1 -- 172.21.15.44:0/2742796755 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f086b5d2c0 con 0x55f0867e21c0 2024-05-27T09:38:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.352+0000 7f1e19400640 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f0867e21c0 0x55f086c77750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:38:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.352+0000 7f1e19400640 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f0867e21c0 0x55f086c77750 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d48617af8ce511b7 server_cookie=9814140f3c142285 in_seq=0 out_seq=0 2024-05-27T09:38:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.352+0000 7f1dfbe00640 10 client.25865 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.372+0000 7f1dfbe00640 1 -- 172.21.15.44:0/2742796755 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f086b5d2c0 con 0x55f0867e21c0 2024-05-27T09:38:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.372+0000 7f1dfbe00640 10 client.25865 handle_command_reply: tid=0 2024-05-27T09:38:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.372+0000 7f1e28008740 1 client.25865 shutdown 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1e28008740 1 -- 172.21.15.44:0/2742796755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f0868252f0 msgr2=0x55f086c03340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1e28008740 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f0868252f0 0x55f086c03340 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c03dfa0 tx=0x7f1e0c030970 comp rx=0 tx=0).stop 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1e28008740 1 -- 172.21.15.44:0/2742796755 shutdown_connections 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1e28008740 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f0867e21c0 0x55f086c77750 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1e28008740 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f08684e1b0 0x55f086c00bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1e28008740 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f08682d4d0 0x55f086c006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1e28008740 1 --2- 172.21.15.44:0/2742796755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f0868252f0 0x55f086c03340 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1e28008740 1 -- 172.21.15.44:0/2742796755 >> 172.21.15.44:0/2742796755 conn(0x55f086842300 msgr2=0x55f086bfc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1dfbe00640 0 client.25865 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1e28008740 1 -- 172.21.15.44:0/2742796755 shutdown_connections 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1e28008740 1 -- 172.21.15.44:0/2742796755 wait complete. 2024-05-27T09:38:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.373+0000 7f1e28008740 20 client.25865 trim_cache size 0 max 16384 2024-05-27T09:38:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:38:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:38:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:38:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 -- 172.21.15.44:0/3915636732 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b2aa0 msgr2=0x7f1e000b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b2aa0 0x7f1e000b4fb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c006010 tx=0x7f1e0c009280 comp rx=0 tx=0).stop 2024-05-27T09:38:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 -- 172.21.15.44:0/3915636732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c10bec0 msgr2=0x7f1e1c19d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:38:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c10bec0 0x7f1e1c19d140 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c10c790 tx=0x7f1e0800ed90 comp rx=0 tx=0).stop 2024-05-27T09:38:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 -- 172.21.15.44:0/3915636732 shutdown_connections 2024-05-27T09:38:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b2aa0 0x7f1e000b4fb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10c360 0x7f1e1c1a1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c10bec0 0x7f1e1c19d140 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 --2- 172.21.15.44:0/3915636732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106610 0x7f1e1c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:38:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 -- 172.21.15.44:0/3915636732 >> 172.21.15.44:0/3915636732 conn(0x7f1e1c1016b0 msgr2=0x7f1e1c101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:38:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 -- 172.21.15.44:0/3915636732 shutdown_connections 2024-05-27T09:38:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:38:48.375+0000 7f1e21000640 1 -- 172.21.15.44:0/3915636732 wait complete. 2024-05-27T09:38:48.670 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:38:48.671 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:38:49.421 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:49.422 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:49.649 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:49.649 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:49.699 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:49.699 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:49.700 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 47 -lt 100 ']' 2024-05-27T09:38:49.701 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 47 + 1 2024-05-27T09:38:49.829 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=48 2024-05-27T09:38:49.830 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:49.851 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:49.851 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:49.856 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:49.856 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:49.857 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:49.871 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-05-27T09:38:49.872 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-05-27T09:38:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:49 smithi044 ceph-mon[31618]: pgmap v110: 97 pgs: 97 active+clean; 2.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-27T09:38:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:49 smithi050 ceph-mon[36371]: pgmap v110: 97 pgs: 97 active+clean; 2.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-27T09:38:50.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:49 smithi189 ceph-mon[36671]: pgmap v110: 97 pgs: 97 active+clean; 2.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-27T09:38:50.510 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:50.510 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:50.774 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:50.774 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:50.795 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:50.795 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:50.795 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 48 -lt 100 ']' 2024-05-27T09:38:50.796 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 48 + 1 2024-05-27T09:38:50.852 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=49 2024-05-27T09:38:50.852 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:50.885 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:50.885 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:50.890 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:50.891 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:50.891 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:51.762 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:51.763 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:51.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:51 smithi044 ceph-mon[31618]: pgmap v111: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 14 op/s 2024-05-27T09:38:52.032 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:52.033 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:51 smithi050 ceph-mon[36371]: pgmap v111: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 14 op/s 2024-05-27T09:38:52.072 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:52.072 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:52.072 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 49 -lt 100 ']' 2024-05-27T09:38:52.072 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 49 + 1 2024-05-27T09:38:52.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:51 smithi189 ceph-mon[36671]: pgmap v111: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 14 op/s 2024-05-27T09:38:52.130 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=50 2024-05-27T09:38:52.130 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:52.163 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:52.164 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:52.169 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:52.171 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:52.171 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:53.292 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:53.292 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:53.699 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:53.699 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:53.730 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:53.730 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:53.730 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 50 -lt 100 ']' 2024-05-27T09:38:53.732 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 50 + 1 2024-05-27T09:38:53.837 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=51 2024-05-27T09:38:53.837 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:53.928 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:53.929 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:53.933 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:53.935 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:53.935 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:53.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:53 smithi044 ceph-mon[31618]: pgmap v112: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-05-27T09:38:54.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:53 smithi050 ceph-mon[36371]: pgmap v112: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-05-27T09:38:54.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:53 smithi189 ceph-mon[36671]: pgmap v112: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-05-27T09:38:55.275 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:55.275 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:55.615 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:55.615 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:55.629 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:55.629 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:55.629 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 51 -lt 100 ']' 2024-05-27T09:38:55.630 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 51 + 1 2024-05-27T09:38:55.704 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=52 2024-05-27T09:38:55.704 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:55.754 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:55.755 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:55.759 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:55.762 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:55.762 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:55.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:55 smithi044 ceph-mon[31618]: pgmap v113: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-05-27T09:38:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:55 smithi050 ceph-mon[36371]: pgmap v113: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-05-27T09:38:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:55 smithi189 ceph-mon[36671]: pgmap v113: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-05-27T09:38:57.058 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:57.058 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:57.223 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:57.224 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:57.247 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:57.247 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:57.247 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 52 -lt 100 ']' 2024-05-27T09:38:57.249 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 52 + 1 2024-05-27T09:38:57.308 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=53 2024-05-27T09:38:57.308 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:57.337 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:57.337 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:57.342 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:57.343 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:57.343 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:57.941 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:57 smithi044 ceph-mon[31618]: pgmap v114: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-05-27T09:38:58.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:57 smithi050 ceph-mon[36371]: pgmap v114: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-05-27T09:38:58.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:57 smithi189 ceph-mon[36671]: pgmap v114: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-05-27T09:38:58.333 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:58.333 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:58.631 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:38:58.631 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:38:58.647 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:58.647 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:38:58.647 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 53 -lt 100 ']' 2024-05-27T09:38:58.648 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 53 + 1 2024-05-27T09:38:58.727 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=54 2024-05-27T09:38:58.727 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:38:58.776 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:58.777 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:58.782 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:38:58.783 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:38:58.785 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:38:59.767 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:38:59.767 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:38:59.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:38:59 smithi044 ceph-mon[31618]: pgmap v115: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-05-27T09:39:00.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:38:59 smithi050 ceph-mon[36371]: pgmap v115: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-05-27T09:39:00.073 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:00.073 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:00.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:38:59 smithi189 ceph-mon[36671]: pgmap v115: 97 pgs: 97 active+clean; 2.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-05-27T09:39:00.089 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:00.089 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:00.089 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 54 -lt 100 ']' 2024-05-27T09:39:00.090 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 54 + 1 2024-05-27T09:39:00.178 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=55 2024-05-27T09:39:00.178 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:00.238 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:00.238 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:00.243 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:00.244 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:00.245 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:00.742 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:39:00.757 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:39:01.057 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:01.058 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:01.256 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:01.256 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:01.278 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:01.279 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:01.279 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 55 -lt 100 ']' 2024-05-27T09:39:01.280 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 55 + 1 2024-05-27T09:39:01.351 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=56 2024-05-27T09:39:01.352 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:01.394 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:01.394 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:01.398 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:01.400 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:01.400 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:01.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:01 smithi044 ceph-mon[31618]: pgmap v116: 97 pgs: 97 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 15 op/s 2024-05-27T09:39:02.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:01 smithi050 ceph-mon[36371]: pgmap v116: 97 pgs: 97 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 15 op/s 2024-05-27T09:39:02.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:01 smithi189 ceph-mon[36671]: pgmap v116: 97 pgs: 97 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 15 op/s 2024-05-27T09:39:02.999 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:02.999 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:03.372 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:03.373 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:03.388 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:03.388 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:03.388 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 56 -lt 100 ']' 2024-05-27T09:39:03.389 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 56 + 1 2024-05-27T09:39:03.556 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=57 2024-05-27T09:39:03.556 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:03.627 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:03.628 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:03.632 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:03.634 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:03.635 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:03.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:03 smithi044 ceph-mon[31618]: pgmap v117: 97 pgs: 97 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-05-27T09:39:04.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:03 smithi050 ceph-mon[36371]: pgmap v117: 97 pgs: 97 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-05-27T09:39:04.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:03 smithi189 ceph-mon[36671]: pgmap v117: 97 pgs: 97 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-05-27T09:39:05.424 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:05.425 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:05.871 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:05.871 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:05.958 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:05.958 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:05.958 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 57 -lt 100 ']' 2024-05-27T09:39:05.959 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 57 + 1 2024-05-27T09:39:05.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:05 smithi044 ceph-mon[31618]: pgmap v118: 97 pgs: 97 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-05-27T09:39:06.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:05 smithi050 ceph-mon[36371]: pgmap v118: 97 pgs: 97 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-05-27T09:39:06.070 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=58 2024-05-27T09:39:06.070 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:06.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:05 smithi189 ceph-mon[36671]: pgmap v118: 97 pgs: 97 active+clean; 2.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-05-27T09:39:06.286 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:06.286 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:06.293 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:06.293 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:06.294 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:07.508 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:07.508 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:07.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:07 smithi044 ceph-mon[31618]: pgmap v119: 97 pgs: 97 active+clean; 2.6 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 40 KiB/s wr, 25 op/s 2024-05-27T09:39:08.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:07 smithi050 ceph-mon[36371]: pgmap v119: 97 pgs: 97 active+clean; 2.6 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 40 KiB/s wr, 25 op/s 2024-05-27T09:39:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.075+0000 7f1f45000640 1 -- 172.21.15.44:0/438644510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f4010beb0 msgr2=0x7f1f40105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.075+0000 7f1f45000640 1 --2- 172.21.15.44:0/438644510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f4010beb0 0x7f1f40105a50 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009860 tx=0x7f1f3002a340 comp rx=0 tx=0).stop 2024-05-27T09:39:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.075+0000 7f1f45000640 1 -- 172.21.15.44:0/438644510 shutdown_connections 2024-05-27T09:39:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.075+0000 7f1f45000640 1 --2- 172.21.15.44:0/438644510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f40106600 0x7f1f401129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.075+0000 7f1f45000640 1 --2- 172.21.15.44:0/438644510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f4010c350 0x7f1f40105f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.075+0000 7f1f45000640 1 --2- 172.21.15.44:0/438644510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f4010beb0 0x7f1f40105a50 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.075+0000 7f1f45000640 1 -- 172.21.15.44:0/438644510 >> 172.21.15.44:0/438644510 conn(0x7f1f401016b0 msgr2=0x7f1f40103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.075+0000 7f1f45000640 1 -- 172.21.15.44:0/438644510 shutdown_connections 2024-05-27T09:39:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.075+0000 7f1f45000640 1 -- 172.21.15.44:0/438644510 wait complete. 2024-05-27T09:39:08.080 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:08.080 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:08.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:07 smithi189 ceph-mon[36671]: pgmap v119: 97 pgs: 97 active+clean; 2.6 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 40 KiB/s wr, 25 op/s 2024-05-27T09:39:08.102 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:08.102 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:08.103 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 58 -lt 100 ']' 2024-05-27T09:39:08.104 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 58 + 1 2024-05-27T09:39:08.135 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.134+0000 7f1f45000640 1 Processor -- start 2024-05-27T09:39:08.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.135+0000 7f1f45000640 1 -- start start 2024-05-27T09:39:08.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.136+0000 7f1f45000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f40106600 0x7f1f401a2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.138 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.136+0000 7f1f45000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f4010beb0 0x7f1f401a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.138 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.136+0000 7f1f45000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f4010c350 0x7f1f4019c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.138 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f45000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1f40116090 con 0x7f1f4010beb0 2024-05-27T09:39:08.138 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f45000640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1f40115f10 con 0x7f1f4010c350 2024-05-27T09:39:08.138 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f35400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f4010beb0 0x7f1f401a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.138 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f45000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1f40116210 con 0x7f1f40106600 2024-05-27T09:39:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f35400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f4010beb0 0x7f1f401a2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46104/0 (socket says 172.21.15.44:46104) 2024-05-27T09:39:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f35400640 1 -- 172.21.15.44:0/1565854814 learned_addr learned my addr 172.21.15.44:0/1565854814 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f3de00640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f40106600 0x7f1f401a2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f3e800640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f4010c350 0x7f1f4019c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f35400640 1 -- 172.21.15.44:0/1565854814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f40106600 msgr2=0x7f1f401a2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f35400640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f40106600 0x7f1f401a2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f35400640 1 -- 172.21.15.44:0/1565854814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f4010c350 msgr2=0x7f1f4019c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f35400640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f4010c350 0x7f1f4019c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.137+0000 7f1f35400640 1 -- 172.21.15.44:0/1565854814 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f4019cfd0 con 0x7f1f4010beb0 2024-05-27T09:39:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.138+0000 7f1f3de00640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f40106600 0x7f1f401a2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.138+0000 7f1f3e800640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f4010c350 0x7f1f4019c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:39:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.138+0000 7f1f35400640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f4010beb0 0x7f1f401a2c60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800ed00 tx=0x7f1f2800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.138+0000 7f1f37400640 1 -- 172.21.15.44:0/1565854814 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28010630 con 0x7f1f4010beb0 2024-05-27T09:39:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.138+0000 7f1f37400640 1 -- 172.21.15.44:0/1565854814 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f28010c80 con 0x7f1f4010beb0 2024-05-27T09:39:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.138+0000 7f1f37400640 1 -- 172.21.15.44:0/1565854814 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28018d50 con 0x7f1f4010beb0 2024-05-27T09:39:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.138+0000 7f1f45000640 1 -- 172.21.15.44:0/1565854814 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f4019d2c0 con 0x7f1f4010beb0 2024-05-27T09:39:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.139+0000 7f1f45000640 1 -- 172.21.15.44:0/1565854814 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f401a95a0 con 0x7f1f4010beb0 2024-05-27T09:39:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.147+0000 7f1f37400640 1 -- 172.21.15.44:0/1565854814 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f1f2801f020 con 0x7f1f4010beb0 2024-05-27T09:39:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.149+0000 7f1f37400640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1f240b2aa0 0x7f1f240b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.149+0000 7f1f37400640 1 -- 172.21.15.44:0/1565854814 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1f280e28e0 con 0x7f1f4010beb0 2024-05-27T09:39:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.150+0000 7f1f3de00640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1f240b2aa0 0x7f1f240b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.151+0000 7f1f3de00640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1f240b2aa0 0x7f1f240b4fb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1f300303f0 tx=0x7f1f30030340 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:08.247 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=59 2024-05-27T09:39:08.247 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:08.330 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:08.331 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:08.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:08.338 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:08.338 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.366+0000 7f1f4c167740 1 Processor -- start 2024-05-27T09:39:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.366+0000 7f1f4c167740 1 -- start start 2024-05-27T09:39:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.367+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 0x556259fec860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.368+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c212f0 0x556259fecda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.368+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c294d0 0x556259fed2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.368+0000 7f1f4c167740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556259f5db90 con 0x556259c212f0 2024-05-27T09:39:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.368+0000 7f1f4c167740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556259f4d730 con 0x556259c4a1b0 2024-05-27T09:39:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.368+0000 7f1f4c167740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556259f51770 con 0x556259c294d0 2024-05-27T09:39:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.368+0000 7f1f3de00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 0x556259fec860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.368+0000 7f1f3de00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 0x556259fec860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45826/0 (socket says 172.21.15.44:45826) 2024-05-27T09:39:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.368+0000 7f1f3de00640 1 -- 172.21.15.44:0/3518906343 learned_addr learned my addr 172.21.15.44:0/3518906343 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.368+0000 7f1f3e800640 1 --2- 172.21.15.44:0/3518906343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c294d0 0x556259fed2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.368+0000 7f1f35400640 1 --2- 172.21.15.44:0/3518906343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c212f0 0x556259fecda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.369+0000 7f1f3de00640 1 -- 172.21.15.44:0/3518906343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c294d0 msgr2=0x556259fed2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.369+0000 7f1f3de00640 1 --2- 172.21.15.44:0/3518906343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c294d0 0x556259fed2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.369+0000 7f1f3de00640 1 -- 172.21.15.44:0/3518906343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c212f0 msgr2=0x556259fecda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.369+0000 7f1f3de00640 1 --2- 172.21.15.44:0/3518906343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c212f0 0x556259fecda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.369+0000 7f1f3de00640 1 -- 172.21.15.44:0/3518906343 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556259ee68b0 con 0x556259c4a1b0 2024-05-27T09:39:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.369+0000 7f1f3e800640 1 --2- 172.21.15.44:0/3518906343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c294d0 0x556259fed2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.369+0000 7f1f3de00640 1 --2- 172.21.15.44:0/3518906343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 0x556259fec860 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1f3002a930 tx=0x7f1f3003c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80d00a57316fd1d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.369+0000 7f1f2f400640 1 -- 172.21.15.44:0/3518906343 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30042070 con 0x556259c4a1b0 2024-05-27T09:39:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.370+0000 7f1f2f400640 1 -- 172.21.15.44:0/3518906343 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f3004e070 con 0x556259c4a1b0 2024-05-27T09:39:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.370+0000 7f1f2f400640 1 -- 172.21.15.44:0/3518906343 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30037040 con 0x556259c4a1b0 2024-05-27T09:39:08.372 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.372+0000 7f1f4c167740 1 -- 172.21.15.44:0/3518906343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 msgr2=0x556259fec860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:08.372 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.372+0000 7f1f4c167740 1 --2- 172.21.15.44:0/3518906343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 0x556259fec860 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1f3002a930 tx=0x7f1f3003c000 comp rx=0 tx=0).stop 2024-05-27T09:39:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.372+0000 7f1f4c167740 1 -- 172.21.15.44:0/3518906343 shutdown_connections 2024-05-27T09:39:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.372+0000 7f1f4c167740 1 --2- 172.21.15.44:0/3518906343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c294d0 0x556259fed2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.372+0000 7f1f4c167740 1 --2- 172.21.15.44:0/3518906343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c212f0 0x556259fecda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.372+0000 7f1f4c167740 1 --2- 172.21.15.44:0/3518906343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 0x556259fec860 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.372+0000 7f1f4c167740 1 -- 172.21.15.44:0/3518906343 >> 172.21.15.44:0/3518906343 conn(0x556259c3e300 msgr2=0x556259fe9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.372+0000 7f1f4c167740 1 -- 172.21.15.44:0/3518906343 shutdown_connections 2024-05-27T09:39:08.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.372+0000 7f1f4c167740 1 -- 172.21.15.44:0/3518906343 wait complete. 2024-05-27T09:39:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.462+0000 7f1f4c167740 1 -- start start 2024-05-27T09:39:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.463+0000 7f1f4c167740 1 Processor -- start 2024-05-27T09:39:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.465+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c212f0 0x556259ff6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.465+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c294d0 0x556259ff6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.465+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 0x55625a002c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.465+0000 7f1f4c167740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556259c33790 con 0x556259c294d0 2024-05-27T09:39:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.465+0000 7f1f4c167740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556259ef5bc0 con 0x556259c4a1b0 2024-05-27T09:39:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.465+0000 7f1f4c167740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556259ff9930 con 0x556259c212f0 2024-05-27T09:39:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.466+0000 7f1f3de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c212f0 0x556259ff6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.466+0000 7f1f3de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c212f0 0x556259ff6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47356/0 (socket says 172.21.15.44:47356) 2024-05-27T09:39:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.466+0000 7f1f3de00640 1 -- 172.21.15.44:0/1594681148 learned_addr learned my addr 172.21.15.44:0/1594681148 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.466+0000 7f1f35400640 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c294d0 0x556259ff6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.466+0000 7f1f3e800640 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 0x55625a002c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.466+0000 7f1f3de00640 1 -- 172.21.15.44:0/1594681148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 msgr2=0x55625a002c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.466+0000 7f1f3de00640 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 0x55625a002c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.466+0000 7f1f3de00640 1 -- 172.21.15.44:0/1594681148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c294d0 msgr2=0x556259ff6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.466+0000 7f1f3de00640 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c294d0 0x556259ff6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.466+0000 7f1f3de00640 1 -- 172.21.15.44:0/1594681148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556259c62210 con 0x556259c212f0 2024-05-27T09:39:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.467+0000 7f1f3de00640 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c212f0 0x556259ff6a40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1f30035040 tx=0x7f1f30030830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbac8fddad834fef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.467+0000 7f1f23e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:39:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.467+0000 7f1f23e00640 1 -- 172.21.15.44:0/1594681148 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30056070 con 0x556259c212f0 2024-05-27T09:39:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.467+0000 7f1f23e00640 1 -- 172.21.15.44:0/1594681148 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1f30032cc0 con 0x556259c212f0 2024-05-27T09:39:08.489 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.488+0000 7f1f4c167740 10 client.0 _ll_register_callbacks cb 0x556259ed3400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:08.489 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.488+0000 7f1f4c167740 1 -- 172.21.15.44:0/1594681148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f3004e070 con 0x556259c212f0 2024-05-27T09:39:08.489 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.489+0000 7f1f23e00640 1 -- 172.21.15.44:0/1594681148 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1f30060050 con 0x556259c212f0 2024-05-27T09:39:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.504+0000 7f1f4c167740 10 client.25891 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.504+0000 7f1f4c167740 1 -- 172.21.15.44:0/1594681148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=26}) -- 0x556259d2e620 con 0x556259c212f0 2024-05-27T09:39:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.505+0000 7f1f23e00640 1 -- 172.21.15.44:0/1594681148 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1f300035f0 con 0x556259c212f0 2024-05-27T09:39:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.505+0000 7f1f4c167740 10 client.25891 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.520+0000 7f1f4c167740 10 client.25891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:39:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.520+0000 7f1f4c167740 20 client.25891 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.520+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x556259c08d80 0x55625a0088f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.520+0000 7f1f4c167740 4 client.25891 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:39:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.520+0000 7f1f4c167740 1 -- 172.21.15.44:0/1594681148 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556259f5fae0 con 0x556259c08d80 2024-05-27T09:39:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.521+0000 7f1f35400640 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x556259c08d80 0x55625a0088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.522+0000 7f1f35400640 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x556259c08d80 0x55625a0088f0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a523b95504326c7 server_cookie=1300113b06d68132 in_seq=0 out_seq=0 2024-05-27T09:39:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.522+0000 7f1f23e00640 10 client.25891 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.523+0000 7f1f23e00640 1 -- 172.21.15.44:0/1594681148 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556259f5fae0 con 0x556259c08d80 2024-05-27T09:39:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.523+0000 7f1f23e00640 10 client.25891 handle_command_reply: tid=0 2024-05-27T09:39:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.523+0000 7f1f4c167740 1 client.25891 shutdown 2024-05-27T09:39:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.560+0000 7f1f4c167740 1 -- 172.21.15.44:0/1594681148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c212f0 msgr2=0x556259ff6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.560+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c212f0 0x556259ff6a40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1f30035040 tx=0x7f1f30030830 comp rx=0 tx=0).stop 2024-05-27T09:39:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.560+0000 7f1f4c167740 1 -- 172.21.15.44:0/1594681148 shutdown_connections 2024-05-27T09:39:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.560+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x556259c08d80 0x55625a0088f0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.560+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c4a1b0 0x55625a002c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.560+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c294d0 0x556259ff6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.560+0000 7f1f23e00640 0 client.25891 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.560+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1594681148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c212f0 0x556259ff6a40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.560+0000 7f1f4c167740 1 -- 172.21.15.44:0/1594681148 >> 172.21.15.44:0/1594681148 conn(0x556259c3e300 msgr2=0x556259fec690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.561+0000 7f1f4c167740 1 -- 172.21.15.44:0/1594681148 shutdown_connections 2024-05-27T09:39:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.561+0000 7f1f4c167740 1 -- 172.21.15.44:0/1594681148 wait complete. 2024-05-27T09:39:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.561+0000 7f1f4c167740 20 client.25891 trim_cache size 0 max 16384 2024-05-27T09:39:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.600+0000 7f1f4c167740 1 Processor -- start 2024-05-27T09:39:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.600+0000 7f1f4c167740 1 -- start start 2024-05-27T09:39:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.601+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c212f0 0x556259ff9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.601+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c294d0 0x556259fe8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.601+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c4a1b0 0x55625a002bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.601+0000 7f1f4c167740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401b4500 con 0x556259c212f0 2024-05-27T09:39:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.601+0000 7f1f4c167740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401b1dd0 con 0x556259c294d0 2024-05-27T09:39:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.601+0000 7f1f4c167740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401afc20 con 0x556259c4a1b0 2024-05-27T09:39:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f3de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c212f0 0x556259ff9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f3e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c4a1b0 0x55625a002bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f35400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c294d0 0x556259fe8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f3e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c4a1b0 0x55625a002bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47370/0 (socket says 172.21.15.44:47370) 2024-05-27T09:39:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f3e800640 1 -- 172.21.15.44:0/737978928 learned_addr learned my addr 172.21.15.44:0/737978928 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f3e800640 1 -- 172.21.15.44:0/737978928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c294d0 msgr2=0x556259fe8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f3e800640 1 --2- 172.21.15.44:0/737978928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c294d0 0x556259fe8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f3e800640 1 -- 172.21.15.44:0/737978928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c212f0 msgr2=0x556259ff9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f3e800640 1 --2- 172.21.15.44:0/737978928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c212f0 0x556259ff9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f3e800640 1 -- 172.21.15.44:0/737978928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556259fec1f0 con 0x556259c4a1b0 2024-05-27T09:39:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f3de00640 1 --2- 172.21.15.44:0/737978928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c212f0 0x556259ff9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.602+0000 7f1f35400640 1 --2- 172.21.15.44:0/737978928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c294d0 0x556259fe8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.603+0000 7f1f3e800640 1 --2- 172.21.15.44:0/737978928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c4a1b0 0x55625a002bd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1f38009ce0 tx=0x7f1f38012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a984eeae06ad6ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.603+0000 7f1f2ea00640 1 -- 172.21.15.44:0/737978928 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f38021070 con 0x556259c4a1b0 2024-05-27T09:39:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.603+0000 7f1f2ea00640 1 -- 172.21.15.44:0/737978928 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f380118f0 con 0x556259c4a1b0 2024-05-27T09:39:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.603+0000 7f1f2ea00640 1 -- 172.21.15.44:0/737978928 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3801c8f0 con 0x556259c4a1b0 2024-05-27T09:39:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.605+0000 7f1f4c167740 1 -- 172.21.15.44:0/737978928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c4a1b0 msgr2=0x55625a002bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.605+0000 7f1f4c167740 1 --2- 172.21.15.44:0/737978928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c4a1b0 0x55625a002bd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1f38009ce0 tx=0x7f1f38012820 comp rx=0 tx=0).stop 2024-05-27T09:39:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.605+0000 7f1f4c167740 1 -- 172.21.15.44:0/737978928 shutdown_connections 2024-05-27T09:39:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.605+0000 7f1f4c167740 1 --2- 172.21.15.44:0/737978928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c4a1b0 0x55625a002bd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.605+0000 7f1f4c167740 1 --2- 172.21.15.44:0/737978928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c294d0 0x556259fe8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.605+0000 7f1f4c167740 1 --2- 172.21.15.44:0/737978928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c212f0 0x556259ff9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.605+0000 7f1f4c167740 1 -- 172.21.15.44:0/737978928 >> 172.21.15.44:0/737978928 conn(0x556259c3e300 msgr2=0x556259ff62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.605+0000 7f1f4c167740 1 -- 172.21.15.44:0/737978928 shutdown_connections 2024-05-27T09:39:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.605+0000 7f1f4c167740 1 -- 172.21.15.44:0/737978928 wait complete. 2024-05-27T09:39:08.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.606+0000 7f1f4c167740 1 -- start start 2024-05-27T09:39:08.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.606+0000 7f1f4c167740 1 Processor -- start 2024-05-27T09:39:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.607+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c212f0 0x55625a008970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.607+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c294d0 0x55625a008eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.607+0000 7f1f4c167740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c4a1b0 0x55625a009430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.607+0000 7f1f4c167740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401b0290 con 0x556259c4a1b0 2024-05-27T09:39:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.607+0000 7f1f4c167740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401b1700 con 0x556259c212f0 2024-05-27T09:39:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.607+0000 7f1f4c167740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401b0960 con 0x556259c294d0 2024-05-27T09:39:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.607+0000 7f1f4c167740 10 client.0 _ll_register_callbacks cb 0x7f1f280186f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.608+0000 7f1f3de00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c212f0 0x55625a008970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.608+0000 7f1f3de00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c212f0 0x55625a008970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45854/0 (socket says 172.21.15.44:45854) 2024-05-27T09:39:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.608+0000 7f1f3de00640 1 -- 172.21.15.44:0/1682581923 learned_addr learned my addr 172.21.15.44:0/1682581923 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.608+0000 7f1f3e800640 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c4a1b0 0x55625a009430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.608+0000 7f1f35400640 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c294d0 0x55625a008eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.608+0000 7f1f3de00640 1 -- 172.21.15.44:0/1682581923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c294d0 msgr2=0x55625a008eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.608+0000 7f1f3de00640 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c294d0 0x55625a008eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.608+0000 7f1f3de00640 1 -- 172.21.15.44:0/1682581923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c4a1b0 msgr2=0x55625a009430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.608+0000 7f1f3de00640 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c4a1b0 0x55625a009430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.608+0000 7f1f3de00640 1 -- 172.21.15.44:0/1682581923 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55625a06ef80 con 0x556259c212f0 2024-05-27T09:39:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.608+0000 7f1f3de00640 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c212f0 0x55625a008970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1f3002a860 tx=0x7f1f3003f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9f6ee017d2f530f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.609+0000 7f1f23e00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:39:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.609+0000 7f1f23e00640 1 -- 172.21.15.44:0/1682581923 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30056040 con 0x556259c212f0 2024-05-27T09:39:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.609+0000 7f1f4c167740 1 -- 172.21.15.44:0/1682581923 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f380118f0 con 0x556259c212f0 2024-05-27T09:39:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.609+0000 7f1f23e00640 1 -- 172.21.15.44:0/1682581923 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1f30032cc0 con 0x556259c212f0 2024-05-27T09:39:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.609+0000 7f1f23e00640 1 -- 172.21.15.44:0/1682581923 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1f30072050 con 0x556259c212f0 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.609+0000 7f1f4c167740 10 client.25883 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.609+0000 7f1f4c167740 1 -- 172.21.15.44:0/1682581923 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55625a070210 con 0x556259c212f0 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.610+0000 7f1f23e00640 1 -- 172.21.15.44:0/1682581923 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f380118f0 con 0x556259c212f0 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.610+0000 7f1f23e00640 1 -- 172.21.15.44:0/1682581923 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1f3005b440 con 0x556259c212f0 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.610+0000 7f1f4c167740 10 client.25883 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.610+0000 7f1f4c167740 10 client.25883 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.610+0000 7f1f4c167740 20 client.25883 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.610+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x556259bde1c0 0x55625a0728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.610+0000 7f1f4c167740 4 client.25883 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.610+0000 7f1f4c167740 1 -- 172.21.15.44:0/1682581923 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556259f592c0 con 0x556259bde1c0 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.611+0000 7f1f35400640 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x556259bde1c0 0x55625a0728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.611+0000 7f1f35400640 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x556259bde1c0 0x55625a0728b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc97ca00b042bc86 server_cookie=1686661c491a727f in_seq=0 out_seq=0 2024-05-27T09:39:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.611+0000 7f1f23e00640 10 client.25883 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.634+0000 7f1f23e00640 1 -- 172.21.15.44:0/1682581923 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x556259f592c0 con 0x556259bde1c0 2024-05-27T09:39:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.634+0000 7f1f23e00640 10 client.25883 handle_command_reply: tid=0 2024-05-27T09:39:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f4c167740 1 client.25883 shutdown 2024-05-27T09:39:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f4c167740 1 -- 172.21.15.44:0/1682581923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c212f0 msgr2=0x55625a008970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c212f0 0x55625a008970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1f3002a860 tx=0x7f1f3003f000 comp rx=0 tx=0).stop 2024-05-27T09:39:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f4c167740 1 -- 172.21.15.44:0/1682581923 shutdown_connections 2024-05-27T09:39:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x556259bde1c0 0x55625a0728b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556259c4a1b0 0x55625a009430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556259c294d0 0x55625a008eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f4c167740 1 --2- 172.21.15.44:0/1682581923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556259c212f0 0x55625a008970 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f4c167740 1 -- 172.21.15.44:0/1682581923 >> 172.21.15.44:0/1682581923 conn(0x556259c3e300 msgr2=0x556259fe7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f23e00640 0 client.25883 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f4c167740 1 -- 172.21.15.44:0/1682581923 shutdown_connections 2024-05-27T09:39:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.635+0000 7f1f4c167740 1 -- 172.21.15.44:0/1682581923 wait complete. 2024-05-27T09:39:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.636+0000 7f1f4c167740 20 client.25883 trim_cache size 0 max 16384 2024-05-27T09:39:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:39:08.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:39:08.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:39:08.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 -- 172.21.15.44:0/1565854814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1f240b2aa0 msgr2=0x7f1f240b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:08.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1f240b2aa0 0x7f1f240b4fb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1f300303f0 tx=0x7f1f30030340 comp rx=0 tx=0).stop 2024-05-27T09:39:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 -- 172.21.15.44:0/1565854814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f4010beb0 msgr2=0x7f1f401a2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f4010beb0 0x7f1f401a2c60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800ed00 tx=0x7f1f2800c6a0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 -- 172.21.15.44:0/1565854814 shutdown_connections 2024-05-27T09:39:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f4010c350 0x7f1f4019c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1f240b2aa0 0x7f1f240b4fb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f4010beb0 0x7f1f401a2c60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 --2- 172.21.15.44:0/1565854814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f40106600 0x7f1f401a2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 -- 172.21.15.44:0/1565854814 >> 172.21.15.44:0/1565854814 conn(0x7f1f401016b0 msgr2=0x7f1f40101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 -- 172.21.15.44:0/1565854814 shutdown_connections 2024-05-27T09:39:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:08.637+0000 7f1f45000640 1 -- 172.21.15.44:0/1565854814 wait complete. 2024-05-27T09:39:09.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:09 smithi044 ceph-mon[31618]: pgmap v120: 97 pgs: 97 active+clean; 2.6 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 KiB/s wr, 22 op/s 2024-05-27T09:39:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:09 smithi050 ceph-mon[36371]: pgmap v120: 97 pgs: 97 active+clean; 2.6 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 KiB/s wr, 22 op/s 2024-05-27T09:39:10.073 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:10.073 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:10.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:09 smithi189 ceph-mon[36671]: pgmap v120: 97 pgs: 97 active+clean; 2.6 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 KiB/s wr, 22 op/s 2024-05-27T09:39:10.236 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:10.236 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:10.252 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:10.252 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:10.252 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 59 -lt 100 ']' 2024-05-27T09:39:10.253 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 59 + 1 2024-05-27T09:39:10.353 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=60 2024-05-27T09:39:10.353 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:10.380 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:10.381 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:10.386 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:10.387 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:10.387 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:11.548 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:11.549 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:11.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:11 smithi044 ceph-mon[31618]: pgmap v121: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-27T09:39:12.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:11 smithi050 ceph-mon[36371]: pgmap v121: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-27T09:39:12.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:11 smithi189 ceph-mon[36671]: pgmap v121: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-05-27T09:39:12.172 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:12.173 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:12.192 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:12.192 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:12.192 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 60 -lt 100 ']' 2024-05-27T09:39:12.193 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 60 + 1 2024-05-27T09:39:12.273 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=61 2024-05-27T09:39:12.273 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:12.334 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:12.335 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:12.338 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:12.339 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:12.340 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:13.798 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:14.532 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:13 smithi189 ceph-mon[36671]: pgmap v122: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-05-27T09:39:14.532 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:13 smithi050 ceph-mon[36371]: pgmap v122: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-05-27T09:39:14.532 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:14.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:13 smithi044 ceph-mon[31618]: pgmap v122: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-05-27T09:39:14.533 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 61 -lt 100 ']' 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 61 + 1 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=62 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:14.534 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:15.506 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:15.506 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:15.929 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:15.929 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:15.946 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:15.946 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:15.946 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 62 -lt 100 ']' 2024-05-27T09:39:15.946 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 62 + 1 2024-05-27T09:39:15.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:15 smithi044 ceph-mon[31618]: pgmap v123: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-05-27T09:39:16.026 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=63 2024-05-27T09:39:16.026 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:16.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:15 smithi050 ceph-mon[36371]: pgmap v123: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-05-27T09:39:16.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:15 smithi189 ceph-mon[36671]: pgmap v123: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-05-27T09:39:16.143 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:16.143 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:16.150 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:16.150 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:16.150 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.034+0000 7f9940200700 1 -- 172.21.15.44:0/4103429290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9938105da0 msgr2=0x7f993810e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.034+0000 7f9940200700 1 --2- 172.21.15.44:0/4103429290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9938105da0 0x7f993810e2f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9924009990 tx=0x7f9924009ca0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.034+0000 7f9940200700 1 -- 172.21.15.44:0/4103429290 shutdown_connections 2024-05-27T09:39:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.034+0000 7f9940200700 1 --2- 172.21.15.44:0/4103429290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f993810e8c0 0x7f9938110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.034+0000 7f9940200700 1 --2- 172.21.15.44:0/4103429290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9938105450 0x7f9938105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.034+0000 7f9940200700 1 --2- 172.21.15.44:0/4103429290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9938105da0 0x7f993810e2f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.034+0000 7f9940200700 1 -- 172.21.15.44:0/4103429290 >> 172.21.15.44:0/4103429290 conn(0x7f99380fedb0 msgr2=0x7f99381011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:17.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.034+0000 7f9940200700 1 -- 172.21.15.44:0/4103429290 shutdown_connections 2024-05-27T09:39:17.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.035+0000 7f9940200700 1 -- 172.21.15.44:0/4103429290 wait complete. 2024-05-27T09:39:17.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.036+0000 7f9940200700 1 Processor -- start 2024-05-27T09:39:17.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.036+0000 7f9940200700 1 -- start start 2024-05-27T09:39:17.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.036+0000 7fad92c00700 1 -- 172.21.15.44:0/1971644862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8c1045e0 msgr2=0x7fad8c1049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:17.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.036+0000 7fad92c00700 1 --2- 172.21.15.44:0/1971644862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8c1045e0 0x7fad8c1049f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fad7c009990 tx=0x7fad7c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.036+0000 7fad92c00700 1 -- 172.21.15.44:0/1971644862 shutdown_connections 2024-05-27T09:39:17.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.036+0000 7fad92c00700 1 --2- 172.21.15.44:0/1971644862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c109860 0x7fad8c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.036+0000 7fad92c00700 1 --2- 172.21.15.44:0/1971644862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8c1045e0 0x7fad8c1049f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.036+0000 7fad92c00700 1 --2- 172.21.15.44:0/1971644862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad8c108f10 0x7fad8c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.036+0000 7fad92c00700 1 -- 172.21.15.44:0/1971644862 >> 172.21.15.44:0/1971644862 conn(0x7fad8c100030 msgr2=0x7fad8c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:17.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.036+0000 7f9940200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9938105450 0x7f993819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.037+0000 7f9940200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9938105da0 0x7f993819d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.037+0000 7f9940200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f993810e8c0 0x7f99381a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.037+0000 7f9940200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9938103ac0 con 0x7f9938105da0 2024-05-27T09:39:17.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.037+0000 7f9940200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9938103940 con 0x7f993810e8c0 2024-05-27T09:39:17.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.037+0000 7f9940200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9938103c40 con 0x7f9938105450 2024-05-27T09:39:17.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.037+0000 7f993d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9938105450 0x7f993819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.037+0000 7f993d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9938105450 0x7f993819cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49968/0 (socket says 172.21.15.44:49968) 2024-05-27T09:39:17.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.037+0000 7f993d800700 1 -- 172.21.15.44:0/3715455651 learned_addr learned my addr 172.21.15.44:0/3715455651 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:17.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.037+0000 7f993ce00700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9938105da0 0x7f993819d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.037+0000 7f993e200700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f993810e8c0 0x7f99381a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.038+0000 7f993d800700 1 -- 172.21.15.44:0/3715455651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f993810e8c0 msgr2=0x7f99381a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:17.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.038+0000 7f993d800700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f993810e8c0 0x7f99381a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.038+0000 7fad92c00700 1 -- 172.21.15.44:0/1971644862 shutdown_connections 2024-05-27T09:39:17.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.038+0000 7f993d800700 1 -- 172.21.15.44:0/3715455651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9938105da0 msgr2=0x7f993819d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:17.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.038+0000 7f993d800700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9938105da0 0x7f993819d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.038+0000 7f993d800700 1 -- 172.21.15.44:0/3715455651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99381a1e60 con 0x7f9938105450 2024-05-27T09:39:17.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.038+0000 7fad92c00700 1 -- 172.21.15.44:0/1971644862 wait complete. 2024-05-27T09:39:17.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.039+0000 7fad92c00700 1 Processor -- start 2024-05-27T09:39:17.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.039+0000 7f993d800700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9938105450 0x7f993819cf90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f992c00c370 tx=0x7f992c00c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:17.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.039+0000 7fad92c00700 1 -- start start 2024-05-27T09:39:17.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7f9936000700 1 -- 172.21.15.44:0/3715455651 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992c0092e0 con 0x7f9938105450 2024-05-27T09:39:17.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7f9936000700 1 -- 172.21.15.44:0/3715455651 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f992c00cbe0 con 0x7f9938105450 2024-05-27T09:39:17.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7f9936000700 1 -- 172.21.15.44:0/3715455651 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992c0046b0 con 0x7f9938105450 2024-05-27T09:39:17.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7f9940200700 1 -- 172.21.15.44:0/3715455651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99381a20f0 con 0x7f9938105450 2024-05-27T09:39:17.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7f9940200700 1 -- 172.21.15.44:0/3715455651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99381a25b0 con 0x7f9938105450 2024-05-27T09:39:17.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7fad92c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad8c1045e0 0x7fad8c19d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7fad92c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8c108f10 0x7fad8c19d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7fad92c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c109860 0x7fad8c1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7fad92c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c1144e0 con 0x7fad8c109860 2024-05-27T09:39:17.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7fad92c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c114360 con 0x7fad8c108f10 2024-05-27T09:39:17.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7fad92c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c114660 con 0x7fad8c1045e0 2024-05-27T09:39:17.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7fad8be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad8c1045e0 0x7fad8c19d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.040+0000 7fad8be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad8c1045e0 0x7fad8c19d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49984/0 (socket says 172.21.15.44:49984) 2024-05-27T09:39:17.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.041+0000 7fad8b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8c108f10 0x7fad8c19d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.041+0000 7fad8be00700 1 -- 172.21.15.44:0/3494422579 learned_addr learned my addr 172.21.15.44:0/3494422579 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:17.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.041+0000 7fad90c00700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c109860 0x7fad8c1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.041+0000 7f9940200700 1 -- 172.21.15.44:0/3715455651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f993806bee0 con 0x7f9938105450 2024-05-27T09:39:17.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.041+0000 7fad8b400700 1 -- 172.21.15.44:0/3494422579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad8c1045e0 msgr2=0x7fad8c19d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:17.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.041+0000 7fad8b400700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad8c1045e0 0x7fad8c19d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.041+0000 7fad8b400700 1 -- 172.21.15.44:0/3494422579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c109860 msgr2=0x7fad8c1a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:17.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.041+0000 7fad8b400700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c109860 0x7fad8c1a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.041+0000 7fad8b400700 1 -- 172.21.15.44:0/3494422579 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8c1a1fe0 con 0x7fad8c108f10 2024-05-27T09:39:17.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.042+0000 7fad8b400700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8c108f10 0x7fad8c19d6b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fad7800e9c0 tx=0x7fad7800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:17.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.042+0000 7fad88c00700 1 -- 172.21.15.44:0/3494422579 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7800c9c0 con 0x7fad8c108f10 2024-05-27T09:39:17.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.042+0000 7fad88c00700 1 -- 172.21.15.44:0/3494422579 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad7800cb60 con 0x7fad8c108f10 2024-05-27T09:39:17.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.042+0000 7fad92c00700 1 -- 172.21.15.44:0/3494422579 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad8c1a22d0 con 0x7fad8c108f10 2024-05-27T09:39:17.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.042+0000 7fad88c00700 1 -- 172.21.15.44:0/3494422579 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78018680 con 0x7fad8c108f10 2024-05-27T09:39:17.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.042+0000 7fad92c00700 1 -- 172.21.15.44:0/3494422579 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad8c10b110 con 0x7fad8c108f10 2024-05-27T09:39:17.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.046+0000 7f9936000700 1 -- 172.21.15.44:0/3715455651 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f992c01a030 con 0x7f9938105450 2024-05-27T09:39:17.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.049+0000 7f9936000700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99200b2a70 0x7f99200b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.049+0000 7f9936000700 1 -- 172.21.15.44:0/3715455651 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f992c0e1ac0 con 0x7f9938105450 2024-05-27T09:39:17.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.049+0000 7f9936000700 1 -- 172.21.15.44:0/3715455651 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f992c010040 con 0x7f9938105450 2024-05-27T09:39:17.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.049+0000 7fad88c00700 1 -- 172.21.15.44:0/3494422579 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fad78018820 con 0x7fad8c108f10 2024-05-27T09:39:17.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.050+0000 7f993ce00700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99200b2a70 0x7f99200b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.051+0000 7f993ce00700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99200b2a70 0x7f99200b4f80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f99240098e0 tx=0x7f9924005960 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:17.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.052+0000 7fad88c00700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad700b29f0 0x7fad700b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.053+0000 7fad8be00700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad700b29f0 0x7fad700b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.053+0000 7fad88c00700 1 -- 172.21.15.44:0/3494422579 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fad780e2620 con 0x7fad8c108f10 2024-05-27T09:39:17.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.054+0000 7fad8be00700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad700b29f0 0x7fad700b4f00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00b790 tx=0x7fad7c00b260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:17.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.070+0000 7fada17f7b80 1 Processor -- start 2024-05-27T09:39:17.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.070+0000 7fada17f7b80 1 -- start start 2024-05-27T09:39:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.071+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 0x5654036d2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.071+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036d3380 0x5654036ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.071+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 0x5654036cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.071+0000 7fada17f7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5654035fafd0 con 0x5654036cd4d0 2024-05-27T09:39:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.071+0000 7fada17f7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5654035eab70 con 0x5654036d3380 2024-05-27T09:39:17.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.071+0000 7fada17f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5654035eebb0 con 0x56540354dfd0 2024-05-27T09:39:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.071+0000 7fad90c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 0x5654036cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.071+0000 7fad8be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 0x5654036d2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.071+0000 7fad90c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 0x5654036cd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50694/0 (socket says 172.21.15.44:50694) 2024-05-27T09:39:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.071+0000 7fad90c00700 1 -- 172.21.15.44:0/1782862889 learned_addr learned my addr 172.21.15.44:0/1782862889 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.072+0000 7fad8b400700 1 --2- 172.21.15.44:0/1782862889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036d3380 0x5654036ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.072+0000 7fad8be00700 1 -- 172.21.15.44:0/1782862889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036d3380 msgr2=0x5654036ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.072+0000 7fad8be00700 1 --2- 172.21.15.44:0/1782862889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036d3380 0x5654036ccf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.072+0000 7fad8be00700 1 -- 172.21.15.44:0/1782862889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 msgr2=0x5654036cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:17.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.072+0000 7fad8be00700 1 --2- 172.21.15.44:0/1782862889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 0x5654036cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.072+0000 7fad8be00700 1 -- 172.21.15.44:0/1782862889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654035b35b0 con 0x56540354dfd0 2024-05-27T09:39:17.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.073+0000 7fad8be00700 1 --2- 172.21.15.44:0/1782862889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 0x5654036d2e40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fad7c000c00 tx=0x7fad7c01e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccecbe5861d9b1c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:17.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.073+0000 7fad81600700 1 -- 172.21.15.44:0/1782862889 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0056c0 con 0x56540354dfd0 2024-05-27T09:39:17.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.073+0000 7fad81600700 1 -- 172.21.15.44:0/1782862889 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad7c003b30 con 0x56540354dfd0 2024-05-27T09:39:17.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.073+0000 7fad81600700 1 -- 172.21.15.44:0/1782862889 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c00fab0 con 0x56540354dfd0 2024-05-27T09:39:17.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.074+0000 7fada17f7b80 1 -- 172.21.15.44:0/1782862889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 msgr2=0x5654036d2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:17.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.074+0000 7fada17f7b80 1 --2- 172.21.15.44:0/1782862889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 0x5654036d2e40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fad7c000c00 tx=0x7fad7c01e000 comp rx=0 tx=0).stop 2024-05-27T09:39:17.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.075+0000 7fada17f7b80 1 -- 172.21.15.44:0/1782862889 shutdown_connections 2024-05-27T09:39:17.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.075+0000 7fada17f7b80 1 --2- 172.21.15.44:0/1782862889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 0x5654036cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.075+0000 7fada17f7b80 1 --2- 172.21.15.44:0/1782862889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 0x5654036d2e40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.075+0000 7fada17f7b80 1 --2- 172.21.15.44:0/1782862889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036d3380 0x5654036ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.075+0000 7fada17f7b80 1 -- 172.21.15.44:0/1782862889 >> 172.21.15.44:0/1782862889 conn(0x5654035ae150 msgr2=0x5654036c8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:17.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.075+0000 7fada17f7b80 1 -- 172.21.15.44:0/1782862889 shutdown_connections 2024-05-27T09:39:17.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.075+0000 7fada17f7b80 1 -- 172.21.15.44:0/1782862889 wait complete. 2024-05-27T09:39:17.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.076+0000 7fada17f7b80 1 -- start start 2024-05-27T09:39:17.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.076+0000 7fada17f7b80 1 Processor -- start 2024-05-27T09:39:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.077+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56540354dfd0 0x5654036d6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.077+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 0x5654036d6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.077+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 0x5654036d74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.077+0000 7fada17f7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5654034f1130 con 0x56540354dfd0 2024-05-27T09:39:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.077+0000 7fada17f7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56540358b8f0 con 0x5654036cd4d0 2024-05-27T09:39:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.077+0000 7fada17f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5654036d9d90 con 0x5654036d3380 2024-05-27T09:39:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.077+0000 7fada17f7b80 10 client.0 _ll_register_callbacks cb 0x565403594e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.077+0000 7fad8be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56540354dfd0 0x5654036d6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.077+0000 7fad8be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56540354dfd0 0x5654036d6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50702/0 (socket says 172.21.15.44:50702) 2024-05-27T09:39:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.077+0000 7fad8be00700 1 -- 172.21.15.44:0/3018496141 learned_addr learned my addr 172.21.15.44:0/3018496141 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.077+0000 7fad8b400700 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 0x5654036d6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.078+0000 7fad90c00700 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 0x5654036d74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.078+0000 7fad8b400700 1 -- 172.21.15.44:0/3018496141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 msgr2=0x5654036d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.078+0000 7fad8b400700 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 0x5654036d74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.078+0000 7fad8b400700 1 -- 172.21.15.44:0/3018496141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56540354dfd0 msgr2=0x5654036d6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.078+0000 7fad8b400700 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56540354dfd0 0x5654036d6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.078+0000 7fad8b400700 1 -- 172.21.15.44:0/3018496141 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56540353aa60 con 0x5654036cd4d0 2024-05-27T09:39:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.078+0000 7fad90c00700 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 0x5654036d74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:39:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.079+0000 7fad8b400700 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 0x5654036d6f80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fad780dfda0 tx=0x7fad780a9110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=576e9cbee03403a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.079+0000 7fad76000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:39:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.079+0000 7fad76000700 1 -- 172.21.15.44:0/3018496141 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad780a9440 con 0x5654036cd4d0 2024-05-27T09:39:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.079+0000 7fad76000700 1 -- 172.21.15.44:0/3018496141 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fad780cadf0 con 0x5654036cd4d0 2024-05-27T09:39:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.079+0000 7fada17f7b80 1 -- 172.21.15.44:0/3018496141 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad7c003b30 con 0x5654036cd4d0 2024-05-27T09:39:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.080+0000 7fad76000700 1 -- 172.21.15.44:0/3018496141 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fad780aa030 con 0x5654036cd4d0 2024-05-27T09:39:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.080+0000 7fada17f7b80 10 client.25901 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.080+0000 7fada17f7b80 1 -- 172.21.15.44:0/3018496141 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5654035bb1f0 con 0x5654036cd4d0 2024-05-27T09:39:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.080+0000 7fad76000700 1 -- 172.21.15.44:0/3018496141 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad7c003b30 con 0x5654036cd4d0 2024-05-27T09:39:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.080+0000 7fad76000700 1 -- 172.21.15.44:0/3018496141 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fad780a9b90 con 0x5654036cd4d0 2024-05-27T09:39:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.081+0000 7fada17f7b80 10 client.25901 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.081+0000 7fada17f7b80 10 client.25901 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:39:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.081+0000 7fada17f7b80 20 client.25901 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.081+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5654036e6bd0 0x5654036e9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.081+0000 7fada17f7b80 4 client.25901 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:39:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.081+0000 7fada17f7b80 1 -- 172.21.15.44:0/3018496141 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654035fcf20 con 0x5654036e6bd0 2024-05-27T09:39:17.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.082+0000 7fad8be00700 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5654036e6bd0 0x5654036e9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.082+0000 7fad8be00700 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5654036e6bd0 0x5654036e9040 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e35769b1f25e1d23 server_cookie=544829b8ca371787 in_seq=0 out_seq=0 2024-05-27T09:39:17.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.082+0000 7fad76000700 10 client.25901 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.083+0000 7fad76000700 1 -- 172.21.15.44:0/3018496141 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5654035fcf20 con 0x5654036e6bd0 2024-05-27T09:39:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.083+0000 7fad76000700 10 client.25901 handle_command_reply: tid=0 2024-05-27T09:39:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.083+0000 7fada17f7b80 1 client.25901 shutdown 2024-05-27T09:39:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.084+0000 7fada17f7b80 1 -- 172.21.15.44:0/3018496141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 msgr2=0x5654036d6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.084+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 0x5654036d6f80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fad780dfda0 tx=0x7fad780a9110 comp rx=0 tx=0).stop 2024-05-27T09:39:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.084+0000 7fada17f7b80 1 -- 172.21.15.44:0/3018496141 shutdown_connections 2024-05-27T09:39:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.084+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5654036e6bd0 0x5654036e9040 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.084+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 0x5654036d74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.084+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56540354dfd0 0x5654036d6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.084+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3018496141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 0x5654036d6f80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.084+0000 7fada17f7b80 1 -- 172.21.15.44:0/3018496141 >> 172.21.15.44:0/3018496141 conn(0x5654035ae150 msgr2=0x5654036cd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.084+0000 7fad76000700 0 client.25901 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.084+0000 7fada17f7b80 1 -- 172.21.15.44:0/3018496141 shutdown_connections 2024-05-27T09:39:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.085+0000 7fada17f7b80 1 -- 172.21.15.44:0/3018496141 wait complete. 2024-05-27T09:39:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.085+0000 7fada17f7b80 20 client.25901 trim_cache size 0 max 16384 2024-05-27T09:39:17.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.126+0000 7fada17f7b80 1 Processor -- start 2024-05-27T09:39:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.126+0000 7fada17f7b80 1 -- start start 2024-05-27T09:39:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.127+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540354dfd0 0x5654036d8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.127+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 0x5654036cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.127+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 0x5654036e2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.127+0000 7fada17f7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c00ff70 con 0x5654036cd4d0 2024-05-27T09:39:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.127+0000 7fada17f7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c00d800 con 0x56540354dfd0 2024-05-27T09:39:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.127+0000 7fada17f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c00b650 con 0x5654036d3380 2024-05-27T09:39:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.127+0000 7fad8b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 0x5654036cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.127+0000 7fad8b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 0x5654036cc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50712/0 (socket says 172.21.15.44:50712) 2024-05-27T09:39:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.127+0000 7fad8b400700 1 -- 172.21.15.44:0/1832041721 learned_addr learned my addr 172.21.15.44:0/1832041721 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.127+0000 7fad8be00700 1 --2- 172.21.15.44:0/1832041721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540354dfd0 0x5654036d8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.127+0000 7fad90c00700 1 --2- 172.21.15.44:0/1832041721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 0x5654036e2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.128+0000 7fad8b400700 1 -- 172.21.15.44:0/1832041721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 msgr2=0x5654036e2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.128+0000 7fad8b400700 1 --2- 172.21.15.44:0/1832041721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 0x5654036e2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.128+0000 7fad8b400700 1 -- 172.21.15.44:0/1832041721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540354dfd0 msgr2=0x5654036d8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:17.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.128+0000 7fad8b400700 1 --2- 172.21.15.44:0/1832041721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540354dfd0 0x5654036d8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.128+0000 7fad8b400700 1 -- 172.21.15.44:0/1832041721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654036d7830 con 0x5654036cd4d0 2024-05-27T09:39:17.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.128+0000 7fad90c00700 1 --2- 172.21.15.44:0/1832041721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 0x5654036e2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:39:17.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.128+0000 7fad8be00700 1 --2- 172.21.15.44:0/1832041721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540354dfd0 0x5654036d8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:17.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.128+0000 7fad8b400700 1 --2- 172.21.15.44:0/1832041721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 0x5654036cc7d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fad780a98c0 tx=0x7fad780a5920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fc1acdd4999b3f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:17.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.128+0000 7fad80c00700 1 -- 172.21.15.44:0/1832041721 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad780adce0 con 0x5654036cd4d0 2024-05-27T09:39:17.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.128+0000 7fad80c00700 1 -- 172.21.15.44:0/1832041721 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad7808fe50 con 0x5654036cd4d0 2024-05-27T09:39:17.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.128+0000 7fad80c00700 1 -- 172.21.15.44:0/1832041721 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad780b24c0 con 0x5654036cd4d0 2024-05-27T09:39:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.129+0000 7fada17f7b80 1 -- 172.21.15.44:0/1832041721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 msgr2=0x5654036cc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.129+0000 7fada17f7b80 1 --2- 172.21.15.44:0/1832041721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 0x5654036cc7d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fad780a98c0 tx=0x7fad780a5920 comp rx=0 tx=0).stop 2024-05-27T09:39:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.130+0000 7fada17f7b80 1 -- 172.21.15.44:0/1832041721 shutdown_connections 2024-05-27T09:39:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.130+0000 7fada17f7b80 1 --2- 172.21.15.44:0/1832041721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654036d3380 0x5654036e2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.130+0000 7fada17f7b80 1 --2- 172.21.15.44:0/1832041721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540354dfd0 0x5654036d8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.130+0000 7fada17f7b80 1 --2- 172.21.15.44:0/1832041721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036cd4d0 0x5654036cc7d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.130+0000 7fada17f7b80 1 -- 172.21.15.44:0/1832041721 >> 172.21.15.44:0/1832041721 conn(0x5654035ae150 msgr2=0x5654036d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.130+0000 7fada17f7b80 1 -- 172.21.15.44:0/1832041721 shutdown_connections 2024-05-27T09:39:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.130+0000 7fada17f7b80 1 -- 172.21.15.44:0/1832041721 wait complete. 2024-05-27T09:39:17.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.130+0000 7fada17f7b80 1 -- start start 2024-05-27T09:39:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.131+0000 7fada17f7b80 1 Processor -- start 2024-05-27T09:39:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.131+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 0x5654036df6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.131+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 0x5654036dc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.131+0000 7fada17f7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036d3380 0x5654036dc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.131+0000 7fada17f7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c00bcc0 con 0x5654036d3380 2024-05-27T09:39:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.131+0000 7fada17f7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c00d130 con 0x5654036cd4d0 2024-05-27T09:39:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.131+0000 7fada17f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c00c390 con 0x56540354dfd0 2024-05-27T09:39:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.131+0000 7fada17f7b80 10 client.0 _ll_register_callbacks cb 0x565403594e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:17.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.132+0000 7fad8b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 0x5654036dc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.132+0000 7fad8b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 0x5654036dc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52906/0 (socket says 172.21.15.44:52906) 2024-05-27T09:39:17.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.132+0000 7fad8b400700 1 -- 172.21.15.44:0/3348548986 learned_addr learned my addr 172.21.15.44:0/3348548986 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:17.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.132+0000 7fad90c00700 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036d3380 0x5654036dc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.132+0000 7fad8be00700 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 0x5654036df6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.132+0000 7fad8be00700 1 -- 172.21.15.44:0/3348548986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 msgr2=0x5654036dc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:17.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.132+0000 7fad8be00700 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 0x5654036dc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.132+0000 7fad8be00700 1 -- 172.21.15.44:0/3348548986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036d3380 msgr2=0x5654036dc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:17.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.132+0000 7fad8be00700 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036d3380 0x5654036dc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.132+0000 7fad8be00700 1 -- 172.21.15.44:0/3348548986 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654036dcc30 con 0x56540354dfd0 2024-05-27T09:39:17.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.132+0000 7fad90c00700 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036d3380 0x5654036dc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:17.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.133+0000 7fad8be00700 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 0x5654036df6b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x5654036e97b0 tx=0x7fad7c00f460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=396ceacf55a4ccaf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:17.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.133+0000 7fad76000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:39:17.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.133+0000 7fad76000700 1 -- 172.21.15.44:0/3348548986 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c00f680 con 0x56540354dfd0 2024-05-27T09:39:17.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.133+0000 7fada17f7b80 1 -- 172.21.15.44:0/3348548986 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad7808fe50 con 0x56540354dfd0 2024-05-27T09:39:17.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.133+0000 7fad76000700 1 -- 172.21.15.44:0/3348548986 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fad7c02b040 con 0x56540354dfd0 2024-05-27T09:39:17.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.133+0000 7fad76000700 1 -- 172.21.15.44:0/3348548986 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fad7c00f820 con 0x56540354dfd0 2024-05-27T09:39:17.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.133+0000 7fada17f7b80 10 client.25918 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:17.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.133+0000 7fada17f7b80 1 -- 172.21.15.44:0/3348548986 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x565403752530 con 0x56540354dfd0 2024-05-27T09:39:17.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.134+0000 7fad76000700 1 -- 172.21.15.44:0/3348548986 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad7808fe50 con 0x56540354dfd0 2024-05-27T09:39:17.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.134+0000 7fad76000700 1 -- 172.21.15.44:0/3348548986 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fad7c022070 con 0x56540354dfd0 2024-05-27T09:39:17.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.134+0000 7fada17f7b80 10 client.25918 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:17.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.134+0000 7fada17f7b80 10 client.25918 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:39:17.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.134+0000 7fada17f7b80 20 client.25918 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:17.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.134+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565403752700 0x565403754b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:17.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.134+0000 7fada17f7b80 4 client.25918 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-05-27T09:39:17.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.134+0000 7fada17f7b80 1 -- 172.21.15.44:0/3348548986 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x5654035f6700 con 0x565403752700 2024-05-27T09:39:17.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.135+0000 7fad8b400700 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565403752700 0x565403754b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.135+0000 7fad76000700 10 client.25918 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:17.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.135+0000 7fad8b400700 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565403752700 0x565403754b20 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d15d48604103b4a4 server_cookie=619113c87110bc22 in_seq=0 out_seq=0 2024-05-27T09:39:17.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.371+0000 7f9940200700 1 -- 172.21.15.44:0/3715455651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f993819e2f0 con 0x7f9938105450 2024-05-27T09:39:17.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.373+0000 7f9936000700 1 -- 172.21.15.44:0/3715455651 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7f992c0a9470 con 0x7f9938105450 2024-05-27T09:39:17.374 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:39:17.374 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:39:17.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.376+0000 7f9940200700 1 -- 172.21.15.44:0/3715455651 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99200b2a70 msgr2=0x7f99200b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:17.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.376+0000 7f9940200700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99200b2a70 0x7f99200b4f80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f99240098e0 tx=0x7f9924005960 comp rx=0 tx=0).stop 2024-05-27T09:39:17.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.376+0000 7f9940200700 1 -- 172.21.15.44:0/3715455651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9938105450 msgr2=0x7f993819cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:17.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.376+0000 7f9940200700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9938105450 0x7f993819cf90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f992c00c370 tx=0x7f992c00c680 comp rx=0 tx=0).stop 2024-05-27T09:39:17.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.376+0000 7f9940200700 1 -- 172.21.15.44:0/3715455651 shutdown_connections 2024-05-27T09:39:17.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.376+0000 7f9940200700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99200b2a70 0x7f99200b4f80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.376+0000 7f9940200700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f993810e8c0 0x7f99381a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.376+0000 7f9940200700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9938105450 0x7f993819cf90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.376+0000 7f9940200700 1 --2- 172.21.15.44:0/3715455651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9938105da0 0x7f993819d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:17.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.376+0000 7f9940200700 1 -- 172.21.15.44:0/3715455651 >> 172.21.15.44:0/3715455651 conn(0x7f99380fedb0 msgr2=0x7f9938101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:17.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.377+0000 7f9940200700 1 -- 172.21.15.44:0/3715455651 shutdown_connections 2024-05-27T09:39:17.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:17.377+0000 7f9940200700 1 -- 172.21.15.44:0/3715455651 wait complete. 2024-05-27T09:39:17.379 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:17.379 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:17.379 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:39:17.895 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:17.895 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:17.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:17 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3715455651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:39:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:17 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3715455651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:39:18.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:17 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3715455651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:39:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:18 smithi050 ceph-mon[36371]: pgmap v124: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 KiB/s wr, 31 op/s 2024-05-27T09:39:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:18 smithi050 ceph-mon[36371]: pgmap v125: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-05-27T09:39:19.072 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:18 smithi044 ceph-mon[31618]: pgmap v124: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 KiB/s wr, 31 op/s 2024-05-27T09:39:19.072 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:18 smithi044 ceph-mon[31618]: pgmap v125: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-05-27T09:39:19.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:18 smithi189 ceph-mon[36671]: pgmap v124: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 KiB/s wr, 31 op/s 2024-05-27T09:39:19.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:18 smithi189 ceph-mon[36671]: pgmap v125: 97 pgs: 97 active+clean; 2.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-05-27T09:39:19.670 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:39:20.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fad76000700 1 -- 172.21.15.44:0/3348548986 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5654035f6700 con 0x565403752700 2024-05-27T09:39:20.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fad76000700 10 client.25918 handle_command_reply: tid=0 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fada17f7b80 1 client.25918 shutdown 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fada17f7b80 1 -- 172.21.15.44:0/3348548986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 msgr2=0x5654036df6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 0x5654036df6b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x5654036e97b0 tx=0x7fad7c00f460 comp rx=0 tx=0).stop 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fada17f7b80 1 -- 172.21.15.44:0/3348548986 shutdown_connections 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565403752700 0x565403754b20 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5654036d3380 0x5654036dc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56540354dfd0 0x5654036df6b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fada17f7b80 1 --2- 172.21.15.44:0/3348548986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654036cd4d0 0x5654036dc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fad76000700 0 client.25918 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.681+0000 7fada17f7b80 1 -- 172.21.15.44:0/3348548986 >> 172.21.15.44:0/3348548986 conn(0x5654035ae150 msgr2=0x5654036d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.682+0000 7fada17f7b80 1 -- 172.21.15.44:0/3348548986 shutdown_connections 2024-05-27T09:39:20.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.682+0000 7fada17f7b80 1 -- 172.21.15.44:0/3348548986 wait complete. 2024-05-27T09:39:20.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.682+0000 7fada17f7b80 20 client.25918 trim_cache size 0 max 16384 2024-05-27T09:39:20.685 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:39:20.685 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 26, 2024-05-27T09:39:20.685 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:39:20.685 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 6, 2024-05-27T09:39:20.685 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:39:20.685 INFO:teuthology.orchestra.run.smithi044.stdout: "f17bf296": { 2024-05-27T09:39:20.685 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 6, 2024-05-27T09:39:20.685 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:39:20.685 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 130.0, 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:39:20.686 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:39:20.687 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:39:20.687 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:39:20.687 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:39:20.687 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:39:20.687 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:39:20.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 -- 172.21.15.44:0/3494422579 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad700b29f0 msgr2=0x7fad700b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:20.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad700b29f0 0x7fad700b4f00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00b790 tx=0x7fad7c00b260 comp rx=0 tx=0).stop 2024-05-27T09:39:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 -- 172.21.15.44:0/3494422579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8c108f10 msgr2=0x7fad8c19d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8c108f10 0x7fad8c19d6b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fad7800e9c0 tx=0x7fad7800ecd0 comp rx=0 tx=0).stop 2024-05-27T09:39:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 -- 172.21.15.44:0/3494422579 shutdown_connections 2024-05-27T09:39:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fad700b29f0 0x7fad700b4f00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c109860 0x7fad8c1a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad8c1045e0 0x7fad8c19d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 --2- 172.21.15.44:0/3494422579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8c108f10 0x7fad8c19d6b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 -- 172.21.15.44:0/3494422579 >> 172.21.15.44:0/3494422579 conn(0x7fad8c100030 msgr2=0x7fad8c1086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 -- 172.21.15.44:0/3494422579 shutdown_connections 2024-05-27T09:39:20.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:20.684+0000 7fad92c00700 1 -- 172.21.15.44:0/3494422579 wait complete. 2024-05-27T09:39:21.857 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'f17bf296', quiesce took 72.4 seconds. Will release after: -62.400000000000006 2024-05-27T09:39:21.857 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:39:21.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:21 smithi044 ceph-mon[31618]: pgmap v126: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 16 op/s 2024-05-27T09:39:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:21 smithi050 ceph-mon[36371]: pgmap v126: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 16 op/s 2024-05-27T09:39:22.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:21 smithi189 ceph-mon[36671]: pgmap v126: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 26 KiB/s wr, 16 op/s 2024-05-27T09:39:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:39:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:39:23.948 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:23 smithi044 ceph-mon[31618]: pgmap v127: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 6 op/s 2024-05-27T09:39:24.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:23 smithi050 ceph-mon[36371]: pgmap v127: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 6 op/s 2024-05-27T09:39:24.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:23 smithi189 ceph-mon[36671]: pgmap v127: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 6 op/s 2024-05-27T09:39:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.602+0000 7fa73be00640 1 -- 172.21.15.44:0/3096637465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa734075420 msgr2=0x7fa734075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.602+0000 7fa73be00640 1 --2- 172.21.15.44:0/3096637465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa734075420 0x7fa734075830 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa728009860 tx=0x7fa72802a340 comp rx=0 tx=0).stop 2024-05-27T09:39:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.602+0000 7fa73be00640 1 -- 172.21.15.44:0/3096637465 shutdown_connections 2024-05-27T09:39:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.602+0000 7fa73be00640 1 --2- 172.21.15.44:0/3096637465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7341104f0 0x7fa734112910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.602+0000 7fa73be00640 1 --2- 172.21.15.44:0/3096637465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa73410db90 0x7fa73410ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.602+0000 7fa73be00640 1 --2- 172.21.15.44:0/3096637465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa734075420 0x7fa734075830 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.602+0000 7fa73be00640 1 -- 172.21.15.44:0/3096637465 >> 172.21.15.44:0/3096637465 conn(0x7fa734079390 msgr2=0x7fa7340797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.602+0000 7fa73be00640 1 -- 172.21.15.44:0/3096637465 shutdown_connections 2024-05-27T09:39:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.602+0000 7fa73be00640 1 -- 172.21.15.44:0/3096637465 wait complete. 2024-05-27T09:39:24.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.616+0000 7fa73be00640 1 Processor -- start 2024-05-27T09:39:24.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.617+0000 7fa73be00640 1 -- start start 2024-05-27T09:39:24.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.617+0000 7fa73be00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa734075420 0x7fa7341a2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.617+0000 7fa73be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa73410db90 0x7fa7341a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.617+0000 7fa73be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7341104f0 0x7fa73419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.617+0000 7fa73be00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa734116050 con 0x7fa7341104f0 2024-05-27T09:39:24.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.617+0000 7fa73be00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa734115ed0 con 0x7fa734075420 2024-05-27T09:39:24.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.617+0000 7fa73be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa7341161d0 con 0x7fa73410db90 2024-05-27T09:39:24.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.618+0000 7fa739400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa734075420 0x7fa7341a2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.618+0000 7fa739400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa734075420 0x7fa7341a2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52100/0 (socket says 172.21.15.44:52100) 2024-05-27T09:39:24.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.618+0000 7fa739e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7341104f0 0x7fa73419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.618+0000 7fa739400640 1 -- 172.21.15.44:0/2567789075 learned_addr learned my addr 172.21.15.44:0/2567789075 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:24.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.618+0000 7fa738a00640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa73410db90 0x7fa7341a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.618+0000 7fa739400640 1 -- 172.21.15.44:0/2567789075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa73410db90 msgr2=0x7fa7341a2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:24.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.618+0000 7fa739400640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa73410db90 0x7fa7341a2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.618+0000 7fa739400640 1 -- 172.21.15.44:0/2567789075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7341104f0 msgr2=0x7fa73419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.618+0000 7fa739400640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7341104f0 0x7fa73419c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.618+0000 7fa739400640 1 -- 172.21.15.44:0/2567789075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa73419cfb0 con 0x7fa734075420 2024-05-27T09:39:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.618+0000 7fa738a00640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa73410db90 0x7fa7341a2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.619+0000 7fa739400640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa734075420 0x7fa7341a2700 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa728035040 tx=0x7fa728032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.619+0000 7fa732000640 1 -- 172.21.15.44:0/2567789075 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa728037040 con 0x7fa734075420 2024-05-27T09:39:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.619+0000 7fa732000640 1 -- 172.21.15.44:0/2567789075 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa728041070 con 0x7fa734075420 2024-05-27T09:39:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.619+0000 7fa732000640 1 -- 172.21.15.44:0/2567789075 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa728003850 con 0x7fa734075420 2024-05-27T09:39:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.619+0000 7fa73be00640 1 -- 172.21.15.44:0/2567789075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa73419d240 con 0x7fa734075420 2024-05-27T09:39:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.620+0000 7fa73be00640 1 -- 172.21.15.44:0/2567789075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa73419d6d0 con 0x7fa734075420 2024-05-27T09:39:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.628+0000 7fa732000640 1 -- 172.21.15.44:0/2567789075 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa7280039f0 con 0x7fa734075420 2024-05-27T09:39:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.630+0000 7fa732000640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa71c0b2aa0 0x7fa71c0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.630+0000 7fa732000640 1 -- 172.21.15.44:0/2567789075 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa728101f50 con 0x7fa734075420 2024-05-27T09:39:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.630+0000 7fa738a00640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa71c0b2aa0 0x7fa71c0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.631+0000 7fa738a00640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa71c0b2aa0 0x7fa71c0b4fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa734079930 tx=0x7fa720005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.822+0000 7fa742dfe740 1 Processor -- start 2024-05-27T09:39:24.823 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.822+0000 7fa742dfe740 1 -- start start 2024-05-27T09:39:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.823+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2aa21b0 0x5567b2e44860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.823+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a792f0 0x5567b2e44da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.823+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a814d0 0x5567b2e452e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.823+0000 7fa742dfe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5567b2db5b90 con 0x5567b2a814d0 2024-05-27T09:39:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.823+0000 7fa742dfe740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5567b2da5730 con 0x5567b2a792f0 2024-05-27T09:39:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.823+0000 7fa742dfe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5567b2da9770 con 0x5567b2aa21b0 2024-05-27T09:39:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.824+0000 7fa738a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a792f0 0x5567b2e44da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.824+0000 7fa738a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a792f0 0x5567b2e44da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52104/0 (socket says 172.21.15.44:52104) 2024-05-27T09:39:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.824+0000 7fa738a00640 1 -- 172.21.15.44:0/2919636527 learned_addr learned my addr 172.21.15.44:0/2919636527 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.824+0000 7fa739400640 1 --2- 172.21.15.44:0/2919636527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2aa21b0 0x5567b2e44860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.824+0000 7fa739e00640 1 --2- 172.21.15.44:0/2919636527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a814d0 0x5567b2e452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.824+0000 7fa738a00640 1 -- 172.21.15.44:0/2919636527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2aa21b0 msgr2=0x5567b2e44860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.824+0000 7fa738a00640 1 --2- 172.21.15.44:0/2919636527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2aa21b0 0x5567b2e44860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.824+0000 7fa738a00640 1 -- 172.21.15.44:0/2919636527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a814d0 msgr2=0x5567b2e452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.824+0000 7fa738a00640 1 --2- 172.21.15.44:0/2919636527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a814d0 0x5567b2e452e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.824+0000 7fa738a00640 1 -- 172.21.15.44:0/2919636527 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567b2d3e8b0 con 0x5567b2a792f0 2024-05-27T09:39:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.824+0000 7fa739400640 1 --2- 172.21.15.44:0/2919636527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2aa21b0 0x5567b2e44860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.825+0000 7fa738a00640 1 --2- 172.21.15.44:0/2919636527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a792f0 0x5567b2e44da0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa720004220 tx=0x7fa720014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd9cb6292087063f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.825+0000 7fa726600640 1 -- 172.21.15.44:0/2919636527 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72000f480 con 0x5567b2a792f0 2024-05-27T09:39:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.825+0000 7fa726600640 1 -- 172.21.15.44:0/2919636527 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa72000fad0 con 0x5567b2a792f0 2024-05-27T09:39:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.825+0000 7fa726600640 1 -- 172.21.15.44:0/2919636527 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72001eb00 con 0x5567b2a792f0 2024-05-27T09:39:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.827+0000 7fa742dfe740 1 -- 172.21.15.44:0/2919636527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a792f0 msgr2=0x5567b2e44da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.827+0000 7fa742dfe740 1 --2- 172.21.15.44:0/2919636527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a792f0 0x5567b2e44da0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa720004220 tx=0x7fa720014000 comp rx=0 tx=0).stop 2024-05-27T09:39:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.827+0000 7fa742dfe740 1 -- 172.21.15.44:0/2919636527 shutdown_connections 2024-05-27T09:39:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.827+0000 7fa742dfe740 1 --2- 172.21.15.44:0/2919636527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a814d0 0x5567b2e452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.827+0000 7fa742dfe740 1 --2- 172.21.15.44:0/2919636527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a792f0 0x5567b2e44da0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.827+0000 7fa742dfe740 1 --2- 172.21.15.44:0/2919636527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2aa21b0 0x5567b2e44860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.827+0000 7fa742dfe740 1 -- 172.21.15.44:0/2919636527 >> 172.21.15.44:0/2919636527 conn(0x5567b2a96300 msgr2=0x5567b2e41010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.827+0000 7fa742dfe740 1 -- 172.21.15.44:0/2919636527 shutdown_connections 2024-05-27T09:39:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.827+0000 7fa742dfe740 1 -- 172.21.15.44:0/2919636527 wait complete. 2024-05-27T09:39:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.896+0000 7fa742dfe740 1 -- start start 2024-05-27T09:39:24.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.897+0000 7fa742dfe740 1 Processor -- start 2024-05-27T09:39:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.898+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a792f0 0x5567b2e4ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.899+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 0x5567b2e4efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.899+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2aa21b0 0x5567b2e5acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.899+0000 7fa742dfe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5567b2a8b790 con 0x5567b2aa21b0 2024-05-27T09:39:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.899+0000 7fa742dfe740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5567b2d4dbc0 con 0x5567b2a814d0 2024-05-27T09:39:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.899+0000 7fa742dfe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5567b2e51990 con 0x5567b2a792f0 2024-05-27T09:39:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.899+0000 7fa739e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2aa21b0 0x5567b2e5acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.899+0000 7fa739400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a792f0 0x5567b2e4ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.899+0000 7fa739e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2aa21b0 0x5567b2e5acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58334/0 (socket says 172.21.15.44:58334) 2024-05-27T09:39:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.899+0000 7fa739e00640 1 -- 172.21.15.44:0/354360976 learned_addr learned my addr 172.21.15.44:0/354360976 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.899+0000 7fa739400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a792f0 0x5567b2e4ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38770/0 (socket says 172.21.15.44:38770) 2024-05-27T09:39:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.899+0000 7fa738a00640 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 0x5567b2e4efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.900+0000 7fa739400640 1 -- 172.21.15.44:0/354360976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 msgr2=0x5567b2e4efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.900+0000 7fa739400640 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 0x5567b2e4efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.900+0000 7fa739400640 1 -- 172.21.15.44:0/354360976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2aa21b0 msgr2=0x5567b2e5acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.900+0000 7fa739400640 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2aa21b0 0x5567b2e5acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.900+0000 7fa739400640 1 -- 172.21.15.44:0/354360976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567b2aba210 con 0x5567b2a792f0 2024-05-27T09:39:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.900+0000 7fa739400640 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a792f0 0x5567b2e4ea60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa7280ff780 tx=0x7fa7280057e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47bfc5c95fa6574b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.900+0000 7fa71aa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:39:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.900+0000 7fa71aa00640 1 -- 172.21.15.44:0/354360976 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7280e3060 con 0x5567b2a792f0 2024-05-27T09:39:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.900+0000 7fa71aa00640 1 -- 172.21.15.44:0/354360976 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa7280c8800 con 0x5567b2a792f0 2024-05-27T09:39:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.904+0000 7fa742dfe740 10 client.0 _ll_register_callbacks cb 0x5567b2d2b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.904+0000 7fa742dfe740 1 -- 172.21.15.44:0/354360976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa72000fad0 con 0x5567b2a792f0 2024-05-27T09:39:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.905+0000 7fa71aa00640 1 -- 172.21.15.44:0/354360976 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa7280e8050 con 0x5567b2a792f0 2024-05-27T09:39:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.912+0000 7fa742dfe740 10 client.25930 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.912+0000 7fa742dfe740 1 -- 172.21.15.44:0/354360976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=26}) -- 0x5567b2b86620 con 0x5567b2a792f0 2024-05-27T09:39:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.913+0000 7fa71aa00640 1 -- 172.21.15.44:0/354360976 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa7280c4d00 con 0x5567b2a792f0 2024-05-27T09:39:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.913+0000 7fa742dfe740 10 client.25930 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.913+0000 7fa742dfe740 10 client.25930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:39:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.913+0000 7fa742dfe740 20 client.25930 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.913+0000 7fa742dfe740 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5567b2a60d80 0x5567b2e60ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.913+0000 7fa742dfe740 4 client.25930 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:39:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.913+0000 7fa742dfe740 1 -- 172.21.15.44:0/354360976 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567b2db7ae0 con 0x5567b2a60d80 2024-05-27T09:39:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.913+0000 7fa738a00640 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5567b2a60d80 0x5567b2e60ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.914+0000 7fa71aa00640 10 client.25930 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.914+0000 7fa738a00640 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5567b2a60d80 0x5567b2e60ae0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a9f94731b19c9ea server_cookie=888f463eca5ea12f in_seq=0 out_seq=0 2024-05-27T09:39:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.915+0000 7fa71aa00640 1 -- 172.21.15.44:0/354360976 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5567b2db7ae0 con 0x5567b2a60d80 2024-05-27T09:39:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.915+0000 7fa71aa00640 10 client.25930 handle_command_reply: tid=0 2024-05-27T09:39:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.915+0000 7fa742dfe740 1 client.25930 shutdown 2024-05-27T09:39:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.921+0000 7fa742dfe740 1 -- 172.21.15.44:0/354360976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a792f0 msgr2=0x5567b2e4ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.921+0000 7fa742dfe740 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a792f0 0x5567b2e4ea60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa7280ff780 tx=0x7fa7280057e0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.921+0000 7fa742dfe740 1 -- 172.21.15.44:0/354360976 shutdown_connections 2024-05-27T09:39:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.921+0000 7fa742dfe740 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5567b2a60d80 0x5567b2e60ae0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.921+0000 7fa742dfe740 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2aa21b0 0x5567b2e5acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.921+0000 7fa742dfe740 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 0x5567b2e4efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.921+0000 7fa742dfe740 1 --2- 172.21.15.44:0/354360976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a792f0 0x5567b2e4ea60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.921+0000 7fa742dfe740 1 -- 172.21.15.44:0/354360976 >> 172.21.15.44:0/354360976 conn(0x5567b2a96300 msgr2=0x5567b2e44690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.921+0000 7fa71aa00640 0 client.25930 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.922+0000 7fa742dfe740 1 -- 172.21.15.44:0/354360976 shutdown_connections 2024-05-27T09:39:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.922+0000 7fa742dfe740 1 -- 172.21.15.44:0/354360976 wait complete. 2024-05-27T09:39:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.922+0000 7fa742dfe740 20 client.25930 trim_cache size 0 max 16384 2024-05-27T09:39:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.972+0000 7fa742dfe740 1 Processor -- start 2024-05-27T09:39:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.972+0000 7fa742dfe740 1 -- start start 2024-05-27T09:39:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.972+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 0x5567b2e51190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.972+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a814d0 0x5567b2e44d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.972+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2aa21b0 0x5567b2e5aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.972+0000 7fa742dfe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa7341b4550 con 0x5567b2a792f0 2024-05-27T09:39:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.972+0000 7fa742dfe740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa7341b1e20 con 0x5567b2aa21b0 2024-05-27T09:39:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.972+0000 7fa742dfe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa7341afc70 con 0x5567b2a814d0 2024-05-27T09:39:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.973+0000 7fa739400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 0x5567b2e51190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.973+0000 7fa739400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 0x5567b2e51190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58336/0 (socket says 172.21.15.44:58336) 2024-05-27T09:39:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.973+0000 7fa739400640 1 -- 172.21.15.44:0/4053962123 learned_addr learned my addr 172.21.15.44:0/4053962123 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.973+0000 7fa738a00640 1 --2- 172.21.15.44:0/4053962123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a814d0 0x5567b2e44d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.973+0000 7fa739e00640 1 --2- 172.21.15.44:0/4053962123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2aa21b0 0x5567b2e5aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.973+0000 7fa739400640 1 -- 172.21.15.44:0/4053962123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a814d0 msgr2=0x5567b2e44d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.973+0000 7fa739400640 1 --2- 172.21.15.44:0/4053962123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a814d0 0x5567b2e44d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.973+0000 7fa739400640 1 -- 172.21.15.44:0/4053962123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2aa21b0 msgr2=0x5567b2e5aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.973+0000 7fa739400640 1 --2- 172.21.15.44:0/4053962123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2aa21b0 0x5567b2e5aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.973+0000 7fa739400640 1 -- 172.21.15.44:0/4053962123 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567b2e42010 con 0x5567b2a792f0 2024-05-27T09:39:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.974+0000 7fa739400640 1 --2- 172.21.15.44:0/4053962123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 0x5567b2e51190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa7280c6600 tx=0x7fa7280cb000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f869dfebfc3ed2c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.974+0000 7fa725c00640 1 -- 172.21.15.44:0/4053962123 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7280c8ce0 con 0x5567b2a792f0 2024-05-27T09:39:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.974+0000 7fa725c00640 1 -- 172.21.15.44:0/4053962123 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa72000fad0 con 0x5567b2a792f0 2024-05-27T09:39:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.974+0000 7fa725c00640 1 -- 172.21.15.44:0/4053962123 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7280c9390 con 0x5567b2a792f0 2024-05-27T09:39:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.975+0000 7fa742dfe740 1 -- 172.21.15.44:0/4053962123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 msgr2=0x5567b2e51190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.975+0000 7fa742dfe740 1 --2- 172.21.15.44:0/4053962123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 0x5567b2e51190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa7280c6600 tx=0x7fa7280cb000 comp rx=0 tx=0).stop 2024-05-27T09:39:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.975+0000 7fa742dfe740 1 -- 172.21.15.44:0/4053962123 shutdown_connections 2024-05-27T09:39:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.975+0000 7fa742dfe740 1 --2- 172.21.15.44:0/4053962123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2aa21b0 0x5567b2e5aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.975+0000 7fa742dfe740 1 --2- 172.21.15.44:0/4053962123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2a814d0 0x5567b2e44d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.975+0000 7fa742dfe740 1 --2- 172.21.15.44:0/4053962123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 0x5567b2e51190 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.975+0000 7fa742dfe740 1 -- 172.21.15.44:0/4053962123 >> 172.21.15.44:0/4053962123 conn(0x5567b2a96300 msgr2=0x5567b2e445f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.975+0000 7fa742dfe740 1 -- 172.21.15.44:0/4053962123 shutdown_connections 2024-05-27T09:39:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.975+0000 7fa742dfe740 1 -- 172.21.15.44:0/4053962123 wait complete. 2024-05-27T09:39:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.976+0000 7fa742dfe740 1 -- start start 2024-05-27T09:39:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa742dfe740 1 Processor -- start 2024-05-27T09:39:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 0x5567b2e53cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 0x5567b2e54230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa742dfe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2aa21b0 0x5567b2ecf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa742dfe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa7341b02e0 con 0x5567b2a792f0 2024-05-27T09:39:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa742dfe740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa7341b1750 con 0x5567b2a814d0 2024-05-27T09:39:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa742dfe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa7341b09b0 con 0x5567b2aa21b0 2024-05-27T09:39:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa742dfe740 10 client.0 _ll_register_callbacks cb 0x7fa7280c5250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa738a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 0x5567b2e54230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa739400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 0x5567b2e53cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa738a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 0x5567b2e54230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52130/0 (socket says 172.21.15.44:52130) 2024-05-27T09:39:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.977+0000 7fa738a00640 1 -- 172.21.15.44:0/1678894317 learned_addr learned my addr 172.21.15.44:0/1678894317 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.978+0000 7fa739e00640 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2aa21b0 0x5567b2ecf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.978+0000 7fa738a00640 1 -- 172.21.15.44:0/1678894317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2aa21b0 msgr2=0x5567b2ecf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.978+0000 7fa738a00640 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2aa21b0 0x5567b2ecf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.978+0000 7fa738a00640 1 -- 172.21.15.44:0/1678894317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 msgr2=0x5567b2e53cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.978+0000 7fa738a00640 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 0x5567b2e53cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.978+0000 7fa738a00640 1 -- 172.21.15.44:0/1678894317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567b2e54a80 con 0x5567b2a814d0 2024-05-27T09:39:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.978+0000 7fa738a00640 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 0x5567b2e54230 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x5567b2e42350 tx=0x7fa720012f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=753b4dbf3bf24c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.978+0000 7fa71aa00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.979+0000 7fa71aa00640 1 -- 172.21.15.44:0/1678894317 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72003b060 con 0x5567b2a814d0 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.979+0000 7fa71aa00640 1 -- 172.21.15.44:0/1678894317 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa72000f480 con 0x5567b2a814d0 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.979+0000 7fa742dfe740 1 -- 172.21.15.44:0/1678894317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa72000fad0 con 0x5567b2a814d0 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.979+0000 7fa71aa00640 1 -- 172.21.15.44:0/1678894317 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa72000fe80 con 0x5567b2a814d0 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.979+0000 7fa742dfe740 10 client.25925 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.979+0000 7fa742dfe740 1 -- 172.21.15.44:0/1678894317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5567b2ed0ee0 con 0x5567b2a814d0 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.979+0000 7fa71aa00640 1 -- 172.21.15.44:0/1678894317 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa72000fad0 con 0x5567b2a814d0 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.980+0000 7fa71aa00640 1 -- 172.21.15.44:0/1678894317 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa720030070 con 0x5567b2a814d0 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.980+0000 7fa742dfe740 10 client.25925 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.980+0000 7fa742dfe740 10 client.25925 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:39:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.980+0000 7fa742dfe740 20 client.25925 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.980+0000 7fa742dfe740 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5567b2a361c0 0x5567b2ed3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.980+0000 7fa742dfe740 4 client.25925 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:39:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.980+0000 7fa742dfe740 1 -- 172.21.15.44:0/1678894317 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5567b2db12c0 con 0x5567b2a361c0 2024-05-27T09:39:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.980+0000 7fa739400640 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5567b2a361c0 0x5567b2ed3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.980+0000 7fa739400640 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5567b2a361c0 0x5567b2ed3340 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85a0496fa5eb9d57 server_cookie=e5044fdbe82024ff in_seq=0 out_seq=0 2024-05-27T09:39:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:24.980+0000 7fa71aa00640 10 client.25925 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.000+0000 7fa71aa00640 1 -- 172.21.15.44:0/1678894317 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5567b2db12c0 con 0x5567b2a361c0 2024-05-27T09:39:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.000+0000 7fa71aa00640 10 client.25925 handle_command_reply: tid=0 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.000+0000 7fa742dfe740 1 client.25925 shutdown 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.000+0000 7fa742dfe740 1 -- 172.21.15.44:0/1678894317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 msgr2=0x5567b2e54230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.000+0000 7fa742dfe740 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 0x5567b2e54230 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x5567b2e42350 tx=0x7fa720012f90 comp rx=0 tx=0).stop 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.001+0000 7fa742dfe740 1 -- 172.21.15.44:0/1678894317 shutdown_connections 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.001+0000 7fa742dfe740 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5567b2a361c0 0x5567b2ed3340 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.001+0000 7fa742dfe740 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567b2aa21b0 0x5567b2ecf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.001+0000 7fa742dfe740 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5567b2a814d0 0x5567b2e54230 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.001+0000 7fa742dfe740 1 --2- 172.21.15.44:0/1678894317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5567b2a792f0 0x5567b2e53cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.001+0000 7fa71aa00640 0 client.25925 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.001+0000 7fa742dfe740 1 -- 172.21.15.44:0/1678894317 >> 172.21.15.44:0/1678894317 conn(0x5567b2a96300 msgr2=0x5567b2e50b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.001+0000 7fa742dfe740 1 -- 172.21.15.44:0/1678894317 shutdown_connections 2024-05-27T09:39:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.001+0000 7fa742dfe740 1 -- 172.21.15.44:0/1678894317 wait complete. 2024-05-27T09:39:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.001+0000 7fa742dfe740 20 client.25925 trim_cache size 0 max 16384 2024-05-27T09:39:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:39:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:39:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:39:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 -- 172.21.15.44:0/2567789075 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa71c0b2aa0 msgr2=0x7fa71c0b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa71c0b2aa0 0x7fa71c0b4fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa734079930 tx=0x7fa720005f90 comp rx=0 tx=0).stop 2024-05-27T09:39:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 -- 172.21.15.44:0/2567789075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa734075420 msgr2=0x7fa7341a2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa734075420 0x7fa7341a2700 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa728035040 tx=0x7fa728032d50 comp rx=0 tx=0).stop 2024-05-27T09:39:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 -- 172.21.15.44:0/2567789075 shutdown_connections 2024-05-27T09:39:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa71c0b2aa0 0x7fa71c0b4fb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:25.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7341104f0 0x7fa73419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:25.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa73410db90 0x7fa7341a2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:25.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 --2- 172.21.15.44:0/2567789075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa734075420 0x7fa7341a2700 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:25.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 -- 172.21.15.44:0/2567789075 >> 172.21.15.44:0/2567789075 conn(0x7fa734079390 msgr2=0x7fa734102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:25.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 -- 172.21.15.44:0/2567789075 shutdown_connections 2024-05-27T09:39:25.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:25.002+0000 7fa73be00640 1 -- 172.21.15.44:0/2567789075 wait complete. 2024-05-27T09:39:25.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:25 smithi044 ceph-mon[31618]: pgmap v128: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 6 op/s 2024-05-27T09:39:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:25 smithi050 ceph-mon[36371]: pgmap v128: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 6 op/s 2024-05-27T09:39:26.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:25 smithi189 ceph-mon[36671]: pgmap v128: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 6 op/s 2024-05-27T09:39:27.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:27 smithi044 ceph-mon[31618]: pgmap v129: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-05-27T09:39:28.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:27 smithi050 ceph-mon[36371]: pgmap v129: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-05-27T09:39:28.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:27 smithi189 ceph-mon[36671]: pgmap v129: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 KiB/s wr, 14 op/s 2024-05-27T09:39:28.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.656+0000 7f3f0ec00700 1 -- 172.21.15.44:0/3028658607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f08106eb0 msgr2=0x7f3f08113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.656+0000 7f3f0ec00700 1 --2- 172.21.15.44:0/3028658607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f08106eb0 0x7f3f08113340 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000b0a0 tx=0x7f3f0000b3b0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.656+0000 7f3f0ec00700 1 -- 172.21.15.44:0/3028658607 shutdown_connections 2024-05-27T09:39:28.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.656+0000 7f3f0ec00700 1 --2- 172.21.15.44:0/3028658607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f08106eb0 0x7f3f08113340 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.656+0000 7f3f0ec00700 1 --2- 172.21.15.44:0/3028658607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f0810c5a0 0x7f3f0810c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.656+0000 7f3f0ec00700 1 --2- 172.21.15.44:0/3028658607 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08106560 0x7f3f08106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.656+0000 7f3f0ec00700 1 -- 172.21.15.44:0/3028658607 >> 172.21.15.44:0/3028658607 conn(0x7f3f08102040 msgr2=0x7f3f08104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:28.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.656+0000 7f3f0ec00700 1 -- 172.21.15.44:0/3028658607 shutdown_connections 2024-05-27T09:39:28.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.656+0000 7f3f0ec00700 1 -- 172.21.15.44:0/3028658607 wait complete. 2024-05-27T09:39:28.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.658+0000 7f3f0ec00700 1 Processor -- start 2024-05-27T09:39:28.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.658+0000 7f3f0ec00700 1 -- start start 2024-05-27T09:39:28.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.659+0000 7f3f0ec00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08106560 0x7f3f0819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.659+0000 7f3f0ec00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f08106eb0 0x7f3f0819d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.659+0000 7f3f0ec00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f0810c5a0 0x7f3f081a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.659+0000 7f3f0ec00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f08116720 con 0x7f3f0810c5a0 2024-05-27T09:39:28.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.659+0000 7f3f0ec00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3f081165a0 con 0x7f3f08106560 2024-05-27T09:39:28.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.659+0000 7f3f0ec00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f081168a0 con 0x7f3f08106eb0 2024-05-27T09:39:28.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.660+0000 7f3f07e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08106560 0x7f3f0819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.660+0000 7f3effe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f08106eb0 0x7f3f0819d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.660+0000 7f3f0cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f0810c5a0 0x7f3f081a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.660+0000 7f3f07e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08106560 0x7f3f0819cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52138/0 (socket says 172.21.15.44:52138) 2024-05-27T09:39:28.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.660+0000 7f3effe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f08106eb0 0x7f3f0819d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38800/0 (socket says 172.21.15.44:38800) 2024-05-27T09:39:28.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.660+0000 7f3f07e00700 1 -- 172.21.15.44:0/631745778 learned_addr learned my addr 172.21.15.44:0/631745778 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:28.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.660+0000 7f3f07e00700 1 -- 172.21.15.44:0/631745778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f08106eb0 msgr2=0x7f3f0819d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.660+0000 7f3f07e00700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f08106eb0 0x7f3f0819d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.660+0000 7f3f07e00700 1 -- 172.21.15.44:0/631745778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f0810c5a0 msgr2=0x7f3f081a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.660+0000 7f3f07e00700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f0810c5a0 0x7f3f081a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.660+0000 7f3f07e00700 1 -- 172.21.15.44:0/631745778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f081a1ec0 con 0x7f3f08106560 2024-05-27T09:39:28.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.661+0000 7f3f07e00700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08106560 0x7f3f0819cff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800e970 tx=0x7f3ef800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:28.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.661+0000 7f3f05600700 1 -- 172.21.15.44:0/631745778 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8004a30 con 0x7f3f08106560 2024-05-27T09:39:28.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.661+0000 7f3f05600700 1 -- 172.21.15.44:0/631745778 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ef800cc90 con 0x7f3f08106560 2024-05-27T09:39:28.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.661+0000 7f3f05600700 1 -- 172.21.15.44:0/631745778 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8005550 con 0x7f3f08106560 2024-05-27T09:39:28.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.662+0000 7f3f0ec00700 1 -- 172.21.15.44:0/631745778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f081a21b0 con 0x7f3f08106560 2024-05-27T09:39:28.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.662+0000 7f3f0ec00700 1 -- 172.21.15.44:0/631745778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f081a9ad0 con 0x7f3f08106560 2024-05-27T09:39:28.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.663+0000 7f3f0ec00700 1 -- 172.21.15.44:0/631745778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f080512b0 con 0x7f3f08106560 2024-05-27T09:39:28.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.667+0000 7f3f05600700 1 -- 172.21.15.44:0/631745778 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3ef8004bd0 con 0x7f3f08106560 2024-05-27T09:39:28.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.670+0000 7f3f05600700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3eec0b2a70 0x7f3eec0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.671+0000 7f3f05600700 1 -- 172.21.15.44:0/631745778 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f3ef80e2960 con 0x7f3f08106560 2024-05-27T09:39:28.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.671+0000 7f3effe00700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3eec0b2a70 0x7f3eec0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.672+0000 7f3effe00700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3eec0b2a70 0x7f3eec0b4f80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0009680 tx=0x7f3ef0009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:28.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.673+0000 7f3f05600700 1 -- 172.21.15.44:0/631745778 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3ef8054df0 con 0x7f3f08106560 2024-05-27T09:39:28.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.673+0000 7f72b9c00700 1 -- 172.21.15.44:0/2385447799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72b410c5a0 msgr2=0x7f72b410c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.673+0000 7f72b9c00700 1 --2- 172.21.15.44:0/2385447799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72b410c5a0 0x7f72b410c9b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009990 tx=0x7f72a4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.673+0000 7f72b9c00700 1 -- 172.21.15.44:0/2385447799 shutdown_connections 2024-05-27T09:39:28.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.673+0000 7f72b9c00700 1 --2- 172.21.15.44:0/2385447799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b4106eb0 0x7f72b4113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.673+0000 7f72b9c00700 1 --2- 172.21.15.44:0/2385447799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72b410c5a0 0x7f72b410c9b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.673+0000 7f72b9c00700 1 --2- 172.21.15.44:0/2385447799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b4106560 0x7f72b4106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.673+0000 7f72b9c00700 1 -- 172.21.15.44:0/2385447799 >> 172.21.15.44:0/2385447799 conn(0x7f72b41020c0 msgr2=0x7f72b4104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:28.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.674+0000 7f72b9c00700 1 -- 172.21.15.44:0/2385447799 shutdown_connections 2024-05-27T09:39:28.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.674+0000 7f72b9c00700 1 -- 172.21.15.44:0/2385447799 wait complete. 2024-05-27T09:39:28.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.675+0000 7f72b9c00700 1 Processor -- start 2024-05-27T09:39:28.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.675+0000 7f72b9c00700 1 -- start start 2024-05-27T09:39:28.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.676+0000 7f72b9c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72b4106560 0x7f72b419d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.676+0000 7f72b9c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b4106eb0 0x7f72b419d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.676+0000 7f72b9c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b410c5a0 0x7f72b41a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.676+0000 7f72b9c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72b4116700 con 0x7f72b4106560 2024-05-27T09:39:28.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.676+0000 7f72b9c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f72b4116580 con 0x7f72b410c5a0 2024-05-27T09:39:28.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.676+0000 7f72b9c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72b4116880 con 0x7f72b4106eb0 2024-05-27T09:39:28.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.676+0000 7f72b2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72b4106560 0x7f72b419d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.676+0000 7f72b2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b4106eb0 0x7f72b419d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.676+0000 7f72b2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72b4106560 0x7f72b419d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58384/0 (socket says 172.21.15.44:58384) 2024-05-27T09:39:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.676+0000 7f72b2a00700 1 -- 172.21.15.44:0/3025263355 learned_addr learned my addr 172.21.15.44:0/3025263355 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.676+0000 7f72b3400700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b410c5a0 0x7f72b41a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.677+0000 7f72b2000700 1 -- 172.21.15.44:0/3025263355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b410c5a0 msgr2=0x7f72b41a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.677+0000 7f72b2000700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b410c5a0 0x7f72b41a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.677+0000 7f72b2000700 1 -- 172.21.15.44:0/3025263355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72b4106560 msgr2=0x7f72b419d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.677+0000 7f72b2000700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72b4106560 0x7f72b419d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.677+0000 7f72b2000700 1 -- 172.21.15.44:0/3025263355 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b41a1f40 con 0x7f72b4106eb0 2024-05-27T09:39:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.677+0000 7f72b3400700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b410c5a0 0x7f72b41a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:39:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.677+0000 7f72b2a00700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72b4106560 0x7f72b419d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:28.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.677+0000 7f72b2000700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b4106eb0 0x7f72b419d610 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f72b4075a20 tx=0x7f72a000eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:28.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.678+0000 7f72ab400700 1 -- 172.21.15.44:0/3025263355 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a000cbd0 con 0x7f72b4106eb0 2024-05-27T09:39:28.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.678+0000 7f72b9c00700 1 -- 172.21.15.44:0/3025263355 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b41a2230 con 0x7f72b4106eb0 2024-05-27T09:39:28.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.678+0000 7f72ab400700 1 -- 172.21.15.44:0/3025263355 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72a000ee80 con 0x7f72b4106eb0 2024-05-27T09:39:28.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.678+0000 7f72ab400700 1 -- 172.21.15.44:0/3025263355 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0018680 con 0x7f72b4106eb0 2024-05-27T09:39:28.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.679+0000 7f72b9c00700 1 -- 172.21.15.44:0/3025263355 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b41a9b50 con 0x7f72b4106eb0 2024-05-27T09:39:28.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.683+0000 7f72ab400700 1 -- 172.21.15.44:0/3025263355 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f72a0010430 con 0x7f72b4106eb0 2024-05-27T09:39:28.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.684+0000 7f72ab400700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72980b2a70 0x7f72980b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.684+0000 7f72ab400700 1 -- 172.21.15.44:0/3025263355 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f72a00e28a0 con 0x7f72b4106eb0 2024-05-27T09:39:28.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.685+0000 7f72b2a00700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72980b2a70 0x7f72980b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.685+0000 7f72b2a00700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72980b2a70 0x7f72980b4f80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f72a4006010 tx=0x7f72a400b280 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:28.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.702+0000 7f72c87dbb80 1 Processor -- start 2024-05-27T09:39:28.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.703+0000 7f72c87dbb80 1 -- start start 2024-05-27T09:39:28.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.703+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0d17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.703+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0d1d30 0x5555ef0cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.703+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 0x5555ef0cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.703+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5555eeff9e40 con 0x5555ef0cbdb0 2024-05-27T09:39:28.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.703+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5555eefe99e0 con 0x5555eef4ce40 2024-05-27T09:39:28.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.703+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5555eefeda20 con 0x5555ef0d1d30 2024-05-27T09:39:28.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.704+0000 7f72b2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0d1d30 0x5555ef0cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.704+0000 7f72b2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0d17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.704+0000 7f72b2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0d1d30 0x5555ef0cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38818/0 (socket says 172.21.15.44:38818) 2024-05-27T09:39:28.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.704+0000 7f72b2000700 1 -- 172.21.15.44:0/2480042268 learned_addr learned my addr 172.21.15.44:0/2480042268 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:28.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.704+0000 7f72b3400700 1 --2- 172.21.15.44:0/2480042268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 0x5555ef0cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.704+0000 7f72b2a00700 1 -- 172.21.15.44:0/2480042268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0d1d30 msgr2=0x5555ef0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:28.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.704+0000 7f72b2a00700 1 --2- 172.21.15.44:0/2480042268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0d1d30 0x5555ef0cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.704+0000 7f72b2a00700 1 -- 172.21.15.44:0/2480042268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 msgr2=0x5555ef0cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:28.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.704+0000 7f72b2a00700 1 --2- 172.21.15.44:0/2480042268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 0x5555ef0cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.704+0000 7f72b2a00700 1 -- 172.21.15.44:0/2480042268 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555eefb2420 con 0x5555eef4ce40 2024-05-27T09:39:28.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.705+0000 7f72b2a00700 1 --2- 172.21.15.44:0/2480042268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0d17f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009750 tx=0x7f72a401e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc9aab926a85029f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:28.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.705+0000 7f729fe00700 1 -- 172.21.15.44:0/2480042268 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4005680 con 0x5555eef4ce40 2024-05-27T09:39:28.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.705+0000 7f729fe00700 1 -- 172.21.15.44:0/2480042268 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72a4003e90 con 0x5555eef4ce40 2024-05-27T09:39:28.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.705+0000 7f729fe00700 1 -- 172.21.15.44:0/2480042268 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4023400 con 0x5555eef4ce40 2024-05-27T09:39:28.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.706+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2480042268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 msgr2=0x5555ef0d17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.706+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/2480042268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0d17f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009750 tx=0x7f72a401e000 comp rx=0 tx=0).stop 2024-05-27T09:39:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.707+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2480042268 shutdown_connections 2024-05-27T09:39:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.707+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/2480042268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 0x5555ef0cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.707+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/2480042268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0d17f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.707+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/2480042268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0d1d30 0x5555ef0cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.707+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2480042268 >> 172.21.15.44:0/2480042268 conn(0x5555eefacfc0 msgr2=0x5555ef0c68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.707+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2480042268 shutdown_connections 2024-05-27T09:39:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.707+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2480042268 wait complete. 2024-05-27T09:39:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.708+0000 7f72c87dbb80 1 -- start start 2024-05-27T09:39:28.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.708+0000 7f72c87dbb80 1 Processor -- start 2024-05-27T09:39:28.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.709+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555eef4ce40 0x5555ef0d54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.709+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555ef0cbdb0 0x5555ef0d5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.709+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0d1d30 0x5555ef0d5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.709+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5555eeeeffa0 con 0x5555ef0d1d30 2024-05-27T09:39:28.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.709+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5555eef8a760 con 0x5555ef0cbdb0 2024-05-27T09:39:28.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.709+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5555ef0d8740 con 0x5555eef4ce40 2024-05-27T09:39:28.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.709+0000 7f72c87dbb80 10 client.0 _ll_register_callbacks cb 0x5555eef93c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:28.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.710+0000 7f72b2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555eef4ce40 0x5555ef0d54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.710+0000 7f72b2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555eef4ce40 0x5555ef0d54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38832/0 (socket says 172.21.15.44:38832) 2024-05-27T09:39:28.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.710+0000 7f72b2a00700 1 -- 172.21.15.44:0/2987077671 learned_addr learned my addr 172.21.15.44:0/2987077671 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:28.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.710+0000 7f72b3400700 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0d1d30 0x5555ef0d5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.710+0000 7f72b2000700 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555ef0cbdb0 0x5555ef0d5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.710+0000 7f72b2a00700 1 -- 172.21.15.44:0/2987077671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555ef0cbdb0 msgr2=0x5555ef0d5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:28.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.710+0000 7f72b2a00700 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555ef0cbdb0 0x5555ef0d5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.710+0000 7f72b2a00700 1 -- 172.21.15.44:0/2987077671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0d1d30 msgr2=0x5555ef0d5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:28.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.710+0000 7f72b2a00700 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0d1d30 0x5555ef0d5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.710+0000 7f72b2a00700 1 -- 172.21.15.44:0/2987077671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555eef398d0 con 0x5555eef4ce40 2024-05-27T09:39:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.711+0000 7f72b2a00700 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555eef4ce40 0x5555ef0d54e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f72a4030040 tx=0x7f72a40056e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e8a59949fd7923d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.711+0000 7f72b2000700 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555ef0cbdb0 0x5555ef0d5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:39:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.711+0000 7f729cc00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:39:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.711+0000 7f729cc00700 1 -- 172.21.15.44:0/2987077671 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a403c060 con 0x5555eef4ce40 2024-05-27T09:39:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.711+0000 7f729cc00700 1 -- 172.21.15.44:0/2987077671 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f72a4024ae0 con 0x5555eef4ce40 2024-05-27T09:39:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.711+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2987077671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72a4003e90 con 0x5555eef4ce40 2024-05-27T09:39:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.711+0000 7f729cc00700 1 -- 172.21.15.44:0/2987077671 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f72a4024ce0 con 0x5555eef4ce40 2024-05-27T09:39:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.712+0000 7f72c87dbb80 10 client.25954 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.712+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2987077671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5555eefba060 con 0x5555eef4ce40 2024-05-27T09:39:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.712+0000 7f729cc00700 1 -- 172.21.15.44:0/2987077671 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72a4003e90 con 0x5555eef4ce40 2024-05-27T09:39:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.712+0000 7f729cc00700 1 -- 172.21.15.44:0/2987077671 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f72a4022070 con 0x5555eef4ce40 2024-05-27T09:39:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.712+0000 7f72c87dbb80 10 client.25954 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.712+0000 7f72c87dbb80 10 client.25954 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:39:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.712+0000 7f72c87dbb80 20 client.25954 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.712+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5555ef0e54a0 0x5555ef0e78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.712+0000 7f72c87dbb80 4 client.25954 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:39:28.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.712+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2987077671 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555eeffbd90 con 0x5555ef0e54a0 2024-05-27T09:39:28.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.713+0000 7f72b2000700 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5555ef0e54a0 0x5555ef0e78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.714+0000 7f729cc00700 10 client.25954 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:39:28.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.714+0000 7f72b2000700 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5555ef0e54a0 0x5555ef0e78c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=29f7fc7307de8097 server_cookie=8e5ed4cbae50864 in_seq=0 out_seq=0 2024-05-27T09:39:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.714+0000 7f729cc00700 1 -- 172.21.15.44:0/2987077671 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5555eeffbd90 con 0x5555ef0e54a0 2024-05-27T09:39:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.714+0000 7f729cc00700 10 client.25954 handle_command_reply: tid=0 2024-05-27T09:39:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 1 client.25954 shutdown 2024-05-27T09:39:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2987077671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555eef4ce40 msgr2=0x5555ef0d54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555eef4ce40 0x5555ef0d54e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f72a4030040 tx=0x7f72a40056e0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2987077671 shutdown_connections 2024-05-27T09:39:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5555ef0e54a0 0x5555ef0e78c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0d1d30 0x5555ef0d5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555eef4ce40 0x5555ef0d54e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/2987077671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555ef0cbdb0 0x5555ef0d5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2987077671 >> 172.21.15.44:0/2987077671 conn(0x5555eefacfc0 msgr2=0x5555ef0c6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:28.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f729cc00700 0 client.25954 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:39:28.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2987077671 shutdown_connections 2024-05-27T09:39:28.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 1 -- 172.21.15.44:0/2987077671 wait complete. 2024-05-27T09:39:28.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.715+0000 7f72c87dbb80 20 client.25954 trim_cache size 0 max 16384 2024-05-27T09:39:28.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.732+0000 7f72c87dbb80 1 Processor -- start 2024-05-27T09:39:28.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.733+0000 7f72c87dbb80 1 -- start start 2024-05-27T09:39:28.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.733+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.733+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0cbdb0 0x5555ef0d7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.733+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0d1d30 0x5555ef0d52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.733+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41b37e0 con 0x5555ef0d1d30 2024-05-27T09:39:28.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.733+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41b1070 con 0x5555eef4ce40 2024-05-27T09:39:28.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.733+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41aeee0 con 0x5555ef0cbdb0 2024-05-27T09:39:28.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.733+0000 7f72b3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0d1d30 0x5555ef0d52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.734+0000 7f72b2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0cbdb0 0x5555ef0d7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.734+0000 7f72b2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0cbdb0 0x5555ef0d7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38844/0 (socket says 172.21.15.44:38844) 2024-05-27T09:39:28.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.734+0000 7f72b2000700 1 -- 172.21.15.44:0/4284494388 learned_addr learned my addr 172.21.15.44:0/4284494388 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:28.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.734+0000 7f72b2a00700 1 --2- 172.21.15.44:0/4284494388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.734+0000 7f72b3400700 1 -- 172.21.15.44:0/4284494388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0cbdb0 msgr2=0x5555ef0d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:28.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.734+0000 7f72b3400700 1 --2- 172.21.15.44:0/4284494388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0cbdb0 0x5555ef0d7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.734+0000 7f72b3400700 1 -- 172.21.15.44:0/4284494388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 msgr2=0x5555ef0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:28.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.734+0000 7f72b3400700 1 --2- 172.21.15.44:0/4284494388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.734+0000 7f72b3400700 1 -- 172.21.15.44:0/4284494388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555ef0c76d0 con 0x5555ef0d1d30 2024-05-27T09:39:28.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.734+0000 7f72b2a00700 1 --2- 172.21.15.44:0/4284494388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:28.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.734+0000 7f72b3400700 1 --2- 172.21.15.44:0/4284494388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0d1d30 0x5555ef0d52c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f72ac007600 tx=0x7f72ac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbc6726f7429ba7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:28.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.735+0000 7f729f400700 1 -- 172.21.15.44:0/4284494388 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac016810 con 0x5555ef0d1d30 2024-05-27T09:39:28.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.735+0000 7f729f400700 1 -- 172.21.15.44:0/4284494388 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72ac007e20 con 0x5555ef0d1d30 2024-05-27T09:39:28.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.735+0000 7f729f400700 1 -- 172.21.15.44:0/4284494388 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac015d10 con 0x5555ef0d1d30 2024-05-27T09:39:28.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.736+0000 7f72c87dbb80 1 -- 172.21.15.44:0/4284494388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0d1d30 msgr2=0x5555ef0d52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.736+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/4284494388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0d1d30 0x5555ef0d52c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f72ac007600 tx=0x7f72ac00b040 comp rx=0 tx=0).stop 2024-05-27T09:39:28.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.736+0000 7f72c87dbb80 1 -- 172.21.15.44:0/4284494388 shutdown_connections 2024-05-27T09:39:28.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.736+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/4284494388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0d1d30 0x5555ef0d52c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.736+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/4284494388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.736+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/4284494388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0cbdb0 0x5555ef0d7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.736+0000 7f72c87dbb80 1 -- 172.21.15.44:0/4284494388 >> 172.21.15.44:0/4284494388 conn(0x5555eefacfc0 msgr2=0x5555ef0d3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:28.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.736+0000 7f72c87dbb80 1 -- 172.21.15.44:0/4284494388 shutdown_connections 2024-05-27T09:39:28.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.736+0000 7f72c87dbb80 1 -- 172.21.15.44:0/4284494388 wait complete. 2024-05-27T09:39:28.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.737+0000 7f72c87dbb80 1 -- start start 2024-05-27T09:39:28.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.737+0000 7f72c87dbb80 1 Processor -- start 2024-05-27T09:39:28.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0e74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 0x5555ef0e79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72c87dbb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0d1d30 0x5555ef0e7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41af530 con 0x5555ef0cbdb0 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41b09a0 con 0x5555eef4ce40 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72c87dbb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41afc00 con 0x5555ef0d1d30 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72c87dbb80 10 client.0 _ll_register_callbacks cb 0x5555eef93c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72b2000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 0x5555ef0e79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72b2000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 0x5555ef0e79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58428/0 (socket says 172.21.15.44:58428) 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72b2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0e74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72b2000700 1 -- 172.21.15.44:0/3146343610 learned_addr learned my addr 172.21.15.44:0/3146343610 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.738+0000 7f72b3400700 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0d1d30 0x5555ef0e7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.739+0000 7f72b2000700 1 -- 172.21.15.44:0/3146343610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0d1d30 msgr2=0x5555ef0e7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:28.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.739+0000 7f72b2000700 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0d1d30 0x5555ef0e7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.739+0000 7f72b2000700 1 -- 172.21.15.44:0/3146343610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 msgr2=0x5555ef0e74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.739+0000 7f72b2000700 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0e74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.739+0000 7f72b2000700 1 -- 172.21.15.44:0/3146343610 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555ef14dd30 con 0x5555ef0cbdb0 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.739+0000 7f72b2000700 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 0x5555ef0e79f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f72a00a5bd0 tx=0x7f72a00e2040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39deb840b2ab4473 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.739+0000 7f729cc00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.739+0000 7f729cc00700 1 -- 172.21.15.44:0/3146343610 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a00c3060 con 0x5555ef0cbdb0 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.739+0000 7f729cc00700 1 -- 172.21.15.44:0/3146343610 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f72a00a6140 con 0x5555ef0cbdb0 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.739+0000 7f72c87dbb80 1 -- 172.21.15.44:0/3146343610 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72ac007e20 con 0x5555ef0cbdb0 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.740+0000 7f729cc00700 1 -- 172.21.15.44:0/3146343610 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f72a00aa250 con 0x5555ef0cbdb0 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.740+0000 7f72c87dbb80 10 client.15822 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.740+0000 7f72c87dbb80 1 -- 172.21.15.44:0/3146343610 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5555ef14eff0 con 0x5555ef0cbdb0 2024-05-27T09:39:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.740+0000 7f729cc00700 1 -- 172.21.15.44:0/3146343610 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72ac007e20 con 0x5555ef0cbdb0 2024-05-27T09:39:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.740+0000 7f729cc00700 1 -- 172.21.15.44:0/3146343610 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f72a0010720 con 0x5555ef0cbdb0 2024-05-27T09:39:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.740+0000 7f72c87dbb80 10 client.15822 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.740+0000 7f72c87dbb80 10 client.15822 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:39:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.740+0000 7f72c87dbb80 20 client.15822 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.741+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5555ef14f6b0 0x5555ef151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.741+0000 7f72c87dbb80 4 client.15822 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:39:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.741+0000 7f72c87dbb80 1 -- 172.21.15.44:0/3146343610 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5555eeff5570 con 0x5555ef14f6b0 2024-05-27T09:39:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.741+0000 7f72b2a00700 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5555ef14f6b0 0x5555ef151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.741+0000 7f72b2a00700 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5555ef14f6b0 0x5555ef151ad0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a31a1211fccb23b2 server_cookie=cf1a18f5b8543c63 in_seq=0 out_seq=0 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.741+0000 7f729cc00700 10 client.15822 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f729cc00700 1 -- 172.21.15.44:0/3146343610 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5555eeff5570 con 0x5555ef14f6b0 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f729cc00700 10 client.15822 handle_command_reply: tid=0 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 1 client.15822 shutdown 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 1 -- 172.21.15.44:0/3146343610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 msgr2=0x5555ef0e79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 0x5555ef0e79f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f72a00a5bd0 tx=0x7f72a00e2040 comp rx=0 tx=0).stop 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 1 -- 172.21.15.44:0/3146343610 shutdown_connections 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555ef0d1d30 0x5555ef0e7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555eef4ce40 0x5555ef0e74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5555ef14f6b0 0x5555ef151ad0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 1 --2- 172.21.15.44:0/3146343610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555ef0cbdb0 0x5555ef0e79f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 1 -- 172.21.15.44:0/3146343610 >> 172.21.15.44:0/3146343610 conn(0x5555eefacfc0 msgr2=0x5555ef0cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:28.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f729cc00700 0 client.15822 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:39:28.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 1 -- 172.21.15.44:0/3146343610 shutdown_connections 2024-05-27T09:39:28.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 1 -- 172.21.15.44:0/3146343610 wait complete. 2024-05-27T09:39:28.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.742+0000 7f72c87dbb80 20 client.15822 trim_cache size 0 max 16384 2024-05-27T09:39:28.746 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:39:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.743+0000 7f72b9c00700 1 -- 172.21.15.44:0/3025263355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72980b2a70 msgr2=0x7f72980b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.743+0000 7f72b9c00700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72980b2a70 0x7f72980b4f80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f72a4006010 tx=0x7f72a400b280 comp rx=0 tx=0).stop 2024-05-27T09:39:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.743+0000 7f72b9c00700 1 -- 172.21.15.44:0/3025263355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b4106eb0 msgr2=0x7f72b419d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.743+0000 7f72b9c00700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b4106eb0 0x7f72b419d610 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f72b4075a20 tx=0x7f72a000eca0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.743+0000 7f72b9c00700 1 -- 172.21.15.44:0/3025263355 shutdown_connections 2024-05-27T09:39:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.743+0000 7f72b9c00700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72980b2a70 0x7f72980b4f80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.743+0000 7f72b9c00700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72b410c5a0 0x7f72b41a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.743+0000 7f72b9c00700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72b4106560 0x7f72b419d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.743+0000 7f72b9c00700 1 --2- 172.21.15.44:0/3025263355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b4106eb0 0x7f72b419d610 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.743+0000 7f72b9c00700 1 -- 172.21.15.44:0/3025263355 >> 172.21.15.44:0/3025263355 conn(0x7f72b41020c0 msgr2=0x7f72b4104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:28.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.744+0000 7f72b9c00700 1 -- 172.21.15.44:0/3025263355 shutdown_connections 2024-05-27T09:39:28.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.744+0000 7f72b9c00700 1 -- 172.21.15.44:0/3025263355 wait complete. 2024-05-27T09:39:28.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.990+0000 7f3f0ec00700 1 -- 172.21.15.44:0/631745778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f0819e380 con 0x7f3f08106560 2024-05-27T09:39:28.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.992+0000 7f3f05600700 1 -- 172.21.15.44:0/631745778 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7f3f0819e380 con 0x7f3f08106560 2024-05-27T09:39:28.993 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:39:28.993 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:39:28.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.996+0000 7f3f0ec00700 1 -- 172.21.15.44:0/631745778 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3eec0b2a70 msgr2=0x7f3eec0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.996+0000 7f3f0ec00700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3eec0b2a70 0x7f3eec0b4f80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0009680 tx=0x7f3ef0009260 comp rx=0 tx=0).stop 2024-05-27T09:39:28.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.996+0000 7f3f0ec00700 1 -- 172.21.15.44:0/631745778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08106560 msgr2=0x7f3f0819cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:28.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.996+0000 7f3f0ec00700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08106560 0x7f3f0819cff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800e970 tx=0x7f3ef800ec80 comp rx=0 tx=0).stop 2024-05-27T09:39:28.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.996+0000 7f3f0ec00700 1 -- 172.21.15.44:0/631745778 shutdown_connections 2024-05-27T09:39:28.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.996+0000 7f3f0ec00700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3eec0b2a70 0x7f3eec0b4f80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.996+0000 7f3f0ec00700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f0810c5a0 0x7f3f081a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.996+0000 7f3f0ec00700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08106560 0x7f3f0819cff0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.996+0000 7f3f0ec00700 1 --2- 172.21.15.44:0/631745778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f08106eb0 0x7f3f0819d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:28.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.996+0000 7f3f0ec00700 1 -- 172.21.15.44:0/631745778 >> 172.21.15.44:0/631745778 conn(0x7f3f08102040 msgr2=0x7f3f08103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:28.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.996+0000 7f3f0ec00700 1 -- 172.21.15.44:0/631745778 shutdown_connections 2024-05-27T09:39:28.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:28.997+0000 7f3f0ec00700 1 -- 172.21.15.44:0/631745778 wait complete. 2024-05-27T09:39:28.998 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:39:29.877 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:39:29.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:29 smithi044 ceph-mon[31618]: pgmap v130: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-05-27T09:39:29.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:29 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/631745778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:39:30.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:29 smithi050 ceph-mon[36371]: pgmap v130: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-05-27T09:39:30.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:29 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/631745778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:39:30.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:29 smithi189 ceph-mon[36671]: pgmap v130: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-05-27T09:39:30.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:29 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/631745778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:39:30.640 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db --set-id f17bf296 --release --await' 2024-05-27T09:39:30.640 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db --set-id f17bf296 --release --await 2024-05-27T09:39:30.749 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:39:31.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:31 smithi044 ceph-mon[31618]: pgmap v131: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-05-27T09:39:32.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:31 smithi050 ceph-mon[36371]: pgmap v131: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-05-27T09:39:32.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:31 smithi189 ceph-mon[36671]: pgmap v131: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-05-27T09:39:32.192 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:39:33.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:33 smithi044 ceph-mon[31618]: pgmap v132: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.2 KiB/s wr, 10 op/s 2024-05-27T09:39:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.642+0000 7f79aac00700 1 -- 172.21.15.44:0/1180194475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a4075d20 msgr2=0x7f79a4076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.642+0000 7f79aac00700 1 --2- 172.21.15.44:0/1180194475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a4075d20 0x7f79a4076130 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7994009990 tx=0x7f7994009ca0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.642+0000 7f79aac00700 1 -- 172.21.15.44:0/1180194475 shutdown_connections 2024-05-27T09:39:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.642+0000 7f79aac00700 1 --2- 172.21.15.44:0/1180194475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a4110b90 0x7f79a4112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.642+0000 7f79aac00700 1 --2- 172.21.15.44:0/1180194475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a4075d20 0x7f79a4076130 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.642+0000 7f79aac00700 1 --2- 172.21.15.44:0/1180194475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79a4076670 0x7f79a4110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.642+0000 7f79aac00700 1 -- 172.21.15.44:0/1180194475 >> 172.21.15.44:0/1180194475 conn(0x7f79a4100030 msgr2=0x7f79a4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.642+0000 7f79aac00700 1 -- 172.21.15.44:0/1180194475 shutdown_connections 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.642+0000 7f79aac00700 1 -- 172.21.15.44:0/1180194475 wait complete. 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.644+0000 7f79aac00700 1 Processor -- start 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.645+0000 7f79aac00700 1 -- start start 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.645+0000 7f79aac00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79a4075d20 0x7f79a419d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.645+0000 7f79aac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a4076670 0x7f79a419d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.645+0000 7f79aac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a4110b90 0x7f79a41a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.645+0000 7f79aac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41166d0 con 0x7f79a4110b90 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.645+0000 7f79aac00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f79a4116550 con 0x7f79a4075d20 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.645+0000 7f79aac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f79a4116850 con 0x7f79a4076670 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.646+0000 7f79a8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a4110b90 0x7f79a41a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.646+0000 7f79a8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a4110b90 0x7f79a41a1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35746/0 (socket says 172.21.15.44:35746) 2024-05-27T09:39:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.646+0000 7f79a8c00700 1 -- 172.21.15.44:0/2705722618 learned_addr learned my addr 172.21.15.44:0/2705722618 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.646+0000 7f79a3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79a4075d20 0x7f79a419d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.646+0000 7f799b400700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a4076670 0x7f79a419d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.646+0000 7f79a8c00700 1 -- 172.21.15.44:0/2705722618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a4076670 msgr2=0x7f79a419d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.646+0000 7f79a8c00700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a4076670 0x7f79a419d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.646+0000 7f79a8c00700 1 -- 172.21.15.44:0/2705722618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79a4075d20 msgr2=0x7f79a419d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.646+0000 7f79a8c00700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79a4075d20 0x7f79a419d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.646+0000 7f79a8c00700 1 -- 172.21.15.44:0/2705722618 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a41a1f10 con 0x7f79a4110b90 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.646+0000 7f79a3e00700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79a4075d20 0x7f79a419d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.647+0000 7f79a8c00700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a4110b90 0x7f79a41a1870 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f799c00c110 tx=0x7f799c00c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.647+0000 7f79a1600700 1 -- 172.21.15.44:0/2705722618 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c0092e0 con 0x7f79a4110b90 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.647+0000 7f79a1600700 1 -- 172.21.15.44:0/2705722618 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f799c003a00 con 0x7f79a4110b90 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.647+0000 7f79aac00700 1 -- 172.21.15.44:0/2705722618 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a41a21a0 con 0x7f79a4110b90 2024-05-27T09:39:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.647+0000 7f79a1600700 1 -- 172.21.15.44:0/2705722618 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c004650 con 0x7f79a4110b90 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.647+0000 7f79aac00700 1 -- 172.21.15.44:0/2705722618 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a41a2690 con 0x7f79a4110b90 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.651+0000 7f79a1600700 1 -- 172.21.15.44:0/2705722618 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f799c01a020 con 0x7f79a4110b90 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.652+0000 7f79b9784b80 1 Processor -- start 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.652+0000 7f79b9784b80 1 -- start start 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79a1600700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79880b29f0 0x7f79880b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec345d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 0x561ec3457870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec3457db0 0x561ec34581c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79b9784b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561ec3385e40 con 0x561ec345dd30 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79b9784b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561ec33759e0 con 0x561ec3457db0 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79a1600700 1 -- 172.21.15.44:0/2705722618 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f799c0e1e70 con 0x7f79a4110b90 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79b9784b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561ec3379a20 con 0x561ec32d8e40 2024-05-27T09:39:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79a3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec345d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79a3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec345d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45874/0 (socket says 172.21.15.44:45874) 2024-05-27T09:39:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f799b400700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79880b29f0 0x7f79880b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79a3e00700 1 -- 172.21.15.44:0/1828422262 learned_addr learned my addr 172.21.15.44:0/1828422262 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.653+0000 7f79a8c00700 1 --2- 172.21.15.44:0/1828422262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec3457db0 0x561ec34581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.654+0000 7f799b400700 1 --2- 172.21.15.44:0/1828422262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 0x561ec3457870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.654+0000 7f79a3e00700 1 -- 172.21.15.44:0/1828422262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec3457db0 msgr2=0x561ec34581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.654+0000 7f79a3e00700 1 --2- 172.21.15.44:0/1828422262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec3457db0 0x561ec34581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.654+0000 7f79a3e00700 1 -- 172.21.15.44:0/1828422262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 msgr2=0x561ec3457870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.654+0000 7f79a3e00700 1 --2- 172.21.15.44:0/1828422262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 0x561ec3457870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.654+0000 7f79a3e00700 1 -- 172.21.15.44:0/1828422262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ec333e420 con 0x561ec32d8e40 2024-05-27T09:39:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.654+0000 7f799b400700 1 --2- 172.21.15.44:0/1828422262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 0x561ec3457870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.654+0000 7f79a3e00700 1 --2- 172.21.15.44:0/1828422262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec345d7f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f79940096a0 tx=0x7f79940096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f10c0740d534252e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.655+0000 7f799b400700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79880b29f0 0x7f79880b4f00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f79a4103430 tx=0x7f798c0099b0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.655+0000 7f7999600700 1 -- 172.21.15.44:0/1828422262 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994005270 con 0x561ec32d8e40 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.655+0000 7f7999600700 1 -- 172.21.15.44:0/1828422262 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7994005410 con 0x561ec32d8e40 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.655+0000 7f7999600700 1 -- 172.21.15.44:0/1828422262 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799400f660 con 0x561ec32d8e40 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.656+0000 7f79b9784b80 1 -- 172.21.15.44:0/1828422262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 msgr2=0x561ec345d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.656+0000 7f79b9784b80 1 --2- 172.21.15.44:0/1828422262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec345d7f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f79940096a0 tx=0x7f79940096d0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.656+0000 7f79b9784b80 1 -- 172.21.15.44:0/1828422262 shutdown_connections 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.656+0000 7f79b9784b80 1 --2- 172.21.15.44:0/1828422262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec3457db0 0x561ec34581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.656+0000 7f79b9784b80 1 --2- 172.21.15.44:0/1828422262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec345d7f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.656+0000 7f79b9784b80 1 --2- 172.21.15.44:0/1828422262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 0x561ec3457870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.656+0000 7f79b9784b80 1 -- 172.21.15.44:0/1828422262 >> 172.21.15.44:0/1828422262 conn(0x561ec3338fc0 msgr2=0x561ec34528d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:34.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.656+0000 7f79b9784b80 1 -- 172.21.15.44:0/1828422262 shutdown_connections 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.656+0000 7f79b9784b80 1 -- 172.21.15.44:0/1828422262 wait complete. 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.656+0000 7f79b9784b80 1 -- start start 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.657+0000 7f79b9784b80 1 Processor -- start 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.657+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec32d8e40 0x561ec3461530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.657+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec3457db0 0x561ec3461a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.657+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec345dd30 0x561ec3461fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.657+0000 7f79b9784b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561ec327bfa0 con 0x561ec32d8e40 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.657+0000 7f79b9784b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561ec3316760 con 0x561ec345dd30 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.657+0000 7f79b9784b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561ec3464740 con 0x561ec3457db0 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.657+0000 7f79b9784b80 10 client.0 _ll_register_callbacks cb 0x561ec331fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.658+0000 7f79a3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec32d8e40 0x561ec3461530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.658+0000 7f79a3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec32d8e40 0x561ec3461530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35778/0 (socket says 172.21.15.44:35778) 2024-05-27T09:39:34.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.658+0000 7f79a3e00700 1 -- 172.21.15.44:0/210118832 learned_addr learned my addr 172.21.15.44:0/210118832 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:34.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.658+0000 7f799b400700 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec3457db0 0x561ec3461a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.658+0000 7f79a8c00700 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec345dd30 0x561ec3461fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.658+0000 7f799b400700 1 -- 172.21.15.44:0/210118832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec345dd30 msgr2=0x561ec3461fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:34.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.658+0000 7f799b400700 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec345dd30 0x561ec3461fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.658+0000 7f799b400700 1 -- 172.21.15.44:0/210118832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec32d8e40 msgr2=0x561ec3461530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:34.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.658+0000 7f799b400700 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec32d8e40 0x561ec3461530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.658+0000 7f799b400700 1 -- 172.21.15.44:0/210118832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ec32c58d0 con 0x561ec3457db0 2024-05-27T09:39:34.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.659+0000 7f799b400700 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec3457db0 0x561ec3461a70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f798c011eb0 tx=0x7f798c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bcc6cb8c4be1e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:34.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.659+0000 7f7992000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.659+0000 7f7992000700 1 -- 172.21.15.44:0/210118832 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c02d060 con 0x561ec3457db0 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.659+0000 7f79b9784b80 1 -- 172.21.15.44:0/210118832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7994005410 con 0x561ec3457db0 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.659+0000 7f7992000700 1 -- 172.21.15.44:0/210118832 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f798c00e840 con 0x561ec3457db0 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.659+0000 7f7992000700 1 -- 172.21.15.44:0/210118832 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f798c00ea40 con 0x561ec3457db0 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.659+0000 7f79b9784b80 10 client.25972 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.659+0000 7f79b9784b80 1 -- 172.21.15.44:0/210118832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561ec3346060 con 0x561ec3457db0 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.659+0000 7f7992000700 1 -- 172.21.15.44:0/210118832 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7994005410 con 0x561ec3457db0 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.660+0000 7f7992000700 1 -- 172.21.15.44:0/210118832 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f798c028ba0 con 0x561ec3457db0 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.660+0000 7f79b9784b80 10 client.25972 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.660+0000 7f79b9784b80 10 client.25972 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.660+0000 7f79b9784b80 20 client.25972 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.660+0000 7f79b9784b80 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561ec3471760 0x561ec3473c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.660+0000 7f79b9784b80 4 client.25972 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.660+0000 7f79b9784b80 1 -- 172.21.15.44:0/210118832 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ec3387d90 con 0x561ec3471760 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.660+0000 7f79a3e00700 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561ec3471760 0x561ec3473c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.661+0000 7f79a3e00700 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561ec3471760 0x561ec3473c00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a606423831b483ad server_cookie=189e369013a7d087 in_seq=0 out_seq=0 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.661+0000 7f7992000700 10 client.25972 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f7992000700 1 -- 172.21.15.44:0/210118832 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561ec3387d90 con 0x561ec3471760 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f7992000700 10 client.25972 handle_command_reply: tid=0 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f79b9784b80 1 client.25972 shutdown 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f79b9784b80 1 -- 172.21.15.44:0/210118832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec3457db0 msgr2=0x561ec3461a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f79b9784b80 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec3457db0 0x561ec3461a70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f798c011eb0 tx=0x7f798c0047c0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f79b9784b80 1 -- 172.21.15.44:0/210118832 shutdown_connections 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f79b9784b80 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec345dd30 0x561ec3461fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f79b9784b80 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561ec3471760 0x561ec3473c00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f79b9784b80 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec32d8e40 0x561ec3461530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f79b9784b80 1 --2- 172.21.15.44:0/210118832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec3457db0 0x561ec3461a70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f79b9784b80 1 -- 172.21.15.44:0/210118832 >> 172.21.15.44:0/210118832 conn(0x561ec3338fc0 msgr2=0x561ec3452580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.662+0000 7f7992000700 0 client.25972 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.663+0000 7f79b9784b80 1 -- 172.21.15.44:0/210118832 shutdown_connections 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.663+0000 7f79b9784b80 1 -- 172.21.15.44:0/210118832 wait complete. 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.663+0000 7f79b9784b80 20 client.25972 trim_cache size 0 max 16384 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.684+0000 7f79b9784b80 1 Processor -- start 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.684+0000 7f79b9784b80 1 -- start start 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.684+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec32d8e40 0x561ec3457870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.684+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec3457db0 0x561ec345f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec345dd30 0x561ec34639a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f79b9784b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41b37c0 con 0x561ec3457db0 2024-05-27T09:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f79b9784b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41b1050 con 0x561ec32d8e40 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f79b9784b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41aeea0 con 0x561ec345dd30 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f799b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec3457db0 0x561ec345f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f79a3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec32d8e40 0x561ec3457870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f799b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec3457db0 0x561ec345f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35786/0 (socket says 172.21.15.44:35786) 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f799b400700 1 -- 172.21.15.44:0/2617722707 learned_addr learned my addr 172.21.15.44:0/2617722707 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f79a8c00700 1 --2- 172.21.15.44:0/2617722707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec345dd30 0x561ec34639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f79a3e00700 1 -- 172.21.15.44:0/2617722707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec345dd30 msgr2=0x561ec34639a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f79a3e00700 1 --2- 172.21.15.44:0/2617722707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec345dd30 0x561ec34639a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f79a3e00700 1 -- 172.21.15.44:0/2617722707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec3457db0 msgr2=0x561ec345f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f79a3e00700 1 --2- 172.21.15.44:0/2617722707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec3457db0 0x561ec345f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.685+0000 7f79a3e00700 1 -- 172.21.15.44:0/2617722707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ec346f130 con 0x561ec32d8e40 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.686+0000 7f79a8c00700 1 --2- 172.21.15.44:0/2617722707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec345dd30 0x561ec34639a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.686+0000 7f799b400700 1 --2- 172.21.15.44:0/2617722707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec3457db0 0x561ec345f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.686+0000 7f79a3e00700 1 --2- 172.21.15.44:0/2617722707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec32d8e40 0x561ec3457870 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7994003d60 tx=0x7f799401a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeb1c30f7638604 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.686+0000 7f7998c00700 1 -- 172.21.15.44:0/2617722707 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994003e60 con 0x561ec32d8e40 2024-05-27T09:39:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.686+0000 7f7998c00700 1 -- 172.21.15.44:0/2617722707 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7994003b30 con 0x561ec32d8e40 2024-05-27T09:39:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.686+0000 7f7998c00700 1 -- 172.21.15.44:0/2617722707 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799400f7c0 con 0x561ec32d8e40 2024-05-27T09:39:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.687+0000 7f79b9784b80 1 -- 172.21.15.44:0/2617722707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec32d8e40 msgr2=0x561ec3457870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.687+0000 7f79b9784b80 1 --2- 172.21.15.44:0/2617722707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec32d8e40 0x561ec3457870 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7994003d60 tx=0x7f799401a040 comp rx=0 tx=0).stop 2024-05-27T09:39:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.687+0000 7f79b9784b80 1 -- 172.21.15.44:0/2617722707 shutdown_connections 2024-05-27T09:39:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.687+0000 7f79b9784b80 1 --2- 172.21.15.44:0/2617722707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec345dd30 0x561ec34639a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.687+0000 7f79b9784b80 1 --2- 172.21.15.44:0/2617722707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec32d8e40 0x561ec3457870 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.687+0000 7f79b9784b80 1 --2- 172.21.15.44:0/2617722707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec3457db0 0x561ec345f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.687+0000 7f79b9784b80 1 -- 172.21.15.44:0/2617722707 >> 172.21.15.44:0/2617722707 conn(0x561ec3338fc0 msgr2=0x561ec34614f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.687+0000 7f79b9784b80 1 -- 172.21.15.44:0/2617722707 shutdown_connections 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.687+0000 7f79b9784b80 1 -- 172.21.15.44:0/2617722707 wait complete. 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.688+0000 7f79b9784b80 1 -- start start 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.688+0000 7f79b9784b80 1 Processor -- start 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec3479c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec3457db0 0x561ec3469990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79b9784b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 0x561ec3469fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79b9784b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41af510 con 0x561ec345dd30 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79b9784b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41b0980 con 0x561ec3457db0 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79b9784b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41afbe0 con 0x561ec32d8e40 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79b9784b80 10 client.0 _ll_register_callbacks cb 0x561ec34609a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:34.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79a8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 0x561ec3469fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79a3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec3479c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79a8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 0x561ec3469fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35796/0 (socket says 172.21.15.44:35796) 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f799b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec3457db0 0x561ec3469990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79a8c00700 1 -- 172.21.15.44:0/653045066 learned_addr learned my addr 172.21.15.44:0/653045066 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.689+0000 7f79a3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec3479c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45890/0 (socket says 172.21.15.44:45890) 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f79a8c00700 1 -- 172.21.15.44:0/653045066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 msgr2=0x561ec3479c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f79a8c00700 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec3479c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f79a8c00700 1 -- 172.21.15.44:0/653045066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec3457db0 msgr2=0x561ec3469990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f79a8c00700 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec3457db0 0x561ec3469990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f79a8c00700 1 -- 172.21.15.44:0/653045066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ec3478420 con 0x561ec345dd30 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f79a3e00700 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec3479c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f79a8c00700 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 0x561ec3469fe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f799c0a6470 tx=0x7f799c0dfaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2a57f7fb45286e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f7992000700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:39:34.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f7992000700 1 -- 172.21.15.44:0/653045066 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c0c2020 con 0x561ec345dd30 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f7992000700 1 -- 172.21.15.44:0/653045066 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f799c0a8460 con 0x561ec345dd30 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f79b9784b80 1 -- 172.21.15.44:0/653045066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7994003b30 con 0x561ec345dd30 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f7992000700 1 -- 172.21.15.44:0/653045066 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f799c0ad4d0 con 0x561ec345dd30 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f79b9784b80 10 client.15840 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.690+0000 7f79b9784b80 1 -- 172.21.15.44:0/653045066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561ec34db590 con 0x561ec345dd30 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.691+0000 7f7992000700 1 -- 172.21.15.44:0/653045066 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7994003b30 con 0x561ec345dd30 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.691+0000 7f7992000700 1 -- 172.21.15.44:0/653045066 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f799c0dfce0 con 0x561ec345dd30 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.691+0000 7f79b9784b80 10 client.15840 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.691+0000 7f79b9784b80 10 client.15840 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.691+0000 7f79b9784b80 20 client.15840 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.691+0000 7f79b9784b80 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561ec34dbc90 0x561ec34de0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.691+0000 7f79b9784b80 4 client.15840 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:39:34.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.691+0000 7f79b9784b80 1 -- 172.21.15.44:0/653045066 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ec3381570 con 0x561ec34dbc90 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.691+0000 7f79a3e00700 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561ec34dbc90 0x561ec34de0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.692+0000 7f79a3e00700 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561ec34dbc90 0x561ec34de0b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d192d2ad3642d477 server_cookie=eeaad2d1e7462da7 in_seq=0 out_seq=0 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.692+0000 7f7992000700 10 client.15840 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.692+0000 7f7992000700 1 -- 172.21.15.44:0/653045066 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ec3381570 con 0x561ec34dbc90 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.692+0000 7f7992000700 10 client.15840 handle_command_reply: tid=0 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.692+0000 7f79b9784b80 1 client.15840 shutdown 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.692+0000 7f79b9784b80 1 -- 172.21.15.44:0/653045066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 msgr2=0x561ec3469fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.692+0000 7f79b9784b80 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 0x561ec3469fe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f799c0a6470 tx=0x7f799c0dfaf0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.693+0000 7f79b9784b80 1 -- 172.21.15.44:0/653045066 shutdown_connections 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.693+0000 7f79b9784b80 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561ec34dbc90 0x561ec34de0b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.693+0000 7f79b9784b80 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561ec345dd30 0x561ec3469fe0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.693+0000 7f79b9784b80 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ec32d8e40 0x561ec3479c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.693+0000 7f79b9784b80 1 --2- 172.21.15.44:0/653045066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ec3457db0 0x561ec3469990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.693+0000 7f7992000700 0 client.15840 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.693+0000 7f79b9784b80 1 -- 172.21.15.44:0/653045066 >> 172.21.15.44:0/653045066 conn(0x561ec3338fc0 msgr2=0x561ec345b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.693+0000 7f79b9784b80 1 -- 172.21.15.44:0/653045066 shutdown_connections 2024-05-27T09:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.693+0000 7f79b9784b80 1 -- 172.21.15.44:0/653045066 wait complete. 2024-05-27T09:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.693+0000 7f79b9784b80 20 client.15840 trim_cache size 0 max 16384 2024-05-27T09:39:34.031 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 -- 172.21.15.44:0/2705722618 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79880b29f0 msgr2=0x7f79880b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79880b29f0 0x7f79880b4f00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f79a4103430 tx=0x7f798c0099b0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 -- 172.21.15.44:0/2705722618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a4110b90 msgr2=0x7f79a41a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a4110b90 0x7f79a41a1870 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f799c00c110 tx=0x7f799c00c420 comp rx=0 tx=0).stop 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 -- 172.21.15.44:0/2705722618 shutdown_connections 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f79880b29f0 0x7f79880b4f00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a4110b90 0x7f79a41a1870 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79a4075d20 0x7f79a419d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 --2- 172.21.15.44:0/2705722618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a4076670 0x7f79a419d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 -- 172.21.15.44:0/2705722618 >> 172.21.15.44:0/2705722618 conn(0x7f79a4100030 msgr2=0x7f79a4100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 -- 172.21.15.44:0/2705722618 shutdown_connections 2024-05-27T09:39:34.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:33.694+0000 7f79aac00700 1 -- 172.21.15.44:0/2705722618 wait complete. 2024-05-27T09:39:34.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:33 smithi050 ceph-mon[36371]: pgmap v132: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.2 KiB/s wr, 10 op/s 2024-05-27T09:39:34.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:33 smithi189 ceph-mon[36671]: pgmap v132: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.2 KiB/s wr, 10 op/s 2024-05-27T09:39:34.747 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:39:35.431 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:39:35.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.498+0000 7f8821800700 1 -- 172.21.15.44:0/2213526940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f881c10e910 msgr2=0x7f881c110dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:35.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.498+0000 7f8821800700 1 --2- 172.21.15.44:0/2213526940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f881c10e910 0x7f881c110dd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8814009990 tx=0x7f8814009ca0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.498+0000 7f8821800700 1 -- 172.21.15.44:0/2213526940 shutdown_connections 2024-05-27T09:39:35.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.498+0000 7f8821800700 1 --2- 172.21.15.44:0/2213526940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f881c10e910 0x7f881c110dd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.498+0000 7f8821800700 1 --2- 172.21.15.44:0/2213526940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881c1054a0 0x7f881c1058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.498+0000 7f8821800700 1 --2- 172.21.15.44:0/2213526940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f881c105df0 0x7f881c10e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.498+0000 7f8821800700 1 -- 172.21.15.44:0/2213526940 >> 172.21.15.44:0/2213526940 conn(0x7f881c0fef10 msgr2=0x7f881c101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:35.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.498+0000 7f8821800700 1 -- 172.21.15.44:0/2213526940 shutdown_connections 2024-05-27T09:39:35.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.498+0000 7f8821800700 1 -- 172.21.15.44:0/2213526940 wait complete. 2024-05-27T09:39:35.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.500+0000 7f8821800700 1 Processor -- start 2024-05-27T09:39:35.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.500+0000 7f8821800700 1 -- start start 2024-05-27T09:39:35.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.501+0000 7f8821800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881c1054a0 0x7f881c19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.501+0000 7f8821800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f881c105df0 0x7f881c19b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.501+0000 7f8821800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f881c10e910 0x7f881c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.501+0000 7f8821800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f881c114470 con 0x7f881c10e910 2024-05-27T09:39:35.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.501+0000 7f8821800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1142f0 con 0x7f881c105df0 2024-05-27T09:39:35.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.501+0000 7f8821800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1145f0 con 0x7f881c1054a0 2024-05-27T09:39:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.501+0000 7f881aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881c1054a0 0x7f881c19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.501+0000 7f881aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881c1054a0 0x7f881c19ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45906/0 (socket says 172.21.15.44:45906) 2024-05-27T09:39:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.501+0000 7f881aa00700 1 -- 172.21.15.44:0/279243236 learned_addr learned my addr 172.21.15.44:0/279243236 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.501+0000 7f881b400700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f881c10e910 0x7f881c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.502+0000 7f8813e00700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f881c105df0 0x7f881c19b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.502+0000 7f881aa00700 1 -- 172.21.15.44:0/279243236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f881c105df0 msgr2=0x7f881c19b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.502+0000 7f881aa00700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f881c105df0 0x7f881c19b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.502+0000 7f881aa00700 1 -- 172.21.15.44:0/279243236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f881c10e910 msgr2=0x7f881c19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.502+0000 7f881aa00700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f881c10e910 0x7f881c19f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.502+0000 7f881aa00700 1 -- 172.21.15.44:0/279243236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881c19fcb0 con 0x7f881c1054a0 2024-05-27T09:39:35.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.502+0000 7f881aa00700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881c1054a0 0x7f881c19ade0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f880c00ef10 tx=0x7f880c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.503+0000 7f8813400700 1 -- 172.21.15.44:0/279243236 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c00ca70 con 0x7f881c1054a0 2024-05-27T09:39:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.503+0000 7f8813400700 1 -- 172.21.15.44:0/279243236 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f880c019070 con 0x7f881c1054a0 2024-05-27T09:39:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.503+0000 7f8821800700 1 -- 172.21.15.44:0/279243236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f881c19ff40 con 0x7f881c1054a0 2024-05-27T09:39:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.503+0000 7f8813400700 1 -- 172.21.15.44:0/279243236 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c014480 con 0x7f881c1054a0 2024-05-27T09:39:35.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.503+0000 7f8821800700 1 -- 172.21.15.44:0/279243236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f881c1a78c0 con 0x7f881c1054a0 2024-05-27T09:39:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.508+0000 7f8813400700 1 -- 172.21.15.44:0/279243236 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f880c012050 con 0x7f881c1054a0 2024-05-27T09:39:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.508+0000 7f883028db80 1 Processor -- start 2024-05-27T09:39:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.508+0000 7f883028db80 1 -- start start 2024-05-27T09:39:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.509+0000 7f883028db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f151c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.509+0000 7f883028db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f0f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.509+0000 7f883028db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.509+0000 7f883028db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d8e4e38f40 con 0x55d8e4f0f850 2024-05-27T09:39:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.509+0000 7f883028db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d8e4e28ae0 con 0x55d8e4f15700 2024-05-27T09:39:35.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.509+0000 7f883028db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d8e4e2cb20 con 0x55d8e4d8bf40 2024-05-27T09:39:35.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.509+0000 7f881b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f151c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.509+0000 7f881aa00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f0f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.509+0000 7f881b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f151c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45914/0 (socket says 172.21.15.44:45914) 2024-05-27T09:39:35.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.509+0000 7f881b400700 1 -- 172.21.15.44:0/2219671009 learned_addr learned my addr 172.21.15.44:0/2219671009 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:35.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.510+0000 7f8813400700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f88000b2a70 0x7f88000b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.510+0000 7f881aa00700 1 -- 172.21.15.44:0/2219671009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 msgr2=0x55d8e4f151c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:35.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.510+0000 7f8813e00700 1 --2- 172.21.15.44:0/2219671009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.510+0000 7f881aa00700 1 --2- 172.21.15.44:0/2219671009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f151c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.510+0000 7f881aa00700 1 -- 172.21.15.44:0/2219671009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 msgr2=0x55d8e4f0fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:35.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.510+0000 7f881aa00700 1 --2- 172.21.15.44:0/2219671009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.510+0000 7f8813400700 1 -- 172.21.15.44:0/279243236 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f880c0e2640 con 0x7f881c1054a0 2024-05-27T09:39:35.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.510+0000 7f881aa00700 1 -- 172.21.15.44:0/2219671009 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8e4df1520 con 0x55d8e4f15700 2024-05-27T09:39:35.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.510+0000 7f881b400700 1 --2- 172.21.15.44:0/2219671009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f151c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:35.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.510+0000 7f8813e00700 1 --2- 172.21.15.44:0/2219671009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:39:35.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.510+0000 7f881b400700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f88000b2a70 0x7f88000b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.511+0000 7f881aa00700 1 --2- 172.21.15.44:0/2219671009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f0f310 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f880c0dfd60 tx=0x7f880c009eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd57a7f2564b1721 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:35.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.511+0000 7f880be00700 1 -- 172.21.15.44:0/2219671009 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c0a57b0 con 0x55d8e4f15700 2024-05-27T09:39:35.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.511+0000 7f880be00700 1 -- 172.21.15.44:0/2219671009 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f880c00cdc0 con 0x55d8e4f15700 2024-05-27T09:39:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.512+0000 7f883028db80 1 -- 172.21.15.44:0/2219671009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 msgr2=0x55d8e4f0f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.512+0000 7f881b400700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f88000b2a70 0x7f88000b4f80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x55d8e4f10500 tx=0x7f8814005040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.512+0000 7f883028db80 1 --2- 172.21.15.44:0/2219671009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f0f310 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f880c0dfd60 tx=0x7f880c009eb0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.512+0000 7f883028db80 1 -- 172.21.15.44:0/2219671009 shutdown_connections 2024-05-27T09:39:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.512+0000 7f883028db80 1 --2- 172.21.15.44:0/2219671009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.512+0000 7f883028db80 1 --2- 172.21.15.44:0/2219671009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f151c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.512+0000 7f883028db80 1 --2- 172.21.15.44:0/2219671009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f0f310 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.512+0000 7f883028db80 1 -- 172.21.15.44:0/2219671009 >> 172.21.15.44:0/2219671009 conn(0x55d8e4dec0c0 msgr2=0x55d8e4f06100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.512+0000 7f883028db80 1 -- 172.21.15.44:0/2219671009 shutdown_connections 2024-05-27T09:39:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.512+0000 7f883028db80 1 -- 172.21.15.44:0/2219671009 wait complete. 2024-05-27T09:39:35.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.513+0000 7f883028db80 1 -- start start 2024-05-27T09:39:35.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.513+0000 7f883028db80 1 Processor -- start 2024-05-27T09:39:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f883028db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f0e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f883028db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f883028db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f0f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f883028db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d8e4d2f0a0 con 0x55d8e4f0f850 2024-05-27T09:39:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f883028db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d8e4dc9860 con 0x55d8e4f15700 2024-05-27T09:39:35.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f883028db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d8e4f0b080 con 0x55d8e4d8bf40 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f883028db80 10 client.0 _ll_register_callbacks cb 0x55d8e4dd2d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f8813e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f881b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f0f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f8813e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35832/0 (socket says 172.21.15.44:35832) 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f8813e00700 1 -- 172.21.15.44:0/1407897868 learned_addr learned my addr 172.21.15.44:0/1407897868 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.514+0000 7f881aa00700 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f0e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.515+0000 7f881b400700 1 -- 172.21.15.44:0/1407897868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 msgr2=0x55d8e4f0e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.515+0000 7f881b400700 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f0e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.515+0000 7f881b400700 1 -- 172.21.15.44:0/1407897868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 msgr2=0x55d8e4f0ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.515+0000 7f881b400700 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.515+0000 7f881b400700 1 -- 172.21.15.44:0/1407897868 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8e4d789d0 con 0x55d8e4f15700 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.515+0000 7f8813e00700 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:35.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.515+0000 7f881b400700 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f0f250 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f88140059d0 tx=0x7f8814005960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b75fec7517e272d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:35.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.515+0000 7f8808c00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:39:35.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.515+0000 7f8808c00700 1 -- 172.21.15.44:0/1407897868 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8814003880 con 0x55d8e4f15700 2024-05-27T09:39:35.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.516+0000 7f8808c00700 1 -- 172.21.15.44:0/1407897868 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f8814003a20 con 0x55d8e4f15700 2024-05-27T09:39:35.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.516+0000 7f883028db80 1 -- 172.21.15.44:0/1407897868 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d8e4e3ae90 con 0x55d8e4f15700 2024-05-27T09:39:35.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.516+0000 7f8808c00700 1 -- 172.21.15.44:0/1407897868 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f881400be60 con 0x55d8e4f15700 2024-05-27T09:39:35.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.516+0000 7f883028db80 10 client.25976 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:35.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.516+0000 7f883028db80 1 -- 172.21.15.44:0/1407897868 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d8e4df9160 con 0x55d8e4f15700 2024-05-27T09:39:35.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.516+0000 7f8808c00700 1 -- 172.21.15.44:0/1407897868 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d8e4e3ae90 con 0x55d8e4f15700 2024-05-27T09:39:35.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.517+0000 7f8808c00700 1 -- 172.21.15.44:0/1407897868 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8814038020 con 0x55d8e4f15700 2024-05-27T09:39:35.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.517+0000 7f883028db80 10 client.25976 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:35.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.517+0000 7f883028db80 10 client.25976 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:39:35.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.517+0000 7f883028db80 20 client.25976 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:35.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.517+0000 7f883028db80 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d8e4f249e0 0x55d8e4f26e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.517+0000 7f883028db80 4 client.25976 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:39:35.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.517+0000 7f883028db80 1 -- 172.21.15.44:0/1407897868 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8e4e34670 con 0x55d8e4f249e0 2024-05-27T09:39:35.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.518+0000 7f881aa00700 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d8e4f249e0 0x55d8e4f26e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.518+0000 7f881aa00700 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d8e4f249e0 0x55d8e4f26e40 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d88d0052bff6273 server_cookie=2d35bf0e124e15b8 in_seq=0 out_seq=0 2024-05-27T09:39:35.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.518+0000 7f8808c00700 10 client.25976 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:35.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f8808c00700 1 -- 172.21.15.44:0/1407897868 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d8e4e34670 con 0x55d8e4f249e0 2024-05-27T09:39:35.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f8808c00700 10 client.25976 handle_command_reply: tid=0 2024-05-27T09:39:35.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f883028db80 1 client.25976 shutdown 2024-05-27T09:39:35.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f883028db80 1 -- 172.21.15.44:0/1407897868 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 msgr2=0x55d8e4f0f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:35.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f883028db80 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f0f250 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f88140059d0 tx=0x7f8814005960 comp rx=0 tx=0).stop 2024-05-27T09:39:35.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f883028db80 1 -- 172.21.15.44:0/1407897868 shutdown_connections 2024-05-27T09:39:35.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f883028db80 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d8e4f249e0 0x55d8e4f26e40 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f883028db80 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f0f250 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f883028db80 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f0e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f883028db80 1 --2- 172.21.15.44:0/1407897868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f883028db80 1 -- 172.21.15.44:0/1407897868 >> 172.21.15.44:0/1407897868 conn(0x55d8e4dec0c0 msgr2=0x55d8e4f09d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:35.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.519+0000 7f8808c00700 0 client.25976 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:35.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.520+0000 7f883028db80 1 -- 172.21.15.44:0/1407897868 shutdown_connections 2024-05-27T09:39:35.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.520+0000 7f883028db80 1 -- 172.21.15.44:0/1407897868 wait complete. 2024-05-27T09:39:35.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.520+0000 7f883028db80 20 client.25976 trim_cache size 0 max 16384 2024-05-27T09:39:35.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.542+0000 7f883028db80 1 Processor -- start 2024-05-27T09:39:35.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.542+0000 7f883028db80 1 -- start start 2024-05-27T09:39:35.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.543+0000 7f883028db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f0a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.543+0000 7f883028db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.543+0000 7f883028db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f19230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.543+0000 7f883028db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1af4d0 con 0x55d8e4f0f850 2024-05-27T09:39:35.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.543+0000 7f883028db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1acd60 con 0x55d8e4f15700 2024-05-27T09:39:35.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.543+0000 7f883028db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1aabd0 con 0x55d8e4d8bf40 2024-05-27T09:39:35.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.543+0000 7f881aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f0a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.543+0000 7f8813e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.543+0000 7f881aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f0a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45934/0 (socket says 172.21.15.44:45934) 2024-05-27T09:39:35.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.543+0000 7f881aa00700 1 -- 172.21.15.44:0/837118666 learned_addr learned my addr 172.21.15.44:0/837118666 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:35.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.543+0000 7f881b400700 1 --2- 172.21.15.44:0/837118666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f19230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.544+0000 7f881aa00700 1 -- 172.21.15.44:0/837118666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 msgr2=0x55d8e4f19230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:35.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.544+0000 7f881aa00700 1 --2- 172.21.15.44:0/837118666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f19230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.544+0000 7f881aa00700 1 -- 172.21.15.44:0/837118666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 msgr2=0x55d8e4f0c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:35.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.544+0000 7f881aa00700 1 --2- 172.21.15.44:0/837118666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.544+0000 7f881aa00700 1 -- 172.21.15.44:0/837118666 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8e4f01e30 con 0x55d8e4d8bf40 2024-05-27T09:39:35.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.544+0000 7f8813e00700 1 --2- 172.21.15.44:0/837118666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:39:35.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.544+0000 7f881aa00700 1 --2- 172.21.15.44:0/837118666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f0a3e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f880c0a9880 tx=0x7f880c00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e623fe67f19a8044 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.544+0000 7f880b400700 1 -- 172.21.15.44:0/837118666 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c0a9240 con 0x55d8e4d8bf40 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.544+0000 7f880b400700 1 -- 172.21.15.44:0/837118666 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f880c0b2090 con 0x55d8e4d8bf40 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.544+0000 7f880b400700 1 -- 172.21.15.44:0/837118666 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c0a96c0 con 0x55d8e4d8bf40 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.545+0000 7f883028db80 1 -- 172.21.15.44:0/837118666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 msgr2=0x55d8e4f0a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.545+0000 7f883028db80 1 --2- 172.21.15.44:0/837118666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f0a3e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f880c0a9880 tx=0x7f880c00c8d0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.545+0000 7f883028db80 1 -- 172.21.15.44:0/837118666 shutdown_connections 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.545+0000 7f883028db80 1 --2- 172.21.15.44:0/837118666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4f15700 0x55d8e4f19230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.545+0000 7f883028db80 1 --2- 172.21.15.44:0/837118666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f0a3e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.545+0000 7f883028db80 1 --2- 172.21.15.44:0/837118666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f0f850 0x55d8e4f0c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.545+0000 7f883028db80 1 -- 172.21.15.44:0/837118666 >> 172.21.15.44:0/837118666 conn(0x55d8e4dec0c0 msgr2=0x55d8e4f0e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.545+0000 7f883028db80 1 -- 172.21.15.44:0/837118666 shutdown_connections 2024-05-27T09:39:35.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.546+0000 7f883028db80 1 -- 172.21.15.44:0/837118666 wait complete. 2024-05-27T09:39:35.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.546+0000 7f883028db80 1 -- start start 2024-05-27T09:39:35.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.546+0000 7f883028db80 1 Processor -- start 2024-05-27T09:39:35.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.547+0000 7f883028db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f21810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.547+0000 7f883028db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4f0f850 0x55d8e4f2c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.547+0000 7f883028db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f15700 0x55d8e4f2c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.547+0000 7f883028db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1ab220 con 0x55d8e4f15700 2024-05-27T09:39:35.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.547+0000 7f883028db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1ac690 con 0x55d8e4d8bf40 2024-05-27T09:39:35.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.547+0000 7f883028db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1ab8f0 con 0x55d8e4f0f850 2024-05-27T09:39:35.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.547+0000 7f883028db80 10 client.0 _ll_register_callbacks cb 0x55d8e4dd2d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:35.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.547+0000 7f8813e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4f0f850 0x55d8e4f2c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.547+0000 7f8813e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4f0f850 0x55d8e4f2c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45946/0 (socket says 172.21.15.44:45946) 2024-05-27T09:39:35.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.547+0000 7f8813e00700 1 -- 172.21.15.44:0/1107107760 learned_addr learned my addr 172.21.15.44:0/1107107760 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:35.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.547+0000 7f881aa00700 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f21810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.548+0000 7f8813e00700 1 -- 172.21.15.44:0/1107107760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4d8bf40 msgr2=0x55d8e4f21810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:35.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.548+0000 7f8813e00700 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f21810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.548+0000 7f8813e00700 1 -- 172.21.15.44:0/1107107760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f15700 msgr2=0x55d8e4f2c820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.548+0000 7f8813e00700 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f15700 0x55d8e4f2c820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.548+0000 7f8813e00700 1 -- 172.21.15.44:0/1107107760 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8e4f2cfc0 con 0x55d8e4f0f850 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.548+0000 7f881aa00700 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f21810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.548+0000 7f8813e00700 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4f0f850 0x55d8e4f2c2a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8804003930 tx=0x7f8804003c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dc9534321cdfc38 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.548+0000 7f8808c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.548+0000 7f8808c00700 1 -- 172.21.15.44:0/1107107760 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8804021060 con 0x55d8e4f0f850 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.548+0000 7f8808c00700 1 -- 172.21.15.44:0/1107107760 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f8804012920 con 0x55d8e4f0f850 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.548+0000 7f883028db80 1 -- 172.21.15.44:0/1107107760 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d8e4cab350 con 0x55d8e4f0f850 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.549+0000 7f8808c00700 1 -- 172.21.15.44:0/1107107760 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8804012b20 con 0x55d8e4f0f850 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.549+0000 7f883028db80 10 client.25996 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.549+0000 7f883028db80 1 -- 172.21.15.44:0/1107107760 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d8e4f8f4e0 con 0x55d8e4f0f850 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.549+0000 7f8808c00700 1 -- 172.21.15.44:0/1107107760 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d8e4cab350 con 0x55d8e4f0f850 2024-05-27T09:39:35.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.549+0000 7f8808c00700 1 -- 172.21.15.44:0/1107107760 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f880401b070 con 0x55d8e4f0f850 2024-05-27T09:39:35.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.550+0000 7f883028db80 10 client.25996 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:35.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.550+0000 7f883028db80 10 client.25996 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:39:35.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.550+0000 7f883028db80 20 client.25996 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:35.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.550+0000 7f883028db80 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d8e4f8fc50 0x55d8e4f92070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:35.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.550+0000 7f883028db80 4 client.25996 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f17bf296", "release": true, "await": true}] 2024-05-27T09:39:35.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.550+0000 7f883028db80 1 -- 172.21.15.44:0/1107107760 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f17bf296", "release": true, "await": true}) -- 0x55d8e4dfcc90 con 0x55d8e4f8fc50 2024-05-27T09:39:35.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.551+0000 7f881aa00700 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d8e4f8fc50 0x55d8e4f92070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:35.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.551+0000 7f881aa00700 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d8e4f8fc50 0x55d8e4f92070 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c92912deaa8aaea5 server_cookie=c4cf36dfecfdd71e in_seq=0 out_seq=0 2024-05-27T09:39:35.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.551+0000 7f8808c00700 10 client.25996 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:35.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f8808c00700 1 -- 172.21.15.44:0/1107107760 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55d8e4dfcc90 con 0x55d8e4f8fc50 2024-05-27T09:39:35.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f8808c00700 10 client.25996 handle_command_reply: tid=0 2024-05-27T09:39:35.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f883028db80 1 client.25996 shutdown 2024-05-27T09:39:35.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f883028db80 1 -- 172.21.15.44:0/1107107760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4f0f850 msgr2=0x55d8e4f2c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:35.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f883028db80 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4f0f850 0x55d8e4f2c2a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8804003930 tx=0x7f8804003c40 comp rx=0 tx=0).stop 2024-05-27T09:39:35.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f883028db80 1 -- 172.21.15.44:0/1107107760 shutdown_connections 2024-05-27T09:39:35.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f883028db80 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d8e4f8fc50 0x55d8e4f92070 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f883028db80 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8e4f15700 0x55d8e4f2c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f883028db80 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8e4d8bf40 0x55d8e4f21810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f883028db80 1 --2- 172.21.15.44:0/1107107760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8e4f0f850 0x55d8e4f2c2a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f8808c00700 0 client.25996 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:35.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.554+0000 7f883028db80 1 -- 172.21.15.44:0/1107107760 >> 172.21.15.44:0/1107107760 conn(0x55d8e4dec0c0 msgr2=0x55d8e4f0efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:35.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.555+0000 7f883028db80 1 -- 172.21.15.44:0/1107107760 shutdown_connections 2024-05-27T09:39:35.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.555+0000 7f883028db80 1 -- 172.21.15.44:0/1107107760 wait complete. 2024-05-27T09:39:35.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.555+0000 7f883028db80 20 client.25996 trim_cache size 0 max 16384 2024-05-27T09:39:35.557 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:39:35.557 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 26, 2024-05-27T09:39:35.557 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:39:35.557 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 8, 2024-05-27T09:39:35.557 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:39:35.557 INFO:teuthology.orchestra.run.smithi044.stdout: "f17bf296": { 2024-05-27T09:39:35.557 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 8, 2024-05-27T09:39:35.557 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:39:35.557 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:39:35.557 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 130.0, 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:39:35.558 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:39:35.559 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:39:35.559 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:39:35.559 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:39:35.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 -- 172.21.15.44:0/279243236 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f88000b2a70 msgr2=0x7f88000b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:35.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f88000b2a70 0x7f88000b4f80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x55d8e4f10500 tx=0x7f8814005040 comp rx=0 tx=0).stop 2024-05-27T09:39:35.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 -- 172.21.15.44:0/279243236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881c1054a0 msgr2=0x7f881c19ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:35.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881c1054a0 0x7f881c19ade0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f880c00ef10 tx=0x7f880c00c3a0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 -- 172.21.15.44:0/279243236 shutdown_connections 2024-05-27T09:39:35.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f88000b2a70 0x7f88000b4f80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f881c10e910 0x7f881c19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881c1054a0 0x7f881c19ade0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 --2- 172.21.15.44:0/279243236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f881c105df0 0x7f881c19b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:35.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 -- 172.21.15.44:0/279243236 >> 172.21.15.44:0/279243236 conn(0x7f881c0fef10 msgr2=0x7f881c0ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:35.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 -- 172.21.15.44:0/279243236 shutdown_connections 2024-05-27T09:39:35.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:35.556+0000 7f8821800700 1 -- 172.21.15.44:0/279243236 wait complete. 2024-05-27T09:39:35.561 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:35.561 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:35.561 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 63 -lt 100 ']' 2024-05-27T09:39:35.561 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 63 + 1 2024-05-27T09:39:35.662 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=64 2024-05-27T09:39:35.662 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:35.714 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:35.715 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:35.720 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:35.721 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:35.722 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:35.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:35 smithi044 ceph-mon[31618]: pgmap v133: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.2 KiB/s wr, 10 op/s 2024-05-27T09:39:36.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:35 smithi050 ceph-mon[36371]: pgmap v133: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.2 KiB/s wr, 10 op/s 2024-05-27T09:39:36.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:35 smithi189 ceph-mon[36671]: pgmap v133: 97 pgs: 97 active+clean; 2.9 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.2 KiB/s wr, 10 op/s 2024-05-27T09:39:36.228 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:36.229 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:36.519 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:36.519 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:36.561 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:36.561 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:36.561 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 64 -lt 100 ']' 2024-05-27T09:39:36.562 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 64 + 1 2024-05-27T09:39:36.595 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=65 2024-05-27T09:39:36.596 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:36.662 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:36.662 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:36.667 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:36.668 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:36.669 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:37.820 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:37 smithi044 ceph-mon[31618]: pgmap v134: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-05-27T09:39:37.881 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'f17bf296', total seconds elapsed: 88.4 2024-05-27T09:39:38.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:37 smithi050 ceph-mon[36371]: pgmap v134: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-05-27T09:39:38.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:37 smithi189 ceph-mon[36671]: pgmap v134: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-05-27T09:39:38.192 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:38.193 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:38.485 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:38.485 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:38.507 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:38.507 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:38.507 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 65 -lt 100 ']' 2024-05-27T09:39:38.508 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 65 + 1 2024-05-27T09:39:38.572 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=66 2024-05-27T09:39:38.572 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:38.656 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:38.657 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:38.661 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:38.662 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:38.663 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:39.577 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:39.578 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:39 smithi044 ceph-mon[31618]: pgmap v135: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-05-27T09:39:40.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:39 smithi050 ceph-mon[36371]: pgmap v135: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-05-27T09:39:40.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:39 smithi189 ceph-mon[36671]: pgmap v135: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-05-27T09:39:40.135 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:40.135 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:40.246 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:40.246 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:40.246 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 66 -lt 100 ']' 2024-05-27T09:39:40.247 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 66 + 1 2024-05-27T09:39:40.281 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=67 2024-05-27T09:39:40.281 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:40.354 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:40.355 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:40.359 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:40.360 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:40.361 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:41.475 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:41.476 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:41.868 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:41.868 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:41.905 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:41.905 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:41.905 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 67 -lt 100 ']' 2024-05-27T09:39:41.906 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 67 + 1 2024-05-27T09:39:41.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:41 smithi044 ceph-mon[31618]: pgmap v136: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-05-27T09:39:42.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:41 smithi050 ceph-mon[36371]: pgmap v136: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-05-27T09:39:42.058 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=68 2024-05-27T09:39:42.058 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:42.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:41 smithi189 ceph-mon[36671]: pgmap v136: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-05-27T09:39:42.098 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:42.099 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:42.104 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:42.105 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:42.106 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:43.410 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:43.411 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:43.675 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:43.676 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:43.725 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:43.725 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:43.725 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 68 -lt 100 ']' 2024-05-27T09:39:43.726 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 68 + 1 2024-05-27T09:39:43.913 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=69 2024-05-27T09:39:43.913 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:43.946 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:43.947 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:43.951 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:43.952 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:43.952 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:43.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:43 smithi044 ceph-mon[31618]: pgmap v137: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.1 KiB/s wr, 6 op/s 2024-05-27T09:39:44.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:43 smithi050 ceph-mon[36371]: pgmap v137: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.1 KiB/s wr, 6 op/s 2024-05-27T09:39:44.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:43 smithi189 ceph-mon[36671]: pgmap v137: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.1 KiB/s wr, 6 op/s 2024-05-27T09:39:44.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:44 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:39:45.047 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:44 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:39:45.097 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:44 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:39:45.153 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:45.154 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:45.601 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.599+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3156248482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f9c104260 msgr2=0x7f8f9c105680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:45.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.599+0000 7f8fa2a00640 1 --2- 172.21.15.44:0/3156248482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f9c104260 0x7f8f9c105680 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8f88009860 tx=0x7f8f8802a3d0 comp rx=0 tx=0).stop 2024-05-27T09:39:45.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.599+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3156248482 shutdown_connections 2024-05-27T09:39:45.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.599+0000 7f8fa2a00640 1 --2- 172.21.15.44:0/3156248482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f9c105cf0 0x7f8f9c110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:45.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.599+0000 7f8fa2a00640 1 --2- 172.21.15.44:0/3156248482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f9c104260 0x7f8f9c105680 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:45.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.599+0000 7f8fa2a00640 1 --2- 172.21.15.44:0/3156248482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f9c103e50 0x7f8f9c105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:45.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.599+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3156248482 >> 172.21.15.44:0/3156248482 conn(0x7f8f9c0791a0 msgr2=0x7f8f9c0795e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:45.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.600+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3156248482 shutdown_connections 2024-05-27T09:39:45.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.600+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3156248482 wait complete. 2024-05-27T09:39:45.625 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:45.626 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:45.666 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:45.666 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:45.666 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 69 -lt 100 ']' 2024-05-27T09:39:45.667 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 69 + 1 2024-05-27T09:39:45.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.672+0000 7f8fa2a00640 1 Processor -- start 2024-05-27T09:39:45.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.672+0000 7f8fa2a00640 1 -- start start 2024-05-27T09:39:45.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.673+0000 7f8fa2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f9c103e50 0x7f8f9c19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:45.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.673+0000 7f8fa2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f9c104260 0x7f8f9c19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:45.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.673+0000 7f8fa2a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f9c105cf0 0x7f8f9c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:45.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.673+0000 7f8fa2a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c113db0 con 0x7f8f9c104260 2024-05-27T09:39:45.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.673+0000 7f8fa2a00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c113c30 con 0x7f8f9c105cf0 2024-05-27T09:39:45.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.673+0000 7f8fa2a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c113f30 con 0x7f8f9c103e50 2024-05-27T09:39:45.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.673+0000 7f8fa2000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f9c105cf0 0x7f8f9c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:45.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.673+0000 7f8fa1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f9c103e50 0x7f8f9c19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:45.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.673+0000 7f8fa0c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f9c104260 0x7f8f9c19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:45.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.674+0000 7f8fa2000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f9c105cf0 0x7f8f9c19efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38836/0 (socket says 172.21.15.44:38836) 2024-05-27T09:39:45.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.674+0000 7f8fa2000640 1 -- 172.21.15.44:0/3464194589 learned_addr learned my addr 172.21.15.44:0/3464194589 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:45.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.674+0000 7f8fa1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f9c103e50 0x7f8f9c19a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47228/0 (socket says 172.21.15.44:47228) 2024-05-27T09:39:45.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.674+0000 7f8fa2000640 1 -- 172.21.15.44:0/3464194589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f9c103e50 msgr2=0x7f8f9c19a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:45.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.674+0000 7f8fa2000640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f9c103e50 0x7f8f9c19a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:45.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.674+0000 7f8fa2000640 1 -- 172.21.15.44:0/3464194589 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f9c104260 msgr2=0x7f8f9c19ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:45.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.674+0000 7f8fa2000640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f9c104260 0x7f8f9c19ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:45.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.674+0000 7f8fa2000640 1 -- 172.21.15.44:0/3464194589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f9c19f610 con 0x7f8f9c105cf0 2024-05-27T09:39:45.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.674+0000 7f8fa1600640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f9c103e50 0x7f8f9c19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:45.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.674+0000 7f8fa0c00640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f9c104260 0x7f8f9c19ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:39:45.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.675+0000 7f8fa2000640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f9c105cf0 0x7f8f9c19efd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009fa0 tx=0x7f8f9400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.675+0000 7f8f9a000640 1 -- 172.21.15.44:0/3464194589 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f94004b20 con 0x7f8f9c105cf0 2024-05-27T09:39:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.675+0000 7f8f9a000640 1 -- 172.21.15.44:0/3464194589 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f94003e20 con 0x7f8f9c105cf0 2024-05-27T09:39:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.676+0000 7f8f9a000640 1 -- 172.21.15.44:0/3464194589 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9400b570 con 0x7f8f9c105cf0 2024-05-27T09:39:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.676+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3464194589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f9c19f900 con 0x7f8f9c105cf0 2024-05-27T09:39:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.676+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3464194589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f9c19fe00 con 0x7f8f9c105cf0 2024-05-27T09:39:45.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.682+0000 7f8f9a000640 1 -- 172.21.15.44:0/3464194589 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f8f9401d020 con 0x7f8f9c105cf0 2024-05-27T09:39:45.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.684+0000 7f8f9a000640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8f840b2a20 0x7f8f840b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:45.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.684+0000 7f8f9a000640 1 -- 172.21.15.44:0/3464194589 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f8f940e2520 con 0x7f8f9c105cf0 2024-05-27T09:39:45.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.685+0000 7f8fa1600640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8f840b2a20 0x7f8f840b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:45.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.686+0000 7f8fa1600640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8f840b2a20 0x7f8f840b4f30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8f900095f0 tx=0x7f8f90012040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:45.742 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=70 2024-05-27T09:39:45.742 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:45.872 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:45.873 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:45.878 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:45.879 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:45.879 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:45.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:45 smithi044 ceph-mon[31618]: pgmap v138: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.1 KiB/s wr, 6 op/s 2024-05-27T09:39:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.974+0000 7f8faaf85740 1 Processor -- start 2024-05-27T09:39:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.974+0000 7f8faaf85740 1 -- start start 2024-05-27T09:39:45.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.975+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662ad141b0 0x55662b0b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:45.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.975+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662aceb2f0 0x55662b0b6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:45.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.975+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662acf34d0 0x55662b0b74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:45.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.975+0000 7f8faaf85740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55662b027b90 con 0x55662acf34d0 2024-05-27T09:39:45.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.975+0000 7f8faaf85740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55662b017730 con 0x55662aceb2f0 2024-05-27T09:39:45.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.975+0000 7f8faaf85740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55662b01b770 con 0x55662ad141b0 2024-05-27T09:39:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.976+0000 7f8fa1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662ad141b0 0x55662b0b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.976+0000 7f8fa0c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662aceb2f0 0x55662b0b6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.976+0000 7f8fa0c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662aceb2f0 0x55662b0b6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38848/0 (socket says 172.21.15.44:38848) 2024-05-27T09:39:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.976+0000 7f8fa0c00640 1 -- 172.21.15.44:0/3017900309 learned_addr learned my addr 172.21.15.44:0/3017900309 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.976+0000 7f8fa1600640 1 --2- 172.21.15.44:0/3017900309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662ad141b0 0x55662b0b6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47230/0 (socket says 172.21.15.44:47230) 2024-05-27T09:39:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.976+0000 7f8fa2000640 1 --2- 172.21.15.44:0/3017900309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662acf34d0 0x55662b0b74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.977+0000 7f8fa0c00640 1 -- 172.21.15.44:0/3017900309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662ad141b0 msgr2=0x55662b0b6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.977+0000 7f8fa0c00640 1 --2- 172.21.15.44:0/3017900309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662ad141b0 0x55662b0b6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.977+0000 7f8fa0c00640 1 -- 172.21.15.44:0/3017900309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662acf34d0 msgr2=0x55662b0b74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.977+0000 7f8fa0c00640 1 --2- 172.21.15.44:0/3017900309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662acf34d0 0x55662b0b74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.977+0000 7f8fa0c00640 1 -- 172.21.15.44:0/3017900309 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55662afb08b0 con 0x55662aceb2f0 2024-05-27T09:39:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.978+0000 7f8fa0c00640 1 --2- 172.21.15.44:0/3017900309 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662aceb2f0 0x55662b0b6fa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8f88009780 tx=0x7f8f88030640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21d27a23fb586066 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.979+0000 7f8f8e600640 1 -- 172.21.15.44:0/3017900309 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f88004250 con 0x55662aceb2f0 2024-05-27T09:39:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.979+0000 7f8f8e600640 1 -- 172.21.15.44:0/3017900309 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f88040030 con 0x55662aceb2f0 2024-05-27T09:39:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.979+0000 7f8f8e600640 1 -- 172.21.15.44:0/3017900309 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f880055b0 con 0x55662aceb2f0 2024-05-27T09:39:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.981+0000 7f8faaf85740 1 -- 172.21.15.44:0/3017900309 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662aceb2f0 msgr2=0x55662b0b6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.981+0000 7f8faaf85740 1 --2- 172.21.15.44:0/3017900309 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662aceb2f0 0x55662b0b6fa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8f88009780 tx=0x7f8f88030640 comp rx=0 tx=0).stop 2024-05-27T09:39:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.981+0000 7f8faaf85740 1 -- 172.21.15.44:0/3017900309 shutdown_connections 2024-05-27T09:39:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.981+0000 7f8faaf85740 1 --2- 172.21.15.44:0/3017900309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662acf34d0 0x55662b0b74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.981+0000 7f8faaf85740 1 --2- 172.21.15.44:0/3017900309 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662aceb2f0 0x55662b0b6fa0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.981+0000 7f8faaf85740 1 --2- 172.21.15.44:0/3017900309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662ad141b0 0x55662b0b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.981+0000 7f8faaf85740 1 -- 172.21.15.44:0/3017900309 >> 172.21.15.44:0/3017900309 conn(0x55662ad08300 msgr2=0x55662b0b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.981+0000 7f8faaf85740 1 -- 172.21.15.44:0/3017900309 shutdown_connections 2024-05-27T09:39:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:45.981+0000 7f8faaf85740 1 -- 172.21.15.44:0/3017900309 wait complete. 2024-05-27T09:39:45.991 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:45 smithi189 ceph-mon[36671]: pgmap v138: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.1 KiB/s wr, 6 op/s 2024-05-27T09:39:46.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:45 smithi050 ceph-mon[36371]: pgmap v138: 97 pgs: 97 active+clean; 3.0 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.1 KiB/s wr, 6 op/s 2024-05-27T09:39:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.049+0000 7f8faaf85740 1 -- start start 2024-05-27T09:39:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.050+0000 7f8faaf85740 1 Processor -- start 2024-05-27T09:39:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 0x55662b0c0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662acf34d0 0x55662b0c1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662ad141b0 0x55662b0cce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8faaf85740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55662acfd790 con 0x55662ad141b0 2024-05-27T09:39:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8faaf85740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55662afbfbc0 con 0x55662acf34d0 2024-05-27T09:39:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8faaf85740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55662b0c3b30 con 0x55662aceb2f0 2024-05-27T09:39:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8fa2000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662ad141b0 0x55662b0cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8fa1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 0x55662b0c0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8fa0c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662acf34d0 0x55662b0c1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8fa0c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662acf34d0 0x55662b0c1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38858/0 (socket says 172.21.15.44:38858) 2024-05-27T09:39:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8fa0c00640 1 -- 172.21.15.44:0/3690039357 learned_addr learned my addr 172.21.15.44:0/3690039357 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.052+0000 7f8fa1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 0x55662b0c0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47232/0 (socket says 172.21.15.44:47232) 2024-05-27T09:39:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.053+0000 7f8fa2000640 1 -- 172.21.15.44:0/3690039357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 msgr2=0x55662b0c0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.053+0000 7f8fa2000640 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 0x55662b0c0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.053+0000 7f8fa2000640 1 -- 172.21.15.44:0/3690039357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662acf34d0 msgr2=0x55662b0c1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.053+0000 7f8fa2000640 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662acf34d0 0x55662b0c1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.053+0000 7f8faaf85740 10 client.0 _ll_register_callbacks cb 0x55662af9d400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.053+0000 7f8fa2000640 1 -- 172.21.15.44:0/3690039357 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55662ad2c210 con 0x55662ad141b0 2024-05-27T09:39:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.053+0000 7f8fa0c00640 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662acf34d0 0x55662b0c1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.053+0000 7f8faaf85740 1 -- 172.21.15.44:0/3690039357 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f88040030 con 0x55662ad141b0 2024-05-27T09:39:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.053+0000 7f8fa2000640 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662ad141b0 0x55662b0cce60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8f940dfc40 tx=0x7f8f940a95a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b63ef004e3117dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.053+0000 7f8f82a00640 10 client.15864 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:39:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.054+0000 7f8f82a00640 1 -- 172.21.15.44:0/3690039357 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9401a040 con 0x55662ad141b0 2024-05-27T09:39:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.054+0000 7f8f82a00640 1 -- 172.21.15.44:0/3690039357 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f8f940b21a0 con 0x55662ad141b0 2024-05-27T09:39:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.054+0000 7f8f82a00640 1 -- 172.21.15.44:0/3690039357 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8f940b23c0 con 0x55662ad141b0 2024-05-27T09:39:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.090+0000 7f8faaf85740 10 client.15864 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.091+0000 7f8faaf85740 1 -- 172.21.15.44:0/3690039357 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=26}) -- 0x55662adf8620 con 0x55662ad141b0 2024-05-27T09:39:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.091+0000 7f8f82a00640 1 -- 172.21.15.44:0/3690039357 <== mon.0 v2:172.21.15.44:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8f9400b040 con 0x55662ad141b0 2024-05-27T09:39:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.091+0000 7f8faaf85740 10 client.15864 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.098+0000 7f8faaf85740 10 client.15864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:39:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.098+0000 7f8faaf85740 20 client.15864 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.098+0000 7f8faaf85740 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662acd2d80 0x55662b0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.098+0000 7f8faaf85740 4 client.15864 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:39:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.098+0000 7f8faaf85740 1 -- 172.21.15.44:0/3690039357 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55662b029ae0 con 0x55662acd2d80 2024-05-27T09:39:46.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.098+0000 7f8fa1600640 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662acd2d80 0x55662b0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.100+0000 7f8f82a00640 10 client.15864 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.100+0000 7f8fa1600640 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662acd2d80 0x55662b0d2cd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d79e77346ae164f server_cookie=3ef0374c9dbd271e in_seq=0 out_seq=0 2024-05-27T09:39:46.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.101+0000 7f8f82a00640 1 -- 172.21.15.44:0/3690039357 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55662b029ae0 con 0x55662acd2d80 2024-05-27T09:39:46.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.101+0000 7f8f82a00640 10 client.15864 handle_command_reply: tid=0 2024-05-27T09:39:46.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.101+0000 7f8faaf85740 1 client.15864 shutdown 2024-05-27T09:39:46.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.103+0000 7f8faaf85740 1 -- 172.21.15.44:0/3690039357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662ad141b0 msgr2=0x55662b0cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.103+0000 7f8faaf85740 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662ad141b0 0x55662b0cce60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8f940dfc40 tx=0x7f8f940a95a0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.103+0000 7f8faaf85740 1 -- 172.21.15.44:0/3690039357 shutdown_connections 2024-05-27T09:39:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.103+0000 7f8faaf85740 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662acd2d80 0x55662b0d2cd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.103+0000 7f8faaf85740 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662ad141b0 0x55662b0cce60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.103+0000 7f8f82a00640 0 client.15864 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.103+0000 7f8faaf85740 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662acf34d0 0x55662b0c1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.103+0000 7f8faaf85740 1 --2- 172.21.15.44:0/3690039357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 0x55662b0c0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.103+0000 7f8faaf85740 1 -- 172.21.15.44:0/3690039357 >> 172.21.15.44:0/3690039357 conn(0x55662ad08300 msgr2=0x55662b0b8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.104+0000 7f8faaf85740 1 -- 172.21.15.44:0/3690039357 shutdown_connections 2024-05-27T09:39:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.104+0000 7f8faaf85740 1 -- 172.21.15.44:0/3690039357 wait complete. 2024-05-27T09:39:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.104+0000 7f8faaf85740 20 client.15864 trim_cache size 0 max 16384 2024-05-27T09:39:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.154+0000 7f8faaf85740 1 Processor -- start 2024-05-27T09:39:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.154+0000 7f8faaf85740 1 -- start start 2024-05-27T09:39:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.155+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662aceb2f0 0x55662b0c3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.155+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662acf34d0 0x55662b0c11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.155+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662ad141b0 0x55662b0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.155+0000 7f8faaf85740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1affd0 con 0x55662aceb2f0 2024-05-27T09:39:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.155+0000 7f8faaf85740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1ad8a0 con 0x55662ad141b0 2024-05-27T09:39:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.155+0000 7f8faaf85740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1ab6f0 con 0x55662acf34d0 2024-05-27T09:39:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.156+0000 7f8fa0c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662acf34d0 0x55662b0c11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.156+0000 7f8fa0c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662acf34d0 0x55662b0c11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47246/0 (socket says 172.21.15.44:47246) 2024-05-27T09:39:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.156+0000 7f8fa1600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662aceb2f0 0x55662b0c3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.156+0000 7f8fa0c00640 1 -- 172.21.15.44:0/1556294949 learned_addr learned my addr 172.21.15.44:0/1556294949 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.156+0000 7f8fa2000640 1 --2- 172.21.15.44:0/1556294949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662ad141b0 0x55662b0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.156+0000 7f8fa0c00640 1 -- 172.21.15.44:0/1556294949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662ad141b0 msgr2=0x55662b0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.156+0000 7f8fa0c00640 1 --2- 172.21.15.44:0/1556294949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662ad141b0 0x55662b0ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.156+0000 7f8fa0c00640 1 -- 172.21.15.44:0/1556294949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662aceb2f0 msgr2=0x55662b0c3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.157+0000 7f8fa0c00640 1 --2- 172.21.15.44:0/1556294949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662aceb2f0 0x55662b0c3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.157+0000 7f8fa0c00640 1 -- 172.21.15.44:0/1556294949 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55662b0b91e0 con 0x55662acf34d0 2024-05-27T09:39:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.157+0000 7f8fa2000640 1 --2- 172.21.15.44:0/1556294949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662ad141b0 0x55662b0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.157+0000 7f8fa0c00640 1 --2- 172.21.15.44:0/1556294949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662acf34d0 0x55662b0c11a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8f8802a940 tx=0x7f8f8803e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2833f154b382ac76 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.158+0000 7f8f8dc00640 1 -- 172.21.15.44:0/1556294949 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f88057070 con 0x55662acf34d0 2024-05-27T09:39:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.158+0000 7f8f8dc00640 1 -- 172.21.15.44:0/1556294949 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f88009450 con 0x55662acf34d0 2024-05-27T09:39:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.158+0000 7f8f8dc00640 1 -- 172.21.15.44:0/1556294949 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f880038e0 con 0x55662acf34d0 2024-05-27T09:39:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.159+0000 7f8faaf85740 1 -- 172.21.15.44:0/1556294949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662acf34d0 msgr2=0x55662b0c11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.159+0000 7f8faaf85740 1 --2- 172.21.15.44:0/1556294949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662acf34d0 0x55662b0c11a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8f8802a940 tx=0x7f8f8803e040 comp rx=0 tx=0).stop 2024-05-27T09:39:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.160+0000 7f8faaf85740 1 -- 172.21.15.44:0/1556294949 shutdown_connections 2024-05-27T09:39:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.160+0000 7f8faaf85740 1 --2- 172.21.15.44:0/1556294949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662ad141b0 0x55662b0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.160+0000 7f8faaf85740 1 --2- 172.21.15.44:0/1556294949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662acf34d0 0x55662b0c11a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.160+0000 7f8faaf85740 1 --2- 172.21.15.44:0/1556294949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662aceb2f0 0x55662b0c3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.160+0000 7f8faaf85740 1 -- 172.21.15.44:0/1556294949 >> 172.21.15.44:0/1556294949 conn(0x55662ad08300 msgr2=0x55662b0b7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.160+0000 7f8faaf85740 1 -- 172.21.15.44:0/1556294949 shutdown_connections 2024-05-27T09:39:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.160+0000 7f8faaf85740 1 -- 172.21.15.44:0/1556294949 wait complete. 2024-05-27T09:39:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.161+0000 7f8faaf85740 1 -- start start 2024-05-27T09:39:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.161+0000 7f8faaf85740 1 Processor -- start 2024-05-27T09:39:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 0x55662b0babf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662ad141b0 0x55662b0bb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8faaf85740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662aca81c0 0x55662b0bb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8faaf85740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1abd60 con 0x55662aca81c0 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8faaf85740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1ad1d0 con 0x55662ad141b0 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8faaf85740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1ac430 con 0x55662aceb2f0 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8faaf85740 10 client.0 _ll_register_callbacks cb 0x7f8f940a5690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8fa1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 0x55662b0babf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8fa1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 0x55662b0babf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47256/0 (socket says 172.21.15.44:47256) 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8fa1600640 1 -- 172.21.15.44:0/535948386 learned_addr learned my addr 172.21.15.44:0/535948386 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8fa0c00640 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662ad141b0 0x55662b0bb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.162+0000 7f8fa2000640 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662aca81c0 0x55662b0bb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.163+0000 7f8fa1600640 1 -- 172.21.15.44:0/535948386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662ad141b0 msgr2=0x55662b0bb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.163+0000 7f8fa1600640 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662ad141b0 0x55662b0bb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.163+0000 7f8fa1600640 1 -- 172.21.15.44:0/535948386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662aca81c0 msgr2=0x55662b0bb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.163+0000 7f8fa1600640 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662aca81c0 0x55662b0bb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.163+0000 7f8fa1600640 1 -- 172.21.15.44:0/535948386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55662b0c7f70 con 0x55662aceb2f0 2024-05-27T09:39:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.163+0000 7f8fa1600640 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 0x55662b0babf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8f90012010 tx=0x7f8f90004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8933d5d52f9b21c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.163+0000 7f8f82a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:39:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.163+0000 7f8f82a00640 1 -- 172.21.15.44:0/535948386 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9002f4d0 con 0x55662aceb2f0 2024-05-27T09:39:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.164+0000 7f8faaf85740 1 -- 172.21.15.44:0/535948386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f88009450 con 0x55662aceb2f0 2024-05-27T09:39:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.164+0000 7f8f82a00640 1 -- 172.21.15.44:0/535948386 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f8f9002f670 con 0x55662aceb2f0 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.164+0000 7f8f82a00640 1 -- 172.21.15.44:0/535948386 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8f9002f870 con 0x55662aceb2f0 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.164+0000 7f8faaf85740 10 client.26014 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.164+0000 7f8faaf85740 1 -- 172.21.15.44:0/535948386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55662b0c9220 con 0x55662aceb2f0 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.164+0000 7f8f82a00640 1 -- 172.21.15.44:0/535948386 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f88009450 con 0x55662aceb2f0 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.165+0000 7f8f82a00640 1 -- 172.21.15.44:0/535948386 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8f90041c20 con 0x55662aceb2f0 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.165+0000 7f8faaf85740 10 client.26014 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.165+0000 7f8faaf85740 10 client.26014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.165+0000 7f8faaf85740 20 client.26014 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.165+0000 7f8faaf85740 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662b13b3a0 0x55662b13d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.165+0000 7f8faaf85740 4 client.26014 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.165+0000 7f8faaf85740 1 -- 172.21.15.44:0/535948386 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55662b0232c0 con 0x55662b13b3a0 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.165+0000 7f8fa0c00640 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662b13b3a0 0x55662b13d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.166+0000 7f8fa0c00640 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662b13b3a0 0x55662b13d7c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b1af91403439603 server_cookie=7717c30215aa26fb in_seq=0 out_seq=0 2024-05-27T09:39:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.166+0000 7f8f82a00640 10 client.26014 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.184+0000 7f8f82a00640 1 -- 172.21.15.44:0/535948386 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55662b0232c0 con 0x55662b13b3a0 2024-05-27T09:39:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.184+0000 7f8f82a00640 10 client.26014 handle_command_reply: tid=0 2024-05-27T09:39:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.184+0000 7f8faaf85740 1 client.26014 shutdown 2024-05-27T09:39:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.185+0000 7f8faaf85740 1 -- 172.21.15.44:0/535948386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 msgr2=0x55662b0babf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.185+0000 7f8faaf85740 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 0x55662b0babf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8f90012010 tx=0x7f8f90004730 comp rx=0 tx=0).stop 2024-05-27T09:39:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.185+0000 7f8faaf85740 1 -- 172.21.15.44:0/535948386 shutdown_connections 2024-05-27T09:39:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.185+0000 7f8faaf85740 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662b13b3a0 0x55662b13d7c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.185+0000 7f8faaf85740 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662aca81c0 0x55662b0bb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.185+0000 7f8faaf85740 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662ad141b0 0x55662b0bb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.185+0000 7f8faaf85740 1 --2- 172.21.15.44:0/535948386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662aceb2f0 0x55662b0babf0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.185+0000 7f8faaf85740 1 -- 172.21.15.44:0/535948386 >> 172.21.15.44:0/535948386 conn(0x55662ad08300 msgr2=0x55662b0b2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.185+0000 7f8f82a00640 0 client.26014 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.185+0000 7f8faaf85740 1 -- 172.21.15.44:0/535948386 shutdown_connections 2024-05-27T09:39:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.185+0000 7f8faaf85740 1 -- 172.21.15.44:0/535948386 wait complete. 2024-05-27T09:39:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.186+0000 7f8faaf85740 20 client.26014 trim_cache size 0 max 16384 2024-05-27T09:39:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:39:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:39:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:39:46.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3464194589 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8f840b2a20 msgr2=0x7f8f840b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:46.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8f840b2a20 0x7f8f840b4f30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8f900095f0 tx=0x7f8f90012040 comp rx=0 tx=0).stop 2024-05-27T09:39:46.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3464194589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f9c105cf0 msgr2=0x7f8f9c19efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:46.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f9c105cf0 0x7f8f9c19efd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009fa0 tx=0x7f8f9400c800 comp rx=0 tx=0).stop 2024-05-27T09:39:46.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3464194589 shutdown_connections 2024-05-27T09:39:46.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f9c105cf0 0x7f8f9c19efd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f8f840b2a20 0x7f8f840b4f30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f9c104260 0x7f8f9c19ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 --2- 172.21.15.44:0/3464194589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f9c103e50 0x7f8f9c19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:46.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3464194589 >> 172.21.15.44:0/3464194589 conn(0x7f8f9c0791a0 msgr2=0x7f8f9c106250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:46.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3464194589 shutdown_connections 2024-05-27T09:39:46.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:39:46.187+0000 7f8fa2a00640 1 -- 172.21.15.44:0/3464194589 wait complete. 2024-05-27T09:39:47.327 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:47.327 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:47.758 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:47.759 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:47.772 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:47.772 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:47.772 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 70 -lt 100 ']' 2024-05-27T09:39:47.773 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 70 + 1 2024-05-27T09:39:47.848 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=71 2024-05-27T09:39:47.848 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:47.892 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:47.892 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:47.896 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:47.898 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:47.898 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:48.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:47 smithi050 ceph-mon[36371]: pgmap v139: 97 pgs: 97 active+clean; 3.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-05-27T09:39:48.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:47 smithi189 ceph-mon[36671]: pgmap v139: 97 pgs: 97 active+clean; 3.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-05-27T09:39:48.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:47 smithi044 ceph-mon[31618]: pgmap v139: 97 pgs: 97 active+clean; 3.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-05-27T09:39:49.001 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:49.002 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:49.158 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:49.158 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:49.187 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:49.188 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:49.188 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 71 -lt 100 ']' 2024-05-27T09:39:49.189 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 71 + 1 2024-05-27T09:39:49.270 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=72 2024-05-27T09:39:49.270 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:49.316 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:49.317 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:49.321 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:49.323 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:49.324 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:49.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:49 smithi044 ceph-mon[31618]: pgmap v140: 97 pgs: 97 active+clean; 3.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-05-27T09:39:50.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:49 smithi050 ceph-mon[36371]: pgmap v140: 97 pgs: 97 active+clean; 3.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-05-27T09:39:50.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:49 smithi189 ceph-mon[36671]: pgmap v140: 97 pgs: 97 active+clean; 3.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-05-27T09:39:50.168 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:50.168 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:50.349 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:50.350 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:50.363 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:50.364 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:50.364 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 72 -lt 100 ']' 2024-05-27T09:39:50.364 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 72 + 1 2024-05-27T09:39:50.484 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=73 2024-05-27T09:39:50.484 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:50.502 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:50.503 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:50.506 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:50.508 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:50.508 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:51.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:50 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:39:51.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:50 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:39:51.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:50 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:39:51.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:50 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:39:51.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:50 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:39:51.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:50 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:39:51.109 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:50 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:39:51.109 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:50 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:39:51.109 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:50 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:39:51.384 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:51.384 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:51.699 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:51.700 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:51.713 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:51.713 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:51.713 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 73 -lt 100 ']' 2024-05-27T09:39:51.715 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 73 + 1 2024-05-27T09:39:51.789 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=74 2024-05-27T09:39:51.789 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:51.853 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:51.854 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:51.859 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:51.860 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:51.861 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:51.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.932+0000 7fc306000700 1 -- 172.21.15.44:0/1206385896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc300109860 msgr2=0x7fc3001111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.932+0000 7fc306000700 1 --2- 172.21.15.44:0/1206385896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc300109860 0x7fc3001111c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800b0d0 tx=0x7fc2f800b3e0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.932+0000 7fc306000700 1 -- 172.21.15.44:0/1206385896 shutdown_connections 2024-05-27T09:39:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.932+0000 7fc306000700 1 --2- 172.21.15.44:0/1206385896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc300109860 0x7fc3001111c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.932+0000 7fc306000700 1 --2- 172.21.15.44:0/1206385896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3001045e0 0x7fc3001049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.932+0000 7fc306000700 1 --2- 172.21.15.44:0/1206385896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc300108f10 0x7fc300109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.932+0000 7fc306000700 1 -- 172.21.15.44:0/1206385896 >> 172.21.15.44:0/1206385896 conn(0x7fc300100030 msgr2=0x7fc300102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.932+0000 7fc306000700 1 -- 172.21.15.44:0/1206385896 shutdown_connections 2024-05-27T09:39:51.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.932+0000 7fc306000700 1 -- 172.21.15.44:0/1206385896 wait complete. 2024-05-27T09:39:51.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.934+0000 7fc306000700 1 Processor -- start 2024-05-27T09:39:51.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.935+0000 7fc306000700 1 -- start start 2024-05-27T09:39:51.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.936+0000 7fc306000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3001045e0 0x7fc30019ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:51.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.936+0000 7fc306000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc300108f10 0x7fc30019b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:51.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.936+0000 7fc306000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc300109860 0x7fc30019f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.936+0000 7fc306000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001144b0 con 0x7fc300109860 2024-05-27T09:39:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.936+0000 7fc306000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc300114330 con 0x7fc3001045e0 2024-05-27T09:39:51.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.936+0000 7fc306000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc300114630 con 0x7fc300108f10 2024-05-27T09:39:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.936+0000 7fc2ff400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3001045e0 0x7fc30019ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.936+0000 7fc2ff400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3001045e0 0x7fc30019ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38896/0 (socket says 172.21.15.44:38896) 2024-05-27T09:39:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.936+0000 7fc2ff400700 1 -- 172.21.15.44:0/2276653377 learned_addr learned my addr 172.21.15.44:0/2276653377 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.936+0000 7fc2f6a00700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc300108f10 0x7fc30019b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.936+0000 7fc2ffe00700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc300109860 0x7fc30019f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.937+0000 7fc2f6a00700 1 -- 172.21.15.44:0/2276653377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3001045e0 msgr2=0x7fc30019ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.937+0000 7fc2f6a00700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3001045e0 0x7fc30019ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.937+0000 7fc2f6a00700 1 -- 172.21.15.44:0/2276653377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc300109860 msgr2=0x7fc30019f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.937+0000 7fc2f6a00700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc300109860 0x7fc30019f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.937+0000 7fc2f6a00700 1 -- 172.21.15.44:0/2276653377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30019fcd0 con 0x7fc300108f10 2024-05-27T09:39:51.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.937+0000 7fc2ff400700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3001045e0 0x7fc30019ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:39:51.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.937+0000 7fc2f6a00700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc300108f10 0x7fc30019b3a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800e9c0 tx=0x7fc2e800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:51.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.938+0000 7fc2fcc00700 1 -- 172.21.15.44:0/2276653377 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8004220 con 0x7fc300108f10 2024-05-27T09:39:51.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.938+0000 7fc2fcc00700 1 -- 172.21.15.44:0/2276653377 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2e80043c0 con 0x7fc300108f10 2024-05-27T09:39:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.938+0000 7fc2fcc00700 1 -- 172.21.15.44:0/2276653377 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8005020 con 0x7fc300108f10 2024-05-27T09:39:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.938+0000 7fc306000700 1 -- 172.21.15.44:0/2276653377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30019ffc0 con 0x7fc300108f10 2024-05-27T09:39:51.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.938+0000 7fc306000700 1 -- 172.21.15.44:0/2276653377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3001a78e0 con 0x7fc300108f10 2024-05-27T09:39:51.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.942+0000 7fc2fcc00700 1 -- 172.21.15.44:0/2276653377 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc2e8004750 con 0x7fc300108f10 2024-05-27T09:39:51.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.944+0000 7fc2fcc00700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2e40b2a70 0x7fc2e40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:51.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.944+0000 7fc2fcc00700 1 -- 172.21.15.44:0/2276653377 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fc2e80e24b0 con 0x7fc300108f10 2024-05-27T09:39:51.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.944+0000 7fc2ff400700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2e40b2a70 0x7fc2e40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:51.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.945+0000 7fc2ff400700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2e40b2a70 0x7fc2e40b4f80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0011550 tx=0x7fc2f0009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:51.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.973+0000 7fc314b7cb80 1 Processor -- start 2024-05-27T09:39:51.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.974+0000 7fc314b7cb80 1 -- start start 2024-05-27T09:39:51.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.974+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002b77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:51.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.975+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 0x55e6002b1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:51.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.975+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b1db0 0x55e6002b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.975+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e6001dfe40 con 0x55e6002b7d30 2024-05-27T09:39:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.975+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e6001cf9e0 con 0x55e600132e40 2024-05-27T09:39:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.975+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e6001d3a20 con 0x55e6002b1db0 2024-05-27T09:39:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.975+0000 7fc2f6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 0x55e6002b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.975+0000 7fc2ff400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002b77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.975+0000 7fc2ff400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002b77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38912/0 (socket says 172.21.15.44:38912) 2024-05-27T09:39:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.975+0000 7fc2f6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 0x55e6002b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53908/0 (socket says 172.21.15.44:53908) 2024-05-27T09:39:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.975+0000 7fc2ff400700 1 -- 172.21.15.44:0/2715064876 learned_addr learned my addr 172.21.15.44:0/2715064876 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.975+0000 7fc2ffe00700 1 --2- 172.21.15.44:0/2715064876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b1db0 0x55e6002b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.976+0000 7fc2f6a00700 1 -- 172.21.15.44:0/2715064876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b1db0 msgr2=0x55e6002b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.976+0000 7fc2f6a00700 1 --2- 172.21.15.44:0/2715064876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b1db0 0x55e6002b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.976+0000 7fc2f6a00700 1 -- 172.21.15.44:0/2715064876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 msgr2=0x55e6002b77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.976+0000 7fc2f6a00700 1 --2- 172.21.15.44:0/2715064876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002b77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.976+0000 7fc2f6a00700 1 -- 172.21.15.44:0/2715064876 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e600198420 con 0x55e6002b7d30 2024-05-27T09:39:51.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.977+0000 7fc2f6a00700 1 --2- 172.21.15.44:0/2715064876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 0x55e6002b1870 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80dfbd0 tx=0x7fc2e8004de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142aafd8ac52f46b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:51.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.978+0000 7fc2f4c00700 1 -- 172.21.15.44:0/2715064876 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e801c070 con 0x55e6002b7d30 2024-05-27T09:39:51.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.978+0000 7fc2f4c00700 1 -- 172.21.15.44:0/2715064876 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2e80a9e60 con 0x55e6002b7d30 2024-05-27T09:39:51.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.978+0000 7fc2f4c00700 1 -- 172.21.15.44:0/2715064876 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8091310 con 0x55e6002b7d30 2024-05-27T09:39:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.979+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2715064876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 msgr2=0x55e6002b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.979+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/2715064876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 0x55e6002b1870 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80dfbd0 tx=0x7fc2e8004de0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.979+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2715064876 shutdown_connections 2024-05-27T09:39:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.979+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/2715064876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b1db0 0x55e6002b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.979+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/2715064876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002b77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.979+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/2715064876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 0x55e6002b1870 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.979+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2715064876 >> 172.21.15.44:0/2715064876 conn(0x55e600192fc0 msgr2=0x55e6002ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:51.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.979+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2715064876 shutdown_connections 2024-05-27T09:39:51.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.980+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2715064876 wait complete. 2024-05-27T09:39:51.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.980+0000 7fc314b7cb80 1 -- start start 2024-05-27T09:39:51.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.981+0000 7fc314b7cb80 1 Processor -- start 2024-05-27T09:39:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002bb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 0x55e6002bbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b7d30 0x55e6002bc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e6000d5fa0 con 0x55e6002b1db0 2024-05-27T09:39:51.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e600170760 con 0x55e600132e40 2024-05-27T09:39:51.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e6002be800 con 0x55e6002b7d30 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc314b7cb80 10 client.0 _ll_register_callbacks cb 0x55e600179c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc2ffe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b7d30 0x55e6002bc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc2f6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 0x55e6002bbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc2ffe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b7d30 0x55e6002bc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47294/0 (socket says 172.21.15.44:47294) 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc2ffe00700 1 -- 172.21.15.44:0/1834691318 learned_addr learned my addr 172.21.15.44:0/1834691318 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.982+0000 7fc2ff400700 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002bb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.983+0000 7fc2ffe00700 1 -- 172.21.15.44:0/1834691318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 msgr2=0x55e6002bb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.983+0000 7fc2ffe00700 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002bb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.983+0000 7fc2ffe00700 1 -- 172.21.15.44:0/1834691318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 msgr2=0x55e6002bbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.983+0000 7fc2ffe00700 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 0x55e6002bbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.983+0000 7fc2ffe00700 1 -- 172.21.15.44:0/1834691318 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e60011f8d0 con 0x55e6002b7d30 2024-05-27T09:39:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.983+0000 7fc2ffe00700 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b7d30 0x55e6002bc040 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005b60 tx=0x7fc2f8005b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b74157e868e22bdf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:51.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.983+0000 7fc2ed600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:39:51.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.983+0000 7fc2ed600700 1 -- 172.21.15.44:0/1834691318 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f800bde0 con 0x55e6002b7d30 2024-05-27T09:39:51.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.983+0000 7fc2ed600700 1 -- 172.21.15.44:0/1834691318 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fc2f8003a00 con 0x55e6002b7d30 2024-05-27T09:39:51.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.983+0000 7fc314b7cb80 1 -- 172.21.15.44:0/1834691318 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2e80a9e60 con 0x55e6002b7d30 2024-05-27T09:39:51.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.984+0000 7fc2ed600700 1 -- 172.21.15.44:0/1834691318 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc2f8003c00 con 0x55e6002b7d30 2024-05-27T09:39:51.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.984+0000 7fc314b7cb80 10 client.26026 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:51.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.984+0000 7fc314b7cb80 1 -- 172.21.15.44:0/1834691318 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e6001a0060 con 0x55e6002b7d30 2024-05-27T09:39:51.987 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:51 smithi044 ceph-mon[31618]: pgmap v141: 97 pgs: 97 active+clean; 3.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 15 op/s 2024-05-27T09:39:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.985+0000 7fc2ed600700 1 -- 172.21.15.44:0/1834691318 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2e80a9e60 con 0x55e6002b7d30 2024-05-27T09:39:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.985+0000 7fc2ed600700 1 -- 172.21.15.44:0/1834691318 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc2f8011070 con 0x55e6002b7d30 2024-05-27T09:39:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.985+0000 7fc314b7cb80 10 client.26026 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.985+0000 7fc314b7cb80 10 client.26026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:39:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.985+0000 7fc314b7cb80 20 client.26026 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.985+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e6002cb760 0x55e6002cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.985+0000 7fc314b7cb80 4 client.26026 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:39:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.985+0000 7fc314b7cb80 1 -- 172.21.15.44:0/1834691318 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6001e1d90 con 0x55e6002cb760 2024-05-27T09:39:51.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.986+0000 7fc2ff400700 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e6002cb760 0x55e6002cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:51.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.986+0000 7fc2ff400700 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e6002cb760 0x55e6002cdbc0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bd50d8c7186247f server_cookie=9c5f8bda1831a89b in_seq=0 out_seq=0 2024-05-27T09:39:51.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.986+0000 7fc2ed600700 10 client.26026 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:51.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.987+0000 7fc2ed600700 1 -- 172.21.15.44:0/1834691318 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e6001e1d90 con 0x55e6002cb760 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.987+0000 7fc2ed600700 10 client.26026 handle_command_reply: tid=0 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.987+0000 7fc314b7cb80 1 client.26026 shutdown 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc314b7cb80 1 -- 172.21.15.44:0/1834691318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b7d30 msgr2=0x55e6002bc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b7d30 0x55e6002bc040 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005b60 tx=0x7fc2f8005b90 comp rx=0 tx=0).stop 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc314b7cb80 1 -- 172.21.15.44:0/1834691318 shutdown_connections 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e6002cb760 0x55e6002cdbc0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b7d30 0x55e6002bc040 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002bb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc2ed600700 0 client.26026 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/1834691318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 0x55e6002bbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc314b7cb80 1 -- 172.21.15.44:0/1834691318 >> 172.21.15.44:0/1834691318 conn(0x55e600192fc0 msgr2=0x55e6002ac580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:51.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc314b7cb80 1 -- 172.21.15.44:0/1834691318 shutdown_connections 2024-05-27T09:39:51.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc314b7cb80 1 -- 172.21.15.44:0/1834691318 wait complete. 2024-05-27T09:39:51.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:51.988+0000 7fc314b7cb80 20 client.26026 trim_cache size 0 max 16384 2024-05-27T09:39:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.008+0000 7fc314b7cb80 1 Processor -- start 2024-05-27T09:39:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.008+0000 7fc314b7cb80 1 -- start start 2024-05-27T09:39:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.009+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e600132e40 0x55e6002b1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.009+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6002b1db0 0x55e6002bda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.009+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 0x55e6002c7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.009+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001b15c0 con 0x55e6002b7d30 2024-05-27T09:39:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.009+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001aee50 con 0x55e6002b1db0 2024-05-27T09:39:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.009+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001acca0 con 0x55e600132e40 2024-05-27T09:39:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.009+0000 7fc2ff400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e600132e40 0x55e6002b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.009+0000 7fc2ff400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e600132e40 0x55e6002b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47304/0 (socket says 172.21.15.44:47304) 2024-05-27T09:39:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.009+0000 7fc2ff400700 1 -- 172.21.15.44:0/4053043939 learned_addr learned my addr 172.21.15.44:0/4053043939 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.009+0000 7fc2f6a00700 1 --2- 172.21.15.44:0/4053043939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6002b1db0 0x55e6002bda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.009+0000 7fc2ffe00700 1 --2- 172.21.15.44:0/4053043939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 0x55e6002c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.010+0000 7fc2ff400700 1 -- 172.21.15.44:0/4053043939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6002b1db0 msgr2=0x55e6002bda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.010+0000 7fc2ff400700 1 --2- 172.21.15.44:0/4053043939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6002b1db0 0x55e6002bda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.010+0000 7fc2ff400700 1 -- 172.21.15.44:0/4053043939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 msgr2=0x55e6002c7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.010+0000 7fc2ff400700 1 --2- 172.21.15.44:0/4053043939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 0x55e6002c7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.010+0000 7fc2ff400700 1 -- 172.21.15.44:0/4053043939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6002ae030 con 0x55e600132e40 2024-05-27T09:39:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.010+0000 7fc2ff400700 1 --2- 172.21.15.44:0/4053043939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e600132e40 0x55e6002b1870 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009cf0 tx=0x7fc2f0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3ef9addff861a1d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.011+0000 7fc2efe00700 1 -- 172.21.15.44:0/4053043939 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0029670 con 0x55e600132e40 2024-05-27T09:39:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.011+0000 7fc2efe00700 1 -- 172.21.15.44:0/4053043939 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2f002f070 con 0x55e600132e40 2024-05-27T09:39:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.011+0000 7fc2efe00700 1 -- 172.21.15.44:0/4053043939 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0028d70 con 0x55e600132e40 2024-05-27T09:39:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.012+0000 7fc314b7cb80 1 -- 172.21.15.44:0/4053043939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e600132e40 msgr2=0x55e6002b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.012+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/4053043939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e600132e40 0x55e6002b1870 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009cf0 tx=0x7fc2f0017000 comp rx=0 tx=0).stop 2024-05-27T09:39:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.012+0000 7fc314b7cb80 1 -- 172.21.15.44:0/4053043939 shutdown_connections 2024-05-27T09:39:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.012+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/4053043939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b7d30 0x55e6002c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.012+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/4053043939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e600132e40 0x55e6002b1870 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.012+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/4053043939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6002b1db0 0x55e6002bda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.012+0000 7fc314b7cb80 1 -- 172.21.15.44:0/4053043939 >> 172.21.15.44:0/4053043939 conn(0x55e600192fc0 msgr2=0x55e6002b5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.013+0000 7fc314b7cb80 1 -- 172.21.15.44:0/4053043939 shutdown_connections 2024-05-27T09:39:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.013+0000 7fc314b7cb80 1 -- 172.21.15.44:0/4053043939 wait complete. 2024-05-27T09:39:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.013+0000 7fc314b7cb80 1 -- start start 2024-05-27T09:39:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.013+0000 7fc314b7cb80 1 Processor -- start 2024-05-27T09:39:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.014+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002c3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.014+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 0x55e6002d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.014+0000 7fc314b7cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b7d30 0x55e6002d2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.014+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001ad310 con 0x55e6002b1db0 2024-05-27T09:39:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.014+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001ae780 con 0x55e600132e40 2024-05-27T09:39:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.014+0000 7fc314b7cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc3001ad9e0 con 0x55e6002b7d30 2024-05-27T09:39:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.014+0000 7fc314b7cb80 10 client.0 _ll_register_callbacks cb 0x55e600179c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:39:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.014+0000 7fc2f6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 0x55e6002d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.015+0000 7fc2f6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 0x55e6002d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53926/0 (socket says 172.21.15.44:53926) 2024-05-27T09:39:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.015+0000 7fc2f6a00700 1 -- 172.21.15.44:0/2334450137 learned_addr learned my addr 172.21.15.44:0/2334450137 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:39:52.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.015+0000 7fc2ff400700 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002c3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.015+0000 7fc2f6a00700 1 -- 172.21.15.44:0/2334450137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b7d30 msgr2=0x55e6002d2dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T09:39:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.015+0000 7fc2f6a00700 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b7d30 0x55e6002d2dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.015+0000 7fc2f6a00700 1 -- 172.21.15.44:0/2334450137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 msgr2=0x55e6002c3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:39:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.015+0000 7fc2f6a00700 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002c3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.015+0000 7fc2f6a00700 1 -- 172.21.15.44:0/2334450137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6002c4640 con 0x55e6002b1db0 2024-05-27T09:39:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.015+0000 7fc2f6a00700 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 0x55e6002d2890 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80dfc30 tx=0x7fc2e800c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dddaa15101e9831 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:39:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.015+0000 7fc2ed600700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:39:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.016+0000 7fc2ed600700 1 -- 172.21.15.44:0/2334450137 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e801c040 con 0x55e6002b1db0 2024-05-27T09:39:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.016+0000 7fc2ed600700 1 -- 172.21.15.44:0/2334450137 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fc2e80b2340 con 0x55e6002b1db0 2024-05-27T09:39:52.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.016+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2334450137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2f002f070 con 0x55e6002b1db0 2024-05-27T09:39:52.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.016+0000 7fc2ed600700 1 -- 172.21.15.44:0/2334450137 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc2e80cc0b0 con 0x55e6002b1db0 2024-05-27T09:39:52.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.016+0000 7fc314b7cb80 10 client.15888 fetch_fsmap learned FSMap version 26 2024-05-27T09:39:52.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.016+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2334450137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e600331d30 con 0x55e6002b1db0 2024-05-27T09:39:52.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.017+0000 7fc2ed600700 1 -- 172.21.15.44:0/2334450137 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2f002f070 con 0x55e6002b1db0 2024-05-27T09:39:52.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.017+0000 7fc2ed600700 1 -- 172.21.15.44:0/2334450137 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc2e80bec40 con 0x55e6002b1db0 2024-05-27T09:39:52.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.017+0000 7fc314b7cb80 10 client.15888 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:39:52.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.017+0000 7fc314b7cb80 10 client.15888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:39:52.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.017+0000 7fc314b7cb80 20 client.15888 populate_metadata read hostname 'smithi044' 2024-05-27T09:39:52.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.017+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e600332010 0x55e600340e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:39:52.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.017+0000 7fc314b7cb80 4 client.15888 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:39:52.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.017+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2334450137 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e6001db570 con 0x55e600332010 2024-05-27T09:39:52.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.018+0000 7fc2ff400700 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e600332010 0x55e600340e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:39:52.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.018+0000 7fc2ff400700 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e600332010 0x55e600340e60 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52545b6ef72d9622 server_cookie=864e37c9592a5068 in_seq=0 out_seq=0 2024-05-27T09:39:52.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.018+0000 7fc2ed600700 10 client.15888 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:52.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc2ed600700 1 -- 172.21.15.44:0/2334450137 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e6001db570 con 0x55e600332010 2024-05-27T09:39:52.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc2ed600700 10 client.15888 handle_command_reply: tid=0 2024-05-27T09:39:52.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 1 client.15888 shutdown 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2334450137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 msgr2=0x55e6002d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 0x55e6002d2890 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80dfc30 tx=0x7fc2e800c9d0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2334450137 shutdown_connections 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e600332010 0x55e600340e60 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6002b7d30 0x55e6002d2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e600132e40 0x55e6002c3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 1 --2- 172.21.15.44:0/2334450137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e6002b1db0 0x55e6002d2890 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2334450137 >> 172.21.15.44:0/2334450137 conn(0x55e600192fc0 msgr2=0x55e6002b5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc2ed600700 0 client.15888 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2334450137 shutdown_connections 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 1 -- 172.21.15.44:0/2334450137 wait complete. 2024-05-27T09:39:52.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.019+0000 7fc314b7cb80 20 client.15888 trim_cache size 0 max 16384 2024-05-27T09:39:52.023 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:39:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.020+0000 7fc306000700 1 -- 172.21.15.44:0/2276653377 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2e40b2a70 msgr2=0x7fc2e40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.020+0000 7fc306000700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2e40b2a70 0x7fc2e40b4f80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0011550 tx=0x7fc2f0009260 comp rx=0 tx=0).stop 2024-05-27T09:39:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.020+0000 7fc306000700 1 -- 172.21.15.44:0/2276653377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc300108f10 msgr2=0x7fc30019b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:39:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.020+0000 7fc306000700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc300108f10 0x7fc30019b3a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800e9c0 tx=0x7fc2e800ecd0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.020+0000 7fc306000700 1 -- 172.21.15.44:0/2276653377 shutdown_connections 2024-05-27T09:39:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.020+0000 7fc306000700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc2e40b2a70 0x7fc2e40b4f80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.020+0000 7fc306000700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc300109860 0x7fc30019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.020+0000 7fc306000700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3001045e0 0x7fc30019ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.020+0000 7fc306000700 1 --2- 172.21.15.44:0/2276653377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc300108f10 0x7fc30019b3a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:39:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.020+0000 7fc306000700 1 -- 172.21.15.44:0/2276653377 >> 172.21.15.44:0/2276653377 conn(0x7fc300100030 msgr2=0x7fc300108650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:39:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.021+0000 7fc306000700 1 -- 172.21.15.44:0/2276653377 shutdown_connections 2024-05-27T09:39:52.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:39:52.021+0000 7fc306000700 1 -- 172.21.15.44:0/2276653377 wait complete. 2024-05-27T09:39:52.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:51 smithi050 ceph-mon[36371]: pgmap v141: 97 pgs: 97 active+clean; 3.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 15 op/s 2024-05-27T09:39:52.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:51 smithi189 ceph-mon[36671]: pgmap v141: 97 pgs: 97 active+clean; 3.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 15 op/s 2024-05-27T09:39:52.820 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:52 smithi044 ceph-mon[31618]: pgmap v142: 97 pgs: 97 active+clean; 3.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-05-27T09:39:52.907 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:52.908 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:53.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:52 smithi050 ceph-mon[36371]: pgmap v142: 97 pgs: 97 active+clean; 3.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-05-27T09:39:53.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:52 smithi189 ceph-mon[36671]: pgmap v142: 97 pgs: 97 active+clean; 3.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-05-27T09:39:53.192 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:53.193 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:53.213 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:53.213 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:53.213 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 74 -lt 100 ']' 2024-05-27T09:39:53.214 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 74 + 1 2024-05-27T09:39:53.246 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=75 2024-05-27T09:39:53.246 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:53.276 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:53.276 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:53.280 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:53.281 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:53.281 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:54.176 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:54.176 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:54.432 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:54.433 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:54.480 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:54.480 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:54.480 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 75 -lt 100 ']' 2024-05-27T09:39:54.482 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 75 + 1 2024-05-27T09:39:54.544 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=76 2024-05-27T09:39:54.544 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:54.598 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:54.598 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:54.602 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:54.603 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:54.604 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:55.084 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:55.084 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:55.199 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:55.199 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:55.211 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:55.212 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:55.212 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 76 -lt 100 ']' 2024-05-27T09:39:55.213 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 76 + 1 2024-05-27T09:39:55.237 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=77 2024-05-27T09:39:55.237 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:55.261 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:55.261 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:55.266 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:55.267 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:55.267 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:55.622 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:39:55.622 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:39:55.651 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:55.651 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:55.774 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:55.774 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:55.796 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:55.796 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:55.796 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 77 -lt 100 ']' 2024-05-27T09:39:55.796 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 77 + 1 2024-05-27T09:39:55.854 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=78 2024-05-27T09:39:55.854 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:55.877 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:55.878 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:55.882 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:55.883 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:55.884 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:55.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:55 smithi044 ceph-mon[31618]: pgmap v143: 97 pgs: 97 active+clean; 3.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-05-27T09:39:56.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:55 smithi050 ceph-mon[36371]: pgmap v143: 97 pgs: 97 active+clean; 3.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-05-27T09:39:56.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:55 smithi189 ceph-mon[36671]: pgmap v143: 97 pgs: 97 active+clean; 3.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-05-27T09:39:56.345 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:56.345 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:56.454 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-05-27T09:39:56.454 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-05-27T09:39:56.454 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:39:56.549 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:56.549 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:56.578 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:56.578 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:56.579 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 78 -lt 100 ']' 2024-05-27T09:39:56.580 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 78 + 1 2024-05-27T09:39:56.608 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=79 2024-05-27T09:39:56.608 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:56.634 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:56.634 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:56.639 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:56.640 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:56.640 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:57.392 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:57.392 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:57.591 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:57.591 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:57.614 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:57.614 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:57.615 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 79 -lt 100 ']' 2024-05-27T09:39:57.616 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 79 + 1 2024-05-27T09:39:57.665 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=80 2024-05-27T09:39:57.665 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:57.697 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:57.697 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:57.703 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:57.704 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:57.705 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:57.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:57 smithi044 ceph-mon[31618]: pgmap v144: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 15 op/s 2024-05-27T09:39:58.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:57 smithi050 ceph-mon[36371]: pgmap v144: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 15 op/s 2024-05-27T09:39:58.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:57 smithi189 ceph-mon[36671]: pgmap v144: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 15 op/s 2024-05-27T09:39:58.518 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:58.725 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:58.932 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:39:58.932 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:39:58.958 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:58.958 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:39:58.958 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 80 -lt 100 ']' 2024-05-27T09:39:58.958 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 80 + 1 2024-05-27T09:39:58.973 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=81 2024-05-27T09:39:58.973 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:39:58.992 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:39:58.992 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:39:58.997 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:39:58.998 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:39:58.999 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:39:59.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:39:59 smithi044 ceph-mon[31618]: pgmap v145: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-05-27T09:40:00.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:39:59 smithi050 ceph-mon[36371]: pgmap v145: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-05-27T09:40:00.075 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:00.078 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:00.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:39:59 smithi189 ceph-mon[36671]: pgmap v145: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-05-27T09:40:00.507 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:00.508 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:00.531 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:00.532 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:00.532 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 81 -lt 100 ']' 2024-05-27T09:40:00.533 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 81 + 1 2024-05-27T09:40:00.636 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=82 2024-05-27T09:40:00.636 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:00.763 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:00.763 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:00.769 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:00.770 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:00.771 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:00.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:00 smithi044 ceph-mon[31618]: overall HEALTH_OK 2024-05-27T09:40:01.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:00 smithi050 ceph-mon[36371]: overall HEALTH_OK 2024-05-27T09:40:01.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:00 smithi189 ceph-mon[36671]: overall HEALTH_OK 2024-05-27T09:40:01.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:01 smithi044 ceph-mon[31618]: pgmap v146: 97 pgs: 97 active+clean; 3.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 15 op/s 2024-05-27T09:40:02.045 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:01 smithi050 ceph-mon[36371]: pgmap v146: 97 pgs: 97 active+clean; 3.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 15 op/s 2024-05-27T09:40:02.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:01 smithi189 ceph-mon[36671]: pgmap v146: 97 pgs: 97 active+clean; 3.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 15 op/s 2024-05-27T09:40:02.096 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:02.096 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:02.440 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:02.440 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:02.476 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:02.476 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:02.476 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 82 -lt 100 ']' 2024-05-27T09:40:02.476 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 82 + 1 2024-05-27T09:40:02.502 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=83 2024-05-27T09:40:02.503 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:02.581 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:02.581 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:02.583 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:02.604 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:02.604 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:03.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:03 smithi044 ceph-mon[31618]: pgmap v147: 97 pgs: 97 active+clean; 3.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-05-27T09:40:04.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:03 smithi050 ceph-mon[36371]: pgmap v147: 97 pgs: 97 active+clean; 3.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-05-27T09:40:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:03 smithi189 ceph-mon[36671]: pgmap v147: 97 pgs: 97 active+clean; 3.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-05-27T09:40:04.258 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:04.258 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:04.656 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:04.657 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:04.694 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:04.694 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:04.694 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 83 -lt 100 ']' 2024-05-27T09:40:04.695 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 83 + 1 2024-05-27T09:40:04.851 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=84 2024-05-27T09:40:04.852 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:04.921 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:04.922 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:04.928 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:04.930 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:04.931 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:05 smithi044 ceph-mon[31618]: pgmap v148: 97 pgs: 97 active+clean; 3.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-05-27T09:40:06.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:05 smithi050 ceph-mon[36371]: pgmap v148: 97 pgs: 97 active+clean; 3.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-05-27T09:40:06.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:05 smithi189 ceph-mon[36671]: pgmap v148: 97 pgs: 97 active+clean; 3.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-05-27T09:40:06.132 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:06.133 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:06.773 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:06.773 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:06.787 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:06.787 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:06.787 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 84 -lt 100 ']' 2024-05-27T09:40:06.788 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 84 + 1 2024-05-27T09:40:06.821 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=85 2024-05-27T09:40:06.822 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:06.912 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:06.912 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:06.917 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:06.918 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:06.918 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:07.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:07 smithi044 ceph-mon[31618]: pgmap v149: 97 pgs: 97 active+clean; 3.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 49 KiB/s wr, 25 op/s 2024-05-27T09:40:08.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:07 smithi050 ceph-mon[36371]: pgmap v149: 97 pgs: 97 active+clean; 3.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 49 KiB/s wr, 25 op/s 2024-05-27T09:40:08.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:07 smithi189 ceph-mon[36671]: pgmap v149: 97 pgs: 97 active+clean; 3.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 49 KiB/s wr, 25 op/s 2024-05-27T09:40:08.532 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:08.532 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:09.439 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:09.439 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:09.461 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:09.461 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:09.461 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 85 -lt 100 ']' 2024-05-27T09:40:09.462 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 85 + 1 2024-05-27T09:40:09.628 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=86 2024-05-27T09:40:09.628 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:09.696 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:09.696 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:09.701 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:09.702 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:09.702 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:09.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:09 smithi044 ceph-mon[31618]: pgmap v150: 97 pgs: 97 active+clean; 3.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 41 KiB/s wr, 21 op/s 2024-05-27T09:40:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:09 smithi050 ceph-mon[36371]: pgmap v150: 97 pgs: 97 active+clean; 3.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 41 KiB/s wr, 21 op/s 2024-05-27T09:40:10.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:09 smithi189 ceph-mon[36671]: pgmap v150: 97 pgs: 97 active+clean; 3.5 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 41 KiB/s wr, 21 op/s 2024-05-27T09:40:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:10.959+0000 7ff7b0400640 1 -- 172.21.15.44:0/3450871186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a8075660 msgr2=0x7ff7a8075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:10.959+0000 7ff7b0400640 1 --2- 172.21.15.44:0/3450871186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a8075660 0x7ff7a8075a70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff79c009900 tx=0x7ff79c02a490 comp rx=0 tx=0).stop 2024-05-27T09:40:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:10.959+0000 7ff7b0400640 1 -- 172.21.15.44:0/3450871186 shutdown_connections 2024-05-27T09:40:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:10.959+0000 7ff7b0400640 1 --2- 172.21.15.44:0/3450871186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7a81104a0 0x7ff7a8112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:10.959+0000 7ff7b0400640 1 --2- 172.21.15.44:0/3450871186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7a810db40 0x7ff7a810ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:10.959+0000 7ff7b0400640 1 --2- 172.21.15.44:0/3450871186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a8075660 0x7ff7a8075a70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:10.959+0000 7ff7b0400640 1 -- 172.21.15.44:0/3450871186 >> 172.21.15.44:0/3450871186 conn(0x7ff7a80ff480 msgr2=0x7ff7a81018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:10.960+0000 7ff7b0400640 1 -- 172.21.15.44:0/3450871186 shutdown_connections 2024-05-27T09:40:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:10.960+0000 7ff7b0400640 1 -- 172.21.15.44:0/3450871186 wait complete. 2024-05-27T09:40:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.107+0000 7ff7b0400640 1 Processor -- start 2024-05-27T09:40:11.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.108+0000 7ff7b0400640 1 -- start start 2024-05-27T09:40:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.109+0000 7ff7b0400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a8075660 0x7ff7a81a2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.109+0000 7ff7b0400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7a810db40 0x7ff7a81a2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.109+0000 7ff7b0400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7a81104a0 0x7ff7a819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.110+0000 7ff7b0400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a8116090 con 0x7ff7a810db40 2024-05-27T09:40:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.110+0000 7ff7b0400640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a8115f10 con 0x7ff7a81104a0 2024-05-27T09:40:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.110+0000 7ff7b0400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a8116210 con 0x7ff7a8075660 2024-05-27T09:40:11.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.110+0000 7ff7ada00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a8075660 0x7ff7a81a2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.110+0000 7ff7ada00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a8075660 0x7ff7a81a2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51958/0 (socket says 172.21.15.44:51958) 2024-05-27T09:40:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.110+0000 7ff7ada00640 1 -- 172.21.15.44:0/3806655125 learned_addr learned my addr 172.21.15.44:0/3806655125 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.110+0000 7ff7ae400640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7a81104a0 0x7ff7a819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.110+0000 7ff7ad000640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7a810db40 0x7ff7a81a2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.111+0000 7ff7ae400640 1 -- 172.21.15.44:0/3806655125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a8075660 msgr2=0x7ff7a81a2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.111+0000 7ff7ae400640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a8075660 0x7ff7a81a2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.111+0000 7ff7ae400640 1 -- 172.21.15.44:0/3806655125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7a810db40 msgr2=0x7ff7a81a2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.111+0000 7ff7ae400640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7a810db40 0x7ff7a81a2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.111+0000 7ff7ae400640 1 -- 172.21.15.44:0/3806655125 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a819cff0 con 0x7ff7a81104a0 2024-05-27T09:40:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.111+0000 7ff7ada00640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a8075660 0x7ff7a81a2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:11.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.112+0000 7ff7ae400640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7a81104a0 0x7ff7a819c950 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000ec50 tx=0x7ff7a000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:11.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.116+0000 7ff7a6000640 1 -- 172.21.15.44:0/3806655125 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0003b70 con 0x7ff7a81104a0 2024-05-27T09:40:11.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.116+0000 7ff7a6000640 1 -- 172.21.15.44:0/3806655125 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7a0003d10 con 0x7ff7a81104a0 2024-05-27T09:40:11.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.116+0000 7ff7a6000640 1 -- 172.21.15.44:0/3806655125 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0010830 con 0x7ff7a81104a0 2024-05-27T09:40:11.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.116+0000 7ff7b0400640 1 -- 172.21.15.44:0/3806655125 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a819d2e0 con 0x7ff7a81104a0 2024-05-27T09:40:11.118 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.117+0000 7ff7b0400640 1 -- 172.21.15.44:0/3806655125 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7a81a95c0 con 0x7ff7a81104a0 2024-05-27T09:40:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.121+0000 7ff7a6000640 1 -- 172.21.15.44:0/3806655125 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff7a0003ec0 con 0x7ff7a81104a0 2024-05-27T09:40:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.124+0000 7ff7a6000640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7900b2b20 0x7ff7900b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.124+0000 7ff7a6000640 1 -- 172.21.15.44:0/3806655125 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff7a00e2a10 con 0x7ff7a81104a0 2024-05-27T09:40:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.125+0000 7ff7ada00640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7900b2b20 0x7ff7900b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.126+0000 7ff7ada00640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7900b2b20 0x7ff7900b5030 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff79c005ec0 tx=0x7ff79c005a20 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:11.415 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:11.415 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.536+0000 7ff7b7415740 1 Processor -- start 2024-05-27T09:40:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.537+0000 7ff7b7415740 1 -- start start 2024-05-27T09:40:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.538+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.538+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd60da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.538+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 0x56037cd612e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.538+0000 7ff7b7415740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56037ccd1b90 con 0x56037c99d4d0 2024-05-27T09:40:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.538+0000 7ff7b7415740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56037ccc1730 con 0x56037c9be1b0 2024-05-27T09:40:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.538+0000 7ff7b7415740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56037ccc5770 con 0x56037c9952f0 2024-05-27T09:40:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.539+0000 7ff7ad000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd60da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.539+0000 7ff7ada00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.539+0000 7ff7ada00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42328/0 (socket says 172.21.15.44:42328) 2024-05-27T09:40:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.539+0000 7ff7ada00640 1 -- 172.21.15.44:0/1498939479 learned_addr learned my addr 172.21.15.44:0/1498939479 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.539+0000 7ff7ae400640 1 --2- 172.21.15.44:0/1498939479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 0x56037cd612e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.540+0000 7ff7ae400640 1 -- 172.21.15.44:0/1498939479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 msgr2=0x56037cd60da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.540+0000 7ff7ae400640 1 --2- 172.21.15.44:0/1498939479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd60da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.540+0000 7ff7ae400640 1 -- 172.21.15.44:0/1498939479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 msgr2=0x56037cd60860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.540+0000 7ff7ae400640 1 --2- 172.21.15.44:0/1498939479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.540+0000 7ff7ae400640 1 -- 172.21.15.44:0/1498939479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56037cc5a8b0 con 0x56037c99d4d0 2024-05-27T09:40:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.540+0000 7ff7ada00640 1 --2- 172.21.15.44:0/1498939479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.540+0000 7ff7ae400640 1 --2- 172.21.15.44:0/1498939479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 0x56037cd612e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00e0130 tx=0x7ff7a00a9430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708d4649d3cfeb16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.540+0000 7ff79aa00640 1 -- 172.21.15.44:0/1498939479 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00c1070 con 0x56037c99d4d0 2024-05-27T09:40:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.541+0000 7ff79aa00640 1 -- 172.21.15.44:0/1498939479 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7a0004660 con 0x56037c99d4d0 2024-05-27T09:40:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.541+0000 7ff79aa00640 1 -- 172.21.15.44:0/1498939479 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00ae070 con 0x56037c99d4d0 2024-05-27T09:40:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.542+0000 7ff7b7415740 1 -- 172.21.15.44:0/1498939479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 msgr2=0x56037cd612e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.542+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1498939479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 0x56037cd612e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00e0130 tx=0x7ff7a00a9430 comp rx=0 tx=0).stop 2024-05-27T09:40:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.542+0000 7ff7b7415740 1 -- 172.21.15.44:0/1498939479 shutdown_connections 2024-05-27T09:40:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.542+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1498939479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 0x56037cd612e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.542+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1498939479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd60da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.542+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1498939479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.542+0000 7ff7b7415740 1 -- 172.21.15.44:0/1498939479 >> 172.21.15.44:0/1498939479 conn(0x56037c9b2300 msgr2=0x56037cd5d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.542+0000 7ff7b7415740 1 -- 172.21.15.44:0/1498939479 shutdown_connections 2024-05-27T09:40:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.542+0000 7ff7b7415740 1 -- 172.21.15.44:0/1498939479 wait complete. 2024-05-27T09:40:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.596+0000 7ff7b7415740 1 -- start start 2024-05-27T09:40:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.597+0000 7ff7b7415740 1 Processor -- start 2024-05-27T09:40:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.599+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9952f0 0x56037cd6ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.599+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c99d4d0 0x56037cd6b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.599+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c9be1b0 0x56037cd76c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.599+0000 7ff7b7415740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56037c9a7790 con 0x56037c9be1b0 2024-05-27T09:40:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.599+0000 7ff7b7415740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56037cc69bc0 con 0x56037c9952f0 2024-05-27T09:40:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.599+0000 7ff7b7415740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56037cd6d930 con 0x56037c99d4d0 2024-05-27T09:40:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.600+0000 7ff7ada00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9952f0 0x56037cd6ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.600+0000 7ff7ad000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c99d4d0 0x56037cd6b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.600+0000 7ff7ad000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c99d4d0 0x56037cd6b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51978/0 (socket says 172.21.15.44:51978) 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.600+0000 7ff7ada00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9952f0 0x56037cd6ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42340/0 (socket says 172.21.15.44:42340) 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.600+0000 7ff7ad000640 1 -- 172.21.15.44:0/911815171 learned_addr learned my addr 172.21.15.44:0/911815171 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.600+0000 7ff7ae400640 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c9be1b0 0x56037cd76c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.600+0000 7ff7ae400640 1 -- 172.21.15.44:0/911815171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c99d4d0 msgr2=0x56037cd6b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.600+0000 7ff7ae400640 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c99d4d0 0x56037cd6b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.600+0000 7ff7ae400640 1 -- 172.21.15.44:0/911815171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9952f0 msgr2=0x56037cd6ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.600+0000 7ff7ae400640 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9952f0 0x56037cd6ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.600+0000 7ff7ae400640 1 -- 172.21.15.44:0/911815171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56037c9d6210 con 0x56037c9be1b0 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.601+0000 7ff7b7415740 10 client.0 _ll_register_callbacks cb 0x56037cc47400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.601+0000 7ff7b7415740 1 -- 172.21.15.44:0/911815171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56037ccd3ae0 con 0x56037c9be1b0 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.601+0000 7ff7ae400640 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c9be1b0 0x56037cd76c60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x56037cd62310 tx=0x7ff7a0004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8abff1da547876a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.601+0000 7ff78f400640 10 client.15900 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:40:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.601+0000 7ff78f400640 1 -- 172.21.15.44:0/911815171 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00ba050 con 0x56037c9be1b0 2024-05-27T09:40:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.601+0000 7ff78f400640 1 -- 172.21.15.44:0/911815171 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff7a00a9130 con 0x56037c9be1b0 2024-05-27T09:40:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.602+0000 7ff78f400640 1 -- 172.21.15.44:0/911815171 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff7a00c8070 con 0x56037c9be1b0 2024-05-27T09:40:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.603+0000 7ff7b7415740 10 client.15900 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.603+0000 7ff7b7415740 1 -- 172.21.15.44:0/911815171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=26}) -- 0x56037caa2620 con 0x56037c9be1b0 2024-05-27T09:40:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.608+0000 7ff78f400640 1 -- 172.21.15.44:0/911815171 <== mon.0 v2:172.21.15.44:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff7a00bd020 con 0x56037c9be1b0 2024-05-27T09:40:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.608+0000 7ff7b7415740 10 client.15900 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.608+0000 7ff7b7415740 10 client.15900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:40:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.608+0000 7ff7b7415740 20 client.15900 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.608+0000 7ff7b7415740 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56037c97cd80 0x56037cd7c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.608+0000 7ff7b7415740 4 client.15900 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:40:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.608+0000 7ff7b7415740 1 -- 172.21.15.44:0/911815171 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56037cccd2c0 con 0x56037c97cd80 2024-05-27T09:40:11.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.609+0000 7ff7ada00640 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56037c97cd80 0x56037cd7c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.610+0000 7ff7ada00640 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56037c97cd80 0x56037cd7c9b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b656a70d0c8f56b server_cookie=de75955c796ef6ea in_seq=0 out_seq=0 2024-05-27T09:40:11.610 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.610+0000 7ff78f400640 10 client.15900 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:11.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.611+0000 7ff78f400640 1 -- 172.21.15.44:0/911815171 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56037cccd2c0 con 0x56037c97cd80 2024-05-27T09:40:11.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.611+0000 7ff78f400640 10 client.15900 handle_command_reply: tid=0 2024-05-27T09:40:11.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.612+0000 7ff7b7415740 1 client.15900 shutdown 2024-05-27T09:40:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.614+0000 7ff7b7415740 1 -- 172.21.15.44:0/911815171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c9be1b0 msgr2=0x56037cd76c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.614+0000 7ff7b7415740 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c9be1b0 0x56037cd76c60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x56037cd62310 tx=0x7ff7a0004440 comp rx=0 tx=0).stop 2024-05-27T09:40:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.615+0000 7ff7b7415740 1 -- 172.21.15.44:0/911815171 shutdown_connections 2024-05-27T09:40:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.615+0000 7ff7b7415740 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56037c97cd80 0x56037cd7c9b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.615+0000 7ff7b7415740 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c9be1b0 0x56037cd76c60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.615+0000 7ff7b7415740 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c99d4d0 0x56037cd6b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.615+0000 7ff7b7415740 1 --2- 172.21.15.44:0/911815171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9952f0 0x56037cd6ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.615+0000 7ff78f400640 0 client.15900 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.615+0000 7ff7b7415740 1 -- 172.21.15.44:0/911815171 >> 172.21.15.44:0/911815171 conn(0x56037c9b2300 msgr2=0x56037cd62580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.615+0000 7ff7b7415740 1 -- 172.21.15.44:0/911815171 shutdown_connections 2024-05-27T09:40:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.615+0000 7ff7b7415740 1 -- 172.21.15.44:0/911815171 wait complete. 2024-05-27T09:40:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.615+0000 7ff7b7415740 20 client.15900 trim_cache size 0 max 16384 2024-05-27T09:40:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.666+0000 7ff7b7415740 1 Processor -- start 2024-05-27T09:40:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.666+0000 7ff7b7415740 1 -- start start 2024-05-27T09:40:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.667+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd69df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.667+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 0x56037cd5c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.667+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.667+0000 7ff7b7415740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a81b45f0 con 0x56037c99d4d0 2024-05-27T09:40:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.667+0000 7ff7b7415740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a81b1ec0 con 0x56037c9be1b0 2024-05-27T09:40:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.667+0000 7ff7b7415740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a81afd10 con 0x56037c9952f0 2024-05-27T09:40:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.667+0000 7ff7ae400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.667+0000 7ff7ada00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd69df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.667+0000 7ff7ae400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42346/0 (socket says 172.21.15.44:42346) 2024-05-27T09:40:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.667+0000 7ff7ae400640 1 -- 172.21.15.44:0/1046468379 learned_addr learned my addr 172.21.15.44:0/1046468379 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.668+0000 7ff7ad000640 1 --2- 172.21.15.44:0/1046468379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 0x56037cd5c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.668+0000 7ff7ae400640 1 -- 172.21.15.44:0/1046468379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 msgr2=0x56037cd69df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.668+0000 7ff7ae400640 1 --2- 172.21.15.44:0/1046468379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd69df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.668+0000 7ff7ae400640 1 -- 172.21.15.44:0/1046468379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 msgr2=0x56037cd5c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.668+0000 7ff7ae400640 1 --2- 172.21.15.44:0/1046468379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 0x56037cd5c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.668+0000 7ff7ae400640 1 -- 172.21.15.44:0/1046468379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56037cd69040 con 0x56037c9be1b0 2024-05-27T09:40:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.668+0000 7ff7ada00640 1 --2- 172.21.15.44:0/1046468379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd69df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.668+0000 7ff7ad000640 1 --2- 172.21.15.44:0/1046468379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 0x56037cd5c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:40:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.669+0000 7ff7ae400640 1 --2- 172.21.15.44:0/1046468379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60ef0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00a6f30 tx=0x7ff7a001c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7be65b8bb170e1ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.669+0000 7ff79a000640 1 -- 172.21.15.44:0/1046468379 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00ba070 con 0x56037c9be1b0 2024-05-27T09:40:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.669+0000 7ff79a000640 1 -- 172.21.15.44:0/1046468379 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56037ccd3ae0 con 0x56037c9be1b0 2024-05-27T09:40:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.669+0000 7ff79a000640 1 -- 172.21.15.44:0/1046468379 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00c1040 con 0x56037c9be1b0 2024-05-27T09:40:11.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.671+0000 7ff7b7415740 1 -- 172.21.15.44:0/1046468379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 msgr2=0x56037cd60ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:11.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.671+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1046468379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60ef0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00a6f30 tx=0x7ff7a001c000 comp rx=0 tx=0).stop 2024-05-27T09:40:11.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.671+0000 7ff7b7415740 1 -- 172.21.15.44:0/1046468379 shutdown_connections 2024-05-27T09:40:11.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.671+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1046468379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c9be1b0 0x56037cd60ef0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.671+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1046468379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c99d4d0 0x56037cd5c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.671+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1046468379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd69df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.671+0000 7ff7b7415740 1 -- 172.21.15.44:0/1046468379 >> 172.21.15.44:0/1046468379 conn(0x56037c9b2300 msgr2=0x56037cd607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:11.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.671+0000 7ff7b7415740 1 -- 172.21.15.44:0/1046468379 shutdown_connections 2024-05-27T09:40:11.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.672+0000 7ff7b7415740 1 -- 172.21.15.44:0/1046468379 wait complete. 2024-05-27T09:40:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.672+0000 7ff7b7415740 1 -- start start 2024-05-27T09:40:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.673+0000 7ff7b7415740 1 Processor -- start 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.673+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd7fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c99d4d0 0x56037cd7b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7b7415740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c9be1b0 0x56037cd7b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7b7415740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a81b0380 con 0x56037c9be1b0 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7b7415740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a81b17f0 con 0x56037c99d4d0 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7b7415740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a81b0a50 con 0x56037c9952f0 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7b7415740 10 client.0 _ll_register_callbacks cb 0x7ff79c040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7ada00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd7fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7ada00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd7fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51990/0 (socket says 172.21.15.44:51990) 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7ada00640 1 -- 172.21.15.44:0/1415342916 learned_addr learned my addr 172.21.15.44:0/1415342916 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7ae400640 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c9be1b0 0x56037cd7b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7ad000640 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c99d4d0 0x56037cd7b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7ada00640 1 -- 172.21.15.44:0/1415342916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c99d4d0 msgr2=0x56037cd7b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.674+0000 7ff7ada00640 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c99d4d0 0x56037cd7b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.675+0000 7ff7ada00640 1 -- 172.21.15.44:0/1415342916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c9be1b0 msgr2=0x56037cd7b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.675+0000 7ff7ada00640 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c9be1b0 0x56037cd7b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.675+0000 7ff7ada00640 1 -- 172.21.15.44:0/1415342916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56037cd7bc50 con 0x56037c9952f0 2024-05-27T09:40:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.675+0000 7ff7ad000640 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c99d4d0 0x56037cd7b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:40:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.675+0000 7ff7ada00640 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd7fe80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x56037cd5d9d0 tx=0x7ff79c02a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a52d4035e6f2a695 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.675+0000 7ff78f400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:40:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.675+0000 7ff78f400640 1 -- 172.21.15.44:0/1415342916 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c035040 con 0x56037c9952f0 2024-05-27T09:40:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.675+0000 7ff78f400640 1 -- 172.21.15.44:0/1415342916 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff79c031870 con 0x56037c9952f0 2024-05-27T09:40:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.675+0000 7ff7b7415740 1 -- 172.21.15.44:0/1415342916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56037ccd3ae0 con 0x56037c9952f0 2024-05-27T09:40:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.676+0000 7ff78f400640 1 -- 172.21.15.44:0/1415342916 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff79c031a70 con 0x56037c9952f0 2024-05-27T09:40:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.676+0000 7ff7b7415740 10 client.26047 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.676+0000 7ff7b7415740 1 -- 172.21.15.44:0/1415342916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56037cde4ad0 con 0x56037c9952f0 2024-05-27T09:40:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.676+0000 7ff78f400640 1 -- 172.21.15.44:0/1415342916 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56037ccd3ae0 con 0x56037c9952f0 2024-05-27T09:40:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.677+0000 7ff78f400640 1 -- 172.21.15.44:0/1415342916 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff79c053aa0 con 0x56037c9952f0 2024-05-27T09:40:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.677+0000 7ff7b7415740 10 client.26047 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.677+0000 7ff7b7415740 10 client.26047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:40:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.677+0000 7ff7b7415740 20 client.26047 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.677+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56037c9521c0 0x56037cde7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.677+0000 7ff7b7415740 4 client.26047 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:40:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.677+0000 7ff7b7415740 1 -- 172.21.15.44:0/1415342916 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56037cc474b0 con 0x56037c9521c0 2024-05-27T09:40:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.677+0000 7ff7ad000640 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56037c9521c0 0x56037cde7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.678+0000 7ff7ad000640 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56037c9521c0 0x56037cde7220 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52794738e6821cac server_cookie=92d30a2def38dd72 in_seq=0 out_seq=0 2024-05-27T09:40:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.678+0000 7ff78f400640 10 client.26047 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff78f400640 1 -- 172.21.15.44:0/1415342916 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56037cc474b0 con 0x56037c9521c0 2024-05-27T09:40:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff78f400640 10 client.26047 handle_command_reply: tid=0 2024-05-27T09:40:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff7b7415740 1 client.26047 shutdown 2024-05-27T09:40:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff7b7415740 1 -- 172.21.15.44:0/1415342916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 msgr2=0x56037cd7fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd7fe80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x56037cd5d9d0 tx=0x7ff79c02a490 comp rx=0 tx=0).stop 2024-05-27T09:40:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff7b7415740 1 -- 172.21.15.44:0/1415342916 shutdown_connections 2024-05-27T09:40:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56037c9521c0 0x56037cde7220 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56037c9be1b0 0x56037cd7b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56037c99d4d0 0x56037cd7b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff7b7415740 1 --2- 172.21.15.44:0/1415342916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56037c9952f0 0x56037cd7fe80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff78f400640 0 client.26047 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.696+0000 7ff7b7415740 1 -- 172.21.15.44:0/1415342916 >> 172.21.15.44:0/1415342916 conn(0x56037c9b2300 msgr2=0x56037cd69b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.697+0000 7ff7b7415740 1 -- 172.21.15.44:0/1415342916 shutdown_connections 2024-05-27T09:40:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.697+0000 7ff7b7415740 1 -- 172.21.15.44:0/1415342916 wait complete. 2024-05-27T09:40:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.697+0000 7ff7b7415740 20 client.26047 trim_cache size 0 max 16384 2024-05-27T09:40:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:40:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:40:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:40:11.713 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:11.713 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.728+0000 7ff7b0400640 1 -- 172.21.15.44:0/3806655125 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7900b2b20 msgr2=0x7ff7900b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.728+0000 7ff7b0400640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7900b2b20 0x7ff7900b5030 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff79c005ec0 tx=0x7ff79c005a20 comp rx=0 tx=0).stop 2024-05-27T09:40:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.728+0000 7ff7b0400640 1 -- 172.21.15.44:0/3806655125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7a81104a0 msgr2=0x7ff7a819c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.728+0000 7ff7b0400640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7a81104a0 0x7ff7a819c950 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000ec50 tx=0x7ff7a000c6a0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.729+0000 7ff7b0400640 1 -- 172.21.15.44:0/3806655125 shutdown_connections 2024-05-27T09:40:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.729+0000 7ff7b0400640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7a81104a0 0x7ff7a819c950 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.729+0000 7ff7b0400640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7900b2b20 0x7ff7900b5030 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.729+0000 7ff7b0400640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7a810db40 0x7ff7a81a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.729+0000 7ff7b0400640 1 --2- 172.21.15.44:0/3806655125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a8075660 0x7ff7a81a2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.729+0000 7ff7b0400640 1 -- 172.21.15.44:0/3806655125 >> 172.21.15.44:0/3806655125 conn(0x7ff7a80ff480 msgr2=0x7ff7a81108f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.729+0000 7ff7b0400640 1 -- 172.21.15.44:0/3806655125 shutdown_connections 2024-05-27T09:40:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:11.729+0000 7ff7b0400640 1 -- 172.21.15.44:0/3806655125 wait complete. 2024-05-27T09:40:11.751 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:11.752 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:11.752 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 86 -lt 100 ']' 2024-05-27T09:40:11.752 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 86 + 1 2024-05-27T09:40:11.894 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=87 2024-05-27T09:40:11.894 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:11.952 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:11.953 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:11.958 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:11.959 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:11.961 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:11.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:11 smithi044 ceph-mon[31618]: pgmap v151: 97 pgs: 97 active+clean; 3.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 50 KiB/s wr, 25 op/s 2024-05-27T09:40:12.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:11 smithi050 ceph-mon[36371]: pgmap v151: 97 pgs: 97 active+clean; 3.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 50 KiB/s wr, 25 op/s 2024-05-27T09:40:12.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:11 smithi189 ceph-mon[36671]: pgmap v151: 97 pgs: 97 active+clean; 3.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 50 KiB/s wr, 25 op/s 2024-05-27T09:40:12.990 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:12.990 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:13.263 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:13.263 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:13.294 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:13.294 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:13.294 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 87 -lt 100 ']' 2024-05-27T09:40:13.295 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 87 + 1 2024-05-27T09:40:13.392 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=88 2024-05-27T09:40:13.392 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:13.479 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:13.479 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:13.485 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:13.486 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:13.486 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:14.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:13 smithi050 ceph-mon[36371]: pgmap v152: 97 pgs: 97 active+clean; 3.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 40 KiB/s wr, 21 op/s 2024-05-27T09:40:14.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:13 smithi189 ceph-mon[36671]: pgmap v152: 97 pgs: 97 active+clean; 3.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 40 KiB/s wr, 21 op/s 2024-05-27T09:40:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:13 smithi044 ceph-mon[31618]: pgmap v152: 97 pgs: 97 active+clean; 3.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 40 KiB/s wr, 21 op/s 2024-05-27T09:40:14.727 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:40:14.727 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:40:14.982 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:14.982 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:15.155 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:15.155 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:15.181 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:15.181 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:15.181 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 88 -lt 100 ']' 2024-05-27T09:40:15.181 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 88 + 1 2024-05-27T09:40:15.256 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=89 2024-05-27T09:40:15.256 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:15.294 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:15.294 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:15.299 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:15.301 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:15.301 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:16.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:15 smithi050 ceph-mon[36371]: pgmap v153: 97 pgs: 97 active+clean; 3.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 40 KiB/s wr, 21 op/s 2024-05-27T09:40:16.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:15 smithi044 ceph-mon[31618]: pgmap v153: 97 pgs: 97 active+clean; 3.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 40 KiB/s wr, 21 op/s 2024-05-27T09:40:16.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:15 smithi189 ceph-mon[36671]: pgmap v153: 97 pgs: 97 active+clean; 3.6 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 40 KiB/s wr, 21 op/s 2024-05-27T09:40:16.390 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:16.390 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:16.821 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:16.822 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:16.863 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:16.863 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:16.863 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 89 -lt 100 ']' 2024-05-27T09:40:16.865 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 89 + 1 2024-05-27T09:40:17.038 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=90 2024-05-27T09:40:17.039 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:17.111 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:17.112 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:17.116 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:17.117 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:17.118 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:18.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:17 smithi050 ceph-mon[36371]: pgmap v154: 97 pgs: 97 active+clean; 3.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 30 op/s 2024-05-27T09:40:18.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:17 smithi189 ceph-mon[36671]: pgmap v154: 97 pgs: 97 active+clean; 3.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 30 op/s 2024-05-27T09:40:18.142 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:17 smithi044 ceph-mon[31618]: pgmap v154: 97 pgs: 97 active+clean; 3.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 30 op/s 2024-05-27T09:40:19.022 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:19.023 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:19.621 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:19.622 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:19.651 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:19.651 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:19.651 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 90 -lt 100 ']' 2024-05-27T09:40:19.652 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 90 + 1 2024-05-27T09:40:19.710 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=91 2024-05-27T09:40:19.711 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:19.762 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:19.763 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:19.768 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:19.769 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:19.769 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:20.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:19 smithi050 ceph-mon[36371]: pgmap v155: 97 pgs: 97 active+clean; 3.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-05-27T09:40:20.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:19 smithi189 ceph-mon[36671]: pgmap v155: 97 pgs: 97 active+clean; 3.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-05-27T09:40:20.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:19 smithi044 ceph-mon[31618]: pgmap v155: 97 pgs: 97 active+clean; 3.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-05-27T09:40:20.714 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:20.714 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:21.263 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:21.263 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:21.335 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:21.335 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:21.336 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 91 -lt 100 ']' 2024-05-27T09:40:21.336 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 91 + 1 2024-05-27T09:40:21.439 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=92 2024-05-27T09:40:21.439 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:21.521 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:21.522 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:21.527 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:21.528 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:21.529 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:22.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:21 smithi050 ceph-mon[36371]: pgmap v156: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 15 op/s 2024-05-27T09:40:22.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:21 smithi189 ceph-mon[36671]: pgmap v156: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 15 op/s 2024-05-27T09:40:22.167 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:21 smithi044 ceph-mon[31618]: pgmap v156: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 15 op/s 2024-05-27T09:40:23.480 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:23.481 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:23.845 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:23.845 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:23.876 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:23.876 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:23.876 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 92 -lt 100 ']' 2024-05-27T09:40:23.877 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 92 + 1 2024-05-27T09:40:24.000 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=93 2024-05-27T09:40:24.000 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:24.022 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:24.022 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:24.027 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:24.027 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:24.028 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:24.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:23 smithi050 ceph-mon[36371]: pgmap v157: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-05-27T09:40:24.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:23 smithi189 ceph-mon[36671]: pgmap v157: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-05-27T09:40:24.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:23 smithi044 ceph-mon[31618]: pgmap v157: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-05-27T09:40:25.071 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:25.072 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:25.487 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:25.487 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:25.534 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:25.534 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:25.535 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 93 -lt 100 ']' 2024-05-27T09:40:25.536 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 93 + 1 2024-05-27T09:40:25.684 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=94 2024-05-27T09:40:25.684 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:25.758 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:25.758 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:25.764 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:25.765 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:25.765 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:26.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:25 smithi050 ceph-mon[36371]: pgmap v158: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-05-27T09:40:26.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:25 smithi189 ceph-mon[36671]: pgmap v158: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-05-27T09:40:26.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:25 smithi044 ceph-mon[31618]: pgmap v158: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-05-27T09:40:27.503 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:27.504 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:28.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:27 smithi050 ceph-mon[36371]: pgmap v159: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 15 op/s 2024-05-27T09:40:28.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:27 smithi189 ceph-mon[36671]: pgmap v159: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 15 op/s 2024-05-27T09:40:28.179 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:28.179 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:28.200 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:28.200 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:28.200 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 94 -lt 100 ']' 2024-05-27T09:40:28.201 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 94 + 1 2024-05-27T09:40:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:27 smithi044 ceph-mon[31618]: pgmap v159: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 15 op/s 2024-05-27T09:40:28.329 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=95 2024-05-27T09:40:28.329 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:28.376 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:28.377 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:28.382 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:28.383 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:28.384 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:29.471 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:29.471 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:29.928 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:29.928 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:29.993 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:29.993 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:29.993 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 95 -lt 100 ']' 2024-05-27T09:40:29.994 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 95 + 1 2024-05-27T09:40:30.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:29 smithi050 ceph-mon[36371]: pgmap v160: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-27T09:40:30.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:29 smithi189 ceph-mon[36671]: pgmap v160: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-27T09:40:30.124 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=96 2024-05-27T09:40:30.124 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:30.161 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:30.162 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:30.166 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:30.167 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:30.168 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:30.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:29 smithi044 ceph-mon[31618]: pgmap v160: 97 pgs: 97 active+clean; 3.8 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-27T09:40:31.796 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:31.796 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:32.042 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:31 smithi044 ceph-mon[31618]: pgmap v161: 97 pgs: 97 active+clean; 3.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 10 op/s 2024-05-27T09:40:32.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:31 smithi050 ceph-mon[36371]: pgmap v161: 97 pgs: 97 active+clean; 3.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 10 op/s 2024-05-27T09:40:32.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:31 smithi189 ceph-mon[36671]: pgmap v161: 97 pgs: 97 active+clean; 3.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 10 op/s 2024-05-27T09:40:32.219 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:32.220 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:32.301 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:32.301 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:32.301 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 96 -lt 100 ']' 2024-05-27T09:40:32.302 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 96 + 1 2024-05-27T09:40:32.440 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=97 2024-05-27T09:40:32.440 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:32.591 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:32.591 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:32.597 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:32.598 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:32.599 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:33.764 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:33.764 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:33.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:33 smithi044 ceph-mon[31618]: pgmap v162: 97 pgs: 97 active+clean; 3.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-27T09:40:34.036 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:34.036 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:34.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:33 smithi050 ceph-mon[36371]: pgmap v162: 97 pgs: 97 active+clean; 3.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-27T09:40:34.058 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:34.058 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:34.058 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 97 -lt 100 ']' 2024-05-27T09:40:34.060 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 97 + 1 2024-05-27T09:40:34.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:33 smithi189 ceph-mon[36671]: pgmap v162: 97 pgs: 97 active+clean; 3.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-27T09:40:34.174 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=98 2024-05-27T09:40:34.174 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:34.222 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:34.223 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:34.227 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:34.228 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:34.229 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:35.333 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:35.333 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:35.569 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:35.569 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.610+0000 7f89a7e00640 1 -- 172.21.15.44:0/581593989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a810bcd0 msgr2=0x7f89a8105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.610+0000 7f89a7e00640 1 --2- 172.21.15.44:0/581593989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a810bcd0 0x7f89a8105870 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8998009900 tx=0x7f899802a3f0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.610+0000 7f89a7e00640 1 -- 172.21.15.44:0/581593989 shutdown_connections 2024-05-27T09:40:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.610+0000 7f89a7e00640 1 --2- 172.21.15.44:0/581593989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89a8106420 0x7f89a81127f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.610+0000 7f89a7e00640 1 --2- 172.21.15.44:0/581593989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a810c170 0x7f89a8105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.610+0000 7f89a7e00640 1 --2- 172.21.15.44:0/581593989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a810bcd0 0x7f89a8105870 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.610+0000 7f89a7e00640 1 -- 172.21.15.44:0/581593989 >> 172.21.15.44:0/581593989 conn(0x7f89a81013d0 msgr2=0x7f89a8103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.610+0000 7f89a7e00640 1 -- 172.21.15.44:0/581593989 shutdown_connections 2024-05-27T09:40:35.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.610+0000 7f89a7e00640 1 -- 172.21.15.44:0/581593989 wait complete. 2024-05-27T09:40:35.613 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:35.613 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:35.614 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 98 -lt 100 ']' 2024-05-27T09:40:35.614 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 98 + 1 2024-05-27T09:40:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.682+0000 7f89a7e00640 1 Processor -- start 2024-05-27T09:40:35.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.682+0000 7f89a7e00640 1 -- start start 2024-05-27T09:40:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.684+0000 7f89a7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a8106420 0x7f89a81a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.684+0000 7f89a7e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a810bcd0 0x7f89a81a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.684+0000 7f89a7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89a810c170 0x7f89a819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.684+0000 7f89a7e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8115ef0 con 0x7f89a810c170 2024-05-27T09:40:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.684+0000 7f89a7e00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8115d70 con 0x7f89a810bcd0 2024-05-27T09:40:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.684+0000 7f89a7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8116070 con 0x7f89a8106420 2024-05-27T09:40:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.684+0000 7f89a6000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a810bcd0 0x7f89a81a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.684+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a8106420 0x7f89a81a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.685+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a8106420 0x7f89a81a25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:36162/0 (socket says 172.21.15.44:36162) 2024-05-27T09:40:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.685+0000 7f89a6000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a810bcd0 0x7f89a81a2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53860/0 (socket says 172.21.15.44:53860) 2024-05-27T09:40:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.685+0000 7f89a6a00640 1 -- 172.21.15.44:0/3757063580 learned_addr learned my addr 172.21.15.44:0/3757063580 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.685+0000 7f89a7400640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89a810c170 0x7f89a819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.685+0000 7f89a6000640 1 -- 172.21.15.44:0/3757063580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a8106420 msgr2=0x7f89a81a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.685+0000 7f89a6000640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a8106420 0x7f89a81a25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.685+0000 7f89a6000640 1 -- 172.21.15.44:0/3757063580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89a810c170 msgr2=0x7f89a819c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.685+0000 7f89a6000640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89a810c170 0x7f89a819c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.685+0000 7f89a6000640 1 -- 172.21.15.44:0/3757063580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a819ce60 con 0x7f89a810bcd0 2024-05-27T09:40:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.685+0000 7f89a7400640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89a810c170 0x7f89a819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:40:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.686+0000 7f89a6000640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a810bcd0 0x7f89a81a2af0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f89a806c3a0 tx=0x7f899400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.687+0000 7f899f400640 1 -- 172.21.15.44:0/3757063580 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899400cd00 con 0x7f89a810bcd0 2024-05-27T09:40:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.687+0000 7f899f400640 1 -- 172.21.15.44:0/3757063580 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89940108e0 con 0x7f89a810bcd0 2024-05-27T09:40:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.687+0000 7f899f400640 1 -- 172.21.15.44:0/3757063580 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940188c0 con 0x7f89a810bcd0 2024-05-27T09:40:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.687+0000 7f89a7e00640 1 -- 172.21.15.44:0/3757063580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a819d0f0 con 0x7f89a810bcd0 2024-05-27T09:40:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.688+0000 7f89a7e00640 1 -- 172.21.15.44:0/3757063580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a81a9350 con 0x7f89a810bcd0 2024-05-27T09:40:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.692+0000 7f899f400640 1 -- 172.21.15.44:0/3757063580 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f899401f020 con 0x7f89a810bcd0 2024-05-27T09:40:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.694+0000 7f899f400640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f898c0b2aa0 0x7f898c0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.695+0000 7f899f400640 1 -- 172.21.15.44:0/3757063580 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f89940e2610 con 0x7f89a810bcd0 2024-05-27T09:40:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.695+0000 7f89a6a00640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f898c0b2aa0 0x7f898c0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.696+0000 7f89a6a00640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f898c0b2aa0 0x7f898c0b4fb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8998000c00 tx=0x7f8998005a20 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:35.758 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=99 2024-05-27T09:40:35.758 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:35.842 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:35.842 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:35.846 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:35.848 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:35.848 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.862+0000 7f89b4636740 1 Processor -- start 2024-05-27T09:40:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.862+0000 7f89b4636740 1 -- start start 2024-05-27T09:40:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.863+0000 7f89b4636740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c030950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.863+0000 7f89b4636740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c030e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.863+0000 7f89b4636740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 0x56097c0313d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.863+0000 7f89b4636740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56097bfa1b90 con 0x56097bc8e1b0 2024-05-27T09:40:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.863+0000 7f89b4636740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56097bf91730 con 0x56097bc6d4d0 2024-05-27T09:40:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.863+0000 7f89b4636740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56097bf95770 con 0x56097bc652f0 2024-05-27T09:40:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.863+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c030950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.864+0000 7f89a7400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 0x56097c0313d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.864+0000 7f89a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c030e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.864+0000 7f89a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c030e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:36172/0 (socket says 172.21.15.44:36172) 2024-05-27T09:40:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.864+0000 7f89a7400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 0x56097c0313d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53870/0 (socket says 172.21.15.44:53870) 2024-05-27T09:40:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.864+0000 7f89a6000640 1 -- 172.21.15.44:0/4045234696 learned_addr learned my addr 172.21.15.44:0/4045234696 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.864+0000 7f89a6000640 1 -- 172.21.15.44:0/4045234696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 msgr2=0x56097c0313d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.864+0000 7f89a6000640 1 --2- 172.21.15.44:0/4045234696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 0x56097c0313d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.864+0000 7f89a6000640 1 -- 172.21.15.44:0/4045234696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 msgr2=0x56097c030950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.864+0000 7f89a6000640 1 --2- 172.21.15.44:0/4045234696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c030950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.864+0000 7f89a6000640 1 -- 172.21.15.44:0/4045234696 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56097bf2a8b0 con 0x56097bc652f0 2024-05-27T09:40:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.865+0000 7f89a6000640 1 --2- 172.21.15.44:0/4045234696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c030e90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f89940dfcd0 tx=0x7f89940a98c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6c3852e61af1a32 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.865+0000 7f8993400640 1 -- 172.21.15.44:0/4045234696 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940b2050 con 0x56097bc652f0 2024-05-27T09:40:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.865+0000 7f8993400640 1 -- 172.21.15.44:0/4045234696 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89940093c0 con 0x56097bc652f0 2024-05-27T09:40:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.865+0000 7f8993400640 1 -- 172.21.15.44:0/4045234696 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940dffb0 con 0x56097bc652f0 2024-05-27T09:40:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.866+0000 7f89b4636740 1 -- 172.21.15.44:0/4045234696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 msgr2=0x56097c030e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.866+0000 7f89b4636740 1 --2- 172.21.15.44:0/4045234696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c030e90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f89940dfcd0 tx=0x7f89940a98c0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.867+0000 7f89b4636740 1 -- 172.21.15.44:0/4045234696 shutdown_connections 2024-05-27T09:40:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.867+0000 7f89b4636740 1 --2- 172.21.15.44:0/4045234696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 0x56097c0313d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.867+0000 7f89b4636740 1 --2- 172.21.15.44:0/4045234696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c030e90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.867+0000 7f89b4636740 1 --2- 172.21.15.44:0/4045234696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c030950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.867+0000 7f89b4636740 1 -- 172.21.15.44:0/4045234696 >> 172.21.15.44:0/4045234696 conn(0x56097bc82300 msgr2=0x56097c02d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.867+0000 7f89b4636740 1 -- 172.21.15.44:0/4045234696 shutdown_connections 2024-05-27T09:40:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.867+0000 7f89b4636740 1 -- 172.21.15.44:0/4045234696 wait complete. 2024-05-27T09:40:35.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.927+0000 7f89b4636740 1 -- start start 2024-05-27T09:40:35.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.928+0000 7f89b4636740 1 Processor -- start 2024-05-27T09:40:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.929+0000 7f89b4636740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c043440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.929+0000 7f89b4636740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 0x56097c043980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.929+0000 7f89b4636740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c04f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.929+0000 7f89b4636740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56097bc77790 con 0x56097bc8e1b0 2024-05-27T09:40:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.929+0000 7f89b4636740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56097bf39bc0 con 0x56097bc6d4d0 2024-05-27T09:40:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.929+0000 7f89b4636740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56097c046340 con 0x56097bc652f0 2024-05-27T09:40:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.929+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c043440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.929+0000 7f89a7400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c04f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.929+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c043440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:36186/0 (socket says 172.21.15.44:36186) 2024-05-27T09:40:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.929+0000 7f89a6a00640 1 -- 172.21.15.44:0/1151845882 learned_addr learned my addr 172.21.15.44:0/1151845882 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.930+0000 7f89a6000640 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 0x56097c043980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.930+0000 7f89a6a00640 1 -- 172.21.15.44:0/1151845882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 msgr2=0x56097c043980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.930+0000 7f89a6a00640 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 0x56097c043980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.930+0000 7f89a6a00640 1 -- 172.21.15.44:0/1151845882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 msgr2=0x56097c04f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.930+0000 7f89a6a00640 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c04f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.930+0000 7f89a6a00640 1 -- 172.21.15.44:0/1151845882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56097bca6210 con 0x56097bc652f0 2024-05-27T09:40:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.930+0000 7f89a6000640 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 0x56097c043980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:40:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.930+0000 7f89a6a00640 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c043440 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f899802a9e0 tx=0x7f8998009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ccc04ce5970c709 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.930+0000 7f8987e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:40:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.931+0000 7f8987e00640 1 -- 172.21.15.44:0/1151845882 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998035040 con 0x56097bc652f0 2024-05-27T09:40:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.931+0000 7f8987e00640 1 -- 172.21.15.44:0/1151845882 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f89980316a0 con 0x56097bc652f0 2024-05-27T09:40:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.938+0000 7f89b4636740 10 client.0 _ll_register_callbacks cb 0x56097bf17400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.938+0000 7f89b4636740 1 -- 172.21.15.44:0/1151845882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89940093c0 con 0x56097bc652f0 2024-05-27T09:40:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.939+0000 7f8987e00640 1 -- 172.21.15.44:0/1151845882 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f89980318a0 con 0x56097bc652f0 2024-05-27T09:40:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.939+0000 7f89b4636740 10 client.26059 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.939+0000 7f89b4636740 1 -- 172.21.15.44:0/1151845882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=26}) -- 0x56097bd72620 con 0x56097bc652f0 2024-05-27T09:40:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.940+0000 7f8987e00640 1 -- 172.21.15.44:0/1151845882 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8998031a40 con 0x56097bc652f0 2024-05-27T09:40:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.940+0000 7f89b4636740 10 client.26059 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.940+0000 7f89b4636740 10 client.26059 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:40:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.940+0000 7f89b4636740 20 client.26059 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.941+0000 7f89b4636740 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56097bc4cd80 0x56097c055450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.941+0000 7f89b4636740 4 client.26059 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:40:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.941+0000 7f89b4636740 1 -- 172.21.15.44:0/1151845882 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56097bfa3ae0 con 0x56097bc4cd80 2024-05-27T09:40:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.941+0000 7f89a6000640 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56097bc4cd80 0x56097c055450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.942+0000 7f89a6000640 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56097bc4cd80 0x56097c055450 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bdb95ba2b197eb4 server_cookie=d3273450191d7834 in_seq=0 out_seq=0 2024-05-27T09:40:35.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.942+0000 7f8987e00640 10 client.26059 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:35.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.942+0000 7f8987e00640 1 -- 172.21.15.44:0/1151845882 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56097bfa3ae0 con 0x56097bc4cd80 2024-05-27T09:40:35.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.943+0000 7f8987e00640 10 client.26059 handle_command_reply: tid=0 2024-05-27T09:40:35.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.943+0000 7f89b4636740 1 client.26059 shutdown 2024-05-27T09:40:35.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.944+0000 7f89b4636740 1 -- 172.21.15.44:0/1151845882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 msgr2=0x56097c043440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.944+0000 7f89b4636740 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c043440 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f899802a9e0 tx=0x7f8998009260 comp rx=0 tx=0).stop 2024-05-27T09:40:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.944+0000 7f89b4636740 1 -- 172.21.15.44:0/1151845882 shutdown_connections 2024-05-27T09:40:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.944+0000 7f89b4636740 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56097bc4cd80 0x56097c055450 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.944+0000 7f89b4636740 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c04f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.944+0000 7f89b4636740 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc6d4d0 0x56097c043980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.944+0000 7f8987e00640 0 client.26059 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.944+0000 7f89b4636740 1 --2- 172.21.15.44:0/1151845882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc652f0 0x56097c043440 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.944+0000 7f89b4636740 1 -- 172.21.15.44:0/1151845882 >> 172.21.15.44:0/1151845882 conn(0x56097bc82300 msgr2=0x56097c031220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.944+0000 7f89b4636740 1 -- 172.21.15.44:0/1151845882 shutdown_connections 2024-05-27T09:40:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.944+0000 7f89b4636740 1 -- 172.21.15.44:0/1151845882 wait complete. 2024-05-27T09:40:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.945+0000 7f89b4636740 20 client.26059 trim_cache size 0 max 16384 2024-05-27T09:40:35.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.979+0000 7f89b4636740 1 Processor -- start 2024-05-27T09:40:35.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.979+0000 7f89b4636740 1 -- start start 2024-05-27T09:40:35.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89b4636740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 0x56097c045b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89b4636740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 0x56097c0307f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89b4636740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c04f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89b4636740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81aff70 con 0x56097bc8e1b0 2024-05-27T09:40:35.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89b4636740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81ad840 con 0x56097bc652f0 2024-05-27T09:40:35.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89b4636740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81ab6b0 con 0x56097bc6d4d0 2024-05-27T09:40:35.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 0x56097c0307f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 0x56097c045b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 0x56097c0307f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:36198/0 (socket says 172.21.15.44:36198) 2024-05-27T09:40:35.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 0x56097c045b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53892/0 (socket says 172.21.15.44:53892) 2024-05-27T09:40:35.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89a6000640 1 -- 172.21.15.44:0/20627905 learned_addr learned my addr 172.21.15.44:0/20627905 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:35.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.980+0000 7f89a7400640 1 --2- 172.21.15.44:0/20627905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c04f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:35.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.981+0000 7f89a6000640 1 -- 172.21.15.44:0/20627905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 msgr2=0x56097c045b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:35.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.981+0000 7f89a6000640 1 --2- 172.21.15.44:0/20627905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 0x56097c045b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.981+0000 7f89a6000640 1 -- 172.21.15.44:0/20627905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 msgr2=0x56097c04f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:35.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.981+0000 7f89a6000640 1 --2- 172.21.15.44:0/20627905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c04f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.981+0000 7f89a6000640 1 -- 172.21.15.44:0/20627905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56097c044220 con 0x56097bc6d4d0 2024-05-27T09:40:35.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.981+0000 7f89a6000640 1 --2- 172.21.15.44:0/20627905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 0x56097c0307f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f899400c6a0 tx=0x7f89940a5040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0485f9fddddfacc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:35.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.981+0000 7f8992a00640 1 -- 172.21.15.44:0/20627905 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940cc0b0 con 0x56097bc6d4d0 2024-05-27T09:40:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.981+0000 7f8992a00640 1 -- 172.21.15.44:0/20627905 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89940a5b50 con 0x56097bc6d4d0 2024-05-27T09:40:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.981+0000 7f8992a00640 1 -- 172.21.15.44:0/20627905 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940b2290 con 0x56097bc6d4d0 2024-05-27T09:40:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.983+0000 7f89b4636740 1 -- 172.21.15.44:0/20627905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 msgr2=0x56097c0307f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.983+0000 7f89b4636740 1 --2- 172.21.15.44:0/20627905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 0x56097c0307f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f899400c6a0 tx=0x7f89940a5040 comp rx=0 tx=0).stop 2024-05-27T09:40:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.983+0000 7f89b4636740 1 -- 172.21.15.44:0/20627905 shutdown_connections 2024-05-27T09:40:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.983+0000 7f89b4636740 1 --2- 172.21.15.44:0/20627905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c04f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.983+0000 7f89b4636740 1 --2- 172.21.15.44:0/20627905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 0x56097c0307f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.983+0000 7f89b4636740 1 --2- 172.21.15.44:0/20627905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 0x56097c045b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:35.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.983+0000 7f89b4636740 1 -- 172.21.15.44:0/20627905 >> 172.21.15.44:0/20627905 conn(0x56097bc82300 msgr2=0x56097c033140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:35.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.983+0000 7f89b4636740 1 -- 172.21.15.44:0/20627905 shutdown_connections 2024-05-27T09:40:35.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.983+0000 7f89b4636740 1 -- 172.21.15.44:0/20627905 wait complete. 2024-05-27T09:40:35.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.984+0000 7f89b4636740 1 -- start start 2024-05-27T09:40:35.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.985+0000 7f89b4636740 1 Processor -- start 2024-05-27T09:40:35.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.985+0000 7f89b4636740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 0x56097c04b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.985+0000 7f89b4636740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 0x56097c03d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.985+0000 7f89b4636740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c03dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.985+0000 7f89b4636740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81abd00 con 0x56097bc8e1b0 2024-05-27T09:40:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.985+0000 7f89b4636740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81ad170 con 0x56097bc652f0 2024-05-27T09:40:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.985+0000 7f89b4636740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81ac3d0 con 0x56097bc6d4d0 2024-05-27T09:40:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.985+0000 7f89b4636740 10 client.0 _ll_register_callbacks cb 0x7f8994005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.986+0000 7f89a7400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c03dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.986+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 0x56097c04b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.986+0000 7f89a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 0x56097c03d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.986+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 0x56097c04b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53900/0 (socket says 172.21.15.44:53900) 2024-05-27T09:40:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.986+0000 7f89a6a00640 1 -- 172.21.15.44:0/7055862 learned_addr learned my addr 172.21.15.44:0/7055862 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.986+0000 7f89a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 0x56097c03d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:36212/0 (socket says 172.21.15.44:36212) 2024-05-27T09:40:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.986+0000 7f89a6a00640 1 -- 172.21.15.44:0/7055862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 msgr2=0x56097c03d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.986+0000 7f89a6a00640 1 --2- 172.21.15.44:0/7055862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 0x56097c03d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.986+0000 7f89a6a00640 1 -- 172.21.15.44:0/7055862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 msgr2=0x56097c03dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.986+0000 7f89a6a00640 1 --2- 172.21.15.44:0/7055862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c03dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.987+0000 7f89a6a00640 1 -- 172.21.15.44:0/7055862 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56097c03e320 con 0x56097bc652f0 2024-05-27T09:40:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.987+0000 7f89a6a00640 1 --2- 172.21.15.44:0/7055862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 0x56097c04b9d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8998005ab0 tx=0x7f8998041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c092686695c1f726 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.987+0000 7f8987e00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:40:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.987+0000 7f8987e00640 1 -- 172.21.15.44:0/7055862 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998035070 con 0x56097bc652f0 2024-05-27T09:40:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.988+0000 7f8987e00640 1 -- 172.21.15.44:0/7055862 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f899803c890 con 0x56097bc652f0 2024-05-27T09:40:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.988+0000 7f89b4636740 1 -- 172.21.15.44:0/7055862 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89940a5b50 con 0x56097bc652f0 2024-05-27T09:40:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.988+0000 7f8987e00640 1 -- 172.21.15.44:0/7055862 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f899803c440 con 0x56097bc652f0 2024-05-27T09:40:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.988+0000 7f89b4636740 10 client.26054 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.988+0000 7f89b4636740 1 -- 172.21.15.44:0/7055862 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56097c04b160 con 0x56097bc652f0 2024-05-27T09:40:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.988+0000 7f8987e00640 1 -- 172.21.15.44:0/7055862 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89940a5b50 con 0x56097bc652f0 2024-05-27T09:40:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.989+0000 7f8987e00640 1 -- 172.21.15.44:0/7055862 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f899804a900 con 0x56097bc652f0 2024-05-27T09:40:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.989+0000 7f89b4636740 10 client.26054 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.989+0000 7f89b4636740 10 client.26054 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:40:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.989+0000 7f89b4636740 20 client.26054 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.989+0000 7f89b4636740 1 --2- 172.21.15.44:0/7055862 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56097bc221c0 0x56097c0bfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.989+0000 7f89b4636740 4 client.26054 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:40:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.989+0000 7f89b4636740 1 -- 172.21.15.44:0/7055862 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56097bf9d2c0 con 0x56097bc221c0 2024-05-27T09:40:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.989+0000 7f89a6000640 1 --2- 172.21.15.44:0/7055862 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56097bc221c0 0x56097c0bfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.990+0000 7f8987e00640 10 client.26054 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:35.990+0000 7f89a6000640 1 --2- 172.21.15.44:0/7055862 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56097bc221c0 0x56097c0bfe70 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e806a5eff8d98783 server_cookie=7bb3de79ccd88466 in_seq=0 out_seq=0 2024-05-27T09:40:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f8987e00640 1 -- 172.21.15.44:0/7055862 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56097bf9d2c0 con 0x56097bc221c0 2024-05-27T09:40:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f8987e00640 10 client.26054 handle_command_reply: tid=0 2024-05-27T09:40:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f89b4636740 1 client.26054 shutdown 2024-05-27T09:40:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f89b4636740 1 -- 172.21.15.44:0/7055862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 msgr2=0x56097c04b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f89b4636740 1 --2- 172.21.15.44:0/7055862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 0x56097c04b9d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8998005ab0 tx=0x7f8998041000 comp rx=0 tx=0).stop 2024-05-27T09:40:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f89b4636740 1 -- 172.21.15.44:0/7055862 shutdown_connections 2024-05-27T09:40:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f89b4636740 1 --2- 172.21.15.44:0/7055862 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56097bc221c0 0x56097c0bfe70 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f89b4636740 1 --2- 172.21.15.44:0/7055862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56097bc8e1b0 0x56097c03dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f89b4636740 1 --2- 172.21.15.44:0/7055862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56097bc6d4d0 0x56097c03d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f89b4636740 1 --2- 172.21.15.44:0/7055862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56097bc652f0 0x56097c04b9d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f8987e00640 0 client.26054 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:36.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f89b4636740 1 -- 172.21.15.44:0/7055862 >> 172.21.15.44:0/7055862 conn(0x56097bc82300 msgr2=0x56097c033140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:36.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f89b4636740 1 -- 172.21.15.44:0/7055862 shutdown_connections 2024-05-27T09:40:36.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.011+0000 7f89b4636740 1 -- 172.21.15.44:0/7055862 wait complete. 2024-05-27T09:40:36.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.012+0000 7f89b4636740 20 client.26054 trim_cache size 0 max 16384 2024-05-27T09:40:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:40:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:40:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:40:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.012+0000 7f89a7e00640 1 -- 172.21.15.44:0/3757063580 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f898c0b2aa0 msgr2=0x7f898c0b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.012+0000 7f89a7e00640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f898c0b2aa0 0x7f898c0b4fb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8998000c00 tx=0x7f8998005a20 comp rx=0 tx=0).stop 2024-05-27T09:40:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.012+0000 7f89a7e00640 1 -- 172.21.15.44:0/3757063580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a810bcd0 msgr2=0x7f89a81a2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.012+0000 7f89a7e00640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a810bcd0 0x7f89a81a2af0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f89a806c3a0 tx=0x7f899400ee50 comp rx=0 tx=0).stop 2024-05-27T09:40:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.013+0000 7f89a7e00640 1 -- 172.21.15.44:0/3757063580 shutdown_connections 2024-05-27T09:40:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.013+0000 7f89a7e00640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f898c0b2aa0 0x7f898c0b4fb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.013+0000 7f89a7e00640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89a810c170 0x7f89a819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.013+0000 7f89a7e00640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a810bcd0 0x7f89a81a2af0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.013+0000 7f89a7e00640 1 --2- 172.21.15.44:0/3757063580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a8106420 0x7f89a81a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.013+0000 7f89a7e00640 1 -- 172.21.15.44:0/3757063580 >> 172.21.15.44:0/3757063580 conn(0x7f89a81013d0 msgr2=0x7f89a81017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.013+0000 7f89a7e00640 1 -- 172.21.15.44:0/3757063580 shutdown_connections 2024-05-27T09:40:36.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:36.013+0000 7f89a7e00640 1 -- 172.21.15.44:0/3757063580 wait complete. 2024-05-27T09:40:36.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:35 smithi050 ceph-mon[36371]: pgmap v163: 97 pgs: 97 active+clean; 3.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-27T09:40:36.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:35 smithi189 ceph-mon[36671]: pgmap v163: 97 pgs: 97 active+clean; 3.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-27T09:40:36.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:35 smithi044 ceph-mon[31618]: pgmap v163: 97 pgs: 97 active+clean; 3.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-27T09:40:36.892 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:36.892 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:36.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.895+0000 7f5ba3000700 1 -- 172.21.15.44:0/2954083345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b9c10c5a0 msgr2=0x7f5b9c10c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:36.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.895+0000 7f5ba3000700 1 --2- 172.21.15.44:0/2954083345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b9c10c5a0 0x7f5b9c10c9b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009990 tx=0x7f5b8c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.896+0000 7f5ba3000700 1 -- 172.21.15.44:0/2954083345 shutdown_connections 2024-05-27T09:40:36.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.896+0000 7f5ba3000700 1 --2- 172.21.15.44:0/2954083345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c106eb0 0x7f5b9c113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.896+0000 7f5ba3000700 1 --2- 172.21.15.44:0/2954083345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b9c10c5a0 0x7f5b9c10c9b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.896+0000 7f5ba3000700 1 --2- 172.21.15.44:0/2954083345 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c106560 0x7f5b9c106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.896+0000 7f5ba3000700 1 -- 172.21.15.44:0/2954083345 >> 172.21.15.44:0/2954083345 conn(0x7f5b9c102040 msgr2=0x7f5b9c104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:36.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.896+0000 7f5ba3000700 1 -- 172.21.15.44:0/2954083345 shutdown_connections 2024-05-27T09:40:36.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.896+0000 7f5ba3000700 1 -- 172.21.15.44:0/2954083345 wait complete. 2024-05-27T09:40:36.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.897+0000 7f5ba3000700 1 Processor -- start 2024-05-27T09:40:36.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.898+0000 7f5ba3000700 1 -- start start 2024-05-27T09:40:36.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.899+0000 7f5ba3000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b9c106560 0x7f5b9c19d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.899+0000 7f5ba3000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c106eb0 0x7f5b9c19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.899+0000 7f5ba3000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c10c5a0 0x7f5b9c1a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.899+0000 7f5ba3000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c116700 con 0x7f5b9c106eb0 2024-05-27T09:40:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.899+0000 7f5ba3000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c116580 con 0x7f5b9c106560 2024-05-27T09:40:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.899+0000 7f5ba3000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c116880 con 0x7f5b9c10c5a0 2024-05-27T09:40:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.899+0000 7f5b9be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b9c106560 0x7f5b9c19d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.899+0000 7f5b9be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b9c106560 0x7f5b9c19d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53926/0 (socket says 172.21.15.44:53926) 2024-05-27T09:40:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.899+0000 7f5b9be00700 1 -- 172.21.15.44:0/1738262256 learned_addr learned my addr 172.21.15.44:0/1738262256 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.899+0000 7f5b9b400700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c106eb0 0x7f5b9c19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:36.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.899+0000 7f5ba1000700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c10c5a0 0x7f5b9c1a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:36.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.900+0000 7f5b9be00700 1 -- 172.21.15.44:0/1738262256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c10c5a0 msgr2=0x7f5b9c1a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:36.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.900+0000 7f5b9be00700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c10c5a0 0x7f5b9c1a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.900+0000 7f5b9be00700 1 -- 172.21.15.44:0/1738262256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c106eb0 msgr2=0x7f5b9c19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:36.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.900+0000 7f5b9be00700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c106eb0 0x7f5b9c19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.900+0000 7f5b9be00700 1 -- 172.21.15.44:0/1738262256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9c1a1f80 con 0x7f5b9c106560 2024-05-27T09:40:36.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.900+0000 7f5ba1000700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c10c5a0 0x7f5b9c1a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:40:36.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.900+0000 7f5b9be00700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b9c106560 0x7f5b9c19d0b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c000c00 tx=0x7f5b8c009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:36.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.900+0000 7f5b98c00700 1 -- 172.21.15.44:0/1738262256 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c027070 con 0x7f5b9c106560 2024-05-27T09:40:36.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.901+0000 7f5ba3000700 1 -- 172.21.15.44:0/1738262256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b9c1a2210 con 0x7f5b9c106560 2024-05-27T09:40:36.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.901+0000 7f5b98c00700 1 -- 172.21.15.44:0/1738262256 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b8c022470 con 0x7f5b9c106560 2024-05-27T09:40:36.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.901+0000 7f5b98c00700 1 -- 172.21.15.44:0/1738262256 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c017650 con 0x7f5b9c106560 2024-05-27T09:40:36.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.901+0000 7f5ba3000700 1 -- 172.21.15.44:0/1738262256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b9c1a9b90 con 0x7f5b9c106560 2024-05-27T09:40:36.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.902+0000 7f5b9b400700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c106eb0 0x7f5b9c19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:40:36.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.901+0000 7fa8c6c00700 1 -- 172.21.15.44:0/1523680119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8c0109880 msgr2=0x7fa8c01111e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:36.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.901+0000 7fa8c6c00700 1 --2- 172.21.15.44:0/1523680119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8c0109880 0x7fa8c01111e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009f90 tx=0x7fa8b800b330 comp rx=0 tx=0).stop 2024-05-27T09:40:36.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.902+0000 7fa8c6c00700 1 -- 172.21.15.44:0/1523680119 shutdown_connections 2024-05-27T09:40:36.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.902+0000 7fa8c6c00700 1 --2- 172.21.15.44:0/1523680119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8c0109880 0x7fa8c01111e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.902+0000 7fa8c6c00700 1 --2- 172.21.15.44:0/1523680119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8c0104600 0x7fa8c0104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.902+0000 7fa8c6c00700 1 --2- 172.21.15.44:0/1523680119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c0108f30 0x7fa8c0109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.902+0000 7fa8c6c00700 1 -- 172.21.15.44:0/1523680119 >> 172.21.15.44:0/1523680119 conn(0x7fa8c007af60 msgr2=0x7fa8c007b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:36.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.902+0000 7fa8c6c00700 1 -- 172.21.15.44:0/1523680119 shutdown_connections 2024-05-27T09:40:36.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.903+0000 7f5ba3000700 1 -- 172.21.15.44:0/1738262256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b9c0512b0 con 0x7f5b9c106560 2024-05-27T09:40:36.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.903+0000 7fa8c6c00700 1 -- 172.21.15.44:0/1523680119 wait complete. 2024-05-27T09:40:36.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.905+0000 7fa8c6c00700 1 Processor -- start 2024-05-27T09:40:36.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.905+0000 7f5b98c00700 1 -- 172.21.15.44:0/1738262256 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5b8c02a070 con 0x7f5b9c106560 2024-05-27T09:40:36.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.905+0000 7fa8c6c00700 1 -- start start 2024-05-27T09:40:36.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.905+0000 7fa8c6c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c0104600 0x7fa8c019dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:36.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.905+0000 7fa8c6c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8c0108f30 0x7fa8c019e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:36.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.905+0000 7fa8c6c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8c0109880 0x7fa8c01a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:36.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.905+0000 7fa8c6c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01144b0 con 0x7fa8c0108f30 2024-05-27T09:40:36.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.905+0000 7fa8c6c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c0114330 con 0x7fa8c0109880 2024-05-27T09:40:36.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.905+0000 7fa8c6c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c0114630 con 0x7fa8c0104600 2024-05-27T09:40:36.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.906+0000 7fa8bf400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8c0108f30 0x7fa8c019e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:36.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.906+0000 7fa8bf400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8c0108f30 0x7fa8c019e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33860/0 (socket says 172.21.15.44:33860) 2024-05-27T09:40:36.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.906+0000 7fa8bf400700 1 -- 172.21.15.44:0/3566298315 learned_addr learned my addr 172.21.15.44:0/3566298315 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:36.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.906+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8c0109880 0x7fa8c01a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:36.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.907+0000 7fa8c4c00700 1 -- 172.21.15.44:0/3566298315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c0104600 msgr2=0x7fa8c019dd90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T09:40:36.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.907+0000 7f5b98c00700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b800b2af0 0x7f5b800b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:36.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.907+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c0104600 0x7fa8c019dd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.907+0000 7fa8c4c00700 1 -- 172.21.15.44:0/3566298315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8c0108f30 msgr2=0x7fa8c019e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:36.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.907+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8c0108f30 0x7fa8c019e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:36.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.907+0000 7fa8c4c00700 1 -- 172.21.15.44:0/3566298315 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c01a9350 con 0x7fa8c0109880 2024-05-27T09:40:36.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.907+0000 7f5b98c00700 1 -- 172.21.15.44:0/1738262256 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f5b8c0e34e0 con 0x7f5b9c106560 2024-05-27T09:40:36.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.907+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8c0109880 0x7fa8c01a8e10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009790 tx=0x7fa8b800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:36.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.908+0000 7f5b9b400700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b800b2af0 0x7f5b800b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:36.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.909+0000 7f5b9b400700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b800b2af0 0x7f5b800b5000 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c105650 tx=0x7f5b88009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:36.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.911+0000 7fa8bcc00700 1 -- 172.21.15.44:0/3566298315 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8021070 con 0x7fa8c0109880 2024-05-27T09:40:36.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.911+0000 7fa8bcc00700 1 -- 172.21.15.44:0/3566298315 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8b801d450 con 0x7fa8c0109880 2024-05-27T09:40:36.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.911+0000 7fa8bcc00700 1 -- 172.21.15.44:0/3566298315 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b80052a0 con 0x7fa8c0109880 2024-05-27T09:40:36.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.911+0000 7fa8c6c00700 1 -- 172.21.15.44:0/3566298315 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c01a9520 con 0x7fa8c0109880 2024-05-27T09:40:36.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.911+0000 7fa8c6c00700 1 -- 172.21.15.44:0/3566298315 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c01a9910 con 0x7fa8c0109880 2024-05-27T09:40:36.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.911+0000 7f5b98c00700 1 -- 172.21.15.44:0/1738262256 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5b8c013070 con 0x7f5b9c106560 2024-05-27T09:40:36.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.915+0000 7fa8bcc00700 1 -- 172.21.15.44:0/3566298315 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa8b8024070 con 0x7fa8c0109880 2024-05-27T09:40:36.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.917+0000 7fa8bcc00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa8a40b2a70 0x7fa8a40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:36.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.917+0000 7fa8bcc00700 1 -- 172.21.15.44:0/3566298315 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa8b80e34a0 con 0x7fa8c0109880 2024-05-27T09:40:36.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.917+0000 7fa8bfe00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa8a40b2a70 0x7fa8a40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:36.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:36.919+0000 7fa8bfe00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa8a40b2a70 0x7fa8a40b4f80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0005fd0 tx=0x7fa8b0009310 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:37.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.055+0000 7fa8d57c0b80 1 Processor -- start 2024-05-27T09:40:37.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.055+0000 7fa8d57c0b80 1 -- start start 2024-05-27T09:40:37.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.056+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1973ec0 0x5624a1af8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.056+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af8db0 0x5624a1af28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.056+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1af2e30 0x5624a1af3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.056+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5624a1a20ec0 con 0x5624a1af8db0 2024-05-27T09:40:37.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.056+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5624a1a10a60 con 0x5624a1af2e30 2024-05-27T09:40:37.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.056+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5624a1a14aa0 con 0x5624a1973ec0 2024-05-27T09:40:37.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.057+0000 7fa8bf400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af8db0 0x5624a1af28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.057+0000 7fa8bf400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af8db0 0x5624a1af28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33868/0 (socket says 172.21.15.44:33868) 2024-05-27T09:40:37.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.057+0000 7fa8bfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1973ec0 0x5624a1af8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.057+0000 7fa8bf400700 1 -- 172.21.15.44:0/3694057482 learned_addr learned my addr 172.21.15.44:0/3694057482 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:37.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.057+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3694057482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1af2e30 0x5624a1af3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.057+0000 7fa8bf400700 1 -- 172.21.15.44:0/3694057482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1973ec0 msgr2=0x5624a1af8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:37.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.057+0000 7fa8bf400700 1 --2- 172.21.15.44:0/3694057482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1973ec0 0x5624a1af8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.057+0000 7fa8bf400700 1 -- 172.21.15.44:0/3694057482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1af2e30 msgr2=0x5624a1af3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:37.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.057+0000 7fa8bf400700 1 --2- 172.21.15.44:0/3694057482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1af2e30 0x5624a1af3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.057+0000 7fa8bf400700 1 -- 172.21.15.44:0/3694057482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a19d94a0 con 0x5624a1af8db0 2024-05-27T09:40:37.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.058+0000 7fa8bf400700 1 --2- 172.21.15.44:0/3694057482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af8db0 0x5624a1af28f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009d30 tx=0x7fa8ac011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e21e3ed2d6edc20 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:37.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.058+0000 7fa8b5600700 1 -- 172.21.15.44:0/3694057482 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac004a30 con 0x5624a1af8db0 2024-05-27T09:40:37.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.058+0000 7fa8b5600700 1 -- 172.21.15.44:0/3694057482 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8ac0054d0 con 0x5624a1af8db0 2024-05-27T09:40:37.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.058+0000 7fa8b5600700 1 -- 172.21.15.44:0/3694057482 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac019910 con 0x5624a1af8db0 2024-05-27T09:40:37.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.060+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3694057482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af8db0 msgr2=0x5624a1af28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:37.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.060+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3694057482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af8db0 0x5624a1af28f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009d30 tx=0x7fa8ac011720 comp rx=0 tx=0).stop 2024-05-27T09:40:37.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.060+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3694057482 shutdown_connections 2024-05-27T09:40:37.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.060+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3694057482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1af2e30 0x5624a1af3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.060+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3694057482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1973ec0 0x5624a1af8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.060+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3694057482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af8db0 0x5624a1af28f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.060+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3694057482 >> 172.21.15.44:0/3694057482 conn(0x5624a19d4040 msgr2=0x5624a1aed950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:37.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.060+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3694057482 shutdown_connections 2024-05-27T09:40:37.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.060+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3694057482 wait complete. 2024-05-27T09:40:37.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.061+0000 7fa8d57c0b80 1 -- start start 2024-05-27T09:40:37.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.061+0000 7fa8d57c0b80 1 Processor -- start 2024-05-27T09:40:37.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1973ec0 0x5624a1afc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 0x5624a1afcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1afd110 0x5624a1b08bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5624a1917020 con 0x5624a1afd110 2024-05-27T09:40:37.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5624a19b17e0 con 0x5624a1973ec0 2024-05-27T09:40:37.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5624a1aff880 con 0x5624a1af2e30 2024-05-27T09:40:37.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8d57c0b80 10 client.0 _ll_register_callbacks cb 0x5624a19bacf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8bf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 0x5624a1afcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8bfe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1973ec0 0x5624a1afc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8bf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 0x5624a1afcbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:36262/0 (socket says 172.21.15.44:36262) 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8bf400700 1 -- 172.21.15.44:0/3493455947 learned_addr learned my addr 172.21.15.44:0/3493455947 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.062+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1afd110 0x5624a1b08bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.063+0000 7fa8bf400700 1 -- 172.21.15.44:0/3493455947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1973ec0 msgr2=0x5624a1afc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.063+0000 7fa8bf400700 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1973ec0 0x5624a1afc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.063+0000 7fa8bf400700 1 -- 172.21.15.44:0/3493455947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1afd110 msgr2=0x5624a1b08bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.063+0000 7fa8bf400700 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1afd110 0x5624a1b08bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.063+0000 7fa8bf400700 1 -- 172.21.15.44:0/3493455947 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624a1960950 con 0x5624a1af2e30 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.063+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1afd110 0x5624a1b08bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.063+0000 7fa8bf400700 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 0x5624a1afcbd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac01b040 tx=0x7fa8ac004980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5040a039c4ad57af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.063+0000 7fa8aa000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:40:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.063+0000 7fa8aa000700 1 -- 172.21.15.44:0/3493455947 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac029030 con 0x5624a1af2e30 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.064+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3493455947 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8ac0054d0 con 0x5624a1af2e30 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.064+0000 7fa8aa000700 1 -- 172.21.15.44:0/3493455947 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa8ac005680 con 0x5624a1af2e30 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.064+0000 7fa8aa000700 1 -- 172.21.15.44:0/3493455947 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa8ac033050 con 0x5624a1af2e30 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.064+0000 7fa8d57c0b80 10 client.26077 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.064+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3493455947 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5624a19e10e0 con 0x5624a1af2e30 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.064+0000 7fa8aa000700 1 -- 172.21.15.44:0/3493455947 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8ac0054d0 con 0x5624a1af2e30 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.065+0000 7fa8aa000700 1 -- 172.21.15.44:0/3493455947 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa8ac025470 con 0x5624a1af2e30 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.065+0000 7fa8d57c0b80 10 client.26077 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.065+0000 7fa8d57c0b80 10 client.26077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.065+0000 7fa8d57c0b80 20 client.26077 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.065+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624a1b0cc50 0x5624a1b0f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.065+0000 7fa8d57c0b80 4 client.26077 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:40:37.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.065+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3493455947 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624a1a22e10 con 0x5624a1b0cc50 2024-05-27T09:40:37.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.065+0000 7fa8bfe00700 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624a1b0cc50 0x5624a1b0f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.066+0000 7fa8bfe00700 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624a1b0cc50 0x5624a1b0f070 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=359450cf623a44bd server_cookie=d6b2699b5ba0449 in_seq=0 out_seq=0 2024-05-27T09:40:37.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.066+0000 7fa8aa000700 10 client.26077 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:37.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.067+0000 7fa8aa000700 1 -- 172.21.15.44:0/3493455947 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5624a1a22e10 con 0x5624a1b0cc50 2024-05-27T09:40:37.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.067+0000 7fa8aa000700 10 client.26077 handle_command_reply: tid=0 2024-05-27T09:40:37.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.067+0000 7fa8d57c0b80 1 client.26077 shutdown 2024-05-27T09:40:37.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.067+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3493455947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 msgr2=0x5624a1afcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:37.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.067+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 0x5624a1afcbd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac01b040 tx=0x7fa8ac004980 comp rx=0 tx=0).stop 2024-05-27T09:40:37.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.068+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3493455947 shutdown_connections 2024-05-27T09:40:37.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.068+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624a1b0cc50 0x5624a1b0f070 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.068+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1afd110 0x5624a1b08bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.068+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1973ec0 0x5624a1afc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.068+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3493455947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 0x5624a1afcbd0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.068+0000 7fa8aa000700 0 client.26077 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:37.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.068+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3493455947 >> 172.21.15.44:0/3493455947 conn(0x5624a19d4040 msgr2=0x5624a1aed600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:37.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.068+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3493455947 shutdown_connections 2024-05-27T09:40:37.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.068+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3493455947 wait complete. 2024-05-27T09:40:37.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.068+0000 7fa8d57c0b80 20 client.26077 trim_cache size 0 max 16384 2024-05-27T09:40:37.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.110+0000 7fa8d57c0b80 1 Processor -- start 2024-05-27T09:40:37.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.110+0000 7fa8d57c0b80 1 -- start start 2024-05-27T09:40:37.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.111+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1973ec0 0x5624a1afeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.111+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 0x5624a1af6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.111+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 0x5624a1af9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.111+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01b37a0 con 0x5624a1973ec0 2024-05-27T09:40:37.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.111+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01b1030 con 0x5624a1afd110 2024-05-27T09:40:37.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.111+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01aeea0 con 0x5624a1af2e30 2024-05-27T09:40:37.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.111+0000 7fa8bfe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1973ec0 0x5624a1afeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.111+0000 7fa8bfe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1973ec0 0x5624a1afeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33890/0 (socket says 172.21.15.44:33890) 2024-05-27T09:40:37.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.111+0000 7fa8bfe00700 1 -- 172.21.15.44:0/3975071057 learned_addr learned my addr 172.21.15.44:0/3975071057 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:37.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.111+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3975071057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 0x5624a1af9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.111+0000 7fa8bf400700 1 --2- 172.21.15.44:0/3975071057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 0x5624a1af6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.112+0000 7fa8c4c00700 1 -- 172.21.15.44:0/3975071057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 msgr2=0x5624a1af6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:37.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.112+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3975071057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 0x5624a1af6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.112+0000 7fa8c4c00700 1 -- 172.21.15.44:0/3975071057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1973ec0 msgr2=0x5624a1afeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:37.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.112+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3975071057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1973ec0 0x5624a1afeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.112+0000 7fa8c4c00700 1 -- 172.21.15.44:0/3975071057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a1afd750 con 0x5624a1afd110 2024-05-27T09:40:37.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.112+0000 7fa8bfe00700 1 --2- 172.21.15.44:0/3975071057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1973ec0 0x5624a1afeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:37.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.112+0000 7fa8bf400700 1 --2- 172.21.15.44:0/3975071057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 0x5624a1af6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:37.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.112+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3975071057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 0x5624a1af9bc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80e0bc0 tx=0x7fa8b80b1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97f6c7a5b5820f68 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.112+0000 7fa8b4c00700 1 -- 172.21.15.44:0/3975071057 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b802b070 con 0x5624a1afd110 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.113+0000 7fa8b4c00700 1 -- 172.21.15.44:0/3975071057 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8b80aa2f0 con 0x5624a1afd110 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.113+0000 7fa8b4c00700 1 -- 172.21.15.44:0/3975071057 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b80a6a70 con 0x5624a1afd110 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.114+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3975071057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 msgr2=0x5624a1af9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.114+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3975071057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 0x5624a1af9bc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80e0bc0 tx=0x7fa8b80b1040 comp rx=0 tx=0).stop 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.114+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3975071057 shutdown_connections 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.114+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3975071057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 0x5624a1af9bc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.114+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3975071057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1973ec0 0x5624a1afeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.114+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3975071057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1af2e30 0x5624a1af6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.114+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3975071057 >> 172.21.15.44:0/3975071057 conn(0x5624a19d4040 msgr2=0x5624a1af94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.114+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3975071057 shutdown_connections 2024-05-27T09:40:37.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.114+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3975071057 wait complete. 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.114+0000 7fa8d57c0b80 1 -- start start 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.115+0000 7fa8d57c0b80 1 Processor -- start 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.115+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1973ec0 0x5624a1b04fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.115+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af2e30 0x5624a1b01ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.115+0000 7fa8d57c0b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 0x5624a1b02020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.115+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01af4f0 con 0x5624a1af2e30 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.115+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01b0960 con 0x5624a1afd110 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.115+0000 7fa8d57c0b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c01afbc0 con 0x5624a1973ec0 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.115+0000 7fa8d57c0b80 10 client.0 _ll_register_callbacks cb 0x5624a19bacf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.116+0000 7fa8c4c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 0x5624a1b02020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.116+0000 7fa8bf400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af2e30 0x5624a1b01ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.116+0000 7fa8c4c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 0x5624a1b02020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:53982/0 (socket says 172.21.15.44:53982) 2024-05-27T09:40:37.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.116+0000 7fa8c4c00700 1 -- 172.21.15.44:0/3089276472 learned_addr learned my addr 172.21.15.44:0/3089276472 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:37.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.116+0000 7fa8bfe00700 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1973ec0 0x5624a1b04fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.116+0000 7fa8c4c00700 1 -- 172.21.15.44:0/3089276472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1973ec0 msgr2=0x5624a1b04fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:37.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.116+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1973ec0 0x5624a1b04fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.116+0000 7fa8c4c00700 1 -- 172.21.15.44:0/3089276472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af2e30 msgr2=0x5624a1b01ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:37.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.116+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af2e30 0x5624a1b01ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.116+0000 7fa8c4c00700 1 -- 172.21.15.44:0/3089276472 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624a1b02560 con 0x5624a1afd110 2024-05-27T09:40:37.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.117+0000 7fa8c4c00700 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 0x5624a1b02020 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80a6810 tx=0x7fa8b80098a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8117262a296839e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:37.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.117+0000 7fa8aa000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:40:37.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.117+0000 7fa8aa000700 1 -- 172.21.15.44:0/3089276472 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b802b070 con 0x5624a1afd110 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.117+0000 7fa8aa000700 1 -- 172.21.15.44:0/3089276472 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fa8b80a6dd0 con 0x5624a1afd110 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.117+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3089276472 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8b80aa2f0 con 0x5624a1afd110 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.117+0000 7fa8aa000700 1 -- 172.21.15.44:0/3089276472 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa8b80b10b0 con 0x5624a1afd110 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.117+0000 7fa8d57c0b80 10 client.26078 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.117+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3089276472 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5624a1b782f0 con 0x5624a1afd110 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.118+0000 7fa8aa000700 1 -- 172.21.15.44:0/3089276472 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8b80aa2f0 con 0x5624a1afd110 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.118+0000 7fa8aa000700 1 -- 172.21.15.44:0/3089276472 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa8b8011070 con 0x5624a1afd110 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.118+0000 7fa8d57c0b80 10 client.26078 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.118+0000 7fa8d57c0b80 10 client.26078 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.118+0000 7fa8d57c0b80 20 client.26078 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.118+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624a1b78890 0x5624a1b7acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.118+0000 7fa8d57c0b80 4 client.26078 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:40:37.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.118+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3089276472 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5624a1a1c5f0 con 0x5624a1b78890 2024-05-27T09:40:37.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.118+0000 7fa8bfe00700 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624a1b78890 0x5624a1b7acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:37.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.119+0000 7fa8bfe00700 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624a1b78890 0x5624a1b7acb0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5f65fb28c32b886 server_cookie=7ffccec9dce6efeb in_seq=0 out_seq=0 2024-05-27T09:40:37.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.119+0000 7fa8aa000700 10 client.26078 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:37.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.120+0000 7fa8aa000700 1 -- 172.21.15.44:0/3089276472 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5624a1a1c5f0 con 0x5624a1b78890 2024-05-27T09:40:37.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.120+0000 7fa8aa000700 10 client.26078 handle_command_reply: tid=0 2024-05-27T09:40:37.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.120+0000 7fa8d57c0b80 1 client.26078 shutdown 2024-05-27T09:40:37.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3089276472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 msgr2=0x5624a1b02020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:37.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 0x5624a1b02020 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80a6810 tx=0x7fa8b80098a0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3089276472 shutdown_connections 2024-05-27T09:40:37.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624a1b78890 0x5624a1b7acb0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624a1afd110 0x5624a1b02020 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a1973ec0 0x5624a1b04fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8d57c0b80 1 --2- 172.21.15.44:0/3089276472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624a1af2e30 0x5624a1b01ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8aa000700 0 client.26078 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:37.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3089276472 >> 172.21.15.44:0/3089276472 conn(0x5624a19d4040 msgr2=0x5624a1af6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:37.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3089276472 shutdown_connections 2024-05-27T09:40:37.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8d57c0b80 1 -- 172.21.15.44:0/3089276472 wait complete. 2024-05-27T09:40:37.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.121+0000 7fa8d57c0b80 20 client.26078 trim_cache size 0 max 16384 2024-05-27T09:40:37.123 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:40:37.123 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:40:37.123 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "21acf208-afa2-4d56-b5ea-fe9d8026120d", 2024-05-27T09:40:37.124 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:40:37.124 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:40:37.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.122+0000 7fa8c6c00700 1 -- 172.21.15.44:0/3566298315 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa8a40b2a70 msgr2=0x7fa8a40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:37.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.122+0000 7fa8c6c00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa8a40b2a70 0x7fa8a40b4f80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0005fd0 tx=0x7fa8b0009310 comp rx=0 tx=0).stop 2024-05-27T09:40:37.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.122+0000 7fa8c6c00700 1 -- 172.21.15.44:0/3566298315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8c0109880 msgr2=0x7fa8c01a8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:37.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.122+0000 7fa8c6c00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8c0109880 0x7fa8c01a8e10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009790 tx=0x7fa8b800ccb0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.123+0000 7fa8c6c00700 1 -- 172.21.15.44:0/3566298315 shutdown_connections 2024-05-27T09:40:37.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.123+0000 7fa8c6c00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa8a40b2a70 0x7fa8a40b4f80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.123+0000 7fa8c6c00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8c0109880 0x7fa8c01a8e10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.123+0000 7fa8c6c00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c0104600 0x7fa8c019dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.123+0000 7fa8c6c00700 1 --2- 172.21.15.44:0/3566298315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8c0108f30 0x7fa8c019e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.123+0000 7fa8c6c00700 1 -- 172.21.15.44:0/3566298315 >> 172.21.15.44:0/3566298315 conn(0x7fa8c007af60 msgr2=0x7fa8c0076190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:37.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.123+0000 7fa8c6c00700 1 -- 172.21.15.44:0/3566298315 shutdown_connections 2024-05-27T09:40:37.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.123+0000 7fa8c6c00700 1 -- 172.21.15.44:0/3566298315 wait complete. 2024-05-27T09:40:37.152 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:37.152 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:37.173 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:37.173 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:37.173 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 99 -lt 100 ']' 2024-05-27T09:40:37.174 INFO:tasks.workunit.client.0.smithi044.stderr:++ expr 99 + 1 2024-05-27T09:40:37.240 INFO:tasks.workunit.client.0.smithi044.stderr:+ c=100 2024-05-27T09:40:37.240 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:37.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.245+0000 7f5ba3000700 1 -- 172.21.15.44:0/1738262256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b9c02e5e0 con 0x7f5b9c106560 2024-05-27T09:40:37.247 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:40:37.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.246+0000 7f5b98c00700 1 -- 172.21.15.44:0/1738262256 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7f5b8c0aaf20 con 0x7f5b9c106560 2024-05-27T09:40:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:40:37.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.249+0000 7f5ba3000700 1 -- 172.21.15.44:0/1738262256 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b800b2af0 msgr2=0x7f5b800b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:37.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.249+0000 7f5ba3000700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b800b2af0 0x7f5b800b5000 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c105650 tx=0x7f5b88009260 comp rx=0 tx=0).stop 2024-05-27T09:40:37.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.249+0000 7f5ba3000700 1 -- 172.21.15.44:0/1738262256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b9c106560 msgr2=0x7f5b9c19d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:37.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.249+0000 7f5ba3000700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b9c106560 0x7f5b9c19d0b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c000c00 tx=0x7f5b8c009510 comp rx=0 tx=0).stop 2024-05-27T09:40:37.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.250+0000 7f5ba3000700 1 -- 172.21.15.44:0/1738262256 shutdown_connections 2024-05-27T09:40:37.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.250+0000 7f5ba3000700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b800b2af0 0x7f5b800b5000 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.250+0000 7f5ba3000700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c10c5a0 0x7f5b9c1a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.250+0000 7f5ba3000700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b9c106560 0x7f5b9c19d0b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.250+0000 7f5ba3000700 1 --2- 172.21.15.44:0/1738262256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b9c106eb0 0x7f5b9c19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:37.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.250+0000 7f5ba3000700 1 -- 172.21.15.44:0/1738262256 >> 172.21.15.44:0/1738262256 conn(0x7f5b9c102040 msgr2=0x7f5b9c1038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:37.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.250+0000 7f5ba3000700 1 -- 172.21.15.44:0/1738262256 shutdown_connections 2024-05-27T09:40:37.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:37.250+0000 7f5ba3000700 1 -- 172.21.15.44:0/1738262256 wait complete. 2024-05-27T09:40:37.252 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:40:37.288 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:37.288 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:37.292 INFO:tasks.workunit.client.0.smithi044.stderr:+ c1=2 2024-05-27T09:40:37.292 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo 3 2024-05-27T09:40:37.293 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-05-27T09:40:38.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:37 smithi189 ceph-mon[36671]: pgmap v164: 97 pgs: 97 active+clean; 4.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-05-27T09:40:38.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:37 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:40:38.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:37 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:40:38.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:37 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:40:38.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:37 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:40:38.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:37 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1738262256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:40:38.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:37 smithi044 ceph-mon[31618]: pgmap v164: 97 pgs: 97 active+clean; 4.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-05-27T09:40:38.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:37 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:40:38.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:37 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:40:38.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:37 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:40:38.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:37 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:40:38.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:37 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1738262256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:40:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:37 smithi050 ceph-mon[36371]: pgmap v164: 97 pgs: 97 active+clean; 4.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-05-27T09:40:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:37 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:40:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:37 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:40:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:37 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:40:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:37 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:40:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:37 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1738262256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:40:38.887 INFO:tasks.workunit.client.0.smithi044.stderr:++ getfacl d1 2024-05-27T09:40:38.887 INFO:tasks.workunit.client.0.smithi044.stderr:++ grep -c nobody:rw 2024-05-27T09:40:39.119 INFO:tasks.workunit.client.0.smithi044.stderr:+ c2=2 2024-05-27T09:40:39.119 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:39.131 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:39.131 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 2 -ne 2 ']' 2024-05-27T09:40:39.131 INFO:tasks.workunit.client.0.smithi044.stderr:+ '[' 100 -lt 100 ']' 2024-05-27T09:40:39.136 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir d1 2024-05-27T09:40:39.174 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n system.posix_acl_access -v 0x02000000 d1 2024-05-27T09:40:39.201 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n system.posix_acl_default -v 0x02000000 . 2024-05-27T09:40:39.204 INFO:tasks.workunit.client.0.smithi044.stderr:+ expect_failure getfattr -n system.posix_acl_access d1 2024-05-27T09:40:39.204 INFO:tasks.workunit.client.0.smithi044.stderr:+ getfattr -n system.posix_acl_access d1 2024-05-27T09:40:39.208 INFO:tasks.workunit.client.0.smithi044.stderr:d1: system.posix_acl_access: No such attribute 2024-05-27T09:40:39.209 INFO:tasks.workunit.client.0.smithi044.stderr:+ return 0 2024-05-27T09:40:39.209 INFO:tasks.workunit.client.0.smithi044.stderr:+ expect_failure getfattr -n system.posix_acl_default . 2024-05-27T09:40:39.209 INFO:tasks.workunit.client.0.smithi044.stderr:+ getfattr -n system.posix_acl_default . 2024-05-27T09:40:39.212 INFO:tasks.workunit.client.0.smithi044.stderr:.: system.posix_acl_default: No such attribute 2024-05-27T09:40:39.212 INFO:tasks.workunit.client.0.smithi044.stderr:+ return 0 2024-05-27T09:40:39.212 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir d1 2024-05-27T09:40:39.217 INFO:tasks.workunit.client.0.smithi044.stderr:+ cd .. 2024-05-27T09:40:39.223 INFO:tasks.workunit.client.0.smithi044.stderr:+ rmdir testdir 2024-05-27T09:40:39.233 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo OK 2024-05-27T09:40:39.234 INFO:tasks.workunit.client.0.smithi044.stdout:OK 2024-05-27T09:40:39.234 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T09:40:39.234 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T09:40:39.862 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:39 smithi044 ceph-mon[31618]: pgmap v165: 97 pgs: 97 active+clean; 4.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-05-27T09:40:40.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:39 smithi189 ceph-mon[36671]: pgmap v165: 97 pgs: 97 active+clean; 4.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-05-27T09:40:40.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:39 smithi050 ceph-mon[36371]: pgmap v165: 97 pgs: 97 active+clean; 4.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-05-27T09:40:40.802 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-05-27T09:40:40.803 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-05-27T09:40:40.845 INFO:tasks.workunit:Running workunit fs/misc/chmod.sh... 2024-05-27T09:40:40.845 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/chmod.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6507388dd5057528934822c0163b0c347ef1d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh 2024-05-27T09:40:40.892 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:40:40.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:40 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:40:41.023 INFO:tasks.workunit.client.0.smithi044.stderr:+ set -e 2024-05-27T09:40:41.023 INFO:tasks.workunit.client.0.smithi044.stderr:+ file=test_chmod.100731 2024-05-27T09:40:41.024 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo foo 2024-05-27T09:40:41.032 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2024-05-27T09:40:41.032 INFO:tasks.workunit.client.0.smithi044.stderr:+ chmod 400 test_chmod.100731 2024-05-27T09:40:41.040 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2024-05-27T09:40:41.040 INFO:tasks.workunit.client.0.smithi044.stderr:+ check_perms test_chmod.100731 -r-------- 2024-05-27T09:40:41.040 INFO:tasks.workunit.client.0.smithi044.stderr:+ file=test_chmod.100731 2024-05-27T09:40:41.041 INFO:tasks.workunit.client.0.smithi044.stderr:++ ls -la test_chmod.100731 2024-05-27T09:40:41.061 INFO:tasks.workunit.client.0.smithi044.stderr:+ r='-r--------. 1 ubuntu ubuntu 4 May 27 09:40 test_chmod.100731' 2024-05-27T09:40:41.061 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2024-05-27T09:40:41.061 INFO:tasks.workunit.client.0.smithi044.stderr:+ perms=-r-------- 2024-05-27T09:40:41.062 INFO:tasks.workunit.client.0.smithi044.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 May 27 09:40 test_chmod.100731 2024-05-27T09:40:41.063 INFO:tasks.workunit.client.0.smithi044.stderr:++ awk '{print $1}' 2024-05-27T09:40:41.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:40 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:40:41.151 INFO:tasks.workunit.client.0.smithi044.stderr:+ test -r-------- '!=' -r--------. 2024-05-27T09:40:41.153 INFO:tasks.workunit.client.0.smithi044.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 May 27 09:40 test_chmod.100731 2024-05-27T09:40:41.154 INFO:tasks.workunit.client.0.smithi044.stderr:++ awk '{print $1}' 2024-05-27T09:40:41.161 INFO:tasks.workunit.client.0.smithi044.stderr:+ test -r--------. '!=' -r--------. 2024-05-27T09:40:41.161 INFO:tasks.workunit.client.0.smithi044.stderr:+ set +e 2024-05-27T09:40:41.161 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo bar 2024-05-27T09:40:41.161 INFO:tasks.workunit.client.0.smithi044.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh: line 40: test_chmod.100731: Permission denied 2024-05-27T09:40:41.162 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 1 = 0 2024-05-27T09:40:41.162 INFO:tasks.workunit.client.0.smithi044.stderr:+ set -e 2024-05-27T09:40:41.162 INFO:tasks.workunit.client.0.smithi044.stderr:+ chmod 600 test_chmod.100731 2024-05-27T09:40:41.165 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo bar 2024-05-27T09:40:41.165 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2024-05-27T09:40:41.165 INFO:tasks.workunit.client.0.smithi044.stderr:+ check_perms test_chmod.100731 -rw------- 2024-05-27T09:40:41.165 INFO:tasks.workunit.client.0.smithi044.stderr:+ file=test_chmod.100731 2024-05-27T09:40:41.166 INFO:tasks.workunit.client.0.smithi044.stderr:++ ls -la test_chmod.100731 2024-05-27T09:40:41.171 INFO:tasks.workunit.client.0.smithi044.stderr:+ r='-rw-------. 1 ubuntu ubuntu 8 May 27 09:40 test_chmod.100731' 2024-05-27T09:40:41.171 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2024-05-27T09:40:41.171 INFO:tasks.workunit.client.0.smithi044.stderr:+ perms=-rw------- 2024-05-27T09:40:41.173 INFO:tasks.workunit.client.0.smithi044.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 May 27 09:40 test_chmod.100731 2024-05-27T09:40:41.174 INFO:tasks.workunit.client.0.smithi044.stderr:++ awk '{print $1}' 2024-05-27T09:40:41.180 INFO:tasks.workunit.client.0.smithi044.stderr:+ test -rw------- '!=' -rw-------. 2024-05-27T09:40:41.183 INFO:tasks.workunit.client.0.smithi044.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 May 27 09:40 test_chmod.100731 2024-05-27T09:40:41.183 INFO:tasks.workunit.client.0.smithi044.stderr:++ awk '{print $1}' 2024-05-27T09:40:41.189 INFO:tasks.workunit.client.0.smithi044.stderr:+ test -rw-------. '!=' -rw-------. 2024-05-27T09:40:41.189 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo foo 2024-05-27T09:40:41.189 INFO:tasks.workunit.client.0.smithi044.stderr:+ test 0 '!=' 0 2024-05-27T09:40:41.191 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T09:40:41.191 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T09:40:41.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:40 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:40:41.350 INFO:tasks.workunit:Running workunit fs/misc/dac_override.sh... 2024-05-27T09:40:41.350 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/dac_override.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6507388dd5057528934822c0163b0c347ef1d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dac_override.sh 2024-05-27T09:40:41.434 INFO:tasks.workunit.client.0.smithi044.stderr:+ set -e 2024-05-27T09:40:41.434 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir -p testdir 2024-05-27T09:40:41.439 INFO:tasks.workunit.client.0.smithi044.stderr:+ file=test_chmod.100786 2024-05-27T09:40:41.439 INFO:tasks.workunit.client.0.smithi044.stderr:+ echo foo 2024-05-27T09:40:41.440 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo chmod 600 testdir 2024-05-27T09:40:41.477 INFO:tasks.workunit.client.0.smithi044.stderr:+ expect_failure cat testdir/test_chmod.100786 2024-05-27T09:40:41.477 INFO:tasks.workunit.client.0.smithi044.stderr:+ cat testdir/test_chmod.100786 2024-05-27T09:40:41.480 INFO:tasks.workunit.client.0.smithi044.stderr:cat: testdir/test_chmod.100786: Permission denied 2024-05-27T09:40:41.480 INFO:tasks.workunit.client.0.smithi044.stderr:+ return 0 2024-05-27T09:40:41.480 INFO:tasks.workunit.client.0.smithi044.stderr:+ sudo cat testdir/test_chmod.100786 2024-05-27T09:40:41.514 INFO:tasks.workunit.client.0.smithi044.stdout:foo 2024-05-27T09:40:41.520 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T09:40:41.520 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T09:40:41.637 INFO:tasks.workunit:Running workunit fs/misc/direct_io.py... 2024-05-27T09:40:41.638 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/direct_io.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6507388dd5057528934822c0163b0c347ef1d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py 2024-05-27T09:40:41.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:41 smithi044 ceph-mon[31618]: pgmap v166: 97 pgs: 97 active+clean; 4.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 18 op/s 2024-05-27T09:40:42.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:41 smithi189 ceph-mon[36671]: pgmap v166: 97 pgs: 97 active+clean; 4.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 18 op/s 2024-05-27T09:40:42.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:41 smithi050 ceph-mon[36371]: pgmap v166: 97 pgs: 97 active+clean; 4.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 18 op/s 2024-05-27T09:40:42.409 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:40:42.415 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:40:43.913 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:43 smithi044 ceph-mon[31618]: pgmap v167: 97 pgs: 97 active+clean; 4.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-05-27T09:40:44.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:43 smithi189 ceph-mon[36671]: pgmap v167: 97 pgs: 97 active+clean; 4.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-05-27T09:40:44.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:43 smithi050 ceph-mon[36371]: pgmap v167: 97 pgs: 97 active+clean; 4.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-05-27T09:40:45.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:45 smithi044 ceph-mon[31618]: pgmap v168: 97 pgs: 97 active+clean; 4.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-05-27T09:40:46.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:45 smithi189 ceph-mon[36671]: pgmap v168: 97 pgs: 97 active+clean; 4.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-05-27T09:40:46.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:45 smithi050 ceph-mon[36371]: pgmap v168: 97 pgs: 97 active+clean; 4.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 31 KiB/s wr, 14 op/s 2024-05-27T09:40:48.034 INFO:tasks.workunit.client.0.smithi044.stdout:ok 2024-05-27T09:40:48.050 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T09:40:48.050 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T09:40:48.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:47 smithi044 ceph-mon[31618]: pgmap v169: 97 pgs: 97 active+clean; 4.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 19 op/s 2024-05-27T09:40:48.150 INFO:tasks.workunit:Running workunit fs/misc/dirfrag.sh... 2024-05-27T09:40:48.150 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/dirfrag.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6507388dd5057528934822c0163b0c347ef1d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh 2024-05-27T09:40:48.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:47 smithi050 ceph-mon[36371]: pgmap v169: 97 pgs: 97 active+clean; 4.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 19 op/s 2024-05-27T09:40:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:47 smithi189 ceph-mon[36671]: pgmap v169: 97 pgs: 97 active+clean; 4.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 19 op/s 2024-05-27T09:40:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.697+0000 7f7335a00700 1 -- 172.21.15.44:0/1561110919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7330108f10 msgr2=0x7f7330109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.697+0000 7f7335a00700 1 --2- 172.21.15.44:0/1561110919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7330108f10 0x7f7330109320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f731c0099c0 tx=0x7f731c009cd0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.698+0000 7f7335a00700 1 -- 172.21.15.44:0/1561110919 shutdown_connections 2024-05-27T09:40:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.698+0000 7f7335a00700 1 --2- 172.21.15.44:0/1561110919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7330109860 0x7f73301111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.698+0000 7f7335a00700 1 --2- 172.21.15.44:0/1561110919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73301045e0 0x7f73301049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.698+0000 7f7335a00700 1 --2- 172.21.15.44:0/1561110919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7330108f10 0x7f7330109320 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.698+0000 7f7335a00700 1 -- 172.21.15.44:0/1561110919 >> 172.21.15.44:0/1561110919 conn(0x7f7330100030 msgr2=0x7f7330102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.698+0000 7f7335a00700 1 -- 172.21.15.44:0/1561110919 shutdown_connections 2024-05-27T09:40:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.698+0000 7f7335a00700 1 -- 172.21.15.44:0/1561110919 wait complete. 2024-05-27T09:40:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.700+0000 7f7335a00700 1 Processor -- start 2024-05-27T09:40:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.700+0000 7f7335a00700 1 -- start start 2024-05-27T09:40:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.701+0000 7f7335a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73301045e0 0x7f733019aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.701+0000 7f7335a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7330108f10 0x7f733019b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.701+0000 7f7335a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7330109860 0x7f733019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.701+0000 7f7335a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7330114470 con 0x7f73301045e0 2024-05-27T09:40:48.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.701+0000 7f7335a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f73301142f0 con 0x7f7330108f10 2024-05-27T09:40:48.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.701+0000 7f7335a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f73301145f0 con 0x7f7330109860 2024-05-27T09:40:48.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.701+0000 7f732ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73301045e0 0x7f733019aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.701+0000 7f732ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73301045e0 0x7f733019aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48362/0 (socket says 172.21.15.44:48362) 2024-05-27T09:40:48.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.701+0000 7f732ea00700 1 -- 172.21.15.44:0/3053654704 learned_addr learned my addr 172.21.15.44:0/3053654704 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:48.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.701+0000 7f732e000700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7330108f10 0x7f733019b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.702+0000 7f732f400700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7330109860 0x7f733019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.702+0000 7f732e000700 1 -- 172.21.15.44:0/3053654704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7330109860 msgr2=0x7f733019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.702+0000 7f732e000700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7330109860 0x7f733019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.702+0000 7f732e000700 1 -- 172.21.15.44:0/3053654704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73301045e0 msgr2=0x7f733019aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.702+0000 7f732e000700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73301045e0 0x7f733019aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.702+0000 7f732e000700 1 -- 172.21.15.44:0/3053654704 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733019fd90 con 0x7f7330108f10 2024-05-27T09:40:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.702+0000 7f732e000700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7330108f10 0x7f733019b460 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f731c0098e0 tx=0x7f731c0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.703+0000 7f7327400700 1 -- 172.21.15.44:0/3053654704 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c026070 con 0x7f7330108f10 2024-05-27T09:40:48.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.703+0000 7f7327400700 1 -- 172.21.15.44:0/3053654704 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f731c00f460 con 0x7f7330108f10 2024-05-27T09:40:48.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.703+0000 7f7327400700 1 -- 172.21.15.44:0/3053654704 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c009e60 con 0x7f7330108f10 2024-05-27T09:40:48.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.703+0000 7f7335a00700 1 -- 172.21.15.44:0/3053654704 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73301a0020 con 0x7f7330108f10 2024-05-27T09:40:48.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.703+0000 7f7335a00700 1 -- 172.21.15.44:0/3053654704 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73301a04b0 con 0x7f7330108f10 2024-05-27T09:40:48.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.707+0000 7f1d4fa00700 1 -- 172.21.15.44:0/3098645491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d48076c00 msgr2=0x7f1d48077010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.707+0000 7f1d4fa00700 1 --2- 172.21.15.44:0/3098645491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d48076c00 0x7f1d48077010 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0098f0 tx=0x7f1d3c009c00 comp rx=0 tx=0).stop 2024-05-27T09:40:48.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.708+0000 7f1d4fa00700 1 -- 172.21.15.44:0/3098645491 shutdown_connections 2024-05-27T09:40:48.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.708+0000 7f1d4fa00700 1 --2- 172.21.15.44:0/3098645491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d48110bd0 0x7f1d48112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.708+0000 7f1d4fa00700 1 --2- 172.21.15.44:0/3098645491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d48076c00 0x7f1d48077010 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.708+0000 7f1d4fa00700 1 --2- 172.21.15.44:0/3098645491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d48077550 0x7f1d48110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.708+0000 7f1d4fa00700 1 -- 172.21.15.44:0/3098645491 >> 172.21.15.44:0/3098645491 conn(0x7f1d480ffe90 msgr2=0x7f1d481022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:48.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.709+0000 7f1d4fa00700 1 -- 172.21.15.44:0/3098645491 shutdown_connections 2024-05-27T09:40:48.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.709+0000 7f1d4fa00700 1 -- 172.21.15.44:0/3098645491 wait complete. 2024-05-27T09:40:48.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.710+0000 7f7327400700 1 -- 172.21.15.44:0/3053654704 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f731c00f8a0 con 0x7f7330108f10 2024-05-27T09:40:48.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.710+0000 7f1d4fa00700 1 Processor -- start 2024-05-27T09:40:48.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.711+0000 7f1d4fa00700 1 -- start start 2024-05-27T09:40:48.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.711+0000 7f1d4fa00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d48076c00 0x7f1d4819d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.711+0000 7f1d4fa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d48077550 0x7f1d4819d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.711+0000 7f1d4fa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d48110bd0 0x7f1d481a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.711+0000 7f1d4fa00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1d48116700 con 0x7f1d48110bd0 2024-05-27T09:40:48.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.711+0000 7f1d4fa00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1d48116580 con 0x7f1d48076c00 2024-05-27T09:40:48.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.711+0000 7f1d4fa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1d48116880 con 0x7f1d48077550 2024-05-27T09:40:48.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.712+0000 7f1d4d000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d48076c00 0x7f1d4819d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.712+0000 7f1d4d000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d48076c00 0x7f1d4819d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37758/0 (socket says 172.21.15.44:37758) 2024-05-27T09:40:48.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.712+0000 7f1d4d000700 1 -- 172.21.15.44:0/1133824269 learned_addr learned my addr 172.21.15.44:0/1133824269 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:48.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.712+0000 7f1d47e00700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d48077550 0x7f1d4819d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.712+0000 7f1d4d000700 1 -- 172.21.15.44:0/1133824269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d48077550 msgr2=0x7f1d4819d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.712+0000 7f1d4d000700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d48077550 0x7f1d4819d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.712+0000 7f1d4d000700 1 -- 172.21.15.44:0/1133824269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d48110bd0 msgr2=0x7f1d481a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.712+0000 7f1d4d000700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d48110bd0 0x7f1d481a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.712+0000 7f1d4d000700 1 -- 172.21.15.44:0/1133824269 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d481a1fa0 con 0x7f1d48076c00 2024-05-27T09:40:48.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.713+0000 7f1d4d000700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d48076c00 0x7f1d4819d0d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009230 tx=0x7f1d3c00fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.713+0000 7f1d45600700 1 -- 172.21.15.44:0/1133824269 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c025070 con 0x7f1d48076c00 2024-05-27T09:40:48.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.713+0000 7f1d45600700 1 -- 172.21.15.44:0/1133824269 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d3c00ba10 con 0x7f1d48076c00 2024-05-27T09:40:48.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.713+0000 7f1d45600700 1 -- 172.21.15.44:0/1133824269 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c003af0 con 0x7f1d48076c00 2024-05-27T09:40:48.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.713+0000 7f1d4fa00700 1 -- 172.21.15.44:0/1133824269 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d481a2230 con 0x7f1d48076c00 2024-05-27T09:40:48.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.713+0000 7f7327400700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73140b2a70 0x7f73140b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.713+0000 7f7327400700 1 -- 172.21.15.44:0/3053654704 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f731c0e2f60 con 0x7f7330108f10 2024-05-27T09:40:48.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.713+0000 7f732ea00700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73140b2a70 0x7f73140b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.714+0000 7f1d4fa00700 1 -- 172.21.15.44:0/1133824269 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d481a26c0 con 0x7f1d48076c00 2024-05-27T09:40:48.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.715+0000 7f732ea00700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73140b2a70 0x7f73140b4f80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7320007c90 tx=0x7f7320007480 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.717+0000 7f1d45600700 1 -- 172.21.15.44:0/1133824269 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f1d3c028070 con 0x7f1d48076c00 2024-05-27T09:40:48.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.719+0000 7f1d45600700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1d300b2a70 0x7f1d300b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.719+0000 7f1d45600700 1 -- 172.21.15.44:0/1133824269 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1d3c0e2f40 con 0x7f1d48076c00 2024-05-27T09:40:48.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.720+0000 7f1d47e00700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1d300b2a70 0x7f1d300b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.720+0000 7f1d47e00700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1d300b2a70 0x7f1d300b4f80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1d48103390 tx=0x7f1d34009310 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f1d5e582b80 1 Processor -- start 2024-05-27T09:40:48.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f73444a6b80 1 Processor -- start 2024-05-27T09:40:48.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f1d5e582b80 1 -- start start 2024-05-27T09:40:48.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f73444a6b80 1 -- start start 2024-05-27T09:40:48.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e784046fd0 0x55e7841cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262319f0 0x5608262c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 0x5608262c4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262d5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f73444a6b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5608261f2e40 con 0x5608262319f0 2024-05-27T09:40:48.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f73444a6b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5608261e29e0 con 0x560826145e40 2024-05-27T09:40:48.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f73444a6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5608261e6a20 con 0x5608262d3510 2024-05-27T09:40:48.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841c64d0 0x55e7841c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e7840f3fd0 con 0x55e784046fd0 2024-05-27T09:40:48.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e7840e3b70 con 0x55e7841c64d0 2024-05-27T09:40:48.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.731+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e7840e7bb0 con 0x55e7841cc380 2024-05-27T09:40:48.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f1d47e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f1d47e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841c5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43188/0 (socket says 172.21.15.44:43188) 2024-05-27T09:40:48.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f732f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262d5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f1d47e00700 1 -- 172.21.15.44:0/745272821 learned_addr learned my addr 172.21.15.44:0/745272821 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:48.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f1d4d000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e784046fd0 0x55e7841cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f732e000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 0x5608262c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f732f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262d5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43202/0 (socket says 172.21.15.44:43202) 2024-05-27T09:40:48.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f732f400700 1 -- 172.21.15.44:0/3270845140 learned_addr learned my addr 172.21.15.44:0/3270845140 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:48.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.732+0000 7f732ea00700 1 --2- 172.21.15.44:0/3270845140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262319f0 0x5608262c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f1d4da00700 1 --2- 172.21.15.44:0/745272821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841c64d0 0x55e7841c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f1d47e00700 1 -- 172.21.15.44:0/745272821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841c64d0 msgr2=0x55e7841c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f732e000700 1 -- 172.21.15.44:0/3270845140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 msgr2=0x5608262d5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f732e000700 1 --2- 172.21.15.44:0/3270845140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262d5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f732e000700 1 -- 172.21.15.44:0/3270845140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262319f0 msgr2=0x5608262c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f732e000700 1 --2- 172.21.15.44:0/3270845140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262319f0 0x5608262c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f1d47e00700 1 --2- 172.21.15.44:0/745272821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841c64d0 0x55e7841c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f1d47e00700 1 -- 172.21.15.44:0/745272821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e784046fd0 msgr2=0x55e7841cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f1d47e00700 1 --2- 172.21.15.44:0/745272821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e784046fd0 0x55e7841cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f1d47e00700 1 -- 172.21.15.44:0/745272821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7840ac5b0 con 0x55e7841cc380 2024-05-27T09:40:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f1d4da00700 1 --2- 172.21.15.44:0/745272821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841c64d0 0x55e7841c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:40:48.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f732e000700 1 -- 172.21.15.44:0/3270845140 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608261ab420 con 0x560826145e40 2024-05-27T09:40:48.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f732e000700 1 --2- 172.21.15.44:0/3270845140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 0x5608262c4930 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f731c0e0680 tx=0x7f731c0096e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=203b5f5c32f648f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.733+0000 7f1d4d000700 1 --2- 172.21.15.44:0/745272821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e784046fd0 0x55e7841cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:48.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.734+0000 7f1d47e00700 1 --2- 172.21.15.44:0/745272821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841c5f90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1d34004140 tx=0x7f1d34014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=859b465a13e4b89b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.734+0000 7f731be00700 1 -- 172.21.15.44:0/3270845140 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c0aa9a0 con 0x560826145e40 2024-05-27T09:40:48.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.734+0000 7f731be00700 1 -- 172.21.15.44:0/3270845140 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f731c0aa2a0 con 0x560826145e40 2024-05-27T09:40:48.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.734+0000 7f731be00700 1 -- 172.21.15.44:0/3270845140 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c0aa760 con 0x560826145e40 2024-05-27T09:40:48.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.734+0000 7f1d3a000700 1 -- 172.21.15.44:0/745272821 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d34003b10 con 0x55e7841cc380 2024-05-27T09:40:48.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.734+0000 7f1d3a000700 1 -- 172.21.15.44:0/745272821 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d34004ce0 con 0x55e7841cc380 2024-05-27T09:40:48.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.734+0000 7f1d3a000700 1 -- 172.21.15.44:0/745272821 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d34005640 con 0x55e7841cc380 2024-05-27T09:40:48.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f1d5e582b80 1 -- 172.21.15.44:0/745272821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 msgr2=0x55e7841c5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/745272821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841c5f90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1d34004140 tx=0x7f1d34014000 comp rx=0 tx=0).stop 2024-05-27T09:40:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f73444a6b80 1 -- 172.21.15.44:0/3270845140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 msgr2=0x5608262c4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f73444a6b80 1 --2- 172.21.15.44:0/3270845140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 0x5608262c4930 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f731c0e0680 tx=0x7f731c0096e0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f1d5e582b80 1 -- 172.21.15.44:0/745272821 shutdown_connections 2024-05-27T09:40:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/745272821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841c64d0 0x55e7841c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/745272821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e784046fd0 0x55e7841cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/745272821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841c5f90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f1d5e582b80 1 -- 172.21.15.44:0/745272821 >> 172.21.15.44:0/745272821 conn(0x55e7840a7150 msgr2=0x55e7841c1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f73444a6b80 1 -- 172.21.15.44:0/3270845140 shutdown_connections 2024-05-27T09:40:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f73444a6b80 1 --2- 172.21.15.44:0/3270845140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262d5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f73444a6b80 1 --2- 172.21.15.44:0/3270845140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262319f0 0x5608262c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f73444a6b80 1 --2- 172.21.15.44:0/3270845140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 0x5608262c4930 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.735+0000 7f73444a6b80 1 -- 172.21.15.44:0/3270845140 >> 172.21.15.44:0/3270845140 conn(0x5608261df390 msgr2=0x5608262c04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.736+0000 7f73444a6b80 1 -- 172.21.15.44:0/3270845140 shutdown_connections 2024-05-27T09:40:48.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.736+0000 7f1d5e582b80 1 -- 172.21.15.44:0/745272821 shutdown_connections 2024-05-27T09:40:48.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.736+0000 7f1d5e582b80 1 -- 172.21.15.44:0/745272821 wait complete. 2024-05-27T09:40:48.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.736+0000 7f73444a6b80 1 -- 172.21.15.44:0/3270845140 wait complete. 2024-05-27T09:40:48.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.737+0000 7f73444a6b80 1 -- start start 2024-05-27T09:40:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.736+0000 7f1d5e582b80 1 -- start start 2024-05-27T09:40:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.737+0000 7f1d5e582b80 1 Processor -- start 2024-05-27T09:40:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.737+0000 7f73444a6b80 1 Processor -- start 2024-05-27T09:40:48.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 0x55e7841d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841d0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e783fea130 con 0x55e7841c64d0 2024-05-27T09:40:48.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e7840848f0 con 0x55e784046fd0 2024-05-27T09:40:48.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e7841d2e50 con 0x55e7841cc380 2024-05-27T09:40:48.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f1d5e582b80 10 client.0 _ll_register_callbacks cb 0x55e78408de00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:48.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f1d4d000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 0x55e7841d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f1d4d000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 0x55e7841d2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37782/0 (socket says 172.21.15.44:37782) 2024-05-27T09:40:48.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f1d4d000700 1 -- 172.21.15.44:0/3748795787 learned_addr learned my addr 172.21.15.44:0/3748795787 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:48.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f1d4da00700 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f1d4d000700 1 -- 172.21.15.44:0/3748795787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 msgr2=0x55e7841d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f1d4d000700 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841d0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f1d4d000700 1 -- 172.21.15.44:0/3748795787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 msgr2=0x55e7841cf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f1d4d000700 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f1d4d000700 1 -- 172.21.15.44:0/3748795787 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e784033a60 con 0x55e784046fd0 2024-05-27T09:40:48.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560826145e40 0x5608262d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608262319f0 0x5608262cc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f73444a6b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5608260e8fa0 con 0x560826145e40 2024-05-27T09:40:48.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f73444a6b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560826183760 con 0x5608262319f0 2024-05-27T09:40:48.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f73444a6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5608261e29e0 con 0x5608262d3510 2024-05-27T09:40:48.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.738+0000 7f73444a6b80 10 client.0 _ll_register_callbacks cb 0x56082618cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:48.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f732ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560826145e40 0x5608262d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f732f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f732ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560826145e40 0x5608262d2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48426/0 (socket says 172.21.15.44:48426) 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f732ea00700 1 -- 172.21.15.44:0/2895984610 learned_addr learned my addr 172.21.15.44:0/2895984610 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f732f400700 1 -- 172.21.15.44:0/2895984610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608262319f0 msgr2=0x5608262cc150 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f732f400700 1 --2- 172.21.15.44:0/2895984610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608262319f0 0x5608262cc150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f732f400700 1 -- 172.21.15.44:0/2895984610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560826145e40 msgr2=0x5608262d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f732f400700 1 --2- 172.21.15.44:0/2895984610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560826145e40 0x5608262d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f732f400700 1 -- 172.21.15.44:0/2895984610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608261328d0 con 0x5608262d3510 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.740+0000 7f732f400700 1 --2- 172.21.15.44:0/2895984610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262cc6b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7328004710 tx=0x7f7328014600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13ee2998a224fcd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.740+0000 7f7318c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.740+0000 7f7318c00700 1 -- 172.21.15.44:0/2895984610 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73280092d0 con 0x5608262d3510 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.740+0000 7f7318c00700 1 -- 172.21.15.44:0/2895984610 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f7328014b90 con 0x5608262d3510 2024-05-27T09:40:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.740+0000 7f73444a6b80 1 -- 172.21.15.44:0/2895984610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f731c0aa2a0 con 0x5608262d3510 2024-05-27T09:40:48.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f1d4d000700 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 0x55e7841d2630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0e06c0 tx=0x7f1d3c003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd6aae675819367f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f1d2ea00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:40:48.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.739+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3748795787 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0a9d90 con 0x55e784046fd0 2024-05-27T09:40:48.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.740+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3748795787 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1d3c0aa6f0 con 0x55e784046fd0 2024-05-27T09:40:48.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.740+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3748795787 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d34004ce0 con 0x55e784046fd0 2024-05-27T09:40:48.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.740+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3748795787 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1d3c0aa980 con 0x55e784046fd0 2024-05-27T09:40:48.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.740+0000 7f1d5e582b80 10 client.26114 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:48.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.740+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3748795787 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e7840b41f0 con 0x55e784046fd0 2024-05-27T09:40:48.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.740+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3748795787 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d34004ce0 con 0x55e784046fd0 2024-05-27T09:40:48.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.741+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3748795787 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1d3c0ae170 con 0x55e784046fd0 2024-05-27T09:40:48.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.741+0000 7f7318c00700 1 -- 172.21.15.44:0/2895984610 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7328014d90 con 0x5608262d3510 2024-05-27T09:40:48.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.741+0000 7f73444a6b80 10 client.26125 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:48.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.741+0000 7f73444a6b80 1 -- 172.21.15.44:0/2895984610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5608261b3060 con 0x5608262d3510 2024-05-27T09:40:48.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.741+0000 7f7318c00700 1 -- 172.21.15.44:0/2895984610 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73280079b0 con 0x5608262d3510 2024-05-27T09:40:48.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.742+0000 7f1d5e582b80 10 client.26114 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:48.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.742+0000 7f1d5e582b80 10 client.26114 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:40:48.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.742+0000 7f1d5e582b80 20 client.26114 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:48.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.742+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e7841e8200 0x55e7841ea680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.742+0000 7f1d5e582b80 4 client.26114 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:40:48.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.742+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3748795787 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7840f5f20 con 0x55e7841e8200 2024-05-27T09:40:48.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.742+0000 7f1d47e00700 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e7841e8200 0x55e7841ea680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.742+0000 7f7318c00700 1 -- 172.21.15.44:0/2895984610 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7328028590 con 0x5608262d3510 2024-05-27T09:40:48.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.743+0000 7f73444a6b80 10 client.26125 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:48.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.743+0000 7f1d47e00700 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e7841e8200 0x55e7841ea680 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5b0246b6790f541 server_cookie=a189b775aab2db5a in_seq=0 out_seq=0 2024-05-27T09:40:48.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.743+0000 7f1d2ea00700 10 client.26114 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:48.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.743+0000 7f73444a6b80 10 client.26125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:40:48.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.743+0000 7f73444a6b80 20 client.26125 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:48.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.743+0000 7f73444a6b80 1 --2- 172.21.15.44:0/2895984610 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5608262e7160 0x5608262e9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.743+0000 7f73444a6b80 4 client.26125 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:40:48.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.743+0000 7f73444a6b80 1 -- 172.21.15.44:0/2895984610 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f731c0aa2a0 con 0x5608262e7160 2024-05-27T09:40:48.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.743+0000 7f732ea00700 1 --2- 172.21.15.44:0/2895984610 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5608262e7160 0x5608262e9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3748795787 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e7840f5f20 con 0x55e7841e8200 2024-05-27T09:40:48.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d2ea00700 10 client.26114 handle_command_reply: tid=0 2024-05-27T09:40:48.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d5e582b80 1 client.26114 shutdown 2024-05-27T09:40:48.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f732ea00700 1 --2- 172.21.15.44:0/2895984610 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5608262e7160 0x5608262e9580 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b17d6af07c0ff9f server_cookie=fb48b9d6da22910a in_seq=0 out_seq=0 2024-05-27T09:40:48.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f7318c00700 10 client.26125 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:48.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3748795787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 msgr2=0x55e7841d2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 0x55e7841d2630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0e06c0 tx=0x7f1d3c003f90 comp rx=0 tx=0).stop 2024-05-27T09:40:48.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3748795787 shutdown_connections 2024-05-27T09:40:48.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e7841e8200 0x55e7841ea680 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 0x55e7841d2630 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3748795787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841cf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3748795787 >> 172.21.15.44:0/3748795787 conn(0x55e7840a7150 msgr2=0x55e7841c4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:48.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.744+0000 7f1d2ea00700 0 client.26114 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:48.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.745+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3748795787 shutdown_connections 2024-05-27T09:40:48.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.745+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3748795787 wait complete. 2024-05-27T09:40:48.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.745+0000 7f1d5e582b80 20 client.26114 trim_cache size 0 max 16384 2024-05-27T09:40:48.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.745+0000 7f7318c00700 1 -- 172.21.15.44:0/2895984610 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f731c0aa2a0 con 0x5608262e7160 2024-05-27T09:40:48.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.745+0000 7f7318c00700 10 client.26125 handle_command_reply: tid=0 2024-05-27T09:40:48.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.745+0000 7f73444a6b80 1 client.26125 shutdown 2024-05-27T09:40:48.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f73444a6b80 1 -- 172.21.15.44:0/2895984610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 msgr2=0x5608262cc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f73444a6b80 1 --2- 172.21.15.44:0/2895984610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262cc6b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7328004710 tx=0x7f7328014600 comp rx=0 tx=0).stop 2024-05-27T09:40:48.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f73444a6b80 1 -- 172.21.15.44:0/2895984610 shutdown_connections 2024-05-27T09:40:48.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f73444a6b80 1 --2- 172.21.15.44:0/2895984610 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5608262e7160 0x5608262e9580 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f73444a6b80 1 --2- 172.21.15.44:0/2895984610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262cc6b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f73444a6b80 1 --2- 172.21.15.44:0/2895984610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560826145e40 0x5608262d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f73444a6b80 1 --2- 172.21.15.44:0/2895984610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608262319f0 0x5608262cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f73444a6b80 1 -- 172.21.15.44:0/2895984610 >> 172.21.15.44:0/2895984610 conn(0x5608261df390 msgr2=0x5608262c01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:48.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f7318c00700 0 client.26125 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:48.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f73444a6b80 1 -- 172.21.15.44:0/2895984610 shutdown_connections 2024-05-27T09:40:48.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f73444a6b80 1 -- 172.21.15.44:0/2895984610 wait complete. 2024-05-27T09:40:48.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.746+0000 7f73444a6b80 20 client.26125 trim_cache size 0 max 16384 2024-05-27T09:40:48.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.769+0000 7f1d5e582b80 1 Processor -- start 2024-05-27T09:40:48.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.769+0000 7f1d5e582b80 1 -- start start 2024-05-27T09:40:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.769+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e784046fd0 0x55e7841dbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.769+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841efe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.769+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841cc380 0x55e7841f0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.769+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1d481af560 con 0x55e7841c64d0 2024-05-27T09:40:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.769+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1d48106860 con 0x55e7841cc380 2024-05-27T09:40:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.769+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1d481046b0 con 0x55e784046fd0 2024-05-27T09:40:48.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d47e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d4d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e784046fd0 0x55e7841dbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d47e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841efe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48436/0 (socket says 172.21.15.44:48436) 2024-05-27T09:40:48.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d47e00700 1 -- 172.21.15.44:0/2031275194 learned_addr learned my addr 172.21.15.44:0/2031275194 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:48.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d4da00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841cc380 0x55e7841f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d4d000700 1 -- 172.21.15.44:0/2031275194 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841cc380 msgr2=0x55e7841f0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d4d000700 1 --2- 172.21.15.44:0/2031275194 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841cc380 0x55e7841f0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d4d000700 1 -- 172.21.15.44:0/2031275194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 msgr2=0x55e7841efe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d4d000700 1 --2- 172.21.15.44:0/2031275194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841efe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d4d000700 1 -- 172.21.15.44:0/2031275194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7841d29d0 con 0x55e784046fd0 2024-05-27T09:40:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d47e00700 1 --2- 172.21.15.44:0/2031275194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841efe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.770+0000 7f1d4da00700 1 --2- 172.21.15.44:0/2031275194 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841cc380 0x55e7841f0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.771+0000 7f1d4d000700 1 --2- 172.21.15.44:0/2031275194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e784046fd0 0x55e7841dbeb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0a9f80 tx=0x7f1d3c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18506fc83077db3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.771+0000 7f1d39600700 1 -- 172.21.15.44:0/2031275194 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0a6140 con 0x55e784046fd0 2024-05-27T09:40:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.771+0000 7f1d39600700 1 -- 172.21.15.44:0/2031275194 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d3c0c4070 con 0x55e784046fd0 2024-05-27T09:40:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.771+0000 7f1d39600700 1 -- 172.21.15.44:0/2031275194 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0b3050 con 0x55e784046fd0 2024-05-27T09:40:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.771+0000 7f73444a6b80 1 Processor -- start 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.771+0000 7f73444a6b80 1 -- start start 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.771+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 0x5608262d4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.771+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262319f0 0x5608262bfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.771+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262c03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f73444a6b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f73301af3c0 con 0x5608262319f0 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f73444a6b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f73301acc50 con 0x560826145e40 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f73444a6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f73301aaac0 con 0x5608262d3510 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f732ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 0x5608262d4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f732ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 0x5608262d4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37800/0 (socket says 172.21.15.44:37800) 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f732ea00700 1 -- 172.21.15.44:0/46536812 learned_addr learned my addr 172.21.15.44:0/46536812 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f732e000700 1 --2- 172.21.15.44:0/46536812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262319f0 0x5608262bfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f732f400700 1 --2- 172.21.15.44:0/46536812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262c03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f1d5e582b80 1 -- 172.21.15.44:0/2031275194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e784046fd0 msgr2=0x55e7841dbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/2031275194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e784046fd0 0x55e7841dbeb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0a9f80 tx=0x7f1d3c009f90 comp rx=0 tx=0).stop 2024-05-27T09:40:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f1d5e582b80 1 -- 172.21.15.44:0/2031275194 shutdown_connections 2024-05-27T09:40:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/2031275194 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7841cc380 0x55e7841f0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/2031275194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e784046fd0 0x55e7841dbeb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/2031275194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841efe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f1d5e582b80 1 -- 172.21.15.44:0/2031275194 >> 172.21.15.44:0/2031275194 conn(0x55e7840a7150 msgr2=0x55e7841c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f1d5e582b80 1 -- 172.21.15.44:0/2031275194 shutdown_connections 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f1d5e582b80 1 -- 172.21.15.44:0/2031275194 wait complete. 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f1d5e582b80 1 -- start start 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f1d5e582b80 1 Processor -- start 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 0x55e7841d87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841d5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f1d5e582b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841d55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1d48104d20 con 0x55e7841c64d0 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1d48106190 con 0x55e784046fd0 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f1d5e582b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1d481053f0 con 0x55e7841cc380 2024-05-27T09:40:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f1d5e582b80 10 client.0 _ll_register_callbacks cb 0x55e78408de00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.775+0000 7f1d4da00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841d55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.775+0000 7f1d4da00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841d55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43264/0 (socket says 172.21.15.44:43264) 2024-05-27T09:40:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.775+0000 7f1d4da00700 1 -- 172.21.15.44:0/3147596034 learned_addr learned my addr 172.21.15.44:0/3147596034 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.775+0000 7f1d4d000700 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 0x55e7841d87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.775+0000 7f1d47e00700 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841d5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.775+0000 7f1d4da00700 1 -- 172.21.15.44:0/3147596034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 msgr2=0x55e7841d87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.775+0000 7f1d4da00700 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 0x55e7841d87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f732ea00700 1 -- 172.21.15.44:0/46536812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 msgr2=0x5608262c03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f732ea00700 1 --2- 172.21.15.44:0/46536812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262c03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f732ea00700 1 -- 172.21.15.44:0/46536812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262319f0 msgr2=0x5608262bfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f732ea00700 1 --2- 172.21.15.44:0/46536812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262319f0 0x5608262bfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.772+0000 7f732ea00700 1 -- 172.21.15.44:0/46536812 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608262c19c0 con 0x560826145e40 2024-05-27T09:40:48.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f732e000700 1 --2- 172.21.15.44:0/46536812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262319f0 0x5608262bfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:40:48.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f732ea00700 1 --2- 172.21.15.44:0/46536812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 0x5608262d4930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7320010a40 tx=0x7f7320013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f85f4098899a867c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f731b400700 1 -- 172.21.15.44:0/46536812 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732002a470 con 0x560826145e40 2024-05-27T09:40:48.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.775+0000 7f1d4da00700 1 -- 172.21.15.44:0/3147596034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 msgr2=0x55e7841d5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f731b400700 1 -- 172.21.15.44:0/46536812 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f732002f070 con 0x560826145e40 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.773+0000 7f731b400700 1 -- 172.21.15.44:0/46536812 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732001f9f0 con 0x560826145e40 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f73444a6b80 1 -- 172.21.15.44:0/46536812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 msgr2=0x5608262d4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f73444a6b80 1 --2- 172.21.15.44:0/46536812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 0x5608262d4930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7320010a40 tx=0x7f7320013000 comp rx=0 tx=0).stop 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f73444a6b80 1 -- 172.21.15.44:0/46536812 shutdown_connections 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f73444a6b80 1 --2- 172.21.15.44:0/46536812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608262d3510 0x5608262c03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f73444a6b80 1 --2- 172.21.15.44:0/46536812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560826145e40 0x5608262d4930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f73444a6b80 1 --2- 172.21.15.44:0/46536812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262319f0 0x5608262bfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.774+0000 7f73444a6b80 1 -- 172.21.15.44:0/46536812 >> 172.21.15.44:0/46536812 conn(0x5608261df390 msgr2=0x5608262d52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.775+0000 7f73444a6b80 1 -- 172.21.15.44:0/46536812 shutdown_connections 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.775+0000 7f73444a6b80 1 -- 172.21.15.44:0/46536812 wait complete. 2024-05-27T09:40:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.776+0000 7f73444a6b80 1 -- start start 2024-05-27T09:40:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.776+0000 7f1d4da00700 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841d5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.776+0000 7f1d4da00700 1 -- 172.21.15.44:0/3147596034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7841d5d30 con 0x55e7841cc380 2024-05-27T09:40:48.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.776+0000 7f73444a6b80 1 Processor -- start 2024-05-27T09:40:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.776+0000 7f1d47e00700 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841d5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:40:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.776+0000 7f1d4da00700 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841d55b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1d40007f00 tx=0x7f1d40014240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d17a742f36dd09f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.776+0000 7f1d2ea00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:40:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.776+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3147596034 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d40004db0 con 0x55e7841cc380 2024-05-27T09:40:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.776+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3147596034 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f1d40014b00 con 0x55e7841cc380 2024-05-27T09:40:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3147596034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d3c0c4070 con 0x55e7841cc380 2024-05-27T09:40:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560826145e40 0x5608262e3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608262319f0 0x5608262e3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f73444a6b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262d3510 0x5608262e42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3147596034 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1d40014d00 con 0x55e7841cc380 2024-05-27T09:40:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f1d5e582b80 10 client.26134 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3147596034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e784253c70 con 0x55e7841cc380 2024-05-27T09:40:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f732ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560826145e40 0x5608262e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f732ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560826145e40 0x5608262e3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43280/0 (socket says 172.21.15.44:43280) 2024-05-27T09:40:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f732ea00700 1 -- 172.21.15.44:0/626115517 learned_addr learned my addr 172.21.15.44:0/626115517 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f73444a6b80 1 -- 172.21.15.44:0/626115517 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f73301ab110 con 0x5608262d3510 2024-05-27T09:40:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f73444a6b80 1 -- 172.21.15.44:0/626115517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f73301ac580 con 0x5608262319f0 2024-05-27T09:40:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.778+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3147596034 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d3c0c4070 con 0x55e7841cc380 2024-05-27T09:40:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f73444a6b80 1 -- 172.21.15.44:0/626115517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f73301ab7e0 con 0x560826145e40 2024-05-27T09:40:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.777+0000 7f732e000700 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608262319f0 0x5608262e3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.778+0000 7f73444a6b80 10 client.0 _ll_register_callbacks cb 0x56082618cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.778+0000 7f732f400700 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262d3510 0x5608262e42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.778+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3147596034 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1d40004740 con 0x55e7841cc380 2024-05-27T09:40:48.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.779+0000 7f732f400700 1 -- 172.21.15.44:0/626115517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560826145e40 msgr2=0x5608262e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.779+0000 7f732f400700 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560826145e40 0x5608262e3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.778+0000 7f1d5e582b80 10 client.26134 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.778+0000 7f1d5e582b80 10 client.26134 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:40:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.778+0000 7f1d5e582b80 20 client.26134 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.778+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e784254440 0x55e784256860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.778+0000 7f1d5e582b80 4 client.26134 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-05-27T09:40:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.778+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3147596034 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x55e7840ef700 con 0x55e784254440 2024-05-27T09:40:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.778+0000 7f1d4d000700 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e784254440 0x55e784256860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.779+0000 7f1d4d000700 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e784254440 0x55e784256860 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=632188ad50a3b6ef server_cookie=66ebbe9de2b163d7 in_seq=0 out_seq=0 2024-05-27T09:40:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.779+0000 7f1d2ea00700 10 client.26134 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:48.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.779+0000 7f732f400700 1 -- 172.21.15.44:0/626115517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608262319f0 msgr2=0x5608262e3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:48.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.779+0000 7f732f400700 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608262319f0 0x5608262e3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.779+0000 7f732f400700 1 -- 172.21.15.44:0/626115517 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56082634ad20 con 0x5608262d3510 2024-05-27T09:40:48.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.779+0000 7f732f400700 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262d3510 0x5608262e42f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f73280289e0 tx=0x7f7328009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3af77a69af9992 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:48.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.779+0000 7f7318c00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.780+0000 7f7318c00700 1 -- 172.21.15.44:0/626115517 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7328028630 con 0x5608262d3510 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.780+0000 7f73444a6b80 1 -- 172.21.15.44:0/626115517 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f732002f070 con 0x5608262d3510 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.780+0000 7f7318c00700 1 -- 172.21.15.44:0/626115517 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f7328004a00 con 0x5608262d3510 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.780+0000 7f7318c00700 1 -- 172.21.15.44:0/626115517 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7328038050 con 0x5608262d3510 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.780+0000 7f73444a6b80 10 client.15993 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.780+0000 7f73444a6b80 1 -- 172.21.15.44:0/626115517 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560826362f00 con 0x5608262d3510 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.781+0000 7f7318c00700 1 -- 172.21.15.44:0/626115517 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f732002f070 con 0x5608262d3510 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.781+0000 7f7318c00700 1 -- 172.21.15.44:0/626115517 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f732801eb70 con 0x5608262d3510 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.781+0000 7f73444a6b80 10 client.15993 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.781+0000 7f73444a6b80 10 client.15993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.781+0000 7f73444a6b80 20 client.15993 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.781+0000 7f73444a6b80 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5608263635b0 0x5608263659d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.781+0000 7f73444a6b80 4 client.15993 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.781+0000 7f73444a6b80 1 -- 172.21.15.44:0/626115517 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5608261f4d90 con 0x5608263635b0 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.782+0000 7f732ea00700 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5608263635b0 0x5608263659d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.782+0000 7f732ea00700 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5608263635b0 0x5608263659d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69872a8d2bda283f server_cookie=68045299c7861fcf in_seq=0 out_seq=0 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.782+0000 7f7318c00700 10 client.15993 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.783+0000 7f7318c00700 1 -- 172.21.15.44:0/626115517 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5608261f4d90 con 0x5608263635b0 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.783+0000 7f7318c00700 10 client.15993 handle_command_reply: tid=0 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.783+0000 7f73444a6b80 1 client.15993 shutdown 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.783+0000 7f73444a6b80 1 -- 172.21.15.44:0/626115517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262d3510 msgr2=0x5608262e42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.783+0000 7f73444a6b80 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262d3510 0x5608262e42f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f73280289e0 tx=0x7f7328009380 comp rx=0 tx=0).stop 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.784+0000 7f73444a6b80 1 -- 172.21.15.44:0/626115517 shutdown_connections 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.784+0000 7f73444a6b80 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5608263635b0 0x5608263659d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.784+0000 7f73444a6b80 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608262d3510 0x5608262e42f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.784+0000 7f73444a6b80 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560826145e40 0x5608262e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.784+0000 7f73444a6b80 1 --2- 172.21.15.44:0/626115517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608262319f0 0x5608262e3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.784+0000 7f7318c00700 0 client.15993 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.784+0000 7f73444a6b80 1 -- 172.21.15.44:0/626115517 >> 172.21.15.44:0/626115517 conn(0x5608261df390 msgr2=0x5608262cbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.784+0000 7f73444a6b80 1 -- 172.21.15.44:0/626115517 shutdown_connections 2024-05-27T09:40:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.784+0000 7f73444a6b80 1 -- 172.21.15.44:0/626115517 wait complete. 2024-05-27T09:40:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.784+0000 7f73444a6b80 20 client.15993 trim_cache size 0 max 16384 2024-05-27T09:40:48.792 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:40:48.792 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:40:48.792 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:40:48.793 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:40:48.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 -- 172.21.15.44:0/3053654704 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73140b2a70 msgr2=0x7f73140b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73140b2a70 0x7f73140b4f80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7320007c90 tx=0x7f7320007480 comp rx=0 tx=0).stop 2024-05-27T09:40:48.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 -- 172.21.15.44:0/3053654704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7330108f10 msgr2=0x7f733019b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:48.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7330108f10 0x7f733019b460 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f731c0098e0 tx=0x7f731c0045e0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 -- 172.21.15.44:0/3053654704 shutdown_connections 2024-05-27T09:40:48.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73140b2a70 0x7f73140b4f80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7330109860 0x7f733019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73301045e0 0x7f733019aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 --2- 172.21.15.44:0/3053654704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7330108f10 0x7f733019b460 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:48.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 -- 172.21.15.44:0/3053654704 >> 172.21.15.44:0/3053654704 conn(0x7f7330100030 msgr2=0x7f733010a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:48.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 -- 172.21.15.44:0/3053654704 shutdown_connections 2024-05-27T09:40:48.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:48.785+0000 7f7335a00700 1 -- 172.21.15.44:0/3053654704 wait complete. 2024-05-27T09:40:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.092+0000 7f2ea6a00640 1 -- 172.21.15.44:0/635629286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ea01043c0 msgr2=0x7f2ea0110790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.092+0000 7f2ea6a00640 1 --2- 172.21.15.44:0/635629286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ea01043c0 0x7f2ea0110790 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2e98009860 tx=0x7f2e9802a340 comp rx=0 tx=0).stop 2024-05-27T09:40:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.092+0000 7f2ea6a00640 1 -- 172.21.15.44:0/635629286 shutdown_connections 2024-05-27T09:40:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.092+0000 7f2ea6a00640 1 --2- 172.21.15.44:0/635629286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ea01043c0 0x7f2ea0110790 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.092+0000 7f2ea6a00640 1 --2- 172.21.15.44:0/635629286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ea010a110 0x7f2ea0103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.092+0000 7f2ea6a00640 1 --2- 172.21.15.44:0/635629286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ea0109c70 0x7f2ea0103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.092+0000 7f2ea6a00640 1 -- 172.21.15.44:0/635629286 >> 172.21.15.44:0/635629286 conn(0x7f2ea00ff350 msgr2=0x7f2ea01017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.092+0000 7f2ea6a00640 1 -- 172.21.15.44:0/635629286 shutdown_connections 2024-05-27T09:40:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.092+0000 7f2ea6a00640 1 -- 172.21.15.44:0/635629286 wait complete. 2024-05-27T09:40:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.093+0000 7f2ea6a00640 1 Processor -- start 2024-05-27T09:40:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.093+0000 7f2ea6a00640 1 -- start start 2024-05-27T09:40:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.094+0000 7f2ea6a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ea01043c0 0x7f2ea01a2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.094+0000 7f2ea6a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ea0109c70 0x7f2ea01a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.094+0000 7f2ea6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ea010a110 0x7f2ea019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.094+0000 7f2ea6a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea0113e60 con 0x7f2ea01043c0 2024-05-27T09:40:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.094+0000 7f2ea6a00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea0113ce0 con 0x7f2ea0109c70 2024-05-27T09:40:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.094+0000 7f2ea6a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea0113fe0 con 0x7f2ea010a110 2024-05-27T09:40:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.094+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ea01043c0 0x7f2ea01a2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.094+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ea01043c0 0x7f2ea01a2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48476/0 (socket says 172.21.15.44:48476) 2024-05-27T09:40:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.094+0000 7f2e9fe00640 1 -- 172.21.15.44:0/1143084378 learned_addr learned my addr 172.21.15.44:0/1143084378 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.094+0000 7f2ea4a00640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ea010a110 0x7f2ea019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.094+0000 7f2e9f400640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ea0109c70 0x7f2ea01a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.095+0000 7f2e9fe00640 1 -- 172.21.15.44:0/1143084378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ea010a110 msgr2=0x7f2ea019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.095+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ea010a110 0x7f2ea019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.095+0000 7f2e9fe00640 1 -- 172.21.15.44:0/1143084378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ea0109c70 msgr2=0x7f2ea01a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.095+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ea0109c70 0x7f2ea01a2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.095+0000 7f2e9fe00640 1 -- 172.21.15.44:0/1143084378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea019cf00 con 0x7f2ea01043c0 2024-05-27T09:40:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.095+0000 7f2e9f400640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ea0109c70 0x7f2ea01a2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.095+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ea01043c0 0x7f2ea01a2650 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000eb90 tx=0x7f2e9000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.096+0000 7f2e9cc00640 1 -- 172.21.15.44:0/1143084378 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90010630 con 0x7f2ea01043c0 2024-05-27T09:40:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.096+0000 7f2e9cc00640 1 -- 172.21.15.44:0/1143084378 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e9000ee90 con 0x7f2ea01043c0 2024-05-27T09:40:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.096+0000 7f2ea6a00640 1 -- 172.21.15.44:0/1143084378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea019d1f0 con 0x7f2ea01043c0 2024-05-27T09:40:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.096+0000 7f2e9cc00640 1 -- 172.21.15.44:0/1143084378 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90018ac0 con 0x7f2ea01043c0 2024-05-27T09:40:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.096+0000 7f2ea6a00640 1 -- 172.21.15.44:0/1143084378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea01a94d0 con 0x7f2ea01043c0 2024-05-27T09:40:50.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.100+0000 7f2e9cc00640 1 -- 172.21.15.44:0/1143084378 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2e90018c60 con 0x7f2ea01043c0 2024-05-27T09:40:50.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.102+0000 7f2e9cc00640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2e840b2950 0x7f2e840b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.102+0000 7f2e9cc00640 1 -- 172.21.15.44:0/1143084378 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f2e900e2c00 con 0x7f2ea01043c0 2024-05-27T09:40:50.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.102+0000 7f2e9f400640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2e840b2950 0x7f2e840b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.103+0000 7f2e9f400640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2e840b2950 0x7f2e840b4e60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c0096c0 tx=0x7f2e8c009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:50.127 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:49 smithi044 ceph-mon[31618]: pgmap v170: 97 pgs: 97 active+clean; 4.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 15 op/s 2024-05-27T09:40:50.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.170+0000 7f2ead992740 1 Processor -- start 2024-05-27T09:40:50.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.171+0000 7f2ead992740 1 -- start start 2024-05-27T09:40:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.171+0000 7f2ead992740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506a71b0 0x565350a49860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.171+0000 7f2ead992740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a49da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.171+0000 7f2ead992740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653506864d0 0x565350a4a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.171+0000 7f2ead992740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5653509bab90 con 0x5653506864d0 2024-05-27T09:40:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.171+0000 7f2ead992740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5653509aa730 con 0x5653506a71b0 2024-05-27T09:40:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.171+0000 7f2ead992740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5653509ae770 con 0x56535067e2f0 2024-05-27T09:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.172+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506a71b0 0x565350a49860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.172+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506a71b0 0x565350a49860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37836/0 (socket says 172.21.15.44:37836) 2024-05-27T09:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.172+0000 7f2e9fe00640 1 -- 172.21.15.44:0/3681973282 learned_addr learned my addr 172.21.15.44:0/3681973282 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.172+0000 7f2e9f400640 1 --2- 172.21.15.44:0/3681973282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a49da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.172+0000 7f2ea4a00640 1 --2- 172.21.15.44:0/3681973282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653506864d0 0x565350a4a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.172+0000 7f2e9fe00640 1 -- 172.21.15.44:0/3681973282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 msgr2=0x565350a49da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.172+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/3681973282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a49da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.172+0000 7f2e9fe00640 1 -- 172.21.15.44:0/3681973282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653506864d0 msgr2=0x565350a4a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.172+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/3681973282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653506864d0 0x565350a4a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.172+0000 7f2e9fe00640 1 -- 172.21.15.44:0/3681973282 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653509438b0 con 0x5653506a71b0 2024-05-27T09:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.173+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/3681973282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506a71b0 0x565350a49860 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2e90005fa0 tx=0x7f2e90009200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c4f4bbf62b09b33 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.173+0000 7f2e95200640 1 -- 172.21.15.44:0/3681973282 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900ae260 con 0x5653506a71b0 2024-05-27T09:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.173+0000 7f2e95200640 1 -- 172.21.15.44:0/3681973282 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e900b2d10 con 0x5653506a71b0 2024-05-27T09:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.173+0000 7f2e95200640 1 -- 172.21.15.44:0/3681973282 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900a8c30 con 0x5653506a71b0 2024-05-27T09:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.174+0000 7f2ead992740 1 -- 172.21.15.44:0/3681973282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506a71b0 msgr2=0x565350a49860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.174+0000 7f2ead992740 1 --2- 172.21.15.44:0/3681973282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506a71b0 0x565350a49860 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2e90005fa0 tx=0x7f2e90009200 comp rx=0 tx=0).stop 2024-05-27T09:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.174+0000 7f2ead992740 1 -- 172.21.15.44:0/3681973282 shutdown_connections 2024-05-27T09:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.174+0000 7f2ead992740 1 --2- 172.21.15.44:0/3681973282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653506864d0 0x565350a4a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.174+0000 7f2ead992740 1 --2- 172.21.15.44:0/3681973282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a49da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.174+0000 7f2ead992740 1 --2- 172.21.15.44:0/3681973282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506a71b0 0x565350a49860 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.174+0000 7f2ead992740 1 -- 172.21.15.44:0/3681973282 >> 172.21.15.44:0/3681973282 conn(0x56535069b300 msgr2=0x565350a46010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.174+0000 7f2ead992740 1 -- 172.21.15.44:0/3681973282 shutdown_connections 2024-05-27T09:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.175+0000 7f2ead992740 1 -- 172.21.15.44:0/3681973282 wait complete. 2024-05-27T09:40:50.183 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:40:50.183 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:40:50.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.185+0000 7f2ead992740 1 -- start start 2024-05-27T09:40:50.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.185+0000 7f2ead992740 1 Processor -- start 2024-05-27T09:40:50.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.186+0000 7f2ead992740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56535067e2f0 0x565350a53a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.186+0000 7f2ead992740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653506864d0 0x565350a53f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.186+0000 7f2ead992740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653506a71b0 0x565350a5fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.186+0000 7f2ead992740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x565350690790 con 0x5653506a71b0 2024-05-27T09:40:50.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.186+0000 7f2ead992740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x565350952bc0 con 0x56535067e2f0 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.186+0000 7f2ead992740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x565350a56990 con 0x5653506864d0 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.186+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56535067e2f0 0x565350a53a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.186+0000 7f2ea4a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653506a71b0 0x565350a5fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.186+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56535067e2f0 0x565350a53a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37844/0 (socket says 172.21.15.44:37844) 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.186+0000 7f2e9fe00640 1 -- 172.21.15.44:0/1605148237 learned_addr learned my addr 172.21.15.44:0/1605148237 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.186+0000 7f2e9f400640 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653506864d0 0x565350a53f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.187+0000 7f2e9fe00640 1 -- 172.21.15.44:0/1605148237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653506864d0 msgr2=0x565350a53f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.187+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653506864d0 0x565350a53f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.187+0000 7f2e9fe00640 1 -- 172.21.15.44:0/1605148237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653506a71b0 msgr2=0x565350a5fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.187+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653506a71b0 0x565350a5fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.187+0000 7f2e9fe00640 1 -- 172.21.15.44:0/1605148237 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653506bf210 con 0x56535067e2f0 2024-05-27T09:40:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.187+0000 7f2e9f400640 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653506864d0 0x565350a53f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:50.191 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.187+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56535067e2f0 0x565350a53a40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2e900a71d0 tx=0x7f2e9000cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fab988f2eab9c28f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:50.191 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.187+0000 7f2e89600640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:40:50.191 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.188+0000 7f2e89600640 1 -- 172.21.15.44:0/1605148237 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900ae260 con 0x56535067e2f0 2024-05-27T09:40:50.191 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.188+0000 7f2e89600640 1 -- 172.21.15.44:0/1605148237 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f2e900a5a40 con 0x56535067e2f0 2024-05-27T09:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.196+0000 7f2ead992740 10 client.0 _ll_register_callbacks cb 0x565350930400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.197+0000 7f2ead992740 1 -- 172.21.15.44:0/1605148237 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e900b2d10 con 0x56535067e2f0 2024-05-27T09:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.197+0000 7f2e89600640 1 -- 172.21.15.44:0/1605148237 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e900a91c0 con 0x56535067e2f0 2024-05-27T09:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.199+0000 7f2ead992740 10 client.26138 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.199+0000 7f2ead992740 1 -- 172.21.15.44:0/1605148237 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({fsmap=26}) -- 0x56535078b620 con 0x56535067e2f0 2024-05-27T09:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.199+0000 7f2e89600640 1 -- 172.21.15.44:0/1605148237 <== mon.2 v2:172.21.15.50:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2e9008d020 con 0x56535067e2f0 2024-05-27T09:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.199+0000 7f2ead992740 10 client.26138 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.199+0000 7f2ead992740 10 client.26138 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.199+0000 7f2ead992740 20 client.26138 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.199+0000 7f2ead992740 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565350665d80 0x565350a65950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.199+0000 7f2ead992740 4 client.26138 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.200+0000 7f2ead992740 1 -- 172.21.15.44:0/1605148237 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653509bcae0 con 0x565350665d80 2024-05-27T09:40:50.202 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.200+0000 7f2e9f400640 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565350665d80 0x565350a65950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.202 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.200+0000 7f2e9f400640 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565350665d80 0x565350a65950 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fe1e5dbefecba02 server_cookie=2f4756d4ac0e8ab8 in_seq=0 out_seq=0 2024-05-27T09:40:50.202 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.200+0000 7f2e89600640 10 client.26138 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:50.203 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.201+0000 7f2e89600640 1 -- 172.21.15.44:0/1605148237 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5653509bcae0 con 0x565350665d80 2024-05-27T09:40:50.203 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.201+0000 7f2e89600640 10 client.26138 handle_command_reply: tid=0 2024-05-27T09:40:50.203 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.201+0000 7f2ead992740 1 client.26138 shutdown 2024-05-27T09:40:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.202+0000 7f2ead992740 1 -- 172.21.15.44:0/1605148237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56535067e2f0 msgr2=0x565350a53a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.202+0000 7f2ead992740 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56535067e2f0 0x565350a53a40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2e900a71d0 tx=0x7f2e9000cf90 comp rx=0 tx=0).stop 2024-05-27T09:40:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.202+0000 7f2ead992740 1 -- 172.21.15.44:0/1605148237 shutdown_connections 2024-05-27T09:40:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.202+0000 7f2ead992740 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x565350665d80 0x565350a65950 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.202+0000 7f2ead992740 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653506a71b0 0x565350a5fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.202+0000 7f2ead992740 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653506864d0 0x565350a53f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.202+0000 7f2ead992740 1 --2- 172.21.15.44:0/1605148237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56535067e2f0 0x565350a53a40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.202+0000 7f2e89600640 0 client.26138 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.202+0000 7f2ead992740 1 -- 172.21.15.44:0/1605148237 >> 172.21.15.44:0/1605148237 conn(0x56535069b300 msgr2=0x565350a49690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:50.205 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.203+0000 7f2ead992740 1 -- 172.21.15.44:0/1605148237 shutdown_connections 2024-05-27T09:40:50.205 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.203+0000 7f2ead992740 1 -- 172.21.15.44:0/1605148237 wait complete. 2024-05-27T09:40:50.205 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.203+0000 7f2ead992740 20 client.26138 trim_cache size 0 max 16384 2024-05-27T09:40:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.223+0000 7f2ead992740 1 Processor -- start 2024-05-27T09:40:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.223+0000 7f2ead992740 1 -- start start 2024-05-27T09:40:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.223+0000 7f2ead992740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 0x565350a56170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.223+0000 7f2ead992740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a45330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.223+0000 7f2ead992740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506864d0 0x565350a5fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.223+0000 7f2ead992740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01b4640 con 0x56535069b300 2024-05-27T09:40:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.223+0000 7f2ead992740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01b1f10 con 0x5653506864d0 2024-05-27T09:40:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.223+0000 7f2ead992740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01afd60 con 0x56535067e2f0 2024-05-27T09:40:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.224+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 0x565350a56170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.224+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 0x565350a56170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48502/0 (socket says 172.21.15.44:48502) 2024-05-27T09:40:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.224+0000 7f2e9f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a45330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.224+0000 7f2e9fe00640 1 -- 172.21.15.44:0/3221090824 learned_addr learned my addr 172.21.15.44:0/3221090824 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.224+0000 7f2ea4a00640 1 --2- 172.21.15.44:0/3221090824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506864d0 0x565350a5fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.224+0000 7f2e9f400640 1 -- 172.21.15.44:0/3221090824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506864d0 msgr2=0x565350a5fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.224+0000 7f2e9f400640 1 --2- 172.21.15.44:0/3221090824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506864d0 0x565350a5fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.224+0000 7f2e9f400640 1 -- 172.21.15.44:0/3221090824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 msgr2=0x565350a56170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.224+0000 7f2e9f400640 1 --2- 172.21.15.44:0/3221090824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 0x565350a56170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.224+0000 7f2e9f400640 1 -- 172.21.15.44:0/3221090824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565350a51f90 con 0x56535067e2f0 2024-05-27T09:40:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.225+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/3221090824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 0x565350a56170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.225+0000 7f2e9f400640 1 --2- 172.21.15.44:0/3221090824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a45330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0110ae0 tx=0x7f2e8c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0694a935f9bcb48 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.225+0000 7f2e8be00640 1 -- 172.21.15.44:0/3221090824 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c01ed90 con 0x56535067e2f0 2024-05-27T09:40:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.225+0000 7f2e8be00640 1 -- 172.21.15.44:0/3221090824 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e8c0348e0 con 0x56535067e2f0 2024-05-27T09:40:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.225+0000 7f2e8be00640 1 -- 172.21.15.44:0/3221090824 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c02a7e0 con 0x56535067e2f0 2024-05-27T09:40:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.226+0000 7f2ead992740 1 -- 172.21.15.44:0/3221090824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 msgr2=0x565350a45330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.226+0000 7f2ead992740 1 --2- 172.21.15.44:0/3221090824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a45330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0110ae0 tx=0x7f2e8c014000 comp rx=0 tx=0).stop 2024-05-27T09:40:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.226+0000 7f2ead992740 1 -- 172.21.15.44:0/3221090824 shutdown_connections 2024-05-27T09:40:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.226+0000 7f2ead992740 1 --2- 172.21.15.44:0/3221090824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506864d0 0x565350a5fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.226+0000 7f2ead992740 1 --2- 172.21.15.44:0/3221090824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a45330 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.226+0000 7f2ead992740 1 --2- 172.21.15.44:0/3221090824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 0x565350a56170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.226+0000 7f2ead992740 1 -- 172.21.15.44:0/3221090824 >> 172.21.15.44:0/3221090824 conn(0x565350a4ddc0 msgr2=0x565350a532d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.227+0000 7f2ead992740 1 -- 172.21.15.44:0/3221090824 shutdown_connections 2024-05-27T09:40:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.227+0000 7f2ead992740 1 -- 172.21.15.44:0/3221090824 wait complete. 2024-05-27T09:40:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.227+0000 7f2ead992740 1 -- start start 2024-05-27T09:40:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.227+0000 7f2ead992740 1 Processor -- start 2024-05-27T09:40:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2ead992740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a65e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2ead992740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506864d0 0x565350a663c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2ead992740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 0x565350acbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2ead992740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01b03d0 con 0x56535069b300 2024-05-27T09:40:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2ead992740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01b1840 con 0x5653506864d0 2024-05-27T09:40:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2ead992740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01b0aa0 con 0x56535067e2f0 2024-05-27T09:40:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2ead992740 10 client.0 _ll_register_callbacks cb 0x7f2e8c00e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a65e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2e9f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506864d0 0x565350a663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2ea4a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 0x565350acbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a65e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43334/0 (socket says 172.21.15.44:43334) 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.228+0000 7f2e9fe00640 1 -- 172.21.15.44:0/2956600572 learned_addr learned my addr 172.21.15.44:0/2956600572 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.229+0000 7f2e9fe00640 1 -- 172.21.15.44:0/2956600572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506864d0 msgr2=0x565350a663c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.229+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506864d0 0x565350a663c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.229+0000 7f2e9fe00640 1 -- 172.21.15.44:0/2956600572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 msgr2=0x565350acbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.229+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 0x565350acbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.229+0000 7f2e9fe00640 1 -- 172.21.15.44:0/2956600572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565350acc2d0 con 0x56535067e2f0 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.229+0000 7f2e9f400640 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506864d0 0x565350a663c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.229+0000 7f2e9fe00640 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a65e80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2e900c42f0 tx=0x7f2e90018590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0b6d5700415bb03 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.229+0000 7f2e89600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:40:50.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.229+0000 7f2e89600640 1 -- 172.21.15.44:0/2956600572 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900b7060 con 0x56535067e2f0 2024-05-27T09:40:50.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.230+0000 7f2e89600640 1 -- 172.21.15.44:0/2956600572 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f2e900c4370 con 0x56535067e2f0 2024-05-27T09:40:50.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.230+0000 7f2ead992740 1 -- 172.21.15.44:0/2956600572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e8c0348e0 con 0x56535067e2f0 2024-05-27T09:40:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.230+0000 7f2e89600640 1 -- 172.21.15.44:0/2956600572 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e900ae260 con 0x56535067e2f0 2024-05-27T09:40:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.230+0000 7f2ead992740 10 client.26155 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.230+0000 7f2ead992740 1 -- 172.21.15.44:0/2956600572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x565350acc2d0 con 0x56535067e2f0 2024-05-27T09:40:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.230+0000 7f2e89600640 1 -- 172.21.15.44:0/2956600572 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e8c0348e0 con 0x56535067e2f0 2024-05-27T09:40:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.230+0000 7f2e89600640 1 -- 172.21.15.44:0/2956600572 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2e9008d040 con 0x56535067e2f0 2024-05-27T09:40:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.231+0000 7f2ead992740 10 client.26155 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.231+0000 7f2ead992740 10 client.26155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:40:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.231+0000 7f2ead992740 20 client.26155 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.231+0000 7f2ead992740 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5653506a71b0 0x565350acf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.231+0000 7f2ead992740 4 client.26155 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:40:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.231+0000 7f2ead992740 1 -- 172.21.15.44:0/2956600572 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5653509b62c0 con 0x5653506a71b0 2024-05-27T09:40:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.231+0000 7f2e9f400640 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5653506a71b0 0x565350acf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.232+0000 7f2e9f400640 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5653506a71b0 0x565350acf860 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=125f3e140c4c8247 server_cookie=8827e5ebc6324fc7 in_seq=0 out_seq=0 2024-05-27T09:40:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.232+0000 7f2e89600640 10 client.26155 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.238+0000 7f1d2ea00700 1 -- 172.21.15.44:0/3147596034 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e7840ef700 con 0x55e784254440 2024-05-27T09:40:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.238+0000 7f1d2ea00700 10 client.26134 handle_command_reply: tid=0 2024-05-27T09:40:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.238+0000 7f1d5e582b80 1 client.26134 shutdown 2024-05-27T09:40:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.238+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3147596034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 msgr2=0x55e7841d55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.238+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841d55b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1d40007f00 tx=0x7f1d40014240 comp rx=0 tx=0).stop 2024-05-27T09:40:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.239+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3147596034 shutdown_connections 2024-05-27T09:40:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.239+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e784254440 0x55e784256860 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.239+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7841cc380 0x55e7841d55b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.239+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e784046fd0 0x55e7841d87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.239+0000 7f1d5e582b80 1 --2- 172.21.15.44:0/3147596034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7841c64d0 0x55e7841d5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.239+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3147596034 >> 172.21.15.44:0/3147596034 conn(0x55e7840a7150 msgr2=0x55e7841ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.239+0000 7f1d2ea00700 0 client.26134 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.239+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3147596034 shutdown_connections 2024-05-27T09:40:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.239+0000 7f1d5e582b80 1 -- 172.21.15.44:0/3147596034 wait complete. 2024-05-27T09:40:50.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.240+0000 7f1d5e582b80 20 client.26134 trim_cache size 0 max 16384 2024-05-27T09:40:50.242 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:40:50.242 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 26, 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 10, 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "17b76354": { 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 10, 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 150.0, 2024-05-27T09:40:50.243 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:40:50.244 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:40:50.244 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:40:50.244 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:40:50.244 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:40:50.244 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:40:50.244 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:40:50.244 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:40:50.244 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:40:50.244 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:40:50.244 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:40:50.244 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:40:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.241+0000 7f1d4fa00700 1 -- 172.21.15.44:0/1133824269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1d300b2a70 msgr2=0x7f1d300b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.241+0000 7f1d4fa00700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1d300b2a70 0x7f1d300b4f80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1d48103390 tx=0x7f1d34009310 comp rx=0 tx=0).stop 2024-05-27T09:40:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.241+0000 7f1d4fa00700 1 -- 172.21.15.44:0/1133824269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d48076c00 msgr2=0x7f1d4819d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.241+0000 7f1d4fa00700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d48076c00 0x7f1d4819d0d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009230 tx=0x7f1d3c00fb20 comp rx=0 tx=0).stop 2024-05-27T09:40:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.241+0000 7f1d4fa00700 1 -- 172.21.15.44:0/1133824269 shutdown_connections 2024-05-27T09:40:50.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.241+0000 7f1d4fa00700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1d300b2a70 0x7f1d300b4f80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.241+0000 7f1d4fa00700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d48110bd0 0x7f1d481a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.241+0000 7f1d4fa00700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d48076c00 0x7f1d4819d0d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.241+0000 7f1d4fa00700 1 --2- 172.21.15.44:0/1133824269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d48077550 0x7f1d4819d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.241+0000 7f1d4fa00700 1 -- 172.21.15.44:0/1133824269 >> 172.21.15.44:0/1133824269 conn(0x7f1d480ffe90 msgr2=0x7f1d48100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.241+0000 7f1d4fa00700 1 -- 172.21.15.44:0/1133824269 shutdown_connections 2024-05-27T09:40:50.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:50.242+0000 7f1d4fa00700 1 -- 172.21.15.44:0/1133824269 wait complete. 2024-05-27T09:40:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.253+0000 7f2e89600640 1 -- 172.21.15.44:0/2956600572 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5653509b62c0 con 0x5653506a71b0 2024-05-27T09:40:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.253+0000 7f2e89600640 10 client.26155 handle_command_reply: tid=0 2024-05-27T09:40:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.253+0000 7f2ead992740 1 client.26155 shutdown 2024-05-27T09:40:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2ead992740 1 -- 172.21.15.44:0/2956600572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 msgr2=0x565350a65e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2ead992740 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a65e80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2e900c42f0 tx=0x7f2e90018590 comp rx=0 tx=0).stop 2024-05-27T09:40:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2ead992740 1 -- 172.21.15.44:0/2956600572 shutdown_connections 2024-05-27T09:40:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2ead992740 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5653506a71b0 0x565350acf860 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2ead992740 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56535069b300 0x565350acbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2ead992740 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5653506864d0 0x565350a663c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2ead992740 1 --2- 172.21.15.44:0/2956600572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56535067e2f0 0x565350a65e80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2ead992740 1 -- 172.21.15.44:0/2956600572 >> 172.21.15.44:0/2956600572 conn(0x565350a4ddc0 msgr2=0x565350a53cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2e89600640 0 client.26155 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:40:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2ead992740 1 -- 172.21.15.44:0/2956600572 shutdown_connections 2024-05-27T09:40:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2ead992740 1 -- 172.21.15.44:0/2956600572 wait complete. 2024-05-27T09:40:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.254+0000 7f2ead992740 20 client.26155 trim_cache size 0 max 16384 2024-05-27T09:40:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:40:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:40:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:40:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.255+0000 7f2ea6a00640 1 -- 172.21.15.44:0/1143084378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2e840b2950 msgr2=0x7f2e840b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.255+0000 7f2ea6a00640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2e840b2950 0x7f2e840b4e60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c0096c0 tx=0x7f2e8c009260 comp rx=0 tx=0).stop 2024-05-27T09:40:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.255+0000 7f2ea6a00640 1 -- 172.21.15.44:0/1143084378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ea01043c0 msgr2=0x7f2ea01a2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.255+0000 7f2ea6a00640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ea01043c0 0x7f2ea01a2650 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000eb90 tx=0x7f2e9000c6a0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.255+0000 7f2ea6a00640 1 -- 172.21.15.44:0/1143084378 shutdown_connections 2024-05-27T09:40:50.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.255+0000 7f2ea6a00640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ea010a110 0x7f2ea019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.255+0000 7f2ea6a00640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ea0109c70 0x7f2ea01a2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.255+0000 7f2ea6a00640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2e840b2950 0x7f2e840b4e60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.255+0000 7f2ea6a00640 1 --2- 172.21.15.44:0/1143084378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ea01043c0 0x7f2ea01a2650 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:50.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.255+0000 7f2ea6a00640 1 -- 172.21.15.44:0/1143084378 >> 172.21.15.44:0/1143084378 conn(0x7f2ea00ff350 msgr2=0x7f2ea0106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:50.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.256+0000 7f2ea6a00640 1 -- 172.21.15.44:0/1143084378 shutdown_connections 2024-05-27T09:40:50.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:40:50.256+0000 7f2ea6a00640 1 -- 172.21.15.44:0/1143084378 wait complete. 2024-05-27T09:40:50.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:49 smithi050 ceph-mon[36371]: pgmap v170: 97 pgs: 97 active+clean; 4.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 15 op/s 2024-05-27T09:40:50.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:49 smithi189 ceph-mon[36671]: pgmap v170: 97 pgs: 97 active+clean; 4.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 15 op/s 2024-05-27T09:40:50.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:50 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:40:51.064 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '17b76354', quiesce took 54.6 seconds. Will release after: -24.6 2024-05-27T09:40:51.064 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:40:51.123 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:40:51.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:50 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:40:51.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:50 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:40:51.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:51 smithi044 ceph-mon[31618]: pgmap v171: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 38 KiB/s wr, 20 op/s 2024-05-27T09:40:52.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:51 smithi050 ceph-mon[36371]: pgmap v171: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 38 KiB/s wr, 20 op/s 2024-05-27T09:40:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:51 smithi189 ceph-mon[36671]: pgmap v171: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 38 KiB/s wr, 20 op/s 2024-05-27T09:40:52.607 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:40:53.176 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:52 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:40:53.176 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:52 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:40:53.176 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:52 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:40:53.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:52 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:40:53.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:52 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:40:53.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:52 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:40:53.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:52 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:40:53.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:52 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:40:53.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:52 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:40:54.174 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:53 smithi044 ceph-mon[31618]: pgmap v172: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-05-27T09:40:54.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:53 smithi050 ceph-mon[36371]: pgmap v172: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-05-27T09:40:54.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:53 smithi189 ceph-mon[36671]: pgmap v172: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-05-27T09:40:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.509+0000 7f7a7e200700 1 -- 172.21.15.44:0/413903510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a7810c5b0 msgr2=0x7f7a7810c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.509+0000 7f7a7e200700 1 --2- 172.21.15.44:0/413903510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a7810c5b0 0x7f7a7810c9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800b0a0 tx=0x7f7a6800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:40:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.509+0000 7f7a7e200700 1 -- 172.21.15.44:0/413903510 shutdown_connections 2024-05-27T09:40:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.509+0000 7f7a7e200700 1 --2- 172.21.15.44:0/413903510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a78106ec0 0x7f7a78113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.509+0000 7f7a7e200700 1 --2- 172.21.15.44:0/413903510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a7810c5b0 0x7f7a7810c9c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.509+0000 7f7a7e200700 1 --2- 172.21.15.44:0/413903510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a78106570 0x7f7a78106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.509+0000 7f7a7e200700 1 -- 172.21.15.44:0/413903510 >> 172.21.15.44:0/413903510 conn(0x7f7a78102070 msgr2=0x7f7a781044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.510+0000 7f7a7e200700 1 -- 172.21.15.44:0/413903510 shutdown_connections 2024-05-27T09:40:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.510+0000 7f7a7e200700 1 -- 172.21.15.44:0/413903510 wait complete. 2024-05-27T09:40:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.511+0000 7f7a7e200700 1 Processor -- start 2024-05-27T09:40:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.511+0000 7f7a7e200700 1 -- start start 2024-05-27T09:40:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.512+0000 7f7a7e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a78106570 0x7f7a7819d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.512+0000 7f7a7e200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a78106ec0 0x7f7a7819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.512+0000 7f7a7e200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a7810c5b0 0x7f7a781a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.512+0000 7f7a7e200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7a78116720 con 0x7f7a78106ec0 2024-05-27T09:40:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.512+0000 7f7a7e200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7a781165a0 con 0x7f7a7810c5b0 2024-05-27T09:40:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.512+0000 7f7a7e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7a781168a0 con 0x7f7a78106570 2024-05-27T09:40:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.512+0000 7f7a6ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a78106ec0 0x7f7a7819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.512+0000 7f7a77400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a78106570 0x7f7a7819d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.512+0000 7f7a77400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a78106570 0x7f7a7819d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38176/0 (socket says 172.21.15.44:38176) 2024-05-27T09:40:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.512+0000 7f7a6ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a78106ec0 0x7f7a7819d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52862/0 (socket says 172.21.15.44:52862) 2024-05-27T09:40:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.512+0000 7f7a77400700 1 -- 172.21.15.44:0/665024667 learned_addr learned my addr 172.21.15.44:0/665024667 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.513+0000 7f7a77e00700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a7810c5b0 0x7f7a781a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.513+0000 7f7a77400700 1 -- 172.21.15.44:0/665024667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a7810c5b0 msgr2=0x7f7a781a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.513+0000 7f7a77400700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a7810c5b0 0x7f7a781a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.513+0000 7f7a77400700 1 -- 172.21.15.44:0/665024667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a78106ec0 msgr2=0x7f7a7819d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.513+0000 7f7a77400700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a78106ec0 0x7f7a7819d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.513+0000 7f7a77400700 1 -- 172.21.15.44:0/665024667 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a781a1f20 con 0x7f7a78106570 2024-05-27T09:40:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.513+0000 7f7a77400700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a78106570 0x7f7a7819d050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800b780 tx=0x7f7a68004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.514+0000 7f7a74c00700 1 -- 172.21.15.44:0/665024667 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68021070 con 0x7f7a78106570 2024-05-27T09:40:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.514+0000 7f7a74c00700 1 -- 172.21.15.44:0/665024667 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a68007e30 con 0x7f7a78106570 2024-05-27T09:40:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.514+0000 7f7a74c00700 1 -- 172.21.15.44:0/665024667 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a680046a0 con 0x7f7a78106570 2024-05-27T09:40:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.514+0000 7f7a7e200700 1 -- 172.21.15.44:0/665024667 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a781a21b0 con 0x7f7a78106570 2024-05-27T09:40:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.514+0000 7f7a7e200700 1 -- 172.21.15.44:0/665024667 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a781a9b30 con 0x7f7a78106570 2024-05-27T09:40:54.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.515+0000 7f7a7e200700 1 -- 172.21.15.44:0/665024667 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a780512b0 con 0x7f7a78106570 2024-05-27T09:40:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.518+0000 7f7a74c00700 1 -- 172.21.15.44:0/665024667 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f7a68007980 con 0x7f7a78106570 2024-05-27T09:40:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.520+0000 7f7a74c00700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a5c0b2a70 0x7f7a5c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:54.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.520+0000 7f7a74c00700 1 -- 172.21.15.44:0/665024667 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f7a6800f020 con 0x7f7a78106570 2024-05-27T09:40:54.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.520+0000 7f7a6ea00700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a5c0b2a70 0x7f7a5c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:54.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.521+0000 7f7a6ea00700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a5c0b2a70 0x7f7a5c0b4f80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a60009680 tx=0x7f7a60009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:54.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.522+0000 7f7a74c00700 1 -- 172.21.15.44:0/665024667 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7a6802a020 con 0x7f7a78106570 2024-05-27T09:40:54.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.820+0000 7f7a7e200700 1 -- 172.21.15.44:0/665024667 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a7819e3b0 con 0x7f7a78106570 2024-05-27T09:40:54.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.821+0000 7f7a74c00700 1 -- 172.21.15.44:0/665024667 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7f7a6802e3e0 con 0x7f7a78106570 2024-05-27T09:40:54.823 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:40:54.823 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:40:54.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 -- 172.21.15.44:0/665024667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a5c0b2a70 msgr2=0x7f7a5c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:54.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a5c0b2a70 0x7f7a5c0b4f80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a60009680 tx=0x7f7a60009260 comp rx=0 tx=0).stop 2024-05-27T09:40:54.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 -- 172.21.15.44:0/665024667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a78106570 msgr2=0x7f7a7819d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:54.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a78106570 0x7f7a7819d050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800b780 tx=0x7f7a68004d00 comp rx=0 tx=0).stop 2024-05-27T09:40:54.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 -- 172.21.15.44:0/665024667 shutdown_connections 2024-05-27T09:40:54.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7a5c0b2a70 0x7f7a5c0b4f80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:54.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a7810c5b0 0x7f7a781a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:54.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a78106570 0x7f7a7819d050 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:54.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 --2- 172.21.15.44:0/665024667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a78106ec0 0x7f7a7819d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:54.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 -- 172.21.15.44:0/665024667 >> 172.21.15.44:0/665024667 conn(0x7f7a78102070 msgr2=0x7f7a78104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:54.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 -- 172.21.15.44:0/665024667 shutdown_connections 2024-05-27T09:40:54.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:54.825+0000 7f7a7e200700 1 -- 172.21.15.44:0/665024667 wait complete. 2024-05-27T09:40:54.829 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:40:55.143 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:54 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/665024667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:40:55.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:54 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/665024667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:40:55.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:54 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/665024667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:40:55.703 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:40:55.907 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:55 smithi044 ceph-mon[31618]: pgmap v173: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-05-27T09:40:56.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.272+0000 7f6cd6000700 1 -- 172.21.15.44:0/2071824278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cd0110c20 msgr2=0x7f6cd0113040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:56.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.272+0000 7f6cd6000700 1 --2- 172.21.15.44:0/2071824278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cd0110c20 0x7f6cd0113040 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8009f90 tx=0x7f6cc800b330 comp rx=0 tx=0).stop 2024-05-27T09:40:56.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.272+0000 7f6cd6000700 1 -- 172.21.15.44:0/2071824278 shutdown_connections 2024-05-27T09:40:56.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.272+0000 7f6cd6000700 1 --2- 172.21.15.44:0/2071824278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cd0110c20 0x7f6cd0113040 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:56.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.272+0000 7f6cd6000700 1 --2- 172.21.15.44:0/2071824278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd0103070 0x7f6cd0103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:56.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.272+0000 7f6cd6000700 1 --2- 172.21.15.44:0/2071824278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cd0103a50 0x7f6cd01106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:56.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.272+0000 7f6cd6000700 1 -- 172.21.15.44:0/2071824278 >> 172.21.15.44:0/2071824278 conn(0x7f6cd00fedb0 msgr2=0x7f6cd01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:56.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.272+0000 7f6cd6000700 1 -- 172.21.15.44:0/2071824278 shutdown_connections 2024-05-27T09:40:56.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.272+0000 7f6cd6000700 1 -- 172.21.15.44:0/2071824278 wait complete. 2024-05-27T09:40:56.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.273+0000 7f6cd6000700 1 Processor -- start 2024-05-27T09:40:56.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.273+0000 7f6cd6000700 1 -- start start 2024-05-27T09:40:56.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.274+0000 7f6cd6000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cd0103070 0x7f6cd019cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:56.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.274+0000 7f6cd6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd0103a50 0x7f6cd019d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:56.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.274+0000 7f6cd6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cd0110c20 0x7f6cd01a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:56.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.274+0000 7f6cd6000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0116680 con 0x7f6cd0110c20 2024-05-27T09:40:56.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.274+0000 7f6cd6000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0116500 con 0x7f6cd0103070 2024-05-27T09:40:56.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.274+0000 7f6cd6000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0116800 con 0x7f6cd0103a50 2024-05-27T09:40:56.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.274+0000 7f6ccea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd0103a50 0x7f6cd019d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:56.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.274+0000 7f6ccfe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cd0110c20 0x7f6cd01a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:56.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.274+0000 7f6ccea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd0103a50 0x7f6cd019d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38198/0 (socket says 172.21.15.44:38198) 2024-05-27T09:40:56.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.274+0000 7f6ccea00700 1 -- 172.21.15.44:0/1761891875 learned_addr learned my addr 172.21.15.44:0/1761891875 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:56.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.274+0000 7f6ccf400700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cd0103070 0x7f6cd019cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:56.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.275+0000 7f6ccea00700 1 -- 172.21.15.44:0/1761891875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cd0103070 msgr2=0x7f6cd019cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:56.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.275+0000 7f6ccea00700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cd0103070 0x7f6cd019cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:56.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.275+0000 7f6ccea00700 1 -- 172.21.15.44:0/1761891875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cd0110c20 msgr2=0x7f6cd01a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:56.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.275+0000 7f6ccea00700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cd0110c20 0x7f6cd01a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:56.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.275+0000 7f6ccea00700 1 -- 172.21.15.44:0/1761891875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd01a1e80 con 0x7f6cd0103a50 2024-05-27T09:40:56.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.275+0000 7f6ccea00700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd0103a50 0x7f6cd019d550 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00e940 tx=0x7f6cbc00ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:56.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.275+0000 7f6cc7e00700 1 -- 172.21.15.44:0/1761891875 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc00cbd0 con 0x7f6cd0103a50 2024-05-27T09:40:56.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.276+0000 7f6cc7e00700 1 -- 172.21.15.44:0/1761891875 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cbc00ee90 con 0x7f6cd0103a50 2024-05-27T09:40:56.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.276+0000 7f6cd6000700 1 -- 172.21.15.44:0/1761891875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd01a2170 con 0x7f6cd0103a50 2024-05-27T09:40:56.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.276+0000 7f6cc7e00700 1 -- 172.21.15.44:0/1761891875 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc003dc0 con 0x7f6cd0103a50 2024-05-27T09:40:56.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.276+0000 7f6cd6000700 1 -- 172.21.15.44:0/1761891875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd01a9a90 con 0x7f6cd0103a50 2024-05-27T09:40:56.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.277+0000 7f6cd6000700 1 -- 172.21.15.44:0/1761891875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cd00512b0 con 0x7f6cd0103a50 2024-05-27T09:40:56.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.280+0000 7f6cc7e00700 1 -- 172.21.15.44:0/1761891875 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6cbc003f60 con 0x7f6cd0103a50 2024-05-27T09:40:56.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.282+0000 7f6cc7e00700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6cb40b29f0 0x7f6cb40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:56.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.283+0000 7f6cc7e00700 1 -- 172.21.15.44:0/1761891875 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f6cbc0e28d0 con 0x7f6cd0103a50 2024-05-27T09:40:56.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.283+0000 7f6ccf400700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6cb40b29f0 0x7f6cb40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:56.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.283+0000 7f6cc7e00700 1 -- 172.21.15.44:0/1761891875 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6cbc0aa310 con 0x7f6cd0103a50 2024-05-27T09:40:56.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.284+0000 7f6ccf400700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6cb40b29f0 0x7f6cb40b4f00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009750 tx=0x7f6cc0009310 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:56.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:55 smithi050 ceph-mon[36371]: pgmap v173: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-05-27T09:40:56.331 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:40:56.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:55 smithi189 ceph-mon[36671]: pgmap v173: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-05-27T09:40:56.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.575+0000 7f6cd6000700 1 -- 172.21.15.44:0/1761891875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cd019e310 con 0x7f6cd0103a50 2024-05-27T09:40:56.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.576+0000 7f6cc7e00700 1 -- 172.21.15.44:0/1761891875 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11865 (secure 0 0 0) 0x7f6cbc0a9c10 con 0x7f6cd0103a50 2024-05-27T09:40:56.579 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:40:56.579 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:35:12.426904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:40:56.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.579+0000 7f6cd6000700 1 -- 172.21.15.44:0/1761891875 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6cb40b29f0 msgr2=0x7f6cb40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:56.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.580+0000 7f6cd6000700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6cb40b29f0 0x7f6cb40b4f00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009750 tx=0x7f6cc0009310 comp rx=0 tx=0).stop 2024-05-27T09:40:56.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.580+0000 7f6cd6000700 1 -- 172.21.15.44:0/1761891875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd0103a50 msgr2=0x7f6cd019d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:56.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.580+0000 7f6cd6000700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd0103a50 0x7f6cd019d550 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00e940 tx=0x7f6cbc00ec50 comp rx=0 tx=0).stop 2024-05-27T09:40:56.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.580+0000 7f6cd6000700 1 -- 172.21.15.44:0/1761891875 shutdown_connections 2024-05-27T09:40:56.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.580+0000 7f6cd6000700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6cb40b29f0 0x7f6cb40b4f00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:56.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.580+0000 7f6cd6000700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cd0110c20 0x7f6cd01a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:56.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.580+0000 7f6cd6000700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cd0103070 0x7f6cd019cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:56.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.580+0000 7f6cd6000700 1 --2- 172.21.15.44:0/1761891875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd0103a50 0x7f6cd019d550 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:56.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.580+0000 7f6cd6000700 1 -- 172.21.15.44:0/1761891875 >> 172.21.15.44:0/1761891875 conn(0x7f6cd00fedb0 msgr2=0x7f6cd0109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:56.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.580+0000 7f6cd6000700 1 -- 172.21.15.44:0/1761891875 shutdown_connections 2024-05-27T09:40:56.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:56.580+0000 7f6cd6000700 1 -- 172.21.15.44:0/1761891875 wait complete. 2024-05-27T09:40:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-05-27T09:40:57.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:56 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1761891875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:40:57.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:56 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1761891875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:40:57.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:56 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1761891875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:40:57.401 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db --set-id 17b76354 --release --await' 2024-05-27T09:40:57.402 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db --set-id 17b76354 --release --await 2024-05-27T09:40:58.206 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:57 smithi044 ceph-mon[31618]: pgmap v174: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-27T09:40:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:57 smithi050 ceph-mon[36371]: pgmap v174: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-27T09:40:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:57 smithi189 ceph-mon[36671]: pgmap v174: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-27T09:40:58.357 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:40:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.170+0000 7f6829600700 1 -- 172.21.15.44:0/910897029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6824110b90 msgr2=0x7f6824112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.170+0000 7f6829600700 1 --2- 172.21.15.44:0/910897029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6824110b90 0x7f6824112ff0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f681c00b0a0 tx=0x7f681c00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.170+0000 7f6829600700 1 -- 172.21.15.44:0/910897029 shutdown_connections 2024-05-27T09:40:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.170+0000 7f6829600700 1 --2- 172.21.15.44:0/910897029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6824110b90 0x7f6824112ff0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.170+0000 7f6829600700 1 --2- 172.21.15.44:0/910897029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6824075d20 0x7f6824076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.170+0000 7f6829600700 1 --2- 172.21.15.44:0/910897029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6824076670 0x7f6824110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.170+0000 7f6829600700 1 -- 172.21.15.44:0/910897029 >> 172.21.15.44:0/910897029 conn(0x7f6824100030 msgr2=0x7f6824102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:59.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.170+0000 7f6829600700 1 -- 172.21.15.44:0/910897029 shutdown_connections 2024-05-27T09:40:59.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.170+0000 7f6829600700 1 -- 172.21.15.44:0/910897029 wait complete. 2024-05-27T09:40:59.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.172+0000 7f6829600700 1 Processor -- start 2024-05-27T09:40:59.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.172+0000 7f6829600700 1 -- start start 2024-05-27T09:40:59.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.172+0000 7f6829600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6824075d20 0x7f682419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.172+0000 7f6829600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6824076670 0x7f682419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.172+0000 7f6829600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6824110b90 0x7f68241a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.172+0000 7f6829600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f68241166c0 con 0x7f6824076670 2024-05-27T09:40:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.172+0000 7f6829600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6824116540 con 0x7f6824110b90 2024-05-27T09:40:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.172+0000 7f6829600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6824116840 con 0x7f6824075d20 2024-05-27T09:40:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.173+0000 7f6822000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6824076670 0x7f682419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.173+0000 7f6822a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6824075d20 0x7f682419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.173+0000 7f6822a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6824075d20 0x7f682419d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38206/0 (socket says 172.21.15.44:38206) 2024-05-27T09:40:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.173+0000 7f6822a00700 1 -- 172.21.15.44:0/3854317130 learned_addr learned my addr 172.21.15.44:0/3854317130 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:59.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.173+0000 7f6823400700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6824110b90 0x7f68241a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.173+0000 7f6822a00700 1 -- 172.21.15.44:0/3854317130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6824110b90 msgr2=0x7f68241a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.173+0000 7f6822a00700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6824110b90 0x7f68241a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.173+0000 7f6822a00700 1 -- 172.21.15.44:0/3854317130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6824076670 msgr2=0x7f682419d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.173+0000 7f6822a00700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6824076670 0x7f682419d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.173+0000 7f6822a00700 1 -- 172.21.15.44:0/3854317130 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68241a1ee0 con 0x7f6824075d20 2024-05-27T09:40:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.174+0000 7f6822a00700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6824075d20 0x7f682419d010 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f681400e9a0 tx=0x7f681400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.174+0000 7f681b400700 1 -- 172.21.15.44:0/3854317130 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681400ca10 con 0x7f6824075d20 2024-05-27T09:40:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.174+0000 7f681b400700 1 -- 172.21.15.44:0/3854317130 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f681400cbb0 con 0x7f6824075d20 2024-05-27T09:40:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.174+0000 7f6829600700 1 -- 172.21.15.44:0/3854317130 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68241a21d0 con 0x7f6824075d20 2024-05-27T09:40:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.174+0000 7f6829600700 1 -- 172.21.15.44:0/3854317130 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68241a9af0 con 0x7f6824075d20 2024-05-27T09:40:59.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.174+0000 7f681b400700 1 -- 172.21.15.44:0/3854317130 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6814018680 con 0x7f6824075d20 2024-05-27T09:40:59.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.178+0000 7f681b400700 1 -- 172.21.15.44:0/3854317130 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6814018820 con 0x7f6824075d20 2024-05-27T09:40:59.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.180+0000 7f6838191b80 1 Processor -- start 2024-05-27T09:40:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.180+0000 7f6838191b80 1 -- start start 2024-05-27T09:40:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.180+0000 7f681b400700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68080b29f0 0x7f68080b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.180+0000 7f681b400700 1 -- 172.21.15.44:0/3854317130 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f68140e2860 con 0x7f6824075d20 2024-05-27T09:40:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.180+0000 7f6838191b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2880330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6838191b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2701e40 0x55a0f2880870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6838191b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2880e70 0x55a0f288d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6838191b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a0f27aee40 con 0x55a0f2880e70 2024-05-27T09:40:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6838191b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a0f279e9e0 con 0x55a0f27ed9f0 2024-05-27T09:40:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6838191b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a0f27a2a20 con 0x55a0f2701e40 2024-05-27T09:40:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6823400700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68080b29f0 0x7f68080b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6822a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2880330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6822a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2880330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34832/0 (socket says 172.21.15.44:34832) 2024-05-27T09:40:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6822a00700 1 -- 172.21.15.44:0/3891284813 learned_addr learned my addr 172.21.15.44:0/3891284813 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6822000700 1 --2- 172.21.15.44:0/3891284813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2701e40 0x55a0f2880870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6823400700 1 --2- 172.21.15.44:0/3891284813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2880e70 0x55a0f288d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6822a00700 1 -- 172.21.15.44:0/3891284813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2701e40 msgr2=0x55a0f2880870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:59.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6822a00700 1 --2- 172.21.15.44:0/3891284813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2701e40 0x55a0f2880870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6822a00700 1 -- 172.21.15.44:0/3891284813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2880e70 msgr2=0x55a0f288d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:59.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6822a00700 1 --2- 172.21.15.44:0/3891284813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2880e70 0x55a0f288d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6822a00700 1 -- 172.21.15.44:0/3891284813 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0f2767420 con 0x55a0f27ed9f0 2024-05-27T09:40:59.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.181+0000 7f6822000700 1 --2- 172.21.15.44:0/3891284813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2701e40 0x55a0f2880870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:40:59.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.182+0000 7f6822a00700 1 --2- 172.21.15.44:0/3891284813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2880330 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f68140a5b90 tx=0x7f68140b2400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f377202e560ecba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:59.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.182+0000 7f6823400700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68080b29f0 0x7f68080b4f00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f681c009fd0 tx=0x7f681c009510 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:59.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.182+0000 7f680fe00700 1 -- 172.21.15.44:0/3891284813 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68140adec0 con 0x55a0f27ed9f0 2024-05-27T09:40:59.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.182+0000 7f680fe00700 1 -- 172.21.15.44:0/3891284813 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68140b2970 con 0x55a0f27ed9f0 2024-05-27T09:40:59.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.182+0000 7f680fe00700 1 -- 172.21.15.44:0/3891284813 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68140b24e0 con 0x55a0f27ed9f0 2024-05-27T09:40:59.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.183+0000 7f6838191b80 1 -- 172.21.15.44:0/3891284813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 msgr2=0x55a0f2880330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:59.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.183+0000 7f6838191b80 1 --2- 172.21.15.44:0/3891284813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2880330 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f68140a5b90 tx=0x7f68140b2400 comp rx=0 tx=0).stop 2024-05-27T09:40:59.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.183+0000 7f6838191b80 1 -- 172.21.15.44:0/3891284813 shutdown_connections 2024-05-27T09:40:59.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.183+0000 7f6838191b80 1 --2- 172.21.15.44:0/3891284813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2880e70 0x55a0f288d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.183+0000 7f6838191b80 1 --2- 172.21.15.44:0/3891284813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2880330 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.183+0000 7f6838191b80 1 --2- 172.21.15.44:0/3891284813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2701e40 0x55a0f2880870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.183+0000 7f6838191b80 1 -- 172.21.15.44:0/3891284813 >> 172.21.15.44:0/3891284813 conn(0x55a0f279b390 msgr2=0x55a0f287c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:59.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.183+0000 7f6838191b80 1 -- 172.21.15.44:0/3891284813 shutdown_connections 2024-05-27T09:40:59.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.184+0000 7f6838191b80 1 -- 172.21.15.44:0/3891284813 wait complete. 2024-05-27T09:40:59.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.184+0000 7f6838191b80 1 -- start start 2024-05-27T09:40:59.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.185+0000 7f6838191b80 1 Processor -- start 2024-05-27T09:40:59.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.185+0000 7f6838191b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2701e40 0x55a0f288a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.185+0000 7f6838191b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f27ed9f0 0x55a0f288ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.185+0000 7f6838191b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2880e70 0x55a0f288b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.185+0000 7f6838191b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a0f26a4fa0 con 0x55a0f2701e40 2024-05-27T09:40:59.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.185+0000 7f6838191b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a0f273f760 con 0x55a0f2880e70 2024-05-27T09:40:59.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.185+0000 7f6838191b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a0f288db80 con 0x55a0f27ed9f0 2024-05-27T09:40:59.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.185+0000 7f6838191b80 10 client.0 _ll_register_callbacks cb 0x55a0f2748c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.186+0000 7f6822000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f27ed9f0 0x55a0f288ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.186+0000 7f6822000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f27ed9f0 0x55a0f288ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38224/0 (socket says 172.21.15.44:38224) 2024-05-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.186+0000 7f6822000700 1 -- 172.21.15.44:0/3463222520 learned_addr learned my addr 172.21.15.44:0/3463222520 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.186+0000 7f6823400700 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2880e70 0x55a0f288b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.186+0000 7f6822a00700 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2701e40 0x55a0f288a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.186+0000 7f6822000700 1 -- 172.21.15.44:0/3463222520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2880e70 msgr2=0x55a0f288b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.186+0000 7f6822000700 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2880e70 0x55a0f288b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.186+0000 7f6822000700 1 -- 172.21.15.44:0/3463222520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2701e40 msgr2=0x55a0f288a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.186+0000 7f6822000700 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2701e40 0x55a0f288a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.186+0000 7f6822000700 1 -- 172.21.15.44:0/3463222520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0f26ee8d0 con 0x55a0f27ed9f0 2024-05-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.186+0000 7f6822a00700 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2701e40 0x55a0f288a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:40:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.187+0000 7f6822000700 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f27ed9f0 0x55a0f288ac20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f68100116d0 tx=0x7f68100119e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b189bb8eccf191c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.187+0000 7f680cc00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:40:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.187+0000 7f680cc00700 1 -- 172.21.15.44:0/3463222520 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6810004830 con 0x55a0f27ed9f0 2024-05-27T09:40:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.187+0000 7f6838191b80 1 -- 172.21.15.44:0/3463222520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68140b2970 con 0x55a0f27ed9f0 2024-05-27T09:40:59.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.187+0000 7f680cc00700 1 -- 172.21.15.44:0/3463222520 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f68100049d0 con 0x55a0f27ed9f0 2024-05-27T09:40:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.187+0000 7f680cc00700 1 -- 172.21.15.44:0/3463222520 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6810004bd0 con 0x55a0f27ed9f0 2024-05-27T09:40:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.188+0000 7f6838191b80 10 client.26179 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.188+0000 7f6838191b80 1 -- 172.21.15.44:0/3463222520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a0f276f060 con 0x55a0f27ed9f0 2024-05-27T09:40:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.188+0000 7f680cc00700 1 -- 172.21.15.44:0/3463222520 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68140b2970 con 0x55a0f27ed9f0 2024-05-27T09:40:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.188+0000 7f680cc00700 1 -- 172.21.15.44:0/3463222520 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6810019070 con 0x55a0f27ed9f0 2024-05-27T09:40:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.188+0000 7f6838191b80 10 client.26179 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.188+0000 7f6838191b80 10 client.26179 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:40:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.188+0000 7f6838191b80 20 client.26179 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.188+0000 7f6838191b80 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55a0f289ab60 0x55a0f289cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.188+0000 7f6838191b80 4 client.26179 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:40:59.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.188+0000 7f6838191b80 1 -- 172.21.15.44:0/3463222520 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0f27b0d90 con 0x55a0f289ab60 2024-05-27T09:40:59.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.189+0000 7f6822a00700 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55a0f289ab60 0x55a0f289cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.189+0000 7f6822a00700 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55a0f289ab60 0x55a0f289cfe0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=218f934cabe15f56 server_cookie=3d75bc0c034341cf in_seq=0 out_seq=0 2024-05-27T09:40:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.189+0000 7f680cc00700 10 client.26179 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:40:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.190+0000 7f680cc00700 1 -- 172.21.15.44:0/3463222520 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a0f27b0d90 con 0x55a0f289ab60 2024-05-27T09:40:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.190+0000 7f680cc00700 10 client.26179 handle_command_reply: tid=0 2024-05-27T09:40:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.190+0000 7f6838191b80 1 client.26179 shutdown 2024-05-27T09:40:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.190+0000 7f6838191b80 1 -- 172.21.15.44:0/3463222520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f27ed9f0 msgr2=0x55a0f288ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.190+0000 7f6838191b80 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f27ed9f0 0x55a0f288ac20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f68100116d0 tx=0x7f68100119e0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.191+0000 7f6838191b80 1 -- 172.21.15.44:0/3463222520 shutdown_connections 2024-05-27T09:40:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.191+0000 7f6838191b80 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2880e70 0x55a0f288b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.191+0000 7f6838191b80 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55a0f289ab60 0x55a0f289cfe0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.191+0000 7f6838191b80 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2701e40 0x55a0f288a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.191+0000 7f6838191b80 1 --2- 172.21.15.44:0/3463222520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f27ed9f0 0x55a0f288ac20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.191+0000 7f680cc00700 0 client.26179 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:40:59.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.191+0000 7f6838191b80 1 -- 172.21.15.44:0/3463222520 >> 172.21.15.44:0/3463222520 conn(0x55a0f279b390 msgr2=0x55a0f28803b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:59.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.191+0000 7f6838191b80 1 -- 172.21.15.44:0/3463222520 shutdown_connections 2024-05-27T09:40:59.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.191+0000 7f6838191b80 1 -- 172.21.15.44:0/3463222520 wait complete. 2024-05-27T09:40:59.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.191+0000 7f6838191b80 20 client.26179 trim_cache size 0 max 16384 2024-05-27T09:40:59.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.208+0000 7f6838191b80 1 Processor -- start 2024-05-27T09:40:59.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.208+0000 7f6838191b80 1 -- start start 2024-05-27T09:40:59.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.209+0000 7f6838191b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2701e40 0x55a0f2889a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.209+0000 7f6838191b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2896c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.209+0000 7f6838191b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2880e70 0x55a0f28a27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.209+0000 7f6838191b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f68241b3970 con 0x55a0f2880e70 2024-05-27T09:40:59.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.209+0000 7f6838191b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f68241b1200 con 0x55a0f27ed9f0 2024-05-27T09:40:59.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.209+0000 7f6838191b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f68241af050 con 0x55a0f2701e40 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.209+0000 7f6822000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2896c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.209+0000 7f6822a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2701e40 0x55a0f2889a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.209+0000 7f6822000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2896c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34846/0 (socket says 172.21.15.44:34846) 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.209+0000 7f6822000700 1 -- 172.21.15.44:0/3797733890 learned_addr learned my addr 172.21.15.44:0/3797733890 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.209+0000 7f6823400700 1 --2- 172.21.15.44:0/3797733890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2880e70 0x55a0f28a27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.210+0000 7f6822000700 1 -- 172.21.15.44:0/3797733890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2701e40 msgr2=0x55a0f2889a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.210+0000 7f6822000700 1 --2- 172.21.15.44:0/3797733890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2701e40 0x55a0f2889a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.210+0000 7f6822000700 1 -- 172.21.15.44:0/3797733890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2880e70 msgr2=0x55a0f28a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.210+0000 7f6822000700 1 --2- 172.21.15.44:0/3797733890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2880e70 0x55a0f28a27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.210+0000 7f6822000700 1 -- 172.21.15.44:0/3797733890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0f2878240 con 0x55a0f27ed9f0 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.210+0000 7f6822000700 1 --2- 172.21.15.44:0/3797733890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2896c20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6810005520 tx=0x7f68100092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4638e092675d307a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:59.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.210+0000 7f680f400700 1 -- 172.21.15.44:0/3797733890 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6810031070 con 0x55a0f27ed9f0 2024-05-27T09:40:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.210+0000 7f680f400700 1 -- 172.21.15.44:0/3797733890 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6810004d70 con 0x55a0f27ed9f0 2024-05-27T09:40:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.210+0000 7f680f400700 1 -- 172.21.15.44:0/3797733890 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681002c440 con 0x55a0f27ed9f0 2024-05-27T09:40:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.211+0000 7f6838191b80 1 -- 172.21.15.44:0/3797733890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 msgr2=0x55a0f2896c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.211+0000 7f6838191b80 1 --2- 172.21.15.44:0/3797733890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2896c20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6810005520 tx=0x7f68100092d0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.211+0000 7f6838191b80 1 -- 172.21.15.44:0/3797733890 shutdown_connections 2024-05-27T09:40:59.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.211+0000 7f6838191b80 1 --2- 172.21.15.44:0/3797733890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f2880e70 0x55a0f28a27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.211+0000 7f6838191b80 1 --2- 172.21.15.44:0/3797733890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2701e40 0x55a0f2889a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.211+0000 7f6838191b80 1 --2- 172.21.15.44:0/3797733890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f27ed9f0 0x55a0f2896c20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.211+0000 7f6838191b80 1 -- 172.21.15.44:0/3797733890 >> 172.21.15.44:0/3797733890 conn(0x55a0f279b390 msgr2=0x55a0f2884550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:59.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.212+0000 7f6838191b80 1 -- 172.21.15.44:0/3797733890 shutdown_connections 2024-05-27T09:40:59.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.212+0000 7f6838191b80 1 -- 172.21.15.44:0/3797733890 wait complete. 2024-05-27T09:40:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.212+0000 7f6838191b80 1 -- start start 2024-05-27T09:40:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.212+0000 7f6838191b80 1 Processor -- start 2024-05-27T09:40:59.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.213+0000 7f6838191b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2701e40 0x55a0f2893530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.213+0000 7f6838191b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f27ed9f0 0x55a0f288e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.213+0000 7f6838191b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2880e70 0x55a0f288edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.213+0000 7f6838191b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f68241af6c0 con 0x55a0f27ed9f0 2024-05-27T09:40:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.213+0000 7f6838191b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f68241b0b30 con 0x55a0f2701e40 2024-05-27T09:40:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.213+0000 7f6838191b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f68241afd90 con 0x55a0f2880e70 2024-05-27T09:40:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.213+0000 7f6838191b80 10 client.0 _ll_register_callbacks cb 0x55a0f2748c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:40:59.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f6822a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2701e40 0x55a0f2893530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f6822a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2701e40 0x55a0f2893530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34862/0 (socket says 172.21.15.44:34862) 2024-05-27T09:40:59.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f6822a00700 1 -- 172.21.15.44:0/3263325393 learned_addr learned my addr 172.21.15.44:0/3263325393 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:40:59.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f6822000700 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f27ed9f0 0x55a0f288e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f6823400700 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2880e70 0x55a0f288edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f6822a00700 1 -- 172.21.15.44:0/3263325393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2880e70 msgr2=0x55a0f288edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:59.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f6822a00700 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2880e70 0x55a0f288edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f6822a00700 1 -- 172.21.15.44:0/3263325393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f27ed9f0 msgr2=0x55a0f288e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:40:59.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f6822a00700 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f27ed9f0 0x55a0f288e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f6822a00700 1 -- 172.21.15.44:0/3263325393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0f288f570 con 0x55a0f2701e40 2024-05-27T09:40:59.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f6822a00700 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2701e40 0x55a0f2893530 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f68140ae080 tx=0x7f68140b2360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7bd5d12035fe6dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:40:59.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.214+0000 7f680cc00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:40:59.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.215+0000 7f680cc00700 1 -- 172.21.15.44:0/3263325393 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68140b2970 con 0x55a0f2701e40 2024-05-27T09:40:59.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.215+0000 7f680cc00700 1 -- 172.21.15.44:0/3263325393 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f6814018b10 con 0x55a0f2701e40 2024-05-27T09:40:59.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.215+0000 7f6838191b80 1 -- 172.21.15.44:0/3263325393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6810004d70 con 0x55a0f2701e40 2024-05-27T09:40:59.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.215+0000 7f680cc00700 1 -- 172.21.15.44:0/3263325393 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f68140a9d70 con 0x55a0f2701e40 2024-05-27T09:40:59.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.215+0000 7f6838191b80 10 client.26174 fetch_fsmap learned FSMap version 26 2024-05-27T09:40:59.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.215+0000 7f6838191b80 1 -- 172.21.15.44:0/3263325393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a0f29055a0 con 0x55a0f2701e40 2024-05-27T09:40:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.215+0000 7f680cc00700 1 -- 172.21.15.44:0/3263325393 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6810004d70 con 0x55a0f2701e40 2024-05-27T09:40:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.216+0000 7f680cc00700 1 -- 172.21.15.44:0/3263325393 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6814070230 con 0x55a0f2701e40 2024-05-27T09:40:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.216+0000 7f6838191b80 10 client.26174 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:40:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.216+0000 7f6838191b80 10 client.26174 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:40:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.216+0000 7f6838191b80 20 client.26174 populate_metadata read hostname 'smithi044' 2024-05-27T09:40:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.216+0000 7f6838191b80 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55a0f2905830 0x55a0f2907c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:40:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.216+0000 7f6838191b80 4 client.26174 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:40:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.216+0000 7f6838191b80 1 -- 172.21.15.44:0/3263325393 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a0f27aa570 con 0x55a0f2905830 2024-05-27T09:40:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.216+0000 7f6822000700 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55a0f2905830 0x55a0f2907c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:40:59.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f6822000700 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55a0f2905830 0x55a0f2907c50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9c891cd0e963b386 server_cookie=6ee0a15cff82f5dc in_seq=0 out_seq=0 2024-05-27T09:40:59.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f680cc00700 10 client.26174 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:40:59.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f680cc00700 1 -- 172.21.15.44:0/3263325393 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a0f27aa570 con 0x55a0f2905830 2024-05-27T09:40:59.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f680cc00700 10 client.26174 handle_command_reply: tid=0 2024-05-27T09:40:59.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f6838191b80 1 client.26174 shutdown 2024-05-27T09:40:59.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f6838191b80 1 -- 172.21.15.44:0/3263325393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2701e40 msgr2=0x55a0f2893530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:59.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f6838191b80 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2701e40 0x55a0f2893530 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f68140ae080 tx=0x7f68140b2360 comp rx=0 tx=0).stop 2024-05-27T09:40:59.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f6838191b80 1 -- 172.21.15.44:0/3263325393 shutdown_connections 2024-05-27T09:40:59.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f6838191b80 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0f2880e70 0x55a0f288edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f6838191b80 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0f2701e40 0x55a0f2893530 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f6838191b80 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55a0f2905830 0x55a0f2907c50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f6838191b80 1 --2- 172.21.15.44:0/3263325393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a0f27ed9f0 0x55a0f288e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f6838191b80 1 -- 172.21.15.44:0/3263325393 >> 172.21.15.44:0/3263325393 conn(0x55a0f279b390 msgr2=0x55a0f2889760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:59.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.217+0000 7f680cc00700 0 client.26174 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:40:59.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.218+0000 7f6838191b80 1 -- 172.21.15.44:0/3263325393 shutdown_connections 2024-05-27T09:40:59.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.218+0000 7f6838191b80 1 -- 172.21.15.44:0/3263325393 wait complete. 2024-05-27T09:40:59.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.218+0000 7f6838191b80 20 client.26174 trim_cache size 0 max 16384 2024-05-27T09:40:59.224 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:40:59.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 -- 172.21.15.44:0/3854317130 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68080b29f0 msgr2=0x7f68080b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:59.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68080b29f0 0x7f68080b4f00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f681c009fd0 tx=0x7f681c009510 comp rx=0 tx=0).stop 2024-05-27T09:40:59.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 -- 172.21.15.44:0/3854317130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6824075d20 msgr2=0x7f682419d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:40:59.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6824075d20 0x7f682419d010 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f681400e9a0 tx=0x7f681400ecb0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 -- 172.21.15.44:0/3854317130 shutdown_connections 2024-05-27T09:40:59.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68080b29f0 0x7f68080b4f00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6824110b90 0x7f68241a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6824075d20 0x7f682419d010 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 --2- 172.21.15.44:0/3854317130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6824076670 0x7f682419d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:40:59.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 -- 172.21.15.44:0/3854317130 >> 172.21.15.44:0/3854317130 conn(0x7f6824100030 msgr2=0x7f6824100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:40:59.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 -- 172.21.15.44:0/3854317130 shutdown_connections 2024-05-27T09:40:59.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:40:59.219+0000 7f6829600700 1 -- 172.21.15.44:0/3854317130 wait complete. 2024-05-27T09:40:59.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:40:59 smithi044 ceph-mon[31618]: pgmap v175: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 13 op/s 2024-05-27T09:41:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:40:59 smithi050 ceph-mon[36371]: pgmap v175: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 13 op/s 2024-05-27T09:41:00.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:40:59 smithi189 ceph-mon[36671]: pgmap v175: 97 pgs: 97 active+clean; 4.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 13 op/s 2024-05-27T09:41:00.794 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:41:01.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.430+0000 7f727d400700 1 -- 172.21.15.44:0/530400993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7278075b50 msgr2=0x7f7278075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:01.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.430+0000 7f727d400700 1 --2- 172.21.15.44:0/530400993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7278075b50 0x7f7278075f60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f72680099c0 tx=0x7f7268009cd0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.430+0000 7f727d400700 1 -- 172.21.15.44:0/530400993 shutdown_connections 2024-05-27T09:41:01.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.430+0000 7f727d400700 1 --2- 172.21.15.44:0/530400993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72781153a0 0x7f72781177c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.430+0000 7f727d400700 1 --2- 172.21.15.44:0/530400993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7278075b50 0x7f7278075f60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.430+0000 7f727d400700 1 --2- 172.21.15.44:0/530400993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72780764a0 0x7f7278114e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.430+0000 7f727d400700 1 -- 172.21.15.44:0/530400993 >> 172.21.15.44:0/530400993 conn(0x7f7278100090 msgr2=0x7f72781024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:01.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.430+0000 7f727d400700 1 -- 172.21.15.44:0/530400993 shutdown_connections 2024-05-27T09:41:01.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.432+0000 7f727d400700 1 -- 172.21.15.44:0/530400993 wait complete. 2024-05-27T09:41:01.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.433+0000 7f727d400700 1 Processor -- start 2024-05-27T09:41:01.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.434+0000 7f727d400700 1 -- start start 2024-05-27T09:41:01.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.434+0000 7f727d400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7278075b50 0x7f72781a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.434+0000 7f727d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72780764a0 0x7f72781a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.434+0000 7f727d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72781153a0 0x7f72781a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.434+0000 7f727d400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f727811aa80 con 0x7f72780764a0 2024-05-27T09:41:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.434+0000 7f727d400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f727811a900 con 0x7f7278075b50 2024-05-27T09:41:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.434+0000 7f727d400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f727811ac00 con 0x7f72781153a0 2024-05-27T09:41:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.435+0000 7f7276800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7278075b50 0x7f72781a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.435+0000 7f7276800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7278075b50 0x7f72781a13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34870/0 (socket says 172.21.15.44:34870) 2024-05-27T09:41:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.435+0000 7f7276800700 1 -- 172.21.15.44:0/3548238574 learned_addr learned my addr 172.21.15.44:0/3548238574 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.435+0000 7f7277200700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72781153a0 0x7f72781a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.435+0000 7f7276800700 1 -- 172.21.15.44:0/3548238574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72781153a0 msgr2=0x7f72781a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.435+0000 7f7276800700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72781153a0 0x7f72781a5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.435+0000 7f7276800700 1 -- 172.21.15.44:0/3548238574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72780764a0 msgr2=0x7f72781a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.435+0000 7f7276800700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72780764a0 0x7f72781a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.435+0000 7f7276800700 1 -- 172.21.15.44:0/3548238574 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72781a62a0 con 0x7f7278075b50 2024-05-27T09:41:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.436+0000 7f7276800700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7278075b50 0x7f72781a13d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7268006010 tx=0x7f7268005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.436+0000 7f726f400700 1 -- 172.21.15.44:0/3548238574 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7268026070 con 0x7f7278075b50 2024-05-27T09:41:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.436+0000 7f727d400700 1 -- 172.21.15.44:0/3548238574 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72781a6530 con 0x7f7278075b50 2024-05-27T09:41:01.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.437+0000 7f726f400700 1 -- 172.21.15.44:0/3548238574 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7268003c70 con 0x7f7278075b50 2024-05-27T09:41:01.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.437+0000 7f727d400700 1 -- 172.21.15.44:0/3548238574 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72781adeb0 con 0x7f7278075b50 2024-05-27T09:41:01.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.437+0000 7f726f400700 1 -- 172.21.15.44:0/3548238574 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726800fc20 con 0x7f7278075b50 2024-05-27T09:41:01.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.440+0000 7f726f400700 1 -- 172.21.15.44:0/3548238574 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f7268029070 con 0x7f7278075b50 2024-05-27T09:41:01.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.442+0000 7f726f400700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f725c0b29f0 0x7f725c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.442+0000 7f726f400700 1 -- 172.21.15.44:0/3548238574 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f72680e2f60 con 0x7f7278075b50 2024-05-27T09:41:01.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.443+0000 7f7275e00700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f725c0b29f0 0x7f725c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.443+0000 7f728c00bb80 1 Processor -- start 2024-05-27T09:41:01.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.443+0000 7f7275e00700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f725c0b29f0 0x7f725c0b4f00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7264005950 tx=0x7f72640058e0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:01.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.443+0000 7f728c00bb80 1 -- start start 2024-05-27T09:41:01.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.444+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 0x55eccb8ba530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.444+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb734f40 0x55eccb8baa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.444+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb8b45e0 0x55eccb8b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.444+0000 7f728c00bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eccb7e1f40 con 0x55eccb734f40 2024-05-27T09:41:01.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.444+0000 7f728c00bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55eccb7d1ae0 con 0x55eccb820af0 2024-05-27T09:41:01.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.444+0000 7f728c00bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55eccb7d5b20 con 0x55eccb8b45e0 2024-05-27T09:41:01.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.445+0000 7f7276800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 0x55eccb8ba530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.445+0000 7f7275e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb734f40 0x55eccb8baa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.445+0000 7f7276800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 0x55eccb8ba530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34876/0 (socket says 172.21.15.44:34876) 2024-05-27T09:41:01.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.445+0000 7f7276800700 1 -- 172.21.15.44:0/907733219 learned_addr learned my addr 172.21.15.44:0/907733219 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:01.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.445+0000 7f7277200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb8b45e0 0x55eccb8b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.447 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:41:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.445+0000 7f7276800700 1 -- 172.21.15.44:0/907733219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb8b45e0 msgr2=0x55eccb8b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.445+0000 7f7276800700 1 --2- 172.21.15.44:0/907733219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb8b45e0 0x55eccb8b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.445+0000 7f7276800700 1 -- 172.21.15.44:0/907733219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb734f40 msgr2=0x55eccb8baa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.445+0000 7f7276800700 1 --2- 172.21.15.44:0/907733219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb734f40 0x55eccb8baa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.445+0000 7f7276800700 1 -- 172.21.15.44:0/907733219 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eccb79a520 con 0x55eccb820af0 2024-05-27T09:41:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.445+0000 7f7276800700 1 --2- 172.21.15.44:0/907733219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 0x55eccb8ba530 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f72680aa230 tx=0x7f7268009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=590318ad50874c8e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.446+0000 7f7263e00700 1 -- 172.21.15.44:0/907733219 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7268003e20 con 0x55eccb820af0 2024-05-27T09:41:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.446+0000 7f7263e00700 1 -- 172.21.15.44:0/907733219 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72680aa2a0 con 0x55eccb820af0 2024-05-27T09:41:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.446+0000 7f7263e00700 1 -- 172.21.15.44:0/907733219 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72680ae120 con 0x55eccb820af0 2024-05-27T09:41:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.447+0000 7f728c00bb80 1 -- 172.21.15.44:0/907733219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 msgr2=0x55eccb8ba530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.447+0000 7f728c00bb80 1 --2- 172.21.15.44:0/907733219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 0x55eccb8ba530 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f72680aa230 tx=0x7f7268009ee0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.447+0000 7f728c00bb80 1 -- 172.21.15.44:0/907733219 shutdown_connections 2024-05-27T09:41:01.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.447+0000 7f728c00bb80 1 --2- 172.21.15.44:0/907733219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb8b45e0 0x55eccb8b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.447+0000 7f728c00bb80 1 --2- 172.21.15.44:0/907733219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 0x55eccb8ba530 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.447+0000 7f728c00bb80 1 --2- 172.21.15.44:0/907733219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb734f40 0x55eccb8baa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.447+0000 7f728c00bb80 1 -- 172.21.15.44:0/907733219 >> 172.21.15.44:0/907733219 conn(0x55eccb7ce490 msgr2=0x55eccb8af650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:01.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.447+0000 7f728c00bb80 1 -- 172.21.15.44:0/907733219 shutdown_connections 2024-05-27T09:41:01.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.447+0000 7f728c00bb80 1 -- 172.21.15.44:0/907733219 wait complete. 2024-05-27T09:41:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.448+0000 7f728c00bb80 1 -- start start 2024-05-27T09:41:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.448+0000 7f728c00bb80 1 Processor -- start 2024-05-27T09:41:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.448+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb734f40 0x55eccb8c3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.448+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb820af0 0x55eccb8c4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 0x55eccb8be120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f728c00bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eccb6d80a0 con 0x55eccb8b45e0 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f728c00bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55eccb772860 con 0x55eccb734f40 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f728c00bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55eccb8b5190 con 0x55eccb820af0 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f728c00bb80 10 client.0 _ll_register_callbacks cb 0x55eccb77bd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f7277200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 0x55eccb8be120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f7275e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb820af0 0x55eccb8c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f7275e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb820af0 0x55eccb8c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38268/0 (socket says 172.21.15.44:38268) 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f7277200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 0x55eccb8be120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52962/0 (socket says 172.21.15.44:52962) 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f7275e00700 1 -- 172.21.15.44:0/1260058520 learned_addr learned my addr 172.21.15.44:0/1260058520 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f7276800700 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb734f40 0x55eccb8c3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f7275e00700 1 -- 172.21.15.44:0/1260058520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb734f40 msgr2=0x55eccb8c3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.449+0000 7f7275e00700 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb734f40 0x55eccb8c3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.450+0000 7f7275e00700 1 -- 172.21.15.44:0/1260058520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 msgr2=0x55eccb8be120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.450+0000 7f7275e00700 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 0x55eccb8be120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.450+0000 7f7275e00700 1 -- 172.21.15.44:0/1260058520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eccb7219d0 con 0x55eccb820af0 2024-05-27T09:41:01.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.450+0000 7f7277200700 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 0x55eccb8be120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.450+0000 7f7275e00700 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb820af0 0x55eccb8c4260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f726401d6e0 tx=0x7f7264004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1ab4498bf9f86c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.450+0000 7f7260c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:41:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.450+0000 7f7260c00700 1 -- 172.21.15.44:0/1260058520 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7264004a30 con 0x55eccb820af0 2024-05-27T09:41:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.451+0000 7f7260c00700 1 -- 172.21.15.44:0/1260058520 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f7264004bd0 con 0x55eccb820af0 2024-05-27T09:41:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.451+0000 7f728c00bb80 1 -- 172.21.15.44:0/1260058520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72680aa2a0 con 0x55eccb820af0 2024-05-27T09:41:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.451+0000 7f7260c00700 1 -- 172.21.15.44:0/1260058520 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7264004dd0 con 0x55eccb820af0 2024-05-27T09:41:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.451+0000 7f728c00bb80 10 client.26197 fetch_fsmap learned FSMap version 26 2024-05-27T09:41:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.451+0000 7f728c00bb80 1 -- 172.21.15.44:0/1260058520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55eccb7a2160 con 0x55eccb820af0 2024-05-27T09:41:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.451+0000 7f7260c00700 1 -- 172.21.15.44:0/1260058520 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72680aa2a0 con 0x55eccb820af0 2024-05-27T09:41:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.451+0000 7f7260c00700 1 -- 172.21.15.44:0/1260058520 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7264024030 con 0x55eccb820af0 2024-05-27T09:41:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.452+0000 7f728c00bb80 10 client.26197 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:41:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.452+0000 7f728c00bb80 10 client.26197 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:41:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.452+0000 7f728c00bb80 20 client.26197 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.452+0000 7f728c00bb80 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eccb8cde60 0x55eccb8d0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.452+0000 7f728c00bb80 4 client.26197 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.452+0000 7f728c00bb80 1 -- 172.21.15.44:0/1260058520 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eccb7e3e90 con 0x55eccb8cde60 2024-05-27T09:41:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.452+0000 7f7276800700 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eccb8cde60 0x55eccb8d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.453+0000 7f7276800700 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eccb8cde60 0x55eccb8d0280 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c70b2dc5c1aaa83f server_cookie=bc70644c7bd2f218 in_seq=0 out_seq=0 2024-05-27T09:41:01.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.453+0000 7f7260c00700 10 client.26197 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:01.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.454+0000 7f7260c00700 1 -- 172.21.15.44:0/1260058520 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55eccb7e3e90 con 0x55eccb8cde60 2024-05-27T09:41:01.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.454+0000 7f7260c00700 10 client.26197 handle_command_reply: tid=0 2024-05-27T09:41:01.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.454+0000 7f728c00bb80 1 client.26197 shutdown 2024-05-27T09:41:01.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.456+0000 7f728c00bb80 1 -- 172.21.15.44:0/1260058520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb820af0 msgr2=0x55eccb8c4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:01.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.456+0000 7f728c00bb80 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb820af0 0x55eccb8c4260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f726401d6e0 tx=0x7f7264004750 comp rx=0 tx=0).stop 2024-05-27T09:41:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.456+0000 7f728c00bb80 1 -- 172.21.15.44:0/1260058520 shutdown_connections 2024-05-27T09:41:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.456+0000 7f728c00bb80 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eccb8cde60 0x55eccb8d0280 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.456+0000 7f728c00bb80 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 0x55eccb8be120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.456+0000 7f728c00bb80 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb734f40 0x55eccb8c3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.456+0000 7f728c00bb80 1 --2- 172.21.15.44:0/1260058520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb820af0 0x55eccb8c4260 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.456+0000 7f728c00bb80 1 -- 172.21.15.44:0/1260058520 >> 172.21.15.44:0/1260058520 conn(0x55eccb7ce490 msgr2=0x55eccb8ae9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:01.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.456+0000 7f7260c00700 0 client.26197 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:01.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.457+0000 7f728c00bb80 1 -- 172.21.15.44:0/1260058520 shutdown_connections 2024-05-27T09:41:01.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.457+0000 7f728c00bb80 1 -- 172.21.15.44:0/1260058520 wait complete. 2024-05-27T09:41:01.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.457+0000 7f728c00bb80 20 client.26197 trim_cache size 0 max 16384 2024-05-27T09:41:01.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.479+0000 7f728c00bb80 1 Processor -- start 2024-05-27T09:41:01.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.479+0000 7f728c00bb80 1 -- start start 2024-05-27T09:41:01.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.480+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb734f40 0x55eccb8b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.480+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb820af0 0x55eccb8bb640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.480+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb8b45e0 0x55eccb8bbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.480+0000 7f728c00bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f725800fd70 con 0x55eccb820af0 2024-05-27T09:41:01.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.480+0000 7f728c00bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f725800d600 con 0x55eccb734f40 2024-05-27T09:41:01.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.480+0000 7f728c00bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f725800b450 con 0x55eccb8b45e0 2024-05-27T09:41:01.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.480+0000 7f7275e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb820af0 0x55eccb8bb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.480+0000 7f7277200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb8b45e0 0x55eccb8bbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.481+0000 7f7277200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb8b45e0 0x55eccb8bbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38278/0 (socket says 172.21.15.44:38278) 2024-05-27T09:41:01.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.481+0000 7f7277200700 1 -- 172.21.15.44:0/405335006 learned_addr learned my addr 172.21.15.44:0/405335006 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:01.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.481+0000 7f7276800700 1 --2- 172.21.15.44:0/405335006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb734f40 0x55eccb8b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.481+0000 7f7276800700 1 -- 172.21.15.44:0/405335006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb8b45e0 msgr2=0x55eccb8bbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:01.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.481+0000 7f7276800700 1 --2- 172.21.15.44:0/405335006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb8b45e0 0x55eccb8bbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.481+0000 7f7276800700 1 -- 172.21.15.44:0/405335006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb820af0 msgr2=0x55eccb8bb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:01.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.481+0000 7f7276800700 1 --2- 172.21.15.44:0/405335006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb820af0 0x55eccb8bb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.481+0000 7f7276800700 1 -- 172.21.15.44:0/405335006 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eccb8c45b0 con 0x55eccb734f40 2024-05-27T09:41:01.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.481+0000 7f7275e00700 1 --2- 172.21.15.44:0/405335006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb820af0 0x55eccb8bb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:01.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.481+0000 7f7276800700 1 --2- 172.21.15.44:0/405335006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb734f40 0x55eccb8b3ce0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7268000c00 tx=0x7f72680092b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65f7b9b6bda5c317 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:01.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.482+0000 7f7263400700 1 -- 172.21.15.44:0/405335006 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7268003e20 con 0x55eccb734f40 2024-05-27T09:41:01.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.482+0000 7f7263400700 1 -- 172.21.15.44:0/405335006 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72680ae650 con 0x55eccb734f40 2024-05-27T09:41:01.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.482+0000 7f7263400700 1 -- 172.21.15.44:0/405335006 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72680ae120 con 0x55eccb734f40 2024-05-27T09:41:01.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.483+0000 7f728c00bb80 1 -- 172.21.15.44:0/405335006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb734f40 msgr2=0x55eccb8b3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:01.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.483+0000 7f728c00bb80 1 --2- 172.21.15.44:0/405335006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb734f40 0x55eccb8b3ce0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7268000c00 tx=0x7f72680092b0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.483+0000 7f728c00bb80 1 -- 172.21.15.44:0/405335006 shutdown_connections 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.483+0000 7f728c00bb80 1 --2- 172.21.15.44:0/405335006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb8b45e0 0x55eccb8bbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.483+0000 7f728c00bb80 1 --2- 172.21.15.44:0/405335006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb734f40 0x55eccb8b3ce0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.483+0000 7f728c00bb80 1 --2- 172.21.15.44:0/405335006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb820af0 0x55eccb8bb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.483+0000 7f728c00bb80 1 -- 172.21.15.44:0/405335006 >> 172.21.15.44:0/405335006 conn(0x55eccb7ce490 msgr2=0x55eccb8b35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.483+0000 7f728c00bb80 1 -- 172.21.15.44:0/405335006 shutdown_connections 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.483+0000 7f728c00bb80 1 -- 172.21.15.44:0/405335006 wait complete. 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.484+0000 7f728c00bb80 1 -- start start 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.484+0000 7f728c00bb80 1 Processor -- start 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb734f40 0x55eccb8c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 0x55eccb8ca530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f728c00bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 0x55eccb8c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f728c00bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f725800bac0 con 0x55eccb8b45e0 2024-05-27T09:41:01.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f728c00bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f725800cf30 con 0x55eccb820af0 2024-05-27T09:41:01.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f728c00bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f725800c190 con 0x55eccb734f40 2024-05-27T09:41:01.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f728c00bb80 10 client.0 _ll_register_callbacks cb 0x55eccb77bd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:01.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f7276800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb734f40 0x55eccb8c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f7275e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 0x55eccb8ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f7276800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb734f40 0x55eccb8c9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38282/0 (socket says 172.21.15.44:38282) 2024-05-27T09:41:01.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f7276800700 1 -- 172.21.15.44:0/632101260 learned_addr learned my addr 172.21.15.44:0/632101260 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:01.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f7275e00700 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 0x55eccb8ca530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34914/0 (socket says 172.21.15.44:34914) 2024-05-27T09:41:01.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.485+0000 7f7277200700 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 0x55eccb8c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.486+0000 7f7276800700 1 -- 172.21.15.44:0/632101260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 msgr2=0x55eccb8ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:01.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.486+0000 7f7276800700 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 0x55eccb8ca530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.486+0000 7f7276800700 1 -- 172.21.15.44:0/632101260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 msgr2=0x55eccb8c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:01.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.486+0000 7f7276800700 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 0x55eccb8c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.486+0000 7f7276800700 1 -- 172.21.15.44:0/632101260 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eccb8c8c10 con 0x55eccb734f40 2024-05-27T09:41:01.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.486+0000 7f7277200700 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 0x55eccb8c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:01.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.486+0000 7f7276800700 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb734f40 0x55eccb8c9ff0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7268009f60 tx=0x7f72680a9db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a33cb81dc1c705a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:01.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.486+0000 7f7260c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:41:01.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.487+0000 7f7260c00700 1 -- 172.21.15.44:0/632101260 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72680aa710 con 0x55eccb734f40 2024-05-27T09:41:01.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.487+0000 7f728c00bb80 1 -- 172.21.15.44:0/632101260 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72680ae650 con 0x55eccb734f40 2024-05-27T09:41:01.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.487+0000 7f7260c00700 1 -- 172.21.15.44:0/632101260 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f72680b2be0 con 0x55eccb734f40 2024-05-27T09:41:01.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.487+0000 7f7260c00700 1 -- 172.21.15.44:0/632101260 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f72680b2e20 con 0x55eccb734f40 2024-05-27T09:41:01.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.487+0000 7f728c00bb80 10 client.26203 fetch_fsmap learned FSMap version 26 2024-05-27T09:41:01.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.487+0000 7f728c00bb80 1 -- 172.21.15.44:0/632101260 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55eccb939400 con 0x55eccb734f40 2024-05-27T09:41:01.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.488+0000 7f7260c00700 1 -- 172.21.15.44:0/632101260 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72680ae650 con 0x55eccb734f40 2024-05-27T09:41:01.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.488+0000 7f7260c00700 1 -- 172.21.15.44:0/632101260 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f72680c4b00 con 0x55eccb734f40 2024-05-27T09:41:01.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.488+0000 7f728c00bb80 10 client.26203 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:41:01.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.488+0000 7f728c00bb80 10 client.26203 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:41:01.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.488+0000 7f728c00bb80 20 client.26203 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:01.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.488+0000 7f728c00bb80 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eccb9396d0 0x55eccb93baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:01.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.488+0000 7f728c00bb80 4 client.26203 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "17b76354", "release": true, "await": true}] 2024-05-27T09:41:01.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.488+0000 7f728c00bb80 1 -- 172.21.15.44:0/632101260 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "set_id": "17b76354", "release": true, "await": true}) -- 0x55eccb8a7810 con 0x55eccb9396d0 2024-05-27T09:41:01.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.489+0000 7f7275e00700 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eccb9396d0 0x55eccb93baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:01.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.489+0000 7f7275e00700 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eccb9396d0 0x55eccb93baf0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55d362017a427791 server_cookie=7299509c94932088 in_seq=0 out_seq=0 2024-05-27T09:41:01.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.489+0000 7f7260c00700 10 client.26203 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:01.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.492+0000 7f7260c00700 1 -- 172.21.15.44:0/632101260 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55eccb8a7810 con 0x55eccb9396d0 2024-05-27T09:41:01.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.492+0000 7f7260c00700 10 client.26203 handle_command_reply: tid=0 2024-05-27T09:41:01.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.492+0000 7f728c00bb80 1 client.26203 shutdown 2024-05-27T09:41:01.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.492+0000 7f728c00bb80 1 -- 172.21.15.44:0/632101260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb734f40 msgr2=0x55eccb8c9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:01.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.492+0000 7f728c00bb80 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb734f40 0x55eccb8c9ff0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7268009f60 tx=0x7f72680a9db0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.492+0000 7f728c00bb80 1 -- 172.21.15.44:0/632101260 shutdown_connections 2024-05-27T09:41:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.492+0000 7f728c00bb80 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eccb9396d0 0x55eccb93baf0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.492+0000 7f728c00bb80 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eccb8b45e0 0x55eccb8c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.492+0000 7f7260c00700 0 client.26203 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.492+0000 7f728c00bb80 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eccb734f40 0x55eccb8c9ff0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.492+0000 7f728c00bb80 1 --2- 172.21.15.44:0/632101260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eccb820af0 0x55eccb8ca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.493+0000 7f728c00bb80 1 -- 172.21.15.44:0/632101260 >> 172.21.15.44:0/632101260 conn(0x55eccb7ce490 msgr2=0x55eccb8b8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.493+0000 7f728c00bb80 1 -- 172.21.15.44:0/632101260 shutdown_connections 2024-05-27T09:41:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.493+0000 7f728c00bb80 1 -- 172.21.15.44:0/632101260 wait complete. 2024-05-27T09:41:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.493+0000 7f728c00bb80 20 client.26203 trim_cache size 0 max 16384 2024-05-27T09:41:01.495 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:41:01.495 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 26, 2024-05-27T09:41:01.495 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:41:01.495 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 12, 2024-05-27T09:41:01.495 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:41:01.495 INFO:teuthology.orchestra.run.smithi044.stdout: "17b76354": { 2024-05-27T09:41:01.495 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 12, 2024-05-27T09:41:01.495 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:41:01.495 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:41:01.495 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 150.0, 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:01.496 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:01.497 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:01.497 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:01.497 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:41:01.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 -- 172.21.15.44:0/3548238574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f725c0b29f0 msgr2=0x7f725c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:01.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f725c0b29f0 0x7f725c0b4f00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7264005950 tx=0x7f72640058e0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 -- 172.21.15.44:0/3548238574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7278075b50 msgr2=0x7f72781a13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:01.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7278075b50 0x7f72781a13d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7268006010 tx=0x7f7268005280 comp rx=0 tx=0).stop 2024-05-27T09:41:01.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 -- 172.21.15.44:0/3548238574 shutdown_connections 2024-05-27T09:41:01.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f725c0b29f0 0x7f725c0b4f00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72781153a0 0x7f72781a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7278075b50 0x7f72781a13d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 --2- 172.21.15.44:0/3548238574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72780764a0 0x7f72781a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:01.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 -- 172.21.15.44:0/3548238574 >> 172.21.15.44:0/3548238574 conn(0x7f7278100090 msgr2=0x7f7278100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:01.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 -- 172.21.15.44:0/3548238574 shutdown_connections 2024-05-27T09:41:01.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:01.494+0000 7f727d400700 1 -- 172.21.15.44:0/3548238574 wait complete. 2024-05-27T09:41:01.576 INFO:tasks.workunit.client.0.smithi044.stdout:creating folder hierarchy 2024-05-27T09:41:02.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:01 smithi044 ceph-mon[31618]: pgmap v176: 97 pgs: 97 active+clean; 4.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 16 op/s 2024-05-27T09:41:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:01 smithi050 ceph-mon[36371]: pgmap v176: 97 pgs: 97 active+clean; 4.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 16 op/s 2024-05-27T09:41:02.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:01 smithi189 ceph-mon[36671]: pgmap v176: 97 pgs: 97 active+clean; 4.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 16 op/s 2024-05-27T09:41:02.601 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '17b76354', total seconds elapsed: 66.1 2024-05-27T09:41:02.601 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-05-27T09:41:02.601 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:41:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.527+0000 7ff64f200640 1 -- 172.21.15.44:0/538045965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff650112100 msgr2=0x7ff650114520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.527+0000 7ff64f200640 1 --2- 172.21.15.44:0/538045965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff650112100 0x7ff650114520 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009900 tx=0x7ff63c02a410 comp rx=0 tx=0).stop 2024-05-27T09:41:03.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.529+0000 7ff64f200640 1 -- 172.21.15.44:0/538045965 shutdown_connections 2024-05-27T09:41:03.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.529+0000 7ff64f200640 1 --2- 172.21.15.44:0/538045965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff650114a60 0x7ff650116e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.529+0000 7ff64f200640 1 --2- 172.21.15.44:0/538045965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff650112100 0x7ff650114520 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.529+0000 7ff64f200640 1 --2- 172.21.15.44:0/538045965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6500765b0 0x7ff6500769c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.529+0000 7ff64f200640 1 -- 172.21.15.44:0/538045965 >> 172.21.15.44:0/538045965 conn(0x7ff6500791a0 msgr2=0x7ff6500795e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.529+0000 7ff64f200640 1 -- 172.21.15.44:0/538045965 shutdown_connections 2024-05-27T09:41:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.529+0000 7ff64f200640 1 -- 172.21.15.44:0/538045965 wait complete. 2024-05-27T09:41:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.530+0000 7ff64f200640 1 Processor -- start 2024-05-27T09:41:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.530+0000 7ff64f200640 1 -- start start 2024-05-27T09:41:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.531+0000 7ff64f200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6500765b0 0x7ff6501a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.531+0000 7ff64f200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff650112100 0x7ff6501a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.531+0000 7ff64f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff650114a60 0x7ff6501a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.531+0000 7ff64f200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff65011a410 con 0x7ff650112100 2024-05-27T09:41:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.531+0000 7ff64f200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff65011a290 con 0x7ff6500765b0 2024-05-27T09:41:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.531+0000 7ff64f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff65011a590 con 0x7ff650114a60 2024-05-27T09:41:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.531+0000 7ff64de00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6500765b0 0x7ff6501a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.531+0000 7ff64de00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6500765b0 0x7ff6501a0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51574/0 (socket says 172.21.15.44:51574) 2024-05-27T09:41:03.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.531+0000 7ff64de00640 1 -- 172.21.15.44:0/442232683 learned_addr learned my addr 172.21.15.44:0/442232683 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:03.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.532+0000 7ff64d400640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff650112100 0x7ff6501a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.532+0000 7ff64d400640 1 -- 172.21.15.44:0/442232683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff650114a60 msgr2=0x7ff6501a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.532+0000 7ff64d400640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff650114a60 0x7ff6501a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.532+0000 7ff64d400640 1 -- 172.21.15.44:0/442232683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6500765b0 msgr2=0x7ff6501a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.532+0000 7ff64d400640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6500765b0 0x7ff6501a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.532+0000 7ff64d400640 1 -- 172.21.15.44:0/442232683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6501a5cb0 con 0x7ff650112100 2024-05-27T09:41:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.533+0000 7ff64d400640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff650112100 0x7ff6501a13b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff63c02a930 tx=0x7ff63c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.533+0000 7ff646a00640 1 -- 172.21.15.44:0/442232683 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c03f070 con 0x7ff650112100 2024-05-27T09:41:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.533+0000 7ff646a00640 1 -- 172.21.15.44:0/442232683 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff63c03bb10 con 0x7ff650112100 2024-05-27T09:41:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.533+0000 7ff64f200640 1 -- 172.21.15.44:0/442232683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6501a5f40 con 0x7ff650112100 2024-05-27T09:41:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.533+0000 7ff646a00640 1 -- 172.21.15.44:0/442232683 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c035030 con 0x7ff650112100 2024-05-27T09:41:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.533+0000 7ff64f200640 1 -- 172.21.15.44:0/442232683 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6501a6480 con 0x7ff650112100 2024-05-27T09:41:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.537+0000 7ff646a00640 1 -- 172.21.15.44:0/442232683 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff63c042070 con 0x7ff650112100 2024-05-27T09:41:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.538+0000 7ff65bc34740 1 Processor -- start 2024-05-27T09:41:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.538+0000 7ff65bc34740 1 -- start start 2024-05-27T09:41:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.538+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d681b0 0x5624d810aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.538+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 0x5624d810afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.538+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 0x5624d810b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.538+0000 7ff65bc34740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5624d807bb90 con 0x5624d7d681b0 2024-05-27T09:41:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.538+0000 7ff65bc34740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5624d806b730 con 0x5624d7d474d0 2024-05-27T09:41:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.538+0000 7ff65bc34740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5624d806f770 con 0x5624d7d3f2f0 2024-05-27T09:41:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.539+0000 7ff64d400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 0x5624d810b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.539+0000 7ff64e800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d681b0 0x5624d810aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.539+0000 7ff64d400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 0x5624d810b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51578/0 (socket says 172.21.15.44:51578) 2024-05-27T09:41:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.539+0000 7ff64d400640 1 -- 172.21.15.44:0/2828225899 learned_addr learned my addr 172.21.15.44:0/2828225899 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.539+0000 7ff64de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 0x5624d810afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.539+0000 7ff64e800640 1 -- 172.21.15.44:0/2828225899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 msgr2=0x5624d810afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.540+0000 7ff64e800640 1 --2- 172.21.15.44:0/2828225899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 0x5624d810afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.540+0000 7ff64e800640 1 -- 172.21.15.44:0/2828225899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 msgr2=0x5624d810b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.540+0000 7ff64e800640 1 --2- 172.21.15.44:0/2828225899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 0x5624d810b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.540+0000 7ff64e800640 1 -- 172.21.15.44:0/2828225899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d80048b0 con 0x5624d7d681b0 2024-05-27T09:41:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.540+0000 7ff64de00640 1 --2- 172.21.15.44:0/2828225899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 0x5624d810afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.540+0000 7ff646a00640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6340b2a50 0x7ff6340b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.540+0000 7ff64e800640 1 --2- 172.21.15.44:0/2828225899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d681b0 0x5624d810aa60 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff648005fd0 tx=0x7ff64800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6868539e10f3b903 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.541+0000 7ff646a00640 1 -- 172.21.15.44:0/442232683 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff63c102c30 con 0x7ff650112100 2024-05-27T09:41:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.541+0000 7ff64de00640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6340b2a50 0x7ff6340b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.541+0000 7ff63b000640 1 -- 172.21.15.44:0/2828225899 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648004b20 con 0x5624d7d681b0 2024-05-27T09:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.541+0000 7ff63b000640 1 -- 172.21.15.44:0/2828225899 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff648003e20 con 0x5624d7d681b0 2024-05-27T09:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.541+0000 7ff63b000640 1 -- 172.21.15.44:0/2828225899 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6480047b0 con 0x5624d7d681b0 2024-05-27T09:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.542+0000 7ff65bc34740 1 -- 172.21.15.44:0/2828225899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d681b0 msgr2=0x5624d810aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.542+0000 7ff64de00640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6340b2a50 0x7ff6340b4f60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x5624d810be10 tx=0x7ff640005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.542+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2828225899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d681b0 0x5624d810aa60 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff648005fd0 tx=0x7ff64800e720 comp rx=0 tx=0).stop 2024-05-27T09:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.542+0000 7ff65bc34740 1 -- 172.21.15.44:0/2828225899 shutdown_connections 2024-05-27T09:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.542+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2828225899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 0x5624d810b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.542+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2828225899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 0x5624d810afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.542+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2828225899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d681b0 0x5624d810aa60 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.542+0000 7ff65bc34740 1 -- 172.21.15.44:0/2828225899 >> 172.21.15.44:0/2828225899 conn(0x5624d7d5c300 msgr2=0x5624d8107210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.543+0000 7ff65bc34740 1 -- 172.21.15.44:0/2828225899 shutdown_connections 2024-05-27T09:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.543+0000 7ff65bc34740 1 -- 172.21.15.44:0/2828225899 wait complete. 2024-05-27T09:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.543+0000 7ff65bc34740 1 -- start start 2024-05-27T09:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.544+0000 7ff65bc34740 1 Processor -- start 2024-05-27T09:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.544+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 0x5624d8114be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.544+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d474d0 0x5624d8115120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.544+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d681b0 0x5624d8120ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.544+0000 7ff65bc34740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5624d7d51790 con 0x5624d7d474d0 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.544+0000 7ff65bc34740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5624d8013bc0 con 0x5624d7d681b0 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.544+0000 7ff65bc34740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5624d8117b90 con 0x5624d7d3f2f0 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff65bc34740 10 client.0 _ll_register_callbacks cb 0x5624d7ff1400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff64de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 0x5624d8114be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff64e800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d681b0 0x5624d8120ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff64de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 0x5624d8114be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39802/0 (socket says 172.21.15.44:39802) 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff64de00640 1 -- 172.21.15.44:0/2883155623 learned_addr learned my addr 172.21.15.44:0/2883155623 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff64d400640 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d474d0 0x5624d8115120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff64de00640 1 -- 172.21.15.44:0/2883155623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d681b0 msgr2=0x5624d8120ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff64de00640 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d681b0 0x5624d8120ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff64de00640 1 -- 172.21.15.44:0/2883155623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d474d0 msgr2=0x5624d8115120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff64de00640 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d474d0 0x5624d8115120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff64de00640 1 -- 172.21.15.44:0/2883155623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624d7d80210 con 0x5624d7d3f2f0 2024-05-27T09:41:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.545+0000 7ff64d400640 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d474d0 0x5624d8115120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:41:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.546+0000 7ff64de00640 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 0x5624d8114be0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff6400209e0 tx=0x7ff640004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2efc76e3c5d1bd67 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.546+0000 7ff62f400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:41:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.546+0000 7ff62f400640 1 -- 172.21.15.44:0/2883155623 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64002d060 con 0x5624d7d3f2f0 2024-05-27T09:41:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.546+0000 7ff62f400640 1 -- 172.21.15.44:0/2883155623 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff640018030 con 0x5624d7d3f2f0 2024-05-27T09:41:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.546+0000 7ff65bc34740 1 -- 172.21.15.44:0/2883155623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff648003e20 con 0x5624d7d3f2f0 2024-05-27T09:41:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.546+0000 7ff62f400640 1 -- 172.21.15.44:0/2883155623 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff6400098c0 con 0x5624d7d3f2f0 2024-05-27T09:41:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.546+0000 7ff65bc34740 10 client.26212 fetch_fsmap learned FSMap version 26 2024-05-27T09:41:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.546+0000 7ff65bc34740 1 -- 172.21.15.44:0/2883155623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5624d7e4c620 con 0x5624d7d3f2f0 2024-05-27T09:41:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.547+0000 7ff62f400640 1 -- 172.21.15.44:0/2883155623 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff648003e20 con 0x5624d7d3f2f0 2024-05-27T09:41:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.547+0000 7ff62f400640 1 -- 172.21.15.44:0/2883155623 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff64001dc70 con 0x5624d7d3f2f0 2024-05-27T09:41:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.547+0000 7ff65bc34740 10 client.26212 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:41:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.547+0000 7ff65bc34740 10 client.26212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.547+0000 7ff65bc34740 20 client.26212 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.547+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624d7d26d80 0x5624d8126f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.547+0000 7ff65bc34740 4 client.26212 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.547+0000 7ff65bc34740 1 -- 172.21.15.44:0/2883155623 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624d807dae0 con 0x5624d7d26d80 2024-05-27T09:41:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.548+0000 7ff64d400640 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624d7d26d80 0x5624d8126f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.548+0000 7ff64d400640 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624d7d26d80 0x5624d8126f20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61fe67f582be5a3a server_cookie=3e60992ef65017de in_seq=0 out_seq=0 2024-05-27T09:41:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.548+0000 7ff62f400640 10 client.26212 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.549+0000 7ff62f400640 1 -- 172.21.15.44:0/2883155623 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5624d807dae0 con 0x5624d7d26d80 2024-05-27T09:41:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.549+0000 7ff62f400640 10 client.26212 handle_command_reply: tid=0 2024-05-27T09:41:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.549+0000 7ff65bc34740 1 client.26212 shutdown 2024-05-27T09:41:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff65bc34740 1 -- 172.21.15.44:0/2883155623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 msgr2=0x5624d8114be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 0x5624d8114be0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff6400209e0 tx=0x7ff640004730 comp rx=0 tx=0).stop 2024-05-27T09:41:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff65bc34740 1 -- 172.21.15.44:0/2883155623 shutdown_connections 2024-05-27T09:41:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624d7d26d80 0x5624d8126f20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d681b0 0x5624d8120ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d474d0 0x5624d8115120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2883155623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d3f2f0 0x5624d8114be0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff65bc34740 1 -- 172.21.15.44:0/2883155623 >> 172.21.15.44:0/2883155623 conn(0x5624d7d5c300 msgr2=0x5624d810a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff62f400640 0 client.26212 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff65bc34740 1 -- 172.21.15.44:0/2883155623 shutdown_connections 2024-05-27T09:41:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff65bc34740 1 -- 172.21.15.44:0/2883155623 wait complete. 2024-05-27T09:41:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.550+0000 7ff65bc34740 20 client.26212 trim_cache size 0 max 16384 2024-05-27T09:41:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff65bc34740 1 Processor -- start 2024-05-27T09:41:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff65bc34740 1 -- start start 2024-05-27T09:41:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 0x5624d8120c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 0x5624d8121170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d681b0 0x5624d812c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff65bc34740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501b8820 con 0x5624d7d3f2f0 2024-05-27T09:41:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff65bc34740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501b60f0 con 0x5624d7d474d0 2024-05-27T09:41:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff65bc34740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501b3f40 con 0x5624d7d681b0 2024-05-27T09:41:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff64d400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 0x5624d8121170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff64d400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 0x5624d8121170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51600/0 (socket says 172.21.15.44:51600) 2024-05-27T09:41:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff64d400640 1 -- 172.21.15.44:0/2552328960 learned_addr learned my addr 172.21.15.44:0/2552328960 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.568+0000 7ff64e800640 1 --2- 172.21.15.44:0/2552328960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d681b0 0x5624d812c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.569+0000 7ff64de00640 1 --2- 172.21.15.44:0/2552328960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 0x5624d8120c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.569+0000 7ff64d400640 1 -- 172.21.15.44:0/2552328960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d681b0 msgr2=0x5624d812c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.569+0000 7ff64d400640 1 --2- 172.21.15.44:0/2552328960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d681b0 0x5624d812c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.569+0000 7ff64d400640 1 -- 172.21.15.44:0/2552328960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 msgr2=0x5624d8120c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.569+0000 7ff64d400640 1 --2- 172.21.15.44:0/2552328960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 0x5624d8120c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.569+0000 7ff64d400640 1 -- 172.21.15.44:0/2552328960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d8110440 con 0x5624d7d474d0 2024-05-27T09:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.569+0000 7ff64de00640 1 --2- 172.21.15.44:0/2552328960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 0x5624d8120c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.569+0000 7ff64d400640 1 --2- 172.21.15.44:0/2552328960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 0x5624d8121170 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff63c000c00 tx=0x7ff63c005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=988f4fe4ef51af3a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.569+0000 7ff63a600640 1 -- 172.21.15.44:0/2552328960 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c049040 con 0x5624d7d474d0 2024-05-27T09:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.569+0000 7ff63a600640 1 -- 172.21.15.44:0/2552328960 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff63c0ca5e0 con 0x5624d7d474d0 2024-05-27T09:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.569+0000 7ff63a600640 1 -- 172.21.15.44:0/2552328960 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c031510 con 0x5624d7d474d0 2024-05-27T09:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.570+0000 7ff65bc34740 1 -- 172.21.15.44:0/2552328960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 msgr2=0x5624d8121170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.570+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2552328960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 0x5624d8121170 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff63c000c00 tx=0x7ff63c005de0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.570+0000 7ff65bc34740 1 -- 172.21.15.44:0/2552328960 shutdown_connections 2024-05-27T09:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.570+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2552328960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d681b0 0x5624d812c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.570+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2552328960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d474d0 0x5624d8121170 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.570+0000 7ff65bc34740 1 --2- 172.21.15.44:0/2552328960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 0x5624d8120c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.570+0000 7ff65bc34740 1 -- 172.21.15.44:0/2552328960 >> 172.21.15.44:0/2552328960 conn(0x5624d7d5c300 msgr2=0x5624d8114ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.571+0000 7ff65bc34740 1 -- 172.21.15.44:0/2552328960 shutdown_connections 2024-05-27T09:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.571+0000 7ff65bc34740 1 -- 172.21.15.44:0/2552328960 wait complete. 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.571+0000 7ff65bc34740 1 -- start start 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.571+0000 7ff65bc34740 1 Processor -- start 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 0x5624d811d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d474d0 0x5624d811d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff65bc34740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d681b0 0x5624d811a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff65bc34740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501b45b0 con 0x5624d7d3f2f0 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff65bc34740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501b5a20 con 0x5624d7d681b0 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff65bc34740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501b4c80 con 0x5624d7d474d0 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff65bc34740 10 client.0 _ll_register_callbacks cb 0x7ff6400208a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff64de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 0x5624d811d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff64d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d474d0 0x5624d811d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff64e800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d681b0 0x5624d811a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff64de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 0x5624d811d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32830/0 (socket says 172.21.15.44:32830) 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff64de00640 1 -- 172.21.15.44:0/3462733315 learned_addr learned my addr 172.21.15.44:0/3462733315 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff64d400640 1 -- 172.21.15.44:0/3462733315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d681b0 msgr2=0x5624d811a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff64d400640 1 --2- 172.21.15.44:0/3462733315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d681b0 0x5624d811a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff64d400640 1 -- 172.21.15.44:0/3462733315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 msgr2=0x5624d811d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff64d400640 1 --2- 172.21.15.44:0/3462733315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 0x5624d811d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.572+0000 7ff64d400640 1 -- 172.21.15.44:0/3462733315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624d811adc0 con 0x5624d7d474d0 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.573+0000 7ff64d400640 1 --2- 172.21.15.44:0/3462733315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d474d0 0x5624d811d6a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff63c000c00 tx=0x7ff63c003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=258c3b2c9cd18a2d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.573+0000 7ff62f400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.573+0000 7ff62f400640 1 -- 172.21.15.44:0/3462733315 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c049070 con 0x5624d7d474d0 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.573+0000 7ff62f400640 1 -- 172.21.15.44:0/3462733315 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff63c0c6170 con 0x5624d7d474d0 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.573+0000 7ff65bc34740 1 -- 172.21.15.44:0/3462733315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff63c0ca5e0 con 0x5624d7d474d0 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.573+0000 7ff62f400640 1 -- 172.21.15.44:0/3462733315 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff63c0c9ee0 con 0x5624d7d474d0 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.573+0000 7ff65bc34740 10 client.26218 fetch_fsmap learned FSMap version 26 2024-05-27T09:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.573+0000 7ff65bc34740 1 -- 172.21.15.44:0/3462733315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5624d818f0f0 con 0x5624d7d474d0 2024-05-27T09:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.573+0000 7ff62f400640 1 -- 172.21.15.44:0/3462733315 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff63c0ca5e0 con 0x5624d7d474d0 2024-05-27T09:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.574+0000 7ff62f400640 1 -- 172.21.15.44:0/3462733315 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff63c0d28b0 con 0x5624d7d474d0 2024-05-27T09:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.574+0000 7ff65bc34740 10 client.26218 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.574+0000 7ff65bc34740 10 client.26218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.574+0000 7ff65bc34740 20 client.26218 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.574+0000 7ff65bc34740 1 --2- 172.21.15.44:0/3462733315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624d7cfc1c0 0x5624d81917f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.574+0000 7ff65bc34740 4 client.26218 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.574+0000 7ff65bc34740 1 -- 172.21.15.44:0/3462733315 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5624d80772c0 con 0x5624d7cfc1c0 2024-05-27T09:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.574+0000 7ff64de00640 1 --2- 172.21.15.44:0/3462733315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624d7cfc1c0 0x5624d81917f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.575+0000 7ff64de00640 1 --2- 172.21.15.44:0/3462733315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624d7cfc1c0 0x5624d81917f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1223a30e7abde1f3 server_cookie=f7400ee340268804 in_seq=0 out_seq=0 2024-05-27T09:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:03.575+0000 7ff62f400640 10 client.26218 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:41:04.149 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:03 smithi044 ceph-mon[31618]: pgmap v177: 97 pgs: 97 active+clean; 4.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.1 KiB/s wr, 11 op/s 2024-05-27T09:41:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:03 smithi050 ceph-mon[36371]: pgmap v177: 97 pgs: 97 active+clean; 4.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.1 KiB/s wr, 11 op/s 2024-05-27T09:41:04.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:03 smithi189 ceph-mon[36671]: pgmap v177: 97 pgs: 97 active+clean; 4.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.1 KiB/s wr, 11 op/s 2024-05-27T09:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff62f400640 1 -- 172.21.15.44:0/3462733315 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5624d80772c0 con 0x5624d7cfc1c0 2024-05-27T09:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff62f400640 10 client.26218 handle_command_reply: tid=0 2024-05-27T09:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff65bc34740 1 client.26218 shutdown 2024-05-27T09:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff65bc34740 1 -- 172.21.15.44:0/3462733315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d474d0 msgr2=0x5624d811d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff65bc34740 1 --2- 172.21.15.44:0/3462733315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d474d0 0x5624d811d6a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff63c000c00 tx=0x7ff63c003f90 comp rx=0 tx=0).stop 2024-05-27T09:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff65bc34740 1 -- 172.21.15.44:0/3462733315 shutdown_connections 2024-05-27T09:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff65bc34740 1 --2- 172.21.15.44:0/3462733315 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624d7cfc1c0 0x5624d81917f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff65bc34740 1 --2- 172.21.15.44:0/3462733315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624d7d681b0 0x5624d811a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff65bc34740 1 --2- 172.21.15.44:0/3462733315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624d7d474d0 0x5624d811d6a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff65bc34740 1 --2- 172.21.15.44:0/3462733315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624d7d3f2f0 0x5624d811d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff65bc34740 1 -- 172.21.15.44:0/3462733315 >> 172.21.15.44:0/3462733315 conn(0x5624d7d5c300 msgr2=0x5624d8114ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff62f400640 0 client.26218 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff65bc34740 1 -- 172.21.15.44:0/3462733315 shutdown_connections 2024-05-27T09:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.843+0000 7ff65bc34740 1 -- 172.21.15.44:0/3462733315 wait complete. 2024-05-27T09:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.844+0000 7ff65bc34740 20 client.26218 trim_cache size 0 max 16384 2024-05-27T09:41:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:41:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:41:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:41:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.845+0000 7ff64f200640 1 -- 172.21.15.44:0/442232683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6340b2a50 msgr2=0x7ff6340b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.845+0000 7ff64f200640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6340b2a50 0x7ff6340b4f60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x5624d810be10 tx=0x7ff640005f90 comp rx=0 tx=0).stop 2024-05-27T09:41:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.845+0000 7ff64f200640 1 -- 172.21.15.44:0/442232683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff650112100 msgr2=0x7ff6501a13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.845+0000 7ff64f200640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff650112100 0x7ff6501a13b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff63c02a930 tx=0x7ff63c003b90 comp rx=0 tx=0).stop 2024-05-27T09:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.845+0000 7ff64f200640 1 -- 172.21.15.44:0/442232683 shutdown_connections 2024-05-27T09:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.845+0000 7ff64f200640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff650114a60 0x7ff6501a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.845+0000 7ff64f200640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6340b2a50 0x7ff6340b4f60 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x5624d810be10 tx=0x7ff640005f90 comp rx=0 tx=0).stop 2024-05-27T09:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.845+0000 7ff64f200640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff650112100 0x7ff6501a13b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.845+0000 7ff64f200640 1 --2- 172.21.15.44:0/442232683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6500765b0 0x7ff6501a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.845+0000 7ff64f200640 1 -- 172.21.15.44:0/442232683 >> 172.21.15.44:0/442232683 conn(0x7ff6500791a0 msgr2=0x7ff650112510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:04.848 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.847+0000 7ff64f200640 1 -- 172.21.15.44:0/442232683 shutdown_connections 2024-05-27T09:41:04.848 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:04.847+0000 7ff64f200640 1 -- 172.21.15.44:0/442232683 wait complete. 2024-05-27T09:41:05.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.829+0000 7f2c4ba00700 1 -- 172.21.15.44:0/2617080352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c441045e0 msgr2=0x7f2c441049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.829+0000 7f2c4ba00700 1 --2- 172.21.15.44:0/2617080352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c441045e0 0x7f2c441049f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009990 tx=0x7f2c38009ca0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.829+0000 7f2c4ba00700 1 -- 172.21.15.44:0/2617080352 shutdown_connections 2024-05-27T09:41:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.829+0000 7f2c4ba00700 1 --2- 172.21.15.44:0/2617080352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c44109860 0x7f2c441111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.829+0000 7f2c4ba00700 1 --2- 172.21.15.44:0/2617080352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c441045e0 0x7f2c441049f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.829+0000 7f2c4ba00700 1 --2- 172.21.15.44:0/2617080352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44108f10 0x7f2c44109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.829+0000 7f2c4ba00700 1 -- 172.21.15.44:0/2617080352 >> 172.21.15.44:0/2617080352 conn(0x7f2c44100030 msgr2=0x7f2c44102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.829+0000 7f2c4ba00700 1 -- 172.21.15.44:0/2617080352 shutdown_connections 2024-05-27T09:41:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.830+0000 7f2c4ba00700 1 -- 172.21.15.44:0/2617080352 wait complete. 2024-05-27T09:41:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.831+0000 7f2c4ba00700 1 Processor -- start 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.831+0000 7f2c4ba00700 1 -- start start 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c4ba00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c441045e0 0x7f2c4419d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c4ba00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c44108f10 0x7f2c4419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c4ba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44109860 0x7f2c441a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c4ba00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2c441144e0 con 0x7f2c441045e0 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c4ba00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2c44114360 con 0x7f2c44108f10 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c4ba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2c44114660 con 0x7f2c44109860 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c49000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c441045e0 0x7f2c4419d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c43e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c44108f10 0x7f2c4419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c49000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c441045e0 0x7f2c4419d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32848/0 (socket says 172.21.15.44:32848) 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c49000700 1 -- 172.21.15.44:0/1055137605 learned_addr learned my addr 172.21.15.44:0/1055137605 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c49a00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44109860 0x7f2c441a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c43e00700 1 -- 172.21.15.44:0/1055137605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44109860 msgr2=0x7f2c441a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c43e00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44109860 0x7f2c441a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.832+0000 7f2c43e00700 1 -- 172.21.15.44:0/1055137605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c441045e0 msgr2=0x7f2c4419d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.833+0000 7f2c43e00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c441045e0 0x7f2c4419d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.833+0000 7f2c43e00700 1 -- 172.21.15.44:0/1055137605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c441a1fe0 con 0x7f2c44108f10 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.833+0000 7f2c49000700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c441045e0 0x7f2c4419d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.833+0000 7f2c43e00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c44108f10 0x7f2c4419d6b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000e9c0 tx=0x7f2c3000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.833+0000 7f2c41600700 1 -- 172.21.15.44:0/1055137605 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3000cb70 con 0x7f2c44108f10 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.833+0000 7f2c41600700 1 -- 172.21.15.44:0/1055137605 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c3000ee60 con 0x7f2c44108f10 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.833+0000 7f2c41600700 1 -- 172.21.15.44:0/1055137605 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c30003d60 con 0x7f2c44108f10 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.833+0000 7f2c4ba00700 1 -- 172.21.15.44:0/1055137605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c441a22d0 con 0x7f2c44108f10 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.833+0000 7f2c4ba00700 1 -- 172.21.15.44:0/1055137605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c441a9bf0 con 0x7f2c44108f10 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.837+0000 7f2c41600700 1 -- 172.21.15.44:0/1055137605 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2c30003f00 con 0x7f2c44108f10 2024-05-27T09:41:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.838+0000 7f2c5a51bb80 1 Processor -- start 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.838+0000 7f2c5a51bb80 1 -- start start 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.838+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c447a9f0 0x5627c450d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.838+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c438ee40 0x5627c450d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.838+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c450de70 0x5627c451a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.838+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5627c443be40 con 0x5627c438ee40 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.838+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5627c442b9e0 con 0x5627c447a9f0 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.838+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5627c442fa20 con 0x5627c450de70 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c49a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c447a9f0 0x5627c450d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c49a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c447a9f0 0x5627c450d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51626/0 (socket says 172.21.15.44:51626) 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c49a00700 1 -- 172.21.15.44:0/1990379872 learned_addr learned my addr 172.21.15.44:0/1990379872 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c41600700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c2c0b29f0 0x7f2c2c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c49000700 1 --2- 172.21.15.44:0/1990379872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c438ee40 0x5627c450d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c43e00700 1 --2- 172.21.15.44:0/1990379872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c450de70 0x5627c451a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c41600700 1 -- 172.21.15.44:0/1055137605 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f2c300e2740 con 0x7f2c44108f10 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c49a00700 1 -- 172.21.15.44:0/1990379872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c450de70 msgr2=0x5627c451a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c49a00700 1 --2- 172.21.15.44:0/1990379872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c450de70 0x5627c451a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c49a00700 1 -- 172.21.15.44:0/1990379872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c438ee40 msgr2=0x5627c450d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c49a00700 1 --2- 172.21.15.44:0/1990379872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c438ee40 0x5627c450d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c49a00700 1 -- 172.21.15.44:0/1990379872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627c43f4420 con 0x5627c447a9f0 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.839+0000 7f2c49000700 1 --2- 172.21.15.44:0/1990379872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c438ee40 0x5627c450d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.840+0000 7f2c49a00700 1 --2- 172.21.15.44:0/1990379872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c447a9f0 0x5627c450d330 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c011790 tx=0x7f2c3c011aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=661ab47bae150f2d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.840+0000 7f2c49a00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c2c0b29f0 0x7f2c2c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.840+0000 7f2c36000700 1 -- 172.21.15.44:0/1990379872 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c01b030 con 0x5627c447a9f0 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.840+0000 7f2c36000700 1 -- 172.21.15.44:0/1990379872 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c3c007bb0 con 0x5627c447a9f0 2024-05-27T09:41:05.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.840+0000 7f2c36000700 1 -- 172.21.15.44:0/1990379872 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c017e50 con 0x5627c447a9f0 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.841+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1990379872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c447a9f0 msgr2=0x5627c450d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.841+0000 7f2c49a00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c2c0b29f0 0x7f2c2c0b4f00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c012010 tx=0x7f2c3c0218c0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.841+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/1990379872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c447a9f0 0x5627c450d330 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c011790 tx=0x7f2c3c011aa0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.841+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1990379872 shutdown_connections 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.841+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/1990379872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c450de70 0x5627c451a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.841+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/1990379872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c447a9f0 0x5627c450d330 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.841+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/1990379872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c438ee40 0x5627c450d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.841+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1990379872 >> 172.21.15.44:0/1990379872 conn(0x5627c4428390 msgr2=0x5627c45094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.843+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1990379872 shutdown_connections 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.843+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1990379872 wait complete. 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.843+0000 7f2c5a51bb80 1 -- start start 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.844+0000 7f2c5a51bb80 1 Processor -- start 2024-05-27T09:41:05.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.845+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c45176a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.845+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c4517be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.845+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c4518120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.845+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5627c4331fa0 con 0x5627c447a9f0 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.845+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5627c43cc760 con 0x5627c450de70 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.845+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5627c451ab80 con 0x5627c438ee40 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.845+0000 7f2c5a51bb80 10 client.0 _ll_register_callbacks cb 0x5627c43d5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.846+0000 7f2c49a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c4518120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.846+0000 7f2c49a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c4518120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51634/0 (socket says 172.21.15.44:51634) 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.846+0000 7f2c49a00700 1 -- 172.21.15.44:0/1921127032 learned_addr learned my addr 172.21.15.44:0/1921127032 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.846+0000 7f2c49000700 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c45176a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.846+0000 7f2c49a00700 1 -- 172.21.15.44:0/1921127032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 msgr2=0x5627c45176a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.846+0000 7f2c49a00700 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c45176a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.846+0000 7f2c49a00700 1 -- 172.21.15.44:0/1921127032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 msgr2=0x5627c4517be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.846+0000 7f2c49a00700 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c4517be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.846+0000 7f2c49a00700 1 -- 172.21.15.44:0/1921127032 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627c437b8d0 con 0x5627c450de70 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.847+0000 7f2c49a00700 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c4518120 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0178b0 tx=0x7f2c3c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7301eb55b9fcc713 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.847+0000 7f2c2aa00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.847+0000 7f2c2aa00700 1 -- 172.21.15.44:0/1921127032 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c02d030 con 0x5627c450de70 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.847+0000 7f2c2aa00700 1 -- 172.21.15.44:0/1921127032 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f2c3c026070 con 0x5627c450de70 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.847+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1921127032 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c3c007bb0 con 0x5627c450de70 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.848+0000 7f2c2aa00700 1 -- 172.21.15.44:0/1921127032 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2c3c038050 con 0x5627c450de70 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.848+0000 7f2c5a51bb80 10 client.26222 fetch_fsmap learned FSMap version 26 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.848+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1921127032 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5627c43fc060 con 0x5627c450de70 2024-05-27T09:41:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.849+0000 7f2c2aa00700 1 -- 172.21.15.44:0/1921127032 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c3c007bb0 con 0x5627c450de70 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.849+0000 7f2c2aa00700 1 -- 172.21.15.44:0/1921127032 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2c3c029420 con 0x5627c450de70 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.849+0000 7f2c5a51bb80 10 client.26222 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.849+0000 7f2c5a51bb80 10 client.26222 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.849+0000 7f2c5a51bb80 20 client.26222 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.849+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x5627c4527a90 0x5627c4529eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.849+0000 7f2c5a51bb80 4 client.26222 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.849+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1921127032 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627c443dd90 con 0x5627c4527a90 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.850+0000 7f2c49000700 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x5627c4527a90 0x5627c4529eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.851+0000 7f2c49000700 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x5627c4527a90 0x5627c4529eb0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41c357f9f87d64ac server_cookie=97854db6034eff93 in_seq=0 out_seq=0 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.851+0000 7f2c2aa00700 10 client.26222 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.852+0000 7f2c2aa00700 1 -- 172.21.15.44:0/1921127032 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5627c443dd90 con 0x5627c4527a90 2024-05-27T09:41:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.852+0000 7f2c2aa00700 10 client.26222 handle_command_reply: tid=0 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.853+0000 7f2c5a51bb80 1 client.26222 shutdown 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.853+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1921127032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 msgr2=0x5627c4518120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.853+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c4518120 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0178b0 tx=0x7f2c3c01a000 comp rx=0 tx=0).stop 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.853+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1921127032 shutdown_connections 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.853+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c4518120 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.853+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c45176a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.853+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x5627c4527a90 0x5627c4529eb0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.853+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/1921127032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c4517be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.853+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1921127032 >> 172.21.15.44:0/1921127032 conn(0x5627c4428390 msgr2=0x5627c450d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.854+0000 7f2c2aa00700 0 client.26222 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.854+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1921127032 shutdown_connections 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.855+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/1921127032 wait complete. 2024-05-27T09:41:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.855+0000 7f2c5a51bb80 20 client.26222 trim_cache size 0 max 16384 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.876+0000 7f2c5a51bb80 1 Processor -- start 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.877+0000 7f2c5a51bb80 1 -- start start 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.877+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c4516600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.877+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c4512e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.877+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c450d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.877+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2c441b3ad0 con 0x5627c447a9f0 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.877+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2c441b1360 con 0x5627c450de70 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.877+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2c441af1b0 con 0x5627c438ee40 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.878+0000 7f2c49a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c450d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.878+0000 7f2c49a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c450d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51638/0 (socket says 172.21.15.44:51638) 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.878+0000 7f2c49a00700 1 -- 172.21.15.44:0/411243147 learned_addr learned my addr 172.21.15.44:0/411243147 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.878+0000 7f2c49000700 1 --2- 172.21.15.44:0/411243147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c4516600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.878+0000 7f2c43e00700 1 --2- 172.21.15.44:0/411243147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c4512e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.878+0000 7f2c49a00700 1 -- 172.21.15.44:0/411243147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 msgr2=0x5627c4516600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.878+0000 7f2c49a00700 1 --2- 172.21.15.44:0/411243147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c4516600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.878+0000 7f2c49a00700 1 -- 172.21.15.44:0/411243147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 msgr2=0x5627c4512e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.879+0000 7f2c49a00700 1 --2- 172.21.15.44:0/411243147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c4512e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.879+0000 7f2c49a00700 1 -- 172.21.15.44:0/411243147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627c4508b10 con 0x5627c450de70 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.879+0000 7f2c43e00700 1 --2- 172.21.15.44:0/411243147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c4512e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.879+0000 7f2c49a00700 1 --2- 172.21.15.44:0/411243147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c450d7c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0112e0 tx=0x7f2c3c011aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=472ace46f9ef451f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.880+0000 7f2c35600700 1 -- 172.21.15.44:0/411243147 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c02d070 con 0x5627c450de70 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.880+0000 7f2c35600700 1 -- 172.21.15.44:0/411243147 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c3c01c040 con 0x5627c450de70 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.880+0000 7f2c35600700 1 -- 172.21.15.44:0/411243147 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c01b070 con 0x5627c450de70 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.881+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/411243147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 msgr2=0x5627c450d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.881+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/411243147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c450d7c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0112e0 tx=0x7f2c3c011aa0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.882+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/411243147 shutdown_connections 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.882+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/411243147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c450d7c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.882+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/411243147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c4516600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.882+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/411243147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c4512e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.882+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/411243147 >> 172.21.15.44:0/411243147 conn(0x5627c4428390 msgr2=0x5627c450d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.882+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/411243147 shutdown_connections 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.882+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/411243147 wait complete. 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.883+0000 7f2c5a51bb80 1 -- start start 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.883+0000 7f2c5a51bb80 1 Processor -- start 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c451add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c451b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c5a51bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c451b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2c441af820 con 0x5627c447a9f0 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2c441b0c90 con 0x5627c450de70 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c5a51bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2c441afef0 con 0x5627c438ee40 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c49000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c451add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c49000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c451add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39850/0 (socket says 172.21.15.44:39850) 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c49000700 1 -- 172.21.15.44:0/3841819294 learned_addr learned my addr 172.21.15.44:0/3841819294 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c5a51bb80 10 client.0 _ll_register_callbacks cb 0x5627c43d5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c43e00700 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c451b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.884+0000 7f2c49a00700 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c451b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.885+0000 7f2c49000700 1 -- 172.21.15.44:0/3841819294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 msgr2=0x5627c451b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.885+0000 7f2c49000700 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c451b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.885+0000 7f2c49000700 1 -- 172.21.15.44:0/3841819294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 msgr2=0x5627c451b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.885+0000 7f2c49000700 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c451b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.885+0000 7f2c49000700 1 -- 172.21.15.44:0/3841819294 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627c4598be0 con 0x5627c438ee40 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.885+0000 7f2c43e00700 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c451b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.885+0000 7f2c49000700 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c451add0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2c380098e0 tx=0x7f2c3800bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=183744ef4840d639 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.885+0000 7f2c2aa00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.885+0000 7f2c2aa00700 1 -- 172.21.15.44:0/3841819294 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3802a020 con 0x5627c438ee40 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.885+0000 7f2c2aa00700 1 -- 172.21.15.44:0/3841819294 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f2c38005450 con 0x5627c438ee40 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.886+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/3841819294 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c3c01c040 con 0x5627c438ee40 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.886+0000 7f2c2aa00700 1 -- 172.21.15.44:0/3841819294 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2c38005650 con 0x5627c438ee40 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.886+0000 7f2c5a51bb80 10 client.26233 fetch_fsmap learned FSMap version 26 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.886+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/3841819294 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5627c4599da0 con 0x5627c438ee40 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.886+0000 7f2c2aa00700 1 -- 172.21.15.44:0/3841819294 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c3c01c040 con 0x5627c438ee40 2024-05-27T09:41:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.886+0000 7f2c2aa00700 1 -- 172.21.15.44:0/3841819294 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2c38037020 con 0x5627c438ee40 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.887+0000 7f2c5a51bb80 10 client.26233 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.887+0000 7f2c5a51bb80 10 client.26233 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.887+0000 7f2c5a51bb80 20 client.26233 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.887+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x5627c459a560 0x5627c459c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.887+0000 7f2c5a51bb80 4 client.26233 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.887+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/3841819294 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5627c4437570 con 0x5627c459a560 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.888+0000 7f2c43e00700 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x5627c459a560 0x5627c459c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.890+0000 7f2c43e00700 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x5627c459a560 0x5627c459c980 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bcc4b4e0b96e3adc server_cookie=b37c3e3af5c4f35a in_seq=0 out_seq=0 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.890+0000 7f2c2aa00700 10 client.26233 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.890+0000 7f2c2aa00700 1 -- 172.21.15.44:0/3841819294 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5627c4437570 con 0x5627c459a560 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.890+0000 7f2c2aa00700 10 client.26233 handle_command_reply: tid=0 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.890+0000 7f2c5a51bb80 1 client.26233 shutdown 2024-05-27T09:41:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.890+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/3841819294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 msgr2=0x5627c451add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.890+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c451add0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2c380098e0 tx=0x7f2c3800bcd0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.891+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/3841819294 shutdown_connections 2024-05-27T09:41:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.891+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c450de70 0x5627c451b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.891+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c438ee40 0x5627c451add0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.891+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x5627c459a560 0x5627c459c980 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.891+0000 7f2c2aa00700 0 client.26233 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:41:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.891+0000 7f2c5a51bb80 1 --2- 172.21.15.44:0/3841819294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c447a9f0 0x5627c451b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.891+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/3841819294 >> 172.21.15.44:0/3841819294 conn(0x5627c4428390 msgr2=0x5627c4517480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.891+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/3841819294 shutdown_connections 2024-05-27T09:41:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.892+0000 7f2c5a51bb80 1 -- 172.21.15.44:0/3841819294 wait complete. 2024-05-27T09:41:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.892+0000 7f2c5a51bb80 20 client.26233 trim_cache size 0 max 16384 2024-05-27T09:41:05.303 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:41:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 -- 172.21.15.44:0/1055137605 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c2c0b29f0 msgr2=0x7f2c2c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c2c0b29f0 0x7f2c2c0b4f00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c012010 tx=0x7f2c3c0218c0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 -- 172.21.15.44:0/1055137605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c44108f10 msgr2=0x7f2c4419d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c44108f10 0x7f2c4419d6b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000e9c0 tx=0x7f2c3000ecd0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 -- 172.21.15.44:0/1055137605 shutdown_connections 2024-05-27T09:41:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c2c0b29f0 0x7f2c2c0b4f00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44109860 0x7f2c441a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c441045e0 0x7f2c4419d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 --2- 172.21.15.44:0/1055137605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c44108f10 0x7f2c4419d6b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:05.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 -- 172.21.15.44:0/1055137605 >> 172.21.15.44:0/1055137605 conn(0x7f2c44100030 msgr2=0x7f2c441086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:05.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 -- 172.21.15.44:0/1055137605 shutdown_connections 2024-05-27T09:41:05.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:04.894+0000 7f2c4ba00700 1 -- 172.21.15.44:0/1055137605 wait complete. 2024-05-27T09:41:06.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.023+0000 7fdaaca00700 1 -- 172.21.15.44:0/645205317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdaa8105e40 msgr2=0x7fdaa810e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:06.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.023+0000 7fdaaca00700 1 --2- 172.21.15.44:0/645205317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdaa8105e40 0x7fdaa810e390 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fda940098f0 tx=0x7fda94009c00 comp rx=0 tx=0).stop 2024-05-27T09:41:06.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.023+0000 7fdaaca00700 1 -- 172.21.15.44:0/645205317 shutdown_connections 2024-05-27T09:41:06.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.023+0000 7fdaaca00700 1 --2- 172.21.15.44:0/645205317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa810e8d0 0x7fdaa8110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:06.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.023+0000 7fdaaca00700 1 --2- 172.21.15.44:0/645205317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa81054f0 0x7fdaa8105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:06.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.023+0000 7fdaaca00700 1 --2- 172.21.15.44:0/645205317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdaa8105e40 0x7fdaa810e390 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:06.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.023+0000 7fdaaca00700 1 -- 172.21.15.44:0/645205317 >> 172.21.15.44:0/645205317 conn(0x7fdaa80fedb0 msgr2=0x7fdaa81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:06.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.023+0000 7fdaaca00700 1 -- 172.21.15.44:0/645205317 shutdown_connections 2024-05-27T09:41:06.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.023+0000 7fdaaca00700 1 -- 172.21.15.44:0/645205317 wait complete. 2024-05-27T09:41:06.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.025+0000 7fdaaca00700 1 Processor -- start 2024-05-27T09:41:06.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.025+0000 7fdaaca00700 1 -- start start 2024-05-27T09:41:06.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.025+0000 7fdaaca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdaa81054f0 0x7fdaa819ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:06.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.025+0000 7fdaaca00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8105e40 0x7fdaa819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:06.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.025+0000 7fdaaca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa810e8d0 0x7fdaa819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:06.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.025+0000 7fdaaca00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa8114430 con 0x7fdaa81054f0 2024-05-27T09:41:06.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.025+0000 7fdaaca00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81142b0 con 0x7fdaa8105e40 2024-05-27T09:41:06.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.025+0000 7fdaaca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81145b0 con 0x7fdaa810e8d0 2024-05-27T09:41:06.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.026+0000 7fdaa5400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8105e40 0x7fdaa819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:06.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.026+0000 7fdaa5400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8105e40 0x7fdaa819b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51662/0 (socket says 172.21.15.44:51662) 2024-05-27T09:41:06.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.026+0000 7fdaa5400700 1 -- 172.21.15.44:0/1836654809 learned_addr learned my addr 172.21.15.44:0/1836654809 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:06.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.026+0000 7fdaa6800700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa810e8d0 0x7fdaa819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:06.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.026+0000 7fdaa5400700 1 -- 172.21.15.44:0/1836654809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa810e8d0 msgr2=0x7fdaa819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:06.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.026+0000 7fdaa5400700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa810e8d0 0x7fdaa819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:06.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.026+0000 7fdaa5400700 1 -- 172.21.15.44:0/1836654809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdaa81054f0 msgr2=0x7fdaa819ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:06.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.026+0000 7fdaa5400700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdaa81054f0 0x7fdaa819ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:06.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.026+0000 7fdaa5400700 1 -- 172.21.15.44:0/1836654809 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa819fcf0 con 0x7fdaa8105e40 2024-05-27T09:41:06.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.026+0000 7fdaa5400700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8105e40 0x7fdaa819b3c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fda9400f990 tx=0x7fda9400fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:06.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.027+0000 7fda9ea00700 1 -- 172.21.15.44:0/1836654809 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94005400 con 0x7fdaa8105e40 2024-05-27T09:41:06.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.027+0000 7fdaaca00700 1 -- 172.21.15.44:0/1836654809 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa819ff80 con 0x7fdaa8105e40 2024-05-27T09:41:06.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.027+0000 7fda9ea00700 1 -- 172.21.15.44:0/1836654809 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda940055a0 con 0x7fdaa8105e40 2024-05-27T09:41:06.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.027+0000 7fda9ea00700 1 -- 172.21.15.44:0/1836654809 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94004dc0 con 0x7fdaa8105e40 2024-05-27T09:41:06.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.027+0000 7fdaaca00700 1 -- 172.21.15.44:0/1836654809 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa81a0460 con 0x7fdaa8105e40 2024-05-27T09:41:06.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.029+0000 7fda93e00700 1 -- 172.21.15.44:0/1836654809 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda80005330 con 0x7fdaa8105e40 2024-05-27T09:41:06.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.031+0000 7fda9ea00700 1 -- 172.21.15.44:0/1836654809 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fda9400b560 con 0x7fdaa8105e40 2024-05-27T09:41:06.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.032+0000 7fda9ea00700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fda8c0b29f0 0x7fda8c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:06.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.032+0000 7fda9ea00700 1 -- 172.21.15.44:0/1836654809 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7fda940e3d00 con 0x7fdaa8105e40 2024-05-27T09:41:06.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.033+0000 7fdaa5e00700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fda8c0b29f0 0x7fda8c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:06.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.033+0000 7fdaa5e00700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fda8c0b29f0 0x7fda8c0b4f00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fda98005e50 tx=0x7fda98005da0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:06.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.040+0000 7fda9ea00700 1 -- 172.21.15.44:0/1836654809 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fda94013030 con 0x7fdaa8105e40 2024-05-27T09:41:06.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:05 smithi044 ceph-mon[31618]: pgmap v178: 97 pgs: 97 active+clean; 4.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.1 KiB/s wr, 11 op/s 2024-05-27T09:41:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:05 smithi050 ceph-mon[36371]: pgmap v178: 97 pgs: 97 active+clean; 4.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.1 KiB/s wr, 11 op/s 2024-05-27T09:41:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:05 smithi189 ceph-mon[36671]: pgmap v178: 97 pgs: 97 active+clean; 4.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.1 KiB/s wr, 11 op/s 2024-05-27T09:41:06.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.344+0000 7fda93e00700 1 -- 172.21.15.44:0/1836654809 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda80005620 con 0x7fdaa8105e40 2024-05-27T09:41:06.349 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:41:06.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.345+0000 7fda9ea00700 1 -- 172.21.15.44:0/1836654809 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11866 (secure 0 0 0) 0x7fda940ab740 con 0x7fdaa8105e40 2024-05-27T09:41:06.350 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:41:05.680635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/2673083839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2673083839},{"type":"v1","addr":"172.21.15.189:6835","nonce":2673083839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:41:06.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.350+0000 7fda93e00700 1 -- 172.21.15.44:0/1836654809 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fda8c0b29f0 msgr2=0x7fda8c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:06.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.350+0000 7fda93e00700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fda8c0b29f0 0x7fda8c0b4f00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fda98005e50 tx=0x7fda98005da0 comp rx=0 tx=0).stop 2024-05-27T09:41:06.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.350+0000 7fda93e00700 1 -- 172.21.15.44:0/1836654809 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8105e40 msgr2=0x7fdaa819b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:06.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.350+0000 7fda93e00700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8105e40 0x7fdaa819b3c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fda9400f990 tx=0x7fda9400fa70 comp rx=0 tx=0).stop 2024-05-27T09:41:06.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.351+0000 7fda93e00700 1 -- 172.21.15.44:0/1836654809 shutdown_connections 2024-05-27T09:41:06.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.351+0000 7fda93e00700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fda8c0b29f0 0x7fda8c0b4f00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:06.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.351+0000 7fda93e00700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa810e8d0 0x7fdaa819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:06.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.351+0000 7fda93e00700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdaa81054f0 0x7fdaa819ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:06.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.351+0000 7fda93e00700 1 --2- 172.21.15.44:0/1836654809 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaa8105e40 0x7fdaa819b3c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:06.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.351+0000 7fda93e00700 1 -- 172.21.15.44:0/1836654809 >> 172.21.15.44:0/1836654809 conn(0x7fdaa80fedb0 msgr2=0x7fdaa8100b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:06.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.351+0000 7fda93e00700 1 -- 172.21.15.44:0/1836654809 shutdown_connections 2024-05-27T09:41:06.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:06.351+0000 7fda93e00700 1 -- 172.21.15.44:0/1836654809 wait complete. 2024-05-27T09:41:06.353 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 27 2024-05-27T09:41:06.508 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:41:07.131 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:06 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:41:07.131 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:06 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1836654809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:41:07.184 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-05-27T09:41:07.184 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-05-27T09:41:07.195 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:06 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:41:07.195 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:06 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1836654809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:41:07.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:06 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:41:07.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:06 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1836654809' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:41:07.340 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:41:08.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:07 smithi044 ceph-mon[31618]: pgmap v179: 97 pgs: 97 active+clean; 5.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 588 KiB/s wr, 303 op/s 2024-05-27T09:41:08.228 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:07 smithi189 ceph-mon[36671]: pgmap v179: 97 pgs: 97 active+clean; 5.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 588 KiB/s wr, 303 op/s 2024-05-27T09:41:08.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:07 smithi050 ceph-mon[36371]: pgmap v179: 97 pgs: 97 active+clean; 5.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 588 KiB/s wr, 303 op/s 2024-05-27T09:41:08.486 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[67248]: *** Caught signal (Segmentation fault) ** 2024-05-27T09:41:08.486 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[67248]: in thread 7fb089c0d700 thread_name:MR_Finisher 2024-05-27T09:41:08.486 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:08 smithi189 podman[72270]: 2024-05-27 09:41:08.272429653 +0000 UTC m=+0.020328417 container died 4d1bcbfcf9a87dbbb87c71b3eb218a2eea56b27b2e064f1bd2114be588fcac51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T09:41:08.739 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:41:09.321 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:09 smithi189 podman[72270]: 2024-05-27 09:41:09.03693229 +0000 UTC m=+0.784831055 container remove 4d1bcbfcf9a87dbbb87c71b3eb218a2eea56b27b2e064f1bd2114be588fcac51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T09:41:09.321 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:09 smithi189 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=139/n/a 2024-05-27T09:41:09.838 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:09 smithi189 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T09:41:09.838 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:09 smithi189 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.f.service: Consumed 2.504s CPU time. 2024-05-27T09:41:10.208 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:09 smithi044 ceph-mon[31618]: pgmap v180: 97 pgs: 97 active+clean; 5.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 581 KiB/s wr, 294 op/s 2024-05-27T09:41:10.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:09 smithi050 ceph-mon[36371]: pgmap v180: 97 pgs: 97 active+clean; 5.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 581 KiB/s wr, 294 op/s 2024-05-27T09:41:10.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:09 smithi189 ceph-mon[36671]: pgmap v180: 97 pgs: 97 active+clean; 5.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 581 KiB/s wr, 294 op/s 2024-05-27T09:41:10.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.654+0000 7f4585000700 1 -- 172.21.15.44:0/336783902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f458010e8f0 msgr2=0x7f4580110dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:10.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.654+0000 7f4585000700 1 --2- 172.21.15.44:0/336783902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f458010e8f0 0x7f4580110dd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f457800b0a0 tx=0x7f457800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.655+0000 7f4585000700 1 -- 172.21.15.44:0/336783902 shutdown_connections 2024-05-27T09:41:10.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.655+0000 7f4585000700 1 --2- 172.21.15.44:0/336783902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f458010e8f0 0x7f4580110dd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.655+0000 7f4585000700 1 --2- 172.21.15.44:0/336783902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4580105510 0x7f4580105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.655+0000 7f4585000700 1 --2- 172.21.15.44:0/336783902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4580105e60 0x7f458010e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.655+0000 7f4585000700 1 -- 172.21.15.44:0/336783902 >> 172.21.15.44:0/336783902 conn(0x7f45800fef20 msgr2=0x7f4580101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:10.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.655+0000 7f4585000700 1 -- 172.21.15.44:0/336783902 shutdown_connections 2024-05-27T09:41:10.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.655+0000 7f4585000700 1 -- 172.21.15.44:0/336783902 wait complete. 2024-05-27T09:41:10.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.656+0000 7f4585000700 1 Processor -- start 2024-05-27T09:41:10.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.656+0000 7f4585000700 1 -- start start 2024-05-27T09:41:10.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f4585000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4580105510 0x7f458019d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f4585000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4580105e60 0x7f458019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f4585000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f458010e8f0 0x7f45801a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f4585000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4580114470 con 0x7f4580105e60 2024-05-27T09:41:10.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f4585000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f45801142f0 con 0x7f4580105510 2024-05-27T09:41:10.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f4585000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f45801145f0 con 0x7f458010e8f0 2024-05-27T09:41:10.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f457de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4580105510 0x7f458019d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f457de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4580105510 0x7f458019d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51670/0 (socket says 172.21.15.44:51670) 2024-05-27T09:41:10.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f457d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4580105e60 0x7f458019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f457de00700 1 -- 172.21.15.44:0/2105312652 learned_addr learned my addr 172.21.15.44:0/2105312652 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:10.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f457de00700 1 -- 172.21.15.44:0/2105312652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f458010e8f0 msgr2=0x7f45801a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:10.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f457de00700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f458010e8f0 0x7f45801a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f457de00700 1 -- 172.21.15.44:0/2105312652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4580105e60 msgr2=0x7f458019d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:10.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f457de00700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4580105e60 0x7f458019d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.657+0000 7f457de00700 1 -- 172.21.15.44:0/2105312652 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45801a1f30 con 0x7f4580105510 2024-05-27T09:41:10.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.658+0000 7f457d400700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4580105e60 0x7f458019d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:41:10.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.658+0000 7f457de00700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4580105510 0x7f458019d060 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f457000e9a0 tx=0x7f457000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:10.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.658+0000 7f4576a00700 1 -- 172.21.15.44:0/2105312652 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457000cbd0 con 0x7f4580105510 2024-05-27T09:41:10.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.658+0000 7f4585000700 1 -- 172.21.15.44:0/2105312652 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45801a2220 con 0x7f4580105510 2024-05-27T09:41:10.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.659+0000 7f4585000700 1 -- 172.21.15.44:0/2105312652 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45801a9b40 con 0x7f4580105510 2024-05-27T09:41:10.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.659+0000 7f4576a00700 1 -- 172.21.15.44:0/2105312652 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f457000cd70 con 0x7f4580105510 2024-05-27T09:41:10.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.659+0000 7f4576a00700 1 -- 172.21.15.44:0/2105312652 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4570026060 con 0x7f4580105510 2024-05-27T09:41:10.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.662+0000 7f4576a00700 1 -- 172.21.15.44:0/2105312652 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4570018680 con 0x7f4580105510 2024-05-27T09:41:10.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.664+0000 7f4576a00700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f45640b2a70 0x7f45640b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.664+0000 7f4576a00700 1 -- 172.21.15.44:0/2105312652 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f45700e27d0 con 0x7f4580105510 2024-05-27T09:41:10.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.665+0000 7f457d400700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f45640b2a70 0x7f45640b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.666+0000 7f457d400700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f45640b2a70 0x7f45640b4f80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4580110740 tx=0x7f456c009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:10.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.667+0000 7f4593c3bb80 1 Processor -- start 2024-05-27T09:41:10.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.667+0000 7f4593c3bb80 1 -- start start 2024-05-27T09:41:10.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.667+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be7f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.667+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385bd00e40 0x56385be7f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.667+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385be7fe70 0x56385be8c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.667+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56385bdade40 con 0x56385be7fe70 2024-05-27T09:41:10.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.667+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56385bd9d9e0 con 0x56385bd00e40 2024-05-27T09:41:10.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.667+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56385bda1a20 con 0x56385bdec9f0 2024-05-27T09:41:10.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.668+0000 7f457de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be7f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.668+0000 7f457de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be7f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39908/0 (socket says 172.21.15.44:39908) 2024-05-27T09:41:10.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.668+0000 7f457de00700 1 -- 172.21.15.44:0/1596767407 learned_addr learned my addr 172.21.15.44:0/1596767407 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:10.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.668+0000 7f457de00700 1 -- 172.21.15.44:0/1596767407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385bd00e40 msgr2=0x56385be7f870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:41:10.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.668+0000 7f457de00700 1 --2- 172.21.15.44:0/1596767407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385bd00e40 0x56385be7f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.668+0000 7f457de00700 1 -- 172.21.15.44:0/1596767407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385be7fe70 msgr2=0x56385be8c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:10.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.668+0000 7f457de00700 1 --2- 172.21.15.44:0/1596767407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385be7fe70 0x56385be8c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.668+0000 7f457de00700 1 -- 172.21.15.44:0/1596767407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385bd66420 con 0x56385bdec9f0 2024-05-27T09:41:10.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.668+0000 7f457de00700 1 --2- 172.21.15.44:0/1596767407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be7f330 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f45700a6b90 tx=0x7f45700b3330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c150dda8c2d0a2fd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:10.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.669+0000 7f456b400700 1 -- 172.21.15.44:0/1596767407 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45700aab10 con 0x56385bdec9f0 2024-05-27T09:41:10.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.669+0000 7f456b400700 1 -- 172.21.15.44:0/1596767407 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45700aeec0 con 0x56385bdec9f0 2024-05-27T09:41:10.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.669+0000 7f456b400700 1 -- 172.21.15.44:0/1596767407 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45700a66e0 con 0x56385bdec9f0 2024-05-27T09:41:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.670+0000 7f4593c3bb80 1 -- 172.21.15.44:0/1596767407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 msgr2=0x56385be7f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.670+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/1596767407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be7f330 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f45700a6b90 tx=0x7f45700b3330 comp rx=0 tx=0).stop 2024-05-27T09:41:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.670+0000 7f4593c3bb80 1 -- 172.21.15.44:0/1596767407 shutdown_connections 2024-05-27T09:41:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.671+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/1596767407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385be7fe70 0x56385be8c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.671+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/1596767407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be7f330 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.671+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/1596767407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385bd00e40 0x56385be7f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.671+0000 7f4593c3bb80 1 -- 172.21.15.44:0/1596767407 >> 172.21.15.44:0/1596767407 conn(0x56385bd9a390 msgr2=0x56385be7b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:10.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.671+0000 7f4593c3bb80 1 -- 172.21.15.44:0/1596767407 shutdown_connections 2024-05-27T09:41:10.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.671+0000 7f4593c3bb80 1 -- 172.21.15.44:0/1596767407 wait complete. 2024-05-27T09:41:10.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.672+0000 7f4593c3bb80 1 -- start start 2024-05-27T09:41:10.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.672+0000 7f4593c3bb80 1 Processor -- start 2024-05-27T09:41:10.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.673+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385bd00e40 0x56385be89720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.673+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be89c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.673+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385be7fe70 0x56385be8a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.673+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56385bca3fa0 con 0x56385be7fe70 2024-05-27T09:41:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.673+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56385bd3e760 con 0x56385bd00e40 2024-05-27T09:41:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.673+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56385be9db30 con 0x56385bdec9f0 2024-05-27T09:41:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.673+0000 7f4593c3bb80 10 client.0 _ll_register_callbacks cb 0x56385bd47c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.673+0000 7f457de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385bd00e40 0x56385be89720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.673+0000 7f457de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385bd00e40 0x56385be89720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51684/0 (socket says 172.21.15.44:51684) 2024-05-27T09:41:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.673+0000 7f457de00700 1 -- 172.21.15.44:0/3227948861 learned_addr learned my addr 172.21.15.44:0/3227948861 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.674+0000 7f457d400700 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be89c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.674+0000 7f457e800700 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385be7fe70 0x56385be8a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.674+0000 7f457de00700 1 -- 172.21.15.44:0/3227948861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 msgr2=0x56385be89c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.674+0000 7f457de00700 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be89c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.674+0000 7f457de00700 1 -- 172.21.15.44:0/3227948861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385be7fe70 msgr2=0x56385be8a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.674+0000 7f457de00700 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385be7fe70 0x56385be8a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.674+0000 7f457de00700 1 -- 172.21.15.44:0/3227948861 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385bced8d0 con 0x56385bd00e40 2024-05-27T09:41:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.674+0000 7f457e800700 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385be7fe70 0x56385be8a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:41:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.675+0000 7f457de00700 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385bd00e40 0x56385be89720 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f45700a7d80 tx=0x7f4570000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba57f93ae1fa0863 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.675+0000 7f455fe00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.675+0000 7f455fe00700 1 -- 172.21.15.44:0/3227948861 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45700c5200 con 0x56385bd00e40 2024-05-27T09:41:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.675+0000 7f455fe00700 1 -- 172.21.15.44:0/3227948861 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f45700a7100 con 0x56385bd00e40 2024-05-27T09:41:10.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.676+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3227948861 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45700aeec0 con 0x56385bd00e40 2024-05-27T09:41:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.676+0000 7f455fe00700 1 -- 172.21.15.44:0/3227948861 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f45700aab10 con 0x56385bd00e40 2024-05-27T09:41:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.676+0000 7f4593c3bb80 10 client.26243 fetch_fsmap learned FSMap version 27 2024-05-27T09:41:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.676+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3227948861 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56385bd6e060 con 0x56385bd00e40 2024-05-27T09:41:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.677+0000 7f455fe00700 1 -- 172.21.15.44:0/3227948861 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45700aeec0 con 0x56385bd00e40 2024-05-27T09:41:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.677+0000 7f455fe00700 1 -- 172.21.15.44:0/3227948861 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7f45700cc1f0 con 0x56385bd00e40 2024-05-27T09:41:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.677+0000 7f4593c3bb80 10 client.26243 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T09:41:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.677+0000 7f4593c3bb80 10 client.26243 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.677+0000 7f4593c3bb80 20 client.26243 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.677+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56385beaaa40 0x56385beace60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.677+0000 7f4593c3bb80 4 client.26243 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:10.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.677+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3227948861 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56385bdafd90 con 0x56385beaaa40 2024-05-27T09:41:10.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.678+0000 7f457d400700 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56385beaaa40 0x56385beace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.679+0000 7f457d400700 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56385beaaa40 0x56385beace60 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=634dcc3683fe995c server_cookie=e2b30620a3e62191 in_seq=0 out_seq=0 2024-05-27T09:41:10.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.679+0000 7f455fe00700 10 client.26243 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:10.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f455fe00700 1 -- 172.21.15.44:0/3227948861 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56385bdafd90 con 0x56385beaaa40 2024-05-27T09:41:10.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f455fe00700 10 client.26243 handle_command_reply: tid=0 2024-05-27T09:41:10.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f4593c3bb80 1 client.26243 shutdown 2024-05-27T09:41:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3227948861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385bd00e40 msgr2=0x56385be89720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385bd00e40 0x56385be89720 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f45700a7d80 tx=0x7f4570000bc0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3227948861 shutdown_connections 2024-05-27T09:41:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56385beaaa40 0x56385beace60 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385be7fe70 0x56385be8a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385bd00e40 0x56385be89720 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f455fe00700 0 client.26243 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:10.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3227948861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be89c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3227948861 >> 172.21.15.44:0/3227948861 conn(0x56385bd9a390 msgr2=0x56385be7f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:10.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3227948861 shutdown_connections 2024-05-27T09:41:10.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.681+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3227948861 wait complete. 2024-05-27T09:41:10.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.682+0000 7f4593c3bb80 20 client.26243 trim_cache size 0 max 16384 2024-05-27T09:41:10.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.701+0000 7f4593c3bb80 1 Processor -- start 2024-05-27T09:41:10.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.701+0000 7f4593c3bb80 1 -- start start 2024-05-27T09:41:10.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.701+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385bd00e40 0x56385be88a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.701+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be84ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.701+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 0x56385be7b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.701+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f456000fd80 con 0x56385bd00e40 2024-05-27T09:41:10.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.701+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f456000d610 con 0x56385be7fe70 2024-05-27T09:41:10.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.701+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f456000b460 con 0x56385bdec9f0 2024-05-27T09:41:10.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.702+0000 7f457de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385bd00e40 0x56385be88a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.702+0000 7f457e800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 0x56385be7b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.702+0000 7f457d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be84ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.702+0000 7f457e800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 0x56385be7b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51688/0 (socket says 172.21.15.44:51688) 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.702+0000 7f457de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385bd00e40 0x56385be88a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32950/0 (socket says 172.21.15.44:32950) 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.702+0000 7f457e800700 1 -- 172.21.15.44:0/1885876253 learned_addr learned my addr 172.21.15.44:0/1885876253 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.702+0000 7f457d400700 1 -- 172.21.15.44:0/1885876253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 msgr2=0x56385be7b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.702+0000 7f457d400700 1 --2- 172.21.15.44:0/1885876253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 0x56385be7b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.702+0000 7f457d400700 1 -- 172.21.15.44:0/1885876253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385bd00e40 msgr2=0x56385be88a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.702+0000 7f457d400700 1 --2- 172.21.15.44:0/1885876253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385bd00e40 0x56385be88a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.702+0000 7f457d400700 1 -- 172.21.15.44:0/1885876253 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385be8a510 con 0x56385bdec9f0 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.703+0000 7f457d400700 1 --2- 172.21.15.44:0/1885876253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be84ef0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x56385be7f180 tx=0x7f456c013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bc597b3a3bf542c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.703+0000 7f456aa00700 1 -- 172.21.15.44:0/1885876253 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c011dd0 con 0x56385bdec9f0 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.703+0000 7f456aa00700 1 -- 172.21.15.44:0/1885876253 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f456c02d070 con 0x56385bdec9f0 2024-05-27T09:41:10.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.703+0000 7f456aa00700 1 -- 172.21.15.44:0/1885876253 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c01f820 con 0x56385bdec9f0 2024-05-27T09:41:10.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.704+0000 7f4593c3bb80 1 -- 172.21.15.44:0/1885876253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 msgr2=0x56385be84ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:10.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.704+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/1885876253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be84ef0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x56385be7f180 tx=0x7f456c013000 comp rx=0 tx=0).stop 2024-05-27T09:41:10.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.704+0000 7f4593c3bb80 1 -- 172.21.15.44:0/1885876253 shutdown_connections 2024-05-27T09:41:10.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.704+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/1885876253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 0x56385be7b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.704+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/1885876253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385bd00e40 0x56385be88a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.704+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/1885876253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bdec9f0 0x56385be84ef0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.704+0000 7f4593c3bb80 1 -- 172.21.15.44:0/1885876253 >> 172.21.15.44:0/1885876253 conn(0x56385bd9a390 msgr2=0x56385be80ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:10.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.704+0000 7f4593c3bb80 1 -- 172.21.15.44:0/1885876253 shutdown_connections 2024-05-27T09:41:10.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.705+0000 7f4593c3bb80 1 -- 172.21.15.44:0/1885876253 wait complete. 2024-05-27T09:41:10.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.705+0000 7f4593c3bb80 1 -- start start 2024-05-27T09:41:10.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.705+0000 7f4593c3bb80 1 Processor -- start 2024-05-27T09:41:10.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.706+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bd00e40 0x56385beb0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.706+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385bdec9f0 0x56385beb06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.706+0000 7f4593c3bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 0x56385beab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.706+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f456000bad0 con 0x56385bdec9f0 2024-05-27T09:41:10.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.706+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f456000cf40 con 0x56385be7fe70 2024-05-27T09:41:10.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.706+0000 7f4593c3bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f456000c1a0 con 0x56385bd00e40 2024-05-27T09:41:10.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.706+0000 7f4593c3bb80 10 client.0 _ll_register_callbacks cb 0x56385bd47c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:10.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.707+0000 7f457e800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 0x56385beab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.707+0000 7f457e800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 0x56385beab600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51698/0 (socket says 172.21.15.44:51698) 2024-05-27T09:41:10.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.707+0000 7f457e800700 1 -- 172.21.15.44:0/3127527444 learned_addr learned my addr 172.21.15.44:0/3127527444 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:10.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.707+0000 7f457d400700 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385bdec9f0 0x56385beb06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.707+0000 7f457e800700 1 -- 172.21.15.44:0/3127527444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bd00e40 msgr2=0x56385beb0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:10.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.707+0000 7f457e800700 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bd00e40 0x56385beb0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.707+0000 7f457e800700 1 -- 172.21.15.44:0/3127527444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385bdec9f0 msgr2=0x56385beb06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:10.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.707+0000 7f457e800700 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385bdec9f0 0x56385beb06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.707+0000 7f457e800700 1 -- 172.21.15.44:0/3127527444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385beabda0 con 0x56385be7fe70 2024-05-27T09:41:10.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.708+0000 7f455fe00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:10.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.708+0000 7f457e800700 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 0x56385beab600 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4578006250 tx=0x7f457800bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44602422f649a7da server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:10.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.708+0000 7f455fe00700 1 -- 172.21.15.44:0/3127527444 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578020070 con 0x56385be7fe70 2024-05-27T09:41:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.709+0000 7f455fe00700 1 -- 172.21.15.44:0/3127527444 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f45780075c0 con 0x56385be7fe70 2024-05-27T09:41:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.709+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3127527444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f456c02d070 con 0x56385be7fe70 2024-05-27T09:41:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.709+0000 7f455fe00700 1 -- 172.21.15.44:0/3127527444 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4578009cc0 con 0x56385be7fe70 2024-05-27T09:41:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.709+0000 7f4593c3bb80 10 client.26249 fetch_fsmap learned FSMap version 27 2024-05-27T09:41:10.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.709+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3127527444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56385bf1d5b0 con 0x56385be7fe70 2024-05-27T09:41:10.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.710+0000 7f455fe00700 1 -- 172.21.15.44:0/3127527444 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f456c02d070 con 0x56385be7fe70 2024-05-27T09:41:10.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.710+0000 7f455fe00700 1 -- 172.21.15.44:0/3127527444 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7f4578011070 con 0x56385be7fe70 2024-05-27T09:41:10.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.710+0000 7f4593c3bb80 10 client.26249 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T09:41:10.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.710+0000 7f4593c3bb80 10 client.26249 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:10.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.710+0000 7f4593c3bb80 20 client.26249 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:10.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.710+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56385bf1dcf0 0x56385bf20130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:10.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.711+0000 7f4593c3bb80 4 client.26249 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:41:10.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.711+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3127527444 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56385bda9570 con 0x56385bf1dcf0 2024-05-27T09:41:10.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.711+0000 7f457de00700 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56385bf1dcf0 0x56385bf20130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:10.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f457de00700 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56385bf1dcf0 0x56385bf20130 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8e80ee272b5a3f4 server_cookie=825750e35ba3dbdb in_seq=0 out_seq=0 2024-05-27T09:41:10.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f455fe00700 10 client.26249 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f455fe00700 1 -- 172.21.15.44:0/3127527444 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56385bda9570 con 0x56385bf1dcf0 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f455fe00700 10 client.26249 handle_command_reply: tid=0 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f4593c3bb80 1 client.26249 shutdown 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3127527444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 msgr2=0x56385beab600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 0x56385beab600 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4578006250 tx=0x7f457800bce0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3127527444 shutdown_connections 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56385bf1dcf0 0x56385bf20130 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385be7fe70 0x56385beab600 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56385bd00e40 0x56385beb0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f4593c3bb80 1 --2- 172.21.15.44:0/3127527444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56385bdec9f0 0x56385beb06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3127527444 >> 172.21.15.44:0/3127527444 conn(0x56385bd9a390 msgr2=0x56385be887a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.712+0000 7f455fe00700 0 client.26249 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:10.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.713+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3127527444 shutdown_connections 2024-05-27T09:41:10.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.713+0000 7f4593c3bb80 1 -- 172.21.15.44:0/3127527444 wait complete. 2024-05-27T09:41:10.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.713+0000 7f4593c3bb80 20 client.26249 trim_cache size 0 max 16384 2024-05-27T09:41:10.715 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 -- 172.21.15.44:0/2105312652 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f45640b2a70 msgr2=0x7f45640b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f45640b2a70 0x7f45640b4f80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4580110740 tx=0x7f456c009260 comp rx=0 tx=0).stop 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 -- 172.21.15.44:0/2105312652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4580105510 msgr2=0x7f458019d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4580105510 0x7f458019d060 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f457000e9a0 tx=0x7f457000ed60 comp rx=0 tx=0).stop 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 -- 172.21.15.44:0/2105312652 shutdown_connections 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f45640b2a70 0x7f45640b4f80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f458010e8f0 0x7f45801a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4580105510 0x7f458019d060 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 --2- 172.21.15.44:0/2105312652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4580105e60 0x7f458019d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 -- 172.21.15.44:0/2105312652 >> 172.21.15.44:0/2105312652 conn(0x7f45800fef20 msgr2=0x7f45801031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 -- 172.21.15.44:0/2105312652 shutdown_connections 2024-05-27T09:41:10.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:10.714+0000 7f4585000700 1 -- 172.21.15.44:0/2105312652 wait complete. 2024-05-27T09:41:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.192+0000 7f3f5f200700 1 -- 172.21.15.44:0/287157772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f580764a0 msgr2=0x7f3f58114e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.192+0000 7f3f5f200700 1 --2- 172.21.15.44:0/287157772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f580764a0 0x7f3f58114e20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009990 tx=0x7f3f44009ca0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.192+0000 7f3f5f200700 1 -- 172.21.15.44:0/287157772 shutdown_connections 2024-05-27T09:41:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.192+0000 7f3f5f200700 1 --2- 172.21.15.44:0/287157772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f58115360 0x7f3f58117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.192+0000 7f3f5f200700 1 --2- 172.21.15.44:0/287157772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f58075b50 0x7f3f58075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.192+0000 7f3f5f200700 1 --2- 172.21.15.44:0/287157772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f580764a0 0x7f3f58114e20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.192+0000 7f3f5f200700 1 -- 172.21.15.44:0/287157772 >> 172.21.15.44:0/287157772 conn(0x7f3f58100030 msgr2=0x7f3f58102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.192+0000 7f3f5f200700 1 -- 172.21.15.44:0/287157772 shutdown_connections 2024-05-27T09:41:12.194 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:11 smithi044 ceph-mon[31618]: pgmap v181: 97 pgs: 97 active+clean; 14 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1.5 MiB/s wr, 329 op/s 2024-05-27T09:41:12.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.192+0000 7f3f5f200700 1 -- 172.21.15.44:0/287157772 wait complete. 2024-05-27T09:41:12.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.194+0000 7f3f5f200700 1 Processor -- start 2024-05-27T09:41:12.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.194+0000 7f3f5f200700 1 -- start start 2024-05-27T09:41:12.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f5f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58075b50 0x7f3f58114d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f5f200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f580764a0 0x7f3f5810de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f5f200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f58115360 0x7f3f5810e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f5f200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5811aab0 con 0x7f3f580764a0 2024-05-27T09:41:12.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f5f200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5811a930 con 0x7f3f58115360 2024-05-27T09:41:12.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f5f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5811ac30 con 0x7f3f58075b50 2024-05-27T09:41:12.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58075b50 0x7f3f58114d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58075b50 0x7f3f58114d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39954/0 (socket says 172.21.15.44:39954) 2024-05-27T09:41:12.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f57e00700 1 -- 172.21.15.44:0/2217105063 learned_addr learned my addr 172.21.15.44:0/2217105063 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:12.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f57400700 1 --2- 172.21.15.44:0/2217105063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f580764a0 0x7f3f5810de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f57e00700 1 -- 172.21.15.44:0/2217105063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f58115360 msgr2=0x7f3f5810e4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T09:41:12.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.195+0000 7f3f5d200700 1 --2- 172.21.15.44:0/2217105063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f58115360 0x7f3f5810e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.196+0000 7f3f57e00700 1 --2- 172.21.15.44:0/2217105063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f58115360 0x7f3f5810e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.196+0000 7f3f57e00700 1 -- 172.21.15.44:0/2217105063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f580764a0 msgr2=0x7f3f5810de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:12.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.196+0000 7f3f57e00700 1 --2- 172.21.15.44:0/2217105063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f580764a0 0x7f3f5810de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.196+0000 7f3f57e00700 1 -- 172.21.15.44:0/2217105063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f58112870 con 0x7f3f58075b50 2024-05-27T09:41:12.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.196+0000 7f3f57400700 1 --2- 172.21.15.44:0/2217105063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f580764a0 0x7f3f5810de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:41:12.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.196+0000 7f3f5d200700 1 --2- 172.21.15.44:0/2217105063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f58115360 0x7f3f5810e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:41:12.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.196+0000 7f3f57e00700 1 --2- 172.21.15.44:0/2217105063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58075b50 0x7f3f58114d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800e9a0 tx=0x7f3f4800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:12.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.196+0000 7f3f54c00700 1 -- 172.21.15.44:0/2217105063 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4800cbd0 con 0x7f3f58075b50 2024-05-27T09:41:12.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.196+0000 7f3f54c00700 1 -- 172.21.15.44:0/2217105063 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f4800cd70 con 0x7f3f58075b50 2024-05-27T09:41:12.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.196+0000 7f3f54c00700 1 -- 172.21.15.44:0/2217105063 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48018680 con 0x7f3f58075b50 2024-05-27T09:41:12.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.196+0000 7f3f5f200700 1 -- 172.21.15.44:0/2217105063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f58112b60 con 0x7f3f58075b50 2024-05-27T09:41:12.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.197+0000 7f3f5f200700 1 -- 172.21.15.44:0/2217105063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f58113050 con 0x7f3f58075b50 2024-05-27T09:41:12.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.202+0000 7f3f54c00700 1 -- 172.21.15.44:0/2217105063 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3f48010970 con 0x7f3f58075b50 2024-05-27T09:41:12.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.204+0000 7f3f6dcb9b80 1 Processor -- start 2024-05-27T09:41:12.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.204+0000 7f3f54c00700 1 --2- 172.21.15.44:0/2217105063 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3f3c0b2a70 0x7f3f3c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.204+0000 7f3f6dcb9b80 1 -- start start 2024-05-27T09:41:12.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.204+0000 7f3f54c00700 1 -- 172.21.15.44:0/2217105063 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f3f480e2960 con 0x7f3f58075b50 2024-05-27T09:41:12.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.204+0000 7f3f6dcb9b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 0x55ca00e2fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.204+0000 7f3f6dcb9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e30380 0x55ca00e29f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.204+0000 7f3f6dcb9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e2a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.204+0000 7f3f6dcb9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ca00d57fd0 con 0x55ca00e30380 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.204+0000 7f3f6dcb9b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ca00d47b70 con 0x55ca00caafd0 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.204+0000 7f3f6dcb9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ca00d4bbb0 con 0x55ca00e2a4d0 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.204+0000 7f3f5d200700 1 --2- 172.21.15.44:0/2217105063 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3f3c0b2a70 0x7f3f3c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.205+0000 7f3f5d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e2a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.205+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 0x55ca00e2fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.205+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 0x55ca00e2fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51740/0 (socket says 172.21.15.44:51740) 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.205+0000 7f3f57e00700 1 -- 172.21.15.44:0/1312616359 learned_addr learned my addr 172.21.15.44:0/1312616359 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.205+0000 7f3f57400700 1 --2- 172.21.15.44:0/1312616359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e30380 0x55ca00e29f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.205+0000 7f3f57e00700 1 -- 172.21.15.44:0/1312616359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 msgr2=0x55ca00e2a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.205+0000 7f3f57e00700 1 --2- 172.21.15.44:0/1312616359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e2a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.205+0000 7f3f57e00700 1 -- 172.21.15.44:0/1312616359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e30380 msgr2=0x55ca00e29f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:12.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.205+0000 7f3f57e00700 1 --2- 172.21.15.44:0/1312616359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e30380 0x55ca00e29f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.205+0000 7f3f57e00700 1 -- 172.21.15.44:0/1312616359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca00d105b0 con 0x55ca00caafd0 2024-05-27T09:41:12.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.206+0000 7f3f57e00700 1 --2- 172.21.15.44:0/1312616359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 0x55ca00e2fe40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3f480a9ba0 tx=0x7f3f480a9810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bab755044982ad84 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:12.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.206+0000 7f3f5d200700 1 --2- 172.21.15.44:0/2217105063 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3f3c0b2a70 0x7f3f3c0b4f80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009550 tx=0x7f3f50009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:12.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.206+0000 7f3f4d600700 1 -- 172.21.15.44:0/1312616359 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f480adfc0 con 0x55ca00caafd0 2024-05-27T09:41:12.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.207+0000 7f3f4d600700 1 -- 172.21.15.44:0/1312616359 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f480b2a70 con 0x55ca00caafd0 2024-05-27T09:41:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.207+0000 7f3f4d600700 1 -- 172.21.15.44:0/1312616359 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f480a57e0 con 0x55ca00caafd0 2024-05-27T09:41:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.208+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/1312616359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 msgr2=0x55ca00e2fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.208+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/1312616359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 0x55ca00e2fe40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3f480a9ba0 tx=0x7f3f480a9810 comp rx=0 tx=0).stop 2024-05-27T09:41:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.208+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/1312616359 shutdown_connections 2024-05-27T09:41:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.208+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/1312616359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e2a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.208+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/1312616359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 0x55ca00e2fe40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.208+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/1312616359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e30380 0x55ca00e29f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.208+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/1312616359 >> 172.21.15.44:0/1312616359 conn(0x55ca00d0b150 msgr2=0x55ca00e25190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.208+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/1312616359 shutdown_connections 2024-05-27T09:41:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.209+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/1312616359 wait complete. 2024-05-27T09:41:12.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.209+0000 7f3f6dcb9b80 1 -- start start 2024-05-27T09:41:12.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.210+0000 7f3f6dcb9b80 1 Processor -- start 2024-05-27T09:41:12.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.211+0000 7f3f6dcb9b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 0x55ca00e33bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.211+0000 7f3f6dcb9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e34130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.211+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 0x55ca00e33bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.211+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 0x55ca00e33bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51750/0 (socket says 172.21.15.44:51750) 2024-05-27T09:41:12.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.211+0000 7f3f57e00700 1 -- 172.21.15.44:0/2373100610 learned_addr learned my addr 172.21.15.44:0/2373100610 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:12.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.211+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e30380 0x55ca00e34670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.211+0000 7f3f57400700 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e34130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.211+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/2373100610 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ca00c4e130 con 0x55ca00e30380 2024-05-27T09:41:12.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.211+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/2373100610 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ca00ce88f0 con 0x55ca00caafd0 2024-05-27T09:41:12.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.211+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/2373100610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ca00e36e50 con 0x55ca00e2a4d0 2024-05-27T09:41:12.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.211+0000 7f3f6dcb9b80 10 client.0 _ll_register_callbacks cb 0x55ca00cf1e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:12.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.212+0000 7f3f57400700 1 -- 172.21.15.44:0/2373100610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 msgr2=0x55ca00e33bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:12.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.212+0000 7f3f57400700 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 0x55ca00e33bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.212+0000 7f3f57400700 1 -- 172.21.15.44:0/2373100610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e30380 msgr2=0x55ca00e34670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.212+0000 7f3f5d200700 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e30380 0x55ca00e34670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.212+0000 7f3f57400700 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e30380 0x55ca00e34670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.212+0000 7f3f57400700 1 -- 172.21.15.44:0/2373100610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca00c97a60 con 0x55ca00e2a4d0 2024-05-27T09:41:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.213+0000 7f3f57400700 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e34130 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3f581035c0 tx=0x7f3f44005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56e65728fd331d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.213+0000 7f3f42000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:41:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.213+0000 7f3f42000700 1 -- 172.21.15.44:0/2373100610 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f44027060 con 0x55ca00e2a4d0 2024-05-27T09:41:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.213+0000 7f3f42000700 1 -- 172.21.15.44:0/2373100610 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f3f44003680 con 0x55ca00e2a4d0 2024-05-27T09:41:12.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.213+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/2373100610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f480b2a70 con 0x55ca00e2a4d0 2024-05-27T09:41:12.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.213+0000 7f3f42000700 1 -- 172.21.15.44:0/2373100610 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3f44003880 con 0x55ca00e2a4d0 2024-05-27T09:41:12.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.214+0000 7f3f6dcb9b80 10 client.26263 fetch_fsmap learned FSMap version 27 2024-05-27T09:41:12.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.214+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/2373100610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ca00d181f0 con 0x55ca00e2a4d0 2024-05-27T09:41:12.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.214+0000 7f3f42000700 1 -- 172.21.15.44:0/2373100610 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f480b2a70 con 0x55ca00e2a4d0 2024-05-27T09:41:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.214+0000 7f3f42000700 1 -- 172.21.15.44:0/2373100610 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7f3f44021030 con 0x55ca00e2a4d0 2024-05-27T09:41:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.214+0000 7f3f6dcb9b80 10 client.26263 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T09:41:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.214+0000 7f3f6dcb9b80 10 client.26263 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:41:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.214+0000 7f3f6dcb9b80 20 client.26263 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.214+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ca00e43e30 0x55ca00e46250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.214+0000 7f3f6dcb9b80 4 client.26263 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.214+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/2373100610 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca00d59f20 con 0x55ca00e43e30 2024-05-27T09:41:12.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.215+0000 7f3f57e00700 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ca00e43e30 0x55ca00e46250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.215+0000 7f3f42000700 10 client.26263 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.215+0000 7f3f57e00700 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ca00e43e30 0x55ca00e46250 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38b1b838d4a528ee server_cookie=4da076e58d3b2510 in_seq=0 out_seq=0 2024-05-27T09:41:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.216+0000 7f3f42000700 1 -- 172.21.15.44:0/2373100610 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ca00d59f20 con 0x55ca00e43e30 2024-05-27T09:41:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.216+0000 7f3f42000700 10 client.26263 handle_command_reply: tid=0 2024-05-27T09:41:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.216+0000 7f3f6dcb9b80 1 client.26263 shutdown 2024-05-27T09:41:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/2373100610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 msgr2=0x55ca00e34130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:12.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e34130 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3f581035c0 tx=0x7f3f44005770 comp rx=0 tx=0).stop 2024-05-27T09:41:12.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/2373100610 shutdown_connections 2024-05-27T09:41:12.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ca00e43e30 0x55ca00e46250 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e30380 0x55ca00e34670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00caafd0 0x55ca00e33bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/2373100610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e34130 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/2373100610 >> 172.21.15.44:0/2373100610 conn(0x55ca00d0b150 msgr2=0x55ca00e28da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:12.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f42000700 0 client.26263 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:12.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/2373100610 shutdown_connections 2024-05-27T09:41:12.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/2373100610 wait complete. 2024-05-27T09:41:12.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.217+0000 7f3f6dcb9b80 20 client.26263 trim_cache size 0 max 16384 2024-05-27T09:41:12.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.245+0000 7f3f6dcb9b80 1 Processor -- start 2024-05-27T09:41:12.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.245+0000 7f3f6dcb9b80 1 -- start start 2024-05-27T09:41:12.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.245+0000 7f3f6dcb9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00caafd0 0x55ca00e297d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.245+0000 7f3f6dcb9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e2a4d0 0x55ca00e2f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f6dcb9b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00e30380 0x55ca00e3feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f6dcb9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581c0660 con 0x55ca00e2a4d0 2024-05-27T09:41:12.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f6dcb9b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581bdef0 con 0x55ca00e30380 2024-05-27T09:41:12.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f6dcb9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581bbd40 con 0x55ca00caafd0 2024-05-27T09:41:12.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00caafd0 0x55ca00e297d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00caafd0 0x55ca00e297d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39986/0 (socket says 172.21.15.44:39986) 2024-05-27T09:41:12.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f57e00700 1 -- 172.21.15.44:0/3669236407 learned_addr learned my addr 172.21.15.44:0/3669236407 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:12.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f57e00700 1 -- 172.21.15.44:0/3669236407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00e30380 msgr2=0x55ca00e3feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:12.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f57e00700 1 --2- 172.21.15.44:0/3669236407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00e30380 0x55ca00e3feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f57e00700 1 -- 172.21.15.44:0/3669236407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e2a4d0 msgr2=0x55ca00e2f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:12.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f57e00700 1 --2- 172.21.15.44:0/3669236407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e2a4d0 0x55ca00e2f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.246+0000 7f3f57e00700 1 -- 172.21.15.44:0/3669236407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca00e349e0 con 0x55ca00caafd0 2024-05-27T09:41:12.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.247+0000 7f3f57e00700 1 --2- 172.21.15.44:0/3669236407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00caafd0 0x55ca00e297d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3f48000c00 tx=0x7f3f480a5b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=977f3ad3e5e3e439 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:12.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.247+0000 7f3f4cc00700 1 -- 172.21.15.44:0/3669236407 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f480c3020 con 0x55ca00caafd0 2024-05-27T09:41:12.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.247+0000 7f3f4cc00700 1 -- 172.21.15.44:0/3669236407 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f480adfc0 con 0x55ca00caafd0 2024-05-27T09:41:12.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.247+0000 7f3f4cc00700 1 -- 172.21.15.44:0/3669236407 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4808d070 con 0x55ca00caafd0 2024-05-27T09:41:12.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.248+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/3669236407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00caafd0 msgr2=0x55ca00e297d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:12.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.248+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/3669236407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00caafd0 0x55ca00e297d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3f48000c00 tx=0x7f3f480a5b80 comp rx=0 tx=0).stop 2024-05-27T09:41:12.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.248+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/3669236407 shutdown_connections 2024-05-27T09:41:12.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.249+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/3669236407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00e30380 0x55ca00e3feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.249+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/3669236407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00caafd0 0x55ca00e297d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.249+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/3669236407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e2a4d0 0x55ca00e2f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.249+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/3669236407 >> 172.21.15.44:0/3669236407 conn(0x55ca00d0b150 msgr2=0x55ca00e2d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:12.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.249+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/3669236407 shutdown_connections 2024-05-27T09:41:12.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.249+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/3669236407 wait complete. 2024-05-27T09:41:12.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.249+0000 7f3f6dcb9b80 1 -- start start 2024-05-27T09:41:12.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.250+0000 7f3f6dcb9b80 1 Processor -- start 2024-05-27T09:41:12.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.250+0000 7f3f6dcb9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e3c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.250+0000 7f3f6dcb9b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00e30380 0x55ca00e3c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.250+0000 7f3f6dcb9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e39070 0x55ca00e39480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.250+0000 7f3f6dcb9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581bc3b0 con 0x55ca00e39070 2024-05-27T09:41:12.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.250+0000 7f3f6dcb9b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581bd820 con 0x55ca00e30380 2024-05-27T09:41:12.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.250+0000 7f3f6dcb9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581bca80 con 0x55ca00e2a4d0 2024-05-27T09:41:12.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.250+0000 7f3f6dcb9b80 10 client.0 _ll_register_callbacks cb 0x55ca00cf1e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:12.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.251+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e3c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.251+0000 7f3f57400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00e30380 0x55ca00e3c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.251+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e3c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39994/0 (socket says 172.21.15.44:39994) 2024-05-27T09:41:12.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.251+0000 7f3f57e00700 1 -- 172.21.15.44:0/497666062 learned_addr learned my addr 172.21.15.44:0/497666062 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:12.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.251+0000 7f3f5d200700 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e39070 0x55ca00e39480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.251+0000 7f3f57e00700 1 -- 172.21.15.44:0/497666062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00e30380 msgr2=0x55ca00e3c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:12.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.251+0000 7f3f57e00700 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00e30380 0x55ca00e3c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.251+0000 7f3f57e00700 1 -- 172.21.15.44:0/497666062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e39070 msgr2=0x55ca00e39480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:12.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.251+0000 7f3f57e00700 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e39070 0x55ca00e39480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:12.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.251+0000 7f3f57e00700 1 -- 172.21.15.44:0/497666062 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca00e39b30 con 0x55ca00e2a4d0 2024-05-27T09:41:12.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.251+0000 7f3f5d200700 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e39070 0x55ca00e39480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:41:12.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.252+0000 7f3f57e00700 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e3c2a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3f480a9ba0 tx=0x7f3f480b2b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7ec0647f17372a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:12.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.252+0000 7f3f42000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:41:12.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.252+0000 7f3f42000700 1 -- 172.21.15.44:0/497666062 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f480c3040 con 0x55ca00e2a4d0 2024-05-27T09:41:12.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.252+0000 7f3f42000700 1 -- 172.21.15.44:0/497666062 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7f3f480b25e0 con 0x55ca00e2a4d0 2024-05-27T09:41:12.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.252+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/497666062 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f480adfc0 con 0x55ca00e2a4d0 2024-05-27T09:41:12.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.253+0000 7f3f42000700 1 -- 172.21.15.44:0/497666062 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3f4808d040 con 0x55ca00e2a4d0 2024-05-27T09:41:12.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.253+0000 7f3f6dcb9b80 10 client.26269 fetch_fsmap learned FSMap version 27 2024-05-27T09:41:12.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.253+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/497666062 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ca00eaf5d0 con 0x55ca00e2a4d0 2024-05-27T09:41:12.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.253+0000 7f3f42000700 1 -- 172.21.15.44:0/497666062 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f480adfc0 con 0x55ca00e2a4d0 2024-05-27T09:41:12.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.253+0000 7f3f42000700 1 -- 172.21.15.44:0/497666062 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7f3f480a88e0 con 0x55ca00e2a4d0 2024-05-27T09:41:12.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.254+0000 7f3f6dcb9b80 10 client.26269 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T09:41:12.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.254+0000 7f3f6dcb9b80 10 client.26269 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:41:12.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.254+0000 7f3f6dcb9b80 20 client.26269 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:12.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.254+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ca00eaf850 0x55ca00eb1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:12.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.254+0000 7f3f6dcb9b80 4 client.26269 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-05-27T09:41:12.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.254+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/497666062 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x55ca00d53700 con 0x55ca00eaf850 2024-05-27T09:41:12.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.254+0000 7f3f57400700 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ca00eaf850 0x55ca00eb1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:12.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.255+0000 7f3f42000700 10 client.26269 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:12.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:12.255+0000 7f3f57400700 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ca00eaf850 0x55ca00eb1c70 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f685a07723c11667 server_cookie=670c91a3b82a75b5 in_seq=0 out_seq=0 2024-05-27T09:41:12.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:11 smithi050 ceph-mon[36371]: pgmap v181: 97 pgs: 97 active+clean; 14 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1.5 MiB/s wr, 329 op/s 2024-05-27T09:41:12.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:11 smithi189 ceph-mon[36671]: pgmap v181: 97 pgs: 97 active+clean; 14 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1.5 MiB/s wr, 329 op/s 2024-05-27T09:41:12.693 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:41:12.694 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:41:13.210 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:41:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:13 smithi044 ceph-mon[31618]: pgmap v182: 97 pgs: 97 active+clean; 14 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.5 MiB/s wr, 326 op/s 2024-05-27T09:41:14.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:13 smithi050 ceph-mon[36371]: pgmap v182: 97 pgs: 97 active+clean; 14 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.5 MiB/s wr, 326 op/s 2024-05-27T09:41:14.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:13 smithi189 ceph-mon[36671]: pgmap v182: 97 pgs: 97 active+clean; 14 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.5 MiB/s wr, 326 op/s 2024-05-27T09:41:15.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.769+0000 7efe39e00700 1 -- 172.21.15.44:0/553919505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe34105e60 msgr2=0x7efe3410e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:15.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.769+0000 7efe39e00700 1 --2- 172.21.15.44:0/553919505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe34105e60 0x7efe3410e3b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7efe200098f0 tx=0x7efe20009c00 comp rx=0 tx=0).stop 2024-05-27T09:41:15.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.770+0000 7efe39e00700 1 -- 172.21.15.44:0/553919505 shutdown_connections 2024-05-27T09:41:15.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.770+0000 7efe39e00700 1 --2- 172.21.15.44:0/553919505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe3410e8f0 0x7efe34110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.770+0000 7efe39e00700 1 --2- 172.21.15.44:0/553919505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe34105510 0x7efe34105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.770+0000 7efe39e00700 1 --2- 172.21.15.44:0/553919505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe34105e60 0x7efe3410e3b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.770+0000 7efe39e00700 1 -- 172.21.15.44:0/553919505 >> 172.21.15.44:0/553919505 conn(0x7efe340fedb0 msgr2=0x7efe341011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:15.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.770+0000 7efe39e00700 1 -- 172.21.15.44:0/553919505 shutdown_connections 2024-05-27T09:41:15.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.770+0000 7efe39e00700 1 -- 172.21.15.44:0/553919505 wait complete. 2024-05-27T09:41:15.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.771+0000 7efe39e00700 1 Processor -- start 2024-05-27T09:41:15.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.772+0000 7efe39e00700 1 -- start start 2024-05-27T09:41:15.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.772+0000 7efe39e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe34105510 0x7efe3419d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.772+0000 7efe39e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe34105e60 0x7efe3419d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.772+0000 7efe39e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe3410e8f0 0x7efe341a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.772+0000 7efe39e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efe34114480 con 0x7efe3410e8f0 2024-05-27T09:41:15.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.772+0000 7efe39e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efe34114300 con 0x7efe34105e60 2024-05-27T09:41:15.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.772+0000 7efe39e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe34114600 con 0x7efe34105510 2024-05-27T09:41:15.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.773+0000 7efe32a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe34105510 0x7efe3419d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.773+0000 7efe32000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe34105e60 0x7efe3419d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.773+0000 7efe32000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe34105e60 0x7efe3419d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43578/0 (socket says 172.21.15.44:43578) 2024-05-27T09:41:15.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.773+0000 7efe32000700 1 -- 172.21.15.44:0/1003226648 learned_addr learned my addr 172.21.15.44:0/1003226648 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:15.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.773+0000 7efe33400700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe3410e8f0 0x7efe341a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.773+0000 7efe32000700 1 -- 172.21.15.44:0/1003226648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe34105510 msgr2=0x7efe3419d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:15.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.773+0000 7efe32000700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe34105510 0x7efe3419d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.773+0000 7efe32000700 1 -- 172.21.15.44:0/1003226648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe3410e8f0 msgr2=0x7efe341a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:15.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.773+0000 7efe32000700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe3410e8f0 0x7efe341a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.773+0000 7efe32000700 1 -- 172.21.15.44:0/1003226648 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe341a1fc0 con 0x7efe34105e60 2024-05-27T09:41:15.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.773+0000 7efe33400700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe3410e8f0 0x7efe341a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:41:15.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.774+0000 7efe32000700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe34105e60 0x7efe3419d690 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efe20009fd0 tx=0x7efe20005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:15.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.774+0000 7efe38a00700 1 -- 172.21.15.44:0/1003226648 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20027070 con 0x7efe34105e60 2024-05-27T09:41:15.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.774+0000 7efe38a00700 1 -- 172.21.15.44:0/1003226648 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe2000b560 con 0x7efe34105e60 2024-05-27T09:41:15.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.774+0000 7efe38a00700 1 -- 172.21.15.44:0/1003226648 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20004930 con 0x7efe34105e60 2024-05-27T09:41:15.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.775+0000 7efe39e00700 1 -- 172.21.15.44:0/1003226648 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe341a2250 con 0x7efe34105e60 2024-05-27T09:41:15.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.775+0000 7efe39e00700 1 -- 172.21.15.44:0/1003226648 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe341a26e0 con 0x7efe34105e60 2024-05-27T09:41:15.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.780+0000 7efe38a00700 1 -- 172.21.15.44:0/1003226648 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7efe2002a070 con 0x7efe34105e60 2024-05-27T09:41:15.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.783+0000 7efe48a1bb80 1 Processor -- start 2024-05-27T09:41:15.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.783+0000 7efe38a00700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe180b29f0 0x7efe180b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.783+0000 7efe38a00700 1 -- 172.21.15.44:0/1003226648 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7efe200e3160 con 0x7efe34105e60 2024-05-27T09:41:15.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.784+0000 7efe48a1bb80 1 -- start start 2024-05-27T09:41:15.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.784+0000 7efe33400700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe180b29f0 0x7efe180b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.784+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b74cec0 0x558f7b8da050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.784+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.784+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8d4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.784+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558f7b7f9ec0 con 0x558f7b74cec0 2024-05-27T09:41:15.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.784+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558f7b7e9a60 con 0x558f7b8da590 2024-05-27T09:41:15.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.784+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558f7b7edaa0 con 0x558f7b8d4640 2024-05-27T09:41:15.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.785+0000 7efe32000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.785+0000 7efe32000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8d4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43590/0 (socket says 172.21.15.44:43590) 2024-05-27T09:41:15.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.785+0000 7efe32000700 1 -- 172.21.15.44:0/2586345467 learned_addr learned my addr 172.21.15.44:0/2586345467 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:15.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.785+0000 7efe32a00700 1 --2- 172.21.15.44:0/2586345467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b74cec0 0x558f7b8da050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.785+0000 7efe32000700 1 -- 172.21.15.44:0/2586345467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 msgr2=0x558f7b8d4a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:41:15.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.786+0000 7efe33400700 1 --2- 172.21.15.44:0/2586345467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8d4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.786+0000 7efe32000700 1 --2- 172.21.15.44:0/2586345467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8d4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.786+0000 7efe32000700 1 -- 172.21.15.44:0/2586345467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b74cec0 msgr2=0x558f7b8da050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:15.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.786+0000 7efe32000700 1 --2- 172.21.15.44:0/2586345467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b74cec0 0x558f7b8da050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.787+0000 7efe32000700 1 -- 172.21.15.44:0/2586345467 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f7b7b24a0 con 0x558f7b8da590 2024-05-27T09:41:15.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.787+0000 7efe32a00700 1 --2- 172.21.15.44:0/2586345467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b74cec0 0x558f7b8da050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:15.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.787+0000 7efe32000700 1 --2- 172.21.15.44:0/2586345467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8d4100 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efe200aa1a0 tx=0x7efe200aaba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=561840b14c1ee6b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:15.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.787+0000 7efe33400700 1 --2- 172.21.15.44:0/2586345467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8d4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:41:15.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.787+0000 7efe33400700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe180b29f0 0x7efe180b4f00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efe2c009db0 tx=0x7efe2c0077e0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:15.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.787+0000 7efe28c00700 1 -- 172.21.15.44:0/2586345467 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe200ae850 con 0x558f7b8da590 2024-05-27T09:41:15.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.788+0000 7efe28c00700 1 -- 172.21.15.44:0/2586345467 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe200b3270 con 0x558f7b8da590 2024-05-27T09:41:15.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.788+0000 7efe28c00700 1 -- 172.21.15.44:0/2586345467 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe200aa700 con 0x558f7b8da590 2024-05-27T09:41:15.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.789+0000 7efe48a1bb80 1 -- 172.21.15.44:0/2586345467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 msgr2=0x558f7b8d4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:15.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.789+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/2586345467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8d4100 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efe200aa1a0 tx=0x7efe200aaba0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.789+0000 7efe48a1bb80 1 -- 172.21.15.44:0/2586345467 shutdown_connections 2024-05-27T09:41:15.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.789+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/2586345467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8d4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.789+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/2586345467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b74cec0 0x558f7b8da050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.789+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/2586345467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8d4100 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.789+0000 7efe48a1bb80 1 -- 172.21.15.44:0/2586345467 >> 172.21.15.44:0/2586345467 conn(0x558f7b7ad040 msgr2=0x558f7b8c6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:15.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.790+0000 7efe48a1bb80 1 -- 172.21.15.44:0/2586345467 shutdown_connections 2024-05-27T09:41:15.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.790+0000 7efe48a1bb80 1 -- 172.21.15.44:0/2586345467 wait complete. 2024-05-27T09:41:15.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.791+0000 7efe48a1bb80 1 -- start start 2024-05-27T09:41:15.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.791+0000 7efe48a1bb80 1 Processor -- start 2024-05-27T09:41:15.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.792+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.792+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.792+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8da590 0x558f7b8cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.792+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558f7b6f0020 con 0x558f7b8da590 2024-05-27T09:41:15.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.792+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558f7b78a7e0 con 0x558f7b74cec0 2024-05-27T09:41:15.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.792+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558f7b8cb8f0 con 0x558f7b8d4640 2024-05-27T09:41:15.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.792+0000 7efe48a1bb80 10 client.0 _ll_register_callbacks cb 0x558f7b793cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:15.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.793+0000 7efe32a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.793+0000 7efe32a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43594/0 (socket says 172.21.15.44:43594) 2024-05-27T09:41:15.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.793+0000 7efe32a00700 1 -- 172.21.15.44:0/835345450 learned_addr learned my addr 172.21.15.44:0/835345450 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:15.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.793+0000 7efe32000700 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.793+0000 7efe33400700 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8da590 0x558f7b8cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.794+0000 7efe32a00700 1 -- 172.21.15.44:0/835345450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 msgr2=0x558f7b8cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:15.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.794+0000 7efe32a00700 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8cf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.794+0000 7efe32a00700 1 -- 172.21.15.44:0/835345450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8da590 msgr2=0x558f7b8cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:15.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.794+0000 7efe32a00700 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8da590 0x558f7b8cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.794+0000 7efe32a00700 1 -- 172.21.15.44:0/835345450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f7b739950 con 0x558f7b74cec0 2024-05-27T09:41:15.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.794+0000 7efe32000700 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:15.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.794+0000 7efe32a00700 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3e70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efe240129c0 tx=0x7efe24012cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adb7d8ff4b3237bd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:15.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.794+0000 7efe1d600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:15.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.795+0000 7efe1d600700 1 -- 172.21.15.44:0/835345450 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe24004020 con 0x558f7b74cec0 2024-05-27T09:41:15.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.795+0000 7efe1d600700 1 -- 172.21.15.44:0/835345450 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7efe240041c0 con 0x558f7b74cec0 2024-05-27T09:41:15.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.795+0000 7efe48a1bb80 1 -- 172.21.15.44:0/835345450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe200b3270 con 0x558f7b74cec0 2024-05-27T09:41:15.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.795+0000 7efe1d600700 1 -- 172.21.15.44:0/835345450 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efe24004400 con 0x558f7b74cec0 2024-05-27T09:41:15.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.795+0000 7efe48a1bb80 10 client.26273 fetch_fsmap learned FSMap version 27 2024-05-27T09:41:15.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.796+0000 7efe48a1bb80 1 -- 172.21.15.44:0/835345450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558f7b7ba0e0 con 0x558f7b74cec0 2024-05-27T09:41:15.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.796+0000 7efe1d600700 1 -- 172.21.15.44:0/835345450 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe200b3270 con 0x558f7b74cec0 2024-05-27T09:41:15.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.796+0000 7efe1d600700 1 -- 172.21.15.44:0/835345450 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7efe240039f0 con 0x558f7b74cec0 2024-05-27T09:41:15.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.796+0000 7efe48a1bb80 10 client.26273 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T09:41:15.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.796+0000 7efe48a1bb80 10 client.26273 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:15.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.796+0000 7efe48a1bb80 20 client.26273 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:15.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.796+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f7b8e55b0 0x558f7b8e79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.796+0000 7efe48a1bb80 4 client.26273 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:15.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.796+0000 7efe48a1bb80 1 -- 172.21.15.44:0/835345450 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f7b7fbe10 con 0x558f7b8e55b0 2024-05-27T09:41:15.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.798+0000 7efe32000700 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f7b8e55b0 0x558f7b8e79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.799+0000 7efe32000700 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f7b8e55b0 0x558f7b8e79d0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=231a51ad1d67ca50 server_cookie=dd08c8eb635e09ca in_seq=0 out_seq=0 2024-05-27T09:41:15.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.799+0000 7efe1d600700 10 client.26273 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:15.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.801+0000 7efe1d600700 1 -- 172.21.15.44:0/835345450 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558f7b7fbe10 con 0x558f7b8e55b0 2024-05-27T09:41:15.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.801+0000 7efe1d600700 10 client.26273 handle_command_reply: tid=0 2024-05-27T09:41:15.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.801+0000 7efe48a1bb80 1 client.26273 shutdown 2024-05-27T09:41:15.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe48a1bb80 1 -- 172.21.15.44:0/835345450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 msgr2=0x558f7b8d3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:15.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3e70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efe240129c0 tx=0x7efe24012cd0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe48a1bb80 1 -- 172.21.15.44:0/835345450 shutdown_connections 2024-05-27T09:41:15.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f7b8e55b0 0x558f7b8e79d0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8da590 0x558f7b8cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3e70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/835345450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe48a1bb80 1 -- 172.21.15.44:0/835345450 >> 172.21.15.44:0/835345450 conn(0x558f7b7ad040 msgr2=0x558f7b8c6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:15.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe1d600700 0 client.26273 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:15.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe48a1bb80 1 -- 172.21.15.44:0/835345450 shutdown_connections 2024-05-27T09:41:15.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe48a1bb80 1 -- 172.21.15.44:0/835345450 wait complete. 2024-05-27T09:41:15.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.802+0000 7efe48a1bb80 20 client.26273 trim_cache size 0 max 16384 2024-05-27T09:41:15.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.840+0000 7efe48a1bb80 1 Processor -- start 2024-05-27T09:41:15.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.840+0000 7efe48a1bb80 1 -- start start 2024-05-27T09:41:15.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.840+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.840+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8cb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.841+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8da590 0x558f7b8ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.841+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efe341af470 con 0x558f7b8da590 2024-05-27T09:41:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.841+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efe341acd00 con 0x558f7b74cec0 2024-05-27T09:41:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.841+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe3410bd00 con 0x558f7b8d4640 2024-05-27T09:41:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.841+0000 7efe32a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.841+0000 7efe32a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43608/0 (socket says 172.21.15.44:43608) 2024-05-27T09:41:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.841+0000 7efe32a00700 1 -- 172.21.15.44:0/1254724084 learned_addr learned my addr 172.21.15.44:0/1254724084 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.841+0000 7efe33400700 1 --2- 172.21.15.44:0/1254724084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8da590 0x558f7b8ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.841+0000 7efe32000700 1 --2- 172.21.15.44:0/1254724084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.842+0000 7efe32a00700 1 -- 172.21.15.44:0/1254724084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 msgr2=0x558f7b8cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.842+0000 7efe32a00700 1 --2- 172.21.15.44:0/1254724084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8cb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.842+0000 7efe32a00700 1 -- 172.21.15.44:0/1254724084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8da590 msgr2=0x558f7b8ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.842+0000 7efe32a00700 1 --2- 172.21.15.44:0/1254724084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8da590 0x558f7b8ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.842+0000 7efe32a00700 1 -- 172.21.15.44:0/1254724084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f7b8e1d70 con 0x558f7b74cec0 2024-05-27T09:41:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.842+0000 7efe32a00700 1 --2- 172.21.15.44:0/1254724084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7efe24003e60 tx=0x7efe2401c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d61a4f3b5a29c5ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:15.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.844+0000 7efe1fe00700 1 -- 172.21.15.44:0/1254724084 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe24029040 con 0x558f7b74cec0 2024-05-27T09:41:15.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.844+0000 7efe1fe00700 1 -- 172.21.15.44:0/1254724084 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe24003b30 con 0x558f7b74cec0 2024-05-27T09:41:15.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.844+0000 7efe1fe00700 1 -- 172.21.15.44:0/1254724084 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe24005030 con 0x558f7b74cec0 2024-05-27T09:41:15.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.845+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1254724084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 msgr2=0x558f7b8d3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:15.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.845+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/1254724084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7efe24003e60 tx=0x7efe2401c000 comp rx=0 tx=0).stop 2024-05-27T09:41:15.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.845+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1254724084 shutdown_connections 2024-05-27T09:41:15.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.845+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/1254724084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8da590 0x558f7b8ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.845+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/1254724084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b74cec0 0x558f7b8d3680 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.845+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/1254724084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b8d4640 0x558f7b8cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.845+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1254724084 >> 172.21.15.44:0/1254724084 conn(0x558f7b7ad040 msgr2=0x558f7b8cf620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:15.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.845+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1254724084 shutdown_connections 2024-05-27T09:41:15.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.846+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1254724084 wait complete. 2024-05-27T09:41:15.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.846+0000 7efe48a1bb80 1 -- start start 2024-05-27T09:41:15.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.847+0000 7efe48a1bb80 1 Processor -- start 2024-05-27T09:41:15.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.847+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b74cec0 0x558f7b8e12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.847+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8d4640 0x558f7b8e1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.848+0000 7efe48a1bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8dd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.848+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efe341ab1c0 con 0x558f7b8d4640 2024-05-27T09:41:15.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.848+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efe341ac630 con 0x558f7b8da590 2024-05-27T09:41:15.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.848+0000 7efe48a1bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe341ab890 con 0x558f7b74cec0 2024-05-27T09:41:15.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.848+0000 7efe48a1bb80 10 client.0 _ll_register_callbacks cb 0x558f7b793cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.848+0000 7efe33400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8dd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.848+0000 7efe32000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8d4640 0x558f7b8e1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.848+0000 7efe33400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8dd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43618/0 (socket says 172.21.15.44:43618) 2024-05-27T09:41:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.848+0000 7efe33400700 1 -- 172.21.15.44:0/1051992056 learned_addr learned my addr 172.21.15.44:0/1051992056 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.849+0000 7efe32a00700 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b74cec0 0x558f7b8e12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.849+0000 7efe33400700 1 -- 172.21.15.44:0/1051992056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b74cec0 msgr2=0x558f7b8e12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.849+0000 7efe33400700 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b74cec0 0x558f7b8e12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.849+0000 7efe33400700 1 -- 172.21.15.44:0/1051992056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8d4640 msgr2=0x558f7b8e1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:15.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.849+0000 7efe33400700 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8d4640 0x558f7b8e1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.849+0000 7efe33400700 1 -- 172.21.15.44:0/1051992056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f7b8ddf80 con 0x558f7b8da590 2024-05-27T09:41:15.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.849+0000 7efe32000700 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8d4640 0x558f7b8e1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:15.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.849+0000 7efe33400700 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8dd7e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7efe2c017f80 tx=0x7efe2c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b0cd1c4210cf02c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:15.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.849+0000 7efe1d600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:15.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.850+0000 7efe1d600700 1 -- 172.21.15.44:0/1051992056 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2c029060 con 0x558f7b8da590 2024-05-27T09:41:15.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.850+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1051992056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe24003b30 con 0x558f7b8da590 2024-05-27T09:41:15.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.850+0000 7efe1d600700 1 -- 172.21.15.44:0/1051992056 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7efe2c0243f0 con 0x558f7b8da590 2024-05-27T09:41:15.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.850+0000 7efe1d600700 1 -- 172.21.15.44:0/1051992056 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efe2c0245f0 con 0x558f7b8da590 2024-05-27T09:41:15.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.850+0000 7efe48a1bb80 10 client.26279 fetch_fsmap learned FSMap version 27 2024-05-27T09:41:15.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.850+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1051992056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558f7b950c50 con 0x558f7b8da590 2024-05-27T09:41:15.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.851+0000 7efe1d600700 1 -- 172.21.15.44:0/1051992056 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe24003b30 con 0x558f7b8da590 2024-05-27T09:41:15.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.852+0000 7efe1d600700 1 -- 172.21.15.44:0/1051992056 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7efe2c023070 con 0x558f7b8da590 2024-05-27T09:41:15.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.853+0000 7efe48a1bb80 10 client.26279 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T09:41:15.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.853+0000 7efe48a1bb80 10 client.26279 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:15.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.853+0000 7efe48a1bb80 20 client.26279 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:15.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.853+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f7b951310 0x558f7b953730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:15.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.853+0000 7efe48a1bb80 4 client.26279 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:41:15.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.853+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1051992056 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558f7b7f55f0 con 0x558f7b951310 2024-05-27T09:41:15.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.853+0000 7efe32a00700 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f7b951310 0x558f7b953730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:15.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.854+0000 7efe1d600700 10 client.26279 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:15.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.854+0000 7efe32a00700 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f7b951310 0x558f7b953730 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46f0fe91d3f1a8d server_cookie=fe632d9449020180 in_seq=0 out_seq=0 2024-05-27T09:41:15.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.862+0000 7efe1d600700 1 -- 172.21.15.44:0/1051992056 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558f7b7f55f0 con 0x558f7b951310 2024-05-27T09:41:15.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.862+0000 7efe1d600700 10 client.26279 handle_command_reply: tid=0 2024-05-27T09:41:15.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.862+0000 7efe48a1bb80 1 client.26279 shutdown 2024-05-27T09:41:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.863+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1051992056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 msgr2=0x558f7b8dd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.863+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8dd7e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7efe2c017f80 tx=0x7efe2c004b40 comp rx=0 tx=0).stop 2024-05-27T09:41:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.863+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1051992056 shutdown_connections 2024-05-27T09:41:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.863+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f7b951310 0x558f7b953730 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.863+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f7b8da590 0x558f7b8dd7e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.863+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f7b74cec0 0x558f7b8e12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.863+0000 7efe48a1bb80 1 --2- 172.21.15.44:0/1051992056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f7b8d4640 0x558f7b8e1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.863+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1051992056 >> 172.21.15.44:0/1051992056 conn(0x558f7b7ad040 msgr2=0x558f7b8cf620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.863+0000 7efe1d600700 0 client.26279 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.863+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1051992056 shutdown_connections 2024-05-27T09:41:15.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.863+0000 7efe48a1bb80 1 -- 172.21.15.44:0/1051992056 wait complete. 2024-05-27T09:41:15.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.864+0000 7efe48a1bb80 20 client.26279 trim_cache size 0 max 16384 2024-05-27T09:41:15.866 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:41:15.866 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:41:15.866 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "67e722f3-2161-48d5-aae5-e2b39f69565f", 2024-05-27T09:41:15.866 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:41:15.866 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:41:15.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.866+0000 7efe39e00700 1 -- 172.21.15.44:0/1003226648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe180b29f0 msgr2=0x7efe180b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:15.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.866+0000 7efe39e00700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe180b29f0 0x7efe180b4f00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efe2c009db0 tx=0x7efe2c0077e0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.866+0000 7efe39e00700 1 -- 172.21.15.44:0/1003226648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe34105e60 msgr2=0x7efe3419d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:15.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.866+0000 7efe39e00700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe34105e60 0x7efe3419d690 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efe20009fd0 tx=0x7efe20005e40 comp rx=0 tx=0).stop 2024-05-27T09:41:15.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.866+0000 7efe39e00700 1 -- 172.21.15.44:0/1003226648 shutdown_connections 2024-05-27T09:41:15.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.866+0000 7efe39e00700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe180b29f0 0x7efe180b4f00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.866+0000 7efe39e00700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe3410e8f0 0x7efe341a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.866+0000 7efe39e00700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe34105510 0x7efe3419d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.866+0000 7efe39e00700 1 --2- 172.21.15.44:0/1003226648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe34105e60 0x7efe3419d690 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:15.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.866+0000 7efe39e00700 1 -- 172.21.15.44:0/1003226648 >> 172.21.15.44:0/1003226648 conn(0x7efe340fedb0 msgr2=0x7efe34103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:15.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.867+0000 7efe39e00700 1 -- 172.21.15.44:0/1003226648 shutdown_connections 2024-05-27T09:41:15.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:15.867+0000 7efe39e00700 1 -- 172.21.15.44:0/1003226648 wait complete. 2024-05-27T09:41:16.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:15 smithi044 ceph-mon[31618]: pgmap v183: 97 pgs: 97 active+clean; 14 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.5 MiB/s wr, 326 op/s 2024-05-27T09:41:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:15 smithi050 ceph-mon[36371]: pgmap v183: 97 pgs: 97 active+clean; 14 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.5 MiB/s wr, 326 op/s 2024-05-27T09:41:16.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:15 smithi189 ceph-mon[36671]: pgmap v183: 97 pgs: 97 active+clean; 14 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.5 MiB/s wr, 326 op/s 2024-05-27T09:41:16.895 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:41:17.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:16 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:41:17.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:16 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:41:17.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:16 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:41:17.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:16 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:41:17.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:16 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:41:17.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:16 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:41:17.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:16 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:41:17.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:16 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:41:17.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:16 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:41:17.507 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:41:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.121+0000 7ff262c00640 1 -- 172.21.15.44:0/418852213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c109ab0 msgr2=0x7ff25c1036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.121+0000 7ff262c00640 1 --2- 172.21.15.44:0/418852213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c109ab0 0x7ff25c1036a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff250009900 tx=0x7ff25002a3f0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.121+0000 7ff262c00640 1 -- 172.21.15.44:0/418852213 shutdown_connections 2024-05-27T09:41:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.121+0000 7ff262c00640 1 --2- 172.21.15.44:0/418852213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c104250 0x7ff25c1105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.121+0000 7ff262c00640 1 --2- 172.21.15.44:0/418852213 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff25c109f50 0x7ff25c103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.121+0000 7ff262c00640 1 --2- 172.21.15.44:0/418852213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c109ab0 0x7ff25c1036a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.121+0000 7ff262c00640 1 -- 172.21.15.44:0/418852213 >> 172.21.15.44:0/418852213 conn(0x7ff25c079240 msgr2=0x7ff25c079680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.121+0000 7ff262c00640 1 -- 172.21.15.44:0/418852213 shutdown_connections 2024-05-27T09:41:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.121+0000 7ff262c00640 1 -- 172.21.15.44:0/418852213 wait complete. 2024-05-27T09:41:18.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.123+0000 7ff262c00640 1 Processor -- start 2024-05-27T09:41:18.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.124+0000 7ff262c00640 1 -- start start 2024-05-27T09:41:18.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff262c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c104250 0x7ff25c074950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff262c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c109ab0 0x7ff25c06da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff262c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff25c109f50 0x7ff25c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff262c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c113cf0 con 0x7ff25c109f50 2024-05-27T09:41:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff262c00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c113b70 con 0x7ff25c109ab0 2024-05-27T09:41:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff262c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c113e70 con 0x7ff25c104250 2024-05-27T09:41:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff261800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c104250 0x7ff25c074950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff261800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c104250 0x7ff25c074950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53990/0 (socket says 172.21.15.44:53990) 2024-05-27T09:41:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff261800640 1 -- 172.21.15.44:0/1717803235 learned_addr learned my addr 172.21.15.44:0/1717803235 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff261800640 1 -- 172.21.15.44:0/1717803235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c109ab0 msgr2=0x7ff25c06da70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:41:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff261800640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c109ab0 0x7ff25c06da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff261800640 1 -- 172.21.15.44:0/1717803235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff25c109f50 msgr2=0x7ff25c06e110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:41:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.125+0000 7ff261800640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff25c109f50 0x7ff25c06e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.126+0000 7ff261800640 1 -- 172.21.15.44:0/1717803235 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25c072470 con 0x7ff25c104250 2024-05-27T09:41:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.126+0000 7ff261800640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c104250 0x7ff25c074950 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff250005ab0 tx=0x7ff2500035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.126+0000 7ff25a000640 1 -- 172.21.15.44:0/1717803235 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25004d070 con 0x7ff25c104250 2024-05-27T09:41:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.126+0000 7ff25a000640 1 -- 172.21.15.44:0/1717803235 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff250003bf0 con 0x7ff25c104250 2024-05-27T09:41:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.126+0000 7ff25a000640 1 -- 172.21.15.44:0/1717803235 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff250035070 con 0x7ff25c104250 2024-05-27T09:41:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.127+0000 7ff262c00640 1 -- 172.21.15.44:0/1717803235 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff25c072700 con 0x7ff25c104250 2024-05-27T09:41:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.127+0000 7ff262c00640 1 -- 172.21.15.44:0/1717803235 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff25c072ba0 con 0x7ff25c104250 2024-05-27T09:41:18.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.132+0000 7ff26b191740 1 Processor -- start 2024-05-27T09:41:18.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.133+0000 7ff26b191740 1 -- start start 2024-05-27T09:41:18.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.133+0000 7ff26b191740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225c121b0 0x55d225fb4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.133+0000 7ff26b191740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225be92f0 0x55d225fb4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.133+0000 7ff26b191740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 0x55d225fb54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.135 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.134+0000 7ff262200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225c121b0 0x55d225fb4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.135 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.134+0000 7ff262200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225c121b0 0x55d225fb4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39946/0 (socket says 172.21.15.44:39946) 2024-05-27T09:41:18.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.135+0000 7ff262200640 1 -- 172.21.15.44:0/4091715002 learned_addr learned my addr 172.21.15.44:0/4091715002 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:18.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.135+0000 7ff26b191740 1 -- 172.21.15.44:0/4091715002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d225f25b90 con 0x55d225c121b0 2024-05-27T09:41:18.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.135+0000 7ff26b191740 1 -- 172.21.15.44:0/4091715002 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d225f15730 con 0x55d225be92f0 2024-05-27T09:41:18.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.135+0000 7ff26b191740 1 -- 172.21.15.44:0/4091715002 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d225f19770 con 0x55d225bf14d0 2024-05-27T09:41:18.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.135+0000 7ff260e00640 1 --2- 172.21.15.44:0/4091715002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 0x55d225fb54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.136+0000 7ff260e00640 1 -- 172.21.15.44:0/4091715002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225be92f0 msgr2=0x55d225fb4fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:41:18.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.136+0000 7ff260e00640 1 --2- 172.21.15.44:0/4091715002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225be92f0 0x55d225fb4fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.136+0000 7ff260e00640 1 -- 172.21.15.44:0/4091715002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225c121b0 msgr2=0x55d225fb4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:18.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.136+0000 7ff260e00640 1 --2- 172.21.15.44:0/4091715002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225c121b0 0x55d225fb4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.138 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.136+0000 7ff260e00640 1 -- 172.21.15.44:0/4091715002 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d225eae8b0 con 0x55d225bf14d0 2024-05-27T09:41:18.138 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.136+0000 7ff262200640 1 --2- 172.21.15.44:0/4091715002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225c121b0 0x55d225fb4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:18.138 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.137+0000 7ff25a000640 1 -- 172.21.15.44:0/1717803235 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff2500319e0 con 0x7ff25c104250 2024-05-27T09:41:18.138 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.137+0000 7ff260e00640 1 --2- 172.21.15.44:0/4091715002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 0x55d225fb54e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff25c06c390 tx=0x7ff248009e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=354819674bd8085f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.137+0000 7ff24ea00640 1 -- 172.21.15.44:0/4091715002 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24800eb70 con 0x55d225bf14d0 2024-05-27T09:41:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.137+0000 7ff24ea00640 1 -- 172.21.15.44:0/4091715002 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff24800ed10 con 0x55d225bf14d0 2024-05-27T09:41:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.137+0000 7ff24ea00640 1 -- 172.21.15.44:0/4091715002 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2480187a0 con 0x55d225bf14d0 2024-05-27T09:41:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.139+0000 7ff26b191740 1 -- 172.21.15.44:0/4091715002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 msgr2=0x55d225fb54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.139+0000 7ff26b191740 1 --2- 172.21.15.44:0/4091715002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 0x55d225fb54e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff25c06c390 tx=0x7ff248009e70 comp rx=0 tx=0).stop 2024-05-27T09:41:18.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.139+0000 7ff26b191740 1 -- 172.21.15.44:0/4091715002 shutdown_connections 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.139+0000 7ff26b191740 1 --2- 172.21.15.44:0/4091715002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 0x55d225fb54e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.139+0000 7ff26b191740 1 --2- 172.21.15.44:0/4091715002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225be92f0 0x55d225fb4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.139+0000 7ff26b191740 1 --2- 172.21.15.44:0/4091715002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225c121b0 0x55d225fb4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.139+0000 7ff26b191740 1 -- 172.21.15.44:0/4091715002 >> 172.21.15.44:0/4091715002 conn(0x55d225c06300 msgr2=0x55d225fb1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.139+0000 7ff26b191740 1 -- 172.21.15.44:0/4091715002 shutdown_connections 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.139+0000 7ff26b191740 1 -- 172.21.15.44:0/4091715002 wait complete. 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.140+0000 7ff26b191740 1 -- start start 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.140+0000 7ff25a000640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2440c39c0 0x7ff2440c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.140+0000 7ff26b191740 1 Processor -- start 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.140+0000 7ff25a000640 1 -- 172.21.15.44:0/1717803235 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff250101eb0 con 0x7ff25c104250 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff262200640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2440c39c0 0x7ff2440c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff26b191740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 0x55d225fbebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff26b191740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 0x55d225fbf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff26b191740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 0x55d225fcaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff26b191740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d225bfb790 con 0x55d225be92f0 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff26b191740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d225ebdbc0 con 0x55d225c121b0 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff26b191740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d225fc1b90 con 0x55d225bf14d0 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff26b191740 10 client.0 _ll_register_callbacks cb 0x55d225e9b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff260e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 0x55d225fbf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff260e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 0x55d225fbf120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53998/0 (socket says 172.21.15.44:53998) 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff261800640 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 0x55d225fbebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.142+0000 7ff262200640 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 0x55d225fcaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.141+0000 7ff260e00640 1 -- 172.21.15.44:0/176912273 learned_addr learned my addr 172.21.15.44:0/176912273 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.142+0000 7ff262200640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2440c39c0 0x7ff2440c5ed0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff2540049a0 tx=0x7ff254009500 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.142+0000 7ff261800640 1 -- 172.21.15.44:0/176912273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 msgr2=0x55d225fbf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.142+0000 7ff261800640 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 0x55d225fbf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.142+0000 7ff261800640 1 -- 172.21.15.44:0/176912273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 msgr2=0x55d225fcaec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.142+0000 7ff261800640 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 0x55d225fcaec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.142+0000 7ff261800640 1 -- 172.21.15.44:0/176912273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d225c2a210 con 0x55d225be92f0 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.142+0000 7ff261800640 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 0x55d225fbebe0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff2500ff5d0 tx=0x7ff2500c8960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1986accb4b5a7129 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.142+0000 7ff243400640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.142+0000 7ff243400640 1 -- 172.21.15.44:0/176912273 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2500c91f0 con 0x55d225be92f0 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.143+0000 7ff243400640 1 -- 172.21.15.44:0/176912273 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff2500cd170 con 0x55d225be92f0 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.143+0000 7ff26b191740 1 -- 172.21.15.44:0/176912273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff24800ed10 con 0x55d225be92f0 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.143+0000 7ff243400640 1 -- 172.21.15.44:0/176912273 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff2500cd5a0 con 0x55d225be92f0 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.143+0000 7ff26b191740 10 client.16125 fetch_fsmap learned FSMap version 27 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.143+0000 7ff26b191740 1 -- 172.21.15.44:0/176912273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d225cf6620 con 0x55d225be92f0 2024-05-27T09:41:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.143+0000 7ff243400640 1 -- 172.21.15.44:0/176912273 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff24800ed10 con 0x55d225be92f0 2024-05-27T09:41:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.144+0000 7ff243400640 1 -- 172.21.15.44:0/176912273 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7ff250047030 con 0x55d225be92f0 2024-05-27T09:41:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.144+0000 7ff26b191740 10 client.16125 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T09:41:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.144+0000 7ff26b191740 10 client.16125 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.144+0000 7ff26b191740 20 client.16125 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.144+0000 7ff26b191740 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d225bd0d80 0x55d225fd0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.144+0000 7ff26b191740 4 client.16125 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.144+0000 7ff26b191740 1 -- 172.21.15.44:0/176912273 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d225f27ae0 con 0x55d225bd0d80 2024-05-27T09:41:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.144+0000 7ff260e00640 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d225bd0d80 0x55d225fd0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.145+0000 7ff260e00640 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d225bd0d80 0x55d225fd0e30 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=581476a36ea156b5 server_cookie=5e028b0bf9e4bc7f in_seq=0 out_seq=0 2024-05-27T09:41:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.145+0000 7ff243400640 10 client.16125 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:18.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff243400640 1 -- 172.21.15.44:0/176912273 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d225f27ae0 con 0x55d225bd0d80 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff243400640 10 client.16125 handle_command_reply: tid=0 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 1 client.16125 shutdown 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 1 -- 172.21.15.44:0/176912273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 msgr2=0x55d225fbebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 0x55d225fbebe0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff2500ff5d0 tx=0x7ff2500c8960 comp rx=0 tx=0).stop 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 1 -- 172.21.15.44:0/176912273 shutdown_connections 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d225bd0d80 0x55d225fd0e30 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 0x55d225fcaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225bf14d0 0x55d225fbf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 1 --2- 172.21.15.44:0/176912273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 0x55d225fbebe0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 1 -- 172.21.15.44:0/176912273 >> 172.21.15.44:0/176912273 conn(0x55d225c06300 msgr2=0x55d225fb6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff243400640 0 client.16125 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 1 -- 172.21.15.44:0/176912273 shutdown_connections 2024-05-27T09:41:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 1 -- 172.21.15.44:0/176912273 wait complete. 2024-05-27T09:41:18.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.146+0000 7ff26b191740 20 client.16125 trim_cache size 0 max 16384 2024-05-27T09:41:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff26b191740 1 Processor -- start 2024-05-27T09:41:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff26b191740 1 -- start start 2024-05-27T09:41:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff26b191740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 0x55d225fba3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff26b191740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225bf14d0 0x55d225fb70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff26b191740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225c121b0 0x55d225fc0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff26b191740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b8660 con 0x55d225be92f0 2024-05-27T09:41:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff26b191740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b5f30 con 0x55d225bf14d0 2024-05-27T09:41:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff26b191740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b3da0 con 0x55d225c121b0 2024-05-27T09:41:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff261800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 0x55d225fba3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff262200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225c121b0 0x55d225fc0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff262200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225c121b0 0x55d225fc0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:54012/0 (socket says 172.21.15.44:54012) 2024-05-27T09:41:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff262200640 1 -- 172.21.15.44:0/3357624723 learned_addr learned my addr 172.21.15.44:0/3357624723 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff261800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 0x55d225fba3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39964/0 (socket says 172.21.15.44:39964) 2024-05-27T09:41:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.163+0000 7ff260e00640 1 --2- 172.21.15.44:0/3357624723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225bf14d0 0x55d225fb70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.164+0000 7ff260e00640 1 -- 172.21.15.44:0/3357624723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225c121b0 msgr2=0x55d225fc0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.164+0000 7ff260e00640 1 --2- 172.21.15.44:0/3357624723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225c121b0 0x55d225fc0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.164+0000 7ff260e00640 1 -- 172.21.15.44:0/3357624723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 msgr2=0x55d225fba3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.164+0000 7ff260e00640 1 --2- 172.21.15.44:0/3357624723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 0x55d225fba3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.164+0000 7ff260e00640 1 -- 172.21.15.44:0/3357624723 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d225fc11b0 con 0x55d225bf14d0 2024-05-27T09:41:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.164+0000 7ff260e00640 1 --2- 172.21.15.44:0/3357624723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225bf14d0 0x55d225fb70d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55d225fd15d0 tx=0x7ff248005d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=935064e438e07a34 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.165+0000 7ff24e000640 1 -- 172.21.15.44:0/3357624723 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24800ea70 con 0x55d225bf14d0 2024-05-27T09:41:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.165+0000 7ff24e000640 1 -- 172.21.15.44:0/3357624723 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff24800ec10 con 0x55d225bf14d0 2024-05-27T09:41:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.165+0000 7ff24e000640 1 -- 172.21.15.44:0/3357624723 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248005630 con 0x55d225bf14d0 2024-05-27T09:41:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.167+0000 7ff26b191740 1 -- 172.21.15.44:0/3357624723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225bf14d0 msgr2=0x55d225fb70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.167+0000 7ff26b191740 1 --2- 172.21.15.44:0/3357624723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225bf14d0 0x55d225fb70d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55d225fd15d0 tx=0x7ff248005d70 comp rx=0 tx=0).stop 2024-05-27T09:41:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.167+0000 7ff26b191740 1 -- 172.21.15.44:0/3357624723 shutdown_connections 2024-05-27T09:41:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.167+0000 7ff26b191740 1 --2- 172.21.15.44:0/3357624723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225c121b0 0x55d225fc0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.167+0000 7ff26b191740 1 --2- 172.21.15.44:0/3357624723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225bf14d0 0x55d225fb70d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.167+0000 7ff26b191740 1 --2- 172.21.15.44:0/3357624723 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225be92f0 0x55d225fba3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.167+0000 7ff26b191740 1 -- 172.21.15.44:0/3357624723 >> 172.21.15.44:0/3357624723 conn(0x55d225c06300 msgr2=0x55d225fbe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.168+0000 7ff26b191740 1 -- 172.21.15.44:0/3357624723 shutdown_connections 2024-05-27T09:41:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.168+0000 7ff26b191740 1 -- 172.21.15.44:0/3357624723 wait complete. 2024-05-27T09:41:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.169+0000 7ff26b191740 1 -- start start 2024-05-27T09:41:18.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.169+0000 7ff26b191740 1 Processor -- start 2024-05-27T09:41:18.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff26b191740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225be92f0 0x55d225fc7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff26b191740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225bf14d0 0x55d225fb9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff26b191740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 0x55d225fb9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff26b191740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b43f0 con 0x55d225bf14d0 2024-05-27T09:41:18.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff26b191740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b5860 con 0x55d225c121b0 2024-05-27T09:41:18.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff26b191740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c1b4ac0 con 0x55d225be92f0 2024-05-27T09:41:18.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff26b191740 10 client.0 _ll_register_callbacks cb 0x7ff250005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:18.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff262200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 0x55d225fb9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff262200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 0x55d225fb9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43654/0 (socket says 172.21.15.44:43654) 2024-05-27T09:41:18.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff262200640 1 -- 172.21.15.44:0/2880222689 learned_addr learned my addr 172.21.15.44:0/2880222689 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:18.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff261800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225be92f0 0x55d225fc7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.170+0000 7ff260e00640 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225bf14d0 0x55d225fb9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.171+0000 7ff262200640 1 -- 172.21.15.44:0/2880222689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225be92f0 msgr2=0x55d225fc7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:18.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.171+0000 7ff262200640 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225be92f0 0x55d225fc7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.171+0000 7ff262200640 1 -- 172.21.15.44:0/2880222689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225bf14d0 msgr2=0x55d225fb9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:18.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.171+0000 7ff262200640 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225bf14d0 0x55d225fb9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.171+0000 7ff262200640 1 -- 172.21.15.44:0/2880222689 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d225fb9aa0 con 0x55d225c121b0 2024-05-27T09:41:18.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.171+0000 7ff261800640 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225be92f0 0x55d225fc7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:41:18.173 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.171+0000 7ff262200640 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 0x55d225fb9560 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff25400ef20 tx=0x7ff254004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9692eade34aa25d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:18.173 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.171+0000 7ff243400640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.172+0000 7ff243400640 1 -- 172.21.15.44:0/2880222689 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254004b20 con 0x55d225c121b0 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.172+0000 7ff26b191740 1 -- 172.21.15.44:0/2880222689 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff24800ec10 con 0x55d225c121b0 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.172+0000 7ff243400640 1 -- 172.21.15.44:0/2880222689 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff254009040 con 0x55d225c121b0 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.172+0000 7ff243400640 1 -- 172.21.15.44:0/2880222689 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff254004cc0 con 0x55d225c121b0 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.172+0000 7ff26b191740 10 client.26291 fetch_fsmap learned FSMap version 27 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.172+0000 7ff26b191740 1 -- 172.21.15.44:0/2880222689 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d225fc6b50 con 0x55d225c121b0 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.172+0000 7ff243400640 1 -- 172.21.15.44:0/2880222689 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff24800ec10 con 0x55d225c121b0 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.173+0000 7ff243400640 1 -- 172.21.15.44:0/2880222689 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7ff2540051e0 con 0x55d225c121b0 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.173+0000 7ff26b191740 10 client.26291 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.173+0000 7ff26b191740 10 client.26291 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.173+0000 7ff26b191740 20 client.26291 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:18.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.173+0000 7ff26b191740 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d225ba61c0 0x55d22603b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:18.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.173+0000 7ff26b191740 4 client.26291 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:41:18.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.173+0000 7ff26b191740 1 -- 172.21.15.44:0/2880222689 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d225f212c0 con 0x55d225ba61c0 2024-05-27T09:41:18.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.173+0000 7ff261800640 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d225ba61c0 0x55d22603b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:18.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.174+0000 7ff261800640 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d225ba61c0 0x55d22603b890 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=385f2b86fec96725 server_cookie=d6757cb59cca596f in_seq=0 out_seq=0 2024-05-27T09:41:18.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.174+0000 7ff243400640 10 client.26291 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:18.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:17 smithi044 ceph-mon[31618]: pgmap v184: 97 pgs: 97 active+clean; 21 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 550 op/s 2024-05-27T09:41:18.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:17 smithi050 ceph-mon[36371]: pgmap v184: 97 pgs: 97 active+clean; 21 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 550 op/s 2024-05-27T09:41:18.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:17 smithi189 ceph-mon[36671]: pgmap v184: 97 pgs: 97 active+clean; 21 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 550 op/s 2024-05-27T09:41:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.621+0000 7ff243400640 1 -- 172.21.15.44:0/2880222689 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55d225f212c0 con 0x55d225ba61c0 2024-05-27T09:41:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.621+0000 7ff243400640 10 client.26291 handle_command_reply: tid=0 2024-05-27T09:41:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.621+0000 7ff26b191740 1 client.26291 shutdown 2024-05-27T09:41:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff26b191740 1 -- 172.21.15.44:0/2880222689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 msgr2=0x55d225fb9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff26b191740 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 0x55d225fb9560 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff25400ef20 tx=0x7ff254004ab0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff26b191740 1 -- 172.21.15.44:0/2880222689 shutdown_connections 2024-05-27T09:41:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff26b191740 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d225ba61c0 0x55d22603b890 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff26b191740 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d225c121b0 0x55d225fb9560 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff26b191740 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d225bf14d0 0x55d225fb9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff26b191740 1 --2- 172.21.15.44:0/2880222689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d225be92f0 0x55d225fc7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff26b191740 1 -- 172.21.15.44:0/2880222689 >> 172.21.15.44:0/2880222689 conn(0x55d225c06300 msgr2=0x55d225fbeea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff243400640 0 client.26291 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff26b191740 1 -- 172.21.15.44:0/2880222689 shutdown_connections 2024-05-27T09:41:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff26b191740 1 -- 172.21.15.44:0/2880222689 wait complete. 2024-05-27T09:41:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.622+0000 7ff26b191740 20 client.26291 trim_cache size 0 max 16384 2024-05-27T09:41:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:41:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:41:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:41:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.623+0000 7ff262c00640 1 -- 172.21.15.44:0/1717803235 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2440c39c0 msgr2=0x7ff2440c5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.623+0000 7ff262c00640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2440c39c0 0x7ff2440c5ed0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff2540049a0 tx=0x7ff254009500 comp rx=0 tx=0).stop 2024-05-27T09:41:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.623+0000 7ff262c00640 1 -- 172.21.15.44:0/1717803235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c104250 msgr2=0x7ff25c074950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.623+0000 7ff262c00640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c104250 0x7ff25c074950 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff250005ab0 tx=0x7ff2500035f0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.624+0000 7ff262c00640 1 -- 172.21.15.44:0/1717803235 shutdown_connections 2024-05-27T09:41:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.624+0000 7ff262c00640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2440c39c0 0x7ff2440c5ed0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.624+0000 7ff262c00640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff25c109f50 0x7ff25c06e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.624+0000 7ff262c00640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c109ab0 0x7ff25c06da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.624+0000 7ff262c00640 1 --2- 172.21.15.44:0/1717803235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff25c104250 0x7ff25c074950 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.624+0000 7ff262c00640 1 -- 172.21.15.44:0/1717803235 >> 172.21.15.44:0/1717803235 conn(0x7ff25c079240 msgr2=0x7ff25c1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.624+0000 7ff262c00640 1 -- 172.21.15.44:0/1717803235 shutdown_connections 2024-05-27T09:41:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:18.624+0000 7ff262c00640 1 -- 172.21.15.44:0/1717803235 wait complete. 2024-05-27T09:41:19.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:18 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:41:19.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:18 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:41:19.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:18 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:41:19.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:18 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:41:19.406 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:18 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:41:19.406 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:18 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:41:19.613 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:19 smithi189 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-05-27T09:41:19.613 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:19 smithi189 systemd[1]: Stopped Ceph mds.f for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:41:19.613 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:19 smithi189 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.f.service: Consumed 2.504s CPU time. 2024-05-27T09:41:19.613 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:19 smithi189 systemd[1]: Starting Ceph mds.f for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:41:19.613 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:19 smithi189 podman[72361]: 2024-05-27 09:41:19.493603577 +0000 UTC m=+0.025669878 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:41:19.979 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:19 smithi189 podman[72361]: 2024-05-27 09:41:19.613279518 +0000 UTC m=+0.145345792 container create a1d538aae68166ff48a930b9af49dccd5a3ef2114af0dee624850f8f659ad3e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:41:20.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.183+0000 7ff2d9000700 1 -- 172.21.15.44:0/3605443668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d4107de0 msgr2=0x7ff2d41081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:20.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.183+0000 7ff2d9000700 1 --2- 172.21.15.44:0/3605443668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d4107de0 0x7ff2d41081f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009990 tx=0x7ff2c0009ca0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.184+0000 7ff2d9000700 1 -- 172.21.15.44:0/3605443668 shutdown_connections 2024-05-27T09:41:20.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.184+0000 7ff2d9000700 1 --2- 172.21.15.44:0/3605443668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d4108730 0x7ff2d4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.184+0000 7ff2d9000700 1 --2- 172.21.15.44:0/3605443668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d41034b0 0x7ff2d41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.184+0000 7ff2d9000700 1 --2- 172.21.15.44:0/3605443668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d4107de0 0x7ff2d41081f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.184+0000 7ff2d9000700 1 -- 172.21.15.44:0/3605443668 >> 172.21.15.44:0/3605443668 conn(0x7ff2d40fedb0 msgr2=0x7ff2d41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:20.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.184+0000 7ff2d9000700 1 -- 172.21.15.44:0/3605443668 shutdown_connections 2024-05-27T09:41:20.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:19 smithi044 ceph-mon[31618]: pgmap v185: 97 pgs: 97 active+clean; 21 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-05-27T09:41:20.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.184+0000 7ff2d9000700 1 -- 172.21.15.44:0/3605443668 wait complete. 2024-05-27T09:41:20.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.185+0000 7ff2d9000700 1 Processor -- start 2024-05-27T09:41:20.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.185+0000 7ff2d9000700 1 -- start start 2024-05-27T09:41:20.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d9000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d41034b0 0x7ff2d419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d9000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d4107de0 0x7ff2d419d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d9000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d4108730 0x7ff2d41a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d9000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d4113340 con 0x7ff2d4108730 2024-05-27T09:41:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d9000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d41131c0 con 0x7ff2d41034b0 2024-05-27T09:41:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d9000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d41134c0 con 0x7ff2d4107de0 2024-05-27T09:41:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d1e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d41034b0 0x7ff2d419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d4107de0 0x7ff2d419d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d1e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d41034b0 0x7ff2d419cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43666/0 (socket says 172.21.15.44:43666) 2024-05-27T09:41:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d1e00700 1 -- 172.21.15.44:0/2417948810 learned_addr learned my addr 172.21.15.44:0/2417948810 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d2800700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d4108730 0x7ff2d41a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d1e00700 1 -- 172.21.15.44:0/2417948810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d4107de0 msgr2=0x7ff2d419d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d4107de0 0x7ff2d419d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d1e00700 1 -- 172.21.15.44:0/2417948810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d4108730 msgr2=0x7ff2d41a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d4108730 0x7ff2d41a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.186+0000 7ff2d1e00700 1 -- 172.21.15.44:0/2417948810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d41a1ec0 con 0x7ff2d41034b0 2024-05-27T09:41:20.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.187+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d41034b0 0x7ff2d419cff0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400eb30 tx=0x7ff2c400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:20.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.187+0000 7ff2caa00700 1 -- 172.21.15.44:0/2417948810 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c400cbd0 con 0x7ff2d41034b0 2024-05-27T09:41:20.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.187+0000 7ff2d9000700 1 -- 172.21.15.44:0/2417948810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d41a21b0 con 0x7ff2d41034b0 2024-05-27T09:41:20.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.187+0000 7ff2caa00700 1 -- 172.21.15.44:0/2417948810 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c40044d0 con 0x7ff2d41034b0 2024-05-27T09:41:20.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.187+0000 7ff2caa00700 1 -- 172.21.15.44:0/2417948810 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4003e30 con 0x7ff2d41034b0 2024-05-27T09:41:20.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.187+0000 7ff2d9000700 1 -- 172.21.15.44:0/2417948810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d41a9ad0 con 0x7ff2d41034b0 2024-05-27T09:41:20.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.191+0000 7ff2caa00700 1 -- 172.21.15.44:0/2417948810 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff2c4004000 con 0x7ff2d41034b0 2024-05-27T09:41:20.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.192+0000 7ff2e7b8bb80 1 Processor -- start 2024-05-27T09:41:20.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.192+0000 7ff2e7b8bb80 1 -- start start 2024-05-27T09:41:20.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.192+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 0x5589b7279330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.192+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b7279870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.192+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b7286110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.192+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5589b71a7e40 con 0x5589b71e69f0 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.192+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5589b71979e0 con 0x5589b7279e70 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.192+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5589b719ba20 con 0x5589b70fae40 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2caa00700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b80b29f0 0x7ff2b80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b7279870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d2800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 0x5589b7279330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b7279870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:54042/0 (socket says 172.21.15.44:54042) 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d2800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 0x5589b7279330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40010/0 (socket says 172.21.15.44:40010) 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d1e00700 1 -- 172.21.15.44:0/239391449 learned_addr learned my addr 172.21.15.44:0/239391449 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:20.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d1400700 1 --2- 172.21.15.44:0/239391449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b7286110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2caa00700 1 -- 172.21.15.44:0/2417948810 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff2c40e29e0 con 0x7ff2d41034b0 2024-05-27T09:41:20.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d1400700 1 -- 172.21.15.44:0/239391449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 msgr2=0x5589b7279870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:20.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d1400700 1 --2- 172.21.15.44:0/239391449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b7279870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d1400700 1 -- 172.21.15.44:0/239391449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 msgr2=0x5589b7279330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:20.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d1400700 1 --2- 172.21.15.44:0/239391449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 0x5589b7279330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d1400700 1 -- 172.21.15.44:0/239391449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589b7160420 con 0x5589b7279e70 2024-05-27T09:41:20.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.193+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/239391449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b7279870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:20.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.194+0000 7ff2d1400700 1 --2- 172.21.15.44:0/239391449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b7286110 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00098e0 tx=0x7ff2c0009570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2ea4eb57e055bdc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:20.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.194+0000 7ff2d2800700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b80b29f0 0x7ff2b80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.194+0000 7ff2bf400700 1 -- 172.21.15.44:0/239391449 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0027070 con 0x5589b7279e70 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.194+0000 7ff2bf400700 1 -- 172.21.15.44:0/239391449 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c00056b0 con 0x5589b7279e70 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.194+0000 7ff2bf400700 1 -- 172.21.15.44:0/239391449 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0003ae0 con 0x5589b7279e70 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.195+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/239391449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 msgr2=0x5589b7286110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.195+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/239391449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b7286110 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00098e0 tx=0x7ff2c0009570 comp rx=0 tx=0).stop 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.195+0000 7ff2d2800700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b80b29f0 0x7ff2b80b4f00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009ec0 tx=0x7ff2cc009040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.195+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/239391449 shutdown_connections 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.195+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/239391449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b7286110 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.195+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/239391449 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 0x5589b7279330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.195+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/239391449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b7279870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.195+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/239391449 >> 172.21.15.44:0/239391449 conn(0x5589b7194390 msgr2=0x5589b72754c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.195+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/239391449 shutdown_connections 2024-05-27T09:41:20.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.195+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/239391449 wait complete. 2024-05-27T09:41:20.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.196+0000 7ff2e7b8bb80 1 -- start start 2024-05-27T09:41:20.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.196+0000 7ff2e7b8bb80 1 Processor -- start 2024-05-27T09:41:20.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b70fae40 0x5589b7283660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b71e69f0 0x5589b7283ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b72840e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5589b709dfa0 con 0x5589b70fae40 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5589b7138760 con 0x5589b7279e70 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5589b7286b20 con 0x5589b71e69f0 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2e7b8bb80 10 client.0 _ll_register_callbacks cb 0x5589b7141c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2d1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b70fae40 0x5589b7283660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2d2800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b72840e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2d1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b70fae40 0x5589b7283660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40022/0 (socket says 172.21.15.44:40022) 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2d1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b71e69f0 0x5589b7283ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2d2800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b72840e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43684/0 (socket says 172.21.15.44:43684) 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.197+0000 7ff2d1e00700 1 -- 172.21.15.44:0/2998590785 learned_addr learned my addr 172.21.15.44:0/2998590785 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2d1400700 1 -- 172.21.15.44:0/2998590785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 msgr2=0x5589b72840e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2d1400700 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b72840e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2d1400700 1 -- 172.21.15.44:0/2998590785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b70fae40 msgr2=0x5589b7283660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2d1400700 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b70fae40 0x5589b7283660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2d1400700 1 -- 172.21.15.44:0/2998590785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589b70e78d0 con 0x5589b71e69f0 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b70fae40 0x5589b7283660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2d2800700 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b72840e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2d1400700 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b71e69f0 0x5589b7283ba0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009990 tx=0x7ff2c000ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc2a282016bb5b29 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2b3e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2998590785 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0031060 con 0x5589b71e69f0 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2998590785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2c00056b0 con 0x5589b71e69f0 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.198+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2998590785 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff2c0022480 con 0x5589b71e69f0 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.199+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2998590785 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff2c0027040 con 0x5589b71e69f0 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.199+0000 7ff2e7b8bb80 10 client.26317 fetch_fsmap learned FSMap version 27 2024-05-27T09:41:20.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.199+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2998590785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5589b7168060 con 0x5589b71e69f0 2024-05-27T09:41:20.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.199+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2998590785 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c00056b0 con 0x5589b71e69f0 2024-05-27T09:41:20.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.199+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2998590785 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7ff2c00227a0 con 0x5589b71e69f0 2024-05-27T09:41:20.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.199+0000 7ff2e7b8bb80 10 client.26317 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T09:41:20.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.199+0000 7ff2e7b8bb80 10 client.26317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:20.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.199+0000 7ff2e7b8bb80 20 client.26317 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:20.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.199+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5589b7293920 0x5589b7295d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.199+0000 7ff2e7b8bb80 4 client.26317 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:20.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.199+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2998590785 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589b71a9d90 con 0x5589b7293920 2024-05-27T09:41:20.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.200+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5589b7293920 0x5589b7295d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.200+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5589b7293920 0x5589b7295d40 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d48f4ee0c6cd5a90 server_cookie=34c16333b5ab0272 in_seq=0 out_seq=0 2024-05-27T09:41:20.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.200+0000 7ff2b3e00700 10 client.26317 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2998590785 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5589b71a9d90 con 0x5589b7293920 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2b3e00700 10 client.26317 handle_command_reply: tid=0 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2e7b8bb80 1 client.26317 shutdown 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2998590785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b71e69f0 msgr2=0x5589b7283ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b71e69f0 0x5589b7283ba0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009990 tx=0x7ff2c000ba20 comp rx=0 tx=0).stop 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2998590785 shutdown_connections 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5589b7293920 0x5589b7295d40 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b72840e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b70fae40 0x5589b7283660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2b3e00700 0 client.26317 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2998590785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b71e69f0 0x5589b7283ba0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.201+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2998590785 >> 172.21.15.44:0/2998590785 conn(0x5589b7194390 msgr2=0x5589b7279310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:20.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.202+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2998590785 shutdown_connections 2024-05-27T09:41:20.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.202+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2998590785 wait complete. 2024-05-27T09:41:20.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.202+0000 7ff2e7b8bb80 20 client.26317 trim_cache size 0 max 16384 2024-05-27T09:41:20.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.224+0000 7ff2e7b8bb80 1 Processor -- start 2024-05-27T09:41:20.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.224+0000 7ff2e7b8bb80 1 -- start start 2024-05-27T09:41:20.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.224+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b728fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.224+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 0x5589b727b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b7283b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d41b3820 con 0x5589b71e69f0 2024-05-27T09:41:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d41b10b0 con 0x5589b7279e70 2024-05-27T09:41:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d41aef20 con 0x5589b70fae40 2024-05-27T09:41:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2d1400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 0x5589b727b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2d2800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b7283b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2d1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b728fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2d1400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 0x5589b727b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40034/0 (socket says 172.21.15.44:40034) 2024-05-27T09:41:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2d1400700 1 -- 172.21.15.44:0/3645032833 learned_addr learned my addr 172.21.15.44:0/3645032833 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2d1e00700 1 -- 172.21.15.44:0/3645032833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 msgr2=0x5589b7283b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/3645032833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b7283b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2d1e00700 1 -- 172.21.15.44:0/3645032833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 msgr2=0x5589b727b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/3645032833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 0x5589b727b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.225+0000 7ff2d1e00700 1 -- 172.21.15.44:0/3645032833 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589b7271240 con 0x5589b70fae40 2024-05-27T09:41:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.226+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/3645032833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b728fdc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4003fd0 tx=0x7ff2c4077860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9143af9fa07453b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.226+0000 7ff2bea00700 1 -- 172.21.15.44:0/3645032833 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c40a5420 con 0x5589b70fae40 2024-05-27T09:41:20.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.226+0000 7ff2bea00700 1 -- 172.21.15.44:0/3645032833 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c40b2430 con 0x5589b70fae40 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.226+0000 7ff2bea00700 1 -- 172.21.15.44:0/3645032833 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c40ae040 con 0x5589b70fae40 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.227+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/3645032833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 msgr2=0x5589b728fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.227+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/3645032833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b728fdc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4003fd0 tx=0x7ff2c4077860 comp rx=0 tx=0).stop 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.227+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/3645032833 shutdown_connections 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.227+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/3645032833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b7279e70 0x5589b7283b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.227+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/3645032833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b728fdc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.227+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/3645032833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b71e69f0 0x5589b727b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.227+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/3645032833 >> 172.21.15.44:0/3645032833 conn(0x5589b7194390 msgr2=0x5589b7283440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.227+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/3645032833 shutdown_connections 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.227+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/3645032833 wait complete. 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.228+0000 7ff2e7b8bb80 1 -- start start 2024-05-27T09:41:20.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.228+0000 7ff2e7b8bb80 1 Processor -- start 2024-05-27T09:41:20.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.228+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b72995e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.228+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b71e69f0 0x5589b72949e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.228+0000 7ff2e7b8bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b7279e70 0x5589b7295030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.228+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d41af570 con 0x5589b7279e70 2024-05-27T09:41:20.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.228+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d41b09e0 con 0x5589b71e69f0 2024-05-27T09:41:20.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.228+0000 7ff2e7b8bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d41afc40 con 0x5589b70fae40 2024-05-27T09:41:20.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.229+0000 7ff2e7b8bb80 10 client.0 _ll_register_callbacks cb 0x5589b7141c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:20.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.229+0000 7ff2d1400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b71e69f0 0x5589b72949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.229+0000 7ff2d2800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b7279e70 0x5589b7295030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.229+0000 7ff2d1400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b71e69f0 0x5589b72949e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43698/0 (socket says 172.21.15.44:43698) 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.229+0000 7ff2d1400700 1 -- 172.21.15.44:0/2363852479 learned_addr learned my addr 172.21.15.44:0/2363852479 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.229+0000 7ff2d1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b72995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.229+0000 7ff2d1400700 1 -- 172.21.15.44:0/2363852479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 msgr2=0x5589b72995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.229+0000 7ff2d1400700 1 --2- 172.21.15.44:0/2363852479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b72995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.229+0000 7ff2d1400700 1 -- 172.21.15.44:0/2363852479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b7279e70 msgr2=0x5589b7295030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.229+0000 7ff2d1400700 1 --2- 172.21.15.44:0/2363852479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b7279e70 0x5589b7295030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.229+0000 7ff2d1400700 1 -- 172.21.15.44:0/2363852479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589b7295800 con 0x5589b71e69f0 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.230+0000 7ff2d1400700 1 --2- 172.21.15.44:0/2363852479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b71e69f0 0x5589b72949e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009ce0 tx=0x7ff2c0005960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ef8ebaef49067a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.230+0000 7ff2b3e00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.230+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2363852479 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0031040 con 0x5589b71e69f0 2024-05-27T09:41:20.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.230+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2363852479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5589b71a3570 con 0x5589b71e69f0 2024-05-27T09:41:20.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.230+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2363852479 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8477+0+0 (secure 0 0 0) 0x7ff2c0022480 con 0x5589b71e69f0 2024-05-27T09:41:20.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.230+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2363852479 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff2c0045050 con 0x5589b71e69f0 2024-05-27T09:41:20.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.230+0000 7ff2e7b8bb80 10 client.26315 fetch_fsmap learned FSMap version 27 2024-05-27T09:41:20.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.230+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2363852479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5589b7306a10 con 0x5589b71e69f0 2024-05-27T09:41:20.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.230+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2363852479 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5589b71a3570 con 0x5589b71e69f0 2024-05-27T09:41:20.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.231+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2363852479 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7ff2c003d020 con 0x5589b71e69f0 2024-05-27T09:41:20.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.231+0000 7ff2e7b8bb80 10 client.26315 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T09:41:20.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.231+0000 7ff2e7b8bb80 10 client.26315 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:20.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.231+0000 7ff2e7b8bb80 20 client.26315 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:20.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.231+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2363852479 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5589b7306e20 0x5589b7309240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:20.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.231+0000 7ff2e7b8bb80 4 client.26315 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:41:20.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.231+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2363852479 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5589b701a250 con 0x5589b7306e20 2024-05-27T09:41:20.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.231+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/2363852479 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5589b7306e20 0x5589b7309240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:20.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2d1e00700 1 --2- 172.21.15.44:0/2363852479 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5589b7306e20 0x5589b7309240 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbe34e0b60a7e3dd server_cookie=78b775c8464f6987 in_seq=0 out_seq=0 2024-05-27T09:41:20.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2b3e00700 10 client.26315 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:20.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2b3e00700 1 -- 172.21.15.44:0/2363852479 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5589b701a250 con 0x5589b7306e20 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2b3e00700 10 client.26315 handle_command_reply: tid=0 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2e7b8bb80 1 client.26315 shutdown 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2363852479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b71e69f0 msgr2=0x5589b72949e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2363852479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b71e69f0 0x5589b72949e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009ce0 tx=0x7ff2c0005960 comp rx=0 tx=0).stop 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2363852479 shutdown_connections 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2363852479 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5589b7306e20 0x5589b7309240 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2363852479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589b7279e70 0x5589b7295030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2363852479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589b70fae40 0x5589b72995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2b3e00700 0 client.26315 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2e7b8bb80 1 --2- 172.21.15.44:0/2363852479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5589b71e69f0 0x5589b72949e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2363852479 >> 172.21.15.44:0/2363852479 conn(0x5589b7194390 msgr2=0x5589b7283e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2363852479 shutdown_connections 2024-05-27T09:41:20.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.232+0000 7ff2e7b8bb80 1 -- 172.21.15.44:0/2363852479 wait complete. 2024-05-27T09:41:20.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.233+0000 7ff2e7b8bb80 20 client.26315 trim_cache size 0 max 16384 2024-05-27T09:41:20.237 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:41:20.237 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:41:20.237 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:41:20.237 INFO:teuthology.orchestra.run.smithi044.stdout: "67e722f3-2161-48d5-aae5-e2b39f69565f": { 2024-05-27T09:41:20.237 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:41:20.237 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "67e722f3-2161-48d5-aae5-e2b39f69565f", 2024-05-27T09:41:20.237 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:41:20.237 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:20.237 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:20.238 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:41:20.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.233+0000 7ff2d9000700 1 -- 172.21.15.44:0/2417948810 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b80b29f0 msgr2=0x7ff2b80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:20.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.233+0000 7ff2d9000700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b80b29f0 0x7ff2b80b4f00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009ec0 tx=0x7ff2cc009040 comp rx=0 tx=0).stop 2024-05-27T09:41:20.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.233+0000 7ff2d9000700 1 -- 172.21.15.44:0/2417948810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d41034b0 msgr2=0x7ff2d419cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:20.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.233+0000 7ff2d9000700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d41034b0 0x7ff2d419cff0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400eb30 tx=0x7ff2c400ee40 comp rx=0 tx=0).stop 2024-05-27T09:41:20.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.233+0000 7ff2d9000700 1 -- 172.21.15.44:0/2417948810 shutdown_connections 2024-05-27T09:41:20.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.233+0000 7ff2d9000700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b80b29f0 0x7ff2b80b4f00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.233+0000 7ff2d9000700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d4108730 0x7ff2d41a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.233+0000 7ff2d9000700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d41034b0 0x7ff2d419cff0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.233+0000 7ff2d9000700 1 --2- 172.21.15.44:0/2417948810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d4107de0 0x7ff2d419d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:20.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.234+0000 7ff2d9000700 1 -- 172.21.15.44:0/2417948810 >> 172.21.15.44:0/2417948810 conn(0x7ff2d40fedb0 msgr2=0x7ff2d410edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:20.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.234+0000 7ff2d9000700 1 -- 172.21.15.44:0/2417948810 shutdown_connections 2024-05-27T09:41:20.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:20.234+0000 7ff2d9000700 1 -- 172.21.15.44:0/2417948810 wait complete. 2024-05-27T09:41:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:19 smithi050 ceph-mon[36371]: pgmap v185: 97 pgs: 97 active+clean; 21 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-05-27T09:41:20.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:19 smithi189 ceph-mon[36671]: pgmap v185: 97 pgs: 97 active+clean; 21 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-05-27T09:41:20.338 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:19 smithi189 podman[72361]: 2024-05-27 09:41:19.982905449 +0000 UTC m=+0.514971724 container init a1d538aae68166ff48a930b9af49dccd5a3ef2114af0dee624850f8f659ad3e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:41:20.338 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:19 smithi189 podman[72361]: 2024-05-27 09:41:19.987286613 +0000 UTC m=+0.519352887 container start a1d538aae68166ff48a930b9af49dccd5a3ef2114af0dee624850f8f659ad3e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T09:41:20.338 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:20 smithi189 ceph-mds[72379]: -- 172.21.15.189:0/85271383 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b2460ffa40 con 0x55b246147180 2024-05-27T09:41:20.338 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:20 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[72375]: starting mds.f at 2024-05-27T09:41:20.339 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:20 smithi189 bash[72361]: a1d538aae68166ff48a930b9af49dccd5a3ef2114af0dee624850f8f659ad3e4 2024-05-27T09:41:20.339 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:41:20 smithi189 systemd[1]: Started Ceph mds.f for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:41:21.168 INFO:tasks.cephfs.filesystem:scrub status for tag:67e722f3-2161-48d5-aae5-e2b39f69565f - {'path': '/', 'tag': '67e722f3-2161-48d5-aae5-e2b39f69565f', 'options': 'recursive,force'} 2024-05-27T09:41:21.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:20 smithi050 ceph-mon[36371]: daemon mds.f restarted 2024-05-27T09:41:21.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:20 smithi189 ceph-mon[36671]: daemon mds.f restarted 2024-05-27T09:41:21.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:20 smithi044 ceph-mon[31618]: daemon mds.f restarted 2024-05-27T09:41:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:21 smithi050 ceph-mon[36371]: pgmap v186: 97 pgs: 97 active+clean; 19 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 983 op/s 2024-05-27T09:41:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:21 smithi050 ceph-mon[36371]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T09:41:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:21 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:41:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:21 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6834/102444567,v1:172.21.15.189:6835/102444567] up:boot 2024-05-27T09:41:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:21 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:41:22.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:21 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:41:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:21 smithi189 ceph-mon[36671]: pgmap v186: 97 pgs: 97 active+clean; 19 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 983 op/s 2024-05-27T09:41:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:21 smithi189 ceph-mon[36671]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T09:41:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:21 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:41:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:21 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6834/102444567,v1:172.21.15.189:6835/102444567] up:boot 2024-05-27T09:41:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:21 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:41:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:21 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:41:22.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:21 smithi044 ceph-mon[31618]: pgmap v186: 97 pgs: 97 active+clean; 19 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 983 op/s 2024-05-27T09:41:22.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:21 smithi044 ceph-mon[31618]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T09:41:22.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:21 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:41:22.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:21 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6834/102444567,v1:172.21.15.189:6835/102444567] up:boot 2024-05-27T09:41:22.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:21 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:41:22.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:21 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:41:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:24 smithi050 ceph-mon[36371]: pgmap v188: 97 pgs: 97 active+clean; 19 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 1.14k op/s 2024-05-27T09:41:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:24 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:41:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:24 smithi189 ceph-mon[36671]: pgmap v188: 97 pgs: 97 active+clean; 19 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 1.14k op/s 2024-05-27T09:41:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:24 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:41:24.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:24 smithi044 ceph-mon[31618]: pgmap v188: 97 pgs: 97 active+clean; 19 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 1.14k op/s 2024-05-27T09:41:24.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:24 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:41:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:26 smithi050 ceph-mon[36371]: pgmap v189: 97 pgs: 97 active+clean; 19 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 1.14k op/s 2024-05-27T09:41:26.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:26 smithi189 ceph-mon[36671]: pgmap v189: 97 pgs: 97 active+clean; 19 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 1.14k op/s 2024-05-27T09:41:26.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:26 smithi044 ceph-mon[31618]: pgmap v189: 97 pgs: 97 active+clean; 19 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 1.14k op/s 2024-05-27T09:41:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:28 smithi050 ceph-mon[36371]: pgmap v190: 97 pgs: 97 active+clean; 24 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.08k op/s 2024-05-27T09:41:28.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:28 smithi189 ceph-mon[36671]: pgmap v190: 97 pgs: 97 active+clean; 24 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.08k op/s 2024-05-27T09:41:28.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:28 smithi044 ceph-mon[31618]: pgmap v190: 97 pgs: 97 active+clean; 24 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.08k op/s 2024-05-27T09:41:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:30 smithi050 ceph-mon[36371]: pgmap v191: 97 pgs: 97 active+clean; 24 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.08k op/s 2024-05-27T09:41:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:30 smithi189 ceph-mon[36671]: pgmap v191: 97 pgs: 97 active+clean; 24 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.08k op/s 2024-05-27T09:41:30.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:30 smithi044 ceph-mon[31618]: pgmap v191: 97 pgs: 97 active+clean; 24 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.08k op/s 2024-05-27T09:41:31.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:31 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:41:31.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:31 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:41:31.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:31 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:41:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.862+0000 7fb65cc00640 1 -- 172.21.15.44:0/125884864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb65810e8b0 msgr2=0x7fb65810ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.862+0000 7fb65cc00640 1 --2- 172.21.15.44:0/125884864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb65810e8b0 0x7fb65810ece0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb648009900 tx=0x7fb64802a410 comp rx=0 tx=0).stop 2024-05-27T09:41:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.862+0000 7fb65cc00640 1 -- 172.21.15.44:0/125884864 shutdown_connections 2024-05-27T09:41:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.862+0000 7fb65cc00640 1 --2- 172.21.15.44:0/125884864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb65810f370 0x7fb658116b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.862+0000 7fb65cc00640 1 --2- 172.21.15.44:0/125884864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb65810e8b0 0x7fb65810ece0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.862+0000 7fb65cc00640 1 --2- 172.21.15.44:0/125884864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810a320 0x7fb65810a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.862+0000 7fb65cc00640 1 -- 172.21.15.44:0/125884864 >> 172.21.15.44:0/125884864 conn(0x7fb658076070 msgr2=0x7fb658076480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.862+0000 7fb65cc00640 1 -- 172.21.15.44:0/125884864 shutdown_connections 2024-05-27T09:41:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.862+0000 7fb65cc00640 1 -- 172.21.15.44:0/125884864 wait complete. 2024-05-27T09:41:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.864+0000 7fb65cc00640 1 Processor -- start 2024-05-27T09:41:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.864+0000 7fb65cc00640 1 -- start start 2024-05-27T09:41:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.864+0000 7fb65cc00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb65810a320 0x7fb6581a68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.864+0000 7fb65cc00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb65810e8b0 0x7fb6581a6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.864+0000 7fb65cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810f370 0x7fb6581a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.864+0000 7fb65cc00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb65811a250 con 0x7fb65810e8b0 2024-05-27T09:41:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.864+0000 7fb65cc00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb65811a0d0 con 0x7fb65810a320 2024-05-27T09:41:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.864+0000 7fb65cc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb65811a3d0 con 0x7fb65810f370 2024-05-27T09:41:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.865+0000 7fb656a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb65810e8b0 0x7fb6581a6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.865+0000 7fb657400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb65810a320 0x7fb6581a68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.865+0000 7fb656a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb65810e8b0 0x7fb6581a6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55692/0 (socket says 172.21.15.44:55692) 2024-05-27T09:41:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.865+0000 7fb656a00640 1 -- 172.21.15.44:0/3529869197 learned_addr learned my addr 172.21.15.44:0/3529869197 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.865+0000 7fb657e00640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810f370 0x7fb6581a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.865+0000 7fb657400640 1 -- 172.21.15.44:0/3529869197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810f370 msgr2=0x7fb6581a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.865+0000 7fb657400640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810f370 0x7fb6581a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.865+0000 7fb657400640 1 -- 172.21.15.44:0/3529869197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb65810e8b0 msgr2=0x7fb6581a6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.865+0000 7fb657400640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb65810e8b0 0x7fb6581a6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.865+0000 7fb657400640 1 -- 172.21.15.44:0/3529869197 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6581a1170 con 0x7fb65810a320 2024-05-27T09:41:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.865+0000 7fb657e00640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810f370 0x7fb6581a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.866+0000 7fb657400640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb65810a320 0x7fb6581a68a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb64000e9f0 tx=0x7fb64000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.866+0000 7fb64fe00640 1 -- 172.21.15.44:0/3529869197 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64000ccc0 con 0x7fb65810a320 2024-05-27T09:41:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.866+0000 7fb65cc00640 1 -- 172.21.15.44:0/3529869197 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6581a1460 con 0x7fb65810a320 2024-05-27T09:41:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.866+0000 7fb64fe00640 1 -- 172.21.15.44:0/3529869197 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6400108e0 con 0x7fb65810a320 2024-05-27T09:41:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.866+0000 7fb64fe00640 1 -- 172.21.15.44:0/3529869197 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640018880 con 0x7fb65810a320 2024-05-27T09:41:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.866+0000 7fb65cc00640 1 -- 172.21.15.44:0/3529869197 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6581ad740 con 0x7fb65810a320 2024-05-27T09:41:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.870+0000 7fb66519d740 1 Processor -- start 2024-05-27T09:41:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.870+0000 7fb66519d740 1 -- start start 2024-05-27T09:41:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.870+0000 7fb64fe00640 1 -- 172.21.15.44:0/3529869197 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb640010430 con 0x7fb65810a320 2024-05-27T09:41:31.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.870+0000 7fb66519d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde274a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.870+0000 7fb66519d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddea92f0 0x562fde274fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.870+0000 7fb66519d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddeb14d0 0x562fde2754e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.870+0000 7fb66519d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562fde1e5b90 con 0x562fddeb14d0 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.870+0000 7fb66519d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562fde1d5730 con 0x562fddea92f0 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.870+0000 7fb66519d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562fde1d9770 con 0x562fdded21b0 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.871+0000 7fb657400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddea92f0 0x562fde274fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.871+0000 7fb657e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde274a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.871+0000 7fb656a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddeb14d0 0x562fde2754e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.871+0000 7fb657e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde274a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:36728/0 (socket says 172.21.15.44:36728) 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.871+0000 7fb657e00640 1 -- 172.21.15.44:0/4254673815 learned_addr learned my addr 172.21.15.44:0/4254673815 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.871+0000 7fb657e00640 1 -- 172.21.15.44:0/4254673815 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddea92f0 msgr2=0x562fde274fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.871+0000 7fb657e00640 1 --2- 172.21.15.44:0/4254673815 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddea92f0 0x562fde274fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.871+0000 7fb657e00640 1 -- 172.21.15.44:0/4254673815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddeb14d0 msgr2=0x562fde2754e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.871+0000 7fb657e00640 1 --2- 172.21.15.44:0/4254673815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddeb14d0 0x562fde2754e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.871+0000 7fb657e00640 1 -- 172.21.15.44:0/4254673815 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fde16e8b0 con 0x562fdded21b0 2024-05-27T09:41:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.872+0000 7fb657e00640 1 --2- 172.21.15.44:0/4254673815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde274a60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb650011790 tx=0x7fb650011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a28b689252ddf34 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.872+0000 7fb647e00640 1 -- 172.21.15.44:0/4254673815 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650003c30 con 0x562fdded21b0 2024-05-27T09:41:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.872+0000 7fb647e00640 1 -- 172.21.15.44:0/4254673815 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb65000fe50 con 0x562fdded21b0 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.872+0000 7fb647e00640 1 -- 172.21.15.44:0/4254673815 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6500193e0 con 0x562fdded21b0 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.872+0000 7fb64fe00640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb63c0b2950 0x7fb63c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.872+0000 7fb64fe00640 1 -- 172.21.15.44:0/3529869197 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7fb6400e29c0 con 0x7fb65810a320 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.872+0000 7fb657400640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb63c0b2950 0x7fb63c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.873+0000 7fb66519d740 1 -- 172.21.15.44:0/4254673815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 msgr2=0x562fde274a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.873+0000 7fb66519d740 1 --2- 172.21.15.44:0/4254673815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde274a60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb650011790 tx=0x7fb650011c50 comp rx=0 tx=0).stop 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.873+0000 7fb66519d740 1 -- 172.21.15.44:0/4254673815 shutdown_connections 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.873+0000 7fb66519d740 1 --2- 172.21.15.44:0/4254673815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddeb14d0 0x562fde2754e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.873+0000 7fb66519d740 1 --2- 172.21.15.44:0/4254673815 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddea92f0 0x562fde274fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.873+0000 7fb66519d740 1 --2- 172.21.15.44:0/4254673815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde274a60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.873+0000 7fb66519d740 1 -- 172.21.15.44:0/4254673815 >> 172.21.15.44:0/4254673815 conn(0x562fddec6300 msgr2=0x562fde271210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.873+0000 7fb66519d740 1 -- 172.21.15.44:0/4254673815 shutdown_connections 2024-05-27T09:41:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.873+0000 7fb66519d740 1 -- 172.21.15.44:0/4254673815 wait complete. 2024-05-27T09:41:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.873+0000 7fb657400640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb63c0b2950 0x7fb63c0b4e60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb6400e01f0 tx=0x7fb6400e6500 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.874+0000 7fb66519d740 1 -- start start 2024-05-27T09:41:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.874+0000 7fb66519d740 1 Processor -- start 2024-05-27T09:41:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.874+0000 7fb66519d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.874+0000 7fb66519d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddeb14d0 0x562fde27f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb66519d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde28aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb66519d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562fddebb790 con 0x562fddea92f0 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb66519d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562fde17dbc0 con 0x562fddeb14d0 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb66519d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562fde281b90 con 0x562fdded21b0 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb66519d740 10 client.0 _ll_register_callbacks cb 0x562fde15b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb657e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb657e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55708/0 (socket says 172.21.15.44:55708) 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb657e00640 1 -- 172.21.15.44:0/1708161985 learned_addr learned my addr 172.21.15.44:0/1708161985 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb656a00640 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddeb14d0 0x562fde27f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb657e00640 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde28aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb656a00640 1 -- 172.21.15.44:0/1708161985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 msgr2=0x562fde28aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb656a00640 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde28aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb656a00640 1 -- 172.21.15.44:0/1708161985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 msgr2=0x562fde27eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb656a00640 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb656a00640 1 -- 172.21.15.44:0/1708161985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fddeea210 con 0x562fddeb14d0 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.875+0000 7fb657e00640 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.876+0000 7fb656a00640 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddeb14d0 0x562fde27f3e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb648004770 tx=0x7fb648031e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9eec3b54d2cecd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.876+0000 7fb644c00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.876+0000 7fb644c00640 1 -- 172.21.15.44:0/1708161985 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648035040 con 0x562fddeb14d0 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.876+0000 7fb644c00640 1 -- 172.21.15.44:0/1708161985 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7fb6480316a0 con 0x562fddeb14d0 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.876+0000 7fb66519d740 1 -- 172.21.15.44:0/1708161985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb65000fe50 con 0x562fddeb14d0 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.876+0000 7fb644c00640 1 -- 172.21.15.44:0/1708161985 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb6480318a0 con 0x562fddeb14d0 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.876+0000 7fb66519d740 10 client.26327 fetch_fsmap learned FSMap version 29 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.876+0000 7fb66519d740 1 -- 172.21.15.44:0/1708161985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562fddfb6620 con 0x562fddeb14d0 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.877+0000 7fb644c00640 1 -- 172.21.15.44:0/1708161985 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb65000fe50 con 0x562fddeb14d0 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.877+0000 7fb644c00640 1 -- 172.21.15.44:0/1708161985 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fb648003cc0 con 0x562fddeb14d0 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.877+0000 7fb66519d740 10 client.26327 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:41:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.877+0000 7fb66519d740 10 client.26327 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:31.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.877+0000 7fb66519d740 20 client.26327 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:31.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.877+0000 7fb66519d740 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562fdde90d80 0x562fde290e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.877+0000 7fb66519d740 4 client.26327 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:31.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.877+0000 7fb66519d740 1 -- 172.21.15.44:0/1708161985 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fde1e7ae0 con 0x562fdde90d80 2024-05-27T09:41:31.881 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.878+0000 7fb657e00640 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562fdde90d80 0x562fde290e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.878+0000 7fb657e00640 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562fdde90d80 0x562fde290e60 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44be11e8a21c73d1 server_cookie=c5fb0eb3331b344f in_seq=0 out_seq=0 2024-05-27T09:41:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.878+0000 7fb644c00640 10 client.26327 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.879+0000 7fb644c00640 1 -- 172.21.15.44:0/1708161985 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562fde1e7ae0 con 0x562fdde90d80 2024-05-27T09:41:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.879+0000 7fb644c00640 10 client.26327 handle_command_reply: tid=0 2024-05-27T09:41:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.879+0000 7fb66519d740 1 client.26327 shutdown 2024-05-27T09:41:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.879+0000 7fb66519d740 1 -- 172.21.15.44:0/1708161985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddeb14d0 msgr2=0x562fde27f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.879+0000 7fb66519d740 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddeb14d0 0x562fde27f3e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb648004770 tx=0x7fb648031e10 comp rx=0 tx=0).stop 2024-05-27T09:41:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.880+0000 7fb66519d740 1 -- 172.21.15.44:0/1708161985 shutdown_connections 2024-05-27T09:41:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.880+0000 7fb66519d740 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562fdde90d80 0x562fde290e60 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.880+0000 7fb66519d740 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde28aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.880+0000 7fb66519d740 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddeb14d0 0x562fde27f3e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.880+0000 7fb66519d740 1 --2- 172.21.15.44:0/1708161985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.880+0000 7fb644c00640 0 client.26327 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.880+0000 7fb66519d740 1 -- 172.21.15.44:0/1708161985 >> 172.21.15.44:0/1708161985 conn(0x562fddec6300 msgr2=0x562fde274890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.880+0000 7fb66519d740 1 -- 172.21.15.44:0/1708161985 shutdown_connections 2024-05-27T09:41:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.880+0000 7fb66519d740 1 -- 172.21.15.44:0/1708161985 wait complete. 2024-05-27T09:41:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.880+0000 7fb66519d740 20 client.26327 trim_cache size 0 max 16384 2024-05-27T09:41:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.897+0000 7fb66519d740 1 Processor -- start 2024-05-27T09:41:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.897+0000 7fb66519d740 1 -- start start 2024-05-27T09:41:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.897+0000 7fb66519d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.897+0000 7fb66519d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddeb14d0 0x562fde27a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.897+0000 7fb66519d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde274f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.897+0000 7fb66519d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581b88c0 con 0x562fddea92f0 2024-05-27T09:41:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.897+0000 7fb66519d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581b6160 con 0x562fddeb14d0 2024-05-27T09:41:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.897+0000 7fb66519d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581b3fb0 con 0x562fdded21b0 2024-05-27T09:41:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.898+0000 7fb657e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.898+0000 7fb657e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55714/0 (socket says 172.21.15.44:55714) 2024-05-27T09:41:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.898+0000 7fb657e00640 1 -- 172.21.15.44:0/3753791245 learned_addr learned my addr 172.21.15.44:0/3753791245 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.898+0000 7fb656a00640 1 --2- 172.21.15.44:0/3753791245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddeb14d0 0x562fde27a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.898+0000 7fb657e00640 1 -- 172.21.15.44:0/3753791245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 msgr2=0x562fde274f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.898+0000 7fb657e00640 1 --2- 172.21.15.44:0/3753791245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde274f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.898+0000 7fb657e00640 1 -- 172.21.15.44:0/3753791245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddeb14d0 msgr2=0x562fde27a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.898+0000 7fb657e00640 1 --2- 172.21.15.44:0/3753791245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddeb14d0 0x562fde27a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.898+0000 7fb657e00640 1 -- 172.21.15.44:0/3753791245 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fde27d1e0 con 0x562fddea92f0 2024-05-27T09:41:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.898+0000 7fb657e00640 1 --2- 172.21.15.44:0/3753791245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27dbd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb650006010 tx=0x7fb650009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04bf281d30f619b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.899+0000 7fb647400640 1 -- 172.21.15.44:0/3753791245 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65002e020 con 0x562fddea92f0 2024-05-27T09:41:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.899+0000 7fb647400640 1 -- 172.21.15.44:0/3753791245 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb65000fe50 con 0x562fddea92f0 2024-05-27T09:41:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.899+0000 7fb647400640 1 -- 172.21.15.44:0/3753791245 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65001f040 con 0x562fddea92f0 2024-05-27T09:41:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.899+0000 7fb66519d740 1 -- 172.21.15.44:0/3753791245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 msgr2=0x562fde27dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.899+0000 7fb66519d740 1 --2- 172.21.15.44:0/3753791245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27dbd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb650006010 tx=0x7fb650009ee0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.900+0000 7fb66519d740 1 -- 172.21.15.44:0/3753791245 shutdown_connections 2024-05-27T09:41:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.900+0000 7fb66519d740 1 --2- 172.21.15.44:0/3753791245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fdded21b0 0x562fde274f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.900+0000 7fb66519d740 1 --2- 172.21.15.44:0/3753791245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fddeb14d0 0x562fde27a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.900+0000 7fb66519d740 1 --2- 172.21.15.44:0/3753791245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde27dbd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.900+0000 7fb66519d740 1 -- 172.21.15.44:0/3753791245 >> 172.21.15.44:0/3753791245 conn(0x562fddec6300 msgr2=0x562fde2747f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.900+0000 7fb66519d740 1 -- 172.21.15.44:0/3753791245 shutdown_connections 2024-05-27T09:41:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.900+0000 7fb66519d740 1 -- 172.21.15.44:0/3753791245 wait complete. 2024-05-27T09:41:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.900+0000 7fb66519d740 1 -- start start 2024-05-27T09:41:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.900+0000 7fb66519d740 1 Processor -- start 2024-05-27T09:41:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.901+0000 7fb66519d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde2823e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.901+0000 7fb66519d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fddeb14d0 0x562fde282920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.901+0000 7fb66519d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fdded21b0 0x562fde2f7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.901+0000 7fb66519d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581b4620 con 0x562fddea92f0 2024-05-27T09:41:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.901+0000 7fb66519d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581b5a90 con 0x562fdded21b0 2024-05-27T09:41:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.901+0000 7fb66519d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581b4cf0 con 0x562fddeb14d0 2024-05-27T09:41:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.901+0000 7fb66519d740 10 client.0 _ll_register_callbacks cb 0x7fb648009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.901+0000 7fb656a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fddeb14d0 0x562fde282920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.901+0000 7fb657e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde2823e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.901+0000 7fb656a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fddeb14d0 0x562fde282920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:36764/0 (socket says 172.21.15.44:36764) 2024-05-27T09:41:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.901+0000 7fb656a00640 1 -- 172.21.15.44:0/3907263539 learned_addr learned my addr 172.21.15.44:0/3907263539 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb657e00640 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fdded21b0 0x562fde2f7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb656a00640 1 -- 172.21.15.44:0/3907263539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fdded21b0 msgr2=0x562fde2f7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb656a00640 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fdded21b0 0x562fde2f7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb656a00640 1 -- 172.21.15.44:0/3907263539 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 msgr2=0x562fde2823e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb656a00640 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde2823e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb656a00640 1 -- 172.21.15.44:0/3907263539 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fde2f7a60 con 0x562fddeb14d0 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb657e00640 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fdded21b0 0x562fde2f7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb656a00640 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fddeb14d0 0x562fde282920 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x562fde275970 tx=0x7fb648000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f17675eaea9920bf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb644c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb644c00640 1 -- 172.21.15.44:0/3907263539 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648035040 con 0x562fddeb14d0 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb644c00640 1 -- 172.21.15.44:0/3907263539 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7fb648046940 con 0x562fddeb14d0 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.902+0000 7fb66519d740 1 -- 172.21.15.44:0/3907263539 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb65000fe50 con 0x562fddeb14d0 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.903+0000 7fb644c00640 1 -- 172.21.15.44:0/3907263539 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb64805f050 con 0x562fddeb14d0 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.903+0000 7fb66519d740 10 client.26338 fetch_fsmap learned FSMap version 29 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.903+0000 7fb66519d740 1 -- 172.21.15.44:0/3907263539 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562fde2f8c60 con 0x562fddeb14d0 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.903+0000 7fb644c00640 1 -- 172.21.15.44:0/3907263539 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb65000fe50 con 0x562fddeb14d0 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.903+0000 7fb644c00640 1 -- 172.21.15.44:0/3907263539 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fb648003680 con 0x562fddeb14d0 2024-05-27T09:41:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.903+0000 7fb66519d740 10 client.26338 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:41:31.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.903+0000 7fb66519d740 10 client.26338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:31.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.903+0000 7fb66519d740 20 client.26338 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:31.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.903+0000 7fb66519d740 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562fdde661c0 0x562fde2fb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:31.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.903+0000 7fb66519d740 4 client.26338 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:41:31.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.903+0000 7fb66519d740 1 -- 172.21.15.44:0/3907263539 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562fde1e12c0 con 0x562fdde661c0 2024-05-27T09:41:31.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.904+0000 7fb657e00640 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562fdde661c0 0x562fde2fb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:31.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.904+0000 7fb657e00640 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562fdde661c0 0x562fde2fb450 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e599a29a7ca1cdcf server_cookie=dad37d447a3fb440 in_seq=0 out_seq=0 2024-05-27T09:41:31.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.904+0000 7fb644c00640 10 client.26338 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.959+0000 7fb644c00640 1 -- 172.21.15.44:0/3907263539 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562fde1e12c0 con 0x562fdde661c0 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.959+0000 7fb644c00640 10 client.26338 handle_command_reply: tid=0 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 1 client.26338 shutdown 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 1 -- 172.21.15.44:0/3907263539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fddeb14d0 msgr2=0x562fde282920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fddeb14d0 0x562fde282920 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x562fde275970 tx=0x7fb648000bc0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 1 -- 172.21.15.44:0/3907263539 shutdown_connections 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562fdde661c0 0x562fde2fb450 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562fdded21b0 0x562fde2f7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fddeb14d0 0x562fde282920 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 1 --2- 172.21.15.44:0/3907263539 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fddea92f0 0x562fde2823e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 1 -- 172.21.15.44:0/3907263539 >> 172.21.15.44:0/3907263539 conn(0x562fddec6300 msgr2=0x562fde274f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb644c00640 0 client.26338 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 1 -- 172.21.15.44:0/3907263539 shutdown_connections 2024-05-27T09:41:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 1 -- 172.21.15.44:0/3907263539 wait complete. 2024-05-27T09:41:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.960+0000 7fb66519d740 20 client.26338 trim_cache size 0 max 16384 2024-05-27T09:41:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:41:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:41:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:41:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.961+0000 7fb65cc00640 1 -- 172.21.15.44:0/3529869197 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb63c0b2950 msgr2=0x7fb63c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.961+0000 7fb65cc00640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb63c0b2950 0x7fb63c0b4e60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb6400e01f0 tx=0x7fb6400e6500 comp rx=0 tx=0).stop 2024-05-27T09:41:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.961+0000 7fb65cc00640 1 -- 172.21.15.44:0/3529869197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb65810a320 msgr2=0x7fb6581a68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.961+0000 7fb65cc00640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb65810a320 0x7fb6581a68a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb64000e9f0 tx=0x7fb64000eeb0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.961+0000 7fb65cc00640 1 -- 172.21.15.44:0/3529869197 shutdown_connections 2024-05-27T09:41:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.961+0000 7fb65cc00640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810f370 0x7fb6581a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.961+0000 7fb65cc00640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb63c0b2950 0x7fb63c0b4e60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.961+0000 7fb65cc00640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb65810e8b0 0x7fb6581a6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.961+0000 7fb65cc00640 1 --2- 172.21.15.44:0/3529869197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb65810a320 0x7fb6581a68a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:31.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.961+0000 7fb65cc00640 1 -- 172.21.15.44:0/3529869197 >> 172.21.15.44:0/3529869197 conn(0x7fb658076070 msgr2=0x7fb658107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:31.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.962+0000 7fb65cc00640 1 -- 172.21.15.44:0/3529869197 shutdown_connections 2024-05-27T09:41:31.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:31.962+0000 7fb65cc00640 1 -- 172.21.15.44:0/3529869197 wait complete. 2024-05-27T09:41:32.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:32 smithi050 ceph-mon[36371]: pgmap v192: 97 pgs: 97 active+clean; 24 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 585 KiB/s wr, 212 op/s 2024-05-27T09:41:32.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:32 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:41:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:32 smithi189 ceph-mon[36671]: pgmap v192: 97 pgs: 97 active+clean; 24 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 585 KiB/s wr, 212 op/s 2024-05-27T09:41:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:32 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:41:32.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:32 smithi044 ceph-mon[31618]: pgmap v192: 97 pgs: 97 active+clean; 24 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 585 KiB/s wr, 212 op/s 2024-05-27T09:41:32.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:32 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:41:34.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:34 smithi050 ceph-mon[36371]: pgmap v193: 97 pgs: 97 active+clean; 24 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 502 KiB/s wr, 181 op/s 2024-05-27T09:41:34.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:34 smithi189 ceph-mon[36671]: pgmap v193: 97 pgs: 97 active+clean; 24 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 502 KiB/s wr, 181 op/s 2024-05-27T09:41:34.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:34 smithi044 ceph-mon[31618]: pgmap v193: 97 pgs: 97 active+clean; 24 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 502 KiB/s wr, 181 op/s 2024-05-27T09:41:36.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:36 smithi050 ceph-mon[36371]: pgmap v194: 97 pgs: 97 active+clean; 24 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 488 KiB/s wr, 176 op/s 2024-05-27T09:41:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:36 smithi189 ceph-mon[36671]: pgmap v194: 97 pgs: 97 active+clean; 24 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 488 KiB/s wr, 176 op/s 2024-05-27T09:41:36.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:36 smithi044 ceph-mon[31618]: pgmap v194: 97 pgs: 97 active+clean; 24 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 488 KiB/s wr, 176 op/s 2024-05-27T09:41:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:38 smithi189 ceph-mon[36671]: pgmap v195: 97 pgs: 97 active+clean; 24 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 562 KiB/s wr, 198 op/s 2024-05-27T09:41:38.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:38 smithi044 ceph-mon[31618]: pgmap v195: 97 pgs: 97 active+clean; 24 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 562 KiB/s wr, 198 op/s 2024-05-27T09:41:38.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:38 smithi050 ceph-mon[36371]: pgmap v195: 97 pgs: 97 active+clean; 24 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 562 KiB/s wr, 198 op/s 2024-05-27T09:41:40.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:40 smithi189 ceph-mon[36671]: pgmap v196: 97 pgs: 97 active+clean; 24 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 79 KiB/s wr, 25 op/s 2024-05-27T09:41:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:40 smithi044 ceph-mon[31618]: pgmap v196: 97 pgs: 97 active+clean; 24 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 79 KiB/s wr, 25 op/s 2024-05-27T09:41:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:40 smithi050 ceph-mon[36371]: pgmap v196: 97 pgs: 97 active+clean; 24 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 79 KiB/s wr, 25 op/s 2024-05-27T09:41:42.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:42 smithi189 ceph-mon[36671]: pgmap v197: 97 pgs: 97 active+clean; 25 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 89 KiB/s wr, 29 op/s 2024-05-27T09:41:42.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:42 smithi044 ceph-mon[31618]: pgmap v197: 97 pgs: 97 active+clean; 25 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 89 KiB/s wr, 29 op/s 2024-05-27T09:41:42.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:42 smithi050 ceph-mon[36371]: pgmap v197: 97 pgs: 97 active+clean; 25 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 89 KiB/s wr, 29 op/s 2024-05-27T09:41:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:44 smithi189 ceph-mon[36671]: pgmap v198: 97 pgs: 97 active+clean; 25 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 84 KiB/s wr, 25 op/s 2024-05-27T09:41:44.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:44 smithi044 ceph-mon[31618]: pgmap v198: 97 pgs: 97 active+clean; 25 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 84 KiB/s wr, 25 op/s 2024-05-27T09:41:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:44 smithi050 ceph-mon[36371]: pgmap v198: 97 pgs: 97 active+clean; 25 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 84 KiB/s wr, 25 op/s 2024-05-27T09:41:45.200 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.199+0000 7f4c93e00640 1 -- 172.21.15.44:0/860054334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c8c109c80 msgr2=0x7f4c8c103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:45.200 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.199+0000 7f4c93e00640 1 --2- 172.21.15.44:0/860054334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c8c109c80 0x7f4c8c103820 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4c80009900 tx=0x7f4c8002a3f0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.199+0000 7f4c93e00640 1 -- 172.21.15.44:0/860054334 shutdown_connections 2024-05-27T09:41:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.199+0000 7f4c93e00640 1 --2- 172.21.15.44:0/860054334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c8c1043d0 0x7f4c8c1107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.199+0000 7f4c93e00640 1 --2- 172.21.15.44:0/860054334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c8c10a120 0x7f4c8c103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.199+0000 7f4c93e00640 1 --2- 172.21.15.44:0/860054334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c8c109c80 0x7f4c8c103820 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.199+0000 7f4c93e00640 1 -- 172.21.15.44:0/860054334 >> 172.21.15.44:0/860054334 conn(0x7f4c8c0ff340 msgr2=0x7f4c8c101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.199+0000 7f4c93e00640 1 -- 172.21.15.44:0/860054334 shutdown_connections 2024-05-27T09:41:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.199+0000 7f4c93e00640 1 -- 172.21.15.44:0/860054334 wait complete. 2024-05-27T09:41:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.201+0000 7f4c93e00640 1 Processor -- start 2024-05-27T09:41:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.201+0000 7f4c93e00640 1 -- start start 2024-05-27T09:41:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.201+0000 7f4c93e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c8c1043d0 0x7f4c8c19d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.201+0000 7f4c93e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c8c109c80 0x7f4c8c19dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.201+0000 7f4c93e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c8c10a120 0x7f4c8c1a87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.201+0000 7f4c93e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c113e70 con 0x7f4c8c1043d0 2024-05-27T09:41:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.201+0000 7f4c93e00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c113cf0 con 0x7f4c8c10a120 2024-05-27T09:41:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.201+0000 7f4c93e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c113ff0 con 0x7f4c8c109c80 2024-05-27T09:41:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.201+0000 7f4c91400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c8c1043d0 0x7f4c8c19d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.202+0000 7f4c91400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c8c1043d0 0x7f4c8c19d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52516/0 (socket says 172.21.15.44:52516) 2024-05-27T09:41:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.202+0000 7f4c91400640 1 -- 172.21.15.44:0/3417186574 learned_addr learned my addr 172.21.15.44:0/3417186574 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.202+0000 7f4c90a00640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c8c109c80 0x7f4c8c19dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.202+0000 7f4c91e00640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c8c10a120 0x7f4c8c1a87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.202+0000 7f4c91400640 1 -- 172.21.15.44:0/3417186574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c8c109c80 msgr2=0x7f4c8c19dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.202+0000 7f4c91400640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c8c109c80 0x7f4c8c19dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.202+0000 7f4c91400640 1 -- 172.21.15.44:0/3417186574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c8c10a120 msgr2=0x7f4c8c1a87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.202+0000 7f4c91400640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c8c10a120 0x7f4c8c1a87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.202+0000 7f4c91400640 1 -- 172.21.15.44:0/3417186574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c8c1a8cf0 con 0x7f4c8c1043d0 2024-05-27T09:41:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.202+0000 7f4c91400640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c8c1043d0 0x7f4c8c19d730 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4c80032fd0 tx=0x7f4c80004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.203+0000 7f4c8a000640 1 -- 172.21.15.44:0/3417186574 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8003f030 con 0x7f4c8c1043d0 2024-05-27T09:41:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.203+0000 7f4c8a000640 1 -- 172.21.15.44:0/3417186574 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c80004490 con 0x7f4c8c1043d0 2024-05-27T09:41:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.203+0000 7f4c8a000640 1 -- 172.21.15.44:0/3417186574 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80035070 con 0x7f4c8c1043d0 2024-05-27T09:41:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.203+0000 7f4c93e00640 1 -- 172.21.15.44:0/3417186574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c8c1a8ec0 con 0x7f4c8c1043d0 2024-05-27T09:41:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.203+0000 7f4c93e00640 1 -- 172.21.15.44:0/3417186574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c8c1a92d0 con 0x7f4c8c1043d0 2024-05-27T09:41:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.207+0000 7f4c9ade8740 1 Processor -- start 2024-05-27T09:41:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.207+0000 7f4c9ade8740 1 -- start start 2024-05-27T09:41:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.207+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3c0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.207+0000 7f4c8a000640 1 -- 172.21.15.44:0/3417186574 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4c80042030 con 0x7f4c8c1043d0 2024-05-27T09:41:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.207+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 0x55bf8e3c0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.207+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3c12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.207+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bf8e331b90 con 0x55bf8dff52f0 2024-05-27T09:41:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.207+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bf8e321730 con 0x55bf8dffd4d0 2024-05-27T09:41:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.207+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bf8e325770 con 0x55bf8e01e1b0 2024-05-27T09:41:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c91400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 0x55bf8e3c0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c90a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3c12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.209 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c91e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3c0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c90a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3c12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52056/0 (socket says 172.21.15.44:52056) 2024-05-27T09:41:45.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c90a00640 1 -- 172.21.15.44:0/2847072084 learned_addr learned my addr 172.21.15.44:0/2847072084 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:45.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c91400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 0x55bf8e3c0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52528/0 (socket says 172.21.15.44:52528) 2024-05-27T09:41:45.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c90a00640 1 -- 172.21.15.44:0/2847072084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8e01e1b0 msgr2=0x55bf8e3c0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:45.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c90a00640 1 --2- 172.21.15.44:0/2847072084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3c0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c90a00640 1 -- 172.21.15.44:0/2847072084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 msgr2=0x55bf8e3c0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:45.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c90a00640 1 --2- 172.21.15.44:0/2847072084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 0x55bf8e3c0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c90a00640 1 -- 172.21.15.44:0/2847072084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf8e2ba8b0 con 0x55bf8dffd4d0 2024-05-27T09:41:45.210 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.208+0000 7f4c90a00640 1 --2- 172.21.15.44:0/2847072084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3c12e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800bbb0 tx=0x7f4c780107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31272fb442bddd65 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:45.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.209+0000 7f4c7e600640 1 -- 172.21.15.44:0/2847072084 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78004030 con 0x55bf8dffd4d0 2024-05-27T09:41:45.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.209+0000 7f4c7e600640 1 -- 172.21.15.44:0/2847072084 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c780054d0 con 0x55bf8dffd4d0 2024-05-27T09:41:45.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.209+0000 7f4c7e600640 1 -- 172.21.15.44:0/2847072084 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78004b80 con 0x55bf8dffd4d0 2024-05-27T09:41:45.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.209+0000 7f4c8a000640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4c740b2950 0x7f4c740b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.210+0000 7f4c8a000640 1 -- 172.21.15.44:0/3417186574 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7f4c80102a80 con 0x7f4c8c1043d0 2024-05-27T09:41:45.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.210+0000 7f4c91e00640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4c740b2950 0x7f4c740b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.210+0000 7f4c9ade8740 1 -- 172.21.15.44:0/2847072084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 msgr2=0x55bf8e3c12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:45.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.210+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/2847072084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3c12e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800bbb0 tx=0x7f4c780107a0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.210+0000 7f4c9ade8740 1 -- 172.21.15.44:0/2847072084 shutdown_connections 2024-05-27T09:41:45.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.210+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/2847072084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3c12e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.210+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/2847072084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 0x55bf8e3c0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.210+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/2847072084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3c0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.210+0000 7f4c9ade8740 1 -- 172.21.15.44:0/2847072084 >> 172.21.15.44:0/2847072084 conn(0x55bf8e012300 msgr2=0x55bf8e3bd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:45.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.210+0000 7f4c9ade8740 1 -- 172.21.15.44:0/2847072084 shutdown_connections 2024-05-27T09:41:45.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.210+0000 7f4c9ade8740 1 -- 172.21.15.44:0/2847072084 wait complete. 2024-05-27T09:41:45.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.211+0000 7f4c91e00640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4c740b2950 0x7f4c740b4e60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55bf8e3bd9d0 tx=0x7f4c840096a0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:45.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.211+0000 7f4c9ade8740 1 -- start start 2024-05-27T09:41:45.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.211+0000 7f4c9ade8740 1 Processor -- start 2024-05-27T09:41:45.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3caf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3d6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bf8e007790 con 0x55bf8e01e1b0 2024-05-27T09:41:45.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bf8e2c9bc0 con 0x55bf8dff52f0 2024-05-27T09:41:45.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bf8e3cd990 con 0x55bf8dffd4d0 2024-05-27T09:41:45.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c9ade8740 10 client.0 _ll_register_callbacks cb 0x55bf8e2a7400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:45.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c91e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3d6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c91e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3d6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52538/0 (socket says 172.21.15.44:52538) 2024-05-27T09:41:45.216 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c91e00640 1 -- 172.21.15.44:0/164258328 learned_addr learned my addr 172.21.15.44:0/164258328 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:45.216 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c91400640 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.216 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.212+0000 7f4c90a00640 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.218 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.213+0000 7f4c91400640 1 -- 172.21.15.44:0/164258328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8dffd4d0 msgr2=0x55bf8e3caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:45.218 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.213+0000 7f4c91400640 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.213+0000 7f4c91400640 1 -- 172.21.15.44:0/164258328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8e01e1b0 msgr2=0x55bf8e3d6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.213+0000 7f4c91400640 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3d6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.213+0000 7f4c91400640 1 -- 172.21.15.44:0/164258328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf8e036210 con 0x55bf8dff52f0 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.213+0000 7f4c91400640 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ca9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4c801001a0 tx=0x7f4c8003b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5684bf30703674dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.213+0000 7f4c72a00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.213+0000 7f4c72a00640 1 -- 172.21.15.44:0/164258328 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80049070 con 0x55bf8dff52f0 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.213+0000 7f4c72a00640 1 -- 172.21.15.44:0/164258328 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7f4c80100420 con 0x55bf8dff52f0 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.213+0000 7f4c9ade8740 1 -- 172.21.15.44:0/164258328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c780054d0 con 0x55bf8dff52f0 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.214+0000 7f4c72a00640 1 -- 172.21.15.44:0/164258328 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4c8010c040 con 0x55bf8dff52f0 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.214+0000 7f4c9ade8740 10 client.26342 fetch_fsmap learned FSMap version 29 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.214+0000 7f4c9ade8740 1 -- 172.21.15.44:0/164258328 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bf8e102620 con 0x55bf8dff52f0 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.214+0000 7f4c72a00640 1 -- 172.21.15.44:0/164258328 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c800e0080 con 0x55bf8dff52f0 2024-05-27T09:41:45.219 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.214+0000 7f4c72a00640 1 -- 172.21.15.44:0/164258328 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f4c800d2700 con 0x55bf8dff52f0 2024-05-27T09:41:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.214+0000 7f4c9ade8740 10 client.26342 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:41:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.214+0000 7f4c9ade8740 10 client.26342 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.214+0000 7f4c9ade8740 20 client.26342 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.214+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf8dfdcd80 0x55bf8e3dcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.214+0000 7f4c9ade8740 4 client.26342 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.214+0000 7f4c9ade8740 1 -- 172.21.15.44:0/164258328 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf8e333ae0 con 0x55bf8dfdcd80 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.215+0000 7f4c90a00640 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf8dfdcd80 0x55bf8e3dcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.215+0000 7f4c90a00640 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf8dfdcd80 0x55bf8e3dcd20 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c842090f2dbca93d server_cookie=9a55b51d7876dce1 in_seq=0 out_seq=0 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.215+0000 7f4c72a00640 10 client.26342 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c72a00640 1 -- 172.21.15.44:0/164258328 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bf8e333ae0 con 0x55bf8dfdcd80 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c72a00640 10 client.26342 handle_command_reply: tid=0 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c9ade8740 1 client.26342 shutdown 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c9ade8740 1 -- 172.21.15.44:0/164258328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dff52f0 msgr2=0x55bf8e3ca9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ca9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4c801001a0 tx=0x7f4c8003b720 comp rx=0 tx=0).stop 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c9ade8740 1 -- 172.21.15.44:0/164258328 shutdown_connections 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf8dfdcd80 0x55bf8e3dcd20 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3d6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/164258328 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ca9c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c72a00640 0 client.26342 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.216+0000 7f4c9ade8740 1 -- 172.21.15.44:0/164258328 >> 172.21.15.44:0/164258328 conn(0x55bf8e012300 msgr2=0x55bf8e3c2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.217+0000 7f4c9ade8740 1 -- 172.21.15.44:0/164258328 shutdown_connections 2024-05-27T09:41:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.217+0000 7f4c9ade8740 1 -- 172.21.15.44:0/164258328 wait complete. 2024-05-27T09:41:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.217+0000 7f4c9ade8740 20 client.26342 trim_cache size 0 max 16384 2024-05-27T09:41:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.234+0000 7f4c9ade8740 1 Processor -- start 2024-05-27T09:41:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.234+0000 7f4c9ade8740 1 -- start start 2024-05-27T09:41:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.234+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.234+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3d6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.234+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3d6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.234+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c1b4590 con 0x55bf8dff52f0 2024-05-27T09:41:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.234+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c1b1e30 con 0x55bf8dffd4d0 2024-05-27T09:41:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.234+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c1afc80 con 0x55bf8e01e1b0 2024-05-27T09:41:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c90a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3d6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c90a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3d6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52086/0 (socket says 172.21.15.44:52086) 2024-05-27T09:41:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c90a00640 1 -- 172.21.15.44:0/586655897 learned_addr learned my addr 172.21.15.44:0/586655897 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c91e00640 1 --2- 172.21.15.44:0/586655897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3d6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c91400640 1 --2- 172.21.15.44:0/586655897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c90a00640 1 -- 172.21.15.44:0/586655897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8e01e1b0 msgr2=0x55bf8e3d6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c90a00640 1 --2- 172.21.15.44:0/586655897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3d6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c90a00640 1 -- 172.21.15.44:0/586655897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 msgr2=0x55bf8e3ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c90a00640 1 --2- 172.21.15.44:0/586655897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c90a00640 1 -- 172.21.15.44:0/586655897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf8e3d79d0 con 0x55bf8dffd4d0 2024-05-27T09:41:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c91400640 1 --2- 172.21.15.44:0/586655897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:41:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.235+0000 7f4c90a00640 1 --2- 172.21.15.44:0/586655897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3d6a30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4c78011010 tx=0x7f4c78013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81a34afa0893eac9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.236+0000 7f4c7dc00640 1 -- 172.21.15.44:0/586655897 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78011440 con 0x55bf8dffd4d0 2024-05-27T09:41:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.236+0000 7f4c7dc00640 1 -- 172.21.15.44:0/586655897 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c78011a90 con 0x55bf8dffd4d0 2024-05-27T09:41:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.236+0000 7f4c7dc00640 1 -- 172.21.15.44:0/586655897 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78029780 con 0x55bf8dffd4d0 2024-05-27T09:41:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.237+0000 7f4c9ade8740 1 -- 172.21.15.44:0/586655897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 msgr2=0x55bf8e3d6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.237+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/586655897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3d6a30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4c78011010 tx=0x7f4c78013040 comp rx=0 tx=0).stop 2024-05-27T09:41:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.237+0000 7f4c9ade8740 1 -- 172.21.15.44:0/586655897 shutdown_connections 2024-05-27T09:41:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.237+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/586655897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8e01e1b0 0x55bf8e3d6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.237+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/586655897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3d6a30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.237+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/586655897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.237+0000 7f4c9ade8740 1 -- 172.21.15.44:0/586655897 >> 172.21.15.44:0/586655897 conn(0x55bf8e012300 msgr2=0x55bf8e3c11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.237+0000 7f4c9ade8740 1 -- 172.21.15.44:0/586655897 shutdown_connections 2024-05-27T09:41:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.237+0000 7f4c9ade8740 1 -- 172.21.15.44:0/586655897 wait complete. 2024-05-27T09:41:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.237+0000 7f4c9ade8740 1 -- start start 2024-05-27T09:41:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.237+0000 7f4c9ade8740 1 Processor -- start 2024-05-27T09:41:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ce450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c9ade8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8e01e1b0 0x55bf8e43d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c1b02f0 con 0x55bf8dffd4d0 2024-05-27T09:41:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c1b1760 con 0x55bf8e01e1b0 2024-05-27T09:41:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c9ade8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c1b09c0 con 0x55bf8dff52f0 2024-05-27T09:41:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c9ade8740 10 client.0 _ll_register_callbacks cb 0x7f4c80005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c90a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c90a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52542/0 (socket says 172.21.15.44:52542) 2024-05-27T09:41:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c90a00640 1 -- 172.21.15.44:0/1775705771 learned_addr learned my addr 172.21.15.44:0/1775705771 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c91400640 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.238+0000 7f4c91e00640 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8e01e1b0 0x55bf8e43d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.239+0000 7f4c90a00640 1 -- 172.21.15.44:0/1775705771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8dff52f0 msgr2=0x55bf8e3ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.239+0000 7f4c90a00640 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.239+0000 7f4c90a00640 1 -- 172.21.15.44:0/1775705771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8e01e1b0 msgr2=0x55bf8e43d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.239+0000 7f4c90a00640 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8e01e1b0 0x55bf8e43d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.239+0000 7f4c90a00640 1 -- 172.21.15.44:0/1775705771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf8e43d8e0 con 0x55bf8dffd4d0 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.239+0000 7f4c91400640 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.239+0000 7f4c90a00640 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3ce990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800b4e0 tx=0x7f4c7800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc21eae8fd0dc2d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.239+0000 7f4c72a00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.239+0000 7f4c72a00640 1 -- 172.21.15.44:0/1775705771 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78005680 con 0x55bf8dffd4d0 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.240+0000 7f4c72a00640 1 -- 172.21.15.44:0/1775705771 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7f4c78011440 con 0x55bf8dffd4d0 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.240+0000 7f4c9ade8740 1 -- 172.21.15.44:0/1775705771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c78011a90 con 0x55bf8dffd4d0 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.240+0000 7f4c72a00640 1 -- 172.21.15.44:0/1775705771 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4c78011660 con 0x55bf8dffd4d0 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.240+0000 7f4c9ade8740 10 client.16188 fetch_fsmap learned FSMap version 29 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.240+0000 7f4c9ade8740 1 -- 172.21.15.44:0/1775705771 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bf8e444c10 con 0x55bf8dffd4d0 2024-05-27T09:41:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.240+0000 7f4c72a00640 1 -- 172.21.15.44:0/1775705771 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c78011a90 con 0x55bf8dffd4d0 2024-05-27T09:41:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.240+0000 7f4c72a00640 1 -- 172.21.15.44:0/1775705771 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f4c7801bcb0 con 0x55bf8dffd4d0 2024-05-27T09:41:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.240+0000 7f4c9ade8740 10 client.16188 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:41:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.240+0000 7f4c9ade8740 10 client.16188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.240+0000 7f4c9ade8740 20 client.16188 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.241+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf8dfb21c0 0x55bf8e447170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.241+0000 7f4c9ade8740 4 client.16188 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:41:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.241+0000 7f4c9ade8740 1 -- 172.21.15.44:0/1775705771 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bf8e32d2c0 con 0x55bf8dfb21c0 2024-05-27T09:41:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.241+0000 7f4c91400640 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf8dfb21c0 0x55bf8e447170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.241+0000 7f4c91400640 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf8dfb21c0 0x55bf8e447170 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36a70c5fc79f6f1c server_cookie=378d73c2f120fba6 in_seq=0 out_seq=0 2024-05-27T09:41:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.241+0000 7f4c72a00640 10 client.16188 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c72a00640 1 -- 172.21.15.44:0/1775705771 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55bf8e32d2c0 con 0x55bf8dfb21c0 2024-05-27T09:41:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c72a00640 10 client.16188 handle_command_reply: tid=0 2024-05-27T09:41:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 1 client.16188 shutdown 2024-05-27T09:41:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 1 -- 172.21.15.44:0/1775705771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dffd4d0 msgr2=0x55bf8e3ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:45.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3ce990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800b4e0 tx=0x7f4c7800bab0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 1 -- 172.21.15.44:0/1775705771 shutdown_connections 2024-05-27T09:41:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf8dfb21c0 0x55bf8e447170 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf8e01e1b0 0x55bf8e43d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf8dffd4d0 0x55bf8e3ce990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 1 --2- 172.21.15.44:0/1775705771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf8dff52f0 0x55bf8e3ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c72a00640 0 client.16188 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 1 -- 172.21.15.44:0/1775705771 >> 172.21.15.44:0/1775705771 conn(0x55bf8e012300 msgr2=0x55bf8e3cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 1 -- 172.21.15.44:0/1775705771 shutdown_connections 2024-05-27T09:41:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 1 -- 172.21.15.44:0/1775705771 wait complete. 2024-05-27T09:41:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.257+0000 7f4c9ade8740 20 client.16188 trim_cache size 0 max 16384 2024-05-27T09:41:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:41:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:41:45.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:41:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 -- 172.21.15.44:0/3417186574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4c740b2950 msgr2=0x7f4c740b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4c740b2950 0x7f4c740b4e60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55bf8e3bd9d0 tx=0x7f4c840096a0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 -- 172.21.15.44:0/3417186574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c8c1043d0 msgr2=0x7f4c8c19d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c8c1043d0 0x7f4c8c19d730 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4c80032fd0 tx=0x7f4c80004270 comp rx=0 tx=0).stop 2024-05-27T09:41:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 -- 172.21.15.44:0/3417186574 shutdown_connections 2024-05-27T09:41:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c8c10a120 0x7f4c8c1a87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c8c109c80 0x7f4c8c19dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4c740b2950 0x7f4c740b4e60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 --2- 172.21.15.44:0/3417186574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c8c1043d0 0x7f4c8c19d730 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 -- 172.21.15.44:0/3417186574 >> 172.21.15.44:0/3417186574 conn(0x7f4c8c0ff340 msgr2=0x7f4c8c0ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 -- 172.21.15.44:0/3417186574 shutdown_connections 2024-05-27T09:41:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:45.258+0000 7f4c93e00640 1 -- 172.21.15.44:0/3417186574 wait complete. 2024-05-27T09:41:46.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:46 smithi189 ceph-mon[36671]: pgmap v199: 97 pgs: 97 active+clean; 25 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 84 KiB/s wr, 25 op/s 2024-05-27T09:41:46.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:46 smithi044 ceph-mon[31618]: pgmap v199: 97 pgs: 97 active+clean; 25 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 84 KiB/s wr, 25 op/s 2024-05-27T09:41:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:46 smithi050 ceph-mon[36371]: pgmap v199: 97 pgs: 97 active+clean; 25 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 84 KiB/s wr, 25 op/s 2024-05-27T09:41:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:48 smithi189 ceph-mon[36671]: pgmap v200: 97 pgs: 97 active+clean; 25 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 137 KiB/s wr, 28 op/s 2024-05-27T09:41:48.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:48 smithi044 ceph-mon[31618]: pgmap v200: 97 pgs: 97 active+clean; 25 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 137 KiB/s wr, 28 op/s 2024-05-27T09:41:48.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:48 smithi050 ceph-mon[36371]: pgmap v200: 97 pgs: 97 active+clean; 25 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 137 KiB/s wr, 28 op/s 2024-05-27T09:41:50.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:50 smithi189 ceph-mon[36671]: pgmap v201: 97 pgs: 97 active+clean; 25 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 63 KiB/s wr, 6 op/s 2024-05-27T09:41:50.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:50 smithi044 ceph-mon[31618]: pgmap v201: 97 pgs: 97 active+clean; 25 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 63 KiB/s wr, 6 op/s 2024-05-27T09:41:50.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:50 smithi050 ceph-mon[36371]: pgmap v201: 97 pgs: 97 active+clean; 25 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 63 KiB/s wr, 6 op/s 2024-05-27T09:41:51.169 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:41:51.636 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:41:52.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:52 smithi189 ceph-mon[36671]: pgmap v202: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 89 KiB/s wr, 12 op/s 2024-05-27T09:41:52.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:52 smithi044 ceph-mon[31618]: pgmap v202: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 89 KiB/s wr, 12 op/s 2024-05-27T09:41:52.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:52 smithi050 ceph-mon[36371]: pgmap v202: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 89 KiB/s wr, 12 op/s 2024-05-27T09:41:53.093 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:53 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:41:53.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:53 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:41:53.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:53 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:41:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.822+0000 7feb48600700 1 -- 172.21.15.44:0/957612333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb401034b0 msgr2=0x7feb401038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.822+0000 7feb48600700 1 --2- 172.21.15.44:0/957612333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb401034b0 0x7feb401038c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7feb340099c0 tx=0x7feb34009cd0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.822+0000 7feb48600700 1 -- 172.21.15.44:0/957612333 shutdown_connections 2024-05-27T09:41:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.822+0000 7feb48600700 1 --2- 172.21.15.44:0/957612333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb40108730 0x7feb40110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.822+0000 7feb48600700 1 --2- 172.21.15.44:0/957612333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb401034b0 0x7feb401038c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.822+0000 7feb48600700 1 --2- 172.21.15.44:0/957612333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb40107de0 0x7feb401081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.822+0000 7feb48600700 1 -- 172.21.15.44:0/957612333 >> 172.21.15.44:0/957612333 conn(0x7feb400fedb0 msgr2=0x7feb401011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.822+0000 7feb48600700 1 -- 172.21.15.44:0/957612333 shutdown_connections 2024-05-27T09:41:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.822+0000 7feb48600700 1 -- 172.21.15.44:0/957612333 wait complete. 2024-05-27T09:41:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.823+0000 7feb48600700 1 Processor -- start 2024-05-27T09:41:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.824+0000 7feb48600700 1 -- start start 2024-05-27T09:41:53.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.824+0000 7feb48600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb401034b0 0x7feb4019ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.824+0000 7feb48600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb40107de0 0x7feb4019b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.824+0000 7feb48600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb40108730 0x7feb4019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.824+0000 7feb48600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feb401133b0 con 0x7feb40108730 2024-05-27T09:41:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.824+0000 7feb48600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feb40113230 con 0x7feb40107de0 2024-05-27T09:41:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.824+0000 7feb48600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feb40113530 con 0x7feb401034b0 2024-05-27T09:41:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.825+0000 7feb45c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb401034b0 0x7feb4019ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.825+0000 7feb45200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb40107de0 0x7feb4019b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.825+0000 7feb45c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb401034b0 0x7feb4019ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42208/0 (socket says 172.21.15.44:42208) 2024-05-27T09:41:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.825+0000 7feb45c00700 1 -- 172.21.15.44:0/2464423212 learned_addr learned my addr 172.21.15.44:0/2464423212 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.825+0000 7feb46600700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb40108730 0x7feb4019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.825+0000 7feb45c00700 1 -- 172.21.15.44:0/2464423212 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb40107de0 msgr2=0x7feb4019b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:53.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.825+0000 7feb45c00700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb40107de0 0x7feb4019b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.825+0000 7feb45c00700 1 -- 172.21.15.44:0/2464423212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb40108730 msgr2=0x7feb4019f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:53.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.825+0000 7feb45c00700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb40108730 0x7feb4019f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.825+0000 7feb45c00700 1 -- 172.21.15.44:0/2464423212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4019fd00 con 0x7feb401034b0 2024-05-27T09:41:53.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.825+0000 7feb45c00700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb401034b0 0x7feb4019ae30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7feb34006010 tx=0x7feb34005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:53.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.826+0000 7feb3e000700 1 -- 172.21.15.44:0/2464423212 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34026070 con 0x7feb401034b0 2024-05-27T09:41:53.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.826+0000 7feb3e000700 1 -- 172.21.15.44:0/2464423212 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb34005680 con 0x7feb401034b0 2024-05-27T09:41:53.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.826+0000 7feb48600700 1 -- 172.21.15.44:0/2464423212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4019ff90 con 0x7feb401034b0 2024-05-27T09:41:53.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.826+0000 7feb3e000700 1 -- 172.21.15.44:0/2464423212 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3400fb20 con 0x7feb401034b0 2024-05-27T09:41:53.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.826+0000 7feb48600700 1 -- 172.21.15.44:0/2464423212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb401a0470 con 0x7feb401034b0 2024-05-27T09:41:53.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.830+0000 7feb571e8b80 1 Processor -- start 2024-05-27T09:41:53.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.831+0000 7feb571e8b80 1 -- start start 2024-05-27T09:41:53.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.831+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e03330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.831+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382c84e40 0x55a382e03870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.831+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382e03e70 0x55a382e10110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.831+0000 7feb571e8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a382d31e40 con 0x55a382d709f0 2024-05-27T09:41:53.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.831+0000 7feb571e8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a382d219e0 con 0x55a382c84e40 2024-05-27T09:41:53.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.831+0000 7feb571e8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a382d25a20 con 0x55a382e03e70 2024-05-27T09:41:53.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.831+0000 7feb3e000700 1 -- 172.21.15.44:0/2464423212 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7feb34029070 con 0x7feb401034b0 2024-05-27T09:41:53.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.831+0000 7feb46600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e03330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.832+0000 7feb46600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e03330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59084/0 (socket says 172.21.15.44:59084) 2024-05-27T09:41:53.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.832+0000 7feb46600700 1 -- 172.21.15.44:0/1486889487 learned_addr learned my addr 172.21.15.44:0/1486889487 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:53.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.832+0000 7feb45c00700 1 --2- 172.21.15.44:0/1486889487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382c84e40 0x55a382e03870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.832+0000 7feb45200700 1 --2- 172.21.15.44:0/1486889487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382e03e70 0x55a382e10110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.832+0000 7feb46600700 1 -- 172.21.15.44:0/1486889487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382e03e70 msgr2=0x55a382e10110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:53.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.832+0000 7feb46600700 1 --2- 172.21.15.44:0/1486889487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382e03e70 0x55a382e10110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.832+0000 7feb46600700 1 -- 172.21.15.44:0/1486889487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382c84e40 msgr2=0x55a382e03870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:53.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.832+0000 7feb46600700 1 --2- 172.21.15.44:0/1486889487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382c84e40 0x55a382e03870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.832+0000 7feb46600700 1 -- 172.21.15.44:0/1486889487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a382cea420 con 0x55a382d709f0 2024-05-27T09:41:53.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.832+0000 7feb46600700 1 --2- 172.21.15.44:0/1486889487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e03330 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feb38011680 tx=0x7feb38009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f640717228c29ae server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:53.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.833+0000 7feb32a00700 1 -- 172.21.15.44:0/1486889487 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38019030 con 0x55a382d709f0 2024-05-27T09:41:53.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.833+0000 7feb32a00700 1 -- 172.21.15.44:0/1486889487 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb380093a0 con 0x55a382d709f0 2024-05-27T09:41:53.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.833+0000 7feb32a00700 1 -- 172.21.15.44:0/1486889487 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38007500 con 0x55a382d709f0 2024-05-27T09:41:53.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.833+0000 7feb3e000700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7feb280b29f0 0x7feb280b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.834+0000 7feb3e000700 1 -- 172.21.15.44:0/2464423212 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7feb340e2ef0 con 0x7feb401034b0 2024-05-27T09:41:53.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.834+0000 7feb571e8b80 1 -- 172.21.15.44:0/1486889487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 msgr2=0x55a382e03330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:53.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.834+0000 7feb571e8b80 1 --2- 172.21.15.44:0/1486889487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e03330 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feb38011680 tx=0x7feb38009920 comp rx=0 tx=0).stop 2024-05-27T09:41:53.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.834+0000 7feb45c00700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7feb280b29f0 0x7feb280b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.834+0000 7feb571e8b80 1 -- 172.21.15.44:0/1486889487 shutdown_connections 2024-05-27T09:41:53.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.834+0000 7feb571e8b80 1 --2- 172.21.15.44:0/1486889487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382e03e70 0x55a382e10110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.834+0000 7feb571e8b80 1 --2- 172.21.15.44:0/1486889487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e03330 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.834+0000 7feb571e8b80 1 --2- 172.21.15.44:0/1486889487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382c84e40 0x55a382e03870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.834+0000 7feb571e8b80 1 -- 172.21.15.44:0/1486889487 >> 172.21.15.44:0/1486889487 conn(0x55a382d1e390 msgr2=0x55a382dff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.835+0000 7feb571e8b80 1 -- 172.21.15.44:0/1486889487 shutdown_connections 2024-05-27T09:41:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.835+0000 7feb571e8b80 1 -- 172.21.15.44:0/1486889487 wait complete. 2024-05-27T09:41:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.835+0000 7feb45c00700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7feb280b29f0 0x7feb280b4f00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7feb340e0720 tx=0x7feb34009550 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.836+0000 7feb571e8b80 1 -- start start 2024-05-27T09:41:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.836+0000 7feb571e8b80 1 Processor -- start 2024-05-27T09:41:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.836+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e0d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.836+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382d709f0 0x55a382e0dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.836+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382e03e70 0x55a382e0e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb571e8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a382c27fa0 con 0x55a382e03e70 2024-05-27T09:41:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb571e8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a382cc2760 con 0x55a382d709f0 2024-05-27T09:41:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb571e8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a382e10b80 con 0x55a382c84e40 2024-05-27T09:41:53.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb571e8b80 10 client.0 _ll_register_callbacks cb 0x55a382ccbc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb46600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e0d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb45200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382d709f0 0x55a382e0dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb46600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e0d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42218/0 (socket says 172.21.15.44:42218) 2024-05-27T09:41:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb46600700 1 -- 172.21.15.44:0/859451775 learned_addr learned my addr 172.21.15.44:0/859451775 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb46600700 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382e03e70 0x55a382e0e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb45200700 1 -- 172.21.15.44:0/859451775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 msgr2=0x55a382e0d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb45200700 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e0d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb45200700 1 -- 172.21.15.44:0/859451775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382e03e70 msgr2=0x55a382e0e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb45200700 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382e03e70 0x55a382e0e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.837+0000 7feb45200700 1 -- 172.21.15.44:0/859451775 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a382c718d0 con 0x55a382d709f0 2024-05-27T09:41:53.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.838+0000 7feb45200700 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382d709f0 0x55a382e0dc10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7feb40076aa0 tx=0x7feb2c015db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f82b066040b70a17 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.838+0000 7feb27400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.838+0000 7feb27400700 1 -- 172.21.15.44:0/859451775 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c019070 con 0x55a382d709f0 2024-05-27T09:41:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.838+0000 7feb27400700 1 -- 172.21.15.44:0/859451775 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7feb2c004050 con 0x55a382d709f0 2024-05-27T09:41:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.838+0000 7feb571e8b80 1 -- 172.21.15.44:0/859451775 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a382d33d90 con 0x55a382d709f0 2024-05-27T09:41:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.838+0000 7feb27400700 1 -- 172.21.15.44:0/859451775 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feb2c024050 con 0x55a382d709f0 2024-05-27T09:41:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.838+0000 7feb571e8b80 10 client.26360 fetch_fsmap learned FSMap version 29 2024-05-27T09:41:53.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.838+0000 7feb571e8b80 1 -- 172.21.15.44:0/859451775 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a382cf2060 con 0x55a382d709f0 2024-05-27T09:41:53.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.839+0000 7feb27400700 1 -- 172.21.15.44:0/859451775 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a382d33d90 con 0x55a382d709f0 2024-05-27T09:41:53.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.839+0000 7feb27400700 1 -- 172.21.15.44:0/859451775 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7feb2c003680 con 0x55a382d709f0 2024-05-27T09:41:53.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.839+0000 7feb571e8b80 10 client.26360 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:41:53.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.839+0000 7feb571e8b80 10 client.26360 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:53.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.839+0000 7feb571e8b80 20 client.26360 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:53.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.839+0000 7feb571e8b80 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a382e1da20 0x55a382e1fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.839+0000 7feb571e8b80 4 client.26360 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.839+0000 7feb571e8b80 1 -- 172.21.15.44:0/859451775 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a382d2d570 con 0x55a382e1da20 2024-05-27T09:41:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.839+0000 7feb46600700 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a382e1da20 0x55a382e1fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.840+0000 7feb27400700 10 client.26360 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:53.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.840+0000 7feb46600700 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a382e1da20 0x55a382e1fe40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=690f38bda8711349 server_cookie=8bc49f32844f3702 in_seq=0 out_seq=0 2024-05-27T09:41:53.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb27400700 1 -- 172.21.15.44:0/859451775 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a382d2d570 con 0x55a382e1da20 2024-05-27T09:41:53.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb27400700 10 client.26360 handle_command_reply: tid=0 2024-05-27T09:41:53.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb571e8b80 1 client.26360 shutdown 2024-05-27T09:41:53.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb571e8b80 1 -- 172.21.15.44:0/859451775 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382d709f0 msgr2=0x55a382e0dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:53.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb571e8b80 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382d709f0 0x55a382e0dc10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7feb40076aa0 tx=0x7feb2c015db0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb571e8b80 1 -- 172.21.15.44:0/859451775 shutdown_connections 2024-05-27T09:41:53.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb571e8b80 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a382e1da20 0x55a382e1fe40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb571e8b80 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382e03e70 0x55a382e0e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb571e8b80 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e0d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb571e8b80 1 --2- 172.21.15.44:0/859451775 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382d709f0 0x55a382e0dc10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb571e8b80 1 -- 172.21.15.44:0/859451775 >> 172.21.15.44:0/859451775 conn(0x55a382d1e390 msgr2=0x55a382e033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:53.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.841+0000 7feb27400700 0 client.26360 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:53.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.842+0000 7feb571e8b80 1 -- 172.21.15.44:0/859451775 shutdown_connections 2024-05-27T09:41:53.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.842+0000 7feb571e8b80 1 -- 172.21.15.44:0/859451775 wait complete. 2024-05-27T09:41:53.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.842+0000 7feb571e8b80 20 client.26360 trim_cache size 0 max 16384 2024-05-27T09:41:53.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.864+0000 7feb571e8b80 1 Processor -- start 2024-05-27T09:41:53.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.864+0000 7feb571e8b80 1 -- start start 2024-05-27T09:41:53.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.864+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e08f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.864+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e19c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.864+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 0x55a382e25620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb571e8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feb401af3d0 con 0x55a382d709f0 2024-05-27T09:41:53.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb571e8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feb401acc30 con 0x55a382e03e70 2024-05-27T09:41:53.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb571e8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feb401aaaa0 con 0x55a382c84e40 2024-05-27T09:41:53.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb46600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e08f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb45200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e19c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb46600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e08f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42222/0 (socket says 172.21.15.44:42222) 2024-05-27T09:41:53.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb46600700 1 -- 172.21.15.44:0/1678244939 learned_addr learned my addr 172.21.15.44:0/1678244939 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:53.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb46600700 1 --2- 172.21.15.44:0/1678244939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 0x55a382e25620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb45200700 1 -- 172.21.15.44:0/1678244939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 msgr2=0x55a382e08f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:53.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb45200700 1 --2- 172.21.15.44:0/1678244939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e08f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb45200700 1 -- 172.21.15.44:0/1678244939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 msgr2=0x55a382e25620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:53.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb45200700 1 --2- 172.21.15.44:0/1678244939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 0x55a382e25620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb45200700 1 -- 172.21.15.44:0/1678244939 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a382e0bc40 con 0x55a382d709f0 2024-05-27T09:41:53.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.865+0000 7feb46600700 1 --2- 172.21.15.44:0/1678244939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 0x55a382e25620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:41:53.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.866+0000 7feb45200700 1 --2- 172.21.15.44:0/1678244939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e19c20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7feb2c003c40 tx=0x7feb2c003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a0e7d7a327f93b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:53.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.866+0000 7feb32000700 1 -- 172.21.15.44:0/1678244939 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c019040 con 0x55a382d709f0 2024-05-27T09:41:53.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.866+0000 7feb32000700 1 -- 172.21.15.44:0/1678244939 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb2c015680 con 0x55a382d709f0 2024-05-27T09:41:53.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.866+0000 7feb32000700 1 -- 172.21.15.44:0/1678244939 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c005300 con 0x55a382d709f0 2024-05-27T09:41:53.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.867+0000 7feb571e8b80 1 -- 172.21.15.44:0/1678244939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 msgr2=0x55a382e19c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:53.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.867+0000 7feb571e8b80 1 --2- 172.21.15.44:0/1678244939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e19c20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7feb2c003c40 tx=0x7feb2c003700 comp rx=0 tx=0).stop 2024-05-27T09:41:53.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.867+0000 7feb571e8b80 1 -- 172.21.15.44:0/1678244939 shutdown_connections 2024-05-27T09:41:53.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.867+0000 7feb571e8b80 1 --2- 172.21.15.44:0/1678244939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 0x55a382e25620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.867+0000 7feb571e8b80 1 --2- 172.21.15.44:0/1678244939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e08f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.867+0000 7feb571e8b80 1 --2- 172.21.15.44:0/1678244939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e19c20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.867+0000 7feb571e8b80 1 -- 172.21.15.44:0/1678244939 >> 172.21.15.44:0/1678244939 conn(0x55a382d1e390 msgr2=0x55a382e0c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:53.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.868+0000 7feb571e8b80 1 -- 172.21.15.44:0/1678244939 shutdown_connections 2024-05-27T09:41:53.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.868+0000 7feb571e8b80 1 -- 172.21.15.44:0/1678244939 wait complete. 2024-05-27T09:41:53.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.868+0000 7feb571e8b80 1 -- start start 2024-05-27T09:41:53.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.868+0000 7feb571e8b80 1 Processor -- start 2024-05-27T09:41:53.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e161f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e11550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb571e8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 0x55a382e11a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb571e8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feb401ab0f0 con 0x55a382d709f0 2024-05-27T09:41:53.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb571e8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feb401ac560 con 0x55a382e03e70 2024-05-27T09:41:53.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb571e8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feb401ab7c0 con 0x55a382c84e40 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb571e8b80 10 client.0 _ll_register_callbacks cb 0x55a382ccbc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb45200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e11550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb46600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb46600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42228/0 (socket says 172.21.15.44:42228) 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb45200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e11550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59110/0 (socket says 172.21.15.44:59110) 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb46600700 1 -- 172.21.15.44:0/2741916524 learned_addr learned my addr 172.21.15.44:0/2741916524 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.869+0000 7feb46600700 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 0x55a382e11a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb46600700 1 -- 172.21.15.44:0/2741916524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 msgr2=0x55a382e161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb46600700 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e161f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb46600700 1 -- 172.21.15.44:0/2741916524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 msgr2=0x55a382e11550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb46600700 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e11550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb46600700 1 -- 172.21.15.44:0/2741916524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a382e11fd0 con 0x55a382e03e70 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb46600700 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 0x55a382e11a90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7feb38022070 tx=0x7feb38004500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=353a901ddaadbc92 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb27400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb27400700 1 -- 172.21.15.44:0/2741916524 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38019040 con 0x55a382e03e70 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb571e8b80 1 -- 172.21.15.44:0/2741916524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb2c015680 con 0x55a382e03e70 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb27400700 1 -- 172.21.15.44:0/2741916524 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7feb38003c00 con 0x55a382e03e70 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb27400700 1 -- 172.21.15.44:0/2741916524 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feb3802ae60 con 0x55a382e03e70 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb571e8b80 10 client.26366 fetch_fsmap learned FSMap version 29 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.870+0000 7feb571e8b80 1 -- 172.21.15.44:0/2741916524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a382e880e0 con 0x55a382e03e70 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.871+0000 7feb27400700 1 -- 172.21.15.44:0/2741916524 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb2c015680 con 0x55a382e03e70 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.871+0000 7feb27400700 1 -- 172.21.15.44:0/2741916524 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7feb380377f0 con 0x55a382e03e70 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.871+0000 7feb571e8b80 10 client.26366 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.871+0000 7feb571e8b80 10 client.26366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:53.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.871+0000 7feb571e8b80 20 client.26366 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.871+0000 7feb571e8b80 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a382e883b0 0x55a382e8a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.871+0000 7feb571e8b80 4 client.26366 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.871+0000 7feb571e8b80 1 -- 172.21.15.44:0/2741916524 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a382dfe120 con 0x55a382e883b0 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.872+0000 7feb45200700 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a382e883b0 0x55a382e8a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.872+0000 7feb45200700 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a382e883b0 0x55a382e8a7d0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5cc659a2516fac7 server_cookie=cdd94401ae9dd092 in_seq=0 out_seq=0 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.872+0000 7feb27400700 10 client.26366 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.872+0000 7feb27400700 1 -- 172.21.15.44:0/2741916524 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a382dfe120 con 0x55a382e883b0 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.872+0000 7feb27400700 10 client.26366 handle_command_reply: tid=0 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.872+0000 7feb571e8b80 1 client.26366 shutdown 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb571e8b80 1 -- 172.21.15.44:0/2741916524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 msgr2=0x55a382e11a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb571e8b80 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 0x55a382e11a90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7feb38022070 tx=0x7feb38004500 comp rx=0 tx=0).stop 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb571e8b80 1 -- 172.21.15.44:0/2741916524 shutdown_connections 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb571e8b80 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a382e883b0 0x55a382e8a7d0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb571e8b80 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a382e03e70 0x55a382e11a90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb571e8b80 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a382c84e40 0x55a382e161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb571e8b80 1 --2- 172.21.15.44:0/2741916524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a382d709f0 0x55a382e11550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb571e8b80 1 -- 172.21.15.44:0/2741916524 >> 172.21.15.44:0/2741916524 conn(0x55a382d1e390 msgr2=0x55a382e0ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:53.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb27400700 0 client.26366 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:53.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb571e8b80 1 -- 172.21.15.44:0/2741916524 shutdown_connections 2024-05-27T09:41:53.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb571e8b80 1 -- 172.21.15.44:0/2741916524 wait complete. 2024-05-27T09:41:53.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.873+0000 7feb571e8b80 20 client.26366 trim_cache size 0 max 16384 2024-05-27T09:41:53.881 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:41:53.881 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:41:53.881 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:41:53.882 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:41:53.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.874+0000 7feb48600700 1 -- 172.21.15.44:0/2464423212 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7feb280b29f0 msgr2=0x7feb280b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:53.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.874+0000 7feb48600700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7feb280b29f0 0x7feb280b4f00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7feb340e0720 tx=0x7feb34009550 comp rx=0 tx=0).stop 2024-05-27T09:41:53.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.874+0000 7feb48600700 1 -- 172.21.15.44:0/2464423212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb401034b0 msgr2=0x7feb4019ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:53.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.874+0000 7feb48600700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb401034b0 0x7feb4019ae30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7feb34006010 tx=0x7feb34005280 comp rx=0 tx=0).stop 2024-05-27T09:41:53.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.874+0000 7feb48600700 1 -- 172.21.15.44:0/2464423212 shutdown_connections 2024-05-27T09:41:53.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.874+0000 7feb48600700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7feb280b29f0 0x7feb280b4f00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.874+0000 7feb48600700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb40108730 0x7feb4019f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.874+0000 7feb48600700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb401034b0 0x7feb4019ae30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.874+0000 7feb48600700 1 --2- 172.21.15.44:0/2464423212 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb40107de0 0x7feb4019b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:53.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.874+0000 7feb48600700 1 -- 172.21.15.44:0/2464423212 >> 172.21.15.44:0/2464423212 conn(0x7feb400fedb0 msgr2=0x7feb4006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:53.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.874+0000 7feb48600700 1 -- 172.21.15.44:0/2464423212 shutdown_connections 2024-05-27T09:41:53.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:53.875+0000 7feb48600700 1 -- 172.21.15.44:0/2464423212 wait complete. 2024-05-27T09:41:54.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:54 smithi044 ceph-mon[31618]: pgmap v203: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 79 KiB/s wr, 9 op/s 2024-05-27T09:41:54.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:54 smithi050 ceph-mon[36371]: pgmap v203: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 79 KiB/s wr, 9 op/s 2024-05-27T09:41:54.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:54 smithi189 ceph-mon[36671]: pgmap v203: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 79 KiB/s wr, 9 op/s 2024-05-27T09:41:54.699 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:41:54.700 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:41:55.173 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:41:55.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:55 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:41:55.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:55 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:41:55.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:55 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:41:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:55 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:41:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:55 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:41:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:55 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:41:55.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:55 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:41:55.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:55 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:41:55.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:55 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:41:56.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:56 smithi044 ceph-mon[31618]: pgmap v204: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 79 KiB/s wr, 9 op/s 2024-05-27T09:41:56.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:56 smithi050 ceph-mon[36371]: pgmap v204: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 79 KiB/s wr, 9 op/s 2024-05-27T09:41:56.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:56 smithi189 ceph-mon[36671]: pgmap v204: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 79 KiB/s wr, 9 op/s 2024-05-27T09:41:57.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.255+0000 7f3f42000700 1 -- 172.21.15.44:0/497666062 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: -115 ) ==== 8+0+595 (crc 0 0 0) 0x55ca00d53700 con 0x55ca00eaf850 2024-05-27T09:41:57.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.255+0000 7f3f42000700 10 client.26269 handle_command_reply: tid=0 2024-05-27T09:41:57.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.255+0000 7f3f6dcb9b80 1 client.26269 shutdown 2024-05-27T09:41:57.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.256+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/497666062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 msgr2=0x55ca00e3c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:57.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.256+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e3c2a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3f480a9ba0 tx=0x7f3f480b2b10 comp rx=0 tx=0).stop 2024-05-27T09:41:57.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.256+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/497666062 shutdown_connections 2024-05-27T09:41:57.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.256+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ca00eaf850 0x55ca00eb1c70 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.256+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca00e39070 0x55ca00e39480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.256+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca00e2a4d0 0x55ca00e3c2a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.256+0000 7f3f6dcb9b80 1 --2- 172.21.15.44:0/497666062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ca00e30380 0x55ca00e3c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.256+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/497666062 >> 172.21.15.44:0/497666062 conn(0x55ca00d0b150 msgr2=0x55ca00e29530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:57.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.256+0000 7f3f42000700 0 client.26269 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:57.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.257+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/497666062 shutdown_connections 2024-05-27T09:41:57.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.257+0000 7f3f6dcb9b80 1 -- 172.21.15.44:0/497666062 wait complete. 2024-05-27T09:41:57.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.257+0000 7f3f6dcb9b80 20 client.26269 trim_cache size 0 max 16384 2024-05-27T09:41:57.261 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:41:57.261 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 29, 2024-05-27T09:41:57.262 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:41:57.262 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 13, 2024-05-27T09:41:57.262 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:41:57.262 INFO:teuthology.orchestra.run.smithi044.stdout: "77ef2bd1": { 2024-05-27T09:41:57.262 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 13, 2024-05-27T09:41:57.262 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:41:57.262 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:41:57.262 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCING", 2024-05-27T09:41:57.262 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 45.0 2024-05-27T09:41:57.262 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:41:57.262 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 150.0, 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCING", 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 45.0 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:41:57.263 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:41:57.264 INFO:teuthology.orchestra.run.smithi044.stderr:Error EINPROGRESS: 2024-05-27T09:41:57.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.497+0000 7f4d6de00700 1 -- 172.21.15.44:0/2164386650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d681034b0 msgr2=0x7f4d681038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.497+0000 7f4d6de00700 1 --2- 172.21.15.44:0/2164386650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d681034b0 0x7f4d681038c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009990 tx=0x7f4d58009ca0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.498+0000 7f4d6de00700 1 -- 172.21.15.44:0/2164386650 shutdown_connections 2024-05-27T09:41:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.498+0000 7f4d6de00700 1 --2- 172.21.15.44:0/2164386650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d68108730 0x7f4d68110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.498+0000 7f4d6de00700 1 --2- 172.21.15.44:0/2164386650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d681034b0 0x7f4d681038c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.498+0000 7f4d6de00700 1 --2- 172.21.15.44:0/2164386650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d68107de0 0x7f4d681081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.498+0000 7f4d6de00700 1 -- 172.21.15.44:0/2164386650 >> 172.21.15.44:0/2164386650 conn(0x7f4d680fedb0 msgr2=0x7f4d681011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.498+0000 7f4d6de00700 1 -- 172.21.15.44:0/2164386650 shutdown_connections 2024-05-27T09:41:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.498+0000 7f4d6de00700 1 -- 172.21.15.44:0/2164386650 wait complete. 2024-05-27T09:41:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.499+0000 7f4d6de00700 1 Processor -- start 2024-05-27T09:41:57.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.499+0000 7f4d6de00700 1 -- start start 2024-05-27T09:41:57.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.500+0000 7f4d6de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d681034b0 0x7f4d6819d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:57.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.500+0000 7f4d6de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d68107de0 0x7f4d6819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:57.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.500+0000 7f4d6de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d68108730 0x7f4d681a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:57.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.500+0000 7f4d6de00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4d681133b0 con 0x7f4d68108730 2024-05-27T09:41:57.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.500+0000 7f4d6de00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4d68113230 con 0x7f4d68107de0 2024-05-27T09:41:57.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.500+0000 7f4d6de00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4d68113530 con 0x7f4d681034b0 2024-05-27T09:41:57.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.500+0000 7f4d66a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d681034b0 0x7f4d6819d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:57.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.500+0000 7f4d5e000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d68107de0 0x7f4d6819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:57.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.500+0000 7f4d67400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d68108730 0x7f4d681a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:57.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.500+0000 7f4d66a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d681034b0 0x7f4d6819d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42242/0 (socket says 172.21.15.44:42242) 2024-05-27T09:41:57.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.500+0000 7f4d66a00700 1 -- 172.21.15.44:0/1489921541 learned_addr learned my addr 172.21.15.44:0/1489921541 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.501+0000 7f4d66a00700 1 -- 172.21.15.44:0/1489921541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d68107de0 msgr2=0x7f4d6819d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.501+0000 7f4d66a00700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d68107de0 0x7f4d6819d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.501+0000 7f4d66a00700 1 -- 172.21.15.44:0/1489921541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d68108730 msgr2=0x7f4d681a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.501+0000 7f4d66a00700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d68108730 0x7f4d681a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.501+0000 7f4d66a00700 1 -- 172.21.15.44:0/1489921541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d681a1f10 con 0x7f4d681034b0 2024-05-27T09:41:57.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.501+0000 7f4d5e000700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d68107de0 0x7f4d6819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:57.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.501+0000 7f4d66a00700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d681034b0 0x7f4d6819d040 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4d58000c00 tx=0x7f4d580057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:57.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.501+0000 7f4d6ca00700 1 -- 172.21.15.44:0/1489921541 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d58027070 con 0x7f4d681034b0 2024-05-27T09:41:57.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.501+0000 7f4d6ca00700 1 -- 172.21.15.44:0/1489921541 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d5800b610 con 0x7f4d681034b0 2024-05-27T09:41:57.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.501+0000 7f4d6de00700 1 -- 172.21.15.44:0/1489921541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d681a21a0 con 0x7f4d681034b0 2024-05-27T09:41:57.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.501+0000 7f4d6ca00700 1 -- 172.21.15.44:0/1489921541 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d580173f0 con 0x7f4d681034b0 2024-05-27T09:41:57.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.502+0000 7f4d6de00700 1 -- 172.21.15.44:0/1489921541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d681a2680 con 0x7f4d681034b0 2024-05-27T09:41:57.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.503+0000 7f4d6de00700 1 -- 172.21.15.44:0/1489921541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d680512b0 con 0x7f4d681034b0 2024-05-27T09:41:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.505+0000 7f4d6ca00700 1 -- 172.21.15.44:0/1489921541 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4d5802a070 con 0x7f4d681034b0 2024-05-27T09:41:57.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.507+0000 7f4d6ca00700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4d4c0b29f0 0x7f4d4c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:57.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.507+0000 7f4d6ca00700 1 -- 172.21.15.44:0/1489921541 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7f4d580e3770 con 0x7f4d681034b0 2024-05-27T09:41:57.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.507+0000 7f4d5e000700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4d4c0b29f0 0x7f4d4c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:57.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.508+0000 7f4d5e000700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4d4c0b29f0 0x7f4d4c0b4f00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d68107cf0 tx=0x7f4d50009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:57.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.509+0000 7f4d6ca00700 1 -- 172.21.15.44:0/1489921541 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4d58013070 con 0x7f4d681034b0 2024-05-27T09:41:57.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.807+0000 7f4d6de00700 1 -- 172.21.15.44:0/1489921541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d6802e5e0 con 0x7f4d681034b0 2024-05-27T09:41:57.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.808+0000 7f4d6ca00700 1 -- 172.21.15.44:0/1489921541 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11863 (secure 0 0 0) 0x7f4d580ab1b0 con 0x7f4d681034b0 2024-05-27T09:41:57.810 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:41:57.810 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:41:21.007340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26297":{"gid":26297,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/102444567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":102444567},{"type":"v1","addr":"172.21.15.189:6835","nonce":102444567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:41:57.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.811+0000 7f4d6de00700 1 -- 172.21.15.44:0/1489921541 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4d4c0b29f0 msgr2=0x7f4d4c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:57.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.811+0000 7f4d6de00700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4d4c0b29f0 0x7f4d4c0b4f00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d68107cf0 tx=0x7f4d50009260 comp rx=0 tx=0).stop 2024-05-27T09:41:57.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.811+0000 7f4d6de00700 1 -- 172.21.15.44:0/1489921541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d681034b0 msgr2=0x7f4d6819d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:57.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.811+0000 7f4d6de00700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d681034b0 0x7f4d6819d040 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4d58000c00 tx=0x7f4d580057c0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.811+0000 7f4d6de00700 1 -- 172.21.15.44:0/1489921541 shutdown_connections 2024-05-27T09:41:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.811+0000 7f4d6de00700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4d4c0b29f0 0x7f4d4c0b4f00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.811+0000 7f4d6de00700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d68108730 0x7f4d681a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.811+0000 7f4d6de00700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d681034b0 0x7f4d6819d040 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.811+0000 7f4d6de00700 1 --2- 172.21.15.44:0/1489921541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d68107de0 0x7f4d6819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.811+0000 7f4d6de00700 1 -- 172.21.15.44:0/1489921541 >> 172.21.15.44:0/1489921541 conn(0x7f4d680fedb0 msgr2=0x7f4d68106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.812+0000 7f4d6de00700 1 -- 172.21.15.44:0/1489921541 shutdown_connections 2024-05-27T09:41:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:41:57.812+0000 7f4d6de00700 1 -- 172.21.15.44:0/1489921541 wait complete. 2024-05-27T09:41:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-05-27T09:41:58.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:58 smithi044 ceph-mon[31618]: pgmap v205: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 82 KiB/s wr, 12 op/s 2024-05-27T09:41:58.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:41:58 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1489921541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:41:58.369 DEBUG:teuthology.orchestra.run:got remote process result: 115 2024-05-27T09:41:58.369 WARNING:tasks.quiescer.fs.[cephfs]:Set '77ef2bd1' hasn't quiesced after 55.8 seconds (timeout: 90). Dumping ops with locks from all ranks. 2024-05-27T09:41:58.370 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:41:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.500+0000 7f246a000640 1 -- 172.21.15.44:0/3596127666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2464075420 msgr2=0x7f2464075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.500+0000 7f246a000640 1 --2- 172.21.15.44:0/3596127666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2464075420 0x7f2464075830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2454009900 tx=0x7f245402a410 comp rx=0 tx=0).stop 2024-05-27T09:41:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.500+0000 7f246a000640 1 -- 172.21.15.44:0/3596127666 shutdown_connections 2024-05-27T09:41:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.500+0000 7f246a000640 1 --2- 172.21.15.44:0/3596127666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24641104f0 0x7f2464112910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.500+0000 7f246a000640 1 --2- 172.21.15.44:0/3596127666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f246410db90 0x7f246410ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.500+0000 7f246a000640 1 --2- 172.21.15.44:0/3596127666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2464075420 0x7f2464075830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.500+0000 7f246a000640 1 -- 172.21.15.44:0/3596127666 >> 172.21.15.44:0/3596127666 conn(0x7f2464079390 msgr2=0x7f24640797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.500+0000 7f246a000640 1 -- 172.21.15.44:0/3596127666 shutdown_connections 2024-05-27T09:41:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.500+0000 7f246a000640 1 -- 172.21.15.44:0/3596127666 wait complete. 2024-05-27T09:41:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.502+0000 7f246a000640 1 Processor -- start 2024-05-27T09:41:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.502+0000 7f246a000640 1 -- start start 2024-05-27T09:41:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.502+0000 7f246a000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2464075420 0x7f24641a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.502+0000 7f246a000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f246410db90 0x7f24641a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.502+0000 7f246a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24641104f0 0x7f246419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.502+0000 7f246a000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f24641160c0 con 0x7f2464075420 2024-05-27T09:41:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.502+0000 7f246a000640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2464115f40 con 0x7f246410db90 2024-05-27T09:41:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.502+0000 7f246a000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2464116240 con 0x7f24641104f0 2024-05-27T09:41:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.502+0000 7f2463400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2464075420 0x7f24641a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.503+0000 7f2463400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2464075420 0x7f24641a26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59148/0 (socket says 172.21.15.44:59148) 2024-05-27T09:41:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.503+0000 7f2462a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f246410db90 0x7f24641a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.503+0000 7f2463400640 1 -- 172.21.15.44:0/882066655 learned_addr learned my addr 172.21.15.44:0/882066655 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.503+0000 7f2463e00640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24641104f0 0x7f246419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.503+0000 7f2463400640 1 -- 172.21.15.44:0/882066655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24641104f0 msgr2=0x7f246419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.503+0000 7f2463400640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24641104f0 0x7f246419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.503+0000 7f2463400640 1 -- 172.21.15.44:0/882066655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f246410db90 msgr2=0x7f24641a2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.503+0000 7f2463400640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f246410db90 0x7f24641a2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.503+0000 7f2463400640 1 -- 172.21.15.44:0/882066655 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246419cfa0 con 0x7f2464075420 2024-05-27T09:41:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.503+0000 7f2462a00640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f246410db90 0x7f24641a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.504+0000 7f2463400640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2464075420 0x7f24641a26f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2454005ab0 tx=0x7f2454003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.504+0000 7f245be00640 1 -- 172.21.15.44:0/882066655 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245403f070 con 0x7f2464075420 2024-05-27T09:41:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.504+0000 7f245be00640 1 -- 172.21.15.44:0/882066655 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2454031b60 con 0x7f2464075420 2024-05-27T09:41:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.504+0000 7f245be00640 1 -- 172.21.15.44:0/882066655 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454035030 con 0x7f2464075420 2024-05-27T09:41:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.504+0000 7f246a000640 1 -- 172.21.15.44:0/882066655 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246419d230 con 0x7f2464075420 2024-05-27T09:41:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.504+0000 7f246a000640 1 -- 172.21.15.44:0/882066655 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f246419d6d0 con 0x7f2464075420 2024-05-27T09:41:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.508+0000 7f2471061740 1 Processor -- start 2024-05-27T09:41:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.508+0000 7f2471061740 1 -- start start 2024-05-27T09:41:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.509+0000 7f2471061740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a16860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.509+0000 7f2471061740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317064b2f0 0x563170a16da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.509+0000 7f2471061740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706534d0 0x563170a172e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.509+0000 7f2471061740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563170987b90 con 0x5631706534d0 2024-05-27T09:41:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.509+0000 7f2471061740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563170977730 con 0x5631706741b0 2024-05-27T09:41:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.509+0000 7f2471061740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56317097b770 con 0x56317064b2f0 2024-05-27T09:41:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.509+0000 7f2462a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706534d0 0x563170a172e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.509+0000 7f2462a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706534d0 0x563170a172e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59150/0 (socket says 172.21.15.44:59150) 2024-05-27T09:41:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.509+0000 7f2462a00640 1 -- 172.21.15.44:0/2919191197 learned_addr learned my addr 172.21.15.44:0/2919191197 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.509+0000 7f2463e00640 1 --2- 172.21.15.44:0/2919191197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a16860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.510+0000 7f2463e00640 1 -- 172.21.15.44:0/2919191197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317064b2f0 msgr2=0x563170a16da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:41:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.510+0000 7f2463e00640 1 --2- 172.21.15.44:0/2919191197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317064b2f0 0x563170a16da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.510+0000 7f2463e00640 1 -- 172.21.15.44:0/2919191197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706534d0 msgr2=0x563170a172e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.510+0000 7f2463e00640 1 --2- 172.21.15.44:0/2919191197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706534d0 0x563170a172e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.510+0000 7f2463e00640 1 -- 172.21.15.44:0/2919191197 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631709108b0 con 0x5631706741b0 2024-05-27T09:41:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.510+0000 7f2462a00640 1 --2- 172.21.15.44:0/2919191197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706534d0 0x563170a172e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:41:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.510+0000 7f245be00640 1 -- 172.21.15.44:0/882066655 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2454042070 con 0x7f2464075420 2024-05-27T09:41:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.511+0000 7f2463e00640 1 --2- 172.21.15.44:0/2919191197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a16860 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f245c005fd0 tx=0x7f245c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=237aa11e81e5d602 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.511+0000 7f244fe00640 1 -- 172.21.15.44:0/2919191197 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c004d20 con 0x5631706741b0 2024-05-27T09:41:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.511+0000 7f244fe00640 1 -- 172.21.15.44:0/2919191197 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f245c004050 con 0x5631706741b0 2024-05-27T09:41:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.511+0000 7f244fe00640 1 -- 172.21.15.44:0/2919191197 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c005220 con 0x5631706741b0 2024-05-27T09:41:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.512+0000 7f2471061740 1 -- 172.21.15.44:0/2919191197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 msgr2=0x563170a16860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.512+0000 7f2471061740 1 --2- 172.21.15.44:0/2919191197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a16860 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f245c005fd0 tx=0x7f245c011810 comp rx=0 tx=0).stop 2024-05-27T09:41:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.512+0000 7f2471061740 1 -- 172.21.15.44:0/2919191197 shutdown_connections 2024-05-27T09:41:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.512+0000 7f2471061740 1 --2- 172.21.15.44:0/2919191197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706534d0 0x563170a172e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.512+0000 7f2471061740 1 --2- 172.21.15.44:0/2919191197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317064b2f0 0x563170a16da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.512+0000 7f2471061740 1 --2- 172.21.15.44:0/2919191197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a16860 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.512+0000 7f2471061740 1 -- 172.21.15.44:0/2919191197 >> 172.21.15.44:0/2919191197 conn(0x563170668300 msgr2=0x563170a13010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.513+0000 7f2471061740 1 -- 172.21.15.44:0/2919191197 shutdown_connections 2024-05-27T09:41:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.513+0000 7f2471061740 1 -- 172.21.15.44:0/2919191197 wait complete. 2024-05-27T09:41:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.513+0000 7f245be00640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f24480b2860 0x7f24480b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.513+0000 7f245be00640 1 -- 172.21.15.44:0/882066655 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7f24541029e0 con 0x7f2464075420 2024-05-27T09:41:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.513+0000 7f2462a00640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f24480b2860 0x7f24480b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.513+0000 7f2471061740 1 -- start start 2024-05-27T09:41:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.514+0000 7f2471061740 1 Processor -- start 2024-05-27T09:41:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.514+0000 7f2462a00640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f24480b2860 0x7f24480b4d70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2450012560 tx=0x7f2450005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.514+0000 7f2471061740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56317064b2f0 0x563170a209e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.514+0000 7f2471061740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 0x563170a20f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.514+0000 7f2471061740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a2ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.514+0000 7f2471061740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56317065d790 con 0x56317064b2f0 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.514+0000 7f2471061740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56317091fbc0 con 0x5631706741b0 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.514+0000 7f2471061740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563170a23990 con 0x5631706534d0 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.514+0000 7f2471061740 10 client.0 _ll_register_callbacks cb 0x5631708fd400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f2463e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a2ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f2463400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56317064b2f0 0x563170a209e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f2463e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a2ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60160/0 (socket says 172.21.15.44:60160) 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f2463e00640 1 -- 172.21.15.44:0/3631982722 learned_addr learned my addr 172.21.15.44:0/3631982722 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f2462a00640 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 0x563170a20f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f2463e00640 1 -- 172.21.15.44:0/3631982722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 msgr2=0x563170a20f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f2463e00640 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 0x563170a20f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f2463e00640 1 -- 172.21.15.44:0/3631982722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56317064b2f0 msgr2=0x563170a209e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f2463e00640 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56317064b2f0 0x563170a209e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f2463e00640 1 -- 172.21.15.44:0/3631982722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56317068c210 con 0x5631706741b0 2024-05-27T09:41:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f2463e00640 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a2ccc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f245c01b040 tx=0x7f245c004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=519959290ef51b26 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.515+0000 7f244cc00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:41:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.516+0000 7f244cc00640 1 -- 172.21.15.44:0/3631982722 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c029020 con 0x5631706741b0 2024-05-27T09:41:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.516+0000 7f2471061740 1 -- 172.21.15.44:0/3631982722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f245c004050 con 0x5631706741b0 2024-05-27T09:41:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.516+0000 7f244cc00640 1 -- 172.21.15.44:0/3631982722 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7f245c005040 con 0x5631706741b0 2024-05-27T09:41:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.516+0000 7f244cc00640 1 -- 172.21.15.44:0/3631982722 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f245c034050 con 0x5631706741b0 2024-05-27T09:41:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.516+0000 7f2471061740 10 client.26384 fetch_fsmap learned FSMap version 29 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.516+0000 7f2471061740 1 -- 172.21.15.44:0/3631982722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563170758620 con 0x5631706741b0 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.516+0000 7f244cc00640 1 -- 172.21.15.44:0/3631982722 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f245c004050 con 0x5631706741b0 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.517+0000 7f244cc00640 1 -- 172.21.15.44:0/3631982722 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f245c0254e0 con 0x5631706741b0 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.517+0000 7f2471061740 10 client.26384 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.517+0000 7f2471061740 10 client.26384 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.517+0000 7f2471061740 20 client.26384 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.517+0000 7f2471061740 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563170632d80 0x563170a32ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.517+0000 7f2471061740 4 client.26384 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.517+0000 7f2471061740 1 -- 172.21.15.44:0/3631982722 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563170989ae0 con 0x563170632d80 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.517+0000 7f2463400640 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563170632d80 0x563170a32ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.518+0000 7f2463400640 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563170632d80 0x563170a32ba0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=714f8649f2b66bdd server_cookie=370a4a112ce11951 in_seq=0 out_seq=0 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.518+0000 7f244cc00640 10 client.26384 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f244cc00640 1 -- 172.21.15.44:0/3631982722 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563170989ae0 con 0x563170632d80 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f244cc00640 10 client.26384 handle_command_reply: tid=0 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 1 client.26384 shutdown 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 1 -- 172.21.15.44:0/3631982722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 msgr2=0x563170a2ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a2ccc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f245c01b040 tx=0x7f245c004cf0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 1 -- 172.21.15.44:0/3631982722 shutdown_connections 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563170632d80 0x563170a32ba0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a2ccc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 0x563170a20f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 1 --2- 172.21.15.44:0/3631982722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56317064b2f0 0x563170a209e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f244cc00640 0 client.26384 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 1 -- 172.21.15.44:0/3631982722 >> 172.21.15.44:0/3631982722 conn(0x563170668300 msgr2=0x563170a16690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 1 -- 172.21.15.44:0/3631982722 shutdown_connections 2024-05-27T09:41:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 1 -- 172.21.15.44:0/3631982722 wait complete. 2024-05-27T09:41:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.519+0000 7f2471061740 20 client.26384 trim_cache size 0 max 16384 2024-05-27T09:41:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.540+0000 7f2471061740 1 Processor -- start 2024-05-27T09:41:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.540+0000 7f2471061740 1 -- start start 2024-05-27T09:41:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.541+0000 7f2471061740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56317064b2f0 0x563170a20ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.541+0000 7f2471061740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 0x563170a2ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.541+0000 7f2471061740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a2cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.541+0000 7f2471061740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f24641b45a0 con 0x56317064b2f0 2024-05-27T09:41:58.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.541+0000 7f2471061740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f24641b1e40 con 0x5631706741b0 2024-05-27T09:41:58.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.541+0000 7f2471061740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f24641afc90 con 0x5631706534d0 2024-05-27T09:41:58.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.541+0000 7f2463400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56317064b2f0 0x563170a20ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.541+0000 7f2462a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 0x563170a2ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.541+0000 7f2462a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 0x563170a2ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42296/0 (socket says 172.21.15.44:42296) 2024-05-27T09:41:58.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.541+0000 7f2462a00640 1 -- 172.21.15.44:0/1233442758 learned_addr learned my addr 172.21.15.44:0/1233442758 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:58.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.541+0000 7f2463e00640 1 --2- 172.21.15.44:0/1233442758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a2cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.542+0000 7f2462a00640 1 -- 172.21.15.44:0/1233442758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 msgr2=0x563170a2cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:58.544 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.542+0000 7f2462a00640 1 --2- 172.21.15.44:0/1233442758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a2cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.542+0000 7f2462a00640 1 -- 172.21.15.44:0/1233442758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56317064b2f0 msgr2=0x563170a20ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.542+0000 7f2462a00640 1 --2- 172.21.15.44:0/1233442758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56317064b2f0 0x563170a20ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.542+0000 7f2462a00640 1 -- 172.21.15.44:0/1233442758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563170a0d050 con 0x5631706534d0 2024-05-27T09:41:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.542+0000 7f2462a00640 1 --2- 172.21.15.44:0/1233442758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 0x563170a2ca30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2450004640 tx=0x7f245001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fa7574d480e5436 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.542+0000 7f244f400640 1 -- 172.21.15.44:0/1233442758 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245000f600 con 0x5631706534d0 2024-05-27T09:41:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.543+0000 7f244f400640 1 -- 172.21.15.44:0/1233442758 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f245000fc50 con 0x5631706534d0 2024-05-27T09:41:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.543+0000 7f244f400640 1 -- 172.21.15.44:0/1233442758 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245001fd50 con 0x5631706534d0 2024-05-27T09:41:58.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:58 smithi050 ceph-mon[36371]: pgmap v205: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 82 KiB/s wr, 12 op/s 2024-05-27T09:41:58.546 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:41:58 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1489921541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:41:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.543+0000 7f2471061740 1 -- 172.21.15.44:0/1233442758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 msgr2=0x563170a2ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.544+0000 7f2471061740 1 --2- 172.21.15.44:0/1233442758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 0x563170a2ca30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2450004640 tx=0x7f245001b000 comp rx=0 tx=0).stop 2024-05-27T09:41:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.544+0000 7f2471061740 1 -- 172.21.15.44:0/1233442758 shutdown_connections 2024-05-27T09:41:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.544+0000 7f2471061740 1 --2- 172.21.15.44:0/1233442758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706741b0 0x563170a2cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.544+0000 7f2471061740 1 --2- 172.21.15.44:0/1233442758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631706534d0 0x563170a2ca30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.544+0000 7f2471061740 1 --2- 172.21.15.44:0/1233442758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56317064b2f0 0x563170a20ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.544+0000 7f2471061740 1 -- 172.21.15.44:0/1233442758 >> 172.21.15.44:0/1233442758 conn(0x563170668300 msgr2=0x563170a1f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.544+0000 7f2471061740 1 -- 172.21.15.44:0/1233442758 shutdown_connections 2024-05-27T09:41:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.544+0000 7f2471061740 1 -- 172.21.15.44:0/1233442758 wait complete. 2024-05-27T09:41:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.544+0000 7f2471061740 1 -- start start 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.544+0000 7f2471061740 1 Processor -- start 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.545+0000 7f2471061740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317064b2f0 0x563170a29480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.545+0000 7f2471061740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706534d0 0x563170a25830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.545+0000 7f2471061740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706741b0 0x563170a25e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.545+0000 7f2471061740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f24641b0300 con 0x5631706741b0 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.545+0000 7f2471061740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f24641b1770 con 0x5631706534d0 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.545+0000 7f2471061740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f24641b09d0 con 0x56317064b2f0 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.545+0000 7f2471061740 10 client.0 _ll_register_callbacks cb 0x7f2450009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.545+0000 7f2463e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706741b0 0x563170a25e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.545+0000 7f2463400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317064b2f0 0x563170a29480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.546+0000 7f2463400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317064b2f0 0x563170a29480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42312/0 (socket says 172.21.15.44:42312) 2024-05-27T09:41:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.546+0000 7f2463400640 1 -- 172.21.15.44:0/2220688073 learned_addr learned my addr 172.21.15.44:0/2220688073 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:41:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.546+0000 7f2462a00640 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706534d0 0x563170a25830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.546+0000 7f2463400640 1 -- 172.21.15.44:0/2220688073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706534d0 msgr2=0x563170a25830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.546+0000 7f2463400640 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706534d0 0x563170a25830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.546+0000 7f2463400640 1 -- 172.21.15.44:0/2220688073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706741b0 msgr2=0x563170a25e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:41:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.546+0000 7f2463400640 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706741b0 0x563170a25e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.546+0000 7f2463400640 1 -- 172.21.15.44:0/2220688073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563170a265f0 con 0x56317064b2f0 2024-05-27T09:41:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.546+0000 7f2463400640 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317064b2f0 0x563170a29480 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f24540c6fb0 tx=0x7f245402a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc0e668b3aa5d4ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:41:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.546+0000 7f244cc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:41:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.546+0000 7f244cc00640 1 -- 172.21.15.44:0/2220688073 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454049040 con 0x56317064b2f0 2024-05-27T09:41:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.547+0000 7f244cc00640 1 -- 172.21.15.44:0/2220688073 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7f24540c5fd0 con 0x56317064b2f0 2024-05-27T09:41:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.547+0000 7f2471061740 1 -- 172.21.15.44:0/2220688073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f245000fc50 con 0x56317064b2f0 2024-05-27T09:41:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.547+0000 7f244cc00640 1 -- 172.21.15.44:0/2220688073 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f24540ce040 con 0x56317064b2f0 2024-05-27T09:41:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.547+0000 7f2471061740 10 client.26395 fetch_fsmap learned FSMap version 29 2024-05-27T09:41:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.547+0000 7f2471061740 1 -- 172.21.15.44:0/2220688073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563170a9a200 con 0x56317064b2f0 2024-05-27T09:41:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.547+0000 7f244cc00640 1 -- 172.21.15.44:0/2220688073 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f245000fc50 con 0x56317064b2f0 2024-05-27T09:41:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.547+0000 7f244cc00640 1 -- 172.21.15.44:0/2220688073 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f245404c060 con 0x56317064b2f0 2024-05-27T09:41:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.547+0000 7f2471061740 10 client.26395 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:41:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.547+0000 7f2471061740 10 client.26395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:41:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.548+0000 7f2471061740 20 client.26395 populate_metadata read hostname 'smithi044' 2024-05-27T09:41:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.548+0000 7f2471061740 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5631706081c0 0x563170a9c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:41:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.548+0000 7f2471061740 4 client.26395 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:41:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.548+0000 7f2471061740 1 -- 172.21.15.44:0/2220688073 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5631709832c0 con 0x5631706081c0 2024-05-27T09:41:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.548+0000 7f2462a00640 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5631706081c0 0x563170a9c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:41:58.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.548+0000 7f2462a00640 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5631706081c0 0x563170a9c970 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f2591c3ba75dd8 server_cookie=1e4d6a44dfc267ae in_seq=0 out_seq=0 2024-05-27T09:41:58.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.548+0000 7f244cc00640 10 client.26395 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f244cc00640 1 -- 172.21.15.44:0/2220688073 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5631709832c0 con 0x5631706081c0 2024-05-27T09:41:58.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f244cc00640 10 client.26395 handle_command_reply: tid=0 2024-05-27T09:41:58.567 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f2471061740 1 client.26395 shutdown 2024-05-27T09:41:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f2471061740 1 -- 172.21.15.44:0/2220688073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317064b2f0 msgr2=0x563170a29480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f2471061740 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317064b2f0 0x563170a29480 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f24540c6fb0 tx=0x7f245402a410 comp rx=0 tx=0).stop 2024-05-27T09:41:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f2471061740 1 -- 172.21.15.44:0/2220688073 shutdown_connections 2024-05-27T09:41:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f2471061740 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5631706081c0 0x563170a9c970 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f2471061740 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5631706741b0 0x563170a25e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f2471061740 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631706534d0 0x563170a25830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f244cc00640 0 client.26395 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:41:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f2471061740 1 --2- 172.21.15.44:0/2220688073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317064b2f0 0x563170a29480 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.561+0000 7f2471061740 1 -- 172.21.15.44:0/2220688073 >> 172.21.15.44:0/2220688073 conn(0x563170668300 msgr2=0x563170a22090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.562+0000 7f2471061740 1 -- 172.21.15.44:0/2220688073 shutdown_connections 2024-05-27T09:41:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.562+0000 7f2471061740 1 -- 172.21.15.44:0/2220688073 wait complete. 2024-05-27T09:41:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.562+0000 7f2471061740 20 client.26395 trim_cache size 0 max 16384 2024-05-27T09:41:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:41:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:41:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:41:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.562+0000 7f246a000640 1 -- 172.21.15.44:0/882066655 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f24480b2860 msgr2=0x7f24480b4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.562+0000 7f246a000640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f24480b2860 0x7f24480b4d70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2450012560 tx=0x7f2450005f90 comp rx=0 tx=0).stop 2024-05-27T09:41:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.563+0000 7f246a000640 1 -- 172.21.15.44:0/882066655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2464075420 msgr2=0x7f24641a26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:41:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.563+0000 7f246a000640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2464075420 0x7f24641a26f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2454005ab0 tx=0x7f2454003c10 comp rx=0 tx=0).stop 2024-05-27T09:41:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.563+0000 7f246a000640 1 -- 172.21.15.44:0/882066655 shutdown_connections 2024-05-27T09:41:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.563+0000 7f246a000640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24641104f0 0x7f246419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.563+0000 7f246a000640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f246410db90 0x7f24641a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.563+0000 7f246a000640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f24480b2860 0x7f24480b4d70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.563+0000 7f246a000640 1 --2- 172.21.15.44:0/882066655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2464075420 0x7f24641a26f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:41:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.563+0000 7f246a000640 1 -- 172.21.15.44:0/882066655 >> 172.21.15.44:0/882066655 conn(0x7f2464079390 msgr2=0x7f2464102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:41:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.563+0000 7f246a000640 1 -- 172.21.15.44:0/882066655 shutdown_connections 2024-05-27T09:41:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:41:58.563+0000 7f246a000640 1 -- 172.21.15.44:0/882066655 wait complete. 2024-05-27T09:41:58.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:58 smithi189 ceph-mon[36671]: pgmap v205: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 82 KiB/s wr, 12 op/s 2024-05-27T09:41:58.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:41:58 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1489921541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:41:58.883 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:41:58.925 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:41:59.780 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:00.207 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:00 smithi044 ceph-mon[31618]: pgmap v206: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 30 KiB/s wr, 8 op/s 2024-05-27T09:42:00.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:00 smithi050 ceph-mon[36371]: pgmap v206: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 30 KiB/s wr, 8 op/s 2024-05-27T09:42:00.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:00 smithi189 ceph-mon[36671]: pgmap v206: 97 pgs: 97 active+clean; 25 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 30 KiB/s wr, 8 op/s 2024-05-27T09:42:01.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:01 smithi044 ceph-mon[31618]: pgmap v207: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 42 KiB/s wr, 14 op/s 2024-05-27T09:42:01.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:01 smithi050 ceph-mon[36371]: pgmap v207: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 42 KiB/s wr, 14 op/s 2024-05-27T09:42:01.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:01 smithi189 ceph-mon[36671]: pgmap v207: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 42 KiB/s wr, 14 op/s 2024-05-27T09:42:02.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.026+0000 7f020fe00700 1 -- 172.21.15.44:0/3572132307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0210105280 msgr2=0x7f0210105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.026+0000 7f020fe00700 1 --2- 172.21.15.44:0/3572132307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0210105280 0x7f0210105690 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0200009990 tx=0x7f0200009ca0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.026+0000 7f020fe00700 1 -- 172.21.15.44:0/3572132307 shutdown_connections 2024-05-27T09:42:02.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.026+0000 7f020fe00700 1 --2- 172.21.15.44:0/3572132307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021010e6f0 0x7f0210110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.026+0000 7f020fe00700 1 --2- 172.21.15.44:0/3572132307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0210105280 0x7f0210105690 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.026+0000 7f020fe00700 1 --2- 172.21.15.44:0/3572132307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0210105bd0 0x7f021010e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.026+0000 7f020fe00700 1 -- 172.21.15.44:0/3572132307 >> 172.21.15.44:0/3572132307 conn(0x7f0210078af0 msgr2=0x7f0210078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:02.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.026+0000 7f020fe00700 1 -- 172.21.15.44:0/3572132307 shutdown_connections 2024-05-27T09:42:02.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.026+0000 7f020fe00700 1 -- 172.21.15.44:0/3572132307 wait complete. 2024-05-27T09:42:02.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.027+0000 7f020fe00700 1 Processor -- start 2024-05-27T09:42:02.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.027+0000 7f020fe00700 1 -- start start 2024-05-27T09:42:02.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.028+0000 7f020fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0210105280 0x7f0210102f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.028+0000 7f020fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0210105bd0 0x7f02101034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.028+0000 7f020fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f021010e6f0 0x7f02101039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.028+0000 7f020fe00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0210114270 con 0x7f021010e6f0 2024-05-27T09:42:02.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.028+0000 7f020fe00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f02101140f0 con 0x7f0210105bd0 2024-05-27T09:42:02.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.028+0000 7f020fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f02101143f0 con 0x7f0210105280 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.028+0000 7f020f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f021010e6f0 0x7f02101039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.028+0000 7f020e000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0210105bd0 0x7f02101034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.028+0000 7f020f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f021010e6f0 0x7f02101039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59180/0 (socket says 172.21.15.44:59180) 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.028+0000 7f020f400700 1 -- 172.21.15.44:0/3670435606 learned_addr learned my addr 172.21.15.44:0/3670435606 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.029+0000 7f020ea00700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0210105280 0x7f0210102f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.029+0000 7f020e000700 1 -- 172.21.15.44:0/3670435606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0210105280 msgr2=0x7f0210102f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.029+0000 7f020e000700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0210105280 0x7f0210102f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.029+0000 7f020e000700 1 -- 172.21.15.44:0/3670435606 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f021010e6f0 msgr2=0x7f02101039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.029+0000 7f020e000700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f021010e6f0 0x7f02101039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.029+0000 7f020e000700 1 -- 172.21.15.44:0/3670435606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02101a8fa0 con 0x7f0210105bd0 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.029+0000 7f020e000700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0210105bd0 0x7f02101034b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f01fc00e9f0 tx=0x7f01fc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:02.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.030+0000 7f0207400700 1 -- 172.21.15.44:0/3670435606 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01fc004220 con 0x7f0210105bd0 2024-05-27T09:42:02.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.030+0000 7f020fe00700 1 -- 172.21.15.44:0/3670435606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02101a9290 con 0x7f0210105bd0 2024-05-27T09:42:02.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.030+0000 7f0207400700 1 -- 172.21.15.44:0/3670435606 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01fc0043c0 con 0x7f0210105bd0 2024-05-27T09:42:02.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.030+0000 7f0207400700 1 -- 172.21.15.44:0/3670435606 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01fc005020 con 0x7f0210105bd0 2024-05-27T09:42:02.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.030+0000 7f020fe00700 1 -- 172.21.15.44:0/3670435606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02101a9780 con 0x7f0210105bd0 2024-05-27T09:42:02.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.031+0000 7f020fe00700 1 -- 172.21.15.44:0/3670435606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f021006c910 con 0x7f0210105bd0 2024-05-27T09:42:02.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.034+0000 7f0207400700 1 -- 172.21.15.44:0/3670435606 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f01fc004750 con 0x7f0210105bd0 2024-05-27T09:42:02.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.036+0000 7f0207400700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01f40b29f0 0x7f01f40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.036+0000 7f0207400700 1 -- 172.21.15.44:0/3670435606 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7f01fc0e2660 con 0x7f0210105bd0 2024-05-27T09:42:02.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.036+0000 7f020ea00700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01f40b29f0 0x7f01f40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.037+0000 7f020ea00700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01f40b29f0 0x7f01f40b4f00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f02000098e0 tx=0x7f0200005960 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:02.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.037+0000 7f0207400700 1 -- 172.21.15.44:0/3670435606 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f01fc0aa0a0 con 0x7f0210105bd0 2024-05-27T09:42:02.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.335+0000 7f020fe00700 1 -- 172.21.15.44:0/3670435606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f021002e5e0 con 0x7f0210105bd0 2024-05-27T09:42:02.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.337+0000 7f0207400700 1 -- 172.21.15.44:0/3670435606 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11863 (secure 0 0 0) 0x7f01fc0a99a0 con 0x7f0210105bd0 2024-05-27T09:42:02.339 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:42:02.339 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:41:21.007340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26297":{"gid":26297,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/102444567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":102444567},{"type":"v1","addr":"172.21.15.189:6835","nonce":102444567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:42:02.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 -- 172.21.15.44:0/3670435606 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01f40b29f0 msgr2=0x7f01f40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01f40b29f0 0x7f01f40b4f00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f02000098e0 tx=0x7f0200005960 comp rx=0 tx=0).stop 2024-05-27T09:42:02.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 -- 172.21.15.44:0/3670435606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0210105bd0 msgr2=0x7f02101034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0210105bd0 0x7f02101034b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f01fc00e9f0 tx=0x7f01fc00ed00 comp rx=0 tx=0).stop 2024-05-27T09:42:02.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 -- 172.21.15.44:0/3670435606 shutdown_connections 2024-05-27T09:42:02.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01f40b29f0 0x7f01f40b4f00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f021010e6f0 0x7f02101039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0210105280 0x7f0210102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 --2- 172.21.15.44:0/3670435606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0210105bd0 0x7f02101034b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 -- 172.21.15.44:0/3670435606 >> 172.21.15.44:0/3670435606 conn(0x7f0210078af0 msgr2=0x7f02100ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:02.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 -- 172.21.15.44:0/3670435606 shutdown_connections 2024-05-27T09:42:02.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.340+0000 7f020fe00700 1 -- 172.21.15.44:0/3670435606 wait complete. 2024-05-27T09:42:02.343 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-05-27T09:42:02.646 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:02 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3670435606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:42:02.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:02 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3670435606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:42:02.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:02 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3670435606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:42:02.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.644+0000 7fd2b8e00700 1 -- 172.21.15.44:0/2238989142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2b410e8c0 msgr2=0x7fd2b4110da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.644+0000 7fd2b8e00700 1 --2- 172.21.15.44:0/2238989142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2b410e8c0 0x7fd2b4110da0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00b0a0 tx=0x7fd2ac00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.644+0000 7fd2b8e00700 1 -- 172.21.15.44:0/2238989142 shutdown_connections 2024-05-27T09:42:02.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.644+0000 7fd2b8e00700 1 --2- 172.21.15.44:0/2238989142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2b410e8c0 0x7fd2b4110da0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.644+0000 7fd2b8e00700 1 --2- 172.21.15.44:0/2238989142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b4105450 0x7fd2b4105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.644+0000 7fd2b8e00700 1 --2- 172.21.15.44:0/2238989142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b4105da0 0x7fd2b410e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.644+0000 7fd2b8e00700 1 -- 172.21.15.44:0/2238989142 >> 172.21.15.44:0/2238989142 conn(0x7fd2b40fedb0 msgr2=0x7fd2b41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.644+0000 7fd2b8e00700 1 -- 172.21.15.44:0/2238989142 shutdown_connections 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.644+0000 7fd2b8e00700 1 -- 172.21.15.44:0/2238989142 wait complete. 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.646+0000 7fd2b8e00700 1 Processor -- start 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.646+0000 7fd2b8e00700 1 -- start start 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.646+0000 7fd2b8e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b4105450 0x7fd2b419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.646+0000 7fd2b8e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2b4105da0 0x7fd2b419d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.646+0000 7fd2b8e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b410e8c0 0x7fd2b41a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.646+0000 7fd2b8e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b4114470 con 0x7fd2b4105450 2024-05-27T09:42:02.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.646+0000 7fd2b8e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41142f0 con 0x7fd2b4105da0 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.646+0000 7fd2b8e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41145f0 con 0x7fd2b410e8c0 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b4105450 0x7fd2b419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b4105450 0x7fd2b419cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59188/0 (socket says 172.21.15.44:59188) 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b1e00700 1 -- 172.21.15.44:0/3646451222 learned_addr learned my addr 172.21.15.44:0/3646451222 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b1400700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2b4105da0 0x7fd2b419d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b2800700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b410e8c0 0x7fd2b41a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b1e00700 1 -- 172.21.15.44:0/3646451222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b410e8c0 msgr2=0x7fd2b41a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b410e8c0 0x7fd2b41a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b1e00700 1 -- 172.21.15.44:0/3646451222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2b4105da0 msgr2=0x7fd2b419d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2b4105da0 0x7fd2b419d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b1e00700 1 -- 172.21.15.44:0/3646451222 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b41a1e30 con 0x7fd2b4105450 2024-05-27T09:42:02.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b1400700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2b4105da0 0x7fd2b419d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.647+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b4105450 0x7fd2b419cf60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400eb30 tx=0x7fd2a400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.648+0000 7fd2aaa00700 1 -- 172.21.15.44:0/3646451222 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a400cbd0 con 0x7fd2b4105450 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.648+0000 7fd2b8e00700 1 -- 172.21.15.44:0/3646451222 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b41a2120 con 0x7fd2b4105450 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.648+0000 7fd2aaa00700 1 -- 172.21.15.44:0/3646451222 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2a40044d0 con 0x7fd2b4105450 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.648+0000 7fd2aaa00700 1 -- 172.21.15.44:0/3646451222 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4003e30 con 0x7fd2b4105450 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.648+0000 7fd2b8e00700 1 -- 172.21.15.44:0/3646451222 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b41a9a40 con 0x7fd2b4105450 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.652+0000 7fd2aaa00700 1 -- 172.21.15.44:0/3646451222 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd2a4004000 con 0x7fd2b4105450 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.653+0000 7fd2c79aeb80 1 Processor -- start 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.653+0000 7fd2c79aeb80 1 -- start start 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.653+0000 7fd2aaa00700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2980b2a70 0x7fd2980b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d17150330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d16fd1e40 0x562d17150870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d1715d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562d1707ee40 con 0x562d170bd9f0 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562d1706e9e0 con 0x562d17150e70 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562d17072a20 con 0x562d16fd1e40 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2aaa00700 1 -- 172.21.15.44:0/3646451222 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7fd2a40e29b0 con 0x7fd2b4105450 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d17150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d17150330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59198/0 (socket says 172.21.15.44:59198) 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2b1e00700 1 -- 172.21.15.44:0/2596407382 learned_addr learned my addr 172.21.15.44:0/2596407382 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2b1400700 1 --2- 172.21.15.44:0/2596407382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d16fd1e40 0x562d17150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2b2800700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2980b2a70 0x7fd2980b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.654+0000 7fd2b2800700 1 --2- 172.21.15.44:0/2596407382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d1715d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.655+0000 7fd2b1e00700 1 -- 172.21.15.44:0/2596407382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d16fd1e40 msgr2=0x562d17150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:02.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.655+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/2596407382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d16fd1e40 0x562d17150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.655+0000 7fd2b1e00700 1 -- 172.21.15.44:0/2596407382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 msgr2=0x562d1715d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.655+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/2596407382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d1715d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.655+0000 7fd2b1e00700 1 -- 172.21.15.44:0/2596407382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d17037420 con 0x562d170bd9f0 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.655+0000 7fd2b1400700 1 --2- 172.21.15.44:0/2596407382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d16fd1e40 0x562d17150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.655+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/2596407382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d17150330 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4003fd0 tx=0x7fd2a40a5ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6bbdaaf934054b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.655+0000 7fd2b2800700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2980b2a70 0x7fd2980b4f80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00b780 tx=0x7fd2ac009d80 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.655+0000 7fd29f400700 1 -- 172.21.15.44:0/2596407382 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a40a93d0 con 0x562d170bd9f0 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.655+0000 7fd29f400700 1 -- 172.21.15.44:0/2596407382 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2a4091040 con 0x562d170bd9f0 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.655+0000 7fd29f400700 1 -- 172.21.15.44:0/2596407382 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a40e0560 con 0x562d170bd9f0 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.657+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/2596407382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 msgr2=0x562d17150330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.657+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/2596407382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d17150330 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4003fd0 tx=0x7fd2a40a5ba0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.657+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/2596407382 shutdown_connections 2024-05-27T09:42:02.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.657+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/2596407382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d1715d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.657+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/2596407382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d17150330 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.657+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/2596407382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d16fd1e40 0x562d17150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.657+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/2596407382 >> 172.21.15.44:0/2596407382 conn(0x562d1706b390 msgr2=0x562d1714c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.657+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/2596407382 shutdown_connections 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.657+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/2596407382 wait complete. 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.658+0000 7fd2c79aeb80 1 -- start start 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.658+0000 7fd2c79aeb80 1 Processor -- start 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.658+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d16fd1e40 0x562d1715a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.658+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d1715ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.658+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d1715b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.658+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562d16f74fa0 con 0x562d170bd9f0 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.658+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562d1700f760 con 0x562d17150e70 2024-05-27T09:42:02.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.658+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562d1715db20 con 0x562d16fd1e40 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2c79aeb80 10 client.0 _ll_register_callbacks cb 0x562d17018c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2b1400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d1715ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2b1400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d1715ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59212/0 (socket says 172.21.15.44:59212) 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2b1400700 1 -- 172.21.15.44:0/1915643001 learned_addr learned my addr 172.21.15.44:0/1915643001 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2b2800700 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d1715b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d16fd1e40 0x562d1715a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2b1400700 1 -- 172.21.15.44:0/1915643001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d16fd1e40 msgr2=0x562d1715a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2b1400700 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d16fd1e40 0x562d1715a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2b1400700 1 -- 172.21.15.44:0/1915643001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 msgr2=0x562d1715b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2b1400700 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d1715b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2b1400700 1 -- 172.21.15.44:0/1915643001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d16fbe8d0 con 0x562d170bd9f0 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.659+0000 7fd2b2800700 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d1715b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:02.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.660+0000 7fd2b1400700 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d1715ac00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0012d20 tx=0x7fd2a000f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ff10a6c9410410 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.660+0000 7fd293e00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.660+0000 7fd293e00700 1 -- 172.21.15.44:0/1915643001 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a000fb80 con 0x562d170bd9f0 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.660+0000 7fd293e00700 1 -- 172.21.15.44:0/1915643001 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7fd2a000fd20 con 0x562d170bd9f0 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.660+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/1915643001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2a4091040 con 0x562d170bd9f0 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.660+0000 7fd293e00700 1 -- 172.21.15.44:0/1915643001 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd2a0004030 con 0x562d170bd9f0 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.660+0000 7fd2c79aeb80 10 client.16248 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.660+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/1915643001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562d1703f060 con 0x562d170bd9f0 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.660+0000 7fd293e00700 1 -- 172.21.15.44:0/1915643001 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2a4091040 con 0x562d170bd9f0 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.661+0000 7fd293e00700 1 -- 172.21.15.44:0/1915643001 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fd2a001c460 con 0x562d170bd9f0 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.661+0000 7fd2c79aeb80 10 client.16248 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.661+0000 7fd2c79aeb80 10 client.16248 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.661+0000 7fd2c79aeb80 20 client.16248 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:02.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.661+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x562d1716aae0 0x562d1716cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.661+0000 7fd2c79aeb80 4 client.16248 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.661+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/1915643001 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d17080d90 con 0x562d1716aae0 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.661+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x562d1716aae0 0x562d1716cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.662+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x562d1716aae0 0x562d1716cf00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ff7298cffb567449 server_cookie=e7e107d52e12028e in_seq=0 out_seq=0 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.662+0000 7fd293e00700 10 client.16248 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd293e00700 1 -- 172.21.15.44:0/1915643001 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562d17080d90 con 0x562d1716aae0 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd293e00700 10 client.16248 handle_command_reply: tid=0 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 1 client.16248 shutdown 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/1915643001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 msgr2=0x562d1715ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d1715ac00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0012d20 tx=0x7fd2a000f6a0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/1915643001 shutdown_connections 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d1715b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d16fd1e40 0x562d1715a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x562d1716aae0 0x562d1716cf00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/1915643001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d170bd9f0 0x562d1715ac00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/1915643001 >> 172.21.15.44:0/1915643001 conn(0x562d1706b390 msgr2=0x562d17150370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:02.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd293e00700 0 client.16248 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:42:02.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/1915643001 shutdown_connections 2024-05-27T09:42:02.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/1915643001 wait complete. 2024-05-27T09:42:02.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.663+0000 7fd2c79aeb80 20 client.16248 trim_cache size 0 max 16384 2024-05-27T09:42:02.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.681+0000 7fd2c79aeb80 1 Processor -- start 2024-05-27T09:42:02.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.681+0000 7fd2c79aeb80 1 -- start start 2024-05-27T09:42:02.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.681+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d17155f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.681+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d170bd9f0 0x562d17159620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.681+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d17150e70 0x562d17166bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.681+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41b38c0 con 0x562d16fd1e40 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.681+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41b1120 con 0x562d170bd9f0 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.681+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41aef70 con 0x562d17150e70 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d17155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b1400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d170bd9f0 0x562d17159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b1400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d170bd9f0 0x562d17159620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60238/0 (socket says 172.21.15.44:60238) 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d17155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59226/0 (socket says 172.21.15.44:59226) 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b1400700 1 -- 172.21.15.44:0/3060619768 learned_addr learned my addr 172.21.15.44:0/3060619768 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b2800700 1 --2- 172.21.15.44:0/3060619768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d17150e70 0x562d17166bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b1400700 1 -- 172.21.15.44:0/3060619768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d17150e70 msgr2=0x562d17166bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b1400700 1 --2- 172.21.15.44:0/3060619768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d17150e70 0x562d17166bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b1400700 1 -- 172.21.15.44:0/3060619768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 msgr2=0x562d17155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b1400700 1 --2- 172.21.15.44:0/3060619768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d17155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b1400700 1 -- 172.21.15.44:0/3060619768 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d1714d000 con 0x562d170bd9f0 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.682+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/3060619768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d17155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.683+0000 7fd2b1400700 1 --2- 172.21.15.44:0/3060619768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d170bd9f0 0x562d17159620 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd2a000a010 tx=0x7fd2a000f9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c04f94b5d01d32b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.683+0000 7fd29ea00700 1 -- 172.21.15.44:0/3060619768 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a001c460 con 0x562d170bd9f0 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.683+0000 7fd29ea00700 1 -- 172.21.15.44:0/3060619768 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2a001c600 con 0x562d170bd9f0 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.683+0000 7fd29ea00700 1 -- 172.21.15.44:0/3060619768 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00055d0 con 0x562d170bd9f0 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.684+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3060619768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d170bd9f0 msgr2=0x562d17159620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.684+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/3060619768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d170bd9f0 0x562d17159620 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd2a000a010 tx=0x7fd2a000f9f0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.684+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3060619768 shutdown_connections 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.684+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/3060619768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d17150e70 0x562d17166bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.684+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/3060619768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d17155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.684+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/3060619768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d170bd9f0 0x562d17159620 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.684+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3060619768 >> 172.21.15.44:0/3060619768 conn(0x562d1706b390 msgr2=0x562d1715bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.684+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3060619768 shutdown_connections 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.684+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3060619768 wait complete. 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.684+0000 7fd2c79aeb80 1 -- start start 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.685+0000 7fd2c79aeb80 1 Processor -- start 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.685+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d171547b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.685+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d170bd9f0 0x562d17154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.685+0000 7fd2c79aeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d17161db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.685+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41af5e0 con 0x562d16fd1e40 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.685+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41b0a50 con 0x562d17150e70 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.685+0000 7fd2c79aeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41afcb0 con 0x562d170bd9f0 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.685+0000 7fd2c79aeb80 10 client.0 _ll_register_callbacks cb 0x562d17018c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:02.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd2b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d171547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd2b1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d171547b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59238/0 (socket says 172.21.15.44:59238) 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd2b1e00700 1 -- 172.21.15.44:0/3445571597 learned_addr learned my addr 172.21.15.44:0/3445571597 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd2b1400700 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d170bd9f0 0x562d17154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd2b2800700 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d17161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd2b1e00700 1 -- 172.21.15.44:0/3445571597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d170bd9f0 msgr2=0x562d17154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d170bd9f0 0x562d17154cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd2b1e00700 1 -- 172.21.15.44:0/3445571597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 msgr2=0x562d17161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d17161db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd2b1e00700 1 -- 172.21.15.44:0/3445571597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d171555d0 con 0x562d16fd1e40 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd2b1e00700 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d171547b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400c5b0 tx=0x7fd2a40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3c02ee72da4bb3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.686+0000 7fd293e00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:42:02.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.687+0000 7fd293e00700 1 -- 172.21.15.44:0/3445571597 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a40a53f0 con 0x562d16fd1e40 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.687+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3445571597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2a001c600 con 0x562d16fd1e40 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.687+0000 7fd293e00700 1 -- 172.21.15.44:0/3445571597 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7fd2a407f2c0 con 0x562d16fd1e40 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.687+0000 7fd293e00700 1 -- 172.21.15.44:0/3445571597 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd2a407cd10 con 0x562d16fd1e40 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.687+0000 7fd2c79aeb80 10 client.16254 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.687+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3445571597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562d171633a0 con 0x562d16fd1e40 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.687+0000 7fd293e00700 1 -- 172.21.15.44:0/3445571597 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2a001c600 con 0x562d16fd1e40 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.687+0000 7fd293e00700 1 -- 172.21.15.44:0/3445571597 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fd2a40c4020 con 0x562d16fd1e40 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.687+0000 7fd2c79aeb80 10 client.16254 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.687+0000 7fd2c79aeb80 10 client.16254 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.688+0000 7fd2c79aeb80 20 client.16254 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.688+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x562d17163630 0x562d171d7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.688+0000 7fd2c79aeb80 4 client.16254 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:42:02.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.688+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3445571597 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562d1707a570 con 0x562d17163630 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.688+0000 7fd2b1400700 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x562d17163630 0x562d171d7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.688+0000 7fd2b1400700 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x562d17163630 0x562d171d7580 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2fd0ecbdbb8ae0a server_cookie=bb6d78eeb28a0f88 in_seq=0 out_seq=0 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.688+0000 7fd293e00700 10 client.16254 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd293e00700 1 -- 172.21.15.44:0/3445571597 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562d1707a570 con 0x562d17163630 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd293e00700 10 client.16254 handle_command_reply: tid=0 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd2c79aeb80 1 client.16254 shutdown 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3445571597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 msgr2=0x562d171547b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d171547b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400c5b0 tx=0x7fd2a40047c0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3445571597 shutdown_connections 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d17150e70 0x562d17161db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x562d17163630 0x562d171d7580 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d16fd1e40 0x562d171547b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd2c79aeb80 1 --2- 172.21.15.44:0/3445571597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d170bd9f0 0x562d17154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd293e00700 0 client.16254 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:42:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3445571597 >> 172.21.15.44:0/3445571597 conn(0x562d1706b390 msgr2=0x562d1715a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3445571597 shutdown_connections 2024-05-27T09:42:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.689+0000 7fd2c79aeb80 1 -- 172.21.15.44:0/3445571597 wait complete. 2024-05-27T09:42:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.690+0000 7fd2c79aeb80 20 client.16254 trim_cache size 0 max 16384 2024-05-27T09:42:02.962 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:42:02.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.690+0000 7fd2b8e00700 1 -- 172.21.15.44:0/3646451222 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2980b2a70 msgr2=0x7fd2980b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.690+0000 7fd2b8e00700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2980b2a70 0x7fd2980b4f80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00b780 tx=0x7fd2ac009d80 comp rx=0 tx=0).stop 2024-05-27T09:42:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.690+0000 7fd2b8e00700 1 -- 172.21.15.44:0/3646451222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b4105450 msgr2=0x7fd2b419cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.690+0000 7fd2b8e00700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b4105450 0x7fd2b419cf60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400eb30 tx=0x7fd2a400ee40 comp rx=0 tx=0).stop 2024-05-27T09:42:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.690+0000 7fd2b8e00700 1 -- 172.21.15.44:0/3646451222 shutdown_connections 2024-05-27T09:42:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.690+0000 7fd2b8e00700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2980b2a70 0x7fd2980b4f80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.690+0000 7fd2b8e00700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b410e8c0 0x7fd2b41a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.690+0000 7fd2b8e00700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b4105450 0x7fd2b419cf60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.690+0000 7fd2b8e00700 1 --2- 172.21.15.44:0/3646451222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2b4105da0 0x7fd2b419d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.690+0000 7fd2b8e00700 1 -- 172.21.15.44:0/3646451222 >> 172.21.15.44:0/3646451222 conn(0x7fd2b40fedb0 msgr2=0x7fd2b41030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.691+0000 7fd2b8e00700 1 -- 172.21.15.44:0/3646451222 shutdown_connections 2024-05-27T09:42:02.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:02.691+0000 7fd2b8e00700 1 -- 172.21.15.44:0/3646451222 wait complete. 2024-05-27T09:42:03.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:03 smithi044 ceph-mon[31618]: pgmap v208: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-05-27T09:42:03.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:03 smithi050 ceph-mon[36371]: pgmap v208: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-05-27T09:42:03.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:03 smithi189 ceph-mon[36671]: pgmap v208: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-05-27T09:42:03.840 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:42:03.963 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 2 (c) to a remote file /var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.c.json 2024-05-27T09:42:03.963 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 ops --flags=locks --path=/var/run/ceph/ops-77ef2bd1-1-mds.c.json 2024-05-27T09:42:04.321 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:05.890 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:05 smithi044 ceph-mon[31618]: pgmap v209: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-05-27T09:42:06.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:05 smithi050 ceph-mon[36371]: pgmap v209: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-05-27T09:42:06.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:05 smithi189 ceph-mon[36671]: pgmap v209: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-05-27T09:42:07.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.583+0000 7f70c8c00700 1 -- 172.21.15.44:0/937548653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70c4077ce0 msgr2=0x7f70c40780f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:07.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.583+0000 7f70c8c00700 1 --2- 172.21.15.44:0/937548653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70c4077ce0 0x7f70c40780f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009990 tx=0x7f70b4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.583+0000 7f70c8c00700 1 -- 172.21.15.44:0/937548653 shutdown_connections 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.583+0000 7f70c8c00700 1 --2- 172.21.15.44:0/937548653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c4115040 0x7f70c4117460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.583+0000 7f70c8c00700 1 --2- 172.21.15.44:0/937548653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70c4077ce0 0x7f70c40780f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.583+0000 7f70c8c00700 1 --2- 172.21.15.44:0/937548653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70c4078630 0x7f70c4114b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.583+0000 7f70c8c00700 1 -- 172.21.15.44:0/937548653 >> 172.21.15.44:0/937548653 conn(0x7f70c4102020 msgr2=0x7f70c4104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.583+0000 7f70c8c00700 1 -- 172.21.15.44:0/937548653 shutdown_connections 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.583+0000 7f70c8c00700 1 -- 172.21.15.44:0/937548653 wait complete. 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.585+0000 7f70c8c00700 1 Processor -- start 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.586+0000 7f70c8c00700 1 -- start start 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.586+0000 7f70c8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70c4077ce0 0x7f70c41a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.586+0000 7f70c8c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70c4078630 0x7f70c41a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.586+0000 7f70c8c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c4115040 0x7f70c41a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c8c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f70c411a8e0 con 0x7f70c4077ce0 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c8c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f70c411a760 con 0x7f70c4078630 2024-05-27T09:42:07.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c8c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f70c411aa60 con 0x7f70c4115040 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70c4078630 0x7f70c41a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70c4077ce0 0x7f70c41a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c4115040 0x7f70c41a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70c4078630 0x7f70c41a17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36830/0 (socket says 172.21.15.44:36830) 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c2a00700 1 -- 172.21.15.44:0/3721108100 learned_addr learned my addr 172.21.15.44:0/3721108100 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c3e00700 1 -- 172.21.15.44:0/3721108100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70c4078630 msgr2=0x7f70c41a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c3e00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70c4078630 0x7f70c41a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c3e00700 1 -- 172.21.15.44:0/3721108100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70c4077ce0 msgr2=0x7f70c41a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c3e00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70c4077ce0 0x7f70c41a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.587+0000 7f70c3e00700 1 -- 172.21.15.44:0/3721108100 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c41a6120 con 0x7f70c4115040 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.588+0000 7f70c3e00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c4115040 0x7f70c41a5a80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00efc0 tx=0x7f70bc00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.588+0000 7f70bbe00700 1 -- 172.21.15.44:0/3721108100 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc0092e0 con 0x7f70c4115040 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.588+0000 7f70bbe00700 1 -- 172.21.15.44:0/3721108100 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70bc004bb0 con 0x7f70c4115040 2024-05-27T09:42:07.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.588+0000 7f70c8c00700 1 -- 172.21.15.44:0/3721108100 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c41a63b0 con 0x7f70c4115040 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.588+0000 7f70bbe00700 1 -- 172.21.15.44:0/3721108100 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc004020 con 0x7f70c4115040 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.588+0000 7f70c8c00700 1 -- 172.21.15.44:0/3721108100 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c41add30 con 0x7f70c4115040 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.592+0000 7f70bbe00700 1 -- 172.21.15.44:0/3721108100 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f70bc01b030 con 0x7f70c4115040 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.593+0000 7f70d8ca3b80 1 Processor -- start 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.593+0000 7f70d8ca3b80 1 -- start start 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d42750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860bc2e40 0x55d860d42c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860d4c2c0 0x55d860d4e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d860c6fe40 con 0x55d860cae9f0 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d860c5f9e0 con 0x55d860d4c2c0 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d860c63a20 con 0x55d860bc2e40 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70bbe00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f70a80b2a70 0x7f70a80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70c2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d42750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70c2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d42750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50660/0 (socket says 172.21.15.44:50660) 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70c2a00700 1 -- 172.21.15.44:0/211641836 learned_addr learned my addr 172.21.15.44:0/211641836 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70c3400700 1 --2- 172.21.15.44:0/211641836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860bc2e40 0x55d860d42c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70bbe00700 1 -- 172.21.15.44:0/3721108100 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7f70bc0e2060 con 0x7f70c4115040 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70c3e00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f70a80b2a70 0x7f70a80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.594+0000 7f70c2a00700 1 --2- 172.21.15.44:0/211641836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860d4c2c0 0x55d860d4e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.595+0000 7f70c2a00700 1 -- 172.21.15.44:0/211641836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860bc2e40 msgr2=0x55d860d42c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.595+0000 7f70c2a00700 1 --2- 172.21.15.44:0/211641836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860bc2e40 0x55d860d42c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.595+0000 7f70c2a00700 1 -- 172.21.15.44:0/211641836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860d4c2c0 msgr2=0x55d860d4e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.595+0000 7f70c2a00700 1 --2- 172.21.15.44:0/211641836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860d4c2c0 0x55d860d4e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.595+0000 7f70c2a00700 1 -- 172.21.15.44:0/211641836 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d860c28420 con 0x55d860cae9f0 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.595+0000 7f70c2a00700 1 --2- 172.21.15.44:0/211641836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d42750 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f70b0009e80 tx=0x7f70b0009980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c043c40a413db2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:07.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.595+0000 7f70c3e00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f70a80b2a70 0x7f70a80b4f80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009fb0 tx=0x7f70bc009040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.595+0000 7f70b8c00700 1 -- 172.21.15.44:0/211641836 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b0010430 con 0x55d860cae9f0 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.595+0000 7f70b8c00700 1 -- 172.21.15.44:0/211641836 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70b0010a80 con 0x55d860cae9f0 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.595+0000 7f70b8c00700 1 -- 172.21.15.44:0/211641836 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b000e930 con 0x55d860cae9f0 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.596+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/211641836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 msgr2=0x55d860d42750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.596+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/211641836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d42750 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f70b0009e80 tx=0x7f70b0009980 comp rx=0 tx=0).stop 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.596+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/211641836 shutdown_connections 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.596+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/211641836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860d4c2c0 0x55d860d4e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.596+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/211641836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d42750 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.596+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/211641836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860bc2e40 0x55d860d42c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.596+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/211641836 >> 172.21.15.44:0/211641836 conn(0x55d860c5c390 msgr2=0x55d860d3d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.597+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/211641836 shutdown_connections 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.597+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/211641836 wait complete. 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.597+0000 7f70d8ca3b80 1 -- start start 2024-05-27T09:42:07.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.598+0000 7f70d8ca3b80 1 Processor -- start 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.598+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860bc2e40 0x55d860d49240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.598+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d49780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.598+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860d4c2c0 0x55d860d49cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.598+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d860b65fa0 con 0x55d860cae9f0 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.598+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d860c00760 con 0x55d860bc2e40 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.598+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d860d4eda0 con 0x55d860d4c2c0 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.598+0000 7f70d8ca3b80 10 client.0 _ll_register_callbacks cb 0x55d860c09c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.599+0000 7f70c2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860bc2e40 0x55d860d49240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.599+0000 7f70c3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d49780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.599+0000 7f70c2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860bc2e40 0x55d860d49240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36846/0 (socket says 172.21.15.44:36846) 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.599+0000 7f70c2a00700 1 -- 172.21.15.44:0/774351033 learned_addr learned my addr 172.21.15.44:0/774351033 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.599+0000 7f70c3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d49780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50664/0 (socket says 172.21.15.44:50664) 2024-05-27T09:42:07.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.599+0000 7f70c2a00700 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860d4c2c0 0x55d860d49cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.599+0000 7f70c3400700 1 -- 172.21.15.44:0/774351033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860d4c2c0 msgr2=0x55d860d49cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.599+0000 7f70c3400700 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860d4c2c0 0x55d860d49cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.599+0000 7f70c3400700 1 -- 172.21.15.44:0/774351033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860bc2e40 msgr2=0x55d860d49240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.599+0000 7f70c3400700 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860bc2e40 0x55d860d49240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.599+0000 7f70c3400700 1 -- 172.21.15.44:0/774351033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d860baf8d0 con 0x55d860cae9f0 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.600+0000 7f70c3400700 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d49780 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f70b400b2e0 tx=0x7f70b4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59654a3eac471582 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.600+0000 7f70ad600700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.600+0000 7f70ad600700 1 -- 172.21.15.44:0/774351033 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b400bd00 con 0x55d860cae9f0 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.600+0000 7f70ad600700 1 -- 172.21.15.44:0/774351033 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7f70b4003680 con 0x55d860cae9f0 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.600+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/774351033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70b0010a80 con 0x55d860cae9f0 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.600+0000 7f70ad600700 1 -- 172.21.15.44:0/774351033 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f70b4031050 con 0x55d860cae9f0 2024-05-27T09:42:07.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.600+0000 7f70d8ca3b80 10 client.16266 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.600+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/774351033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d860c30060 con 0x55d860cae9f0 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.600+0000 7f70ad600700 1 -- 172.21.15.44:0/774351033 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70b0010a80 con 0x55d860cae9f0 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.601+0000 7f70ad600700 1 -- 172.21.15.44:0/774351033 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f70b40237f0 con 0x55d860cae9f0 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.601+0000 7f70d8ca3b80 10 client.16266 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.601+0000 7f70d8ca3b80 10 client.16266 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.601+0000 7f70d8ca3b80 20 client.16266 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.601+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55d860d5bcf0 0x55d860d5e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.601+0000 7f70d8ca3b80 4 client.16266 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.601+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/774351033 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d860c71d90 con 0x55d860d5bcf0 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.601+0000 7f70c2a00700 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55d860d5bcf0 0x55d860d5e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.602+0000 7f70c2a00700 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55d860d5bcf0 0x55d860d5e110 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5ddc97d98379983f server_cookie=7a743e3f617642b5 in_seq=0 out_seq=0 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.602+0000 7f70ad600700 10 client.16266 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:42:07.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70ad600700 1 -- 172.21.15.44:0/774351033 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d860c71d90 con 0x55d860d5bcf0 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70ad600700 10 client.16266 handle_command_reply: tid=0 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 1 client.16266 shutdown 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/774351033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 msgr2=0x55d860d49780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d49780 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f70b400b2e0 tx=0x7f70b4005800 comp rx=0 tx=0).stop 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/774351033 shutdown_connections 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860d4c2c0 0x55d860d49cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860bc2e40 0x55d860d49240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55d860d5bcf0 0x55d860d5e110 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/774351033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860cae9f0 0x55d860d49780 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/774351033 >> 172.21.15.44:0/774351033 conn(0x55d860c5c390 msgr2=0x55d860d3d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70ad600700 0 client.16266 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/774351033 shutdown_connections 2024-05-27T09:42:07.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/774351033 wait complete. 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.603+0000 7f70d8ca3b80 20 client.16266 trim_cache size 0 max 16384 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.621+0000 7f70d8ca3b80 1 Processor -- start 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.621+0000 7f70d8ca3b80 1 -- start start 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.621+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 0x55d860d44aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.621+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860cae9f0 0x55d860d481a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.621+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860d4c2c0 0x55d860d57e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.621+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41b79d0 con 0x55d860bc2e40 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.621+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41b5230 con 0x55d860cae9f0 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.621+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41b30a0 con 0x55d860d4c2c0 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.622+0000 7f70c3400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860cae9f0 0x55d860d481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.622+0000 7f70c3400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860cae9f0 0x55d860d481a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36854/0 (socket says 172.21.15.44:36854) 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.622+0000 7f70c3400700 1 -- 172.21.15.44:0/533413201 learned_addr learned my addr 172.21.15.44:0/533413201 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.622+0000 7f70c2a00700 1 --2- 172.21.15.44:0/533413201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 0x55d860d44aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.622+0000 7f70c3400700 1 -- 172.21.15.44:0/533413201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860d4c2c0 msgr2=0x55d860d57e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.622+0000 7f70c3400700 1 --2- 172.21.15.44:0/533413201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860d4c2c0 0x55d860d57e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.622+0000 7f70c3400700 1 -- 172.21.15.44:0/533413201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 msgr2=0x55d860d44aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.622+0000 7f70c3400700 1 --2- 172.21.15.44:0/533413201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 0x55d860d44aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.622+0000 7f70c3400700 1 -- 172.21.15.44:0/533413201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d860d43d60 con 0x55d860cae9f0 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.622+0000 7f70c2a00700 1 --2- 172.21.15.44:0/533413201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 0x55d860d44aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.623+0000 7f70c3400700 1 --2- 172.21.15.44:0/533413201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860cae9f0 0x55d860d481a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f70b402f040 tx=0x7f70b4009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80f0f98f155ea024 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.623+0000 7f70afe00700 1 -- 172.21.15.44:0/533413201 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b400ba00 con 0x55d860cae9f0 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.623+0000 7f70afe00700 1 -- 172.21.15.44:0/533413201 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70b4039020 con 0x55d860cae9f0 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.623+0000 7f70afe00700 1 -- 172.21.15.44:0/533413201 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b40038e0 con 0x55d860cae9f0 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.624+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/533413201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860cae9f0 msgr2=0x55d860d481a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.624+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/533413201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860cae9f0 0x55d860d481a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f70b402f040 tx=0x7f70b4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.624+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/533413201 shutdown_connections 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.624+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/533413201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860d4c2c0 0x55d860d57e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.624+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/533413201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 0x55d860d44aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.624+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/533413201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860cae9f0 0x55d860d481a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.624+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/533413201 >> 172.21.15.44:0/533413201 conn(0x55d860c5c390 msgr2=0x55d860d4a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.624+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/533413201 shutdown_connections 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.624+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/533413201 wait complete. 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.625+0000 7f70d8ca3b80 1 -- start start 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.625+0000 7f70d8ca3b80 1 Processor -- start 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.626+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 0x55d860d4f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.626+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860cae9f0 0x55d860d4f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.626+0000 7f70d8ca3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860d4c2c0 0x55d860dc4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.626+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41b36f0 con 0x55d860bc2e40 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.626+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41b4b60 con 0x55d860d4c2c0 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.626+0000 7f70d8ca3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41b3dc0 con 0x55d860cae9f0 2024-05-27T09:42:07.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.626+0000 7f70d8ca3b80 10 client.0 _ll_register_callbacks cb 0x55d860c09c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.626+0000 7f70c2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 0x55d860d4f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.626+0000 7f70c2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 0x55d860d4f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50668/0 (socket says 172.21.15.44:50668) 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.626+0000 7f70c2a00700 1 -- 172.21.15.44:0/1505486990 learned_addr learned my addr 172.21.15.44:0/1505486990 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.626+0000 7f70c3400700 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860cae9f0 0x55d860d4f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.627+0000 7f70c2a00700 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860d4c2c0 0x55d860dc4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.627+0000 7f70c3400700 1 -- 172.21.15.44:0/1505486990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860d4c2c0 msgr2=0x55d860dc4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.627+0000 7f70c3400700 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860d4c2c0 0x55d860dc4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.627+0000 7f70c3400700 1 -- 172.21.15.44:0/1505486990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 msgr2=0x55d860d4f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.627+0000 7f70c3400700 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 0x55d860d4f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.627+0000 7f70c3400700 1 -- 172.21.15.44:0/1505486990 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d860dc4cc0 con 0x55d860cae9f0 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.627+0000 7f70c2a00700 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 0x55d860d4f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.627+0000 7f70c3400700 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860cae9f0 0x55d860d4f940 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f70b4023bc0 tx=0x7f70b4005ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec7459b4638d105b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:07.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.627+0000 7f70ad600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.628+0000 7f70ad600700 1 -- 172.21.15.44:0/1505486990 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4023880 con 0x55d860cae9f0 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.628+0000 7f70ad600700 1 -- 172.21.15.44:0/1505486990 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7f70b4003680 con 0x55d860cae9f0 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.628+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/1505486990 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70b4039020 con 0x55d860cae9f0 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.628+0000 7f70ad600700 1 -- 172.21.15.44:0/1505486990 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f70b404c050 con 0x55d860cae9f0 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.628+0000 7f70d8ca3b80 10 client.26434 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.628+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/1505486990 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d860dc5d80 con 0x55d860cae9f0 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.629+0000 7f70ad600700 1 -- 172.21.15.44:0/1505486990 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70b4039020 con 0x55d860cae9f0 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.629+0000 7f70ad600700 1 -- 172.21.15.44:0/1505486990 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f70b400f920 con 0x55d860cae9f0 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.629+0000 7f70d8ca3b80 10 client.26434 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.629+0000 7f70d8ca3b80 10 client.26434 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.629+0000 7f70d8ca3b80 20 client.26434 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.629+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55d860dc6240 0x55d860dc8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.629+0000 7f70d8ca3b80 4 client.26434 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:42:07.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.629+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/1505486990 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d860c6b570 con 0x55d860dc6240 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.630+0000 7f70c2a00700 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55d860dc6240 0x55d860dc8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.630+0000 7f70c2a00700 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55d860dc6240 0x55d860dc8660 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=69808242c1cf632f server_cookie=4412042ee55a4d5c in_seq=0 out_seq=0 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.630+0000 7f70ad600700 10 client.26434 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.630+0000 7f70ad600700 1 -- 172.21.15.44:0/1505486990 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d860c6b570 con 0x55d860dc6240 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.630+0000 7f70ad600700 10 client.26434 handle_command_reply: tid=0 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.630+0000 7f70d8ca3b80 1 client.26434 shutdown 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/1505486990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860cae9f0 msgr2=0x55d860d4f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860cae9f0 0x55d860d4f940 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f70b4023bc0 tx=0x7f70b4005ac0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/1505486990 shutdown_connections 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d860d4c2c0 0x55d860dc4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55d860dc6240 0x55d860dc8660 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d860bc2e40 0x55d860d4f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70d8ca3b80 1 --2- 172.21.15.44:0/1505486990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d860cae9f0 0x55d860d4f940 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70ad600700 0 client.26434 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:42:07.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/1505486990 >> 172.21.15.44:0/1505486990 conn(0x55d860c5c390 msgr2=0x55d860d44800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:07.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/1505486990 shutdown_connections 2024-05-27T09:42:07.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70d8ca3b80 1 -- 172.21.15.44:0/1505486990 wait complete. 2024-05-27T09:42:07.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.631+0000 7f70d8ca3b80 20 client.26434 trim_cache size 0 max 16384 2024-05-27T09:42:07.270 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.632+0000 7f70c8c00700 1 -- 172.21.15.44:0/3721108100 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f70a80b2a70 msgr2=0x7f70a80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.632+0000 7f70c8c00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f70a80b2a70 0x7f70a80b4f80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009fb0 tx=0x7f70bc009040 comp rx=0 tx=0).stop 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.632+0000 7f70c8c00700 1 -- 172.21.15.44:0/3721108100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c4115040 msgr2=0x7f70c41a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.632+0000 7f70c8c00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c4115040 0x7f70c41a5a80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00efc0 tx=0x7f70bc00c3a0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.632+0000 7f70c8c00700 1 -- 172.21.15.44:0/3721108100 shutdown_connections 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.632+0000 7f70c8c00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f70a80b2a70 0x7f70a80b4f80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.632+0000 7f70c8c00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c4115040 0x7f70c41a5a80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.632+0000 7f70c8c00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70c4077ce0 0x7f70c41a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.632+0000 7f70c8c00700 1 --2- 172.21.15.44:0/3721108100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70c4078630 0x7f70c41a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.632+0000 7f70c8c00700 1 -- 172.21.15.44:0/3721108100 >> 172.21.15.44:0/3721108100 conn(0x7f70c4102020 msgr2=0x7f70c407ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.633+0000 7f70c8c00700 1 -- 172.21.15.44:0/3721108100 shutdown_connections 2024-05-27T09:42:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:06.633+0000 7f70c8c00700 1 -- 172.21.15.44:0/3721108100 wait complete. 2024-05-27T09:42:08.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:07 smithi044 ceph-mon[31618]: pgmap v210: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 60 KiB/s wr, 28 op/s 2024-05-27T09:42:08.006 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:42:08.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:07 smithi050 ceph-mon[36371]: pgmap v210: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 60 KiB/s wr, 28 op/s 2024-05-27T09:42:08.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:07 smithi189 ceph-mon[36671]: pgmap v210: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 60 KiB/s wr, 28 op/s 2024-05-27T09:42:08.641 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:08.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.724+0000 7ff2d4a00700 1 -- 172.21.15.44:0/3629423821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d01045e0 msgr2=0x7ff2d01049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:08.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.724+0000 7ff2d4a00700 1 --2- 172.21.15.44:0/3629423821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d01045e0 0x7ff2d01049f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00099c0 tx=0x7ff2c0009cd0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.724+0000 7ff2d4a00700 1 -- 172.21.15.44:0/3629423821 shutdown_connections 2024-05-27T09:42:08.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.724+0000 7ff2d4a00700 1 --2- 172.21.15.44:0/3629423821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d0109860 0x7ff2d01111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.724+0000 7ff2d4a00700 1 --2- 172.21.15.44:0/3629423821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d01045e0 0x7ff2d01049f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.725+0000 7ff2d4a00700 1 --2- 172.21.15.44:0/3629423821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d0108f10 0x7ff2d0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.725+0000 7ff2d4a00700 1 -- 172.21.15.44:0/3629423821 >> 172.21.15.44:0/3629423821 conn(0x7ff2d0100030 msgr2=0x7ff2d0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:08.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.725+0000 7ff2d4a00700 1 -- 172.21.15.44:0/3629423821 shutdown_connections 2024-05-27T09:42:08.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.725+0000 7ff2d4a00700 1 -- 172.21.15.44:0/3629423821 wait complete. 2024-05-27T09:42:08.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.726+0000 7ff2d4a00700 1 Processor -- start 2024-05-27T09:42:08.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.727+0000 7ff2d4a00700 1 -- start start 2024-05-27T09:42:08.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.727+0000 7ff2d4a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d01045e0 0x7ff2d019aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.727+0000 7ff2d4a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d0108f10 0x7ff2d019b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.727+0000 7ff2d4a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d0109860 0x7ff2d019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.727+0000 7ff2d4a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01144e0 con 0x7ff2d0108f10 2024-05-27T09:42:08.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.727+0000 7ff2d4a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d0114360 con 0x7ff2d01045e0 2024-05-27T09:42:08.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.727+0000 7ff2d4a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d0114660 con 0x7ff2d0109860 2024-05-27T09:42:08.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.728+0000 7ff2cd400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d0108f10 0x7ff2d019b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.728+0000 7ff2cd400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d0108f10 0x7ff2d019b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50686/0 (socket says 172.21.15.44:50686) 2024-05-27T09:42:08.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.728+0000 7ff2cd400700 1 -- 172.21.15.44:0/151524497 learned_addr learned my addr 172.21.15.44:0/151524497 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:08.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.728+0000 7ff2cde00700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d01045e0 0x7ff2d019aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.728+0000 7ff2ce800700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d0109860 0x7ff2d019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.728+0000 7ff2cd400700 1 -- 172.21.15.44:0/151524497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d0109860 msgr2=0x7ff2d019f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:08.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.728+0000 7ff2cd400700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d0109860 0x7ff2d019f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.728+0000 7ff2cd400700 1 -- 172.21.15.44:0/151524497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d01045e0 msgr2=0x7ff2d019aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:08.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.728+0000 7ff2cd400700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d01045e0 0x7ff2d019aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.728+0000 7ff2cd400700 1 -- 172.21.15.44:0/151524497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d019fd80 con 0x7ff2d0108f10 2024-05-27T09:42:08.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.729+0000 7ff2cd400700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d0108f10 0x7ff2d019b450 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc00d950 tx=0x7ff2bc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:08.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.732+0000 7ff2c6a00700 1 -- 172.21.15.44:0/151524497 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc016070 con 0x7ff2d0108f10 2024-05-27T09:42:08.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.733+0000 7ff2d4a00700 1 -- 172.21.15.44:0/151524497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d01a0070 con 0x7ff2d0108f10 2024-05-27T09:42:08.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.733+0000 7ff2c6a00700 1 -- 172.21.15.44:0/151524497 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2bc004830 con 0x7ff2d0108f10 2024-05-27T09:42:08.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.733+0000 7ff2c6a00700 1 -- 172.21.15.44:0/151524497 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc005520 con 0x7ff2d0108f10 2024-05-27T09:42:08.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.733+0000 7ff2d4a00700 1 -- 172.21.15.44:0/151524497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d01a7990 con 0x7ff2d0108f10 2024-05-27T09:42:08.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.738+0000 7ff2c6a00700 1 -- 172.21.15.44:0/151524497 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff2bc004220 con 0x7ff2d0108f10 2024-05-27T09:42:08.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.738+0000 7ff2e349fb80 1 Processor -- start 2024-05-27T09:42:08.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.738+0000 7ff2e349fb80 1 -- start start 2024-05-27T09:42:08.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.739+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561522966ad0 0x561522a00510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.739+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x561522a00a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.739+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615229fa5c0 0x5615229fa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.739+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561522927f20 con 0x561522966ad0 2024-05-27T09:42:08.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.739+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561522917ac0 con 0x56152287af20 2024-05-27T09:42:08.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.739+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56152291bb00 con 0x5615229fa5c0 2024-05-27T09:42:08.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.739+0000 7ff2ce800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561522966ad0 0x561522a00510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.739+0000 7ff2cd400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615229fa5c0 0x5615229fa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.739+0000 7ff2ce800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561522966ad0 0x561522a00510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50692/0 (socket says 172.21.15.44:50692) 2024-05-27T09:42:08.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.739+0000 7ff2ce800700 1 -- 172.21.15.44:0/2936048676 learned_addr learned my addr 172.21.15.44:0/2936048676 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:08.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.739+0000 7ff2cde00700 1 --2- 172.21.15.44:0/2936048676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x561522a00a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.740+0000 7ff2c6a00700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b40b2a70 0x7ff2b40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.740+0000 7ff2c6a00700 1 -- 172.21.15.44:0/151524497 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7ff2bc0e22c0 con 0x7ff2d0108f10 2024-05-27T09:42:08.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.740+0000 7ff2cde00700 1 -- 172.21.15.44:0/2936048676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615229fa5c0 msgr2=0x5615229fa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:08.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.740+0000 7ff2cde00700 1 --2- 172.21.15.44:0/2936048676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615229fa5c0 0x5615229fa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.740+0000 7ff2cde00700 1 -- 172.21.15.44:0/2936048676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561522966ad0 msgr2=0x561522a00510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:08.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.740+0000 7ff2cde00700 1 --2- 172.21.15.44:0/2936048676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561522966ad0 0x561522a00510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.740+0000 7ff2cde00700 1 -- 172.21.15.44:0/2936048676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615228e0500 con 0x56152287af20 2024-05-27T09:42:08.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.740+0000 7ff2ce800700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b40b2a70 0x7ff2b40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.740+0000 7ff2cde00700 1 --2- 172.21.15.44:0/2936048676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x561522a00a70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009990 tx=0x7ff2c0005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4b1a18301016efe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:08.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.741+0000 7ff2bb400700 1 -- 172.21.15.44:0/2936048676 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0027070 con 0x56152287af20 2024-05-27T09:42:08.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.741+0000 7ff2bb400700 1 -- 172.21.15.44:0/2936048676 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c00037c0 con 0x56152287af20 2024-05-27T09:42:08.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.741+0000 7ff2bb400700 1 -- 172.21.15.44:0/2936048676 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c000b9f0 con 0x56152287af20 2024-05-27T09:42:08.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.742+0000 7ff2e349fb80 1 -- 172.21.15.44:0/2936048676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 msgr2=0x561522a00a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:08.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.742+0000 7ff2ce800700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b40b2a70 0x7ff2b40b4f80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff2c80076c0 tx=0x7ff2c8009040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:08.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.742+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/2936048676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x561522a00a70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009990 tx=0x7ff2c0005040 comp rx=0 tx=0).stop 2024-05-27T09:42:08.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.742+0000 7ff2e349fb80 1 -- 172.21.15.44:0/2936048676 shutdown_connections 2024-05-27T09:42:08.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.742+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/2936048676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615229fa5c0 0x5615229fa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.742+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/2936048676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561522966ad0 0x561522a00510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.742+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/2936048676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x561522a00a70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.742+0000 7ff2e349fb80 1 -- 172.21.15.44:0/2936048676 >> 172.21.15.44:0/2936048676 conn(0x561522914470 msgr2=0x5615229f5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:08.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.742+0000 7ff2e349fb80 1 -- 172.21.15.44:0/2936048676 shutdown_connections 2024-05-27T09:42:08.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.742+0000 7ff2e349fb80 1 -- 172.21.15.44:0/2936048676 wait complete. 2024-05-27T09:42:08.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.743+0000 7ff2e349fb80 1 -- start start 2024-05-27T09:42:08.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.743+0000 7ff2e349fb80 1 Processor -- start 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56152287af20 0x561522a042e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a04820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615229fa5c0 0x561522a04d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56152281e080 con 0x56152287af20 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5615228b8840 con 0x5615229fa5c0 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5615229fb350 con 0x561522966ad0 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2e349fb80 10 client.0 _ll_register_callbacks cb 0x5615228c1d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2cde00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56152287af20 0x561522a042e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2cde00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56152287af20 0x561522a042e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50704/0 (socket says 172.21.15.44:50704) 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2cde00700 1 -- 172.21.15.44:0/1020486148 learned_addr learned my addr 172.21.15.44:0/1020486148 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2cd400700 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a04820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.744+0000 7ff2ce800700 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615229fa5c0 0x561522a04d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.745+0000 7ff2ce800700 1 -- 172.21.15.44:0/1020486148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 msgr2=0x561522a04820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:08.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.745+0000 7ff2ce800700 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a04820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.745+0000 7ff2ce800700 1 -- 172.21.15.44:0/1020486148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56152287af20 msgr2=0x561522a042e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.745+0000 7ff2ce800700 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56152287af20 0x561522a042e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.745+0000 7ff2ce800700 1 -- 172.21.15.44:0/1020486148 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615228679b0 con 0x5615229fa5c0 2024-05-27T09:42:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.745+0000 7ff2ce800700 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615229fa5c0 0x561522a04d60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8017df0 tx=0x7ff2c8004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5e0aef94c7fac7f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.745+0000 7ff2afe00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.745+0000 7ff2afe00700 1 -- 172.21.15.44:0/1020486148 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8029060 con 0x5615229fa5c0 2024-05-27T09:42:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.745+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1020486148 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2c00037c0 con 0x5615229fa5c0 2024-05-27T09:42:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.745+0000 7ff2afe00700 1 -- 172.21.15.44:0/1020486148 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7ff2c8003a00 con 0x5615229fa5c0 2024-05-27T09:42:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.746+0000 7ff2afe00700 1 -- 172.21.15.44:0/1020486148 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2c8003c00 con 0x5615229fa5c0 2024-05-27T09:42:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.746+0000 7ff2e349fb80 10 client.26444 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.746+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1020486148 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5615228e8140 con 0x5615229fa5c0 2024-05-27T09:42:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.746+0000 7ff2afe00700 1 -- 172.21.15.44:0/1020486148 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c00037c0 con 0x5615229fa5c0 2024-05-27T09:42:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.746+0000 7ff2afe00700 1 -- 172.21.15.44:0/1020486148 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7ff2c802d3e0 con 0x5615229fa5c0 2024-05-27T09:42:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.746+0000 7ff2e349fb80 10 client.26444 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.746+0000 7ff2e349fb80 10 client.26444 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:42:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.746+0000 7ff2e349fb80 20 client.26444 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.746+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x561522a183f0 0x561522a1a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.746+0000 7ff2e349fb80 4 client.26444 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.746+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1020486148 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561522929e70 con 0x561522a183f0 2024-05-27T09:42:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.747+0000 7ff2cde00700 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x561522a183f0 0x561522a1a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.748+0000 7ff2cde00700 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x561522a183f0 0x561522a1a830 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6e7d0e1c6e606351 server_cookie=9062d84951428165 in_seq=0 out_seq=0 2024-05-27T09:42:08.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.748+0000 7ff2afe00700 10 client.26444 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:42:08.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.748+0000 7ff2afe00700 1 -- 172.21.15.44:0/1020486148 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561522929e70 con 0x561522a183f0 2024-05-27T09:42:08.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.748+0000 7ff2afe00700 10 client.26444 handle_command_reply: tid=0 2024-05-27T09:42:08.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 1 client.26444 shutdown 2024-05-27T09:42:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1020486148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615229fa5c0 msgr2=0x561522a04d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615229fa5c0 0x561522a04d60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8017df0 tx=0x7ff2c8004ad0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1020486148 shutdown_connections 2024-05-27T09:42:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615229fa5c0 0x561522a04d60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x561522a183f0 0x561522a1a830 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56152287af20 0x561522a042e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1020486148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a04820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2afe00700 0 client.26444 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:42:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1020486148 >> 172.21.15.44:0/1020486148 conn(0x561522914470 msgr2=0x5615229f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1020486148 shutdown_connections 2024-05-27T09:42:08.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1020486148 wait complete. 2024-05-27T09:42:08.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.749+0000 7ff2e349fb80 20 client.26444 trim_cache size 0 max 16384 2024-05-27T09:42:08.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.770+0000 7ff2e349fb80 1 Processor -- start 2024-05-27T09:42:08.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.770+0000 7ff2e349fb80 1 -- start start 2024-05-27T09:42:08.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.770+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x5615229fe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.770+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5615229fa5c0 0x561522a04d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01af570 con 0x5615229fa5c0 2024-05-27T09:42:08.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01acdd0 con 0x56152287af20 2024-05-27T09:42:08.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01aac20 con 0x561522966ad0 2024-05-27T09:42:08.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2cd400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2cd400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:58964/0 (socket says 172.21.15.44:58964) 2024-05-27T09:42:08.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2cd400700 1 -- 172.21.15.44:0/3934703679 learned_addr learned my addr 172.21.15.44:0/3934703679 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:08.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2cde00700 1 --2- 172.21.15.44:0/3934703679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x5615229fe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2ce800700 1 --2- 172.21.15.44:0/3934703679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5615229fa5c0 0x561522a04d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2cd400700 1 -- 172.21.15.44:0/3934703679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 msgr2=0x5615229fe9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:08.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2cd400700 1 --2- 172.21.15.44:0/3934703679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x5615229fe9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2cd400700 1 -- 172.21.15.44:0/3934703679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5615229fa5c0 msgr2=0x561522a04d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2cd400700 1 --2- 172.21.15.44:0/3934703679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5615229fa5c0 0x561522a04d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.771+0000 7ff2cd400700 1 -- 172.21.15.44:0/3934703679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561522a15ec0 con 0x561522966ad0 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.772+0000 7ff2cd400700 1 --2- 172.21.15.44:0/3934703679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a047d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc007770 tx=0x7ff2bc0076b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54a73e020d165d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.772+0000 7ff2baa00700 1 -- 172.21.15.44:0/3934703679 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc0a9080 con 0x561522966ad0 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.772+0000 7ff2baa00700 1 -- 172.21.15.44:0/3934703679 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2bc0a9c70 con 0x561522966ad0 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.772+0000 7ff2baa00700 1 -- 172.21.15.44:0/3934703679 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc0a97d0 con 0x561522966ad0 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.773+0000 7ff2e349fb80 1 -- 172.21.15.44:0/3934703679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 msgr2=0x561522a047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.773+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/3934703679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a047d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc007770 tx=0x7ff2bc0076b0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.773+0000 7ff2e349fb80 1 -- 172.21.15.44:0/3934703679 shutdown_connections 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.773+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/3934703679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5615229fa5c0 0x561522a04d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.773+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/3934703679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x5615229fe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.773+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/3934703679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a047d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.773+0000 7ff2e349fb80 1 -- 172.21.15.44:0/3934703679 >> 172.21.15.44:0/3934703679 conn(0x561522914470 msgr2=0x561522a040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.773+0000 7ff2e349fb80 1 -- 172.21.15.44:0/3934703679 shutdown_connections 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.773+0000 7ff2e349fb80 1 -- 172.21.15.44:0/3934703679 wait complete. 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.774+0000 7ff2e349fb80 1 -- start start 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.774+0000 7ff2e349fb80 1 Processor -- start 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x561522a0c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a0cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2e349fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5615229fa5c0 0x561522a0d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01ab290 con 0x5615229fa5c0 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01ac700 con 0x56152287af20 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2e349fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01ab960 con 0x561522966ad0 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2e349fb80 10 client.0 _ll_register_callbacks cb 0x5615228c1d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2cde00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x561522a0c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2cde00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x561522a0c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36928/0 (socket says 172.21.15.44:36928) 2024-05-27T09:42:08.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2cde00700 1 -- 172.21.15.44:0/1672240710 learned_addr learned my addr 172.21.15.44:0/1672240710 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:08.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2cd400700 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a0cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2ce800700 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5615229fa5c0 0x561522a0d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2cde00700 1 -- 172.21.15.44:0/1672240710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 msgr2=0x561522a0cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:08.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2cde00700 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a0cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2cde00700 1 -- 172.21.15.44:0/1672240710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5615229fa5c0 msgr2=0x561522a0d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:08.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2cde00700 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5615229fa5c0 0x561522a0d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.775+0000 7ff2cde00700 1 -- 172.21.15.44:0/1672240710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561522a820b0 con 0x56152287af20 2024-05-27T09:42:08.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.776+0000 7ff2cde00700 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x561522a0c710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000b580 tx=0x7ff2c000b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bccd52c924b9d99a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:08.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.776+0000 7ff2afe00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.776+0000 7ff2afe00700 1 -- 172.21.15.44:0/1672240710 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0027040 con 0x56152287af20 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.776+0000 7ff2afe00700 1 -- 172.21.15.44:0/1672240710 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8477+0+0 (secure 0 0 0) 0x7ff2c00054c0 con 0x56152287af20 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.776+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1672240710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2bc0a9c70 con 0x56152287af20 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.776+0000 7ff2afe00700 1 -- 172.21.15.44:0/1672240710 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2c0040050 con 0x56152287af20 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.776+0000 7ff2e349fb80 10 client.26450 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.776+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1672240710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561522a832f0 con 0x56152287af20 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.777+0000 7ff2afe00700 1 -- 172.21.15.44:0/1672240710 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2bc0a9c70 con 0x56152287af20 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.777+0000 7ff2afe00700 1 -- 172.21.15.44:0/1672240710 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7ff2c0021030 con 0x56152287af20 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.777+0000 7ff2e349fb80 10 client.26450 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.777+0000 7ff2e349fb80 10 client.26450 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.777+0000 7ff2e349fb80 20 client.26450 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.777+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x561522a834c0 0x561522a858e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.777+0000 7ff2e349fb80 4 client.26450 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-77ef2bd1-1-mds.c.json"}] 2024-05-27T09:42:08.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.777+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1672240710 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-77ef2bd1-1-mds.c.json"}) -- 0x5615229ed7f0 con 0x561522a834c0 2024-05-27T09:42:08.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.778+0000 7ff2cd400700 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x561522a834c0 0x561522a858e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:08.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.778+0000 7ff2cd400700 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x561522a834c0 0x561522a858e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=345bdab5a378d1f8 server_cookie=394167c905049921 in_seq=0 out_seq=0 2024-05-27T09:42:08.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.778+0000 7ff2afe00700 10 client.26450 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:42:08.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2afe00700 1 -- 172.21.15.44:0/1672240710 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x5615229ed7f0 con 0x561522a834c0 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2afe00700 10 client.26450 handle_command_reply: tid=0 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 1 client.26450 shutdown 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1672240710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 msgr2=0x561522a0c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x561522a0c710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000b580 tx=0x7ff2c000b670 comp rx=0 tx=0).stop 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1672240710 shutdown_connections 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x561522a834c0 0x561522a858e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5615229fa5c0 0x561522a0d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56152287af20 0x561522a0c710 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 1 --2- 172.21.15.44:0/1672240710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561522966ad0 0x561522a0cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2afe00700 0 client.26450 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1672240710 >> 172.21.15.44:0/1672240710 conn(0x561522914470 msgr2=0x561522a04af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1672240710 shutdown_connections 2024-05-27T09:42:08.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 1 -- 172.21.15.44:0/1672240710 wait complete. 2024-05-27T09:42:08.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.780+0000 7ff2e349fb80 20 client.26450 trim_cache size 0 max 16384 2024-05-27T09:42:08.783 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:42:08.783 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/var/run/ceph/ops-77ef2bd1-1-mds.c.json" 2024-05-27T09:42:08.783 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:42:08.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.781+0000 7ff2d4a00700 1 -- 172.21.15.44:0/151524497 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b40b2a70 msgr2=0x7ff2b40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:08.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.781+0000 7ff2d4a00700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b40b2a70 0x7ff2b40b4f80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff2c80076c0 tx=0x7ff2c8009040 comp rx=0 tx=0).stop 2024-05-27T09:42:08.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.781+0000 7ff2d4a00700 1 -- 172.21.15.44:0/151524497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d0108f10 msgr2=0x7ff2d019b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:08.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.781+0000 7ff2d4a00700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d0108f10 0x7ff2d019b450 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc00d950 tx=0x7ff2bc00dc60 comp rx=0 tx=0).stop 2024-05-27T09:42:08.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.781+0000 7ff2d4a00700 1 -- 172.21.15.44:0/151524497 shutdown_connections 2024-05-27T09:42:08.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.781+0000 7ff2d4a00700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2b40b2a70 0x7ff2b40b4f80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.781+0000 7ff2d4a00700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d0109860 0x7ff2d019f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.781+0000 7ff2d4a00700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2d01045e0 0x7ff2d019aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.781+0000 7ff2d4a00700 1 --2- 172.21.15.44:0/151524497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2d0108f10 0x7ff2d019b450 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:08.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.781+0000 7ff2d4a00700 1 -- 172.21.15.44:0/151524497 >> 172.21.15.44:0/151524497 conn(0x7ff2d0100030 msgr2=0x7ff2d0108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:08.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.782+0000 7ff2d4a00700 1 -- 172.21.15.44:0/151524497 shutdown_connections 2024-05-27T09:42:08.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:08.782+0000 7ff2d4a00700 1 -- 172.21.15.44:0/151524497 wait complete. 2024-05-27T09:42:09.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:09 smithi044 ceph-mon[31618]: pgmap v211: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 57 KiB/s wr, 25 op/s 2024-05-27T09:42:10.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:09 smithi050 ceph-mon[36371]: pgmap v211: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 57 KiB/s wr, 25 op/s 2024-05-27T09:42:10.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:09 smithi189 ceph-mon[36671]: pgmap v211: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 57 KiB/s wr, 25 op/s 2024-05-27T09:42:10.873 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 1 (i) to a remote file /var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.i.json 2024-05-27T09:42:10.873 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 ops --flags=locks --path=/var/run/ceph/ops-77ef2bd1-1-mds.i.json 2024-05-27T09:42:11.537 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:11.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.571+0000 7f4801c00700 1 -- 172.21.15.44:0/3235923313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47fc104400 msgr2=0x7f47fc104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.571+0000 7f4801c00700 1 --2- 172.21.15.44:0/3235923313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47fc104400 0x7f47fc104810 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009990 tx=0x7f47ec009ca0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.571+0000 7f4801c00700 1 -- 172.21.15.44:0/3235923313 shutdown_connections 2024-05-27T09:42:11.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.571+0000 7f4801c00700 1 --2- 172.21.15.44:0/3235923313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47fc109680 0x7f47fc110fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.571+0000 7f4801c00700 1 --2- 172.21.15.44:0/3235923313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47fc104400 0x7f47fc104810 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.571+0000 7f4801c00700 1 --2- 172.21.15.44:0/3235923313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47fc108d30 0x7f47fc109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.571+0000 7f4801c00700 1 -- 172.21.15.44:0/3235923313 >> 172.21.15.44:0/3235923313 conn(0x7f47fc079c10 msgr2=0x7f47fc07a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.571+0000 7f4801c00700 1 -- 172.21.15.44:0/3235923313 shutdown_connections 2024-05-27T09:42:11.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.571+0000 7f4801c00700 1 -- 172.21.15.44:0/3235923313 wait complete. 2024-05-27T09:42:11.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.573+0000 7f4801c00700 1 Processor -- start 2024-05-27T09:42:11.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.573+0000 7f4801c00700 1 -- start start 2024-05-27T09:42:11.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.574+0000 7f4801c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47fc104400 0x7f47fc1a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.574+0000 7f4801c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47fc108d30 0x7f47fc1a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.574+0000 7f4801c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47fc109680 0x7f47fc19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.574+0000 7f4801c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc114300 con 0x7f47fc109680 2024-05-27T09:42:11.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.574+0000 7f4801c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc114180 con 0x7f47fc104400 2024-05-27T09:42:11.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.574+0000 7f4801c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc114480 con 0x7f47fc108d30 2024-05-27T09:42:11.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.574+0000 7f47fbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47fc104400 0x7f47fc1a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.574+0000 7f47fbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47fc104400 0x7f47fc1a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36944/0 (socket says 172.21.15.44:36944) 2024-05-27T09:42:11.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.574+0000 7f47fbe00700 1 -- 172.21.15.44:0/967485392 learned_addr learned my addr 172.21.15.44:0/967485392 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:11.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.574+0000 7f4801200700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47fc109680 0x7f47fc19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.574+0000 7f47fbe00700 1 -- 172.21.15.44:0/967485392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47fc108d30 msgr2=0x7f47fc1a0e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:42:11.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.575+0000 7f47fbe00700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47fc108d30 0x7f47fc1a0e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.575+0000 7f47fbe00700 1 -- 172.21.15.44:0/967485392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47fc109680 msgr2=0x7f47fc19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.575+0000 7f47fbe00700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47fc109680 0x7f47fc19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.575+0000 7f47fbe00700 1 -- 172.21.15.44:0/967485392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc19b1e0 con 0x7f47fc104400 2024-05-27T09:42:11.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.575+0000 7f4801200700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47fc109680 0x7f47fc19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:11.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.575+0000 7f47fbe00700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47fc104400 0x7f47fc1a08c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f47ec005910 tx=0x7f47ec00bfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.575+0000 7f47f8c00700 1 -- 172.21.15.44:0/967485392 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ec027070 con 0x7f47fc104400 2024-05-27T09:42:11.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.576+0000 7f47f8c00700 1 -- 172.21.15.44:0/967485392 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47ec00b610 con 0x7f47fc104400 2024-05-27T09:42:11.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.576+0000 7f4801c00700 1 -- 172.21.15.44:0/967485392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47fc19b470 con 0x7f47fc104400 2024-05-27T09:42:11.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.576+0000 7f47f8c00700 1 -- 172.21.15.44:0/967485392 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ec0173f0 con 0x7f47fc104400 2024-05-27T09:42:11.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.576+0000 7f4801c00700 1 -- 172.21.15.44:0/967485392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47fc1a7770 con 0x7f47fc104400 2024-05-27T09:42:11.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.580+0000 7f47f8c00700 1 -- 172.21.15.44:0/967485392 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f47ec009d10 con 0x7f47fc104400 2024-05-27T09:42:11.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.580+0000 7f4811dd1b80 1 Processor -- start 2024-05-27T09:42:11.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.581+0000 7f4811dd1b80 1 -- start start 2024-05-27T09:42:11.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.581+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 0x563312926450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.581+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312926990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.581+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x5633129366e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.581+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563312853e40 con 0x563312926f70 2024-05-27T09:42:11.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.581+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5633128439e0 con 0x5633127a6e40 2024-05-27T09:42:11.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.581+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563312847a20 con 0x5633128929f0 2024-05-27T09:42:11.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.582+0000 7f47fbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312926990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.582+0000 7f47fb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x5633129366e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.582+0000 7f47f8c00700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f47e00b2a70 0x7f47e00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.582+0000 7f47fbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312926990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36950/0 (socket says 172.21.15.44:36950) 2024-05-27T09:42:11.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.582+0000 7f47fb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x5633129366e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50746/0 (socket says 172.21.15.44:50746) 2024-05-27T09:42:11.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.582+0000 7f47fbe00700 1 -- 172.21.15.44:0/824845611 learned_addr learned my addr 172.21.15.44:0/824845611 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:11.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.582+0000 7f4801200700 1 --2- 172.21.15.44:0/824845611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 0x563312926450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.582+0000 7f47f8c00700 1 -- 172.21.15.44:0/967485392 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7f47ec0e3270 con 0x7f47fc104400 2024-05-27T09:42:11.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.582+0000 7f4801200700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f47e00b2a70 0x7f47e00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.583+0000 7f4801200700 1 -- 172.21.15.44:0/824845611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 msgr2=0x563312926990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.583+0000 7f4801200700 1 --2- 172.21.15.44:0/824845611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312926990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.583+0000 7f4801200700 1 -- 172.21.15.44:0/824845611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 msgr2=0x5633129366e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.583+0000 7f4801200700 1 --2- 172.21.15.44:0/824845611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x5633129366e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.583+0000 7f4801200700 1 -- 172.21.15.44:0/824845611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56331280c420 con 0x5633128929f0 2024-05-27T09:42:11.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.584+0000 7f4801200700 1 --2- 172.21.15.44:0/824845611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 0x563312926450 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f47f4004460 tx=0x7f47f4004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5d4776ec88c2162 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.584+0000 7f4801200700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f47e00b2a70 0x7f47e00b4f80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009000 tx=0x7f47f401e000 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.584+0000 7f47f1600700 1 -- 172.21.15.44:0/824845611 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4007700 con 0x5633128929f0 2024-05-27T09:42:11.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.584+0000 7f47f1600700 1 -- 172.21.15.44:0/824845611 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47f4009430 con 0x5633128929f0 2024-05-27T09:42:11.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.584+0000 7f47f1600700 1 -- 172.21.15.44:0/824845611 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4022940 con 0x5633128929f0 2024-05-27T09:42:11.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.585+0000 7f4811dd1b80 1 -- 172.21.15.44:0/824845611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 msgr2=0x563312926450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.585+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/824845611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 0x563312926450 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f47f4004460 tx=0x7f47f4004770 comp rx=0 tx=0).stop 2024-05-27T09:42:11.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.585+0000 7f4811dd1b80 1 -- 172.21.15.44:0/824845611 shutdown_connections 2024-05-27T09:42:11.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.585+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/824845611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x5633129366e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.585+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/824845611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 0x563312926450 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.585+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/824845611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312926990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.585+0000 7f4811dd1b80 1 -- 172.21.15.44:0/824845611 >> 172.21.15.44:0/824845611 conn(0x563312840390 msgr2=0x563312921710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.586+0000 7f4811dd1b80 1 -- 172.21.15.44:0/824845611 shutdown_connections 2024-05-27T09:42:11.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.586+0000 7f4811dd1b80 1 -- 172.21.15.44:0/824845611 wait complete. 2024-05-27T09:42:11.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.586+0000 7f4811dd1b80 1 -- start start 2024-05-27T09:42:11.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.587+0000 7f4811dd1b80 1 Processor -- start 2024-05-27T09:42:11.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.587+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312933d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.587+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 0x5633129342c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.587+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x563312934800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.587+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563312749fa0 con 0x563312926f70 2024-05-27T09:42:11.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.587+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5633127e4760 con 0x5633127a6e40 2024-05-27T09:42:11.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.587+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563312937130 con 0x5633128929f0 2024-05-27T09:42:11.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.587+0000 7f4811dd1b80 10 client.0 _ll_register_callbacks cb 0x5633127edc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:11.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.588+0000 7f4801200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x563312934800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.588+0000 7f47fbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312933d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.588+0000 7f47fb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 0x5633129342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.588+0000 7f47fbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312933d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36958/0 (socket says 172.21.15.44:36958) 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.588+0000 7f47fbe00700 1 -- 172.21.15.44:0/224972156 learned_addr learned my addr 172.21.15.44:0/224972156 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.588+0000 7f47fb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 0x5633129342c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:58994/0 (socket says 172.21.15.44:58994) 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.588+0000 7f4801200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x563312934800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50754/0 (socket says 172.21.15.44:50754) 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.589+0000 7f47fb400700 1 -- 172.21.15.44:0/224972156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 msgr2=0x563312933d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.589+0000 7f47fb400700 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312933d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.589+0000 7f47fb400700 1 -- 172.21.15.44:0/224972156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 msgr2=0x563312934800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.589+0000 7f47fb400700 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x563312934800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.589+0000 7f47fb400700 1 -- 172.21.15.44:0/224972156 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633127938d0 con 0x5633128929f0 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.589+0000 7f47fbe00700 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312933d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:11.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.589+0000 7f47fb400700 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 0x5633129342c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f47fc06ce90 tx=0x7f47e8011790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d30abd558015267 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.589+0000 7f47e6000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:42:11.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.589+0000 7f47e6000700 1 -- 172.21.15.44:0/224972156 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8004520 con 0x5633128929f0 2024-05-27T09:42:11.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.589+0000 7f4811dd1b80 1 -- 172.21.15.44:0/224972156 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47f4009430 con 0x5633128929f0 2024-05-27T09:42:11.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.589+0000 7f47e6000700 1 -- 172.21.15.44:0/224972156 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7f47e8004830 con 0x5633128929f0 2024-05-27T09:42:11.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.590+0000 7f47e6000700 1 -- 172.21.15.44:0/224972156 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f47e8004a30 con 0x5633128929f0 2024-05-27T09:42:11.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.590+0000 7f4811dd1b80 10 client.26461 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:11.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.590+0000 7f4811dd1b80 1 -- 172.21.15.44:0/224972156 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563312814060 con 0x5633128929f0 2024-05-27T09:42:11.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.590+0000 7f47e6000700 1 -- 172.21.15.44:0/224972156 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47f4009430 con 0x5633128929f0 2024-05-27T09:42:11.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.590+0000 7f47e6000700 1 -- 172.21.15.44:0/224972156 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f47e8018070 con 0x5633128929f0 2024-05-27T09:42:11.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.590+0000 7f4811dd1b80 10 client.26461 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:11.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.590+0000 7f4811dd1b80 10 client.26461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:11.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.591+0000 7f4811dd1b80 20 client.26461 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:11.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.591+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56331292a860 0x56331292acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.591+0000 7f4811dd1b80 4 client.26461 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:11.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.591+0000 7f4811dd1b80 1 -- 172.21.15.44:0/224972156 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563312855d90 con 0x56331292a860 2024-05-27T09:42:11.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.591+0000 7f47fbe00700 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56331292a860 0x56331292acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.592+0000 7f47fbe00700 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56331292a860 0x56331292acc0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc3ab7a1f4f3c918 server_cookie=1199d2364d08faf6 in_seq=0 out_seq=0 2024-05-27T09:42:11.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.592+0000 7f47e6000700 10 client.26461 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:11.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.592+0000 7f47e6000700 1 -- 172.21.15.44:0/224972156 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563312855d90 con 0x56331292a860 2024-05-27T09:42:11.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.592+0000 7f47e6000700 10 client.26461 handle_command_reply: tid=0 2024-05-27T09:42:11.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.593+0000 7f4811dd1b80 1 client.26461 shutdown 2024-05-27T09:42:11.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.593+0000 7f4811dd1b80 1 -- 172.21.15.44:0/224972156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 msgr2=0x5633129342c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.593+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 0x5633129342c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f47fc06ce90 tx=0x7f47e8011790 comp rx=0 tx=0).stop 2024-05-27T09:42:11.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.593+0000 7f4811dd1b80 1 -- 172.21.15.44:0/224972156 shutdown_connections 2024-05-27T09:42:11.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.593+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56331292a860 0x56331292acc0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.593+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x563312934800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.593+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312933d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.593+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/224972156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633128929f0 0x5633129342c0 secure :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f47fc06ce90 tx=0x7f47e8011790 comp rx=0 tx=0).stop 2024-05-27T09:42:11.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.593+0000 7f4811dd1b80 1 -- 172.21.15.44:0/224972156 >> 172.21.15.44:0/224972156 conn(0x563312840390 msgr2=0x5633129214e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.593+0000 7f47e6000700 0 client.26461 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:11.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.594+0000 7f4811dd1b80 1 -- 172.21.15.44:0/224972156 shutdown_connections 2024-05-27T09:42:11.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.594+0000 7f4811dd1b80 1 -- 172.21.15.44:0/224972156 wait complete. 2024-05-27T09:42:11.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.594+0000 7f4811dd1b80 20 client.26461 trim_cache size 0 max 16384 2024-05-27T09:42:11.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.612+0000 7f4811dd1b80 1 Processor -- start 2024-05-27T09:42:11.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.612+0000 7f4811dd1b80 1 -- start start 2024-05-27T09:42:11.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.612+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312926980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.612+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x563312940400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.613+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331292f630 0x563312932d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.613+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc1af2e0 con 0x563312926f70 2024-05-27T09:42:11.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.613+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc1acb40 con 0x5633127a6e40 2024-05-27T09:42:11.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.613+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc1aa9b0 con 0x56331292f630 2024-05-27T09:42:11.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.613+0000 7f47fb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x563312940400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.613+0000 7f47fbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312926980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.613+0000 7f4801200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331292f630 0x563312932d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.613+0000 7f4801200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331292f630 0x563312932d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59008/0 (socket says 172.21.15.44:59008) 2024-05-27T09:42:11.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.613+0000 7f47fbe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312926980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36974/0 (socket says 172.21.15.44:36974) 2024-05-27T09:42:11.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.613+0000 7f4801200700 1 -- 172.21.15.44:0/3524589047 learned_addr learned my addr 172.21.15.44:0/3524589047 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:11.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.614+0000 7f47fb400700 1 -- 172.21.15.44:0/3524589047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331292f630 msgr2=0x563312932d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.614+0000 7f47fb400700 1 --2- 172.21.15.44:0/3524589047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331292f630 0x563312932d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.614+0000 7f47fb400700 1 -- 172.21.15.44:0/3524589047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 msgr2=0x563312926980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.614+0000 7f47fb400700 1 --2- 172.21.15.44:0/3524589047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312926980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.614+0000 7f47fb400700 1 -- 172.21.15.44:0/3524589047 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633129282e0 con 0x563312926f70 2024-05-27T09:42:11.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.614+0000 7f47fbe00700 1 --2- 172.21.15.44:0/3524589047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312926980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:11.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.614+0000 7f47fb400700 1 --2- 172.21.15.44:0/3524589047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x563312940400 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x563312935350 tx=0x7f47e80216a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd0667c62b31b90 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.615+0000 7f47f0c00700 1 -- 172.21.15.44:0/3524589047 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e802e070 con 0x563312926f70 2024-05-27T09:42:11.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.615+0000 7f47f0c00700 1 -- 172.21.15.44:0/3524589047 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47e8003680 con 0x563312926f70 2024-05-27T09:42:11.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.615+0000 7f47f0c00700 1 -- 172.21.15.44:0/3524589047 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8029670 con 0x563312926f70 2024-05-27T09:42:11.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.616+0000 7f4811dd1b80 1 -- 172.21.15.44:0/3524589047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 msgr2=0x563312940400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.616+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/3524589047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x563312940400 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x563312935350 tx=0x7f47e80216a0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.616+0000 7f4811dd1b80 1 -- 172.21.15.44:0/3524589047 shutdown_connections 2024-05-27T09:42:11.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.616+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/3524589047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331292f630 0x563312932d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.616+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/3524589047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5633127a6e40 0x563312926980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.616+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/3524589047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563312926f70 0x563312940400 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.616+0000 7f4811dd1b80 1 -- 172.21.15.44:0/3524589047 >> 172.21.15.44:0/3524589047 conn(0x563312840390 msgr2=0x563312932670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.616+0000 7f4811dd1b80 1 -- 172.21.15.44:0/3524589047 shutdown_connections 2024-05-27T09:42:11.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.616+0000 7f4811dd1b80 1 -- 172.21.15.44:0/3524589047 wait complete. 2024-05-27T09:42:11.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.617+0000 7f4811dd1b80 1 -- start start 2024-05-27T09:42:11.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.617+0000 7f4811dd1b80 1 Processor -- start 2024-05-27T09:42:11.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633127a6e40 0x56331294b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563312926f70 0x56331294b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f4811dd1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331292f630 0x56331294fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc1ab000 con 0x5633127a6e40 2024-05-27T09:42:11.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc1ac470 con 0x563312926f70 2024-05-27T09:42:11.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f4811dd1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc1ab6d0 con 0x56331292f630 2024-05-27T09:42:11.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f4811dd1b80 10 client.0 _ll_register_callbacks cb 0x5633127edc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:11.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f47fbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633127a6e40 0x56331294b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f47fbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633127a6e40 0x56331294b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50780/0 (socket says 172.21.15.44:50780) 2024-05-27T09:42:11.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f47fbe00700 1 -- 172.21.15.44:0/2425454533 learned_addr learned my addr 172.21.15.44:0/2425454533 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:11.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f47fb400700 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563312926f70 0x56331294b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.618+0000 7f4801200700 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331292f630 0x56331294fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.619+0000 7f47fb400700 1 -- 172.21.15.44:0/2425454533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331292f630 msgr2=0x56331294fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.619+0000 7f47fb400700 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331292f630 0x56331294fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.619+0000 7f47fb400700 1 -- 172.21.15.44:0/2425454533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633127a6e40 msgr2=0x56331294b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.619+0000 7f47fb400700 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633127a6e40 0x56331294b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.619+0000 7f47fb400700 1 -- 172.21.15.44:0/2425454533 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563312934b70 con 0x563312926f70 2024-05-27T09:42:11.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.619+0000 7f47fb400700 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563312926f70 0x56331294b980 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f47e8005950 tx=0x7f47e8011ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95391c375bf4f327 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.619+0000 7f47e6000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:11.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.620+0000 7f47e6000700 1 -- 172.21.15.44:0/2425454533 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8037020 con 0x563312926f70 2024-05-27T09:42:11.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.620+0000 7f47e6000700 1 -- 172.21.15.44:0/2425454533 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7f47e8004320 con 0x563312926f70 2024-05-27T09:42:11.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.620+0000 7f4811dd1b80 1 -- 172.21.15.44:0/2425454533 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47e8003680 con 0x563312926f70 2024-05-27T09:42:11.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.620+0000 7f47e6000700 1 -- 172.21.15.44:0/2425454533 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f47e802e050 con 0x563312926f70 2024-05-27T09:42:11.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.620+0000 7f4811dd1b80 10 client.26468 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:11.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.620+0000 7f4811dd1b80 1 -- 172.21.15.44:0/2425454533 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5633129b26c0 con 0x563312926f70 2024-05-27T09:42:11.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.620+0000 7f47e6000700 1 -- 172.21.15.44:0/2425454533 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47e8003680 con 0x563312926f70 2024-05-27T09:42:11.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.621+0000 7f47e6000700 1 -- 172.21.15.44:0/2425454533 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f47e8018040 con 0x563312926f70 2024-05-27T09:42:11.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.621+0000 7f4811dd1b80 10 client.26468 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.621+0000 7f4811dd1b80 10 client.26468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.621+0000 7f4811dd1b80 20 client.26468 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.621+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5633129b2cb0 0x5633129b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.621+0000 7f4811dd1b80 4 client.26468 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.621+0000 7f4811dd1b80 1 -- 172.21.15.44:0/2425454533 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56331292b270 con 0x5633129b2cb0 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.621+0000 7f47fbe00700 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5633129b2cb0 0x5633129b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f47fbe00700 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5633129b2cb0 0x5633129b50d0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=365b4f597000776d server_cookie=773dc3f413224316 in_seq=0 out_seq=0 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f47e6000700 10 client.26468 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f47e6000700 1 -- 172.21.15.44:0/2425454533 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56331292b270 con 0x5633129b2cb0 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f47e6000700 10 client.26468 handle_command_reply: tid=0 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f4811dd1b80 1 client.26468 shutdown 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f4811dd1b80 1 -- 172.21.15.44:0/2425454533 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563312926f70 msgr2=0x56331294b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563312926f70 0x56331294b980 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f47e8005950 tx=0x7f47e8011ae0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f4811dd1b80 1 -- 172.21.15.44:0/2425454533 shutdown_connections 2024-05-27T09:42:11.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5633129b2cb0 0x5633129b50d0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f47e6000700 0 client.26468 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:11.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331292f630 0x56331294fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5633127a6e40 0x56331294b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f4811dd1b80 1 --2- 172.21.15.44:0/2425454533 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563312926f70 0x56331294b980 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.622+0000 7f4811dd1b80 1 -- 172.21.15.44:0/2425454533 >> 172.21.15.44:0/2425454533 conn(0x563312840390 msgr2=0x563312925120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.623+0000 7f4811dd1b80 1 -- 172.21.15.44:0/2425454533 shutdown_connections 2024-05-27T09:42:11.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.623+0000 7f4811dd1b80 1 -- 172.21.15.44:0/2425454533 wait complete. 2024-05-27T09:42:11.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.623+0000 7f4811dd1b80 20 client.26468 trim_cache size 0 max 16384 2024-05-27T09:42:11.626 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:42:11.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.623+0000 7f4801c00700 1 -- 172.21.15.44:0/967485392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f47e00b2a70 msgr2=0x7f47e00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.623+0000 7f4801c00700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f47e00b2a70 0x7f47e00b4f80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009000 tx=0x7f47f401e000 comp rx=0 tx=0).stop 2024-05-27T09:42:11.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.623+0000 7f4801c00700 1 -- 172.21.15.44:0/967485392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47fc104400 msgr2=0x7f47fc1a08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.624+0000 7f4801c00700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47fc104400 0x7f47fc1a08c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f47ec005910 tx=0x7f47ec00bfc0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.624+0000 7f4801c00700 1 -- 172.21.15.44:0/967485392 shutdown_connections 2024-05-27T09:42:11.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.624+0000 7f4801c00700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f47e00b2a70 0x7f47e00b4f80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.624+0000 7f4801c00700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47fc109680 0x7f47fc19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.624+0000 7f4801c00700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47fc104400 0x7f47fc1a08c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.624+0000 7f4801c00700 1 --2- 172.21.15.44:0/967485392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47fc108d30 0x7f47fc1a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.624+0000 7f4801c00700 1 -- 172.21.15.44:0/967485392 >> 172.21.15.44:0/967485392 conn(0x7f47fc079c10 msgr2=0x7f47fc102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.624+0000 7f4801c00700 1 -- 172.21.15.44:0/967485392 shutdown_connections 2024-05-27T09:42:11.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:11.624+0000 7f4801c00700 1 -- 172.21.15.44:0/967485392 wait complete. 2024-05-27T09:42:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.795+0000 7fa721600640 1 -- 172.21.15.44:0/846915095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa71c10da80 msgr2=0x7fa71c10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.795+0000 7fa721600640 1 --2- 172.21.15.44:0/846915095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa71c10da80 0x7fa71c10fea0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa708009860 tx=0x7fa70802a340 comp rx=0 tx=0).stop 2024-05-27T09:42:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.796+0000 7fa721600640 1 -- 172.21.15.44:0/846915095 shutdown_connections 2024-05-27T09:42:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.796+0000 7fa721600640 1 --2- 172.21.15.44:0/846915095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa71c1103e0 0x7fa71c112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.796+0000 7fa721600640 1 --2- 172.21.15.44:0/846915095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa71c10da80 0x7fa71c10fea0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.796+0000 7fa721600640 1 --2- 172.21.15.44:0/846915095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa71c102840 0x7fa71c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.796+0000 7fa721600640 1 -- 172.21.15.44:0/846915095 >> 172.21.15.44:0/846915095 conn(0x7fa71c0fe210 msgr2=0x7fa71c100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.798+0000 7fa721600640 1 -- 172.21.15.44:0/846915095 shutdown_connections 2024-05-27T09:42:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.798+0000 7fa721600640 1 -- 172.21.15.44:0/846915095 wait complete. 2024-05-27T09:42:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.799+0000 7fa721600640 1 Processor -- start 2024-05-27T09:42:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.799+0000 7fa721600640 1 -- start start 2024-05-27T09:42:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.799+0000 7fa721600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa71c102840 0x7fa71c1a25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.799+0000 7fa721600640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa71c10da80 0x7fa71c1a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.799+0000 7fa721600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa71c1103e0 0x7fa71c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa721600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c115e80 con 0x7fa71c102840 2024-05-27T09:42:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa721600640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c115d00 con 0x7fa71c10da80 2024-05-27T09:42:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa721600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c116000 con 0x7fa71c1103e0 2024-05-27T09:42:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa71be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa71c102840 0x7fa71c1a25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa71b400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa71c10da80 0x7fa71c1a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa71b400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa71c10da80 0x7fa71c1a2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36992/0 (socket says 172.21.15.44:36992) 2024-05-27T09:42:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa71be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa71c102840 0x7fa71c1a25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50810/0 (socket says 172.21.15.44:50810) 2024-05-27T09:42:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa71b400640 1 -- 172.21.15.44:0/1922496828 learned_addr learned my addr 172.21.15.44:0/1922496828 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa720c00640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa71c1103e0 0x7fa71c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa71b400640 1 -- 172.21.15.44:0/1922496828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa71c1103e0 msgr2=0x7fa71c19c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa71b400640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa71c1103e0 0x7fa71c19c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa71b400640 1 -- 172.21.15.44:0/1922496828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa71c102840 msgr2=0x7fa71c1a25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa71b400640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa71c102840 0x7fa71c1a25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa71b400640 1 -- 172.21.15.44:0/1922496828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71c19ce50 con 0x7fa71c10da80 2024-05-27T09:42:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.800+0000 7fa720c00640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa71c1103e0 0x7fa71c19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.801+0000 7fa71b400640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa71c10da80 0x7fa71c1a2ae0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa708035040 tx=0x7fa708032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.801+0000 7fa718c00640 1 -- 172.21.15.44:0/1922496828 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708037040 con 0x7fa71c10da80 2024-05-27T09:42:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.801+0000 7fa721600640 1 -- 172.21.15.44:0/1922496828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa71c19d080 con 0x7fa71c10da80 2024-05-27T09:42:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.801+0000 7fa718c00640 1 -- 172.21.15.44:0/1922496828 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa708030c10 con 0x7fa71c10da80 2024-05-27T09:42:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.801+0000 7fa718c00640 1 -- 172.21.15.44:0/1922496828 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7080035f0 con 0x7fa71c10da80 2024-05-27T09:42:11.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.801+0000 7fa721600640 1 -- 172.21.15.44:0/1922496828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa71c19d560 con 0x7fa71c10da80 2024-05-27T09:42:11.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.805+0000 7fa729c61740 1 Processor -- start 2024-05-27T09:42:11.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.805+0000 7fa729c61740 1 -- start start 2024-05-27T09:42:11.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.805+0000 7fa729c61740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e1304a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.805+0000 7fa718c00640 1 -- 172.21.15.44:0/1922496828 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa708043020 con 0x7fa71c10da80 2024-05-27T09:42:11.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.805+0000 7fa729c61740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 0x55f4e1304fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.805+0000 7fa729c61740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f414d0 0x55f4e13054e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.805+0000 7fa729c61740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f4e1275b90 con 0x55f4e0f621b0 2024-05-27T09:42:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.805+0000 7fa729c61740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f4e1265730 con 0x55f4e0f392f0 2024-05-27T09:42:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.805+0000 7fa729c61740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f4e1269770 con 0x55f4e0f414d0 2024-05-27T09:42:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.806+0000 7fa720c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e1304a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.806+0000 7fa71b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f414d0 0x55f4e13054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.806+0000 7fa720c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e1304a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50816/0 (socket says 172.21.15.44:50816) 2024-05-27T09:42:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.806+0000 7fa720c00640 1 -- 172.21.15.44:0/298940433 learned_addr learned my addr 172.21.15.44:0/298940433 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.806+0000 7fa71be00640 1 --2- 172.21.15.44:0/298940433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 0x55f4e1304fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.806+0000 7fa71b400640 1 -- 172.21.15.44:0/298940433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 msgr2=0x55f4e1304fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.806+0000 7fa71b400640 1 --2- 172.21.15.44:0/298940433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 0x55f4e1304fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.806+0000 7fa71b400640 1 -- 172.21.15.44:0/298940433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 msgr2=0x55f4e1304a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.806+0000 7fa71b400640 1 --2- 172.21.15.44:0/298940433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e1304a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.806+0000 7fa71b400640 1 -- 172.21.15.44:0/298940433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4e11fe8b0 con 0x55f4e0f414d0 2024-05-27T09:42:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.807+0000 7fa71b400640 1 --2- 172.21.15.44:0/298940433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f414d0 0x55f4e13054e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa7080fef70 tx=0x7fa708107b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f885c72c4b9e3d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.807+0000 7fa711200640 1 -- 172.21.15.44:0/298940433 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708110070 con 0x55f4e0f414d0 2024-05-27T09:42:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.807+0000 7fa711200640 1 -- 172.21.15.44:0/298940433 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa708103070 con 0x55f4e0f414d0 2024-05-27T09:42:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.807+0000 7fa711200640 1 -- 172.21.15.44:0/298940433 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70810c6a0 con 0x55f4e0f414d0 2024-05-27T09:42:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.808+0000 7fa718c00640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7000b29d0 0x7fa7000b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.808+0000 7fa718c00640 1 -- 172.21.15.44:0/1922496828 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7fa708101740 con 0x7fa71c10da80 2024-05-27T09:42:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.808+0000 7fa720c00640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7000b29d0 0x7fa7000b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.808+0000 7fa729c61740 1 -- 172.21.15.44:0/298940433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f414d0 msgr2=0x55f4e13054e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.808+0000 7fa729c61740 1 --2- 172.21.15.44:0/298940433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f414d0 0x55f4e13054e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa7080fef70 tx=0x7fa708107b40 comp rx=0 tx=0).stop 2024-05-27T09:42:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.809+0000 7fa729c61740 1 -- 172.21.15.44:0/298940433 shutdown_connections 2024-05-27T09:42:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.809+0000 7fa729c61740 1 --2- 172.21.15.44:0/298940433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f414d0 0x55f4e13054e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.809+0000 7fa729c61740 1 --2- 172.21.15.44:0/298940433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 0x55f4e1304fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.809+0000 7fa729c61740 1 --2- 172.21.15.44:0/298940433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e1304a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.809+0000 7fa729c61740 1 -- 172.21.15.44:0/298940433 >> 172.21.15.44:0/298940433 conn(0x55f4e0f56300 msgr2=0x55f4e1301210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.809+0000 7fa729c61740 1 -- 172.21.15.44:0/298940433 shutdown_connections 2024-05-27T09:42:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.809+0000 7fa720c00640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7000b29d0 0x7fa7000b4ee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55f4e1301bd0 tx=0x7fa714010040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.809+0000 7fa729c61740 1 -- 172.21.15.44:0/298940433 wait complete. 2024-05-27T09:42:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.809+0000 7fa729c61740 1 -- start start 2024-05-27T09:42:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.810+0000 7fa729c61740 1 Processor -- start 2024-05-27T09:42:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.810+0000 7fa729c61740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 0x55f4e130ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.810+0000 7fa729c61740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f414d0 0x55f4e130f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.810+0000 7fa729c61740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e131abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.810+0000 7fa729c61740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f4e0f4b790 con 0x55f4e0f621b0 2024-05-27T09:42:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.810+0000 7fa729c61740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f4e120dbc0 con 0x55f4e0f392f0 2024-05-27T09:42:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.810+0000 7fa729c61740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f4e1311950 con 0x55f4e0f414d0 2024-05-27T09:42:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.810+0000 7fa729c61740 10 client.0 _ll_register_callbacks cb 0x55f4e11eb400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.811+0000 7fa71be00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 0x55f4e130ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.811+0000 7fa720c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e131abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.811+0000 7fa71be00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 0x55f4e130ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37000/0 (socket says 172.21.15.44:37000) 2024-05-27T09:42:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.811+0000 7fa71be00640 1 -- 172.21.15.44:0/797185786 learned_addr learned my addr 172.21.15.44:0/797185786 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.811+0000 7fa71b400640 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f414d0 0x55f4e130f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.811+0000 7fa71be00640 1 -- 172.21.15.44:0/797185786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f414d0 msgr2=0x55f4e130f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.811+0000 7fa71be00640 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f414d0 0x55f4e130f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.811+0000 7fa71be00640 1 -- 172.21.15.44:0/797185786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 msgr2=0x55f4e131abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.811+0000 7fa71be00640 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e131abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.811+0000 7fa71be00640 1 -- 172.21.15.44:0/797185786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4e0f7a210 con 0x55f4e0f392f0 2024-05-27T09:42:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.811+0000 7fa720c00640 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e131abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.812+0000 7fa71be00640 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 0x55f4e130ebe0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0126e0 tx=0x7fa70c012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7ef7bba778da165 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.813+0000 7fa705600640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.813+0000 7fa705600640 1 -- 172.21.15.44:0/797185786 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70c0120d0 con 0x55f4e0f392f0 2024-05-27T09:42:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.813+0000 7fa705600640 1 -- 172.21.15.44:0/797185786 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7fa70c00f8a0 con 0x55f4e0f392f0 2024-05-27T09:42:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.813+0000 7fa729c61740 1 -- 172.21.15.44:0/797185786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa708103070 con 0x55f4e0f392f0 2024-05-27T09:42:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.813+0000 7fa705600640 1 -- 172.21.15.44:0/797185786 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa70c012270 con 0x55f4e0f392f0 2024-05-27T09:42:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.813+0000 7fa729c61740 10 client.26480 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.813+0000 7fa729c61740 1 -- 172.21.15.44:0/797185786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f4e1046620 con 0x55f4e0f392f0 2024-05-27T09:42:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.814+0000 7fa705600640 1 -- 172.21.15.44:0/797185786 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa708103070 con 0x55f4e0f392f0 2024-05-27T09:42:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.814+0000 7fa705600640 1 -- 172.21.15.44:0/797185786 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fa70c024cc0 con 0x55f4e0f392f0 2024-05-27T09:42:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.814+0000 7fa729c61740 10 client.26480 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.814+0000 7fa729c61740 10 client.26480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.814+0000 7fa729c61740 20 client.26480 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.814+0000 7fa729c61740 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4e0f20d80 0x55f4e1320c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.814+0000 7fa729c61740 4 client.26480 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.814+0000 7fa729c61740 1 -- 172.21.15.44:0/797185786 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4e1277ae0 con 0x55f4e0f20d80 2024-05-27T09:42:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.815+0000 7fa71b400640 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4e0f20d80 0x55f4e1320c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.815+0000 7fa71b400640 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4e0f20d80 0x55f4e1320c50 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=115e3e653f2c4d4f server_cookie=3253a83ddb4712d8 in_seq=0 out_seq=0 2024-05-27T09:42:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.815+0000 7fa705600640 10 client.26480 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.816+0000 7fa705600640 1 -- 172.21.15.44:0/797185786 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f4e1277ae0 con 0x55f4e0f20d80 2024-05-27T09:42:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.816+0000 7fa705600640 10 client.26480 handle_command_reply: tid=0 2024-05-27T09:42:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.816+0000 7fa729c61740 1 client.26480 shutdown 2024-05-27T09:42:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa729c61740 1 -- 172.21.15.44:0/797185786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 msgr2=0x55f4e130ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa729c61740 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 0x55f4e130ebe0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0126e0 tx=0x7fa70c012ba0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa729c61740 1 -- 172.21.15.44:0/797185786 shutdown_connections 2024-05-27T09:42:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa729c61740 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4e0f20d80 0x55f4e1320c50 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa729c61740 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e131abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa729c61740 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f414d0 0x55f4e130f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa729c61740 1 --2- 172.21.15.44:0/797185786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f392f0 0x55f4e130ebe0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa729c61740 1 -- 172.21.15.44:0/797185786 >> 172.21.15.44:0/797185786 conn(0x55f4e0f56300 msgr2=0x55f4e1306780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa705600640 0 client.26480 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa729c61740 1 -- 172.21.15.44:0/797185786 shutdown_connections 2024-05-27T09:42:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa729c61740 1 -- 172.21.15.44:0/797185786 wait complete. 2024-05-27T09:42:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.817+0000 7fa729c61740 20 client.26480 trim_cache size 0 max 16384 2024-05-27T09:42:11.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:11 smithi044 ceph-mon[31618]: pgmap v212: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 57 KiB/s wr, 27 op/s 2024-05-27T09:42:11.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:11 smithi044 ceph-mon[31618]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T09:42:11.837 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.836+0000 7fa729c61740 1 Processor -- start 2024-05-27T09:42:11.837 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.837+0000 7fa729c61740 1 -- start start 2024-05-27T09:42:11.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.837+0000 7fa729c61740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 0x55f4e1311310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.837+0000 7fa729c61740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 0x55f4e131a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.837+0000 7fa729c61740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e131aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.837+0000 7fa729c61740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c1b4600 con 0x55f4e0f621b0 2024-05-27T09:42:11.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.837+0000 7fa729c61740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c1b1ea0 con 0x55f4e0f414d0 2024-05-27T09:42:11.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.837+0000 7fa729c61740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c1afd10 con 0x55f4e0f392f0 2024-05-27T09:42:11.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71b400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 0x55f4e131a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 0x55f4e1311310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa720c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e131aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 0x55f4e1311310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59044/0 (socket says 172.21.15.44:59044) 2024-05-27T09:42:11.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71b400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 0x55f4e131a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37014/0 (socket says 172.21.15.44:37014) 2024-05-27T09:42:11.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71be00640 1 -- 172.21.15.44:0/265810357 learned_addr learned my addr 172.21.15.44:0/265810357 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:11.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71be00640 1 -- 172.21.15.44:0/265810357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 msgr2=0x55f4e131a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.840 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71be00640 1 --2- 172.21.15.44:0/265810357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 0x55f4e131a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71be00640 1 -- 172.21.15.44:0/265810357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 msgr2=0x55f4e131aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71be00640 1 --2- 172.21.15.44:0/265810357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e131aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71be00640 1 -- 172.21.15.44:0/265810357 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4e13022d0 con 0x55f4e0f392f0 2024-05-27T09:42:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71b400640 1 --2- 172.21.15.44:0/265810357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 0x55f4e131a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.838+0000 7fa71be00640 1 --2- 172.21.15.44:0/265810357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 0x55f4e1311310 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa70c010500 tx=0x7fa70c005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa3eb3f84c3aa981 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.839+0000 7fa707e00640 1 -- 172.21.15.44:0/265810357 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70c010770 con 0x55f4e0f392f0 2024-05-27T09:42:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.839+0000 7fa707e00640 1 -- 172.21.15.44:0/265810357 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa70c010910 con 0x55f4e0f392f0 2024-05-27T09:42:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.839+0000 7fa707e00640 1 -- 172.21.15.44:0/265810357 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70c005360 con 0x55f4e0f392f0 2024-05-27T09:42:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.840+0000 7fa729c61740 1 -- 172.21.15.44:0/265810357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 msgr2=0x55f4e1311310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.840+0000 7fa729c61740 1 --2- 172.21.15.44:0/265810357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 0x55f4e1311310 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa70c010500 tx=0x7fa70c005970 comp rx=0 tx=0).stop 2024-05-27T09:42:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.840+0000 7fa729c61740 1 -- 172.21.15.44:0/265810357 shutdown_connections 2024-05-27T09:42:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.840+0000 7fa729c61740 1 --2- 172.21.15.44:0/265810357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e131aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.840+0000 7fa729c61740 1 --2- 172.21.15.44:0/265810357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 0x55f4e131a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.840+0000 7fa729c61740 1 --2- 172.21.15.44:0/265810357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 0x55f4e1311310 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.840+0000 7fa729c61740 1 -- 172.21.15.44:0/265810357 >> 172.21.15.44:0/265810357 conn(0x55f4e0f56300 msgr2=0x55f4e1309dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.840+0000 7fa729c61740 1 -- 172.21.15.44:0/265810357 shutdown_connections 2024-05-27T09:42:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.840+0000 7fa729c61740 1 -- 172.21.15.44:0/265810357 wait complete. 2024-05-27T09:42:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.840+0000 7fa729c61740 1 -- start start 2024-05-27T09:42:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.841+0000 7fa729c61740 1 Processor -- start 2024-05-27T09:42:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa729c61740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 0x55f4e1312180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa729c61740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 0x55f4e13126c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa729c61740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e13813a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa729c61740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c1b0360 con 0x55f4e0f621b0 2024-05-27T09:42:11.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa729c61740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c1b17d0 con 0x55f4e0f414d0 2024-05-27T09:42:11.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa729c61740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c1b0a30 con 0x55f4e0f392f0 2024-05-27T09:42:11.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa729c61740 10 client.0 _ll_register_callbacks cb 0x7fa70c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:11.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa720c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e13813a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa71be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 0x55f4e1312180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa720c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e13813a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50826/0 (socket says 172.21.15.44:50826) 2024-05-27T09:42:11.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa720c00640 1 -- 172.21.15.44:0/1858190322 learned_addr learned my addr 172.21.15.44:0/1858190322 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:11.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.842+0000 7fa71b400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 0x55f4e13126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa71b400640 1 -- 172.21.15.44:0/1858190322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 msgr2=0x55f4e1312180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa71b400640 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 0x55f4e1312180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa71b400640 1 -- 172.21.15.44:0/1858190322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 msgr2=0x55f4e13813a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa71b400640 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e13813a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa71b400640 1 -- 172.21.15.44:0/1858190322 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4e1381910 con 0x55f4e0f414d0 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa71be00640 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 0x55f4e1312180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa720c00640 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e13813a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa71b400640 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 0x55f4e13126c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa7080c4b80 tx=0x7fa708030fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae6ec15518f521af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa705600640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa705600640 1 -- 172.21.15.44:0/1858190322 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708110040 con 0x55f4e0f414d0 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa705600640 1 -- 172.21.15.44:0/1858190322 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7fa7080ff050 con 0x55f4e0f414d0 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa729c61740 1 -- 172.21.15.44:0/1858190322 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa70c010910 con 0x55f4e0f414d0 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa705600640 1 -- 172.21.15.44:0/1858190322 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7080c49c0 con 0x55f4e0f414d0 2024-05-27T09:42:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa729c61740 10 client.26486 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:11.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.843+0000 7fa729c61740 1 -- 172.21.15.44:0/1858190322 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f4e1388c40 con 0x55f4e0f414d0 2024-05-27T09:42:11.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.844+0000 7fa705600640 1 -- 172.21.15.44:0/1858190322 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa70c010910 con 0x55f4e0f414d0 2024-05-27T09:42:11.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.844+0000 7fa705600640 1 -- 172.21.15.44:0/1858190322 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fa7080c8d60 con 0x55f4e0f414d0 2024-05-27T09:42:11.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.844+0000 7fa729c61740 10 client.26486 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:11.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.844+0000 7fa729c61740 10 client.26486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:11.848 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.844+0000 7fa729c61740 20 client.26486 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:11.848 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.844+0000 7fa729c61740 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4e0ef61c0 0x55f4e138afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:11.848 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.844+0000 7fa729c61740 4 client.26486 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:42:11.848 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.844+0000 7fa729c61740 1 -- 172.21.15.44:0/1858190322 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f4e12712c0 con 0x55f4e0ef61c0 2024-05-27T09:42:11.848 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.845+0000 7fa71be00640 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4e0ef61c0 0x55f4e138afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:11.848 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.845+0000 7fa71be00640 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4e0ef61c0 0x55f4e138afe0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13700a2bf397ca4f server_cookie=e903e71fda69aec1 in_seq=0 out_seq=0 2024-05-27T09:42:11.848 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.845+0000 7fa705600640 10 client.26486 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa705600640 1 -- 172.21.15.44:0/1858190322 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f4e12712c0 con 0x55f4e0ef61c0 2024-05-27T09:42:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa705600640 10 client.26486 handle_command_reply: tid=0 2024-05-27T09:42:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa729c61740 1 client.26486 shutdown 2024-05-27T09:42:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa729c61740 1 -- 172.21.15.44:0/1858190322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 msgr2=0x55f4e13126c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa729c61740 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 0x55f4e13126c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa7080c4b80 tx=0x7fa708030fc0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa729c61740 1 -- 172.21.15.44:0/1858190322 shutdown_connections 2024-05-27T09:42:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa729c61740 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f4e0ef61c0 0x55f4e138afe0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa729c61740 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f4e0f621b0 0x55f4e13813a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa729c61740 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f4e0f414d0 0x55f4e13126c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa729c61740 1 --2- 172.21.15.44:0/1858190322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4e0f392f0 0x55f4e1312180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa705600640 0 client.26486 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.855+0000 7fa729c61740 1 -- 172.21.15.44:0/1858190322 >> 172.21.15.44:0/1858190322 conn(0x55f4e0f56300 msgr2=0x55f4e1309330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.856+0000 7fa729c61740 1 -- 172.21.15.44:0/1858190322 shutdown_connections 2024-05-27T09:42:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.856+0000 7fa729c61740 1 -- 172.21.15.44:0/1858190322 wait complete. 2024-05-27T09:42:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.856+0000 7fa729c61740 20 client.26486 trim_cache size 0 max 16384 2024-05-27T09:42:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:42:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:42:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 -- 172.21.15.44:0/1922496828 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7000b29d0 msgr2=0x7fa7000b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7000b29d0 0x7fa7000b4ee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55f4e1301bd0 tx=0x7fa714010040 comp rx=0 tx=0).stop 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 -- 172.21.15.44:0/1922496828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa71c10da80 msgr2=0x7fa71c1a2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa71c10da80 0x7fa71c1a2ae0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa708035040 tx=0x7fa708032d50 comp rx=0 tx=0).stop 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 -- 172.21.15.44:0/1922496828 shutdown_connections 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa71c1103e0 0x7fa71c19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa71c10da80 0x7fa71c1a2ae0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa7000b29d0 0x7fa7000b4ee0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 --2- 172.21.15.44:0/1922496828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa71c102840 0x7fa71c1a25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 -- 172.21.15.44:0/1922496828 >> 172.21.15.44:0/1922496828 conn(0x7fa71c0fe210 msgr2=0x7fa71c100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 -- 172.21.15.44:0/1922496828 shutdown_connections 2024-05-27T09:42:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:11.857+0000 7fa721600640 1 -- 172.21.15.44:0/1922496828 wait complete. 2024-05-27T09:42:12.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:11 smithi050 ceph-mon[36371]: pgmap v212: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 57 KiB/s wr, 27 op/s 2024-05-27T09:42:12.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:11 smithi050 ceph-mon[36371]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T09:42:12.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:11 smithi189 ceph-mon[36671]: pgmap v212: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 57 KiB/s wr, 27 op/s 2024-05-27T09:42:12.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:11 smithi189 ceph-mon[36671]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T09:42:13.775 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:42:13.776 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:42:14.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:13 smithi050 ceph-mon[36371]: pgmap v214: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-27T09:42:14.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:13 smithi044 ceph-mon[31618]: pgmap v214: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-27T09:42:14.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:13 smithi189 ceph-mon[36671]: pgmap v214: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-27T09:42:14.434 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.464+0000 7fc77fa00700 1 -- 172.21.15.44:0/548979494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7781045e0 msgr2=0x7fc7781049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.464+0000 7fc77fa00700 1 --2- 172.21.15.44:0/548979494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7781045e0 0x7fc7781049f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc768009990 tx=0x7fc768009ca0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.464+0000 7fc77fa00700 1 -- 172.21.15.44:0/548979494 shutdown_connections 2024-05-27T09:42:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.464+0000 7fc77fa00700 1 --2- 172.21.15.44:0/548979494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc778109860 0x7fc7781111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.464+0000 7fc77fa00700 1 --2- 172.21.15.44:0/548979494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7781045e0 0x7fc7781049f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.464+0000 7fc77fa00700 1 --2- 172.21.15.44:0/548979494 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc778108f10 0x7fc778109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.464+0000 7fc77fa00700 1 -- 172.21.15.44:0/548979494 >> 172.21.15.44:0/548979494 conn(0x7fc778100030 msgr2=0x7fc778102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.464+0000 7fc77fa00700 1 -- 172.21.15.44:0/548979494 shutdown_connections 2024-05-27T09:42:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.465+0000 7fc77fa00700 1 -- 172.21.15.44:0/548979494 wait complete. 2024-05-27T09:42:14.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.466+0000 7fc77fa00700 1 Processor -- start 2024-05-27T09:42:14.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.466+0000 7fc77fa00700 1 -- start start 2024-05-27T09:42:14.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.467+0000 7fc77fa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7781045e0 0x7fc77810a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.467+0000 7fc77fa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc778108f10 0x7fc77810a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.467+0000 7fc77fa00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc778109860 0x7fc77810abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.467+0000 7fc77fa00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781144e0 con 0x7fc778108f10 2024-05-27T09:42:14.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.467+0000 7fc77fa00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc778114360 con 0x7fc778109860 2024-05-27T09:42:14.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.467+0000 7fc77fa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc778114660 con 0x7fc7781045e0 2024-05-27T09:42:14.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.467+0000 7fc77d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7781045e0 0x7fc77810a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.467+0000 7fc77d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7781045e0 0x7fc77810a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50320/0 (socket says 172.21.15.44:50320) 2024-05-27T09:42:14.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.467+0000 7fc77d000700 1 -- 172.21.15.44:0/3525928527 learned_addr learned my addr 172.21.15.44:0/3525928527 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:14.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.467+0000 7fc77da00700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc778109860 0x7fc77810abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.468+0000 7fc76fe00700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc778108f10 0x7fc77810a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.468+0000 7fc77d000700 1 -- 172.21.15.44:0/3525928527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc778109860 msgr2=0x7fc77810abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:14.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.468+0000 7fc77d000700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc778109860 0x7fc77810abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.468+0000 7fc77d000700 1 -- 172.21.15.44:0/3525928527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc778108f10 msgr2=0x7fc77810a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:14.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.468+0000 7fc77d000700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc778108f10 0x7fc77810a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.468+0000 7fc77d000700 1 -- 172.21.15.44:0/3525928527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7781ad630 con 0x7fc7781045e0 2024-05-27T09:42:14.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.468+0000 7fc76fe00700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc778108f10 0x7fc77810a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:42:14.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.468+0000 7fc77d000700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7781045e0 0x7fc77810a140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc768000c00 tx=0x7fc7680057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:14.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.469+0000 7fc776000700 1 -- 172.21.15.44:0/3525928527 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768027070 con 0x7fc7781045e0 2024-05-27T09:42:14.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.469+0000 7fc77fa00700 1 -- 172.21.15.44:0/3525928527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7781ad800 con 0x7fc7781045e0 2024-05-27T09:42:14.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.469+0000 7fc776000700 1 -- 172.21.15.44:0/3525928527 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc76800b610 con 0x7fc7781045e0 2024-05-27T09:42:14.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.469+0000 7fc77fa00700 1 -- 172.21.15.44:0/3525928527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7781adcb0 con 0x7fc7781045e0 2024-05-27T09:42:14.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.470+0000 7fc776000700 1 -- 172.21.15.44:0/3525928527 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7680173f0 con 0x7fc7781045e0 2024-05-27T09:42:14.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.474+0000 7fc78e4eab80 1 Processor -- start 2024-05-27T09:42:14.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.474+0000 7fc78e4eab80 1 -- start start 2024-05-27T09:42:14.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091005f20 0x55b09118ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b091184ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091185420 0x55b091185830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b0910b2f20 con 0x55b091005f20 2024-05-27T09:42:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b0910a2ac0 con 0x55b091185420 2024-05-27T09:42:14.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b0910a6b00 con 0x55b09118b2d0 2024-05-27T09:42:14.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc776000700 1 -- 172.21.15.44:0/3525928527 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc76802a070 con 0x7fc7781045e0 2024-05-27T09:42:14.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc77da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091005f20 0x55b09118ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc76fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091185420 0x55b091185830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc77da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091005f20 0x55b09118ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43930/0 (socket says 172.21.15.44:43930) 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc76fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091185420 0x55b091185830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38582/0 (socket says 172.21.15.44:38582) 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.475+0000 7fc77da00700 1 -- 172.21.15.44:0/3952702937 learned_addr learned my addr 172.21.15.44:0/3952702937 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.476+0000 7fc77d000700 1 --2- 172.21.15.44:0/3952702937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b091184ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.476+0000 7fc76fe00700 1 -- 172.21.15.44:0/3952702937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 msgr2=0x55b091184ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.476+0000 7fc76fe00700 1 --2- 172.21.15.44:0/3952702937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b091184ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.476+0000 7fc76fe00700 1 -- 172.21.15.44:0/3952702937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091005f20 msgr2=0x55b09118ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.476+0000 7fc76fe00700 1 --2- 172.21.15.44:0/3952702937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091005f20 0x55b09118ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.476+0000 7fc76fe00700 1 -- 172.21.15.44:0/3952702937 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b09106b500 con 0x55b091185420 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.476+0000 7fc77d000700 1 --2- 172.21.15.44:0/3952702937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b091184ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.476+0000 7fc76fe00700 1 --2- 172.21.15.44:0/3952702937 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091185420 0x55b091185830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc778077ab0 tx=0x7fc760009eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2da679bf6298bbe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.477+0000 7fc76e000700 1 -- 172.21.15.44:0/3952702937 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc760010af0 con 0x55b091185420 2024-05-27T09:42:14.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.477+0000 7fc76e000700 1 -- 172.21.15.44:0/3952702937 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc760004ce0 con 0x55b091185420 2024-05-27T09:42:14.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.477+0000 7fc76e000700 1 -- 172.21.15.44:0/3952702937 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc760005610 con 0x55b091185420 2024-05-27T09:42:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.477+0000 7fc776000700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc75c0b29f0 0x7fc75c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.478+0000 7fc776000700 1 -- 172.21.15.44:0/3525928527 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7fc7680e32b0 con 0x7fc7781045e0 2024-05-27T09:42:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.478+0000 7fc77da00700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc75c0b29f0 0x7fc75c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.478+0000 7fc78e4eab80 1 -- 172.21.15.44:0/3952702937 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091185420 msgr2=0x55b091185830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.478+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/3952702937 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091185420 0x55b091185830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc778077ab0 tx=0x7fc760009eb0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.478+0000 7fc78e4eab80 1 -- 172.21.15.44:0/3952702937 shutdown_connections 2024-05-27T09:42:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.478+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/3952702937 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091185420 0x55b091185830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.478+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/3952702937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091005f20 0x55b09118ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.478+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/3952702937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b091184ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.478+0000 7fc78e4eab80 1 -- 172.21.15.44:0/3952702937 >> 172.21.15.44:0/3952702937 conn(0x55b0910660a0 msgr2=0x55b0911800e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.479+0000 7fc78e4eab80 1 -- 172.21.15.44:0/3952702937 shutdown_connections 2024-05-27T09:42:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.479+0000 7fc78e4eab80 1 -- 172.21.15.44:0/3952702937 wait complete. 2024-05-27T09:42:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.479+0000 7fc77da00700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc75c0b29f0 0x7fc75c0b4f00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55b0911860d0 tx=0x7fc77000b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.480+0000 7fc78e4eab80 1 -- start start 2024-05-27T09:42:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.480+0000 7fc78e4eab80 1 Processor -- start 2024-05-27T09:42:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.481+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b091005f20 0x55b09118e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.481+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b09118ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.481+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b09118b2d0 0x55b09118f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.481+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b090fa9080 con 0x55b091185420 2024-05-27T09:42:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.481+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b091043840 con 0x55b09118b2d0 2024-05-27T09:42:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.481+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b091191da0 con 0x55b091005f20 2024-05-27T09:42:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.481+0000 7fc78e4eab80 10 client.0 _ll_register_callbacks cb 0x55b09104cd50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.482+0000 7fc76fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b09118ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.482+0000 7fc77da00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b09118b2d0 0x55b09118f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.482+0000 7fc76fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b09118ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43936/0 (socket says 172.21.15.44:43936) 2024-05-27T09:42:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.482+0000 7fc76fe00700 1 -- 172.21.15.44:0/2568428996 learned_addr learned my addr 172.21.15.44:0/2568428996 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.482+0000 7fc77d000700 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b091005f20 0x55b09118e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.482+0000 7fc77da00700 1 -- 172.21.15.44:0/2568428996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b091005f20 msgr2=0x55b09118e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.482+0000 7fc77da00700 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b091005f20 0x55b09118e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.482+0000 7fc77da00700 1 -- 172.21.15.44:0/2568428996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 msgr2=0x55b09118ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.482+0000 7fc77da00700 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b09118ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.482+0000 7fc77da00700 1 -- 172.21.15.44:0/2568428996 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b090ff29b0 con 0x55b09118b2d0 2024-05-27T09:42:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.482+0000 7fc76fe00700 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b09118ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.483+0000 7fc77da00700 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b09118b2d0 0x55b09118f440 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc770016bc0 tx=0x7fc770004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35490b7ec80009ab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.483+0000 7fc766a00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.483+0000 7fc766a00700 1 -- 172.21.15.44:0/2568428996 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc770027030 con 0x55b09118b2d0 2024-05-27T09:42:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.483+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2568428996 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc760004ce0 con 0x55b09118b2d0 2024-05-27T09:42:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.483+0000 7fc766a00700 1 -- 172.21.15.44:0/2568428996 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7fc770004bb0 con 0x55b09118b2d0 2024-05-27T09:42:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.483+0000 7fc766a00700 1 -- 172.21.15.44:0/2568428996 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc770031050 con 0x55b09118b2d0 2024-05-27T09:42:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.484+0000 7fc78e4eab80 10 client.26498 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.484+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2568428996 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b091073140 con 0x55b09118b2d0 2024-05-27T09:42:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.484+0000 7fc766a00700 1 -- 172.21.15.44:0/2568428996 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc760004ce0 con 0x55b09118b2d0 2024-05-27T09:42:14.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.484+0000 7fc766a00700 1 -- 172.21.15.44:0/2568428996 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fc770023a30 con 0x55b09118b2d0 2024-05-27T09:42:14.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.484+0000 7fc78e4eab80 10 client.26498 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:14.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.484+0000 7fc78e4eab80 10 client.26498 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:42:14.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.485+0000 7fc78e4eab80 20 client.26498 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:14.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.485+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55b09119ec70 0x55b0911a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.485+0000 7fc78e4eab80 4 client.26498 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:14.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.485+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2568428996 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0910b4e70 con 0x55b09119ec70 2024-05-27T09:42:14.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.485+0000 7fc77d000700 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55b09119ec70 0x55b0911a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.486+0000 7fc77d000700 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55b09119ec70 0x55b0911a10b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3974f21473af35a9 server_cookie=45b14ed5609916a7 in_seq=0 out_seq=0 2024-05-27T09:42:14.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.486+0000 7fc766a00700 10 client.26498 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:42:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc766a00700 1 -- 172.21.15.44:0/2568428996 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b0910b4e70 con 0x55b09119ec70 2024-05-27T09:42:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc766a00700 10 client.26498 handle_command_reply: tid=0 2024-05-27T09:42:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc78e4eab80 1 client.26498 shutdown 2024-05-27T09:42:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2568428996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b09118b2d0 msgr2=0x55b09118f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:14.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b09118b2d0 0x55b09118f440 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc770016bc0 tx=0x7fc770004b40 comp rx=0 tx=0).stop 2024-05-27T09:42:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2568428996 shutdown_connections 2024-05-27T09:42:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b09118b2d0 0x55b09118f440 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b091005f20 0x55b09118e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55b09119ec70 0x55b0911a10b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2568428996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b09118ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2568428996 >> 172.21.15.44:0/2568428996 conn(0x55b0910660a0 msgr2=0x55b091183cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc766a00700 0 client.26498 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:42:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.487+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2568428996 shutdown_connections 2024-05-27T09:42:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.488+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2568428996 wait complete. 2024-05-27T09:42:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.488+0000 7fc78e4eab80 20 client.26498 trim_cache size 0 max 16384 2024-05-27T09:42:14.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.509+0000 7fc78e4eab80 1 Processor -- start 2024-05-27T09:42:14.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.509+0000 7fc78e4eab80 1 -- start start 2024-05-27T09:42:14.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.510+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 0x55b09118a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.510+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b091189110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.510+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b09119ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.510+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781b7db0 con 0x55b091185420 2024-05-27T09:42:14.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.510+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781b5610 con 0x55b091005f20 2024-05-27T09:42:14.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.510+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781b3480 con 0x55b09118b2d0 2024-05-27T09:42:14.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.510+0000 7fc77d000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 0x55b09118a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.510+0000 7fc76fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b091189110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.510+0000 7fc77d000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 0x55b09118a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38594/0 (socket says 172.21.15.44:38594) 2024-05-27T09:42:14.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.510+0000 7fc77d000700 1 -- 172.21.15.44:0/1553845890 learned_addr learned my addr 172.21.15.44:0/1553845890 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:14.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.510+0000 7fc77da00700 1 --2- 172.21.15.44:0/1553845890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b09119ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.511+0000 7fc77d000700 1 -- 172.21.15.44:0/1553845890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 msgr2=0x55b09119ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:14.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.511+0000 7fc77d000700 1 --2- 172.21.15.44:0/1553845890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b09119ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.511+0000 7fc77d000700 1 -- 172.21.15.44:0/1553845890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 msgr2=0x55b091189110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:14.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.511+0000 7fc77d000700 1 --2- 172.21.15.44:0/1553845890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b091189110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.511+0000 7fc77d000700 1 -- 172.21.15.44:0/1553845890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0911795c0 con 0x55b091005f20 2024-05-27T09:42:14.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.511+0000 7fc76fe00700 1 --2- 172.21.15.44:0/1553845890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b091189110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:14.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.511+0000 7fc77d000700 1 --2- 172.21.15.44:0/1553845890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 0x55b09118a380 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc7680a6610 tx=0x7fc768009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc99186a7bc2f0a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.511+0000 7fc76d600700 1 -- 172.21.15.44:0/1553845890 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7680aa590 con 0x55b091005f20 2024-05-27T09:42:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.511+0000 7fc76d600700 1 -- 172.21.15.44:0/1553845890 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7680ae940 con 0x55b091005f20 2024-05-27T09:42:14.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.511+0000 7fc76d600700 1 -- 172.21.15.44:0/1553845890 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7680aa7f0 con 0x55b091005f20 2024-05-27T09:42:14.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.512+0000 7fc78e4eab80 1 -- 172.21.15.44:0/1553845890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 msgr2=0x55b09118a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:14.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.512+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/1553845890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 0x55b09118a380 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc7680a6610 tx=0x7fc768009310 comp rx=0 tx=0).stop 2024-05-27T09:42:14.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.512+0000 7fc78e4eab80 1 -- 172.21.15.44:0/1553845890 shutdown_connections 2024-05-27T09:42:14.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.512+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/1553845890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b09119ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.512+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/1553845890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 0x55b09118a380 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.512+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/1553845890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b091189110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.512+0000 7fc78e4eab80 1 -- 172.21.15.44:0/1553845890 >> 172.21.15.44:0/1553845890 conn(0x55b0910660a0 msgr2=0x55b091183cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:14.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.513+0000 7fc78e4eab80 1 -- 172.21.15.44:0/1553845890 shutdown_connections 2024-05-27T09:42:14.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.513+0000 7fc78e4eab80 1 -- 172.21.15.44:0/1553845890 wait complete. 2024-05-27T09:42:14.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.513+0000 7fc78e4eab80 1 -- start start 2024-05-27T09:42:14.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.513+0000 7fc78e4eab80 1 Processor -- start 2024-05-27T09:42:14.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.514+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 0x55b091197230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.514+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b091197770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.514+0000 7fc78e4eab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b091194000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.514+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781b3ad0 con 0x55b091185420 2024-05-27T09:42:14.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.514+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781b4f40 con 0x55b091005f20 2024-05-27T09:42:14.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.514+0000 7fc78e4eab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781b41a0 con 0x55b09118b2d0 2024-05-27T09:42:14.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.514+0000 7fc78e4eab80 10 client.0 _ll_register_callbacks cb 0x55b09104cd50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:14.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.514+0000 7fc76fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b091197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.514+0000 7fc77da00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b091194000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.515+0000 7fc77da00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b091194000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50354/0 (socket says 172.21.15.44:50354) 2024-05-27T09:42:14.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.515+0000 7fc77d000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 0x55b091197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.515+0000 7fc77da00700 1 -- 172.21.15.44:0/2725669856 learned_addr learned my addr 172.21.15.44:0/2725669856 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:14.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.515+0000 7fc77da00700 1 -- 172.21.15.44:0/2725669856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 msgr2=0x55b091197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:14.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.515+0000 7fc77da00700 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 0x55b091197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.515+0000 7fc77da00700 1 -- 172.21.15.44:0/2725669856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 msgr2=0x55b091197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:14.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.515+0000 7fc77da00700 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b091197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:14.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.515+0000 7fc77da00700 1 -- 172.21.15.44:0/2725669856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b09118c9e0 con 0x55b09118b2d0 2024-05-27T09:42:14.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.515+0000 7fc77d000700 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 0x55b091197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:14.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.515+0000 7fc77da00700 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b091194000 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc770009a90 tx=0x7fc7700156f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9ecd28dce3600e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:14.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.515+0000 7fc766a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:42:14.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.516+0000 7fc766a00700 1 -- 172.21.15.44:0/2725669856 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc770027040 con 0x55b09118b2d0 2024-05-27T09:42:14.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.516+0000 7fc766a00700 1 -- 172.21.15.44:0/2725669856 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7fc7700159b0 con 0x55b09118b2d0 2024-05-27T09:42:14.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.516+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2725669856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7680ae940 con 0x55b09118b2d0 2024-05-27T09:42:14.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.516+0000 7fc766a00700 1 -- 172.21.15.44:0/2725669856 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc770005e90 con 0x55b09118b2d0 2024-05-27T09:42:14.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.516+0000 7fc78e4eab80 10 client.26500 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:14.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.516+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2725669856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b091209e70 con 0x55b09118b2d0 2024-05-27T09:42:14.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.517+0000 7fc766a00700 1 -- 172.21.15.44:0/2725669856 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7680ae940 con 0x55b09118b2d0 2024-05-27T09:42:14.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.517+0000 7fc766a00700 1 -- 172.21.15.44:0/2725669856 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fc7700050a0 con 0x55b09118b2d0 2024-05-27T09:42:14.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.517+0000 7fc78e4eab80 10 client.26500 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:14.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.517+0000 7fc78e4eab80 10 client.26500 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:42:14.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.517+0000 7fc78e4eab80 20 client.26500 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:14.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.517+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55b09120a570 0x55b09120c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:14.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.517+0000 7fc78e4eab80 4 client.26500 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-77ef2bd1-1-mds.i.json"}] 2024-05-27T09:42:14.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.517+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2725669856 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-77ef2bd1-1-mds.i.json"}) -- 0x55b0910ae650 con 0x55b09120a570 2024-05-27T09:42:14.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.518+0000 7fc77d000700 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55b09120a570 0x55b09120c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:14.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.518+0000 7fc77d000700 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55b09120a570 0x55b09120c990 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67d37c533bc21bbc server_cookie=e49e44f4930cc8d1 in_seq=0 out_seq=0 2024-05-27T09:42:14.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:14.518+0000 7fc766a00700 10 client.26500 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:42:15.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.013+0000 7fc766a00700 1 -- 172.21.15.44:0/2725669856 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x55b0910ae650 con 0x55b09120a570 2024-05-27T09:42:15.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.013+0000 7fc766a00700 10 client.26500 handle_command_reply: tid=0 2024-05-27T09:42:15.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.013+0000 7fc78e4eab80 1 client.26500 shutdown 2024-05-27T09:42:15.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.014+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2725669856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 msgr2=0x55b091194000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:15.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.014+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b091194000 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc770009a90 tx=0x7fc7700156f0 comp rx=0 tx=0).stop 2024-05-27T09:42:15.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.014+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2725669856 shutdown_connections 2024-05-27T09:42:15.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.014+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b09118b2d0 0x55b091194000 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:15.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.014+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b091005f20 0x55b091197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:15.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.014+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55b09120a570 0x55b09120c990 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:15.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.014+0000 7fc78e4eab80 1 --2- 172.21.15.44:0/2725669856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b091185420 0x55b091197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:15.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.014+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2725669856 >> 172.21.15.44:0/2725669856 conn(0x55b0910660a0 msgr2=0x55b091188e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:15.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.014+0000 7fc766a00700 0 client.26500 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:42:15.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.015+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2725669856 shutdown_connections 2024-05-27T09:42:15.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.015+0000 7fc78e4eab80 1 -- 172.21.15.44:0/2725669856 wait complete. 2024-05-27T09:42:15.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.015+0000 7fc78e4eab80 20 client.26500 trim_cache size 0 max 16384 2024-05-27T09:42:15.017 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:42:15.017 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/var/run/ceph/ops-77ef2bd1-1-mds.i.json" 2024-05-27T09:42:15.017 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:42:15.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 -- 172.21.15.44:0/3525928527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc75c0b29f0 msgr2=0x7fc75c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:15.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc75c0b29f0 0x7fc75c0b4f00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55b0911860d0 tx=0x7fc77000b040 comp rx=0 tx=0).stop 2024-05-27T09:42:15.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 -- 172.21.15.44:0/3525928527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7781045e0 msgr2=0x7fc77810a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:15.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7781045e0 0x7fc77810a140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc768000c00 tx=0x7fc7680057c0 comp rx=0 tx=0).stop 2024-05-27T09:42:15.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 -- 172.21.15.44:0/3525928527 shutdown_connections 2024-05-27T09:42:15.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc75c0b29f0 0x7fc75c0b4f00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:15.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc778109860 0x7fc77810abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:15.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7781045e0 0x7fc77810a140 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:15.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 --2- 172.21.15.44:0/3525928527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc778108f10 0x7fc77810a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:15.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 -- 172.21.15.44:0/3525928527 >> 172.21.15.44:0/3525928527 conn(0x7fc778100030 msgr2=0x7fc778100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:15.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 -- 172.21.15.44:0/3525928527 shutdown_connections 2024-05-27T09:42:15.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:15.017+0000 7fc77fa00700 1 -- 172.21.15.44:0/3525928527 wait complete. 2024-05-27T09:42:15.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:15 smithi044 ceph-mon[31618]: pgmap v215: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-27T09:42:16.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:15 smithi050 ceph-mon[36371]: pgmap v215: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-27T09:42:16.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:15 smithi189 ceph-mon[36671]: pgmap v215: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-27T09:42:16.420 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 0 (b) to a remote file /var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.b.json 2024-05-27T09:42:16.420 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 ops --flags=locks --path=/var/run/ceph/ops-77ef2bd1-1-mds.b.json 2024-05-27T09:42:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.068+0000 7f3a25e00700 1 -- 172.21.15.44:0/4200516383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a201045e0 msgr2=0x7f3a201049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.068+0000 7f3a25e00700 1 --2- 172.21.15.44:0/4200516383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a201045e0 0x7f3a201049f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3a100099c0 tx=0x7f3a10009cd0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.068+0000 7f3a25e00700 1 -- 172.21.15.44:0/4200516383 shutdown_connections 2024-05-27T09:42:17.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.068+0000 7f3a25e00700 1 --2- 172.21.15.44:0/4200516383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a20109860 0x7f3a201111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.068+0000 7f3a25e00700 1 --2- 172.21.15.44:0/4200516383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a201045e0 0x7f3a201049f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.068+0000 7f3a25e00700 1 --2- 172.21.15.44:0/4200516383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a20108f10 0x7f3a20109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.068+0000 7f3a25e00700 1 -- 172.21.15.44:0/4200516383 >> 172.21.15.44:0/4200516383 conn(0x7f3a20100030 msgr2=0x7f3a20102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.068+0000 7f3a25e00700 1 -- 172.21.15.44:0/4200516383 shutdown_connections 2024-05-27T09:42:17.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.068+0000 7f3a25e00700 1 -- 172.21.15.44:0/4200516383 wait complete. 2024-05-27T09:42:17.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.069+0000 7f3a25e00700 1 Processor -- start 2024-05-27T09:42:17.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.069+0000 7f3a25e00700 1 -- start start 2024-05-27T09:42:17.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.070+0000 7f3a25e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a201045e0 0x7f3a2019d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.070+0000 7f3a25e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a20108f10 0x7f3a2019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.070+0000 7f3a25e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a20109860 0x7f3a201a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.070+0000 7f3a25e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3a201144e0 con 0x7f3a20108f10 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.070+0000 7f3a25e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3a20114360 con 0x7f3a201045e0 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.070+0000 7f3a25e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a20114660 con 0x7f3a20109860 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.070+0000 7f3a1e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a20108f10 0x7f3a2019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.070+0000 7f3a1e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a20108f10 0x7f3a2019d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43974/0 (socket says 172.21.15.44:43974) 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.070+0000 7f3a1e000700 1 -- 172.21.15.44:0/3544576485 learned_addr learned my addr 172.21.15.44:0/3544576485 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.070+0000 7f3a1ea00700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a201045e0 0x7f3a2019d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.070+0000 7f3a1f400700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a20109860 0x7f3a201a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.071+0000 7f3a1e000700 1 -- 172.21.15.44:0/3544576485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a20109860 msgr2=0x7f3a201a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.071+0000 7f3a1e000700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a20109860 0x7f3a201a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.071+0000 7f3a1e000700 1 -- 172.21.15.44:0/3544576485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a201045e0 msgr2=0x7f3a2019d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:17.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.071+0000 7f3a1e000700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a201045e0 0x7f3a2019d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.071+0000 7f3a1e000700 1 -- 172.21.15.44:0/3544576485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a201a1f80 con 0x7f3a20108f10 2024-05-27T09:42:17.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.071+0000 7f3a1ea00700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a201045e0 0x7f3a2019d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:17.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.071+0000 7f3a1e000700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a20108f10 0x7f3a2019d650 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00e970 tx=0x7f3a0c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:17.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.072+0000 7f3a24a00700 1 -- 172.21.15.44:0/3544576485 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c004a30 con 0x7f3a20108f10 2024-05-27T09:42:17.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.072+0000 7f3a25e00700 1 -- 172.21.15.44:0/3544576485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a201a2270 con 0x7f3a20108f10 2024-05-27T09:42:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.072+0000 7f3a24a00700 1 -- 172.21.15.44:0/3544576485 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a0c00cc90 con 0x7f3a20108f10 2024-05-27T09:42:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.072+0000 7f3a24a00700 1 -- 172.21.15.44:0/3544576485 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c005610 con 0x7f3a20108f10 2024-05-27T09:42:17.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.072+0000 7f3a25e00700 1 -- 172.21.15.44:0/3544576485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a201a9b90 con 0x7f3a20108f10 2024-05-27T09:42:17.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.076+0000 7f3a24a00700 1 -- 172.21.15.44:0/3544576485 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3a0c00c7e0 con 0x7f3a20108f10 2024-05-27T09:42:17.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.077+0000 7f3a34864b80 1 Processor -- start 2024-05-27T09:42:17.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.077+0000 7f3a34864b80 1 -- start start 2024-05-27T09:42:17.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe18aec0 0x55dbbe30f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 0x55dbbe3098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe30a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a34864b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55dbbe237ec0 con 0x55dbbe309e30 2024-05-27T09:42:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a34864b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55dbbe227a60 con 0x55dbbe18aec0 2024-05-27T09:42:17.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a34864b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dbbe22baa0 con 0x55dbbe30fdb0 2024-05-27T09:42:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a1ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 0x55dbbe3098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a1e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe30a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a1f400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe18aec0 0x55dbbe30f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a1e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe30a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43982/0 (socket says 172.21.15.44:43982) 2024-05-27T09:42:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a1ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 0x55dbbe3098f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50384/0 (socket says 172.21.15.44:50384) 2024-05-27T09:42:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.078+0000 7f3a1e000700 1 -- 172.21.15.44:0/594727513 learned_addr learned my addr 172.21.15.44:0/594727513 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.079+0000 7f3a24a00700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a040b29f0 0x7f3a040b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.079+0000 7f3a1f400700 1 -- 172.21.15.44:0/594727513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 msgr2=0x55dbbe3098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.079+0000 7f3a24a00700 1 -- 172.21.15.44:0/3544576485 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7f3a0c0e2810 con 0x7f3a20108f10 2024-05-27T09:42:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.079+0000 7f3a1f400700 1 --2- 172.21.15.44:0/594727513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 0x55dbbe3098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.079+0000 7f3a1f400700 1 -- 172.21.15.44:0/594727513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 msgr2=0x55dbbe30a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.079+0000 7f3a1f400700 1 --2- 172.21.15.44:0/594727513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe30a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.079+0000 7f3a1f400700 1 -- 172.21.15.44:0/594727513 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbbe1f04a0 con 0x55dbbe18aec0 2024-05-27T09:42:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.079+0000 7f3a1e000700 1 --2- 172.21.15.44:0/594727513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe30a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.079+0000 7f3a1ea00700 1 --2- 172.21.15.44:0/594727513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 0x55dbbe3098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.079+0000 7f3a1f400700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a040b29f0 0x7f3a040b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.080+0000 7f3a1f400700 1 --2- 172.21.15.44:0/594727513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe18aec0 0x55dbbe30f870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3a18011650 tx=0x7f3a18011960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebdff05c46be52ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.080+0000 7f3a14c00700 1 -- 172.21.15.44:0/594727513 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1801b070 con 0x55dbbe18aec0 2024-05-27T09:42:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.080+0000 7f3a14c00700 1 -- 172.21.15.44:0/594727513 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a18017b00 con 0x55dbbe18aec0 2024-05-27T09:42:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.080+0000 7f3a14c00700 1 -- 172.21.15.44:0/594727513 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a18020850 con 0x55dbbe18aec0 2024-05-27T09:42:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.081+0000 7f3a34864b80 1 -- 172.21.15.44:0/594727513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe18aec0 msgr2=0x55dbbe30f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.081+0000 7f3a1f400700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a040b29f0 0x7f3a040b4f00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a18022000 tx=0x7f3a1801f000 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.081+0000 7f3a34864b80 1 --2- 172.21.15.44:0/594727513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe18aec0 0x55dbbe30f870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3a18011650 tx=0x7f3a18011960 comp rx=0 tx=0).stop 2024-05-27T09:42:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.081+0000 7f3a34864b80 1 -- 172.21.15.44:0/594727513 shutdown_connections 2024-05-27T09:42:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.081+0000 7f3a34864b80 1 --2- 172.21.15.44:0/594727513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe30a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.081+0000 7f3a34864b80 1 --2- 172.21.15.44:0/594727513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe18aec0 0x55dbbe30f870 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.081+0000 7f3a34864b80 1 --2- 172.21.15.44:0/594727513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 0x55dbbe3098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.081+0000 7f3a34864b80 1 -- 172.21.15.44:0/594727513 >> 172.21.15.44:0/594727513 conn(0x55dbbe1eb040 msgr2=0x55dbbe304950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.081+0000 7f3a34864b80 1 -- 172.21.15.44:0/594727513 shutdown_connections 2024-05-27T09:42:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.082+0000 7f3a34864b80 1 -- 172.21.15.44:0/594727513 wait complete. 2024-05-27T09:42:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.082+0000 7f3a34864b80 1 -- start start 2024-05-27T09:42:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.083+0000 7f3a34864b80 1 Processor -- start 2024-05-27T09:42:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.083+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 0x55dbbe313590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.083+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe309e30 0x55dbbe313ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.083+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe30fdb0 0x55dbbe314010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.083+0000 7f3a34864b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55dbbe12e020 con 0x55dbbe18aec0 2024-05-27T09:42:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.083+0000 7f3a34864b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55dbbe1c87e0 con 0x55dbbe30fdb0 2024-05-27T09:42:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.083+0000 7f3a34864b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dbbe316880 con 0x55dbbe309e30 2024-05-27T09:42:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.083+0000 7f3a34864b80 10 client.0 _ll_register_callbacks cb 0x55dbbe1d1cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.084+0000 7f3a1ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 0x55dbbe313590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.084+0000 7f3a1e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe309e30 0x55dbbe313ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.084+0000 7f3a1ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 0x55dbbe313590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43998/0 (socket says 172.21.15.44:43998) 2024-05-27T09:42:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.084+0000 7f3a1ea00700 1 -- 172.21.15.44:0/693659755 learned_addr learned my addr 172.21.15.44:0/693659755 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.084+0000 7f3a1f400700 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe30fdb0 0x55dbbe314010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.084+0000 7f3a1e000700 1 -- 172.21.15.44:0/693659755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe30fdb0 msgr2=0x55dbbe314010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:17.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.084+0000 7f3a1e000700 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe30fdb0 0x55dbbe314010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.084+0000 7f3a1e000700 1 -- 172.21.15.44:0/693659755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 msgr2=0x55dbbe313590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.084+0000 7f3a1e000700 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 0x55dbbe313590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.084+0000 7f3a1e000700 1 -- 172.21.15.44:0/693659755 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbbe177950 con 0x55dbbe309e30 2024-05-27T09:42:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.085+0000 7f3a1e000700 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe309e30 0x55dbbe313ad0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0a9760 tx=0x7f3a0c0b10b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c357f9d6fc4a9603 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.085+0000 7f3a09600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:42:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.085+0000 7f3a09600700 1 -- 172.21.15.44:0/693659755 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0a93b0 con 0x55dbbe309e30 2024-05-27T09:42:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.085+0000 7f3a09600700 1 -- 172.21.15.44:0/693659755 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7f3a0c0a9cc0 con 0x55dbbe309e30 2024-05-27T09:42:17.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.085+0000 7f3a34864b80 1 -- 172.21.15.44:0/693659755 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a18017b00 con 0x55dbbe309e30 2024-05-27T09:42:17.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.086+0000 7f3a09600700 1 -- 172.21.15.44:0/693659755 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3a0c0a51f0 con 0x55dbbe309e30 2024-05-27T09:42:17.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.086+0000 7f3a34864b80 10 client.26512 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:17.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.086+0000 7f3a34864b80 1 -- 172.21.15.44:0/693659755 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dbbe1f80e0 con 0x55dbbe309e30 2024-05-27T09:42:17.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.086+0000 7f3a09600700 1 -- 172.21.15.44:0/693659755 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a18017b00 con 0x55dbbe309e30 2024-05-27T09:42:17.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.086+0000 7f3a09600700 1 -- 172.21.15.44:0/693659755 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f3a0c0b2430 con 0x55dbbe309e30 2024-05-27T09:42:17.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.086+0000 7f3a34864b80 10 client.26512 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:17.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.086+0000 7f3a34864b80 10 client.26512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:17.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.087+0000 7f3a34864b80 20 client.26512 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.087+0000 7f3a34864b80 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dbbe323740 0x55dbbe325bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.087+0000 7f3a34864b80 4 client.26512 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.087+0000 7f3a34864b80 1 -- 172.21.15.44:0/693659755 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbbe239e10 con 0x55dbbe323740 2024-05-27T09:42:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.087+0000 7f3a1ea00700 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dbbe323740 0x55dbbe325bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.087+0000 7f3a09600700 10 client.26512 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.088+0000 7f3a1ea00700 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dbbe323740 0x55dbbe325bb0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d66aa3fc0cc0c745 server_cookie=ef2015ef8a83ad26 in_seq=0 out_seq=0 2024-05-27T09:42:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a09600700 1 -- 172.21.15.44:0/693659755 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dbbe239e10 con 0x55dbbe323740 2024-05-27T09:42:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a09600700 10 client.26512 handle_command_reply: tid=0 2024-05-27T09:42:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a34864b80 1 client.26512 shutdown 2024-05-27T09:42:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a34864b80 1 -- 172.21.15.44:0/693659755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe309e30 msgr2=0x55dbbe313ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a34864b80 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe309e30 0x55dbbe313ad0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0a9760 tx=0x7f3a0c0b10b0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a34864b80 1 -- 172.21.15.44:0/693659755 shutdown_connections 2024-05-27T09:42:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a34864b80 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dbbe323740 0x55dbbe325bb0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a34864b80 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe30fdb0 0x55dbbe314010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a34864b80 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 0x55dbbe313590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a34864b80 1 --2- 172.21.15.44:0/693659755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe309e30 0x55dbbe313ad0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a34864b80 1 -- 172.21.15.44:0/693659755 >> 172.21.15.44:0/693659755 conn(0x55dbbe1eb040 msgr2=0x55dbbe304600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a09600700 0 client.26512 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a34864b80 1 -- 172.21.15.44:0/693659755 shutdown_connections 2024-05-27T09:42:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.089+0000 7f3a34864b80 1 -- 172.21.15.44:0/693659755 wait complete. 2024-05-27T09:42:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.090+0000 7f3a34864b80 20 client.26512 trim_cache size 0 max 16384 2024-05-27T09:42:17.104 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.114+0000 7f3a34864b80 1 Processor -- start 2024-05-27T09:42:17.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.114+0000 7f3a34864b80 1 -- start start 2024-05-27T09:42:17.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.114+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 0x55dbbe3098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.114+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe309e30 0x55dbbe31f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.114+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 0x55dbbe313bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.114+0000 7f3a34864b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3a201b3ad0 con 0x55dbbe18aec0 2024-05-27T09:42:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.114+0000 7f3a34864b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3a201b1330 con 0x55dbbe309e30 2024-05-27T09:42:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.114+0000 7f3a34864b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a201af140 con 0x55dbbe30fdb0 2024-05-27T09:42:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.115+0000 7f3a1ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 0x55dbbe3098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.115+0000 7f3a1ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 0x55dbbe3098f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44008/0 (socket says 172.21.15.44:44008) 2024-05-27T09:42:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.115+0000 7f3a1ea00700 1 -- 172.21.15.44:0/2249875158 learned_addr learned my addr 172.21.15.44:0/2249875158 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.115+0000 7f3a1e000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe309e30 0x55dbbe31f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.115+0000 7f3a1f400700 1 --2- 172.21.15.44:0/2249875158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 0x55dbbe313bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.115+0000 7f3a1e000700 1 -- 172.21.15.44:0/2249875158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 msgr2=0x55dbbe313bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.115+0000 7f3a1e000700 1 --2- 172.21.15.44:0/2249875158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 0x55dbbe313bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.115+0000 7f3a1e000700 1 -- 172.21.15.44:0/2249875158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 msgr2=0x55dbbe3098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.115+0000 7f3a1e000700 1 --2- 172.21.15.44:0/2249875158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 0x55dbbe3098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.115+0000 7f3a1e000700 1 -- 172.21.15.44:0/2249875158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbbe30f210 con 0x55dbbe309e30 2024-05-27T09:42:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.115+0000 7f3a1ea00700 1 --2- 172.21.15.44:0/2249875158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 0x55dbbe3098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.116+0000 7f3a1e000700 1 --2- 172.21.15.44:0/2249875158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe309e30 0x55dbbe31f920 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c000c00 tx=0x7f3a0c004bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d6e408a6ce5e760 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.116+0000 7f3a0be00700 1 -- 172.21.15.44:0/2249875158 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c06d070 con 0x55dbbe309e30 2024-05-27T09:42:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.116+0000 7f3a0be00700 1 -- 172.21.15.44:0/2249875158 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a0c0a5040 con 0x55dbbe309e30 2024-05-27T09:42:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.116+0000 7f3a0be00700 1 -- 172.21.15.44:0/2249875158 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0a91d0 con 0x55dbbe309e30 2024-05-27T09:42:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.117+0000 7f3a34864b80 1 -- 172.21.15.44:0/2249875158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe309e30 msgr2=0x55dbbe31f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.117+0000 7f3a34864b80 1 --2- 172.21.15.44:0/2249875158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe309e30 0x55dbbe31f920 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c000c00 tx=0x7f3a0c004bd0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.117+0000 7f3a34864b80 1 -- 172.21.15.44:0/2249875158 shutdown_connections 2024-05-27T09:42:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.117+0000 7f3a34864b80 1 --2- 172.21.15.44:0/2249875158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe30fdb0 0x55dbbe313bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.117+0000 7f3a34864b80 1 --2- 172.21.15.44:0/2249875158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe18aec0 0x55dbbe3098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.117+0000 7f3a34864b80 1 --2- 172.21.15.44:0/2249875158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe309e30 0x55dbbe31f920 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.117+0000 7f3a34864b80 1 -- 172.21.15.44:0/2249875158 >> 172.21.15.44:0/2249875158 conn(0x55dbbe1eb040 msgr2=0x55dbbe310ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.117+0000 7f3a34864b80 1 -- 172.21.15.44:0/2249875158 shutdown_connections 2024-05-27T09:42:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.117+0000 7f3a34864b80 1 -- 172.21.15.44:0/2249875158 wait complete. 2024-05-27T09:42:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.118+0000 7f3a34864b80 1 -- start start 2024-05-27T09:42:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.118+0000 7f3a34864b80 1 Processor -- start 2024-05-27T09:42:17.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe18aec0 0x55dbbe319100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe319640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a34864b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe30fdb0 0x55dbbe38e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a34864b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3a201af7f0 con 0x55dbbe309e30 2024-05-27T09:42:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a34864b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3a201b0c60 con 0x55dbbe30fdb0 2024-05-27T09:42:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a34864b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a201afec0 con 0x55dbbe18aec0 2024-05-27T09:42:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a34864b80 10 client.0 _ll_register_callbacks cb 0x55dbbe3036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a1e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe319640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a1ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe18aec0 0x55dbbe319100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a1e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe319640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44020/0 (socket says 172.21.15.44:44020) 2024-05-27T09:42:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a1e000700 1 -- 172.21.15.44:0/2990103246 learned_addr learned my addr 172.21.15.44:0/2990103246 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.119+0000 7f3a1f400700 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe30fdb0 0x55dbbe38e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.120+0000 7f3a1ea00700 1 -- 172.21.15.44:0/2990103246 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe30fdb0 msgr2=0x55dbbe38e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:17.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.120+0000 7f3a1ea00700 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe30fdb0 0x55dbbe38e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.120+0000 7f3a1ea00700 1 -- 172.21.15.44:0/2990103246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 msgr2=0x55dbbe319640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.120+0000 7f3a1ea00700 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe319640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.120+0000 7f3a1ea00700 1 -- 172.21.15.44:0/2990103246 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbbe38e7e0 con 0x55dbbe18aec0 2024-05-27T09:42:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.120+0000 7f3a1e000700 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe319640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.120+0000 7f3a1ea00700 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe18aec0 0x55dbbe319100 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3a10000c00 tx=0x7f3a1000be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7262fae73702baff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.120+0000 7f3a09600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:42:17.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.120+0000 7f3a09600700 1 -- 172.21.15.44:0/2990103246 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10005580 con 0x55dbbe18aec0 2024-05-27T09:42:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.120+0000 7f3a34864b80 1 -- 172.21.15.44:0/2990103246 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a0c0a5040 con 0x55dbbe18aec0 2024-05-27T09:42:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.120+0000 7f3a09600700 1 -- 172.21.15.44:0/2990103246 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8565+0+0 (secure 0 0 0) 0x7f3a100037c0 con 0x55dbbe18aec0 2024-05-27T09:42:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.121+0000 7f3a09600700 1 -- 172.21.15.44:0/2990103246 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3a100039c0 con 0x55dbbe18aec0 2024-05-27T09:42:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.121+0000 7f3a34864b80 10 client.26518 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.121+0000 7f3a34864b80 1 -- 172.21.15.44:0/2990103246 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dbbe38fac0 con 0x55dbbe18aec0 2024-05-27T09:42:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.121+0000 7f3a09600700 1 -- 172.21.15.44:0/2990103246 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a0c0a5040 con 0x55dbbe18aec0 2024-05-27T09:42:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.122+0000 7f3a09600700 1 -- 172.21.15.44:0/2990103246 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f3a10038070 con 0x55dbbe18aec0 2024-05-27T09:42:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.122+0000 7f3a34864b80 10 client.26518 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.122+0000 7f3a34864b80 10 client.26518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.122+0000 7f3a34864b80 20 client.26518 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:17.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.122+0000 7f3a34864b80 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dbbe390170 0x55dbbe392590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:17.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.122+0000 7f3a34864b80 4 client.26518 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:42:17.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.122+0000 7f3a34864b80 1 -- 172.21.15.44:0/2990103246 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dbbe2335f0 con 0x55dbbe390170 2024-05-27T09:42:17.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.122+0000 7f3a1e000700 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dbbe390170 0x55dbbe392590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:17.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.123+0000 7f3a1e000700 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dbbe390170 0x55dbbe392590 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ecc7c2c0aa81234 server_cookie=c7c56be7c54c04d2 in_seq=0 out_seq=0 2024-05-27T09:42:17.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.123+0000 7f3a09600700 10 client.26518 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:17.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.124+0000 7f3a09600700 1 -- 172.21.15.44:0/2990103246 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dbbe2335f0 con 0x55dbbe390170 2024-05-27T09:42:17.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.124+0000 7f3a09600700 10 client.26518 handle_command_reply: tid=0 2024-05-27T09:42:17.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.124+0000 7f3a34864b80 1 client.26518 shutdown 2024-05-27T09:42:17.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.124+0000 7f3a34864b80 1 -- 172.21.15.44:0/2990103246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe18aec0 msgr2=0x55dbbe319100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:17.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.124+0000 7f3a34864b80 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe18aec0 0x55dbbe319100 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3a10000c00 tx=0x7f3a1000be60 comp rx=0 tx=0).stop 2024-05-27T09:42:17.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.124+0000 7f3a34864b80 1 -- 172.21.15.44:0/2990103246 shutdown_connections 2024-05-27T09:42:17.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.124+0000 7f3a34864b80 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dbbe390170 0x55dbbe392590 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.124+0000 7f3a34864b80 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dbbe30fdb0 0x55dbbe38e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.124+0000 7f3a34864b80 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbbe18aec0 0x55dbbe319100 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.124+0000 7f3a34864b80 1 --2- 172.21.15.44:0/2990103246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dbbe309e30 0x55dbbe319640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.124+0000 7f3a34864b80 1 -- 172.21.15.44:0/2990103246 >> 172.21.15.44:0/2990103246 conn(0x55dbbe1eb040 msgr2=0x55dbbe310ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:17.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.125+0000 7f3a09600700 0 client.26518 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:17.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.125+0000 7f3a34864b80 1 -- 172.21.15.44:0/2990103246 shutdown_connections 2024-05-27T09:42:17.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.125+0000 7f3a34864b80 1 -- 172.21.15.44:0/2990103246 wait complete. 2024-05-27T09:42:17.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.125+0000 7f3a34864b80 20 client.26518 trim_cache size 0 max 16384 2024-05-27T09:42:17.128 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:42:17.128 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:42:17.128 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "7e532cb2-6cd4-4494-953d-0430f62f7964", 2024-05-27T09:42:17.128 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:42:17.128 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:42:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.126+0000 7f3a25e00700 1 -- 172.21.15.44:0/3544576485 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a040b29f0 msgr2=0x7f3a040b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.126+0000 7f3a25e00700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a040b29f0 0x7f3a040b4f00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a18022000 tx=0x7f3a1801f000 comp rx=0 tx=0).stop 2024-05-27T09:42:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.126+0000 7f3a25e00700 1 -- 172.21.15.44:0/3544576485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a20108f10 msgr2=0x7f3a2019d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.126+0000 7f3a25e00700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a20108f10 0x7f3a2019d650 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00e970 tx=0x7f3a0c00ec80 comp rx=0 tx=0).stop 2024-05-27T09:42:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.126+0000 7f3a25e00700 1 -- 172.21.15.44:0/3544576485 shutdown_connections 2024-05-27T09:42:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.126+0000 7f3a25e00700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3a040b29f0 0x7f3a040b4f00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.126+0000 7f3a25e00700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a20109860 0x7f3a201a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.126+0000 7f3a25e00700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a201045e0 0x7f3a2019d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.126+0000 7f3a25e00700 1 --2- 172.21.15.44:0/3544576485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a20108f10 0x7f3a2019d650 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.126+0000 7f3a25e00700 1 -- 172.21.15.44:0/3544576485 >> 172.21.15.44:0/3544576485 conn(0x7f3a20100030 msgr2=0x7f3a2010a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.127+0000 7f3a25e00700 1 -- 172.21.15.44:0/3544576485 shutdown_connections 2024-05-27T09:42:17.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:17.127+0000 7f3a25e00700 1 -- 172.21.15.44:0/3544576485 wait complete. 2024-05-27T09:42:18.043 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:17 smithi050 ceph-mon[36371]: pgmap v216: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2024-05-27T09:42:18.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:17 smithi189 ceph-mon[36671]: pgmap v216: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2024-05-27T09:42:18.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:17 smithi044 ceph-mon[31618]: pgmap v216: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2024-05-27T09:42:18.381 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:42:18.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:18 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:42:18.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:18 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:42:18.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:18 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:42:18.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:18 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:42:18.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:18 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:42:18.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:18 smithi044 ceph-mon[31618]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T09:42:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:18 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:42:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:18 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:42:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:18 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:42:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:18 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:42:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:18 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:42:19.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:18 smithi050 ceph-mon[36371]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T09:42:19.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:18 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:42:19.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:18 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:42:19.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:18 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:42:19.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:18 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:42:19.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:18 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:42:19.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:18 smithi189 ceph-mon[36671]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T09:42:19.512 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:19.941 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:19 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:42:19.941 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:19 smithi044 ceph-mon[31618]: pgmap v218: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 5 op/s 2024-05-27T09:42:20.083 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:19 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:42:20.083 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:19 smithi189 ceph-mon[36671]: pgmap v218: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 5 op/s 2024-05-27T09:42:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:19 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:42:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:19 smithi050 ceph-mon[36371]: pgmap v218: 97 pgs: 97 active+clean; 26 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 5 op/s 2024-05-27T09:42:20.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.584+0000 7fd5fbc00700 1 -- 172.21.15.44:0/3492753745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f4108ef0 msgr2=0x7fd5f4109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:20.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.584+0000 7fd5fbc00700 1 --2- 172.21.15.44:0/3492753745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f4108ef0 0x7fd5f4109300 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00099c0 tx=0x7fd5e0009cd0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.584+0000 7fd5fbc00700 1 -- 172.21.15.44:0/3492753745 shutdown_connections 2024-05-27T09:42:20.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.584+0000 7fd5fbc00700 1 --2- 172.21.15.44:0/3492753745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5f4109840 0x7fd5f41111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.584+0000 7fd5fbc00700 1 --2- 172.21.15.44:0/3492753745 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5f41045c0 0x7fd5f41049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.584+0000 7fd5fbc00700 1 --2- 172.21.15.44:0/3492753745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f4108ef0 0x7fd5f4109300 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.584+0000 7fd5fbc00700 1 -- 172.21.15.44:0/3492753745 >> 172.21.15.44:0/3492753745 conn(0x7fd5f40ffe70 msgr2=0x7fd5f41022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:20.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.584+0000 7fd5fbc00700 1 -- 172.21.15.44:0/3492753745 shutdown_connections 2024-05-27T09:42:20.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.585+0000 7fd5fbc00700 1 -- 172.21.15.44:0/3492753745 wait complete. 2024-05-27T09:42:20.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.586+0000 7fd5fbc00700 1 Processor -- start 2024-05-27T09:42:20.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.587+0000 7fd5fbc00700 1 -- start start 2024-05-27T09:42:20.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.587+0000 7fd5fbc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f41045c0 0x7fd5f419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.587+0000 7fd5fbc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5f4108ef0 0x7fd5f419d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.587+0000 7fd5fbc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5f4109840 0x7fd5f41a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.587+0000 7fd5fbc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f4114450 con 0x7fd5f4108ef0 2024-05-27T09:42:20.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.587+0000 7fd5fbc00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41142d0 con 0x7fd5f4109840 2024-05-27T09:42:20.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.587+0000 7fd5fbc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41145d0 con 0x7fd5f41045c0 2024-05-27T09:42:20.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.588+0000 7fd5f3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5f4108ef0 0x7fd5f419d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.588+0000 7fd5f9c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5f4109840 0x7fd5f41a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.588+0000 7fd5f9c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5f4109840 0x7fd5f41a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38684/0 (socket says 172.21.15.44:38684) 2024-05-27T09:42:20.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.588+0000 7fd5f9c00700 1 -- 172.21.15.44:0/1656504596 learned_addr learned my addr 172.21.15.44:0/1656504596 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:20.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.588+0000 7fd5f9200700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f41045c0 0x7fd5f419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.588+0000 7fd5f9c00700 1 -- 172.21.15.44:0/1656504596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f41045c0 msgr2=0x7fd5f419d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:20.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.588+0000 7fd5f9c00700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f41045c0 0x7fd5f419d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.588+0000 7fd5f9c00700 1 -- 172.21.15.44:0/1656504596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5f4108ef0 msgr2=0x7fd5f419d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:20.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.588+0000 7fd5f9c00700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5f4108ef0 0x7fd5f419d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.588+0000 7fd5f9c00700 1 -- 172.21.15.44:0/1656504596 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f41a1ed0 con 0x7fd5f4109840 2024-05-27T09:42:20.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.589+0000 7fd5f9c00700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5f4109840 0x7fd5f41a1830 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00ef30 tx=0x7fd5ec00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:20.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.589+0000 7fd5f1600700 1 -- 172.21.15.44:0/1656504596 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0092e0 con 0x7fd5f4109840 2024-05-27T09:42:20.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.589+0000 7fd5fbc00700 1 -- 172.21.15.44:0/1656504596 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f41a2160 con 0x7fd5f4109840 2024-05-27T09:42:20.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.589+0000 7fd5f1600700 1 -- 172.21.15.44:0/1656504596 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5ec004bb0 con 0x7fd5f4109840 2024-05-27T09:42:20.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.589+0000 7fd5f1600700 1 -- 172.21.15.44:0/1656504596 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec004280 con 0x7fd5f4109840 2024-05-27T09:42:20.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.589+0000 7fd5fbc00700 1 -- 172.21.15.44:0/1656504596 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f41a2670 con 0x7fd5f4109840 2024-05-27T09:42:20.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.593+0000 7fd5f1600700 1 -- 172.21.15.44:0/1656504596 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd5ec01b030 con 0x7fd5f4109840 2024-05-27T09:42:20.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.595+0000 7fd60a71bb80 1 Processor -- start 2024-05-27T09:42:20.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.595+0000 7fd60a71bb80 1 -- start start 2024-05-27T09:42:20.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.595+0000 7fd5f1600700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5dc0b2a70 0x7fd5dc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.595+0000 7fd5f1600700 1 -- 172.21.15.44:0/1656504596 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fd5ec0e20e0 con 0x7fd5f4109840 2024-05-27T09:42:20.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd5f3e00700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5dc0b2a70 0x7fd5dc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 0x55e52694f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 0x55e52694fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e5269499d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e526876f20 con 0x55e5269495c0 2024-05-27T09:42:20.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e526866ac0 con 0x55e5268b5ad0 2024-05-27T09:42:20.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e52686ab00 con 0x55e5267c9f20 2024-05-27T09:42:20.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd5f3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 0x55e52694fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd5f3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 0x55e52694fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50458/0 (socket says 172.21.15.44:50458) 2024-05-27T09:42:20.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd5f3e00700 1 -- 172.21.15.44:0/2475463646 learned_addr learned my addr 172.21.15.44:0/2475463646 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:20.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd5f9200700 1 --2- 172.21.15.44:0/2475463646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 0x55e52694f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.596+0000 7fd5f9c00700 1 --2- 172.21.15.44:0/2475463646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e5269499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.597+0000 7fd5f3e00700 1 -- 172.21.15.44:0/2475463646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 msgr2=0x55e52694f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:20.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.597+0000 7fd5f3e00700 1 --2- 172.21.15.44:0/2475463646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 0x55e52694f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.597+0000 7fd5f3e00700 1 -- 172.21.15.44:0/2475463646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 msgr2=0x55e5269499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:20.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.597+0000 7fd5f3e00700 1 --2- 172.21.15.44:0/2475463646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e5269499d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.597+0000 7fd5f3e00700 1 -- 172.21.15.44:0/2475463646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e52682f500 con 0x55e5267c9f20 2024-05-27T09:42:20.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.597+0000 7fd5f3e00700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5dc0b2a70 0x7fd5dc0b4f80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd5f406c120 tx=0x7fd5e0005fb0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:20.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.597+0000 7fd5f3e00700 1 --2- 172.21.15.44:0/2475463646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 0x55e52694fa70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0003760 tx=0x7fd5e002a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a136c2199f29da6d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.597+0000 7fd5e6000700 1 -- 172.21.15.44:0/2475463646 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0021040 con 0x55e5267c9f20 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.597+0000 7fd5e6000700 1 -- 172.21.15.44:0/2475463646 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5e002d070 con 0x55e5267c9f20 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.597+0000 7fd5e6000700 1 -- 172.21.15.44:0/2475463646 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e000f6c0 con 0x55e5267c9f20 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.598+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2475463646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 msgr2=0x55e52694fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.598+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/2475463646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 0x55e52694fa70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0003760 tx=0x7fd5e002a000 comp rx=0 tx=0).stop 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.599+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2475463646 shutdown_connections 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.599+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/2475463646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e5269499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.599+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/2475463646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 0x55e52694f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.599+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/2475463646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 0x55e52694fa70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.599+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2475463646 >> 172.21.15.44:0/2475463646 conn(0x55e526863470 msgr2=0x55e526944630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.599+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2475463646 shutdown_connections 2024-05-27T09:42:20.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.599+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2475463646 wait complete. 2024-05-27T09:42:20.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.599+0000 7fd60a71bb80 1 -- start start 2024-05-27T09:42:20.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.600+0000 7fd60a71bb80 1 Processor -- start 2024-05-27T09:42:20.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.600+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5267c9f20 0x55e5269533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.600+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5268b5ad0 0x55e526953900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.600+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e526953e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.600+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e52676d080 con 0x55e5269495c0 2024-05-27T09:42:20.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.600+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e526807840 con 0x55e5267c9f20 2024-05-27T09:42:20.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.600+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e52694a350 con 0x55e5268b5ad0 2024-05-27T09:42:20.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.600+0000 7fd60a71bb80 10 client.0 _ll_register_callbacks cb 0x55e526810d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:20.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.601+0000 7fd5f9200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5267c9f20 0x55e5269533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.601+0000 7fd5f3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5268b5ad0 0x55e526953900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.601+0000 7fd5f9200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5267c9f20 0x55e5269533c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38696/0 (socket says 172.21.15.44:38696) 2024-05-27T09:42:20.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.601+0000 7fd5f9200700 1 -- 172.21.15.44:0/1135341158 learned_addr learned my addr 172.21.15.44:0/1135341158 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:20.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.601+0000 7fd5f9c00700 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e526953e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.601+0000 7fd5f9200700 1 -- 172.21.15.44:0/1135341158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5268b5ad0 msgr2=0x55e526953900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:20.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.601+0000 7fd5f9200700 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5268b5ad0 0x55e526953900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.601+0000 7fd5f9200700 1 -- 172.21.15.44:0/1135341158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 msgr2=0x55e526953e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:20.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.601+0000 7fd5f9200700 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e526953e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.601+0000 7fd5f9200700 1 -- 172.21.15.44:0/1135341158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5267b69b0 con 0x55e5267c9f20 2024-05-27T09:42:20.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.602+0000 7fd5f9200700 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5267c9f20 0x55e5269533c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8012810 tx=0x7fd5e8012b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=354baa401ffeb077 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:20.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.602+0000 7fd5daa00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.602+0000 7fd5daa00700 1 -- 172.21.15.44:0/1135341158 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e80117f0 con 0x55e5267c9f20 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.602+0000 7fd60a71bb80 1 -- 172.21.15.44:0/1135341158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5e002d070 con 0x55e5267c9f20 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.602+0000 7fd5daa00700 1 -- 172.21.15.44:0/1135341158 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fd5e8011990 con 0x55e5267c9f20 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.602+0000 7fd5daa00700 1 -- 172.21.15.44:0/1135341158 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd5e8011b90 con 0x55e5267c9f20 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.602+0000 7fd60a71bb80 10 client.26534 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.602+0000 7fd60a71bb80 1 -- 172.21.15.44:0/1135341158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e526837140 con 0x55e5267c9f20 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.603+0000 7fd5daa00700 1 -- 172.21.15.44:0/1135341158 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5e002d070 con 0x55e5267c9f20 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.603+0000 7fd5daa00700 1 -- 172.21.15.44:0/1135341158 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fd5e801c420 con 0x55e5267c9f20 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.603+0000 7fd60a71bb80 10 client.26534 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.603+0000 7fd60a71bb80 10 client.26534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.603+0000 7fd60a71bb80 20 client.26534 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.603+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5269632a0 0x55e526965750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.603+0000 7fd60a71bb80 4 client.26534 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:20.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.603+0000 7fd60a71bb80 1 -- 172.21.15.44:0/1135341158 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e526878e70 con 0x55e5269632a0 2024-05-27T09:42:20.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.603+0000 7fd5f3e00700 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5269632a0 0x55e526965750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.604+0000 7fd5f3e00700 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5269632a0 0x55e526965750 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e12629d59c8e397 server_cookie=a7dd9c69ed1c2e05 in_seq=0 out_seq=0 2024-05-27T09:42:20.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.604+0000 7fd5daa00700 10 client.26534 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:20.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd5daa00700 1 -- 172.21.15.44:0/1135341158 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e526878e70 con 0x55e5269632a0 2024-05-27T09:42:20.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd5daa00700 10 client.26534 handle_command_reply: tid=0 2024-05-27T09:42:20.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd60a71bb80 1 client.26534 shutdown 2024-05-27T09:42:20.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd60a71bb80 1 -- 172.21.15.44:0/1135341158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5267c9f20 msgr2=0x55e5269533c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:20.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5267c9f20 0x55e5269533c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8012810 tx=0x7fd5e8012b20 comp rx=0 tx=0).stop 2024-05-27T09:42:20.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd60a71bb80 1 -- 172.21.15.44:0/1135341158 shutdown_connections 2024-05-27T09:42:20.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5269632a0 0x55e526965750 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e526953e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5267c9f20 0x55e5269533c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/1135341158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5268b5ad0 0x55e526953900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd60a71bb80 1 -- 172.21.15.44:0/1135341158 >> 172.21.15.44:0/1135341158 conn(0x55e526863470 msgr2=0x55e5269485b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:20.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.605+0000 7fd5daa00700 0 client.26534 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:20.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.606+0000 7fd60a71bb80 1 -- 172.21.15.44:0/1135341158 shutdown_connections 2024-05-27T09:42:20.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.606+0000 7fd60a71bb80 1 -- 172.21.15.44:0/1135341158 wait complete. 2024-05-27T09:42:20.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.606+0000 7fd60a71bb80 20 client.26534 trim_cache size 0 max 16384 2024-05-27T09:42:20.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.626+0000 7fd60a71bb80 1 Processor -- start 2024-05-27T09:42:20.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.626+0000 7fd60a71bb80 1 -- start start 2024-05-27T09:42:20.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.627+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5267c9f20 0x55e5269526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.627+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5268b5ad0 0x55e526955860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.627+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5269495c0 0x55e526943f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.627+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41b3810 con 0x55e5267c9f20 2024-05-27T09:42:20.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.627+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41b1070 con 0x55e5269495c0 2024-05-27T09:42:20.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.627+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41aeee0 con 0x55e5268b5ad0 2024-05-27T09:42:20.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.627+0000 7fd5f9200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5267c9f20 0x55e5269526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.627+0000 7fd5f9200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5267c9f20 0x55e5269526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44052/0 (socket says 172.21.15.44:44052) 2024-05-27T09:42:20.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.627+0000 7fd5f9c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5269495c0 0x55e526943f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.627+0000 7fd5f9200700 1 -- 172.21.15.44:0/3022698169 learned_addr learned my addr 172.21.15.44:0/3022698169 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:20.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.627+0000 7fd5f3e00700 1 --2- 172.21.15.44:0/3022698169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5268b5ad0 0x55e526955860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.628+0000 7fd5f9c00700 1 -- 172.21.15.44:0/3022698169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5268b5ad0 msgr2=0x55e526955860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:20.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.628+0000 7fd5f9c00700 1 --2- 172.21.15.44:0/3022698169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5268b5ad0 0x55e526955860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.628+0000 7fd5f9c00700 1 -- 172.21.15.44:0/3022698169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5267c9f20 msgr2=0x55e5269526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:20.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.628+0000 7fd5f9c00700 1 --2- 172.21.15.44:0/3022698169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5267c9f20 0x55e5269526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.628+0000 7fd5f9c00700 1 -- 172.21.15.44:0/3022698169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e526951930 con 0x55e5269495c0 2024-05-27T09:42:20.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.628+0000 7fd5f9c00700 1 --2- 172.21.15.44:0/3022698169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5269495c0 0x55e526943f90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0df7d0 tx=0x7fd5ec0a8e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d4f846d49c1c6e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:20.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.628+0000 7fd5e5600700 1 -- 172.21.15.44:0/3022698169 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0a93f0 con 0x55e5269495c0 2024-05-27T09:42:20.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.628+0000 7fd5e5600700 1 -- 172.21.15.44:0/3022698169 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5ec0ad7a0 con 0x55e5269495c0 2024-05-27T09:42:20.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.628+0000 7fd5e5600700 1 -- 172.21.15.44:0/3022698169 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec004420 con 0x55e5269495c0 2024-05-27T09:42:20.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.629+0000 7fd60a71bb80 1 -- 172.21.15.44:0/3022698169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5269495c0 msgr2=0x55e526943f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:20.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.629+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/3022698169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5269495c0 0x55e526943f90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0df7d0 tx=0x7fd5ec0a8e70 comp rx=0 tx=0).stop 2024-05-27T09:42:20.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.629+0000 7fd60a71bb80 1 -- 172.21.15.44:0/3022698169 shutdown_connections 2024-05-27T09:42:20.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.629+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/3022698169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5269495c0 0x55e526943f90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.629+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/3022698169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5267c9f20 0x55e5269526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.629+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/3022698169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5268b5ad0 0x55e526955860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.629+0000 7fd60a71bb80 1 -- 172.21.15.44:0/3022698169 >> 172.21.15.44:0/3022698169 conn(0x55e526863470 msgr2=0x55e526948c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:20.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.629+0000 7fd60a71bb80 1 -- 172.21.15.44:0/3022698169 shutdown_connections 2024-05-27T09:42:20.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.629+0000 7fd60a71bb80 1 -- 172.21.15.44:0/3022698169 wait complete. 2024-05-27T09:42:20.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.630+0000 7fd60a71bb80 1 -- start start 2024-05-27T09:42:20.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.630+0000 7fd60a71bb80 1 Processor -- start 2024-05-27T09:42:20.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 0x55e52695bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 0x55e52696a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd60a71bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e52696af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41af530 con 0x55e5269495c0 2024-05-27T09:42:20.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41b09a0 con 0x55e5268b5ad0 2024-05-27T09:42:20.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd60a71bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41afc00 con 0x55e5267c9f20 2024-05-27T09:42:20.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd60a71bb80 10 client.0 _ll_register_callbacks cb 0x55e526810d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:20.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd5f3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 0x55e52696a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd5f3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 0x55e52696a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38724/0 (socket says 172.21.15.44:38724) 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd5f3e00700 1 -- 172.21.15.44:0/2806991962 learned_addr learned my addr 172.21.15.44:0/2806991962 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd5f9c00700 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e52696af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.631+0000 7fd5f9200700 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 0x55e52695bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.632+0000 7fd5f3e00700 1 -- 172.21.15.44:0/2806991962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 msgr2=0x55e52695bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.632+0000 7fd5f3e00700 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 0x55e52695bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.632+0000 7fd5f3e00700 1 -- 172.21.15.44:0/2806991962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 msgr2=0x55e52696af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.632+0000 7fd5f3e00700 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e52696af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.632+0000 7fd5f3e00700 1 -- 172.21.15.44:0/2806991962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e52696b6f0 con 0x55e5268b5ad0 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.632+0000 7fd5f3e00700 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 0x55e52696a900 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009990 tx=0x7fd5e0005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=126f0f97a9155cda server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.632+0000 7fd5daa00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:20.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.632+0000 7fd5daa00700 1 -- 172.21.15.44:0/2806991962 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e000b790 con 0x55e5268b5ad0 2024-05-27T09:42:20.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.632+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2806991962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5ec0ad7a0 con 0x55e5268b5ad0 2024-05-27T09:42:20.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd5daa00700 1 -- 172.21.15.44:0/2806991962 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fd5e000b930 con 0x55e5268b5ad0 2024-05-27T09:42:20.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd5daa00700 1 -- 172.21.15.44:0/2806991962 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd5e000bb30 con 0x55e5268b5ad0 2024-05-27T09:42:20.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd60a71bb80 10 client.26540 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:20.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2806991962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e5269cde00 con 0x55e5268b5ad0 2024-05-27T09:42:20.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd5daa00700 1 -- 172.21.15.44:0/2806991962 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5ec0ad7a0 con 0x55e5268b5ad0 2024-05-27T09:42:20.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd5daa00700 1 -- 172.21.15.44:0/2806991962 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fd5e0029430 con 0x55e5268b5ad0 2024-05-27T09:42:20.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd60a71bb80 10 client.26540 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:20.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd60a71bb80 10 client.26540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:20.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd60a71bb80 20 client.26540 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:20.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5269ce0f0 0x55e5269d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:20.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd60a71bb80 4 client.26540 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-77ef2bd1-1-mds.b.json"}] 2024-05-27T09:42:20.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.633+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2806991962 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-77ef2bd1-1-mds.b.json"}) -- 0x55e52693c7f0 con 0x55e5269ce0f0 2024-05-27T09:42:20.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.634+0000 7fd5f9200700 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5269ce0f0 0x55e5269d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:20.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.634+0000 7fd5f9200700 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5269ce0f0 0x55e5269d0510 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fdecb6d58819126 server_cookie=2bd9007537f6b118 in_seq=0 out_seq=0 2024-05-27T09:42:20.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.634+0000 7fd5daa00700 10 client.26540 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:20.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.795+0000 7fd5daa00700 1 -- 172.21.15.44:0/2806991962 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x55e52693c7f0 con 0x55e5269ce0f0 2024-05-27T09:42:20.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.795+0000 7fd5daa00700 10 client.26540 handle_command_reply: tid=0 2024-05-27T09:42:20.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.795+0000 7fd60a71bb80 1 client.26540 shutdown 2024-05-27T09:42:20.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.796+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2806991962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 msgr2=0x55e52696a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:20.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.796+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 0x55e52696a900 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009990 tx=0x7fd5e0005770 comp rx=0 tx=0).stop 2024-05-27T09:42:20.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.796+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2806991962 shutdown_connections 2024-05-27T09:42:20.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.796+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e5269ce0f0 0x55e5269d0510 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.796+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e5269495c0 0x55e52696af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.796+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5267c9f20 0x55e52695bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.796+0000 7fd5daa00700 0 client.26540 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:20.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.796+0000 7fd60a71bb80 1 --2- 172.21.15.44:0/2806991962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e5268b5ad0 0x55e52696a900 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.796+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2806991962 >> 172.21.15.44:0/2806991962 conn(0x55e526863470 msgr2=0x55e526948c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:20.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.797+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2806991962 shutdown_connections 2024-05-27T09:42:20.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.797+0000 7fd60a71bb80 1 -- 172.21.15.44:0/2806991962 wait complete. 2024-05-27T09:42:20.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.797+0000 7fd60a71bb80 20 client.26540 trim_cache size 0 max 16384 2024-05-27T09:42:20.799 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:42:20.799 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/var/run/ceph/ops-77ef2bd1-1-mds.b.json" 2024-05-27T09:42:20.799 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:42:20.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.798+0000 7fd5fbc00700 1 -- 172.21.15.44:0/1656504596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5dc0b2a70 msgr2=0x7fd5dc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:20.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.798+0000 7fd5fbc00700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5dc0b2a70 0x7fd5dc0b4f80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd5f406c120 tx=0x7fd5e0005fb0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.798+0000 7fd5fbc00700 1 -- 172.21.15.44:0/1656504596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5f4109840 msgr2=0x7fd5f41a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:20.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.798+0000 7fd5fbc00700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5f4109840 0x7fd5f41a1830 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00ef30 tx=0x7fd5ec00c3a0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.799+0000 7fd5fbc00700 1 -- 172.21.15.44:0/1656504596 shutdown_connections 2024-05-27T09:42:20.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.799+0000 7fd5fbc00700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd5dc0b2a70 0x7fd5dc0b4f80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.799+0000 7fd5fbc00700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5f4109840 0x7fd5f41a1830 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.799+0000 7fd5fbc00700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f41045c0 0x7fd5f419d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.799+0000 7fd5fbc00700 1 --2- 172.21.15.44:0/1656504596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5f4108ef0 0x7fd5f419d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:20.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.799+0000 7fd5fbc00700 1 -- 172.21.15.44:0/1656504596 >> 172.21.15.44:0/1656504596 conn(0x7fd5f40ffe70 msgr2=0x7fd5f410a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:20.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.799+0000 7fd5fbc00700 1 -- 172.21.15.44:0/1656504596 shutdown_connections 2024-05-27T09:42:20.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:20.799+0000 7fd5fbc00700 1 -- 172.21.15.44:0/1656504596 wait complete. 2024-05-27T09:42:21.683 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-05-27T09:42:21.703 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.Zg1GVDCDLw 2024-05-27T09:42:21.703 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.c.json /tmp/tmp.Zg1GVDCDLw 2024-05-27T09:42:21.774 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.Zg1GVDCDLw 2024-05-27T09:42:21.901 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.Zg1GVDCDLw is 21KB 2024-05-27T09:42:21.949 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.Zg1GVDCDLw 2024-05-27T09:42:21.964 DEBUG:tasks.quiescer.fs.[cephfs]:read 21510B of ops from '/var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.c.json' on mds.2 (c) 2024-05-27T09:42:21.968 INFO:tasks.quiescer.fs.[cephfs]:Pulled 9 ops from rank 2 (c) into home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/ops-77ef2bd1-1-mds.c.json 2024-05-27T09:42:21.968 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -- /var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.c.json 2024-05-27T09:42:21.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:21 smithi044 ceph-mon[31618]: pgmap v219: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 9.0 KiB/s wr, 318 op/s 2024-05-27T09:42:22.024 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:21 smithi189 ceph-mon[36671]: pgmap v219: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 9.0 KiB/s wr, 318 op/s 2024-05-27T09:42:22.028 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-05-27T09:42:22.084 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.0rzUZhsI9S 2024-05-27T09:42:22.084 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.i.json /tmp/tmp.0rzUZhsI9S 2024-05-27T09:42:22.239 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.0rzUZhsI9S 2024-05-27T09:42:22.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:21 smithi050 ceph-mon[36371]: pgmap v219: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 9.0 KiB/s wr, 318 op/s 2024-05-27T09:42:22.323 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.0rzUZhsI9S is 16MB 2024-05-27T09:42:22.713 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.0rzUZhsI9S 2024-05-27T09:42:22.759 DEBUG:tasks.quiescer.fs.[cephfs]:read 16861483B of ops from '/var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.i.json' on mds.1 (i) 2024-05-27T09:42:23.195 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1984' 2024-05-27T09:42:23.243 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1985' 2024-05-27T09:42:23.291 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1986' 2024-05-27T09:42:23.338 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1987' 2024-05-27T09:42:23.386 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1988' 2024-05-27T09:42:23.425 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'client_request(client.25070:8315 create owner_uid=1000, owner_gid=1316 #0x10000000272/file1602 2024-05-27T09:41:12.326574+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-05-27T09:42:23.438 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1989' 2024-05-27T09:42:23.485 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1990' 2024-05-27T09:42:23.533 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1991' 2024-05-27T09:42:23.580 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1992' 2024-05-27T09:42:23.627 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1993' 2024-05-27T09:42:23.679 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'client_request(client.25070:8318 create owner_uid=1000, owner_gid=1316 #0x10000000273/file1604 2024-05-27T09:41:13.926550+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-05-27T09:42:23.679 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1994' 2024-05-27T09:42:23.774 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1964' 2024-05-27T09:42:23.821 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1965' 2024-05-27T09:42:23.870 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1966' 2024-05-27T09:42:23.921 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1967' 2024-05-27T09:42:23.969 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1968' 2024-05-27T09:42:23.982 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'client_request(client.25070:8321 create owner_uid=1000, owner_gid=1316 #0x10000000274/file1605 2024-05-27T09:41:16.583509+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-05-27T09:42:24.017 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1969' 2024-05-27T09:42:24.065 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1970' 2024-05-27T09:42:24.112 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1971' 2024-05-27T09:42:24.138 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'client_request(client.25070:8184 create owner_uid=1000, owner_gid=1316 #0x10000000271/file1254 2024-05-27T09:41:12.159577+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-05-27T09:42:24.164 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1972' 2024-05-27T09:42:24.212 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1973' 2024-05-27T09:42:24.260 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1974' 2024-05-27T09:42:24.306 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1975' 2024-05-27T09:42:24.351 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1976' 2024-05-27T09:42:24.402 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1977' 2024-05-27T09:42:24.450 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1978' 2024-05-27T09:42:24.497 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1979' 2024-05-27T09:42:24.544 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1980' 2024-05-27T09:42:24.591 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1981' 2024-05-27T09:42:24.642 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1982' 2024-05-27T09:42:24.689 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (i) for 77ef2bd1-1: 'peer_request:mds.0:1983' 2024-05-27T09:42:24.715 INFO:tasks.quiescer.fs.[cephfs]:Pulled 6111 ops from rank 1 (i) into home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/ops-77ef2bd1-1-mds.i.json 2024-05-27T09:42:24.716 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -- /var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.i.json 2024-05-27T09:42:24.717 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:23 smithi050 ceph-mon[36371]: pgmap v220: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 8.9 KiB/s wr, 316 op/s 2024-05-27T09:42:24.717 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:23 smithi189 ceph-mon[36671]: pgmap v220: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 8.9 KiB/s wr, 316 op/s 2024-05-27T09:42:24.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.192+0000 7fd30ba00700 1 -- 172.21.15.44:0/3710339308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304110b90 msgr2=0x7fd304112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:24.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.192+0000 7fd30ba00700 1 --2- 172.21.15.44:0/3710339308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304110b90 0x7fd304112ff0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009f90 tx=0x7fd2fc00b330 comp rx=0 tx=0).stop 2024-05-27T09:42:24.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.193+0000 7fd30ba00700 1 -- 172.21.15.44:0/3710339308 shutdown_connections 2024-05-27T09:42:24.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.193+0000 7fd30ba00700 1 --2- 172.21.15.44:0/3710339308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304110b90 0x7fd304112ff0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.193+0000 7fd30ba00700 1 --2- 172.21.15.44:0/3710339308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd304075d20 0x7fd304076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.193+0000 7fd30ba00700 1 --2- 172.21.15.44:0/3710339308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd304076670 0x7fd304110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.193+0000 7fd30ba00700 1 -- 172.21.15.44:0/3710339308 >> 172.21.15.44:0/3710339308 conn(0x7fd304100030 msgr2=0x7fd304102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.193+0000 7fd30ba00700 1 -- 172.21.15.44:0/3710339308 shutdown_connections 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.193+0000 7fd30ba00700 1 -- 172.21.15.44:0/3710339308 wait complete. 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.195+0000 7fd30ba00700 1 Processor -- start 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.195+0000 7fd30ba00700 1 -- start start 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.196+0000 7fd30ba00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304075d20 0x7fd30419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.196+0000 7fd30ba00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd304076670 0x7fd30419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.196+0000 7fd30ba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd304110b90 0x7fd3041a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.196+0000 7fd30ba00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041166c0 con 0x7fd304075d20 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.196+0000 7fd30ba00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd304116540 con 0x7fd304076670 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.196+0000 7fd30ba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd304116840 con 0x7fd304110b90 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.196+0000 7fd309000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304075d20 0x7fd30419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.196+0000 7fd309000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304075d20 0x7fd30419d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36876/0 (socket says 172.21.15.44:36876) 2024-05-27T09:42:24.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.196+0000 7fd309000700 1 -- 172.21.15.44:0/2640103627 learned_addr learned my addr 172.21.15.44:0/2640103627 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.196+0000 7fd303e00700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd304076670 0x7fd30419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.196+0000 7fd309a00700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd304110b90 0x7fd3041a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.197+0000 7fd309000700 1 -- 172.21.15.44:0/2640103627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd304110b90 msgr2=0x7fd3041a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.197+0000 7fd309000700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd304110b90 0x7fd3041a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.197+0000 7fd309000700 1 -- 172.21.15.44:0/2640103627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd304076670 msgr2=0x7fd30419d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.197+0000 7fd309000700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd304076670 0x7fd30419d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.197+0000 7fd309000700 1 -- 172.21.15.44:0/2640103627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3041a1ee0 con 0x7fd304075d20 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.197+0000 7fd303e00700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd304076670 0x7fd30419d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.197+0000 7fd309000700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304075d20 0x7fd30419d010 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800e9c0 tx=0x7fd2f800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.197+0000 7fd301600700 1 -- 172.21.15.44:0/2640103627 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8004a30 con 0x7fd304075d20 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.197+0000 7fd30ba00700 1 -- 172.21.15.44:0/2640103627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3041a21d0 con 0x7fd304075d20 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.197+0000 7fd301600700 1 -- 172.21.15.44:0/2640103627 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2f800cc70 con 0x7fd304075d20 2024-05-27T09:42:24.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.197+0000 7fd301600700 1 -- 172.21.15.44:0/2640103627 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80054a0 con 0x7fd304075d20 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.198+0000 7fd30ba00700 1 -- 172.21.15.44:0/2640103627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3041a9af0 con 0x7fd304075d20 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.201+0000 7fd301600700 1 -- 172.21.15.44:0/2640103627 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd2f8004bd0 con 0x7fd304075d20 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.202+0000 7fd31a5d1b80 1 Processor -- start 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.203+0000 7fd31a5d1b80 1 -- start start 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.203+0000 7fd301600700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2ec0b29f0 0x7fd2ec0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.203+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fdb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.203+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fdb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.203+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3fdbe70 0x5606b3fe8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.203+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5606b3f09e40 con 0x5606b3e5ce40 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.203+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5606b3ef99e0 con 0x5606b3fdbe70 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.203+0000 7fd301600700 1 -- 172.21.15.44:0/2640103627 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fd2f80e2730 con 0x7fd304075d20 2024-05-27T09:42:24.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.203+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5606b3efda20 con 0x5606b3f489f0 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.203+0000 7fd309a00700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2ec0b29f0 0x7fd2ec0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.203+0000 7fd303e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fdb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.204+0000 7fd303e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fdb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36882/0 (socket says 172.21.15.44:36882) 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.204+0000 7fd303e00700 1 -- 172.21.15.44:0/3793437464 learned_addr learned my addr 172.21.15.44:0/3793437464 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.204+0000 7fd309a00700 1 --2- 172.21.15.44:0/3793437464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3fdbe70 0x5606b3fe8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.204+0000 7fd309000700 1 --2- 172.21.15.44:0/3793437464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fdb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.204+0000 7fd303e00700 1 -- 172.21.15.44:0/3793437464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 msgr2=0x5606b3fdb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.204+0000 7fd303e00700 1 --2- 172.21.15.44:0/3793437464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fdb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.204+0000 7fd303e00700 1 -- 172.21.15.44:0/3793437464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3fdbe70 msgr2=0x5606b3fe8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.204+0000 7fd303e00700 1 --2- 172.21.15.44:0/3793437464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3fdbe70 0x5606b3fe8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.204+0000 7fd303e00700 1 -- 172.21.15.44:0/3793437464 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606b3ec2420 con 0x5606b3e5ce40 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.205+0000 7fd309a00700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2ec0b29f0 0x7fd2ec0b4f00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009550 tx=0x7fd2fc009300 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:24.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.205+0000 7fd303e00700 1 --2- 172.21.15.44:0/3793437464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fdb870 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00114f0 tx=0x7fd2f0011800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62bd7621fe39395b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.205+0000 7fd2f6000700 1 -- 172.21.15.44:0/3793437464 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00112f0 con 0x5606b3e5ce40 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.205+0000 7fd2f6000700 1 -- 172.21.15.44:0/3793437464 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2f00044e0 con 0x5606b3e5ce40 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.205+0000 7fd2f6000700 1 -- 172.21.15.44:0/3793437464 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f001a6e0 con 0x5606b3e5ce40 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.206+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/3793437464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 msgr2=0x5606b3fdb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.206+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/3793437464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fdb870 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00114f0 tx=0x7fd2f0011800 comp rx=0 tx=0).stop 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.206+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/3793437464 shutdown_connections 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.206+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/3793437464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3fdbe70 0x5606b3fe8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.206+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/3793437464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fdb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.206+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/3793437464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fdb870 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.206+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/3793437464 >> 172.21.15.44:0/3793437464 conn(0x5606b3ef6390 msgr2=0x5606b3fd74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.206+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/3793437464 shutdown_connections 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.206+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/3793437464 wait complete. 2024-05-27T09:42:24.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.207+0000 7fd31a5d1b80 1 -- start start 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.207+0000 7fd31a5d1b80 1 Processor -- start 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3e5ce40 0x5606b3fe5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fe5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3fdbe70 0x5606b3fe5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5606b3dfffa0 con 0x5606b3fdbe70 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5606b3e9a760 con 0x5606b3e5ce40 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5606b3fe8b80 con 0x5606b3f489f0 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd31a5d1b80 10 client.0 _ll_register_callbacks cb 0x5606b3ea3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd303e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fe5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd309a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3fdbe70 0x5606b3fe5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd303e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fe5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50260/0 (socket says 172.21.15.44:50260) 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd303e00700 1 -- 172.21.15.44:0/843538924 learned_addr learned my addr 172.21.15.44:0/843538924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:24.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.208+0000 7fd309000700 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3e5ce40 0x5606b3fe5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.209+0000 7fd309a00700 1 -- 172.21.15.44:0/843538924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 msgr2=0x5606b3fe5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.209+0000 7fd309a00700 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fe5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.209+0000 7fd309a00700 1 -- 172.21.15.44:0/843538924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3e5ce40 msgr2=0x5606b3fe5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.209+0000 7fd309a00700 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3e5ce40 0x5606b3fe5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.209+0000 7fd309a00700 1 -- 172.21.15.44:0/843538924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606b3e498d0 con 0x5606b3fdbe70 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.209+0000 7fd309000700 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3e5ce40 0x5606b3fe5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.209+0000 7fd309a00700 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3fdbe70 0x5606b3fe5f90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009790 tx=0x7fd2fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b3dccae7fc7726 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.209+0000 7fd2eaa00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.209+0000 7fd2eaa00700 1 -- 172.21.15.44:0/843538924 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc031060 con 0x5606b3fdbe70 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.210+0000 7fd2eaa00700 1 -- 172.21.15.44:0/843538924 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fd2fc003a00 con 0x5606b3fdbe70 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.210+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/843538924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2f00044e0 con 0x5606b3fdbe70 2024-05-27T09:42:24.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.210+0000 7fd2eaa00700 1 -- 172.21.15.44:0/843538924 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd2fc0050e0 con 0x5606b3fdbe70 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.210+0000 7fd31a5d1b80 10 client.16389 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.210+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/843538924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5606b3eca060 con 0x5606b3fdbe70 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.210+0000 7fd2eaa00700 1 -- 172.21.15.44:0/843538924 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2f00044e0 con 0x5606b3fdbe70 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.210+0000 7fd2eaa00700 1 -- 172.21.15.44:0/843538924 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fd2fc011070 con 0x5606b3fdbe70 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.211+0000 7fd31a5d1b80 10 client.16389 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.211+0000 7fd31a5d1b80 10 client.16389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.211+0000 7fd31a5d1b80 20 client.16389 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.211+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5606b3ff59e0 0x5606b3ff7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.211+0000 7fd31a5d1b80 4 client.16389 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.211+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/843538924 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606b3f0bd90 con 0x5606b3ff59e0 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.211+0000 7fd309000700 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5606b3ff59e0 0x5606b3ff7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.212+0000 7fd309000700 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5606b3ff59e0 0x5606b3ff7e20 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4ec8edcb97ed799 server_cookie=2978e6185ab1ba6c in_seq=0 out_seq=0 2024-05-27T09:42:24.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.212+0000 7fd2eaa00700 10 client.16389 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:24.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.212+0000 7fd2eaa00700 1 -- 172.21.15.44:0/843538924 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5606b3f0bd90 con 0x5606b3ff59e0 2024-05-27T09:42:24.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.212+0000 7fd2eaa00700 10 client.16389 handle_command_reply: tid=0 2024-05-27T09:42:24.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd31a5d1b80 1 client.16389 shutdown 2024-05-27T09:42:24.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/843538924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3fdbe70 msgr2=0x5606b3fe5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:24.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3fdbe70 0x5606b3fe5f90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009790 tx=0x7fd2fc00b330 comp rx=0 tx=0).stop 2024-05-27T09:42:24.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/843538924 shutdown_connections 2024-05-27T09:42:24.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5606b3ff59e0 0x5606b3ff7e20 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3fdbe70 0x5606b3fe5f90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3e5ce40 0x5606b3fe5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/843538924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fe5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/843538924 >> 172.21.15.44:0/843538924 conn(0x5606b3ef6390 msgr2=0x5606b3fdb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd2eaa00700 0 client.16389 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/843538924 shutdown_connections 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.213+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/843538924 wait complete. 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.214+0000 7fd31a5d1b80 20 client.16389 trim_cache size 0 max 16384 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.237+0000 7fd31a5d1b80 1 Processor -- start 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.237+0000 7fd31a5d1b80 1 -- start start 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.237+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3e5ce40 0x5606b3fe4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.237+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3f489f0 0x5606b3ff1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.237+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3fdbe70 0x5606b3fdf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.237+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041b3a00 con 0x5606b3fdbe70 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.237+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041b1260 con 0x5606b3f489f0 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.237+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041af0d0 con 0x5606b3e5ce40 2024-05-27T09:42:24.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.238+0000 7fd309000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3e5ce40 0x5606b3fe4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.238+0000 7fd309000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3e5ce40 0x5606b3fe4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50266/0 (socket says 172.21.15.44:50266) 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.238+0000 7fd309000700 1 -- 172.21.15.44:0/4121965399 learned_addr learned my addr 172.21.15.44:0/4121965399 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.238+0000 7fd309a00700 1 --2- 172.21.15.44:0/4121965399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3fdbe70 0x5606b3fdf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.238+0000 7fd303e00700 1 --2- 172.21.15.44:0/4121965399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3f489f0 0x5606b3ff1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.238+0000 7fd309000700 1 -- 172.21.15.44:0/4121965399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3f489f0 msgr2=0x5606b3ff1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.238+0000 7fd309000700 1 --2- 172.21.15.44:0/4121965399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3f489f0 0x5606b3ff1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.238+0000 7fd309000700 1 -- 172.21.15.44:0/4121965399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3fdbe70 msgr2=0x5606b3fdf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.238+0000 7fd309000700 1 --2- 172.21.15.44:0/4121965399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3fdbe70 0x5606b3fdf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.238+0000 7fd309000700 1 -- 172.21.15.44:0/4121965399 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606b3fdb880 con 0x5606b3e5ce40 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.238+0000 7fd309000700 1 --2- 172.21.15.44:0/4121965399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3e5ce40 0x5606b3fe4470 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80a96b0 tx=0x7fd2f8021000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5da609a1092265d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.239+0000 7fd2f5600700 1 -- 172.21.15.44:0/4121965399 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80a94c0 con 0x5606b3e5ce40 2024-05-27T09:42:24.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.239+0000 7fd2f5600700 1 -- 172.21.15.44:0/4121965399 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2f808e030 con 0x5606b3e5ce40 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.239+0000 7fd2f5600700 1 -- 172.21.15.44:0/4121965399 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80cc0b0 con 0x5606b3e5ce40 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.240+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/4121965399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3e5ce40 msgr2=0x5606b3fe4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.240+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/4121965399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3e5ce40 0x5606b3fe4470 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80a96b0 tx=0x7fd2f8021000 comp rx=0 tx=0).stop 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.240+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/4121965399 shutdown_connections 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.240+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/4121965399 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3fdbe70 0x5606b3fdf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.240+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/4121965399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3e5ce40 0x5606b3fe4470 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.240+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/4121965399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3f489f0 0x5606b3ff1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.240+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/4121965399 >> 172.21.15.44:0/4121965399 conn(0x5606b3ef6390 msgr2=0x5606b3fdeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.240+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/4121965399 shutdown_connections 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.240+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/4121965399 wait complete. 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.240+0000 7fd31a5d1b80 1 -- start start 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.241+0000 7fd31a5d1b80 1 Processor -- start 2024-05-27T09:42:24.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.241+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fe9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.241+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fea0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.241+0000 7fd31a5d1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3fdbe70 0x5606b3fea620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.241+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041af720 con 0x5606b3e5ce40 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.241+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041b0b90 con 0x5606b3fdbe70 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.241+0000 7fd31a5d1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041afdf0 con 0x5606b3f489f0 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.241+0000 7fd31a5d1b80 10 client.0 _ll_register_callbacks cb 0x5606b3ea3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.242+0000 7fd309000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fe9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.242+0000 7fd309000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fe9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36910/0 (socket says 172.21.15.44:36910) 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.242+0000 7fd309000700 1 -- 172.21.15.44:0/1579427239 learned_addr learned my addr 172.21.15.44:0/1579427239 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.242+0000 7fd303e00700 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fea0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.242+0000 7fd309a00700 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3fdbe70 0x5606b3fea620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.242+0000 7fd303e00700 1 -- 172.21.15.44:0/1579427239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3fdbe70 msgr2=0x5606b3fea620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.242+0000 7fd303e00700 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3fdbe70 0x5606b3fea620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.242+0000 7fd303e00700 1 -- 172.21.15.44:0/1579427239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 msgr2=0x5606b3fe9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.242+0000 7fd303e00700 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fe9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.242+0000 7fd303e00700 1 -- 172.21.15.44:0/1579427239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606b405f4f0 con 0x5606b3f489f0 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.242+0000 7fd309000700 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fe9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.243+0000 7fd303e00700 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fea0e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009790 tx=0x7fd2f0005440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec236adce1efbf3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.243+0000 7fd2eaa00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.243+0000 7fd2eaa00700 1 -- 172.21.15.44:0/1579427239 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0003890 con 0x5606b3f489f0 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.243+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/1579427239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2f808e030 con 0x5606b3f489f0 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.243+0000 7fd2eaa00700 1 -- 172.21.15.44:0/1579427239 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fd2f0003a30 con 0x5606b3f489f0 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.243+0000 7fd2eaa00700 1 -- 172.21.15.44:0/1579427239 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd2f0035050 con 0x5606b3f489f0 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.243+0000 7fd31a5d1b80 10 client.26554 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.243+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/1579427239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5606b4060730 con 0x5606b3f489f0 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.244+0000 7fd2eaa00700 1 -- 172.21.15.44:0/1579427239 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2f808e030 con 0x5606b3f489f0 2024-05-27T09:42:24.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.244+0000 7fd2eaa00700 1 -- 172.21.15.44:0/1579427239 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fd2f0027440 con 0x5606b3f489f0 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.244+0000 7fd31a5d1b80 10 client.26554 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.244+0000 7fd31a5d1b80 10 client.26554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.244+0000 7fd31a5d1b80 20 client.26554 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.244+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5606b4060c50 0x5606b4063070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.244+0000 7fd31a5d1b80 4 client.26554 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.244+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/1579427239 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5606b3f05570 con 0x5606b4060c50 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.244+0000 7fd309000700 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5606b4060c50 0x5606b4063070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.245+0000 7fd309000700 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5606b4060c50 0x5606b4063070 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f701b18f4480f50 server_cookie=8da4dd05cec45292 in_seq=0 out_seq=0 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.245+0000 7fd2eaa00700 10 client.26554 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.245+0000 7fd2eaa00700 1 -- 172.21.15.44:0/1579427239 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5606b3f05570 con 0x5606b4060c50 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.245+0000 7fd2eaa00700 10 client.26554 handle_command_reply: tid=0 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.245+0000 7fd31a5d1b80 1 client.26554 shutdown 2024-05-27T09:42:24.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.245+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/1579427239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 msgr2=0x5606b3fea0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.245+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fea0e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009790 tx=0x7fd2f0005440 comp rx=0 tx=0).stop 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.246+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/1579427239 shutdown_connections 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.246+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5606b4060c50 0x5606b4063070 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.246+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606b3fdbe70 0x5606b3fea620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.246+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5606b3e5ce40 0x5606b3fe9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.246+0000 7fd2eaa00700 0 client.26554 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.246+0000 7fd31a5d1b80 1 --2- 172.21.15.44:0/1579427239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606b3f489f0 0x5606b3fea0e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.246+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/1579427239 >> 172.21.15.44:0/1579427239 conn(0x5606b3ef6390 msgr2=0x5606b3fe5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.246+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/1579427239 shutdown_connections 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.246+0000 7fd31a5d1b80 1 -- 172.21.15.44:0/1579427239 wait complete. 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.246+0000 7fd31a5d1b80 20 client.26554 trim_cache size 0 max 16384 2024-05-27T09:42:24.743 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:42:24.744 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:42:24.744 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:42:24.744 INFO:teuthology.orchestra.run.smithi044.stdout: "7e532cb2-6cd4-4494-953d-0430f62f7964": { 2024-05-27T09:42:24.744 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:42:24.744 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "7e532cb2-6cd4-4494-953d-0430f62f7964", 2024-05-27T09:42:24.744 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:42:24.744 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:24.744 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:24.744 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:42:24.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 -- 172.21.15.44:0/2640103627 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2ec0b29f0 msgr2=0x7fd2ec0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:24.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2ec0b29f0 0x7fd2ec0b4f00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009550 tx=0x7fd2fc009300 comp rx=0 tx=0).stop 2024-05-27T09:42:24.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 -- 172.21.15.44:0/2640103627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304075d20 msgr2=0x7fd30419d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:24.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304075d20 0x7fd30419d010 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800e9c0 tx=0x7fd2f800ecd0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 -- 172.21.15.44:0/2640103627 shutdown_connections 2024-05-27T09:42:24.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd2ec0b29f0 0x7fd2ec0b4f00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd304110b90 0x7fd3041a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304075d20 0x7fd30419d010 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 --2- 172.21.15.44:0/2640103627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd304076670 0x7fd30419d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:24.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 -- 172.21.15.44:0/2640103627 >> 172.21.15.44:0/2640103627 conn(0x7fd304100030 msgr2=0x7fd304100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:24.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 -- 172.21.15.44:0/2640103627 shutdown_connections 2024-05-27T09:42:24.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:23.247+0000 7fd30ba00700 1 -- 172.21.15.44:0/2640103627 wait complete. 2024-05-27T09:42:24.746 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:23 smithi044 ceph-mon[31618]: pgmap v220: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 8.9 KiB/s wr, 316 op/s 2024-05-27T09:42:24.747 INFO:tasks.cephfs.filesystem:scrub status for tag:7e532cb2-6cd4-4494-953d-0430f62f7964 - {'path': '/', 'tag': '7e532cb2-6cd4-4494-953d-0430f62f7964', 'options': 'recursive,force'} 2024-05-27T09:42:24.748 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-05-27T09:42:24.767 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.g2FoefVyJy 2024-05-27T09:42:24.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.b.json /tmp/tmp.g2FoefVyJy 2024-05-27T09:42:24.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.g2FoefVyJy 2024-05-27T09:42:24.975 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.g2FoefVyJy is 5MB 2024-05-27T09:42:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.089+0000 7f05daa00640 1 -- 172.21.15.44:0/1456291462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05d410c180 msgr2=0x7f05d4105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.089+0000 7f05daa00640 1 --2- 172.21.15.44:0/1456291462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05d410c180 0x7f05d4105dc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f05c0009860 tx=0x7f05c002a340 comp rx=0 tx=0).stop 2024-05-27T09:42:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.089+0000 7f05daa00640 1 -- 172.21.15.44:0/1456291462 shutdown_connections 2024-05-27T09:42:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.089+0000 7f05daa00640 1 --2- 172.21.15.44:0/1456291462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d4106430 0x7f05d4112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.089+0000 7f05daa00640 1 --2- 172.21.15.44:0/1456291462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05d410c180 0x7f05d4105dc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.089+0000 7f05daa00640 1 --2- 172.21.15.44:0/1456291462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d410bce0 0x7f05d4105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.089+0000 7f05daa00640 1 -- 172.21.15.44:0/1456291462 >> 172.21.15.44:0/1456291462 conn(0x7f05d4101380 msgr2=0x7f05d41037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.090+0000 7f05daa00640 1 -- 172.21.15.44:0/1456291462 shutdown_connections 2024-05-27T09:42:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.090+0000 7f05daa00640 1 -- 172.21.15.44:0/1456291462 wait complete. 2024-05-27T09:42:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.091+0000 7f05daa00640 1 Processor -- start 2024-05-27T09:42:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.091+0000 7f05daa00640 1 -- start start 2024-05-27T09:42:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.091+0000 7f05daa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d4106430 0x7f05d41a2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.091+0000 7f05daa00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d410bce0 0x7f05d41a2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.091+0000 7f05daa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05d410c180 0x7f05d419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05daa00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f05d4115ed0 con 0x7f05d410c180 2024-05-27T09:42:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05daa00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f05d4115d50 con 0x7f05d410bce0 2024-05-27T09:42:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05daa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05d4116050 con 0x7f05d4106430 2024-05-27T09:42:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05d8c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d410bce0 0x7f05d41a2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05da000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05d410c180 0x7f05d419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05d8c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d410bce0 0x7f05d41a2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52944/0 (socket says 172.21.15.44:52944) 2024-05-27T09:42:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05d8c00640 1 -- 172.21.15.44:0/3861890214 learned_addr learned my addr 172.21.15.44:0/3861890214 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05d9600640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d4106430 0x7f05d41a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05d8c00640 1 -- 172.21.15.44:0/3861890214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d4106430 msgr2=0x7f05d41a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:25.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05d8c00640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d4106430 0x7f05d41a2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05d8c00640 1 -- 172.21.15.44:0/3861890214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05d410c180 msgr2=0x7f05d419c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:25.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05d8c00640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05d410c180 0x7f05d419c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.092+0000 7f05d8c00640 1 -- 172.21.15.44:0/3861890214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d419cec0 con 0x7f05d410bce0 2024-05-27T09:42:25.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.093+0000 7f05d8c00640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d410bce0 0x7f05d41a2b50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f05c0035040 tx=0x7f05c0032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:25.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.093+0000 7f05d2000640 1 -- 172.21.15.44:0/3861890214 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c0037040 con 0x7f05d410bce0 2024-05-27T09:42:25.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.093+0000 7f05daa00640 1 -- 172.21.15.44:0/3861890214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d419d0f0 con 0x7f05d410bce0 2024-05-27T09:42:25.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.093+0000 7f05d2000640 1 -- 172.21.15.44:0/3861890214 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c0030c10 con 0x7f05d410bce0 2024-05-27T09:42:25.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.093+0000 7f05d2000640 1 -- 172.21.15.44:0/3861890214 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c00035f0 con 0x7f05d410bce0 2024-05-27T09:42:25.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.093+0000 7f05daa00640 1 -- 172.21.15.44:0/3861890214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d419d580 con 0x7f05d410bce0 2024-05-27T09:42:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.097+0000 7f05e2f9a740 1 Processor -- start 2024-05-27T09:42:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.097+0000 7f05d2000640 1 -- 172.21.15.44:0/3861890214 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f05c0043020 con 0x7f05d410bce0 2024-05-27T09:42:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.097+0000 7f05e2f9a740 1 -- start start 2024-05-27T09:42:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.097+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d89765da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.097+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8972922f0 0x55d89765dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.097+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d89729a4d0 0x55d89765e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.097+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d8975ceb90 con 0x55d8972bb1b0 2024-05-27T09:42:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.097+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d8975be730 con 0x55d89729a4d0 2024-05-27T09:42:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.097+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d8975c2770 con 0x55d8972922f0 2024-05-27T09:42:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.098+0000 7f05da000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d89765da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.098+0000 7f05da000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d89765da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36926/0 (socket says 172.21.15.44:36926) 2024-05-27T09:42:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.098+0000 7f05da000640 1 -- 172.21.15.44:0/4137923478 learned_addr learned my addr 172.21.15.44:0/4137923478 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.098+0000 7f05d9600640 1 --2- 172.21.15.44:0/4137923478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8972922f0 0x55d89765dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.098+0000 7f05d8c00640 1 --2- 172.21.15.44:0/4137923478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d89729a4d0 0x55d89765e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.098+0000 7f05da000640 1 -- 172.21.15.44:0/4137923478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8972922f0 msgr2=0x55d89765dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.098+0000 7f05da000640 1 --2- 172.21.15.44:0/4137923478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8972922f0 0x55d89765dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.098+0000 7f05da000640 1 -- 172.21.15.44:0/4137923478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d89729a4d0 msgr2=0x55d89765e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.098+0000 7f05da000640 1 --2- 172.21.15.44:0/4137923478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d89729a4d0 0x55d89765e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.098+0000 7f05da000640 1 -- 172.21.15.44:0/4137923478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8975578b0 con 0x55d8972bb1b0 2024-05-27T09:42:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.098+0000 7f05d8c00640 1 --2- 172.21.15.44:0/4137923478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d89729a4d0 0x55d89765e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:42:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.099+0000 7f05da000640 1 --2- 172.21.15.44:0/4137923478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d89765da60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009fd0 tx=0x7f05cc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbfbeb99c064b688 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.099+0000 7f05c6600640 1 -- 172.21.15.44:0/4137923478 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc004d20 con 0x55d8972bb1b0 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.099+0000 7f05c6600640 1 -- 172.21.15.44:0/4137923478 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05cc004050 con 0x55d8972bb1b0 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.099+0000 7f05c6600640 1 -- 172.21.15.44:0/4137923478 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc005280 con 0x55d8972bb1b0 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.099+0000 7f05d2000640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05bc0b2950 0x7f05bc0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.099+0000 7f05d2000640 1 -- 172.21.15.44:0/3861890214 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f05c0101820 con 0x7f05d410bce0 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.099+0000 7f05d9600640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05bc0b2950 0x7f05bc0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.100+0000 7f05e2f9a740 1 -- 172.21.15.44:0/4137923478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 msgr2=0x55d89765da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.100+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/4137923478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d89765da60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009fd0 tx=0x7f05cc0118d0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.100+0000 7f05e2f9a740 1 -- 172.21.15.44:0/4137923478 shutdown_connections 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.100+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/4137923478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d89729a4d0 0x55d89765e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.100+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/4137923478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8972922f0 0x55d89765dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.100+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/4137923478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d89765da60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.100+0000 7f05e2f9a740 1 -- 172.21.15.44:0/4137923478 >> 172.21.15.44:0/4137923478 conn(0x55d8972af300 msgr2=0x55d89765a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.100+0000 7f05e2f9a740 1 -- 172.21.15.44:0/4137923478 shutdown_connections 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.100+0000 7f05d9600640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05bc0b2950 0x7f05bc0b4e60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55d89765ee10 tx=0x7f05c8009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.100+0000 7f05e2f9a740 1 -- 172.21.15.44:0/4137923478 wait complete. 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.101+0000 7f05e2f9a740 1 -- start start 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.101+0000 7f05e2f9a740 1 Processor -- start 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972922f0 0x55d897667c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d89729a4d0 0x55d897668190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972bb1b0 0x55d897673f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d8972a4790 con 0x55d8972922f0 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d897566bc0 con 0x55d8972bb1b0 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d89766ab90 con 0x55d89729a4d0 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05e2f9a740 10 client.0 _ll_register_callbacks cb 0x55d897544400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05d9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972922f0 0x55d897667c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05d9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972922f0 0x55d897667c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36934/0 (socket says 172.21.15.44:36934) 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05d9600640 1 -- 172.21.15.44:0/1123875327 learned_addr learned my addr 172.21.15.44:0/1123875327 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05d8c00640 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d89729a4d0 0x55d897668190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.102+0000 7f05da000640 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972bb1b0 0x55d897673f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.103+0000 7f05d9600640 1 -- 172.21.15.44:0/1123875327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d89729a4d0 msgr2=0x55d897668190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.103+0000 7f05d9600640 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d89729a4d0 0x55d897668190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.103+0000 7f05d9600640 1 -- 172.21.15.44:0/1123875327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972bb1b0 msgr2=0x55d897673f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.103+0000 7f05d9600640 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972bb1b0 0x55d897673f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.103+0000 7f05d9600640 1 -- 172.21.15.44:0/1123875327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8972d3210 con 0x55d8972922f0 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.103+0000 7f05d9600640 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972922f0 0x55d897667c50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009b60 tx=0x7f05c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a71e929af6d92 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.103+0000 7f05baa00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:42:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.103+0000 7f05baa00640 1 -- 172.21.15.44:0/1123875327 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c802d060 con 0x55d8972922f0 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.103+0000 7f05baa00640 1 -- 172.21.15.44:0/1123875327 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f05c8018030 con 0x55d8972922f0 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.103+0000 7f05e2f9a740 1 -- 172.21.15.44:0/1123875327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05cc004050 con 0x55d8972922f0 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.104+0000 7f05baa00640 1 -- 172.21.15.44:0/1123875327 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f05c8028420 con 0x55d8972922f0 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.104+0000 7f05e2f9a740 10 client.16407 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.104+0000 7f05e2f9a740 1 -- 172.21.15.44:0/1123875327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d89739f620 con 0x55d8972922f0 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.104+0000 7f05baa00640 1 -- 172.21.15.44:0/1123875327 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05cc004050 con 0x55d8972922f0 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.104+0000 7f05baa00640 1 -- 172.21.15.44:0/1123875327 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f05c80303b0 con 0x55d8972922f0 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.104+0000 7f05e2f9a740 10 client.16407 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.104+0000 7f05e2f9a740 10 client.16407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.104+0000 7f05e2f9a740 20 client.16407 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.104+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d897279d80 0x55d89767a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.104+0000 7f05e2f9a740 4 client.16407 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.104+0000 7f05e2f9a740 1 -- 172.21.15.44:0/1123875327 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8975d0ae0 con 0x55d897279d80 2024-05-27T09:42:25.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.105+0000 7f05d8c00640 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d897279d80 0x55d89767a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.105+0000 7f05d8c00640 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d897279d80 0x55d89767a010 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67bf0d0ec0e9e45a server_cookie=921325ca85e1ed6a in_seq=0 out_seq=0 2024-05-27T09:42:25.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.105+0000 7f05baa00640 10 client.16407 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05baa00640 1 -- 172.21.15.44:0/1123875327 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d8975d0ae0 con 0x55d897279d80 2024-05-27T09:42:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05baa00640 10 client.16407 handle_command_reply: tid=0 2024-05-27T09:42:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05e2f9a740 1 client.16407 shutdown 2024-05-27T09:42:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05e2f9a740 1 -- 172.21.15.44:0/1123875327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972922f0 msgr2=0x55d897667c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972922f0 0x55d897667c50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009b60 tx=0x7f05c8004730 comp rx=0 tx=0).stop 2024-05-27T09:42:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05e2f9a740 1 -- 172.21.15.44:0/1123875327 shutdown_connections 2024-05-27T09:42:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d897279d80 0x55d89767a010 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972bb1b0 0x55d897673f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d89729a4d0 0x55d897668190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/1123875327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972922f0 0x55d897667c50 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05baa00640 0 client.16407 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.106+0000 7f05e2f9a740 1 -- 172.21.15.44:0/1123875327 >> 172.21.15.44:0/1123875327 conn(0x55d8972af300 msgr2=0x55d89765d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.107+0000 7f05e2f9a740 1 -- 172.21.15.44:0/1123875327 shutdown_connections 2024-05-27T09:42:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.107+0000 7f05e2f9a740 1 -- 172.21.15.44:0/1123875327 wait complete. 2024-05-27T09:42:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.107+0000 7f05e2f9a740 20 client.16407 trim_cache size 0 max 16384 2024-05-27T09:42:25.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.125+0000 7f05e2f9a740 1 Processor -- start 2024-05-27T09:42:25.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.125+0000 7f05e2f9a740 1 -- start start 2024-05-27T09:42:25.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.126+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972922f0 0x55d8976633e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.126+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d89729a4d0 0x55d897673c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.126+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d8976741d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.126+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41b4410 con 0x55d8972bb1b0 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.126+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41b1cb0 con 0x55d8972922f0 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.126+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41afb20 con 0x55d89729a4d0 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.126+0000 7f05da000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d8976741d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.126+0000 7f05da000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d8976741d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36940/0 (socket says 172.21.15.44:36940) 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.126+0000 7f05da000640 1 -- 172.21.15.44:0/2787578438 learned_addr learned my addr 172.21.15.44:0/2787578438 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.126+0000 7f05d8c00640 1 --2- 172.21.15.44:0/2787578438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d89729a4d0 0x55d897673c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.126+0000 7f05d9600640 1 --2- 172.21.15.44:0/2787578438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972922f0 0x55d8976633e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.127+0000 7f05da000640 1 -- 172.21.15.44:0/2787578438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d89729a4d0 msgr2=0x55d897673c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.127+0000 7f05da000640 1 --2- 172.21.15.44:0/2787578438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d89729a4d0 0x55d897673c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.127+0000 7f05da000640 1 -- 172.21.15.44:0/2787578438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972922f0 msgr2=0x55d8976633e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:25.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.127+0000 7f05da000640 1 --2- 172.21.15.44:0/2787578438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972922f0 0x55d8976633e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.127+0000 7f05da000640 1 -- 172.21.15.44:0/2787578438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d89765ee40 con 0x55d8972bb1b0 2024-05-27T09:42:25.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.127+0000 7f05d8c00640 1 --2- 172.21.15.44:0/2787578438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d89729a4d0 0x55d897673c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:25.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.127+0000 7f05d9600640 1 --2- 172.21.15.44:0/2787578438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972922f0 0x55d8976633e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:42:25.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.127+0000 7f05da000640 1 --2- 172.21.15.44:0/2787578438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d8976741d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f05cc0099b0 tx=0x7f05cc009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772555668b7fe596 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.127+0000 7f05c5c00640 1 -- 172.21.15.44:0/2787578438 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc003bf0 con 0x55d8972bb1b0 2024-05-27T09:42:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.127+0000 7f05c5c00640 1 -- 172.21.15.44:0/2787578438 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05cc003d90 con 0x55d8972bb1b0 2024-05-27T09:42:25.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.127+0000 7f05c5c00640 1 -- 172.21.15.44:0/2787578438 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc005020 con 0x55d8972bb1b0 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.128+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2787578438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 msgr2=0x55d8976741d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.128+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/2787578438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d8976741d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f05cc0099b0 tx=0x7f05cc009d70 comp rx=0 tx=0).stop 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.128+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2787578438 shutdown_connections 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.128+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/2787578438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d8972bb1b0 0x55d8976741d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.129+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/2787578438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d89729a4d0 0x55d897673c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.129+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/2787578438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972922f0 0x55d8976633e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.129+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2787578438 >> 172.21.15.44:0/2787578438 conn(0x55d8972af300 msgr2=0x55d897667f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.129+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2787578438 shutdown_connections 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.129+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2787578438 wait complete. 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.129+0000 7f05e2f9a740 1 -- start start 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.129+0000 7f05e2f9a740 1 Processor -- start 2024-05-27T09:42:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8972922f0 0x55d8976daaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d89729a4d0 0x55d8976dafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05e2f9a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972bb1b0 0x55d89766b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41b0170 con 0x55d89729a4d0 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41b15e0 con 0x55d8972bb1b0 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05e2f9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41b0840 con 0x55d8972922f0 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05e2f9a740 10 client.0 _ll_register_callbacks cb 0x7f05c8009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05d9600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8972922f0 0x55d8976daaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05da000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972bb1b0 0x55d89766b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05d8c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d89729a4d0 0x55d8976dafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05d9600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8972922f0 0x55d8976daaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:50316/0 (socket says 172.21.15.44:50316) 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.130+0000 7f05d9600640 1 -- 172.21.15.44:0/2718705558 learned_addr learned my addr 172.21.15.44:0/2718705558 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.131+0000 7f05da000640 1 -- 172.21.15.44:0/2718705558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8972922f0 msgr2=0x55d8976daaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.131+0000 7f05da000640 1 --2- 172.21.15.44:0/2718705558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8972922f0 0x55d8976daaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.131+0000 7f05da000640 1 -- 172.21.15.44:0/2718705558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d89729a4d0 msgr2=0x55d8976dafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.131+0000 7f05da000640 1 --2- 172.21.15.44:0/2718705558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d89729a4d0 0x55d8976dafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.131+0000 7f05da000640 1 -- 172.21.15.44:0/2718705558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d897658610 con 0x55d8972bb1b0 2024-05-27T09:42:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.131+0000 7f05da000640 1 --2- 172.21.15.44:0/2718705558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972bb1b0 0x55d89766b440 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f05cc006010 tx=0x7f05cc004170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f672b0156d45b5a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.131+0000 7f05baa00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:25.135 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.g2FoefVyJy 2024-05-27T09:42:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.131+0000 7f05baa00640 1 -- 172.21.15.44:0/2718705558 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc004be0 con 0x55d8972bb1b0 2024-05-27T09:42:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.131+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2718705558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05cc003d90 con 0x55d8972bb1b0 2024-05-27T09:42:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.132+0000 7f05baa00640 1 -- 172.21.15.44:0/2718705558 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f05cc004d80 con 0x55d8972bb1b0 2024-05-27T09:42:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.132+0000 7f05baa00640 1 -- 172.21.15.44:0/2718705558 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f05cc005020 con 0x55d8972bb1b0 2024-05-27T09:42:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.132+0000 7f05e2f9a740 10 client.26576 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.132+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2718705558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d8976e1f10 con 0x55d8972bb1b0 2024-05-27T09:42:25.136 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.132+0000 7f05baa00640 1 -- 172.21.15.44:0/2718705558 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05cc003d90 con 0x55d8972bb1b0 2024-05-27T09:42:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.133+0000 7f05baa00640 1 -- 172.21.15.44:0/2718705558 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f05cc00f5e0 con 0x55d8972bb1b0 2024-05-27T09:42:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.133+0000 7f05e2f9a740 10 client.26576 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.133+0000 7f05e2f9a740 10 client.26576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.133+0000 7f05e2f9a740 20 client.26576 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.133+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/2718705558 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d89724f1c0 0x55d8976e4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.133+0000 7f05e2f9a740 4 client.26576 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:42:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.133+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2718705558 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d8975ca2c0 con 0x55d89724f1c0 2024-05-27T09:42:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.133+0000 7f05d9600640 1 --2- 172.21.15.44:0/2718705558 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d89724f1c0 0x55d8976e4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.133+0000 7f05d9600640 1 --2- 172.21.15.44:0/2718705558 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d89724f1c0 0x55d8976e4400 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2e5ce00cedad3fc server_cookie=e4436c937c10996a in_seq=0 out_seq=0 2024-05-27T09:42:25.137 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.133+0000 7f05baa00640 10 client.26576 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.139+0000 7f05baa00640 1 -- 172.21.15.44:0/2718705558 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55d8975ca2c0 con 0x55d89724f1c0 2024-05-27T09:42:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.139+0000 7f05baa00640 10 client.26576 handle_command_reply: tid=0 2024-05-27T09:42:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.139+0000 7f05e2f9a740 1 client.26576 shutdown 2024-05-27T09:42:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.139+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2718705558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972bb1b0 msgr2=0x55d89766b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.139+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/2718705558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972bb1b0 0x55d89766b440 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f05cc006010 tx=0x7f05cc004170 comp rx=0 tx=0).stop 2024-05-27T09:42:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.140+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2718705558 shutdown_connections 2024-05-27T09:42:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.140+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/2718705558 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d89724f1c0 0x55d8976e4400 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.140+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/2718705558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8972bb1b0 0x55d89766b440 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.140+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/2718705558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d89729a4d0 0x55d8976dafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.140+0000 7f05e2f9a740 1 --2- 172.21.15.44:0/2718705558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8972922f0 0x55d8976daaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.140+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2718705558 >> 172.21.15.44:0/2718705558 conn(0x55d8972af300 msgr2=0x55d897659240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.140+0000 7f05baa00640 0 client.26576 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.140+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2718705558 shutdown_connections 2024-05-27T09:42:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.140+0000 7f05e2f9a740 1 -- 172.21.15.44:0/2718705558 wait complete. 2024-05-27T09:42:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.140+0000 7f05e2f9a740 20 client.26576 trim_cache size 0 max 16384 2024-05-27T09:42:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:42:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:42:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:42:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 -- 172.21.15.44:0/3861890214 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05bc0b2950 msgr2=0x7f05bc0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05bc0b2950 0x7f05bc0b4e60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55d89765ee10 tx=0x7f05c8009260 comp rx=0 tx=0).stop 2024-05-27T09:42:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 -- 172.21.15.44:0/3861890214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d410bce0 msgr2=0x7f05d41a2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d410bce0 0x7f05d41a2b50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f05c0035040 tx=0x7f05c0032d50 comp rx=0 tx=0).stop 2024-05-27T09:42:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 -- 172.21.15.44:0/3861890214 shutdown_connections 2024-05-27T09:42:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05bc0b2950 0x7f05bc0b4e60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05d410c180 0x7f05d419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d410bce0 0x7f05d41a2b50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 --2- 172.21.15.44:0/3861890214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d4106430 0x7f05d41a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:25.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 -- 172.21.15.44:0/3861890214 >> 172.21.15.44:0/3861890214 conn(0x7f05d4101380 msgr2=0x7f05d41017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:25.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 -- 172.21.15.44:0/3861890214 shutdown_connections 2024-05-27T09:42:25.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:25.141+0000 7f05daa00640 1 -- 172.21.15.44:0/3861890214 wait complete. 2024-05-27T09:42:25.162 DEBUG:tasks.quiescer.fs.[cephfs]:read 5179752B of ops from '/var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.b.json' on mds.0 (b) 2024-05-27T09:42:25.427 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.2:32' 2024-05-27T09:42:25.428 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.1:68' 2024-05-27T09:42:25.442 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.2:33' 2024-05-27T09:42:25.458 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.2:34' 2024-05-27T09:42:25.488 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.1:59' 2024-05-27T09:42:25.532 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.1:60' 2024-05-27T09:42:25.578 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.1:61' 2024-05-27T09:42:25.651 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.1:62' 2024-05-27T09:42:25.666 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.1:63' 2024-05-27T09:42:25.730 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.1:64' 2024-05-27T09:42:25.820 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.2:27' 2024-05-27T09:42:25.835 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.1:65' 2024-05-27T09:42:25.850 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.2:28' 2024-05-27T09:42:25.865 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.2:29' 2024-05-27T09:42:25.865 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'client_request(client.25070:8295 create owner_uid=1000, owner_gid=1316 #0x10000000270/file1877 2024-05-27T09:41:12.292575+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-05-27T09:42:25.880 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.2:30' 2024-05-27T09:42:25.880 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.1:66' 2024-05-27T09:42:25.894 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.2:31' 2024-05-27T09:42:25.894 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for 77ef2bd1-1: 'peer_request:mds.1:67' 2024-05-27T09:42:25.911 INFO:tasks.quiescer.fs.[cephfs]:Pulled 1943 ops from rank 0 (b) into home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/ops-77ef2bd1-1-mds.b.json 2024-05-27T09:42:25.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -- /var/run/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ops-77ef2bd1-1-mds.b.json 2024-05-27T09:42:25.952 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:42:26.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:25 smithi189 ceph-mon[36671]: pgmap v221: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 8.9 KiB/s wr, 316 op/s 2024-05-27T09:42:26.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:25 smithi044 ceph-mon[31618]: pgmap v221: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 8.9 KiB/s wr, 316 op/s 2024-05-27T09:42:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:25 smithi050 ceph-mon[36371]: pgmap v221: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 8.9 KiB/s wr, 316 op/s 2024-05-27T09:42:26.425 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:26.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:26 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:42:26.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:26 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:42:27.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:26 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:42:27.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:26 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:42:27.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:26 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:42:27.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:26 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:42:28.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:27 smithi189 ceph-mon[36671]: pgmap v222: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 7.7 KiB/s wr, 314 op/s 2024-05-27T09:42:28.151 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:27 smithi044 ceph-mon[31618]: pgmap v222: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 7.7 KiB/s wr, 314 op/s 2024-05-27T09:42:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:27 smithi050 ceph-mon[36371]: pgmap v222: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 7.7 KiB/s wr, 314 op/s 2024-05-27T09:42:28.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.767+0000 7fb6e0800700 1 -- 172.21.15.44:0/667655923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6d81045e0 msgr2=0x7fb6d81049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:28.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.767+0000 7fb6e0800700 1 --2- 172.21.15.44:0/667655923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6d81045e0 0x7fb6d81049f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc009990 tx=0x7fb6cc009ca0 comp rx=0 tx=0).stop 2024-05-27T09:42:28.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.768+0000 7fb6e0800700 1 -- 172.21.15.44:0/667655923 shutdown_connections 2024-05-27T09:42:28.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.768+0000 7fb6e0800700 1 --2- 172.21.15.44:0/667655923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6d8109860 0x7fb6d81111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:28.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.768+0000 7fb6e0800700 1 --2- 172.21.15.44:0/667655923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6d81045e0 0x7fb6d81049f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:28.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.768+0000 7fb6e0800700 1 --2- 172.21.15.44:0/667655923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d8108f10 0x7fb6d8109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:28.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.768+0000 7fb6e0800700 1 -- 172.21.15.44:0/667655923 >> 172.21.15.44:0/667655923 conn(0x7fb6d80ffeb0 msgr2=0x7fb6d8102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:28.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.768+0000 7fb6e0800700 1 -- 172.21.15.44:0/667655923 shutdown_connections 2024-05-27T09:42:28.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.768+0000 7fb6e0800700 1 -- 172.21.15.44:0/667655923 wait complete. 2024-05-27T09:42:28.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.769+0000 7fb6e0800700 1 Processor -- start 2024-05-27T09:42:28.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.770+0000 7fb6e0800700 1 -- start start 2024-05-27T09:42:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.770+0000 7fb6e0800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d81045e0 0x7fb6d819ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.770+0000 7fb6e0800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6d8108f10 0x7fb6d819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.770+0000 7fb6e0800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6d8109860 0x7fb6d819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.770+0000 7fb6e0800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d8114500 con 0x7fb6d81045e0 2024-05-27T09:42:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.770+0000 7fb6e0800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d8114380 con 0x7fb6d8108f10 2024-05-27T09:42:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.770+0000 7fb6e0800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d8114680 con 0x7fb6d8109860 2024-05-27T09:42:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6dde00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d81045e0 0x7fb6d819ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6dd400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6d8108f10 0x7fb6d819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6dde00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d81045e0 0x7fb6d819ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36960/0 (socket says 172.21.15.44:36960) 2024-05-27T09:42:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6dde00700 1 -- 172.21.15.44:0/1389579584 learned_addr learned my addr 172.21.15.44:0/1389579584 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6dd400700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6d8108f10 0x7fb6d819b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52992/0 (socket says 172.21.15.44:52992) 2024-05-27T09:42:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6de800700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6d8109860 0x7fb6d819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:28.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6dde00700 1 -- 172.21.15.44:0/1389579584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6d8109860 msgr2=0x7fb6d819f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:28.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6dde00700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6d8109860 0x7fb6d819f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:28.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6dde00700 1 -- 172.21.15.44:0/1389579584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6d8108f10 msgr2=0x7fb6d819b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:28.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6dde00700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6d8108f10 0x7fb6d819b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:28.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6dde00700 1 -- 172.21.15.44:0/1389579584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d819fce0 con 0x7fb6d81045e0 2024-05-27T09:42:28.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.771+0000 7fb6dd400700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6d8108f10 0x7fb6d819b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:28.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.772+0000 7fb6dde00700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d81045e0 0x7fb6d819ae10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc005910 tx=0x7fb6cc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:28.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.772+0000 7fb6d2a00700 1 -- 172.21.15.44:0/1389579584 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc027070 con 0x7fb6d81045e0 2024-05-27T09:42:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.772+0000 7fb6d2a00700 1 -- 172.21.15.44:0/1389579584 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6cc00b610 con 0x7fb6d81045e0 2024-05-27T09:42:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.772+0000 7fb6d2a00700 1 -- 172.21.15.44:0/1389579584 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc0173f0 con 0x7fb6d81045e0 2024-05-27T09:42:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.772+0000 7fb6e0800700 1 -- 172.21.15.44:0/1389579584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d819ff70 con 0x7fb6d81045e0 2024-05-27T09:42:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.772+0000 7fb6e0800700 1 -- 172.21.15.44:0/1389579584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d81a0450 con 0x7fb6d81045e0 2024-05-27T09:42:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.774+0000 7fb6e0800700 1 -- 172.21.15.44:0/1389579584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d80512b0 con 0x7fb6d81045e0 2024-05-27T09:42:28.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.776+0000 7fb6d2a00700 1 -- 172.21.15.44:0/1389579584 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb6cc02a070 con 0x7fb6d81045e0 2024-05-27T09:42:28.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.778+0000 7fb6d2a00700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb6c00b29f0 0x7fb6c00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:28.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.778+0000 7fb6d2a00700 1 -- 172.21.15.44:0/1389579584 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fb6cc0e3620 con 0x7fb6d81045e0 2024-05-27T09:42:28.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.778+0000 7fb6dd400700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb6c00b29f0 0x7fb6c00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:28.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.779+0000 7fb6dd400700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb6c00b29f0 0x7fb6c00b4f00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8108e20 tx=0x7fb6c80073d0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:28.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:28.781+0000 7fb6d2a00700 1 -- 172.21.15.44:0/1389579584 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb6cc013070 con 0x7fb6d81045e0 2024-05-27T09:42:29.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.077+0000 7fb6e0800700 1 -- 172.21.15.44:0/1389579584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6d802e5e0 con 0x7fb6d81045e0 2024-05-27T09:42:29.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.079+0000 7fb6d2a00700 1 -- 172.21.15.44:0/1389579584 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11863 (secure 0 0 0) 0x7fb6cc0ab030 con 0x7fb6d81045e0 2024-05-27T09:42:29.080 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:42:29.081 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:41:21.007340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26297":{"gid":26297,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/102444567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":102444567},{"type":"v1","addr":"172.21.15.189:6835","nonce":102444567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:42:29.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.082+0000 7fb6e0800700 1 -- 172.21.15.44:0/1389579584 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb6c00b29f0 msgr2=0x7fb6c00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:29.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.082+0000 7fb6e0800700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb6c00b29f0 0x7fb6c00b4f00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8108e20 tx=0x7fb6c80073d0 comp rx=0 tx=0).stop 2024-05-27T09:42:29.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.082+0000 7fb6e0800700 1 -- 172.21.15.44:0/1389579584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d81045e0 msgr2=0x7fb6d819ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:29.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.082+0000 7fb6e0800700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d81045e0 0x7fb6d819ae10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc005910 tx=0x7fb6cc0047b0 comp rx=0 tx=0).stop 2024-05-27T09:42:29.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.083+0000 7fb6e0800700 1 -- 172.21.15.44:0/1389579584 shutdown_connections 2024-05-27T09:42:29.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.083+0000 7fb6e0800700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb6c00b29f0 0x7fb6c00b4f00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:29.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.083+0000 7fb6e0800700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6d8109860 0x7fb6d819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:29.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.083+0000 7fb6e0800700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d81045e0 0x7fb6d819ae10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:29.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.083+0000 7fb6e0800700 1 --2- 172.21.15.44:0/1389579584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6d8108f10 0x7fb6d819b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:29.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.083+0000 7fb6e0800700 1 -- 172.21.15.44:0/1389579584 >> 172.21.15.44:0/1389579584 conn(0x7fb6d80ffeb0 msgr2=0x7fb6d8102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:29.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.083+0000 7fb6e0800700 1 -- 172.21.15.44:0/1389579584 shutdown_connections 2024-05-27T09:42:29.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:29.083+0000 7fb6e0800700 1 -- 172.21.15.44:0/1389579584 wait complete. 2024-05-27T09:42:29.086 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-05-27T09:42:29.944 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db --set-id 77ef2bd1 --await-for 45.0' 2024-05-27T09:42:29.944 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db --set-id 77ef2bd1 --await-for 45.0 2024-05-27T09:42:30.152 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:29 smithi189 ceph-mon[36671]: pgmap v223: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 7.1 KiB/s wr, 289 op/s 2024-05-27T09:42:30.153 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:29 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1389579584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:42:30.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:29 smithi044 ceph-mon[31618]: pgmap v223: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 7.1 KiB/s wr, 289 op/s 2024-05-27T09:42:30.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:29 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1389579584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:42:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:29 smithi050 ceph-mon[36371]: pgmap v223: 97 pgs: 97 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 7.1 KiB/s wr, 289 op/s 2024-05-27T09:42:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:29 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1389579584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:42:30.429 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:32.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:31 smithi044 ceph-mon[31618]: pgmap v224: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 6.5 KiB/s wr, 265 op/s 2024-05-27T09:42:32.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:31 smithi050 ceph-mon[36371]: pgmap v224: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 6.5 KiB/s wr, 265 op/s 2024-05-27T09:42:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:31 smithi189 ceph-mon[36671]: pgmap v224: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 6.5 KiB/s wr, 265 op/s 2024-05-27T09:42:32.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.836+0000 7f99a5e00700 1 -- 172.21.15.44:0/2576058650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a0108ef0 msgr2=0x7f99a0109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:32.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.836+0000 7f99a5e00700 1 --2- 172.21.15.44:0/2576058650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a0108ef0 0x7f99a0109300 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f99880099c0 tx=0x7f9988009cd0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.836+0000 7f99a5e00700 1 -- 172.21.15.44:0/2576058650 shutdown_connections 2024-05-27T09:42:32.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.836+0000 7f99a5e00700 1 --2- 172.21.15.44:0/2576058650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99a0109840 0x7f99a01111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.836+0000 7f99a5e00700 1 --2- 172.21.15.44:0/2576058650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99a01045c0 0x7f99a01049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.836+0000 7f99a5e00700 1 --2- 172.21.15.44:0/2576058650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a0108ef0 0x7f99a0109300 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.836+0000 7f99a5e00700 1 -- 172.21.15.44:0/2576058650 >> 172.21.15.44:0/2576058650 conn(0x7f99a00ffeb0 msgr2=0x7f99a0102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:32.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.836+0000 7f99a5e00700 1 -- 172.21.15.44:0/2576058650 shutdown_connections 2024-05-27T09:42:32.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.836+0000 7f99a5e00700 1 -- 172.21.15.44:0/2576058650 wait complete. 2024-05-27T09:42:32.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.838+0000 7f99a5e00700 1 Processor -- start 2024-05-27T09:42:32.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.838+0000 7f99a5e00700 1 -- start start 2024-05-27T09:42:32.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.838+0000 7f99a5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99a01045c0 0x7f99a019cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.838+0000 7f99a5e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99a0108ef0 0x7f99a019d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.838+0000 7f99a5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a0109840 0x7f99a01a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.838+0000 7f99a5e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f99a0114450 con 0x7f99a0109840 2024-05-27T09:42:32.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.838+0000 7f99a5e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f99a01142d0 con 0x7f99a0108ef0 2024-05-27T09:42:32.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.838+0000 7f99a5e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99a01145d0 con 0x7f99a01045c0 2024-05-27T09:42:32.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.839+0000 7f999f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a0109840 0x7f99a01a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.839+0000 7f9996000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99a0108ef0 0x7f99a019d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.839+0000 7f999ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99a01045c0 0x7f99a019cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.839+0000 7f9996000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99a0108ef0 0x7f99a019d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35128/0 (socket says 172.21.15.44:35128) 2024-05-27T09:42:32.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.839+0000 7f9996000700 1 -- 172.21.15.44:0/4208501812 learned_addr learned my addr 172.21.15.44:0/4208501812 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:32.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.839+0000 7f999f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a0109840 0x7f99a01a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45772/0 (socket says 172.21.15.44:45772) 2024-05-27T09:42:32.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.839+0000 7f9996000700 1 -- 172.21.15.44:0/4208501812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99a01045c0 msgr2=0x7f99a019cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:32.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.839+0000 7f9996000700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99a01045c0 0x7f99a019cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.839+0000 7f9996000700 1 -- 172.21.15.44:0/4208501812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a0109840 msgr2=0x7f99a01a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:32.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.839+0000 7f9996000700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a0109840 0x7f99a01a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.839+0000 7f9996000700 1 -- 172.21.15.44:0/4208501812 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a01a1eb0 con 0x7f99a0108ef0 2024-05-27T09:42:32.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.840+0000 7f9996000700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99a0108ef0 0x7f99a019d580 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9988006010 tx=0x7f9988005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:32.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.840+0000 7f99a4a00700 1 -- 172.21.15.44:0/4208501812 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9988026070 con 0x7f99a0108ef0 2024-05-27T09:42:32.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.840+0000 7f99a4a00700 1 -- 172.21.15.44:0/4208501812 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9988005680 con 0x7f99a0108ef0 2024-05-27T09:42:32.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.840+0000 7f99a5e00700 1 -- 172.21.15.44:0/4208501812 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a01a2140 con 0x7f99a0108ef0 2024-05-27T09:42:32.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.840+0000 7f99a4a00700 1 -- 172.21.15.44:0/4208501812 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998800fb70 con 0x7f99a0108ef0 2024-05-27T09:42:32.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.840+0000 7f99a5e00700 1 -- 172.21.15.44:0/4208501812 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a01a2620 con 0x7f99a0108ef0 2024-05-27T09:42:32.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.844+0000 7f99a4a00700 1 -- 172.21.15.44:0/4208501812 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f9988029070 con 0x7f99a0108ef0 2024-05-27T09:42:32.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.845+0000 7f99b49ceb80 1 Processor -- start 2024-05-27T09:42:32.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.845+0000 7f99b49ceb80 1 -- start start 2024-05-27T09:42:32.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.846+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 0x561b95d0adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.846+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d0b2f0 0x561b95d04f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.846+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95d05440 0x561b95d05850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.846+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561b95c32f40 con 0x561b95d05440 2024-05-27T09:42:32.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.846+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561b95c22ae0 con 0x561b95d0b2f0 2024-05-27T09:42:32.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.846+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561b95c26b20 con 0x561b95b85f40 2024-05-27T09:42:32.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.846+0000 7f999ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d0b2f0 0x561b95d04f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.846+0000 7f999f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 0x561b95d0adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.846+0000 7f999f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 0x561b95d0adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45936/0 (socket says 172.21.15.44:45936) 2024-05-27T09:42:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.846+0000 7f999f400700 1 -- 172.21.15.44:0/4126576601 learned_addr learned my addr 172.21.15.44:0/4126576601 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.847+0000 7f99a4a00700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99840b29f0 0x7f99840b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.847+0000 7f9996000700 1 --2- 172.21.15.44:0/4126576601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95d05440 0x561b95d05850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.846+0000 7f999ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d0b2f0 0x561b95d04f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35132/0 (socket says 172.21.15.44:35132) 2024-05-27T09:42:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.847+0000 7f999f400700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99840b29f0 0x7f99840b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.847+0000 7f99a4a00700 1 -- 172.21.15.44:0/4208501812 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f99880e2d00 con 0x7f99a0108ef0 2024-05-27T09:42:32.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.847+0000 7f999ea00700 1 -- 172.21.15.44:0/4126576601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 msgr2=0x561b95d0adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:32.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.847+0000 7f999ea00700 1 --2- 172.21.15.44:0/4126576601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 0x561b95d0adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.847+0000 7f999ea00700 1 -- 172.21.15.44:0/4126576601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95d05440 msgr2=0x561b95d05850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:32.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.847+0000 7f999ea00700 1 --2- 172.21.15.44:0/4126576601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95d05440 0x561b95d05850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.847+0000 7f999ea00700 1 -- 172.21.15.44:0/4126576601 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b95beb520 con 0x561b95d0b2f0 2024-05-27T09:42:32.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.848+0000 7f999ea00700 1 --2- 172.21.15.44:0/4126576601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d0b2f0 0x561b95d04f00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9990009cb0 tx=0x7f9990011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb77b5a6f291c4c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:32.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.848+0000 7f9994c00700 1 -- 172.21.15.44:0/4126576601 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990011d20 con 0x561b95d0b2f0 2024-05-27T09:42:32.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.848+0000 7f9994c00700 1 -- 172.21.15.44:0/4126576601 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99900044e0 con 0x561b95d0b2f0 2024-05-27T09:42:32.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.848+0000 7f9994c00700 1 -- 172.21.15.44:0/4126576601 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990005280 con 0x561b95d0b2f0 2024-05-27T09:42:32.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.848+0000 7f999f400700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99840b29f0 0x7f99840b4f00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x561b95d060f0 tx=0x7f9998004b40 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:32.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.849+0000 7f99b49ceb80 1 -- 172.21.15.44:0/4126576601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d0b2f0 msgr2=0x561b95d04f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:32.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.849+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/4126576601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d0b2f0 0x561b95d04f00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9990009cb0 tx=0x7f9990011440 comp rx=0 tx=0).stop 2024-05-27T09:42:32.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.849+0000 7f99b49ceb80 1 -- 172.21.15.44:0/4126576601 shutdown_connections 2024-05-27T09:42:32.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.849+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/4126576601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95d05440 0x561b95d05850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.849+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/4126576601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 0x561b95d0adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.849+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/4126576601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d0b2f0 0x561b95d04f00 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.849+0000 7f99b49ceb80 1 -- 172.21.15.44:0/4126576601 >> 172.21.15.44:0/4126576601 conn(0x561b95be60c0 msgr2=0x561b95d00100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:32.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.849+0000 7f99b49ceb80 1 -- 172.21.15.44:0/4126576601 shutdown_connections 2024-05-27T09:42:32.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.849+0000 7f99b49ceb80 1 -- 172.21.15.44:0/4126576601 wait complete. 2024-05-27T09:42:32.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.850+0000 7f99b49ceb80 1 -- start start 2024-05-27T09:42:32.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.850+0000 7f99b49ceb80 1 Processor -- start 2024-05-27T09:42:32.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 0x561b95d0ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d0ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95d0b2f0 0x561b95d0f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561b95b290a0 con 0x561b95b85f40 2024-05-27T09:42:32.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561b95bc3860 con 0x561b95d05440 2024-05-27T09:42:32.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561b95d11dc0 con 0x561b95d0b2f0 2024-05-27T09:42:32.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f99b49ceb80 10 client.0 _ll_register_callbacks cb 0x561b95bccd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:32.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f999ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 0x561b95d0ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f999ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 0x561b95d0ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45784/0 (socket says 172.21.15.44:45784) 2024-05-27T09:42:32.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f999ea00700 1 -- 172.21.15.44:0/2772965001 learned_addr learned my addr 172.21.15.44:0/2772965001 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:32.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f999f400700 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95d0b2f0 0x561b95d0f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.851+0000 7f9996000700 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d0ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.852+0000 7f999ea00700 1 -- 172.21.15.44:0/2772965001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95d0b2f0 msgr2=0x561b95d0f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:32.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.852+0000 7f999ea00700 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95d0b2f0 0x561b95d0f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.852+0000 7f999ea00700 1 -- 172.21.15.44:0/2772965001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 msgr2=0x561b95d0ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:32.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.852+0000 7f999ea00700 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d0ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.852+0000 7f999ea00700 1 -- 172.21.15.44:0/2772965001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b95b729d0 con 0x561b95b85f40 2024-05-27T09:42:32.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.852+0000 7f999ea00700 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 0x561b95d0ea00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9990004e00 tx=0x7f9990011d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8547c55ac0beafea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:32.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.852+0000 7f998d600700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:42:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.852+0000 7f998d600700 1 -- 172.21.15.44:0/2772965001 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990005310 con 0x561b95b85f40 2024-05-27T09:42:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.852+0000 7f99b49ceb80 1 -- 172.21.15.44:0/2772965001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99900044e0 con 0x561b95b85f40 2024-05-27T09:42:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.852+0000 7f998d600700 1 -- 172.21.15.44:0/2772965001 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f99900054b0 con 0x561b95b85f40 2024-05-27T09:42:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.853+0000 7f998d600700 1 -- 172.21.15.44:0/2772965001 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f99900056b0 con 0x561b95b85f40 2024-05-27T09:42:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.853+0000 7f99b49ceb80 10 client.16431 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.853+0000 7f99b49ceb80 1 -- 172.21.15.44:0/2772965001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b95bf3160 con 0x561b95b85f40 2024-05-27T09:42:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.853+0000 7f998d600700 1 -- 172.21.15.44:0/2772965001 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99900044e0 con 0x561b95b85f40 2024-05-27T09:42:32.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.853+0000 7f998d600700 1 -- 172.21.15.44:0/2772965001 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f9990018070 con 0x561b95b85f40 2024-05-27T09:42:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.853+0000 7f99b49ceb80 10 client.16431 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.853+0000 7f99b49ceb80 10 client.16431 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:42:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.853+0000 7f99b49ceb80 20 client.16431 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.853+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561b95d1eb30 0x561b95d20f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.853+0000 7f99b49ceb80 4 client.16431 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.853+0000 7f99b49ceb80 1 -- 172.21.15.44:0/2772965001 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b95c34e90 con 0x561b95d1eb30 2024-05-27T09:42:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.854+0000 7f9996000700 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561b95d1eb30 0x561b95d20f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.855+0000 7f9996000700 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561b95d1eb30 0x561b95d20f50 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47a647534c916715 server_cookie=b7f6f7c28e2eef9d in_seq=0 out_seq=0 2024-05-27T09:42:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.855+0000 7f998d600700 10 client.16431 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.855+0000 7f998d600700 1 -- 172.21.15.44:0/2772965001 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561b95c34e90 con 0x561b95d1eb30 2024-05-27T09:42:32.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.855+0000 7f998d600700 10 client.16431 handle_command_reply: tid=0 2024-05-27T09:42:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.855+0000 7f99b49ceb80 1 client.16431 shutdown 2024-05-27T09:42:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f99b49ceb80 1 -- 172.21.15.44:0/2772965001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 msgr2=0x561b95d0ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 0x561b95d0ea00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9990004e00 tx=0x7f9990011d50 comp rx=0 tx=0).stop 2024-05-27T09:42:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f99b49ceb80 1 -- 172.21.15.44:0/2772965001 shutdown_connections 2024-05-27T09:42:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561b95d1eb30 0x561b95d20f50 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95d0b2f0 0x561b95d0f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 0x561b95d0ea00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/2772965001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d0ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f99b49ceb80 1 -- 172.21.15.44:0/2772965001 >> 172.21.15.44:0/2772965001 conn(0x561b95be60c0 msgr2=0x561b95d03d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f998d600700 0 client.16431 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:32.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f99b49ceb80 1 -- 172.21.15.44:0/2772965001 shutdown_connections 2024-05-27T09:42:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f99b49ceb80 1 -- 172.21.15.44:0/2772965001 wait complete. 2024-05-27T09:42:32.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.856+0000 7f99b49ceb80 20 client.16431 trim_cache size 0 max 16384 2024-05-27T09:42:32.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.879+0000 7f99b49ceb80 1 Processor -- start 2024-05-27T09:42:32.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.879+0000 7f99b49ceb80 1 -- start start 2024-05-27T09:42:32.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 0x561b95d09130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d1ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95d0b2f0 0x561b95d04420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f99a01b3910 con 0x561b95d0b2f0 2024-05-27T09:42:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f99a01b1170 con 0x561b95d05440 2024-05-27T09:42:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99a01aefe0 con 0x561b95b85f40 2024-05-27T09:42:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f999ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 0x561b95d09130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f9996000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d1ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f999ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 0x561b95d09130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45940/0 (socket says 172.21.15.44:45940) 2024-05-27T09:42:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f999ea00700 1 -- 172.21.15.44:0/1561045339 learned_addr learned my addr 172.21.15.44:0/1561045339 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:32.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f999f400700 1 --2- 172.21.15.44:0/1561045339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95d0b2f0 0x561b95d04420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f999ea00700 1 -- 172.21.15.44:0/1561045339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 msgr2=0x561b95d1ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:32.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f999ea00700 1 --2- 172.21.15.44:0/1561045339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d1ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.880+0000 7f999ea00700 1 -- 172.21.15.44:0/1561045339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95d0b2f0 msgr2=0x561b95d04420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:32.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.881+0000 7f999ea00700 1 --2- 172.21.15.44:0/1561045339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95d0b2f0 0x561b95d04420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.881+0000 7f999ea00700 1 -- 172.21.15.44:0/1561045339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b95d0b780 con 0x561b95b85f40 2024-05-27T09:42:32.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.881+0000 7f9996000700 1 --2- 172.21.15.44:0/1561045339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d1ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:32.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.881+0000 7f999ea00700 1 --2- 172.21.15.44:0/1561045339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 0x561b95d09130 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9990004e30 tx=0x7f9990011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38713c8242b5515 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:32.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.881+0000 7f998fe00700 1 -- 172.21.15.44:0/1561045339 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999003b070 con 0x561b95b85f40 2024-05-27T09:42:32.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.881+0000 7f998fe00700 1 -- 172.21.15.44:0/1561045339 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9990029070 con 0x561b95b85f40 2024-05-27T09:42:32.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.881+0000 7f998fe00700 1 -- 172.21.15.44:0/1561045339 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99900366b0 con 0x561b95b85f40 2024-05-27T09:42:32.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.882+0000 7f99b49ceb80 1 -- 172.21.15.44:0/1561045339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 msgr2=0x561b95d09130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:32.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.882+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/1561045339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 0x561b95d09130 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9990004e30 tx=0x7f9990011440 comp rx=0 tx=0).stop 2024-05-27T09:42:32.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.882+0000 7f99b49ceb80 1 -- 172.21.15.44:0/1561045339 shutdown_connections 2024-05-27T09:42:32.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.882+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/1561045339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95d0b2f0 0x561b95d04420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.882+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/1561045339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95b85f40 0x561b95d09130 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.882+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/1561045339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d1ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.882+0000 7f99b49ceb80 1 -- 172.21.15.44:0/1561045339 >> 172.21.15.44:0/1561045339 conn(0x561b95be60c0 msgr2=0x561b95d03d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:32.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.882+0000 7f99b49ceb80 1 -- 172.21.15.44:0/1561045339 shutdown_connections 2024-05-27T09:42:32.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.882+0000 7f99b49ceb80 1 -- 172.21.15.44:0/1561045339 wait complete. 2024-05-27T09:42:32.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.883+0000 7f99b49ceb80 1 -- start start 2024-05-27T09:42:32.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.883+0000 7f99b49ceb80 1 Processor -- start 2024-05-27T09:42:32.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 0x561b95d17010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d17550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f99b49ceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95d0b2f0 0x561b95d13fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f99a01af630 con 0x561b95b85f40 2024-05-27T09:42:32.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f99a01b0aa0 con 0x561b95d05440 2024-05-27T09:42:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f99b49ceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99a01afd00 con 0x561b95d0b2f0 2024-05-27T09:42:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f99b49ceb80 10 client.0 _ll_register_callbacks cb 0x561b95bccd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f9996000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d17550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f9996000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d17550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35162/0 (socket says 172.21.15.44:35162) 2024-05-27T09:42:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f999ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 0x561b95d17010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f9996000700 1 -- 172.21.15.44:0/3870877792 learned_addr learned my addr 172.21.15.44:0/3870877792 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.884+0000 7f999f400700 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95d0b2f0 0x561b95d13fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.885+0000 7f9996000700 1 -- 172.21.15.44:0/3870877792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95d0b2f0 msgr2=0x561b95d13fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.885+0000 7f9996000700 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95d0b2f0 0x561b95d13fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.885+0000 7f9996000700 1 -- 172.21.15.44:0/3870877792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 msgr2=0x561b95d17010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:32.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.885+0000 7f9996000700 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 0x561b95d17010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.885+0000 7f9996000700 1 -- 172.21.15.44:0/3870877792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b95d17b90 con 0x561b95d05440 2024-05-27T09:42:32.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.885+0000 7f9996000700 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d17550 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f99880a7280 tx=0x7f99880a5f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86dd1fc1232d46ab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:32.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.885+0000 7f998d600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:32.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.885+0000 7f998d600700 1 -- 172.21.15.44:0/3870877792 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99880aa710 con 0x561b95d05440 2024-05-27T09:42:32.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.885+0000 7f998d600700 1 -- 172.21.15.44:0/3870877792 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f99880aa270 con 0x561b95d05440 2024-05-27T09:42:32.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.885+0000 7f99b49ceb80 1 -- 172.21.15.44:0/3870877792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9990029070 con 0x561b95d05440 2024-05-27T09:42:32.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.886+0000 7f998d600700 1 -- 172.21.15.44:0/3870877792 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f99880aa010 con 0x561b95d05440 2024-05-27T09:42:32.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.886+0000 7f99b49ceb80 10 client.26600 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:32.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.886+0000 7f99b49ceb80 1 -- 172.21.15.44:0/3870877792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b95d92220 con 0x561b95d05440 2024-05-27T09:42:32.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.886+0000 7f998d600700 1 -- 172.21.15.44:0/3870877792 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9990029070 con 0x561b95d05440 2024-05-27T09:42:32.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.886+0000 7f998d600700 1 -- 172.21.15.44:0/3870877792 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f99880a9270 con 0x561b95d05440 2024-05-27T09:42:32.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.886+0000 7f99b49ceb80 10 client.26600 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:32.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.886+0000 7f99b49ceb80 10 client.26600 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:42:32.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.886+0000 7f99b49ceb80 20 client.26600 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:32.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.886+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561b95d928e0 0x561b95d94d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.886+0000 7f99b49ceb80 4 client.26600 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "77ef2bd1", "await_for": 45.0}] 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.886+0000 7f99b49ceb80 1 -- 172.21.15.44:0/3870877792 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "set_id": "77ef2bd1", "await_for": 45.0}) -- 0x561b95c2e670 con 0x561b95d928e0 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.887+0000 7f999ea00700 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561b95d928e0 0x561b95d94d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.887+0000 7f999ea00700 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561b95d928e0 0x561b95d94d00 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f2e2ede05578495 server_cookie=38b81a991ad4b8b4 in_seq=0 out_seq=0 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.887+0000 7f998d600700 10 client.26600 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f998d600700 1 -- 172.21.15.44:0/3870877792 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+592 (crc 0 0 0) 0x561b95c2e670 con 0x561b95d928e0 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f998d600700 10 client.26600 handle_command_reply: tid=0 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 1 client.26600 shutdown 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 1 -- 172.21.15.44:0/3870877792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 msgr2=0x561b95d17550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d17550 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f99880a7280 tx=0x7f99880a5f30 comp rx=0 tx=0).stop 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 1 -- 172.21.15.44:0/3870877792 shutdown_connections 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x561b95d928e0 0x561b95d94d00 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b95d0b2f0 0x561b95d13fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561b95b85f40 0x561b95d17010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 1 --2- 172.21.15.44:0/3870877792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b95d05440 0x561b95d17550 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f998d600700 0 client.26600 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 1 -- 172.21.15.44:0/3870877792 >> 172.21.15.44:0/3870877792 conn(0x561b95be60c0 msgr2=0x561b95d08e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 1 -- 172.21.15.44:0/3870877792 shutdown_connections 2024-05-27T09:42:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 1 -- 172.21.15.44:0/3870877792 wait complete. 2024-05-27T09:42:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.888+0000 7f99b49ceb80 20 client.26600 trim_cache size 0 max 16384 2024-05-27T09:42:32.893 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:42:32.893 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 29, 2024-05-27T09:42:32.893 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:42:32.893 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 15, 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "77ef2bd1": { 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 15, 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 150.0, 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:42:32.894 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:42:32.895 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:42:32.895 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:42:32.895 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 35.6 2024-05-27T09:42:32.895 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:32.895 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:32.895 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:32.895 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:32.895 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:32.895 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:42:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.889+0000 7f99a5e00700 1 -- 172.21.15.44:0/4208501812 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99840b29f0 msgr2=0x7f99840b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.889+0000 7f99a5e00700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99840b29f0 0x7f99840b4f00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x561b95d060f0 tx=0x7f9998004b40 comp rx=0 tx=0).stop 2024-05-27T09:42:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.889+0000 7f99a5e00700 1 -- 172.21.15.44:0/4208501812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99a0108ef0 msgr2=0x7f99a019d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.889+0000 7f99a5e00700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99a0108ef0 0x7f99a019d580 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9988006010 tx=0x7f9988005280 comp rx=0 tx=0).stop 2024-05-27T09:42:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.889+0000 7f99a5e00700 1 -- 172.21.15.44:0/4208501812 shutdown_connections 2024-05-27T09:42:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.889+0000 7f99a5e00700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f99840b29f0 0x7f99840b4f00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.889+0000 7f99a5e00700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f99a0109840 0x7f99a01a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.889+0000 7f99a5e00700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99a01045c0 0x7f99a019cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.889+0000 7f99a5e00700 1 --2- 172.21.15.44:0/4208501812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99a0108ef0 0x7f99a019d580 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.889+0000 7f99a5e00700 1 -- 172.21.15.44:0/4208501812 >> 172.21.15.44:0/4208501812 conn(0x7f99a00ffeb0 msgr2=0x7f99a010a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.890+0000 7f99a5e00700 1 -- 172.21.15.44:0/4208501812 shutdown_connections 2024-05-27T09:42:32.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:32.890+0000 7f99a5e00700 1 -- 172.21.15.44:0/4208501812 wait complete. 2024-05-27T09:42:33.774 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '77ef2bd1', quiesce took 91.2 seconds. Will release after: -61.2 2024-05-27T09:42:33.774 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:42:33.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:33 smithi044 ceph-mon[31618]: pgmap v225: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:34.261 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:34.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:33 smithi050 ceph-mon[36371]: pgmap v225: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:34.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:33 smithi189 ceph-mon[36671]: pgmap v225: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:35.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:35 smithi044 ceph-mon[31618]: pgmap v226: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:36.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:35 smithi050 ceph-mon[36371]: pgmap v226: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:35 smithi189 ceph-mon[36671]: pgmap v226: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:36.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.593+0000 7fc8a3200700 1 -- 172.21.15.44:0/425675934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc89c106eb0 msgr2=0x7fc89c113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:36.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.593+0000 7fc8a3200700 1 --2- 172.21.15.44:0/425675934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc89c106eb0 0x7fc89c113340 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc89400b0a0 tx=0x7fc89400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:42:36.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.593+0000 7fc8a3200700 1 -- 172.21.15.44:0/425675934 shutdown_connections 2024-05-27T09:42:36.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.593+0000 7fc8a3200700 1 --2- 172.21.15.44:0/425675934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc89c106eb0 0x7fc89c113340 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:36.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.593+0000 7fc8a3200700 1 --2- 172.21.15.44:0/425675934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89c10c5a0 0x7fc89c10c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:36.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.593+0000 7fc8a3200700 1 --2- 172.21.15.44:0/425675934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc89c106560 0x7fc89c106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:36.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.593+0000 7fc8a3200700 1 -- 172.21.15.44:0/425675934 >> 172.21.15.44:0/425675934 conn(0x7fc89c102040 msgr2=0x7fc89c104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:36.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.593+0000 7fc8a3200700 1 -- 172.21.15.44:0/425675934 shutdown_connections 2024-05-27T09:42:36.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.593+0000 7fc8a3200700 1 -- 172.21.15.44:0/425675934 wait complete. 2024-05-27T09:42:36.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.594+0000 7fc8a3200700 1 Processor -- start 2024-05-27T09:42:36.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.595+0000 7fc8a3200700 1 -- start start 2024-05-27T09:42:36.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.595+0000 7fc8a3200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89c106560 0x7fc89c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:36.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.595+0000 7fc8a3200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc89c106eb0 0x7fc89c19d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:36.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.595+0000 7fc8a3200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc89c10c5a0 0x7fc89c1a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:36.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.595+0000 7fc8a3200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc89c116720 con 0x7fc89c106eb0 2024-05-27T09:42:36.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.595+0000 7fc8a3200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc89c1165a0 con 0x7fc89c10c5a0 2024-05-27T09:42:36.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.595+0000 7fc8a3200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc89c1168a0 con 0x7fc89c106560 2024-05-27T09:42:36.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.596+0000 7fc89b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc89c106eb0 0x7fc89c19d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:36.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.596+0000 7fc8a1200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc89c10c5a0 0x7fc89c1a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:36.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.596+0000 7fc89be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89c106560 0x7fc89c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:36.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.596+0000 7fc8a1200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc89c10c5a0 0x7fc89c1a1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35184/0 (socket says 172.21.15.44:35184) 2024-05-27T09:42:36.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.596+0000 7fc8a1200700 1 -- 172.21.15.44:0/2281384639 learned_addr learned my addr 172.21.15.44:0/2281384639 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:36.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.596+0000 7fc89be00700 1 -- 172.21.15.44:0/2281384639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc89c10c5a0 msgr2=0x7fc89c1a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:36.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.596+0000 7fc89be00700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc89c10c5a0 0x7fc89c1a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:36.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.596+0000 7fc89be00700 1 -- 172.21.15.44:0/2281384639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc89c106eb0 msgr2=0x7fc89c19d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:36.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.596+0000 7fc89be00700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc89c106eb0 0x7fc89c19d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:36.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.596+0000 7fc89be00700 1 -- 172.21.15.44:0/2281384639 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc89c1a1ea0 con 0x7fc89c106560 2024-05-27T09:42:36.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.596+0000 7fc89b400700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc89c106eb0 0x7fc89c19d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:36.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.597+0000 7fc89be00700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89c106560 0x7fc89c19cfd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00e9c0 tx=0x7fc88c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:36.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.597+0000 7fc898c00700 1 -- 172.21.15.44:0/2281384639 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88c00c9c0 con 0x7fc89c106560 2024-05-27T09:42:36.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.597+0000 7fc8a3200700 1 -- 172.21.15.44:0/2281384639 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc89c1a2190 con 0x7fc89c106560 2024-05-27T09:42:36.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.597+0000 7fc898c00700 1 -- 172.21.15.44:0/2281384639 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc88c00cb60 con 0x7fc89c106560 2024-05-27T09:42:36.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.597+0000 7fc898c00700 1 -- 172.21.15.44:0/2281384639 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88c018680 con 0x7fc89c106560 2024-05-27T09:42:36.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.597+0000 7fc8a3200700 1 -- 172.21.15.44:0/2281384639 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc89c1a9ab0 con 0x7fc89c106560 2024-05-27T09:42:36.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.598+0000 7fc8a3200700 1 -- 172.21.15.44:0/2281384639 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc89c0512b0 con 0x7fc89c106560 2024-05-27T09:42:36.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.601+0000 7fc898c00700 1 -- 172.21.15.44:0/2281384639 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc88c010430 con 0x7fc89c106560 2024-05-27T09:42:36.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.602+0000 7fc898c00700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc8800b29f0 0x7fc8800b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:36.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.602+0000 7fc898c00700 1 -- 172.21.15.44:0/2281384639 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fc88c0e2b50 con 0x7fc89c106560 2024-05-27T09:42:36.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.603+0000 7fc89b400700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc8800b29f0 0x7fc8800b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:36.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.603+0000 7fc89b400700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc8800b29f0 0x7fc8800b4f00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc89c105350 tx=0x7fc888009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:36.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.604+0000 7fc898c00700 1 -- 172.21.15.44:0/2281384639 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc88c0aa4d0 con 0x7fc89c106560 2024-05-27T09:42:36.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.893+0000 7fc8a3200700 1 -- 172.21.15.44:0/2281384639 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc89c02e5e0 con 0x7fc89c106560 2024-05-27T09:42:36.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.895+0000 7fc898c00700 1 -- 172.21.15.44:0/2281384639 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11863 (secure 0 0 0) 0x7fc88c0a9dd0 con 0x7fc89c106560 2024-05-27T09:42:36.896 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:42:36.896 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:41:21.007340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26297":{"gid":26297,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/102444567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":102444567},{"type":"v1","addr":"172.21.15.189:6835","nonce":102444567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:42:36.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.898+0000 7fc8a3200700 1 -- 172.21.15.44:0/2281384639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc8800b29f0 msgr2=0x7fc8800b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:36.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.898+0000 7fc8a3200700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc8800b29f0 0x7fc8800b4f00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc89c105350 tx=0x7fc888009260 comp rx=0 tx=0).stop 2024-05-27T09:42:36.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.898+0000 7fc8a3200700 1 -- 172.21.15.44:0/2281384639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89c106560 msgr2=0x7fc89c19cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:36.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.898+0000 7fc8a3200700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89c106560 0x7fc89c19cfd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00e9c0 tx=0x7fc88c00ecd0 comp rx=0 tx=0).stop 2024-05-27T09:42:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.898+0000 7fc8a3200700 1 -- 172.21.15.44:0/2281384639 shutdown_connections 2024-05-27T09:42:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.898+0000 7fc8a3200700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc8800b29f0 0x7fc8800b4f00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.898+0000 7fc8a3200700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc89c10c5a0 0x7fc89c1a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.898+0000 7fc8a3200700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89c106560 0x7fc89c19cfd0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.898+0000 7fc8a3200700 1 --2- 172.21.15.44:0/2281384639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc89c106eb0 0x7fc89c19d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.898+0000 7fc8a3200700 1 -- 172.21.15.44:0/2281384639 >> 172.21.15.44:0/2281384639 conn(0x7fc89c102040 msgr2=0x7fc89c103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.899+0000 7fc8a3200700 1 -- 172.21.15.44:0/2281384639 shutdown_connections 2024-05-27T09:42:36.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:36.899+0000 7fc8a3200700 1 -- 172.21.15.44:0/2281384639 wait complete. 2024-05-27T09:42:36.901 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-05-27T09:42:37.684 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db --set-id 77ef2bd1 --release --await' 2024-05-27T09:42:37.685 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db --set-id 77ef2bd1 --release --await 2024-05-27T09:42:37.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:37 smithi044 ceph-mon[31618]: pgmap v227: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2024-05-27T09:42:37.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:37 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2281384639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:42:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:37 smithi050 ceph-mon[36371]: pgmap v227: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2024-05-27T09:42:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:37 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2281384639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:42:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:37 smithi189 ceph-mon[36671]: pgmap v227: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2024-05-27T09:42:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:37 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2281384639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:42:38.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.383+0000 7f6c82800640 1 -- 172.21.15.44:0/65364615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c10d990 msgr2=0x7f6c7c10fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.383+0000 7f6c82800640 1 --2- 172.21.15.44:0/65364615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c10d990 0x7f6c7c10fdb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009900 tx=0x7f6c6802a480 comp rx=0 tx=0).stop 2024-05-27T09:42:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.383+0000 7f6c82800640 1 -- 172.21.15.44:0/65364615 shutdown_connections 2024-05-27T09:42:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.383+0000 7f6c82800640 1 --2- 172.21.15.44:0/65364615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c7c1102f0 0x7f6c7c112710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.383+0000 7f6c82800640 1 --2- 172.21.15.44:0/65364615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c10d990 0x7f6c7c10fdb0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.383+0000 7f6c82800640 1 --2- 172.21.15.44:0/65364615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c7c075480 0x7f6c7c075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.383+0000 7f6c82800640 1 -- 172.21.15.44:0/65364615 >> 172.21.15.44:0/65364615 conn(0x7f6c7c079240 msgr2=0x7f6c7c079680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.383+0000 7f6c82800640 1 -- 172.21.15.44:0/65364615 shutdown_connections 2024-05-27T09:42:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.383+0000 7f6c82800640 1 -- 172.21.15.44:0/65364615 wait complete. 2024-05-27T09:42:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.384+0000 7f6c82800640 1 Processor -- start 2024-05-27T09:42:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c82800640 1 -- start start 2024-05-27T09:42:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c82800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c075480 0x7f6c7c1a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c82800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c7c10d990 0x7f6c7c1a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c82800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c7c1102f0 0x7f6c7c19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c82800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c115ed0 con 0x7f6c7c1102f0 2024-05-27T09:42:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c82800640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c115d50 con 0x7f6c7c075480 2024-05-27T09:42:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c82800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c116050 con 0x7f6c7c10d990 2024-05-27T09:42:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c81400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c075480 0x7f6c7c1a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c81e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c7c1102f0 0x7f6c7c19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c81400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c075480 0x7f6c7c1a25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35204/0 (socket says 172.21.15.44:35204) 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c81400640 1 -- 172.21.15.44:0/2575268183 learned_addr learned my addr 172.21.15.44:0/2575268183 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.385+0000 7f6c80a00640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c7c10d990 0x7f6c7c1a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c81400640 1 -- 172.21.15.44:0/2575268183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c7c10d990 msgr2=0x7f6c7c1a2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c81400640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c7c10d990 0x7f6c7c1a2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c81400640 1 -- 172.21.15.44:0/2575268183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c7c1102f0 msgr2=0x7f6c7c19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c81400640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c7c1102f0 0x7f6c7c19c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c81400640 1 -- 172.21.15.44:0/2575268183 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7c19ce80 con 0x7f6c7c075480 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c80a00640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c7c10d990 0x7f6c7c1a2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c81e00640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c7c1102f0 0x7f6c7c19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c81400640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c075480 0x7f6c7c1a25d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6c7000e9c0 tx=0x7f6c7000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c7a000640 1 -- 172.21.15.44:0/2575268183 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7000cd80 con 0x7f6c7c075480 2024-05-27T09:42:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c7a000640 1 -- 172.21.15.44:0/2575268183 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c700035f0 con 0x7f6c7c075480 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c7a000640 1 -- 172.21.15.44:0/2575268183 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c70010690 con 0x7f6c7c075480 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.386+0000 7f6c82800640 1 -- 172.21.15.44:0/2575268183 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7c19d110 con 0x7f6c7c075480 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.387+0000 7f6c82800640 1 -- 172.21.15.44:0/2575268183 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c7c1a9370 con 0x7f6c7c075480 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.390+0000 7f6c8ada4740 1 Processor -- start 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.390+0000 7f6c8ada4740 1 -- start start 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb851b0 0x55ad2ff27a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff27fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 0x55ad2ff284e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ad2fe98b90 con 0x55ad2fb5c2f0 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ad2fe88730 con 0x55ad2fb851b0 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ad2fe8c770 con 0x55ad2fb644d0 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c7a000640 1 -- 172.21.15.44:0/2575268183 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6c700037a0 con 0x7f6c7c075480 2024-05-27T09:42:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c81e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb851b0 0x55ad2ff27a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c81e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb851b0 0x55ad2ff27a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35208/0 (socket says 172.21.15.44:35208) 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c81e00640 1 -- 172.21.15.44:0/2283496771 learned_addr learned my addr 172.21.15.44:0/2283496771 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c80a00640 1 --2- 172.21.15.44:0/2283496771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 0x55ad2ff284e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.391+0000 7f6c81400640 1 --2- 172.21.15.44:0/2283496771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff27fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.392+0000 7f6c81e00640 1 -- 172.21.15.44:0/2283496771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 msgr2=0x55ad2ff284e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.392+0000 7f6c81e00640 1 --2- 172.21.15.44:0/2283496771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 0x55ad2ff284e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.392+0000 7f6c81e00640 1 -- 172.21.15.44:0/2283496771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb5c2f0 msgr2=0x55ad2ff27fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.392+0000 7f6c81e00640 1 --2- 172.21.15.44:0/2283496771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff27fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.392+0000 7f6c81e00640 1 -- 172.21.15.44:0/2283496771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad2fe218b0 con 0x55ad2fb851b0 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.392+0000 7f6c81400640 1 --2- 172.21.15.44:0/2283496771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff27fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.392+0000 7f6c81e00640 1 --2- 172.21.15.44:0/2283496771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb851b0 0x55ad2ff27a60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6c74011790 tx=0x7f6c74011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70836bcf462a1423 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.392+0000 7f6c6e600640 1 -- 172.21.15.44:0/2283496771 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74003c30 con 0x55ad2fb851b0 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.392+0000 7f6c6e600640 1 -- 172.21.15.44:0/2283496771 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c7400fe50 con 0x55ad2fb851b0 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.392+0000 7f6c6e600640 1 -- 172.21.15.44:0/2283496771 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c740193e0 con 0x55ad2fb851b0 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.393+0000 7f6c7a000640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c640b29d0 0x7f6c640b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.393+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2283496771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb851b0 msgr2=0x55ad2ff27a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.393+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2283496771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb851b0 0x55ad2ff27a60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6c74011790 tx=0x7f6c74011c50 comp rx=0 tx=0).stop 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.393+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2283496771 shutdown_connections 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.393+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2283496771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 0x55ad2ff284e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.393+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2283496771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff27fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.393+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2283496771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb851b0 0x55ad2ff27a60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.393+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2283496771 >> 172.21.15.44:0/2283496771 conn(0x55ad2fb79300 msgr2=0x55ad2ff24210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.393+0000 7f6c7a000640 1 -- 172.21.15.44:0/2575268183 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f6c700e2a40 con 0x7f6c7c075480 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.393+0000 7f6c81400640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c640b29d0 0x7f6c640b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.393+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2283496771 shutdown_connections 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.394+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2283496771 wait complete. 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.394+0000 7f6c81400640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c640b29d0 0x7f6c640b4ee0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55ad2ff25550 tx=0x7f6c70003d90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.394+0000 7f6c8ada4740 1 -- start start 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.395+0000 7f6c8ada4740 1 Processor -- start 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.395+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff31b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.395+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 0x55ad2ff320d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.395+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff3dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.395+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ad2fb6e790 con 0x55ad2fb851b0 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.395+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ad2fe30bc0 con 0x55ad2fb644d0 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.395+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ad2ff34b90 con 0x55ad2fb5c2f0 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.395+0000 7f6c8ada4740 10 client.0 _ll_register_callbacks cb 0x55ad2fe0e400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.395+0000 7f6c80a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 0x55ad2ff320d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.395+0000 7f6c80a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 0x55ad2ff320d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35218/0 (socket says 172.21.15.44:35218) 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.395+0000 7f6c80a00640 1 -- 172.21.15.44:0/2142343524 learned_addr learned my addr 172.21.15.44:0/2142343524 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.396+0000 7f6c81e00640 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff3dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.396+0000 7f6c81e00640 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff31b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.396+0000 7f6c80a00640 1 -- 172.21.15.44:0/2142343524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 msgr2=0x55ad2ff31b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.396+0000 7f6c80a00640 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff31b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.396+0000 7f6c80a00640 1 -- 172.21.15.44:0/2142343524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 msgr2=0x55ad2ff3dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.396+0000 7f6c80a00640 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff3dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.396+0000 7f6c80a00640 1 -- 172.21.15.44:0/2142343524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad2fb9d210 con 0x55ad2fb644d0 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.396+0000 7f6c80a00640 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 0x55ad2ff320d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c06c620 tx=0x7f6c68005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c233b58698d9a412 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.396+0000 7f6c62a00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.397+0000 7f6c62a00640 1 -- 172.21.15.44:0/2142343524 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68035040 con 0x55ad2fb644d0 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.397+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2142343524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c7400fe50 con 0x55ad2fb644d0 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.397+0000 7f6c62a00640 1 -- 172.21.15.44:0/2142343524 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f6c680046c0 con 0x55ad2fb644d0 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.397+0000 7f6c62a00640 1 -- 172.21.15.44:0/2142343524 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6c680048c0 con 0x55ad2fb644d0 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.397+0000 7f6c8ada4740 10 client.26618 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.397+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2142343524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ad2fc69620 con 0x55ad2fb644d0 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.397+0000 7f6c62a00640 1 -- 172.21.15.44:0/2142343524 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c7400fe50 con 0x55ad2fb644d0 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.398+0000 7f6c62a00640 1 -- 172.21.15.44:0/2142343524 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f6c68003c40 con 0x55ad2fb644d0 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.398+0000 7f6c8ada4740 10 client.26618 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.398+0000 7f6c8ada4740 10 client.26618 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.398+0000 7f6c8ada4740 20 client.26618 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.398+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ad2fb43d80 0x55ad2ff43e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.398+0000 7f6c8ada4740 4 client.26618 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.398+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2142343524 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad2fe9aae0 con 0x55ad2fb43d80 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.398+0000 7f6c81e00640 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ad2fb43d80 0x55ad2ff43e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.399+0000 7f6c81e00640 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ad2fb43d80 0x55ad2ff43e80 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23c5fec0a89c22a server_cookie=9dc4f5a764a6264b in_seq=0 out_seq=0 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.399+0000 7f6c62a00640 10 client.26618 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c62a00640 1 -- 172.21.15.44:0/2142343524 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ad2fe9aae0 con 0x55ad2fb43d80 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c62a00640 10 client.26618 handle_command_reply: tid=0 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 1 client.26618 shutdown 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2142343524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 msgr2=0x55ad2ff320d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 0x55ad2ff320d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c06c620 tx=0x7f6c68005270 comp rx=0 tx=0).stop 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2142343524 shutdown_connections 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ad2fb43d80 0x55ad2ff43e80 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff3dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 0x55ad2ff320d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2142343524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff31b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c62a00640 0 client.26618 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2142343524 >> 172.21.15.44:0/2142343524 conn(0x55ad2fb79300 msgr2=0x55ad2ff27890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2142343524 shutdown_connections 2024-05-27T09:42:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2142343524 wait complete. 2024-05-27T09:42:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.400+0000 7f6c8ada4740 20 client.26618 trim_cache size 0 max 16384 2024-05-27T09:42:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.418+0000 7f6c8ada4740 1 Processor -- start 2024-05-27T09:42:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.418+0000 7f6c8ada4740 1 -- start start 2024-05-27T09:42:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.418+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff342c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.418+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 0x55ad2ff30ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.418+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff3dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.418+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c1b44f0 con 0x55ad2fb851b0 2024-05-27T09:42:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.418+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c1b1d90 con 0x55ad2fb644d0 2024-05-27T09:42:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.418+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c1afc00 con 0x55ad2fb5c2f0 2024-05-27T09:42:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.419+0000 7f6c81e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff3dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.419+0000 7f6c81e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.419+0000 7f6c81e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff342c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:46008/0 (socket says 172.21.15.44:46008) 2024-05-27T09:42:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.419+0000 7f6c81e00640 1 -- 172.21.15.44:0/2418041662 learned_addr learned my addr 172.21.15.44:0/2418041662 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.419+0000 7f6c80a00640 1 --2- 172.21.15.44:0/2418041662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 0x55ad2ff30ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.419+0000 7f6c81e00640 1 -- 172.21.15.44:0/2418041662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 msgr2=0x55ad2ff30ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:38.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.419+0000 7f6c81e00640 1 --2- 172.21.15.44:0/2418041662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 0x55ad2ff30ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.419+0000 7f6c81e00640 1 -- 172.21.15.44:0/2418041662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 msgr2=0x55ad2ff3dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:38.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.419+0000 7f6c81e00640 1 --2- 172.21.15.44:0/2418041662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff3dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.419+0000 7f6c81e00640 1 -- 172.21.15.44:0/2418041662 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad2ff24de0 con 0x55ad2fb5c2f0 2024-05-27T09:42:38.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.419+0000 7f6c81e00640 1 --2- 172.21.15.44:0/2418041662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff342c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6c74004550 tx=0x7f6c74004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=307be22de88da054 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.420+0000 7f6c6dc00640 1 -- 172.21.15.44:0/2418041662 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74004310 con 0x55ad2fb5c2f0 2024-05-27T09:42:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.420+0000 7f6c6dc00640 1 -- 172.21.15.44:0/2418041662 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c7400f9a0 con 0x55ad2fb5c2f0 2024-05-27T09:42:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.420+0000 7f6c6dc00640 1 -- 172.21.15.44:0/2418041662 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7401a440 con 0x55ad2fb5c2f0 2024-05-27T09:42:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.421+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2418041662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 msgr2=0x55ad2ff342c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.421+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2418041662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff342c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6c74004550 tx=0x7f6c74004580 comp rx=0 tx=0).stop 2024-05-27T09:42:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.421+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2418041662 shutdown_connections 2024-05-27T09:42:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.421+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2418041662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff3dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.421+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2418041662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb644d0 0x55ad2ff30ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.421+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/2418041662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff342c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.421+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2418041662 >> 172.21.15.44:0/2418041662 conn(0x55ad2fb79300 msgr2=0x55ad2ff31e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.421+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2418041662 shutdown_connections 2024-05-27T09:42:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.421+0000 7f6c8ada4740 1 -- 172.21.15.44:0/2418041662 wait complete. 2024-05-27T09:42:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.421+0000 7f6c8ada4740 1 -- start start 2024-05-27T09:42:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.421+0000 7f6c8ada4740 1 Processor -- start 2024-05-27T09:42:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.422+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff3a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.422+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 0x55ad2ff35500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.422+0000 7f6c8ada4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff35a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.422+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c1b0250 con 0x55ad2fb851b0 2024-05-27T09:42:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.422+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c1b16c0 con 0x55ad2fb5c2f0 2024-05-27T09:42:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.422+0000 7f6c8ada4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c1b0920 con 0x55ad2fb644d0 2024-05-27T09:42:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.422+0000 7f6c8ada4740 10 client.0 _ll_register_callbacks cb 0x7f6c740093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.422+0000 7f6c80a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 0x55ad2ff35500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.422+0000 7f6c81e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff3a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.422+0000 7f6c80a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 0x55ad2ff35500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:46010/0 (socket says 172.21.15.44:46010) 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.422+0000 7f6c80a00640 1 -- 172.21.15.44:0/3832968593 learned_addr learned my addr 172.21.15.44:0/3832968593 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.423+0000 7f6c81e00640 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff35a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.423+0000 7f6c80a00640 1 -- 172.21.15.44:0/3832968593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb5c2f0 msgr2=0x55ad2ff3a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.423+0000 7f6c80a00640 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff3a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.423+0000 7f6c80a00640 1 -- 172.21.15.44:0/3832968593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 msgr2=0x55ad2ff35a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.423+0000 7f6c80a00640 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff35a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.423+0000 7f6c80a00640 1 -- 172.21.15.44:0/3832968593 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad2ff35f80 con 0x55ad2fb644d0 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.423+0000 7f6c81e00640 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff35a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.423+0000 7f6c80a00640 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 0x55ad2ff35500 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6c6802a9a0 tx=0x7f6c68031ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63353dc0ec0199b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.423+0000 7f6c62a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:42:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.424+0000 7f6c62a00640 1 -- 172.21.15.44:0/3832968593 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68052040 con 0x55ad2fb644d0 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.424+0000 7f6c8ada4740 1 -- 172.21.15.44:0/3832968593 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c7400f9a0 con 0x55ad2fb644d0 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.424+0000 7f6c62a00640 1 -- 172.21.15.44:0/3832968593 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f6c680055f0 con 0x55ad2fb644d0 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.424+0000 7f6c62a00640 1 -- 172.21.15.44:0/3832968593 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6c68035040 con 0x55ad2fb644d0 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.424+0000 7f6c8ada4740 10 client.26620 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.424+0000 7f6c8ada4740 1 -- 172.21.15.44:0/3832968593 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ad2ffac080 con 0x55ad2fb644d0 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.424+0000 7f6c62a00640 1 -- 172.21.15.44:0/3832968593 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c7400f9a0 con 0x55ad2fb644d0 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.424+0000 7f6c62a00640 1 -- 172.21.15.44:0/3832968593 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f6c68003a30 con 0x55ad2fb644d0 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.424+0000 7f6c8ada4740 10 client.26620 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.424+0000 7f6c8ada4740 10 client.26620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.424+0000 7f6c8ada4740 20 client.26620 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.425+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ad2fb191c0 0x55ad2ffae870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.425+0000 7f6c8ada4740 4 client.26620 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:42:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.425+0000 7f6c8ada4740 1 -- 172.21.15.44:0/3832968593 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ad2fe942c0 con 0x55ad2fb191c0 2024-05-27T09:42:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.425+0000 7f6c81e00640 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ad2fb191c0 0x55ad2ffae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.425+0000 7f6c81e00640 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ad2fb191c0 0x55ad2ffae870 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4d30efdc5b21cb0 server_cookie=5763ed716567667 in_seq=0 out_seq=0 2024-05-27T09:42:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.425+0000 7f6c62a00640 10 client.26620 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:38.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c62a00640 1 -- 172.21.15.44:0/3832968593 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55ad2fe942c0 con 0x55ad2fb191c0 2024-05-27T09:42:38.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c62a00640 10 client.26620 handle_command_reply: tid=0 2024-05-27T09:42:38.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c8ada4740 1 client.26620 shutdown 2024-05-27T09:42:38.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c8ada4740 1 -- 172.21.15.44:0/3832968593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 msgr2=0x55ad2ff35500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:38.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 0x55ad2ff35500 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6c6802a9a0 tx=0x7f6c68031ac0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c8ada4740 1 -- 172.21.15.44:0/3832968593 shutdown_connections 2024-05-27T09:42:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ad2fb191c0 0x55ad2ffae870 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ad2fb851b0 0x55ad2ff35a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad2fb644d0 0x55ad2ff35500 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c8ada4740 1 --2- 172.21.15.44:0/3832968593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ad2fb5c2f0 0x55ad2ff3a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c62a00640 0 client.26620 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.431+0000 7f6c8ada4740 1 -- 172.21.15.44:0/3832968593 >> 172.21.15.44:0/3832968593 conn(0x55ad2fb79300 msgr2=0x55ad2ff2c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.432+0000 7f6c8ada4740 1 -- 172.21.15.44:0/3832968593 shutdown_connections 2024-05-27T09:42:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.432+0000 7f6c8ada4740 1 -- 172.21.15.44:0/3832968593 wait complete. 2024-05-27T09:42:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.432+0000 7f6c8ada4740 20 client.26620 trim_cache size 0 max 16384 2024-05-27T09:42:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:42:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:42:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:42:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 -- 172.21.15.44:0/2575268183 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c640b29d0 msgr2=0x7f6c640b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c640b29d0 0x7f6c640b4ee0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55ad2ff25550 tx=0x7f6c70003d90 comp rx=0 tx=0).stop 2024-05-27T09:42:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 -- 172.21.15.44:0/2575268183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c075480 msgr2=0x7f6c7c1a25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c075480 0x7f6c7c1a25d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6c7000e9c0 tx=0x7f6c7000ee80 comp rx=0 tx=0).stop 2024-05-27T09:42:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 -- 172.21.15.44:0/2575268183 shutdown_connections 2024-05-27T09:42:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6c640b29d0 0x7f6c640b4ee0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6c7c1102f0 0x7f6c7c19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c7c10d990 0x7f6c7c1a2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 --2- 172.21.15.44:0/2575268183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c075480 0x7f6c7c1a25d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 -- 172.21.15.44:0/2575268183 >> 172.21.15.44:0/2575268183 conn(0x7f6c7c079240 msgr2=0x7f6c7c102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 -- 172.21.15.44:0/2575268183 shutdown_connections 2024-05-27T09:42:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:38.433+0000 7f6c82800640 1 -- 172.21.15.44:0/2575268183 wait complete. 2024-05-27T09:42:40.073 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:39 smithi044 ceph-mon[31618]: pgmap v228: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:42:40.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:39 smithi050 ceph-mon[36371]: pgmap v228: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:42:40.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:39 smithi189 ceph-mon[36671]: pgmap v228: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:42:40.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.694+0000 7f320ce00700 1 -- 172.21.15.44:0/3210093423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3208110a20 msgr2=0x7f3208112e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.694+0000 7f320ce00700 1 --2- 172.21.15.44:0/3210093423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3208110a20 0x7f3208112e40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f320000b0a0 tx=0x7f320000b3b0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.694+0000 7f320ce00700 1 -- 172.21.15.44:0/3210093423 shutdown_connections 2024-05-27T09:42:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.694+0000 7f320ce00700 1 --2- 172.21.15.44:0/3210093423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3208110a20 0x7f3208112e40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.694+0000 7f320ce00700 1 --2- 172.21.15.44:0/3210093423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3208076be0 0x7f3208076ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.694+0000 7f320ce00700 1 --2- 172.21.15.44:0/3210093423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3208077530 0x7f32081104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.694+0000 7f320ce00700 1 -- 172.21.15.44:0/3210093423 >> 172.21.15.44:0/3210093423 conn(0x7f3208079d40 msgr2=0x7f320807a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.694+0000 7f320ce00700 1 -- 172.21.15.44:0/3210093423 shutdown_connections 2024-05-27T09:42:40.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.694+0000 7f320ce00700 1 -- 172.21.15.44:0/3210093423 wait complete. 2024-05-27T09:42:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.696+0000 7f320ce00700 1 Processor -- start 2024-05-27T09:42:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.696+0000 7f320ce00700 1 -- start start 2024-05-27T09:42:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.696+0000 7f320ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3208076be0 0x7f320819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.696+0000 7f320ce00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3208077530 0x7f320819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.696+0000 7f320ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3208110a20 0x7f32081a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.696+0000 7f320ce00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3208116500 con 0x7f3208076be0 2024-05-27T09:42:40.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.696+0000 7f320ce00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3208116380 con 0x7f3208077530 2024-05-27T09:42:40.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.696+0000 7f320ce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3208116680 con 0x7f3208110a20 2024-05-27T09:42:40.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.697+0000 7f3207400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3208076be0 0x7f320819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.697+0000 7f31ffe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3208077530 0x7f320819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.697+0000 7f3207400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3208076be0 0x7f320819ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45906/0 (socket says 172.21.15.44:45906) 2024-05-27T09:42:40.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.697+0000 7f3207400700 1 -- 172.21.15.44:0/1922547878 learned_addr learned my addr 172.21.15.44:0/1922547878 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:40.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.697+0000 7f3207e00700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3208110a20 0x7f32081a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.697+0000 7f3207400700 1 -- 172.21.15.44:0/1922547878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3208110a20 msgr2=0x7f32081a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:40.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.697+0000 7f3207400700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3208110a20 0x7f32081a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.697+0000 7f3207400700 1 -- 172.21.15.44:0/1922547878 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3208077530 msgr2=0x7f320819d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:40.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.697+0000 7f3207400700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3208077530 0x7f320819d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.697+0000 7f3207400700 1 -- 172.21.15.44:0/1922547878 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32081a1d20 con 0x7f3208076be0 2024-05-27T09:42:40.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.698+0000 7f3207400700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3208076be0 0x7f320819ce50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f31f800d980 tx=0x7f31f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:40.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.698+0000 7f3204c00700 1 -- 172.21.15.44:0/1922547878 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f8016070 con 0x7f3208076be0 2024-05-27T09:42:40.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.698+0000 7f320ce00700 1 -- 172.21.15.44:0/1922547878 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32081a2010 con 0x7f3208076be0 2024-05-27T09:42:40.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.698+0000 7f3204c00700 1 -- 172.21.15.44:0/1922547878 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31f8004830 con 0x7f3208076be0 2024-05-27T09:42:40.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.698+0000 7f320ce00700 1 -- 172.21.15.44:0/1922547878 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32081a9930 con 0x7f3208076be0 2024-05-27T09:42:40.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.698+0000 7f3204c00700 1 -- 172.21.15.44:0/1922547878 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f80054e0 con 0x7f3208076be0 2024-05-27T09:42:40.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.702+0000 7f3204c00700 1 -- 172.21.15.44:0/1922547878 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f31f8004220 con 0x7f3208076be0 2024-05-27T09:42:40.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.703+0000 7f321cf76b80 1 Processor -- start 2024-05-27T09:42:40.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.703+0000 7f321cf76b80 1 -- start start 2024-05-27T09:42:40.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 0x55a71bc754b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc759f0 0x55a71bc7f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc75e00 0x55a71bc815a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f321cf76b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a71bba2f40 con 0x55a71bc759f0 2024-05-27T09:42:40.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f321cf76b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a71bb92ae0 con 0x55a71baf5f40 2024-05-27T09:42:40.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f321cf76b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a71bb96b20 con 0x55a71bc75e00 2024-05-27T09:42:40.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f3207400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc759f0 0x55a71bc7f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f3207e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 0x55a71bc754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f3207e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 0x55a71bc754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35260/0 (socket says 172.21.15.44:35260) 2024-05-27T09:42:40.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f3207e00700 1 -- 172.21.15.44:0/1765909277 learned_addr learned my addr 172.21.15.44:0/1765909277 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:40.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f31ffe00700 1 --2- 172.21.15.44:0/1765909277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc75e00 0x55a71bc815a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.704+0000 7f3204c00700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f31ec0b29f0 0x7f31ec0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.705+0000 7f3204c00700 1 -- 172.21.15.44:0/1922547878 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f31f80e2400 con 0x7f3208076be0 2024-05-27T09:42:40.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.705+0000 7f31ffe00700 1 -- 172.21.15.44:0/1765909277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 msgr2=0x55a71bc754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:40.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.705+0000 7f31ffe00700 1 --2- 172.21.15.44:0/1765909277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 0x55a71bc754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.705+0000 7f31ffe00700 1 -- 172.21.15.44:0/1765909277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc759f0 msgr2=0x55a71bc7f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.705+0000 7f31ffe00700 1 --2- 172.21.15.44:0/1765909277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc759f0 0x55a71bc7f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.705+0000 7f31ffe00700 1 -- 172.21.15.44:0/1765909277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a71bb5b520 con 0x55a71bc75e00 2024-05-27T09:42:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.705+0000 7f3207400700 1 --2- 172.21.15.44:0/1765909277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc759f0 0x55a71bc7f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.705+0000 7f3207e00700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f31ec0b29f0 0x7f31ec0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.705+0000 7f31ffe00700 1 --2- 172.21.15.44:0/1765909277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc75e00 0x55a71bc815a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f320806c020 tx=0x7f31f0007e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f76167205a183b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.706+0000 7f31fcc00700 1 -- 172.21.15.44:0/1765909277 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f0004310 con 0x55a71bc75e00 2024-05-27T09:42:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.706+0000 7f31fcc00700 1 -- 172.21.15.44:0/1765909277 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31f0015070 con 0x55a71bc75e00 2024-05-27T09:42:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.706+0000 7f31fcc00700 1 -- 172.21.15.44:0/1765909277 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f0004dd0 con 0x55a71bc75e00 2024-05-27T09:42:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.706+0000 7f3207e00700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f31ec0b29f0 0x7f31ec0b4f00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55a71bc70c70 tx=0x7f320000b3b0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.707+0000 7f321cf76b80 1 -- 172.21.15.44:0/1765909277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc75e00 msgr2=0x55a71bc815a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.707+0000 7f321cf76b80 1 --2- 172.21.15.44:0/1765909277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc75e00 0x55a71bc815a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f320806c020 tx=0x7f31f0007e40 comp rx=0 tx=0).stop 2024-05-27T09:42:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.707+0000 7f321cf76b80 1 -- 172.21.15.44:0/1765909277 shutdown_connections 2024-05-27T09:42:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.707+0000 7f321cf76b80 1 --2- 172.21.15.44:0/1765909277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc75e00 0x55a71bc815a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.707+0000 7f321cf76b80 1 --2- 172.21.15.44:0/1765909277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 0x55a71bc754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.707+0000 7f321cf76b80 1 --2- 172.21.15.44:0/1765909277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc759f0 0x55a71bc7f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.707+0000 7f321cf76b80 1 -- 172.21.15.44:0/1765909277 >> 172.21.15.44:0/1765909277 conn(0x55a71bb560c0 msgr2=0x55a71bc702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:40.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.707+0000 7f321cf76b80 1 -- 172.21.15.44:0/1765909277 shutdown_connections 2024-05-27T09:42:40.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.707+0000 7f321cf76b80 1 -- 172.21.15.44:0/1765909277 wait complete. 2024-05-27T09:42:40.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.708+0000 7f321cf76b80 1 -- start start 2024-05-27T09:42:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.708+0000 7f321cf76b80 1 Processor -- start 2024-05-27T09:42:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.708+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 0x55a71bc7eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.708+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 0x55a71bc7f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.708+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc7f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.708+0000 7f321cf76b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a71ba990a0 con 0x55a71bc75e00 2024-05-27T09:42:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f321cf76b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a71bb33860 con 0x55a71bc759f0 2024-05-27T09:42:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f321cf76b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a71bc81f10 con 0x55a71baf5f40 2024-05-27T09:42:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f321cf76b80 10 client.0 _ll_register_callbacks cb 0x55a71bb3cd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f31ffe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 0x55a71bc7f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f31ffe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 0x55a71bc7f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35276/0 (socket says 172.21.15.44:35276) 2024-05-27T09:42:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f31ffe00700 1 -- 172.21.15.44:0/3133560594 learned_addr learned my addr 172.21.15.44:0/3133560594 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f3207e00700 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc7f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f3207400700 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 0x55a71bc7eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f3207400700 1 -- 172.21.15.44:0/3133560594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 msgr2=0x55a71bc7f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f3207400700 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 0x55a71bc7f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f3207400700 1 -- 172.21.15.44:0/3133560594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 msgr2=0x55a71bc7f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f3207400700 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc7f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.709+0000 7f3207400700 1 -- 172.21.15.44:0/3133560594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a71bae29d0 con 0x55a71baf5f40 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.710+0000 7f3207400700 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 0x55a71bc7eae0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f31f80dfaf0 tx=0x7f31f8004f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f842e7da0020788 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.710+0000 7f31f5600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.710+0000 7f31f5600700 1 -- 172.21.15.44:0/3133560594 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f8022550 con 0x55a71baf5f40 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.710+0000 7f31f5600700 1 -- 172.21.15.44:0/3133560594 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f31f80dfd70 con 0x55a71baf5f40 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.710+0000 7f321cf76b80 1 -- 172.21.15.44:0/3133560594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31f0015070 con 0x55a71baf5f40 2024-05-27T09:42:40.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.710+0000 7f31f5600700 1 -- 172.21.15.44:0/3133560594 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f31f80a9d80 con 0x55a71baf5f40 2024-05-27T09:42:40.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.710+0000 7f321cf76b80 10 client.26632 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:40.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.710+0000 7f321cf76b80 1 -- 172.21.15.44:0/3133560594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a71bb63160 con 0x55a71baf5f40 2024-05-27T09:42:40.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.711+0000 7f31f5600700 1 -- 172.21.15.44:0/3133560594 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31f0015070 con 0x55a71baf5f40 2024-05-27T09:42:40.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.711+0000 7f31f5600700 1 -- 172.21.15.44:0/3133560594 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f31f80a98e0 con 0x55a71baf5f40 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.711+0000 7f321cf76b80 10 client.26632 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.711+0000 7f321cf76b80 10 client.26632 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.711+0000 7f321cf76b80 20 client.26632 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.711+0000 7f321cf76b80 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a71bc8ee30 0x55a71bc91290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.711+0000 7f321cf76b80 4 client.26632 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.711+0000 7f321cf76b80 1 -- 172.21.15.44:0/3133560594 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a71bba4e90 con 0x55a71bc8ee30 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.712+0000 7f31ffe00700 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a71bc8ee30 0x55a71bc91290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.712+0000 7f31ffe00700 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a71bc8ee30 0x55a71bc91290 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b510672bbb8c1da3 server_cookie=d3a1143499ea1e3e in_seq=0 out_seq=0 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.712+0000 7f31f5600700 10 client.26632 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f31f5600700 1 -- 172.21.15.44:0/3133560594 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a71bba4e90 con 0x55a71bc8ee30 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f31f5600700 10 client.26632 handle_command_reply: tid=0 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f321cf76b80 1 client.26632 shutdown 2024-05-27T09:42:40.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f321cf76b80 1 -- 172.21.15.44:0/3133560594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 msgr2=0x55a71bc7eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:40.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f321cf76b80 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 0x55a71bc7eae0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f31f80dfaf0 tx=0x7f31f8004f10 comp rx=0 tx=0).stop 2024-05-27T09:42:40.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f321cf76b80 1 -- 172.21.15.44:0/3133560594 shutdown_connections 2024-05-27T09:42:40.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f321cf76b80 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a71bc8ee30 0x55a71bc91290 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f321cf76b80 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc7f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f321cf76b80 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 0x55a71bc7eae0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f321cf76b80 1 --2- 172.21.15.44:0/3133560594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 0x55a71bc7f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f321cf76b80 1 -- 172.21.15.44:0/3133560594 >> 172.21.15.44:0/3133560594 conn(0x55a71bb560c0 msgr2=0x55a71bc73e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:40.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.713+0000 7f31f5600700 0 client.26632 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:40.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.714+0000 7f321cf76b80 1 -- 172.21.15.44:0/3133560594 shutdown_connections 2024-05-27T09:42:40.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.714+0000 7f321cf76b80 1 -- 172.21.15.44:0/3133560594 wait complete. 2024-05-27T09:42:40.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.714+0000 7f321cf76b80 20 client.26632 trim_cache size 0 max 16384 2024-05-27T09:42:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.735+0000 7f321cf76b80 1 Processor -- start 2024-05-27T09:42:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.736+0000 7f321cf76b80 1 -- start start 2024-05-27T09:42:40.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.736+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 0x55a71bc80f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.736+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 0x55a71bc79ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.736+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc6fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.736+0000 7f321cf76b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f32081b3850 con 0x55a71bc75e00 2024-05-27T09:42:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.736+0000 7f321cf76b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f32081b10b0 con 0x55a71bc759f0 2024-05-27T09:42:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.736+0000 7f321cf76b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f32081aef20 con 0x55a71baf5f40 2024-05-27T09:42:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.736+0000 7f3207e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc6fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.737+0000 7f31ffe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 0x55a71bc79ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.737+0000 7f31ffe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 0x55a71bc79ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35288/0 (socket says 172.21.15.44:35288) 2024-05-27T09:42:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.737+0000 7f31ffe00700 1 -- 172.21.15.44:0/2581677208 learned_addr learned my addr 172.21.15.44:0/2581677208 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.737+0000 7f3207400700 1 --2- 172.21.15.44:0/2581677208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 0x55a71bc80f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.737+0000 7f31ffe00700 1 -- 172.21.15.44:0/2581677208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 msgr2=0x55a71bc80f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.737+0000 7f31ffe00700 1 --2- 172.21.15.44:0/2581677208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 0x55a71bc80f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.737+0000 7f31ffe00700 1 -- 172.21.15.44:0/2581677208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 msgr2=0x55a71bc6fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.737+0000 7f31ffe00700 1 --2- 172.21.15.44:0/2581677208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc6fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.737+0000 7f31ffe00700 1 -- 172.21.15.44:0/2581677208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a71bc750e0 con 0x55a71bc759f0 2024-05-27T09:42:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.737+0000 7f3207400700 1 --2- 172.21.15.44:0/2581677208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 0x55a71bc80f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:42:40.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.737+0000 7f31ffe00700 1 --2- 172.21.15.44:0/2581677208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 0x55a71bc79ef0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f31f0005c20 tx=0x7f31f0005960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f74c650993d8ace server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:40.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.738+0000 7f31f7e00700 1 -- 172.21.15.44:0/2581677208 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f002c030 con 0x55a71bc759f0 2024-05-27T09:42:40.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.738+0000 7f31f7e00700 1 -- 172.21.15.44:0/2581677208 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31f0015030 con 0x55a71bc759f0 2024-05-27T09:42:40.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.738+0000 7f31f7e00700 1 -- 172.21.15.44:0/2581677208 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f00055e0 con 0x55a71bc759f0 2024-05-27T09:42:40.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.739+0000 7f321cf76b80 1 -- 172.21.15.44:0/2581677208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 msgr2=0x55a71bc79ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:40.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.739+0000 7f321cf76b80 1 --2- 172.21.15.44:0/2581677208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 0x55a71bc79ef0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f31f0005c20 tx=0x7f31f0005960 comp rx=0 tx=0).stop 2024-05-27T09:42:40.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.739+0000 7f321cf76b80 1 -- 172.21.15.44:0/2581677208 shutdown_connections 2024-05-27T09:42:40.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.739+0000 7f321cf76b80 1 --2- 172.21.15.44:0/2581677208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc6fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.739+0000 7f321cf76b80 1 --2- 172.21.15.44:0/2581677208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71baf5f40 0x55a71bc80f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.739+0000 7f321cf76b80 1 --2- 172.21.15.44:0/2581677208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71bc759f0 0x55a71bc79ef0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.739+0000 7f321cf76b80 1 -- 172.21.15.44:0/2581677208 >> 172.21.15.44:0/2581677208 conn(0x55a71bb560c0 msgr2=0x55a71bc6f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:40.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.739+0000 7f321cf76b80 1 -- 172.21.15.44:0/2581677208 shutdown_connections 2024-05-27T09:42:40.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.739+0000 7f321cf76b80 1 -- 172.21.15.44:0/2581677208 wait complete. 2024-05-27T09:42:40.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.739+0000 7f321cf76b80 1 -- start start 2024-05-27T09:42:40.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.740+0000 7f321cf76b80 1 Processor -- start 2024-05-27T09:42:40.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.740+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 0x55a71bc83010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.740+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc759f0 0x55a71bc83550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.740+0000 7f321cf76b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc83a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.740+0000 7f321cf76b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f32081af570 con 0x55a71bc75e00 2024-05-27T09:42:40.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.740+0000 7f321cf76b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f32081b09e0 con 0x55a71baf5f40 2024-05-27T09:42:40.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.740+0000 7f321cf76b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f32081afc40 con 0x55a71bc759f0 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.740+0000 7f321cf76b80 10 client.0 _ll_register_callbacks cb 0x55a71bb3cd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 0x55a71bc83010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc83a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f31ffe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc759f0 0x55a71bc83550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 0x55a71bc83010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:35304/0 (socket says 172.21.15.44:35304) 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc83a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45958/0 (socket says 172.21.15.44:45958) 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207400700 1 -- 172.21.15.44:0/1577062231 learned_addr learned my addr 172.21.15.44:0/1577062231 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207400700 1 -- 172.21.15.44:0/1577062231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc759f0 msgr2=0x55a71bc83550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207400700 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc759f0 0x55a71bc83550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207400700 1 -- 172.21.15.44:0/1577062231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 msgr2=0x55a71bc83a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207400700 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc83a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207400700 1 -- 172.21.15.44:0/1577062231 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a71bcf89d0 con 0x55a71baf5f40 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f31ffe00700 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc759f0 0x55a71bc83550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207e00700 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc83a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f3207400700 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 0x55a71bc83010 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f31f80ac050 tx=0x7f31f80a9b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4150c871b7ed4ee7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.741+0000 7f31f5600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.742+0000 7f31f5600700 1 -- 172.21.15.44:0/1577062231 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f80b20f0 con 0x55a71baf5f40 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.742+0000 7f31f5600700 1 -- 172.21.15.44:0/1577062231 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f31f808fba0 con 0x55a71baf5f40 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.742+0000 7f321cf76b80 1 -- 172.21.15.44:0/1577062231 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31f0015030 con 0x55a71baf5f40 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.742+0000 7f31f5600700 1 -- 172.21.15.44:0/1577062231 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f31f80b22e0 con 0x55a71baf5f40 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.742+0000 7f321cf76b80 10 client.26642 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.742+0000 7f321cf76b80 1 -- 172.21.15.44:0/1577062231 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a71bcf9c70 con 0x55a71baf5f40 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.742+0000 7f31f5600700 1 -- 172.21.15.44:0/1577062231 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31f0015030 con 0x55a71baf5f40 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.742+0000 7f31f5600700 1 -- 172.21.15.44:0/1577062231 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f31f80e5ae0 con 0x55a71baf5f40 2024-05-27T09:42:40.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.742+0000 7f321cf76b80 10 client.26642 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:40.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.742+0000 7f321cf76b80 10 client.26642 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:42:40.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.743+0000 7f321cf76b80 20 client.26642 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:40.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.743+0000 7f321cf76b80 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a71bcfa350 0x55a71bcfc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:40.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.743+0000 7f321cf76b80 4 client.26642 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "77ef2bd1", "release": true, "await": true}] 2024-05-27T09:42:40.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.743+0000 7f321cf76b80 1 -- 172.21.15.44:0/1577062231 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "set_id": "77ef2bd1", "release": true, "await": true}) -- 0x55a71bb9e670 con 0x55a71bcfa350 2024-05-27T09:42:40.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.743+0000 7f31ffe00700 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a71bcfa350 0x55a71bcfc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:40.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.744+0000 7f31ffe00700 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a71bcfa350 0x55a71bcfc770 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fedb55dfe87423a server_cookie=4223c3f9eba09d0f in_seq=0 out_seq=0 2024-05-27T09:42:40.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.744+0000 7f31f5600700 10 client.26642 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f31f5600700 1 -- 172.21.15.44:0/1577062231 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55a71bb9e670 con 0x55a71bcfa350 2024-05-27T09:42:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f31f5600700 10 client.26642 handle_command_reply: tid=0 2024-05-27T09:42:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f321cf76b80 1 client.26642 shutdown 2024-05-27T09:42:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f321cf76b80 1 -- 172.21.15.44:0/1577062231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 msgr2=0x55a71bc83010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:40.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f321cf76b80 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 0x55a71bc83010 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f31f80ac050 tx=0x7f31f80a9b10 comp rx=0 tx=0).stop 2024-05-27T09:42:40.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f321cf76b80 1 -- 172.21.15.44:0/1577062231 shutdown_connections 2024-05-27T09:42:40.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f321cf76b80 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a71bcfa350 0x55a71bcfc770 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f321cf76b80 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a71bc75e00 0x55a71bc83a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f321cf76b80 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a71baf5f40 0x55a71bc83010 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f321cf76b80 1 --2- 172.21.15.44:0/1577062231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a71bc759f0 0x55a71bc83550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f321cf76b80 1 -- 172.21.15.44:0/1577062231 >> 172.21.15.44:0/1577062231 conn(0x55a71bb560c0 msgr2=0x55a71bc70030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.845+0000 7f31f5600700 0 client.26642 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.846+0000 7f321cf76b80 1 -- 172.21.15.44:0/1577062231 shutdown_connections 2024-05-27T09:42:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.846+0000 7f321cf76b80 1 -- 172.21.15.44:0/1577062231 wait complete. 2024-05-27T09:42:40.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.846+0000 7f321cf76b80 20 client.26642 trim_cache size 0 max 16384 2024-05-27T09:42:40.848 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:42:40.849 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 29, 2024-05-27T09:42:40.849 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:42:40.849 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 17, 2024-05-27T09:42:40.849 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:42:40.849 INFO:teuthology.orchestra.run.smithi044.stdout: "77ef2bd1": { 2024-05-27T09:42:40.849 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 17, 2024-05-27T09:42:40.849 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:42:40.849 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:42:40.849 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:42:40.850 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:42:40.850 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:42:40.850 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:42:40.850 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 150.0, 2024-05-27T09:42:40.850 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:42:40.850 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:42:40.850 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:42:40.850 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:42:40.850 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:42:40.850 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:42:40.850 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:40.851 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:40.851 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:40.851 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:40.851 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:42:40.851 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:42:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.847+0000 7f320ce00700 1 -- 172.21.15.44:0/1922547878 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f31ec0b29f0 msgr2=0x7f31ec0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.847+0000 7f320ce00700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f31ec0b29f0 0x7f31ec0b4f00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55a71bc70c70 tx=0x7f320000b3b0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.847+0000 7f320ce00700 1 -- 172.21.15.44:0/1922547878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3208076be0 msgr2=0x7f320819ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.847+0000 7f320ce00700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3208076be0 0x7f320819ce50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f31f800d980 tx=0x7f31f800dc90 comp rx=0 tx=0).stop 2024-05-27T09:42:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.848+0000 7f320ce00700 1 -- 172.21.15.44:0/1922547878 shutdown_connections 2024-05-27T09:42:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.848+0000 7f320ce00700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f31ec0b29f0 0x7f31ec0b4f00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.848+0000 7f320ce00700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3208110a20 0x7f32081a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.848+0000 7f320ce00700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3208076be0 0x7f320819ce50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.848+0000 7f320ce00700 1 --2- 172.21.15.44:0/1922547878 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3208077530 0x7f320819d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.848+0000 7f320ce00700 1 -- 172.21.15.44:0/1922547878 >> 172.21.15.44:0/1922547878 conn(0x7f3208079d40 msgr2=0x7f3208100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.848+0000 7f320ce00700 1 -- 172.21.15.44:0/1922547878 shutdown_connections 2024-05-27T09:42:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:40.848+0000 7f320ce00700 1 -- 172.21.15.44:0/1922547878 wait complete. 2024-05-27T09:42:41.606 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '77ef2bd1', total seconds elapsed: 99.0 2024-05-27T09:42:42.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:41 smithi044 ceph-mon[31618]: pgmap v229: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-05-27T09:42:42.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:41 smithi050 ceph-mon[36371]: pgmap v229: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-05-27T09:42:42.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:41 smithi189 ceph-mon[36671]: pgmap v229: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-05-27T09:42:44.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:43 smithi044 ceph-mon[31618]: pgmap v230: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:44.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:43 smithi050 ceph-mon[36371]: pgmap v230: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:43 smithi189 ceph-mon[36671]: pgmap v230: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:46.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:45 smithi044 ceph-mon[31618]: pgmap v231: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:46.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:45 smithi050 ceph-mon[36371]: pgmap v231: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:46.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:45 smithi189 ceph-mon[36671]: pgmap v231: 97 pgs: 97 active+clean; 26 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-05-27T09:42:48.138 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:47 smithi189 ceph-mon[36671]: pgmap v232: 97 pgs: 97 active+clean; 33 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 682 KiB/s wr, 13 op/s 2024-05-27T09:42:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:47 smithi044 ceph-mon[31618]: pgmap v232: 97 pgs: 97 active+clean; 33 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 682 KiB/s wr, 13 op/s 2024-05-27T09:42:48.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:47 smithi050 ceph-mon[36371]: pgmap v232: 97 pgs: 97 active+clean; 33 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 682 KiB/s wr, 13 op/s 2024-05-27T09:42:50.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:49 smithi044 ceph-mon[31618]: pgmap v233: 97 pgs: 97 active+clean; 33 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 682 KiB/s wr, 10 op/s 2024-05-27T09:42:50.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:49 smithi050 ceph-mon[36371]: pgmap v233: 97 pgs: 97 active+clean; 33 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 682 KiB/s wr, 10 op/s 2024-05-27T09:42:50.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:49 smithi189 ceph-mon[36671]: pgmap v233: 97 pgs: 97 active+clean; 33 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 682 KiB/s wr, 10 op/s 2024-05-27T09:42:51.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.675+0000 7fba93600640 1 -- 172.21.15.44:0/1162593120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba8c102b80 msgr2=0x7fba8c102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:51.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.675+0000 7fba93600640 1 --2- 172.21.15.44:0/1162593120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba8c102b80 0x7fba8c102f90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fba80009860 tx=0x7fba8002a340 comp rx=0 tx=0).stop 2024-05-27T09:42:51.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.676+0000 7fba93600640 1 -- 172.21.15.44:0/1162593120 shutdown_connections 2024-05-27T09:42:51.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.676+0000 7fba93600640 1 --2- 172.21.15.44:0/1162593120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba8c108000 0x7fba8c108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.676+0000 7fba93600640 1 --2- 172.21.15.44:0/1162593120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba8c1075d0 0x7fba8c1079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.676+0000 7fba93600640 1 --2- 172.21.15.44:0/1162593120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba8c102b80 0x7fba8c102f90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.676+0000 7fba93600640 1 -- 172.21.15.44:0/1162593120 >> 172.21.15.44:0/1162593120 conn(0x7fba8c0782d0 msgr2=0x7fba8c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:51.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.676+0000 7fba93600640 1 -- 172.21.15.44:0/1162593120 shutdown_connections 2024-05-27T09:42:51.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.676+0000 7fba93600640 1 -- 172.21.15.44:0/1162593120 wait complete. 2024-05-27T09:42:51.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.677+0000 7fba93600640 1 Processor -- start 2024-05-27T09:42:51.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.677+0000 7fba93600640 1 -- start start 2024-05-27T09:42:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.678+0000 7fba93600640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba8c102b80 0x7fba8c1a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.678+0000 7fba93600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba8c1075d0 0x7fba8c1a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.678+0000 7fba93600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba8c108000 0x7fba8c1a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.678+0000 7fba93600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c11a420 con 0x7fba8c108000 2024-05-27T09:42:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.678+0000 7fba93600640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c11a2a0 con 0x7fba8c102b80 2024-05-27T09:42:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.678+0000 7fba93600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c11a5a0 con 0x7fba8c1075d0 2024-05-27T09:42:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.678+0000 7fba8be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba8c1075d0 0x7fba8c1a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.678+0000 7fba90c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba8c102b80 0x7fba8c1a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.678+0000 7fba8be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba8c1075d0 0x7fba8c1a13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44200/0 (socket says 172.21.15.44:44200) 2024-05-27T09:42:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.678+0000 7fba8be00640 1 -- 172.21.15.44:0/566342133 learned_addr learned my addr 172.21.15.44:0/566342133 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.679+0000 7fba91600640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba8c108000 0x7fba8c1a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.679+0000 7fba8be00640 1 -- 172.21.15.44:0/566342133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba8c102b80 msgr2=0x7fba8c1a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:51.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.679+0000 7fba8be00640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba8c102b80 0x7fba8c1a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.679+0000 7fba8be00640 1 -- 172.21.15.44:0/566342133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba8c108000 msgr2=0x7fba8c1a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:51.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.679+0000 7fba8be00640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba8c108000 0x7fba8c1a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.679+0000 7fba8be00640 1 -- 172.21.15.44:0/566342133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8c1a5cc0 con 0x7fba8c1075d0 2024-05-27T09:42:51.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.679+0000 7fba90c00640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba8c102b80 0x7fba8c1a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:51.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.679+0000 7fba8be00640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba8c1075d0 0x7fba8c1a13c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fba7800e9f0 tx=0x7fba7800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:51.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.680+0000 7fba89600640 1 -- 172.21.15.44:0/566342133 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7800cd80 con 0x7fba8c1075d0 2024-05-27T09:42:51.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.680+0000 7fba89600640 1 -- 172.21.15.44:0/566342133 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba780108e0 con 0x7fba8c1075d0 2024-05-27T09:42:51.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.680+0000 7fba93600640 1 -- 172.21.15.44:0/566342133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba8c1a5fb0 con 0x7fba8c1075d0 2024-05-27T09:42:51.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.680+0000 7fba89600640 1 -- 172.21.15.44:0/566342133 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba78018980 con 0x7fba8c1075d0 2024-05-27T09:42:51.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.680+0000 7fba93600640 1 -- 172.21.15.44:0/566342133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba8c1ad930 con 0x7fba8c1075d0 2024-05-27T09:42:51.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.684+0000 7fba9a5a8740 1 Processor -- start 2024-05-27T09:42:51.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.684+0000 7fba9a5a8740 1 -- start start 2024-05-27T09:42:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.685+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead06860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.685+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead06da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.685+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead072e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.685+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5640eac77b90 con 0x5640ea9641b0 2024-05-27T09:42:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.685+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5640eac67730 con 0x5640ea93b2f0 2024-05-27T09:42:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.685+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5640eac6b770 con 0x5640ea9434d0 2024-05-27T09:42:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.686+0000 7fba91600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead06860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.686+0000 7fba91600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead06860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36710/0 (socket says 172.21.15.44:36710) 2024-05-27T09:42:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.686+0000 7fba91600640 1 -- 172.21.15.44:0/1023730284 learned_addr learned my addr 172.21.15.44:0/1023730284 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.686+0000 7fba90c00640 1 --2- 172.21.15.44:0/1023730284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead06da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.686+0000 7fba89600640 1 -- 172.21.15.44:0/566342133 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fba78010430 con 0x7fba8c1075d0 2024-05-27T09:42:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.686+0000 7fba8be00640 1 --2- 172.21.15.44:0/1023730284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead072e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.687+0000 7fba90c00640 1 -- 172.21.15.44:0/1023730284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 msgr2=0x5640ead072e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.687+0000 7fba90c00640 1 --2- 172.21.15.44:0/1023730284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead072e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.687+0000 7fba90c00640 1 -- 172.21.15.44:0/1023730284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 msgr2=0x5640ead06860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.687+0000 7fba90c00640 1 --2- 172.21.15.44:0/1023730284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead06860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.687+0000 7fba90c00640 1 -- 172.21.15.44:0/1023730284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640eac008b0 con 0x5640ea93b2f0 2024-05-27T09:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.687+0000 7fba8be00640 1 --2- 172.21.15.44:0/1023730284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.687+0000 7fba90c00640 1 --2- 172.21.15.44:0/1023730284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead06da0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fba80005940 tx=0x7fba80005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7147d397000b2b17 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.687+0000 7fba7dc00640 1 -- 172.21.15.44:0/1023730284 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80037040 con 0x5640ea93b2f0 2024-05-27T09:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.687+0000 7fba7dc00640 1 -- 172.21.15.44:0/1023730284 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba80030450 con 0x5640ea93b2f0 2024-05-27T09:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.687+0000 7fba7dc00640 1 -- 172.21.15.44:0/1023730284 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80032b70 con 0x5640ea93b2f0 2024-05-27T09:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.689+0000 7fba9a5a8740 1 -- 172.21.15.44:0/1023730284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 msgr2=0x5640ead06da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.689+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/1023730284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead06da0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fba80005940 tx=0x7fba80005d90 comp rx=0 tx=0).stop 2024-05-27T09:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.689+0000 7fba89600640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fba740b29d0 0x7fba740b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.689+0000 7fba9a5a8740 1 -- 172.21.15.44:0/1023730284 shutdown_connections 2024-05-27T09:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.689+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/1023730284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.689+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/1023730284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead06da0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.689+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/1023730284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead06860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.689+0000 7fba9a5a8740 1 -- 172.21.15.44:0/1023730284 >> 172.21.15.44:0/1023730284 conn(0x5640ea958300 msgr2=0x5640ead03010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.689+0000 7fba89600640 1 -- 172.21.15.44:0/566342133 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fba780e2990 con 0x7fba8c1075d0 2024-05-27T09:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.690+0000 7fba9a5a8740 1 -- 172.21.15.44:0/1023730284 shutdown_connections 2024-05-27T09:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.690+0000 7fba91600640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fba740b29d0 0x7fba740b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.690+0000 7fba9a5a8740 1 -- 172.21.15.44:0/1023730284 wait complete. 2024-05-27T09:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.690+0000 7fba9a5a8740 1 -- start start 2024-05-27T09:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.691+0000 7fba9a5a8740 1 Processor -- start 2024-05-27T09:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.691+0000 7fba91600640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fba740b29d0 0x7fba740b4ee0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fba84005fd0 tx=0x7fba84011040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.691+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead10cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.691+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead111f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.691+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead1cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.691+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5640ea94d790 con 0x5640ea9641b0 2024-05-27T09:42:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.691+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5640eac0fbc0 con 0x5640ea93b2f0 2024-05-27T09:42:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.691+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5640ead13930 con 0x5640ea9434d0 2024-05-27T09:42:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.692+0000 7fba9a5a8740 10 client.0 _ll_register_callbacks cb 0x5640eabed400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.692+0000 7fba90c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead10cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.692+0000 7fba90c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead10cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51494/0 (socket says 172.21.15.44:51494) 2024-05-27T09:42:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.692+0000 7fba90c00640 1 -- 172.21.15.44:0/2849769958 learned_addr learned my addr 172.21.15.44:0/2849769958 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.692+0000 7fba8be00640 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead111f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.692+0000 7fba91600640 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead1cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.692+0000 7fba90c00640 1 -- 172.21.15.44:0/2849769958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 msgr2=0x5640ead111f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.692+0000 7fba90c00640 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead111f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.692+0000 7fba90c00640 1 -- 172.21.15.44:0/2849769958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 msgr2=0x5640ead1cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.692+0000 7fba90c00640 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead1cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.692+0000 7fba90c00640 1 -- 172.21.15.44:0/2849769958 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640ea97c210 con 0x5640ea93b2f0 2024-05-27T09:42:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.693+0000 7fba90c00640 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead10cb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fba8002a860 tx=0x7fba80030970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1eacd4a54efded9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.693+0000 7fba72000640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.693+0000 7fba72000640 1 -- 172.21.15.44:0/2849769958 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80037030 con 0x5640ea93b2f0 2024-05-27T09:42:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.693+0000 7fba72000640 1 -- 172.21.15.44:0/2849769958 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fba80004240 con 0x5640ea93b2f0 2024-05-27T09:42:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.693+0000 7fba9a5a8740 1 -- 172.21.15.44:0/2849769958 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba80030450 con 0x5640ea93b2f0 2024-05-27T09:42:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.693+0000 7fba72000640 1 -- 172.21.15.44:0/2849769958 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fba80054050 con 0x5640ea93b2f0 2024-05-27T09:42:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.693+0000 7fba9a5a8740 10 client.26654 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.693+0000 7fba9a5a8740 1 -- 172.21.15.44:0/2849769958 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5640eaa48620 con 0x5640ea93b2f0 2024-05-27T09:42:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.694+0000 7fba72000640 1 -- 172.21.15.44:0/2849769958 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba80030450 con 0x5640ea93b2f0 2024-05-27T09:42:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.694+0000 7fba72000640 1 -- 172.21.15.44:0/2849769958 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fba800046c0 con 0x5640ea93b2f0 2024-05-27T09:42:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.694+0000 7fba9a5a8740 10 client.26654 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.694+0000 7fba9a5a8740 10 client.26654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.694+0000 7fba9a5a8740 20 client.26654 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.694+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5640ea922d80 0x5640ead22a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.694+0000 7fba9a5a8740 4 client.26654 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.694+0000 7fba9a5a8740 1 -- 172.21.15.44:0/2849769958 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640eac79ae0 con 0x5640ea922d80 2024-05-27T09:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.694+0000 7fba8be00640 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5640ea922d80 0x5640ead22a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.695+0000 7fba8be00640 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5640ea922d80 0x5640ead22a50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bb22c05ee088c44 server_cookie=f2c0559797690f6e in_seq=0 out_seq=0 2024-05-27T09:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.695+0000 7fba72000640 10 client.26654 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba72000640 1 -- 172.21.15.44:0/2849769958 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5640eac79ae0 con 0x5640ea922d80 2024-05-27T09:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba72000640 10 client.26654 handle_command_reply: tid=0 2024-05-27T09:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba9a5a8740 1 client.26654 shutdown 2024-05-27T09:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba9a5a8740 1 -- 172.21.15.44:0/2849769958 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 msgr2=0x5640ead10cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead10cb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fba8002a860 tx=0x7fba80030970 comp rx=0 tx=0).stop 2024-05-27T09:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba9a5a8740 1 -- 172.21.15.44:0/2849769958 shutdown_connections 2024-05-27T09:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5640ea922d80 0x5640ead22a50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead1cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead111f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/2849769958 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead10cb0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba72000640 0 client.26654 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.696+0000 7fba9a5a8740 1 -- 172.21.15.44:0/2849769958 >> 172.21.15.44:0/2849769958 conn(0x5640ea958300 msgr2=0x5640ead08580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.697+0000 7fba9a5a8740 1 -- 172.21.15.44:0/2849769958 shutdown_connections 2024-05-27T09:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.697+0000 7fba9a5a8740 1 -- 172.21.15.44:0/2849769958 wait complete. 2024-05-27T09:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.697+0000 7fba9a5a8740 20 client.26654 trim_cache size 0 max 16384 2024-05-27T09:42:51.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.720+0000 7fba9a5a8740 1 Processor -- start 2024-05-27T09:42:51.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.720+0000 7fba9a5a8740 1 -- start start 2024-05-27T09:42:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.721+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead1cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.721+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead25a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.721+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead25fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.721+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1b89c0 con 0x5640ea9641b0 2024-05-27T09:42:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.721+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1b6260 con 0x5640ea93b2f0 2024-05-27T09:42:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.721+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1b40d0 con 0x5640ea9434d0 2024-05-27T09:42:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.721+0000 7fba90c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead1cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.721+0000 7fba8be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead25a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.721+0000 7fba8be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead25a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44224/0 (socket says 172.21.15.44:44224) 2024-05-27T09:42:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.721+0000 7fba8be00640 1 -- 172.21.15.44:0/3690062036 learned_addr learned my addr 172.21.15.44:0/3690062036 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.721+0000 7fba90c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead1cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51504/0 (socket says 172.21.15.44:51504) 2024-05-27T09:42:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.722+0000 7fba91600640 1 --2- 172.21.15.44:0/3690062036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead25fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.722+0000 7fba8be00640 1 -- 172.21.15.44:0/3690062036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 msgr2=0x5640ead1cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.722+0000 7fba8be00640 1 --2- 172.21.15.44:0/3690062036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead1cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.722+0000 7fba8be00640 1 -- 172.21.15.44:0/3690062036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 msgr2=0x5640ead25fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.722+0000 7fba8be00640 1 --2- 172.21.15.44:0/3690062036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead25fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.722+0000 7fba8be00640 1 -- 172.21.15.44:0/3690062036 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640ead1daa0 con 0x5640ea9434d0 2024-05-27T09:42:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.722+0000 7fba8be00640 1 --2- 172.21.15.44:0/3690062036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead25a70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fba780a5cf0 tx=0x7fba780cc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75ec87791fb3d52e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:51.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.723+0000 7fba7d200640 1 -- 172.21.15.44:0/3690062036 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba780aa370 con 0x5640ea9434d0 2024-05-27T09:42:51.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.723+0000 7fba7d200640 1 -- 172.21.15.44:0/3690062036 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba780a9c70 con 0x5640ea9434d0 2024-05-27T09:42:51.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.723+0000 7fba7d200640 1 -- 172.21.15.44:0/3690062036 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba780a5f00 con 0x5640ea9434d0 2024-05-27T09:42:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.724+0000 7fba9a5a8740 1 -- 172.21.15.44:0/3690062036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 msgr2=0x5640ead25a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.724+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/3690062036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead25a70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fba780a5cf0 tx=0x7fba780cc040 comp rx=0 tx=0).stop 2024-05-27T09:42:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.724+0000 7fba9a5a8740 1 -- 172.21.15.44:0/3690062036 shutdown_connections 2024-05-27T09:42:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.724+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/3690062036 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead25fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.724+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/3690062036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead25a70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.724+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/3690062036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead1cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.724+0000 7fba9a5a8740 1 -- 172.21.15.44:0/3690062036 >> 172.21.15.44:0/3690062036 conn(0x5640ea958300 msgr2=0x5640ead10d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.724+0000 7fba9a5a8740 1 -- 172.21.15.44:0/3690062036 shutdown_connections 2024-05-27T09:42:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.724+0000 7fba9a5a8740 1 -- 172.21.15.44:0/3690062036 wait complete. 2024-05-27T09:42:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.724+0000 7fba9a5a8740 1 -- start start 2024-05-27T09:42:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.725+0000 7fba9a5a8740 1 Processor -- start 2024-05-27T09:42:51.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.725+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead22bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.725+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead23100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba9a5a8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead88e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1b4720 con 0x5640ea9641b0 2024-05-27T09:42:51.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1b5b90 con 0x5640ea93b2f0 2024-05-27T09:42:51.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba9a5a8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1b4df0 con 0x5640ea9434d0 2024-05-27T09:42:51.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba9a5a8740 10 client.0 _ll_register_callbacks cb 0x7fba84009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:51.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba90c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead22bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba90c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead22bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:51518/0 (socket says 172.21.15.44:51518) 2024-05-27T09:42:51.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba90c00640 1 -- 172.21.15.44:0/4163551854 learned_addr learned my addr 172.21.15.44:0/4163551854 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:51.728 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba8be00640 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead23100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba91600640 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead88e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba90c00640 1 -- 172.21.15.44:0/4163551854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 msgr2=0x5640ead23100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:51.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba90c00640 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead23100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba90c00640 1 -- 172.21.15.44:0/4163551854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 msgr2=0x5640ead88e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:51.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba90c00640 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead88e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:51.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba90c00640 1 -- 172.21.15.44:0/4163551854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640ead239d0 con 0x5640ea93b2f0 2024-05-27T09:42:51.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.726+0000 7fba91600640 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead88e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:42:51.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.727+0000 7fba90c00640 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead22bc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fba80004c20 tx=0x7fba800048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a270c64caa309bd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:51.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.727+0000 7fba72000640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:51.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.727+0000 7fba72000640 1 -- 172.21.15.44:0/4163551854 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8005c020 con 0x5640ea93b2f0 2024-05-27T09:42:51.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.727+0000 7fba72000640 1 -- 172.21.15.44:0/4163551854 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7fba80032c00 con 0x5640ea93b2f0 2024-05-27T09:42:51.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.727+0000 7fba9a5a8740 1 -- 172.21.15.44:0/4163551854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba780a9c70 con 0x5640ea93b2f0 2024-05-27T09:42:51.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.727+0000 7fba72000640 1 -- 172.21.15.44:0/4163551854 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fba80067050 con 0x5640ea93b2f0 2024-05-27T09:42:51.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.728+0000 7fba9a5a8740 10 client.26660 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:51.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.728+0000 7fba9a5a8740 1 -- 172.21.15.44:0/4163551854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5640ead8a460 con 0x5640ea93b2f0 2024-05-27T09:42:51.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.728+0000 7fba72000640 1 -- 172.21.15.44:0/4163551854 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba780a9c70 con 0x5640ea93b2f0 2024-05-27T09:42:51.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.728+0000 7fba72000640 1 -- 172.21.15.44:0/4163551854 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fba80005420 con 0x5640ea93b2f0 2024-05-27T09:42:51.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.728+0000 7fba9a5a8740 10 client.26660 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:51.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.728+0000 7fba9a5a8740 10 client.26660 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:51.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.728+0000 7fba9a5a8740 20 client.26660 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:51.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.728+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5640ea8f81c0 0x5640ead8cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:51.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.728+0000 7fba9a5a8740 4 client.26660 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:42:51.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.729+0000 7fba9a5a8740 1 -- 172.21.15.44:0/4163551854 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640eac732c0 con 0x5640ea8f81c0 2024-05-27T09:42:51.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.729+0000 7fba8be00640 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5640ea8f81c0 0x5640ead8cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:51.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.729+0000 7fba8be00640 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5640ea8f81c0 0x5640ead8cbb0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2e8eeaf497b3fe1 server_cookie=8935a5101cfb3ee9 in_seq=0 out_seq=0 2024-05-27T09:42:51.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:51.729+0000 7fba72000640 10 client.26660 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:52.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:51 smithi044 ceph-mon[31618]: pgmap v234: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 19 op/s 2024-05-27T09:42:52.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:51 smithi050 ceph-mon[36371]: pgmap v234: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 19 op/s 2024-05-27T09:42:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:51 smithi189 ceph-mon[36671]: pgmap v234: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 19 op/s 2024-05-27T09:42:54.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:53 smithi044 ceph-mon[31618]: pgmap v235: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2024-05-27T09:42:54.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:53 smithi050 ceph-mon[36371]: pgmap v235: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2024-05-27T09:42:54.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:53 smithi189 ceph-mon[36671]: pgmap v235: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2024-05-27T09:42:54.748 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:42:55.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:54 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:42:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.239+0000 7fba72000640 1 -- 172.21.15.44:0/4163551854 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5640eac732c0 con 0x5640ea8f81c0 2024-05-27T09:42:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.239+0000 7fba72000640 10 client.26660 handle_command_reply: tid=0 2024-05-27T09:42:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.240+0000 7fba9a5a8740 1 client.26660 shutdown 2024-05-27T09:42:55.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.240+0000 7fba9a5a8740 1 -- 172.21.15.44:0/4163551854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 msgr2=0x5640ead22bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:55.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.240+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead22bc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fba80004c20 tx=0x7fba800048f0 comp rx=0 tx=0).stop 2024-05-27T09:42:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.241+0000 7fba9a5a8740 1 -- 172.21.15.44:0/4163551854 shutdown_connections 2024-05-27T09:42:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.241+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5640ea8f81c0 0x5640ead8cbb0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.241+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640ea9641b0 0x5640ead88e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.241+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5640ea9434d0 0x5640ead23100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.241+0000 7fba9a5a8740 1 --2- 172.21.15.44:0/4163551854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5640ea93b2f0 0x5640ead22bc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.241+0000 7fba9a5a8740 1 -- 172.21.15.44:0/4163551854 >> 172.21.15.44:0/4163551854 conn(0x5640ea958300 msgr2=0x5640ead067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.241+0000 7fba72000640 0 client.26660 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.241+0000 7fba9a5a8740 1 -- 172.21.15.44:0/4163551854 shutdown_connections 2024-05-27T09:42:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.242+0000 7fba9a5a8740 1 -- 172.21.15.44:0/4163551854 wait complete. 2024-05-27T09:42:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.242+0000 7fba9a5a8740 20 client.26660 trim_cache size 0 max 16384 2024-05-27T09:42:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:42:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:42:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:42:55.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.246+0000 7fba93600640 1 -- 172.21.15.44:0/566342133 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fba740b29d0 msgr2=0x7fba740b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:55.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.246+0000 7fba93600640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fba740b29d0 0x7fba740b4ee0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fba84005fd0 tx=0x7fba84011040 comp rx=0 tx=0).stop 2024-05-27T09:42:55.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.246+0000 7fba93600640 1 -- 172.21.15.44:0/566342133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba8c1075d0 msgr2=0x7fba8c1a13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:55.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.246+0000 7fba93600640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba8c1075d0 0x7fba8c1a13c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fba7800e9f0 tx=0x7fba7800eeb0 comp rx=0 tx=0).stop 2024-05-27T09:42:55.247 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.246+0000 7fba93600640 1 -- 172.21.15.44:0/566342133 shutdown_connections 2024-05-27T09:42:55.247 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.246+0000 7fba93600640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fba740b29d0 0x7fba740b4ee0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:55.247 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.246+0000 7fba93600640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba8c108000 0x7fba8c1a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:55.247 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.246+0000 7fba93600640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba8c1075d0 0x7fba8c1a13c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:55.247 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.246+0000 7fba93600640 1 --2- 172.21.15.44:0/566342133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba8c102b80 0x7fba8c1a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:55.247 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.246+0000 7fba93600640 1 -- 172.21.15.44:0/566342133 >> 172.21.15.44:0/566342133 conn(0x7fba8c0782d0 msgr2=0x7fba8c105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.247+0000 7fba93600640 1 -- 172.21.15.44:0/566342133 shutdown_connections 2024-05-27T09:42:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:42:55.247+0000 7fba93600640 1 -- 172.21.15.44:0/566342133 wait complete. 2024-05-27T09:42:55.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:54 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:42:55.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:54 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:42:55.516 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:42:55.967 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:55 smithi044 ceph-mon[31618]: pgmap v236: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2024-05-27T09:42:56.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:55 smithi050 ceph-mon[36371]: pgmap v236: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2024-05-27T09:42:56.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:55 smithi189 ceph-mon[36671]: pgmap v236: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2024-05-27T09:42:57.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:56 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:42:57.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:56 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:42:57.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:56 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:42:57.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:56 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:42:57.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:56 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:42:57.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:56 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:42:57.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:56 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:42:57.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:56 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:42:57.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:56 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:42:57.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.809+0000 7f7287600700 1 -- 172.21.15.44:0/1481456086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72801054f0 msgr2=0x7f7280105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:57.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.809+0000 7f7287600700 1 --2- 172.21.15.44:0/1481456086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72801054f0 0x7f7280105900 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7274009990 tx=0x7f7274009ca0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.809+0000 7f7287600700 1 -- 172.21.15.44:0/1481456086 shutdown_connections 2024-05-27T09:42:57.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.809+0000 7f7287600700 1 --2- 172.21.15.44:0/1481456086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f728010e8d0 0x7f7280110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.809+0000 7f7287600700 1 --2- 172.21.15.44:0/1481456086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72801054f0 0x7f7280105900 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.809+0000 7f7287600700 1 --2- 172.21.15.44:0/1481456086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7280105e40 0x7f728010e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.809+0000 7f7287600700 1 -- 172.21.15.44:0/1481456086 >> 172.21.15.44:0/1481456086 conn(0x7f72800fedb0 msgr2=0x7f72801011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:57.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.809+0000 7f7287600700 1 -- 172.21.15.44:0/1481456086 shutdown_connections 2024-05-27T09:42:57.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.810+0000 7f7287600700 1 -- 172.21.15.44:0/1481456086 wait complete. 2024-05-27T09:42:57.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.811+0000 7f7287600700 1 Processor -- start 2024-05-27T09:42:57.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.811+0000 7f7287600700 1 -- start start 2024-05-27T09:42:57.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.812+0000 7f7287600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72801054f0 0x7f728019d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.812+0000 7f7287600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7280105e40 0x7f728019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.812+0000 7f7287600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f728010e8d0 0x7f72801a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.812+0000 7f7287600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7280114470 con 0x7f72801054f0 2024-05-27T09:42:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.812+0000 7f7287600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f72801142f0 con 0x7f728010e8d0 2024-05-27T09:42:57.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.812+0000 7f7287600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72801145f0 con 0x7f7280105e40 2024-05-27T09:42:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.812+0000 7f727fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7280105e40 0x7f728019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.812+0000 7f727fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7280105e40 0x7f728019d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:32838/0 (socket says 172.21.15.44:32838) 2024-05-27T09:42:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.812+0000 7f727fe00700 1 -- 172.21.15.44:0/1291385746 learned_addr learned my addr 172.21.15.44:0/1291385746 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.812+0000 7f7284c00700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72801054f0 0x7f728019d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.812+0000 7f7285600700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f728010e8d0 0x7f72801a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.813+0000 7f7285600700 1 -- 172.21.15.44:0/1291385746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7280105e40 msgr2=0x7f728019d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.813+0000 7f7285600700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7280105e40 0x7f728019d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.813+0000 7f7285600700 1 -- 172.21.15.44:0/1291385746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72801054f0 msgr2=0x7f728019d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.813+0000 7f7285600700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72801054f0 0x7f728019d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.813+0000 7f7285600700 1 -- 172.21.15.44:0/1291385746 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72801a1ed0 con 0x7f728010e8d0 2024-05-27T09:42:57.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.813+0000 7f7284c00700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72801054f0 0x7f728019d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:42:57.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.814+0000 7f7285600700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f728010e8d0 0x7f72801a1830 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f727800efc0 tx=0x7f727800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:57.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.814+0000 7f727d600700 1 -- 172.21.15.44:0/1291385746 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72780092e0 con 0x7f728010e8d0 2024-05-27T09:42:57.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.814+0000 7f727d600700 1 -- 172.21.15.44:0/1291385746 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7278004bb0 con 0x7f728010e8d0 2024-05-27T09:42:57.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.814+0000 7f727d600700 1 -- 172.21.15.44:0/1291385746 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7278004080 con 0x7f728010e8d0 2024-05-27T09:42:57.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.814+0000 7f7287600700 1 -- 172.21.15.44:0/1291385746 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72801a2160 con 0x7f728010e8d0 2024-05-27T09:42:57.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.814+0000 7f7287600700 1 -- 172.21.15.44:0/1291385746 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72801a2650 con 0x7f728010e8d0 2024-05-27T09:42:57.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.818+0000 7f727d600700 1 -- 172.21.15.44:0/1291385746 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f727801b030 con 0x7f728010e8d0 2024-05-27T09:42:57.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.820+0000 7f727d600700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72680b2a70 0x7f72680b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.820+0000 7f727d600700 1 -- 172.21.15.44:0/1291385746 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f72780e2150 con 0x7f728010e8d0 2024-05-27T09:42:57.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.821+0000 7f7284c00700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72680b2a70 0x7f72680b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.821+0000 7f72960dcb80 1 Processor -- start 2024-05-27T09:42:57.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.821+0000 7f72960dcb80 1 -- start start 2024-05-27T09:42:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.821+0000 7f7284c00700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72680b2a70 0x7f72680b4f80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f727400b2e0 tx=0x7f7274005fb0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 0x55de69cc47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cbe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cbedb0 0x55de69cbf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f72960dcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55de69bece40 con 0x55de69b3fe40 2024-05-27T09:42:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f72960dcb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55de69bdc9e0 con 0x55de69cbedb0 2024-05-27T09:42:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f72960dcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55de69be0a20 con 0x55de69cc4d30 2024-05-27T09:42:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f727fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cbe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f7284c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 0x55de69cc47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f727fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cbe870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:32852/0 (socket says 172.21.15.44:32852) 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f727fe00700 1 -- 172.21.15.44:0/1278648658 learned_addr learned my addr 172.21.15.44:0/1278648658 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f7285600700 1 --2- 172.21.15.44:0/1278648658 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cbedb0 0x55de69cbf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f727fe00700 1 -- 172.21.15.44:0/1278648658 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cbedb0 msgr2=0x55de69cbf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f727fe00700 1 --2- 172.21.15.44:0/1278648658 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cbedb0 0x55de69cbf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f727fe00700 1 -- 172.21.15.44:0/1278648658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 msgr2=0x55de69cc47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f727fe00700 1 --2- 172.21.15.44:0/1278648658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 0x55de69cc47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.822+0000 7f727fe00700 1 -- 172.21.15.44:0/1278648658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de69ba5420 con 0x55de69cc4d30 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.823+0000 7f7284c00700 1 --2- 172.21.15.44:0/1278648658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 0x55de69cc47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.823+0000 7f727fe00700 1 --2- 172.21.15.44:0/1278648658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cbe870 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f726c009df0 tx=0x7f726c011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3990cc9dbc90be05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.823+0000 7f7272000700 1 -- 172.21.15.44:0/1278648658 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c004230 con 0x55de69cc4d30 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.823+0000 7f7272000700 1 -- 172.21.15.44:0/1278648658 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f726c0043d0 con 0x55de69cc4d30 2024-05-27T09:42:57.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.823+0000 7f7272000700 1 -- 172.21.15.44:0/1278648658 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c0056c0 con 0x55de69cc4d30 2024-05-27T09:42:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.824+0000 7f72960dcb80 1 -- 172.21.15.44:0/1278648658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 msgr2=0x55de69cbe870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.824+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1278648658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cbe870 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f726c009df0 tx=0x7f726c011720 comp rx=0 tx=0).stop 2024-05-27T09:42:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.824+0000 7f72960dcb80 1 -- 172.21.15.44:0/1278648658 shutdown_connections 2024-05-27T09:42:57.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.824+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1278648658 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cbedb0 0x55de69cbf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.824+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1278648658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 0x55de69cc47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.824+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1278648658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cbe870 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.824+0000 7f72960dcb80 1 -- 172.21.15.44:0/1278648658 >> 172.21.15.44:0/1278648658 conn(0x55de69b9ffc0 msgr2=0x55de69cb98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:57.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.824+0000 7f72960dcb80 1 -- 172.21.15.44:0/1278648658 shutdown_connections 2024-05-27T09:42:57.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.824+0000 7f72960dcb80 1 -- 172.21.15.44:0/1278648658 wait complete. 2024-05-27T09:42:57.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.825+0000 7f72960dcb80 1 -- start start 2024-05-27T09:42:57.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.825+0000 7f72960dcb80 1 Processor -- start 2024-05-27T09:42:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 0x55de69cc84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cbedb0 0x55de69cc89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cc8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f72960dcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55de69ae2fa0 con 0x55de69b3fe40 2024-05-27T09:42:57.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f72960dcb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55de69b7d760 con 0x55de69cbedb0 2024-05-27T09:42:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f72960dcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55de69ccb800 con 0x55de69cc4d30 2024-05-27T09:42:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f72960dcb80 10 client.0 _ll_register_callbacks cb 0x55de69b86c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f7285600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cc8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f7285600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cc8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:32866/0 (socket says 172.21.15.44:32866) 2024-05-27T09:42:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f7285600700 1 -- 172.21.15.44:0/1339734174 learned_addr learned my addr 172.21.15.44:0/1339734174 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f7284c00700 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 0x55de69cc84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.826+0000 7f727fe00700 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cbedb0 0x55de69cc89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.827+0000 7f7285600700 1 -- 172.21.15.44:0/1339734174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cbedb0 msgr2=0x55de69cc89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.827+0000 7f7285600700 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cbedb0 0x55de69cc89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.827+0000 7f7285600700 1 -- 172.21.15.44:0/1339734174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 msgr2=0x55de69cc84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:57.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.827+0000 7f7285600700 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 0x55de69cc84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.827+0000 7f7285600700 1 -- 172.21.15.44:0/1339734174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de69b2c8d0 con 0x55de69cc4d30 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.827+0000 7f7284c00700 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 0x55de69cc84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.827+0000 7f7285600700 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cc8f30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f72780a66d0 tx=0x7f72780b2080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cebed26968feec8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.827+0000 7f7266a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.827+0000 7f7266a00700 1 -- 172.21.15.44:0/1339734174 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72780c3060 con 0x55de69cc4d30 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.827+0000 7f72960dcb80 1 -- 172.21.15.44:0/1339734174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f726c0043d0 con 0x55de69cc4d30 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.827+0000 7f7266a00700 1 -- 172.21.15.44:0/1339734174 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f72780a88d0 con 0x55de69cc4d30 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.828+0000 7f7266a00700 1 -- 172.21.15.44:0/1339734174 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f72780ca050 con 0x55de69cc4d30 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.828+0000 7f72960dcb80 10 client.26665 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.828+0000 7f72960dcb80 1 -- 172.21.15.44:0/1339734174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55de69bad060 con 0x55de69cc4d30 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.828+0000 7f7266a00700 1 -- 172.21.15.44:0/1339734174 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f726c0043d0 con 0x55de69cc4d30 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.828+0000 7f7266a00700 1 -- 172.21.15.44:0/1339734174 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f72780ca1f0 con 0x55de69cc4d30 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.828+0000 7f72960dcb80 10 client.26665 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:57.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.828+0000 7f72960dcb80 10 client.26665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:57.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.828+0000 7f72960dcb80 20 client.26665 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:57.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.828+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de69cd8650 0x55de69cdaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.828+0000 7f72960dcb80 4 client.26665 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:42:57.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.828+0000 7f72960dcb80 1 -- 172.21.15.44:0/1339734174 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de69beed90 con 0x55de69cd8650 2024-05-27T09:42:57.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.829+0000 7f7284c00700 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de69cd8650 0x55de69cdaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.829+0000 7f7284c00700 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de69cd8650 0x55de69cdaa70 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31dfe184fb1ef5e7 server_cookie=91ffa74d0685248f in_seq=0 out_seq=0 2024-05-27T09:42:57.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.829+0000 7f7266a00700 10 client.26665 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:57.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.830+0000 7f7266a00700 1 -- 172.21.15.44:0/1339734174 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55de69beed90 con 0x55de69cd8650 2024-05-27T09:42:57.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.830+0000 7f7266a00700 10 client.26665 handle_command_reply: tid=0 2024-05-27T09:42:57.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.830+0000 7f72960dcb80 1 client.26665 shutdown 2024-05-27T09:42:57.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.830+0000 7f72960dcb80 1 -- 172.21.15.44:0/1339734174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 msgr2=0x55de69cc8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:57.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.830+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cc8f30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f72780a66d0 tx=0x7f72780b2080 comp rx=0 tx=0).stop 2024-05-27T09:42:57.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.831+0000 7f72960dcb80 1 -- 172.21.15.44:0/1339734174 shutdown_connections 2024-05-27T09:42:57.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.831+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de69cd8650 0x55de69cdaa70 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.831+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69cc4d30 0x55de69cc8f30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.831+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69b3fe40 0x55de69cc84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.831+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1339734174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cbedb0 0x55de69cc89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.831+0000 7f7266a00700 0 client.26665 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:57.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.831+0000 7f72960dcb80 1 -- 172.21.15.44:0/1339734174 >> 172.21.15.44:0/1339734174 conn(0x55de69b9ffc0 msgr2=0x55de69cb9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:57.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.831+0000 7f72960dcb80 1 -- 172.21.15.44:0/1339734174 shutdown_connections 2024-05-27T09:42:57.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.831+0000 7f72960dcb80 1 -- 172.21.15.44:0/1339734174 wait complete. 2024-05-27T09:42:57.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.831+0000 7f72960dcb80 20 client.26665 trim_cache size 0 max 16384 2024-05-27T09:42:57.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.857+0000 7f72960dcb80 1 Processor -- start 2024-05-27T09:42:57.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.857+0000 7f72960dcb80 1 -- start start 2024-05-27T09:42:57.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cbe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69cbedb0 0x55de69cca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 0x55de69cd4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f72960dcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72801b39a0 con 0x55de69cbedb0 2024-05-27T09:42:57.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f72960dcb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f72801b1200 con 0x55de69cc4d30 2024-05-27T09:42:57.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f72960dcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72801af070 con 0x55de69b3fe40 2024-05-27T09:42:57.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f7285600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 0x55de69cd4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f7284c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cbe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f7284c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cbe870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:32874/0 (socket says 172.21.15.44:32874) 2024-05-27T09:42:57.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f7285600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 0x55de69cd4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:41360/0 (socket says 172.21.15.44:41360) 2024-05-27T09:42:57.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f727fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69cbedb0 0x55de69cca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.858+0000 7f7284c00700 1 -- 172.21.15.44:0/1481239872 learned_addr learned my addr 172.21.15.44:0/1481239872 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:57.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.859+0000 7f7284c00700 1 -- 172.21.15.44:0/1481239872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 msgr2=0x55de69cd4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:57.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.859+0000 7f7284c00700 1 --2- 172.21.15.44:0/1481239872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 0x55de69cd4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.859+0000 7f7284c00700 1 -- 172.21.15.44:0/1481239872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69cbedb0 msgr2=0x55de69cca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:57.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.859+0000 7f7284c00700 1 --2- 172.21.15.44:0/1481239872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69cbedb0 0x55de69cca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.859+0000 7f7284c00700 1 -- 172.21.15.44:0/1481239872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de69cc6a40 con 0x55de69b3fe40 2024-05-27T09:42:57.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.859+0000 7f7284c00700 1 --2- 172.21.15.44:0/1481239872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cbe870 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7274009990 tx=0x7f727401e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dc922e4c9010e77 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.859+0000 7f7271600700 1 -- 172.21.15.44:0/1481239872 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7274037070 con 0x55de69b3fe40 2024-05-27T09:42:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.859+0000 7f7271600700 1 -- 172.21.15.44:0/1481239872 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7274031480 con 0x55de69b3fe40 2024-05-27T09:42:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.859+0000 7f7271600700 1 -- 172.21.15.44:0/1481239872 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72740236c0 con 0x55de69b3fe40 2024-05-27T09:42:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.860+0000 7f72960dcb80 1 -- 172.21.15.44:0/1481239872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 msgr2=0x55de69cbe870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.860+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1481239872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cbe870 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7274009990 tx=0x7f727401e000 comp rx=0 tx=0).stop 2024-05-27T09:42:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.860+0000 7f72960dcb80 1 -- 172.21.15.44:0/1481239872 shutdown_connections 2024-05-27T09:42:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.860+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1481239872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 0x55de69cd4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.860+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1481239872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cbe870 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.860+0000 7f72960dcb80 1 --2- 172.21.15.44:0/1481239872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69cbedb0 0x55de69cca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.860+0000 7f72960dcb80 1 -- 172.21.15.44:0/1481239872 >> 172.21.15.44:0/1481239872 conn(0x55de69b9ffc0 msgr2=0x55de69cc2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.861+0000 7f72960dcb80 1 -- 172.21.15.44:0/1481239872 shutdown_connections 2024-05-27T09:42:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.861+0000 7f72960dcb80 1 -- 172.21.15.44:0/1481239872 wait complete. 2024-05-27T09:42:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.861+0000 7f72960dcb80 1 -- start start 2024-05-27T09:42:57.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.861+0000 7f72960dcb80 1 Processor -- start 2024-05-27T09:42:57.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cd0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69cbedb0 0x55de69ccda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f72960dcb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 0x55de69ccdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f72960dcb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72801af6c0 con 0x55de69cbedb0 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f72960dcb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f72801b0b30 con 0x55de69cc4d30 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f72960dcb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72801afd90 con 0x55de69b3fe40 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f72960dcb80 10 client.0 _ll_register_callbacks cb 0x55de69b86c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f7284c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cd0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f7285600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 0x55de69ccdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f7284c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cd0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:32890/0 (socket says 172.21.15.44:32890) 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f727fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69cbedb0 0x55de69ccda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.862+0000 7f7284c00700 1 -- 172.21.15.44:0/37724550 learned_addr learned my addr 172.21.15.44:0/37724550 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f7285600700 1 -- 172.21.15.44:0/37724550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 msgr2=0x55de69cd0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f7285600700 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cd0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f7285600700 1 -- 172.21.15.44:0/37724550 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69cbedb0 msgr2=0x55de69ccda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f7285600700 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69cbedb0 0x55de69ccda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f7285600700 1 -- 172.21.15.44:0/37724550 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de69cce4a0 con 0x55de69cc4d30 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f7284c00700 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cd0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f7285600700 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 0x55de69ccdf60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f727800ca10 tx=0x7f72780b2230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=733a04ca7f3ac079 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f7266a00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f7266a00700 1 -- 172.21.15.44:0/37724550 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72780a8f70 con 0x55de69cc4d30 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f7266a00700 1 -- 172.21.15.44:0/37724550 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8521+0+0 (secure 0 0 0) 0x7f72780b0d10 con 0x55de69cc4d30 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f72960dcb80 1 -- 172.21.15.44:0/37724550 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7274031480 con 0x55de69cc4d30 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f7266a00700 1 -- 172.21.15.44:0/37724550 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f72780a9b60 con 0x55de69cc4d30 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f72960dcb80 10 client.26678 fetch_fsmap learned FSMap version 29 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.863+0000 7f72960dcb80 1 -- 172.21.15.44:0/37724550 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55de69d4c5a0 con 0x55de69cc4d30 2024-05-27T09:42:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.864+0000 7f7266a00700 1 -- 172.21.15.44:0/37724550 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7274031480 con 0x55de69cc4d30 2024-05-27T09:42:57.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.864+0000 7f7266a00700 1 -- 172.21.15.44:0/37724550 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f72780a4dd0 con 0x55de69cc4d30 2024-05-27T09:42:57.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.864+0000 7f72960dcb80 10 client.26678 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:42:57.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.864+0000 7f72960dcb80 10 client.26678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:42:57.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.864+0000 7f72960dcb80 20 client.26678 populate_metadata read hostname 'smithi044' 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.864+0000 7f72960dcb80 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de69d4cc80 0x55de69d4f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.864+0000 7f72960dcb80 4 client.26678 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.864+0000 7f72960dcb80 1 -- 172.21.15.44:0/37724550 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55de69be8570 con 0x55de69d4cc80 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.865+0000 7f7284c00700 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de69d4cc80 0x55de69d4f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.865+0000 7f7284c00700 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de69d4cc80 0x55de69d4f0a0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bd423e8f9b3f0bb server_cookie=7efba75e7b658750 in_seq=0 out_seq=0 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.865+0000 7f7266a00700 10 client.26678 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.865+0000 7f7266a00700 1 -- 172.21.15.44:0/37724550 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55de69be8570 con 0x55de69d4cc80 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.865+0000 7f7266a00700 10 client.26678 handle_command_reply: tid=0 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.865+0000 7f72960dcb80 1 client.26678 shutdown 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f72960dcb80 1 -- 172.21.15.44:0/37724550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 msgr2=0x55de69ccdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f72960dcb80 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 0x55de69ccdf60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f727800ca10 tx=0x7f72780b2230 comp rx=0 tx=0).stop 2024-05-27T09:42:57.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f72960dcb80 1 -- 172.21.15.44:0/37724550 shutdown_connections 2024-05-27T09:42:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f72960dcb80 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de69d4cc80 0x55de69d4f0a0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f72960dcb80 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de69cc4d30 0x55de69ccdf60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f72960dcb80 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de69b3fe40 0x55de69cd0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f72960dcb80 1 --2- 172.21.15.44:0/37724550 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de69cbedb0 0x55de69ccda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f72960dcb80 1 -- 172.21.15.44:0/37724550 >> 172.21.15.44:0/37724550 conn(0x55de69b9ffc0 msgr2=0x55de69cc5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f7266a00700 0 client.26678 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:42:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f72960dcb80 1 -- 172.21.15.44:0/37724550 shutdown_connections 2024-05-27T09:42:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f72960dcb80 1 -- 172.21.15.44:0/37724550 wait complete. 2024-05-27T09:42:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.866+0000 7f72960dcb80 20 client.26678 trim_cache size 0 max 16384 2024-05-27T09:42:57.871 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:42:57.872 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:42:57.872 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:42:57.872 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:42:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 -- 172.21.15.44:0/1291385746 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72680b2a70 msgr2=0x7f72680b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72680b2a70 0x7f72680b4f80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f727400b2e0 tx=0x7f7274005fb0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 -- 172.21.15.44:0/1291385746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f728010e8d0 msgr2=0x7f72801a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:42:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f728010e8d0 0x7f72801a1830 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f727800efc0 tx=0x7f727800c3a0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 -- 172.21.15.44:0/1291385746 shutdown_connections 2024-05-27T09:42:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f72680b2a70 0x7f72680b4f80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f728010e8d0 0x7f72801a1830 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72801054f0 0x7f728019d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 --2- 172.21.15.44:0/1291385746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7280105e40 0x7f728019d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:42:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 -- 172.21.15.44:0/1291385746 >> 172.21.15.44:0/1291385746 conn(0x7f72800fedb0 msgr2=0x7f7280103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:42:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 -- 172.21.15.44:0/1291385746 shutdown_connections 2024-05-27T09:42:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:42:57.867+0000 7f7287600700 1 -- 172.21.15.44:0/1291385746 wait complete. 2024-05-27T09:42:58.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:57 smithi044 ceph-mon[31618]: pgmap v237: 97 pgs: 97 active+clean; 54 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 465 op/s 2024-05-27T09:42:58.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:57 smithi050 ceph-mon[36371]: pgmap v237: 97 pgs: 97 active+clean; 54 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 465 op/s 2024-05-27T09:42:58.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:57 smithi189 ceph-mon[36671]: pgmap v237: 97 pgs: 97 active+clean; 54 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 465 op/s 2024-05-27T09:42:58.608 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:42:58.608 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:42:59.061 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:43:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:42:59 smithi044 ceph-mon[31618]: pgmap v238: 97 pgs: 97 active+clean; 54 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 457 op/s 2024-05-27T09:43:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:42:59 smithi050 ceph-mon[36371]: pgmap v238: 97 pgs: 97 active+clean; 54 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 457 op/s 2024-05-27T09:43:00.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:42:59 smithi189 ceph-mon[36671]: pgmap v238: 97 pgs: 97 active+clean; 54 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 457 op/s 2024-05-27T09:43:01.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.315+0000 7f84c3e00700 1 -- 172.21.15.44:0/587134481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84c4110a00 msgr2=0x7f84c4112e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:01.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.315+0000 7f84c3e00700 1 --2- 172.21.15.44:0/587134481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84c4110a00 0x7f84c4112e20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00b0a0 tx=0x7f84bc00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:43:01.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.315+0000 7f84c3e00700 1 -- 172.21.15.44:0/587134481 shutdown_connections 2024-05-27T09:43:01.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.315+0000 7f84c3e00700 1 --2- 172.21.15.44:0/587134481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84c4110a00 0x7f84c4112e20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:01.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.315+0000 7f84c3e00700 1 --2- 172.21.15.44:0/587134481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84c4075af0 0x7f84c4075f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:01.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.315+0000 7f84c3e00700 1 --2- 172.21.15.44:0/587134481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84c4076440 0x7f84c41104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:01.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.315+0000 7f84c3e00700 1 -- 172.21.15.44:0/587134481 >> 172.21.15.44:0/587134481 conn(0x7f84c4079d00 msgr2=0x7f84c407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:01.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.315+0000 7f84c3e00700 1 -- 172.21.15.44:0/587134481 shutdown_connections 2024-05-27T09:43:01.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.315+0000 7f84c3e00700 1 -- 172.21.15.44:0/587134481 wait complete. 2024-05-27T09:43:01.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.316+0000 7f84c3e00700 1 Processor -- start 2024-05-27T09:43:01.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.317+0000 7f84c3e00700 1 -- start start 2024-05-27T09:43:01.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.317+0000 7f84c3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84c4075af0 0x7f84c419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:01.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.317+0000 7f84c3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84c4076440 0x7f84c419d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.317+0000 7f84c3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84c4110a00 0x7f84c41a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.317+0000 7f84c3e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f84c4116510 con 0x7f84c4076440 2024-05-27T09:43:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.317+0000 7f84c3e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f84c4116390 con 0x7f84c4075af0 2024-05-27T09:43:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.317+0000 7f84c3e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f84c4116690 con 0x7f84c4110a00 2024-05-27T09:43:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.317+0000 7f84c2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84c4075af0 0x7f84c419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.317+0000 7f84c2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84c4075af0 0x7f84c419ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:41376/0 (socket says 172.21.15.44:41376) 2024-05-27T09:43:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.317+0000 7f84c2a00700 1 -- 172.21.15.44:0/4004948411 learned_addr learned my addr 172.21.15.44:0/4004948411 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.317+0000 7f84c3400700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84c4110a00 0x7f84c41a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.318+0000 7f84ba000700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84c4076440 0x7f84c419d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.318+0000 7f84c2a00700 1 -- 172.21.15.44:0/4004948411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84c4110a00 msgr2=0x7f84c41a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:01.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.318+0000 7f84c2a00700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84c4110a00 0x7f84c41a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:01.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.318+0000 7f84c2a00700 1 -- 172.21.15.44:0/4004948411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84c4076440 msgr2=0x7f84c419d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:01.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.318+0000 7f84c2a00700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84c4076440 0x7f84c419d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:01.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.318+0000 7f84c2a00700 1 -- 172.21.15.44:0/4004948411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c41a1cf0 con 0x7f84c4075af0 2024-05-27T09:43:01.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.318+0000 7f84ba000700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84c4076440 0x7f84c419d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:43:01.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.318+0000 7f84c2a00700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84c4075af0 0x7f84c419ce20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f84b400eb30 tx=0x7f84b400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:01.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.318+0000 7f84bbe00700 1 -- 172.21.15.44:0/4004948411 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b400cc90 con 0x7f84c4075af0 2024-05-27T09:43:01.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.319+0000 7f84c3e00700 1 -- 172.21.15.44:0/4004948411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c41a1fe0 con 0x7f84c4075af0 2024-05-27T09:43:01.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.319+0000 7f84bbe00700 1 -- 172.21.15.44:0/4004948411 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84b40044d0 con 0x7f84c4075af0 2024-05-27T09:43:01.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.319+0000 7f84bbe00700 1 -- 172.21.15.44:0/4004948411 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b4010430 con 0x7f84c4075af0 2024-05-27T09:43:01.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.319+0000 7f84c3e00700 1 -- 172.21.15.44:0/4004948411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c41a9900 con 0x7f84c4075af0 2024-05-27T09:43:01.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.320+0000 7f84c3e00700 1 -- 172.21.15.44:0/4004948411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c40512b0 con 0x7f84c4075af0 2024-05-27T09:43:01.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.323+0000 7f84bbe00700 1 -- 172.21.15.44:0/4004948411 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f84b40479e0 con 0x7f84c4075af0 2024-05-27T09:43:01.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.324+0000 7f84bbe00700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f84a80b29f0 0x7f84a80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:01.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.325+0000 7f84bbe00700 1 -- 172.21.15.44:0/4004948411 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7f84b40e2990 con 0x7f84c4075af0 2024-05-27T09:43:01.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.325+0000 7f84ba000700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f84a80b29f0 0x7f84a80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:01.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.325+0000 7f84ba000700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f84a80b29f0 0x7f84a80b4f00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84c41035a0 tx=0x7f84ac009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:01.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.329+0000 7f84bbe00700 1 -- 172.21.15.44:0/4004948411 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f84b40aa2e0 con 0x7f84c4075af0 2024-05-27T09:43:01.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.647+0000 7f84c3e00700 1 -- 172.21.15.44:0/4004948411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84c402e5e0 con 0x7f84c4075af0 2024-05-27T09:43:01.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.648+0000 7f84bbe00700 1 -- 172.21.15.44:0/4004948411 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11863 (secure 0 0 0) 0x7f84b4003680 con 0x7f84c4075af0 2024-05-27T09:43:01.650 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:43:01.650 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:41:21.007340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26297":{"gid":26297,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/102444567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":102444567},{"type":"v1","addr":"172.21.15.189:6835","nonce":102444567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:43:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.652+0000 7f84c3e00700 1 -- 172.21.15.44:0/4004948411 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f84a80b29f0 msgr2=0x7f84a80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.652+0000 7f84c3e00700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f84a80b29f0 0x7f84a80b4f00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84c41035a0 tx=0x7f84ac009260 comp rx=0 tx=0).stop 2024-05-27T09:43:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.652+0000 7f84c3e00700 1 -- 172.21.15.44:0/4004948411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84c4075af0 msgr2=0x7f84c419ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.652+0000 7f84c3e00700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84c4075af0 0x7f84c419ce20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f84b400eb30 tx=0x7f84b400ee40 comp rx=0 tx=0).stop 2024-05-27T09:43:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.652+0000 7f84c3e00700 1 -- 172.21.15.44:0/4004948411 shutdown_connections 2024-05-27T09:43:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.652+0000 7f84c3e00700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f84a80b29f0 0x7f84a80b4f00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.652+0000 7f84c3e00700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84c4110a00 0x7f84c41a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.652+0000 7f84c3e00700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84c4075af0 0x7f84c419ce20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.652+0000 7f84c3e00700 1 --2- 172.21.15.44:0/4004948411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84c4076440 0x7f84c419d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.652+0000 7f84c3e00700 1 -- 172.21.15.44:0/4004948411 >> 172.21.15.44:0/4004948411 conn(0x7f84c4079d00 msgr2=0x7f84c4100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.653+0000 7f84c3e00700 1 -- 172.21.15.44:0/4004948411 shutdown_connections 2024-05-27T09:43:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:01.653+0000 7f84c3e00700 1 -- 172.21.15.44:0/4004948411 wait complete. 2024-05-27T09:43:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-05-27T09:43:02.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:01 smithi044 ceph-mon[31618]: pgmap v239: 97 pgs: 97 active+clean; 57 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 473 op/s 2024-05-27T09:43:02.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:01 smithi044 ceph-mon[31618]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T09:43:02.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:01 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4004948411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:43:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:01 smithi050 ceph-mon[36371]: pgmap v239: 97 pgs: 97 active+clean; 57 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 473 op/s 2024-05-27T09:43:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:01 smithi050 ceph-mon[36371]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T09:43:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:01 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/4004948411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:43:02.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:01 smithi189 ceph-mon[36671]: pgmap v239: 97 pgs: 97 active+clean; 57 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 473 op/s 2024-05-27T09:43:02.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:01 smithi189 ceph-mon[36671]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T09:43:02.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:01 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/4004948411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:43:02.484 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:43:02.956 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:43:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:03 smithi044 ceph-mon[31618]: pgmap v241: 97 pgs: 97 active+clean; 57 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 558 op/s 2024-05-27T09:43:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:03 smithi050 ceph-mon[36371]: pgmap v241: 97 pgs: 97 active+clean; 57 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 558 op/s 2024-05-27T09:43:04.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:03 smithi189 ceph-mon[36671]: pgmap v241: 97 pgs: 97 active+clean; 57 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 558 op/s 2024-05-27T09:43:05.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.216+0000 7fc439200700 1 -- 172.21.15.44:0/4287446913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4340764a0 msgr2=0x7fc434114e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:05.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.216+0000 7fc439200700 1 --2- 172.21.15.44:0/4287446913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4340764a0 0x7fc434114e20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc420009990 tx=0x7fc420009ca0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.216+0000 7fc439200700 1 -- 172.21.15.44:0/4287446913 shutdown_connections 2024-05-27T09:43:05.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.217+0000 7fc439200700 1 --2- 172.21.15.44:0/4287446913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc434115360 0x7fc434117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.217+0000 7fc439200700 1 --2- 172.21.15.44:0/4287446913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc434075b50 0x7fc434075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.217+0000 7fc439200700 1 --2- 172.21.15.44:0/4287446913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4340764a0 0x7fc434114e20 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.217+0000 7fc439200700 1 -- 172.21.15.44:0/4287446913 >> 172.21.15.44:0/4287446913 conn(0x7fc434100030 msgr2=0x7fc434102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:05.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.217+0000 7fc439200700 1 -- 172.21.15.44:0/4287446913 shutdown_connections 2024-05-27T09:43:05.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.217+0000 7fc439200700 1 -- 172.21.15.44:0/4287446913 wait complete. 2024-05-27T09:43:05.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.218+0000 7fc439200700 1 Processor -- start 2024-05-27T09:43:05.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.218+0000 7fc439200700 1 -- start start 2024-05-27T09:43:05.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.219+0000 7fc439200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc434075b50 0x7fc4341a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.219+0000 7fc439200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4340764a0 0x7fc4341a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.219+0000 7fc439200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc434115360 0x7fc4341a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.219+0000 7fc439200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc43411aab0 con 0x7fc434115360 2024-05-27T09:43:05.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.219+0000 7fc439200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc43411a930 con 0x7fc4340764a0 2024-05-27T09:43:05.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.219+0000 7fc439200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc43411ac30 con 0x7fc434075b50 2024-05-27T09:43:05.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.219+0000 7fc431e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc434075b50 0x7fc4341a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.219+0000 7fc431e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc434075b50 0x7fc4341a1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44598/0 (socket says 172.21.15.44:44598) 2024-05-27T09:43:05.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.219+0000 7fc431e00700 1 -- 172.21.15.44:0/1770921599 learned_addr learned my addr 172.21.15.44:0/1770921599 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:05.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.219+0000 7fc431400700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4340764a0 0x7fc4341a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.219+0000 7fc432800700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc434115360 0x7fc4341a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.220+0000 7fc431e00700 1 -- 172.21.15.44:0/1770921599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4340764a0 msgr2=0x7fc4341a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:05.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.220+0000 7fc431e00700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4340764a0 0x7fc4341a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.220+0000 7fc431e00700 1 -- 172.21.15.44:0/1770921599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc434115360 msgr2=0x7fc4341a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:05.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.220+0000 7fc431e00700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc434115360 0x7fc4341a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.220+0000 7fc431e00700 1 -- 172.21.15.44:0/1770921599 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4341a6350 con 0x7fc434075b50 2024-05-27T09:43:05.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.220+0000 7fc431e00700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc434075b50 0x7fc4341a1480 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc42400e9a0 tx=0x7fc42400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:05.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.220+0000 7fc42aa00700 1 -- 172.21.15.44:0/1770921599 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42400cbd0 con 0x7fc434075b50 2024-05-27T09:43:05.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.220+0000 7fc42aa00700 1 -- 172.21.15.44:0/1770921599 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc42400cd70 con 0x7fc434075b50 2024-05-27T09:43:05.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.220+0000 7fc439200700 1 -- 172.21.15.44:0/1770921599 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4341a6640 con 0x7fc434075b50 2024-05-27T09:43:05.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.220+0000 7fc42aa00700 1 -- 172.21.15.44:0/1770921599 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc424018680 con 0x7fc434075b50 2024-05-27T09:43:05.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.221+0000 7fc439200700 1 -- 172.21.15.44:0/1770921599 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4341adf60 con 0x7fc434075b50 2024-05-27T09:43:05.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.224+0000 7fc42aa00700 1 -- 172.21.15.44:0/1770921599 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc424010430 con 0x7fc434075b50 2024-05-27T09:43:05.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.226+0000 7fc447cf5b80 1 Processor -- start 2024-05-27T09:43:05.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.226+0000 7fc42aa00700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4180b2a70 0x7fc4180b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.226+0000 7fc447cf5b80 1 -- start start 2024-05-27T09:43:05.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.226+0000 7fc42aa00700 1 -- 172.21.15.44:0/1770921599 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7fc4240e2820 con 0x7fc434075b50 2024-05-27T09:43:05.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.226+0000 7fc432800700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4180b2a70 0x7fc4180b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.227+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.227+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2b2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.227+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb2b2e70 0x558bfb2bf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.227+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558bfb1e0e40 con 0x558bfb21f9f0 2024-05-27T09:43:05.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.227+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558bfb1d09e0 con 0x558bfb133e40 2024-05-27T09:43:05.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.227+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558bfb1d4a20 con 0x558bfb2b2e70 2024-05-27T09:43:05.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.227+0000 7fc431400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.227+0000 7fc431400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2b2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49552/0 (socket says 172.21.15.44:49552) 2024-05-27T09:43:05.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.227+0000 7fc431400700 1 -- 172.21.15.44:0/4208607762 learned_addr learned my addr 172.21.15.44:0/4208607762 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:05.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.227+0000 7fc431e00700 1 --2- 172.21.15.44:0/4208607762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.228+0000 7fc431400700 1 -- 172.21.15.44:0/4208607762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb2b2e70 msgr2=0x558bfb2bf110 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T09:43:05.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.228+0000 7fc431400700 1 --2- 172.21.15.44:0/4208607762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb2b2e70 0x558bfb2bf110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.228+0000 7fc431400700 1 -- 172.21.15.44:0/4208607762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 msgr2=0x558bfb2b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:05.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.228+0000 7fc431400700 1 --2- 172.21.15.44:0/4208607762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2b2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.228+0000 7fc431400700 1 -- 172.21.15.44:0/4208607762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bfb199420 con 0x558bfb133e40 2024-05-27T09:43:05.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.228+0000 7fc432800700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4180b2a70 0x7fc4180b4f80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0093d0 tx=0x7fc42c00b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:05.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.228+0000 7fc431400700 1 --2- 172.21.15.44:0/4208607762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2b2870 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc420015000 tx=0x7fc4200096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8679ad52a2dfa6d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:05.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.228+0000 7fc41f400700 1 -- 172.21.15.44:0/4208607762 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420027070 con 0x558bfb133e40 2024-05-27T09:43:05.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.229+0000 7fc41f400700 1 -- 172.21.15.44:0/4208607762 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc420005660 con 0x558bfb133e40 2024-05-27T09:43:05.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.229+0000 7fc41f400700 1 -- 172.21.15.44:0/4208607762 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420003ab0 con 0x558bfb133e40 2024-05-27T09:43:05.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.230+0000 7fc447cf5b80 1 -- 172.21.15.44:0/4208607762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 msgr2=0x558bfb2b2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:05.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.230+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/4208607762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2b2870 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc420015000 tx=0x7fc4200096a0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.230+0000 7fc447cf5b80 1 -- 172.21.15.44:0/4208607762 shutdown_connections 2024-05-27T09:43:05.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.230+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/4208607762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb2b2e70 0x558bfb2bf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.230+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/4208607762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.230+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/4208607762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2b2870 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.230+0000 7fc447cf5b80 1 -- 172.21.15.44:0/4208607762 >> 172.21.15.44:0/4208607762 conn(0x558bfb1cd390 msgr2=0x558bfb2ae4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:05.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.230+0000 7fc447cf5b80 1 -- 172.21.15.44:0/4208607762 shutdown_connections 2024-05-27T09:43:05.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.230+0000 7fc447cf5b80 1 -- 172.21.15.44:0/4208607762 wait complete. 2024-05-27T09:43:05.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.231+0000 7fc447cf5b80 1 -- start start 2024-05-27T09:43:05.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.231+0000 7fc447cf5b80 1 Processor -- start 2024-05-27T09:43:05.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb133e40 0x558bfb2bc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb21f9f0 0x558bfb2bcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb2b2e70 0x558bfb2bd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558bfb0d6fa0 con 0x558bfb133e40 2024-05-27T09:43:05.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558bfb171760 con 0x558bfb2b2e70 2024-05-27T09:43:05.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558bfb2bfb20 con 0x558bfb21f9f0 2024-05-27T09:43:05.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc447cf5b80 10 client.0 _ll_register_callbacks cb 0x558bfb17ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:05.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc431400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb21f9f0 0x558bfb2bcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc431400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb21f9f0 0x558bfb2bcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44616/0 (socket says 172.21.15.44:44616) 2024-05-27T09:43:05.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc431400700 1 -- 172.21.15.44:0/3574374630 learned_addr learned my addr 172.21.15.44:0/3574374630 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:05.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc432800700 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb2b2e70 0x558bfb2bd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc431e00700 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb133e40 0x558bfb2bc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc431400700 1 -- 172.21.15.44:0/3574374630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb2b2e70 msgr2=0x558bfb2bd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:05.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc431400700 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb2b2e70 0x558bfb2bd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc431400700 1 -- 172.21.15.44:0/3574374630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb133e40 msgr2=0x558bfb2bc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:05.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc431400700 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb133e40 0x558bfb2bc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.232+0000 7fc431400700 1 -- 172.21.15.44:0/3574374630 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bfb1208d0 con 0x558bfb21f9f0 2024-05-27T09:43:05.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.233+0000 7fc431400700 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb21f9f0 0x558bfb2bcbc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc420009750 tx=0x7fc42000b950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=422078a63b211c69 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:05.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.233+0000 7fc413e00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:43:05.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.233+0000 7fc413e00700 1 -- 172.21.15.44:0/3574374630 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420027040 con 0x558bfb21f9f0 2024-05-27T09:43:05.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.233+0000 7fc447cf5b80 1 -- 172.21.15.44:0/3574374630 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc420005660 con 0x558bfb21f9f0 2024-05-27T09:43:05.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.233+0000 7fc413e00700 1 -- 172.21.15.44:0/3574374630 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7fc420003680 con 0x558bfb21f9f0 2024-05-27T09:43:05.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.233+0000 7fc413e00700 1 -- 172.21.15.44:0/3574374630 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc42003a050 con 0x558bfb21f9f0 2024-05-27T09:43:05.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.233+0000 7fc447cf5b80 10 client.26686 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:05.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.233+0000 7fc447cf5b80 1 -- 172.21.15.44:0/3574374630 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558bfb1a1060 con 0x558bfb21f9f0 2024-05-27T09:43:05.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.234+0000 7fc413e00700 1 -- 172.21.15.44:0/3574374630 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc420005660 con 0x558bfb21f9f0 2024-05-27T09:43:05.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.234+0000 7fc413e00700 1 -- 172.21.15.44:0/3574374630 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fc42000f650 con 0x558bfb21f9f0 2024-05-27T09:43:05.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.234+0000 7fc447cf5b80 10 client.26686 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:05.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.234+0000 7fc447cf5b80 10 client.26686 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:43:05.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.234+0000 7fc447cf5b80 20 client.26686 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:05.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.234+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x558bfb2cc940 0x558bfb2ced60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.234+0000 7fc447cf5b80 4 client.26686 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:43:05.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.234+0000 7fc447cf5b80 1 -- 172.21.15.44:0/3574374630 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bfb1e2d90 con 0x558bfb2cc940 2024-05-27T09:43:05.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.235+0000 7fc431e00700 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x558bfb2cc940 0x558bfb2ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.235+0000 7fc431e00700 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x558bfb2cc940 0x558bfb2ced60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4933233fd3d30089 server_cookie=96d5e8e0900298a8 in_seq=0 out_seq=0 2024-05-27T09:43:05.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.235+0000 7fc413e00700 10 client.26686 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:43:05.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc413e00700 1 -- 172.21.15.44:0/3574374630 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558bfb1e2d90 con 0x558bfb2cc940 2024-05-27T09:43:05.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc413e00700 10 client.26686 handle_command_reply: tid=0 2024-05-27T09:43:05.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc447cf5b80 1 client.26686 shutdown 2024-05-27T09:43:05.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc447cf5b80 1 -- 172.21.15.44:0/3574374630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb21f9f0 msgr2=0x558bfb2bcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:05.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb21f9f0 0x558bfb2bcbc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc420009750 tx=0x7fc42000b950 comp rx=0 tx=0).stop 2024-05-27T09:43:05.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc447cf5b80 1 -- 172.21.15.44:0/3574374630 shutdown_connections 2024-05-27T09:43:05.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb2b2e70 0x558bfb2bd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x558bfb2cc940 0x558bfb2ced60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb133e40 0x558bfb2bc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/3574374630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb21f9f0 0x558bfb2bcbc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc413e00700 0 client.26686 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:43:05.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc447cf5b80 1 -- 172.21.15.44:0/3574374630 >> 172.21.15.44:0/3574374630 conn(0x558bfb1cd390 msgr2=0x558bfb2b2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:05.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc447cf5b80 1 -- 172.21.15.44:0/3574374630 shutdown_connections 2024-05-27T09:43:05.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.236+0000 7fc447cf5b80 1 -- 172.21.15.44:0/3574374630 wait complete. 2024-05-27T09:43:05.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.237+0000 7fc447cf5b80 20 client.26686 trim_cache size 0 max 16384 2024-05-27T09:43:05.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.254+0000 7fc447cf5b80 1 Processor -- start 2024-05-27T09:43:05.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.254+0000 7fc447cf5b80 1 -- start start 2024-05-27T09:43:05.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.254+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb133e40 0x558bfb2b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.254+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.254+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb2b2e70 0x558bfb2bcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.254+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc4341b7d70 con 0x558bfb21f9f0 2024-05-27T09:43:05.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.254+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4341b55d0 con 0x558bfb2b2e70 2024-05-27T09:43:05.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.254+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc4341b3440 con 0x558bfb133e40 2024-05-27T09:43:05.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.255+0000 7fc431e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb133e40 0x558bfb2b7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.255+0000 7fc431e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb133e40 0x558bfb2b7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44622/0 (socket says 172.21.15.44:44622) 2024-05-27T09:43:05.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.255+0000 7fc431e00700 1 -- 172.21.15.44:0/1840860626 learned_addr learned my addr 172.21.15.44:0/1840860626 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:05.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.255+0000 7fc432800700 1 --2- 172.21.15.44:0/1840860626 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb2b2e70 0x558bfb2bcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.255+0000 7fc431400700 1 --2- 172.21.15.44:0/1840860626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.255+0000 7fc431e00700 1 -- 172.21.15.44:0/1840860626 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb2b2e70 msgr2=0x558bfb2bcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:05.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.255+0000 7fc431e00700 1 --2- 172.21.15.44:0/1840860626 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb2b2e70 0x558bfb2bcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.255+0000 7fc431e00700 1 -- 172.21.15.44:0/1840860626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 msgr2=0x558bfb2c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:05.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.255+0000 7fc431e00700 1 --2- 172.21.15.44:0/1840860626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.255+0000 7fc431e00700 1 -- 172.21.15.44:0/1840860626 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bfb2babf0 con 0x558bfb133e40 2024-05-27T09:43:05.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.256+0000 7fc431e00700 1 --2- 172.21.15.44:0/1840860626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb133e40 0x558bfb2b7e50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc4240a6d40 tx=0x7fc424010cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9896d6514b499a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:05.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.256+0000 7fc41ea00700 1 -- 172.21.15.44:0/1840860626 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4240aa1d0 con 0x558bfb133e40 2024-05-27T09:43:05.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.256+0000 7fc41ea00700 1 -- 172.21.15.44:0/1840860626 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4240a9ad0 con 0x558bfb133e40 2024-05-27T09:43:05.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.256+0000 7fc41ea00700 1 -- 172.21.15.44:0/1840860626 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4240b24a0 con 0x558bfb133e40 2024-05-27T09:43:05.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.257+0000 7fc447cf5b80 1 -- 172.21.15.44:0/1840860626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb133e40 msgr2=0x558bfb2b7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:05.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.257+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/1840860626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb133e40 0x558bfb2b7e50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc4240a6d40 tx=0x7fc424010cb0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.257+0000 7fc447cf5b80 1 -- 172.21.15.44:0/1840860626 shutdown_connections 2024-05-27T09:43:05.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.257+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/1840860626 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb2b2e70 0x558bfb2bcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.257+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/1840860626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb133e40 0x558bfb2b7e50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.257+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/1840860626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.257+0000 7fc447cf5b80 1 -- 172.21.15.44:0/1840860626 >> 172.21.15.44:0/1840860626 conn(0x558bfb1cd390 msgr2=0x558bfb2bc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:05.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.257+0000 7fc447cf5b80 1 -- 172.21.15.44:0/1840860626 shutdown_connections 2024-05-27T09:43:05.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.257+0000 7fc447cf5b80 1 -- 172.21.15.44:0/1840860626 wait complete. 2024-05-27T09:43:05.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.258+0000 7fc447cf5b80 1 -- start start 2024-05-27T09:43:05.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.258+0000 7fc447cf5b80 1 Processor -- start 2024-05-27T09:43:05.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2cf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc447cf5b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb2b2e70 0x558bfb32f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc4341b3a90 con 0x558bfb21f9f0 2024-05-27T09:43:05.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4341b4f00 con 0x558bfb133e40 2024-05-27T09:43:05.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc447cf5b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc4341b4160 con 0x558bfb2b2e70 2024-05-27T09:43:05.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc447cf5b80 10 client.0 _ll_register_callbacks cb 0x558bfb17ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:05.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc431e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc432800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb2b2e70 0x558bfb32f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc431e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2cf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49576/0 (socket says 172.21.15.44:49576) 2024-05-27T09:43:05.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc431e00700 1 -- 172.21.15.44:0/243142434 learned_addr learned my addr 172.21.15.44:0/243142434 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:05.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.259+0000 7fc431400700 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.260+0000 7fc431e00700 1 -- 172.21.15.44:0/243142434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb2b2e70 msgr2=0x558bfb32f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.260+0000 7fc431e00700 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb2b2e70 0x558bfb32f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.260+0000 7fc431e00700 1 -- 172.21.15.44:0/243142434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 msgr2=0x558bfb2cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.260+0000 7fc431e00700 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2cf860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.260+0000 7fc431e00700 1 -- 172.21.15.44:0/243142434 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bfb32f6e0 con 0x558bfb133e40 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.260+0000 7fc432800700 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb2b2e70 0x558bfb32f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.260+0000 7fc431e00700 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2cf320 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc4240a97d0 tx=0x7fc4240a9fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26fd78f6fa2c07b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.260+0000 7fc413e00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.260+0000 7fc413e00700 1 -- 172.21.15.44:0/243142434 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4240ade80 con 0x558bfb133e40 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.260+0000 7fc413e00700 1 -- 172.21.15.44:0/243142434 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8565+0+0 (secure 0 0 0) 0x7fc4240a60c0 con 0x558bfb133e40 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.260+0000 7fc447cf5b80 1 -- 172.21.15.44:0/243142434 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4240a9ad0 con 0x558bfb133e40 2024-05-27T09:43:05.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.261+0000 7fc413e00700 1 -- 172.21.15.44:0/243142434 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc4240b2930 con 0x558bfb133e40 2024-05-27T09:43:05.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.261+0000 7fc447cf5b80 10 client.26702 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:05.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.261+0000 7fc447cf5b80 1 -- 172.21.15.44:0/243142434 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558bfb3369e0 con 0x558bfb133e40 2024-05-27T09:43:05.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.261+0000 7fc413e00700 1 -- 172.21.15.44:0/243142434 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4240a9ad0 con 0x558bfb133e40 2024-05-27T09:43:05.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.261+0000 7fc413e00700 1 -- 172.21.15.44:0/243142434 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fc4240cc0b0 con 0x558bfb133e40 2024-05-27T09:43:05.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.261+0000 7fc447cf5b80 10 client.26702 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:05.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.261+0000 7fc447cf5b80 10 client.26702 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:43:05.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.261+0000 7fc447cf5b80 20 client.26702 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:05.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.261+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x558bfb336fe0 0x558bfb339400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:05.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.261+0000 7fc447cf5b80 4 client.26702 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:43:05.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.261+0000 7fc447cf5b80 1 -- 172.21.15.44:0/243142434 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558bfb1dc570 con 0x558bfb336fe0 2024-05-27T09:43:05.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.262+0000 7fc431400700 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x558bfb336fe0 0x558bfb339400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:05.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.262+0000 7fc431400700 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x558bfb336fe0 0x558bfb339400 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9287f82c6f4ee436 server_cookie=b61b799e33300b4e in_seq=0 out_seq=0 2024-05-27T09:43:05.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.262+0000 7fc413e00700 10 client.26702 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:43:05.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.262+0000 7fc413e00700 1 -- 172.21.15.44:0/243142434 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558bfb1dc570 con 0x558bfb336fe0 2024-05-27T09:43:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.262+0000 7fc413e00700 10 client.26702 handle_command_reply: tid=0 2024-05-27T09:43:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.262+0000 7fc447cf5b80 1 client.26702 shutdown 2024-05-27T09:43:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc447cf5b80 1 -- 172.21.15.44:0/243142434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 msgr2=0x558bfb2cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2cf320 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc4240a97d0 tx=0x7fc4240a9fe0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc447cf5b80 1 -- 172.21.15.44:0/243142434 shutdown_connections 2024-05-27T09:43:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558bfb2b2e70 0x558bfb32f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558bfb133e40 0x558bfb2cf320 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x558bfb336fe0 0x558bfb339400 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc447cf5b80 1 --2- 172.21.15.44:0/243142434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558bfb21f9f0 0x558bfb2cf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc413e00700 0 client.26702 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:43:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc447cf5b80 1 -- 172.21.15.44:0/243142434 >> 172.21.15.44:0/243142434 conn(0x558bfb1cd390 msgr2=0x558bfb2bb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:05.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc447cf5b80 1 -- 172.21.15.44:0/243142434 shutdown_connections 2024-05-27T09:43:05.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc447cf5b80 1 -- 172.21.15.44:0/243142434 wait complete. 2024-05-27T09:43:05.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.263+0000 7fc447cf5b80 20 client.26702 trim_cache size 0 max 16384 2024-05-27T09:43:05.269 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.264+0000 7fc439200700 1 -- 172.21.15.44:0/1770921599 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4180b2a70 msgr2=0x7fc4180b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.264+0000 7fc439200700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4180b2a70 0x7fc4180b4f80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0093d0 tx=0x7fc42c00b040 comp rx=0 tx=0).stop 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.264+0000 7fc439200700 1 -- 172.21.15.44:0/1770921599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc434075b50 msgr2=0x7fc4341a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.264+0000 7fc439200700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc434075b50 0x7fc4341a1480 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc42400e9a0 tx=0x7fc42400ecb0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.264+0000 7fc439200700 1 -- 172.21.15.44:0/1770921599 shutdown_connections 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.264+0000 7fc439200700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4180b2a70 0x7fc4180b4f80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.264+0000 7fc439200700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc434115360 0x7fc4341a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.264+0000 7fc439200700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc434075b50 0x7fc4341a1480 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.265+0000 7fc439200700 1 --2- 172.21.15.44:0/1770921599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4340764a0 0x7fc4341a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.265+0000 7fc439200700 1 -- 172.21.15.44:0/1770921599 >> 172.21.15.44:0/1770921599 conn(0x7fc434100030 msgr2=0x7fc4341006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.265+0000 7fc439200700 1 -- 172.21.15.44:0/1770921599 shutdown_connections 2024-05-27T09:43:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:05.265+0000 7fc439200700 1 -- 172.21.15.44:0/1770921599 wait complete. 2024-05-27T09:43:06.040 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:43:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:05 smithi044 ceph-mon[31618]: pgmap v242: 97 pgs: 97 active+clean; 57 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 558 op/s 2024-05-27T09:43:06.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:05 smithi050 ceph-mon[36371]: pgmap v242: 97 pgs: 97 active+clean; 57 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 558 op/s 2024-05-27T09:43:06.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:05 smithi189 ceph-mon[36671]: pgmap v242: 97 pgs: 97 active+clean; 57 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 558 op/s 2024-05-27T09:43:06.493 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:43:08.097 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:07 smithi044 ceph-mon[31618]: pgmap v243: 97 pgs: 97 active+clean; 101 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.7 MiB/s wr, 102 op/s 2024-05-27T09:43:08.098 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:07 smithi044 ceph-mon[31618]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T09:43:08.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:07 smithi050 ceph-mon[36371]: pgmap v243: 97 pgs: 97 active+clean; 101 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.7 MiB/s wr, 102 op/s 2024-05-27T09:43:08.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:07 smithi050 ceph-mon[36371]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T09:43:08.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:07 smithi189 ceph-mon[36671]: pgmap v243: 97 pgs: 97 active+clean; 101 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.7 MiB/s wr, 102 op/s 2024-05-27T09:43:08.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:07 smithi189 ceph-mon[36671]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T09:43:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.510+0000 7f1e4ac00640 1 -- 172.21.15.44:0/3421425032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e4410b610 msgr2=0x7f1e4410da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.510+0000 7f1e4ac00640 1 --2- 172.21.15.44:0/3421425032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e4410b610 0x7f1e4410da30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1e30009900 tx=0x7f1e3002a3f0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.510+0000 7f1e4ac00640 1 -- 172.21.15.44:0/3421425032 shutdown_connections 2024-05-27T09:43:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.510+0000 7f1e4ac00640 1 --2- 172.21.15.44:0/3421425032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e4410e000 0x7f1e441104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.510+0000 7f1e4ac00640 1 --2- 172.21.15.44:0/3421425032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e4410b610 0x7f1e4410da30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.510+0000 7f1e4ac00640 1 --2- 172.21.15.44:0/3421425032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e44104bb0 0x7f1e44104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.510+0000 7f1e4ac00640 1 -- 172.21.15.44:0/3421425032 >> 172.21.15.44:0/3421425032 conn(0x7f1e440fe210 msgr2=0x7f1e44100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.511+0000 7f1e4ac00640 1 -- 172.21.15.44:0/3421425032 shutdown_connections 2024-05-27T09:43:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.511+0000 7f1e4ac00640 1 -- 172.21.15.44:0/3421425032 wait complete. 2024-05-27T09:43:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.512+0000 7f1e4ac00640 1 Processor -- start 2024-05-27T09:43:08.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.512+0000 7f1e4ac00640 1 -- start start 2024-05-27T09:43:08.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.513+0000 7f1e4ac00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e44104bb0 0x7f1e44105690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.513+0000 7f1e4ac00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e4410b610 0x7f1e44105bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.513+0000 7f1e4ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e4410e000 0x7f1e44106110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.513+0000 7f1e4ac00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44113c60 con 0x7f1e4410b610 2024-05-27T09:43:08.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.513+0000 7f1e4ac00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44113ae0 con 0x7f1e44104bb0 2024-05-27T09:43:08.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.513+0000 7f1e4ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44113de0 con 0x7f1e4410e000 2024-05-27T09:43:08.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.513+0000 7f1e49800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e44104bb0 0x7f1e44105690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.513+0000 7f1e49800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e44104bb0 0x7f1e44105690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49584/0 (socket says 172.21.15.44:49584) 2024-05-27T09:43:08.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.513+0000 7f1e49800640 1 -- 172.21.15.44:0/2918115299 learned_addr learned my addr 172.21.15.44:0/2918115299 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:08.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.513+0000 7f1e48e00640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e4410b610 0x7f1e44105bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.514+0000 7f1e4a200640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e4410e000 0x7f1e44106110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.514+0000 7f1e49800640 1 -- 172.21.15.44:0/2918115299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e4410e000 msgr2=0x7f1e44106110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.514+0000 7f1e49800640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e4410e000 0x7f1e44106110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.514+0000 7f1e49800640 1 -- 172.21.15.44:0/2918115299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e4410b610 msgr2=0x7f1e44105bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.515+0000 7f1e49800640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e4410b610 0x7f1e44105bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.515+0000 7f1e49800640 1 -- 172.21.15.44:0/2918115299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e441ace50 con 0x7f1e44104bb0 2024-05-27T09:43:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.515+0000 7f1e48e00640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e4410b610 0x7f1e44105bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.515+0000 7f1e49800640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e44104bb0 0x7f1e44105690 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800e9f0 tx=0x7f1e3800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.516+0000 7f1e42000640 1 -- 172.21.15.44:0/2918115299 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3800cd80 con 0x7f1e44104bb0 2024-05-27T09:43:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.516+0000 7f1e42000640 1 -- 172.21.15.44:0/2918115299 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e380108e0 con 0x7f1e44104bb0 2024-05-27T09:43:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.516+0000 7f1e4ac00640 1 -- 172.21.15.44:0/2918115299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e441ad080 con 0x7f1e44104bb0 2024-05-27T09:43:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.516+0000 7f1e42000640 1 -- 172.21.15.44:0/2918115299 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e38018980 con 0x7f1e44104bb0 2024-05-27T09:43:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.516+0000 7f1e4ac00640 1 -- 172.21.15.44:0/2918115299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e441ad5a0 con 0x7f1e44104bb0 2024-05-27T09:43:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.519+0000 7f1e53272740 1 Processor -- start 2024-05-27T09:43:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.519+0000 7f1e53272740 1 -- start start 2024-05-27T09:43:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.520+0000 7f1e53272740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85fffd1b0 0x55b86039fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.520+0000 7f1e53272740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffd42f0 0x55b86039ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.520+0000 7f1e53272740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffdc4d0 0x55b8603a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.520+0000 7f1e53272740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b860310b90 con 0x55b85ffd42f0 2024-05-27T09:43:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.520+0000 7f1e53272740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b860300730 con 0x55b85ffdc4d0 2024-05-27T09:43:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.520+0000 7f1e53272740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b860304770 con 0x55b85fffd1b0 2024-05-27T09:43:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.520+0000 7f1e48e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffdc4d0 0x55b8603a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.520+0000 7f1e4a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85fffd1b0 0x55b86039fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.520+0000 7f1e48e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffdc4d0 0x55b8603a04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49600/0 (socket says 172.21.15.44:49600) 2024-05-27T09:43:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.520+0000 7f1e48e00640 1 -- 172.21.15.44:0/4201401481 learned_addr learned my addr 172.21.15.44:0/4201401481 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.521+0000 7f1e48e00640 1 -- 172.21.15.44:0/4201401481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85fffd1b0 msgr2=0x55b86039fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.521+0000 7f1e48e00640 1 --2- 172.21.15.44:0/4201401481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85fffd1b0 0x55b86039fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.521+0000 7f1e48e00640 1 -- 172.21.15.44:0/4201401481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffd42f0 msgr2=0x55b86039ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:43:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.521+0000 7f1e48e00640 1 --2- 172.21.15.44:0/4201401481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffd42f0 0x55b86039ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.521+0000 7f1e48e00640 1 -- 172.21.15.44:0/4201401481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8602998b0 con 0x55b85ffdc4d0 2024-05-27T09:43:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.521+0000 7f1e48e00640 1 --2- 172.21.15.44:0/4201401481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffdc4d0 0x55b8603a04e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1e3002a910 tx=0x7f1e30031550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f326ac779372cc97 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.522+0000 7f1e42000640 1 -- 172.21.15.44:0/2918115299 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f1e38018b20 con 0x7f1e44104bb0 2024-05-27T09:43:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.522+0000 7f1e36a00640 1 -- 172.21.15.44:0/4201401481 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e30040070 con 0x55b85ffdc4d0 2024-05-27T09:43:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.522+0000 7f1e36a00640 1 -- 172.21.15.44:0/4201401481 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e30005040 con 0x55b85ffdc4d0 2024-05-27T09:43:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.522+0000 7f1e36a00640 1 -- 172.21.15.44:0/4201401481 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e30035040 con 0x55b85ffdc4d0 2024-05-27T09:43:08.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.523+0000 7f1e53272740 1 -- 172.21.15.44:0/4201401481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffdc4d0 msgr2=0x55b8603a04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.523+0000 7f1e53272740 1 --2- 172.21.15.44:0/4201401481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffdc4d0 0x55b8603a04e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1e3002a910 tx=0x7f1e30031550 comp rx=0 tx=0).stop 2024-05-27T09:43:08.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.523+0000 7f1e53272740 1 -- 172.21.15.44:0/4201401481 shutdown_connections 2024-05-27T09:43:08.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.523+0000 7f1e53272740 1 --2- 172.21.15.44:0/4201401481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffdc4d0 0x55b8603a04e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.523+0000 7f1e53272740 1 --2- 172.21.15.44:0/4201401481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffd42f0 0x55b86039ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.523+0000 7f1e53272740 1 --2- 172.21.15.44:0/4201401481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85fffd1b0 0x55b86039fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.523+0000 7f1e53272740 1 -- 172.21.15.44:0/4201401481 >> 172.21.15.44:0/4201401481 conn(0x55b85fff1300 msgr2=0x55b86039c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:08.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.523+0000 7f1e53272740 1 -- 172.21.15.44:0/4201401481 shutdown_connections 2024-05-27T09:43:08.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.524+0000 7f1e42000640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e2c0b28e0 0x7f1e2c0b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.524+0000 7f1e53272740 1 -- 172.21.15.44:0/4201401481 wait complete. 2024-05-27T09:43:08.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.524+0000 7f1e42000640 1 -- 172.21.15.44:0/2918115299 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f1e380e2910 con 0x7f1e44104bb0 2024-05-27T09:43:08.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.524+0000 7f1e48e00640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e2c0b28e0 0x7f1e2c0b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.524+0000 7f1e53272740 1 -- start start 2024-05-27T09:43:08.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.525+0000 7f1e53272740 1 Processor -- start 2024-05-27T09:43:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.525+0000 7f1e53272740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffd42f0 0x55b8603a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.525+0000 7f1e53272740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 0x55b8603aa150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.525+0000 7f1e53272740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85fffd1b0 0x55b8603b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e53272740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b85ffe6790 con 0x55b85fffd1b0 2024-05-27T09:43:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e53272740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b8602a8bc0 con 0x55b85ffd42f0 2024-05-27T09:43:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e53272740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b8603acb30 con 0x55b85ffdc4d0 2024-05-27T09:43:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e49800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffd42f0 0x55b8603a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e49800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffd42f0 0x55b8603a9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49602/0 (socket says 172.21.15.44:49602) 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e49800640 1 -- 172.21.15.44:0/3028029751 learned_addr learned my addr 172.21.15.44:0/3028029751 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e53272740 10 client.0 _ll_register_callbacks cb 0x55b860286400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e48e00640 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 0x55b8603aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e48e00640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e2c0b28e0 0x7f1e2c0b4df0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1e30005ef0 tx=0x7f1e30005fd0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e49800640 1 -- 172.21.15.44:0/3028029751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 msgr2=0x55b8603aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e49800640 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 0x55b8603aa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e49800640 1 -- 172.21.15.44:0/3028029751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85fffd1b0 msgr2=0x55b8603b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e49800640 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85fffd1b0 0x55b8603b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.526+0000 7f1e49800640 1 -- 172.21.15.44:0/3028029751 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b860015210 con 0x55b85ffd42f0 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.527+0000 7f1e49800640 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffd42f0 0x55b8603a9c10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1e380e0110 tx=0x7f1e380a98f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d9ef3ffe6cee809 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.527+0000 7f1e2b400640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:43:08.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.527+0000 7f1e2b400640 1 -- 172.21.15.44:0/3028029751 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e380b2a50 con 0x55b85ffd42f0 2024-05-27T09:43:08.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.527+0000 7f1e2b400640 1 -- 172.21.15.44:0/3028029751 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f1e38010430 con 0x55b85ffd42f0 2024-05-27T09:43:08.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.527+0000 7f1e53272740 1 -- 172.21.15.44:0/3028029751 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e30005040 con 0x55b85ffd42f0 2024-05-27T09:43:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.528+0000 7f1e2b400640 1 -- 172.21.15.44:0/3028029751 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1e38010630 con 0x55b85ffd42f0 2024-05-27T09:43:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.529+0000 7f1e53272740 10 client.26714 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.529+0000 7f1e53272740 1 -- 172.21.15.44:0/3028029751 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b8600e1620 con 0x55b85ffd42f0 2024-05-27T09:43:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.529+0000 7f1e2b400640 1 -- 172.21.15.44:0/3028029751 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e30005040 con 0x55b85ffd42f0 2024-05-27T09:43:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.529+0000 7f1e2b400640 1 -- 172.21.15.44:0/3028029751 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f1e380a8970 con 0x55b85ffd42f0 2024-05-27T09:43:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.529+0000 7f1e53272740 10 client.26714 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.529+0000 7f1e53272740 10 client.26714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.529+0000 7f1e53272740 20 client.26714 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.529+0000 7f1e53272740 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b85ffbbd80 0x55b8603bbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.529+0000 7f1e53272740 4 client.26714 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:43:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.530+0000 7f1e53272740 1 -- 172.21.15.44:0/3028029751 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b860312ae0 con 0x55b85ffbbd80 2024-05-27T09:43:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.530+0000 7f1e48e00640 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b85ffbbd80 0x55b8603bbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.531+0000 7f1e48e00640 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b85ffbbd80 0x55b8603bbea0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5144aecb234dc1a3 server_cookie=9e10020061a94352 in_seq=0 out_seq=0 2024-05-27T09:43:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.531+0000 7f1e2b400640 10 client.26714 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.533+0000 7f1e2b400640 1 -- 172.21.15.44:0/3028029751 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b860312ae0 con 0x55b85ffbbd80 2024-05-27T09:43:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.533+0000 7f1e2b400640 10 client.26714 handle_command_reply: tid=0 2024-05-27T09:43:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.533+0000 7f1e53272740 1 client.26714 shutdown 2024-05-27T09:43:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e53272740 1 -- 172.21.15.44:0/3028029751 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffd42f0 msgr2=0x55b8603a9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e53272740 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffd42f0 0x55b8603a9c10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1e380e0110 tx=0x7f1e380a98f0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e53272740 1 -- 172.21.15.44:0/3028029751 shutdown_connections 2024-05-27T09:43:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e53272740 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b85ffbbd80 0x55b8603bbea0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e53272740 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85fffd1b0 0x55b8603b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e53272740 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 0x55b8603aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e53272740 1 --2- 172.21.15.44:0/3028029751 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85ffd42f0 0x55b8603a9c10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e53272740 1 -- 172.21.15.44:0/3028029751 >> 172.21.15.44:0/3028029751 conn(0x55b85fff1300 msgr2=0x55b8603a1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e2b400640 0 client.26714 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e53272740 1 -- 172.21.15.44:0/3028029751 shutdown_connections 2024-05-27T09:43:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e53272740 1 -- 172.21.15.44:0/3028029751 wait complete. 2024-05-27T09:43:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.534+0000 7f1e53272740 20 client.26714 trim_cache size 0 max 16384 2024-05-27T09:43:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.552+0000 7f1e53272740 1 Processor -- start 2024-05-27T09:43:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e53272740 1 -- start start 2024-05-27T09:43:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e53272740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffd42f0 0x55b8603a5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e53272740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 0x55b8603a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e53272740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85fffd1b0 0x55b8603a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e53272740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1e441b8800 con 0x55b85ffd42f0 2024-05-27T09:43:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e53272740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e441b60a0 con 0x55b85fffd1b0 2024-05-27T09:43:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e53272740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1e441b3f10 con 0x55b85ffdc4d0 2024-05-27T09:43:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e48e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 0x55b8603a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e4a200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85fffd1b0 0x55b8603a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e49800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffd42f0 0x55b8603a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e48e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 0x55b8603a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44672/0 (socket says 172.21.15.44:44672) 2024-05-27T09:43:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e4a200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85fffd1b0 0x55b8603a37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49618/0 (socket says 172.21.15.44:49618) 2024-05-27T09:43:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.553+0000 7f1e48e00640 1 -- 172.21.15.44:0/663234633 learned_addr learned my addr 172.21.15.44:0/663234633 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.554+0000 7f1e48e00640 1 -- 172.21.15.44:0/663234633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85fffd1b0 msgr2=0x55b8603a37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.554+0000 7f1e48e00640 1 --2- 172.21.15.44:0/663234633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85fffd1b0 0x55b8603a37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.554+0000 7f1e48e00640 1 -- 172.21.15.44:0/663234633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffd42f0 msgr2=0x55b8603a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.554+0000 7f1e48e00640 1 --2- 172.21.15.44:0/663234633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffd42f0 0x55b8603a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.554+0000 7f1e48e00640 1 -- 172.21.15.44:0/663234633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b86039a610 con 0x55b85ffdc4d0 2024-05-27T09:43:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.554+0000 7f1e48e00640 1 --2- 172.21.15.44:0/663234633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 0x55b8603a94c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1e300319f0 tx=0x7f1e30031550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d11c827e8907db0d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.555+0000 7f1e36000640 1 -- 172.21.15.44:0/663234633 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e30040040 con 0x55b85ffdc4d0 2024-05-27T09:43:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.555+0000 7f1e36000640 1 -- 172.21.15.44:0/663234633 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e3005b450 con 0x55b85ffdc4d0 2024-05-27T09:43:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.555+0000 7f1e36000640 1 -- 172.21.15.44:0/663234633 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3005be60 con 0x55b85ffdc4d0 2024-05-27T09:43:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.556+0000 7f1e53272740 1 -- 172.21.15.44:0/663234633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 msgr2=0x55b8603a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.556+0000 7f1e53272740 1 --2- 172.21.15.44:0/663234633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 0x55b8603a94c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1e300319f0 tx=0x7f1e30031550 comp rx=0 tx=0).stop 2024-05-27T09:43:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.556+0000 7f1e53272740 1 -- 172.21.15.44:0/663234633 shutdown_connections 2024-05-27T09:43:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.556+0000 7f1e53272740 1 --2- 172.21.15.44:0/663234633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85fffd1b0 0x55b8603a37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.556+0000 7f1e53272740 1 --2- 172.21.15.44:0/663234633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffdc4d0 0x55b8603a94c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.556+0000 7f1e53272740 1 --2- 172.21.15.44:0/663234633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffd42f0 0x55b8603a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.556+0000 7f1e53272740 1 -- 172.21.15.44:0/663234633 >> 172.21.15.44:0/663234633 conn(0x55b85fff1300 msgr2=0x55b8603a0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.557+0000 7f1e53272740 1 -- 172.21.15.44:0/663234633 shutdown_connections 2024-05-27T09:43:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.557+0000 7f1e53272740 1 -- 172.21.15.44:0/663234633 wait complete. 2024-05-27T09:43:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.557+0000 7f1e53272740 1 -- start start 2024-05-27T09:43:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.557+0000 7f1e53272740 1 Processor -- start 2024-05-27T09:43:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.558+0000 7f1e53272740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffd42f0 0x55b8603b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.558+0000 7f1e53272740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffdc4d0 0x55b8603ad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.558+0000 7f1e53272740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85fffd1b0 0x55b8603adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.558+0000 7f1e53272740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1e441b4560 con 0x55b85ffdc4d0 2024-05-27T09:43:08.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.558+0000 7f1e53272740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e441b59d0 con 0x55b85fffd1b0 2024-05-27T09:43:08.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.558+0000 7f1e53272740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1e441b4c30 con 0x55b85ffd42f0 2024-05-27T09:43:08.560 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.558+0000 7f1e53272740 10 client.0 _ll_register_callbacks cb 0x7f1e38009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.558+0000 7f1e49800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffd42f0 0x55b8603b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.558+0000 7f1e49800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffd42f0 0x55b8603b21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44674/0 (socket says 172.21.15.44:44674) 2024-05-27T09:43:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.558+0000 7f1e49800640 1 -- 172.21.15.44:0/2249830517 learned_addr learned my addr 172.21.15.44:0/2249830517 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.558+0000 7f1e48e00640 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffdc4d0 0x55b8603ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.559+0000 7f1e4a200640 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85fffd1b0 0x55b8603adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.559+0000 7f1e49800640 1 -- 172.21.15.44:0/2249830517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85fffd1b0 msgr2=0x55b8603adaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.559+0000 7f1e49800640 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85fffd1b0 0x55b8603adaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.559+0000 7f1e49800640 1 -- 172.21.15.44:0/2249830517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffdc4d0 msgr2=0x55b8603ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.559+0000 7f1e49800640 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffdc4d0 0x55b8603ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.559+0000 7f1e49800640 1 -- 172.21.15.44:0/2249830517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8603adfe0 con 0x55b85ffd42f0 2024-05-27T09:43:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.559+0000 7f1e49800640 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffd42f0 0x55b8603b21e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1e380a8dd0 tx=0x7f1e38018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13284055a28eae84 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.559+0000 7f1e2b400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:43:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.560+0000 7f1e2b400640 1 -- 172.21.15.44:0/2249830517 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e380a9410 con 0x55b85ffd42f0 2024-05-27T09:43:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.560+0000 7f1e2b400640 1 -- 172.21.15.44:0/2249830517 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f1e380a6290 con 0x55b85ffd42f0 2024-05-27T09:43:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.560+0000 7f1e53272740 1 -- 172.21.15.44:0/2249830517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e3005b450 con 0x55b85ffd42f0 2024-05-27T09:43:08.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.560+0000 7f1e2b400640 1 -- 172.21.15.44:0/2249830517 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1e380a9a70 con 0x55b85ffd42f0 2024-05-27T09:43:08.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.561+0000 7f1e53272740 10 client.26710 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:08.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.561+0000 7f1e53272740 1 -- 172.21.15.44:0/2249830517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b860424140 con 0x55b85ffd42f0 2024-05-27T09:43:08.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.561+0000 7f1e2b400640 1 -- 172.21.15.44:0/2249830517 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e3005b450 con 0x55b85ffd42f0 2024-05-27T09:43:08.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.561+0000 7f1e2b400640 1 -- 172.21.15.44:0/2249830517 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f1e380a5730 con 0x55b85ffd42f0 2024-05-27T09:43:08.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.562+0000 7f1e53272740 10 client.26710 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:08.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.562+0000 7f1e53272740 10 client.26710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:08.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.562+0000 7f1e53272740 20 client.26710 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:08.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.562+0000 7f1e53272740 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b85ff911c0 0x55b8604268d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.562+0000 7f1e53272740 4 client.26710 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:43:08.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.562+0000 7f1e53272740 1 -- 172.21.15.44:0/2249830517 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b86030c2c0 con 0x55b85ff911c0 2024-05-27T09:43:08.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.562+0000 7f1e48e00640 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b85ff911c0 0x55b8604268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.563+0000 7f1e2b400640 10 client.26710 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:08.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:08.563+0000 7f1e48e00640 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b85ff911c0 0x55b8604268d0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a5a254f5f282cd1 server_cookie=e53737350339d193 in_seq=0 out_seq=0 2024-05-27T09:43:08.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.646+0000 7f09aea00700 1 -- 172.21.15.44:0/3957133363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09a8107de0 msgr2=0x7f09a81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.646+0000 7f09aea00700 1 --2- 172.21.15.44:0/3957133363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09a8107de0 0x7f09a81081f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f09940099c0 tx=0x7f0994009cd0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.647+0000 7f09aea00700 1 -- 172.21.15.44:0/3957133363 shutdown_connections 2024-05-27T09:43:08.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.647+0000 7f09aea00700 1 --2- 172.21.15.44:0/3957133363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a8108730 0x7f09a8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.647+0000 7f09aea00700 1 --2- 172.21.15.44:0/3957133363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09a81034b0 0x7f09a81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.647+0000 7f09aea00700 1 --2- 172.21.15.44:0/3957133363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09a8107de0 0x7f09a81081f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.647+0000 7f09aea00700 1 -- 172.21.15.44:0/3957133363 >> 172.21.15.44:0/3957133363 conn(0x7f09a80fedb0 msgr2=0x7f09a81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:08.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.647+0000 7f09aea00700 1 -- 172.21.15.44:0/3957133363 shutdown_connections 2024-05-27T09:43:08.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.647+0000 7f09aea00700 1 -- 172.21.15.44:0/3957133363 wait complete. 2024-05-27T09:43:08.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.648+0000 7f09aea00700 1 Processor -- start 2024-05-27T09:43:08.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.649+0000 7f09aea00700 1 -- start start 2024-05-27T09:43:08.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.649+0000 7f09aea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09a81034b0 0x7f09a81a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.649+0000 7f09aea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a8107de0 0x7f09a81a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.649+0000 7f09aea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09a8108730 0x7f09a81a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.649+0000 7f09aea00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09a8112310 con 0x7f09a8107de0 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.649+0000 7f09aea00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f09a8112190 con 0x7f09a8108730 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.649+0000 7f09aea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09a8112490 con 0x7f09a81034b0 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.649+0000 7f09a7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09a81034b0 0x7f09a81a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.649+0000 7f09a7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09a81034b0 0x7f09a81a1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44704/0 (socket says 172.21.15.44:44704) 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.649+0000 7f09a7e00700 1 -- 172.21.15.44:0/969619821 learned_addr learned my addr 172.21.15.44:0/969619821 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.650+0000 7f09aca00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09a8108730 0x7f09a81a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.650+0000 7f09a7e00700 1 -- 172.21.15.44:0/969619821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09a8108730 msgr2=0x7f09a81a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.650+0000 7f09a7e00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09a8108730 0x7f09a81a5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.650+0000 7f09a7e00700 1 -- 172.21.15.44:0/969619821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a8107de0 msgr2=0x7f09a81a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.650+0000 7f09a7e00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a8107de0 0x7f09a81a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.650+0000 7f09a7e00700 1 -- 172.21.15.44:0/969619821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a81a6340 con 0x7f09a81034b0 2024-05-27T09:43:08.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.650+0000 7f09a7e00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09a81034b0 0x7f09a81a1470 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f09980097d0 tx=0x7f0998009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.651+0000 7f09a4c00700 1 -- 172.21.15.44:0/969619821 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998018030 con 0x7f09a81034b0 2024-05-27T09:43:08.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.651+0000 7f09a4c00700 1 -- 172.21.15.44:0/969619821 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0998014ac0 con 0x7f09a81034b0 2024-05-27T09:43:08.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.651+0000 7f09a4c00700 1 -- 172.21.15.44:0/969619821 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998013b20 con 0x7f09a81034b0 2024-05-27T09:43:08.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.651+0000 7f09aea00700 1 -- 172.21.15.44:0/969619821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a81a6630 con 0x7f09a81034b0 2024-05-27T09:43:08.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.651+0000 7f09aea00700 1 -- 172.21.15.44:0/969619821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a81adf50 con 0x7f09a81034b0 2024-05-27T09:43:08.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.655+0000 7f09a4c00700 1 -- 172.21.15.44:0/969619821 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f099801b070 con 0x7f09a81034b0 2024-05-27T09:43:08.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.655+0000 7f09bd504b80 1 Processor -- start 2024-05-27T09:43:08.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.656+0000 7f09bd504b80 1 -- start start 2024-05-27T09:43:08.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.656+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 0x561e42978330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.657+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 0x561e42978870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.657+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e42985110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.657+0000 7f09bd504b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561e428a6e40 con 0x561e42978e70 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.657+0000 7f09bd504b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561e428969e0 con 0x561e427f9e40 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.657+0000 7f09bd504b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561e4289aa20 con 0x561e428e59f0 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.657+0000 7f09aca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 0x561e42978330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.657+0000 7f09aca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 0x561e42978330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44720/0 (socket says 172.21.15.44:44720) 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.657+0000 7f09aca00700 1 -- 172.21.15.44:0/1527953864 learned_addr learned my addr 172.21.15.44:0/1527953864 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.657+0000 7f09a7e00700 1 --2- 172.21.15.44:0/1527953864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 0x561e42978870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.657+0000 7f09a4c00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f098c0b2a70 0x7f098c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.657+0000 7f09a7400700 1 --2- 172.21.15.44:0/1527953864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e42985110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.658+0000 7f09a4c00700 1 -- 172.21.15.44:0/969619821 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f09980e20d0 con 0x7f09a81034b0 2024-05-27T09:43:08.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.658+0000 7f09aca00700 1 -- 172.21.15.44:0/1527953864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 msgr2=0x561e42978870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.658+0000 7f09aca00700 1 --2- 172.21.15.44:0/1527953864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 0x561e42978870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.658+0000 7f09aca00700 1 -- 172.21.15.44:0/1527953864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 msgr2=0x561e42985110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.658+0000 7f09aca00700 1 --2- 172.21.15.44:0/1527953864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e42985110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.658+0000 7f09aca00700 1 -- 172.21.15.44:0/1527953864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e4285f420 con 0x561e428e59f0 2024-05-27T09:43:08.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.658+0000 7f09a7e00700 1 --2- 172.21.15.44:0/1527953864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 0x561e42978870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:43:08.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.658+0000 7f09aca00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f098c0b2a70 0x7f098c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.659+0000 7f09a7400700 1 --2- 172.21.15.44:0/1527953864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e42985110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:08.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.659+0000 7f09aca00700 1 --2- 172.21.15.44:0/1527953864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 0x561e42978330 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f09a0011740 tx=0x7f09a0011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bb5544cec8e7564 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.660+0000 7f09aca00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f098c0b2a70 0x7f098c0b4f80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f09a0022000 tx=0x7f09a0023000 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.660+0000 7f099d600700 1 -- 172.21.15.44:0/1527953864 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a001c040 con 0x561e428e59f0 2024-05-27T09:43:08.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.660+0000 7f099d600700 1 -- 172.21.15.44:0/1527953864 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09a0009430 con 0x561e428e59f0 2024-05-27T09:43:08.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.660+0000 7f099d600700 1 -- 172.21.15.44:0/1527953864 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a00236e0 con 0x561e428e59f0 2024-05-27T09:43:08.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.661+0000 7f09bd504b80 1 -- 172.21.15.44:0/1527953864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 msgr2=0x561e42978330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.661+0000 7f09bd504b80 1 --2- 172.21.15.44:0/1527953864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 0x561e42978330 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f09a0011740 tx=0x7f09a0011a50 comp rx=0 tx=0).stop 2024-05-27T09:43:08.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.661+0000 7f09bd504b80 1 -- 172.21.15.44:0/1527953864 shutdown_connections 2024-05-27T09:43:08.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.661+0000 7f09bd504b80 1 --2- 172.21.15.44:0/1527953864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e42985110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.661+0000 7f09bd504b80 1 --2- 172.21.15.44:0/1527953864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 0x561e42978330 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.661+0000 7f09bd504b80 1 --2- 172.21.15.44:0/1527953864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 0x561e42978870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.661+0000 7f09bd504b80 1 -- 172.21.15.44:0/1527953864 >> 172.21.15.44:0/1527953864 conn(0x561e42893390 msgr2=0x561e429744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:08.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.661+0000 7f09bd504b80 1 -- 172.21.15.44:0/1527953864 shutdown_connections 2024-05-27T09:43:08.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.661+0000 7f09bd504b80 1 -- 172.21.15.44:0/1527953864 wait complete. 2024-05-27T09:43:08.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.662+0000 7f09bd504b80 1 -- start start 2024-05-27T09:43:08.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.662+0000 7f09bd504b80 1 Processor -- start 2024-05-27T09:43:08.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.663+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e427f9e40 0x561e429826e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.663+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e428e59f0 0x561e42982c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.663+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e42978e70 0x561e42983160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.663+0000 7f09bd504b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561e4279cfa0 con 0x561e427f9e40 2024-05-27T09:43:08.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.663+0000 7f09bd504b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561e42837760 con 0x561e428e59f0 2024-05-27T09:43:08.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.663+0000 7f09bd504b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561e42985b80 con 0x561e42978e70 2024-05-27T09:43:08.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.663+0000 7f09a7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e427f9e40 0x561e429826e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.663+0000 7f09bd504b80 10 client.0 _ll_register_callbacks cb 0x561e42840c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:08.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.664+0000 7f09aca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e42978e70 0x561e42983160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.664+0000 7f09a7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e428e59f0 0x561e42982c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.664+0000 7f09a7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e428e59f0 0x561e42982c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49670/0 (socket says 172.21.15.44:49670) 2024-05-27T09:43:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.664+0000 7f09a7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e427f9e40 0x561e429826e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46668/0 (socket says 172.21.15.44:46668) 2024-05-27T09:43:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.664+0000 7f09aca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e42978e70 0x561e42983160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44724/0 (socket says 172.21.15.44:44724) 2024-05-27T09:43:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.664+0000 7f09a7400700 1 -- 172.21.15.44:0/2567650293 learned_addr learned my addr 172.21.15.44:0/2567650293 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.664+0000 7f09aca00700 1 -- 172.21.15.44:0/2567650293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e428e59f0 msgr2=0x561e42982c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.664+0000 7f09aca00700 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e428e59f0 0x561e42982c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.664+0000 7f09aca00700 1 -- 172.21.15.44:0/2567650293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e427f9e40 msgr2=0x561e429826e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.664+0000 7f09aca00700 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e427f9e40 0x561e429826e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.664+0000 7f09aca00700 1 -- 172.21.15.44:0/2567650293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e427e68d0 con 0x561e42978e70 2024-05-27T09:43:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.665+0000 7f09a7e00700 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e427f9e40 0x561e429826e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.665+0000 7f09aca00700 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e42978e70 0x561e42983160 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f09a0011740 tx=0x7f09a0007cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ada4a9577446ce9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.665+0000 7f0992000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:43:08.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.665+0000 7f0992000700 1 -- 172.21.15.44:0/2567650293 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a001c070 con 0x561e42978e70 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.665+0000 7f0992000700 1 -- 172.21.15.44:0/2567650293 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f09a0023650 con 0x561e42978e70 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.665+0000 7f09bd504b80 1 -- 172.21.15.44:0/2567650293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a0009430 con 0x561e42978e70 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.665+0000 7f0992000700 1 -- 172.21.15.44:0/2567650293 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f09a0039050 con 0x561e42978e70 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.665+0000 7f09bd504b80 10 client.26722 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.665+0000 7f09bd504b80 1 -- 172.21.15.44:0/2567650293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561e42867060 con 0x561e42978e70 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.666+0000 7f0992000700 1 -- 172.21.15.44:0/2567650293 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09a0009430 con 0x561e42978e70 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.666+0000 7f0992000700 1 -- 172.21.15.44:0/2567650293 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f09a002a040 con 0x561e42978e70 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.666+0000 7f09bd504b80 10 client.26722 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.666+0000 7f09bd504b80 10 client.26722 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.666+0000 7f09bd504b80 20 client.26722 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.666+0000 7f09bd504b80 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561e42992ab0 0x561e42994ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.666+0000 7f09bd504b80 4 client.26722 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:43:08.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.666+0000 7f09bd504b80 1 -- 172.21.15.44:0/2567650293 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e428a8d90 con 0x561e42992ab0 2024-05-27T09:43:08.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.667+0000 7f09a7e00700 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561e42992ab0 0x561e42994ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.667+0000 7f09a7e00700 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561e42992ab0 0x561e42994ed0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eca621bceb4d3c83 server_cookie=2f70377aa2d46cde in_seq=0 out_seq=0 2024-05-27T09:43:08.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.667+0000 7f0992000700 10 client.26722 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:43:08.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.670+0000 7f0992000700 1 -- 172.21.15.44:0/2567650293 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561e428a8d90 con 0x561e42992ab0 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.670+0000 7f0992000700 10 client.26722 handle_command_reply: tid=0 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f09bd504b80 1 client.26722 shutdown 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f09bd504b80 1 -- 172.21.15.44:0/2567650293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e42978e70 msgr2=0x561e42983160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f09bd504b80 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e42978e70 0x561e42983160 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f09a0011740 tx=0x7f09a0007cb0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f09bd504b80 1 -- 172.21.15.44:0/2567650293 shutdown_connections 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f09bd504b80 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e42978e70 0x561e42983160 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f09bd504b80 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561e42992ab0 0x561e42994ed0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f09bd504b80 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e427f9e40 0x561e429826e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f09bd504b80 1 --2- 172.21.15.44:0/2567650293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e428e59f0 0x561e42982c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f09bd504b80 1 -- 172.21.15.44:0/2567650293 >> 172.21.15.44:0/2567650293 conn(0x561e42893390 msgr2=0x561e42978310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f0992000700 0 client.26722 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:43:08.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f09bd504b80 1 -- 172.21.15.44:0/2567650293 shutdown_connections 2024-05-27T09:43:08.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.671+0000 7f09bd504b80 1 -- 172.21.15.44:0/2567650293 wait complete. 2024-05-27T09:43:08.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.672+0000 7f09bd504b80 20 client.26722 trim_cache size 0 max 16384 2024-05-27T09:43:08.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.691+0000 7f09bd504b80 1 Processor -- start 2024-05-27T09:43:08.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.692+0000 7f09bd504b80 1 -- start start 2024-05-27T09:43:08.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.692+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 0x561e4297de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.692+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 0x561e4297c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.692+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e4298ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.692+0000 7f09bd504b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b7ed0 con 0x561e42978e70 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.692+0000 7f09bd504b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b5730 con 0x561e427f9e40 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.692+0000 7f09bd504b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b35a0 con 0x561e428e59f0 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.692+0000 7f09a7e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 0x561e4297de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.692+0000 7f09a7e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 0x561e4297de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49684/0 (socket says 172.21.15.44:49684) 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.692+0000 7f09a7e00700 1 -- 172.21.15.44:0/4111324131 learned_addr learned my addr 172.21.15.44:0/4111324131 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.692+0000 7f09a7400700 1 --2- 172.21.15.44:0/4111324131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 0x561e4297c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.693+0000 7f09aca00700 1 --2- 172.21.15.44:0/4111324131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e4298ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.693+0000 7f09a7e00700 1 -- 172.21.15.44:0/4111324131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 msgr2=0x561e4297c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.693+0000 7f09a7e00700 1 --2- 172.21.15.44:0/4111324131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 0x561e4297c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.693+0000 7f09a7e00700 1 -- 172.21.15.44:0/4111324131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 msgr2=0x561e4298ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.693+0000 7f09a7e00700 1 --2- 172.21.15.44:0/4111324131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e4298ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.693+0000 7f09a7e00700 1 -- 172.21.15.44:0/4111324131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e42979800 con 0x561e427f9e40 2024-05-27T09:43:08.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.693+0000 7f09a7e00700 1 --2- 172.21.15.44:0/4111324131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 0x561e4297de90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0998009f90 tx=0x7f0998014f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5eea049a3001cc1e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.693+0000 7f099cc00700 1 -- 172.21.15.44:0/4111324131 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09980a9a50 con 0x561e427f9e40 2024-05-27T09:43:08.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.693+0000 7f099cc00700 1 -- 172.21.15.44:0/4111324131 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09980a9350 con 0x561e427f9e40 2024-05-27T09:43:08.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.693+0000 7f099cc00700 1 -- 172.21.15.44:0/4111324131 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09980a95b0 con 0x561e427f9e40 2024-05-27T09:43:08.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.694+0000 7f09bd504b80 1 -- 172.21.15.44:0/4111324131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 msgr2=0x561e4297de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.694+0000 7f09bd504b80 1 --2- 172.21.15.44:0/4111324131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 0x561e4297de90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0998009f90 tx=0x7f0998014f90 comp rx=0 tx=0).stop 2024-05-27T09:43:08.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.694+0000 7f09bd504b80 1 -- 172.21.15.44:0/4111324131 shutdown_connections 2024-05-27T09:43:08.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.694+0000 7f09bd504b80 1 --2- 172.21.15.44:0/4111324131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e4298ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.694+0000 7f09bd504b80 1 --2- 172.21.15.44:0/4111324131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e427f9e40 0x561e4297de90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.694+0000 7f09bd504b80 1 --2- 172.21.15.44:0/4111324131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e428e59f0 0x561e4297c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.694+0000 7f09bd504b80 1 -- 172.21.15.44:0/4111324131 >> 172.21.15.44:0/4111324131 conn(0x561e42893390 msgr2=0x561e4297bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:08.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.694+0000 7f09bd504b80 1 -- 172.21.15.44:0/4111324131 shutdown_connections 2024-05-27T09:43:08.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.694+0000 7f09bd504b80 1 -- 172.21.15.44:0/4111324131 wait complete. 2024-05-27T09:43:08.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.695+0000 7f09bd504b80 1 -- start start 2024-05-27T09:43:08.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.695+0000 7f09bd504b80 1 Processor -- start 2024-05-27T09:43:08.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e427f9e40 0x561e4298b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e428e59f0 0x561e42985dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09bd504b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e42986470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09bd504b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b3bf0 con 0x561e42978e70 2024-05-27T09:43:08.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09bd504b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b5060 con 0x561e428e59f0 2024-05-27T09:43:08.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09bd504b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b42c0 con 0x561e427f9e40 2024-05-27T09:43:08.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09bd504b80 10 client.0 _ll_register_callbacks cb 0x561e42840c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:08.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09a7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e427f9e40 0x561e4298b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09aca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e42986470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09a7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e428e59f0 0x561e42985dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09a7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e427f9e40 0x561e4298b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44740/0 (socket says 172.21.15.44:44740) 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09a7e00700 1 -- 172.21.15.44:0/828185729 learned_addr learned my addr 172.21.15.44:0/828185729 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09a7e00700 1 -- 172.21.15.44:0/828185729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e428e59f0 msgr2=0x561e42985dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09a7e00700 1 --2- 172.21.15.44:0/828185729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e428e59f0 0x561e42985dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09a7e00700 1 -- 172.21.15.44:0/828185729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 msgr2=0x561e42986470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09a7e00700 1 --2- 172.21.15.44:0/828185729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e42986470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.696+0000 7f09a7e00700 1 -- 172.21.15.44:0/828185729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e42986bf0 con 0x561e427f9e40 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.697+0000 7f09a7e00700 1 --2- 172.21.15.44:0/828185729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e427f9e40 0x561e4298b610 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f099800b3c0 tx=0x7f09980a8f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd342573e03ffecc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.697+0000 7f0992000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.697+0000 7f0992000700 1 -- 172.21.15.44:0/828185729 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998013ed0 con 0x561e427f9e40 2024-05-27T09:43:08.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.697+0000 7f0992000700 1 -- 172.21.15.44:0/828185729 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f09980e40f0 con 0x561e427f9e40 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.697+0000 7f09bd504b80 1 -- 172.21.15.44:0/828185729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09980a9350 con 0x561e427f9e40 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.697+0000 7f0992000700 1 -- 172.21.15.44:0/828185729 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f09980e42f0 con 0x561e427f9e40 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.697+0000 7f09bd504b80 10 client.26728 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.697+0000 7f09bd504b80 1 -- 172.21.15.44:0/828185729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561e429fc3a0 con 0x561e427f9e40 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.698+0000 7f0992000700 1 -- 172.21.15.44:0/828185729 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09980a9350 con 0x561e427f9e40 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.698+0000 7f0992000700 1 -- 172.21.15.44:0/828185729 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f0998025060 con 0x561e427f9e40 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.698+0000 7f09bd504b80 10 client.26728 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.698+0000 7f09bd504b80 10 client.26728 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.698+0000 7f09bd504b80 20 client.26728 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.698+0000 7f09bd504b80 1 --2- 172.21.15.44:0/828185729 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561e429fc710 0x561e429feb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.698+0000 7f09bd504b80 4 client.26728 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.698+0000 7f09bd504b80 1 -- 172.21.15.44:0/828185729 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561e428a2570 con 0x561e429fc710 2024-05-27T09:43:08.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.698+0000 7f09a7400700 1 --2- 172.21.15.44:0/828185729 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561e429fc710 0x561e429feb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:08.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.700+0000 7f09a7400700 1 --2- 172.21.15.44:0/828185729 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561e429fc710 0x561e429feb30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3c2ce4a75dfc8ff1 server_cookie=548effa61e293551 in_seq=0 out_seq=0 2024-05-27T09:43:08.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.700+0000 7f0992000700 10 client.26728 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:43:08.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.700+0000 7f0992000700 1 -- 172.21.15.44:0/828185729 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561e428a2570 con 0x561e429fc710 2024-05-27T09:43:08.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.700+0000 7f0992000700 10 client.26728 handle_command_reply: tid=0 2024-05-27T09:43:08.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.700+0000 7f09bd504b80 1 client.26728 shutdown 2024-05-27T09:43:08.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.700+0000 7f09bd504b80 1 -- 172.21.15.44:0/828185729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e427f9e40 msgr2=0x561e4298b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.700+0000 7f09bd504b80 1 --2- 172.21.15.44:0/828185729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e427f9e40 0x561e4298b610 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f099800b3c0 tx=0x7f09980a8f90 comp rx=0 tx=0).stop 2024-05-27T09:43:08.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.701+0000 7f09bd504b80 1 -- 172.21.15.44:0/828185729 shutdown_connections 2024-05-27T09:43:08.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.701+0000 7f09bd504b80 1 --2- 172.21.15.44:0/828185729 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x561e429fc710 0x561e429feb30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.701+0000 7f09bd504b80 1 --2- 172.21.15.44:0/828185729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561e42978e70 0x561e42986470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.701+0000 7f09bd504b80 1 --2- 172.21.15.44:0/828185729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e427f9e40 0x561e4298b610 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.701+0000 7f09bd504b80 1 --2- 172.21.15.44:0/828185729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561e428e59f0 0x561e42985dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.701+0000 7f0992000700 0 client.26728 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:43:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.701+0000 7f09bd504b80 1 -- 172.21.15.44:0/828185729 >> 172.21.15.44:0/828185729 conn(0x561e42893390 msgr2=0x561e429824c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.701+0000 7f09bd504b80 1 -- 172.21.15.44:0/828185729 shutdown_connections 2024-05-27T09:43:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.701+0000 7f09bd504b80 1 -- 172.21.15.44:0/828185729 wait complete. 2024-05-27T09:43:08.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.701+0000 7f09bd504b80 20 client.26728 trim_cache size 0 max 16384 2024-05-27T09:43:08.705 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:43:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 -- 172.21.15.44:0/969619821 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f098c0b2a70 msgr2=0x7f098c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f098c0b2a70 0x7f098c0b4f80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f09a0022000 tx=0x7f09a0023000 comp rx=0 tx=0).stop 2024-05-27T09:43:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 -- 172.21.15.44:0/969619821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09a81034b0 msgr2=0x7f09a81a1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09a81034b0 0x7f09a81a1470 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f09980097d0 tx=0x7f0998009b90 comp rx=0 tx=0).stop 2024-05-27T09:43:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 -- 172.21.15.44:0/969619821 shutdown_connections 2024-05-27T09:43:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f098c0b2a70 0x7f098c0b4f80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09a8108730 0x7f09a81a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09a81034b0 0x7f09a81a1470 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 --2- 172.21.15.44:0/969619821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a8107de0 0x7f09a81a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:08.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 -- 172.21.15.44:0/969619821 >> 172.21.15.44:0/969619821 conn(0x7f09a80fedb0 msgr2=0x7f09a81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:08.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 -- 172.21.15.44:0/969619821 shutdown_connections 2024-05-27T09:43:08.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:08.702+0000 7f09aea00700 1 -- 172.21.15.44:0/969619821 wait complete. 2024-05-27T09:43:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.269+0000 7f1e2b400640 1 -- 172.21.15.44:0/2249830517 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b86030c2c0 con 0x55b85ff911c0 2024-05-27T09:43:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.269+0000 7f1e2b400640 10 client.26710 handle_command_reply: tid=0 2024-05-27T09:43:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.269+0000 7f1e53272740 1 client.26710 shutdown 2024-05-27T09:43:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e53272740 1 -- 172.21.15.44:0/2249830517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffd42f0 msgr2=0x55b8603b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e53272740 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffd42f0 0x55b8603b21e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1e380a8dd0 tx=0x7f1e38018420 comp rx=0 tx=0).stop 2024-05-27T09:43:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e53272740 1 -- 172.21.15.44:0/2249830517 shutdown_connections 2024-05-27T09:43:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e53272740 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b85ff911c0 0x55b8604268d0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e53272740 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b85fffd1b0 0x55b8603adaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e53272740 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b85ffdc4d0 0x55b8603ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e53272740 1 --2- 172.21.15.44:0/2249830517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b85ffd42f0 0x55b8603b21e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e53272740 1 -- 172.21.15.44:0/2249830517 >> 172.21.15.44:0/2249830517 conn(0x55b85fff1300 msgr2=0x55b86039ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e2b400640 0 client.26710 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e53272740 1 -- 172.21.15.44:0/2249830517 shutdown_connections 2024-05-27T09:43:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e53272740 1 -- 172.21.15.44:0/2249830517 wait complete. 2024-05-27T09:43:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.270+0000 7f1e53272740 20 client.26710 trim_cache size 0 max 16384 2024-05-27T09:43:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:43:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:43:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:43:09.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 -- 172.21.15.44:0/2918115299 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e2c0b28e0 msgr2=0x7f1e2c0b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:09.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e2c0b28e0 0x7f1e2c0b4df0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1e30005ef0 tx=0x7f1e30005fd0 comp rx=0 tx=0).stop 2024-05-27T09:43:09.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 -- 172.21.15.44:0/2918115299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e44104bb0 msgr2=0x7f1e44105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:09.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e44104bb0 0x7f1e44105690 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800e9f0 tx=0x7f1e3800eeb0 comp rx=0 tx=0).stop 2024-05-27T09:43:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 -- 172.21.15.44:0/2918115299 shutdown_connections 2024-05-27T09:43:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e4410e000 0x7f1e44106110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e2c0b28e0 0x7f1e2c0b4df0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e4410b610 0x7f1e44105bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 --2- 172.21.15.44:0/2918115299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e44104bb0 0x7f1e44105690 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 -- 172.21.15.44:0/2918115299 >> 172.21.15.44:0/2918115299 conn(0x7f1e440fe210 msgr2=0x7f1e44109db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 -- 172.21.15.44:0/2918115299 shutdown_connections 2024-05-27T09:43:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:09.271+0000 7f1e4ac00640 1 -- 172.21.15.44:0/2918115299 wait complete. 2024-05-27T09:43:09.552 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:43:09.945 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:09 smithi044 ceph-mon[31618]: pgmap v245: 97 pgs: 97 active+clean; 101 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.7 MiB/s wr, 103 op/s 2024-05-27T09:43:10.095 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:43:10.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:09 smithi050 ceph-mon[36371]: pgmap v245: 97 pgs: 97 active+clean; 101 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.7 MiB/s wr, 103 op/s 2024-05-27T09:43:10.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:09 smithi189 ceph-mon[36671]: pgmap v245: 97 pgs: 97 active+clean; 101 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.7 MiB/s wr, 103 op/s 2024-05-27T09:43:12.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.066+0000 7fcd69800700 1 -- 172.21.15.44:0/1580837379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd641043f0 msgr2=0x7fcd64104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:12.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.066+0000 7fcd69800700 1 --2- 172.21.15.44:0/1580837379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd641043f0 0x7fcd64104800 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009990 tx=0x7fcd54009ca0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.066+0000 7fcd69800700 1 -- 172.21.15.44:0/1580837379 shutdown_connections 2024-05-27T09:43:12.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.066+0000 7fcd69800700 1 --2- 172.21.15.44:0/1580837379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd64109670 0x7fcd64110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.066+0000 7fcd69800700 1 --2- 172.21.15.44:0/1580837379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd641043f0 0x7fcd64104800 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.066+0000 7fcd69800700 1 --2- 172.21.15.44:0/1580837379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd64108d20 0x7fcd64109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.066+0000 7fcd69800700 1 -- 172.21.15.44:0/1580837379 >> 172.21.15.44:0/1580837379 conn(0x7fcd64079d00 msgr2=0x7fcd6407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:12.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.066+0000 7fcd69800700 1 -- 172.21.15.44:0/1580837379 shutdown_connections 2024-05-27T09:43:12.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.066+0000 7fcd69800700 1 -- 172.21.15.44:0/1580837379 wait complete. 2024-05-27T09:43:12.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.068+0000 7fcd69800700 1 Processor -- start 2024-05-27T09:43:12.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.068+0000 7fcd69800700 1 -- start start 2024-05-27T09:43:12.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.069+0000 7fcd69800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd641043f0 0x7fcd641a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.069+0000 7fcd69800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd64108d20 0x7fcd641a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.069+0000 7fcd69800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd64109670 0x7fcd6419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.069+0000 7fcd69800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcd641142a0 con 0x7fcd641043f0 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.069+0000 7fcd69800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcd64114120 con 0x7fcd64109670 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.069+0000 7fcd69800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcd64114420 con 0x7fcd64108d20 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.069+0000 7fcd63400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd64108d20 0x7fcd641a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.069+0000 7fcd63400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd64108d20 0x7fcd641a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44752/0 (socket says 172.21.15.44:44752) 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.069+0000 7fcd63400700 1 -- 172.21.15.44:0/2335547149 learned_addr learned my addr 172.21.15.44:0/2335547149 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.069+0000 7fcd63e00700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd641043f0 0x7fcd641a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.069+0000 7fcd68e00700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd64109670 0x7fcd6419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.070+0000 7fcd63400700 1 -- 172.21.15.44:0/2335547149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd64109670 msgr2=0x7fcd6419aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.070+0000 7fcd63400700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd64109670 0x7fcd6419aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.070+0000 7fcd63400700 1 -- 172.21.15.44:0/2335547149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd641043f0 msgr2=0x7fcd641a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:12.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.070+0000 7fcd63400700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd641043f0 0x7fcd641a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.070+0000 7fcd63400700 1 -- 172.21.15.44:0/2335547149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd6419b150 con 0x7fcd64108d20 2024-05-27T09:43:12.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.070+0000 7fcd63400700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd64108d20 0x7fcd641a0dc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcd5000e970 tx=0x7fcd5000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:12.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.070+0000 7fcd60c00700 1 -- 172.21.15.44:0/2335547149 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd50004a30 con 0x7fcd64108d20 2024-05-27T09:43:12.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.070+0000 7fcd60c00700 1 -- 172.21.15.44:0/2335547149 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd5000cc90 con 0x7fcd64108d20 2024-05-27T09:43:12.073 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:11 smithi044 ceph-mon[31618]: pgmap v246: 97 pgs: 97 active+clean; 97 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.1 MiB/s wr, 810 op/s 2024-05-27T09:43:12.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.070+0000 7fcd60c00700 1 -- 172.21.15.44:0/2335547149 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd50005610 con 0x7fcd64108d20 2024-05-27T09:43:12.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.070+0000 7fcd69800700 1 -- 172.21.15.44:0/2335547149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd6419b440 con 0x7fcd64108d20 2024-05-27T09:43:12.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.071+0000 7fcd69800700 1 -- 172.21.15.44:0/2335547149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd641a7710 con 0x7fcd64108d20 2024-05-27T09:43:12.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.076+0000 7fcd798a4b80 1 Processor -- start 2024-05-27T09:43:12.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.076+0000 7fcd798a4b80 1 -- start start 2024-05-27T09:43:12.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.077+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e117c5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.077+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e11645e40 0x559e117c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.077+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e117cf2c0 0x559e117d16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.077+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559e116f2e40 con 0x559e117cf2c0 2024-05-27T09:43:12.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.077+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559e116e29e0 con 0x559e11645e40 2024-05-27T09:43:12.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.077+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559e116e6a20 con 0x559e117319f0 2024-05-27T09:43:12.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.077+0000 7fcd68e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e117c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.077+0000 7fcd63e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e11645e40 0x559e117c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.077+0000 7fcd68e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e117c5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44762/0 (socket says 172.21.15.44:44762) 2024-05-27T09:43:12.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.077+0000 7fcd68e00700 1 -- 172.21.15.44:0/113589176 learned_addr learned my addr 172.21.15.44:0/113589176 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:12.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.078+0000 7fcd63e00700 1 -- 172.21.15.44:0/113589176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 msgr2=0x559e117c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:12.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.078+0000 7fcd63e00700 1 --2- 172.21.15.44:0/113589176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e117c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.078+0000 7fcd63e00700 1 -- 172.21.15.44:0/113589176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e117cf2c0 msgr2=0x559e117d16e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:43:12.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.078+0000 7fcd63e00700 1 --2- 172.21.15.44:0/113589176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e117cf2c0 0x559e117d16e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.078+0000 7fcd63e00700 1 -- 172.21.15.44:0/113589176 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e116ab420 con 0x559e11645e40 2024-05-27T09:43:12.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.078+0000 7fcd60c00700 1 -- 172.21.15.44:0/2335547149 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fcd50021440 con 0x7fcd64108d20 2024-05-27T09:43:12.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.078+0000 7fcd63e00700 1 --2- 172.21.15.44:0/113589176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e11645e40 0x559e117c5c90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fcd54006010 tx=0x7fcd54005f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b27908f686957f03 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:12.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.078+0000 7fcd59600700 1 -- 172.21.15.44:0/113589176 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54027070 con 0x559e11645e40 2024-05-27T09:43:12.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.078+0000 7fcd59600700 1 -- 172.21.15.44:0/113589176 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd540056b0 con 0x559e11645e40 2024-05-27T09:43:12.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.078+0000 7fcd59600700 1 -- 172.21.15.44:0/113589176 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54003ad0 con 0x559e11645e40 2024-05-27T09:43:12.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.079+0000 7fcd60c00700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcd480b2af0 0x7fcd480b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.080+0000 7fcd60c00700 1 -- 172.21.15.44:0/2335547149 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fcd500e2790 con 0x7fcd64108d20 2024-05-27T09:43:12.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.081+0000 7fcd798a4b80 1 -- 172.21.15.44:0/113589176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e11645e40 msgr2=0x559e117c5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:12.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.081+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/113589176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e11645e40 0x559e117c5c90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fcd54006010 tx=0x7fcd54005f00 comp rx=0 tx=0).stop 2024-05-27T09:43:12.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.081+0000 7fcd798a4b80 1 -- 172.21.15.44:0/113589176 shutdown_connections 2024-05-27T09:43:12.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.081+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/113589176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e117cf2c0 0x559e117d16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.081+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/113589176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e117c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.081+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/113589176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e11645e40 0x559e117c5c90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.081+0000 7fcd798a4b80 1 -- 172.21.15.44:0/113589176 >> 172.21.15.44:0/113589176 conn(0x559e116df390 msgr2=0x559e117c0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:12.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.081+0000 7fcd68e00700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcd480b2af0 0x7fcd480b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.081+0000 7fcd798a4b80 1 -- 172.21.15.44:0/113589176 shutdown_connections 2024-05-27T09:43:12.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.081+0000 7fcd798a4b80 1 -- 172.21.15.44:0/113589176 wait complete. 2024-05-27T09:43:12.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.082+0000 7fcd68e00700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcd480b2af0 0x7fcd480b5000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x559e117c1a90 tx=0x7fcd5c00b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:12.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.082+0000 7fcd798a4b80 1 -- start start 2024-05-27T09:43:12.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.082+0000 7fcd798a4b80 1 Processor -- start 2024-05-27T09:43:12.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.083+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e11645e40 0x559e117ced70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.083+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e117319f0 0x559e117cc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.083+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.083+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559e115e8fa0 con 0x559e117319f0 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.083+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559e11683760 con 0x559e117cf2c0 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.083+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559e117d61b0 con 0x559e11645e40 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.083+0000 7fcd798a4b80 10 client.0 _ll_register_callbacks cb 0x559e1168cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.083+0000 7fcd68e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.083+0000 7fcd68e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117cc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49722/0 (socket says 172.21.15.44:49722) 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.083+0000 7fcd68e00700 1 -- 172.21.15.44:0/1278329378 learned_addr learned my addr 172.21.15.44:0/1278329378 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.084+0000 7fcd63e00700 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e11645e40 0x559e117ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.084+0000 7fcd68e00700 1 -- 172.21.15.44:0/1278329378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e11645e40 msgr2=0x559e117ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.084+0000 7fcd68e00700 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e11645e40 0x559e117ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.084+0000 7fcd68e00700 1 -- 172.21.15.44:0/1278329378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e117319f0 msgr2=0x559e117cc360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.084+0000 7fcd63400700 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e117319f0 0x559e117cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.084+0000 7fcd68e00700 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e117319f0 0x559e117cc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.084+0000 7fcd68e00700 1 -- 172.21.15.44:0/1278329378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e116328d0 con 0x559e117cf2c0 2024-05-27T09:43:12.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.084+0000 7fcd63e00700 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e11645e40 0x559e117ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:43:12.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.084+0000 7fcd68e00700 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117cc8c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c016de0 tx=0x7fcd5c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f776d61957ecbfd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:12.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.084+0000 7fcd4e000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:43:12.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.085+0000 7fcd4e000700 1 -- 172.21.15.44:0/1278329378 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c027020 con 0x559e117cf2c0 2024-05-27T09:43:12.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.085+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1278329378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd540056b0 con 0x559e117cf2c0 2024-05-27T09:43:12.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.085+0000 7fcd4e000700 1 -- 172.21.15.44:0/1278329378 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fcd5c015c30 con 0x559e117cf2c0 2024-05-27T09:43:12.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.085+0000 7fcd4e000700 1 -- 172.21.15.44:0/1278329378 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcd5c015e30 con 0x559e117cf2c0 2024-05-27T09:43:12.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.085+0000 7fcd798a4b80 10 client.26750 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:12.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.085+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1278329378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559e116b3060 con 0x559e117cf2c0 2024-05-27T09:43:12.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.086+0000 7fcd4e000700 1 -- 172.21.15.44:0/1278329378 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd540056b0 con 0x559e117cf2c0 2024-05-27T09:43:12.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.086+0000 7fcd4e000700 1 -- 172.21.15.44:0/1278329378 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fcd5c023bf0 con 0x559e117cf2c0 2024-05-27T09:43:12.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.086+0000 7fcd798a4b80 10 client.26750 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:12.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.086+0000 7fcd798a4b80 10 client.26750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:12.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.086+0000 7fcd798a4b80 20 client.26750 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:12.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.086+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559e117e7670 0x559e117e9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.087+0000 7fcd798a4b80 4 client.26750 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:43:12.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.087+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1278329378 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e116f4d90 con 0x559e117e7670 2024-05-27T09:43:12.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.087+0000 7fcd63e00700 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559e117e7670 0x559e117e9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.087+0000 7fcd63e00700 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559e117e7670 0x559e117e9a90 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc8d4b17ce48d221 server_cookie=f4d5ba46a41db927 in_seq=0 out_seq=0 2024-05-27T09:43:12.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.087+0000 7fcd4e000700 10 client.26750 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:12.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.088+0000 7fcd4e000700 1 -- 172.21.15.44:0/1278329378 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559e116f4d90 con 0x559e117e7670 2024-05-27T09:43:12.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.088+0000 7fcd4e000700 10 client.26750 handle_command_reply: tid=0 2024-05-27T09:43:12.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.088+0000 7fcd798a4b80 1 client.26750 shutdown 2024-05-27T09:43:12.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.089+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1278329378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 msgr2=0x559e117cc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:12.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.089+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117cc8c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c016de0 tx=0x7fcd5c004b40 comp rx=0 tx=0).stop 2024-05-27T09:43:12.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.089+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1278329378 shutdown_connections 2024-05-27T09:43:12.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.089+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559e117e7670 0x559e117e9a90 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.089+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117cc8c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.089+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e11645e40 0x559e117ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.089+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/1278329378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e117319f0 0x559e117cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.089+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1278329378 >> 172.21.15.44:0/1278329378 conn(0x559e116df390 msgr2=0x559e117c57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:12.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.089+0000 7fcd4e000700 0 client.26750 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:12.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.090+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1278329378 shutdown_connections 2024-05-27T09:43:12.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.090+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1278329378 wait complete. 2024-05-27T09:43:12.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.090+0000 7fcd798a4b80 20 client.26750 trim_cache size 0 max 16384 2024-05-27T09:43:12.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.110+0000 7fcd798a4b80 1 Processor -- start 2024-05-27T09:43:12.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.110+0000 7fcd798a4b80 1 -- start start 2024-05-27T09:43:12.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.110+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e11645e40 0x559e117cc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.110+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e117d4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.110+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117ef270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.110+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcd641af370 con 0x559e11645e40 2024-05-27T09:43:12.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.110+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcd641acbd0 con 0x559e117cf2c0 2024-05-27T09:43:12.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.110+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcd641aaa40 con 0x559e117319f0 2024-05-27T09:43:12.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.111+0000 7fcd68e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.111+0000 7fcd68e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117ef270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49730/0 (socket says 172.21.15.44:49730) 2024-05-27T09:43:12.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.111+0000 7fcd68e00700 1 -- 172.21.15.44:0/1334821942 learned_addr learned my addr 172.21.15.44:0/1334821942 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:12.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.111+0000 7fcd63e00700 1 --2- 172.21.15.44:0/1334821942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e11645e40 0x559e117cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.111+0000 7fcd63400700 1 --2- 172.21.15.44:0/1334821942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e117d4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.111+0000 7fcd68e00700 1 -- 172.21.15.44:0/1334821942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 msgr2=0x559e117d4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:12.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.111+0000 7fcd68e00700 1 --2- 172.21.15.44:0/1334821942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e117d4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.111+0000 7fcd68e00700 1 -- 172.21.15.44:0/1334821942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e11645e40 msgr2=0x559e117cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:12.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.111+0000 7fcd68e00700 1 --2- 172.21.15.44:0/1334821942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e11645e40 0x559e117cc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.111+0000 7fcd68e00700 1 -- 172.21.15.44:0/1334821942 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e117ca8d0 con 0x559e117cf2c0 2024-05-27T09:43:12.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.112+0000 7fcd68e00700 1 --2- 172.21.15.44:0/1334821942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117ef270 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c016010 tx=0x7fcd5c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6616ed973e36a39c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:12.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.112+0000 7fcd58c00700 1 -- 172.21.15.44:0/1334821942 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c027030 con 0x559e117cf2c0 2024-05-27T09:43:12.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.112+0000 7fcd58c00700 1 -- 172.21.15.44:0/1334821942 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd5c03b020 con 0x559e117cf2c0 2024-05-27T09:43:12.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.112+0000 7fcd58c00700 1 -- 172.21.15.44:0/1334821942 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c037b50 con 0x559e117cf2c0 2024-05-27T09:43:12.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.113+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1334821942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 msgr2=0x559e117ef270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:12.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.113+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/1334821942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117ef270 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c016010 tx=0x7fcd5c009920 comp rx=0 tx=0).stop 2024-05-27T09:43:12.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.113+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1334821942 shutdown_connections 2024-05-27T09:43:12.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.113+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/1334821942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117ef270 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.113+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/1334821942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e11645e40 0x559e117cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.113+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/1334821942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e117d4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.113+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1334821942 >> 172.21.15.44:0/1334821942 conn(0x559e116df390 msgr2=0x559e117cc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:12.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.114+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1334821942 shutdown_connections 2024-05-27T09:43:12.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.114+0000 7fcd798a4b80 1 -- 172.21.15.44:0/1334821942 wait complete. 2024-05-27T09:43:12.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.114+0000 7fcd798a4b80 1 -- start start 2024-05-27T09:43:12.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.115+0000 7fcd798a4b80 1 Processor -- start 2024-05-27T09:43:12.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.115+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e11645e40 0x559e1184a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.115+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e1184aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.115+0000 7fcd798a4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117df480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.115+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcd641ab090 con 0x559e11645e40 2024-05-27T09:43:12.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.115+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcd641ac500 con 0x559e117cf2c0 2024-05-27T09:43:12.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.115+0000 7fcd798a4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcd641ab760 con 0x559e117319f0 2024-05-27T09:43:12.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.115+0000 7fcd798a4b80 10 client.0 _ll_register_callbacks cb 0x559e1168cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:12.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd63400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e1184aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd68e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117df480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd63e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e11645e40 0x559e1184a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd63400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e1184aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44808/0 (socket says 172.21.15.44:44808) 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd63400700 1 -- 172.21.15.44:0/480671538 learned_addr learned my addr 172.21.15.44:0/480671538 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd68e00700 1 -- 172.21.15.44:0/480671538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 msgr2=0x559e1184aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd68e00700 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e1184aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd68e00700 1 -- 172.21.15.44:0/480671538 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e11645e40 msgr2=0x559e1184a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd68e00700 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e11645e40 0x559e1184a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd68e00700 1 -- 172.21.15.44:0/480671538 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e117dfac0 con 0x559e117cf2c0 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd63e00700 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e11645e40 0x559e1184a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.116+0000 7fcd63400700 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e1184aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.117+0000 7fcd68e00700 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117df480 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c0095b0 tx=0x7fcd5c004df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa31d26fd119f726 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.117+0000 7fcd4e000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.117+0000 7fcd4e000700 1 -- 172.21.15.44:0/480671538 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c044070 con 0x559e117cf2c0 2024-05-27T09:43:12.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.117+0000 7fcd4e000700 1 -- 172.21.15.44:0/480671538 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7fcd5c037850 con 0x559e117cf2c0 2024-05-27T09:43:12.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.118+0000 7fcd798a4b80 1 -- 172.21.15.44:0/480671538 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd5c03b020 con 0x559e117cf2c0 2024-05-27T09:43:12.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.118+0000 7fcd4e000700 1 -- 172.21.15.44:0/480671538 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcd5c04e050 con 0x559e117cf2c0 2024-05-27T09:43:12.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.118+0000 7fcd798a4b80 10 client.26756 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:12.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.118+0000 7fcd798a4b80 1 -- 172.21.15.44:0/480671538 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559e11851b50 con 0x559e117cf2c0 2024-05-27T09:43:12.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.119+0000 7fcd4e000700 1 -- 172.21.15.44:0/480671538 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd5c03b020 con 0x559e117cf2c0 2024-05-27T09:43:12.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.119+0000 7fcd4e000700 1 -- 172.21.15.44:0/480671538 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7fcd5c049c30 con 0x559e117cf2c0 2024-05-27T09:43:12.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.120+0000 7fcd798a4b80 10 client.26756 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:12.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.120+0000 7fcd798a4b80 10 client.26756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:12.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.120+0000 7fcd798a4b80 20 client.26756 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:12.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.120+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559e11852170 0x559e11854590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:12.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.120+0000 7fcd798a4b80 4 client.26756 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:43:12.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.120+0000 7fcd798a4b80 1 -- 172.21.15.44:0/480671538 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559e116ee570 con 0x559e11852170 2024-05-27T09:43:12.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.120+0000 7fcd63e00700 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559e11852170 0x559e11854590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.121+0000 7fcd63e00700 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559e11852170 0x559e11854590 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d361c99c24728468 server_cookie=7a99a5481002370 in_seq=0 out_seq=0 2024-05-27T09:43:12.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.121+0000 7fcd4e000700 10 client.26756 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:12.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd4e000700 1 -- 172.21.15.44:0/480671538 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559e116ee570 con 0x559e11852170 2024-05-27T09:43:12.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd4e000700 10 client.26756 handle_command_reply: tid=0 2024-05-27T09:43:12.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd798a4b80 1 client.26756 shutdown 2024-05-27T09:43:12.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd798a4b80 1 -- 172.21.15.44:0/480671538 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 msgr2=0x559e117df480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:12.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117df480 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c0095b0 tx=0x7fcd5c004df0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd798a4b80 1 -- 172.21.15.44:0/480671538 shutdown_connections 2024-05-27T09:43:12.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559e11852170 0x559e11854590 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e117cf2c0 0x559e117df480 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559e11645e40 0x559e1184a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd798a4b80 1 --2- 172.21.15.44:0/480671538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559e117319f0 0x559e1184aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd798a4b80 1 -- 172.21.15.44:0/480671538 >> 172.21.15.44:0/480671538 conn(0x559e116df390 msgr2=0x559e117c7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:12.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.123+0000 7fcd4e000700 0 client.26756 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:12.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.124+0000 7fcd798a4b80 1 -- 172.21.15.44:0/480671538 shutdown_connections 2024-05-27T09:43:12.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.124+0000 7fcd798a4b80 1 -- 172.21.15.44:0/480671538 wait complete. 2024-05-27T09:43:12.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.124+0000 7fcd798a4b80 20 client.26756 trim_cache size 0 max 16384 2024-05-27T09:43:12.126 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:43:12.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 -- 172.21.15.44:0/2335547149 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcd480b2af0 msgr2=0x7fcd480b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:12.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcd480b2af0 0x7fcd480b5000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x559e117c1a90 tx=0x7fcd5c00b040 comp rx=0 tx=0).stop 2024-05-27T09:43:12.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 -- 172.21.15.44:0/2335547149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd64108d20 msgr2=0x7fcd641a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:12.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd64108d20 0x7fcd641a0dc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcd5000e970 tx=0x7fcd5000ec80 comp rx=0 tx=0).stop 2024-05-27T09:43:12.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 -- 172.21.15.44:0/2335547149 shutdown_connections 2024-05-27T09:43:12.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcd480b2af0 0x7fcd480b5000 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd64109670 0x7fcd6419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd641043f0 0x7fcd641a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 --2- 172.21.15.44:0/2335547149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd64108d20 0x7fcd641a0dc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 -- 172.21.15.44:0/2335547149 >> 172.21.15.44:0/2335547149 conn(0x7fcd64079d00 msgr2=0x7fcd64102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 -- 172.21.15.44:0/2335547149 shutdown_connections 2024-05-27T09:43:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:12.125+0000 7fcd69800700 1 -- 172.21.15.44:0/2335547149 wait complete. 2024-05-27T09:43:12.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:11 smithi050 ceph-mon[36371]: pgmap v246: 97 pgs: 97 active+clean; 97 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.1 MiB/s wr, 810 op/s 2024-05-27T09:43:12.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:11 smithi189 ceph-mon[36671]: pgmap v246: 97 pgs: 97 active+clean; 97 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.1 MiB/s wr, 810 op/s 2024-05-27T09:43:13.854 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:43:13.854 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:43:14.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:13 smithi044 ceph-mon[31618]: pgmap v247: 97 pgs: 97 active+clean; 97 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.9 MiB/s wr, 791 op/s 2024-05-27T09:43:14.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:13 smithi050 ceph-mon[36371]: pgmap v247: 97 pgs: 97 active+clean; 97 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.9 MiB/s wr, 791 op/s 2024-05-27T09:43:14.300 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:43:14.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:13 smithi189 ceph-mon[36671]: pgmap v247: 97 pgs: 97 active+clean; 97 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.9 MiB/s wr, 791 op/s 2024-05-27T09:43:16.093 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:15 smithi044 ceph-mon[31618]: pgmap v248: 97 pgs: 97 active+clean; 97 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.9 MiB/s wr, 791 op/s 2024-05-27T09:43:16.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:15 smithi050 ceph-mon[36371]: pgmap v248: 97 pgs: 97 active+clean; 97 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.9 MiB/s wr, 791 op/s 2024-05-27T09:43:16.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.328+0000 7ff608a00700 1 -- 172.21.15.44:0/2538425447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff604108d10 msgr2=0x7ff604109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:16.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.328+0000 7ff608a00700 1 --2- 172.21.15.44:0/2538425447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff604108d10 0x7ff604109120 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009920 tx=0x7ff5f0009c30 comp rx=0 tx=0).stop 2024-05-27T09:43:16.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.329+0000 7ff608a00700 1 -- 172.21.15.44:0/2538425447 shutdown_connections 2024-05-27T09:43:16.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.329+0000 7ff608a00700 1 --2- 172.21.15.44:0/2538425447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff604109660 0x7ff604110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.329+0000 7ff608a00700 1 --2- 172.21.15.44:0/2538425447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6041043e0 0x7ff6041047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.329+0000 7ff608a00700 1 --2- 172.21.15.44:0/2538425447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff604108d10 0x7ff604109120 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.329+0000 7ff608a00700 1 -- 172.21.15.44:0/2538425447 >> 172.21.15.44:0/2538425447 conn(0x7ff604079c00 msgr2=0x7ff60407a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:16.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.329+0000 7ff608a00700 1 -- 172.21.15.44:0/2538425447 shutdown_connections 2024-05-27T09:43:16.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.329+0000 7ff608a00700 1 -- 172.21.15.44:0/2538425447 wait complete. 2024-05-27T09:43:16.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.330+0000 7ff608a00700 1 Processor -- start 2024-05-27T09:43:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.331+0000 7ff608a00700 1 -- start start 2024-05-27T09:43:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.331+0000 7ff608a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6041043e0 0x7ff604075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.331+0000 7ff608a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff604108d10 0x7ff604075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.331+0000 7ff608a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff604109660 0x7ff604076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.331+0000 7ff608a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041142c0 con 0x7ff604109660 2024-05-27T09:43:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.331+0000 7ff608a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff604114140 con 0x7ff604108d10 2024-05-27T09:43:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.331+0000 7ff608a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff604114440 con 0x7ff6041043e0 2024-05-27T09:43:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.331+0000 7ff603400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6041043e0 0x7ff604075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.331+0000 7ff603400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6041043e0 0x7ff604075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59888/0 (socket says 172.21.15.44:59888) 2024-05-27T09:43:16.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.331+0000 7ff603400700 1 -- 172.21.15.44:0/509588420 learned_addr learned my addr 172.21.15.44:0/509588420 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.332+0000 7ff603400700 1 -- 172.21.15.44:0/509588420 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff604108d10 msgr2=0x7ff604075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.332+0000 7ff603400700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff604108d10 0x7ff604075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.332+0000 7ff603400700 1 -- 172.21.15.44:0/509588420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff604109660 msgr2=0x7ff604076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.332+0000 7ff603400700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff604109660 0x7ff604076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.332+0000 7ff603400700 1 -- 172.21.15.44:0/509588420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6041a90d0 con 0x7ff6041043e0 2024-05-27T09:43:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.333+0000 7ff603400700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6041043e0 0x7ff604075ab0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400ec10 tx=0x7ff5f400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.333+0000 7ff5fbe00700 1 -- 172.21.15.44:0/509588420 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f400cd60 con 0x7ff6041043e0 2024-05-27T09:43:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.333+0000 7ff5fbe00700 1 -- 172.21.15.44:0/509588420 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5f40044d0 con 0x7ff6041043e0 2024-05-27T09:43:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.333+0000 7ff608a00700 1 -- 172.21.15.44:0/509588420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6041a93c0 con 0x7ff6041043e0 2024-05-27T09:43:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.333+0000 7ff5fbe00700 1 -- 172.21.15.44:0/509588420 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4010690 con 0x7ff6041043e0 2024-05-27T09:43:16.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.333+0000 7ff608a00700 1 -- 172.21.15.44:0/509588420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6041a9950 con 0x7ff6041043e0 2024-05-27T09:43:16.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:15 smithi189 ceph-mon[36671]: pgmap v248: 97 pgs: 97 active+clean; 97 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.9 MiB/s wr, 791 op/s 2024-05-27T09:43:16.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.337+0000 7ff5fbe00700 1 -- 172.21.15.44:0/509588420 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff5f4010830 con 0x7ff6041043e0 2024-05-27T09:43:16.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.338+0000 7ff618ac6b80 1 Processor -- start 2024-05-27T09:43:16.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.339+0000 7ff618ac6b80 1 -- start start 2024-05-27T09:43:16.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.339+0000 7ff5fbe00700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff5e80b2a70 0x7ff5e80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.339+0000 7ff5fbe00700 1 -- 172.21.15.44:0/509588420 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7ff5f40e29d0 con 0x7ff6041043e0 2024-05-27T09:43:16.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.339+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 0x55eb3951ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.339+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39529b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.339+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3952c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.339+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eb3944dec0 con 0x55eb3951f8a0 2024-05-27T09:43:16.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.339+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55eb3943da60 con 0x55eb3951f490 2024-05-27T09:43:16.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.339+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55eb39441aa0 con 0x55eb393a0ec0 2024-05-27T09:43:16.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.339+0000 7ff603e00700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff5e80b2a70 0x7ff5e80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.340+0000 7ff603400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 0x55eb3951ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.340+0000 7ff602a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39529b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.340+0000 7ff603400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 0x55eb3951ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59898/0 (socket says 172.21.15.44:59898) 2024-05-27T09:43:16.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.340+0000 7ff603400700 1 -- 172.21.15.44:0/2408960932 learned_addr learned my addr 172.21.15.44:0/2408960932 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:16.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.340+0000 7ff603e00700 1 --2- 172.21.15.44:0/2408960932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3952c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.340+0000 7ff603400700 1 -- 172.21.15.44:0/2408960932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 msgr2=0x55eb39529b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:16.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.340+0000 7ff603400700 1 --2- 172.21.15.44:0/2408960932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39529b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.340+0000 7ff603400700 1 -- 172.21.15.44:0/2408960932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 msgr2=0x55eb3952c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:16.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.340+0000 7ff603400700 1 --2- 172.21.15.44:0/2408960932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3952c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.340+0000 7ff603400700 1 -- 172.21.15.44:0/2408960932 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb394064a0 con 0x55eb393a0ec0 2024-05-27T09:43:16.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.341+0000 7ff603400700 1 --2- 172.21.15.44:0/2408960932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 0x55eb3951ef50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4005d70 tx=0x7ff5f40a9aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3404844ad9015756 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:16.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.341+0000 7ff603e00700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff5e80b2a70 0x7ff5e80b4f80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55eb3951fd30 tx=0x7ff5fc00b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:16.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.341+0000 7ff5f8c00700 1 -- 172.21.15.44:0/2408960932 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40a9760 con 0x55eb393a0ec0 2024-05-27T09:43:16.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.341+0000 7ff5f8c00700 1 -- 172.21.15.44:0/2408960932 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5f40aa350 con 0x55eb393a0ec0 2024-05-27T09:43:16.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.341+0000 7ff5f8c00700 1 -- 172.21.15.44:0/2408960932 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40a5ee0 con 0x55eb393a0ec0 2024-05-27T09:43:16.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.342+0000 7ff618ac6b80 1 -- 172.21.15.44:0/2408960932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 msgr2=0x55eb3951ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:16.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.342+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/2408960932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 0x55eb3951ef50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4005d70 tx=0x7ff5f40a9aa0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.343+0000 7ff618ac6b80 1 -- 172.21.15.44:0/2408960932 shutdown_connections 2024-05-27T09:43:16.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.343+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/2408960932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3952c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.343+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/2408960932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 0x55eb3951ef50 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.343+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/2408960932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39529b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.343+0000 7ff618ac6b80 1 -- 172.21.15.44:0/2408960932 >> 172.21.15.44:0/2408960932 conn(0x55eb39401040 msgr2=0x55eb3951adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:16.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.343+0000 7ff618ac6b80 1 -- 172.21.15.44:0/2408960932 shutdown_connections 2024-05-27T09:43:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.343+0000 7ff618ac6b80 1 -- 172.21.15.44:0/2408960932 wait complete. 2024-05-27T09:43:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.344+0000 7ff618ac6b80 1 -- start start 2024-05-27T09:43:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.344+0000 7ff618ac6b80 1 Processor -- start 2024-05-27T09:43:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.345+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 0x55eb39529680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.345+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39529bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.345+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3952a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.345+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eb39344020 con 0x55eb3951f8a0 2024-05-27T09:43:16.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.345+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55eb393de7e0 con 0x55eb3951f490 2024-05-27T09:43:16.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.345+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55eb3952ca10 con 0x55eb393a0ec0 2024-05-27T09:43:16.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.345+0000 7ff618ac6b80 10 client.0 _ll_register_callbacks cb 0x55eb393e7cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:16.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.345+0000 7ff602a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39529bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.345+0000 7ff602a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39529bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42930/0 (socket says 172.21.15.44:42930) 2024-05-27T09:43:16.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.345+0000 7ff602a00700 1 -- 172.21.15.44:0/1434297652 learned_addr learned my addr 172.21.15.44:0/1434297652 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:16.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.346+0000 7ff603e00700 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3952a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.346+0000 7ff603400700 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 0x55eb39529680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.346+0000 7ff603400700 1 -- 172.21.15.44:0/1434297652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 msgr2=0x55eb39529bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:16.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.346+0000 7ff603400700 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39529bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.346+0000 7ff603400700 1 -- 172.21.15.44:0/1434297652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 msgr2=0x55eb3952a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:16.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.346+0000 7ff603400700 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3952a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.346+0000 7ff603400700 1 -- 172.21.15.44:0/1434297652 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb3938d950 con 0x55eb393a0ec0 2024-05-27T09:43:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.346+0000 7ff603e00700 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3952a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.347+0000 7ff603400700 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 0x55eb39529680 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55eb3951b710 tx=0x7ff5f40a5480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=315bd96e44d11b5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.347+0000 7ff5ed600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:43:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.347+0000 7ff5ed600700 1 -- 172.21.15.44:0/1434297652 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40c8070 con 0x55eb393a0ec0 2024-05-27T09:43:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.347+0000 7ff5ed600700 1 -- 172.21.15.44:0/1434297652 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7ff5f4010b20 con 0x55eb393a0ec0 2024-05-27T09:43:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.348+0000 7ff618ac6b80 1 -- 172.21.15.44:0/1434297652 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5f40aa350 con 0x55eb393a0ec0 2024-05-27T09:43:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.348+0000 7ff5ed600700 1 -- 172.21.15.44:0/1434297652 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff5f408b050 con 0x55eb393a0ec0 2024-05-27T09:43:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.348+0000 7ff618ac6b80 10 client.26755 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.348+0000 7ff618ac6b80 1 -- 172.21.15.44:0/1434297652 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55eb3940e0e0 con 0x55eb393a0ec0 2024-05-27T09:43:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.349+0000 7ff5ed600700 1 -- 172.21.15.44:0/1434297652 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5f40aa350 con 0x55eb393a0ec0 2024-05-27T09:43:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.349+0000 7ff5ed600700 1 -- 172.21.15.44:0/1434297652 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7ff5f40a9eb0 con 0x55eb393a0ec0 2024-05-27T09:43:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.349+0000 7ff618ac6b80 10 client.26755 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.349+0000 7ff618ac6b80 10 client.26755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.349+0000 7ff618ac6b80 20 client.26755 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.349+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eb39539860 0x55eb3953bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.349+0000 7ff618ac6b80 4 client.26755 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:43:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.349+0000 7ff618ac6b80 1 -- 172.21.15.44:0/1434297652 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb3944fe10 con 0x55eb39539860 2024-05-27T09:43:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.350+0000 7ff602a00700 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eb39539860 0x55eb3953bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.351+0000 7ff602a00700 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eb39539860 0x55eb3953bc80 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2db71239e542358 server_cookie=43cd360e1bfc1631 in_seq=0 out_seq=0 2024-05-27T09:43:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.351+0000 7ff5ed600700 10 client.26755 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.352+0000 7ff5ed600700 1 -- 172.21.15.44:0/1434297652 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55eb3944fe10 con 0x55eb39539860 2024-05-27T09:43:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.352+0000 7ff5ed600700 10 client.26755 handle_command_reply: tid=0 2024-05-27T09:43:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.352+0000 7ff618ac6b80 1 client.26755 shutdown 2024-05-27T09:43:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.352+0000 7ff618ac6b80 1 -- 172.21.15.44:0/1434297652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 msgr2=0x55eb39529680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.352+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 0x55eb39529680 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55eb3951b710 tx=0x7ff5f40a5480 comp rx=0 tx=0).stop 2024-05-27T09:43:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.352+0000 7ff618ac6b80 1 -- 172.21.15.44:0/1434297652 shutdown_connections 2024-05-27T09:43:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.352+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eb39539860 0x55eb3953bc80 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.352+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3952a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.352+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb393a0ec0 0x55eb39529680 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.352+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/1434297652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39529bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.352+0000 7ff618ac6b80 1 -- 172.21.15.44:0/1434297652 >> 172.21.15.44:0/1434297652 conn(0x55eb39401040 msgr2=0x55eb3951a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.353+0000 7ff5ed600700 0 client.26755 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:16.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.353+0000 7ff618ac6b80 1 -- 172.21.15.44:0/1434297652 shutdown_connections 2024-05-27T09:43:16.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.353+0000 7ff618ac6b80 1 -- 172.21.15.44:0/1434297652 wait complete. 2024-05-27T09:43:16.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.354+0000 7ff618ac6b80 20 client.26755 trim_cache size 0 max 16384 2024-05-27T09:43:16.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.378+0000 7ff618ac6b80 1 Processor -- start 2024-05-27T09:43:16.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.378+0000 7ff618ac6b80 1 -- start start 2024-05-27T09:43:16.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.379+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb393a0ec0 0x55eb39523820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.379+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39524e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.379+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f8a0 0x55eb39529b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.379+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041b3a60 con 0x55eb393a0ec0 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.379+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041b12c0 con 0x55eb3951f490 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.379+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041af130 con 0x55eb3951f8a0 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.379+0000 7ff602a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39524e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.379+0000 7ff602a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39524e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42942/0 (socket says 172.21.15.44:42942) 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.379+0000 7ff602a00700 1 -- 172.21.15.44:0/276939095 learned_addr learned my addr 172.21.15.44:0/276939095 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.379+0000 7ff603e00700 1 --2- 172.21.15.44:0/276939095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f8a0 0x55eb39529b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.379+0000 7ff603400700 1 --2- 172.21.15.44:0/276939095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb393a0ec0 0x55eb39523820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.380+0000 7ff602a00700 1 -- 172.21.15.44:0/276939095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f8a0 msgr2=0x55eb39529b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.380+0000 7ff602a00700 1 --2- 172.21.15.44:0/276939095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f8a0 0x55eb39529b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.380+0000 7ff602a00700 1 -- 172.21.15.44:0/276939095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb393a0ec0 msgr2=0x55eb39523820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.380+0000 7ff602a00700 1 --2- 172.21.15.44:0/276939095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb393a0ec0 0x55eb39523820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.380+0000 7ff602a00700 1 -- 172.21.15.44:0/276939095 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb3951b950 con 0x55eb3951f490 2024-05-27T09:43:16.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.380+0000 7ff603e00700 1 --2- 172.21.15.44:0/276939095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f8a0 0x55eb39529b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:16.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.380+0000 7ff603400700 1 --2- 172.21.15.44:0/276939095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb393a0ec0 0x55eb39523820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:16.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.380+0000 7ff602a00700 1 --2- 172.21.15.44:0/276939095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39524e50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009920 tx=0x7ff5f0005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbd91286a13de206 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.380+0000 7ff5efe00700 1 -- 172.21.15.44:0/276939095 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f002b020 con 0x55eb3951f490 2024-05-27T09:43:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.381+0000 7ff5efe00700 1 -- 172.21.15.44:0/276939095 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5f000f910 con 0x55eb3951f490 2024-05-27T09:43:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.381+0000 7ff5efe00700 1 -- 172.21.15.44:0/276939095 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0003680 con 0x55eb3951f490 2024-05-27T09:43:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.382+0000 7ff618ac6b80 1 -- 172.21.15.44:0/276939095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 msgr2=0x55eb39524e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.382+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/276939095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39524e50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009920 tx=0x7ff5f0005dc0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.382+0000 7ff618ac6b80 1 -- 172.21.15.44:0/276939095 shutdown_connections 2024-05-27T09:43:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.382+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/276939095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f8a0 0x55eb39529b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.382+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/276939095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb393a0ec0 0x55eb39523820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.382+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/276939095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb3951f490 0x55eb39524e50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.382+0000 7ff618ac6b80 1 -- 172.21.15.44:0/276939095 >> 172.21.15.44:0/276939095 conn(0x55eb39401040 msgr2=0x55eb39529460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:16.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.382+0000 7ff618ac6b80 1 -- 172.21.15.44:0/276939095 shutdown_connections 2024-05-27T09:43:16.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.382+0000 7ff618ac6b80 1 -- 172.21.15.44:0/276939095 wait complete. 2024-05-27T09:43:16.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.383+0000 7ff618ac6b80 1 -- start start 2024-05-27T09:43:16.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.383+0000 7ff618ac6b80 1 Processor -- start 2024-05-27T09:43:16.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.384+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb393a0ec0 0x55eb3953edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.384+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f490 0x55eb3953d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.384+0000 7ff618ac6b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3953d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.384+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041af780 con 0x55eb3951f8a0 2024-05-27T09:43:16.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.384+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041b0bf0 con 0x55eb393a0ec0 2024-05-27T09:43:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.384+0000 7ff618ac6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041afe50 con 0x55eb3951f490 2024-05-27T09:43:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.384+0000 7ff618ac6b80 10 client.0 _ll_register_callbacks cb 0x55eb393e7cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.384+0000 7ff603e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3953d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.384+0000 7ff603400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb393a0ec0 0x55eb3953edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.384+0000 7ff603400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb393a0ec0 0x55eb3953edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42944/0 (socket says 172.21.15.44:42944) 2024-05-27T09:43:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.385+0000 7ff603400700 1 -- 172.21.15.44:0/530222123 learned_addr learned my addr 172.21.15.44:0/530222123 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:16.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.384+0000 7ff602a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f490 0x55eb3953d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.385+0000 7ff603400700 1 -- 172.21.15.44:0/530222123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f490 msgr2=0x55eb3953d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.385+0000 7ff603400700 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f490 0x55eb3953d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.385+0000 7ff603400700 1 -- 172.21.15.44:0/530222123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 msgr2=0x55eb3953d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.385+0000 7ff603400700 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3953d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.385+0000 7ff603400700 1 -- 172.21.15.44:0/530222123 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb3951c3d0 con 0x55eb393a0ec0 2024-05-27T09:43:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.385+0000 7ff602a00700 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f490 0x55eb3953d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:16.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.386+0000 7ff603400700 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb393a0ec0 0x55eb3953edb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40a6ec0 tx=0x7ff5f4010d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3535d5d9ee01a97 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:16.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.386+0000 7ff5ed600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:43:16.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.386+0000 7ff5ed600700 1 -- 172.21.15.44:0/530222123 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40c8040 con 0x55eb393a0ec0 2024-05-27T09:43:16.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.386+0000 7ff618ac6b80 1 -- 172.21.15.44:0/530222123 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5f000f910 con 0x55eb393a0ec0 2024-05-27T09:43:16.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.386+0000 7ff5ed600700 1 -- 172.21.15.44:0/530222123 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7ff5f40a90c0 con 0x55eb393a0ec0 2024-05-27T09:43:16.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.386+0000 7ff5ed600700 1 -- 172.21.15.44:0/530222123 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff5f40a9760 con 0x55eb393a0ec0 2024-05-27T09:43:16.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.386+0000 7ff618ac6b80 10 client.26771 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:16.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.386+0000 7ff618ac6b80 1 -- 172.21.15.44:0/530222123 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55eb3951c3d0 con 0x55eb393a0ec0 2024-05-27T09:43:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.387+0000 7ff5ed600700 1 -- 172.21.15.44:0/530222123 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5f000f910 con 0x55eb393a0ec0 2024-05-27T09:43:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.387+0000 7ff5ed600700 1 -- 172.21.15.44:0/530222123 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7ff5f40a8920 con 0x55eb393a0ec0 2024-05-27T09:43:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.387+0000 7ff618ac6b80 10 client.26771 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.387+0000 7ff618ac6b80 10 client.26771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.387+0000 7ff618ac6b80 20 client.26771 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.387+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eb395b5f30 0x55eb395b8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.387+0000 7ff618ac6b80 4 client.26771 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:43:16.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.387+0000 7ff618ac6b80 1 -- 172.21.15.44:0/530222123 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55eb394495f0 con 0x55eb395b5f30 2024-05-27T09:43:16.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.388+0000 7ff602a00700 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eb395b5f30 0x55eb395b8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:16.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.388+0000 7ff602a00700 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eb395b5f30 0x55eb395b8350 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=accf76081d63020 server_cookie=31f0c70d2575e748 in_seq=0 out_seq=0 2024-05-27T09:43:16.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.388+0000 7ff5ed600700 10 client.26771 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:16.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.389+0000 7ff5ed600700 1 -- 172.21.15.44:0/530222123 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55eb394495f0 con 0x55eb395b5f30 2024-05-27T09:43:16.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.389+0000 7ff5ed600700 10 client.26771 handle_command_reply: tid=0 2024-05-27T09:43:16.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.389+0000 7ff618ac6b80 1 client.26771 shutdown 2024-05-27T09:43:16.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.390+0000 7ff618ac6b80 1 -- 172.21.15.44:0/530222123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb393a0ec0 msgr2=0x55eb3953edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:16.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.390+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb393a0ec0 0x55eb3953edb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40a6ec0 tx=0x7ff5f4010d50 comp rx=0 tx=0).stop 2024-05-27T09:43:16.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.391+0000 7ff618ac6b80 1 -- 172.21.15.44:0/530222123 shutdown_connections 2024-05-27T09:43:16.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.391+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55eb395b5f30 0x55eb395b8350 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.391+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eb3951f8a0 0x55eb3953d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.391+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55eb393a0ec0 0x55eb3953edb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.391+0000 7ff618ac6b80 1 --2- 172.21.15.44:0/530222123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eb3951f490 0x55eb3953d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.391+0000 7ff618ac6b80 1 -- 172.21.15.44:0/530222123 >> 172.21.15.44:0/530222123 conn(0x55eb39401040 msgr2=0x55eb39529e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:16.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.392+0000 7ff5ed600700 0 client.26771 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:16.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.392+0000 7ff618ac6b80 1 -- 172.21.15.44:0/530222123 shutdown_connections 2024-05-27T09:43:16.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.392+0000 7ff618ac6b80 1 -- 172.21.15.44:0/530222123 wait complete. 2024-05-27T09:43:16.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.393+0000 7ff618ac6b80 20 client.26771 trim_cache size 0 max 16384 2024-05-27T09:43:16.395 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:43:16.395 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:43:16.395 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "23fa7c2c-265d-4199-a6d4-f39b4f4a28a7", 2024-05-27T09:43:16.395 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:43:16.395 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:43:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.394+0000 7ff608a00700 1 -- 172.21.15.44:0/509588420 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff5e80b2a70 msgr2=0x7ff5e80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.394+0000 7ff608a00700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff5e80b2a70 0x7ff5e80b4f80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55eb3951fd30 tx=0x7ff5fc00b040 comp rx=0 tx=0).stop 2024-05-27T09:43:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.394+0000 7ff608a00700 1 -- 172.21.15.44:0/509588420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6041043e0 msgr2=0x7ff604075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.394+0000 7ff608a00700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6041043e0 0x7ff604075ab0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400ec10 tx=0x7ff5f400c2d0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.394+0000 7ff608a00700 1 -- 172.21.15.44:0/509588420 shutdown_connections 2024-05-27T09:43:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.394+0000 7ff608a00700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff5e80b2a70 0x7ff5e80b4f80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.394+0000 7ff608a00700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff604109660 0x7ff604076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.394+0000 7ff608a00700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6041043e0 0x7ff604075ab0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.394+0000 7ff608a00700 1 --2- 172.21.15.44:0/509588420 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff604108d10 0x7ff604075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.394+0000 7ff608a00700 1 -- 172.21.15.44:0/509588420 >> 172.21.15.44:0/509588420 conn(0x7ff604079c00 msgr2=0x7ff604101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.395+0000 7ff608a00700 1 -- 172.21.15.44:0/509588420 shutdown_connections 2024-05-27T09:43:16.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:16.395+0000 7ff608a00700 1 -- 172.21.15.44:0/509588420 wait complete. 2024-05-27T09:43:17.174 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:16 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:43:17.174 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:16 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:43:17.175 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:16 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:43:17.221 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:43:17.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:16 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:43:17.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:16 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:43:17.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:16 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:43:17.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:16 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:43:17.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:16 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:43:17.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:16 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:43:17.659 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:43:18.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:17 smithi044 ceph-mon[31618]: pgmap v249: 97 pgs: 97 active+clean; 111 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 759 op/s 2024-05-27T09:43:18.238 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:17 smithi189 ceph-mon[36671]: pgmap v249: 97 pgs: 97 active+clean; 111 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 759 op/s 2024-05-27T09:43:18.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:17 smithi050 ceph-mon[36371]: pgmap v249: 97 pgs: 97 active+clean; 111 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 759 op/s 2024-05-27T09:43:18.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:18 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:43:19.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:18 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:43:19.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:18 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:43:19.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.932+0000 7effb2a00700 1 -- 172.21.15.44:0/970050490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effac105bd0 msgr2=0x7effac10e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:19.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.932+0000 7effb2a00700 1 --2- 172.21.15.44:0/970050490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effac105bd0 0x7effac10e120 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7eff98009990 tx=0x7eff98009ca0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.932+0000 7effb2a00700 1 -- 172.21.15.44:0/970050490 shutdown_connections 2024-05-27T09:43:19.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.932+0000 7effb2a00700 1 --2- 172.21.15.44:0/970050490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effac10e6f0 0x7effac110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.932+0000 7effb2a00700 1 --2- 172.21.15.44:0/970050490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7effac105280 0x7effac105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.932+0000 7effb2a00700 1 --2- 172.21.15.44:0/970050490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effac105bd0 0x7effac10e120 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.932+0000 7effb2a00700 1 -- 172.21.15.44:0/970050490 >> 172.21.15.44:0/970050490 conn(0x7effac078ab0 msgr2=0x7effac078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:19.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.933+0000 7effb2a00700 1 -- 172.21.15.44:0/970050490 shutdown_connections 2024-05-27T09:43:19.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.933+0000 7effb2a00700 1 -- 172.21.15.44:0/970050490 wait complete. 2024-05-27T09:43:19.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.934+0000 7effb2a00700 1 Processor -- start 2024-05-27T09:43:19.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.934+0000 7effb2a00700 1 -- start start 2024-05-27T09:43:19.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.935+0000 7effb2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effac105280 0x7effac1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.935+0000 7effb2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effac105bd0 0x7effac1a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.935+0000 7effb2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7effac10e6f0 0x7effac19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.935+0000 7effb2a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7effac1142a0 con 0x7effac10e6f0 2024-05-27T09:43:19.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.935+0000 7effb2a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7effac114120 con 0x7effac105280 2024-05-27T09:43:19.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.935+0000 7effb2a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7effac114420 con 0x7effac105bd0 2024-05-27T09:43:19.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.935+0000 7effb1600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effac105280 0x7effac1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.935+0000 7effb0c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effac105bd0 0x7effac1a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.935+0000 7effb1600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effac105280 0x7effac1a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42960/0 (socket says 172.21.15.44:42960) 2024-05-27T09:43:19.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.935+0000 7effb1600700 1 -- 172.21.15.44:0/3222388544 learned_addr learned my addr 172.21.15.44:0/3222388544 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:19.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.935+0000 7effb2000700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7effac10e6f0 0x7effac19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.936+0000 7effb1600700 1 -- 172.21.15.44:0/3222388544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effac105bd0 msgr2=0x7effac1a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:19.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.936+0000 7effb1600700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effac105bd0 0x7effac1a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.936+0000 7effb1600700 1 -- 172.21.15.44:0/3222388544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7effac10e6f0 msgr2=0x7effac19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:19.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.936+0000 7effb1600700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7effac10e6f0 0x7effac19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.936+0000 7effb1600700 1 -- 172.21.15.44:0/3222388544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effac19b110 con 0x7effac105280 2024-05-27T09:43:19.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.936+0000 7effb0c00700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effac105bd0 0x7effac1a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:43:19.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.936+0000 7effb1600700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effac105280 0x7effac1a0840 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effa000e9a0 tx=0x7effa000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:19.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.937+0000 7effaa000700 1 -- 172.21.15.44:0/3222388544 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa000cbd0 con 0x7effac105280 2024-05-27T09:43:19.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.937+0000 7effaa000700 1 -- 172.21.15.44:0/3222388544 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effa000cd70 con 0x7effac105280 2024-05-27T09:43:19.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.937+0000 7effb2a00700 1 -- 172.21.15.44:0/3222388544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effac19b400 con 0x7effac105280 2024-05-27T09:43:19.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.937+0000 7effaa000700 1 -- 172.21.15.44:0/3222388544 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa0018680 con 0x7effac105280 2024-05-27T09:43:19.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.937+0000 7effb2a00700 1 -- 172.21.15.44:0/3222388544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effac1a76d0 con 0x7effac105280 2024-05-27T09:43:19.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.943+0000 7effaa000700 1 -- 172.21.15.44:0/3222388544 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7effa0018820 con 0x7effac105280 2024-05-27T09:43:19.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.944+0000 7effc2b83b80 1 Processor -- start 2024-05-27T09:43:19.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.945+0000 7effc2b83b80 1 -- start start 2024-05-27T09:43:19.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.945+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f42de40 0x562d4f5abed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.945+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac410 0x562d4f5b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.946+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac820 0x562d4f5b9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.946+0000 7effc2b83b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562d4f4dae40 con 0x562d4f5ac410 2024-05-27T09:43:19.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.946+0000 7effc2b83b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562d4f4ca9e0 con 0x562d4f42de40 2024-05-27T09:43:19.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.946+0000 7effc2b83b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562d4f4cea20 con 0x562d4f5ac820 2024-05-27T09:43:19.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.946+0000 7effb2000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f42de40 0x562d4f5abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.947+0000 7effb2000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f42de40 0x562d4f5abed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42970/0 (socket says 172.21.15.44:42970) 2024-05-27T09:43:19.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.947+0000 7effb2000700 1 -- 172.21.15.44:0/3476438747 learned_addr learned my addr 172.21.15.44:0/3476438747 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:19.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.947+0000 7effb0c00700 1 --2- 172.21.15.44:0/3476438747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac820 0x562d4f5b9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.947+0000 7effaa000700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff940b2af0 0x7eff940b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.947+0000 7effb1600700 1 --2- 172.21.15.44:0/3476438747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac410 0x562d4f5b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.947+0000 7effb0c00700 1 -- 172.21.15.44:0/3476438747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f42de40 msgr2=0x562d4f5abed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:19.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.947+0000 7effb0c00700 1 --2- 172.21.15.44:0/3476438747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f42de40 0x562d4f5abed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.947+0000 7effb0c00700 1 -- 172.21.15.44:0/3476438747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac410 msgr2=0x562d4f5b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:19.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.947+0000 7effb0c00700 1 --2- 172.21.15.44:0/3476438747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac410 0x562d4f5b6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.947+0000 7effb0c00700 1 -- 172.21.15.44:0/3476438747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d4f493420 con 0x562d4f5ac820 2024-05-27T09:43:19.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.947+0000 7effaa000700 1 -- 172.21.15.44:0/3222388544 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7effa00e28b0 con 0x7effac105280 2024-05-27T09:43:19.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.948+0000 7effb0c00700 1 --2- 172.21.15.44:0/3476438747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac820 0x562d4f5b9020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7effac1026e0 tx=0x7eff98005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecb95e02c8b4392d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:19.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.949+0000 7eff9ea00700 1 -- 172.21.15.44:0/3476438747 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff98009510 con 0x562d4f5ac820 2024-05-27T09:43:19.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.949+0000 7eff9ea00700 1 -- 172.21.15.44:0/3476438747 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff98004830 con 0x562d4f5ac820 2024-05-27T09:43:19.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.949+0000 7eff9ea00700 1 -- 172.21.15.44:0/3476438747 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff980054c0 con 0x562d4f5ac820 2024-05-27T09:43:19.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.949+0000 7effb2000700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff940b2af0 0x7eff940b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.951+0000 7effc2b83b80 1 -- 172.21.15.44:0/3476438747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac820 msgr2=0x562d4f5b9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:19.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.951+0000 7effb2000700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff940b2af0 0x7eff940b5000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x562d4f5a8690 tx=0x7effa4004b40 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:19.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.951+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3476438747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac820 0x562d4f5b9020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7effac1026e0 tx=0x7eff98005f00 comp rx=0 tx=0).stop 2024-05-27T09:43:19.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.951+0000 7effc2b83b80 1 -- 172.21.15.44:0/3476438747 shutdown_connections 2024-05-27T09:43:19.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.951+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3476438747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac820 0x562d4f5b9020 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.951+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3476438747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f42de40 0x562d4f5abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.951+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3476438747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac410 0x562d4f5b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.951+0000 7effc2b83b80 1 -- 172.21.15.44:0/3476438747 >> 172.21.15.44:0/3476438747 conn(0x562d4f48dfc0 msgr2=0x562d4f5a7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:19.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.951+0000 7effc2b83b80 1 -- 172.21.15.44:0/3476438747 shutdown_connections 2024-05-27T09:43:19.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.952+0000 7effc2b83b80 1 -- 172.21.15.44:0/3476438747 wait complete. 2024-05-27T09:43:19.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.952+0000 7effc2b83b80 1 -- start start 2024-05-27T09:43:19.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.953+0000 7effc2b83b80 1 Processor -- start 2024-05-27T09:43:19.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.954+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f42de40 0x562d4f5c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.954+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac410 0x562d4f5c5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.954+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 0x562d4f5bf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.954+0000 7effc2b83b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562d4f3d0fa0 con 0x562d4f5ac410 2024-05-27T09:43:19.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.954+0000 7effc2b83b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562d4f46b760 con 0x562d4f5ac820 2024-05-27T09:43:19.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.954+0000 7effc2b83b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562d4f5b98d0 con 0x562d4f42de40 2024-05-27T09:43:19.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.954+0000 7effc2b83b80 10 client.0 _ll_register_callbacks cb 0x562d4f474c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:19.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.954+0000 7effb2000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 0x562d4f5bf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.954+0000 7effb2000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 0x562d4f5bf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42984/0 (socket says 172.21.15.44:42984) 2024-05-27T09:43:19.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.954+0000 7effb2000700 1 -- 172.21.15.44:0/3312369890 learned_addr learned my addr 172.21.15.44:0/3312369890 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:19.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.955+0000 7effb2000700 1 -- 172.21.15.44:0/3312369890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f42de40 msgr2=0x562d4f5c5040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T09:43:19.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.955+0000 7effb2000700 1 --2- 172.21.15.44:0/3312369890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f42de40 0x562d4f5c5040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.955+0000 7effb2000700 1 -- 172.21.15.44:0/3312369890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac410 msgr2=0x562d4f5c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:19.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.955+0000 7effb2000700 1 --2- 172.21.15.44:0/3312369890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac410 0x562d4f5c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.955+0000 7effb2000700 1 -- 172.21.15.44:0/3312369890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d4f41a8d0 con 0x562d4f5ac820 2024-05-27T09:43:19.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.955+0000 7effb2000700 1 --2- 172.21.15.44:0/3312369890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 0x562d4f5bf440 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7effa4007b10 tx=0x7effa4004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=705cce3bf85e0cd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:19.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.956+0000 7eff93400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:43:19.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.956+0000 7eff93400700 1 -- 172.21.15.44:0/3312369890 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa4003e10 con 0x562d4f5ac820 2024-05-27T09:43:19.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.956+0000 7eff93400700 1 -- 172.21.15.44:0/3312369890 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7effa4019810 con 0x562d4f5ac820 2024-05-27T09:43:19.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.956+0000 7effc2b83b80 1 -- 172.21.15.44:0/3312369890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff98004830 con 0x562d4f5ac820 2024-05-27T09:43:19.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.956+0000 7eff93400700 1 -- 172.21.15.44:0/3312369890 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7effa40093b0 con 0x562d4f5ac820 2024-05-27T09:43:19.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.956+0000 7effc2b83b80 10 client.26783 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:19.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.956+0000 7effc2b83b80 1 -- 172.21.15.44:0/3312369890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562d4f49b060 con 0x562d4f5ac820 2024-05-27T09:43:19.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.957+0000 7eff93400700 1 -- 172.21.15.44:0/3312369890 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff98004830 con 0x562d4f5ac820 2024-05-27T09:43:19.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.957+0000 7eff93400700 1 -- 172.21.15.44:0/3312369890 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7effa40243f0 con 0x562d4f5ac820 2024-05-27T09:43:19.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.957+0000 7effc2b83b80 10 client.26783 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:19.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.957+0000 7effc2b83b80 10 client.26783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:19.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.957+0000 7effc2b83b80 20 client.26783 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:19.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.957+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3312369890 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d4f5cf220 0x562d4f5d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.957+0000 7effc2b83b80 4 client.26783 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:43:19.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.957+0000 7effc2b83b80 1 -- 172.21.15.44:0/3312369890 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d4f4dcd90 con 0x562d4f5cf220 2024-05-27T09:43:19.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.958+0000 7effb1600700 1 --2- 172.21.15.44:0/3312369890 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d4f5cf220 0x562d4f5d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.958+0000 7eff93400700 10 client.26783 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:19.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.958+0000 7effb1600700 1 --2- 172.21.15.44:0/3312369890 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d4f5cf220 0x562d4f5d1710 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9722c4287aaa4ba server_cookie=8b9461d146f49421 in_seq=0 out_seq=0 2024-05-27T09:43:19.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.959+0000 7eff93400700 1 -- 172.21.15.44:0/3312369890 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562d4f4dcd90 con 0x562d4f5cf220 2024-05-27T09:43:19.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.959+0000 7eff93400700 10 client.26783 handle_command_reply: tid=0 2024-05-27T09:43:19.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.959+0000 7effc2b83b80 1 client.26783 shutdown 2024-05-27T09:43:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.959+0000 7effc2b83b80 1 -- 172.21.15.44:0/3312369890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 msgr2=0x562d4f5bf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.959+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3312369890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 0x562d4f5bf440 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7effa4007b10 tx=0x7effa4004ad0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.960+0000 7effc2b83b80 1 -- 172.21.15.44:0/3312369890 shutdown_connections 2024-05-27T09:43:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.960+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3312369890 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d4f5cf220 0x562d4f5d1710 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.960+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3312369890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 0x562d4f5bf440 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.960+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3312369890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f42de40 0x562d4f5c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.960+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3312369890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac410 0x562d4f5c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.960+0000 7eff93400700 0 client.26783 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.960+0000 7effc2b83b80 1 -- 172.21.15.44:0/3312369890 >> 172.21.15.44:0/3312369890 conn(0x562d4f48dfc0 msgr2=0x562d4f5a6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.960+0000 7effc2b83b80 1 -- 172.21.15.44:0/3312369890 shutdown_connections 2024-05-27T09:43:19.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.960+0000 7effc2b83b80 1 -- 172.21.15.44:0/3312369890 wait complete. 2024-05-27T09:43:19.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.960+0000 7effc2b83b80 20 client.26783 trim_cache size 0 max 16384 2024-05-27T09:43:19.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.985+0000 7effc2b83b80 1 Processor -- start 2024-05-27T09:43:19.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.985+0000 7effc2b83b80 1 -- start start 2024-05-27T09:43:19.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.986+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f42de40 0x562d4f5bdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.986+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac410 0x562d4f5be270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.986+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac820 0x562d4f5d6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.986+0000 7effc2b83b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7effac1af2c0 con 0x562d4f5ac820 2024-05-27T09:43:19.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.986+0000 7effc2b83b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7effac1acb20 con 0x562d4f5ac410 2024-05-27T09:43:19.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.986+0000 7effc2b83b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7effac1aa990 con 0x562d4f42de40 2024-05-27T09:43:19.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.986+0000 7effb1600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f42de40 0x562d4f5bdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.986+0000 7effb1600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f42de40 0x562d4f5bdd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59970/0 (socket says 172.21.15.44:59970) 2024-05-27T09:43:19.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.986+0000 7effb1600700 1 -- 172.21.15.44:0/2018968522 learned_addr learned my addr 172.21.15.44:0/2018968522 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:19.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.986+0000 7effb2000700 1 --2- 172.21.15.44:0/2018968522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac820 0x562d4f5d6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.986+0000 7effb0c00700 1 --2- 172.21.15.44:0/2018968522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac410 0x562d4f5be270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.987+0000 7effb1600700 1 -- 172.21.15.44:0/2018968522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac410 msgr2=0x562d4f5be270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:19.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.987+0000 7effb1600700 1 --2- 172.21.15.44:0/2018968522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac410 0x562d4f5be270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.987+0000 7effb1600700 1 -- 172.21.15.44:0/2018968522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac820 msgr2=0x562d4f5d6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:19.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.987+0000 7effb1600700 1 --2- 172.21.15.44:0/2018968522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac820 0x562d4f5d6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.987+0000 7effb1600700 1 -- 172.21.15.44:0/2018968522 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d4f5a9350 con 0x562d4f42de40 2024-05-27T09:43:19.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.987+0000 7effb2000700 1 --2- 172.21.15.44:0/2018968522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac820 0x562d4f5d6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:43:19.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.987+0000 7effb1600700 1 --2- 172.21.15.44:0/2018968522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f42de40 0x562d4f5bdd30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7effa00a99f0 tx=0x7effa000cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb363d9edf3a00c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:19.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.987+0000 7eff9e000700 1 -- 172.21.15.44:0/2018968522 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa00900c0 con 0x562d4f42de40 2024-05-27T09:43:19.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.987+0000 7eff9e000700 1 -- 172.21.15.44:0/2018968522 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effa00a9800 con 0x562d4f42de40 2024-05-27T09:43:19.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.987+0000 7eff9e000700 1 -- 172.21.15.44:0/2018968522 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa00a58c0 con 0x562d4f42de40 2024-05-27T09:43:19.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.989+0000 7effc2b83b80 1 -- 172.21.15.44:0/2018968522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f42de40 msgr2=0x562d4f5bdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:19.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.989+0000 7effc2b83b80 1 --2- 172.21.15.44:0/2018968522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f42de40 0x562d4f5bdd30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7effa00a99f0 tx=0x7effa000cf20 comp rx=0 tx=0).stop 2024-05-27T09:43:19.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.989+0000 7effc2b83b80 1 -- 172.21.15.44:0/2018968522 shutdown_connections 2024-05-27T09:43:19.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.989+0000 7effc2b83b80 1 --2- 172.21.15.44:0/2018968522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f5ac820 0x562d4f5d6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.989+0000 7effc2b83b80 1 --2- 172.21.15.44:0/2018968522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f42de40 0x562d4f5bdd30 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.989+0000 7effc2b83b80 1 --2- 172.21.15.44:0/2018968522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac410 0x562d4f5be270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.989+0000 7effc2b83b80 1 -- 172.21.15.44:0/2018968522 >> 172.21.15.44:0/2018968522 conn(0x562d4f48dfc0 msgr2=0x562d4f5a77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:19.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.989+0000 7effc2b83b80 1 -- 172.21.15.44:0/2018968522 shutdown_connections 2024-05-27T09:43:19.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.989+0000 7effc2b83b80 1 -- 172.21.15.44:0/2018968522 wait complete. 2024-05-27T09:43:19.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.990+0000 7effc2b83b80 1 -- start start 2024-05-27T09:43:19.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.990+0000 7effc2b83b80 1 Processor -- start 2024-05-27T09:43:19.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.990+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f42de40 0x562d4f5c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.990+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac410 0x562d4f5c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.990+0000 7effc2b83b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 0x562d4f5c9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.990+0000 7effc2b83b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7effac1aafe0 con 0x562d4f42de40 2024-05-27T09:43:19.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.990+0000 7effc2b83b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7effac1ac450 con 0x562d4f5ac820 2024-05-27T09:43:19.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.990+0000 7effc2b83b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7effac1ab6b0 con 0x562d4f5ac410 2024-05-27T09:43:19.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.991+0000 7effc2b83b80 10 client.0 _ll_register_callbacks cb 0x562d4f474c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:19.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.991+0000 7effb0c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac410 0x562d4f5c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.991+0000 7effb0c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac410 0x562d4f5c9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59976/0 (socket says 172.21.15.44:59976) 2024-05-27T09:43:19.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.991+0000 7effb0c00700 1 -- 172.21.15.44:0/3913866099 learned_addr learned my addr 172.21.15.44:0/3913866099 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:19.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.991+0000 7effb1600700 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f42de40 0x562d4f5c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.991+0000 7effb2000700 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 0x562d4f5c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.991+0000 7effb0c00700 1 -- 172.21.15.44:0/3913866099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 msgr2=0x562d4f5c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:19.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.991+0000 7effb0c00700 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 0x562d4f5c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.991+0000 7effb0c00700 1 -- 172.21.15.44:0/3913866099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f42de40 msgr2=0x562d4f5c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:19.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.991+0000 7effb0c00700 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f42de40 0x562d4f5c8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:19.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.991+0000 7effb0c00700 1 -- 172.21.15.44:0/3913866099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d4f638df0 con 0x562d4f5ac410 2024-05-27T09:43:19.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.992+0000 7effb0c00700 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac410 0x562d4f5c9310 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7eff9800b2e0 tx=0x7eff98009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f2d5e949a84ac3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:19.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.992+0000 7eff93400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:43:19.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.992+0000 7eff93400700 1 -- 172.21.15.44:0/3913866099 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff98004830 con 0x562d4f5ac410 2024-05-27T09:43:19.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.992+0000 7eff93400700 1 -- 172.21.15.44:0/3913866099 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7eff9801e040 con 0x562d4f5ac410 2024-05-27T09:43:19.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.992+0000 7effc2b83b80 1 -- 172.21.15.44:0/3913866099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effa00a9800 con 0x562d4f5ac410 2024-05-27T09:43:19.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.992+0000 7eff93400700 1 -- 172.21.15.44:0/3913866099 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7eff980049d0 con 0x562d4f5ac410 2024-05-27T09:43:19.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.992+0000 7effc2b83b80 10 client.26776 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:19.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.992+0000 7effc2b83b80 1 -- 172.21.15.44:0/3913866099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562d4f639ff0 con 0x562d4f5ac410 2024-05-27T09:43:19.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.993+0000 7eff93400700 1 -- 172.21.15.44:0/3913866099 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effa00a9800 con 0x562d4f5ac410 2024-05-27T09:43:19.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.993+0000 7eff93400700 1 -- 172.21.15.44:0/3913866099 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7eff9802c8c0 con 0x562d4f5ac410 2024-05-27T09:43:19.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.993+0000 7effc2b83b80 10 client.26776 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.993+0000 7effc2b83b80 10 client.26776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.993+0000 7effc2b83b80 20 client.26776 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.993+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d4f63a370 0x562d4f63c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.993+0000 7effc2b83b80 4 client.26776 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.993+0000 7effc2b83b80 1 -- 172.21.15.44:0/3913866099 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562d4f4d6570 con 0x562d4f63a370 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.993+0000 7effb1600700 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d4f63a370 0x562d4f63c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.994+0000 7effb1600700 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d4f63a370 0x562d4f63c790 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9ac65c1e3eb8251 server_cookie=cae0c71a2adf3c1b in_seq=0 out_seq=0 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.994+0000 7eff93400700 10 client.26776 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.994+0000 7eff93400700 1 -- 172.21.15.44:0/3913866099 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562d4f4d6570 con 0x562d4f63a370 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.994+0000 7eff93400700 10 client.26776 handle_command_reply: tid=0 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.994+0000 7effc2b83b80 1 client.26776 shutdown 2024-05-27T09:43:19.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.994+0000 7effc2b83b80 1 -- 172.21.15.44:0/3913866099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac410 msgr2=0x562d4f5c9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:19.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.994+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac410 0x562d4f5c9310 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7eff9800b2e0 tx=0x7eff98009620 comp rx=0 tx=0).stop 2024-05-27T09:43:19.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.995+0000 7effc2b83b80 1 -- 172.21.15.44:0/3913866099 shutdown_connections 2024-05-27T09:43:20.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.995+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d4f63a370 0x562d4f63c790 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:20.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.995+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d4f5ac820 0x562d4f5c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:20.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.995+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d4f42de40 0x562d4f5c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:20.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.995+0000 7effc2b83b80 1 --2- 172.21.15.44:0/3913866099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d4f5ac410 0x562d4f5c9310 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:20.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.995+0000 7effc2b83b80 1 -- 172.21.15.44:0/3913866099 >> 172.21.15.44:0/3913866099 conn(0x562d4f48dfc0 msgr2=0x562d4f5bd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:20.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.995+0000 7eff93400700 0 client.26776 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:20.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.995+0000 7effc2b83b80 1 -- 172.21.15.44:0/3913866099 shutdown_connections 2024-05-27T09:43:20.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.995+0000 7effc2b83b80 1 -- 172.21.15.44:0/3913866099 wait complete. 2024-05-27T09:43:20.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.995+0000 7effc2b83b80 20 client.26776 trim_cache size 0 max 16384 2024-05-27T09:43:20.001 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:43:20.001 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:43:20.001 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:43:20.001 INFO:teuthology.orchestra.run.smithi044.stdout: "23fa7c2c-265d-4199-a6d4-f39b4f4a28a7": { 2024-05-27T09:43:20.001 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:43:20.001 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "23fa7c2c-265d-4199-a6d4-f39b4f4a28a7", 2024-05-27T09:43:20.001 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:43:20.001 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:43:20.002 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:43:20.002 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:43:20.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.996+0000 7effb2a00700 1 -- 172.21.15.44:0/3222388544 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff940b2af0 msgr2=0x7eff940b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:20.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.996+0000 7effb2a00700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff940b2af0 0x7eff940b5000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x562d4f5a8690 tx=0x7effa4004b40 comp rx=0 tx=0).stop 2024-05-27T09:43:20.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.996+0000 7effb2a00700 1 -- 172.21.15.44:0/3222388544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effac105280 msgr2=0x7effac1a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:20.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.996+0000 7effb2a00700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effac105280 0x7effac1a0840 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effa000e9a0 tx=0x7effa000ecb0 comp rx=0 tx=0).stop 2024-05-27T09:43:20.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.996+0000 7effb2a00700 1 -- 172.21.15.44:0/3222388544 shutdown_connections 2024-05-27T09:43:20.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.996+0000 7effb2a00700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7eff940b2af0 0x7eff940b5000 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:20.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.996+0000 7effb2a00700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7effac10e6f0 0x7effac19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:20.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.996+0000 7effb2a00700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effac105280 0x7effac1a0840 secure :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effa000e9a0 tx=0x7effa000ecb0 comp rx=0 tx=0).stop 2024-05-27T09:43:20.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.996+0000 7effb2a00700 1 --2- 172.21.15.44:0/3222388544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effac105bd0 0x7effac1a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:20.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.996+0000 7effb2a00700 1 -- 172.21.15.44:0/3222388544 >> 172.21.15.44:0/3222388544 conn(0x7effac078ab0 msgr2=0x7effac0ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:20.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.996+0000 7effb2a00700 1 -- 172.21.15.44:0/3222388544 shutdown_connections 2024-05-27T09:43:20.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:19.997+0000 7effb2a00700 1 -- 172.21.15.44:0/3222388544 wait complete. 2024-05-27T09:43:20.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:19 smithi044 ceph-mon[31618]: pgmap v250: 97 pgs: 97 active+clean; 111 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 698 op/s 2024-05-27T09:43:20.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:19 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:43:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:19 smithi050 ceph-mon[36371]: pgmap v250: 97 pgs: 97 active+clean; 111 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 698 op/s 2024-05-27T09:43:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:19 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:43:20.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:19 smithi189 ceph-mon[36671]: pgmap v250: 97 pgs: 97 active+clean; 111 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 698 op/s 2024-05-27T09:43:20.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:19 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:43:20.782 INFO:tasks.cephfs.filesystem:scrub status for tag:23fa7c2c-265d-4199-a6d4-f39b4f4a28a7 - {'path': '/', 'tag': '23fa7c2c-265d-4199-a6d4-f39b4f4a28a7', 'options': 'recursive,force'} 2024-05-27T09:43:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:21 smithi050 ceph-mon[36371]: pgmap v251: 97 pgs: 97 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 1.34k op/s 2024-05-27T09:43:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:21 smithi189 ceph-mon[36671]: pgmap v251: 97 pgs: 97 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 1.34k op/s 2024-05-27T09:43:22.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:21 smithi044 ceph-mon[31618]: pgmap v251: 97 pgs: 97 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 1.34k op/s 2024-05-27T09:43:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.517+0000 7f02f5e00640 1 -- 172.21.15.44:0/256577346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02f0075480 msgr2=0x7f02f0075890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.517+0000 7f02f5e00640 1 --2- 172.21.15.44:0/256577346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02f0075480 0x7f02f0075890 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009900 tx=0x7f02e402a480 comp rx=0 tx=0).stop 2024-05-27T09:43:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.518+0000 7f02f5e00640 1 -- 172.21.15.44:0/256577346 shutdown_connections 2024-05-27T09:43:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.518+0000 7f02f5e00640 1 --2- 172.21.15.44:0/256577346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f01102f0 0x7f02f0112710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.518+0000 7f02f5e00640 1 --2- 172.21.15.44:0/256577346 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02f010d990 0x7f02f010fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.518+0000 7f02f5e00640 1 --2- 172.21.15.44:0/256577346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02f0075480 0x7f02f0075890 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.518+0000 7f02f5e00640 1 -- 172.21.15.44:0/256577346 >> 172.21.15.44:0/256577346 conn(0x7f02f0079240 msgr2=0x7f02f0079680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.518+0000 7f02f5e00640 1 -- 172.21.15.44:0/256577346 shutdown_connections 2024-05-27T09:43:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.518+0000 7f02f5e00640 1 -- 172.21.15.44:0/256577346 wait complete. 2024-05-27T09:43:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.519+0000 7f02f5e00640 1 Processor -- start 2024-05-27T09:43:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.519+0000 7f02f5e00640 1 -- start start 2024-05-27T09:43:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02f0075480 0x7f02f01a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f5e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02f010d990 0x7f02f01a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f5e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f01102f0 0x7f02f019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f5e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f02f0115ed0 con 0x7f02f010d990 2024-05-27T09:43:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f5e00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f02f0115d50 con 0x7f02f01102f0 2024-05-27T09:43:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f5e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f02f0116050 con 0x7f02f0075480 2024-05-27T09:43:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02f0075480 0x7f02f01a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02f0075480 0x7f02f01a25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59992/0 (socket says 172.21.15.44:59992) 2024-05-27T09:43:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f4a00640 1 -- 172.21.15.44:0/889594906 learned_addr learned my addr 172.21.15.44:0/889594906 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02efe00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02f010d990 0x7f02f01a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f5400640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f01102f0 0x7f02f019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f4a00640 1 -- 172.21.15.44:0/889594906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f01102f0 msgr2=0x7f02f019c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f4a00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f01102f0 0x7f02f019c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f4a00640 1 -- 172.21.15.44:0/889594906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02f010d990 msgr2=0x7f02f01a2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f4a00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02f010d990 0x7f02f01a2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.520+0000 7f02f4a00640 1 -- 172.21.15.44:0/889594906 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f019ce80 con 0x7f02f0075480 2024-05-27T09:43:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.521+0000 7f02f5400640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f01102f0 0x7f02f019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.521+0000 7f02f4a00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02f0075480 0x7f02f01a25d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f02e4005ab0 tx=0x7f02e4003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.521+0000 7f02ed600640 1 -- 172.21.15.44:0/889594906 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e403f070 con 0x7f02f0075480 2024-05-27T09:43:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.521+0000 7f02ed600640 1 -- 172.21.15.44:0/889594906 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02e4031b50 con 0x7f02f0075480 2024-05-27T09:43:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.521+0000 7f02ed600640 1 -- 172.21.15.44:0/889594906 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e4035030 con 0x7f02f0075480 2024-05-27T09:43:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.521+0000 7f02f5e00640 1 -- 172.21.15.44:0/889594906 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f019d0b0 con 0x7f02f0075480 2024-05-27T09:43:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.521+0000 7f02f5e00640 1 -- 172.21.15.44:0/889594906 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f019d550 con 0x7f02f0075480 2024-05-27T09:43:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.526+0000 7f02fe37c740 1 Processor -- start 2024-05-27T09:43:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.526+0000 7f02ed600640 1 -- 172.21.15.44:0/889594906 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f02e4042070 con 0x7f02f0075480 2024-05-27T09:43:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.526+0000 7f02fe37c740 1 -- start start 2024-05-27T09:43:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.526+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x55655017aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.526+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdaf2f0 0x55655017afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.526+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 0x55655017b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.526+0000 7f02fe37c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5565500ebb90 con 0x55654fdaf2f0 2024-05-27T09:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.526+0000 7f02fe37c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5565500db730 con 0x55654fdd81b0 2024-05-27T09:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.526+0000 7f02fe37c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5565500df770 con 0x55654fdb74d0 2024-05-27T09:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.527+0000 7f02f5400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x55655017aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.527+0000 7f02f5400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x55655017aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:43028/0 (socket says 172.21.15.44:43028) 2024-05-27T09:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.527+0000 7f02f5400640 1 -- 172.21.15.44:0/2677855572 learned_addr learned my addr 172.21.15.44:0/2677855572 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.527+0000 7f02efe00640 1 --2- 172.21.15.44:0/2677855572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 0x55655017b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.527+0000 7f02f4a00640 1 --2- 172.21.15.44:0/2677855572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdaf2f0 0x55655017afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.527+0000 7f02f5400640 1 -- 172.21.15.44:0/2677855572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 msgr2=0x55655017b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.527+0000 7f02f5400640 1 --2- 172.21.15.44:0/2677855572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 0x55655017b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.527+0000 7f02f5400640 1 -- 172.21.15.44:0/2677855572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdaf2f0 msgr2=0x55655017afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.527+0000 7f02f5400640 1 --2- 172.21.15.44:0/2677855572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdaf2f0 0x55655017afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.527+0000 7f02f5400640 1 -- 172.21.15.44:0/2677855572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565500748b0 con 0x55654fdd81b0 2024-05-27T09:43:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.527+0000 7f02f5400640 1 --2- 172.21.15.44:0/2677855572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x55655017aa60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f02e8011790 tx=0x7f02e8011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b08578f0d37b0aec server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.528+0000 7f02ed600640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f02d80b2950 0x7f02d80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.528+0000 7f02e1c00640 1 -- 172.21.15.44:0/2677855572 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8003c30 con 0x55654fdd81b0 2024-05-27T09:43:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.528+0000 7f02e1c00640 1 -- 172.21.15.44:0/2677855572 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02e800fe50 con 0x55654fdd81b0 2024-05-27T09:43:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.528+0000 7f02ed600640 1 -- 172.21.15.44:0/889594906 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f02e4102910 con 0x7f02f0075480 2024-05-27T09:43:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.528+0000 7f02e1c00640 1 -- 172.21.15.44:0/2677855572 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8004840 con 0x55654fdd81b0 2024-05-27T09:43:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.529+0000 7f02f4a00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f02d80b2950 0x7f02d80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.530+0000 7f02fe37c740 1 -- 172.21.15.44:0/2677855572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 msgr2=0x55655017aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.530+0000 7f02f4a00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f02d80b2950 0x7f02d80b4e60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x556550178550 tx=0x7f02e402a480 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.530+0000 7f02fe37c740 1 --2- 172.21.15.44:0/2677855572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x55655017aa60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f02e8011790 tx=0x7f02e8011c50 comp rx=0 tx=0).stop 2024-05-27T09:43:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.530+0000 7f02fe37c740 1 -- 172.21.15.44:0/2677855572 shutdown_connections 2024-05-27T09:43:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.530+0000 7f02fe37c740 1 --2- 172.21.15.44:0/2677855572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 0x55655017b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.530+0000 7f02fe37c740 1 --2- 172.21.15.44:0/2677855572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdaf2f0 0x55655017afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.530+0000 7f02fe37c740 1 --2- 172.21.15.44:0/2677855572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x55655017aa60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.530+0000 7f02fe37c740 1 -- 172.21.15.44:0/2677855572 >> 172.21.15.44:0/2677855572 conn(0x55654fdcc300 msgr2=0x556550177210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:22.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.530+0000 7f02fe37c740 1 -- 172.21.15.44:0/2677855572 shutdown_connections 2024-05-27T09:43:22.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.530+0000 7f02fe37c740 1 -- 172.21.15.44:0/2677855572 wait complete. 2024-05-27T09:43:22.532 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.531+0000 7f02fe37c740 1 -- start start 2024-05-27T09:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.531+0000 7f02fe37c740 1 Processor -- start 2024-05-27T09:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 0x556550184bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 0x556550185110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x556550190ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02fe37c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55654fdc1790 con 0x55654fdb74d0 2024-05-27T09:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02fe37c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556550083bc0 con 0x55654fdd81b0 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02fe37c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556550187b90 con 0x55654fdaf2f0 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02fe37c740 10 client.0 _ll_register_callbacks cb 0x556550061400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02f5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 0x556550184bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02f5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 0x556550184bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60002/0 (socket says 172.21.15.44:60002) 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02f5400640 1 -- 172.21.15.44:0/1491106327 learned_addr learned my addr 172.21.15.44:0/1491106327 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02efe00640 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 0x556550185110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.532+0000 7f02f5400640 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x556550190ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.533+0000 7f02f5400640 1 -- 172.21.15.44:0/1491106327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 msgr2=0x556550190ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.533+0000 7f02f5400640 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x556550190ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.533+0000 7f02f5400640 1 -- 172.21.15.44:0/1491106327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 msgr2=0x556550185110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.533+0000 7f02f5400640 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 0x556550185110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.533+0000 7f02f5400640 1 -- 172.21.15.44:0/1491106327 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55654fdf0210 con 0x55654fdaf2f0 2024-05-27T09:43:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.533+0000 7f02efe00640 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 0x556550185110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.533+0000 7f02f5400640 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 0x556550184bd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f02e8004520 tx=0x7f02e8004420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a9baef33b883e5e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.533+0000 7f02d6000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.534+0000 7f02d6000640 1 -- 172.21.15.44:0/1491106327 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8003a30 con 0x55654fdaf2f0 2024-05-27T09:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.534+0000 7f02d6000640 1 -- 172.21.15.44:0/1491106327 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f02e8003bd0 con 0x55654fdaf2f0 2024-05-27T09:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.534+0000 7f02fe37c740 1 -- 172.21.15.44:0/1491106327 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02e800fe50 con 0x55654fdaf2f0 2024-05-27T09:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.534+0000 7f02d6000640 1 -- 172.21.15.44:0/1491106327 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f02e8003dd0 con 0x55654fdaf2f0 2024-05-27T09:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.534+0000 7f02fe37c740 10 client.26788 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.534+0000 7f02fe37c740 1 -- 172.21.15.44:0/1491106327 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55654febc620 con 0x55654fdaf2f0 2024-05-27T09:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.535+0000 7f02d6000640 1 -- 172.21.15.44:0/1491106327 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02e800fe50 con 0x55654fdaf2f0 2024-05-27T09:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.535+0000 7f02d6000640 1 -- 172.21.15.44:0/1491106327 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f02e8018070 con 0x55654fdaf2f0 2024-05-27T09:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.535+0000 7f02fe37c740 10 client.26788 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.535+0000 7f02fe37c740 10 client.26788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.535+0000 7f02fe37c740 20 client.26788 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.535+0000 7f02fe37c740 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55654fd96d80 0x556550196d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.535+0000 7f02fe37c740 4 client.26788 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:43:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.535+0000 7f02fe37c740 1 -- 172.21.15.44:0/1491106327 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565500edae0 con 0x55654fd96d80 2024-05-27T09:43:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.536+0000 7f02efe00640 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55654fd96d80 0x556550196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.536+0000 7f02efe00640 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55654fd96d80 0x556550196d90 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5a7f575352b801d server_cookie=cec651ebc1429529 in_seq=0 out_seq=0 2024-05-27T09:43:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.536+0000 7f02d6000640 10 client.26788 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02d6000640 1 -- 172.21.15.44:0/1491106327 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5565500edae0 con 0x55654fd96d80 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02d6000640 10 client.26788 handle_command_reply: tid=0 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02fe37c740 1 client.26788 shutdown 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02fe37c740 1 -- 172.21.15.44:0/1491106327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 msgr2=0x556550184bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02fe37c740 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 0x556550184bd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f02e8004520 tx=0x7f02e8004420 comp rx=0 tx=0).stop 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02fe37c740 1 -- 172.21.15.44:0/1491106327 shutdown_connections 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02fe37c740 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55654fd96d80 0x556550196d90 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02fe37c740 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x556550190ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02fe37c740 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 0x556550185110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02fe37c740 1 --2- 172.21.15.44:0/1491106327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 0x556550184bd0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02fe37c740 1 -- 172.21.15.44:0/1491106327 >> 172.21.15.44:0/1491106327 conn(0x55654fdcc300 msgr2=0x55655017a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.537+0000 7f02d6000640 0 client.26788 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.538+0000 7f02fe37c740 1 -- 172.21.15.44:0/1491106327 shutdown_connections 2024-05-27T09:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.538+0000 7f02fe37c740 1 -- 172.21.15.44:0/1491106327 wait complete. 2024-05-27T09:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.538+0000 7f02fe37c740 20 client.26788 trim_cache size 0 max 16384 2024-05-27T09:43:22.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.555+0000 7f02fe37c740 1 Processor -- start 2024-05-27T09:43:22.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.555+0000 7f02fe37c740 1 -- start start 2024-05-27T09:43:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.555+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdaf2f0 0x556550187300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.555+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 0x55655017b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.555+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdd81b0 0x556550184b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.555+0000 7f02fe37c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01b4500 con 0x55654fdd81b0 2024-05-27T09:43:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.555+0000 7f02fe37c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01b1da0 con 0x55654fdaf2f0 2024-05-27T09:43:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.555+0000 7f02fe37c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01afc10 con 0x55654fdb74d0 2024-05-27T09:43:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.555+0000 7f02efe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 0x55655017b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.555+0000 7f02efe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 0x55655017b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60016/0 (socket says 172.21.15.44:60016) 2024-05-27T09:43:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.555+0000 7f02efe00640 1 -- 172.21.15.44:0/2327516551 learned_addr learned my addr 172.21.15.44:0/2327516551 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.556+0000 7f02f5400640 1 --2- 172.21.15.44:0/2327516551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdaf2f0 0x556550187300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.556+0000 7f02f5400640 1 --2- 172.21.15.44:0/2327516551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdd81b0 0x556550184b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.556+0000 7f02efe00640 1 -- 172.21.15.44:0/2327516551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdaf2f0 msgr2=0x556550187300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.556+0000 7f02efe00640 1 --2- 172.21.15.44:0/2327516551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdaf2f0 0x556550187300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.556+0000 7f02efe00640 1 -- 172.21.15.44:0/2327516551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdd81b0 msgr2=0x556550184b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.556+0000 7f02efe00640 1 --2- 172.21.15.44:0/2327516551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdd81b0 0x556550184b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.556+0000 7f02efe00640 1 -- 172.21.15.44:0/2327516551 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556550175610 con 0x55654fdb74d0 2024-05-27T09:43:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.556+0000 7f02efe00640 1 --2- 172.21.15.44:0/2327516551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 0x55655017b220 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f02dc005e00 tx=0x7f02dc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e94776290053e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.557+0000 7f02e1200640 1 -- 172.21.15.44:0/2327516551 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc004030 con 0x55654fdb74d0 2024-05-27T09:43:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.557+0000 7f02e1200640 1 -- 172.21.15.44:0/2327516551 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02dc0035f0 con 0x55654fdb74d0 2024-05-27T09:43:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.557+0000 7f02e1200640 1 -- 172.21.15.44:0/2327516551 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc00f7c0 con 0x55654fdb74d0 2024-05-27T09:43:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.558+0000 7f02fe37c740 1 -- 172.21.15.44:0/2327516551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 msgr2=0x55655017b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.558+0000 7f02fe37c740 1 --2- 172.21.15.44:0/2327516551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 0x55655017b220 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f02dc005e00 tx=0x7f02dc005f90 comp rx=0 tx=0).stop 2024-05-27T09:43:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.558+0000 7f02fe37c740 1 -- 172.21.15.44:0/2327516551 shutdown_connections 2024-05-27T09:43:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.558+0000 7f02fe37c740 1 --2- 172.21.15.44:0/2327516551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdd81b0 0x556550184b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.558+0000 7f02fe37c740 1 --2- 172.21.15.44:0/2327516551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdb74d0 0x55655017b220 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.558+0000 7f02fe37c740 1 --2- 172.21.15.44:0/2327516551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdaf2f0 0x556550187300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.558+0000 7f02fe37c740 1 -- 172.21.15.44:0/2327516551 >> 172.21.15.44:0/2327516551 conn(0x55654fdcc300 msgr2=0x556550184460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.558+0000 7f02fe37c740 1 -- 172.21.15.44:0/2327516551 shutdown_connections 2024-05-27T09:43:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.558+0000 7f02fe37c740 1 -- 172.21.15.44:0/2327516551 wait complete. 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.559+0000 7f02fe37c740 1 -- start start 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.559+0000 7f02fe37c740 1 Processor -- start 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 0x556550187de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 0x556550188320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02fe37c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x556550188860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02fe37c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01b0260 con 0x55654fdb74d0 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02fe37c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01b16d0 con 0x55654fdd81b0 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02fe37c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01b0930 con 0x55654fdaf2f0 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02fe37c740 10 client.0 _ll_register_callbacks cb 0x7f02dc009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02efe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 0x556550188320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02f5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 0x556550187de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02f5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 0x556550187de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60018/0 (socket says 172.21.15.44:60018) 2024-05-27T09:43:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02f5400640 1 -- 172.21.15.44:0/4078061912 learned_addr learned my addr 172.21.15.44:0/4078061912 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02efe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 0x556550188320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49906/0 (socket says 172.21.15.44:49906) 2024-05-27T09:43:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.560+0000 7f02f5400640 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x556550188860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.561+0000 7f02efe00640 1 -- 172.21.15.44:0/4078061912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 msgr2=0x556550187de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.561+0000 7f02efe00640 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 0x556550187de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.561+0000 7f02efe00640 1 -- 172.21.15.44:0/4078061912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 msgr2=0x556550188860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.561+0000 7f02efe00640 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x556550188860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.561+0000 7f02efe00640 1 -- 172.21.15.44:0/4078061912 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565501fd310 con 0x55654fdb74d0 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.561+0000 7f02efe00640 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 0x556550188320 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f02dc000c00 tx=0x7f02dc012310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99d31a397dad5f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.561+0000 7f02d6000640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.562+0000 7f02d6000640 1 -- 172.21.15.44:0/4078061912 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc0038a0 con 0x55654fdb74d0 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.562+0000 7f02d6000640 1 -- 172.21.15.44:0/4078061912 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f02dc003a40 con 0x55654fdb74d0 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.562+0000 7f02fe37c740 1 -- 172.21.15.44:0/4078061912 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02dc0035f0 con 0x55654fdb74d0 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.562+0000 7f02d6000640 1 -- 172.21.15.44:0/4078061912 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f02dc003c80 con 0x55654fdb74d0 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.562+0000 7f02fe37c740 10 client.16611 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.562+0000 7f02fe37c740 1 -- 172.21.15.44:0/4078061912 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5565501fe580 con 0x55654fdb74d0 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.562+0000 7f02d6000640 1 -- 172.21.15.44:0/4078061912 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02dc0035f0 con 0x55654fdb74d0 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.562+0000 7f02d6000640 1 -- 172.21.15.44:0/4078061912 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f02dc00fb40 con 0x55654fdb74d0 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.563+0000 7f02fe37c740 10 client.16611 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.563+0000 7f02fe37c740 10 client.16611 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.563+0000 7f02fe37c740 20 client.16611 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.563+0000 7f02fe37c740 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55654fd6c1c0 0x556550200b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.563+0000 7f02fe37c740 4 client.16611 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.563+0000 7f02fe37c740 1 -- 172.21.15.44:0/4078061912 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5565500e72c0 con 0x55654fd6c1c0 2024-05-27T09:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.563+0000 7f02f5400640 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55654fd6c1c0 0x556550200b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.563+0000 7f02d6000640 10 client.16611 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:22.564+0000 7f02f5400640 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55654fd6c1c0 0x556550200b60 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=121f338b34655b6b server_cookie=47ab3861555bcfb in_seq=0 out_seq=0 2024-05-27T09:43:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.743+0000 7f02d6000640 1 -- 172.21.15.44:0/4078061912 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5565500e72c0 con 0x55654fd6c1c0 2024-05-27T09:43:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.743+0000 7f02d6000640 10 client.16611 handle_command_reply: tid=0 2024-05-27T09:43:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.743+0000 7f02fe37c740 1 client.16611 shutdown 2024-05-27T09:43:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.744+0000 7f02fe37c740 1 -- 172.21.15.44:0/4078061912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 msgr2=0x556550188320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.744+0000 7f02fe37c740 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 0x556550188320 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f02dc000c00 tx=0x7f02dc012310 comp rx=0 tx=0).stop 2024-05-27T09:43:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.744+0000 7f02fe37c740 1 -- 172.21.15.44:0/4078061912 shutdown_connections 2024-05-27T09:43:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.744+0000 7f02fe37c740 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55654fd6c1c0 0x556550200b60 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.744+0000 7f02fe37c740 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55654fdd81b0 0x556550188860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.744+0000 7f02fe37c740 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55654fdb74d0 0x556550188320 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.744+0000 7f02fe37c740 1 --2- 172.21.15.44:0/4078061912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654fdaf2f0 0x556550187de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.744+0000 7f02fe37c740 1 -- 172.21.15.44:0/4078061912 >> 172.21.15.44:0/4078061912 conn(0x55654fdcc300 msgr2=0x556550184e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.744+0000 7f02d6000640 0 client.16611 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:23.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.745+0000 7f02fe37c740 1 -- 172.21.15.44:0/4078061912 shutdown_connections 2024-05-27T09:43:23.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.745+0000 7f02fe37c740 1 -- 172.21.15.44:0/4078061912 wait complete. 2024-05-27T09:43:23.746 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.745+0000 7f02fe37c740 20 client.16611 trim_cache size 0 max 16384 2024-05-27T09:43:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:43:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:43:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:43:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.746+0000 7f02f5e00640 1 -- 172.21.15.44:0/889594906 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f02d80b2950 msgr2=0x7f02d80b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.746+0000 7f02f5e00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f02d80b2950 0x7f02d80b4e60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x556550178550 tx=0x7f02e402a480 comp rx=0 tx=0).stop 2024-05-27T09:43:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.746+0000 7f02f5e00640 1 -- 172.21.15.44:0/889594906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02f0075480 msgr2=0x7f02f01a25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.746+0000 7f02f5e00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02f0075480 0x7f02f01a25d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f02e4005ab0 tx=0x7f02e4003b90 comp rx=0 tx=0).stop 2024-05-27T09:43:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.746+0000 7f02f5e00640 1 -- 172.21.15.44:0/889594906 shutdown_connections 2024-05-27T09:43:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.746+0000 7f02f5e00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f01102f0 0x7f02f019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.746+0000 7f02f5e00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f02d80b2950 0x7f02d80b4e60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.746+0000 7f02f5e00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02f010d990 0x7f02f01a2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.746+0000 7f02f5e00640 1 --2- 172.21.15.44:0/889594906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02f0075480 0x7f02f01a25d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.746+0000 7f02f5e00640 1 -- 172.21.15.44:0/889594906 >> 172.21.15.44:0/889594906 conn(0x7f02f0079240 msgr2=0x7f02f0102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.747+0000 7f02f5e00640 1 -- 172.21.15.44:0/889594906 shutdown_connections 2024-05-27T09:43:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:23.747+0000 7f02f5e00640 1 -- 172.21.15.44:0/889594906 wait complete. 2024-05-27T09:43:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:23 smithi050 ceph-mon[36371]: pgmap v252: 97 pgs: 97 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 752 op/s 2024-05-27T09:43:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:23 smithi189 ceph-mon[36671]: pgmap v252: 97 pgs: 97 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 752 op/s 2024-05-27T09:43:24.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:23 smithi044 ceph-mon[31618]: pgmap v252: 97 pgs: 97 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 752 op/s 2024-05-27T09:43:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:25 smithi050 ceph-mon[36371]: pgmap v253: 97 pgs: 97 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 752 op/s 2024-05-27T09:43:26.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:25 smithi189 ceph-mon[36671]: pgmap v253: 97 pgs: 97 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 752 op/s 2024-05-27T09:43:26.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:25 smithi044 ceph-mon[31618]: pgmap v253: 97 pgs: 97 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 752 op/s 2024-05-27T09:43:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:27 smithi050 ceph-mon[36371]: pgmap v254: 97 pgs: 97 active+clean; 140 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.2 MiB/s wr, 1.45k op/s 2024-05-27T09:43:28.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:27 smithi189 ceph-mon[36671]: pgmap v254: 97 pgs: 97 active+clean; 140 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.2 MiB/s wr, 1.45k op/s 2024-05-27T09:43:28.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:27 smithi044 ceph-mon[31618]: pgmap v254: 97 pgs: 97 active+clean; 140 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.2 MiB/s wr, 1.45k op/s 2024-05-27T09:43:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:30 smithi050 ceph-mon[36371]: pgmap v255: 97 pgs: 97 active+clean; 140 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 1.41k op/s 2024-05-27T09:43:30.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:30 smithi189 ceph-mon[36671]: pgmap v255: 97 pgs: 97 active+clean; 140 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 1.41k op/s 2024-05-27T09:43:30.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:30 smithi044 ceph-mon[31618]: pgmap v255: 97 pgs: 97 active+clean; 140 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 1.41k op/s 2024-05-27T09:43:32.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:32 smithi050 ceph-mon[36371]: pgmap v256: 97 pgs: 97 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.0 MiB/s wr, 1.54k op/s 2024-05-27T09:43:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:32 smithi189 ceph-mon[36671]: pgmap v256: 97 pgs: 97 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.0 MiB/s wr, 1.54k op/s 2024-05-27T09:43:32.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:32 smithi044 ceph-mon[31618]: pgmap v256: 97 pgs: 97 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.0 MiB/s wr, 1.54k op/s 2024-05-27T09:43:34.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:34 smithi050 ceph-mon[36371]: pgmap v257: 97 pgs: 97 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 833 op/s 2024-05-27T09:43:34.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:34 smithi189 ceph-mon[36671]: pgmap v257: 97 pgs: 97 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 833 op/s 2024-05-27T09:43:34.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:34 smithi044 ceph-mon[31618]: pgmap v257: 97 pgs: 97 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 833 op/s 2024-05-27T09:43:36.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:36 smithi050 ceph-mon[36371]: pgmap v258: 97 pgs: 97 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 833 op/s 2024-05-27T09:43:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:36 smithi189 ceph-mon[36671]: pgmap v258: 97 pgs: 97 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 833 op/s 2024-05-27T09:43:36.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:36 smithi044 ceph-mon[31618]: pgmap v258: 97 pgs: 97 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 833 op/s 2024-05-27T09:43:36.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.985+0000 7f266ee00640 1 -- 172.21.15.44:0/424731773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26681104a0 msgr2=0x7f2668112900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:36.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.985+0000 7f266ee00640 1 --2- 172.21.15.44:0/424731773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26681104a0 0x7f2668112900 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2660009ca0 tx=0x7f266002a950 comp rx=0 tx=0).stop 2024-05-27T09:43:36.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.985+0000 7f266ee00640 1 -- 172.21.15.44:0/424731773 shutdown_connections 2024-05-27T09:43:36.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.985+0000 7f266ee00640 1 --2- 172.21.15.44:0/424731773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26681104a0 0x7f2668112900 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:36.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.985+0000 7f266ee00640 1 --2- 172.21.15.44:0/424731773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f266810db40 0x7f266810ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:36.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.985+0000 7f266ee00640 1 --2- 172.21.15.44:0/424731773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2668075660 0x7f2668075a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:36.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.985+0000 7f266ee00640 1 -- 172.21.15.44:0/424731773 >> 172.21.15.44:0/424731773 conn(0x7f26680ff480 msgr2=0x7f26681018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.985+0000 7f266ee00640 1 -- 172.21.15.44:0/424731773 shutdown_connections 2024-05-27T09:43:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.985+0000 7f266ee00640 1 -- 172.21.15.44:0/424731773 wait complete. 2024-05-27T09:43:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.986+0000 7f266ee00640 1 Processor -- start 2024-05-27T09:43:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.987+0000 7f266ee00640 1 -- start start 2024-05-27T09:43:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.987+0000 7f266ee00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2668075660 0x7f26681a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.987+0000 7f266ee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f266810db40 0x7f26681a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.987+0000 7f266ee00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26681104a0 0x7f266819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.987+0000 7f266ee00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26681160b0 con 0x7f2668075660 2024-05-27T09:43:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.987+0000 7f266ee00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2668115f30 con 0x7f26681104a0 2024-05-27T09:43:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.987+0000 7f266ee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2668116230 con 0x7f266810db40 2024-05-27T09:43:36.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.987+0000 7f2667400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f266810db40 0x7f26681a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:36.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.987+0000 7f2667e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2668075660 0x7f26681a26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.987+0000 7f2667400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f266810db40 0x7f26681a2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:46992/0 (socket says 172.21.15.44:46992) 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.987+0000 7f2667400640 1 -- 172.21.15.44:0/3592572795 learned_addr learned my addr 172.21.15.44:0/3592572795 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.988+0000 7f266ce00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26681104a0 0x7f266819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.988+0000 7f2667e00640 1 -- 172.21.15.44:0/3592572795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f266810db40 msgr2=0x7f26681a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.988+0000 7f2667e00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f266810db40 0x7f26681a2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.988+0000 7f2667e00640 1 -- 172.21.15.44:0/3592572795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26681104a0 msgr2=0x7f266819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.988+0000 7f2667e00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26681104a0 0x7f266819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.988+0000 7f2667e00640 1 -- 172.21.15.44:0/3592572795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266819cf70 con 0x7f2668075660 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.988+0000 7f266ce00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26681104a0 0x7f266819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.989+0000 7f2667e00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2668075660 0x7f26681a26c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f265800e9f0 tx=0x7f265800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.989+0000 7f2664c00640 1 -- 172.21.15.44:0/3592572795 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265800ccc0 con 0x7f2668075660 2024-05-27T09:43:36.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.989+0000 7f2664c00640 1 -- 172.21.15.44:0/3592572795 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26580108e0 con 0x7f2668075660 2024-05-27T09:43:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.989+0000 7f2664c00640 1 -- 172.21.15.44:0/3592572795 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658018880 con 0x7f2668075660 2024-05-27T09:43:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.989+0000 7f266ee00640 1 -- 172.21.15.44:0/3592572795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f266819d260 con 0x7f2668075660 2024-05-27T09:43:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.989+0000 7f266ee00640 1 -- 172.21.15.44:0/3592572795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26681a9540 con 0x7f2668075660 2024-05-27T09:43:36.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.994+0000 7f2675f7b740 1 Processor -- start 2024-05-27T09:43:36.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.994+0000 7f2675f7b740 1 -- start start 2024-05-27T09:43:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.994+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a183860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.994+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69db82f0 0x55ab6a183da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.994+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a1842e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.994+0000 7f2675f7b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ab6a0f4b90 con 0x55ab69db82f0 2024-05-27T09:43:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.994+0000 7f2675f7b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ab6a0e4730 con 0x55ab69de11b0 2024-05-27T09:43:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.994+0000 7f2675f7b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ab6a0e8770 con 0x55ab69dc04d0 2024-05-27T09:43:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.995+0000 7f266ce00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a183860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.995+0000 7f2667400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a1842e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.995+0000 7f266ce00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a183860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54690/0 (socket says 172.21.15.44:54690) 2024-05-27T09:43:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.995+0000 7f266ce00640 1 -- 172.21.15.44:0/1181906913 learned_addr learned my addr 172.21.15.44:0/1181906913 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.995+0000 7f266ce00640 1 -- 172.21.15.44:0/1181906913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 msgr2=0x55ab6a1842e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.995+0000 7f266ce00640 1 --2- 172.21.15.44:0/1181906913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a1842e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.995+0000 7f266ce00640 1 -- 172.21.15.44:0/1181906913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69db82f0 msgr2=0x55ab6a183da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:43:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.995+0000 7f266ce00640 1 --2- 172.21.15.44:0/1181906913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69db82f0 0x55ab6a183da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.995+0000 7f266ce00640 1 -- 172.21.15.44:0/1181906913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab6a07d8b0 con 0x55ab69de11b0 2024-05-27T09:43:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.996+0000 7f266ce00640 1 --2- 172.21.15.44:0/1181906913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a183860 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2660009ca0 tx=0x7f26600093f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caab4a633c9789fe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.996+0000 7f2664c00640 1 -- 172.21.15.44:0/3592572795 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2658010430 con 0x7f2668075660 2024-05-27T09:43:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.996+0000 7f265d600640 1 -- 172.21.15.44:0/1181906913 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660048070 con 0x55ab69de11b0 2024-05-27T09:43:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.996+0000 7f265d600640 1 -- 172.21.15.44:0/1181906913 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f266002f870 con 0x55ab69de11b0 2024-05-27T09:43:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.996+0000 7f265d600640 1 -- 172.21.15.44:0/1181906913 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660034040 con 0x55ab69de11b0 2024-05-27T09:43:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.997+0000 7f2675f7b740 1 -- 172.21.15.44:0/1181906913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 msgr2=0x55ab6a183860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.997+0000 7f2675f7b740 1 --2- 172.21.15.44:0/1181906913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a183860 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2660009ca0 tx=0x7f26600093f0 comp rx=0 tx=0).stop 2024-05-27T09:43:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.997+0000 7f2675f7b740 1 -- 172.21.15.44:0/1181906913 shutdown_connections 2024-05-27T09:43:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.997+0000 7f2675f7b740 1 --2- 172.21.15.44:0/1181906913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a1842e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.997+0000 7f2675f7b740 1 --2- 172.21.15.44:0/1181906913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69db82f0 0x55ab6a183da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.997+0000 7f2675f7b740 1 --2- 172.21.15.44:0/1181906913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a183860 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.997+0000 7f2675f7b740 1 -- 172.21.15.44:0/1181906913 >> 172.21.15.44:0/1181906913 conn(0x55ab69dd5300 msgr2=0x55ab6a180010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.997+0000 7f2675f7b740 1 -- 172.21.15.44:0/1181906913 shutdown_connections 2024-05-27T09:43:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.997+0000 7f2675f7b740 1 -- 172.21.15.44:0/1181906913 wait complete. 2024-05-27T09:43:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.997+0000 7f2664c00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f264c0b2860 0x7f264c0b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.998+0000 7f2664c00640 1 -- 172.21.15.44:0/3592572795 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f26580e2960 con 0x7f2668075660 2024-05-27T09:43:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.998+0000 7f2667400640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f264c0b2860 0x7f264c0b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.998+0000 7f2675f7b740 1 -- start start 2024-05-27T09:43:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.998+0000 7f2675f7b740 1 Processor -- start 2024-05-27T09:43:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.999+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69db82f0 0x55ab6a18da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.999+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a18df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.999+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a199c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.999+0000 7f2675f7b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ab69dca790 con 0x55ab69db82f0 2024-05-27T09:43:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.999+0000 7f2675f7b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ab6a08cbc0 con 0x55ab69de11b0 2024-05-27T09:43:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.999+0000 7f2675f7b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ab6a190930 con 0x55ab69dc04d0 2024-05-27T09:43:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:36.999+0000 7f2675f7b740 10 client.0 _ll_register_callbacks cb 0x55ab6a06a400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.000+0000 7f2667400640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f264c0b2860 0x7f264c0b4d70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2654009640 tx=0x7f2654005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.000+0000 7f266ce00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.000+0000 7f2667e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69db82f0 0x55ab6a18da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.000+0000 7f266ce00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54706/0 (socket says 172.21.15.44:54706) 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.000+0000 7f266ce00640 1 -- 172.21.15.44:0/479807112 learned_addr learned my addr 172.21.15.44:0/479807112 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.000+0000 7f2667400640 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a18df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.000+0000 7f2667e00640 1 -- 172.21.15.44:0/479807112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 msgr2=0x55ab6a18df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.000+0000 7f2667e00640 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a18df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.000+0000 7f2667e00640 1 -- 172.21.15.44:0/479807112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 msgr2=0x55ab6a199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.000+0000 7f2667e00640 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.000+0000 7f2667e00640 1 -- 172.21.15.44:0/479807112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab69df9210 con 0x55ab69db82f0 2024-05-27T09:43:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.001+0000 7f2667400640 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a18df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:43:37.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.001+0000 7f2667e00640 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69db82f0 0x55ab6a18da40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f26580e00b0 tx=0x7f2658010850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443a43f4899f0ec2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:37.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.001+0000 7f2652000640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:43:37.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.002+0000 7f2652000640 1 -- 172.21.15.44:0/479807112 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26580adff0 con 0x55ab69db82f0 2024-05-27T09:43:37.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.002+0000 7f2652000640 1 -- 172.21.15.44:0/479807112 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f26580a8f50 con 0x55ab69db82f0 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.002+0000 7f2675f7b740 1 -- 172.21.15.44:0/479807112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f266002f870 con 0x55ab69db82f0 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.002+0000 7f2652000640 1 -- 172.21.15.44:0/479807112 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f26580b2aa0 con 0x55ab69db82f0 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.002+0000 7f2675f7b740 10 client.16620 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.002+0000 7f2675f7b740 1 -- 172.21.15.44:0/479807112 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ab69ec5620 con 0x55ab69db82f0 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.002+0000 7f2652000640 1 -- 172.21.15.44:0/479807112 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f266002f870 con 0x55ab69db82f0 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.002+0000 7f2652000640 1 -- 172.21.15.44:0/479807112 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f26580c4050 con 0x55ab69db82f0 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.003+0000 7f2675f7b740 10 client.16620 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.003+0000 7f2675f7b740 10 client.16620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.003+0000 7f2675f7b740 20 client.16620 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.003+0000 7f2675f7b740 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab69d9fd80 0x55ab6a19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.003+0000 7f2675f7b740 4 client.16620 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.003+0000 7f2675f7b740 1 -- 172.21.15.44:0/479807112 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab6a0f6ae0 con 0x55ab69d9fd80 2024-05-27T09:43:37.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.003+0000 7f2667400640 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab69d9fd80 0x55ab6a19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.004+0000 7f2667400640 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab69d9fd80 0x55ab6a19fc60 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e0111fab88303fc server_cookie=95c5bc8c7aa6cb49 in_seq=0 out_seq=0 2024-05-27T09:43:37.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.004+0000 7f2652000640 10 client.16620 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:37.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2652000640 1 -- 172.21.15.44:0/479807112 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ab6a0f6ae0 con 0x55ab69d9fd80 2024-05-27T09:43:37.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2652000640 10 client.16620 handle_command_reply: tid=0 2024-05-27T09:43:37.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2675f7b740 1 client.16620 shutdown 2024-05-27T09:43:37.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2675f7b740 1 -- 172.21.15.44:0/479807112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69db82f0 msgr2=0x55ab6a18da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:37.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2675f7b740 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69db82f0 0x55ab6a18da40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f26580e00b0 tx=0x7f2658010850 comp rx=0 tx=0).stop 2024-05-27T09:43:37.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2675f7b740 1 -- 172.21.15.44:0/479807112 shutdown_connections 2024-05-27T09:43:37.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2675f7b740 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab69d9fd80 0x55ab6a19fc60 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2675f7b740 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2675f7b740 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a18df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2675f7b740 1 --2- 172.21.15.44:0/479807112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69db82f0 0x55ab6a18da40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2652000640 0 client.16620 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:37.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.005+0000 7f2675f7b740 1 -- 172.21.15.44:0/479807112 >> 172.21.15.44:0/479807112 conn(0x55ab69dd5300 msgr2=0x55ab6a183630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:37.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.006+0000 7f2675f7b740 1 -- 172.21.15.44:0/479807112 shutdown_connections 2024-05-27T09:43:37.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.006+0000 7f2675f7b740 1 -- 172.21.15.44:0/479807112 wait complete. 2024-05-27T09:43:37.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.006+0000 7f2675f7b740 20 client.16620 trim_cache size 0 max 16384 2024-05-27T09:43:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.024+0000 7f2675f7b740 1 Processor -- start 2024-05-27T09:43:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.024+0000 7f2675f7b740 1 -- start start 2024-05-27T09:43:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69db82f0 0x55ab6a189280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a18d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69de11b0 0x55ab6a1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2675f7b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26681b4550 con 0x55ab69de11b0 2024-05-27T09:43:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2675f7b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f26681b1df0 con 0x55ab69db82f0 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2675f7b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f26681afc60 con 0x55ab69dc04d0 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2667e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69db82f0 0x55ab6a189280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2667400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a18d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2667e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69db82f0 0x55ab6a189280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54710/0 (socket says 172.21.15.44:54710) 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2667400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a18d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47014/0 (socket says 172.21.15.44:47014) 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2667e00640 1 -- 172.21.15.44:0/2908439860 learned_addr learned my addr 172.21.15.44:0/2908439860 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f266ce00640 1 --2- 172.21.15.44:0/2908439860 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69de11b0 0x55ab6a1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2667400640 1 -- 172.21.15.44:0/2908439860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69db82f0 msgr2=0x55ab6a189280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2667400640 1 --2- 172.21.15.44:0/2908439860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69db82f0 0x55ab6a189280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2667400640 1 -- 172.21.15.44:0/2908439860 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69de11b0 msgr2=0x55ab6a1a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2667400640 1 --2- 172.21.15.44:0/2908439860 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69de11b0 0x55ab6a1a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2667400640 1 -- 172.21.15.44:0/2908439860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab6a180b10 con 0x55ab69dc04d0 2024-05-27T09:43:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.026+0000 7f266ce00640 1 --2- 172.21.15.44:0/2908439860 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69de11b0 0x55ab6a1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:43:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.025+0000 7f2667e00640 1 --2- 172.21.15.44:0/2908439860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69db82f0 0x55ab6a189280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.026+0000 7f2667400640 1 --2- 172.21.15.44:0/2908439860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a18d2f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55ab6a1a0430 tx=0x7f2654019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e9d0ebe17f2d428 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.026+0000 7f265cc00640 1 -- 172.21.15.44:0/2908439860 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654005020 con 0x55ab69dc04d0 2024-05-27T09:43:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.026+0000 7f265cc00640 1 -- 172.21.15.44:0/2908439860 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26540047a0 con 0x55ab69dc04d0 2024-05-27T09:43:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.026+0000 7f265cc00640 1 -- 172.21.15.44:0/2908439860 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265402a660 con 0x55ab69dc04d0 2024-05-27T09:43:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.027+0000 7f2675f7b740 1 -- 172.21.15.44:0/2908439860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 msgr2=0x55ab6a18d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.027+0000 7f2675f7b740 1 --2- 172.21.15.44:0/2908439860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a18d2f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55ab6a1a0430 tx=0x7f2654019000 comp rx=0 tx=0).stop 2024-05-27T09:43:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.027+0000 7f2675f7b740 1 -- 172.21.15.44:0/2908439860 shutdown_connections 2024-05-27T09:43:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.027+0000 7f2675f7b740 1 --2- 172.21.15.44:0/2908439860 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69de11b0 0x55ab6a1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.027+0000 7f2675f7b740 1 --2- 172.21.15.44:0/2908439860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69dc04d0 0x55ab6a18d2f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.027+0000 7f2675f7b740 1 --2- 172.21.15.44:0/2908439860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69db82f0 0x55ab6a189280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.027+0000 7f2675f7b740 1 -- 172.21.15.44:0/2908439860 >> 172.21.15.44:0/2908439860 conn(0x55ab69dd5300 msgr2=0x55ab6a18fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.027+0000 7f2675f7b740 1 -- 172.21.15.44:0/2908439860 shutdown_connections 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.027+0000 7f2675f7b740 1 -- 172.21.15.44:0/2908439860 wait complete. 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.028+0000 7f2675f7b740 1 -- start start 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.028+0000 7f2675f7b740 1 Processor -- start 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.028+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69db82f0 0x55ab6a193a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.028+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69dc04d0 0x55ab6a193f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.028+0000 7f2675f7b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a194490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2675f7b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26681b02b0 con 0x55ab69dc04d0 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2675f7b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f26681b1720 con 0x55ab69de11b0 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2675f7b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f26681b0980 con 0x55ab69db82f0 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2675f7b740 10 client.0 _ll_register_callbacks cb 0x7f26580a5bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2667e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69db82f0 0x55ab6a193a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2667e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69db82f0 0x55ab6a193a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47030/0 (socket says 172.21.15.44:47030) 2024-05-27T09:43:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2667e00640 1 -- 172.21.15.44:0/2504175317 learned_addr learned my addr 172.21.15.44:0/2504175317 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2667400640 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69dc04d0 0x55ab6a193f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f266ce00640 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a194490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2667e00640 1 -- 172.21.15.44:0/2504175317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 msgr2=0x55ab6a194490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2667e00640 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a194490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2667e00640 1 -- 172.21.15.44:0/2504175317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69dc04d0 msgr2=0x55ab6a193f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2667e00640 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69dc04d0 0x55ab6a193f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.029+0000 7f2667e00640 1 -- 172.21.15.44:0/2504175317 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab6a2073c0 con 0x55ab69db82f0 2024-05-27T09:43:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.030+0000 7f2667e00640 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69db82f0 0x55ab6a193a10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f26580a9c80 tx=0x7f26580a9180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0e507ae35284a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.030+0000 7f2652000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.030+0000 7f2652000640 1 -- 172.21.15.44:0/2504175317 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26580c8030 con 0x55ab69db82f0 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.030+0000 7f2652000640 1 -- 172.21.15.44:0/2504175317 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f26580a62e0 con 0x55ab69db82f0 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.030+0000 7f2675f7b740 1 -- 172.21.15.44:0/2504175317 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26540047a0 con 0x55ab69db82f0 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.030+0000 7f2652000640 1 -- 172.21.15.44:0/2504175317 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f26580a9460 con 0x55ab69db82f0 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.030+0000 7f2675f7b740 10 client.26812 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.030+0000 7f2675f7b740 1 -- 172.21.15.44:0/2504175317 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ab6a208630 con 0x55ab69db82f0 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.030+0000 7f2652000640 1 -- 172.21.15.44:0/2504175317 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26540047a0 con 0x55ab69db82f0 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.030+0000 7f2652000640 1 -- 172.21.15.44:0/2504175317 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f26580c4040 con 0x55ab69db82f0 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.031+0000 7f2675f7b740 10 client.26812 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.031+0000 7f2675f7b740 10 client.26812 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.031+0000 7f2675f7b740 20 client.26812 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.031+0000 7f2675f7b740 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab69d751c0 0x55ab6a20acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.031+0000 7f2675f7b740 4 client.26812 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:43:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.031+0000 7f2675f7b740 1 -- 172.21.15.44:0/2504175317 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab6a0f02c0 con 0x55ab69d751c0 2024-05-27T09:43:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.031+0000 7f2667400640 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab69d751c0 0x55ab6a20acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.032+0000 7f2667400640 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab69d751c0 0x55ab6a20acb0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6deeb626ee385548 server_cookie=4f8fe9b5a432c72c in_seq=0 out_seq=0 2024-05-27T09:43:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.032+0000 7f2652000640 10 client.26812 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.393+0000 7f2652000640 1 -- 172.21.15.44:0/2504175317 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55ab6a0f02c0 con 0x55ab69d751c0 2024-05-27T09:43:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.393+0000 7f2652000640 10 client.26812 handle_command_reply: tid=0 2024-05-27T09:43:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.393+0000 7f2675f7b740 1 client.26812 shutdown 2024-05-27T09:43:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.394+0000 7f2675f7b740 1 -- 172.21.15.44:0/2504175317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69db82f0 msgr2=0x55ab6a193a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.394+0000 7f2675f7b740 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69db82f0 0x55ab6a193a10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f26580a9c80 tx=0x7f26580a9180 comp rx=0 tx=0).stop 2024-05-27T09:43:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.394+0000 7f2675f7b740 1 -- 172.21.15.44:0/2504175317 shutdown_connections 2024-05-27T09:43:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.394+0000 7f2675f7b740 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ab69d751c0 0x55ab6a20acb0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.394+0000 7f2675f7b740 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ab69de11b0 0x55ab6a194490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.394+0000 7f2652000640 0 client.26812 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.394+0000 7f2675f7b740 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab69dc04d0 0x55ab6a193f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.394+0000 7f2675f7b740 1 --2- 172.21.15.44:0/2504175317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ab69db82f0 0x55ab6a193a10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.394+0000 7f2675f7b740 1 -- 172.21.15.44:0/2504175317 >> 172.21.15.44:0/2504175317 conn(0x55ab69dd5300 msgr2=0x55ab6a18efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.394+0000 7f2675f7b740 1 -- 172.21.15.44:0/2504175317 shutdown_connections 2024-05-27T09:43:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.395+0000 7f2675f7b740 1 -- 172.21.15.44:0/2504175317 wait complete. 2024-05-27T09:43:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.395+0000 7f2675f7b740 20 client.26812 trim_cache size 0 max 16384 2024-05-27T09:43:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:43:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:43:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:43:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.396+0000 7f266ee00640 1 -- 172.21.15.44:0/3592572795 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f264c0b2860 msgr2=0x7f264c0b4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.396+0000 7f266ee00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f264c0b2860 0x7f264c0b4d70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2654009640 tx=0x7f2654005f90 comp rx=0 tx=0).stop 2024-05-27T09:43:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.396+0000 7f266ee00640 1 -- 172.21.15.44:0/3592572795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2668075660 msgr2=0x7f26681a26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.396+0000 7f266ee00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2668075660 0x7f26681a26c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f265800e9f0 tx=0x7f265800eeb0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.396+0000 7f266ee00640 1 -- 172.21.15.44:0/3592572795 shutdown_connections 2024-05-27T09:43:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.396+0000 7f266ee00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26681104a0 0x7f266819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.396+0000 7f266ee00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f266810db40 0x7f26681a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.396+0000 7f266ee00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f264c0b2860 0x7f264c0b4d70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.396+0000 7f266ee00640 1 --2- 172.21.15.44:0/3592572795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2668075660 0x7f26681a26c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:37.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.396+0000 7f266ee00640 1 -- 172.21.15.44:0/3592572795 >> 172.21.15.44:0/3592572795 conn(0x7f26680ff480 msgr2=0x7f26681108b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:37.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.397+0000 7f266ee00640 1 -- 172.21.15.44:0/3592572795 shutdown_connections 2024-05-27T09:43:37.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:37.397+0000 7f266ee00640 1 -- 172.21.15.44:0/3592572795 wait complete. 2024-05-27T09:43:38.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:38 smithi044 ceph-mon[31618]: pgmap v259: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 6.1 MiB/s wr, 899 op/s 2024-05-27T09:43:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:38 smithi050 ceph-mon[36371]: pgmap v259: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 6.1 MiB/s wr, 899 op/s 2024-05-27T09:43:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:38 smithi189 ceph-mon[36671]: pgmap v259: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 6.1 MiB/s wr, 899 op/s 2024-05-27T09:43:40.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:40 smithi044 ceph-mon[31618]: pgmap v260: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 201 op/s 2024-05-27T09:43:40.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:40 smithi050 ceph-mon[36371]: pgmap v260: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 201 op/s 2024-05-27T09:43:40.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:40 smithi189 ceph-mon[36671]: pgmap v260: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 201 op/s 2024-05-27T09:43:42.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:42 smithi050 ceph-mon[36371]: pgmap v261: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 365 op/s 2024-05-27T09:43:42.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:42 smithi189 ceph-mon[36671]: pgmap v261: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 365 op/s 2024-05-27T09:43:42.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:42 smithi044 ceph-mon[31618]: pgmap v261: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 365 op/s 2024-05-27T09:43:44.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:44 smithi050 ceph-mon[36371]: pgmap v262: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 230 op/s 2024-05-27T09:43:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:44 smithi189 ceph-mon[36671]: pgmap v262: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 230 op/s 2024-05-27T09:43:44.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:44 smithi044 ceph-mon[31618]: pgmap v262: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 230 op/s 2024-05-27T09:43:46.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:46 smithi050 ceph-mon[36371]: pgmap v263: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 230 op/s 2024-05-27T09:43:46.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:46 smithi189 ceph-mon[36671]: pgmap v263: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 230 op/s 2024-05-27T09:43:46.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:46 smithi044 ceph-mon[31618]: pgmap v263: 97 pgs: 97 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 230 op/s 2024-05-27T09:43:48.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:48 smithi050 ceph-mon[36371]: pgmap v264: 97 pgs: 97 active+clean; 185 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-05-27T09:43:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:48 smithi189 ceph-mon[36671]: pgmap v264: 97 pgs: 97 active+clean; 185 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-05-27T09:43:48.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:48 smithi044 ceph-mon[31618]: pgmap v264: 97 pgs: 97 active+clean; 185 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-05-27T09:43:50.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:50 smithi050 ceph-mon[36371]: pgmap v265: 97 pgs: 97 active+clean; 185 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 188 op/s 2024-05-27T09:43:50.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:50 smithi189 ceph-mon[36671]: pgmap v265: 97 pgs: 97 active+clean; 185 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 188 op/s 2024-05-27T09:43:50.388 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:50 smithi044 ceph-mon[31618]: pgmap v265: 97 pgs: 97 active+clean; 185 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 188 op/s 2024-05-27T09:43:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.634+0000 7f30b8a00640 1 -- 172.21.15.44:0/3989924380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30b410f390 msgr2=0x7f30b4116b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.634+0000 7f30b8a00640 1 --2- 172.21.15.44:0/3989924380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30b410f390 0x7f30b4116b30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009900 tx=0x7f30ac02a410 comp rx=0 tx=0).stop 2024-05-27T09:43:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.634+0000 7f30b8a00640 1 -- 172.21.15.44:0/3989924380 shutdown_connections 2024-05-27T09:43:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.634+0000 7f30b8a00640 1 --2- 172.21.15.44:0/3989924380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30b410f390 0x7f30b4116b30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.634+0000 7f30b8a00640 1 --2- 172.21.15.44:0/3989924380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b410e8d0 0x7f30b410ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.634+0000 7f30b8a00640 1 --2- 172.21.15.44:0/3989924380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30b410a340 0x7f30b410a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.634+0000 7f30b8a00640 1 -- 172.21.15.44:0/3989924380 >> 172.21.15.44:0/3989924380 conn(0x7f30b4076cd0 msgr2=0x7f30b4077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.634+0000 7f30b8a00640 1 -- 172.21.15.44:0/3989924380 shutdown_connections 2024-05-27T09:43:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.634+0000 7f30b8a00640 1 -- 172.21.15.44:0/3989924380 wait complete. 2024-05-27T09:43:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.635+0000 7f30b8a00640 1 Processor -- start 2024-05-27T09:43:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30b8a00640 1 -- start start 2024-05-27T09:43:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30b8a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30b410a340 0x7f30b41a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30b8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b410e8d0 0x7f30b41a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30b8a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30b410f390 0x7f30b41a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30b8a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f30b411a2f0 con 0x7f30b410f390 2024-05-27T09:43:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30b8a00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f30b411a170 con 0x7f30b410a340 2024-05-27T09:43:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30b8a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f30b411a470 con 0x7f30b410e8d0 2024-05-27T09:43:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30b2800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30b410a340 0x7f30b41a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30b2800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30b410a340 0x7f30b41a6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:46038/0 (socket says 172.21.15.44:46038) 2024-05-27T09:43:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30b2800640 1 -- 172.21.15.44:0/2759501741 learned_addr learned my addr 172.21.15.44:0/2759501741 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30b3200640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30b410f390 0x7f30b41a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.636+0000 7f30a9e00640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b410e8d0 0x7f30b41a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.637+0000 7f30b2800640 1 -- 172.21.15.44:0/2759501741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b410e8d0 msgr2=0x7f30b41a6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.637+0000 7f30b2800640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b410e8d0 0x7f30b41a6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.637+0000 7f30b2800640 1 -- 172.21.15.44:0/2759501741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30b410f390 msgr2=0x7f30b41a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.637+0000 7f30b2800640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30b410f390 0x7f30b41a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.637+0000 7f30b2800640 1 -- 172.21.15.44:0/2759501741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b41a1130 con 0x7f30b410a340 2024-05-27T09:43:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.637+0000 7f30a9e00640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b410e8d0 0x7f30b41a6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:43:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.637+0000 7f30b2800640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30b410a340 0x7f30b41a6880 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f30a400ebc0 tx=0x7f30a400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.637+0000 7f30abe00640 1 -- 172.21.15.44:0/2759501741 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a4010630 con 0x7f30b410a340 2024-05-27T09:43:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.637+0000 7f30abe00640 1 -- 172.21.15.44:0/2759501741 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30a4010c80 con 0x7f30b410a340 2024-05-27T09:43:50.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.637+0000 7f30b8a00640 1 -- 172.21.15.44:0/2759501741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30b41a1420 con 0x7f30b410a340 2024-05-27T09:43:50.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.637+0000 7f30abe00640 1 -- 172.21.15.44:0/2759501741 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a4018b60 con 0x7f30b410a340 2024-05-27T09:43:50.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.638+0000 7f30b8a00640 1 -- 172.21.15.44:0/2759501741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30b41ad700 con 0x7f30b410a340 2024-05-27T09:43:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.641+0000 7f30c03f0740 1 Processor -- start 2024-05-27T09:43:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.641+0000 7f30c03f0740 1 -- start start 2024-05-27T09:43:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7fc71b0 0x559ae8369a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7f9e2f0 0x559ae8369fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae836a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30abe00640 1 -- 172.21.15.44:0/2759501741 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f30a4018d00 con 0x7f30b410a340 2024-05-27T09:43:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30c03f0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559ae82dab90 con 0x559ae7fc71b0 2024-05-27T09:43:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30c03f0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559ae82ca730 con 0x559ae7f9e2f0 2024-05-27T09:43:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30c03f0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559ae82ce770 con 0x559ae7fa64d0 2024-05-27T09:43:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30b3200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7fc71b0 0x559ae8369a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30b3200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7fc71b0 0x559ae8369a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34314/0 (socket says 172.21.15.44:34314) 2024-05-27T09:43:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30b3200640 1 -- 172.21.15.44:0/2605893033 learned_addr learned my addr 172.21.15.44:0/2605893033 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30a9e00640 1 --2- 172.21.15.44:0/2605893033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae836a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.642+0000 7f30b2800640 1 --2- 172.21.15.44:0/2605893033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7f9e2f0 0x559ae8369fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.643+0000 7f30b3200640 1 -- 172.21.15.44:0/2605893033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 msgr2=0x559ae836a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.643+0000 7f30b3200640 1 --2- 172.21.15.44:0/2605893033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae836a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.643+0000 7f30b3200640 1 -- 172.21.15.44:0/2605893033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7f9e2f0 msgr2=0x559ae8369fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.643+0000 7f30b3200640 1 --2- 172.21.15.44:0/2605893033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7f9e2f0 0x559ae8369fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.643+0000 7f30b3200640 1 -- 172.21.15.44:0/2605893033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ae82638b0 con 0x559ae7fc71b0 2024-05-27T09:43:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.643+0000 7f30b3200640 1 --2- 172.21.15.44:0/2605893033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7fc71b0 0x559ae8369a60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f30ac02aa00 tx=0x7f30ac0040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbbe838042609902 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.643+0000 7f30a3400640 1 -- 172.21.15.44:0/2605893033 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac035030 con 0x559ae7fc71b0 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30a3400640 1 -- 172.21.15.44:0/2605893033 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30ac0044c0 con 0x559ae7fc71b0 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30a3400640 1 -- 172.21.15.44:0/2605893033 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac0035f0 con 0x559ae7fc71b0 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30abe00640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30980b29d0 0x7f30980b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30c03f0740 1 -- 172.21.15.44:0/2605893033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7fc71b0 msgr2=0x559ae8369a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30c03f0740 1 --2- 172.21.15.44:0/2605893033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7fc71b0 0x559ae8369a60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f30ac02aa00 tx=0x7f30ac0040d0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30abe00640 1 -- 172.21.15.44:0/2759501741 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f30a40e2d30 con 0x7f30b410a340 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30b2800640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30980b29d0 0x7f30980b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30c03f0740 1 -- 172.21.15.44:0/2605893033 shutdown_connections 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30c03f0740 1 --2- 172.21.15.44:0/2605893033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae836a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30c03f0740 1 --2- 172.21.15.44:0/2605893033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7f9e2f0 0x559ae8369fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30c03f0740 1 --2- 172.21.15.44:0/2605893033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7fc71b0 0x559ae8369a60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.644+0000 7f30c03f0740 1 -- 172.21.15.44:0/2605893033 >> 172.21.15.44:0/2605893033 conn(0x559ae7fbb300 msgr2=0x559ae8366210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.645+0000 7f30c03f0740 1 -- 172.21.15.44:0/2605893033 shutdown_connections 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.645+0000 7f30c03f0740 1 -- 172.21.15.44:0/2605893033 wait complete. 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.645+0000 7f30b2800640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30980b29d0 0x7f30980b4ee0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x559ae8367550 tx=0x7f30a40e6820 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.645+0000 7f30c03f0740 1 -- start start 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.646+0000 7f30c03f0740 1 Processor -- start 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.646+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae8373bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.646+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83740f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.646+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fc71b0 0x559ae837fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.646+0000 7f30c03f0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559ae7fb0790 con 0x559ae7f9e2f0 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.646+0000 7f30c03f0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559ae8272bc0 con 0x559ae7fc71b0 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.646+0000 7f30c03f0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559ae8376b30 con 0x559ae7fa64d0 2024-05-27T09:43:50.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.646+0000 7f30c03f0740 10 client.0 _ll_register_callbacks cb 0x559ae8250400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f30a9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83740f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f30a9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83740f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:56254/0 (socket says 172.21.15.44:56254) 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f30a9e00640 1 -- 172.21.15.44:0/1012919773 learned_addr learned my addr 172.21.15.44:0/1012919773 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f30b3200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae8373bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f30b3200640 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fc71b0 0x559ae837fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f30a9e00640 1 -- 172.21.15.44:0/1012919773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fc71b0 msgr2=0x559ae837fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f30a9e00640 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fc71b0 0x559ae837fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f30a9e00640 1 -- 172.21.15.44:0/1012919773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 msgr2=0x559ae8373bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f30a9e00640 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae8373bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f30a9e00640 1 -- 172.21.15.44:0/1012919773 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ae7fdf210 con 0x559ae7fa64d0 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f30a9e00640 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83740f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f309c012700 tx=0x7f309c012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8afa76865712845 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.647+0000 7f3097e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.648+0000 7f3097e00640 1 -- 172.21.15.44:0/1012919773 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c011bb0 con 0x559ae7fa64d0 2024-05-27T09:43:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.648+0000 7f3097e00640 1 -- 172.21.15.44:0/1012919773 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f309c011d50 con 0x559ae7fa64d0 2024-05-27T09:43:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.648+0000 7f30c03f0740 1 -- 172.21.15.44:0/1012919773 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559ae82dcae0 con 0x559ae7fa64d0 2024-05-27T09:43:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.648+0000 7f3097e00640 1 -- 172.21.15.44:0/1012919773 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f309c005020 con 0x559ae7fa64d0 2024-05-27T09:43:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.648+0000 7f30c03f0740 10 client.26824 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.648+0000 7f30c03f0740 1 -- 172.21.15.44:0/1012919773 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559ae80ab620 con 0x559ae7fa64d0 2024-05-27T09:43:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.648+0000 7f3097e00640 1 -- 172.21.15.44:0/1012919773 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x559ae82dcae0 con 0x559ae7fa64d0 2024-05-27T09:43:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.649+0000 7f3097e00640 1 -- 172.21.15.44:0/1012919773 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f309c025480 con 0x559ae7fa64d0 2024-05-27T09:43:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.649+0000 7f30c03f0740 10 client.26824 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.649+0000 7f30c03f0740 10 client.26824 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.649+0000 7f30c03f0740 20 client.26824 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.649+0000 7f30c03f0740 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559ae7f85d80 0x559ae8385da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.649+0000 7f30c03f0740 4 client.26824 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:43:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.649+0000 7f30c03f0740 1 -- 172.21.15.44:0/1012919773 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ae82d62c0 con 0x559ae7f85d80 2024-05-27T09:43:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.649+0000 7f30b3200640 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559ae7f85d80 0x559ae8385da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.650+0000 7f30b3200640 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559ae7f85d80 0x559ae8385da0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d1a04aab9ea4087 server_cookie=804f0101845c5e17 in_seq=0 out_seq=0 2024-05-27T09:43:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.650+0000 7f3097e00640 10 client.26824 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f3097e00640 1 -- 172.21.15.44:0/1012919773 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559ae82d62c0 con 0x559ae7f85d80 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f3097e00640 10 client.26824 handle_command_reply: tid=0 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 1 client.26824 shutdown 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 1 -- 172.21.15.44:0/1012919773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 msgr2=0x559ae83740f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83740f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f309c012700 tx=0x7f309c012bc0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 1 -- 172.21.15.44:0/1012919773 shutdown_connections 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559ae7f85d80 0x559ae8385da0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fc71b0 0x559ae837fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83740f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 1 --2- 172.21.15.44:0/1012919773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae8373bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 1 -- 172.21.15.44:0/1012919773 >> 172.21.15.44:0/1012919773 conn(0x559ae7fbb300 msgr2=0x559ae8369890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f3097e00640 0 client.26824 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 1 -- 172.21.15.44:0/1012919773 shutdown_connections 2024-05-27T09:43:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 1 -- 172.21.15.44:0/1012919773 wait complete. 2024-05-27T09:43:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.651+0000 7f30c03f0740 20 client.26824 trim_cache size 0 max 16384 2024-05-27T09:43:50.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.669+0000 7f30c03f0740 1 Processor -- start 2024-05-27T09:43:50.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.669+0000 7f30c03f0740 1 -- start start 2024-05-27T09:43:50.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.669+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae83762e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.669+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fa64d0 0x559ae8365240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.669+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fc71b0 0x559ae837fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.669+0000 7f30c03f0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f30b41b88b0 con 0x559ae7f9e2f0 2024-05-27T09:43:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.669+0000 7f30c03f0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f30b41b6150 con 0x559ae7fa64d0 2024-05-27T09:43:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.669+0000 7f30c03f0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f30b41b3fc0 con 0x559ae7fc71b0 2024-05-27T09:43:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.670+0000 7f30a9e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fa64d0 0x559ae8365240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.670+0000 7f30a9e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fa64d0 0x559ae8365240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:46056/0 (socket says 172.21.15.44:46056) 2024-05-27T09:43:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.670+0000 7f30a9e00640 1 -- 172.21.15.44:0/4035834093 learned_addr learned my addr 172.21.15.44:0/4035834093 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.670+0000 7f30b3200640 1 --2- 172.21.15.44:0/4035834093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae83762e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.670+0000 7f30a9e00640 1 -- 172.21.15.44:0/4035834093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fc71b0 msgr2=0x559ae837fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.670+0000 7f30b3200640 1 --2- 172.21.15.44:0/4035834093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fc71b0 0x559ae837fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.670+0000 7f30a9e00640 1 --2- 172.21.15.44:0/4035834093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fc71b0 0x559ae837fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.670+0000 7f30a9e00640 1 -- 172.21.15.44:0/4035834093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 msgr2=0x559ae83762e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.670+0000 7f30a9e00640 1 --2- 172.21.15.44:0/4035834093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae83762e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.670+0000 7f30a9e00640 1 -- 172.21.15.44:0/4035834093 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ae8374450 con 0x559ae7fa64d0 2024-05-27T09:43:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.670+0000 7f30b3200640 1 --2- 172.21.15.44:0/4035834093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fc71b0 0x559ae837fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:43:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.671+0000 7f30a9e00640 1 --2- 172.21.15.44:0/4035834093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fa64d0 0x559ae8365240 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f309c0126d0 tx=0x7f309c009fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faee3269ecaaa42b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.671+0000 7f30a2a00640 1 -- 172.21.15.44:0/4035834093 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c025500 con 0x559ae7fa64d0 2024-05-27T09:43:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.671+0000 7f30a2a00640 1 -- 172.21.15.44:0/4035834093 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f309c011bb0 con 0x559ae7fa64d0 2024-05-27T09:43:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.671+0000 7f30a2a00640 1 -- 172.21.15.44:0/4035834093 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c005650 con 0x559ae7fa64d0 2024-05-27T09:43:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.672+0000 7f30c03f0740 1 -- 172.21.15.44:0/4035834093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fa64d0 msgr2=0x559ae8365240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.672+0000 7f30c03f0740 1 --2- 172.21.15.44:0/4035834093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fa64d0 0x559ae8365240 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f309c0126d0 tx=0x7f309c009fa0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.672+0000 7f30c03f0740 1 -- 172.21.15.44:0/4035834093 shutdown_connections 2024-05-27T09:43:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.672+0000 7f30c03f0740 1 --2- 172.21.15.44:0/4035834093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fc71b0 0x559ae837fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.672+0000 7f30c03f0740 1 --2- 172.21.15.44:0/4035834093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fa64d0 0x559ae8365240 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.672+0000 7f30c03f0740 1 --2- 172.21.15.44:0/4035834093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae83762e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.672+0000 7f30c03f0740 1 -- 172.21.15.44:0/4035834093 >> 172.21.15.44:0/4035834093 conn(0x559ae7fbb300 msgr2=0x559ae83660d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.672+0000 7f30c03f0740 1 -- 172.21.15.44:0/4035834093 shutdown_connections 2024-05-27T09:43:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.672+0000 7f30c03f0740 1 -- 172.21.15.44:0/4035834093 wait complete. 2024-05-27T09:43:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.672+0000 7f30c03f0740 1 -- start start 2024-05-27T09:43:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.673+0000 7f30c03f0740 1 Processor -- start 2024-05-27T09:43:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.673+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae83771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.673+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83776f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.673+0000 7f30c03f0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fc71b0 0x559ae83ec460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.673+0000 7f30c03f0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f30b41b4610 con 0x559ae7f9e2f0 2024-05-27T09:43:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.673+0000 7f30c03f0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f30b41b5a80 con 0x559ae7fc71b0 2024-05-27T09:43:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.673+0000 7f30c03f0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f30b41b4ce0 con 0x559ae7fa64d0 2024-05-27T09:43:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.673+0000 7f30c03f0740 10 client.0 _ll_register_callbacks cb 0x7f309c0097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.674+0000 7f30a9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83776f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.674+0000 7f30a9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83776f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:56262/0 (socket says 172.21.15.44:56262) 2024-05-27T09:43:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.674+0000 7f30a9e00640 1 -- 172.21.15.44:0/2409738896 learned_addr learned my addr 172.21.15.44:0/2409738896 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.674+0000 7f30b3200640 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae83771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.674+0000 7f30b3200640 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fc71b0 0x559ae83ec460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.674+0000 7f30a9e00640 1 -- 172.21.15.44:0/2409738896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fc71b0 msgr2=0x559ae83ec460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.674+0000 7f30a9e00640 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fc71b0 0x559ae83ec460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.674+0000 7f30a9e00640 1 -- 172.21.15.44:0/2409738896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 msgr2=0x559ae83771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.674+0000 7f30a9e00640 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae83771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.674+0000 7f30a9e00640 1 -- 172.21.15.44:0/2409738896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ae83ec9a0 con 0x559ae7fa64d0 2024-05-27T09:43:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.675+0000 7f30b3200640 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fc71b0 0x559ae83ec460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:43:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.675+0000 7f30a9e00640 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83776f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f309c0054d0 tx=0x7f309c005a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1924c28b142db89f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.675+0000 7f3097e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:43:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.675+0000 7f3097e00640 1 -- 172.21.15.44:0/2409738896 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c005630 con 0x559ae7fa64d0 2024-05-27T09:43:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.675+0000 7f3097e00640 1 -- 172.21.15.44:0/2409738896 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8521+0+0 (secure 0 0 0) 0x7f309c011d60 con 0x559ae7fa64d0 2024-05-27T09:43:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.675+0000 7f30c03f0740 1 -- 172.21.15.44:0/2409738896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f309c011bb0 con 0x559ae7fa64d0 2024-05-27T09:43:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.676+0000 7f3097e00640 1 -- 172.21.15.44:0/2409738896 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f309c004660 con 0x559ae7fa64d0 2024-05-27T09:43:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.676+0000 7f30c03f0740 10 client.26827 fetch_fsmap learned FSMap version 29 2024-05-27T09:43:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.676+0000 7f30c03f0740 1 -- 172.21.15.44:0/2409738896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559ae83eda40 con 0x559ae7fa64d0 2024-05-27T09:43:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.676+0000 7f3097e00640 1 -- 172.21.15.44:0/2409738896 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f309c011bb0 con 0x559ae7fa64d0 2024-05-27T09:43:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.676+0000 7f3097e00640 1 -- 172.21.15.44:0/2409738896 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8342+0+0 (secure 0 0 0) 0x7f309c01c7e0 con 0x559ae7fa64d0 2024-05-27T09:43:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.676+0000 7f30c03f0740 10 client.26827 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T09:43:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.676+0000 7f30c03f0740 10 client.26827 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.676+0000 7f30c03f0740 20 client.26827 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:50.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.676+0000 7f30c03f0740 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559ae7f5b1c0 0x559ae83f0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:50.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.676+0000 7f30c03f0740 4 client.26827 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:43:50.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.676+0000 7f30c03f0740 1 -- 172.21.15.44:0/2409738896 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559ae82504b0 con 0x559ae7f5b1c0 2024-05-27T09:43:50.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.677+0000 7f30b3200640 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559ae7f5b1c0 0x559ae83f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:50.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.677+0000 7f30b3200640 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559ae7f5b1c0 0x559ae83f0100 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0bfa681c99dbc22 server_cookie=d32d9831e012fd8f in_seq=0 out_seq=0 2024-05-27T09:43:50.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.677+0000 7f3097e00640 10 client.26827 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:50.784 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:43:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.987+0000 7f3097e00640 1 -- 172.21.15.44:0/2409738896 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x559ae82504b0 con 0x559ae7f5b1c0 2024-05-27T09:43:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.987+0000 7f3097e00640 10 client.26827 handle_command_reply: tid=0 2024-05-27T09:43:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.987+0000 7f30c03f0740 1 client.26827 shutdown 2024-05-27T09:43:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f30c03f0740 1 -- 172.21.15.44:0/2409738896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 msgr2=0x559ae83776f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f30c03f0740 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83776f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f309c0054d0 tx=0x7f309c005a70 comp rx=0 tx=0).stop 2024-05-27T09:43:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f30c03f0740 1 -- 172.21.15.44:0/2409738896 shutdown_connections 2024-05-27T09:43:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f30c03f0740 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559ae7f5b1c0 0x559ae83f0100 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f30c03f0740 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559ae7fc71b0 0x559ae83ec460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f30c03f0740 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559ae7fa64d0 0x559ae83776f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f30c03f0740 1 --2- 172.21.15.44:0/2409738896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559ae7f9e2f0 0x559ae83771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f3097e00640 0 client.26827 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f30c03f0740 1 -- 172.21.15.44:0/2409738896 >> 172.21.15.44:0/2409738896 conn(0x559ae7fbb300 msgr2=0x559ae83660d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f30c03f0740 1 -- 172.21.15.44:0/2409738896 shutdown_connections 2024-05-27T09:43:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f30c03f0740 1 -- 172.21.15.44:0/2409738896 wait complete. 2024-05-27T09:43:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.988+0000 7f30c03f0740 20 client.26827 trim_cache size 0 max 16384 2024-05-27T09:43:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:43:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:43:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:43:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 -- 172.21.15.44:0/2759501741 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30980b29d0 msgr2=0x7f30980b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30980b29d0 0x7f30980b4ee0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x559ae8367550 tx=0x7f30a40e6820 comp rx=0 tx=0).stop 2024-05-27T09:43:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 -- 172.21.15.44:0/2759501741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30b410a340 msgr2=0x7f30b41a6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30b410a340 0x7f30b41a6880 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f30a400ebc0 tx=0x7f30a400c6a0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 -- 172.21.15.44:0/2759501741 shutdown_connections 2024-05-27T09:43:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30980b29d0 0x7f30980b4ee0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30b410f390 0x7f30b41a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30b410e8d0 0x7f30b41a6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 --2- 172.21.15.44:0/2759501741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30b410a340 0x7f30b41a6880 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 -- 172.21.15.44:0/2759501741 >> 172.21.15.44:0/2759501741 conn(0x7f30b4076cd0 msgr2=0x7f30b4107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 -- 172.21.15.44:0/2759501741 shutdown_connections 2024-05-27T09:43:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:43:50.989+0000 7f30b8a00640 1 -- 172.21.15.44:0/2759501741 wait complete. 2024-05-27T09:43:51.309 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:43:51.587 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:43:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[72375]: ignoring --setuser ceph since I am not root 2024-05-27T09:43:51.588 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:43:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[72375]: ignoring --setgroup ceph since I am not root 2024-05-27T09:43:51.588 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:43:51 smithi189 ceph-mds[72379]: -- 172.21.15.189:0/1122422511 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56540e8b1a40 con 0x56540e8f9180 2024-05-27T09:43:51.588 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:43:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[72375]: starting mds.f at 2024-05-27T09:43:52.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:52 smithi044 ceph-mon[31618]: pgmap v266: 97 pgs: 97 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 208 op/s 2024-05-27T09:43:52.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:52 smithi044 ceph-mon[31618]: daemon mds.f restarted 2024-05-27T09:43:52.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:52 smithi050 ceph-mon[36371]: pgmap v266: 97 pgs: 97 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 208 op/s 2024-05-27T09:43:52.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:52 smithi050 ceph-mon[36371]: daemon mds.f restarted 2024-05-27T09:43:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:52 smithi189 ceph-mon[36671]: pgmap v266: 97 pgs: 97 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 208 op/s 2024-05-27T09:43:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:52 smithi189 ceph-mon[36671]: daemon mds.f restarted 2024-05-27T09:43:53.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:53 smithi050 ceph-mon[36371]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T09:43:53.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:53 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6834/3188237380,v1:172.21.15.189:6835/3188237380] up:boot 2024-05-27T09:43:53.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:53 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:43:53.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:53 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:43:53.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:53 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:43:53.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:53 smithi044 ceph-mon[31618]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T09:43:53.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:53 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6834/3188237380,v1:172.21.15.189:6835/3188237380] up:boot 2024-05-27T09:43:53.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:53 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:43:53.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:53 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:43:53.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:53 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:43:53.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:53 smithi189 ceph-mon[36671]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T09:43:53.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:53 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6834/3188237380,v1:172.21.15.189:6835/3188237380] up:boot 2024-05-27T09:43:53.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:53 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:43:53.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:53 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:43:53.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:53 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:43:53.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.673+0000 7f21dfc00700 1 -- 172.21.15.44:0/3156357479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21d810e8e0 msgr2=0x7f21d8110dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:53.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.673+0000 7f21dfc00700 1 --2- 172.21.15.44:0/3156357479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21d810e8e0 0x7f21d8110dc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009f90 tx=0x7f21d000b330 comp rx=0 tx=0).stop 2024-05-27T09:43:53.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.673+0000 7f21dfc00700 1 -- 172.21.15.44:0/3156357479 shutdown_connections 2024-05-27T09:43:53.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.673+0000 7f21dfc00700 1 --2- 172.21.15.44:0/3156357479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21d810e8e0 0x7f21d8110dc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.673+0000 7f21dfc00700 1 --2- 172.21.15.44:0/3156357479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d81054b0 0x7f21d81058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.673+0000 7f21dfc00700 1 --2- 172.21.15.44:0/3156357479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21d8105e00 0x7f21d810e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.673+0000 7f21dfc00700 1 -- 172.21.15.44:0/3156357479 >> 172.21.15.44:0/3156357479 conn(0x7f21d80fedb0 msgr2=0x7f21d81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:53.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.673+0000 7f21dfc00700 1 -- 172.21.15.44:0/3156357479 shutdown_connections 2024-05-27T09:43:53.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.673+0000 7f21dfc00700 1 -- 172.21.15.44:0/3156357479 wait complete. 2024-05-27T09:43:53.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.674+0000 7f21dfc00700 1 Processor -- start 2024-05-27T09:43:53.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.675+0000 7f21dfc00700 1 -- start start 2024-05-27T09:43:53.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.675+0000 7f21dfc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21d81054b0 0x7f21d819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.676+0000 7f21dfc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d8105e00 0x7f21d819d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.676+0000 7f21dfc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21d810e8e0 0x7f21d81a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.676+0000 7f21dfc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f21d8114490 con 0x7f21d810e8e0 2024-05-27T09:43:53.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.676+0000 7f21dfc00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f21d8114310 con 0x7f21d81054b0 2024-05-27T09:43:53.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.676+0000 7f21dfc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f21d8114610 con 0x7f21d8105e00 2024-05-27T09:43:53.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.676+0000 7f21dd200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21d81054b0 0x7f21d819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.676+0000 7f21dd200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21d81054b0 0x7f21d819cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42334/0 (socket says 172.21.15.44:42334) 2024-05-27T09:43:53.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.676+0000 7f21dd200700 1 -- 172.21.15.44:0/1322166730 learned_addr learned my addr 172.21.15.44:0/1322166730 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:53.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.676+0000 7f21ddc00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21d810e8e0 0x7f21d81a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.676+0000 7f21d7e00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d8105e00 0x7f21d819d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.677+0000 7f21d7e00700 1 -- 172.21.15.44:0/1322166730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21d81054b0 msgr2=0x7f21d819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:53.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.677+0000 7f21d7e00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21d81054b0 0x7f21d819cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.677+0000 7f21d7e00700 1 -- 172.21.15.44:0/1322166730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21d810e8e0 msgr2=0x7f21d81a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:53.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.677+0000 7f21d7e00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21d810e8e0 0x7f21d81a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.677+0000 7f21d7e00700 1 -- 172.21.15.44:0/1322166730 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d81a1e30 con 0x7f21d8105e00 2024-05-27T09:43:53.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.677+0000 7f21dd200700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21d81054b0 0x7f21d819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:53.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.677+0000 7f21ddc00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21d810e8e0 0x7f21d81a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:43:53.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.678+0000 7f21d7e00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d8105e00 0x7f21d819d500 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f21c400e9f0 tx=0x7f21c400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:53.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.679+0000 7f21d5600700 1 -- 172.21.15.44:0/1322166730 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c400c9c0 con 0x7f21d8105e00 2024-05-27T09:43:53.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.679+0000 7f21d5600700 1 -- 172.21.15.44:0/1322166730 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21c400cb60 con 0x7f21d8105e00 2024-05-27T09:43:53.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.679+0000 7f21d5600700 1 -- 172.21.15.44:0/1322166730 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4018680 con 0x7f21d8105e00 2024-05-27T09:43:53.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.679+0000 7f21dfc00700 1 -- 172.21.15.44:0/1322166730 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d81a2120 con 0x7f21d8105e00 2024-05-27T09:43:53.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.679+0000 7f21dfc00700 1 -- 172.21.15.44:0/1322166730 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d81a9a40 con 0x7f21d8105e00 2024-05-27T09:43:53.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.683+0000 7f21d5600700 1 -- 172.21.15.44:0/1322166730 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f21c4010430 con 0x7f21d8105e00 2024-05-27T09:43:53.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.684+0000 7f21ee769b80 1 Processor -- start 2024-05-27T09:43:53.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.685+0000 7f21ee769b80 1 -- start start 2024-05-27T09:43:53.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.685+0000 7f21d5600700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f21c00b2a70 0x7f21c00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.685+0000 7f21d5600700 1 -- 172.21.15.44:0/1322166730 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8477+0+0 (secure 0 0 0) 0x7f21c40e26a0 con 0x7f21d8105e00 2024-05-27T09:43:53.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.685+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 0x563da494b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.685+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da47cce40 0x563da494b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.685+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da494be70 0x563da4958110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.686+0000 7f21ee769b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563da4879e40 con 0x563da47cce40 2024-05-27T09:43:53.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.686+0000 7f21ee769b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563da48699e0 con 0x563da48b89f0 2024-05-27T09:43:53.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.686+0000 7f21ee769b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563da486da20 con 0x563da494be70 2024-05-27T09:43:53.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.686+0000 7f21dd200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 0x563da494b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.686+0000 7f21dd200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 0x563da494b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42344/0 (socket says 172.21.15.44:42344) 2024-05-27T09:43:53.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.686+0000 7f21dd200700 1 -- 172.21.15.44:0/1081364991 learned_addr learned my addr 172.21.15.44:0/1081364991 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.686+0000 7f21dd200700 1 -- 172.21.15.44:0/1081364991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da494be70 msgr2=0x563da4958110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:43:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.686+0000 7f21dd200700 1 --2- 172.21.15.44:0/1081364991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da494be70 0x563da4958110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.686+0000 7f21dd200700 1 -- 172.21.15.44:0/1081364991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da47cce40 msgr2=0x563da494b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.686+0000 7f21dd200700 1 --2- 172.21.15.44:0/1081364991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da47cce40 0x563da494b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.686+0000 7f21dd200700 1 -- 172.21.15.44:0/1081364991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563da4832420 con 0x563da48b89f0 2024-05-27T09:43:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.687+0000 7f21ddc00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f21c00b2a70 0x7f21c00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.687+0000 7f21dd200700 1 --2- 172.21.15.44:0/1081364991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 0x563da494b330 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f21cc011650 tx=0x7f21cc011960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e98c142830f52f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.688+0000 7f21ca000700 1 -- 172.21.15.44:0/1081364991 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc027070 con 0x563da48b89f0 2024-05-27T09:43:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.688+0000 7f21ca000700 1 -- 172.21.15.44:0/1081364991 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21cc022440 con 0x563da48b89f0 2024-05-27T09:43:53.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.688+0000 7f21ca000700 1 -- 172.21.15.44:0/1081364991 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc00e610 con 0x563da48b89f0 2024-05-27T09:43:53.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.689+0000 7f21ee769b80 1 -- 172.21.15.44:0/1081364991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 msgr2=0x563da494b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:53.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.689+0000 7f21ee769b80 1 --2- 172.21.15.44:0/1081364991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 0x563da494b330 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f21cc011650 tx=0x7f21cc011960 comp rx=0 tx=0).stop 2024-05-27T09:43:53.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.690+0000 7f21ddc00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f21c00b2a70 0x7f21c00b4f80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009da0 tx=0x7f21d0009700 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:53.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.690+0000 7f21ee769b80 1 -- 172.21.15.44:0/1081364991 shutdown_connections 2024-05-27T09:43:53.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.690+0000 7f21ee769b80 1 --2- 172.21.15.44:0/1081364991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da494be70 0x563da4958110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.690+0000 7f21ee769b80 1 --2- 172.21.15.44:0/1081364991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 0x563da494b330 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.690+0000 7f21ee769b80 1 --2- 172.21.15.44:0/1081364991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da47cce40 0x563da494b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.690+0000 7f21ee769b80 1 -- 172.21.15.44:0/1081364991 >> 172.21.15.44:0/1081364991 conn(0x563da4866390 msgr2=0x563da49474c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:53.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.690+0000 7f21ee769b80 1 -- 172.21.15.44:0/1081364991 shutdown_connections 2024-05-27T09:43:53.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.690+0000 7f21ee769b80 1 -- 172.21.15.44:0/1081364991 wait complete. 2024-05-27T09:43:53.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.691+0000 7f21ee769b80 1 -- start start 2024-05-27T09:43:53.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.692+0000 7f21ee769b80 1 Processor -- start 2024-05-27T09:43:53.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.692+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da47cce40 0x563da49557b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.692+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 0x563da4955cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.692+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da494be70 0x563da4956230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.692+0000 7f21ee769b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563da476ffa0 con 0x563da494be70 2024-05-27T09:43:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.692+0000 7f21ee769b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563da480a760 con 0x563da47cce40 2024-05-27T09:43:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.692+0000 7f21ee769b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563da4958b80 con 0x563da48b89f0 2024-05-27T09:43:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.692+0000 7f21ee769b80 10 client.0 _ll_register_callbacks cb 0x563da4813c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.693+0000 7f21d7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 0x563da4955cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.693+0000 7f21d7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 0x563da4955cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49674/0 (socket says 172.21.15.44:49674) 2024-05-27T09:43:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.693+0000 7f21d7e00700 1 -- 172.21.15.44:0/2508622202 learned_addr learned my addr 172.21.15.44:0/2508622202 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:53.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.693+0000 7f21dd200700 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da47cce40 0x563da49557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.693+0000 7f21d7e00700 1 -- 172.21.15.44:0/2508622202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da47cce40 msgr2=0x563da49557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:53.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.693+0000 7f21d7e00700 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da47cce40 0x563da49557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.693+0000 7f21d7e00700 1 -- 172.21.15.44:0/2508622202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da494be70 msgr2=0x563da4956230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:53.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.693+0000 7f21d7e00700 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da494be70 0x563da4956230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.693+0000 7f21d7e00700 1 -- 172.21.15.44:0/2508622202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563da47b98d0 con 0x563da48b89f0 2024-05-27T09:43:53.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.694+0000 7f21d7e00700 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 0x563da4955cf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f21c400e9c0 tx=0x7f21c40a51d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62ceb54b56877646 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:53.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.694+0000 7f21bea00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:43:53.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.694+0000 7f21bea00700 1 -- 172.21.15.44:0/2508622202 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c40a9950 con 0x563da48b89f0 2024-05-27T09:43:53.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.694+0000 7f21ee769b80 1 -- 172.21.15.44:0/2508622202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21cc022440 con 0x563da48b89f0 2024-05-27T09:43:53.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.695+0000 7f21bea00700 1 -- 172.21.15.44:0/2508622202 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8477+0+0 (secure 0 0 0) 0x7f21c40b2320 con 0x563da48b89f0 2024-05-27T09:43:53.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.695+0000 7f21bea00700 1 -- 172.21.15.44:0/2508622202 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f21c40add00 con 0x563da48b89f0 2024-05-27T09:43:53.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.695+0000 7f21ee769b80 10 client.26842 fetch_fsmap learned FSMap version 31 2024-05-27T09:43:53.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.695+0000 7f21ee769b80 1 -- 172.21.15.44:0/2508622202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563da483a060 con 0x563da48b89f0 2024-05-27T09:43:53.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.696+0000 7f21bea00700 1 -- 172.21.15.44:0/2508622202 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21cc022440 con 0x563da48b89f0 2024-05-27T09:43:53.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.696+0000 7f21bea00700 1 -- 172.21.15.44:0/2508622202 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f21c40c7070 con 0x563da48b89f0 2024-05-27T09:43:53.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.696+0000 7f21ee769b80 10 client.26842 fetch_fsmap finished waiting for FSMap version 31 2024-05-27T09:43:53.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.696+0000 7f21ee769b80 10 client.26842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:53.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.696+0000 7f21ee769b80 20 client.26842 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:53.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.696+0000 7f21ee769b80 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563da4965ba0 0x563da4967fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.696+0000 7f21ee769b80 4 client.26842 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:43:53.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.696+0000 7f21ee769b80 1 -- 172.21.15.44:0/2508622202 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563da487bd90 con 0x563da4965ba0 2024-05-27T09:43:53.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.697+0000 7f21dd200700 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563da4965ba0 0x563da4967fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.698+0000 7f21dd200700 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563da4965ba0 0x563da4967fc0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce4a832e42add963 server_cookie=f2cd351d3182fc92 in_seq=0 out_seq=0 2024-05-27T09:43:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.698+0000 7f21bea00700 10 client.26842 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.699+0000 7f21bea00700 1 -- 172.21.15.44:0/2508622202 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563da487bd90 con 0x563da4965ba0 2024-05-27T09:43:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.699+0000 7f21bea00700 10 client.26842 handle_command_reply: tid=0 2024-05-27T09:43:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.699+0000 7f21ee769b80 1 client.26842 shutdown 2024-05-27T09:43:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.699+0000 7f21ee769b80 1 -- 172.21.15.44:0/2508622202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 msgr2=0x563da4955cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.699+0000 7f21ee769b80 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 0x563da4955cf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f21c400e9c0 tx=0x7f21c40a51d0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.699+0000 7f21ee769b80 1 -- 172.21.15.44:0/2508622202 shutdown_connections 2024-05-27T09:43:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.699+0000 7f21ee769b80 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563da4965ba0 0x563da4967fc0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.699+0000 7f21ee769b80 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da494be70 0x563da4956230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.699+0000 7f21ee769b80 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da47cce40 0x563da49557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.699+0000 7f21ee769b80 1 --2- 172.21.15.44:0/2508622202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 0x563da4955cf0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.699+0000 7f21ee769b80 1 -- 172.21.15.44:0/2508622202 >> 172.21.15.44:0/2508622202 conn(0x563da4866390 msgr2=0x563da494b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.700+0000 7f21bea00700 0 client.26842 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.701+0000 7f21ee769b80 1 -- 172.21.15.44:0/2508622202 shutdown_connections 2024-05-27T09:43:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.701+0000 7f21ee769b80 1 -- 172.21.15.44:0/2508622202 wait complete. 2024-05-27T09:43:53.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.701+0000 7f21ee769b80 20 client.26842 trim_cache size 0 max 16384 2024-05-27T09:43:53.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.729+0000 7f21ee769b80 1 Processor -- start 2024-05-27T09:43:53.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.729+0000 7f21ee769b80 1 -- start start 2024-05-27T09:43:53.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.729+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da47cce40 0x563da4947640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.729+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 0x563da49545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.729+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da494be70 0x563da4957890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.729+0000 7f21ee769b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f21d81af4e0 con 0x563da494be70 2024-05-27T09:43:53.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.729+0000 7f21ee769b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f21d81acd40 con 0x563da48b89f0 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.729+0000 7f21ee769b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f21d80512b0 con 0x563da47cce40 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.730+0000 7f21d7e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 0x563da49545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.730+0000 7f21dd200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da47cce40 0x563da4947640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.730+0000 7f21d7e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 0x563da49545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42352/0 (socket says 172.21.15.44:42352) 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.730+0000 7f21dd200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da47cce40 0x563da4947640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49688/0 (socket says 172.21.15.44:49688) 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.730+0000 7f21d7e00700 1 -- 172.21.15.44:0/3027808333 learned_addr learned my addr 172.21.15.44:0/3027808333 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.730+0000 7f21ddc00700 1 --2- 172.21.15.44:0/3027808333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da494be70 0x563da4957890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.730+0000 7f21dd200700 1 -- 172.21.15.44:0/3027808333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 msgr2=0x563da49545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.730+0000 7f21dd200700 1 --2- 172.21.15.44:0/3027808333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 0x563da49545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.730+0000 7f21dd200700 1 -- 172.21.15.44:0/3027808333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da494be70 msgr2=0x563da4957890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.730+0000 7f21dd200700 1 --2- 172.21.15.44:0/3027808333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da494be70 0x563da4957890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.730+0000 7f21dd200700 1 -- 172.21.15.44:0/3027808333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563da494b160 con 0x563da47cce40 2024-05-27T09:43:53.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.731+0000 7f21dd200700 1 --2- 172.21.15.44:0/3027808333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da47cce40 0x563da4947640 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f21cc000c00 tx=0x7f21cc022ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b24d469d957bc61c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:53.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.731+0000 7f21c9600700 1 -- 172.21.15.44:0/3027808333 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc027040 con 0x563da47cce40 2024-05-27T09:43:53.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.731+0000 7f21c9600700 1 -- 172.21.15.44:0/3027808333 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21cc028040 con 0x563da47cce40 2024-05-27T09:43:53.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.731+0000 7f21c9600700 1 -- 172.21.15.44:0/3027808333 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc005330 con 0x563da47cce40 2024-05-27T09:43:53.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.732+0000 7f21ee769b80 1 -- 172.21.15.44:0/3027808333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da47cce40 msgr2=0x563da4947640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:53.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.732+0000 7f21ee769b80 1 --2- 172.21.15.44:0/3027808333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da47cce40 0x563da4947640 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f21cc000c00 tx=0x7f21cc022ec0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.732+0000 7f21ee769b80 1 -- 172.21.15.44:0/3027808333 shutdown_connections 2024-05-27T09:43:53.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.732+0000 7f21ee769b80 1 --2- 172.21.15.44:0/3027808333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da494be70 0x563da4957890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.732+0000 7f21ee769b80 1 --2- 172.21.15.44:0/3027808333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da47cce40 0x563da4947640 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.732+0000 7f21ee769b80 1 --2- 172.21.15.44:0/3027808333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da48b89f0 0x563da49545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.732+0000 7f21ee769b80 1 -- 172.21.15.44:0/3027808333 >> 172.21.15.44:0/3027808333 conn(0x563da4866390 msgr2=0x563da494ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.733+0000 7f21ee769b80 1 -- 172.21.15.44:0/3027808333 shutdown_connections 2024-05-27T09:43:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.733+0000 7f21ee769b80 1 -- 172.21.15.44:0/3027808333 wait complete. 2024-05-27T09:43:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.733+0000 7f21ee769b80 1 -- start start 2024-05-27T09:43:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.733+0000 7f21ee769b80 1 Processor -- start 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.734+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da47cce40 0x563da495de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.734+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 0x563da495e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.734+0000 7f21ee769b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da494be70 0x563da4958e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.734+0000 7f21ee769b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f21d810b3d0 con 0x563da47cce40 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.734+0000 7f21ee769b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f21d81ac670 con 0x563da494be70 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.734+0000 7f21ee769b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f21d81ab8f0 con 0x563da48b89f0 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.734+0000 7f21ee769b80 10 client.0 _ll_register_callbacks cb 0x563da4813c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.735+0000 7f21ddc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da494be70 0x563da4958e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.735+0000 7f21d7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 0x563da495e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.735+0000 7f21ddc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da494be70 0x563da4958e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42364/0 (socket says 172.21.15.44:42364) 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.735+0000 7f21dd200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da47cce40 0x563da495de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.735+0000 7f21ddc00700 1 -- 172.21.15.44:0/3055433185 learned_addr learned my addr 172.21.15.44:0/3055433185 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:43:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.735+0000 7f21ddc00700 1 -- 172.21.15.44:0/3055433185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 msgr2=0x563da495e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.736+0000 7f21ddc00700 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 0x563da495e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.736+0000 7f21ddc00700 1 -- 172.21.15.44:0/3055433185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da47cce40 msgr2=0x563da495de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:43:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.736+0000 7f21ddc00700 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da47cce40 0x563da495de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.736+0000 7f21ddc00700 1 -- 172.21.15.44:0/3055433185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563da49565a0 con 0x563da494be70 2024-05-27T09:43:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.736+0000 7f21dd200700 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da47cce40 0x563da495de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:43:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.736+0000 7f21ddc00700 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da494be70 0x563da4958e10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f21d0007780 tx=0x7f21d00077b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc30a5a7eafa997 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:43:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.736+0000 7f21bea00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:43:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.737+0000 7f21bea00700 1 -- 172.21.15.44:0/3055433185 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d002c070 con 0x563da494be70 2024-05-27T09:43:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.737+0000 7f21ee769b80 1 -- 172.21.15.44:0/3055433185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21cc028040 con 0x563da494be70 2024-05-27T09:43:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.737+0000 7f21bea00700 1 -- 172.21.15.44:0/3055433185 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8477+0+0 (secure 0 0 0) 0x7f21d0031460 con 0x563da494be70 2024-05-27T09:43:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.737+0000 7f21bea00700 1 -- 172.21.15.44:0/3055433185 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f21d002c070 con 0x563da494be70 2024-05-27T09:43:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.737+0000 7f21ee769b80 10 client.26864 fetch_fsmap learned FSMap version 31 2024-05-27T09:43:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.737+0000 7f21ee769b80 1 -- 172.21.15.44:0/3055433185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563da4943240 con 0x563da494be70 2024-05-27T09:43:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.738+0000 7f21bea00700 1 -- 172.21.15.44:0/3055433185 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21cc028040 con 0x563da494be70 2024-05-27T09:43:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.738+0000 7f21bea00700 1 -- 172.21.15.44:0/3055433185 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f21d0011070 con 0x563da494be70 2024-05-27T09:43:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.738+0000 7f21ee769b80 10 client.26864 fetch_fsmap finished waiting for FSMap version 31 2024-05-27T09:43:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.738+0000 7f21ee769b80 10 client.26864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:43:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.738+0000 7f21ee769b80 20 client.26864 populate_metadata read hostname 'smithi044' 2024-05-27T09:43:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.738+0000 7f21ee769b80 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563da49d1070 0x563da49d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:43:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.738+0000 7f21ee769b80 4 client.26864 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:43:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.738+0000 7f21ee769b80 1 -- 172.21.15.44:0/3055433185 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563da4946120 con 0x563da49d1070 2024-05-27T09:43:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.739+0000 7f21dd200700 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563da49d1070 0x563da49d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:43:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.739+0000 7f21dd200700 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563da49d1070 0x563da49d3490 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6321b11de560f8f4 server_cookie=12c5095a40be5891 in_seq=0 out_seq=0 2024-05-27T09:43:53.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.739+0000 7f21bea00700 10 client.26864 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21bea00700 1 -- 172.21.15.44:0/3055433185 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563da4946120 con 0x563da49d1070 2024-05-27T09:43:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21bea00700 10 client.26864 handle_command_reply: tid=0 2024-05-27T09:43:53.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21ee769b80 1 client.26864 shutdown 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21ee769b80 1 -- 172.21.15.44:0/3055433185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da494be70 msgr2=0x563da4958e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21ee769b80 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da494be70 0x563da4958e10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f21d0007780 tx=0x7f21d00077b0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21ee769b80 1 -- 172.21.15.44:0/3055433185 shutdown_connections 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21ee769b80 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563da49d1070 0x563da49d3490 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21ee769b80 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da494be70 0x563da4958e10 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21ee769b80 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563da47cce40 0x563da495de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21ee769b80 1 --2- 172.21.15.44:0/3055433185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563da48b89f0 0x563da495e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21ee769b80 1 -- 172.21.15.44:0/3055433185 >> 172.21.15.44:0/3055433185 conn(0x563da4866390 msgr2=0x563da4955d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21bea00700 0 client.26864 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21ee769b80 1 -- 172.21.15.44:0/3055433185 shutdown_connections 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.740+0000 7f21ee769b80 1 -- 172.21.15.44:0/3055433185 wait complete. 2024-05-27T09:43:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.741+0000 7f21ee769b80 20 client.26864 trim_cache size 0 max 16384 2024-05-27T09:43:53.743 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:43:53.743 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:43:53.743 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:43:53.743 INFO:teuthology.orchestra.run.smithi044.stdout: "23fa7c2c-265d-4199-a6d4-f39b4f4a28a7": { 2024-05-27T09:43:53.743 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:43:53.743 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "23fa7c2c-265d-4199-a6d4-f39b4f4a28a7", 2024-05-27T09:43:53.743 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:43:53.743 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:43:53.743 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:43:53.744 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:43:53.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.741+0000 7f21dfc00700 1 -- 172.21.15.44:0/1322166730 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f21c00b2a70 msgr2=0x7f21c00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:53.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.741+0000 7f21dfc00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f21c00b2a70 0x7f21c00b4f80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009da0 tx=0x7f21d0009700 comp rx=0 tx=0).stop 2024-05-27T09:43:53.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.741+0000 7f21dfc00700 1 -- 172.21.15.44:0/1322166730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d8105e00 msgr2=0x7f21d819d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:43:53.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.741+0000 7f21dfc00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d8105e00 0x7f21d819d500 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f21c400e9f0 tx=0x7f21c400ed00 comp rx=0 tx=0).stop 2024-05-27T09:43:53.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.742+0000 7f21dfc00700 1 -- 172.21.15.44:0/1322166730 shutdown_connections 2024-05-27T09:43:53.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.742+0000 7f21dfc00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f21c00b2a70 0x7f21c00b4f80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.742+0000 7f21dfc00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21d810e8e0 0x7f21d81a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.742+0000 7f21dfc00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21d81054b0 0x7f21d819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.742+0000 7f21dfc00700 1 --2- 172.21.15.44:0/1322166730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d8105e00 0x7f21d819d500 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:43:53.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.742+0000 7f21dfc00700 1 -- 172.21.15.44:0/1322166730 >> 172.21.15.44:0/1322166730 conn(0x7f21d80fedb0 msgr2=0x7f21d80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:43:53.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.742+0000 7f21dfc00700 1 -- 172.21.15.44:0/1322166730 shutdown_connections 2024-05-27T09:43:53.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:43:53.742+0000 7f21dfc00700 1 -- 172.21.15.44:0/1322166730 wait complete. 2024-05-27T09:43:54.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:54 smithi189 ceph-mon[36671]: pgmap v268: 97 pgs: 97 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 53 op/s 2024-05-27T09:43:54.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:54 smithi044 ceph-mon[31618]: pgmap v268: 97 pgs: 97 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 53 op/s 2024-05-27T09:43:54.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:54 smithi050 ceph-mon[36371]: pgmap v268: 97 pgs: 97 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 53 op/s 2024-05-27T09:43:54.582 INFO:tasks.cephfs.filesystem:scrub status for tag:23fa7c2c-265d-4199-a6d4-f39b4f4a28a7 - {'path': '/', 'tag': '23fa7c2c-265d-4199-a6d4-f39b4f4a28a7', 'options': 'recursive,force'} 2024-05-27T09:43:56.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:56 smithi189 ceph-mon[36671]: pgmap v269: 97 pgs: 97 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 53 op/s 2024-05-27T09:43:56.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:56 smithi044 ceph-mon[31618]: pgmap v269: 97 pgs: 97 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 53 op/s 2024-05-27T09:43:56.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:56 smithi050 ceph-mon[36371]: pgmap v269: 97 pgs: 97 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 53 op/s 2024-05-27T09:43:57.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:57 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:43:57.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:57 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:43:57.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:57 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:43:58.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:58 smithi189 ceph-mon[36671]: pgmap v270: 97 pgs: 97 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-05-27T09:43:58.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:58 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:43:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:58 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:43:58.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:58 smithi044 ceph-mon[31618]: pgmap v270: 97 pgs: 97 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-05-27T09:43:58.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:58 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:43:58.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:58 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:43:58.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:58 smithi050 ceph-mon[36371]: pgmap v270: 97 pgs: 97 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-05-27T09:43:58.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:58 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:43:58.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:58 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:43:59.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:59 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:43:59.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:59 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:43:59.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:43:59 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:43:59.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:59 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:43:59.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:59 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:43:59.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:43:59 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:43:59.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:59 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:43:59.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:59 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:43:59.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:43:59 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:44:00.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:00 smithi189 ceph-mon[36671]: pgmap v271: 97 pgs: 97 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-05-27T09:44:00.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:00 smithi044 ceph-mon[31618]: pgmap v271: 97 pgs: 97 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-05-27T09:44:00.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:00 smithi050 ceph-mon[36371]: pgmap v271: 97 pgs: 97 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-05-27T09:44:02.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:02 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:44:02.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:02 smithi044 ceph-mon[31618]: pgmap v272: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 240 op/s 2024-05-27T09:44:02.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:02 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:44:02.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:02 smithi050 ceph-mon[36371]: pgmap v272: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 240 op/s 2024-05-27T09:44:02.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:02 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:44:02.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:02 smithi189 ceph-mon[36671]: pgmap v272: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 240 op/s 2024-05-27T09:44:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.236+0000 7fc0e7000640 1 -- 172.21.15.44:0/2289015770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e0102b70 msgr2=0x7fc0e0102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.236+0000 7fc0e7000640 1 --2- 172.21.15.44:0/2289015770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e0102b70 0x7fc0e0102f80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8009900 tx=0x7fc0d802a3f0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.236+0000 7fc0e7000640 1 -- 172.21.15.44:0/2289015770 shutdown_connections 2024-05-27T09:44:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.236+0000 7fc0e7000640 1 --2- 172.21.15.44:0/2289015770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0e0107ff0 0x7fc0e0108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.236+0000 7fc0e7000640 1 --2- 172.21.15.44:0/2289015770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e01075c0 0x7fc0e01079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.236+0000 7fc0e7000640 1 --2- 172.21.15.44:0/2289015770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e0102b70 0x7fc0e0102f80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.236+0000 7fc0e7000640 1 -- 172.21.15.44:0/2289015770 >> 172.21.15.44:0/2289015770 conn(0x7fc0e00fe380 msgr2=0x7fc0e01007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.236+0000 7fc0e7000640 1 -- 172.21.15.44:0/2289015770 shutdown_connections 2024-05-27T09:44:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.236+0000 7fc0e7000640 1 -- 172.21.15.44:0/2289015770 wait complete. 2024-05-27T09:44:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.238+0000 7fc0e7000640 1 Processor -- start 2024-05-27T09:44:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.238+0000 7fc0e7000640 1 -- start start 2024-05-27T09:44:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.239+0000 7fc0e7000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e0102b70 0x7fc0e01a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.239+0000 7fc0e7000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0e01075c0 0x7fc0e01a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.239+0000 7fc0e7000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e0107ff0 0x7fc0e01a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.239+0000 7fc0e7000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e011a3f0 con 0x7fc0e01075c0 2024-05-27T09:44:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.239+0000 7fc0e7000640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e011a270 con 0x7fc0e0102b70 2024-05-27T09:44:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.239+0000 7fc0e7000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e011a570 con 0x7fc0e0107ff0 2024-05-27T09:44:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.239+0000 7fc0dfe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e0102b70 0x7fc0e01a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.239+0000 7fc0df400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0e01075c0 0x7fc0e01a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.239+0000 7fc0e5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e0107ff0 0x7fc0e01a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.239+0000 7fc0dfe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e0102b70 0x7fc0e01a0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38912/0 (socket says 172.21.15.44:38912) 2024-05-27T09:44:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.239+0000 7fc0dfe00640 1 -- 172.21.15.44:0/4008772452 learned_addr learned my addr 172.21.15.44:0/4008772452 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0e5000640 1 -- 172.21.15.44:0/4008772452 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e0102b70 msgr2=0x7fc0e01a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0e5000640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e0102b70 0x7fc0e01a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0e5000640 1 -- 172.21.15.44:0/4008772452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0e01075c0 msgr2=0x7fc0e01a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0e5000640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0e01075c0 0x7fc0e01a1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0e5000640 1 -- 172.21.15.44:0/4008772452 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e01a5d10 con 0x7fc0e0107ff0 2024-05-27T09:44:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e0102b70 0x7fc0e01a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0e5000640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e0107ff0 0x7fc0e01a56d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009fd0 tx=0x7fc0d000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0dcc00640 1 -- 172.21.15.44:0/4008772452 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d000ce20 con 0x7fc0e0107ff0 2024-05-27T09:44:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0dcc00640 1 -- 172.21.15.44:0/4008772452 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0d0005020 con 0x7fc0e0107ff0 2024-05-27T09:44:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0dcc00640 1 -- 172.21.15.44:0/4008772452 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0010690 con 0x7fc0e0107ff0 2024-05-27T09:44:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0e7000640 1 -- 172.21.15.44:0/4008772452 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e01a6000 con 0x7fc0e0107ff0 2024-05-27T09:44:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.240+0000 7fc0e7000640 1 -- 172.21.15.44:0/4008772452 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e01ad980 con 0x7fc0e0107ff0 2024-05-27T09:44:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.244+0000 7fc0ee181740 1 Processor -- start 2024-05-27T09:44:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.245+0000 7fc0dcc00640 1 -- 172.21.15.44:0/4008772452 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc0d00054d0 con 0x7fc0e0107ff0 2024-05-27T09:44:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.245+0000 7fc0ee181740 1 -- start start 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.245+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad5e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.245+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9932f0 0x55a8bad5eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.245+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad5f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.245+0000 7fc0ee181740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a8baccfb90 con 0x55a8ba99b4d0 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.245+0000 7fc0ee181740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a8bacbf730 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.245+0000 7fc0ee181740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a8bacc3770 con 0x55a8ba9932f0 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.246+0000 7fc0df400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad5e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.246+0000 7fc0df400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad5e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38914/0 (socket says 172.21.15.44:38914) 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.246+0000 7fc0df400640 1 -- 172.21.15.44:0/4165493219 learned_addr learned my addr 172.21.15.44:0/4165493219 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.246+0000 7fc0dfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9932f0 0x55a8bad5eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.246+0000 7fc0df400640 1 --2- 172.21.15.44:0/4165493219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad5f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.246+0000 7fc0dfe00640 1 -- 172.21.15.44:0/4165493219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9bc1b0 msgr2=0x55a8bad5e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.246+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/4165493219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad5e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.246+0000 7fc0dfe00640 1 -- 172.21.15.44:0/4165493219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 msgr2=0x55a8bad5f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.246+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/4165493219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad5f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.246+0000 7fc0dfe00640 1 -- 172.21.15.44:0/4165493219 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8bac588b0 con 0x55a8ba9932f0 2024-05-27T09:44:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.246+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/4165493219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9932f0 0x55a8bad5eda0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8004770 tx=0x7fc0d8005ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caa86932c286a076 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.247+0000 7fc0dcc00640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc0c40b29d0 0x7fc0c40b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.247+0000 7fc0d5600640 1 -- 172.21.15.44:0/4165493219 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8035040 con 0x55a8ba9932f0 2024-05-27T09:44:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.247+0000 7fc0d5600640 1 -- 172.21.15.44:0/4165493219 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0d8031820 con 0x55a8ba9932f0 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.247+0000 7fc0d5600640 1 -- 172.21.15.44:0/4165493219 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d80049e0 con 0x55a8ba9932f0 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.247+0000 7fc0dcc00640 1 -- 172.21.15.44:0/4008772452 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8477+0+0 (secure 0 0 0) 0x7fc0d00e2c80 con 0x7fc0e0107ff0 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.247+0000 7fc0df400640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc0c40b29d0 0x7fc0c40b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.248+0000 7fc0ee181740 1 -- 172.21.15.44:0/4165493219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9932f0 msgr2=0x55a8bad5eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.248+0000 7fc0ee181740 1 --2- 172.21.15.44:0/4165493219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9932f0 0x55a8bad5eda0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8004770 tx=0x7fc0d8005ed0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.248+0000 7fc0df400640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc0c40b29d0 0x7fc0c40b4ee0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55a8bad5b9d0 tx=0x7fc0c8005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.248+0000 7fc0ee181740 1 -- 172.21.15.44:0/4165493219 shutdown_connections 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.248+0000 7fc0ee181740 1 --2- 172.21.15.44:0/4165493219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad5f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.248+0000 7fc0ee181740 1 --2- 172.21.15.44:0/4165493219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9932f0 0x55a8bad5eda0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.248+0000 7fc0ee181740 1 --2- 172.21.15.44:0/4165493219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad5e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.248+0000 7fc0ee181740 1 -- 172.21.15.44:0/4165493219 >> 172.21.15.44:0/4165493219 conn(0x55a8ba9b0300 msgr2=0x55a8bad5b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.248+0000 7fc0ee181740 1 -- 172.21.15.44:0/4165493219 shutdown_connections 2024-05-27T09:44:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.248+0000 7fc0ee181740 1 -- 172.21.15.44:0/4165493219 wait complete. 2024-05-27T09:44:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.249+0000 7fc0ee181740 1 -- start start 2024-05-27T09:44:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.249+0000 7fc0ee181740 1 Processor -- start 2024-05-27T09:44:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.249+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad68a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.249+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad68f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.249+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad74c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.249+0000 7fc0ee181740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a8ba9a5790 con 0x55a8ba99b4d0 2024-05-27T09:44:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.249+0000 7fc0ee181740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a8bac67bc0 con 0x55a8ba9932f0 2024-05-27T09:44:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.249+0000 7fc0ee181740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a8bad6b930 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0ee181740 10 client.0 _ll_register_callbacks cb 0x55a8bac45400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0df400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad68f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0dfe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad68a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0e5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad74c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0e5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad74c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47860/0 (socket says 172.21.15.44:47860) 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0df400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad68f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52716/0 (socket says 172.21.15.44:52716) 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0df400640 1 -- 172.21.15.44:0/3863062221 learned_addr learned my addr 172.21.15.44:0/3863062221 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0dfe00640 1 -- 172.21.15.44:0/3863062221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 msgr2=0x55a8bad74c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad74c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0dfe00640 1 -- 172.21.15.44:0/3863062221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 msgr2=0x55a8bad68f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad68f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0dfe00640 1 -- 172.21.15.44:0/3863062221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8ba9d4210 con 0x55a8ba9932f0 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.250+0000 7fc0e5000640 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad74c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.251+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad68a20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x55a8bad5fc10 tx=0x7fc0d80319d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c6c419f5ce4725 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.251+0000 7fc0ce000640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.251+0000 7fc0ce000640 1 -- 172.21.15.44:0/3863062221 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8035040 con 0x55a8ba9932f0 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.251+0000 7fc0ee181740 1 -- 172.21.15.44:0/3863062221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0d8031820 con 0x55a8ba9932f0 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.251+0000 7fc0ce000640 1 -- 172.21.15.44:0/3863062221 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8477+0+0 (secure 0 0 0) 0x7fc0d80035f0 con 0x55a8ba9932f0 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.251+0000 7fc0ce000640 1 -- 172.21.15.44:0/3863062221 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc0d8058050 con 0x55a8ba9932f0 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.251+0000 7fc0ee181740 10 client.26876 fetch_fsmap learned FSMap version 31 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.251+0000 7fc0ee181740 1 -- 172.21.15.44:0/3863062221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a8baaa0620 con 0x55a8ba9932f0 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.252+0000 7fc0ce000640 1 -- 172.21.15.44:0/3863062221 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0d8031820 con 0x55a8ba9932f0 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.252+0000 7fc0ce000640 1 -- 172.21.15.44:0/3863062221 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7fc0d80466f0 con 0x55a8ba9932f0 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.252+0000 7fc0ee181740 10 client.26876 fetch_fsmap finished waiting for FSMap version 31 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.252+0000 7fc0ee181740 10 client.26876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.252+0000 7fc0ee181740 20 client.26876 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.252+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a8ba97ad80 0x55a8bad7aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.252+0000 7fc0ee181740 4 client.26876 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.252+0000 7fc0ee181740 1 -- 172.21.15.44:0/3863062221 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8bacd1ae0 con 0x55a8ba97ad80 2024-05-27T09:44:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.252+0000 7fc0df400640 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a8ba97ad80 0x55a8bad7aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.253+0000 7fc0df400640 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a8ba97ad80 0x55a8bad7aa50 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=176ab92fe19d718c server_cookie=6989c83003462260 in_seq=0 out_seq=0 2024-05-27T09:44:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.253+0000 7fc0ce000640 10 client.26876 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ce000640 1 -- 172.21.15.44:0/3863062221 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a8bacd1ae0 con 0x55a8ba97ad80 2024-05-27T09:44:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ce000640 10 client.26876 handle_command_reply: tid=0 2024-05-27T09:44:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 1 client.26876 shutdown 2024-05-27T09:44:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 1 -- 172.21.15.44:0/3863062221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 msgr2=0x55a8bad68a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad68a20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x55a8bad5fc10 tx=0x7fc0d80319d0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 1 -- 172.21.15.44:0/3863062221 shutdown_connections 2024-05-27T09:44:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a8ba97ad80 0x55a8bad7aa50 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad74c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad68f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3863062221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad68a20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ce000640 0 client.26876 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 1 -- 172.21.15.44:0/3863062221 >> 172.21.15.44:0/3863062221 conn(0x55a8ba9b0300 msgr2=0x55a8bad5e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 1 -- 172.21.15.44:0/3863062221 shutdown_connections 2024-05-27T09:44:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 1 -- 172.21.15.44:0/3863062221 wait complete. 2024-05-27T09:44:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.254+0000 7fc0ee181740 20 client.26876 trim_cache size 0 max 16384 2024-05-27T09:44:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.272+0000 7fc0ee181740 1 Processor -- start 2024-05-27T09:44:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.272+0000 7fc0ee181740 1 -- start start 2024-05-27T09:44:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.273+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad6a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.273+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad749d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.273+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad74f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.273+0000 7fc0ee181740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b8880 con 0x55a8ba99b4d0 2024-05-27T09:44:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.273+0000 7fc0ee181740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b6120 con 0x55a8ba9932f0 2024-05-27T09:44:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.273+0000 7fc0ee181740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b3f70 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.273+0000 7fc0dfe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad6a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.273+0000 7fc0dfe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad6a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:38932/0 (socket says 172.21.15.44:38932) 2024-05-27T09:44:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.273+0000 7fc0df400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.273+0000 7fc0dfe00640 1 -- 172.21.15.44:0/3031059197 learned_addr learned my addr 172.21.15.44:0/3031059197 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.273+0000 7fc0e5000640 1 --2- 172.21.15.44:0/3031059197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad74f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.274+0000 7fc0dfe00640 1 -- 172.21.15.44:0/3031059197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 msgr2=0x55a8bad74f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.274+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/3031059197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad74f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.274+0000 7fc0dfe00640 1 -- 172.21.15.44:0/3031059197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 msgr2=0x55a8bad749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.274+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/3031059197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.274+0000 7fc0dfe00640 1 -- 172.21.15.44:0/3031059197 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8bad5fc40 con 0x55a8ba9932f0 2024-05-27T09:44:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.274+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/3031059197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad6a850 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8000c00 tx=0x7fc0d802a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=969aab30ae51f531 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.274+0000 7fc0d4c00640 1 -- 172.21.15.44:0/3031059197 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8061070 con 0x55a8ba9932f0 2024-05-27T09:44:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.274+0000 7fc0d4c00640 1 -- 172.21.15.44:0/3031059197 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0d8031820 con 0x55a8ba9932f0 2024-05-27T09:44:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.274+0000 7fc0d4c00640 1 -- 172.21.15.44:0/3031059197 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8035040 con 0x55a8ba9932f0 2024-05-27T09:44:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.275+0000 7fc0ee181740 1 -- 172.21.15.44:0/3031059197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 msgr2=0x55a8bad6a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.275+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3031059197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad6a850 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8000c00 tx=0x7fc0d802a3f0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.276+0000 7fc0ee181740 1 -- 172.21.15.44:0/3031059197 shutdown_connections 2024-05-27T09:44:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.276+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3031059197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad74f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.276+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3031059197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.276+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3031059197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad6a850 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.276+0000 7fc0ee181740 1 -- 172.21.15.44:0/3031059197 >> 172.21.15.44:0/3031059197 conn(0x55a8ba9b0300 msgr2=0x55a8bad6a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.276+0000 7fc0ee181740 1 -- 172.21.15.44:0/3031059197 shutdown_connections 2024-05-27T09:44:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.276+0000 7fc0ee181740 1 -- 172.21.15.44:0/3031059197 wait complete. 2024-05-27T09:44:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.276+0000 7fc0ee181740 1 -- start start 2024-05-27T09:44:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.276+0000 7fc0ee181740 1 Processor -- start 2024-05-27T09:44:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad6dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad6e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0ee181740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad6e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0ee181740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b45e0 con 0x55a8ba99b4d0 2024-05-27T09:44:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0ee181740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b5a50 con 0x55a8ba9932f0 2024-05-27T09:44:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0ee181740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b4cb0 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0ee181740 10 client.0 _ll_register_callbacks cb 0x7fc0d8040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0e5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad6e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0e5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad6e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47886/0 (socket says 172.21.15.44:47886) 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0e5000640 1 -- 172.21.15.44:0/3881198441 learned_addr learned my addr 172.21.15.44:0/3881198441 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad6dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.277+0000 7fc0df400640 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad6e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0e5000640 1 -- 172.21.15.44:0/3881198441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 msgr2=0x55a8bad6dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0e5000640 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad6dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0e5000640 1 -- 172.21.15.44:0/3881198441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 msgr2=0x55a8bad6e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0e5000640 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad6e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0e5000640 1 -- 172.21.15.44:0/3881198441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8bade1140 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0df400640 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad6e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0e5000640 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad6e6c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc0d00a9c30 tx=0x7fc0d00aa6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c58347da45baae1a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0ce000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0ce000640 1 -- 172.21.15.44:0/3881198441 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d00c8070 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0ce000640 1 -- 172.21.15.44:0/3881198441 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8477+0+0 (secure 0 0 0) 0x7fc0d00b2b10 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0ee181740 1 -- 172.21.15.44:0/3881198441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0d8031820 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0ce000640 1 -- 172.21.15.44:0/3881198441 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc0d008b050 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0ee181740 10 client.26866 fetch_fsmap learned FSMap version 31 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.278+0000 7fc0ee181740 1 -- 172.21.15.44:0/3881198441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a8bade2430 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.279+0000 7fc0ce000640 1 -- 172.21.15.44:0/3881198441 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0d8031820 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.279+0000 7fc0ce000640 1 -- 172.21.15.44:0/3881198441 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7fc0d00a5a60 con 0x55a8ba9bc1b0 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.279+0000 7fc0ee181740 10 client.26866 fetch_fsmap finished waiting for FSMap version 31 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.279+0000 7fc0ee181740 10 client.26866 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.279+0000 7fc0ee181740 20 client.26866 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.279+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a8ba9501c0 0x55a8bade4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.279+0000 7fc0ee181740 4 client.26866 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:44:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.279+0000 7fc0ee181740 1 -- 172.21.15.44:0/3881198441 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a8baccb2c0 con 0x55a8ba9501c0 2024-05-27T09:44:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.279+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a8ba9501c0 0x55a8bade4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.280+0000 7fc0dfe00640 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a8ba9501c0 0x55a8bade4c20 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40d55c206bd24a59 server_cookie=a424e870559f35ef in_seq=0 out_seq=0 2024-05-27T09:44:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.280+0000 7fc0ce000640 10 client.26866 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.403+0000 7fc0ce000640 1 -- 172.21.15.44:0/3881198441 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a8baccb2c0 con 0x55a8ba9501c0 2024-05-27T09:44:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.403+0000 7fc0ce000640 10 client.26866 handle_command_reply: tid=0 2024-05-27T09:44:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 1 client.26866 shutdown 2024-05-27T09:44:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 1 -- 172.21.15.44:0/3881198441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 msgr2=0x55a8bad6e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad6e6c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc0d00a9c30 tx=0x7fc0d00aa6f0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 1 -- 172.21.15.44:0/3881198441 shutdown_connections 2024-05-27T09:44:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a8ba9501c0 0x55a8bade4c20 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a8ba9bc1b0 0x55a8bad6e6c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8ba99b4d0 0x55a8bad6e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 1 --2- 172.21.15.44:0/3881198441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a8ba9932f0 0x55a8bad6dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ce000640 0 client.26866 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 1 -- 172.21.15.44:0/3881198441 >> 172.21.15.44:0/3881198441 conn(0x55a8ba9b0300 msgr2=0x55a8bad68190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 1 -- 172.21.15.44:0/3881198441 shutdown_connections 2024-05-27T09:44:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 1 -- 172.21.15.44:0/3881198441 wait complete. 2024-05-27T09:44:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.404+0000 7fc0ee181740 20 client.26866 trim_cache size 0 max 16384 2024-05-27T09:44:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:44:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:44:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:44:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.405+0000 7fc0e7000640 1 -- 172.21.15.44:0/4008772452 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc0c40b29d0 msgr2=0x7fc0c40b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.405+0000 7fc0e7000640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc0c40b29d0 0x7fc0c40b4ee0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55a8bad5b9d0 tx=0x7fc0c8005f90 comp rx=0 tx=0).stop 2024-05-27T09:44:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.405+0000 7fc0e7000640 1 -- 172.21.15.44:0/4008772452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e0107ff0 msgr2=0x7fc0e01a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.405+0000 7fc0e7000640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e0107ff0 0x7fc0e01a56d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009fd0 tx=0x7fc0d000ef10 comp rx=0 tx=0).stop 2024-05-27T09:44:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.405+0000 7fc0e7000640 1 -- 172.21.15.44:0/4008772452 shutdown_connections 2024-05-27T09:44:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.405+0000 7fc0e7000640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e0107ff0 0x7fc0e01a56d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.405+0000 7fc0e7000640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc0c40b29d0 0x7fc0c40b4ee0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.405+0000 7fc0e7000640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0e01075c0 0x7fc0e01a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.405+0000 7fc0e7000640 1 --2- 172.21.15.44:0/4008772452 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e0102b70 0x7fc0e01a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.405+0000 7fc0e7000640 1 -- 172.21.15.44:0/4008772452 >> 172.21.15.44:0/4008772452 conn(0x7fc0e00fe380 msgr2=0x7fc0e0105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.405+0000 7fc0e7000640 1 -- 172.21.15.44:0/4008772452 shutdown_connections 2024-05-27T09:44:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:04.406+0000 7fc0e7000640 1 -- 172.21.15.44:0/4008772452 wait complete. 2024-05-27T09:44:04.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:04 smithi044 ceph-mon[31618]: pgmap v273: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 225 op/s 2024-05-27T09:44:04.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:04 smithi050 ceph-mon[36371]: pgmap v273: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 225 op/s 2024-05-27T09:44:04.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:04 smithi189 ceph-mon[36671]: pgmap v273: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 225 op/s 2024-05-27T09:44:06.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:06 smithi044 ceph-mon[31618]: pgmap v274: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 200 op/s 2024-05-27T09:44:06.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:06 smithi050 ceph-mon[36371]: pgmap v274: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 200 op/s 2024-05-27T09:44:06.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:06 smithi189 ceph-mon[36671]: pgmap v274: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 200 op/s 2024-05-27T09:44:07.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:07 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:44:07.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:07 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:44:07.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:07 smithi044 ceph-mon[31618]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T09:44:07.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:07 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:44:07.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:07 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:44:07.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:07 smithi050 ceph-mon[36371]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T09:44:07.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:07 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:44:07.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:07 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:44:07.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:07 smithi189 ceph-mon[36671]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T09:44:08.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:08 smithi044 ceph-mon[31618]: pgmap v276: 97 pgs: 97 active+clean; 235 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 218 op/s 2024-05-27T09:44:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:08 smithi050 ceph-mon[36371]: pgmap v276: 97 pgs: 97 active+clean; 235 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 218 op/s 2024-05-27T09:44:08.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:08 smithi189 ceph-mon[36671]: pgmap v276: 97 pgs: 97 active+clean; 235 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 218 op/s 2024-05-27T09:44:09.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:09 smithi044 ceph-mon[31618]: pgmap v277: 97 pgs: 97 active+clean; 235 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 218 op/s 2024-05-27T09:44:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:09 smithi050 ceph-mon[36371]: pgmap v277: 97 pgs: 97 active+clean; 235 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 218 op/s 2024-05-27T09:44:09.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:09 smithi189 ceph-mon[36671]: pgmap v277: 97 pgs: 97 active+clean; 235 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 218 op/s 2024-05-27T09:44:12.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:11 smithi050 ceph-mon[36371]: pgmap v278: 97 pgs: 97 active+clean; 256 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.9 MiB/s wr, 1.14k op/s 2024-05-27T09:44:12.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:11 smithi189 ceph-mon[36671]: pgmap v278: 97 pgs: 97 active+clean; 256 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.9 MiB/s wr, 1.14k op/s 2024-05-27T09:44:12.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:11 smithi044 ceph-mon[31618]: pgmap v278: 97 pgs: 97 active+clean; 256 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.9 MiB/s wr, 1.14k op/s 2024-05-27T09:44:13.044 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:12 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:44:13.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:12 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:44:13.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:12 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:44:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:13 smithi044 ceph-mon[31618]: pgmap v279: 97 pgs: 97 active+clean; 256 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.9 MiB/s wr, 1.14k op/s 2024-05-27T09:44:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:13 smithi044 ceph-mon[31618]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T09:44:14.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:13 smithi050 ceph-mon[36371]: pgmap v279: 97 pgs: 97 active+clean; 256 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.9 MiB/s wr, 1.14k op/s 2024-05-27T09:44:14.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:13 smithi050 ceph-mon[36371]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T09:44:14.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:13 smithi189 ceph-mon[36671]: pgmap v279: 97 pgs: 97 active+clean; 256 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.9 MiB/s wr, 1.14k op/s 2024-05-27T09:44:14.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:13 smithi189 ceph-mon[36671]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T09:44:16.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:15 smithi044 ceph-mon[31618]: pgmap v281: 97 pgs: 97 active+clean; 256 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 1.14k op/s 2024-05-27T09:44:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:15 smithi050 ceph-mon[36371]: pgmap v281: 97 pgs: 97 active+clean; 256 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 1.14k op/s 2024-05-27T09:44:16.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:15 smithi189 ceph-mon[36671]: pgmap v281: 97 pgs: 97 active+clean; 256 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 1.14k op/s 2024-05-27T09:44:17.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.637+0000 7fb35a000640 1 -- 172.21.15.44:0/1300036101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb354109cb0 msgr2=0x7fb354103850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:17.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.637+0000 7fb35a000640 1 --2- 172.21.15.44:0/1300036101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb354109cb0 0x7fb354103850 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb344009900 tx=0x7fb34402a410 comp rx=0 tx=0).stop 2024-05-27T09:44:17.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.637+0000 7fb35a000640 1 -- 172.21.15.44:0/1300036101 shutdown_connections 2024-05-27T09:44:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.637+0000 7fb35a000640 1 --2- 172.21.15.44:0/1300036101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb354104400 0x7fb3541107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.637+0000 7fb35a000640 1 --2- 172.21.15.44:0/1300036101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb35410a150 0x7fb354103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.637+0000 7fb35a000640 1 --2- 172.21.15.44:0/1300036101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb354109cb0 0x7fb354103850 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.637+0000 7fb35a000640 1 -- 172.21.15.44:0/1300036101 >> 172.21.15.44:0/1300036101 conn(0x7fb3540ff480 msgr2=0x7fb3541018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.637+0000 7fb35a000640 1 -- 172.21.15.44:0/1300036101 shutdown_connections 2024-05-27T09:44:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.638+0000 7fb35a000640 1 -- 172.21.15.44:0/1300036101 wait complete. 2024-05-27T09:44:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.639+0000 7fb35a000640 1 Processor -- start 2024-05-27T09:44:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.639+0000 7fb35a000640 1 -- start start 2024-05-27T09:44:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.640+0000 7fb35a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb354104400 0x7fb354074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.640+0000 7fb35a000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb354109cb0 0x7fb354075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.640+0000 7fb35a000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35410a150 0x7fb354075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.640+0000 7fb35a000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb354113ea0 con 0x7fb35410a150 2024-05-27T09:44:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.640+0000 7fb35a000640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb354113d20 con 0x7fb354109cb0 2024-05-27T09:44:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.640+0000 7fb35a000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb354114020 con 0x7fb354104400 2024-05-27T09:44:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.640+0000 7fb353400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb354104400 0x7fb354074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.640+0000 7fb353400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb354104400 0x7fb354074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:52872/0 (socket says 172.21.15.44:52872) 2024-05-27T09:44:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.640+0000 7fb353400640 1 -- 172.21.15.44:0/918366721 learned_addr learned my addr 172.21.15.44:0/918366721 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.640+0000 7fb352a00640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb354109cb0 0x7fb354075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.640+0000 7fb353e00640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35410a150 0x7fb354075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.641+0000 7fb353400640 1 -- 172.21.15.44:0/918366721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb354109cb0 msgr2=0x7fb354075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.641+0000 7fb353400640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb354109cb0 0x7fb354075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.641+0000 7fb353400640 1 -- 172.21.15.44:0/918366721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35410a150 msgr2=0x7fb354075a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.641+0000 7fb353400640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35410a150 0x7fb354075a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.641+0000 7fb353400640 1 -- 172.21.15.44:0/918366721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3541a8c40 con 0x7fb354104400 2024-05-27T09:44:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.641+0000 7fb353400640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb354104400 0x7fb354074fc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb344032fd0 tx=0x7fb344031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.641+0000 7fb34be00640 1 -- 172.21.15.44:0/918366721 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34403f030 con 0x7fb354104400 2024-05-27T09:44:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.641+0000 7fb34be00640 1 -- 172.21.15.44:0/918366721 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb344004490 con 0x7fb354104400 2024-05-27T09:44:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.641+0000 7fb34be00640 1 -- 172.21.15.44:0/918366721 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb344035070 con 0x7fb354104400 2024-05-27T09:44:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.641+0000 7fb35a000640 1 -- 172.21.15.44:0/918366721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3541a8ed0 con 0x7fb354104400 2024-05-27T09:44:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.642+0000 7fb35a000640 1 -- 172.21.15.44:0/918366721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3541a9330 con 0x7fb354104400 2024-05-27T09:44:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.645+0000 7fb361072740 1 Processor -- start 2024-05-27T09:44:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.645+0000 7fb361072740 1 -- start start 2024-05-27T09:44:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.645+0000 7fb34be00640 1 -- 172.21.15.44:0/918366721 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb344042030 con 0x7fb354104400 2024-05-27T09:44:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb361072740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de52779860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb361072740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523ae2f0 0x55de52779da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb361072740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 0x55de5277a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb361072740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55de526eab90 con 0x55de523ae2f0 2024-05-27T09:44:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb361072740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55de526da730 con 0x55de523b64d0 2024-05-27T09:44:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb361072740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55de526de770 con 0x55de523d71b0 2024-05-27T09:44:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb353e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de52779860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb353e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de52779860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:52878/0 (socket says 172.21.15.44:52878) 2024-05-27T09:44:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb353e00640 1 -- 172.21.15.44:0/3603211057 learned_addr learned my addr 172.21.15.44:0/3603211057 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb353400640 1 --2- 172.21.15.44:0/3603211057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523ae2f0 0x55de52779da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb352a00640 1 --2- 172.21.15.44:0/3603211057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 0x55de5277a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb353e00640 1 -- 172.21.15.44:0/3603211057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 msgr2=0x55de5277a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb353e00640 1 --2- 172.21.15.44:0/3603211057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 0x55de5277a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb353e00640 1 -- 172.21.15.44:0/3603211057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523ae2f0 msgr2=0x55de52779da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb353e00640 1 --2- 172.21.15.44:0/3603211057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523ae2f0 0x55de52779da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb353e00640 1 -- 172.21.15.44:0/3603211057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de526738b0 con 0x55de523d71b0 2024-05-27T09:44:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.646+0000 7fb352a00640 1 --2- 172.21.15.44:0/3603211057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 0x55de5277a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.647+0000 7fb353e00640 1 --2- 172.21.15.44:0/3603211057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de52779860 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb34c009fd0 tx=0x7fb34c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e034cb93d3935b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.647+0000 7fb343e00640 1 -- 172.21.15.44:0/3603211057 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c004d20 con 0x55de523d71b0 2024-05-27T09:44:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.647+0000 7fb343e00640 1 -- 172.21.15.44:0/3603211057 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb34c004050 con 0x55de523d71b0 2024-05-27T09:44:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.647+0000 7fb343e00640 1 -- 172.21.15.44:0/3603211057 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c005280 con 0x55de523d71b0 2024-05-27T09:44:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.648+0000 7fb34be00640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb3380b2950 0x7fb3380b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.648+0000 7fb34be00640 1 -- 172.21.15.44:0/918366721 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7fb344102980 con 0x7fb354104400 2024-05-27T09:44:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.648+0000 7fb353400640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb3380b2950 0x7fb3380b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.648+0000 7fb361072740 1 -- 172.21.15.44:0/3603211057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 msgr2=0x55de52779860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.648+0000 7fb361072740 1 --2- 172.21.15.44:0/3603211057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de52779860 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb34c009fd0 tx=0x7fb34c0118d0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.648+0000 7fb361072740 1 -- 172.21.15.44:0/3603211057 shutdown_connections 2024-05-27T09:44:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.648+0000 7fb361072740 1 --2- 172.21.15.44:0/3603211057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 0x55de5277a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.648+0000 7fb361072740 1 --2- 172.21.15.44:0/3603211057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523ae2f0 0x55de52779da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.648+0000 7fb361072740 1 --2- 172.21.15.44:0/3603211057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de52779860 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.648+0000 7fb361072740 1 -- 172.21.15.44:0/3603211057 >> 172.21.15.44:0/3603211057 conn(0x55de523cb300 msgr2=0x55de52776010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.648+0000 7fb361072740 1 -- 172.21.15.44:0/3603211057 shutdown_connections 2024-05-27T09:44:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.649+0000 7fb361072740 1 -- 172.21.15.44:0/3603211057 wait complete. 2024-05-27T09:44:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.649+0000 7fb353400640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb3380b2950 0x7fb3380b4e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x55de52777350 tx=0x7fb344000bc0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.649+0000 7fb361072740 1 -- start start 2024-05-27T09:44:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.649+0000 7fb361072740 1 Processor -- start 2024-05-27T09:44:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.650+0000 7fb361072740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 0x55de52783cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.650+0000 7fb361072740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523b64d0 0x55de527841f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.650+0000 7fb361072740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de5278fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.650+0000 7fb361072740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55de523c0790 con 0x55de523b64d0 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.650+0000 7fb361072740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55de52682bc0 con 0x55de523ae2f0 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.650+0000 7fb361072740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55de52786990 con 0x55de523d71b0 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.650+0000 7fb361072740 10 client.0 _ll_register_callbacks cb 0x55de52660400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.650+0000 7fb353e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 0x55de52783cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.650+0000 7fb352a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523b64d0 0x55de527841f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.650+0000 7fb353e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 0x55de52783cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54854/0 (socket says 172.21.15.44:54854) 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.650+0000 7fb353e00640 1 -- 172.21.15.44:0/846914441 learned_addr learned my addr 172.21.15.44:0/846914441 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.651+0000 7fb353e00640 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de5278fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.651+0000 7fb352a00640 1 -- 172.21.15.44:0/846914441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 msgr2=0x55de5278fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.651+0000 7fb352a00640 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de5278fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.651+0000 7fb352a00640 1 -- 172.21.15.44:0/846914441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 msgr2=0x55de52783cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.651+0000 7fb352a00640 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 0x55de52783cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.651+0000 7fb352a00640 1 -- 172.21.15.44:0/846914441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de523ef210 con 0x55de523b64d0 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.651+0000 7fb353e00640 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 0x55de52783cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.651+0000 7fb352a00640 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523b64d0 0x55de527841f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb35410a580 tx=0x7fb33c012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870126421ceb2b1a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.651+0000 7fb340c00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.652+0000 7fb340c00640 1 -- 172.21.15.44:0/846914441 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33c004020 con 0x55de523b64d0 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.652+0000 7fb340c00640 1 -- 172.21.15.44:0/846914441 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7fb33c0041c0 con 0x55de523b64d0 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.652+0000 7fb361072740 1 -- 172.21.15.44:0/846914441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb34c004050 con 0x55de523b64d0 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.652+0000 7fb340c00640 1 -- 172.21.15.44:0/846914441 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb33c0043c0 con 0x55de523b64d0 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.652+0000 7fb361072740 10 client.16689 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.652+0000 7fb361072740 1 -- 172.21.15.44:0/846914441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55de524bb620 con 0x55de523b64d0 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.652+0000 7fb340c00640 1 -- 172.21.15.44:0/846914441 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb34c004050 con 0x55de523b64d0 2024-05-27T09:44:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.653+0000 7fb340c00640 1 -- 172.21.15.44:0/846914441 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7fb33c003ac0 con 0x55de523b64d0 2024-05-27T09:44:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.653+0000 7fb361072740 10 client.16689 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.653+0000 7fb361072740 10 client.16689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.653+0000 7fb361072740 20 client.16689 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.653+0000 7fb361072740 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de52395d80 0x55de52795d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.653+0000 7fb361072740 4 client.16689 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.653+0000 7fb361072740 1 -- 172.21.15.44:0/846914441 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de526ecae0 con 0x55de52395d80 2024-05-27T09:44:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.653+0000 7fb353e00640 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de52395d80 0x55de52795d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.654+0000 7fb353e00640 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de52395d80 0x55de52795d10 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9059371360fb45bf server_cookie=166a331752d823f0 in_seq=0 out_seq=0 2024-05-27T09:44:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.654+0000 7fb340c00640 10 client.16689 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb340c00640 1 -- 172.21.15.44:0/846914441 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55de526ecae0 con 0x55de52395d80 2024-05-27T09:44:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb340c00640 10 client.16689 handle_command_reply: tid=0 2024-05-27T09:44:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 1 client.16689 shutdown 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 1 -- 172.21.15.44:0/846914441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523b64d0 msgr2=0x55de527841f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523b64d0 0x55de527841f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb35410a580 tx=0x7fb33c012b60 comp rx=0 tx=0).stop 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 1 -- 172.21.15.44:0/846914441 shutdown_connections 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de52395d80 0x55de52795d10 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de5278fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523b64d0 0x55de527841f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 1 --2- 172.21.15.44:0/846914441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 0x55de52783cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 1 -- 172.21.15.44:0/846914441 >> 172.21.15.44:0/846914441 conn(0x55de523cb300 msgr2=0x55de52779690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb340c00640 0 client.16689 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 1 -- 172.21.15.44:0/846914441 shutdown_connections 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 1 -- 172.21.15.44:0/846914441 wait complete. 2024-05-27T09:44:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.655+0000 7fb361072740 20 client.16689 trim_cache size 0 max 16384 2024-05-27T09:44:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.672+0000 7fb361072740 1 Processor -- start 2024-05-27T09:44:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb361072740 1 -- start start 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb361072740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523ae2f0 0x55de5277dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb361072740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 0x55de5278fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb361072740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523d71b0 0x55de5278ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb361072740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb3541b4720 con 0x55de523d71b0 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb361072740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb3541b1ff0 con 0x55de523b64d0 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb361072740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb3541afe60 con 0x55de523ae2f0 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb353e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523ae2f0 0x55de5277dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb353e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523ae2f0 0x55de5277dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:52900/0 (socket says 172.21.15.44:52900) 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb353e00640 1 -- 172.21.15.44:0/1185885179 learned_addr learned my addr 172.21.15.44:0/1185885179 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb352a00640 1 --2- 172.21.15.44:0/1185885179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 0x55de5278fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.673+0000 7fb353e00640 1 --2- 172.21.15.44:0/1185885179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523d71b0 0x55de5278ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.674+0000 7fb352a00640 1 -- 172.21.15.44:0/1185885179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523ae2f0 msgr2=0x55de5277dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.674+0000 7fb352a00640 1 --2- 172.21.15.44:0/1185885179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523ae2f0 0x55de5277dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.674+0000 7fb352a00640 1 -- 172.21.15.44:0/1185885179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523d71b0 msgr2=0x55de5278ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.674+0000 7fb352a00640 1 --2- 172.21.15.44:0/1185885179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523d71b0 0x55de5278ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.674+0000 7fb352a00640 1 -- 172.21.15.44:0/1185885179 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de52395fb0 con 0x55de523b64d0 2024-05-27T09:44:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.674+0000 7fb352a00640 1 --2- 172.21.15.44:0/1185885179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 0x55de5278fa30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb35410a580 tx=0x7fb33c001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0ad9073a31a426d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.674+0000 7fb343400640 1 -- 172.21.15.44:0/1185885179 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33c025500 con 0x55de523b64d0 2024-05-27T09:44:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.674+0000 7fb343400640 1 -- 172.21.15.44:0/1185885179 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb33c025e50 con 0x55de523b64d0 2024-05-27T09:44:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.674+0000 7fb343400640 1 -- 172.21.15.44:0/1185885179 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33c00fbd0 con 0x55de523b64d0 2024-05-27T09:44:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.675+0000 7fb361072740 1 -- 172.21.15.44:0/1185885179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 msgr2=0x55de5278fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.675+0000 7fb361072740 1 --2- 172.21.15.44:0/1185885179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 0x55de5278fa30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb35410a580 tx=0x7fb33c001e50 comp rx=0 tx=0).stop 2024-05-27T09:44:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.675+0000 7fb361072740 1 -- 172.21.15.44:0/1185885179 shutdown_connections 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.675+0000 7fb361072740 1 --2- 172.21.15.44:0/1185885179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523d71b0 0x55de5278ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.675+0000 7fb361072740 1 --2- 172.21.15.44:0/1185885179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523b64d0 0x55de5278fa30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.675+0000 7fb361072740 1 --2- 172.21.15.44:0/1185885179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523ae2f0 0x55de5277dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.675+0000 7fb361072740 1 -- 172.21.15.44:0/1185885179 >> 172.21.15.44:0/1185885179 conn(0x55de523cb300 msgr2=0x55de5277d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.676+0000 7fb361072740 1 -- 172.21.15.44:0/1185885179 shutdown_connections 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.676+0000 7fb361072740 1 -- 172.21.15.44:0/1185885179 wait complete. 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.676+0000 7fb361072740 1 -- start start 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.676+0000 7fb361072740 1 Processor -- start 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb361072740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 0x55de5278c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb361072740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523b64d0 0x55de527886b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb361072740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de52788bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb361072740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb3541b04b0 con 0x55de523b64d0 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb361072740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb3541b1920 con 0x55de523ae2f0 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb361072740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb3541b0b80 con 0x55de523d71b0 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb361072740 10 client.0 _ll_register_callbacks cb 0x7fb33c009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb353e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 0x55de5278c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb353e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 0x55de5278c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54872/0 (socket says 172.21.15.44:54872) 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb353e00640 1 -- 172.21.15.44:0/972705098 learned_addr learned my addr 172.21.15.44:0/972705098 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb353e00640 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de52788bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.677+0000 7fb352a00640 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523b64d0 0x55de527886b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb353e00640 1 -- 172.21.15.44:0/972705098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 msgr2=0x55de5278c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb353e00640 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 0x55de5278c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb353e00640 1 -- 172.21.15.44:0/972705098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523b64d0 msgr2=0x55de527886b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb353e00640 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523b64d0 0x55de527886b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb353e00640 1 -- 172.21.15.44:0/972705098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de52789360 con 0x55de523d71b0 2024-05-27T09:44:17.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb353e00640 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de52788bf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x55de52787f10 tx=0x7fb34c000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a19ab2295fe52dce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:17.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb340c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:17.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb340c00640 1 -- 172.21.15.44:0/972705098 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0042f0 con 0x55de523d71b0 2024-05-27T09:44:17.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb340c00640 1 -- 172.21.15.44:0/972705098 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7fb34c01f040 con 0x55de523d71b0 2024-05-27T09:44:17.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb361072740 1 -- 172.21.15.44:0/972705098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb33c025e50 con 0x55de523d71b0 2024-05-27T09:44:17.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb340c00640 1 -- 172.21.15.44:0/972705098 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb34c004490 con 0x55de523d71b0 2024-05-27T09:44:17.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.678+0000 7fb361072740 10 client.26884 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:17.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.679+0000 7fb361072740 1 -- 172.21.15.44:0/972705098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55de527fd2f0 con 0x55de523d71b0 2024-05-27T09:44:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.679+0000 7fb340c00640 1 -- 172.21.15.44:0/972705098 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb33c025e50 con 0x55de523d71b0 2024-05-27T09:44:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.679+0000 7fb340c00640 1 -- 172.21.15.44:0/972705098 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7fb34c028450 con 0x55de523d71b0 2024-05-27T09:44:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.679+0000 7fb361072740 10 client.26884 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.679+0000 7fb361072740 10 client.26884 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.679+0000 7fb361072740 20 client.26884 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.679+0000 7fb361072740 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de5236b1c0 0x55de527ff8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.679+0000 7fb361072740 4 client.26884 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:44:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.679+0000 7fb361072740 1 -- 172.21.15.44:0/972705098 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de526e62c0 con 0x55de5236b1c0 2024-05-27T09:44:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.679+0000 7fb352a00640 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de5236b1c0 0x55de527ff8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.680+0000 7fb352a00640 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de5236b1c0 0x55de527ff8f0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93239f8a42f26002 server_cookie=b1713fbe3e5dabc5 in_seq=0 out_seq=0 2024-05-27T09:44:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.680+0000 7fb340c00640 10 client.26884 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb340c00640 1 -- 172.21.15.44:0/972705098 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55de526e62c0 con 0x55de5236b1c0 2024-05-27T09:44:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb340c00640 10 client.26884 handle_command_reply: tid=0 2024-05-27T09:44:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 1 client.26884 shutdown 2024-05-27T09:44:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 1 -- 172.21.15.44:0/972705098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 msgr2=0x55de52788bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de52788bf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x55de52787f10 tx=0x7fb34c000f40 comp rx=0 tx=0).stop 2024-05-27T09:44:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 1 -- 172.21.15.44:0/972705098 shutdown_connections 2024-05-27T09:44:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55de5236b1c0 0x55de527ff8f0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de523d71b0 0x55de52788bf0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55de523b64d0 0x55de527886b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb340c00640 0 client.26884 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 1 --2- 172.21.15.44:0/972705098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55de523ae2f0 0x55de5278c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 1 -- 172.21.15.44:0/972705098 >> 172.21.15.44:0/972705098 conn(0x55de523cb300 msgr2=0x55de527795f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 1 -- 172.21.15.44:0/972705098 shutdown_connections 2024-05-27T09:44:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 1 -- 172.21.15.44:0/972705098 wait complete. 2024-05-27T09:44:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.706+0000 7fb361072740 20 client.26884 trim_cache size 0 max 16384 2024-05-27T09:44:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:44:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:44:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:44:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 -- 172.21.15.44:0/918366721 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb3380b2950 msgr2=0x7fb3380b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb3380b2950 0x7fb3380b4e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x55de52777350 tx=0x7fb344000bc0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 -- 172.21.15.44:0/918366721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb354104400 msgr2=0x7fb354074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb354104400 0x7fb354074fc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb344032fd0 tx=0x7fb344031a20 comp rx=0 tx=0).stop 2024-05-27T09:44:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 -- 172.21.15.44:0/918366721 shutdown_connections 2024-05-27T09:44:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb3380b2950 0x7fb3380b4e60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35410a150 0x7fb354075a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb354109cb0 0x7fb354075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 --2- 172.21.15.44:0/918366721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb354104400 0x7fb354074fc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 -- 172.21.15.44:0/918366721 >> 172.21.15.44:0/918366721 conn(0x7fb3540ff480 msgr2=0x7fb3541011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 -- 172.21.15.44:0/918366721 shutdown_connections 2024-05-27T09:44:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:17.707+0000 7fb35a000640 1 -- 172.21.15.44:0/918366721 wait complete. 2024-05-27T09:44:18.138 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:17 smithi189 ceph-mon[36671]: pgmap v282: 97 pgs: 97 active+clean; 265 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 991 op/s 2024-05-27T09:44:18.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:17 smithi044 ceph-mon[31618]: pgmap v282: 97 pgs: 97 active+clean; 265 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 991 op/s 2024-05-27T09:44:18.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:17 smithi050 ceph-mon[36371]: pgmap v282: 97 pgs: 97 active+clean; 265 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 991 op/s 2024-05-27T09:44:20.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:19 smithi044 ceph-mon[31618]: pgmap v283: 97 pgs: 97 active+clean; 265 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 991 op/s 2024-05-27T09:44:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:19 smithi050 ceph-mon[36371]: pgmap v283: 97 pgs: 97 active+clean; 265 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 991 op/s 2024-05-27T09:44:20.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:19 smithi189 ceph-mon[36671]: pgmap v283: 97 pgs: 97 active+clean; 265 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 991 op/s 2024-05-27T09:44:22.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:21 smithi044 ceph-mon[31618]: pgmap v284: 97 pgs: 97 active+clean; 276 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 217 op/s 2024-05-27T09:44:22.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:21 smithi050 ceph-mon[36371]: pgmap v284: 97 pgs: 97 active+clean; 276 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 217 op/s 2024-05-27T09:44:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:21 smithi189 ceph-mon[36671]: pgmap v284: 97 pgs: 97 active+clean; 276 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 217 op/s 2024-05-27T09:44:24.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:23 smithi044 ceph-mon[31618]: pgmap v285: 97 pgs: 97 active+clean; 276 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 217 op/s 2024-05-27T09:44:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:23 smithi050 ceph-mon[36371]: pgmap v285: 97 pgs: 97 active+clean; 276 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 217 op/s 2024-05-27T09:44:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:23 smithi189 ceph-mon[36671]: pgmap v285: 97 pgs: 97 active+clean; 276 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 217 op/s 2024-05-27T09:44:24.583 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:44:25.111 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:44:26.129 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:25 smithi044 ceph-mon[31618]: pgmap v286: 97 pgs: 97 active+clean; 276 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 183 op/s 2024-05-27T09:44:26.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:25 smithi050 ceph-mon[36371]: pgmap v286: 97 pgs: 97 active+clean; 276 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 183 op/s 2024-05-27T09:44:26.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:25 smithi189 ceph-mon[36671]: pgmap v286: 97 pgs: 97 active+clean; 276 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 183 op/s 2024-05-27T09:44:27.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.914+0000 7f5271c00700 1 -- 172.21.15.44:0/419555983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f526c076670 msgr2=0x7f526c110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:27.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.914+0000 7f5271c00700 1 --2- 172.21.15.44:0/419555983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f526c076670 0x7f526c110650 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f52580098f0 tx=0x7f5258009c00 comp rx=0 tx=0).stop 2024-05-27T09:44:27.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.914+0000 7f5271c00700 1 -- 172.21.15.44:0/419555983 shutdown_connections 2024-05-27T09:44:27.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.914+0000 7f5271c00700 1 --2- 172.21.15.44:0/419555983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f526c110b90 0x7f526c112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.914+0000 7f5271c00700 1 --2- 172.21.15.44:0/419555983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f526c075d20 0x7f526c076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.914+0000 7f5271c00700 1 --2- 172.21.15.44:0/419555983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f526c076670 0x7f526c110650 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.914+0000 7f5271c00700 1 -- 172.21.15.44:0/419555983 >> 172.21.15.44:0/419555983 conn(0x7f526c100030 msgr2=0x7f526c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:27.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.915+0000 7f5271c00700 1 -- 172.21.15.44:0/419555983 shutdown_connections 2024-05-27T09:44:27.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.915+0000 7f5271c00700 1 -- 172.21.15.44:0/419555983 wait complete. 2024-05-27T09:44:27.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.916+0000 7f5271c00700 1 Processor -- start 2024-05-27T09:44:27.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.916+0000 7f5271c00700 1 -- start start 2024-05-27T09:44:27.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.917+0000 7f5271c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f526c075d20 0x7f526c19d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.917+0000 7f5271c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f526c076670 0x7f526c19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.917+0000 7f5271c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f526c110b90 0x7f526c1a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.917+0000 7f5271c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f526c1166e0 con 0x7f526c076670 2024-05-27T09:44:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.917+0000 7f5271c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f526c116560 con 0x7f526c075d20 2024-05-27T09:44:27.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.917+0000 7f5271c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f526c116860 con 0x7f526c110b90 2024-05-27T09:44:27.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.917+0000 7f526a000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f526c076670 0x7f526c19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.917+0000 7f526b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f526c110b90 0x7f526c1a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.917+0000 7f526b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f526c110b90 0x7f526c1a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:57970/0 (socket says 172.21.15.44:57970) 2024-05-27T09:44:27.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.917+0000 7f526aa00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f526c075d20 0x7f526c19d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.917+0000 7f526b400700 1 -- 172.21.15.44:0/386411664 learned_addr learned my addr 172.21.15.44:0/386411664 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:27.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.918+0000 7f526b400700 1 -- 172.21.15.44:0/386411664 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f526c075d20 msgr2=0x7f526c19d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.918+0000 7f526b400700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f526c075d20 0x7f526c19d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.918+0000 7f526b400700 1 -- 172.21.15.44:0/386411664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f526c076670 msgr2=0x7f526c19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.918+0000 7f526b400700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f526c076670 0x7f526c19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.918+0000 7f526b400700 1 -- 172.21.15.44:0/386411664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526c1a1f80 con 0x7f526c110b90 2024-05-27T09:44:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.918+0000 7f526a000700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f526c076670 0x7f526c19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.918+0000 7f526b400700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f526c110b90 0x7f526c1a18e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f526400ef90 tx=0x7f526400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.918+0000 7f5263400700 1 -- 172.21.15.44:0/386411664 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52640092e0 con 0x7f526c110b90 2024-05-27T09:44:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.918+0000 7f5263400700 1 -- 172.21.15.44:0/386411664 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5264004bb0 con 0x7f526c110b90 2024-05-27T09:44:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.918+0000 7f5271c00700 1 -- 172.21.15.44:0/386411664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526c1a2210 con 0x7f526c110b90 2024-05-27T09:44:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.918+0000 7f5263400700 1 -- 172.21.15.44:0/386411664 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5264004020 con 0x7f526c110b90 2024-05-27T09:44:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.919+0000 7f5271c00700 1 -- 172.21.15.44:0/386411664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f526c1a2720 con 0x7f526c110b90 2024-05-27T09:44:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.923+0000 7f5280785b80 1 Processor -- start 2024-05-27T09:44:27.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.924+0000 7f5263400700 1 -- 172.21.15.44:0/386411664 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f526401b030 con 0x7f526c110b90 2024-05-27T09:44:27.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.924+0000 7f5280785b80 1 -- start start 2024-05-27T09:44:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.924+0000 7f5280785b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x56244225a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.925+0000 7f5280785b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 0x56244225a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.925+0000 7f5280785b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 0x562442267110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.925+0000 7f5280785b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562442188e40 con 0x5624420dbe40 2024-05-27T09:44:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.925+0000 7f5280785b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5624421789e0 con 0x5624421c79f0 2024-05-27T09:44:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.925+0000 7f5280785b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56244217ca20 con 0x56244225ae70 2024-05-27T09:44:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.925+0000 7f526aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 0x56244225a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.925+0000 7f526aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 0x56244225a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51312/0 (socket says 172.21.15.44:51312) 2024-05-27T09:44:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.925+0000 7f526aa00700 1 -- 172.21.15.44:0/3317695865 learned_addr learned my addr 172.21.15.44:0/3317695865 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:27.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.925+0000 7f526a000700 1 --2- 172.21.15.44:0/3317695865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x56244225a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.925+0000 7f526a000700 1 --2- 172.21.15.44:0/3317695865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 0x562442267110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.926+0000 7f526aa00700 1 -- 172.21.15.44:0/3317695865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 msgr2=0x562442267110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.926+0000 7f526aa00700 1 --2- 172.21.15.44:0/3317695865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 0x562442267110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.926+0000 7f526aa00700 1 -- 172.21.15.44:0/3317695865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 msgr2=0x56244225a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.926+0000 7f526aa00700 1 --2- 172.21.15.44:0/3317695865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x56244225a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.926+0000 7f526aa00700 1 -- 172.21.15.44:0/3317695865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562442141420 con 0x5624420dbe40 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.926+0000 7f526a000700 1 --2- 172.21.15.44:0/3317695865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 0x562442267110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.926+0000 7f526aa00700 1 --2- 172.21.15.44:0/3317695865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 0x56244225a870 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f525c00bd50 tx=0x7f525c0106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33f1f641d530a8d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.926+0000 7f5263400700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f52500b2a70 0x7f52500b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.926+0000 7f5263400700 1 -- 172.21.15.44:0/386411664 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f52640e2060 con 0x7f526c110b90 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.927+0000 7f5257e00700 1 -- 172.21.15.44:0/3317695865 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c004830 con 0x5624420dbe40 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.927+0000 7f5257e00700 1 -- 172.21.15.44:0/3317695865 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f525c005020 con 0x5624420dbe40 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.927+0000 7f5257e00700 1 -- 172.21.15.44:0/3317695865 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c0116a0 con 0x5624420dbe40 2024-05-27T09:44:27.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.927+0000 7f526a000700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f52500b2a70 0x7f52500b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.928+0000 7f5280785b80 1 -- 172.21.15.44:0/3317695865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 msgr2=0x56244225a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:27.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.928+0000 7f5280785b80 1 --2- 172.21.15.44:0/3317695865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 0x56244225a870 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f525c00bd50 tx=0x7f525c0106b0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.928+0000 7f526a000700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f52500b2a70 0x7f52500b4f80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f52580098c0 tx=0x7f5258019040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:27.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.928+0000 7f5280785b80 1 -- 172.21.15.44:0/3317695865 shutdown_connections 2024-05-27T09:44:27.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.928+0000 7f5280785b80 1 --2- 172.21.15.44:0/3317695865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 0x562442267110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.928+0000 7f5280785b80 1 --2- 172.21.15.44:0/3317695865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x56244225a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.928+0000 7f5280785b80 1 --2- 172.21.15.44:0/3317695865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 0x56244225a870 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.928+0000 7f5280785b80 1 -- 172.21.15.44:0/3317695865 >> 172.21.15.44:0/3317695865 conn(0x562442175390 msgr2=0x5624422564c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:27.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:27 smithi044 ceph-mon[31618]: pgmap v287: 97 pgs: 97 active+clean; 260 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 669 op/s 2024-05-27T09:44:27.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.928+0000 7f5280785b80 1 -- 172.21.15.44:0/3317695865 shutdown_connections 2024-05-27T09:44:27.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.928+0000 7f5280785b80 1 -- 172.21.15.44:0/3317695865 wait complete. 2024-05-27T09:44:27.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.929+0000 7f5280785b80 1 -- start start 2024-05-27T09:44:27.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.929+0000 7f5280785b80 1 Processor -- start 2024-05-27T09:44:27.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f5280785b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 0x562442264650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f5280785b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x562442264b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f5280785b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 0x5624422650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f5280785b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56244207efa0 con 0x5624420dbe40 2024-05-27T09:44:27.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f5280785b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562442119760 con 0x5624421c79f0 2024-05-27T09:44:27.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f5280785b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562442267b80 con 0x56244225ae70 2024-05-27T09:44:27.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f5280785b80 10 client.0 _ll_register_callbacks cb 0x562442122c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:27.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f526a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x562442264b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f526aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 0x562442264650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f526a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x562442264b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50502/0 (socket says 172.21.15.44:50502) 2024-05-27T09:44:27.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f526aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 0x562442264650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51316/0 (socket says 172.21.15.44:51316) 2024-05-27T09:44:27.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f526a000700 1 -- 172.21.15.44:0/4238299261 learned_addr learned my addr 172.21.15.44:0/4238299261 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:27.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.930+0000 7f526b400700 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 0x5624422650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f526b400700 1 -- 172.21.15.44:0/4238299261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 msgr2=0x562442264b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:27.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f526b400700 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x562442264b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f526b400700 1 -- 172.21.15.44:0/4238299261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 msgr2=0x562442264650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:27.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f526b400700 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 0x562442264650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f526b400700 1 -- 172.21.15.44:0/4238299261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624420c88d0 con 0x56244225ae70 2024-05-27T09:44:27.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f526b400700 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 0x5624422650d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f52640df750 tx=0x7f52640a8c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b690d6a8e3dbad4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:27.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f5254c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:27.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f5254c00700 1 -- 172.21.15.44:0/4238299261 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52640a9370 con 0x56244225ae70 2024-05-27T09:44:27.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f5254c00700 1 -- 172.21.15.44:0/4238299261 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f52640a97e0 con 0x56244225ae70 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f5280785b80 1 -- 172.21.15.44:0/4238299261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f525c005020 con 0x56244225ae70 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f5254c00700 1 -- 172.21.15.44:0/4238299261 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f52640ad720 con 0x56244225ae70 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f5280785b80 10 client.26896 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.931+0000 7f5280785b80 1 -- 172.21.15.44:0/4238299261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562442149060 con 0x56244225ae70 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.932+0000 7f5254c00700 1 -- 172.21.15.44:0/4238299261 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f525c005020 con 0x56244225ae70 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.932+0000 7f5254c00700 1 -- 172.21.15.44:0/4238299261 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f52640a8040 con 0x56244225ae70 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.932+0000 7f5280785b80 10 client.26896 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.932+0000 7f5280785b80 10 client.26896 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.932+0000 7f5280785b80 20 client.26896 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.932+0000 7f5280785b80 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562442274ad0 0x562442276f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.932+0000 7f5280785b80 4 client.26896 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.932+0000 7f5280785b80 1 -- 172.21.15.44:0/4238299261 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56244218ad90 con 0x562442274ad0 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.933+0000 7f526aa00700 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562442274ad0 0x562442276f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.933+0000 7f5254c00700 10 client.26896 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.933+0000 7f526aa00700 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562442274ad0 0x562442276f50 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5cc36fe2d2f3acd server_cookie=8cd9ffdf44f56f7b in_seq=0 out_seq=0 2024-05-27T09:44:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.934+0000 7f5254c00700 1 -- 172.21.15.44:0/4238299261 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56244218ad90 con 0x562442274ad0 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.934+0000 7f5254c00700 10 client.26896 handle_command_reply: tid=0 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.934+0000 7f5280785b80 1 client.26896 shutdown 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5280785b80 1 -- 172.21.15.44:0/4238299261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 msgr2=0x5624422650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5280785b80 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 0x5624422650d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f52640df750 tx=0x7f52640a8c00 comp rx=0 tx=0).stop 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5280785b80 1 -- 172.21.15.44:0/4238299261 shutdown_connections 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5280785b80 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562442274ad0 0x562442276f50 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5280785b80 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56244225ae70 0x5624422650d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5280785b80 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624420dbe40 0x562442264650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5280785b80 1 --2- 172.21.15.44:0/4238299261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x562442264b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5254c00700 0 client.26896 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5280785b80 1 -- 172.21.15.44:0/4238299261 >> 172.21.15.44:0/4238299261 conn(0x562442175390 msgr2=0x56244225a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5280785b80 1 -- 172.21.15.44:0/4238299261 shutdown_connections 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5280785b80 1 -- 172.21.15.44:0/4238299261 wait complete. 2024-05-27T09:44:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.935+0000 7f5280785b80 20 client.26896 trim_cache size 0 max 16384 2024-05-27T09:44:27.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.958+0000 7f5280785b80 1 Processor -- start 2024-05-27T09:44:27.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.958+0000 7f5280785b80 1 -- start start 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.958+0000 7f5280785b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 0x56244225fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.958+0000 7f5280785b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x56244227c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.958+0000 7f5280785b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56244225ae70 0x56244227cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f5280785b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f526c1b3940 con 0x56244225ae70 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f5280785b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f526c1b11d0 con 0x5624421c79f0 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f5280785b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f526c1af040 con 0x5624420dbe40 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x56244227c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 0x56244225fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x56244227c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50518/0 (socket says 172.21.15.44:50518) 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526a000700 1 -- 172.21.15.44:0/714674811 learned_addr learned my addr 172.21.15.44:0/714674811 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 0x56244225fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:57992/0 (socket says 172.21.15.44:57992) 2024-05-27T09:44:27.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526b400700 1 --2- 172.21.15.44:0/714674811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56244225ae70 0x56244227cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526aa00700 1 -- 172.21.15.44:0/714674811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 msgr2=0x56244227c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:27.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526aa00700 1 --2- 172.21.15.44:0/714674811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x56244227c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526aa00700 1 -- 172.21.15.44:0/714674811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56244225ae70 msgr2=0x56244227cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:27.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526aa00700 1 --2- 172.21.15.44:0/714674811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56244225ae70 0x56244227cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526aa00700 1 -- 172.21.15.44:0/714674811 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56244225ba50 con 0x5624420dbe40 2024-05-27T09:44:27.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.959+0000 7f526a000700 1 --2- 172.21.15.44:0/714674811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x56244227c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:27.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.960+0000 7f526aa00700 1 --2- 172.21.15.44:0/714674811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 0x56244225fe60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f525c00bd20 tx=0x7f525c00b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7c4cc00d34b4b93 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:27.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.960+0000 7f5257400700 1 -- 172.21.15.44:0/714674811 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c005040 con 0x5624420dbe40 2024-05-27T09:44:27.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.960+0000 7f5257400700 1 -- 172.21.15.44:0/714674811 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f525c0046d0 con 0x5624420dbe40 2024-05-27T09:44:27.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.960+0000 7f5257400700 1 -- 172.21.15.44:0/714674811 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c011680 con 0x5624420dbe40 2024-05-27T09:44:27.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.961+0000 7f5280785b80 1 -- 172.21.15.44:0/714674811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 msgr2=0x56244225fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:27.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.961+0000 7f5280785b80 1 --2- 172.21.15.44:0/714674811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 0x56244225fe60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f525c00bd20 tx=0x7f525c00b820 comp rx=0 tx=0).stop 2024-05-27T09:44:27.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.961+0000 7f5280785b80 1 -- 172.21.15.44:0/714674811 shutdown_connections 2024-05-27T09:44:27.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.961+0000 7f5280785b80 1 --2- 172.21.15.44:0/714674811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56244225ae70 0x56244227cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.961+0000 7f5280785b80 1 --2- 172.21.15.44:0/714674811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 0x56244225fe60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.961+0000 7f5280785b80 1 --2- 172.21.15.44:0/714674811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624421c79f0 0x56244227c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.961+0000 7f5280785b80 1 -- 172.21.15.44:0/714674811 >> 172.21.15.44:0/714674811 conn(0x562442175390 msgr2=0x56244225a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:27.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.961+0000 7f5280785b80 1 -- 172.21.15.44:0/714674811 shutdown_connections 2024-05-27T09:44:27.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.961+0000 7f5280785b80 1 -- 172.21.15.44:0/714674811 wait complete. 2024-05-27T09:44:27.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.962+0000 7f5280785b80 1 -- start start 2024-05-27T09:44:27.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.962+0000 7f5280785b80 1 Processor -- start 2024-05-27T09:44:27.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f5280785b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 0x562442268d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f5280785b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624421c79f0 0x562442269270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f5280785b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56244225ae70 0x5624422697b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f5280785b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f526c1af690 con 0x5624421c79f0 2024-05-27T09:44:27.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f5280785b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f526c1b0b00 con 0x56244225ae70 2024-05-27T09:44:27.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f5280785b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f526c1afd60 con 0x5624420dbe40 2024-05-27T09:44:27.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f5280785b80 10 client.0 _ll_register_callbacks cb 0x562442122c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:27.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f526a000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624421c79f0 0x562442269270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f526b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56244225ae70 0x5624422697b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f526a000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624421c79f0 0x562442269270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51322/0 (socket says 172.21.15.44:51322) 2024-05-27T09:44:27.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f526a000700 1 -- 172.21.15.44:0/2832786135 learned_addr learned my addr 172.21.15.44:0/2832786135 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:27.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.963+0000 7f526aa00700 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 0x562442268d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.964+0000 7f526b400700 1 -- 172.21.15.44:0/2832786135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 msgr2=0x562442268d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:27.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.964+0000 7f526b400700 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 0x562442268d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.964+0000 7f526b400700 1 -- 172.21.15.44:0/2832786135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624421c79f0 msgr2=0x562442269270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:27.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.964+0000 7f526b400700 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624421c79f0 0x562442269270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.964+0000 7f526b400700 1 -- 172.21.15.44:0/2832786135 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624422de7a0 con 0x56244225ae70 2024-05-27T09:44:27.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.964+0000 7f526aa00700 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 0x562442268d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:27.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.964+0000 7f526b400700 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56244225ae70 0x5624422697b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f52640a53f0 tx=0x7f52640a9370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=413371ccac1bd417 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:27.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.964+0000 7f5254c00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:44:27.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.964+0000 7f5254c00700 1 -- 172.21.15.44:0/2832786135 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52640b2140 con 0x56244225ae70 2024-05-27T09:44:27.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.964+0000 7f5254c00700 1 -- 172.21.15.44:0/2832786135 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f52640c3bd0 con 0x56244225ae70 2024-05-27T09:44:27.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.964+0000 7f5280785b80 1 -- 172.21.15.44:0/2832786135 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f525c0046d0 con 0x56244225ae70 2024-05-27T09:44:27.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.965+0000 7f5254c00700 1 -- 172.21.15.44:0/2832786135 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f52640c3dd0 con 0x56244225ae70 2024-05-27T09:44:27.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.965+0000 7f5280785b80 10 client.26918 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:27.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.965+0000 7f5280785b80 1 -- 172.21.15.44:0/2832786135 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5624422dfa10 con 0x56244225ae70 2024-05-27T09:44:27.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.965+0000 7f5254c00700 1 -- 172.21.15.44:0/2832786135 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f525c0046d0 con 0x56244225ae70 2024-05-27T09:44:27.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.965+0000 7f5254c00700 1 -- 172.21.15.44:0/2832786135 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f52640a3a90 con 0x56244225ae70 2024-05-27T09:44:27.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.965+0000 7f5280785b80 10 client.26918 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:27.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.965+0000 7f5280785b80 10 client.26918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:27.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.965+0000 7f5280785b80 20 client.26918 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:27.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.965+0000 7f5280785b80 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624422e0120 0x5624422e2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:27.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.965+0000 7f5280785b80 4 client.26918 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:44:27.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.965+0000 7f5280785b80 1 -- 172.21.15.44:0/2832786135 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562442184570 con 0x5624422e0120 2024-05-27T09:44:27.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.966+0000 7f526aa00700 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624422e0120 0x5624422e2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.966+0000 7f526aa00700 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624422e0120 0x5624422e2540 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1be43f6cdc6f7ac5 server_cookie=a20d32c398cd6a6d in_seq=0 out_seq=0 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.966+0000 7f5254c00700 10 client.26918 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.966+0000 7f5254c00700 1 -- 172.21.15.44:0/2832786135 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562442184570 con 0x5624422e0120 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.966+0000 7f5254c00700 10 client.26918 handle_command_reply: tid=0 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.966+0000 7f5280785b80 1 client.26918 shutdown 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.966+0000 7f5280785b80 1 -- 172.21.15.44:0/2832786135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56244225ae70 msgr2=0x5624422697b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.966+0000 7f5280785b80 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56244225ae70 0x5624422697b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f52640a53f0 tx=0x7f52640a9370 comp rx=0 tx=0).stop 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.967+0000 7f5280785b80 1 -- 172.21.15.44:0/2832786135 shutdown_connections 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.967+0000 7f5280785b80 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624422e0120 0x5624422e2540 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.967+0000 7f5280785b80 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56244225ae70 0x5624422697b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.967+0000 7f5280785b80 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624420dbe40 0x562442268d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.967+0000 7f5280785b80 1 --2- 172.21.15.44:0/2832786135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624421c79f0 0x562442269270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.967+0000 7f5280785b80 1 -- 172.21.15.44:0/2832786135 >> 172.21.15.44:0/2832786135 conn(0x562442175390 msgr2=0x56244225a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.967+0000 7f5254c00700 0 client.26918 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.967+0000 7f5280785b80 1 -- 172.21.15.44:0/2832786135 shutdown_connections 2024-05-27T09:44:27.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.967+0000 7f5280785b80 1 -- 172.21.15.44:0/2832786135 wait complete. 2024-05-27T09:44:27.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.967+0000 7f5280785b80 20 client.26918 trim_cache size 0 max 16384 2024-05-27T09:44:27.970 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:44:27.970 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:44:27.970 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:44:27.970 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:44:27.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 -- 172.21.15.44:0/386411664 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f52500b2a70 msgr2=0x7f52500b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:27.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f52500b2a70 0x7f52500b4f80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f52580098c0 tx=0x7f5258019040 comp rx=0 tx=0).stop 2024-05-27T09:44:27.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 -- 172.21.15.44:0/386411664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f526c110b90 msgr2=0x7f526c1a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:27.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f526c110b90 0x7f526c1a18e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f526400ef90 tx=0x7f526400c3a0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 -- 172.21.15.44:0/386411664 shutdown_connections 2024-05-27T09:44:27.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f52500b2a70 0x7f52500b4f80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f526c110b90 0x7f526c1a18e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f526c075d20 0x7f526c19d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 --2- 172.21.15.44:0/386411664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f526c076670 0x7f526c19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:27.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 -- 172.21.15.44:0/386411664 >> 172.21.15.44:0/386411664 conn(0x7f526c100030 msgr2=0x7f526c100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:27.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 -- 172.21.15.44:0/386411664 shutdown_connections 2024-05-27T09:44:27.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:27.968+0000 7f5271c00700 1 -- 172.21.15.44:0/386411664 wait complete. 2024-05-27T09:44:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:27 smithi050 ceph-mon[36371]: pgmap v287: 97 pgs: 97 active+clean; 260 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 669 op/s 2024-05-27T09:44:28.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:27 smithi189 ceph-mon[36671]: pgmap v287: 97 pgs: 97 active+clean; 260 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 669 op/s 2024-05-27T09:44:28.441 INFO:tasks.workunit.client.0.smithi044.stdout:created hierarchy, now cleaning up 2024-05-27T09:44:29.213 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:44:29.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:44:29.737 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:44:30.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:29 smithi044 ceph-mon[31618]: pgmap v288: 97 pgs: 97 active+clean; 260 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 662 op/s 2024-05-27T09:44:30.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:29 smithi050 ceph-mon[36371]: pgmap v288: 97 pgs: 97 active+clean; 260 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 662 op/s 2024-05-27T09:44:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:29 smithi189 ceph-mon[36671]: pgmap v288: 97 pgs: 97 active+clean; 260 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 662 op/s 2024-05-27T09:44:30.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.941+0000 7f6760a00640 1 -- 172.21.15.44:0/3579089666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f675c10c180 msgr2=0x7f675c105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:30.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.941+0000 7f6760a00640 1 --2- 172.21.15.44:0/3579089666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f675c10c180 0x7f675c105dc0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6748009900 tx=0x7f674802a3f0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.941+0000 7f6760a00640 1 -- 172.21.15.44:0/3579089666 shutdown_connections 2024-05-27T09:44:30.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.941+0000 7f6760a00640 1 --2- 172.21.15.44:0/3579089666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f675c106430 0x7f675c112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.941+0000 7f6760a00640 1 --2- 172.21.15.44:0/3579089666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f675c10c180 0x7f675c105dc0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.941+0000 7f6760a00640 1 --2- 172.21.15.44:0/3579089666 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f675c10bce0 0x7f675c105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.941+0000 7f6760a00640 1 -- 172.21.15.44:0/3579089666 >> 172.21.15.44:0/3579089666 conn(0x7f675c101380 msgr2=0x7f675c1037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.941+0000 7f6760a00640 1 -- 172.21.15.44:0/3579089666 shutdown_connections 2024-05-27T09:44:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.941+0000 7f6760a00640 1 -- 172.21.15.44:0/3579089666 wait complete. 2024-05-27T09:44:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.942+0000 7f6760a00640 1 Processor -- start 2024-05-27T09:44:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.942+0000 7f6760a00640 1 -- start start 2024-05-27T09:44:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.943+0000 7f6760a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f675c106430 0x7f675c1a25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.943+0000 7f6760a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f675c10bce0 0x7f675c1a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.943+0000 7f6760a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f675c10c180 0x7f675c19c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.943+0000 7f6760a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f675c115eb0 con 0x7f675c10c180 2024-05-27T09:44:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.943+0000 7f6760a00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f675c115d30 con 0x7f675c10bce0 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.943+0000 7f6760a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f675c116030 con 0x7f675c106430 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.943+0000 7f675a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f675c106430 0x7f675c1a25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.943+0000 7f675a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f675c106430 0x7f675c1a25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:58022/0 (socket says 172.21.15.44:58022) 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.943+0000 7f6759e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f675c10bce0 0x7f675c1a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.943+0000 7f675a800640 1 -- 172.21.15.44:0/4195338247 learned_addr learned my addr 172.21.15.44:0/4195338247 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.943+0000 7f675b200640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f675c10c180 0x7f675c19c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.944+0000 7f675a800640 1 -- 172.21.15.44:0/4195338247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f675c10bce0 msgr2=0x7f675c1a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.944+0000 7f675a800640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f675c10bce0 0x7f675c1a2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.944+0000 7f675a800640 1 -- 172.21.15.44:0/4195338247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f675c10c180 msgr2=0x7f675c19c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.944+0000 7f675a800640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f675c10c180 0x7f675c19c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.944+0000 7f675a800640 1 -- 172.21.15.44:0/4195338247 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675c19cea0 con 0x7f675c106430 2024-05-27T09:44:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.944+0000 7f675b200640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f675c10c180 0x7f675c19c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.944+0000 7f6759e00640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f675c10bce0 0x7f675c1a2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.944+0000 7f675a800640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f675c106430 0x7f675c1a25f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f674c00d770 tx=0x7f674c00dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.944+0000 7f6753400640 1 -- 172.21.15.44:0/4195338247 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c017070 con 0x7f675c106430 2024-05-27T09:44:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.945+0000 7f6760a00640 1 -- 172.21.15.44:0/4195338247 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f675c19d130 con 0x7f675c106430 2024-05-27T09:44:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.945+0000 7f6753400640 1 -- 172.21.15.44:0/4195338247 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f674c009c30 con 0x7f675c106430 2024-05-27T09:44:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.945+0000 7f6753400640 1 -- 172.21.15.44:0/4195338247 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c00f620 con 0x7f675c106430 2024-05-27T09:44:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.945+0000 7f6760a00640 1 -- 172.21.15.44:0/4195338247 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f675c1a9390 con 0x7f675c106430 2024-05-27T09:44:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.948+0000 7f676841c740 1 Processor -- start 2024-05-27T09:44:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f676841c740 1 -- start start 2024-05-27T09:44:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f676841c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0196e1b0 0x563a01d10a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f676841c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a019452f0 0x563a01d10fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f676841c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0194d4d0 0x563a01d114e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f676841c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563a01c81b90 con 0x563a0196e1b0 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f676841c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563a01c71730 con 0x563a0194d4d0 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f676841c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563a01c75770 con 0x563a019452f0 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f6759e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0194d4d0 0x563a01d114e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f675b200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0196e1b0 0x563a01d10a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f6759e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0194d4d0 0x563a01d114e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50560/0 (socket says 172.21.15.44:50560) 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f6759e00640 1 -- 172.21.15.44:0/3544185316 learned_addr learned my addr 172.21.15.44:0/3544185316 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.949+0000 7f6753400640 1 -- 172.21.15.44:0/4195338247 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f674c01a070 con 0x7f675c106430 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.950+0000 7f6759e00640 1 -- 172.21.15.44:0/3544185316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a019452f0 msgr2=0x563a01d10fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.950+0000 7f6759e00640 1 --2- 172.21.15.44:0/3544185316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a019452f0 0x563a01d10fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.950+0000 7f6759e00640 1 -- 172.21.15.44:0/3544185316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0196e1b0 msgr2=0x563a01d10a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.950+0000 7f6759e00640 1 --2- 172.21.15.44:0/3544185316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0196e1b0 0x563a01d10a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.950+0000 7f6759e00640 1 -- 172.21.15.44:0/3544185316 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a01c0a8b0 con 0x563a0194d4d0 2024-05-27T09:44:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.950+0000 7f6759e00640 1 --2- 172.21.15.44:0/3544185316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0194d4d0 0x563a01d114e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6748009900 tx=0x7f6748000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd59660ffb14800a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.950+0000 7f6747400640 1 -- 172.21.15.44:0/3544185316 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748040070 con 0x563a0194d4d0 2024-05-27T09:44:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.950+0000 7f6747400640 1 -- 172.21.15.44:0/3544185316 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67480317a0 con 0x563a0194d4d0 2024-05-27T09:44:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.950+0000 7f6747400640 1 -- 172.21.15.44:0/3544185316 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748035040 con 0x563a0194d4d0 2024-05-27T09:44:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.951+0000 7f6753400640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f67400b2950 0x7f67400b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.951+0000 7f676841c740 1 -- 172.21.15.44:0/3544185316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0194d4d0 msgr2=0x563a01d114e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.951+0000 7f676841c740 1 --2- 172.21.15.44:0/3544185316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0194d4d0 0x563a01d114e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6748009900 tx=0x7f6748000bc0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.951+0000 7f6753400640 1 -- 172.21.15.44:0/4195338247 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f674c0e2160 con 0x7f675c106430 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.951+0000 7f676841c740 1 -- 172.21.15.44:0/3544185316 shutdown_connections 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.951+0000 7f676841c740 1 --2- 172.21.15.44:0/3544185316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0194d4d0 0x563a01d114e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.951+0000 7f676841c740 1 --2- 172.21.15.44:0/3544185316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a019452f0 0x563a01d10fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.951+0000 7f676841c740 1 --2- 172.21.15.44:0/3544185316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0196e1b0 0x563a01d10a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.951+0000 7f676841c740 1 -- 172.21.15.44:0/3544185316 >> 172.21.15.44:0/3544185316 conn(0x563a01962300 msgr2=0x563a01d0d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.951+0000 7f675b200640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f67400b2950 0x7f67400b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.952+0000 7f676841c740 1 -- 172.21.15.44:0/3544185316 shutdown_connections 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.952+0000 7f676841c740 1 -- 172.21.15.44:0/3544185316 wait complete. 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.952+0000 7f675b200640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f67400b2950 0x7f67400b4e60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x563a01d0dbd0 tx=0x7f6754009500 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.952+0000 7f676841c740 1 -- start start 2024-05-27T09:44:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.953+0000 7f676841c740 1 Processor -- start 2024-05-27T09:44:30.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.953+0000 7f676841c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 0x563a01d1ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.953+0000 7f676841c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d1b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.953+0000 7f676841c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0196e1b0 0x563a01d26e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.953+0000 7f676841c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563a01957790 con 0x563a0194d4d0 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.953+0000 7f676841c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563a01c19bc0 con 0x563a019452f0 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.953+0000 7f676841c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563a01d1db30 con 0x563a0196e1b0 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.953+0000 7f676841c740 10 client.0 _ll_register_callbacks cb 0x563a01bf7400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f6759e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d1b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f675a800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 0x563a01d1ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f675b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0196e1b0 0x563a01d26e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f675a800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 0x563a01d1ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50566/0 (socket says 172.21.15.44:50566) 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f675b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0196e1b0 0x563a01d26e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:58030/0 (socket says 172.21.15.44:58030) 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f675a800640 1 -- 172.21.15.44:0/1305413020 learned_addr learned my addr 172.21.15.44:0/1305413020 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f6759e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d1b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51360/0 (socket says 172.21.15.44:51360) 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f675a800640 1 -- 172.21.15.44:0/1305413020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0196e1b0 msgr2=0x563a01d26e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f675a800640 1 --2- 172.21.15.44:0/1305413020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0196e1b0 0x563a01d26e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f675a800640 1 -- 172.21.15.44:0/1305413020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 msgr2=0x563a01d1b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f675a800640 1 --2- 172.21.15.44:0/1305413020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d1b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f675a800640 1 -- 172.21.15.44:0/1305413020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a01986210 con 0x563a019452f0 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f675a800640 1 --2- 172.21.15.44:0/1305413020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 0x563a01d1ac40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f674c0a9370 tx=0x7f674c0a8b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcdd576d9bbc02d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f673be00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.954+0000 7f673be00640 1 -- 172.21.15.44:0/1305413020 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c021050 con 0x563a019452f0 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f673be00640 1 -- 172.21.15.44:0/1305413020 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f674c0e4110 con 0x563a019452f0 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f676841c740 1 -- 172.21.15.44:0/1305413020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67480317a0 con 0x563a019452f0 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f673be00640 1 -- 172.21.15.44:0/1305413020 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f674c0dfce0 con 0x563a019452f0 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f676841c740 10 client.26930 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f676841c740 1 -- 172.21.15.44:0/1305413020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563a01a52620 con 0x563a019452f0 2024-05-27T09:44:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f673be00640 1 -- 172.21.15.44:0/1305413020 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67480317a0 con 0x563a019452f0 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f673be00640 1 -- 172.21.15.44:0/1305413020 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f674c0a3b00 con 0x563a019452f0 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f676841c740 10 client.26930 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f676841c740 10 client.26930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f676841c740 20 client.26930 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f676841c740 1 --2- 172.21.15.44:0/1305413020 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563a0192cd80 0x563a01d2cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f676841c740 4 client.26930 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.955+0000 7f676841c740 1 -- 172.21.15.44:0/1305413020 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a01c83ae0 con 0x563a0192cd80 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.956+0000 7f6759e00640 1 --2- 172.21.15.44:0/1305413020 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563a0192cd80 0x563a01d2cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.956+0000 7f6759e00640 1 --2- 172.21.15.44:0/1305413020 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563a0192cd80 0x563a01d2cde0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cb279c6ffe95dcb server_cookie=125d04e46864ac0a in_seq=0 out_seq=0 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.956+0000 7f673be00640 10 client.26930 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.957+0000 7f673be00640 1 -- 172.21.15.44:0/1305413020 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563a01c83ae0 con 0x563a0192cd80 2024-05-27T09:44:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.957+0000 7f673be00640 10 client.26930 handle_command_reply: tid=0 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.957+0000 7f676841c740 1 client.26930 shutdown 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.957+0000 7f676841c740 1 -- 172.21.15.44:0/1305413020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 msgr2=0x563a01d1ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.957+0000 7f676841c740 1 --2- 172.21.15.44:0/1305413020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 0x563a01d1ac40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f674c0a9370 tx=0x7f674c0a8b50 comp rx=0 tx=0).stop 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.957+0000 7f676841c740 1 -- 172.21.15.44:0/1305413020 shutdown_connections 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.957+0000 7f676841c740 1 --2- 172.21.15.44:0/1305413020 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563a0192cd80 0x563a01d2cde0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.957+0000 7f673be00640 0 client.26930 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.957+0000 7f676841c740 1 --2- 172.21.15.44:0/1305413020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0196e1b0 0x563a01d26e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.958+0000 7f676841c740 1 --2- 172.21.15.44:0/1305413020 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d1b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.958+0000 7f676841c740 1 --2- 172.21.15.44:0/1305413020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 0x563a01d1ac40 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.958+0000 7f676841c740 1 -- 172.21.15.44:0/1305413020 >> 172.21.15.44:0/1305413020 conn(0x563a01962300 msgr2=0x563a01d12780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.958+0000 7f676841c740 1 -- 172.21.15.44:0/1305413020 shutdown_connections 2024-05-27T09:44:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.958+0000 7f676841c740 1 -- 172.21.15.44:0/1305413020 wait complete. 2024-05-27T09:44:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.958+0000 7f676841c740 20 client.26930 trim_cache size 0 max 16384 2024-05-27T09:44:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.977+0000 7f676841c740 1 Processor -- start 2024-05-27T09:44:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.977+0000 7f676841c740 1 -- start start 2024-05-27T09:44:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.977+0000 7f676841c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0192cd80 0x563a01d1d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.977+0000 7f676841c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a019452f0 0x563a01d16440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.977+0000 7f676841c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d26bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.977+0000 7f676841c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b43c0 con 0x563a0194d4d0 2024-05-27T09:44:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.977+0000 7f676841c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b1c90 con 0x563a0192cd80 2024-05-27T09:44:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.977+0000 7f676841c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1afb00 con 0x563a019452f0 2024-05-27T09:44:30.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f675a800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0192cd80 0x563a01d1d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f6759e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a019452f0 0x563a01d16440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f675a800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0192cd80 0x563a01d1d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:50580/0 (socket says 172.21.15.44:50580) 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f675a800640 1 -- 172.21.15.44:0/2760491662 learned_addr learned my addr 172.21.15.44:0/2760491662 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f675b200640 1 --2- 172.21.15.44:0/2760491662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d26bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f675a800640 1 -- 172.21.15.44:0/2760491662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a019452f0 msgr2=0x563a01d16440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f675a800640 1 --2- 172.21.15.44:0/2760491662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a019452f0 0x563a01d16440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f675a800640 1 -- 172.21.15.44:0/2760491662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 msgr2=0x563a01d26bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f675a800640 1 --2- 172.21.15.44:0/2760491662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d26bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f675a800640 1 -- 172.21.15.44:0/2760491662 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a01d1ba20 con 0x563a0192cd80 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f675b200640 1 --2- 172.21.15.44:0/2760491662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d26bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.978+0000 7f675a800640 1 --2- 172.21.15.44:0/2760491662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0192cd80 0x563a01d1d370 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f674c0e20d0 tx=0x7f674c0a8c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d7ebb15e05fbcdc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.979+0000 7f6746a00640 1 -- 172.21.15.44:0/2760491662 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c014020 con 0x563a0192cd80 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.979+0000 7f6746a00640 1 -- 172.21.15.44:0/2760491662 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f674c0b2240 con 0x563a0192cd80 2024-05-27T09:44:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.979+0000 7f6746a00640 1 -- 172.21.15.44:0/2760491662 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c021070 con 0x563a0192cd80 2024-05-27T09:44:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.980+0000 7f676841c740 1 -- 172.21.15.44:0/2760491662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0192cd80 msgr2=0x563a01d1d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.980+0000 7f676841c740 1 --2- 172.21.15.44:0/2760491662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0192cd80 0x563a01d1d370 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f674c0e20d0 tx=0x7f674c0a8c80 comp rx=0 tx=0).stop 2024-05-27T09:44:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.980+0000 7f676841c740 1 -- 172.21.15.44:0/2760491662 shutdown_connections 2024-05-27T09:44:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.980+0000 7f676841c740 1 --2- 172.21.15.44:0/2760491662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d26bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.980+0000 7f676841c740 1 --2- 172.21.15.44:0/2760491662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a019452f0 0x563a01d16440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.980+0000 7f676841c740 1 --2- 172.21.15.44:0/2760491662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a0192cd80 0x563a01d1d370 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.980+0000 7f676841c740 1 -- 172.21.15.44:0/2760491662 >> 172.21.15.44:0/2760491662 conn(0x563a01962300 msgr2=0x563a01d1af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.980+0000 7f676841c740 1 -- 172.21.15.44:0/2760491662 shutdown_connections 2024-05-27T09:44:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.980+0000 7f676841c740 1 -- 172.21.15.44:0/2760491662 wait complete. 2024-05-27T09:44:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.980+0000 7f676841c740 1 -- start start 2024-05-27T09:44:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f676841c740 1 Processor -- start 2024-05-27T09:44:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f676841c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0192cd80 0x563a01d23180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f676841c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 0x563a01d1e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f676841c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d1ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f676841c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b0150 con 0x563a0194d4d0 2024-05-27T09:44:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f676841c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b15c0 con 0x563a019452f0 2024-05-27T09:44:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f676841c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b0820 con 0x563a0192cd80 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f676841c740 10 client.0 _ll_register_callbacks cb 0x7f674c010b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f675a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0192cd80 0x563a01d23180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f675a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0192cd80 0x563a01d23180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:58048/0 (socket says 172.21.15.44:58048) 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f675a800640 1 -- 172.21.15.44:0/319074904 learned_addr learned my addr 172.21.15.44:0/319074904 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.981+0000 7f6759e00640 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 0x563a01d1e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f675b200640 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d1ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f675a800640 1 -- 172.21.15.44:0/319074904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 msgr2=0x563a01d1e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f675a800640 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 0x563a01d1e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f675a800640 1 -- 172.21.15.44:0/319074904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 msgr2=0x563a01d1ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f675a800640 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d1ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f675a800640 1 -- 172.21.15.44:0/319074904 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a01d1ef50 con 0x563a0192cd80 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f6759e00640 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 0x563a01d1e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f675a800640 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0192cd80 0x563a01d23180 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f674c00d740 tx=0x7f674c0a8b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e441eb84b6969e12 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f673be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f673be00640 1 -- 172.21.15.44:0/319074904 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c014070 con 0x563a0192cd80 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f676841c740 1 -- 172.21.15.44:0/319074904 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f674c0b2240 con 0x563a0192cd80 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.982+0000 7f673be00640 1 -- 172.21.15.44:0/319074904 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f674c0a5030 con 0x563a0192cd80 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.983+0000 7f673be00640 1 -- 172.21.15.44:0/319074904 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f674c021070 con 0x563a0192cd80 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.983+0000 7f676841c740 10 client.26917 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.983+0000 7f676841c740 1 -- 172.21.15.44:0/319074904 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563a01d1ef50 con 0x563a0192cd80 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.983+0000 7f673be00640 1 -- 172.21.15.44:0/319074904 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f674c0b2240 con 0x563a0192cd80 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.983+0000 7f673be00640 1 -- 172.21.15.44:0/319074904 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f674c0a81d0 con 0x563a0192cd80 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.983+0000 7f676841c740 10 client.26917 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.983+0000 7f676841c740 10 client.26917 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.983+0000 7f676841c740 20 client.26917 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.983+0000 7f676841c740 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563a0196e1b0 0x563a01d97310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.983+0000 7f676841c740 4 client.26917 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:44:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.983+0000 7f676841c740 1 -- 172.21.15.44:0/319074904 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563a01c7d2c0 con 0x563a0196e1b0 2024-05-27T09:44:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.984+0000 7f6759e00640 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563a0196e1b0 0x563a01d97310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.984+0000 7f6759e00640 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563a0196e1b0 0x563a01d97310 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2af66792023a5592 server_cookie=3db61f9088ccd3de in_seq=0 out_seq=0 2024-05-27T09:44:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:30.984+0000 7f673be00640 10 client.26917 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:31.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.033+0000 7f673be00640 1 -- 172.21.15.44:0/319074904 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x563a01c7d2c0 con 0x563a0196e1b0 2024-05-27T09:44:31.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.033+0000 7f673be00640 10 client.26917 handle_command_reply: tid=0 2024-05-27T09:44:31.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.033+0000 7f676841c740 1 client.26917 shutdown 2024-05-27T09:44:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.034+0000 7f676841c740 1 -- 172.21.15.44:0/319074904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0192cd80 msgr2=0x563a01d23180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.034+0000 7f676841c740 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0192cd80 0x563a01d23180 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f674c00d740 tx=0x7f674c0a8b50 comp rx=0 tx=0).stop 2024-05-27T09:44:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.034+0000 7f676841c740 1 -- 172.21.15.44:0/319074904 shutdown_connections 2024-05-27T09:44:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.034+0000 7f676841c740 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563a0196e1b0 0x563a01d97310 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.034+0000 7f676841c740 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a0194d4d0 0x563a01d1ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.034+0000 7f673be00640 0 client.26917 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.034+0000 7f676841c740 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563a019452f0 0x563a01d1e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.034+0000 7f676841c740 1 --2- 172.21.15.44:0/319074904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a0192cd80 0x563a01d23180 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.034+0000 7f676841c740 1 -- 172.21.15.44:0/319074904 >> 172.21.15.44:0/319074904 conn(0x563a01962300 msgr2=0x563a01d1aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.034+0000 7f676841c740 1 -- 172.21.15.44:0/319074904 shutdown_connections 2024-05-27T09:44:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.034+0000 7f676841c740 1 -- 172.21.15.44:0/319074904 wait complete. 2024-05-27T09:44:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.035+0000 7f676841c740 20 client.26917 trim_cache size 0 max 16384 2024-05-27T09:44:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:44:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:44:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:44:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.036+0000 7f6760a00640 1 -- 172.21.15.44:0/4195338247 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f67400b2950 msgr2=0x7f67400b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.036+0000 7f6760a00640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f67400b2950 0x7f67400b4e60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x563a01d0dbd0 tx=0x7f6754009500 comp rx=0 tx=0).stop 2024-05-27T09:44:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.036+0000 7f6760a00640 1 -- 172.21.15.44:0/4195338247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f675c106430 msgr2=0x7f675c1a25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.036+0000 7f6760a00640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f675c106430 0x7f675c1a25f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f674c00d770 tx=0x7f674c00dc30 comp rx=0 tx=0).stop 2024-05-27T09:44:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.036+0000 7f6760a00640 1 -- 172.21.15.44:0/4195338247 shutdown_connections 2024-05-27T09:44:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.036+0000 7f6760a00640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f67400b2950 0x7f67400b4e60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.036+0000 7f6760a00640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f675c10c180 0x7f675c19c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.036+0000 7f6760a00640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f675c10bce0 0x7f675c1a2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.036+0000 7f6760a00640 1 --2- 172.21.15.44:0/4195338247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f675c106430 0x7f675c1a25f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.036+0000 7f6760a00640 1 -- 172.21.15.44:0/4195338247 >> 172.21.15.44:0/4195338247 conn(0x7f675c101380 msgr2=0x7f675c101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.037+0000 7f6760a00640 1 -- 172.21.15.44:0/4195338247 shutdown_connections 2024-05-27T09:44:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:31.037+0000 7f6760a00640 1 -- 172.21.15.44:0/4195338247 wait complete. 2024-05-27T09:44:31.975 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:31 smithi044 ceph-mon[31618]: pgmap v289: 97 pgs: 97 active+clean; 269 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 1.67k op/s 2024-05-27T09:44:32.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.241+0000 7f1e21c00700 1 -- 172.21.15.44:0/67986556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c106eb0 msgr2=0x7f1e1c113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:32.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.241+0000 7f1e21c00700 1 --2- 172.21.15.44:0/67986556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c106eb0 0x7f1e1c113340 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400b0a0 tx=0x7f1e1400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:44:32.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.241+0000 7f1e21c00700 1 -- 172.21.15.44:0/67986556 shutdown_connections 2024-05-27T09:44:32.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.241+0000 7f1e21c00700 1 --2- 172.21.15.44:0/67986556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c106eb0 0x7f1e1c113340 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:32.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.241+0000 7f1e21c00700 1 --2- 172.21.15.44:0/67986556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c10c5a0 0x7f1e1c10c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:32.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.241+0000 7f1e21c00700 1 --2- 172.21.15.44:0/67986556 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c106560 0x7f1e1c106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:32.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.241+0000 7f1e21c00700 1 -- 172.21.15.44:0/67986556 >> 172.21.15.44:0/67986556 conn(0x7f1e1c102040 msgr2=0x7f1e1c104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:32.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.241+0000 7f1e21c00700 1 -- 172.21.15.44:0/67986556 shutdown_connections 2024-05-27T09:44:32.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.241+0000 7f1e21c00700 1 -- 172.21.15.44:0/67986556 wait complete. 2024-05-27T09:44:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.242+0000 7f1e21c00700 1 Processor -- start 2024-05-27T09:44:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.242+0000 7f1e21c00700 1 -- start start 2024-05-27T09:44:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.243+0000 7f1e21c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c106560 0x7f1e1c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.243+0000 7f1e21c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106eb0 0x7f1e1c19d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.243+0000 7f1e21c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10c5a0 0x7f1e1c1a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.243+0000 7f1e21c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c116720 con 0x7f1e1c10c5a0 2024-05-27T09:44:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.243+0000 7f1e21c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c1165a0 con 0x7f1e1c106eb0 2024-05-27T09:44:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.243+0000 7f1e21c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c1168a0 con 0x7f1e1c106560 2024-05-27T09:44:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.243+0000 7f1e1aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c106560 0x7f1e1c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.243+0000 7f1e1a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106eb0 0x7f1e1c19d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.243+0000 7f1e1aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c106560 0x7f1e1c19cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:58074/0 (socket says 172.21.15.44:58074) 2024-05-27T09:44:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.243+0000 7f1e1aa00700 1 -- 172.21.15.44:0/1564316173 learned_addr learned my addr 172.21.15.44:0/1564316173 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:32.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.244+0000 7f1e1b400700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10c5a0 0x7f1e1c1a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.244+0000 7f1e1a000700 1 -- 172.21.15.44:0/1564316173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c106560 msgr2=0x7f1e1c19cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.244+0000 7f1e1a000700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c106560 0x7f1e1c19cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.244+0000 7f1e1a000700 1 -- 172.21.15.44:0/1564316173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10c5a0 msgr2=0x7f1e1c1a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.244+0000 7f1e1a000700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10c5a0 0x7f1e1c1a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.244+0000 7f1e1a000700 1 -- 172.21.15.44:0/1564316173 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1c1a1ea0 con 0x7f1e1c106eb0 2024-05-27T09:44:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.244+0000 7f1e1aa00700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c106560 0x7f1e1c19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.244+0000 7f1e1a000700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106eb0 0x7f1e1c19d570 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c075b70 tx=0x7f1e0800eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.244+0000 7f1e13400700 1 -- 172.21.15.44:0/1564316173 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0800cbd0 con 0x7f1e1c106eb0 2024-05-27T09:44:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.245+0000 7f1e21c00700 1 -- 172.21.15.44:0/1564316173 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e1c1a2190 con 0x7f1e1c106eb0 2024-05-27T09:44:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.245+0000 7f1e13400700 1 -- 172.21.15.44:0/1564316173 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e0800ee80 con 0x7f1e1c106eb0 2024-05-27T09:44:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.245+0000 7f1e13400700 1 -- 172.21.15.44:0/1564316173 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e08018680 con 0x7f1e1c106eb0 2024-05-27T09:44:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.245+0000 7f1e21c00700 1 -- 172.21.15.44:0/1564316173 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e1c1a9ab0 con 0x7f1e1c106eb0 2024-05-27T09:44:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.246+0000 7f1e21c00700 1 -- 172.21.15.44:0/1564316173 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e1c0512b0 con 0x7f1e1c106eb0 2024-05-27T09:44:32.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.249+0000 7f1e13400700 1 -- 172.21.15.44:0/1564316173 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f1e08010430 con 0x7f1e1c106eb0 2024-05-27T09:44:32.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.251+0000 7f1e13400700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b29f0 0x7f1e000b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:32.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.251+0000 7f1e13400700 1 -- 172.21.15.44:0/1564316173 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f1e080e2b60 con 0x7f1e1c106eb0 2024-05-27T09:44:32.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.251+0000 7f1e1aa00700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b29f0 0x7f1e000b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:32.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.252+0000 7f1e1aa00700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b29f0 0x7f1e000b4f00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c011550 tx=0x7f1e0c009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:32.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.256+0000 7f1e13400700 1 -- 172.21.15.44:0/1564316173 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1e080aa4e0 con 0x7f1e1c106eb0 2024-05-27T09:44:32.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:31 smithi050 ceph-mon[36371]: pgmap v289: 97 pgs: 97 active+clean; 269 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 1.67k op/s 2024-05-27T09:44:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:31 smithi189 ceph-mon[36671]: pgmap v289: 97 pgs: 97 active+clean; 269 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 1.67k op/s 2024-05-27T09:44:32.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.549+0000 7f1e21c00700 1 -- 172.21.15.44:0/1564316173 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e1c02e5e0 con 0x7f1e1c106eb0 2024-05-27T09:44:32.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.550+0000 7f1e13400700 1 -- 172.21.15.44:0/1564316173 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11865 (secure 0 0 0) 0x7f1e080a9de0 con 0x7f1e1c106eb0 2024-05-27T09:44:32.560 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:44:32.560 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:44:10.855246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:44:32.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 -- 172.21.15.44:0/1564316173 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b29f0 msgr2=0x7f1e000b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:32.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b29f0 0x7f1e000b4f00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c011550 tx=0x7f1e0c009260 comp rx=0 tx=0).stop 2024-05-27T09:44:32.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 -- 172.21.15.44:0/1564316173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106eb0 msgr2=0x7f1e1c19d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:32.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106eb0 0x7f1e1c19d570 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c075b70 tx=0x7f1e0800eca0 comp rx=0 tx=0).stop 2024-05-27T09:44:32.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 -- 172.21.15.44:0/1564316173 shutdown_connections 2024-05-27T09:44:32.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1e000b29f0 0x7f1e000b4f00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:32.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1c10c5a0 0x7f1e1c1a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:32.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c106560 0x7f1e1c19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:32.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 --2- 172.21.15.44:0/1564316173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e1c106eb0 0x7f1e1c19d570 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:32.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 -- 172.21.15.44:0/1564316173 >> 172.21.15.44:0/1564316173 conn(0x7f1e1c102040 msgr2=0x7f1e1c103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:32.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 -- 172.21.15.44:0/1564316173 shutdown_connections 2024-05-27T09:44:32.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:32.554+0000 7f1e21c00700 1 -- 172.21.15.44:0/1564316173 wait complete. 2024-05-27T09:44:32.563 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 32 2024-05-27T09:44:33.035 INFO:tasks.quiescer.fs.[cephfs]:Total duration (37.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-05-27T09:44:33.036 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.95 2024-05-27T09:44:33.036 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:44:33.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:32 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1564316173' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:44:33.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:32 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1564316173' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:44:33.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:32 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1564316173' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:44:33.409 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:44:33.534 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:44:33.943 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:33 smithi044 ceph-mon[31618]: pgmap v290: 97 pgs: 97 active+clean; 269 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 1.50k op/s 2024-05-27T09:44:34.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:33 smithi050 ceph-mon[36371]: pgmap v290: 97 pgs: 97 active+clean; 269 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 1.50k op/s 2024-05-27T09:44:34.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:33 smithi189 ceph-mon[36671]: pgmap v290: 97 pgs: 97 active+clean; 269 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 1.50k op/s 2024-05-27T09:44:34.904 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:44:35.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:35 smithi044 ceph-mon[31618]: pgmap v291: 97 pgs: 97 active+clean; 269 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 1.50k op/s 2024-05-27T09:44:36.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:35 smithi050 ceph-mon[36371]: pgmap v291: 97 pgs: 97 active+clean; 269 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 1.50k op/s 2024-05-27T09:44:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:35 smithi189 ceph-mon[36671]: pgmap v291: 97 pgs: 97 active+clean; 269 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 1.50k op/s 2024-05-27T09:44:36.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.720+0000 7f0f7e000700 1 -- 172.21.15.44:0/3237389514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f78102f40 msgr2=0x7f0f78103350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:36.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.720+0000 7f0f7e000700 1 --2- 172.21.15.44:0/3237389514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f78102f40 0x7f0f78103350 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009990 tx=0x7f0f6c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:44:36.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.721+0000 7f0f7e000700 1 -- 172.21.15.44:0/3237389514 shutdown_connections 2024-05-27T09:44:36.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.721+0000 7f0f7e000700 1 --2- 172.21.15.44:0/3237389514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f78110af0 0x7f0f78112f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:36.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.721+0000 7f0f7e000700 1 --2- 172.21.15.44:0/3237389514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f78102f40 0x7f0f78103350 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:36.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.721+0000 7f0f7e000700 1 --2- 172.21.15.44:0/3237389514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f78103890 0x7f0f781105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:36.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.721+0000 7f0f7e000700 1 -- 172.21.15.44:0/3237389514 >> 172.21.15.44:0/3237389514 conn(0x7f0f78078ab0 msgr2=0x7f0f78078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:36.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.721+0000 7f0f7e000700 1 -- 172.21.15.44:0/3237389514 shutdown_connections 2024-05-27T09:44:36.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.721+0000 7f0f7e000700 1 -- 172.21.15.44:0/3237389514 wait complete. 2024-05-27T09:44:36.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.722+0000 7f0f7e000700 1 Processor -- start 2024-05-27T09:44:36.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.722+0000 7f0f7e000700 1 -- start start 2024-05-27T09:44:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f7e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f78102f40 0x7f0f7819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f7e000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f78103890 0x7f0f7819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f7e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f78110af0 0x7f0f781a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f7e000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0f781164e0 con 0x7f0f78102f40 2024-05-27T09:44:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f7e000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0f78116360 con 0x7f0f78103890 2024-05-27T09:44:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f7e000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0f78116660 con 0x7f0f78110af0 2024-05-27T09:44:36.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f7cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f78102f40 0x7f0f7819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:36.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f7d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f78110af0 0x7f0f781a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:36.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f7d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f78110af0 0x7f0f781a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53084/0 (socket says 172.21.15.44:53084) 2024-05-27T09:44:36.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f7cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f78102f40 0x7f0f7819ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40324/0 (socket says 172.21.15.44:40324) 2024-05-27T09:44:36.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f7d600700 1 -- 172.21.15.44:0/2712615667 learned_addr learned my addr 172.21.15.44:0/2712615667 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:36.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.723+0000 7f0f77e00700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f78103890 0x7f0f7819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:36.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.724+0000 7f0f7d600700 1 -- 172.21.15.44:0/2712615667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f78103890 msgr2=0x7f0f7819d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:36.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.724+0000 7f0f7d600700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f78103890 0x7f0f7819d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:36.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.724+0000 7f0f7d600700 1 -- 172.21.15.44:0/2712615667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f78102f40 msgr2=0x7f0f7819ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:36.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.724+0000 7f0f7d600700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f78102f40 0x7f0f7819ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:36.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.724+0000 7f0f7d600700 1 -- 172.21.15.44:0/2712615667 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f781a1d20 con 0x7f0f78110af0 2024-05-27T09:44:36.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.724+0000 7f0f77e00700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f78103890 0x7f0f7819d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:36.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.724+0000 7f0f7d600700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f78110af0 0x7f0f781a1680 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000ef30 tx=0x7f0f7000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:36.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.724+0000 7f0f75600700 1 -- 172.21.15.44:0/2712615667 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f700092e0 con 0x7f0f78110af0 2024-05-27T09:44:36.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.725+0000 7f0f7e000700 1 -- 172.21.15.44:0/2712615667 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f781a1fb0 con 0x7f0f78110af0 2024-05-27T09:44:36.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.725+0000 7f0f75600700 1 -- 172.21.15.44:0/2712615667 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f70004bb0 con 0x7f0f78110af0 2024-05-27T09:44:36.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.725+0000 7f0f75600700 1 -- 172.21.15.44:0/2712615667 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f70004020 con 0x7f0f78110af0 2024-05-27T09:44:36.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.725+0000 7f0f7e000700 1 -- 172.21.15.44:0/2712615667 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f781a9930 con 0x7f0f78110af0 2024-05-27T09:44:36.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.726+0000 7f0f7e000700 1 -- 172.21.15.44:0/2712615667 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f780512b0 con 0x7f0f78110af0 2024-05-27T09:44:36.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.728+0000 7f0f75600700 1 -- 172.21.15.44:0/2712615667 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0f7001b030 con 0x7f0f78110af0 2024-05-27T09:44:36.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.730+0000 7f0f75600700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f600b29f0 0x7f0f600b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:36.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.730+0000 7f0f75600700 1 -- 172.21.15.44:0/2712615667 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f0f700e2240 con 0x7f0f78110af0 2024-05-27T09:44:36.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.730+0000 7f0f7cc00700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f600b29f0 0x7f0f600b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:36.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.731+0000 7f0f7cc00700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f600b29f0 0x7f0f600b4f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00b2e0 tx=0x7f0f6c005fd0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:36.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:36.732+0000 7f0f75600700 1 -- 172.21.15.44:0/2712615667 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0f70022020 con 0x7f0f78110af0 2024-05-27T09:44:37.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.047+0000 7f0f7e000700 1 -- 172.21.15.44:0/2712615667 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f7802e5e0 con 0x7f0f78110af0 2024-05-27T09:44:37.054 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:44:37.054 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:44:10.855246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:44:37.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.051+0000 7f0f75600700 1 -- 172.21.15.44:0/2712615667 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11865 (secure 0 0 0) 0x7f0f70010040 con 0x7f0f78110af0 2024-05-27T09:44:37.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.061+0000 7f0f7e000700 1 -- 172.21.15.44:0/2712615667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f600b29f0 msgr2=0x7f0f600b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:37.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.061+0000 7f0f7e000700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f600b29f0 0x7f0f600b4f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00b2e0 tx=0x7f0f6c005fd0 comp rx=0 tx=0).stop 2024-05-27T09:44:37.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.061+0000 7f0f7e000700 1 -- 172.21.15.44:0/2712615667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f78110af0 msgr2=0x7f0f781a1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:37.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.061+0000 7f0f7e000700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f78110af0 0x7f0f781a1680 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000ef30 tx=0x7f0f7000c3a0 comp rx=0 tx=0).stop 2024-05-27T09:44:37.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.062+0000 7f0f7e000700 1 -- 172.21.15.44:0/2712615667 shutdown_connections 2024-05-27T09:44:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.062+0000 7f0f7e000700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f600b29f0 0x7f0f600b4f00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.062+0000 7f0f7e000700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f78110af0 0x7f0f781a1680 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:37.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.062+0000 7f0f7e000700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f78102f40 0x7f0f7819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.062+0000 7f0f7e000700 1 --2- 172.21.15.44:0/2712615667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f78103890 0x7f0f7819d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.062+0000 7f0f7e000700 1 -- 172.21.15.44:0/2712615667 >> 172.21.15.44:0/2712615667 conn(0x7f0f78078ab0 msgr2=0x7f0f78111de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.064+0000 7f0f7e000700 1 -- 172.21.15.44:0/2712615667 shutdown_connections 2024-05-27T09:44:37.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:37.065+0000 7f0f7e000700 1 -- 172.21.15.44:0/2712615667 wait complete. 2024-05-27T09:44:37.069 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 32 2024-05-27T09:44:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.112+0000 7ffab0e00700 1 -- 172.21.15.44:0/1398713985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffaac10af40 msgr2=0x7ffaac10b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.112+0000 7ffab0e00700 1 --2- 172.21.15.44:0/1398713985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffaac10af40 0x7ffaac10b350 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ffa940099c0 tx=0x7ffa94009cd0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.113+0000 7ffab0e00700 1 -- 172.21.15.44:0/1398713985 shutdown_connections 2024-05-27T09:44:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.113+0000 7ffab0e00700 1 --2- 172.21.15.44:0/1398713985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac10b890 0x7ffaac1131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.113+0000 7ffab0e00700 1 --2- 172.21.15.44:0/1398713985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaac106610 0x7ffaac106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.113+0000 7ffab0e00700 1 --2- 172.21.15.44:0/1398713985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffaac10af40 0x7ffaac10b350 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.113+0000 7ffab0e00700 1 -- 172.21.15.44:0/1398713985 >> 172.21.15.44:0/1398713985 conn(0x7ffaac102020 msgr2=0x7ffaac104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.113+0000 7ffab0e00700 1 -- 172.21.15.44:0/1398713985 shutdown_connections 2024-05-27T09:44:38.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:37 smithi044 ceph-mon[31618]: pgmap v292: 97 pgs: 97 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 1.50k op/s 2024-05-27T09:44:38.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:37 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2712615667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:44:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.113+0000 7ffab0e00700 1 -- 172.21.15.44:0/1398713985 wait complete. 2024-05-27T09:44:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.114+0000 7ffab0e00700 1 Processor -- start 2024-05-27T09:44:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.115+0000 7ffab0e00700 1 -- start start 2024-05-27T09:44:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.115+0000 7ffab0e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac106610 0x7ffaac110610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.115+0000 7ffab0e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffaac10af40 0x7ffaac110b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.115+0000 7ffab0e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaac10b890 0x7ffaac111090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.115+0000 7ffab0e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac1164a0 con 0x7ffaac10b890 2024-05-27T09:44:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.115+0000 7ffab0e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac116320 con 0x7ffaac106610 2024-05-27T09:44:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.115+0000 7ffab0e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac116620 con 0x7ffaac10af40 2024-05-27T09:44:38.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.116+0000 7ffaab400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac106610 0x7ffaac110610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.116+0000 7ffaab400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac106610 0x7ffaac110610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45450/0 (socket says 172.21.15.44:45450) 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.116+0000 7ffaab400700 1 -- 172.21.15.44:0/3733552053 learned_addr learned my addr 172.21.15.44:0/3733552053 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.116+0000 7ffaa2a00700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffaac10af40 0x7ffaac110b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.116+0000 7ffaabe00700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaac10b890 0x7ffaac111090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.116+0000 7ffaab400700 1 -- 172.21.15.44:0/3733552053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffaac10af40 msgr2=0x7ffaac110b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.116+0000 7ffaab400700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffaac10af40 0x7ffaac110b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.116+0000 7ffaab400700 1 -- 172.21.15.44:0/3733552053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaac10b890 msgr2=0x7ffaac111090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.116+0000 7ffaab400700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaac10b890 0x7ffaac111090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.116+0000 7ffaab400700 1 -- 172.21.15.44:0/3733552053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac1ad370 con 0x7ffaac106610 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.116+0000 7ffaab400700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac106610 0x7ffaac110610 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c00ef10 tx=0x7ffa9c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.117+0000 7ffaa8c00700 1 -- 172.21.15.44:0/3733552053 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c00c9c0 con 0x7ffaac106610 2024-05-27T09:44:38.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.117+0000 7ffaa8c00700 1 -- 172.21.15.44:0/3733552053 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9c019070 con 0x7ffaac106610 2024-05-27T09:44:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.117+0000 7ffaa8c00700 1 -- 172.21.15.44:0/3733552053 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c014480 con 0x7ffaac106610 2024-05-27T09:44:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.117+0000 7ffab0e00700 1 -- 172.21.15.44:0/3733552053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaac1ad600 con 0x7ffaac106610 2024-05-27T09:44:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.117+0000 7ffab0e00700 1 -- 172.21.15.44:0/3733552053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaac1adac0 con 0x7ffaac106610 2024-05-27T09:44:38.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.121+0000 7ffaa8c00700 1 -- 172.21.15.44:0/3733552053 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ffa9c012050 con 0x7ffaac106610 2024-05-27T09:44:38.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.122+0000 7ffac0f83b80 1 Processor -- start 2024-05-27T09:44:38.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.122+0000 7ffac0f83b80 1 -- start start 2024-05-27T09:44:38.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.123+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bdf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.123+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bdfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.123+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 0x55e7b5beb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.123+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e7b5b0ce40 con 0x55e7b5a5fe40 2024-05-27T09:44:38.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.123+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e7b5afc9e0 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.123+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e7b5b00a20 con 0x55e7b5be92c0 2024-05-27T09:44:38.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.123+0000 7ffaabe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bdf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.123+0000 7ffaab400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bdfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.123+0000 7ffaabe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bdf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45452/0 (socket says 172.21.15.44:45452) 2024-05-27T09:44:38.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.123+0000 7ffaabe00700 1 -- 172.21.15.44:0/1617030715 learned_addr learned my addr 172.21.15.44:0/1617030715 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:38.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.123+0000 7ffaa2a00700 1 --2- 172.21.15.44:0/1617030715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 0x55e7b5beb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.124+0000 7ffaa8c00700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ffa900b2a70 0x7ffa900b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.124+0000 7ffaabe00700 1 -- 172.21.15.44:0/1617030715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 msgr2=0x55e7b5beb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:38.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.124+0000 7ffaabe00700 1 --2- 172.21.15.44:0/1617030715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 0x55e7b5beb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.124+0000 7ffaabe00700 1 -- 172.21.15.44:0/1617030715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5a5fe40 msgr2=0x55e7b5bdfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:38.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.124+0000 7ffaabe00700 1 --2- 172.21.15.44:0/1617030715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bdfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.124+0000 7ffaabe00700 1 -- 172.21.15.44:0/1617030715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7b5ac5420 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.124+0000 7ffaab400700 1 --2- 172.21.15.44:0/1617030715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bdfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:38.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.124+0000 7ffaa8c00700 1 -- 172.21.15.44:0/3733552053 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7ffa9c0e24d0 con 0x7ffaac106610 2024-05-27T09:44:38.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.124+0000 7ffaabe00700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ffa900b2a70 0x7ffa900b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.124+0000 7ffaabe00700 1 --2- 172.21.15.44:0/1617030715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bdf750 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009cb0 tx=0x7ffaa4011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71ece9962a050476 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:38.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.125+0000 7ffaa0c00700 1 -- 172.21.15.44:0/1617030715 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4011d50 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.125+0000 7ffaa0c00700 1 -- 172.21.15.44:0/1617030715 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaa40198b0 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.125+0000 7ffaa0c00700 1 -- 172.21.15.44:0/1617030715 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa400ea00 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.126+0000 7ffac0f83b80 1 -- 172.21.15.44:0/1617030715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 msgr2=0x55e7b5bdf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:38.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.126+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/1617030715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bdf750 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009cb0 tx=0x7ffaa4011440 comp rx=0 tx=0).stop 2024-05-27T09:44:38.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.126+0000 7ffaabe00700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ffa900b2a70 0x7ffa900b4f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4019fd0 tx=0x7ffaa400ec80 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:38.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.126+0000 7ffac0f83b80 1 -- 172.21.15.44:0/1617030715 shutdown_connections 2024-05-27T09:44:38.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.126+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/1617030715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 0x55e7b5beb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.126+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/1617030715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bdf750 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.126+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/1617030715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bdfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.126+0000 7ffac0f83b80 1 -- 172.21.15.44:0/1617030715 >> 172.21.15.44:0/1617030715 conn(0x55e7b5af9390 msgr2=0x55e7b5bda710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:38.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.126+0000 7ffac0f83b80 1 -- 172.21.15.44:0/1617030715 shutdown_connections 2024-05-27T09:44:38.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.126+0000 7ffac0f83b80 1 -- 172.21.15.44:0/1617030715 wait complete. 2024-05-27T09:44:38.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.127+0000 7ffac0f83b80 1 -- start start 2024-05-27T09:44:38.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.127+0000 7ffac0f83b80 1 Processor -- start 2024-05-27T09:44:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.127+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5be6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.127+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5be66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.127+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5be92c0 0x55e7b5be6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.127+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e7b5a02fa0 con 0x55e7b5be92c0 2024-05-27T09:44:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.127+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e7b5a9d760 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.127+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e7b5bebe00 con 0x55e7b5a5fe40 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffac0f83b80 10 client.0 _ll_register_callbacks cb 0x55e7b5aa6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaab400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5be6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaa2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5be66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaab400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5be6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53132/0 (socket says 172.21.15.44:53132) 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaa2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5be66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45468/0 (socket says 172.21.15.44:45468) 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaab400700 1 -- 172.21.15.44:0/2303734822 learned_addr learned my addr 172.21.15.44:0/2303734822 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaabe00700 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5be92c0 0x55e7b5be6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaa2a00700 1 -- 172.21.15.44:0/2303734822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 msgr2=0x55e7b5be6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaa2a00700 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5be6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaa2a00700 1 -- 172.21.15.44:0/2303734822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5be92c0 msgr2=0x55e7b5be6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaa2a00700 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5be92c0 0x55e7b5be6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaa2a00700 1 -- 172.21.15.44:0/2303734822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7b5a4c8d0 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.128+0000 7ffaab400700 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5be6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.129+0000 7ffaa2a00700 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5be66c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ffa940098e0 tx=0x7ffa94003720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22b9e97e2c399f57 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.129+0000 7ffa99600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:44:38.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.129+0000 7ffa99600700 1 -- 172.21.15.44:0/2303734822 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa94003bd0 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.129+0000 7ffa99600700 1 -- 172.21.15.44:0/2303734822 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7ffa94003d70 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.129+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2303734822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaa40198b0 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.129+0000 7ffa99600700 1 -- 172.21.15.44:0/2303734822 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa94031050 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.130+0000 7ffac0f83b80 10 client.26960 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.130+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2303734822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e7b5acd060 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.130+0000 7ffa99600700 1 -- 172.21.15.44:0/2303734822 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaa40198b0 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.130+0000 7ffa99600700 1 -- 172.21.15.44:0/2303734822 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7ffa94023460 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.130+0000 7ffac0f83b80 10 client.26960 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.130+0000 7ffac0f83b80 10 client.26960 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.131+0000 7ffac0f83b80 20 client.26960 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.131+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e7b5bf8c40 0x55e7b5bfb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.131+0000 7ffac0f83b80 4 client.26960 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.131+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2303734822 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7b5b0ed90 con 0x55e7b5bf8c40 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.131+0000 7ffaab400700 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e7b5bf8c40 0x55e7b5bfb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.132+0000 7ffaab400700 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e7b5bf8c40 0x55e7b5bfb060 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3f855c8d9aea2f81 server_cookie=e24d0a0e566dcc4f in_seq=0 out_seq=0 2024-05-27T09:44:38.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.132+0000 7ffa99600700 10 client.26960 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:44:38.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffa99600700 1 -- 172.21.15.44:0/2303734822 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e7b5b0ed90 con 0x55e7b5bf8c40 2024-05-27T09:44:38.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffa99600700 10 client.26960 handle_command_reply: tid=0 2024-05-27T09:44:38.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffac0f83b80 1 client.26960 shutdown 2024-05-27T09:44:38.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2303734822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 msgr2=0x55e7b5be66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:38.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5be66c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ffa940098e0 tx=0x7ffa94003720 comp rx=0 tx=0).stop 2024-05-27T09:44:38.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2303734822 shutdown_connections 2024-05-27T09:44:38.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e7b5bf8c40 0x55e7b5bfb060 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5be92c0 0x55e7b5be6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5be6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/2303734822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5be66c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffa99600700 0 client.26960 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:44:38.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2303734822 >> 172.21.15.44:0/2303734822 conn(0x55e7b5af9390 msgr2=0x55e7b5bda260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:38.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2303734822 shutdown_connections 2024-05-27T09:44:38.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.133+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2303734822 wait complete. 2024-05-27T09:44:38.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.134+0000 7ffac0f83b80 20 client.26960 trim_cache size 0 max 16384 2024-05-27T09:44:38.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffac0f83b80 1 Processor -- start 2024-05-27T09:44:38.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffac0f83b80 1 -- start start 2024-05-27T09:44:38.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bda640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5be85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5be92c0 0x55e7b5bf4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac1b7b20 con 0x55e7b5be92c0 2024-05-27T09:44:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac1b53b0 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac1b3220 con 0x55e7b5a5fe40 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffaab400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bda640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffaa2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5be85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffaab400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bda640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53134/0 (socket says 172.21.15.44:53134) 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.151+0000 7ffaab400700 1 -- 172.21.15.44:0/2840862325 learned_addr learned my addr 172.21.15.44:0/2840862325 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.152+0000 7ffaabe00700 1 --2- 172.21.15.44:0/2840862325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5be92c0 0x55e7b5bf4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.152+0000 7ffaab400700 1 -- 172.21.15.44:0/2840862325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 msgr2=0x55e7b5be85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.152+0000 7ffaab400700 1 --2- 172.21.15.44:0/2840862325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5be85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.152+0000 7ffaab400700 1 -- 172.21.15.44:0/2840862325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5be92c0 msgr2=0x55e7b5bf4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.152+0000 7ffaab400700 1 --2- 172.21.15.44:0/2840862325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5be92c0 0x55e7b5bf4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.152+0000 7ffaab400700 1 -- 172.21.15.44:0/2840862325 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7b5bf5410 con 0x55e7b5a5fe40 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.152+0000 7ffaa2a00700 1 --2- 172.21.15.44:0/2840862325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5be85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.152+0000 7ffaabe00700 1 --2- 172.21.15.44:0/2840862325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5be92c0 0x55e7b5bf4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:38.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.152+0000 7ffaab400700 1 --2- 172.21.15.44:0/2840862325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bda640 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c0a5d50 tx=0x7ffa9c0e4a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce44a8c3c2383714 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.152+0000 7ffa9be00700 1 -- 172.21.15.44:0/2840862325 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c0a9100 con 0x55e7b5a5fe40 2024-05-27T09:44:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.153+0000 7ffa9be00700 1 -- 172.21.15.44:0/2840862325 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9c0a92c0 con 0x55e7b5a5fe40 2024-05-27T09:44:38.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.153+0000 7ffa9be00700 1 -- 172.21.15.44:0/2840862325 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c0cc110 con 0x55e7b5a5fe40 2024-05-27T09:44:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.154+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2840862325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 msgr2=0x55e7b5bda640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.154+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/2840862325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bda640 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c0a5d50 tx=0x7ffa9c0e4a20 comp rx=0 tx=0).stop 2024-05-27T09:44:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.154+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2840862325 shutdown_connections 2024-05-27T09:44:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.154+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/2840862325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5be92c0 0x55e7b5bf4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.154+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/2840862325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bda640 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.154+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/2840862325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5be85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.154+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2840862325 >> 172.21.15.44:0/2840862325 conn(0x55e7b5af9390 msgr2=0x55e7b5bd9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.154+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2840862325 shutdown_connections 2024-05-27T09:44:38.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.154+0000 7ffac0f83b80 1 -- 172.21.15.44:0/2840862325 wait complete. 2024-05-27T09:44:38.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.155+0000 7ffac0f83b80 1 -- start start 2024-05-27T09:44:38.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.155+0000 7ffac0f83b80 1 Processor -- start 2024-05-27T09:44:38.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.155+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bf10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.155+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bf1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.155+0000 7ffac0f83b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 0x55e7b5bec660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.155+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac1b3870 con 0x55e7b5a5fe40 2024-05-27T09:44:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.155+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac1b4ce0 con 0x55e7b5b4b9f0 2024-05-27T09:44:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.155+0000 7ffac0f83b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac1b3f40 con 0x55e7b5be92c0 2024-05-27T09:44:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.155+0000 7ffac0f83b80 10 client.0 _ll_register_callbacks cb 0x55e7b5aa6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaa2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bf1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaabe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 0x55e7b5bec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaab400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bf10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaabe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 0x55e7b5bec660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53148/0 (socket says 172.21.15.44:53148) 2024-05-27T09:44:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaa2a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bf1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45486/0 (socket says 172.21.15.44:45486) 2024-05-27T09:44:38.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaabe00700 1 -- 172.21.15.44:0/3897428755 learned_addr learned my addr 172.21.15.44:0/3897428755 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:38.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaabe00700 1 -- 172.21.15.44:0/3897428755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 msgr2=0x55e7b5bf1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaabe00700 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bf1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaabe00700 1 -- 172.21.15.44:0/3897428755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5a5fe40 msgr2=0x55e7b5bf10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaabe00700 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bf10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaabe00700 1 -- 172.21.15.44:0/3897428755 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7b5becd10 con 0x55e7b5be92c0 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaa2a00700 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bf1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.156+0000 7ffaabe00700 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 0x55e7b5bec660 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ffaa401d000 tx=0x7ffaa4025000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c087490be17f55cc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.157+0000 7ffa99600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.157+0000 7ffa99600700 1 -- 172.21.15.44:0/3897428755 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4035070 con 0x55e7b5be92c0 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.157+0000 7ffa99600700 1 -- 172.21.15.44:0/3897428755 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7ffaa4004050 con 0x55e7b5be92c0 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.157+0000 7ffac0f83b80 1 -- 172.21.15.44:0/3897428755 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa9c0a92c0 con 0x55e7b5be92c0 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.157+0000 7ffa99600700 1 -- 172.21.15.44:0/3897428755 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffaa400ea00 con 0x55e7b5be92c0 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.157+0000 7ffac0f83b80 10 client.26947 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:38.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.157+0000 7ffac0f83b80 1 -- 172.21.15.44:0/3897428755 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e7b5c63030 con 0x55e7b5be92c0 2024-05-27T09:44:38.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.158+0000 7ffa99600700 1 -- 172.21.15.44:0/3897428755 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9c0a92c0 con 0x55e7b5be92c0 2024-05-27T09:44:38.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.158+0000 7ffa99600700 1 -- 172.21.15.44:0/3897428755 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7ffaa4003680 con 0x55e7b5be92c0 2024-05-27T09:44:38.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.158+0000 7ffac0f83b80 10 client.26947 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:38.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.158+0000 7ffac0f83b80 10 client.26947 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:44:38.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.158+0000 7ffac0f83b80 20 client.26947 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:38.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.158+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e7b5c63600 0x55e7b5c65a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:38.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.158+0000 7ffac0f83b80 4 client.26947 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:44:38.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.158+0000 7ffac0f83b80 1 -- 172.21.15.44:0/3897428755 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7b5b08570 con 0x55e7b5c63600 2024-05-27T09:44:38.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.158+0000 7ffaab400700 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e7b5c63600 0x55e7b5c65a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.159+0000 7ffaab400700 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e7b5c63600 0x55e7b5c65a20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f4703bbf970810b9 server_cookie=501a7d2cb5b8122 in_seq=0 out_seq=0 2024-05-27T09:44:38.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.159+0000 7ffa99600700 10 client.26947 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:44:38.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffa99600700 1 -- 172.21.15.44:0/3897428755 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7b5b08570 con 0x55e7b5c63600 2024-05-27T09:44:38.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffa99600700 10 client.26947 handle_command_reply: tid=0 2024-05-27T09:44:38.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffac0f83b80 1 client.26947 shutdown 2024-05-27T09:44:38.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffac0f83b80 1 -- 172.21.15.44:0/3897428755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 msgr2=0x55e7b5bec660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:38.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 0x55e7b5bec660 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ffaa401d000 tx=0x7ffaa4025000 comp rx=0 tx=0).stop 2024-05-27T09:44:38.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffac0f83b80 1 -- 172.21.15.44:0/3897428755 shutdown_connections 2024-05-27T09:44:38.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e7b5be92c0 0x55e7b5bec660 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e7b5c63600 0x55e7b5c65a20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e7b5a5fe40 0x55e7b5bf10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffac0f83b80 1 --2- 172.21.15.44:0/3897428755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e7b5b4b9f0 0x55e7b5bf1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffa99600700 0 client.26947 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:44:38.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.166+0000 7ffac0f83b80 1 -- 172.21.15.44:0/3897428755 >> 172.21.15.44:0/3897428755 conn(0x55e7b5af9390 msgr2=0x55e7b5be4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:38.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.167+0000 7ffac0f83b80 1 -- 172.21.15.44:0/3897428755 shutdown_connections 2024-05-27T09:44:38.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.167+0000 7ffac0f83b80 1 -- 172.21.15.44:0/3897428755 wait complete. 2024-05-27T09:44:38.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.167+0000 7ffac0f83b80 20 client.26947 trim_cache size 0 max 16384 2024-05-27T09:44:38.169 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:44:38.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.168+0000 7ffab0e00700 1 -- 172.21.15.44:0/3733552053 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ffa900b2a70 msgr2=0x7ffa900b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:38.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.168+0000 7ffab0e00700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ffa900b2a70 0x7ffa900b4f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4019fd0 tx=0x7ffaa400ec80 comp rx=0 tx=0).stop 2024-05-27T09:44:38.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.168+0000 7ffab0e00700 1 -- 172.21.15.44:0/3733552053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac106610 msgr2=0x7ffaac110610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:38.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.168+0000 7ffab0e00700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac106610 0x7ffaac110610 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c00ef10 tx=0x7ffa9c00c3a0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.168+0000 7ffab0e00700 1 -- 172.21.15.44:0/3733552053 shutdown_connections 2024-05-27T09:44:38.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.168+0000 7ffab0e00700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ffa900b2a70 0x7ffa900b4f80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.168+0000 7ffab0e00700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaac10b890 0x7ffaac111090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.168+0000 7ffab0e00700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac106610 0x7ffaac110610 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.168+0000 7ffab0e00700 1 --2- 172.21.15.44:0/3733552053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffaac10af40 0x7ffaac110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:38.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.168+0000 7ffab0e00700 1 -- 172.21.15.44:0/3733552053 >> 172.21.15.44:0/3733552053 conn(0x7ffaac102020 msgr2=0x7ffaac10a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:38.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.169+0000 7ffab0e00700 1 -- 172.21.15.44:0/3733552053 shutdown_connections 2024-05-27T09:44:38.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:38.169+0000 7ffab0e00700 1 -- 172.21.15.44:0/3733552053 wait complete. 2024-05-27T09:44:38.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:37 smithi050 ceph-mon[36371]: pgmap v292: 97 pgs: 97 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 1.50k op/s 2024-05-27T09:44:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:37 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2712615667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:44:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:37 smithi189 ceph-mon[36671]: pgmap v292: 97 pgs: 97 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 1.50k op/s 2024-05-27T09:44:38.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:37 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2712615667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:44:38.765 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db / --timeout 90 --expiration 138.95 --await-for 45.0' 2024-05-27T09:44:38.765 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db / --timeout 90 --expiration 138.95 --await-for 45.0 2024-05-27T09:44:39.083 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:44:39.261 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:44:39.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:39 smithi044 ceph-mon[31618]: pgmap v293: 97 pgs: 97 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 1.01k op/s 2024-05-27T09:44:40.109 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:44:40.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:39 smithi050 ceph-mon[36371]: pgmap v293: 97 pgs: 97 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 1.01k op/s 2024-05-27T09:44:40.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:39 smithi189 ceph-mon[36671]: pgmap v293: 97 pgs: 97 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 1.01k op/s 2024-05-27T09:44:42.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.004+0000 7fb202a00700 1 -- 172.21.15.44:0/3337841842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1fc105bd0 msgr2=0x7fb1fc10e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:42.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.004+0000 7fb202a00700 1 --2- 172.21.15.44:0/3337841842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1fc105bd0 0x7fb1fc10e120 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8009990 tx=0x7fb1e8009ca0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.004+0000 7fb202a00700 1 -- 172.21.15.44:0/3337841842 shutdown_connections 2024-05-27T09:44:42.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.004+0000 7fb202a00700 1 --2- 172.21.15.44:0/3337841842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc10e6f0 0x7fb1fc110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.004+0000 7fb202a00700 1 --2- 172.21.15.44:0/3337841842 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1fc105280 0x7fb1fc105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.004+0000 7fb202a00700 1 --2- 172.21.15.44:0/3337841842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1fc105bd0 0x7fb1fc10e120 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.004+0000 7fb202a00700 1 -- 172.21.15.44:0/3337841842 >> 172.21.15.44:0/3337841842 conn(0x7fb1fc078ab0 msgr2=0x7fb1fc078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:42.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.004+0000 7fb202a00700 1 -- 172.21.15.44:0/3337841842 shutdown_connections 2024-05-27T09:44:42.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.004+0000 7fb202a00700 1 -- 172.21.15.44:0/3337841842 wait complete. 2024-05-27T09:44:42.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.005+0000 7fb202a00700 1 Processor -- start 2024-05-27T09:44:42.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.005+0000 7fb202a00700 1 -- start start 2024-05-27T09:44:42.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.006+0000 7fb202a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1fc105280 0x7fb1fc1a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.006+0000 7fb202a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc105bd0 0x7fb1fc1a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.006+0000 7fb202a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1fc10e6f0 0x7fb1fc19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.006+0000 7fb202a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc1142a0 con 0x7fb1fc10e6f0 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.006+0000 7fb202a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc114120 con 0x7fb1fc105280 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.006+0000 7fb202a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc114420 con 0x7fb1fc105bd0 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.006+0000 7fb201600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1fc105280 0x7fb1fc1a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.006+0000 7fb200c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc105bd0 0x7fb1fc1a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.006+0000 7fb200c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc105bd0 0x7fb1fc1a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53160/0 (socket says 172.21.15.44:53160) 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.006+0000 7fb200c00700 1 -- 172.21.15.44:0/940709310 learned_addr learned my addr 172.21.15.44:0/940709310 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.006+0000 7fb201600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1fc105280 0x7fb1fc1a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45492/0 (socket says 172.21.15.44:45492) 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.007+0000 7fb202000700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1fc10e6f0 0x7fb1fc19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.007+0000 7fb200c00700 1 -- 172.21.15.44:0/940709310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1fc105280 msgr2=0x7fb1fc1a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.007+0000 7fb200c00700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1fc105280 0x7fb1fc1a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.007+0000 7fb200c00700 1 -- 172.21.15.44:0/940709310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1fc10e6f0 msgr2=0x7fb1fc19aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.007+0000 7fb200c00700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1fc10e6f0 0x7fb1fc19aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.007+0000 7fb200c00700 1 -- 172.21.15.44:0/940709310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc19b150 con 0x7fb1fc105bd0 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.007+0000 7fb202000700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1fc10e6f0 0x7fb1fc19aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.007+0000 7fb200c00700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc105bd0 0x7fb1fc1a0dc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8005910 tx=0x7fb1e800b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.008+0000 7fb1fa000700 1 -- 172.21.15.44:0/940709310 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8027070 con 0x7fb1fc105bd0 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.008+0000 7fb1fa000700 1 -- 172.21.15.44:0/940709310 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1e8022470 con 0x7fb1fc105bd0 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.008+0000 7fb1fa000700 1 -- 172.21.15.44:0/940709310 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8017650 con 0x7fb1fc105bd0 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.008+0000 7fb202a00700 1 -- 172.21.15.44:0/940709310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1fc19b3e0 con 0x7fb1fc105bd0 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.008+0000 7fb202a00700 1 -- 172.21.15.44:0/940709310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1fc19b870 con 0x7fb1fc105bd0 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.011+0000 7fb1fa000700 1 -- 172.21.15.44:0/940709310 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb1e802a070 con 0x7fb1fc105bd0 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.013+0000 7fb212bfeb80 1 Processor -- start 2024-05-27T09:44:42.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.013+0000 7fb212bfeb80 1 -- start start 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.013+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b349fe0 0x55671b4cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.013+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b4cdea0 0x55671b4d7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.013+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4d9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.013+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55671b3f6fe0 con 0x55671b4cdea0 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.013+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55671b3e6b80 con 0x55671b4ce2b0 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.013+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55671b3eabc0 con 0x55671b349fe0 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb1fa000700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb1e40b2a70 0x7fb1e40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb200c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4d9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb200c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4d9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45504/0 (socket says 172.21.15.44:45504) 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb200c00700 1 -- 172.21.15.44:0/231865925 learned_addr learned my addr 172.21.15.44:0/231865925 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb201600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b4cdea0 0x55671b4d7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb202000700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb1e40b2a70 0x7fb1e40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb1fa000700 1 -- 172.21.15.44:0/940709310 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7fb1e80e36b0 con 0x7fb1fc105bd0 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb202000700 1 --2- 172.21.15.44:0/231865925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b349fe0 0x55671b4cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb200c00700 1 -- 172.21.15.44:0/231865925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b349fe0 msgr2=0x55671b4cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb200c00700 1 --2- 172.21.15.44:0/231865925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b349fe0 0x55671b4cd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb200c00700 1 -- 172.21.15.44:0/231865925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b4cdea0 msgr2=0x55671b4d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb200c00700 1 --2- 172.21.15.44:0/231865925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b4cdea0 0x55671b4d7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb200c00700 1 -- 172.21.15.44:0/231865925 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55671b3af5c0 con 0x55671b4ce2b0 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.014+0000 7fb202000700 1 --2- 172.21.15.44:0/231865925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b349fe0 0x55671b4cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.015+0000 7fb200c00700 1 --2- 172.21.15.44:0/231865925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4d9a50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80e0ce0 tx=0x7fb1e8022cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8081307ac1290244 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.015+0000 7fb1eea00700 1 -- 172.21.15.44:0/231865925 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e80aa4d0 con 0x55671b4ce2b0 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.015+0000 7fb202000700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb1e40b2a70 0x7fb1e40b4f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55671b4c4d10 tx=0x7fb1f400d040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.015+0000 7fb1eea00700 1 -- 172.21.15.44:0/231865925 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1e80b30c0 con 0x55671b4ce2b0 2024-05-27T09:44:42.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.015+0000 7fb1eea00700 1 -- 172.21.15.44:0/231865925 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e80b3790 con 0x55671b4ce2b0 2024-05-27T09:44:42.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.016+0000 7fb212bfeb80 1 -- 172.21.15.44:0/231865925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 msgr2=0x55671b4d9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:42.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.016+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/231865925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4d9a50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80e0ce0 tx=0x7fb1e8022cc0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.016+0000 7fb212bfeb80 1 -- 172.21.15.44:0/231865925 shutdown_connections 2024-05-27T09:44:42.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.016+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/231865925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4d9a50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.016+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/231865925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b349fe0 0x55671b4cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.016+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/231865925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b4cdea0 0x55671b4d7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.016+0000 7fb212bfeb80 1 -- 172.21.15.44:0/231865925 >> 172.21.15.44:0/231865925 conn(0x55671b3aa160 msgr2=0x55671b4c4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:42.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.016+0000 7fb212bfeb80 1 -- 172.21.15.44:0/231865925 shutdown_connections 2024-05-27T09:44:42.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.017+0000 7fb212bfeb80 1 -- 172.21.15.44:0/231865925 wait complete. 2024-05-27T09:44:42.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.017+0000 7fb212bfeb80 1 -- start start 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.017+0000 7fb212bfeb80 1 Processor -- start 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4d7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 0x55671b4d7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 0x55671b4d7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55671b2ed140 con 0x55671b349fe0 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55671b387900 con 0x55671b4cdea0 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55671b4da180 con 0x55671b4ce2b0 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb212bfeb80 10 client.0 _ll_register_callbacks cb 0x55671b390e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb201600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4d7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb200c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 0x55671b4d7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb202000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 0x55671b4d7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb200c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 0x55671b4d7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45510/0 (socket says 172.21.15.44:45510) 2024-05-27T09:44:42.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.018+0000 7fb200c00700 1 -- 172.21.15.44:0/2502304887 learned_addr learned my addr 172.21.15.44:0/2502304887 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.019+0000 7fb200c00700 1 -- 172.21.15.44:0/2502304887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 msgr2=0x55671b4d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.019+0000 7fb200c00700 1 --2- 172.21.15.44:0/2502304887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 0x55671b4d7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.019+0000 7fb200c00700 1 -- 172.21.15.44:0/2502304887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 msgr2=0x55671b4d7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.019+0000 7fb200c00700 1 --2- 172.21.15.44:0/2502304887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4d7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.019+0000 7fb200c00700 1 -- 172.21.15.44:0/2502304887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55671b336a70 con 0x55671b4cdea0 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.019+0000 7fb200c00700 1 --2- 172.21.15.44:0/2502304887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 0x55671b4d7540 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800b230 tx=0x7fb1e8022bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67252d35d6e3241 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.019+0000 7fb1e3400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.019+0000 7fb1e3400700 1 -- 172.21.15.44:0/2502304887 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e80aa4d0 con 0x55671b4cdea0 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.019+0000 7fb1e3400700 1 -- 172.21.15.44:0/2502304887 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7fb1e80cbde0 con 0x55671b4cdea0 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.019+0000 7fb212bfeb80 1 -- 172.21.15.44:0/2502304887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1e80b30c0 con 0x55671b4cdea0 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.020+0000 7fb1e3400700 1 -- 172.21.15.44:0/2502304887 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1e80b12d0 con 0x55671b4cdea0 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.020+0000 7fb212bfeb80 10 client.26978 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:42.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.020+0000 7fb212bfeb80 1 -- 172.21.15.44:0/2502304887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55671b3b7200 con 0x55671b4cdea0 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.020+0000 7fb1e3400700 1 -- 172.21.15.44:0/2502304887 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1e80b30c0 con 0x55671b4cdea0 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.020+0000 7fb1e3400700 1 -- 172.21.15.44:0/2502304887 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7fb1e80aac20 con 0x55671b4cdea0 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.020+0000 7fb212bfeb80 10 client.26978 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.020+0000 7fb212bfeb80 10 client.26978 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.020+0000 7fb212bfeb80 20 client.26978 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.020+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/2502304887 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55671b4e6ed0 0x55671b4e92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.020+0000 7fb212bfeb80 4 client.26978 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.020+0000 7fb212bfeb80 1 -- 172.21.15.44:0/2502304887 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55671b3f8f30 con 0x55671b4e6ed0 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.021+0000 7fb201600700 1 --2- 172.21.15.44:0/2502304887 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55671b4e6ed0 0x55671b4e92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.021+0000 7fb201600700 1 --2- 172.21.15.44:0/2502304887 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55671b4e6ed0 0x55671b4e92f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a05417b26071a119 server_cookie=bc3a29b3ac29f84c in_seq=0 out_seq=0 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.021+0000 7fb1e3400700 10 client.26978 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.027+0000 7fb1e3400700 1 -- 172.21.15.44:0/2502304887 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55671b3f8f30 con 0x55671b4e6ed0 2024-05-27T09:44:42.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.027+0000 7fb1e3400700 10 client.26978 handle_command_reply: tid=0 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.027+0000 7fb212bfeb80 1 client.26978 shutdown 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb212bfeb80 1 -- 172.21.15.44:0/2502304887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 msgr2=0x55671b4d7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/2502304887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 0x55671b4d7540 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800b230 tx=0x7fb1e8022bd0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb212bfeb80 1 -- 172.21.15.44:0/2502304887 shutdown_connections 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/2502304887 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55671b4e6ed0 0x55671b4e92f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/2502304887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 0x55671b4d7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/2502304887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4d7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/2502304887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 0x55671b4d7540 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb1e3400700 0 client.26978 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb212bfeb80 1 -- 172.21.15.44:0/2502304887 >> 172.21.15.44:0/2502304887 conn(0x55671b3aa160 msgr2=0x55671b4cc320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb212bfeb80 1 -- 172.21.15.44:0/2502304887 shutdown_connections 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb212bfeb80 1 -- 172.21.15.44:0/2502304887 wait complete. 2024-05-27T09:44:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.028+0000 7fb212bfeb80 20 client.26978 trim_cache size 0 max 16384 2024-05-27T09:44:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.051+0000 7fb212bfeb80 1 Processor -- start 2024-05-27T09:44:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.051+0000 7fb212bfeb80 1 -- start start 2024-05-27T09:44:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.052+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4e31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.052+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 0x55671b4d6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.052+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 0x55671b4d6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.052+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc1af350 con 0x55671b349fe0 2024-05-27T09:44:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.052+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc1acbe0 con 0x55671b4cdea0 2024-05-27T09:44:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.052+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc1aaa70 con 0x55671b4ce2b0 2024-05-27T09:44:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.052+0000 7fb200c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 0x55671b4d6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.052+0000 7fb202000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 0x55671b4d6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.052+0000 7fb202000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 0x55671b4d6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53188/0 (socket says 172.21.15.44:53188) 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.052+0000 7fb202000700 1 -- 172.21.15.44:0/3042454819 learned_addr learned my addr 172.21.15.44:0/3042454819 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.052+0000 7fb201600700 1 --2- 172.21.15.44:0/3042454819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4e31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.053+0000 7fb202000700 1 -- 172.21.15.44:0/3042454819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 msgr2=0x55671b4d6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.053+0000 7fb202000700 1 --2- 172.21.15.44:0/3042454819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 0x55671b4d6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.053+0000 7fb202000700 1 -- 172.21.15.44:0/3042454819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 msgr2=0x55671b4e31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.053+0000 7fb202000700 1 --2- 172.21.15.44:0/3042454819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4e31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.053+0000 7fb202000700 1 -- 172.21.15.44:0/3042454819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55671b4c5840 con 0x55671b4ce2b0 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.053+0000 7fb200c00700 1 --2- 172.21.15.44:0/3042454819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 0x55671b4d6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.053+0000 7fb202000700 1 --2- 172.21.15.44:0/3042454819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 0x55671b4d6540 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009420 tx=0x7fb1f4014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c30a9b3ece87830 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.053+0000 7fb1ee000700 1 -- 172.21.15.44:0/3042454819 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4007bf0 con 0x55671b4ce2b0 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.053+0000 7fb1ee000700 1 -- 172.21.15.44:0/3042454819 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1f4007d90 con 0x55671b4ce2b0 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.053+0000 7fb1ee000700 1 -- 172.21.15.44:0/3042454819 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4022400 con 0x55671b4ce2b0 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.054+0000 7fb212bfeb80 1 -- 172.21.15.44:0/3042454819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 msgr2=0x55671b4d6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.054+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/3042454819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 0x55671b4d6540 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009420 tx=0x7fb1f4014000 comp rx=0 tx=0).stop 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.054+0000 7fb212bfeb80 1 -- 172.21.15.44:0/3042454819 shutdown_connections 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.054+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/3042454819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4ce2b0 0x55671b4d6540 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.054+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/3042454819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4e31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.054+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/3042454819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4cdea0 0x55671b4d6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.055+0000 7fb212bfeb80 1 -- 172.21.15.44:0/3042454819 >> 172.21.15.44:0/3042454819 conn(0x55671b3aa160 msgr2=0x55671b4d58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.055+0000 7fb212bfeb80 1 -- 172.21.15.44:0/3042454819 shutdown_connections 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.055+0000 7fb212bfeb80 1 -- 172.21.15.44:0/3042454819 wait complete. 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.055+0000 7fb212bfeb80 1 -- start start 2024-05-27T09:44:42.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.056+0000 7fb212bfeb80 1 Processor -- start 2024-05-27T09:44:42.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.056+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4dc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.056+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4cdea0 0x55671b4dc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.056+0000 7fb212bfeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4dcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.056+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc1ab0c0 con 0x55671b349fe0 2024-05-27T09:44:42.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.056+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc1ac510 con 0x55671b4ce2b0 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.056+0000 7fb212bfeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc1ab770 con 0x55671b4cdea0 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.056+0000 7fb212bfeb80 10 client.0 _ll_register_callbacks cb 0x55671b390e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.056+0000 7fb202000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4dcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb200c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4cdea0 0x55671b4dc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb202000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4dcd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:45530/0 (socket says 172.21.15.44:45530) 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb202000700 1 -- 172.21.15.44:0/1041721225 learned_addr learned my addr 172.21.15.44:0/1041721225 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb201600700 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4dc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb202000700 1 -- 172.21.15.44:0/1041721225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4cdea0 msgr2=0x55671b4dc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb202000700 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4cdea0 0x55671b4dc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb202000700 1 -- 172.21.15.44:0/1041721225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 msgr2=0x55671b4dc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb202000700 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4dc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:42.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb202000700 1 -- 172.21.15.44:0/1041721225 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55671b551960 con 0x55671b4ce2b0 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb201600700 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4dc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb202000700 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4dcd30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4006130 tx=0x7fb1f4004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f16e9639e0dae48d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.057+0000 7fb1e3400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.058+0000 7fb1e3400700 1 -- 172.21.15.44:0/1041721225 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4007bf0 con 0x55671b4ce2b0 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.058+0000 7fb212bfeb80 1 -- 172.21.15.44:0/1041721225 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1f4007d90 con 0x55671b4ce2b0 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.058+0000 7fb1e3400700 1 -- 172.21.15.44:0/1041721225 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7fb1f4009040 con 0x55671b4ce2b0 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.058+0000 7fb1e3400700 1 -- 172.21.15.44:0/1041721225 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1f4039050 con 0x55671b4ce2b0 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.058+0000 7fb212bfeb80 10 client.26984 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.058+0000 7fb212bfeb80 1 -- 172.21.15.44:0/1041721225 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55671b552b80 con 0x55671b4ce2b0 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.058+0000 7fb1e3400700 1 -- 172.21.15.44:0/1041721225 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1f4007d90 con 0x55671b4ce2b0 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.058+0000 7fb1e3400700 1 -- 172.21.15.44:0/1041721225 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7fb1f4022400 con 0x55671b4ce2b0 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.059+0000 7fb212bfeb80 10 client.26984 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:42.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.059+0000 7fb212bfeb80 10 client.26984 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:44:42.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.059+0000 7fb212bfeb80 20 client.26984 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:42.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.059+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55671b553270 0x55671b555690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:42.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.059+0000 7fb212bfeb80 4 client.26984 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.95, "await_for": 45.0}] 2024-05-27T09:44:42.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.059+0000 7fb212bfeb80 1 -- 172.21.15.44:0/1041721225 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.95, "await_for": 45.0}) -- 0x55671b3f2710 con 0x55671b553270 2024-05-27T09:44:42.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.059+0000 7fb201600700 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55671b553270 0x55671b555690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:42.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.060+0000 7fb201600700 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55671b553270 0x55671b555690 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceea4f73366634f9 server_cookie=4216b595b97d481a in_seq=0 out_seq=0 2024-05-27T09:44:42.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:42.060+0000 7fb1e3400700 10 client.26984 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:42.179 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:41 smithi044 ceph-mon[31618]: pgmap v294: 97 pgs: 97 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 1.45k op/s 2024-05-27T09:44:42.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:41 smithi050 ceph-mon[36371]: pgmap v294: 97 pgs: 97 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 1.45k op/s 2024-05-27T09:44:42.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:41 smithi189 ceph-mon[36671]: pgmap v294: 97 pgs: 97 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 1.45k op/s 2024-05-27T09:44:43.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.062+0000 7f866ec00700 1 -- 172.21.15.44:0/2537086731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8668110c10 msgr2=0x7f8668113030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:43.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.062+0000 7f866ec00700 1 --2- 172.21.15.44:0/2537086731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8668110c10 0x7f8668113030 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f86600098f0 tx=0x7f8660009c00 comp rx=0 tx=0).stop 2024-05-27T09:44:43.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.062+0000 7f866ec00700 1 -- 172.21.15.44:0/2537086731 shutdown_connections 2024-05-27T09:44:43.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.062+0000 7f866ec00700 1 --2- 172.21.15.44:0/2537086731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8668110c10 0x7f8668113030 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.062+0000 7f866ec00700 1 --2- 172.21.15.44:0/2537086731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668075ad0 0x7f8668075ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.062+0000 7f866ec00700 1 --2- 172.21.15.44:0/2537086731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8668076420 0x7f86681106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.062+0000 7f866ec00700 1 -- 172.21.15.44:0/2537086731 >> 172.21.15.44:0/2537086731 conn(0x7f8668100030 msgr2=0x7f8668102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:43.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.062+0000 7f866ec00700 1 -- 172.21.15.44:0/2537086731 shutdown_connections 2024-05-27T09:44:43.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.062+0000 7f866ec00700 1 -- 172.21.15.44:0/2537086731 wait complete. 2024-05-27T09:44:43.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.063+0000 7f866ec00700 1 Processor -- start 2024-05-27T09:44:43.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.064+0000 7f866ec00700 1 -- start start 2024-05-27T09:44:43.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.064+0000 7f866ec00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668075ad0 0x7f866819d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.064+0000 7f866ec00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8668076420 0x7f866819d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.064+0000 7f866ec00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8668110c10 0x7f86681a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.064+0000 7f866ec00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f86681166f0 con 0x7f8668110c10 2024-05-27T09:44:43.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.064+0000 7f866ec00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8668116570 con 0x7f8668076420 2024-05-27T09:44:43.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.064+0000 7f866ec00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8668116870 con 0x7f8668075ad0 2024-05-27T09:44:43.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f8667e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668075ad0 0x7f866819d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f8667e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668075ad0 0x7f866819d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41882/0 (socket says 172.21.15.44:41882) 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f8667e00700 1 -- 172.21.15.44:0/1030617707 learned_addr learned my addr 172.21.15.44:0/1030617707 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f8667400700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8668076420 0x7f866819d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f866cc00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8668110c10 0x7f86681a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f8667e00700 1 -- 172.21.15.44:0/1030617707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8668076420 msgr2=0x7f866819d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f8667e00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8668076420 0x7f866819d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f8667e00700 1 -- 172.21.15.44:0/1030617707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8668110c10 msgr2=0x7f86681a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f8667e00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8668110c10 0x7f86681a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f8667e00700 1 -- 172.21.15.44:0/1030617707 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86681a1f30 con 0x7f8668075ad0 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f866cc00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8668110c10 0x7f86681a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f8667400700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8668076420 0x7f866819d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:43.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.065+0000 7f8667e00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668075ad0 0x7f866819d060 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f865800e3c0 tx=0x7f865800e6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:43.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.066+0000 7f8664c00700 1 -- 172.21.15.44:0/1030617707 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86580090d0 con 0x7f8668075ad0 2024-05-27T09:44:43.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.066+0000 7f8664c00700 1 -- 172.21.15.44:0/1030617707 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8658009270 con 0x7f8668075ad0 2024-05-27T09:44:43.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.066+0000 7f866ec00700 1 -- 172.21.15.44:0/1030617707 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86681a21c0 con 0x7f8668075ad0 2024-05-27T09:44:43.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.066+0000 7f8664c00700 1 -- 172.21.15.44:0/1030617707 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658004680 con 0x7f8668075ad0 2024-05-27T09:44:43.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.066+0000 7f866ec00700 1 -- 172.21.15.44:0/1030617707 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86681a9b40 con 0x7f8668075ad0 2024-05-27T09:44:43.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.072+0000 7f8664c00700 1 -- 172.21.15.44:0/1030617707 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f865801b030 con 0x7f8668075ad0 2024-05-27T09:44:43.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.072+0000 7f867d666b80 1 Processor -- start 2024-05-27T09:44:43.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.073+0000 7f867d666b80 1 -- start start 2024-05-27T09:44:43.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.073+0000 7f867d666b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f248048330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.073+0000 7f867d666b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f247ec9e40 0x55f248048870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.073+0000 7f867d666b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f248048e70 0x55f248055110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.073+0000 7f867d666b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f247f76e40 con 0x55f247ec9e40 2024-05-27T09:44:43.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.073+0000 7f867d666b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f247f669e0 con 0x55f248048e70 2024-05-27T09:44:43.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.073+0000 7f867d666b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f247f6aa20 con 0x55f247fb59f0 2024-05-27T09:44:43.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f866cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f248048330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f866cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f248048330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41890/0 (socket says 172.21.15.44:41890) 2024-05-27T09:44:43.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f8667400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f248048e70 0x55f248055110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f866cc00700 1 -- 172.21.15.44:0/2188869084 learned_addr learned my addr 172.21.15.44:0/2188869084 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:43.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f8667e00700 1 --2- 172.21.15.44:0/2188869084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f247ec9e40 0x55f248048870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f8664c00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f864c0b29f0 0x7f864c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f866cc00700 1 -- 172.21.15.44:0/2188869084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f248048e70 msgr2=0x55f248055110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:43.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f866cc00700 1 --2- 172.21.15.44:0/2188869084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f248048e70 0x55f248055110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f866cc00700 1 -- 172.21.15.44:0/2188869084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f247ec9e40 msgr2=0x55f248048870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:43.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f866cc00700 1 --2- 172.21.15.44:0/2188869084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f247ec9e40 0x55f248048870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f866cc00700 1 -- 172.21.15.44:0/2188869084 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f247f2f420 con 0x55f247fb59f0 2024-05-27T09:44:43.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f8667400700 1 --2- 172.21.15.44:0/2188869084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f248048e70 0x55f248055110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:43.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f8667e00700 1 --2- 172.21.15.44:0/2188869084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f247ec9e40 0x55f248048870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:44:43.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.074+0000 7f8664c00700 1 -- 172.21.15.44:0/1030617707 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f86580e2060 con 0x7f8668075ad0 2024-05-27T09:44:43.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.075+0000 7f866cc00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f864c0b29f0 0x7f864c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.075+0000 7f866cc00700 1 --2- 172.21.15.44:0/2188869084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f248048330 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8660016000 tx=0x7f866000fbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3b8a845d17ee9b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:43.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.075+0000 7f865d600700 1 -- 172.21.15.44:0/2188869084 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866000fde0 con 0x55f247fb59f0 2024-05-27T09:44:43.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.075+0000 7f865d600700 1 -- 172.21.15.44:0/2188869084 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86600054f0 con 0x55f247fb59f0 2024-05-27T09:44:43.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.075+0000 7f865d600700 1 -- 172.21.15.44:0/2188869084 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866000b560 con 0x55f247fb59f0 2024-05-27T09:44:43.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.076+0000 7f867d666b80 1 -- 172.21.15.44:0/2188869084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 msgr2=0x55f248048330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:43.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.076+0000 7f867d666b80 1 --2- 172.21.15.44:0/2188869084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f248048330 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8660016000 tx=0x7f866000fbc0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.076+0000 7f866cc00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f864c0b29f0 0x7f864c0b4f00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f866002b000 tx=0x7f866002e000 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:43.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.077+0000 7f867d666b80 1 -- 172.21.15.44:0/2188869084 shutdown_connections 2024-05-27T09:44:43.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.077+0000 7f867d666b80 1 --2- 172.21.15.44:0/2188869084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f248048e70 0x55f248055110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.077+0000 7f867d666b80 1 --2- 172.21.15.44:0/2188869084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f248048330 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.077+0000 7f867d666b80 1 --2- 172.21.15.44:0/2188869084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f247ec9e40 0x55f248048870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.077+0000 7f867d666b80 1 -- 172.21.15.44:0/2188869084 >> 172.21.15.44:0/2188869084 conn(0x55f247f63390 msgr2=0x55f2480444c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:43.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.077+0000 7f867d666b80 1 -- 172.21.15.44:0/2188869084 shutdown_connections 2024-05-27T09:44:43.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.077+0000 7f867d666b80 1 -- 172.21.15.44:0/2188869084 wait complete. 2024-05-27T09:44:43.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.078+0000 7f867d666b80 1 -- start start 2024-05-27T09:44:43.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.078+0000 7f867d666b80 1 Processor -- start 2024-05-27T09:44:43.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f867d666b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f247ec9e40 0x55f248052640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f867d666b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247fb59f0 0x55f248052b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f867d666b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f248048e70 0x55f2480530c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f867d666b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f247e6cfa0 con 0x55f247ec9e40 2024-05-27T09:44:43.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f867d666b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f247f07760 con 0x55f247fb59f0 2024-05-27T09:44:43.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f867d666b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f248055b80 con 0x55f248048e70 2024-05-27T09:44:43.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f867d666b80 10 client.0 _ll_register_callbacks cb 0x55f247f10c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:43.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f866cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f248048e70 0x55f2480530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f8667400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247fb59f0 0x55f248052b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f866cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f248048e70 0x55f2480530c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41900/0 (socket says 172.21.15.44:41900) 2024-05-27T09:44:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f866cc00700 1 -- 172.21.15.44:0/4037468423 learned_addr learned my addr 172.21.15.44:0/4037468423 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f8667400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247fb59f0 0x55f248052b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:55000/0 (socket says 172.21.15.44:55000) 2024-05-27T09:44:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f8667e00700 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f247ec9e40 0x55f248052640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f866cc00700 1 -- 172.21.15.44:0/4037468423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247fb59f0 msgr2=0x55f248052b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f866cc00700 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247fb59f0 0x55f248052b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f866cc00700 1 -- 172.21.15.44:0/4037468423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f247ec9e40 msgr2=0x55f248052640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.079+0000 7f866cc00700 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f247ec9e40 0x55f248052640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.080+0000 7f866cc00700 1 -- 172.21.15.44:0/4037468423 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f247eb68d0 con 0x55f248048e70 2024-05-27T09:44:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.080+0000 7f8667400700 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247fb59f0 0x55f248052b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.080+0000 7f866cc00700 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f248048e70 0x55f2480530c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8660004af0 tx=0x7f8660004a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9906149f5851fd1d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.080+0000 7f8652000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:43.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.080+0000 7f8652000700 1 -- 172.21.15.44:0/4037468423 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660035070 con 0x55f248048e70 2024-05-27T09:44:43.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.080+0000 7f8652000700 1 -- 172.21.15.44:0/4037468423 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f8660005110 con 0x55f248048e70 2024-05-27T09:44:43.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.080+0000 7f867d666b80 1 -- 172.21.15.44:0/4037468423 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86600054f0 con 0x55f248048e70 2024-05-27T09:44:43.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.081+0000 7f8652000700 1 -- 172.21.15.44:0/4037468423 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8660047050 con 0x55f248048e70 2024-05-27T09:44:43.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.081+0000 7f867d666b80 10 client.26974 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:43.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.081+0000 7f867d666b80 1 -- 172.21.15.44:0/4037468423 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f247f37060 con 0x55f248048e70 2024-05-27T09:44:43.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.081+0000 7f8652000700 1 -- 172.21.15.44:0/4037468423 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86600054f0 con 0x55f248048e70 2024-05-27T09:44:43.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.081+0000 7f8652000700 1 -- 172.21.15.44:0/4037468423 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f8660042030 con 0x55f248048e70 2024-05-27T09:44:43.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.081+0000 7f867d666b80 10 client.26974 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:43.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.081+0000 7f867d666b80 10 client.26974 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:44:43.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.081+0000 7f867d666b80 20 client.26974 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.081+0000 7f867d666b80 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55f2480628c0 0x55f248064ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.081+0000 7f867d666b80 4 client.26974 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.081+0000 7f867d666b80 1 -- 172.21.15.44:0/4037468423 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f247f78d90 con 0x55f2480628c0 2024-05-27T09:44:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.082+0000 7f8667e00700 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55f2480628c0 0x55f248064ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.082+0000 7f8667e00700 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55f2480628c0 0x55f248064ce0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9b1a7674604645d1 server_cookie=c4cad4112d295a43 in_seq=0 out_seq=0 2024-05-27T09:44:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.082+0000 7f8652000700 10 client.26974 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:44:43.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f8652000700 1 -- 172.21.15.44:0/4037468423 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f247f78d90 con 0x55f2480628c0 2024-05-27T09:44:43.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f8652000700 10 client.26974 handle_command_reply: tid=0 2024-05-27T09:44:43.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 1 client.26974 shutdown 2024-05-27T09:44:43.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 1 -- 172.21.15.44:0/4037468423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f248048e70 msgr2=0x55f2480530c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:43.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f248048e70 0x55f2480530c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8660004af0 tx=0x7f8660004a00 comp rx=0 tx=0).stop 2024-05-27T09:44:43.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 1 -- 172.21.15.44:0/4037468423 shutdown_connections 2024-05-27T09:44:43.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f248048e70 0x55f2480530c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55f2480628c0 0x55f248064ce0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f247ec9e40 0x55f248052640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 1 --2- 172.21.15.44:0/4037468423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247fb59f0 0x55f248052b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f8652000700 0 client.26974 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:44:43.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 1 -- 172.21.15.44:0/4037468423 >> 172.21.15.44:0/4037468423 conn(0x55f247f63390 msgr2=0x55f248048310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:43.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 1 -- 172.21.15.44:0/4037468423 shutdown_connections 2024-05-27T09:44:43.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 1 -- 172.21.15.44:0/4037468423 wait complete. 2024-05-27T09:44:43.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.090+0000 7f867d666b80 20 client.26974 trim_cache size 0 max 16384 2024-05-27T09:44:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f867d666b80 1 Processor -- start 2024-05-27T09:44:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f867d666b80 1 -- start start 2024-05-27T09:44:43.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f867d666b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f2480515a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f867d666b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f2480487c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f867d666b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f248048e70 0x55f24804ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f867d666b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b3960 con 0x55f248048e70 2024-05-27T09:44:43.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f867d666b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b11f0 con 0x55f247ec9e40 2024-05-27T09:44:43.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f867d666b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86681af060 con 0x55f247fb59f0 2024-05-27T09:44:43.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f8667e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f2480515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f8667400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f2480487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f8667e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f2480515a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:55016/0 (socket says 172.21.15.44:55016) 2024-05-27T09:44:43.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f8667400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f2480487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41912/0 (socket says 172.21.15.44:41912) 2024-05-27T09:44:43.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.109+0000 7f8667e00700 1 -- 172.21.15.44:0/3319846160 learned_addr learned my addr 172.21.15.44:0/3319846160 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:43.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.110+0000 7f866cc00700 1 --2- 172.21.15.44:0/3319846160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f248048e70 0x55f24804ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.110+0000 7f8667e00700 1 -- 172.21.15.44:0/3319846160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 msgr2=0x55f2480487c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:43.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.110+0000 7f8667e00700 1 --2- 172.21.15.44:0/3319846160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f2480487c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.110+0000 7f8667e00700 1 -- 172.21.15.44:0/3319846160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f248048e70 msgr2=0x55f24804ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:43.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.110+0000 7f8667e00700 1 --2- 172.21.15.44:0/3319846160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f248048e70 0x55f24804ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.110+0000 7f8667e00700 1 -- 172.21.15.44:0/3319846160 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f248050bb0 con 0x55f247ec9e40 2024-05-27T09:44:43.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.110+0000 7f8667e00700 1 --2- 172.21.15.44:0/3319846160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f2480515a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f86580048b0 tx=0x7f865801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e54d5ffee09a5d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:43.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.110+0000 7f865cc00700 1 -- 172.21.15.44:0/3319846160 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86580a8cc0 con 0x55f247ec9e40 2024-05-27T09:44:43.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.110+0000 7f865cc00700 1 -- 172.21.15.44:0/3319846160 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86580a8e80 con 0x55f247ec9e40 2024-05-27T09:44:43.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.111+0000 7f865cc00700 1 -- 172.21.15.44:0/3319846160 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86580a8ae0 con 0x55f247ec9e40 2024-05-27T09:44:43.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.111+0000 7f867d666b80 1 -- 172.21.15.44:0/3319846160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 msgr2=0x55f2480515a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:43.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.111+0000 7f867d666b80 1 --2- 172.21.15.44:0/3319846160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f2480515a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f86580048b0 tx=0x7f865801c000 comp rx=0 tx=0).stop 2024-05-27T09:44:43.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.112+0000 7f867d666b80 1 -- 172.21.15.44:0/3319846160 shutdown_connections 2024-05-27T09:44:43.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.112+0000 7f867d666b80 1 --2- 172.21.15.44:0/3319846160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f248048e70 0x55f24804ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.112+0000 7f867d666b80 1 --2- 172.21.15.44:0/3319846160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f2480515a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.112+0000 7f867d666b80 1 --2- 172.21.15.44:0/3319846160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f2480487c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.112+0000 7f867d666b80 1 -- 172.21.15.44:0/3319846160 >> 172.21.15.44:0/3319846160 conn(0x55f247f63390 msgr2=0x55f2480480b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:43.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.112+0000 7f867d666b80 1 -- 172.21.15.44:0/3319846160 shutdown_connections 2024-05-27T09:44:43.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.112+0000 7f867d666b80 1 -- 172.21.15.44:0/3319846160 wait complete. 2024-05-27T09:44:43.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.112+0000 7f867d666b80 1 -- start start 2024-05-27T09:44:43.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.112+0000 7f867d666b80 1 Processor -- start 2024-05-27T09:44:43.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.113+0000 7f867d666b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f248065180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.113+0000 7f867d666b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f2480656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.113+0000 7f867d666b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f248048e70 0x55f2480cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.113+0000 7f867d666b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f86681af6b0 con 0x55f248048e70 2024-05-27T09:44:43.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.113+0000 7f867d666b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b0b20 con 0x55f247ec9e40 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.113+0000 7f867d666b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86681afd80 con 0x55f247fb59f0 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.113+0000 7f867d666b80 10 client.0 _ll_register_callbacks cb 0x55f247f10c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.113+0000 7f8667e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f248065180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.113+0000 7f8667400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f2480656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.113+0000 7f8667e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f248065180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:55020/0 (socket says 172.21.15.44:55020) 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.113+0000 7f8667e00700 1 -- 172.21.15.44:0/2357782136 learned_addr learned my addr 172.21.15.44:0/2357782136 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.114+0000 7f866cc00700 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f248048e70 0x55f2480cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.114+0000 7f8667e00700 1 -- 172.21.15.44:0/2357782136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 msgr2=0x55f2480656c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.114+0000 7f8667e00700 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f2480656c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.114+0000 7f8667e00700 1 -- 172.21.15.44:0/2357782136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f248048e70 msgr2=0x55f2480cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.114+0000 7f8667e00700 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f248048e70 0x55f2480cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.114+0000 7f8667e00700 1 -- 172.21.15.44:0/2357782136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2480cb660 con 0x55f247ec9e40 2024-05-27T09:44:43.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.114+0000 7f8667e00700 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f248065180 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f865800ead0 tx=0x7f86580b1f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d63da5a5f999403b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:43.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.114+0000 7f8652000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:44:43.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.114+0000 7f8652000700 1 -- 172.21.15.44:0/2357782136 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86580a8ae0 con 0x55f247ec9e40 2024-05-27T09:44:43.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.114+0000 7f8652000700 1 -- 172.21.15.44:0/2357782136 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f86580a85d0 con 0x55f247ec9e40 2024-05-27T09:44:43.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.114+0000 7f867d666b80 1 -- 172.21.15.44:0/2357782136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86580a8e80 con 0x55f247ec9e40 2024-05-27T09:44:43.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.115+0000 7f8652000700 1 -- 172.21.15.44:0/2357782136 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f86580a8cc0 con 0x55f247ec9e40 2024-05-27T09:44:43.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.115+0000 7f867d666b80 10 client.27002 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:43.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.115+0000 7f867d666b80 1 -- 172.21.15.44:0/2357782136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f2480cc820 con 0x55f247ec9e40 2024-05-27T09:44:43.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.115+0000 7f8652000700 1 -- 172.21.15.44:0/2357782136 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86580a8e80 con 0x55f247ec9e40 2024-05-27T09:44:43.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.115+0000 7f8652000700 1 -- 172.21.15.44:0/2357782136 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f8660004bf0 con 0x55f247ec9e40 2024-05-27T09:44:43.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.115+0000 7f867d666b80 10 client.27002 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:43.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.115+0000 7f867d666b80 10 client.27002 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:44:43.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.115+0000 7f867d666b80 20 client.27002 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:43.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.115+0000 7f867d666b80 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55f2480cce90 0x55f2480cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:43.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.115+0000 7f867d666b80 4 client.27002 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:44:43.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.115+0000 7f867d666b80 1 -- 172.21.15.44:0/2357782136 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f247f72570 con 0x55f2480cce90 2024-05-27T09:44:43.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.116+0000 7f8667400700 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55f2480cce90 0x55f2480cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:43.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.116+0000 7f8667400700 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55f2480cce90 0x55f2480cf2b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c571361b016ccfd2 server_cookie=45a99d9ceeb03470 in_seq=0 out_seq=0 2024-05-27T09:44:43.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.116+0000 7f8652000700 10 client.27002 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:44:43.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.121+0000 7f8652000700 1 -- 172.21.15.44:0/2357782136 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f247f72570 con 0x55f2480cce90 2024-05-27T09:44:43.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.121+0000 7f8652000700 10 client.27002 handle_command_reply: tid=0 2024-05-27T09:44:43.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.121+0000 7f867d666b80 1 client.27002 shutdown 2024-05-27T09:44:43.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f867d666b80 1 -- 172.21.15.44:0/2357782136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 msgr2=0x55f248065180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:43.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f867d666b80 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f248065180 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f865800ead0 tx=0x7f86580b1f60 comp rx=0 tx=0).stop 2024-05-27T09:44:43.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f867d666b80 1 -- 172.21.15.44:0/2357782136 shutdown_connections 2024-05-27T09:44:43.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f867d666b80 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55f2480cce90 0x55f2480cf2b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f867d666b80 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f248048e70 0x55f2480cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f8652000700 0 client.27002 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:44:43.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f867d666b80 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f247ec9e40 0x55f248065180 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f867d666b80 1 --2- 172.21.15.44:0/2357782136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f247fb59f0 0x55f2480656c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f867d666b80 1 -- 172.21.15.44:0/2357782136 >> 172.21.15.44:0/2357782136 conn(0x55f247f63390 msgr2=0x55f248053bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:43.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f867d666b80 1 -- 172.21.15.44:0/2357782136 shutdown_connections 2024-05-27T09:44:43.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f867d666b80 1 -- 172.21.15.44:0/2357782136 wait complete. 2024-05-27T09:44:43.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.122+0000 7f867d666b80 20 client.27002 trim_cache size 0 max 16384 2024-05-27T09:44:43.124 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:44:43.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.123+0000 7f866ec00700 1 -- 172.21.15.44:0/1030617707 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f864c0b29f0 msgr2=0x7f864c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:43.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.123+0000 7f866ec00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f864c0b29f0 0x7f864c0b4f00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f866002b000 tx=0x7f866002e000 comp rx=0 tx=0).stop 2024-05-27T09:44:43.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.123+0000 7f866ec00700 1 -- 172.21.15.44:0/1030617707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668075ad0 msgr2=0x7f866819d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:43.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.123+0000 7f866ec00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668075ad0 0x7f866819d060 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f865800e3c0 tx=0x7f865800e6d0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.123+0000 7f866ec00700 1 -- 172.21.15.44:0/1030617707 shutdown_connections 2024-05-27T09:44:43.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.123+0000 7f866ec00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f864c0b29f0 0x7f864c0b4f00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.123+0000 7f866ec00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8668110c10 0x7f86681a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.123+0000 7f866ec00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668075ad0 0x7f866819d060 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.123+0000 7f866ec00700 1 --2- 172.21.15.44:0/1030617707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8668076420 0x7f866819d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:43.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.123+0000 7f866ec00700 1 -- 172.21.15.44:0/1030617707 >> 172.21.15.44:0/1030617707 conn(0x7f8668100030 msgr2=0x7f8668100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:43.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.124+0000 7f866ec00700 1 -- 172.21.15.44:0/1030617707 shutdown_connections 2024-05-27T09:44:43.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:43.124+0000 7f866ec00700 1 -- 172.21.15.44:0/1030617707 wait complete. 2024-05-27T09:44:43.872 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:44:44.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:43 smithi044 ceph-mon[31618]: pgmap v295: 97 pgs: 97 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 920 KiB/s wr, 442 op/s 2024-05-27T09:44:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.280+0000 7f3bb5a00640 1 -- 172.21.15.44:0/759037444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bb0104d90 msgr2=0x7f3bb01051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.280+0000 7f3bb5a00640 1 --2- 172.21.15.44:0/759037444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bb0104d90 0x7f3bb01051a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0009900 tx=0x7f3ba002a410 comp rx=0 tx=0).stop 2024-05-27T09:44:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.280+0000 7f3bb5a00640 1 -- 172.21.15.44:0/759037444 shutdown_connections 2024-05-27T09:44:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.280+0000 7f3bb5a00640 1 --2- 172.21.15.44:0/759037444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb010e1e0 0x7f3bb01106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.280+0000 7f3bb5a00640 1 --2- 172.21.15.44:0/759037444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bb010b7f0 0x7f3bb010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.280+0000 7f3bb5a00640 1 --2- 172.21.15.44:0/759037444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bb0104d90 0x7f3bb01051a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.280+0000 7f3bb5a00640 1 -- 172.21.15.44:0/759037444 >> 172.21.15.44:0/759037444 conn(0x7f3bb00782d0 msgr2=0x7f3bb0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.280+0000 7f3bb5a00640 1 -- 172.21.15.44:0/759037444 shutdown_connections 2024-05-27T09:44:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.280+0000 7f3bb5a00640 1 -- 172.21.15.44:0/759037444 wait complete. 2024-05-27T09:44:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.281+0000 7f3bb5a00640 1 Processor -- start 2024-05-27T09:44:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.282+0000 7f3bb5a00640 1 -- start start 2024-05-27T09:44:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.282+0000 7f3bb5a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bb0104d90 0x7f3bb01a27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.282+0000 7f3bb5a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb010b7f0 0x7f3bb01a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.282+0000 7f3bb5a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bb010e1e0 0x7f3bb019c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.282+0000 7f3bb5a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb0113e90 con 0x7f3bb0104d90 2024-05-27T09:44:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.282+0000 7f3bb5a00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb0113d10 con 0x7f3bb010e1e0 2024-05-27T09:44:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.282+0000 7f3bb5a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb0114010 con 0x7f3bb010b7f0 2024-05-27T09:44:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.283+0000 7f3baea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bb0104d90 0x7f3bb01a27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.283+0000 7f3bae000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb010b7f0 0x7f3bb01a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.283+0000 7f3bae000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb010b7f0 0x7f3bb01a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41938/0 (socket says 172.21.15.44:41938) 2024-05-27T09:44:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.283+0000 7f3baea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bb0104d90 0x7f3bb01a27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38516/0 (socket says 172.21.15.44:38516) 2024-05-27T09:44:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.283+0000 7f3bae000640 1 -- 172.21.15.44:0/865228559 learned_addr learned my addr 172.21.15.44:0/865228559 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.283+0000 7f3baf400640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bb010e1e0 0x7f3bb019c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.283+0000 7f3bae000640 1 -- 172.21.15.44:0/865228559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bb010e1e0 msgr2=0x7f3bb019c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.283+0000 7f3bae000640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bb010e1e0 0x7f3bb019c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.283+0000 7f3bae000640 1 -- 172.21.15.44:0/865228559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bb0104d90 msgr2=0x7f3bb01a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.283+0000 7f3bae000640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bb0104d90 0x7f3bb01a27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.283+0000 7f3bae000640 1 -- 172.21.15.44:0/865228559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb019d050 con 0x7f3bb010b7f0 2024-05-27T09:44:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.284+0000 7f3baf400640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bb010e1e0 0x7f3bb019c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.284+0000 7f3bae000640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb010b7f0 0x7f3bb01a2ce0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00e9f0 tx=0x7f3b9c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.284+0000 7f3ba7400640 1 -- 172.21.15.44:0/865228559 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c00cd80 con 0x7f3bb010b7f0 2024-05-27T09:44:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.284+0000 7f3ba7400640 1 -- 172.21.15.44:0/865228559 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b9c0108e0 con 0x7f3bb010b7f0 2024-05-27T09:44:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.284+0000 7f3bb5a00640 1 -- 172.21.15.44:0/865228559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bb019d340 con 0x7f3bb010b7f0 2024-05-27T09:44:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.284+0000 7f3ba7400640 1 -- 172.21.15.44:0/865228559 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c018980 con 0x7f3bb010b7f0 2024-05-27T09:44:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.284+0000 7f3bb5a00640 1 -- 172.21.15.44:0/865228559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bb01a9620 con 0x7f3bb010b7f0 2024-05-27T09:44:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.288+0000 7f3bbca29740 1 Processor -- start 2024-05-27T09:44:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.289+0000 7f3bbca29740 1 -- start start 2024-05-27T09:44:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.289+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94c021b0 0x55cf94fa4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.289+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 0x55cf94fa4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.289+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94be14d0 0x55cf94fa52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.289+0000 7f3bbca29740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cf94f15b90 con 0x55cf94c021b0 2024-05-27T09:44:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.289+0000 7f3bbca29740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cf94f05730 con 0x55cf94be14d0 2024-05-27T09:44:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.289+0000 7f3bbca29740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cf94f09770 con 0x55cf94bd92f0 2024-05-27T09:44:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.289+0000 7f3baea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 0x55cf94fa4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.289+0000 7f3baea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 0x55cf94fa4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41946/0 (socket says 172.21.15.44:41946) 2024-05-27T09:44:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.289+0000 7f3baea00640 1 -- 172.21.15.44:0/1604603668 learned_addr learned my addr 172.21.15.44:0/1604603668 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.289+0000 7f3baf400640 1 --2- 172.21.15.44:0/1604603668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94c021b0 0x55cf94fa4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.290+0000 7f3ba7400640 1 -- 172.21.15.44:0/865228559 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3b9c010430 con 0x7f3bb010b7f0 2024-05-27T09:44:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.290+0000 7f3baf400640 1 -- 172.21.15.44:0/1604603668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 msgr2=0x55cf94fa4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.290+0000 7f3baf400640 1 --2- 172.21.15.44:0/1604603668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 0x55cf94fa4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.290+0000 7f3baf400640 1 -- 172.21.15.44:0/1604603668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94be14d0 msgr2=0x55cf94fa52e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:44:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.290+0000 7f3baf400640 1 --2- 172.21.15.44:0/1604603668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94be14d0 0x55cf94fa52e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.290+0000 7f3baf400640 1 -- 172.21.15.44:0/1604603668 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf94e9e8b0 con 0x55cf94c021b0 2024-05-27T09:44:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.291+0000 7f3baf400640 1 --2- 172.21.15.44:0/1604603668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94c021b0 0x55cf94fa4860 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8010710 tx=0x7f3ba8010bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3d001b0ab12ea1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.291+0000 7f3b9be00640 1 -- 172.21.15.44:0/1604603668 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8004b20 con 0x55cf94c021b0 2024-05-27T09:44:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.291+0000 7f3b9be00640 1 -- 172.21.15.44:0/1604603668 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ba800ed80 con 0x55cf94c021b0 2024-05-27T09:44:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.292+0000 7f3b9be00640 1 -- 172.21.15.44:0/1604603668 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8005600 con 0x55cf94c021b0 2024-05-27T09:44:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.292+0000 7f3ba7400640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3b940b2950 0x7f3b940b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.293+0000 7f3ba7400640 1 -- 172.21.15.44:0/865228559 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f3b9c0e29a0 con 0x7f3bb010b7f0 2024-05-27T09:44:44.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:43 smithi050 ceph-mon[36371]: pgmap v295: 97 pgs: 97 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 920 KiB/s wr, 442 op/s 2024-05-27T09:44:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.293+0000 7f3baea00640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3b940b2950 0x7f3b940b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.293+0000 7f3bbca29740 1 -- 172.21.15.44:0/1604603668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94c021b0 msgr2=0x55cf94fa4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.293+0000 7f3bbca29740 1 --2- 172.21.15.44:0/1604603668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94c021b0 0x55cf94fa4860 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8010710 tx=0x7f3ba8010bd0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.293+0000 7f3bbca29740 1 -- 172.21.15.44:0/1604603668 shutdown_connections 2024-05-27T09:44:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.293+0000 7f3bbca29740 1 --2- 172.21.15.44:0/1604603668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94be14d0 0x55cf94fa52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.293+0000 7f3bbca29740 1 --2- 172.21.15.44:0/1604603668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 0x55cf94fa4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.293+0000 7f3bbca29740 1 --2- 172.21.15.44:0/1604603668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94c021b0 0x55cf94fa4860 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.293+0000 7f3bbca29740 1 -- 172.21.15.44:0/1604603668 >> 172.21.15.44:0/1604603668 conn(0x55cf94bf6300 msgr2=0x55cf94fa1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.294+0000 7f3bbca29740 1 -- 172.21.15.44:0/1604603668 shutdown_connections 2024-05-27T09:44:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.294+0000 7f3bbca29740 1 -- 172.21.15.44:0/1604603668 wait complete. 2024-05-27T09:44:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.294+0000 7f3baea00640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3b940b2950 0x7f3b940b4e60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0000c00 tx=0x7f3ba0005a20 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.295+0000 7f3bbca29740 1 -- start start 2024-05-27T09:44:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.295+0000 7f3bbca29740 1 Processor -- start 2024-05-27T09:44:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94bd92f0 0x55cf94faecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94be14d0 0x55cf94faf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 0x55cf94fbacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3bbca29740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cf94beb790 con 0x55cf94bd92f0 2024-05-27T09:44:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3bbca29740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cf94eadbc0 con 0x55cf94c021b0 2024-05-27T09:44:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3bbca29740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cf94fb1990 con 0x55cf94be14d0 2024-05-27T09:44:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3bbca29740 10 client.0 _ll_register_callbacks cb 0x55cf94e8b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3baf400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 0x55cf94fbacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3bae000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94be14d0 0x55cf94faf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3baea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94bd92f0 0x55cf94faecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3bae000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94be14d0 0x55cf94faf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41958/0 (socket says 172.21.15.44:41958) 2024-05-27T09:44:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3baf400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 0x55cf94fbacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:55054/0 (socket says 172.21.15.44:55054) 2024-05-27T09:44:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.296+0000 7f3bae000640 1 -- 172.21.15.44:0/473288858 learned_addr learned my addr 172.21.15.44:0/473288858 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.297+0000 7f3bae000640 1 -- 172.21.15.44:0/473288858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 msgr2=0x55cf94fbacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.297+0000 7f3bae000640 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 0x55cf94fbacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.297+0000 7f3bae000640 1 -- 172.21.15.44:0/473288858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94bd92f0 msgr2=0x55cf94faecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.297+0000 7f3bae000640 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94bd92f0 0x55cf94faecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.297+0000 7f3bae000640 1 -- 172.21.15.44:0/473288858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf94c1a210 con 0x55cf94be14d0 2024-05-27T09:44:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.297+0000 7f3baf400640 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 0x55cf94fbacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.297+0000 7f3bae000640 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94be14d0 0x55cf94faf210 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00c6a0 tx=0x7f3b9c010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76646be49c3edf3c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.297+0000 7f3b98c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.297+0000 7f3b98c00640 1 -- 172.21.15.44:0/473288858 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c0e0520 con 0x55cf94be14d0 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.298+0000 7f3b98c00640 1 -- 172.21.15.44:0/473288858 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f3b9c0a8f30 con 0x55cf94be14d0 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.298+0000 7f3bbca29740 1 -- 172.21.15.44:0/473288858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ba800ed80 con 0x55cf94be14d0 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.298+0000 7f3b98c00640 1 -- 172.21.15.44:0/473288858 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3b9c0adfd0 con 0x55cf94be14d0 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.298+0000 7f3bbca29740 10 client.26992 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.298+0000 7f3bbca29740 1 -- 172.21.15.44:0/473288858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cf94ce6620 con 0x55cf94be14d0 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.298+0000 7f3b98c00640 1 -- 172.21.15.44:0/473288858 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ba800ed80 con 0x55cf94be14d0 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.298+0000 7f3b98c00640 1 -- 172.21.15.44:0/473288858 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f3b9c0a89a0 con 0x55cf94be14d0 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.299+0000 7f3bbca29740 10 client.26992 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.299+0000 7f3bbca29740 10 client.26992 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.299+0000 7f3bbca29740 20 client.26992 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.299+0000 7f3bbca29740 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf94bc0d80 0x55cf94fc0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.299+0000 7f3bbca29740 4 client.26992 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.299+0000 7f3bbca29740 1 -- 172.21.15.44:0/473288858 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf94f17ae0 con 0x55cf94bc0d80 2024-05-27T09:44:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.299+0000 7f3baea00640 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf94bc0d80 0x55cf94fc0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.300+0000 7f3baea00640 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf94bc0d80 0x55cf94fc0cd0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b02219e45d3ebcd server_cookie=a30edd1bbd73b0e5 in_seq=0 out_seq=0 2024-05-27T09:44:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.300+0000 7f3b98c00640 10 client.26992 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.301+0000 7f3b98c00640 1 -- 172.21.15.44:0/473288858 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cf94f17ae0 con 0x55cf94bc0d80 2024-05-27T09:44:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3b98c00640 10 client.26992 handle_command_reply: tid=0 2024-05-27T09:44:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3bbca29740 1 client.26992 shutdown 2024-05-27T09:44:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3bbca29740 1 -- 172.21.15.44:0/473288858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94be14d0 msgr2=0x55cf94faf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3bbca29740 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94be14d0 0x55cf94faf210 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00c6a0 tx=0x7f3b9c010860 comp rx=0 tx=0).stop 2024-05-27T09:44:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3bbca29740 1 -- 172.21.15.44:0/473288858 shutdown_connections 2024-05-27T09:44:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3bbca29740 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf94bc0d80 0x55cf94fc0cd0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3bbca29740 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 0x55cf94fbacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3bbca29740 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94be14d0 0x55cf94faf210 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3bbca29740 1 --2- 172.21.15.44:0/473288858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94bd92f0 0x55cf94faecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3bbca29740 1 -- 172.21.15.44:0/473288858 >> 172.21.15.44:0/473288858 conn(0x55cf94bf6300 msgr2=0x55cf94fa4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3b98c00640 0 client.26992 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3bbca29740 1 -- 172.21.15.44:0/473288858 shutdown_connections 2024-05-27T09:44:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.302+0000 7f3bbca29740 1 -- 172.21.15.44:0/473288858 wait complete. 2024-05-27T09:44:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.303+0000 7f3bbca29740 20 client.26992 trim_cache size 0 max 16384 2024-05-27T09:44:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.322+0000 7f3bbca29740 1 Processor -- start 2024-05-27T09:44:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.322+0000 7f3bbca29740 1 -- start start 2024-05-27T09:44:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.323+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94bd92f0 0x55cf94fa8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.323+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94be14d0 0x55cf94fbaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.323+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94c021b0 0x55cf94fbaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.323+0000 7f3bbca29740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb01b45c0 con 0x55cf94be14d0 2024-05-27T09:44:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.323+0000 7f3bbca29740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb01b1e90 con 0x55cf94bd92f0 2024-05-27T09:44:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.323+0000 7f3bbca29740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb01afd00 con 0x55cf94c021b0 2024-05-27T09:44:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.323+0000 7f3baea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94bd92f0 0x55cf94fa8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.323+0000 7f3baea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94bd92f0 0x55cf94fa8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:55056/0 (socket says 172.21.15.44:55056) 2024-05-27T09:44:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.323+0000 7f3baea00640 1 -- 172.21.15.44:0/1244929157 learned_addr learned my addr 172.21.15.44:0/1244929157 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.323+0000 7f3bae000640 1 --2- 172.21.15.44:0/1244929157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94be14d0 0x55cf94fbaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.323+0000 7f3baf400640 1 --2- 172.21.15.44:0/1244929157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94c021b0 0x55cf94fbaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.324+0000 7f3baea00640 1 -- 172.21.15.44:0/1244929157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94c021b0 msgr2=0x55cf94fbaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.324+0000 7f3baea00640 1 --2- 172.21.15.44:0/1244929157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94c021b0 0x55cf94fbaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.324+0000 7f3baea00640 1 -- 172.21.15.44:0/1244929157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94be14d0 msgr2=0x55cf94fbaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.324+0000 7f3baea00640 1 --2- 172.21.15.44:0/1244929157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94be14d0 0x55cf94fbaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.324+0000 7f3baea00640 1 -- 172.21.15.44:0/1244929157 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf94fb1870 con 0x55cf94bd92f0 2024-05-27T09:44:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.324+0000 7f3baea00640 1 --2- 172.21.15.44:0/1244929157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94bd92f0 0x55cf94fa8bc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0005ab0 tx=0x7f3ba003e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f421b6f8d2641b88 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.324+0000 7f3b9b400640 1 -- 172.21.15.44:0/1244929157 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0035040 con 0x55cf94bd92f0 2024-05-27T09:44:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.324+0000 7f3b9b400640 1 -- 172.21.15.44:0/1244929157 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ba0004c20 con 0x55cf94bd92f0 2024-05-27T09:44:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.324+0000 7f3b9b400640 1 -- 172.21.15.44:0/1244929157 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0003cb0 con 0x55cf94bd92f0 2024-05-27T09:44:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.325+0000 7f3bbca29740 1 -- 172.21.15.44:0/1244929157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94bd92f0 msgr2=0x55cf94fa8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.325+0000 7f3bbca29740 1 --2- 172.21.15.44:0/1244929157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94bd92f0 0x55cf94fa8bc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0005ab0 tx=0x7f3ba003e000 comp rx=0 tx=0).stop 2024-05-27T09:44:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.325+0000 7f3bbca29740 1 -- 172.21.15.44:0/1244929157 shutdown_connections 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.325+0000 7f3bbca29740 1 --2- 172.21.15.44:0/1244929157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94c021b0 0x55cf94fbaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.325+0000 7f3bbca29740 1 --2- 172.21.15.44:0/1244929157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94be14d0 0x55cf94fbaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.325+0000 7f3bbca29740 1 --2- 172.21.15.44:0/1244929157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94bd92f0 0x55cf94fa8bc0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.325+0000 7f3bbca29740 1 -- 172.21.15.44:0/1244929157 >> 172.21.15.44:0/1244929157 conn(0x55cf94bf6300 msgr2=0x55cf94fa84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.325+0000 7f3bbca29740 1 -- 172.21.15.44:0/1244929157 shutdown_connections 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.325+0000 7f3bbca29740 1 -- 172.21.15.44:0/1244929157 wait complete. 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.326+0000 7f3bbca29740 1 -- start start 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.326+0000 7f3bbca29740 1 Processor -- start 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 0x55cf94fb3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94be14d0 0x55cf94fb3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3bbca29740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 0x55cf94fb4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3bbca29740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb01b0350 con 0x55cf94be14d0 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3bbca29740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb01b17c0 con 0x55cf94c021b0 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3bbca29740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb01b0a20 con 0x55cf94bd92f0 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3bbca29740 10 client.0 _ll_register_callbacks cb 0x7f3b9c009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3baea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 0x55cf94fb3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3baea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 0x55cf94fb3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41968/0 (socket says 172.21.15.44:41968) 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3baea00640 1 -- 172.21.15.44:0/3163925377 learned_addr learned my addr 172.21.15.44:0/3163925377 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3baf400640 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 0x55cf94fb4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3bae000640 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94be14d0 0x55cf94fb3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3baea00640 1 -- 172.21.15.44:0/3163925377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 msgr2=0x55cf94fb4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3baea00640 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 0x55cf94fb4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3baea00640 1 -- 172.21.15.44:0/3163925377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94be14d0 msgr2=0x55cf94fb3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3baea00640 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94be14d0 0x55cf94fb3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3baea00640 1 -- 172.21.15.44:0/3163925377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf95027310 con 0x55cf94bd92f0 2024-05-27T09:44:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.327+0000 7f3bae000640 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94be14d0 0x55cf94fb3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.328+0000 7f3baea00640 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 0x55cf94fb3880 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0031440 tx=0x7f3ba00043c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f468155de912910 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.328+0000 7f3b98c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.328+0000 7f3b98c00640 1 -- 172.21.15.44:0/3163925377 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0004770 con 0x55cf94bd92f0 2024-05-27T09:44:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.328+0000 7f3bbca29740 1 -- 172.21.15.44:0/3163925377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ba0004c20 con 0x55cf94bd92f0 2024-05-27T09:44:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.328+0000 7f3b98c00640 1 -- 172.21.15.44:0/3163925377 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f3ba0004910 con 0x55cf94bd92f0 2024-05-27T09:44:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.328+0000 7f3b98c00640 1 -- 172.21.15.44:0/3163925377 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3ba006b050 con 0x55cf94bd92f0 2024-05-27T09:44:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.328+0000 7f3bbca29740 10 client.26998 fetch_fsmap learned FSMap version 32 2024-05-27T09:44:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.328+0000 7f3bbca29740 1 -- 172.21.15.44:0/3163925377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cf95028440 con 0x55cf94bd92f0 2024-05-27T09:44:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.329+0000 7f3b98c00640 1 -- 172.21.15.44:0/3163925377 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ba0004c20 con 0x55cf94bd92f0 2024-05-27T09:44:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.329+0000 7f3b98c00640 1 -- 172.21.15.44:0/3163925377 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f3ba0053820 con 0x55cf94bd92f0 2024-05-27T09:44:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.329+0000 7f3bbca29740 10 client.26998 fetch_fsmap finished waiting for FSMap version 32 2024-05-27T09:44:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.329+0000 7f3bbca29740 10 client.26998 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.329+0000 7f3bbca29740 20 client.26998 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.329+0000 7f3bbca29740 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf94b961c0 0x55cf9502ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.329+0000 7f3bbca29740 4 client.26998 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:44:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.329+0000 7f3bbca29740 1 -- 172.21.15.44:0/3163925377 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cf94f112c0 con 0x55cf94b961c0 2024-05-27T09:44:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.329+0000 7f3bae000640 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf94b961c0 0x55cf9502ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:44.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.330+0000 7f3bae000640 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf94b961c0 0x55cf9502ab00 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7ad511ca89ec784 server_cookie=10e43afc2c2853d2 in_seq=0 out_seq=0 2024-05-27T09:44:44.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:44.330+0000 7f3b98c00640 10 client.26998 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:43 smithi189 ceph-mon[36671]: pgmap v295: 97 pgs: 97 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 920 KiB/s wr, 442 op/s 2024-05-27T09:44:44.345 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:44:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.988+0000 7f3b98c00640 1 -- 172.21.15.44:0/3163925377 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55cf94f112c0 con 0x55cf94b961c0 2024-05-27T09:44:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.988+0000 7f3b98c00640 10 client.26998 handle_command_reply: tid=0 2024-05-27T09:44:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.988+0000 7f3bbca29740 1 client.26998 shutdown 2024-05-27T09:44:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.989+0000 7f3bbca29740 1 -- 172.21.15.44:0/3163925377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 msgr2=0x55cf94fb3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.989+0000 7f3bbca29740 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 0x55cf94fb3880 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0031440 tx=0x7f3ba00043c0 comp rx=0 tx=0).stop 2024-05-27T09:44:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.989+0000 7f3bbca29740 1 -- 172.21.15.44:0/3163925377 shutdown_connections 2024-05-27T09:44:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.989+0000 7f3bbca29740 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf94b961c0 0x55cf9502ab00 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.989+0000 7f3bbca29740 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf94c021b0 0x55cf94fb4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.989+0000 7f3bbca29740 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf94be14d0 0x55cf94fb3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.989+0000 7f3bbca29740 1 --2- 172.21.15.44:0/3163925377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf94bd92f0 0x55cf94fb3880 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.989+0000 7f3b98c00640 0 client.26998 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.989+0000 7f3bbca29740 1 -- 172.21.15.44:0/3163925377 >> 172.21.15.44:0/3163925377 conn(0x55cf94bf6300 msgr2=0x55cf94fb0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.989+0000 7f3bbca29740 1 -- 172.21.15.44:0/3163925377 shutdown_connections 2024-05-27T09:44:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.989+0000 7f3bbca29740 1 -- 172.21.15.44:0/3163925377 wait complete. 2024-05-27T09:44:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.990+0000 7f3bbca29740 20 client.26998 trim_cache size 0 max 16384 2024-05-27T09:44:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:44:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:44:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:44:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.991+0000 7f3bb5a00640 1 -- 172.21.15.44:0/865228559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3b940b2950 msgr2=0x7f3b940b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.991+0000 7f3bb5a00640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3b940b2950 0x7f3b940b4e60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0000c00 tx=0x7f3ba0005a20 comp rx=0 tx=0).stop 2024-05-27T09:44:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.991+0000 7f3bb5a00640 1 -- 172.21.15.44:0/865228559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb010b7f0 msgr2=0x7f3bb01a2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.991+0000 7f3bb5a00640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb010b7f0 0x7f3bb01a2ce0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00e9f0 tx=0x7f3b9c00eeb0 comp rx=0 tx=0).stop 2024-05-27T09:44:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.991+0000 7f3bb5a00640 1 -- 172.21.15.44:0/865228559 shutdown_connections 2024-05-27T09:44:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.991+0000 7f3bb5a00640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bb010e1e0 0x7f3bb019c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.991+0000 7f3bb5a00640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb010b7f0 0x7f3bb01a2ce0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.991+0000 7f3bb5a00640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3b940b2950 0x7f3b940b4e60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.991+0000 7f3bb5a00640 1 --2- 172.21.15.44:0/865228559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bb0104d90 0x7f3bb01a27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.991+0000 7f3bb5a00640 1 -- 172.21.15.44:0/865228559 >> 172.21.15.44:0/865228559 conn(0x7f3bb00782d0 msgr2=0x7f3bb0109c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.992+0000 7f3bb5a00640 1 -- 172.21.15.44:0/865228559 shutdown_connections 2024-05-27T09:44:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:45.992+0000 7f3bb5a00640 1 -- 172.21.15.44:0/865228559 wait complete. 2024-05-27T09:44:46.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:45 smithi044 ceph-mon[31618]: pgmap v296: 97 pgs: 97 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 920 KiB/s wr, 442 op/s 2024-05-27T09:44:46.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:45 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:44:46.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:45 smithi050 ceph-mon[36371]: pgmap v296: 97 pgs: 97 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 920 KiB/s wr, 442 op/s 2024-05-27T09:44:46.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:45 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:44:46.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:45 smithi189 ceph-mon[36671]: pgmap v296: 97 pgs: 97 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 920 KiB/s wr, 442 op/s 2024-05-27T09:44:46.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:45 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:44:46.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.592+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2326942169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c441030f0 msgr2=0x7f0c44103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:46.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.592+0000 7f0c4ae00700 1 --2- 172.21.15.44:0/2326942169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c441030f0 0x7f0c44103500 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009990 tx=0x7f0c34009ca0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.592+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2326942169 shutdown_connections 2024-05-27T09:44:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.592+0000 7f0c4ae00700 1 --2- 172.21.15.44:0/2326942169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c44110ca0 0x7f0c441130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.592+0000 7f0c4ae00700 1 --2- 172.21.15.44:0/2326942169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c441030f0 0x7f0c44103500 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.592+0000 7f0c4ae00700 1 --2- 172.21.15.44:0/2326942169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c44103a40 0x7f0c44110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.592+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2326942169 >> 172.21.15.44:0/2326942169 conn(0x7f0c440fedb0 msgr2=0x7f0c441011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.592+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2326942169 shutdown_connections 2024-05-27T09:44:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.592+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2326942169 wait complete. 2024-05-27T09:44:46.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.594+0000 7f0c4ae00700 1 Processor -- start 2024-05-27T09:44:46.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.594+0000 7f0c4ae00700 1 -- start start 2024-05-27T09:44:46.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.594+0000 7f0c4ae00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c441030f0 0x7f0c4419cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.594+0000 7f0c4ae00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c44103a40 0x7f0c4419d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.594+0000 7f0c4ae00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c44110ca0 0x7f0c441a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.594+0000 7f0c4ae00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c44116650 con 0x7f0c441030f0 2024-05-27T09:44:46.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.594+0000 7f0c4ae00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0c441164d0 con 0x7f0c44110ca0 2024-05-27T09:44:46.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.594+0000 7f0c4ae00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0c441167d0 con 0x7f0c44103a40 2024-05-27T09:44:46.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c48e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c44110ca0 0x7f0c441a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c48e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c44110ca0 0x7f0c441a17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:55080/0 (socket says 172.21.15.44:55080) 2024-05-27T09:44:46.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c48e00700 1 -- 172.21.15.44:0/2200732479 learned_addr learned my addr 172.21.15.44:0/2200732479 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:46.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c3be00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c44103a40 0x7f0c4419d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c43e00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c441030f0 0x7f0c4419cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c48e00700 1 -- 172.21.15.44:0/2200732479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c44103a40 msgr2=0x7f0c4419d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:46.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c48e00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c44103a40 0x7f0c4419d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c48e00700 1 -- 172.21.15.44:0/2200732479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c441030f0 msgr2=0x7f0c4419cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:46.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c48e00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c441030f0 0x7f0c4419cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c48e00700 1 -- 172.21.15.44:0/2200732479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c441a1e90 con 0x7f0c44110ca0 2024-05-27T09:44:46.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c43e00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c441030f0 0x7f0c4419cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:44:46.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.595+0000 7f0c48e00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c44110ca0 0x7f0c441a17f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00c370 tx=0x7f0c3c00c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:46.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.596+0000 7f0c41600700 1 -- 172.21.15.44:0/2200732479 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c00ce20 con 0x7f0c44110ca0 2024-05-27T09:44:46.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.596+0000 7f0c41600700 1 -- 172.21.15.44:0/2200732479 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c3c0090d0 con 0x7f0c44110ca0 2024-05-27T09:44:46.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.596+0000 7f0c41600700 1 -- 172.21.15.44:0/2200732479 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c004730 con 0x7f0c44110ca0 2024-05-27T09:44:46.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.597+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2200732479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c441a2120 con 0x7f0c44110ca0 2024-05-27T09:44:46.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.597+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2200732479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c441a2610 con 0x7f0c44110ca0 2024-05-27T09:44:46.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.602+0000 7f0c41600700 1 -- 172.21.15.44:0/2200732479 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0c3c01a020 con 0x7f0c44110ca0 2024-05-27T09:44:46.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.603+0000 7f0c59a0ab80 1 Processor -- start 2024-05-27T09:44:46.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.604+0000 7f0c59a0ab80 1 -- start start 2024-05-27T09:44:46.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.604+0000 7f0c41600700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c280b2b40 0x7f0c280b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.604+0000 7f0c3be00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c280b2b40 0x7f0c280b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.604+0000 7f0c41600700 1 -- 172.21.15.44:0/2200732479 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f0c3c0e1f00 con 0x7f0c44110ca0 2024-05-27T09:44:46.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.604+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7a7fe40 0x5595d7c047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.604+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7c04d30 0x5595d7bfe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.604+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7bff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.604+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5595d7b2ce40 con 0x5595d7c04d30 2024-05-27T09:44:46.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.604+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5595d7b1c9e0 con 0x5595d7a7fe40 2024-05-27T09:44:46.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.604+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5595d7b20a20 con 0x5595d7bfedb0 2024-05-27T09:44:46.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.605+0000 7f0c48e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7bff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.605+0000 7f0c48e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7bff1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:41998/0 (socket says 172.21.15.44:41998) 2024-05-27T09:44:46.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.605+0000 7f0c48e00700 1 -- 172.21.15.44:0/2904970942 learned_addr learned my addr 172.21.15.44:0/2904970942 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:46.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.605+0000 7f0c3be00700 1 --2- 172.21.15.44:0/2904970942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7c04d30 0x5595d7bfe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.605+0000 7f0c43e00700 1 --2- 172.21.15.44:0/2904970942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7a7fe40 0x5595d7c047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.606+0000 7f0c43e00700 1 -- 172.21.15.44:0/2904970942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 msgr2=0x5595d7bff1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:46.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.606+0000 7f0c43e00700 1 --2- 172.21.15.44:0/2904970942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7bff1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.606+0000 7f0c43e00700 1 -- 172.21.15.44:0/2904970942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7c04d30 msgr2=0x5595d7bfe870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:46.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.606+0000 7f0c43e00700 1 --2- 172.21.15.44:0/2904970942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7c04d30 0x5595d7bfe870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.606+0000 7f0c43e00700 1 -- 172.21.15.44:0/2904970942 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595d7ae5420 con 0x5595d7a7fe40 2024-05-27T09:44:46.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.607+0000 7f0c43e00700 1 --2- 172.21.15.44:0/2904970942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7a7fe40 0x5595d7c047f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400b2e0 tx=0x7f0c34005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6cdc7d1c4bf5378 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:46.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.607+0000 7f0c3be00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c280b2b40 0x7f0c280b5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0c441049e0 tx=0x7f0c2c013900 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:46.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.607+0000 7f0c39600700 1 -- 172.21.15.44:0/2904970942 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c340095d0 con 0x5595d7a7fe40 2024-05-27T09:44:46.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.607+0000 7f0c39600700 1 -- 172.21.15.44:0/2904970942 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c34004830 con 0x5595d7a7fe40 2024-05-27T09:44:46.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.607+0000 7f0c39600700 1 -- 172.21.15.44:0/2904970942 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34005380 con 0x5595d7a7fe40 2024-05-27T09:44:46.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.608+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/2904970942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7a7fe40 msgr2=0x5595d7c047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:46.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.608+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/2904970942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7a7fe40 0x5595d7c047f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400b2e0 tx=0x7f0c34005e80 comp rx=0 tx=0).stop 2024-05-27T09:44:46.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.609+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/2904970942 shutdown_connections 2024-05-27T09:44:46.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.609+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/2904970942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7bff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.609+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/2904970942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7a7fe40 0x5595d7c047f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.609+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/2904970942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7c04d30 0x5595d7bfe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.609+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/2904970942 >> 172.21.15.44:0/2904970942 conn(0x5595d7adffc0 msgr2=0x5595d7bf98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:46.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.609+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/2904970942 shutdown_connections 2024-05-27T09:44:46.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.610+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/2904970942 wait complete. 2024-05-27T09:44:46.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.610+0000 7f0c59a0ab80 1 -- start start 2024-05-27T09:44:46.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.611+0000 7f0c59a0ab80 1 Processor -- start 2024-05-27T09:44:46.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.612+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 0x5595d7c08510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.612+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c08a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.612+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c08f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.612+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5595d7a22fa0 con 0x5595d7a7fe40 2024-05-27T09:44:46.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.612+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5595d7abd760 con 0x5595d7c04d30 2024-05-27T09:44:46.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.612+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5595d7c0b740 con 0x5595d7bfedb0 2024-05-27T09:44:46.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.612+0000 7f0c59a0ab80 10 client.0 _ll_register_callbacks cb 0x5595d7ac6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:46.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.612+0000 7f0c3be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c08a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.612+0000 7f0c3be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c08a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42004/0 (socket says 172.21.15.44:42004) 2024-05-27T09:44:46.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.612+0000 7f0c3be00700 1 -- 172.21.15.44:0/564872714 learned_addr learned my addr 172.21.15.44:0/564872714 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:46.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.613+0000 7f0c43e00700 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 0x5595d7c08510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.613+0000 7f0c3be00700 1 -- 172.21.15.44:0/564872714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 msgr2=0x5595d7c08f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:46.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.613+0000 7f0c48e00700 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c08f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.614+0000 7f0c3be00700 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c08f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.614+0000 7f0c3be00700 1 -- 172.21.15.44:0/564872714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 msgr2=0x5595d7c08510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:46.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.614+0000 7f0c3be00700 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 0x5595d7c08510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.614+0000 7f0c3be00700 1 -- 172.21.15.44:0/564872714 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595d7a6c8d0 con 0x5595d7bfedb0 2024-05-27T09:44:46.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.614+0000 7f0c48e00700 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c08f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:44:46.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.614+0000 7f0c43e00700 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 0x5595d7c08510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:46.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.614+0000 7f0c3be00700 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c08a50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0218e0 tx=0x7f0c2c01b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1e3d9efe4664792 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:46.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.614+0000 7f0c32000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:46.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.615+0000 7f0c32000700 1 -- 172.21.15.44:0/564872714 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c004220 con 0x5595d7bfedb0 2024-05-27T09:44:46.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.615+0000 7f0c32000700 1 -- 172.21.15.44:0/564872714 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f0c2c0043c0 con 0x5595d7bfedb0 2024-05-27T09:44:46.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.615+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/564872714 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c34004830 con 0x5595d7bfedb0 2024-05-27T09:44:46.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.615+0000 7f0c32000700 1 -- 172.21.15.44:0/564872714 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0c2c003680 con 0x5595d7bfedb0 2024-05-27T09:44:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.615+0000 7f0c59a0ab80 10 client.27010 fetch_fsmap learned FSMap version 33 2024-05-27T09:44:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.615+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/564872714 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5595d7aed060 con 0x5595d7bfedb0 2024-05-27T09:44:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.616+0000 7f0c32000700 1 -- 172.21.15.44:0/564872714 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c34004830 con 0x5595d7bfedb0 2024-05-27T09:44:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.616+0000 7f0c32000700 1 -- 172.21.15.44:0/564872714 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f0c2c020050 con 0x5595d7bfedb0 2024-05-27T09:44:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.616+0000 7f0c59a0ab80 10 client.27010 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:44:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.616+0000 7f0c59a0ab80 10 client.27010 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.616+0000 7f0c59a0ab80 20 client.27010 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.616+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5595d7c186f0 0x5595d7c1ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.616+0000 7f0c59a0ab80 4 client.27010 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:46.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.616+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/564872714 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595d7b2ed90 con 0x5595d7c186f0 2024-05-27T09:44:46.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.617+0000 7f0c43e00700 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5595d7c186f0 0x5595d7c1ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.618+0000 7f0c32000700 10 client.27010 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:46.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.618+0000 7f0c43e00700 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5595d7c186f0 0x5595d7c1ab10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd7f71b6be0af797 server_cookie=df64c29e22bad2f4 in_seq=0 out_seq=0 2024-05-27T09:44:46.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.619+0000 7f0c32000700 1 -- 172.21.15.44:0/564872714 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5595d7b2ed90 con 0x5595d7c186f0 2024-05-27T09:44:46.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.619+0000 7f0c32000700 10 client.27010 handle_command_reply: tid=0 2024-05-27T09:44:46.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.620+0000 7f0c59a0ab80 1 client.27010 shutdown 2024-05-27T09:44:46.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.620+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/564872714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 msgr2=0x5595d7c08a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:46.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.620+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c08a50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0218e0 tx=0x7f0c2c01b000 comp rx=0 tx=0).stop 2024-05-27T09:44:46.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.620+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/564872714 shutdown_connections 2024-05-27T09:44:46.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.620+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5595d7c186f0 0x5595d7c1ab10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.620+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c08f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.620+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 0x5595d7c08510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.620+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/564872714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c08a50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.620+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/564872714 >> 172.21.15.44:0/564872714 conn(0x5595d7adffc0 msgr2=0x5595d7bf9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.620+0000 7f0c32000700 0 client.27010 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.621+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/564872714 shutdown_connections 2024-05-27T09:44:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.621+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/564872714 wait complete. 2024-05-27T09:44:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.621+0000 7f0c59a0ab80 20 client.27010 trim_cache size 0 max 16384 2024-05-27T09:44:46.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.648+0000 7f0c59a0ab80 1 Processor -- start 2024-05-27T09:44:46.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.648+0000 7f0c59a0ab80 1 -- start start 2024-05-27T09:44:46.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.648+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7a7fe40 0x5595d7bfe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.648+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7bfedb0 0x5595d7c0a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.648+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c147e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.648+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c441b3880 con 0x5595d7bfedb0 2024-05-27T09:44:46.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.648+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0c441b10e0 con 0x5595d7c04d30 2024-05-27T09:44:46.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.648+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0c441aef50 con 0x5595d7a7fe40 2024-05-27T09:44:46.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.649+0000 7f0c43e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7a7fe40 0x5595d7bfe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.649+0000 7f0c43e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7a7fe40 0x5595d7bfe870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42008/0 (socket says 172.21.15.44:42008) 2024-05-27T09:44:46.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.649+0000 7f0c43e00700 1 -- 172.21.15.44:0/1865203453 learned_addr learned my addr 172.21.15.44:0/1865203453 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:46.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.649+0000 7f0c48e00700 1 --2- 172.21.15.44:0/1865203453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c147e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.649+0000 7f0c3be00700 1 --2- 172.21.15.44:0/1865203453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7bfedb0 0x5595d7c0a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.649+0000 7f0c43e00700 1 -- 172.21.15.44:0/1865203453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 msgr2=0x5595d7c147e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:46.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.649+0000 7f0c43e00700 1 --2- 172.21.15.44:0/1865203453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c147e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.649+0000 7f0c43e00700 1 -- 172.21.15.44:0/1865203453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7bfedb0 msgr2=0x5595d7c0a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:46.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.649+0000 7f0c43e00700 1 --2- 172.21.15.44:0/1865203453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7bfedb0 0x5595d7c0a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.649+0000 7f0c43e00700 1 -- 172.21.15.44:0/1865203453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595d7c06a80 con 0x5595d7a7fe40 2024-05-27T09:44:46.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.649+0000 7f0c3be00700 1 --2- 172.21.15.44:0/1865203453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7bfedb0 0x5595d7c0a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:44:46.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.650+0000 7f0c43e00700 1 --2- 172.21.15.44:0/1865203453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7a7fe40 0x5595d7bfe870 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009990 tx=0x7f0c34005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d52b0508a20eaf8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:46.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.650+0000 7f0c38c00700 1 -- 172.21.15.44:0/1865203453 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34004830 con 0x5595d7a7fe40 2024-05-27T09:44:46.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.650+0000 7f0c38c00700 1 -- 172.21.15.44:0/1865203453 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c34021920 con 0x5595d7a7fe40 2024-05-27T09:44:46.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.650+0000 7f0c38c00700 1 -- 172.21.15.44:0/1865203453 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3400f880 con 0x5595d7a7fe40 2024-05-27T09:44:46.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.651+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/1865203453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7a7fe40 msgr2=0x5595d7bfe870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.651+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/1865203453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7a7fe40 0x5595d7bfe870 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009990 tx=0x7f0c34005ef0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.651+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/1865203453 shutdown_connections 2024-05-27T09:44:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.651+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/1865203453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c147e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.651+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/1865203453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7a7fe40 0x5595d7bfe870 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.651+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/1865203453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7bfedb0 0x5595d7c0a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.651+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/1865203453 >> 172.21.15.44:0/1865203453 conn(0x5595d7adffc0 msgr2=0x5595d7c08d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.651+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/1865203453 shutdown_connections 2024-05-27T09:44:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.652+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/1865203453 wait complete. 2024-05-27T09:44:46.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.652+0000 7f0c59a0ab80 1 -- start start 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.652+0000 7f0c59a0ab80 1 Processor -- start 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.653+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 0x5595d7c109d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.653+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c10f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.653+0000 7f0c59a0ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c1f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.653+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c441af5a0 con 0x5595d7a7fe40 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.653+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0c441b0a10 con 0x5595d7c04d30 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.653+0000 7f0c59a0ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0c441afc70 con 0x5595d7bfedb0 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.653+0000 7f0c59a0ab80 10 client.0 _ll_register_callbacks cb 0x5595d7ac6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.653+0000 7f0c3be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c10f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.653+0000 7f0c3be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c10f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42010/0 (socket says 172.21.15.44:42010) 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.653+0000 7f0c3be00700 1 -- 172.21.15.44:0/3333201438 learned_addr learned my addr 172.21.15.44:0/3333201438 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:46.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.654+0000 7f0c48e00700 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c1f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.654+0000 7f0c43e00700 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 0x5595d7c109d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.654+0000 7f0c3be00700 1 -- 172.21.15.44:0/3333201438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 msgr2=0x5595d7c1f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:46.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.654+0000 7f0c3be00700 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c1f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.654+0000 7f0c3be00700 1 -- 172.21.15.44:0/3333201438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 msgr2=0x5595d7c109d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:46.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.654+0000 7f0c3be00700 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 0x5595d7c109d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.654+0000 7f0c3be00700 1 -- 172.21.15.44:0/3333201438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595d7c1f970 con 0x5595d7bfedb0 2024-05-27T09:44:46.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.654+0000 7f0c43e00700 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 0x5595d7c109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:46.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.654+0000 7f0c3be00700 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c10f10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c013fd0 tx=0x7f0c2c01fdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ad0d9d81fdb2c6e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:46.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.654+0000 7f0c32000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:46.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.655+0000 7f0c32000700 1 -- 172.21.15.44:0/3333201438 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c01f5f0 con 0x5595d7bfedb0 2024-05-27T09:44:46.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.655+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/3333201438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c34021920 con 0x5595d7bfedb0 2024-05-27T09:44:46.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.655+0000 7f0c32000700 1 -- 172.21.15.44:0/3333201438 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f0c2c010430 con 0x5595d7bfedb0 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.655+0000 7f0c32000700 1 -- 172.21.15.44:0/3333201438 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0c2c044050 con 0x5595d7bfedb0 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.655+0000 7f0c59a0ab80 10 client.27016 fetch_fsmap learned FSMap version 33 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.655+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/3333201438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5595d7c8b110 con 0x5595d7bfedb0 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.655+0000 7f0c32000700 1 -- 172.21.15.44:0/3333201438 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c34021920 con 0x5595d7bfedb0 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.656+0000 7f0c32000700 1 -- 172.21.15.44:0/3333201438 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f0c2c029550 con 0x5595d7bfedb0 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.656+0000 7f0c59a0ab80 10 client.27016 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.656+0000 7f0c59a0ab80 10 client.27016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.656+0000 7f0c59a0ab80 20 client.27016 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.656+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5595d7c8b850 0x5595d7c8dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.656+0000 7f0c59a0ab80 4 client.27016 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.656+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/3333201438 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5595d7b28570 con 0x5595d7c8b850 2024-05-27T09:44:46.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.656+0000 7f0c43e00700 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5595d7c8b850 0x5595d7c8dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:46.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.657+0000 7f0c43e00700 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5595d7c8b850 0x5595d7c8dc70 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c64d0b1a3ef6df69 server_cookie=eadb7844b1ff9457 in_seq=0 out_seq=0 2024-05-27T09:44:46.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.657+0000 7f0c32000700 10 client.27016 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:46.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.659+0000 7f0c32000700 1 -- 172.21.15.44:0/3333201438 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5595d7b28570 con 0x5595d7c8b850 2024-05-27T09:44:46.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.659+0000 7f0c32000700 10 client.27016 handle_command_reply: tid=0 2024-05-27T09:44:46.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.659+0000 7f0c59a0ab80 1 client.27016 shutdown 2024-05-27T09:44:46.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.659+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/3333201438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 msgr2=0x5595d7c10f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:46.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.659+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c10f10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c013fd0 tx=0x7f0c2c01fdc0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.659+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/3333201438 shutdown_connections 2024-05-27T09:44:46.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.659+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5595d7c8b850 0x5595d7c8dc70 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.659+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595d7c04d30 0x5595d7c1f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.659+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5595d7a7fe40 0x5595d7c109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.659+0000 7f0c59a0ab80 1 --2- 172.21.15.44:0/3333201438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5595d7bfedb0 0x5595d7c10f10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.660+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/3333201438 >> 172.21.15.44:0/3333201438 conn(0x5595d7adffc0 msgr2=0x5595d7c08d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:46.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.660+0000 7f0c32000700 0 client.27016 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:46.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.660+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/3333201438 shutdown_connections 2024-05-27T09:44:46.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.660+0000 7f0c59a0ab80 1 -- 172.21.15.44:0/3333201438 wait complete. 2024-05-27T09:44:46.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.660+0000 7f0c59a0ab80 20 client.27016 trim_cache size 0 max 16384 2024-05-27T09:44:46.662 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:44:46.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.661+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2200732479 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c280b2b40 msgr2=0x7f0c280b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.661+0000 7f0c4ae00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c280b2b40 0x7f0c280b5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0c441049e0 tx=0x7f0c2c013900 comp rx=0 tx=0).stop 2024-05-27T09:44:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.661+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2200732479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c44110ca0 msgr2=0x7f0c441a17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.661+0000 7f0c4ae00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c44110ca0 0x7f0c441a17f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00c370 tx=0x7f0c3c00c680 comp rx=0 tx=0).stop 2024-05-27T09:44:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.661+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2200732479 shutdown_connections 2024-05-27T09:44:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.661+0000 7f0c4ae00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c280b2b40 0x7f0c280b5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.661+0000 7f0c4ae00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c44110ca0 0x7f0c441a17f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.661+0000 7f0c4ae00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c441030f0 0x7f0c4419cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.661+0000 7f0c4ae00700 1 --2- 172.21.15.44:0/2200732479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c44103a40 0x7f0c4419d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.661+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2200732479 >> 172.21.15.44:0/2200732479 conn(0x7f0c440fedb0 msgr2=0x7f0c441127b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.662+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2200732479 shutdown_connections 2024-05-27T09:44:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:46.662+0000 7f0c4ae00700 1 -- 172.21.15.44:0/2200732479 wait complete. 2024-05-27T09:44:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:47 smithi044 ceph-mon[31618]: pgmap v297: 97 pgs: 97 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 498 op/s 2024-05-27T09:44:48.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:47 smithi050 ceph-mon[36371]: pgmap v297: 97 pgs: 97 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 498 op/s 2024-05-27T09:44:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:47 smithi189 ceph-mon[36671]: pgmap v297: 97 pgs: 97 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 498 op/s 2024-05-27T09:44:48.514 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:44:48.514 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:44:49.023 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:44:50.211 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:49 smithi044 ceph-mon[31618]: pgmap v298: 97 pgs: 97 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 489 op/s 2024-05-27T09:44:50.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:49 smithi050 ceph-mon[36371]: pgmap v298: 97 pgs: 97 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 489 op/s 2024-05-27T09:44:50.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:49 smithi189 ceph-mon[36671]: pgmap v298: 97 pgs: 97 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 489 op/s 2024-05-27T09:44:51.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.392+0000 7f5947e00700 1 -- 172.21.15.44:0/735362272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5948108d20 msgr2=0x7f5948109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:51.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.392+0000 7f5947e00700 1 --2- 172.21.15.44:0/735362272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5948108d20 0x7f5948109130 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f59340099c0 tx=0x7f5934009cd0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.392+0000 7f5947e00700 1 -- 172.21.15.44:0/735362272 shutdown_connections 2024-05-27T09:44:51.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.392+0000 7f5947e00700 1 --2- 172.21.15.44:0/735362272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5948109670 0x7f5948110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.392+0000 7f5947e00700 1 --2- 172.21.15.44:0/735362272 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59481043f0 0x7f5948104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.392+0000 7f5947e00700 1 --2- 172.21.15.44:0/735362272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5948108d20 0x7f5948109130 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.392+0000 7f5947e00700 1 -- 172.21.15.44:0/735362272 >> 172.21.15.44:0/735362272 conn(0x7f5948079d00 msgr2=0x7f594807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:51.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.392+0000 7f5947e00700 1 -- 172.21.15.44:0/735362272 shutdown_connections 2024-05-27T09:44:51.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.392+0000 7f5947e00700 1 -- 172.21.15.44:0/735362272 wait complete. 2024-05-27T09:44:51.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.394+0000 7f5947e00700 1 Processor -- start 2024-05-27T09:44:51.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.394+0000 7f5947e00700 1 -- start start 2024-05-27T09:44:51.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.394+0000 7f5947e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59481043f0 0x7f5948075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.394+0000 7f5947e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5948108d20 0x7f5948075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.394+0000 7f5947e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5948109670 0x7f5948076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.394+0000 7f5947e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5948114280 con 0x7f5948109670 2024-05-27T09:44:51.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.394+0000 7f5947e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5948114100 con 0x7f59481043f0 2024-05-27T09:44:51.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.394+0000 7f5947e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5948114400 con 0x7f5948108d20 2024-05-27T09:44:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.395+0000 7f5946a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59481043f0 0x7f5948075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.395+0000 7f5946a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59481043f0 0x7f5948075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:55136/0 (socket says 172.21.15.44:55136) 2024-05-27T09:44:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.395+0000 7f5946000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5948108d20 0x7f5948075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.395+0000 7f5946a00700 1 -- 172.21.15.44:0/4050491639 learned_addr learned my addr 172.21.15.44:0/4050491639 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.395+0000 7f5947400700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5948109670 0x7f5948076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.395+0000 7f5946a00700 1 -- 172.21.15.44:0/4050491639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5948108d20 msgr2=0x7f5948075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.395+0000 7f5946a00700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5948108d20 0x7f5948075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.395+0000 7f5946a00700 1 -- 172.21.15.44:0/4050491639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5948109670 msgr2=0x7f5948076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.395+0000 7f5946a00700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5948109670 0x7f5948076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.395+0000 7f5946a00700 1 -- 172.21.15.44:0/4050491639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59481a9030 con 0x7f59481043f0 2024-05-27T09:44:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.395+0000 7f5946a00700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59481043f0 0x7f5948075ab0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f593800eb30 tx=0x7f593800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.396+0000 7f593f400700 1 -- 172.21.15.44:0/4050491639 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593800cc90 con 0x7f59481043f0 2024-05-27T09:44:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.396+0000 7f593f400700 1 -- 172.21.15.44:0/4050491639 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59380044d0 con 0x7f59481043f0 2024-05-27T09:44:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.396+0000 7f593f400700 1 -- 172.21.15.44:0/4050491639 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5938010430 con 0x7f59481043f0 2024-05-27T09:44:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.396+0000 7f5947e00700 1 -- 172.21.15.44:0/4050491639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59481a9320 con 0x7f59481043f0 2024-05-27T09:44:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.396+0000 7f5947e00700 1 -- 172.21.15.44:0/4050491639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59481a9880 con 0x7f59481043f0 2024-05-27T09:44:51.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.400+0000 7f593f400700 1 -- 172.21.15.44:0/4050491639 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5938003ea0 con 0x7f59481043f0 2024-05-27T09:44:51.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.401+0000 7f595c5c1b80 1 Processor -- start 2024-05-27T09:44:51.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.401+0000 7f595c5c1b80 1 -- start start 2024-05-27T09:44:51.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f593f400700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f592c0b2a70 0x7f592c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe4fda70 0x5613fe5917d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe591d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f593f400700 1 -- 172.21.15.44:0/4050491639 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f59380e2890 con 0x7f59481043f0 2024-05-27T09:44:51.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe59b340 0x5613fe59d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5613fe4beec0 con 0x5613fe4fda70 2024-05-27T09:44:51.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5613fe4aea60 con 0x5613fe59b340 2024-05-27T09:44:51.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5613fe4b2aa0 con 0x5613fe411ec0 2024-05-27T09:44:51.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f5947400700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f592c0b2a70 0x7f592c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f5946000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe591d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f5946000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe591d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42046/0 (socket says 172.21.15.44:42046) 2024-05-27T09:44:51.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.402+0000 7f5946000700 1 -- 172.21.15.44:0/3112547796 learned_addr learned my addr 172.21.15.44:0/3112547796 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:51.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.403+0000 7f5946a00700 1 --2- 172.21.15.44:0/3112547796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe4fda70 0x5613fe5917d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.403+0000 7f5946000700 1 -- 172.21.15.44:0/3112547796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe59b340 msgr2=0x5613fe59d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:51.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.403+0000 7f5946000700 1 --2- 172.21.15.44:0/3112547796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe59b340 0x5613fe59d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.403+0000 7f5946000700 1 -- 172.21.15.44:0/3112547796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe4fda70 msgr2=0x5613fe5917d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:51.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.403+0000 7f5946000700 1 --2- 172.21.15.44:0/3112547796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe4fda70 0x5613fe5917d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.403+0000 7f5946000700 1 -- 172.21.15.44:0/3112547796 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613fe4774a0 con 0x5613fe411ec0 2024-05-27T09:44:51.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.403+0000 7f5946000700 1 --2- 172.21.15.44:0/3112547796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe591d10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f59340099c0 tx=0x7f5934005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b8fa85a1aad67ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:51.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.404+0000 7f5947400700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f592c0b2a70 0x7f592c0b4f80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x5613fe593290 tx=0x7f5940009380 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:51.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.404+0000 7f5933e00700 1 -- 172.21.15.44:0/3112547796 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5934027070 con 0x5613fe411ec0 2024-05-27T09:44:51.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.404+0000 7f5933e00700 1 -- 172.21.15.44:0/3112547796 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59340037c0 con 0x5613fe411ec0 2024-05-27T09:44:51.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.404+0000 7f5933e00700 1 -- 172.21.15.44:0/3112547796 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593400bb10 con 0x5613fe411ec0 2024-05-27T09:44:51.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.405+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3112547796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 msgr2=0x5613fe591d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:51.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.405+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/3112547796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe591d10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f59340099c0 tx=0x7f5934005040 comp rx=0 tx=0).stop 2024-05-27T09:44:51.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.405+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3112547796 shutdown_connections 2024-05-27T09:44:51.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.405+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/3112547796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe59b340 0x5613fe59d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.405+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/3112547796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe4fda70 0x5613fe5917d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.405+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/3112547796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe591d10 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.405+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3112547796 >> 172.21.15.44:0/3112547796 conn(0x5613fe4ab410 msgr2=0x5613fe58c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:51.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.405+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3112547796 shutdown_connections 2024-05-27T09:44:51.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.405+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3112547796 wait complete. 2024-05-27T09:44:51.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.406+0000 7f595c5c1b80 1 -- start start 2024-05-27T09:44:51.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.406+0000 7f595c5c1b80 1 Processor -- start 2024-05-27T09:44:51.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.407+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe411ec0 0x5613fe5982b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.407+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5987f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.407+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe59b340 0x5613fe598d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.407+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5613fe3b5020 con 0x5613fe411ec0 2024-05-27T09:44:51.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.407+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5613fe44f7e0 con 0x5613fe4fda70 2024-05-27T09:44:51.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.407+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5613fe59de20 con 0x5613fe59b340 2024-05-27T09:44:51.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.407+0000 7f595c5c1b80 10 client.0 _ll_register_callbacks cb 0x5613fe458cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:51.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.407+0000 7f5946a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe411ec0 0x5613fe5982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.407+0000 7f5946a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe411ec0 0x5613fe5982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38634/0 (socket says 172.21.15.44:38634) 2024-05-27T09:44:51.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.407+0000 7f5946a00700 1 -- 172.21.15.44:0/1954082830 learned_addr learned my addr 172.21.15.44:0/1954082830 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:51.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.407+0000 7f5946000700 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.408+0000 7f5947400700 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe59b340 0x5613fe598d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.408+0000 7f5946000700 1 -- 172.21.15.44:0/1954082830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe59b340 msgr2=0x5613fe598d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.408+0000 7f5946000700 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe59b340 0x5613fe598d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.408+0000 7f5946000700 1 -- 172.21.15.44:0/1954082830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe411ec0 msgr2=0x5613fe5982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.408+0000 7f5946000700 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe411ec0 0x5613fe5982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.408+0000 7f5946000700 1 -- 172.21.15.44:0/1954082830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613fe3fe950 con 0x5613fe4fda70 2024-05-27T09:44:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.408+0000 7f5946000700 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5987f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5934005010 tx=0x7f5934005360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d44c71a07c5315f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.408+0000 7f5930c00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:44:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.409+0000 7f5930c00700 1 -- 172.21.15.44:0/1954082830 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5934027050 con 0x5613fe4fda70 2024-05-27T09:44:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.409+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1954082830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59340037c0 con 0x5613fe4fda70 2024-05-27T09:44:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.409+0000 7f5930c00700 1 -- 172.21.15.44:0/1954082830 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f593400b790 con 0x5613fe4fda70 2024-05-27T09:44:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.409+0000 7f5930c00700 1 -- 172.21.15.44:0/1954082830 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f593403b050 con 0x5613fe4fda70 2024-05-27T09:44:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.409+0000 7f595c5c1b80 10 client.27041 fetch_fsmap learned FSMap version 33 2024-05-27T09:44:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.409+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1954082830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5613fe47f0e0 con 0x5613fe4fda70 2024-05-27T09:44:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.409+0000 7f5930c00700 1 -- 172.21.15.44:0/1954082830 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59340037c0 con 0x5613fe4fda70 2024-05-27T09:44:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.409+0000 7f5930c00700 1 -- 172.21.15.44:0/1954082830 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f5934021030 con 0x5613fe4fda70 2024-05-27T09:44:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.409+0000 7f595c5c1b80 10 client.27041 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:44:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.409+0000 7f595c5c1b80 10 client.27041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:51.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.409+0000 7f595c5c1b80 20 client.27041 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:51.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.410+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5613fe5aaba0 0x5613fe5acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.410+0000 7f595c5c1b80 4 client.27041 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:51.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.410+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1954082830 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613fe4c0e10 con 0x5613fe5aaba0 2024-05-27T09:44:51.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.410+0000 7f5946a00700 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5613fe5aaba0 0x5613fe5acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.411+0000 7f5946a00700 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5613fe5aaba0 0x5613fe5acfc0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a294cc3b2387ee03 server_cookie=b7364bd83e847f34 in_seq=0 out_seq=0 2024-05-27T09:44:51.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.411+0000 7f5930c00700 10 client.27041 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:51.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.411+0000 7f5930c00700 1 -- 172.21.15.44:0/1954082830 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5613fe4c0e10 con 0x5613fe5aaba0 2024-05-27T09:44:51.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.411+0000 7f5930c00700 10 client.27041 handle_command_reply: tid=0 2024-05-27T09:44:51.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.411+0000 7f595c5c1b80 1 client.27041 shutdown 2024-05-27T09:44:51.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1954082830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 msgr2=0x5613fe5987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:51.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5987f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5934005010 tx=0x7f5934005360 comp rx=0 tx=0).stop 2024-05-27T09:44:51.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1954082830 shutdown_connections 2024-05-27T09:44:51.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5613fe5aaba0 0x5613fe5acfc0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe59b340 0x5613fe598d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe411ec0 0x5613fe5982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/1954082830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5987f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f5930c00700 0 client.27041 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:51.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1954082830 >> 172.21.15.44:0/1954082830 conn(0x5613fe4ab410 msgr2=0x5613fe58c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:51.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1954082830 shutdown_connections 2024-05-27T09:44:51.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1954082830 wait complete. 2024-05-27T09:44:51.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.412+0000 7f595c5c1b80 20 client.27041 trim_cache size 0 max 16384 2024-05-27T09:44:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.438+0000 7f595c5c1b80 1 Processor -- start 2024-05-27T09:44:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.438+0000 7f595c5c1b80 1 -- start start 2024-05-27T09:44:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.438+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe5a70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.438+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5972b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.438+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe59b340 0x5613fe5affe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.438+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f59481b3740 con 0x5613fe59b340 2024-05-27T09:44:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.438+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f59481b0fa0 con 0x5613fe4fda70 2024-05-27T09:44:51.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.438+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59481aee10 con 0x5613fe411ec0 2024-05-27T09:44:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.439+0000 7f5946a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe5a70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.439+0000 7f5946a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe5a70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42058/0 (socket says 172.21.15.44:42058) 2024-05-27T09:44:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.439+0000 7f5946a00700 1 -- 172.21.15.44:0/1149039374 learned_addr learned my addr 172.21.15.44:0/1149039374 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.439+0000 7f5946000700 1 --2- 172.21.15.44:0/1149039374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.439+0000 7f5947400700 1 --2- 172.21.15.44:0/1149039374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe59b340 0x5613fe5affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.439+0000 7f5946a00700 1 -- 172.21.15.44:0/1149039374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 msgr2=0x5613fe5972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.439+0000 7f5946a00700 1 --2- 172.21.15.44:0/1149039374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.439+0000 7f5946a00700 1 -- 172.21.15.44:0/1149039374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe59b340 msgr2=0x5613fe5affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.439+0000 7f5946a00700 1 --2- 172.21.15.44:0/1149039374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe59b340 0x5613fe5affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.439+0000 7f5946a00700 1 -- 172.21.15.44:0/1149039374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613fe5869b0 con 0x5613fe411ec0 2024-05-27T09:44:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.440+0000 7f5946a00700 1 --2- 172.21.15.44:0/1149039374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe5a70f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f59380dff20 tx=0x7f59380ac040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b4078c0be6ebcee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.440+0000 7f5933400700 1 -- 172.21.15.44:0/1149039374 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59380a9460 con 0x5613fe411ec0 2024-05-27T09:44:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.440+0000 7f5933400700 1 -- 172.21.15.44:0/1149039374 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5938009500 con 0x5613fe411ec0 2024-05-27T09:44:51.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.440+0000 7f5933400700 1 -- 172.21.15.44:0/1149039374 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59380adec0 con 0x5613fe411ec0 2024-05-27T09:44:51.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.441+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1149039374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 msgr2=0x5613fe5a70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:51.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.441+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/1149039374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe5a70f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f59380dff20 tx=0x7f59380ac040 comp rx=0 tx=0).stop 2024-05-27T09:44:51.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.441+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1149039374 shutdown_connections 2024-05-27T09:44:51.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.441+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/1149039374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe59b340 0x5613fe5affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.441+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/1149039374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe5a70f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.441+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/1149039374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.441+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1149039374 >> 172.21.15.44:0/1149039374 conn(0x5613fe4ab410 msgr2=0x5613fe596ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:51.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.441+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1149039374 shutdown_connections 2024-05-27T09:44:51.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.441+0000 7f595c5c1b80 1 -- 172.21.15.44:0/1149039374 wait complete. 2024-05-27T09:44:51.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.441+0000 7f595c5c1b80 1 -- start start 2024-05-27T09:44:51.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.442+0000 7f595c5c1b80 1 Processor -- start 2024-05-27T09:44:51.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.442+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe5ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.442+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.442+0000 7f595c5c1b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe59b340 0x5613fe5aea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.442+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f59481af460 con 0x5613fe59b340 2024-05-27T09:44:51.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.442+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f59481b08d0 con 0x5613fe4fda70 2024-05-27T09:44:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.442+0000 7f595c5c1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59481afb30 con 0x5613fe411ec0 2024-05-27T09:44:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.442+0000 7f595c5c1b80 10 client.0 _ll_register_callbacks cb 0x5613fe458cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5946000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5946a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe5ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5946000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:55168/0 (socket says 172.21.15.44:55168) 2024-05-27T09:44:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5946000700 1 -- 172.21.15.44:0/3496152085 learned_addr learned my addr 172.21.15.44:0/3496152085 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5947400700 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe59b340 0x5613fe5aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5946000700 1 -- 172.21.15.44:0/3496152085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 msgr2=0x5613fe5ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5946000700 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe5ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5946000700 1 -- 172.21.15.44:0/3496152085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe59b340 msgr2=0x5613fe5aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5946000700 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe59b340 0x5613fe5aea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5946000700 1 -- 172.21.15.44:0/3496152085 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613fe614600 con 0x5613fe4fda70 2024-05-27T09:44:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5946000700 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5ae540 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5934009520 tx=0x7f5934009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=759f43317f6d81b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.443+0000 7f5930c00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:44:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.444+0000 7f5930c00700 1 -- 172.21.15.44:0/3496152085 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5934027070 con 0x5613fe4fda70 2024-05-27T09:44:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.444+0000 7f5930c00700 1 -- 172.21.15.44:0/3496152085 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f5934033040 con 0x5613fe4fda70 2024-05-27T09:44:51.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.444+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3496152085 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5938009500 con 0x5613fe4fda70 2024-05-27T09:44:51.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.444+0000 7f5930c00700 1 -- 172.21.15.44:0/3496152085 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f593400fb70 con 0x5613fe4fda70 2024-05-27T09:44:51.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.444+0000 7f595c5c1b80 10 client.27047 fetch_fsmap learned FSMap version 33 2024-05-27T09:44:51.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.444+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3496152085 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5613fe6158a0 con 0x5613fe4fda70 2024-05-27T09:44:51.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.445+0000 7f5930c00700 1 -- 172.21.15.44:0/3496152085 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5938009500 con 0x5613fe4fda70 2024-05-27T09:44:51.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.445+0000 7f5930c00700 1 -- 172.21.15.44:0/3496152085 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f5934021040 con 0x5613fe4fda70 2024-05-27T09:44:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.445+0000 7f595c5c1b80 10 client.27047 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:44:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.445+0000 7f595c5c1b80 10 client.27047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.445+0000 7f595c5c1b80 20 client.27047 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.445+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5613fe615e30 0x5613fe618250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.445+0000 7f595c5c1b80 4 client.27047 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:44:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.445+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3496152085 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5613fe4ba5f0 con 0x5613fe615e30 2024-05-27T09:44:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.446+0000 7f5946a00700 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5613fe615e30 0x5613fe618250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.446+0000 7f5946a00700 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5613fe615e30 0x5613fe618250 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe491fce6c6db429 server_cookie=7d0d93d31893be28 in_seq=0 out_seq=0 2024-05-27T09:44:51.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.446+0000 7f5930c00700 10 client.27047 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f5930c00700 1 -- 172.21.15.44:0/3496152085 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5613fe4ba5f0 con 0x5613fe615e30 2024-05-27T09:44:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f5930c00700 10 client.27047 handle_command_reply: tid=0 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f595c5c1b80 1 client.27047 shutdown 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3496152085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 msgr2=0x5613fe5ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5ae540 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5934009520 tx=0x7f5934009cd0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3496152085 shutdown_connections 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5613fe615e30 0x5613fe618250 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5613fe59b340 0x5613fe5aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613fe411ec0 0x5613fe5ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f595c5c1b80 1 --2- 172.21.15.44:0/3496152085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5613fe4fda70 0x5613fe5ae540 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3496152085 >> 172.21.15.44:0/3496152085 conn(0x5613fe4ab410 msgr2=0x5613fe5901a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f5930c00700 0 client.27047 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3496152085 shutdown_connections 2024-05-27T09:44:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.448+0000 7f595c5c1b80 1 -- 172.21.15.44:0/3496152085 wait complete. 2024-05-27T09:44:51.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.449+0000 7f595c5c1b80 20 client.27047 trim_cache size 0 max 16384 2024-05-27T09:44:51.451 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:44:51.451 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:44:51.451 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "ada1416b-784c-4c2b-8fb2-2ae21fafa772", 2024-05-27T09:44:51.451 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:44:51.451 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:44:51.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.449+0000 7f5947e00700 1 -- 172.21.15.44:0/4050491639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f592c0b2a70 msgr2=0x7f592c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:51.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.449+0000 7f5947e00700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f592c0b2a70 0x7f592c0b4f80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x5613fe593290 tx=0x7f5940009380 comp rx=0 tx=0).stop 2024-05-27T09:44:51.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.449+0000 7f5947e00700 1 -- 172.21.15.44:0/4050491639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59481043f0 msgr2=0x7f5948075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:51.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.449+0000 7f5947e00700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59481043f0 0x7f5948075ab0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f593800eb30 tx=0x7f593800ee40 comp rx=0 tx=0).stop 2024-05-27T09:44:51.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.450+0000 7f5947e00700 1 -- 172.21.15.44:0/4050491639 shutdown_connections 2024-05-27T09:44:51.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.450+0000 7f5947e00700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f592c0b2a70 0x7f592c0b4f80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.450+0000 7f5947e00700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5948109670 0x7f5948076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.450+0000 7f5947e00700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59481043f0 0x7f5948075ab0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.450+0000 7f5947e00700 1 --2- 172.21.15.44:0/4050491639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5948108d20 0x7f5948075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:51.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.450+0000 7f5947e00700 1 -- 172.21.15.44:0/4050491639 >> 172.21.15.44:0/4050491639 conn(0x7f5948079d00 msgr2=0x7f5948101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.451+0000 7f5947e00700 1 -- 172.21.15.44:0/4050491639 shutdown_connections 2024-05-27T09:44:51.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:51.451+0000 7f5947e00700 1 -- 172.21.15.44:0/4050491639 wait complete. 2024-05-27T09:44:52.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:51 smithi044 ceph-mon[31618]: pgmap v299: 97 pgs: 97 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 687 op/s 2024-05-27T09:44:52.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:51 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:44:52.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:51 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:44:52.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:51 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:44:52.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:51 smithi050 ceph-mon[36371]: pgmap v299: 97 pgs: 97 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 687 op/s 2024-05-27T09:44:52.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:51 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:44:52.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:51 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:44:52.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:51 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:44:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:51 smithi189 ceph-mon[36671]: pgmap v299: 97 pgs: 97 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 687 op/s 2024-05-27T09:44:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:51 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:44:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:51 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:44:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:51 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:44:52.633 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:44:53.144 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:44:54.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:53 smithi044 ceph-mon[31618]: pgmap v300: 97 pgs: 97 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.1 MiB/s wr, 253 op/s 2024-05-27T09:44:54.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:53 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:44:54.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:53 smithi050 ceph-mon[36371]: pgmap v300: 97 pgs: 97 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.1 MiB/s wr, 253 op/s 2024-05-27T09:44:54.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:53 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:44:54.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:53 smithi189 ceph-mon[36671]: pgmap v300: 97 pgs: 97 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.1 MiB/s wr, 253 op/s 2024-05-27T09:44:54.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:53 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:44:55.205 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:54 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:44:55.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:54 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:44:55.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:54 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:44:55.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.590+0000 7f2cbfe00700 1 -- 172.21.15.44:0/3985308099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cc010e6f0 msgr2=0x7f2cc0110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:55.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.590+0000 7f2cbfe00700 1 --2- 172.21.15.44:0/3985308099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cc010e6f0 0x7f2cc0110bd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800b0a0 tx=0x7f2cb800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.590+0000 7f2cbfe00700 1 -- 172.21.15.44:0/3985308099 shutdown_connections 2024-05-27T09:44:55.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.590+0000 7f2cbfe00700 1 --2- 172.21.15.44:0/3985308099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cc010e6f0 0x7f2cc0110bd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.590+0000 7f2cbfe00700 1 --2- 172.21.15.44:0/3985308099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cc0105280 0x7f2cc0105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.590+0000 7f2cbfe00700 1 --2- 172.21.15.44:0/3985308099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc0105bd0 0x7f2cc010e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.590+0000 7f2cbfe00700 1 -- 172.21.15.44:0/3985308099 >> 172.21.15.44:0/3985308099 conn(0x7f2cc0078ab0 msgr2=0x7f2cc0078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:55.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.590+0000 7f2cbfe00700 1 -- 172.21.15.44:0/3985308099 shutdown_connections 2024-05-27T09:44:55.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.590+0000 7f2cbfe00700 1 -- 172.21.15.44:0/3985308099 wait complete. 2024-05-27T09:44:55.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.592+0000 7f2cbfe00700 1 Processor -- start 2024-05-27T09:44:55.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.592+0000 7f2cbfe00700 1 -- start start 2024-05-27T09:44:55.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.592+0000 7f2cbfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc0105280 0x7f2cc019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.592+0000 7f2cbfe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cc0105bd0 0x7f2cc019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.592+0000 7f2cbfe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cc010e6f0 0x7f2cc01a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.592+0000 7f2cbfe00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc01142a0 con 0x7f2cc0105bd0 2024-05-27T09:44:55.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.592+0000 7f2cbfe00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc0114120 con 0x7f2cc010e6f0 2024-05-27T09:44:55.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.592+0000 7f2cbfe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc0114420 con 0x7f2cc0105280 2024-05-27T09:44:55.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc0105280 0x7f2cc019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc0105280 0x7f2cc019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59478/0 (socket says 172.21.15.44:59478) 2024-05-27T09:44:55.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbea00700 1 -- 172.21.15.44:0/2414000353 learned_addr learned my addr 172.21.15.44:0/2414000353 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:55.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbf400700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cc010e6f0 0x7f2cc01a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbe000700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cc0105bd0 0x7f2cc019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbea00700 1 -- 172.21.15.44:0/2414000353 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cc010e6f0 msgr2=0x7f2cc01a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbea00700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cc010e6f0 0x7f2cc01a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbea00700 1 -- 172.21.15.44:0/2414000353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cc0105bd0 msgr2=0x7f2cc019d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbea00700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cc0105bd0 0x7f2cc019d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbea00700 1 -- 172.21.15.44:0/2414000353 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc01a1ca0 con 0x7f2cc0105280 2024-05-27T09:44:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbe000700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cc0105bd0 0x7f2cc019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.593+0000 7f2cbea00700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc0105280 0x7f2cc019cdd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000e9a0 tx=0x7f2cb000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.594+0000 7f2cb7400700 1 -- 172.21.15.44:0/2414000353 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb000cbd0 con 0x7f2cc0105280 2024-05-27T09:44:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.594+0000 7f2cb7400700 1 -- 172.21.15.44:0/2414000353 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cb000cd70 con 0x7f2cc0105280 2024-05-27T09:44:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.594+0000 7f2cb7400700 1 -- 172.21.15.44:0/2414000353 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0018680 con 0x7f2cc0105280 2024-05-27T09:44:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.594+0000 7f2cbfe00700 1 -- 172.21.15.44:0/2414000353 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc01a1f90 con 0x7f2cc0105280 2024-05-27T09:44:55.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.594+0000 7f2cbfe00700 1 -- 172.21.15.44:0/2414000353 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc01a98b0 con 0x7f2cc0105280 2024-05-27T09:44:55.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.598+0000 7f2cb7400700 1 -- 172.21.15.44:0/2414000353 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2cb0010430 con 0x7f2cc0105280 2024-05-27T09:44:55.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.599+0000 7f2cd41d7b80 1 Processor -- start 2024-05-27T09:44:55.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.599+0000 7f2cd41d7b80 1 -- start start 2024-05-27T09:44:55.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e1166e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e116c20 0x55601e1212e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cb7400700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2ca40b2a70 0x7f2ca40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e117030 0x55601e123830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55601e03ce40 con 0x55601e116c20 2024-05-27T09:44:55.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55601e02c9e0 con 0x55601df8fe40 2024-05-27T09:44:55.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55601e030a20 con 0x55601e117030 2024-05-27T09:44:55.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cb7400700 1 -- 172.21.15.44:0/2414000353 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f2cb00e2890 con 0x7f2cc0105280 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cbf400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e1166e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cbe000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e116c20 0x55601e1212e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cbf400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e1166e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47750/0 (socket says 172.21.15.44:47750) 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cbf400700 1 -- 172.21.15.44:0/3805543875 learned_addr learned my addr 172.21.15.44:0/3805543875 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3805543875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e117030 0x55601e123830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.600+0000 7f2cbea00700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2ca40b2a70 0x7f2ca40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.601+0000 7f2cbf400700 1 -- 172.21.15.44:0/3805543875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e117030 msgr2=0x55601e123830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.601+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3805543875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e117030 0x55601e123830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.601+0000 7f2cbf400700 1 -- 172.21.15.44:0/3805543875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e116c20 msgr2=0x55601e1212e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.601+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3805543875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e116c20 0x55601e1212e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.601+0000 7f2cbf400700 1 -- 172.21.15.44:0/3805543875 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55601dff5420 con 0x55601df8fe40 2024-05-27T09:44:55.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.601+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3805543875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e1166e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800b0a0 tx=0x7f2cb8009860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d346ede65e5a014 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:55.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.602+0000 7f2cabe00700 1 -- 172.21.15.44:0/3805543875 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8003a00 con 0x55601df8fe40 2024-05-27T09:44:55.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.602+0000 7f2cbea00700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2ca40b2a70 0x7f2ca40b4f80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0000c00 tx=0x7f2cb000cf90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:55.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.602+0000 7f2cabe00700 1 -- 172.21.15.44:0/3805543875 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cb8003ba0 con 0x55601df8fe40 2024-05-27T09:44:55.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.602+0000 7f2cabe00700 1 -- 172.21.15.44:0/3805543875 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb801e460 con 0x55601df8fe40 2024-05-27T09:44:55.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.603+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3805543875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 msgr2=0x55601e1166e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:55.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.603+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3805543875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e1166e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800b0a0 tx=0x7f2cb8009860 comp rx=0 tx=0).stop 2024-05-27T09:44:55.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.603+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3805543875 shutdown_connections 2024-05-27T09:44:55.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.603+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3805543875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e117030 0x55601e123830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.603+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3805543875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e1166e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.603+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3805543875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e116c20 0x55601e1212e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.603+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3805543875 >> 172.21.15.44:0/3805543875 conn(0x55601dfeffc0 msgr2=0x55601e109d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:55.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.603+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3805543875 shutdown_connections 2024-05-27T09:44:55.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.603+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3805543875 wait complete. 2024-05-27T09:44:55.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.604+0000 7f2cd41d7b80 1 -- start start 2024-05-27T09:44:55.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.604+0000 7f2cd41d7b80 1 Processor -- start 2024-05-27T09:44:55.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.604+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601df8fe40 0x55601e120e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.604+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e116c20 0x55601e121380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.604+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e117030 0x55601e1218c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.604+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55601df32fa0 con 0x55601e117030 2024-05-27T09:44:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.604+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55601dfcd760 con 0x55601e116c20 2024-05-27T09:44:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.604+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55601e1240e0 con 0x55601df8fe40 2024-05-27T09:44:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.605+0000 7f2cd41d7b80 10 client.0 _ll_register_callbacks cb 0x55601dfd6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.605+0000 7f2cbe000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e116c20 0x55601e121380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.605+0000 7f2cbe000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e116c20 0x55601e121380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47760/0 (socket says 172.21.15.44:47760) 2024-05-27T09:44:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.605+0000 7f2cbe000700 1 -- 172.21.15.44:0/3665808490 learned_addr learned my addr 172.21.15.44:0/3665808490 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.605+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601df8fe40 0x55601e120e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.605+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e117030 0x55601e1218c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.605+0000 7f2cbe000700 1 -- 172.21.15.44:0/3665808490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601df8fe40 msgr2=0x55601e120e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:55.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.605+0000 7f2cbe000700 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601df8fe40 0x55601e120e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.605+0000 7f2cbe000700 1 -- 172.21.15.44:0/3665808490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e117030 msgr2=0x55601e1218c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:55.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.605+0000 7f2cbe000700 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e117030 0x55601e1218c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.605+0000 7f2cbe000700 1 -- 172.21.15.44:0/3665808490 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55601df7c8d0 con 0x55601e116c20 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.606+0000 7f2cbe000700 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e116c20 0x55601e121380 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009d20 tx=0x7f2cac003960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=737861cefadaf186 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.606+0000 7f2ca8c00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.606+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3665808490 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac009e00 con 0x55601e116c20 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.606+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3665808490 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cb8003ba0 con 0x55601e116c20 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.606+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3665808490 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f2cac011650 con 0x55601e116c20 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.606+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3665808490 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2cac011850 con 0x55601e116c20 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.606+0000 7f2cd41d7b80 10 client.27059 fetch_fsmap learned FSMap version 33 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.606+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3665808490 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55601dffd060 con 0x55601e116c20 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.606+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3665808490 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cb8003ba0 con 0x55601e116c20 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.607+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3665808490 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f2cac01bbc0 con 0x55601e116c20 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.607+0000 7f2cd41d7b80 10 client.27059 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:44:55.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.607+0000 7f2cd41d7b80 10 client.27059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:55.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.607+0000 7f2cd41d7b80 20 client.27059 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:55.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.607+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55601e131120 0x55601e133540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.607+0000 7f2cd41d7b80 4 client.27059 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:55.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.607+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3665808490 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55601e03ed90 con 0x55601e131120 2024-05-27T09:44:55.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.607+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55601e131120 0x55601e133540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.608+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55601e131120 0x55601e133540 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7226ac4ec996b2b server_cookie=6c4db98b946bbc29 in_seq=0 out_seq=0 2024-05-27T09:44:55.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.608+0000 7f2ca8c00700 10 client.27059 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:55.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3665808490 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55601e03ed90 con 0x55601e131120 2024-05-27T09:44:55.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2ca8c00700 10 client.27059 handle_command_reply: tid=0 2024-05-27T09:44:55.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2cd41d7b80 1 client.27059 shutdown 2024-05-27T09:44:55.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3665808490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e116c20 msgr2=0x55601e121380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:55.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e116c20 0x55601e121380 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009d20 tx=0x7f2cac003960 comp rx=0 tx=0).stop 2024-05-27T09:44:55.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3665808490 shutdown_connections 2024-05-27T09:44:55.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55601e131120 0x55601e133540 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e117030 0x55601e1218c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601df8fe40 0x55601e120e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3665808490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e116c20 0x55601e121380 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2ca8c00700 0 client.27059 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:55.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3665808490 >> 172.21.15.44:0/3665808490 conn(0x55601dfeffc0 msgr2=0x55601e108ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:55.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3665808490 shutdown_connections 2024-05-27T09:44:55.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.609+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3665808490 wait complete. 2024-05-27T09:44:55.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.610+0000 7f2cd41d7b80 20 client.27059 trim_cache size 0 max 16384 2024-05-27T09:44:55.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.635+0000 7f2cd41d7b80 1 Processor -- start 2024-05-27T09:44:55.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.635+0000 7f2cd41d7b80 1 -- start start 2024-05-27T09:44:55.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.635+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e11b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.635+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e116c20 0x55601e1231a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.635+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e117030 0x55601e12d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.635+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc01af340 con 0x55601e116c20 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.635+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc01acba0 con 0x55601df8fe40 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.635+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc00512b0 con 0x55601e117030 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.636+0000 7f2cbf400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e11b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.636+0000 7f2cbf400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e11b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47774/0 (socket says 172.21.15.44:47774) 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.636+0000 7f2cbf400700 1 -- 172.21.15.44:0/4085589760 learned_addr learned my addr 172.21.15.44:0/4085589760 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.636+0000 7f2cbf400700 1 --2- 172.21.15.44:0/4085589760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e117030 0x55601e12d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.636+0000 7f2cbe000700 1 --2- 172.21.15.44:0/4085589760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e116c20 0x55601e1231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.636+0000 7f2cbf400700 1 -- 172.21.15.44:0/4085589760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e117030 msgr2=0x55601e12d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.636+0000 7f2cbf400700 1 --2- 172.21.15.44:0/4085589760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e117030 0x55601e12d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.636+0000 7f2cbf400700 1 -- 172.21.15.44:0/4085589760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e116c20 msgr2=0x55601e1231a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:55.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.636+0000 7f2cbf400700 1 --2- 172.21.15.44:0/4085589760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e116c20 0x55601e1231a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.636+0000 7f2cbf400700 1 -- 172.21.15.44:0/4085589760 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55601e10b350 con 0x55601df8fe40 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.636+0000 7f2cbe000700 1 --2- 172.21.15.44:0/4085589760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e116c20 0x55601e1231a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.637+0000 7f2cbf400700 1 --2- 172.21.15.44:0/4085589760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e11b090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009fd0 tx=0x7f2cb8000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4629e8c9ba786ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.637+0000 7f2cab400700 1 -- 172.21.15.44:0/4085589760 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb80075c0 con 0x55601df8fe40 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.637+0000 7f2cab400700 1 -- 172.21.15.44:0/4085589760 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cb8007760 con 0x55601df8fe40 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.637+0000 7f2cab400700 1 -- 172.21.15.44:0/4085589760 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb802fc00 con 0x55601df8fe40 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.638+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/4085589760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 msgr2=0x55601e11b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.638+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/4085589760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e11b090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009fd0 tx=0x7f2cb8000f40 comp rx=0 tx=0).stop 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.638+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/4085589760 shutdown_connections 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.638+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/4085589760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e117030 0x55601e12d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.638+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/4085589760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601df8fe40 0x55601e11b090 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.638+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/4085589760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601e116c20 0x55601e1231a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.638+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/4085589760 >> 172.21.15.44:0/4085589760 conn(0x55601dfeffc0 msgr2=0x55601e108ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:55.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.638+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/4085589760 shutdown_connections 2024-05-27T09:44:55.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.638+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/4085589760 wait complete. 2024-05-27T09:44:55.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.639+0000 7f2cd41d7b80 1 -- start start 2024-05-27T09:44:55.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.639+0000 7f2cd41d7b80 1 Processor -- start 2024-05-27T09:44:55.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.639+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601df8fe40 0x55601e1251a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.639+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e116c20 0x55601e1256e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cd41d7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e117030 0x55601e125c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc010b0e0 con 0x55601df8fe40 2024-05-27T09:44:55.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc01ac4d0 con 0x55601e117030 2024-05-27T09:44:55.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cd41d7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc01ab730 con 0x55601e116c20 2024-05-27T09:44:55.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cd41d7b80 10 client.0 _ll_register_callbacks cb 0x55601dfd6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cbe000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e116c20 0x55601e1256e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cbe000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e116c20 0x55601e1256e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59530/0 (socket says 172.21.15.44:59530) 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cbe000700 1 -- 172.21.15.44:0/3591218060 learned_addr learned my addr 172.21.15.44:0/3591218060 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601df8fe40 0x55601e1251a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e117030 0x55601e125c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cbe000700 1 -- 172.21.15.44:0/3591218060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e117030 msgr2=0x55601e125c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cbe000700 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e117030 0x55601e125c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cbe000700 1 -- 172.21.15.44:0/3591218060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601df8fe40 msgr2=0x55601e1251a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cbe000700 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601df8fe40 0x55601e1251a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cbe000700 1 -- 172.21.15.44:0/3591218060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55601e19ac40 con 0x55601e116c20 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.640+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e117030 0x55601e125c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.641+0000 7f2cbe000700 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e116c20 0x55601e1256e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x55601e104310 tx=0x7f2cac003d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=478a5711788c6f8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.641+0000 7f2ca8c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.641+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3591218060 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac02f020 con 0x55601e116c20 2024-05-27T09:44:55.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.641+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3591218060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cb8007760 con 0x55601e116c20 2024-05-27T09:44:55.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.641+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3591218060 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f2cac011650 con 0x55601e116c20 2024-05-27T09:44:55.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.641+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3591218060 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2cac005250 con 0x55601e116c20 2024-05-27T09:44:55.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.641+0000 7f2cd41d7b80 10 client.27052 fetch_fsmap learned FSMap version 33 2024-05-27T09:44:55.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.641+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3591218060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55601e19beb0 con 0x55601e116c20 2024-05-27T09:44:55.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.642+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3591218060 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cb8007760 con 0x55601e116c20 2024-05-27T09:44:55.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.642+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3591218060 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f2cac01a3f0 con 0x55601e116c20 2024-05-27T09:44:55.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.642+0000 7f2cd41d7b80 10 client.27052 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:44:55.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.642+0000 7f2cd41d7b80 10 client.27052 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:55.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.642+0000 7f2cd41d7b80 20 client.27052 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:55.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.642+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55601e19c5f0 0x55601e19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:55.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.642+0000 7f2cd41d7b80 4 client.27052 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.642+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3591218060 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55601e038570 con 0x55601e19c5f0 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.642+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55601e19c5f0 0x55601e19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.643+0000 7f2cbf400700 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55601e19c5f0 0x55601e19ea10 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2418cfe7a19037be server_cookie=bce565e2fd756b91 in_seq=0 out_seq=0 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.643+0000 7f2ca8c00700 10 client.27052 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.643+0000 7f2ca8c00700 1 -- 172.21.15.44:0/3591218060 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55601e038570 con 0x55601e19c5f0 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.643+0000 7f2ca8c00700 10 client.27052 handle_command_reply: tid=0 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.643+0000 7f2cd41d7b80 1 client.27052 shutdown 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.643+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3591218060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e116c20 msgr2=0x55601e1256e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.643+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e116c20 0x55601e1256e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x55601e104310 tx=0x7f2cac003d60 comp rx=0 tx=0).stop 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.644+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3591218060 shutdown_connections 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.644+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55601e19c5f0 0x55601e19ea10 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.644+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55601e117030 0x55601e125c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.644+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55601df8fe40 0x55601e1251a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.644+0000 7f2ca8c00700 0 client.27052 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:55.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.644+0000 7f2cd41d7b80 1 --2- 172.21.15.44:0/3591218060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55601e116c20 0x55601e1256e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.644+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3591218060 >> 172.21.15.44:0/3591218060 conn(0x55601dfeffc0 msgr2=0x55601e11c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:55.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.644+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3591218060 shutdown_connections 2024-05-27T09:44:55.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.644+0000 7f2cd41d7b80 1 -- 172.21.15.44:0/3591218060 wait complete. 2024-05-27T09:44:55.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.644+0000 7f2cd41d7b80 20 client.27052 trim_cache size 0 max 16384 2024-05-27T09:44:55.647 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:44:55.647 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:44:55.647 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:44:55.648 INFO:teuthology.orchestra.run.smithi044.stdout: "ada1416b-784c-4c2b-8fb2-2ae21fafa772": { 2024-05-27T09:44:55.648 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:44:55.648 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "ada1416b-784c-4c2b-8fb2-2ae21fafa772", 2024-05-27T09:44:55.648 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:44:55.648 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:44:55.648 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:44:55.648 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 -- 172.21.15.44:0/2414000353 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2ca40b2a70 msgr2=0x7f2ca40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2ca40b2a70 0x7f2ca40b4f80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0000c00 tx=0x7f2cb000cf90 comp rx=0 tx=0).stop 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 -- 172.21.15.44:0/2414000353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc0105280 msgr2=0x7f2cc019cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc0105280 0x7f2cc019cdd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000e9a0 tx=0x7f2cb000ecb0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 -- 172.21.15.44:0/2414000353 shutdown_connections 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2ca40b2a70 0x7f2ca40b4f80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cc010e6f0 0x7f2cc01a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc0105280 0x7f2cc019cdd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 --2- 172.21.15.44:0/2414000353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cc0105bd0 0x7f2cc019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 -- 172.21.15.44:0/2414000353 >> 172.21.15.44:0/2414000353 conn(0x7f2cc0078ab0 msgr2=0x7f2cc0102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 -- 172.21.15.44:0/2414000353 shutdown_connections 2024-05-27T09:44:55.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:44:55.645+0000 7f2cbfe00700 1 -- 172.21.15.44:0/2414000353 wait complete. 2024-05-27T09:44:56.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:55 smithi044 ceph-mon[31618]: pgmap v301: 97 pgs: 97 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.1 MiB/s wr, 253 op/s 2024-05-27T09:44:56.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:55 smithi050 ceph-mon[36371]: pgmap v301: 97 pgs: 97 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.1 MiB/s wr, 253 op/s 2024-05-27T09:44:56.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:55 smithi189 ceph-mon[36671]: pgmap v301: 97 pgs: 97 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.1 MiB/s wr, 253 op/s 2024-05-27T09:44:56.730 INFO:tasks.cephfs.filesystem:scrub status for tag:ada1416b-784c-4c2b-8fb2-2ae21fafa772 - {'path': '/', 'tag': 'ada1416b-784c-4c2b-8fb2-2ae21fafa772', 'options': 'recursive,force'} 2024-05-27T09:44:58.147 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:57 smithi044 ceph-mon[31618]: pgmap v302: 97 pgs: 97 active+clean; 260 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 1.14k op/s 2024-05-27T09:44:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:57 smithi050 ceph-mon[36371]: pgmap v302: 97 pgs: 97 active+clean; 260 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 1.14k op/s 2024-05-27T09:44:58.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:57 smithi189 ceph-mon[36671]: pgmap v302: 97 pgs: 97 active+clean; 260 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 1.14k op/s 2024-05-27T09:44:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.249+0000 7f0154a00640 1 -- 172.21.15.44:0/4256305268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01501102f0 msgr2=0x7f0150112710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.249+0000 7f0154a00640 1 --2- 172.21.15.44:0/4256305268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01501102f0 0x7f0150112710 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0148009ca0 tx=0x7f014802a950 comp rx=0 tx=0).stop 2024-05-27T09:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.250+0000 7f0154a00640 1 -- 172.21.15.44:0/4256305268 shutdown_connections 2024-05-27T09:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.250+0000 7f0154a00640 1 --2- 172.21.15.44:0/4256305268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01501102f0 0x7f0150112710 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.250+0000 7f0154a00640 1 --2- 172.21.15.44:0/4256305268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f015010d990 0x7f015010fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.250+0000 7f0154a00640 1 --2- 172.21.15.44:0/4256305268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0150075480 0x7f0150075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.250+0000 7f0154a00640 1 -- 172.21.15.44:0/4256305268 >> 172.21.15.44:0/4256305268 conn(0x7f0150079240 msgr2=0x7f0150079680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.250+0000 7f0154a00640 1 -- 172.21.15.44:0/4256305268 shutdown_connections 2024-05-27T09:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.250+0000 7f0154a00640 1 -- 172.21.15.44:0/4256305268 wait complete. 2024-05-27T09:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.251+0000 7f0154a00640 1 Processor -- start 2024-05-27T09:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.252+0000 7f0154a00640 1 -- start start 2024-05-27T09:44:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.252+0000 7f0154a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0150075480 0x7f01501a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.252+0000 7f0154a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f015010d990 0x7f01501a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.252+0000 7f0154a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01501102f0 0x7f015019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.252+0000 7f0154a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0150115e80 con 0x7f0150075480 2024-05-27T09:44:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f0154a00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0150115d00 con 0x7f015010d990 2024-05-27T09:44:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f0154a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0150116000 con 0x7f01501102f0 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f014ea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f015010d990 0x7f01501a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f014fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01501102f0 0x7f015019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f014ea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f015010d990 0x7f01501a2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47788/0 (socket says 172.21.15.44:47788) 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f014ea00640 1 -- 172.21.15.44:0/2835308461 learned_addr learned my addr 172.21.15.44:0/2835308461 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f014f400640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0150075480 0x7f01501a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f014fe00640 1 -- 172.21.15.44:0/2835308461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f015010d990 msgr2=0x7f01501a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f014fe00640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f015010d990 0x7f01501a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f014fe00640 1 -- 172.21.15.44:0/2835308461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0150075480 msgr2=0x7f01501a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f014fe00640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0150075480 0x7f01501a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.253+0000 7f014fe00640 1 -- 172.21.15.44:0/2835308461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015019cdf0 con 0x7f01501102f0 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.254+0000 7f014fe00640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01501102f0 0x7f015019c6c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0148005cc0 tx=0x7f0148031f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.254+0000 7f0147e00640 1 -- 172.21.15.44:0/2835308461 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0148034040 con 0x7f01501102f0 2024-05-27T09:44:59.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.254+0000 7f0147e00640 1 -- 172.21.15.44:0/2835308461 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01480308f0 con 0x7f01501102f0 2024-05-27T09:44:59.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.254+0000 7f0154a00640 1 -- 172.21.15.44:0/2835308461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f015019d020 con 0x7f01501102f0 2024-05-27T09:44:59.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.254+0000 7f0154a00640 1 -- 172.21.15.44:0/2835308461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f015019d510 con 0x7f01501102f0 2024-05-27T09:44:59.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.254+0000 7f0147e00640 1 -- 172.21.15.44:0/2835308461 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01480044d0 con 0x7f01501102f0 2024-05-27T09:44:59.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.258+0000 7f015cff2740 1 Processor -- start 2024-05-27T09:44:59.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.259+0000 7f0147e00640 1 -- 172.21.15.44:0/2835308461 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0148037020 con 0x7f01501102f0 2024-05-27T09:44:59.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.259+0000 7f015cff2740 1 -- start start 2024-05-27T09:44:59.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.260+0000 7f015cff2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648445141b0 0x5648448b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.260+0000 7f015cff2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444eb2f0 0x5648448b6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.260+0000 7f015cff2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448b74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.260+0000 7f015cff2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564844827b90 con 0x5648444f34d0 2024-05-27T09:44:59.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.260+0000 7f015cff2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564844817730 con 0x5648445141b0 2024-05-27T09:44:59.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.260+0000 7f015cff2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56484481b770 con 0x5648444eb2f0 2024-05-27T09:44:59.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.260+0000 7f014ea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648445141b0 0x5648448b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.260+0000 7f014ea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648445141b0 0x5648448b6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47798/0 (socket says 172.21.15.44:47798) 2024-05-27T09:44:59.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.260+0000 7f014ea00640 1 -- 172.21.15.44:0/1979163170 learned_addr learned my addr 172.21.15.44:0/1979163170 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:59.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.260+0000 7f014f400640 1 --2- 172.21.15.44:0/1979163170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444eb2f0 0x5648448b6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.260+0000 7f014ea00640 1 --2- 172.21.15.44:0/1979163170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448b74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.261+0000 7f014ea00640 1 -- 172.21.15.44:0/1979163170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444eb2f0 msgr2=0x5648448b6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.261+0000 7f014ea00640 1 --2- 172.21.15.44:0/1979163170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444eb2f0 0x5648448b6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.261+0000 7f014ea00640 1 -- 172.21.15.44:0/1979163170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 msgr2=0x5648448b74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.261+0000 7f014ea00640 1 --2- 172.21.15.44:0/1979163170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448b74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.261+0000 7f014ea00640 1 -- 172.21.15.44:0/1979163170 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648447b08b0 con 0x5648445141b0 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.261+0000 7f014ea00640 1 --2- 172.21.15.44:0/1979163170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648445141b0 0x5648448b6a60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f013c011440 tx=0x7f013c0099b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccf46f7653b53e7a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.261+0000 7f0147e00640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01340b2950 0x7f01340b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.261+0000 7f013be00640 1 -- 172.21.15.44:0/1979163170 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c004050 con 0x5648445141b0 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.261+0000 7f013be00640 1 -- 172.21.15.44:0/1979163170 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f013c0041f0 con 0x5648445141b0 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.261+0000 7f013be00640 1 -- 172.21.15.44:0/1979163170 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c00e620 con 0x5648445141b0 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.262+0000 7f0147e00640 1 -- 172.21.15.44:0/2835308461 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f0148103340 con 0x7f01501102f0 2024-05-27T09:44:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.262+0000 7f014f400640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01340b2950 0x7f01340b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.263+0000 7f014f400640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01340b2950 0x7f01340b4e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x5648448b7e10 tx=0x7f0140005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.263+0000 7f015cff2740 1 -- 172.21.15.44:0/1979163170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648445141b0 msgr2=0x5648448b6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.263+0000 7f015cff2740 1 --2- 172.21.15.44:0/1979163170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648445141b0 0x5648448b6a60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f013c011440 tx=0x7f013c0099b0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.263+0000 7f015cff2740 1 -- 172.21.15.44:0/1979163170 shutdown_connections 2024-05-27T09:44:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.263+0000 7f015cff2740 1 --2- 172.21.15.44:0/1979163170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448b74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.263+0000 7f015cff2740 1 --2- 172.21.15.44:0/1979163170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444eb2f0 0x5648448b6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.263+0000 7f015cff2740 1 --2- 172.21.15.44:0/1979163170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648445141b0 0x5648448b6a60 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.263+0000 7f015cff2740 1 -- 172.21.15.44:0/1979163170 >> 172.21.15.44:0/1979163170 conn(0x564844508300 msgr2=0x5648448b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.263+0000 7f015cff2740 1 -- 172.21.15.44:0/1979163170 shutdown_connections 2024-05-27T09:44:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.263+0000 7f015cff2740 1 -- 172.21.15.44:0/1979163170 wait complete. 2024-05-27T09:44:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.264+0000 7f015cff2740 1 -- start start 2024-05-27T09:44:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.264+0000 7f015cff2740 1 Processor -- start 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f015cff2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448c0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f015cff2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448c1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f015cff2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 0x5648448ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f015cff2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5648444fd790 con 0x5648444f34d0 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f015cff2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5648447bfbc0 con 0x5648444eb2f0 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f015cff2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5648448c3b90 con 0x5648445141b0 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f015cff2740 10 client.0 _ll_register_callbacks cb 0x56484479d400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f014f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448c0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f014ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f014f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448c0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47804/0 (socket says 172.21.15.44:47804) 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f014f400640 1 -- 172.21.15.44:0/3028625162 learned_addr learned my addr 172.21.15.44:0/3028625162 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f014fe00640 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 0x5648448ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f014f400640 1 -- 172.21.15.44:0/3028625162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 msgr2=0x5648448ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f014f400640 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 0x5648448ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f014f400640 1 -- 172.21.15.44:0/3028625162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 msgr2=0x5648448c1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f014f400640 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448c1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.265+0000 7f014f400640 1 -- 172.21.15.44:0/3028625162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56484452c210 con 0x5648444eb2f0 2024-05-27T09:44:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.266+0000 7f014f400640 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448c0be0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0140013010 tx=0x7f0140004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46a50458dcedda24 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.266+0000 7f0138c00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:44:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.266+0000 7f0138c00640 1 -- 172.21.15.44:0/3028625162 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140003d00 con 0x5648444eb2f0 2024-05-27T09:44:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.266+0000 7f015cff2740 1 -- 172.21.15.44:0/3028625162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f013c0041f0 con 0x5648444eb2f0 2024-05-27T09:44:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.266+0000 7f0138c00640 1 -- 172.21.15.44:0/3028625162 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f014002a420 con 0x5648444eb2f0 2024-05-27T09:44:59.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.267+0000 7f0138c00640 1 -- 172.21.15.44:0/3028625162 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f014002a620 con 0x5648444eb2f0 2024-05-27T09:44:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.267+0000 7f015cff2740 10 client.27074 fetch_fsmap learned FSMap version 33 2024-05-27T09:44:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.267+0000 7f015cff2740 1 -- 172.21.15.44:0/3028625162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5648445f8620 con 0x5648444eb2f0 2024-05-27T09:44:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.267+0000 7f0138c00640 1 -- 172.21.15.44:0/3028625162 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f013c0041f0 con 0x5648444eb2f0 2024-05-27T09:44:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.267+0000 7f0138c00640 1 -- 172.21.15.44:0/3028625162 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f0140033750 con 0x5648444eb2f0 2024-05-27T09:44:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.267+0000 7f015cff2740 10 client.27074 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:44:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.267+0000 7f015cff2740 10 client.27074 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.267+0000 7f015cff2740 20 client.27074 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.267+0000 7f015cff2740 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5648444d2d80 0x5648448d2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.267+0000 7f015cff2740 4 client.27074 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:44:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.267+0000 7f015cff2740 1 -- 172.21.15.44:0/3028625162 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564844829ae0 con 0x5648444d2d80 2024-05-27T09:44:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.268+0000 7f014ea00640 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5648444d2d80 0x5648448d2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.268+0000 7f014ea00640 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5648444d2d80 0x5648448d2f40 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91abb5a0799602dc server_cookie=5c6b04957a497ad in_seq=0 out_seq=0 2024-05-27T09:44:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.268+0000 7f0138c00640 10 client.27074 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.269+0000 7f0138c00640 1 -- 172.21.15.44:0/3028625162 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564844829ae0 con 0x5648444d2d80 2024-05-27T09:44:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.269+0000 7f0138c00640 10 client.27074 handle_command_reply: tid=0 2024-05-27T09:44:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.269+0000 7f015cff2740 1 client.27074 shutdown 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f015cff2740 1 -- 172.21.15.44:0/3028625162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 msgr2=0x5648448c0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f015cff2740 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448c0be0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0140013010 tx=0x7f0140004730 comp rx=0 tx=0).stop 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f015cff2740 1 -- 172.21.15.44:0/3028625162 shutdown_connections 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f015cff2740 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5648444d2d80 0x5648448d2f40 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f015cff2740 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 0x5648448ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f015cff2740 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f015cff2740 1 --2- 172.21.15.44:0/3028625162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448c0be0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f015cff2740 1 -- 172.21.15.44:0/3028625162 >> 172.21.15.44:0/3028625162 conn(0x564844508300 msgr2=0x5648448b6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f0138c00640 0 client.27074 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f015cff2740 1 -- 172.21.15.44:0/3028625162 shutdown_connections 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f015cff2740 1 -- 172.21.15.44:0/3028625162 wait complete. 2024-05-27T09:44:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.270+0000 7f015cff2740 20 client.27074 trim_cache size 0 max 16384 2024-05-27T09:44:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.289+0000 7f015cff2740 1 Processor -- start 2024-05-27T09:44:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.289+0000 7f015cff2740 1 -- start start 2024-05-27T09:44:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f015cff2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f015cff2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444f34d0 0x5648448b7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f015cff2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648445141b0 0x5648448c0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f015cff2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f01501b4400 con 0x5648445141b0 2024-05-27T09:44:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f015cff2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01501b1ca0 con 0x5648444eb2f0 2024-05-27T09:44:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f015cff2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f01501afb10 con 0x5648444f34d0 2024-05-27T09:44:59.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f014fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648445141b0 0x5648448c0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f014ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444f34d0 0x5648448b7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f014f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f014ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444f34d0 0x5648448b7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59586/0 (socket says 172.21.15.44:59586) 2024-05-27T09:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f014ea00640 1 -- 172.21.15.44:0/62998927 learned_addr learned my addr 172.21.15.44:0/62998927 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f014f400640 1 -- 172.21.15.44:0/62998927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444f34d0 msgr2=0x5648448b7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.290+0000 7f014f400640 1 --2- 172.21.15.44:0/62998927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444f34d0 0x5648448b7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.291+0000 7f014f400640 1 -- 172.21.15.44:0/62998927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648445141b0 msgr2=0x5648448c0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.291+0000 7f014f400640 1 --2- 172.21.15.44:0/62998927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648445141b0 0x5648448c0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.291+0000 7f014f400640 1 -- 172.21.15.44:0/62998927 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648448b3c00 con 0x5648444eb2f0 2024-05-27T09:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.291+0000 7f014ea00640 1 --2- 172.21.15.44:0/62998927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444f34d0 0x5648448b7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.291+0000 7f014f400640 1 --2- 172.21.15.44:0/62998927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448b2240 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0140033d60 tx=0x7f01400039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0cb47f080958f25 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.291+0000 7f013b400640 1 -- 172.21.15.44:0/62998927 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014002a420 con 0x5648444eb2f0 2024-05-27T09:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.291+0000 7f013b400640 1 -- 172.21.15.44:0/62998927 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0140025030 con 0x5648444eb2f0 2024-05-27T09:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.291+0000 7f013b400640 1 -- 172.21.15.44:0/62998927 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140021970 con 0x5648444eb2f0 2024-05-27T09:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.292+0000 7f015cff2740 1 -- 172.21.15.44:0/62998927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 msgr2=0x5648448b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.292+0000 7f015cff2740 1 --2- 172.21.15.44:0/62998927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448b2240 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0140033d60 tx=0x7f01400039c0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.293+0000 7f015cff2740 1 -- 172.21.15.44:0/62998927 shutdown_connections 2024-05-27T09:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.293+0000 7f015cff2740 1 --2- 172.21.15.44:0/62998927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648445141b0 0x5648448c0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.293+0000 7f015cff2740 1 --2- 172.21.15.44:0/62998927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648444f34d0 0x5648448b7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.293+0000 7f015cff2740 1 --2- 172.21.15.44:0/62998927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448b2240 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.293+0000 7f015cff2740 1 -- 172.21.15.44:0/62998927 >> 172.21.15.44:0/62998927 conn(0x564844508300 msgr2=0x5648448c0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.293+0000 7f015cff2740 1 -- 172.21.15.44:0/62998927 shutdown_connections 2024-05-27T09:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.293+0000 7f015cff2740 1 -- 172.21.15.44:0/62998927 wait complete. 2024-05-27T09:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.293+0000 7f015cff2740 1 -- start start 2024-05-27T09:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.293+0000 7f015cff2740 1 Processor -- start 2024-05-27T09:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.294+0000 7f015cff2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.294+0000 7f015cff2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448c4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.294+0000 7f015cff2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 0x5648448c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.294+0000 7f015cff2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f01501b0160 con 0x5648444f34d0 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.294+0000 7f015cff2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01501b15d0 con 0x5648444eb2f0 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.294+0000 7f015cff2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f01501b0830 con 0x5648445141b0 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.294+0000 7f015cff2740 10 client.0 _ll_register_callbacks cb 0x7f0140009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.294+0000 7f014fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 0x5648448c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.294+0000 7f014ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f014fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 0x5648448c48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59598/0 (socket says 172.21.15.44:59598) 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f014fe00640 1 -- 172.21.15.44:0/1361257099 learned_addr learned my addr 172.21.15.44:0/1361257099 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f014f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f014fe00640 1 -- 172.21.15.44:0/1361257099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 msgr2=0x5648448c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f014fe00640 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448c3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f014fe00640 1 -- 172.21.15.44:0/1361257099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 msgr2=0x5648448c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:44:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f014fe00640 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f014fe00640 1 -- 172.21.15.44:0/1361257099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564844939730 con 0x5648445141b0 2024-05-27T09:44:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f014ea00640 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:44:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f014fe00640 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 0x5648448c48c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f01480ca660 tx=0x7f0148100f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59823cb784b8d5f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:44:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f0138c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:44:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.295+0000 7f0138c00640 1 -- 172.21.15.44:0/1361257099 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01480e3070 con 0x5648445141b0 2024-05-27T09:44:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f015cff2740 1 -- 172.21.15.44:0/1361257099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0140025030 con 0x5648445141b0 2024-05-27T09:44:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f0138c00640 1 -- 172.21.15.44:0/1361257099 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8521+0+0 (secure 0 0 0) 0x7f01480c6cc0 con 0x5648445141b0 2024-05-27T09:44:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f0138c00640 1 -- 172.21.15.44:0/1361257099 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f01480e9050 con 0x5648445141b0 2024-05-27T09:44:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f015cff2740 10 client.27070 fetch_fsmap learned FSMap version 33 2024-05-27T09:44:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f015cff2740 1 -- 172.21.15.44:0/1361257099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564844943210 con 0x5648445141b0 2024-05-27T09:44:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f0138c00640 1 -- 172.21.15.44:0/1361257099 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0140025030 con 0x5648445141b0 2024-05-27T09:44:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f0138c00640 1 -- 172.21.15.44:0/1361257099 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f014804d030 con 0x5648445141b0 2024-05-27T09:44:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f015cff2740 10 client.27070 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:44:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f015cff2740 10 client.27070 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:44:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f015cff2740 20 client.27070 populate_metadata read hostname 'smithi044' 2024-05-27T09:44:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f015cff2740 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5648444a81c0 0x564844945940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:44:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.296+0000 7f015cff2740 4 client.27070 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:44:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.297+0000 7f015cff2740 1 -- 172.21.15.44:0/1361257099 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5648448232c0 con 0x5648444a81c0 2024-05-27T09:44:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.297+0000 7f014f400640 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5648444a81c0 0x564844945940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:44:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.297+0000 7f0138c00640 10 client.27070 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.298+0000 7f014f400640 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5648444a81c0 0x564844945940 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b06801e0370afa2 server_cookie=fcc49e59ade1a96e in_seq=0 out_seq=0 2024-05-27T09:44:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.771+0000 7f0138c00640 1 -- 172.21.15.44:0/1361257099 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5648448232c0 con 0x5648444a81c0 2024-05-27T09:44:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.771+0000 7f0138c00640 10 client.27070 handle_command_reply: tid=0 2024-05-27T09:44:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.771+0000 7f015cff2740 1 client.27070 shutdown 2024-05-27T09:44:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.771+0000 7f015cff2740 1 -- 172.21.15.44:0/1361257099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 msgr2=0x5648448c48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.771+0000 7f015cff2740 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 0x5648448c48c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f01480ca660 tx=0x7f0148100f30 comp rx=0 tx=0).stop 2024-05-27T09:44:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.772+0000 7f015cff2740 1 -- 172.21.15.44:0/1361257099 shutdown_connections 2024-05-27T09:44:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.772+0000 7f015cff2740 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5648444a81c0 0x564844945940 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.772+0000 7f015cff2740 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648445141b0 0x5648448c48c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.772+0000 7f015cff2740 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5648444f34d0 0x5648448c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.772+0000 7f015cff2740 1 --2- 172.21.15.44:0/1361257099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5648444eb2f0 0x5648448c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.772+0000 7f0138c00640 0 client.27070 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:44:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.772+0000 7f015cff2740 1 -- 172.21.15.44:0/1361257099 >> 172.21.15.44:0/1361257099 conn(0x564844508300 msgr2=0x5648448b1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.772+0000 7f015cff2740 1 -- 172.21.15.44:0/1361257099 shutdown_connections 2024-05-27T09:44:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.772+0000 7f015cff2740 1 -- 172.21.15.44:0/1361257099 wait complete. 2024-05-27T09:44:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.773+0000 7f015cff2740 20 client.27070 trim_cache size 0 max 16384 2024-05-27T09:44:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:44:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:44:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:44:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 -- 172.21.15.44:0/2835308461 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01340b2950 msgr2=0x7f01340b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01340b2950 0x7f01340b4e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x5648448b7e10 tx=0x7f0140005f90 comp rx=0 tx=0).stop 2024-05-27T09:44:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 -- 172.21.15.44:0/2835308461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01501102f0 msgr2=0x7f015019c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:44:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01501102f0 0x7f015019c6c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0148005cc0 tx=0x7f0148031f40 comp rx=0 tx=0).stop 2024-05-27T09:44:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 -- 172.21.15.44:0/2835308461 shutdown_connections 2024-05-27T09:44:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01501102f0 0x7f015019c6c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f015010d990 0x7f01501a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01340b2950 0x7f01340b4e60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 --2- 172.21.15.44:0/2835308461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0150075480 0x7f01501a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:44:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 -- 172.21.15.44:0/2835308461 >> 172.21.15.44:0/2835308461 conn(0x7f0150079240 msgr2=0x7f0150102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:44:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 -- 172.21.15.44:0/2835308461 shutdown_connections 2024-05-27T09:44:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:44:59.774+0000 7f0154a00640 1 -- 172.21.15.44:0/2835308461 wait complete. 2024-05-27T09:45:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:59 smithi044 ceph-mon[31618]: pgmap v303: 97 pgs: 97 active+clean; 260 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 669 KiB/s wr, 1.09k op/s 2024-05-27T09:45:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:59 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:45:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:59 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:59 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:59 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:00.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:44:59 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:00.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:59 smithi050 ceph-mon[36371]: pgmap v303: 97 pgs: 97 active+clean; 260 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 669 KiB/s wr, 1.09k op/s 2024-05-27T09:45:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:59 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:45:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:59 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:59 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:59 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:44:59 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:00.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:59 smithi189 ceph-mon[36671]: pgmap v303: 97 pgs: 97 active+clean; 260 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 669 KiB/s wr, 1.09k op/s 2024-05-27T09:45:00.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:59 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:45:00.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:59 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:00.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:59 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:00.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:59 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:00.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:44:59 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:01.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:00 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:45:01.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:00 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:45:01.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:00 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:01.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:00 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:45:01.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:00 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:45:01.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:00 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:01.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:00 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:45:01.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:00 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:45:01.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:00 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:45:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:01 smithi050 ceph-mon[36371]: pgmap v304: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 785 KiB/s wr, 1.32k op/s 2024-05-27T09:45:02.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:01 smithi189 ceph-mon[36671]: pgmap v304: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 785 KiB/s wr, 1.32k op/s 2024-05-27T09:45:02.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:01 smithi044 ceph-mon[31618]: pgmap v304: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 785 KiB/s wr, 1.32k op/s 2024-05-27T09:45:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:03 smithi050 ceph-mon[36371]: pgmap v305: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 265 KiB/s wr, 1.12k op/s 2024-05-27T09:45:04.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:03 smithi189 ceph-mon[36671]: pgmap v305: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 265 KiB/s wr, 1.12k op/s 2024-05-27T09:45:04.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:03 smithi044 ceph-mon[31618]: pgmap v305: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 265 KiB/s wr, 1.12k op/s 2024-05-27T09:45:06.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:05 smithi050 ceph-mon[36371]: pgmap v306: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 265 KiB/s wr, 1.12k op/s 2024-05-27T09:45:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:05 smithi189 ceph-mon[36671]: pgmap v306: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 265 KiB/s wr, 1.12k op/s 2024-05-27T09:45:06.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:05 smithi044 ceph-mon[31618]: pgmap v306: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 265 KiB/s wr, 1.12k op/s 2024-05-27T09:45:07.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:07 smithi050 ceph-mon[36371]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T09:45:07.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:07 smithi189 ceph-mon[36671]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T09:45:07.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:07 smithi044 ceph-mon[31618]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T09:45:08.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:08 smithi050 ceph-mon[36371]: pgmap v308: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 222 KiB/s wr, 636 op/s 2024-05-27T09:45:08.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:08 smithi189 ceph-mon[36671]: pgmap v308: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 222 KiB/s wr, 636 op/s 2024-05-27T09:45:08.375 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:08 smithi044 ceph-mon[31618]: pgmap v308: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 222 KiB/s wr, 636 op/s 2024-05-27T09:45:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.682+0000 7fb1e3400700 1 -- 172.21.15.44:0/1041721225 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55671b3f2710 con 0x55671b553270 2024-05-27T09:45:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.682+0000 7fb1e3400700 10 client.26984 handle_command_reply: tid=0 2024-05-27T09:45:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.682+0000 7fb212bfeb80 1 client.26984 shutdown 2024-05-27T09:45:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.683+0000 7fb212bfeb80 1 -- 172.21.15.44:0/1041721225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 msgr2=0x55671b4dcd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.683+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4dcd30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4006130 tx=0x7fb1f4004d00 comp rx=0 tx=0).stop 2024-05-27T09:45:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.683+0000 7fb212bfeb80 1 -- 172.21.15.44:0/1041721225 shutdown_connections 2024-05-27T09:45:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.683+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55671b553270 0x55671b555690 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:08.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.683+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55671b4ce2b0 0x55671b4dcd30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.683+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55671b349fe0 0x55671b4dc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.683+0000 7fb212bfeb80 1 --2- 172.21.15.44:0/1041721225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55671b4cdea0 0x55671b4dc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.683+0000 7fb212bfeb80 1 -- 172.21.15.44:0/1041721225 >> 172.21.15.44:0/1041721225 conn(0x55671b3aa160 msgr2=0x55671b4d6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.683+0000 7fb1e3400700 0 client.26984 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.684+0000 7fb212bfeb80 1 -- 172.21.15.44:0/1041721225 shutdown_connections 2024-05-27T09:45:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.684+0000 7fb212bfeb80 1 -- 172.21.15.44:0/1041721225 wait complete. 2024-05-27T09:45:08.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.684+0000 7fb212bfeb80 20 client.26984 trim_cache size 0 max 16384 2024-05-27T09:45:08.687 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 33, 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 19, 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "eed35f50": { 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 19, 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 138.9, 2024-05-27T09:45:08.688 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:45:08.689 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:45:08.689 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:45:08.689 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:45:08.689 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:45:08.689 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:45:08.689 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:08.689 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:08.689 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:08.689 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:08.689 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:08.689 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:45:08.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.686+0000 7fb202a00700 1 -- 172.21.15.44:0/940709310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb1e40b2a70 msgr2=0x7fb1e40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:08.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.686+0000 7fb202a00700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb1e40b2a70 0x7fb1e40b4f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55671b4c4d10 tx=0x7fb1f400d040 comp rx=0 tx=0).stop 2024-05-27T09:45:08.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.686+0000 7fb202a00700 1 -- 172.21.15.44:0/940709310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc105bd0 msgr2=0x7fb1fc1a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:08.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.686+0000 7fb202a00700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc105bd0 0x7fb1fc1a0dc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8005910 tx=0x7fb1e800b670 comp rx=0 tx=0).stop 2024-05-27T09:45:08.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.686+0000 7fb202a00700 1 -- 172.21.15.44:0/940709310 shutdown_connections 2024-05-27T09:45:08.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.686+0000 7fb202a00700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb1e40b2a70 0x7fb1e40b4f80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:08.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.686+0000 7fb202a00700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1fc10e6f0 0x7fb1fc19aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:08.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.686+0000 7fb202a00700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1fc105280 0x7fb1fc1a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:08.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.686+0000 7fb202a00700 1 --2- 172.21.15.44:0/940709310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc105bd0 0x7fb1fc1a0dc0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:08.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.686+0000 7fb202a00700 1 -- 172.21.15.44:0/940709310 >> 172.21.15.44:0/940709310 conn(0x7fb1fc078ab0 msgr2=0x7fb1fc0ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:08.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.687+0000 7fb202a00700 1 -- 172.21.15.44:0/940709310 shutdown_connections 2024-05-27T09:45:08.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:08.688+0000 7fb202a00700 1 -- 172.21.15.44:0/940709310 wait complete. 2024-05-27T09:45:09.537 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'eed35f50', quiesce took 36.5 seconds. Will release after: -17.55 2024-05-27T09:45:09.537 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:45:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:10.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:10 smithi050 ceph-mon[36371]: pgmap v309: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 222 KiB/s wr, 636 op/s 2024-05-27T09:45:10.331 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:10 smithi044 ceph-mon[31618]: pgmap v309: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 222 KiB/s wr, 636 op/s 2024-05-27T09:45:10.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:10 smithi189 ceph-mon[36671]: pgmap v309: 97 pgs: 97 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 222 KiB/s wr, 636 op/s 2024-05-27T09:45:12.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.236+0000 7f96e3c00700 1 -- 172.21.15.44:0/4006285184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96dc105510 msgr2=0x7f96dc105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:12.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.236+0000 7f96e3c00700 1 --2- 172.21.15.44:0/4006285184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96dc105510 0x7f96dc105920 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009990 tx=0x7f96d0009ca0 comp rx=0 tx=0).stop 2024-05-27T09:45:12.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.236+0000 7f96e3c00700 1 -- 172.21.15.44:0/4006285184 shutdown_connections 2024-05-27T09:45:12.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.236+0000 7f96e3c00700 1 --2- 172.21.15.44:0/4006285184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96dc10e8f0 0x7f96dc110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:12.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.236+0000 7f96e3c00700 1 --2- 172.21.15.44:0/4006285184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96dc105510 0x7f96dc105920 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:12.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.236+0000 7f96e3c00700 1 --2- 172.21.15.44:0/4006285184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96dc105e60 0x7f96dc10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:12.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.236+0000 7f96e3c00700 1 -- 172.21.15.44:0/4006285184 >> 172.21.15.44:0/4006285184 conn(0x7f96dc0fedb0 msgr2=0x7f96dc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:12.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.236+0000 7f96e3c00700 1 -- 172.21.15.44:0/4006285184 shutdown_connections 2024-05-27T09:45:12.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.236+0000 7f96e3c00700 1 -- 172.21.15.44:0/4006285184 wait complete. 2024-05-27T09:45:12.239 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:12 smithi044 ceph-mon[31618]: pgmap v310: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 87 KiB/s wr, 925 op/s 2024-05-27T09:45:12.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.237+0000 7f96e3c00700 1 Processor -- start 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.237+0000 7f96e3c00700 1 -- start start 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.238+0000 7f96e3c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96dc105510 0x7f96dc19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.238+0000 7f96e3c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96dc105e60 0x7f96dc19b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.238+0000 7f96e3c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96dc10e8f0 0x7f96dc19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.238+0000 7f96e3c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc114470 con 0x7f96dc10e8f0 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.238+0000 7f96e3c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc1142f0 con 0x7f96dc105e60 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.238+0000 7f96e3c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc1145f0 con 0x7f96dc105510 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.238+0000 7f96e1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96dc105510 0x7f96dc19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.238+0000 7f96e1c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96dc10e8f0 0x7f96dc19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.238+0000 7f96e1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96dc105510 0x7f96dc19ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:57270/0 (socket says 172.21.15.44:57270) 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.238+0000 7f96e1200700 1 -- 172.21.15.44:0/2660918402 learned_addr learned my addr 172.21.15.44:0/2660918402 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:12.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.238+0000 7f96dbe00700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96dc105e60 0x7f96dc19b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:12.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.239+0000 7f96e1200700 1 -- 172.21.15.44:0/2660918402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96dc105e60 msgr2=0x7f96dc19b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:12.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.239+0000 7f96e1200700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96dc105e60 0x7f96dc19b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:12.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.239+0000 7f96e1200700 1 -- 172.21.15.44:0/2660918402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96dc10e8f0 msgr2=0x7f96dc19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:12.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.239+0000 7f96e1200700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96dc10e8f0 0x7f96dc19f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:12.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.239+0000 7f96e1200700 1 -- 172.21.15.44:0/2660918402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc19fcb0 con 0x7f96dc105510 2024-05-27T09:45:12.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.239+0000 7f96e1c00700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96dc10e8f0 0x7f96dc19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:12.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.239+0000 7f96e1200700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96dc105510 0x7f96dc19ade0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f96d0000c00 tx=0x7f96d00057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:12.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.239+0000 7f96d9600700 1 -- 172.21.15.44:0/2660918402 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0027070 con 0x7f96dc105510 2024-05-27T09:45:12.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.240+0000 7f96e3c00700 1 -- 172.21.15.44:0/2660918402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96dc19ff40 con 0x7f96dc105510 2024-05-27T09:45:12.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.240+0000 7f96d9600700 1 -- 172.21.15.44:0/2660918402 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96d0022470 con 0x7f96dc105510 2024-05-27T09:45:12.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.240+0000 7f96d9600700 1 -- 172.21.15.44:0/2660918402 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0017650 con 0x7f96dc105510 2024-05-27T09:45:12.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.240+0000 7f96e3c00700 1 -- 172.21.15.44:0/2660918402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96dc103130 con 0x7f96dc105510 2024-05-27T09:45:12.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.241+0000 7f96e3c00700 1 -- 172.21.15.44:0/2660918402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96dc06be20 con 0x7f96dc105510 2024-05-27T09:45:12.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.243+0000 7f96d9600700 1 -- 172.21.15.44:0/2660918402 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f96d002a070 con 0x7f96dc105510 2024-05-27T09:45:12.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.245+0000 7f96d9600700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f96c40b29f0 0x7f96c40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:12.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.245+0000 7f96d9600700 1 -- 172.21.15.44:0/2660918402 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8565+0+0 (secure 0 0 0) 0x7f96d00e38e0 con 0x7f96dc105510 2024-05-27T09:45:12.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.245+0000 7f96dbe00700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f96c40b29f0 0x7f96c40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:12.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.246+0000 7f96dbe00700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f96c40b29f0 0x7f96c40b4f00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f96dc102240 tx=0x7f96c8009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:12.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.247+0000 7f96d9600700 1 -- 172.21.15.44:0/2660918402 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f96d0013070 con 0x7f96dc105510 2024-05-27T09:45:12.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:12 smithi050 ceph-mon[36371]: pgmap v310: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 87 KiB/s wr, 925 op/s 2024-05-27T09:45:12.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:12 smithi189 ceph-mon[36671]: pgmap v310: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 87 KiB/s wr, 925 op/s 2024-05-27T09:45:12.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.535+0000 7f96e3c00700 1 -- 172.21.15.44:0/2660918402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96dc0512b0 con 0x7f96dc105510 2024-05-27T09:45:12.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.537+0000 7f96d9600700 1 -- 172.21.15.44:0/2660918402 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11866 (secure 0 0 0) 0x7f96d00ab2c0 con 0x7f96dc105510 2024-05-27T09:45:12.539 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:45:12.539 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:44:44.299923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:45:12.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.540+0000 7f96e3c00700 1 -- 172.21.15.44:0/2660918402 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f96c40b29f0 msgr2=0x7f96c40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:12.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.540+0000 7f96e3c00700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f96c40b29f0 0x7f96c40b4f00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f96dc102240 tx=0x7f96c8009260 comp rx=0 tx=0).stop 2024-05-27T09:45:12.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.540+0000 7f96e3c00700 1 -- 172.21.15.44:0/2660918402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96dc105510 msgr2=0x7f96dc19ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:12.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.540+0000 7f96e3c00700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96dc105510 0x7f96dc19ade0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f96d0000c00 tx=0x7f96d00057c0 comp rx=0 tx=0).stop 2024-05-27T09:45:12.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.540+0000 7f96e3c00700 1 -- 172.21.15.44:0/2660918402 shutdown_connections 2024-05-27T09:45:12.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.540+0000 7f96e3c00700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f96c40b29f0 0x7f96c40b4f00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:12.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.540+0000 7f96e3c00700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96dc10e8f0 0x7f96dc19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:12.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.540+0000 7f96e3c00700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96dc105510 0x7f96dc19ade0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:12.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.540+0000 7f96e3c00700 1 --2- 172.21.15.44:0/2660918402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96dc105e60 0x7f96dc19b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:12.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.540+0000 7f96e3c00700 1 -- 172.21.15.44:0/2660918402 >> 172.21.15.44:0/2660918402 conn(0x7f96dc0fedb0 msgr2=0x7f96dc0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:12.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.541+0000 7f96e3c00700 1 -- 172.21.15.44:0/2660918402 shutdown_connections 2024-05-27T09:45:12.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:12.541+0000 7f96e3c00700 1 -- 172.21.15.44:0/2660918402 wait complete. 2024-05-27T09:45:12.544 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 33 2024-05-27T09:45:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.016+0000 7f9a5be00640 1 -- 172.21.15.44:0/1712288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a54110c30 msgr2=0x7f9a54111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.016+0000 7f9a5be00640 1 --2- 172.21.15.44:0/1712288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a54110c30 0x7f9a54111060 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009900 tx=0x7f9a4002a410 comp rx=0 tx=0).stop 2024-05-27T09:45:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.016+0000 7f9a5be00640 1 -- 172.21.15.44:0/1712288 shutdown_connections 2024-05-27T09:45:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.016+0000 7f9a5be00640 1 --2- 172.21.15.44:0/1712288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a541116f0 0x7f9a54118e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.016+0000 7f9a5be00640 1 --2- 172.21.15.44:0/1712288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a54110c30 0x7f9a54111060 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.017+0000 7f9a5be00640 1 --2- 172.21.15.44:0/1712288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a5410c6a0 0x7f9a5410cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.017+0000 7f9a5be00640 1 -- 172.21.15.44:0/1712288 >> 172.21.15.44:0/1712288 conn(0x7f9a5406fa80 msgr2=0x7f9a5406fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.017+0000 7f9a5be00640 1 -- 172.21.15.44:0/1712288 shutdown_connections 2024-05-27T09:45:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.017+0000 7f9a5be00640 1 -- 172.21.15.44:0/1712288 wait complete. 2024-05-27T09:45:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.018+0000 7f9a5be00640 1 Processor -- start 2024-05-27T09:45:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.018+0000 7f9a5be00640 1 -- start start 2024-05-27T09:45:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.018+0000 7f9a5be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5410c6a0 0x7f9a541a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.018+0000 7f9a5be00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a54110c30 0x7f9a541a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a5be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a541116f0 0x7f9a541b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a5be00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5411c640 con 0x7f9a541116f0 2024-05-27T09:45:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a5be00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5411c4c0 con 0x7f9a54110c30 2024-05-27T09:45:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a5be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5411c7c0 con 0x7f9a5410c6a0 2024-05-27T09:45:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a59e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a541116f0 0x7f9a541b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a58a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a54110c30 0x7f9a541a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a58a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a54110c30 0x7f9a541a6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:32936/0 (socket says 172.21.15.44:32936) 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a58a00640 1 -- 172.21.15.44:0/2756462392 learned_addr learned my addr 172.21.15.44:0/2756462392 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a59400640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5410c6a0 0x7f9a541a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a59400640 1 -- 172.21.15.44:0/2756462392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a54110c30 msgr2=0x7f9a541a6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a59400640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a54110c30 0x7f9a541a6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a59400640 1 -- 172.21.15.44:0/2756462392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a541116f0 msgr2=0x7f9a541b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a59400640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a541116f0 0x7f9a541b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a59400640 1 -- 172.21.15.44:0/2756462392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a541b14e0 con 0x7f9a5410c6a0 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a58a00640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a54110c30 0x7f9a541a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.019+0000 7f9a59e00640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a541116f0 0x7f9a541b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.020+0000 7f9a59400640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5410c6a0 0x7f9a541a6100 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800eb00 tx=0x7f9a4800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.020+0000 7f9a52000640 1 -- 172.21.15.44:0/2756462392 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a48010430 con 0x7f9a5410c6a0 2024-05-27T09:45:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.020+0000 7f9a52000640 1 -- 172.21.15.44:0/2756462392 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a48010a80 con 0x7f9a5410c6a0 2024-05-27T09:45:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.020+0000 7f9a52000640 1 -- 172.21.15.44:0/2756462392 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a48017a90 con 0x7f9a5410c6a0 2024-05-27T09:45:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.020+0000 7f9a5be00640 1 -- 172.21.15.44:0/2756462392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a541b1770 con 0x7f9a5410c6a0 2024-05-27T09:45:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.020+0000 7f9a5be00640 1 -- 172.21.15.44:0/2756462392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a541b1c90 con 0x7f9a5410c6a0 2024-05-27T09:45:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.024+0000 7f9a62f82740 1 Processor -- start 2024-05-27T09:45:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.024+0000 7f9a62f82740 1 -- start start 2024-05-27T09:45:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.024+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0511b0 0x564f0d3f3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.024+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0282f0 0x564f0d3f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.024+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0304d0 0x564f0d3f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.024+0000 7f9a62f82740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564f0d364b90 con 0x564f0d0304d0 2024-05-27T09:45:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.024+0000 7f9a62f82740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564f0d354730 con 0x564f0d0511b0 2024-05-27T09:45:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.024+0000 7f9a62f82740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564f0d358770 con 0x564f0d0282f0 2024-05-27T09:45:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.025+0000 7f9a59e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0511b0 0x564f0d3f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.025+0000 7f9a58a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0304d0 0x564f0d3f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.025+0000 7f9a59e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0511b0 0x564f0d3f3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:32942/0 (socket says 172.21.15.44:32942) 2024-05-27T09:45:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.025+0000 7f9a59e00640 1 -- 172.21.15.44:0/205347924 learned_addr learned my addr 172.21.15.44:0/205347924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.025+0000 7f9a59e00640 1 -- 172.21.15.44:0/205347924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0282f0 msgr2=0x564f0d3f3da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:45:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.025+0000 7f9a59e00640 1 --2- 172.21.15.44:0/205347924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0282f0 0x564f0d3f3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.025+0000 7f9a59e00640 1 -- 172.21.15.44:0/205347924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0304d0 msgr2=0x564f0d3f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.025+0000 7f9a59e00640 1 --2- 172.21.15.44:0/205347924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0304d0 0x564f0d3f42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.025+0000 7f9a59e00640 1 -- 172.21.15.44:0/205347924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f0d2ed8b0 con 0x564f0d0511b0 2024-05-27T09:45:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.026+0000 7f9a59e00640 1 --2- 172.21.15.44:0/205347924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0511b0 0x564f0d3f3860 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c011440 tx=0x7f9a4c011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a99c4a400137267 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.026+0000 7f9a52000640 1 -- 172.21.15.44:0/2756462392 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f9a48017c70 con 0x7f9a5410c6a0 2024-05-27T09:45:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.026+0000 7f9a46600640 1 -- 172.21.15.44:0/205347924 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c004020 con 0x564f0d0511b0 2024-05-27T09:45:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.027+0000 7f9a46600640 1 -- 172.21.15.44:0/205347924 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a4c0041c0 con 0x564f0d0511b0 2024-05-27T09:45:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.027+0000 7f9a46600640 1 -- 172.21.15.44:0/205347924 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c00f8b0 con 0x564f0d0511b0 2024-05-27T09:45:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.027+0000 7f9a52000640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9a3c0b2950 0x7f9a3c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.028+0000 7f9a62f82740 1 -- 172.21.15.44:0/205347924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0511b0 msgr2=0x564f0d3f3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.028+0000 7f9a62f82740 1 --2- 172.21.15.44:0/205347924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0511b0 0x564f0d3f3860 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c011440 tx=0x7f9a4c011900 comp rx=0 tx=0).stop 2024-05-27T09:45:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.028+0000 7f9a52000640 1 -- 172.21.15.44:0/2756462392 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f9a480e3090 con 0x7f9a5410c6a0 2024-05-27T09:45:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.028+0000 7f9a59400640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9a3c0b2950 0x7f9a3c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.028+0000 7f9a62f82740 1 -- 172.21.15.44:0/205347924 shutdown_connections 2024-05-27T09:45:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.028+0000 7f9a62f82740 1 --2- 172.21.15.44:0/205347924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0304d0 0x564f0d3f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.028+0000 7f9a62f82740 1 --2- 172.21.15.44:0/205347924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0282f0 0x564f0d3f3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.028+0000 7f9a62f82740 1 --2- 172.21.15.44:0/205347924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0511b0 0x564f0d3f3860 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.028+0000 7f9a62f82740 1 -- 172.21.15.44:0/205347924 >> 172.21.15.44:0/205347924 conn(0x564f0d045300 msgr2=0x564f0d3f0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.028+0000 7f9a62f82740 1 -- 172.21.15.44:0/205347924 shutdown_connections 2024-05-27T09:45:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.028+0000 7f9a62f82740 1 -- 172.21.15.44:0/205347924 wait complete. 2024-05-27T09:45:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.029+0000 7f9a59400640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9a3c0b2950 0x7f9a3c0b4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x564f0d3f1350 tx=0x7f9a4800cf10 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.029+0000 7f9a62f82740 1 -- start start 2024-05-27T09:45:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.029+0000 7f9a62f82740 1 Processor -- start 2024-05-27T09:45:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0282f0 0x564f0d3fda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0304d0 0x564f0d3fdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0511b0 0x564f0d409cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a62f82740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564f0d03a790 con 0x564f0d0511b0 2024-05-27T09:45:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a62f82740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564f0d2fcbc0 con 0x564f0d0282f0 2024-05-27T09:45:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a62f82740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564f0d400990 con 0x564f0d0304d0 2024-05-27T09:45:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a62f82740 10 client.0 _ll_register_callbacks cb 0x564f0d2da400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a59e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0282f0 0x564f0d3fda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a59e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0282f0 0x564f0d3fda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:32956/0 (socket says 172.21.15.44:32956) 2024-05-27T09:45:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a59e00640 1 -- 172.21.15.44:0/1498141905 learned_addr learned my addr 172.21.15.44:0/1498141905 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a59e00640 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0511b0 0x564f0d409cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.030+0000 7f9a58a00640 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0304d0 0x564f0d3fdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.031+0000 7f9a59e00640 1 -- 172.21.15.44:0/1498141905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0304d0 msgr2=0x564f0d3fdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.031+0000 7f9a59e00640 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0304d0 0x564f0d3fdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.031+0000 7f9a59e00640 1 -- 172.21.15.44:0/1498141905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0511b0 msgr2=0x564f0d409cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.031+0000 7f9a59e00640 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0511b0 0x564f0d409cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.031+0000 7f9a59e00640 1 -- 172.21.15.44:0/1498141905 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f0d069210 con 0x564f0d0282f0 2024-05-27T09:45:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.031+0000 7f9a59e00640 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0282f0 0x564f0d3fda30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c011e00 tx=0x7f9a4c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94e801ef44108204 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.031+0000 7f9a3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.032+0000 7f9a3aa00640 1 -- 172.21.15.44:0/1498141905 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c02d060 con 0x564f0d0282f0 2024-05-27T09:45:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.032+0000 7f9a3aa00640 1 -- 172.21.15.44:0/1498141905 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f9a4c005050 con 0x564f0d0282f0 2024-05-27T09:45:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.032+0000 7f9a62f82740 1 -- 172.21.15.44:0/1498141905 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a4c0041c0 con 0x564f0d0282f0 2024-05-27T09:45:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.033+0000 7f9a3aa00640 1 -- 172.21.15.44:0/1498141905 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9a4c01d070 con 0x564f0d0282f0 2024-05-27T09:45:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.033+0000 7f9a62f82740 10 client.27098 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.034+0000 7f9a62f82740 1 -- 172.21.15.44:0/1498141905 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564f0d135620 con 0x564f0d0282f0 2024-05-27T09:45:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.034+0000 7f9a3aa00640 1 -- 172.21.15.44:0/1498141905 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a4c0041c0 con 0x564f0d0282f0 2024-05-27T09:45:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.034+0000 7f9a3aa00640 1 -- 172.21.15.44:0/1498141905 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f9a4c00f5f0 con 0x564f0d0282f0 2024-05-27T09:45:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.034+0000 7f9a62f82740 10 client.27098 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.034+0000 7f9a62f82740 10 client.27098 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.034+0000 7f9a62f82740 20 client.27098 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.034+0000 7f9a62f82740 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564f0d00fd80 0x564f0d40fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.034+0000 7f9a62f82740 4 client.27098 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.034+0000 7f9a62f82740 1 -- 172.21.15.44:0/1498141905 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f0d366ae0 con 0x564f0d00fd80 2024-05-27T09:45:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.035+0000 7f9a58a00640 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564f0d00fd80 0x564f0d40fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.035+0000 7f9a58a00640 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564f0d00fd80 0x564f0d40fba0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fce2760b7be1d75b server_cookie=d87e154d195bfdd0 in_seq=0 out_seq=0 2024-05-27T09:45:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.035+0000 7f9a3aa00640 10 client.27098 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.036+0000 7f9a3aa00640 1 -- 172.21.15.44:0/1498141905 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564f0d366ae0 con 0x564f0d00fd80 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.036+0000 7f9a3aa00640 10 client.27098 handle_command_reply: tid=0 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.036+0000 7f9a62f82740 1 client.27098 shutdown 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a62f82740 1 -- 172.21.15.44:0/1498141905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0282f0 msgr2=0x564f0d3fda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a62f82740 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0282f0 0x564f0d3fda30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c011e00 tx=0x7f9a4c009310 comp rx=0 tx=0).stop 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a62f82740 1 -- 172.21.15.44:0/1498141905 shutdown_connections 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a62f82740 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564f0d00fd80 0x564f0d40fba0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a62f82740 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0511b0 0x564f0d409cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a62f82740 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0304d0 0x564f0d3fdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a3aa00640 0 client.27098 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a62f82740 1 --2- 172.21.15.44:0/1498141905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0282f0 0x564f0d3fda30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a62f82740 1 -- 172.21.15.44:0/1498141905 >> 172.21.15.44:0/1498141905 conn(0x564f0d045300 msgr2=0x564f0d3f3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a62f82740 1 -- 172.21.15.44:0/1498141905 shutdown_connections 2024-05-27T09:45:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a62f82740 1 -- 172.21.15.44:0/1498141905 wait complete. 2024-05-27T09:45:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.037+0000 7f9a62f82740 20 client.27098 trim_cache size 0 max 16384 2024-05-27T09:45:13.057 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.056+0000 7f9a62f82740 1 Processor -- start 2024-05-27T09:45:13.058 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.056+0000 7f9a62f82740 1 -- start start 2024-05-27T09:45:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.057+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0282f0 0x564f0d400160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.057+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 0x564f0d4151a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.057+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0511b0 0x564f0d4156e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.057+0000 7f9a62f82740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9a541bce20 con 0x564f0d0511b0 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.057+0000 7f9a62f82740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9a541ba6c0 con 0x564f0d0304d0 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.057+0000 7f9a62f82740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9a541b8530 con 0x564f0d0282f0 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.057+0000 7f9a59e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0282f0 0x564f0d400160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.057+0000 7f9a58a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 0x564f0d4151a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.057+0000 7f9a59e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0282f0 0x564f0d400160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53104/0 (socket says 172.21.15.44:53104) 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.057+0000 7f9a59e00640 1 -- 172.21.15.44:0/617989804 learned_addr learned my addr 172.21.15.44:0/617989804 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.057+0000 7f9a59e00640 1 --2- 172.21.15.44:0/617989804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0511b0 0x564f0d4156e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.058+0000 7f9a58a00640 1 -- 172.21.15.44:0/617989804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0282f0 msgr2=0x564f0d400160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.058+0000 7f9a58a00640 1 --2- 172.21.15.44:0/617989804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0282f0 0x564f0d400160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.058+0000 7f9a58a00640 1 -- 172.21.15.44:0/617989804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0511b0 msgr2=0x564f0d4156e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.058+0000 7f9a58a00640 1 --2- 172.21.15.44:0/617989804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0511b0 0x564f0d4156e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.058+0000 7f9a58a00640 1 -- 172.21.15.44:0/617989804 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f0d3ea050 con 0x564f0d0304d0 2024-05-27T09:45:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.058+0000 7f9a59e00640 1 --2- 172.21.15.44:0/617989804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0282f0 0x564f0d400160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.058+0000 7f9a58a00640 1 --2- 172.21.15.44:0/617989804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 0x564f0d4151a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9a40004770 tx=0x7f9a40003980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c13bcbf074a81a12 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.058+0000 7f9a45c00640 1 -- 172.21.15.44:0/617989804 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40035040 con 0x564f0d0304d0 2024-05-27T09:45:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.058+0000 7f9a45c00640 1 -- 172.21.15.44:0/617989804 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a400054d0 con 0x564f0d0304d0 2024-05-27T09:45:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.058+0000 7f9a45c00640 1 -- 172.21.15.44:0/617989804 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40004b90 con 0x564f0d0304d0 2024-05-27T09:45:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.059+0000 7f9a62f82740 1 -- 172.21.15.44:0/617989804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 msgr2=0x564f0d4151a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.059+0000 7f9a62f82740 1 --2- 172.21.15.44:0/617989804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 0x564f0d4151a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9a40004770 tx=0x7f9a40003980 comp rx=0 tx=0).stop 2024-05-27T09:45:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.059+0000 7f9a62f82740 1 -- 172.21.15.44:0/617989804 shutdown_connections 2024-05-27T09:45:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.059+0000 7f9a62f82740 1 --2- 172.21.15.44:0/617989804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0511b0 0x564f0d4156e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.059+0000 7f9a62f82740 1 --2- 172.21.15.44:0/617989804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 0x564f0d4151a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.059+0000 7f9a62f82740 1 --2- 172.21.15.44:0/617989804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0282f0 0x564f0d400160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.060+0000 7f9a62f82740 1 -- 172.21.15.44:0/617989804 >> 172.21.15.44:0/617989804 conn(0x564f0d045300 msgr2=0x564f0d3f8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.060+0000 7f9a62f82740 1 -- 172.21.15.44:0/617989804 shutdown_connections 2024-05-27T09:45:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.060+0000 7f9a62f82740 1 -- 172.21.15.44:0/617989804 wait complete. 2024-05-27T09:45:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.060+0000 7f9a62f82740 1 -- start start 2024-05-27T09:45:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.060+0000 7f9a62f82740 1 Processor -- start 2024-05-27T09:45:13.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0282f0 0x564f0d4031c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 0x564f0d403700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a62f82740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0511b0 0x564f0d470040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a62f82740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9a541b8b80 con 0x564f0d0282f0 2024-05-27T09:45:13.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a62f82740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9a541b9ff0 con 0x564f0d0304d0 2024-05-27T09:45:13.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a62f82740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9a541b9250 con 0x564f0d0511b0 2024-05-27T09:45:13.066 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a62f82740 10 client.0 _ll_register_callbacks cb 0x7f9a4c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a59e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0282f0 0x564f0d4031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a58a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 0x564f0d403700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a59e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0282f0 0x564f0d4031c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33198/0 (socket says 172.21.15.44:33198) 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a59e00640 1 -- 172.21.15.44:0/2596436832 learned_addr learned my addr 172.21.15.44:0/2596436832 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.061+0000 7f9a59e00640 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0511b0 0x564f0d470040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.062+0000 7f9a58a00640 1 -- 172.21.15.44:0/2596436832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0511b0 msgr2=0x564f0d470040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.062+0000 7f9a58a00640 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0511b0 0x564f0d470040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.062+0000 7f9a58a00640 1 -- 172.21.15.44:0/2596436832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0282f0 msgr2=0x564f0d4031c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.062+0000 7f9a58a00640 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0282f0 0x564f0d4031c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.062+0000 7f9a58a00640 1 -- 172.21.15.44:0/2596436832 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f0d470580 con 0x564f0d0304d0 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.062+0000 7f9a58a00640 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 0x564f0d403700 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x564f0d4006c0 tx=0x7f9a40005f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a07367166f1ec021 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.062+0000 7f9a3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.062+0000 7f9a3aa00640 1 -- 172.21.15.44:0/2596436832 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40054070 con 0x564f0d0304d0 2024-05-27T09:45:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.062+0000 7f9a62f82740 1 -- 172.21.15.44:0/2596436832 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a400054d0 con 0x564f0d0304d0 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.062+0000 7f9a3aa00640 1 -- 172.21.15.44:0/2596436832 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f9a40003c70 con 0x564f0d0304d0 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.063+0000 7f9a3aa00640 1 -- 172.21.15.44:0/2596436832 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9a40060050 con 0x564f0d0304d0 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.063+0000 7f9a62f82740 10 client.27104 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.063+0000 7f9a62f82740 1 -- 172.21.15.44:0/2596436832 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564f0d4778c0 con 0x564f0d0304d0 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.063+0000 7f9a3aa00640 1 -- 172.21.15.44:0/2596436832 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a400054d0 con 0x564f0d0304d0 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.063+0000 7f9a3aa00640 1 -- 172.21.15.44:0/2596436832 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f9a40046670 con 0x564f0d0304d0 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.063+0000 7f9a62f82740 10 client.27104 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.063+0000 7f9a62f82740 10 client.27104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.063+0000 7f9a62f82740 20 client.27104 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.063+0000 7f9a62f82740 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564f0cfe51c0 0x564f0d479fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.063+0000 7f9a62f82740 4 client.27104 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.063+0000 7f9a62f82740 1 -- 172.21.15.44:0/2596436832 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564f0d3602c0 con 0x564f0cfe51c0 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.064+0000 7f9a59e00640 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564f0cfe51c0 0x564f0d479fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.064+0000 7f9a59e00640 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564f0cfe51c0 0x564f0d479fa0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7689c19ec1e816cc server_cookie=da666315064fd395 in_seq=0 out_seq=0 2024-05-27T09:45:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.064+0000 7f9a3aa00640 10 client.27104 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.092+0000 7f9a3aa00640 1 -- 172.21.15.44:0/2596436832 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x564f0d3602c0 con 0x564f0cfe51c0 2024-05-27T09:45:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.092+0000 7f9a3aa00640 10 client.27104 handle_command_reply: tid=0 2024-05-27T09:45:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.092+0000 7f9a62f82740 1 client.27104 shutdown 2024-05-27T09:45:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a62f82740 1 -- 172.21.15.44:0/2596436832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 msgr2=0x564f0d403700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a62f82740 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 0x564f0d403700 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x564f0d4006c0 tx=0x7f9a40005f30 comp rx=0 tx=0).stop 2024-05-27T09:45:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a62f82740 1 -- 172.21.15.44:0/2596436832 shutdown_connections 2024-05-27T09:45:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a62f82740 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564f0cfe51c0 0x564f0d479fa0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a62f82740 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564f0d0511b0 0x564f0d470040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a62f82740 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564f0d0304d0 0x564f0d403700 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a62f82740 1 --2- 172.21.15.44:0/2596436832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564f0d0282f0 0x564f0d4031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a3aa00640 0 client.27104 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a62f82740 1 -- 172.21.15.44:0/2596436832 >> 172.21.15.44:0/2596436832 conn(0x564f0d045300 msgr2=0x564f0d408a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a62f82740 1 -- 172.21.15.44:0/2596436832 shutdown_connections 2024-05-27T09:45:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a62f82740 1 -- 172.21.15.44:0/2596436832 wait complete. 2024-05-27T09:45:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.093+0000 7f9a62f82740 20 client.27104 trim_cache size 0 max 16384 2024-05-27T09:45:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:45:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:45:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:45:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 -- 172.21.15.44:0/2756462392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9a3c0b2950 msgr2=0x7f9a3c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9a3c0b2950 0x7f9a3c0b4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x564f0d3f1350 tx=0x7f9a4800cf10 comp rx=0 tx=0).stop 2024-05-27T09:45:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 -- 172.21.15.44:0/2756462392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5410c6a0 msgr2=0x7f9a541a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5410c6a0 0x7f9a541a6100 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800eb00 tx=0x7f9a4800c2d0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 -- 172.21.15.44:0/2756462392 shutdown_connections 2024-05-27T09:45:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9a3c0b2950 0x7f9a3c0b4e60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a541116f0 0x7f9a541b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a54110c30 0x7f9a541a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 --2- 172.21.15.44:0/2756462392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5410c6a0 0x7f9a541a6100 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 -- 172.21.15.44:0/2756462392 >> 172.21.15.44:0/2756462392 conn(0x7f9a5406fa80 msgr2=0x7f9a54109cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 -- 172.21.15.44:0/2756462392 shutdown_connections 2024-05-27T09:45:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:13.094+0000 7f9a5be00640 1 -- 172.21.15.44:0/2756462392 wait complete. 2024-05-27T09:45:13.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:13 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:45:13.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:13 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2660918402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:13.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:13 smithi044 ceph-mon[31618]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T09:45:13.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:13 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:45:13.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:13 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2660918402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:13.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:13 smithi050 ceph-mon[36371]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T09:45:13.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:13 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:45:13.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:13 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2660918402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:13.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:13 smithi189 ceph-mon[36671]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T09:45:13.367 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db --set-id eed35f50 --release --await' 2024-05-27T09:45:13.367 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db --set-id eed35f50 --release --await 2024-05-27T09:45:13.884 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:14.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:14 smithi050 ceph-mon[36371]: pgmap v311: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 87 KiB/s wr, 925 op/s 2024-05-27T09:45:14.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:14 smithi189 ceph-mon[36671]: pgmap v311: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 87 KiB/s wr, 925 op/s 2024-05-27T09:45:14.355 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:14 smithi044 ceph-mon[31618]: pgmap v311: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 87 KiB/s wr, 925 op/s 2024-05-27T09:45:16.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.084+0000 7f9356400700 1 -- 172.21.15.44:0/1346971192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9350105df0 msgr2=0x7f935010e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:16.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.084+0000 7f9356400700 1 --2- 172.21.15.44:0/1346971192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9350105df0 0x7f935010e340 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f93380098f0 tx=0x7f9338009c00 comp rx=0 tx=0).stop 2024-05-27T09:45:16.087 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:16 smithi044 ceph-mon[31618]: pgmap v313: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 4.6 KiB/s wr, 650 op/s 2024-05-27T09:45:16.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.084+0000 7f9356400700 1 -- 172.21.15.44:0/1346971192 shutdown_connections 2024-05-27T09:45:16.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.084+0000 7f9356400700 1 --2- 172.21.15.44:0/1346971192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f935010e910 0x7f9350110dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.084+0000 7f9356400700 1 --2- 172.21.15.44:0/1346971192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93501054a0 0x7f93501058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.084+0000 7f9356400700 1 --2- 172.21.15.44:0/1346971192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9350105df0 0x7f935010e340 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.084+0000 7f9356400700 1 -- 172.21.15.44:0/1346971192 >> 172.21.15.44:0/1346971192 conn(0x7f93500fef10 msgr2=0x7f9350101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:16.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.084+0000 7f9356400700 1 -- 172.21.15.44:0/1346971192 shutdown_connections 2024-05-27T09:45:16.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.084+0000 7f9356400700 1 -- 172.21.15.44:0/1346971192 wait complete. 2024-05-27T09:45:16.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.086+0000 7f9356400700 1 Processor -- start 2024-05-27T09:45:16.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.086+0000 7f9356400700 1 -- start start 2024-05-27T09:45:16.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.086+0000 7f9356400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93501054a0 0x7f935019d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.086+0000 7f9356400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9350105df0 0x7f935019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.086+0000 7f9356400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f935010e910 0x7f93501a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.086+0000 7f9356400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9350114450 con 0x7f9350105df0 2024-05-27T09:45:16.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.086+0000 7f9356400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f93501142d0 con 0x7f93501054a0 2024-05-27T09:45:16.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.086+0000 7f9356400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93501145d0 con 0x7f935010e910 2024-05-27T09:45:16.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.087+0000 7f9346a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9350105df0 0x7f935019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.087+0000 7f9346a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9350105df0 0x7f935019d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33214/0 (socket says 172.21.15.44:33214) 2024-05-27T09:45:16.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.087+0000 7f9346a00700 1 -- 172.21.15.44:0/1542194051 learned_addr learned my addr 172.21.15.44:0/1542194051 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:16.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.087+0000 7f934fe00700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f935010e910 0x7f93501a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.087+0000 7f934f400700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93501054a0 0x7f935019d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.087+0000 7f9346a00700 1 -- 172.21.15.44:0/1542194051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f935010e910 msgr2=0x7f93501a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.087+0000 7f9346a00700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f935010e910 0x7f93501a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.087+0000 7f9346a00700 1 -- 172.21.15.44:0/1542194051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93501054a0 msgr2=0x7f935019d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.087+0000 7f9346a00700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93501054a0 0x7f935019d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.087+0000 7f9346a00700 1 -- 172.21.15.44:0/1542194051 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93501a1f10 con 0x7f9350105df0 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.087+0000 7f934fe00700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f935010e910 0x7f93501a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.088+0000 7f9346a00700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9350105df0 0x7f935019d5e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9338009230 tx=0x7f933800f990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.088+0000 7f934cc00700 1 -- 172.21.15.44:0/1542194051 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93380050c0 con 0x7f9350105df0 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.088+0000 7f934cc00700 1 -- 172.21.15.44:0/1542194051 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f933800b560 con 0x7f9350105df0 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.088+0000 7f9356400700 1 -- 172.21.15.44:0/1542194051 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93501a21a0 con 0x7f9350105df0 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.088+0000 7f934cc00700 1 -- 172.21.15.44:0/1542194051 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338003a40 con 0x7f9350105df0 2024-05-27T09:45:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.088+0000 7f9356400700 1 -- 172.21.15.44:0/1542194051 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93501a2680 con 0x7f9350105df0 2024-05-27T09:45:16.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.093+0000 7f934cc00700 1 -- 172.21.15.44:0/1542194051 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f933800ba10 con 0x7f9350105df0 2024-05-27T09:45:16.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.094+0000 7f934cc00700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93340b29f0 0x7f93340b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.094+0000 7f9364f33b80 1 Processor -- start 2024-05-27T09:45:16.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.094+0000 7f9364f33b80 1 -- start start 2024-05-27T09:45:16.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.094+0000 7f934cc00700 1 -- 172.21.15.44:0/1542194051 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f93380e3510 con 0x7f9350105df0 2024-05-27T09:45:16.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.095+0000 7f934f400700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93340b29f0 0x7f93340b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.095+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558399f19f40 0x55839a0a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.095+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.095+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55839a0a1c50 0x55839a0a2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.095+0000 7f9364f33b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558399fc6f40 con 0x55839a0a7b00 2024-05-27T09:45:16.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.095+0000 7f9364f33b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558399fb6ae0 con 0x558399f19f40 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.095+0000 7f9364f33b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558399fbab20 con 0x55839a0a1c50 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.095+0000 7f9346a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.095+0000 7f9346a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33224/0 (socket says 172.21.15.44:33224) 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.095+0000 7f9346a00700 1 -- 172.21.15.44:0/2673696446 learned_addr learned my addr 172.21.15.44:0/2673696446 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.096+0000 7f934f400700 1 --2- 172.21.15.44:0/2673696446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558399f19f40 0x55839a0a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.096+0000 7f934f400700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93340b29f0 0x7f93340b4f00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9340009cb0 tx=0x7f9340009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.096+0000 7f934fe00700 1 --2- 172.21.15.44:0/2673696446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55839a0a1c50 0x55839a0a2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.096+0000 7f9346a00700 1 -- 172.21.15.44:0/2673696446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55839a0a1c50 msgr2=0x55839a0a2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.096+0000 7f9346a00700 1 --2- 172.21.15.44:0/2673696446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55839a0a1c50 0x55839a0a2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.096+0000 7f9346a00700 1 -- 172.21.15.44:0/2673696446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558399f19f40 msgr2=0x55839a0a75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.096+0000 7f9346a00700 1 --2- 172.21.15.44:0/2673696446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558399f19f40 0x55839a0a75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.096+0000 7f9346a00700 1 -- 172.21.15.44:0/2673696446 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558399f7f520 con 0x55839a0a7b00 2024-05-27T09:45:16.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.096+0000 7f934fe00700 1 --2- 172.21.15.44:0/2673696446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55839a0a1c50 0x55839a0a2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:45:16.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.096+0000 7f934f400700 1 --2- 172.21.15.44:0/2673696446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558399f19f40 0x55839a0a75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:16.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.096+0000 7f9346a00700 1 --2- 172.21.15.44:0/2673696446 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0a1710 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f933800bd60 tx=0x7f93380a67e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4351c0be4bd3d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:16.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.097+0000 7f9344c00700 1 -- 172.21.15.44:0/2673696446 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93380cd0f0 con 0x55839a0a7b00 2024-05-27T09:45:16.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.097+0000 7f9344c00700 1 -- 172.21.15.44:0/2673696446 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93380aa330 con 0x55839a0a7b00 2024-05-27T09:45:16.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.097+0000 7f9344c00700 1 -- 172.21.15.44:0/2673696446 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93380a6ab0 con 0x55839a0a7b00 2024-05-27T09:45:16.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.098+0000 7f9364f33b80 1 -- 172.21.15.44:0/2673696446 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 msgr2=0x55839a0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:16.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.098+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2673696446 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0a1710 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f933800bd60 tx=0x7f93380a67e0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.098+0000 7f9364f33b80 1 -- 172.21.15.44:0/2673696446 shutdown_connections 2024-05-27T09:45:16.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.098+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2673696446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55839a0a1c50 0x55839a0a2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.098+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2673696446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558399f19f40 0x55839a0a75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.098+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2673696446 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0a1710 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.098+0000 7f9364f33b80 1 -- 172.21.15.44:0/2673696446 >> 172.21.15.44:0/2673696446 conn(0x558399f7a0c0 msgr2=0x55839a094100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.098+0000 7f9364f33b80 1 -- 172.21.15.44:0/2673696446 shutdown_connections 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.098+0000 7f9364f33b80 1 -- 172.21.15.44:0/2673696446 wait complete. 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.099+0000 7f9364f33b80 1 -- start start 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.099+0000 7f9364f33b80 1 Processor -- start 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.099+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.099+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0ab730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0abc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9364f33b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558399ebd0a0 con 0x55839a0a7b00 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9364f33b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558399f57860 con 0x55839a0a1c50 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9364f33b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55839a0ae390 con 0x558399f19f40 2024-05-27T09:45:16.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9364f33b80 10 client.0 _ll_register_callbacks cb 0x558399f60d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9346a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f934f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f934fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9346a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0ab730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:33014/0 (socket says 172.21.15.44:33014) 2024-05-27T09:45:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9346a00700 1 -- 172.21.15.44:0/2094853199 learned_addr learned my addr 172.21.15.44:0/2094853199 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9346a00700 1 -- 172.21.15.44:0/2094853199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 msgr2=0x55839a0ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9346a00700 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0ab1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9346a00700 1 -- 172.21.15.44:0/2094853199 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 msgr2=0x55839a0abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9346a00700 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0abc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f9346a00700 1 -- 172.21.15.44:0/2094853199 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558399f069d0 con 0x55839a0a1c50 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.100+0000 7f934f400700 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.101+0000 7f9346a00700 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0ab730 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f93380aa520 tx=0x7f93380a6d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49ceda4daf821535 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.101+0000 7f933d600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.101+0000 7f933d600700 1 -- 172.21.15.44:0/2094853199 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93380cd0f0 con 0x55839a0a1c50 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.101+0000 7f933d600700 1 -- 172.21.15.44:0/2094853199 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f93380bfbe0 con 0x55839a0a1c50 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.101+0000 7f9364f33b80 1 -- 172.21.15.44:0/2094853199 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93380aa330 con 0x55839a0a1c50 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.101+0000 7f933d600700 1 -- 172.21.15.44:0/2094853199 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f93380aa820 con 0x55839a0a1c50 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.101+0000 7f9364f33b80 10 client.27116 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.101+0000 7f9364f33b80 1 -- 172.21.15.44:0/2094853199 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558399f87160 con 0x55839a0a1c50 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.101+0000 7f933d600700 1 -- 172.21.15.44:0/2094853199 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93380aa330 con 0x55839a0a1c50 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.102+0000 7f933d600700 1 -- 172.21.15.44:0/2094853199 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f9338003be0 con 0x55839a0a1c50 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.102+0000 7f9364f33b80 10 client.27116 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.102+0000 7f9364f33b80 10 client.27116 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:45:16.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.102+0000 7f9364f33b80 20 client.27116 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:16.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.102+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55839a0baf10 0x55839a0bd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.102+0000 7f9364f33b80 4 client.27116 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:16.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.102+0000 7f9364f33b80 1 -- 172.21.15.44:0/2094853199 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558399fc8e90 con 0x55839a0baf10 2024-05-27T09:45:16.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.102+0000 7f934f400700 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55839a0baf10 0x55839a0bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.103+0000 7f934f400700 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55839a0baf10 0x55839a0bd330 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d73ec39b5650b2a server_cookie=a7fba8a7560eeec5 in_seq=0 out_seq=0 2024-05-27T09:45:16.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.103+0000 7f933d600700 10 client.27116 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:16.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f933d600700 1 -- 172.21.15.44:0/2094853199 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558399fc8e90 con 0x55839a0baf10 2024-05-27T09:45:16.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f933d600700 10 client.27116 handle_command_reply: tid=0 2024-05-27T09:45:16.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f9364f33b80 1 client.27116 shutdown 2024-05-27T09:45:16.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f9364f33b80 1 -- 172.21.15.44:0/2094853199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 msgr2=0x55839a0ab730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:16.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0ab730 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f93380aa520 tx=0x7f93380a6d30 comp rx=0 tx=0).stop 2024-05-27T09:45:16.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f9364f33b80 1 -- 172.21.15.44:0/2094853199 shutdown_connections 2024-05-27T09:45:16.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55839a0baf10 0x55839a0bd330 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0abc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2094853199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0ab730 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f933d600700 0 client.27116 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:16.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f9364f33b80 1 -- 172.21.15.44:0/2094853199 >> 172.21.15.44:0/2094853199 conn(0x558399f7a0c0 msgr2=0x55839a097d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:16.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.104+0000 7f9364f33b80 1 -- 172.21.15.44:0/2094853199 shutdown_connections 2024-05-27T09:45:16.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.105+0000 7f9364f33b80 1 -- 172.21.15.44:0/2094853199 wait complete. 2024-05-27T09:45:16.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.105+0000 7f9364f33b80 20 client.27116 trim_cache size 0 max 16384 2024-05-27T09:45:16.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.127+0000 7f9364f33b80 1 Processor -- start 2024-05-27T09:45:16.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.127+0000 7f9364f33b80 1 -- start start 2024-05-27T09:45:16.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.127+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0ad690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.127+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0b7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.127+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f9364f33b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f93501aeee0 con 0x55839a0a7b00 2024-05-27T09:45:16.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f9364f33b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f93501af5b0 con 0x55839a0a1c50 2024-05-27T09:45:16.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f9364f33b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93501ace10 con 0x558399f19f40 2024-05-27T09:45:16.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f934fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f934f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f934f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0ad690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53172/0 (socket says 172.21.15.44:53172) 2024-05-27T09:45:16.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f934f400700 1 -- 172.21.15.44:0/246692383 learned_addr learned my addr 172.21.15.44:0/246692383 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:16.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f9346a00700 1 --2- 172.21.15.44:0/246692383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f9346a00700 1 -- 172.21.15.44:0/246692383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 msgr2=0x55839a0ad690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:16.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f9346a00700 1 --2- 172.21.15.44:0/246692383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0ad690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f9346a00700 1 -- 172.21.15.44:0/246692383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 msgr2=0x55839a0c29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:16.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.128+0000 7f9346a00700 1 --2- 172.21.15.44:0/246692383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0c29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.129+0000 7f9346a00700 1 -- 172.21.15.44:0/246692383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55839a0abfe0 con 0x55839a0a1c50 2024-05-27T09:45:16.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.129+0000 7f934f400700 1 --2- 172.21.15.44:0/246692383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0ad690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:16.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.129+0000 7f934fe00700 1 --2- 172.21.15.44:0/246692383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:16.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.129+0000 7f9346a00700 1 --2- 172.21.15.44:0/246692383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0b7360 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f93380aa7b0 tx=0x7f9338009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef8874b0112b92b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:16.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.129+0000 7f933fe00700 1 -- 172.21.15.44:0/246692383 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93380e1090 con 0x55839a0a1c50 2024-05-27T09:45:16.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.129+0000 7f933fe00700 1 -- 172.21.15.44:0/246692383 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93380aa330 con 0x55839a0a1c50 2024-05-27T09:45:16.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.129+0000 7f933fe00700 1 -- 172.21.15.44:0/246692383 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93380bfbe0 con 0x55839a0a1c50 2024-05-27T09:45:16.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.130+0000 7f9364f33b80 1 -- 172.21.15.44:0/246692383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 msgr2=0x55839a0b7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:16.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.130+0000 7f9364f33b80 1 --2- 172.21.15.44:0/246692383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0b7360 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f93380aa7b0 tx=0x7f9338009260 comp rx=0 tx=0).stop 2024-05-27T09:45:16.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.130+0000 7f9364f33b80 1 -- 172.21.15.44:0/246692383 shutdown_connections 2024-05-27T09:45:16.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.130+0000 7f9364f33b80 1 --2- 172.21.15.44:0/246692383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.130+0000 7f9364f33b80 1 --2- 172.21.15.44:0/246692383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0ad690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.130+0000 7f9364f33b80 1 --2- 172.21.15.44:0/246692383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0b7360 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.130+0000 7f9364f33b80 1 -- 172.21.15.44:0/246692383 >> 172.21.15.44:0/246692383 conn(0x558399f7a0c0 msgr2=0x55839a0a5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:16.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.130+0000 7f9364f33b80 1 -- 172.21.15.44:0/246692383 shutdown_connections 2024-05-27T09:45:16.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.130+0000 7f9364f33b80 1 -- 172.21.15.44:0/246692383 wait complete. 2024-05-27T09:45:16.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.131+0000 7f9364f33b80 1 -- start start 2024-05-27T09:45:16.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.131+0000 7f9364f33b80 1 Processor -- start 2024-05-27T09:45:16.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f9364f33b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f9364f33b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f935010b480 con 0x55839a0a7b00 2024-05-27T09:45:16.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f9364f33b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f93501ac740 con 0x55839a0a1c50 2024-05-27T09:45:16.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f9364f33b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93501ab9a0 con 0x558399f19f40 2024-05-27T09:45:16.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f9364f33b80 10 client.0 _ll_register_callbacks cb 0x558399f60d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:16.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f934fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f934fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33252/0 (socket says 172.21.15.44:33252) 2024-05-27T09:45:16.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f934fe00700 1 -- 172.21.15.44:0/2733659877 learned_addr learned my addr 172.21.15.44:0/2733659877 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:16.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f9346a00700 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f934f400700 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f934fe00700 1 -- 172.21.15.44:0/2733659877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 msgr2=0x55839a0b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f934fe00700 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.132+0000 7f934fe00700 1 -- 172.21.15.44:0/2733659877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 msgr2=0x55839a0b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.133+0000 7f934fe00700 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0b04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.133+0000 7f934fe00700 1 -- 172.21.15.44:0/2733659877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55839a0b1180 con 0x55839a0a7b00 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.133+0000 7f9346a00700 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.133+0000 7f934fe00700 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0b09e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9348007fc0 tx=0x7f9348009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78010c2b398a911 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.133+0000 7f933d600700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.133+0000 7f933d600700 1 -- 172.21.15.44:0/2733659877 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9348003a00 con 0x55839a0a7b00 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.133+0000 7f9364f33b80 1 -- 172.21.15.44:0/2733659877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93380aa330 con 0x55839a0a7b00 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.133+0000 7f933d600700 1 -- 172.21.15.44:0/2733659877 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f9348003ba0 con 0x55839a0a7b00 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.134+0000 7f933d600700 1 -- 172.21.15.44:0/2733659877 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9348003da0 con 0x55839a0a7b00 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.134+0000 7f9364f33b80 10 client.16914 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:16.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.134+0000 7f9364f33b80 1 -- 172.21.15.44:0/2733659877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55839a1268c0 con 0x55839a0a7b00 2024-05-27T09:45:16.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.134+0000 7f933d600700 1 -- 172.21.15.44:0/2733659877 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93380aa330 con 0x55839a0a7b00 2024-05-27T09:45:16.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.134+0000 7f933d600700 1 -- 172.21.15.44:0/2733659877 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f9348029610 con 0x55839a0a7b00 2024-05-27T09:45:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.134+0000 7f9364f33b80 10 client.16914 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.134+0000 7f9364f33b80 10 client.16914 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:45:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.134+0000 7f9364f33b80 20 client.16914 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.134+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55839a126f60 0x55839a129380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.134+0000 7f9364f33b80 4 client.16914 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "eed35f50", "release": true, "await": true}] 2024-05-27T09:45:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.134+0000 7f9364f33b80 1 -- 172.21.15.44:0/2733659877 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "set_id": "eed35f50", "release": true, "await": true}) -- 0x558399fc2670 con 0x55839a126f60 2024-05-27T09:45:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.135+0000 7f934f400700 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55839a126f60 0x55839a129380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.135+0000 7f934f400700 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55839a126f60 0x55839a129380 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbcd496ed5eedb82 server_cookie=eacfa80f230a0454 in_seq=0 out_seq=0 2024-05-27T09:45:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.135+0000 7f933d600700 10 client.16914 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:16.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:16 smithi050 ceph-mon[36371]: pgmap v313: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 4.6 KiB/s wr, 650 op/s 2024-05-27T09:45:16.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:16 smithi189 ceph-mon[36671]: pgmap v313: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 4.6 KiB/s wr, 650 op/s 2024-05-27T09:45:16.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f933d600700 1 -- 172.21.15.44:0/2733659877 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x558399fc2670 con 0x55839a126f60 2024-05-27T09:45:16.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f933d600700 10 client.16914 handle_command_reply: tid=0 2024-05-27T09:45:16.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f9364f33b80 1 client.16914 shutdown 2024-05-27T09:45:16.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f9364f33b80 1 -- 172.21.15.44:0/2733659877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 msgr2=0x55839a0b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:16.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0b09e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9348007fc0 tx=0x7f9348009490 comp rx=0 tx=0).stop 2024-05-27T09:45:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f9364f33b80 1 -- 172.21.15.44:0/2733659877 shutdown_connections 2024-05-27T09:45:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55839a126f60 0x55839a129380 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55839a0a7b00 0x55839a0b09e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558399f19f40 0x55839a0b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f9364f33b80 1 --2- 172.21.15.44:0/2733659877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839a0a1c50 0x55839a0b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f933d600700 0 client.16914 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.592+0000 7f9364f33b80 1 -- 172.21.15.44:0/2733659877 >> 172.21.15.44:0/2733659877 conn(0x558399f7a0c0 msgr2=0x55839a0a5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.593+0000 7f9364f33b80 1 -- 172.21.15.44:0/2733659877 shutdown_connections 2024-05-27T09:45:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.593+0000 7f9364f33b80 1 -- 172.21.15.44:0/2733659877 wait complete. 2024-05-27T09:45:16.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.593+0000 7f9364f33b80 20 client.16914 trim_cache size 0 max 16384 2024-05-27T09:45:16.596 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:45:16.596 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 33, 2024-05-27T09:45:16.596 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 21, 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "eed35f50": { 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 21, 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 138.9, 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:45:16.597 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:45:16.598 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:45:16.598 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:45:16.598 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:45:16.598 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:16.598 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:16.598 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:16.598 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:16.598 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:16.598 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:45:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.595+0000 7f9356400700 1 -- 172.21.15.44:0/1542194051 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93340b29f0 msgr2=0x7f93340b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.595+0000 7f9356400700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93340b29f0 0x7f93340b4f00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9340009cb0 tx=0x7f9340009260 comp rx=0 tx=0).stop 2024-05-27T09:45:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.595+0000 7f9356400700 1 -- 172.21.15.44:0/1542194051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9350105df0 msgr2=0x7f935019d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.595+0000 7f9356400700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9350105df0 0x7f935019d5e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9338009230 tx=0x7f933800f990 comp rx=0 tx=0).stop 2024-05-27T09:45:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.595+0000 7f9356400700 1 -- 172.21.15.44:0/1542194051 shutdown_connections 2024-05-27T09:45:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.596+0000 7f9356400700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93340b29f0 0x7f93340b4f00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.596+0000 7f9356400700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f935010e910 0x7f93501a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.596+0000 7f9356400700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93501054a0 0x7f935019d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.596+0000 7f9356400700 1 --2- 172.21.15.44:0/1542194051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9350105df0 0x7f935019d5e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:16.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.596+0000 7f9356400700 1 -- 172.21.15.44:0/1542194051 >> 172.21.15.44:0/1542194051 conn(0x7f93500fef10 msgr2=0x7f93500ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:16.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.596+0000 7f9356400700 1 -- 172.21.15.44:0/1542194051 shutdown_connections 2024-05-27T09:45:16.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:16.596+0000 7f9356400700 1 -- 172.21.15.44:0/1542194051 wait complete. 2024-05-27T09:45:17.635 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'eed35f50', total seconds elapsed: 44.6 2024-05-27T09:45:17.635 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.95 2024-05-27T09:45:17.636 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:45:18.116 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:18.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:18 smithi050 ceph-mon[36371]: pgmap v314: 97 pgs: 97 active+clean; 263 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 138 KiB/s wr, 1.17k op/s 2024-05-27T09:45:18.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:18 smithi044 ceph-mon[31618]: pgmap v314: 97 pgs: 97 active+clean; 263 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 138 KiB/s wr, 1.17k op/s 2024-05-27T09:45:18.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:18 smithi189 ceph-mon[36671]: pgmap v314: 97 pgs: 97 active+clean; 263 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 138 KiB/s wr, 1.17k op/s 2024-05-27T09:45:20.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.179+0000 7fd393200700 1 -- 172.21.15.44:0/1133552327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd38c10e6f0 msgr2=0x7fd38c110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:20.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.179+0000 7fd393200700 1 --2- 172.21.15.44:0/1133552327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd38c10e6f0 0x7fd38c110bd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd38800b0a0 tx=0x7fd38800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:45:20.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.179+0000 7fd393200700 1 -- 172.21.15.44:0/1133552327 shutdown_connections 2024-05-27T09:45:20.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.179+0000 7fd393200700 1 --2- 172.21.15.44:0/1133552327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd38c10e6f0 0x7fd38c110bd0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:20.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.179+0000 7fd393200700 1 --2- 172.21.15.44:0/1133552327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c105280 0x7fd38c105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:20.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.179+0000 7fd393200700 1 --2- 172.21.15.44:0/1133552327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38c105bd0 0x7fd38c10e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:20.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.179+0000 7fd393200700 1 -- 172.21.15.44:0/1133552327 >> 172.21.15.44:0/1133552327 conn(0x7fd38c078b40 msgr2=0x7fd38c078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:20.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.179+0000 7fd393200700 1 -- 172.21.15.44:0/1133552327 shutdown_connections 2024-05-27T09:45:20.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.179+0000 7fd393200700 1 -- 172.21.15.44:0/1133552327 wait complete. 2024-05-27T09:45:20.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.180+0000 7fd393200700 1 Processor -- start 2024-05-27T09:45:20.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.180+0000 7fd393200700 1 -- start start 2024-05-27T09:45:20.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.181+0000 7fd393200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd38c105280 0x7fd38c19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:20.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.181+0000 7fd393200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c105bd0 0x7fd38c19d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:20.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.181+0000 7fd393200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38c10e6f0 0x7fd38c1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.181+0000 7fd393200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c1142a0 con 0x7fd38c105bd0 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.181+0000 7fd393200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c114120 con 0x7fd38c105280 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.181+0000 7fd393200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c114420 con 0x7fd38c10e6f0 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.181+0000 7fd392800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38c10e6f0 0x7fd38c1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.181+0000 7fd391e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd38c105280 0x7fd38c19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.181+0000 7fd392800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38c10e6f0 0x7fd38c1a15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53196/0 (socket says 172.21.15.44:53196) 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.181+0000 7fd392800700 1 -- 172.21.15.44:0/1528388683 learned_addr learned my addr 172.21.15.44:0/1528388683 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.181+0000 7fd391400700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c105bd0 0x7fd38c19d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.182+0000 7fd392800700 1 -- 172.21.15.44:0/1528388683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd38c105280 msgr2=0x7fd38c19cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.182+0000 7fd392800700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd38c105280 0x7fd38c19cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.182+0000 7fd392800700 1 -- 172.21.15.44:0/1528388683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c105bd0 msgr2=0x7fd38c19d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:20.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.182+0000 7fd392800700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c105bd0 0x7fd38c19d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:20.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.182+0000 7fd392800700 1 -- 172.21.15.44:0/1528388683 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c1a1c60 con 0x7fd38c10e6f0 2024-05-27T09:45:20.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.182+0000 7fd392800700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38c10e6f0 0x7fd38c1a15c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd38800b070 tx=0x7fd388009c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:20.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.182+0000 7fd386a00700 1 -- 172.21.15.44:0/1528388683 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388021070 con 0x7fd38c10e6f0 2024-05-27T09:45:20.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.182+0000 7fd386a00700 1 -- 172.21.15.44:0/1528388683 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd388007e30 con 0x7fd38c10e6f0 2024-05-27T09:45:20.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.182+0000 7fd386a00700 1 -- 172.21.15.44:0/1528388683 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388004680 con 0x7fd38c10e6f0 2024-05-27T09:45:20.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.183+0000 7fd393200700 1 -- 172.21.15.44:0/1528388683 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38c1a1ef0 con 0x7fd38c10e6f0 2024-05-27T09:45:20.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.183+0000 7fd393200700 1 -- 172.21.15.44:0/1528388683 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd38c1a23e0 con 0x7fd38c10e6f0 2024-05-27T09:45:20.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.184+0000 7fd393200700 1 -- 172.21.15.44:0/1528388683 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd38c0512b0 con 0x7fd38c10e6f0 2024-05-27T09:45:20.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.186+0000 7fd386a00700 1 -- 172.21.15.44:0/1528388683 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd388007980 con 0x7fd38c10e6f0 2024-05-27T09:45:20.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.188+0000 7fd386a00700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3740b2a70 0x7fd3740b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:20.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.188+0000 7fd386a00700 1 -- 172.21.15.44:0/1528388683 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fd38800f020 con 0x7fd38c10e6f0 2024-05-27T09:45:20.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.188+0000 7fd391e00700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3740b2a70 0x7fd3740b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:20.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.189+0000 7fd391e00700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3740b2a70 0x7fd3740b4f80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd380009c60 tx=0x7fd380009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:20.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.192+0000 7fd386a00700 1 -- 172.21.15.44:0/1528388683 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd38802a020 con 0x7fd38c10e6f0 2024-05-27T09:45:20.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:20 smithi044 ceph-mon[31618]: pgmap v315: 97 pgs: 97 active+clean; 263 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 138 KiB/s wr, 1.17k op/s 2024-05-27T09:45:20.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:20 smithi189 ceph-mon[36671]: pgmap v315: 97 pgs: 97 active+clean; 263 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 138 KiB/s wr, 1.17k op/s 2024-05-27T09:45:20.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.506+0000 7fd393200700 1 -- 172.21.15.44:0/1528388683 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd38c19e0f0 con 0x7fd38c10e6f0 2024-05-27T09:45:20.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.507+0000 7fd386a00700 1 -- 172.21.15.44:0/1528388683 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11866 (secure 0 0 0) 0x7fd388004000 con 0x7fd38c10e6f0 2024-05-27T09:45:20.509 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:45:20.509 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:44:44.299923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:45:20.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 -- 172.21.15.44:0/1528388683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3740b2a70 msgr2=0x7fd3740b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:20.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3740b2a70 0x7fd3740b4f80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd380009c60 tx=0x7fd380009260 comp rx=0 tx=0).stop 2024-05-27T09:45:20.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 -- 172.21.15.44:0/1528388683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38c10e6f0 msgr2=0x7fd38c1a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:20.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38c10e6f0 0x7fd38c1a15c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd38800b070 tx=0x7fd388009c90 comp rx=0 tx=0).stop 2024-05-27T09:45:20.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 -- 172.21.15.44:0/1528388683 shutdown_connections 2024-05-27T09:45:20.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd3740b2a70 0x7fd3740b4f80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:20.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38c10e6f0 0x7fd38c1a15c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:20.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd38c105280 0x7fd38c19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:20.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 --2- 172.21.15.44:0/1528388683 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c105bd0 0x7fd38c19d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:20.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 -- 172.21.15.44:0/1528388683 >> 172.21.15.44:0/1528388683 conn(0x7fd38c078b40 msgr2=0x7fd38c102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:20.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 -- 172.21.15.44:0/1528388683 shutdown_connections 2024-05-27T09:45:20.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:20.510+0000 7fd393200700 1 -- 172.21.15.44:0/1528388683 wait complete. 2024-05-27T09:45:20.513 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 33 2024-05-27T09:45:20.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:20 smithi050 ceph-mon[36371]: pgmap v315: 97 pgs: 97 active+clean; 263 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 138 KiB/s wr, 1.17k op/s 2024-05-27T09:45:21.197 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:21 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1528388683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:21.240 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db / --timeout 90 --expiration 138.95 --await-for 45.0' 2024-05-27T09:45:21.240 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db / --timeout 90 --expiration 138.95 --await-for 45.0 2024-05-27T09:45:21.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:21 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1528388683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:21.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:21 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1528388683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:21.688 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:22.109 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:22 smithi044 ceph-mon[31618]: pgmap v316: 97 pgs: 97 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 574 KiB/s wr, 608 op/s 2024-05-27T09:45:22.109 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:22 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:45:22.109 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:22 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:45:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:22 smithi189 ceph-mon[36671]: pgmap v316: 97 pgs: 97 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 574 KiB/s wr, 608 op/s 2024-05-27T09:45:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:22 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:45:22.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:22 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:45:22.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:22 smithi050 ceph-mon[36371]: pgmap v316: 97 pgs: 97 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 574 KiB/s wr, 608 op/s 2024-05-27T09:45:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:22 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:45:22.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:22 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:45:24.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.010+0000 7f7c61800700 1 -- 172.21.15.44:0/2019402279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5c110640 msgr2=0x7f7c5c110a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:24.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.010+0000 7f7c61800700 1 --2- 172.21.15.44:0/2019402279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5c110640 0x7f7c5c110a50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009990 tx=0x7f7c48009ca0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.011+0000 7f7c61800700 1 -- 172.21.15.44:0/2019402279 shutdown_connections 2024-05-27T09:45:24.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.011+0000 7f7c61800700 1 --2- 172.21.15.44:0/2019402279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c5c110f90 0x7f7c5c1188f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.011+0000 7f7c61800700 1 --2- 172.21.15.44:0/2019402279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c5c10c820 0x7f7c5c10cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.011+0000 7f7c61800700 1 --2- 172.21.15.44:0/2019402279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5c110640 0x7f7c5c110a50 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.011+0000 7f7c61800700 1 -- 172.21.15.44:0/2019402279 >> 172.21.15.44:0/2019402279 conn(0x7f7c5c06f980 msgr2=0x7f7c5c06fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:24.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.012+0000 7f7c61800700 1 -- 172.21.15.44:0/2019402279 shutdown_connections 2024-05-27T09:45:24.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.012+0000 7f7c61800700 1 -- 172.21.15.44:0/2019402279 wait complete. 2024-05-27T09:45:24.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.013+0000 7f7c61800700 1 Processor -- start 2024-05-27T09:45:24.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.014+0000 7f7c61800700 1 -- start start 2024-05-27T09:45:24.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.014+0000 7f7c61800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c5c10c820 0x7f7c5c1ab560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.014+0000 7f7c61800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5c110640 0x7f7c5c1abaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.014+0000 7f7c61800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c5c110f90 0x7f7c5c1a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.014+0000 7f7c61800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c11bbe0 con 0x7f7c5c10c820 2024-05-27T09:45:24.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.014+0000 7f7c61800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c11ba60 con 0x7f7c5c110640 2024-05-27T09:45:24.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.014+0000 7f7c61800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c11bd60 con 0x7f7c5c110f90 2024-05-27T09:45:24.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.015+0000 7f7c5aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c5c10c820 0x7f7c5c1ab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.015+0000 7f7c5b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c5c110f90 0x7f7c5c1a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.015+0000 7f7c5aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c5c10c820 0x7f7c5c1ab560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55814/0 (socket says 172.21.15.44:55814) 2024-05-27T09:45:24.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.015+0000 7f7c5aa00700 1 -- 172.21.15.44:0/330645355 learned_addr learned my addr 172.21.15.44:0/330645355 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:24.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.015+0000 7f7c5a000700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5c110640 0x7f7c5c1abaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.016+0000 7f7c5a000700 1 -- 172.21.15.44:0/330645355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c5c110f90 msgr2=0x7f7c5c1a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:24.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.016+0000 7f7c5a000700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c5c110f90 0x7f7c5c1a56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.016+0000 7f7c5a000700 1 -- 172.21.15.44:0/330645355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c5c10c820 msgr2=0x7f7c5c1ab560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:24.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.016+0000 7f7c5a000700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c5c10c820 0x7f7c5c1ab560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.016+0000 7f7c5a000700 1 -- 172.21.15.44:0/330645355 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5c1a5e20 con 0x7f7c5c110640 2024-05-27T09:45:24.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.016+0000 7f7c5a000700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5c110640 0x7f7c5c1abaa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7c48000c00 tx=0x7f7c48005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:24.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.017+0000 7f7c53400700 1 -- 172.21.15.44:0/330645355 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48027070 con 0x7f7c5c110640 2024-05-27T09:45:24.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.017+0000 7f7c53400700 1 -- 172.21.15.44:0/330645355 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c48003f20 con 0x7f7c5c110640 2024-05-27T09:45:24.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.017+0000 7f7c53400700 1 -- 172.21.15.44:0/330645355 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48022bf0 con 0x7f7c5c110640 2024-05-27T09:45:24.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.017+0000 7f7c61800700 1 -- 172.21.15.44:0/330645355 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c5c1a60b0 con 0x7f7c5c110640 2024-05-27T09:45:24.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.017+0000 7f7c61800700 1 -- 172.21.15.44:0/330645355 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c5c1a6540 con 0x7f7c5c110640 2024-05-27T09:45:24.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.022+0000 7f7c53400700 1 -- 172.21.15.44:0/330645355 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f7c4800f460 con 0x7f7c5c110640 2024-05-27T09:45:24.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.025+0000 7f7c53400700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c400b2b40 0x7f7c400b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.025+0000 7f7c70409b80 1 Processor -- start 2024-05-27T09:45:24.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.025+0000 7f7c53400700 1 -- 172.21.15.44:0/330645355 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f7c48022470 con 0x7f7c5c110640 2024-05-27T09:45:24.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.025+0000 7f7c70409b80 1 -- start start 2024-05-27T09:45:24.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.026+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86fd7b90 0x558b870715d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.026+0000 7f7c5aa00700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c400b2b40 0x7f7c400b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.026+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86eebfe0 0x558b87071b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.026+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b8706b680 0x558b8706ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.026+0000 7f7c70409b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558b86f98fe0 con 0x558b86fd7b90 2024-05-27T09:45:24.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.026+0000 7f7c70409b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558b86f88b80 con 0x558b86eebfe0 2024-05-27T09:45:24.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.026+0000 7f7c70409b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558b86f8cbc0 con 0x558b8706b680 2024-05-27T09:45:24.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.027+0000 7f7c5a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86eebfe0 0x558b87071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.027+0000 7f7c5a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86eebfe0 0x558b87071b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54440/0 (socket says 172.21.15.44:54440) 2024-05-27T09:45:24.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.027+0000 7f7c5a000700 1 -- 172.21.15.44:0/721095238 learned_addr learned my addr 172.21.15.44:0/721095238 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:24.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.027+0000 7f7c5b400700 1 --2- 172.21.15.44:0/721095238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b8706b680 0x558b8706ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.027+0000 7f7c5aa00700 1 --2- 172.21.15.44:0/721095238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86fd7b90 0x558b870715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.027+0000 7f7c5a000700 1 -- 172.21.15.44:0/721095238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b8706b680 msgr2=0x558b8706ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:24.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.027+0000 7f7c5a000700 1 --2- 172.21.15.44:0/721095238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b8706b680 0x558b8706ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.027+0000 7f7c5a000700 1 -- 172.21.15.44:0/721095238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86fd7b90 msgr2=0x558b870715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:24.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.028+0000 7f7c5a000700 1 --2- 172.21.15.44:0/721095238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86fd7b90 0x558b870715d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.028+0000 7f7c5a000700 1 -- 172.21.15.44:0/721095238 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b86f515c0 con 0x558b86eebfe0 2024-05-27T09:45:24.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.028+0000 7f7c5aa00700 1 --2- 172.21.15.44:0/721095238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86fd7b90 0x558b870715d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:45:24.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.028+0000 7f7c5a000700 1 --2- 172.21.15.44:0/721095238 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86eebfe0 0x558b87071b30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7c480a7990 tx=0x7f7c48009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=463258290dabc574 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:24.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.029+0000 7f7c5aa00700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c400b2b40 0x7f7c400b5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00af60 tx=0x7f7c4c00a2b0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:24.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.029+0000 7f7c47e00700 1 -- 172.21.15.44:0/721095238 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c480ab910 con 0x558b86eebfe0 2024-05-27T09:45:24.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.029+0000 7f7c47e00700 1 -- 172.21.15.44:0/721095238 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c480b0e50 con 0x558b86eebfe0 2024-05-27T09:45:24.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.029+0000 7f7c47e00700 1 -- 172.21.15.44:0/721095238 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c480abdd0 con 0x558b86eebfe0 2024-05-27T09:45:24.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.030+0000 7f7c70409b80 1 -- 172.21.15.44:0/721095238 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86eebfe0 msgr2=0x558b87071b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:24.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.030+0000 7f7c70409b80 1 --2- 172.21.15.44:0/721095238 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86eebfe0 0x558b87071b30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7c480a7990 tx=0x7f7c48009f90 comp rx=0 tx=0).stop 2024-05-27T09:45:24.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.031+0000 7f7c70409b80 1 -- 172.21.15.44:0/721095238 shutdown_connections 2024-05-27T09:45:24.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.031+0000 7f7c70409b80 1 --2- 172.21.15.44:0/721095238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b8706b680 0x558b8706ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.031+0000 7f7c70409b80 1 --2- 172.21.15.44:0/721095238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86fd7b90 0x558b870715d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.031+0000 7f7c70409b80 1 --2- 172.21.15.44:0/721095238 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86eebfe0 0x558b87071b30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.031+0000 7f7c70409b80 1 -- 172.21.15.44:0/721095238 >> 172.21.15.44:0/721095238 conn(0x558b86f85530 msgr2=0x558b870666f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:24.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.031+0000 7f7c70409b80 1 -- 172.21.15.44:0/721095238 shutdown_connections 2024-05-27T09:45:24.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.031+0000 7f7c70409b80 1 -- 172.21.15.44:0/721095238 wait complete. 2024-05-27T09:45:24.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.032+0000 7f7c70409b80 1 -- start start 2024-05-27T09:45:24.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.032+0000 7f7c70409b80 1 Processor -- start 2024-05-27T09:45:24.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.033+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86eebfe0 0x558b87083600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.033+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86fd7b90 0x558b87083b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.033+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b8706b680 0x558b8707da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.033+0000 7f7c70409b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558b86e8f140 con 0x558b86eebfe0 2024-05-27T09:45:24.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.033+0000 7f7c70409b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558b86f29900 con 0x558b8706b680 2024-05-27T09:45:24.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.033+0000 7f7c70409b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558b8706c230 con 0x558b86fd7b90 2024-05-27T09:45:24.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.033+0000 7f7c70409b80 10 client.0 _ll_register_callbacks cb 0x558b86f32e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:24.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.034+0000 7f7c5b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b8706b680 0x558b8707da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.034+0000 7f7c5b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b8706b680 0x558b8707da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54442/0 (socket says 172.21.15.44:54442) 2024-05-27T09:45:24.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.034+0000 7f7c5b400700 1 -- 172.21.15.44:0/140500272 learned_addr learned my addr 172.21.15.44:0/140500272 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:24.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.034+0000 7f7c5b400700 1 -- 172.21.15.44:0/140500272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86fd7b90 msgr2=0x558b87083b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:24.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.034+0000 7f7c5b400700 1 --2- 172.21.15.44:0/140500272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86fd7b90 0x558b87083b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.034+0000 7f7c5b400700 1 -- 172.21.15.44:0/140500272 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86eebfe0 msgr2=0x558b87083600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:24.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.034+0000 7f7c5b400700 1 --2- 172.21.15.44:0/140500272 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86eebfe0 0x558b87083600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.034+0000 7f7c5b400700 1 -- 172.21.15.44:0/140500272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b86ed8a70 con 0x558b8706b680 2024-05-27T09:45:24.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.035+0000 7f7c5b400700 1 --2- 172.21.15.44:0/140500272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b8706b680 0x558b8707da00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7c54007a70 tx=0x7f7c54007e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf78d99dd04e26fa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:24.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.036+0000 7f7c44c00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:24.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.036+0000 7f7c44c00700 1 -- 172.21.15.44:0/140500272 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5401a070 con 0x558b8706b680 2024-05-27T09:45:24.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.037+0000 7f7c44c00700 1 -- 172.21.15.44:0/140500272 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f7c54012c00 con 0x558b8706b680 2024-05-27T09:45:24.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.037+0000 7f7c70409b80 1 -- 172.21.15.44:0/140500272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c480b0e50 con 0x558b8706b680 2024-05-27T09:45:24.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.038+0000 7f7c44c00700 1 -- 172.21.15.44:0/140500272 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7c54024050 con 0x558b8706b680 2024-05-27T09:45:24.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.038+0000 7f7c70409b80 10 client.27140 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:24.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.038+0000 7f7c70409b80 1 -- 172.21.15.44:0/140500272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558b86f59200 con 0x558b8706b680 2024-05-27T09:45:24.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.038+0000 7f7c44c00700 1 -- 172.21.15.44:0/140500272 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c480b0e50 con 0x558b8706b680 2024-05-27T09:45:24.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.039+0000 7f7c44c00700 1 -- 172.21.15.44:0/140500272 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f7c54016420 con 0x558b8706b680 2024-05-27T09:45:24.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.040+0000 7f7c70409b80 10 client.27140 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:24.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.040+0000 7f7c70409b80 10 client.27140 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:45:24.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.040+0000 7f7c70409b80 20 client.27140 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:24.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.041+0000 7f7c70409b80 1 --2- 172.21.15.44:0/140500272 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558b8708d6f0 0x558b8708fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.041+0000 7f7c70409b80 4 client.27140 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:24.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.041+0000 7f7c70409b80 1 -- 172.21.15.44:0/140500272 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b86f9af30 con 0x558b8708d6f0 2024-05-27T09:45:24.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.042+0000 7f7c5aa00700 1 --2- 172.21.15.44:0/140500272 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558b8708d6f0 0x558b8708fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.043+0000 7f7c5aa00700 1 --2- 172.21.15.44:0/140500272 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558b8708d6f0 0x558b8708fbe0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f36f48cdd0704163 server_cookie=17076f67f628c556 in_seq=0 out_seq=0 2024-05-27T09:45:24.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.043+0000 7f7c44c00700 10 client.27140 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.044+0000 7f7c44c00700 1 -- 172.21.15.44:0/140500272 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558b86f9af30 con 0x558b8708d6f0 2024-05-27T09:45:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.044+0000 7f7c44c00700 10 client.27140 handle_command_reply: tid=0 2024-05-27T09:45:24.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.044+0000 7f7c70409b80 1 client.27140 shutdown 2024-05-27T09:45:24.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.045+0000 7f7c70409b80 1 -- 172.21.15.44:0/140500272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b8706b680 msgr2=0x558b8707da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:24.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.045+0000 7f7c70409b80 1 --2- 172.21.15.44:0/140500272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b8706b680 0x558b8707da00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7c54007a70 tx=0x7f7c54007e30 comp rx=0 tx=0).stop 2024-05-27T09:45:24.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.045+0000 7f7c70409b80 1 -- 172.21.15.44:0/140500272 shutdown_connections 2024-05-27T09:45:24.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.045+0000 7f7c70409b80 1 --2- 172.21.15.44:0/140500272 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558b8708d6f0 0x558b8708fbe0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.045+0000 7f7c70409b80 1 --2- 172.21.15.44:0/140500272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b8706b680 0x558b8707da00 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.045+0000 7f7c70409b80 1 --2- 172.21.15.44:0/140500272 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86eebfe0 0x558b87083600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.045+0000 7f7c70409b80 1 --2- 172.21.15.44:0/140500272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86fd7b90 0x558b87083b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.045+0000 7f7c70409b80 1 -- 172.21.15.44:0/140500272 >> 172.21.15.44:0/140500272 conn(0x558b86f85530 msgr2=0x558b8706a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:24.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.045+0000 7f7c44c00700 0 client.27140 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:24.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.045+0000 7f7c70409b80 1 -- 172.21.15.44:0/140500272 shutdown_connections 2024-05-27T09:45:24.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.046+0000 7f7c70409b80 1 -- 172.21.15.44:0/140500272 wait complete. 2024-05-27T09:45:24.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.046+0000 7f7c70409b80 20 client.27140 trim_cache size 0 max 16384 2024-05-27T09:45:24.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.094+0000 7f7c70409b80 1 Processor -- start 2024-05-27T09:45:24.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.095+0000 7f7c70409b80 1 -- start start 2024-05-27T09:45:24.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.095+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86eebfe0 0x558b870953e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.095+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86fd7b90 0x558b87095920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b8706b680 0x558b87070330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c70409b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c1b7cf0 con 0x558b86eebfe0 2024-05-27T09:45:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c70409b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c1b5550 con 0x558b8706b680 2024-05-27T09:45:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c70409b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c116a10 con 0x558b86fd7b90 2024-05-27T09:45:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c5a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86fd7b90 0x558b87095920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c5a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86fd7b90 0x558b87095920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53266/0 (socket says 172.21.15.44:53266) 2024-05-27T09:45:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c5a000700 1 -- 172.21.15.44:0/1009705192 learned_addr learned my addr 172.21.15.44:0/1009705192 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c5a000700 1 -- 172.21.15.44:0/1009705192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b8706b680 msgr2=0x558b87070330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:45:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c5a000700 1 --2- 172.21.15.44:0/1009705192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b8706b680 0x558b87070330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c5a000700 1 -- 172.21.15.44:0/1009705192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86eebfe0 msgr2=0x558b870953e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:45:24.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c5a000700 1 --2- 172.21.15.44:0/1009705192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86eebfe0 0x558b870953e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.096+0000 7f7c5a000700 1 -- 172.21.15.44:0/1009705192 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b87083eb0 con 0x558b86fd7b90 2024-05-27T09:45:24.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.097+0000 7f7c5a000700 1 --2- 172.21.15.44:0/1009705192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86fd7b90 0x558b87095920 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7c4801f040 tx=0x7f7c48009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aca45369fe43891 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:24.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.097+0000 7f7c47400700 1 -- 172.21.15.44:0/1009705192 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c480ab910 con 0x558b86fd7b90 2024-05-27T09:45:24.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.097+0000 7f7c47400700 1 -- 172.21.15.44:0/1009705192 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c480b0e50 con 0x558b86fd7b90 2024-05-27T09:45:24.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.097+0000 7f7c47400700 1 -- 172.21.15.44:0/1009705192 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c480abdd0 con 0x558b86fd7b90 2024-05-27T09:45:24.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.098+0000 7f7c70409b80 1 -- 172.21.15.44:0/1009705192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86fd7b90 msgr2=0x558b87095920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:24.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.098+0000 7f7c70409b80 1 --2- 172.21.15.44:0/1009705192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86fd7b90 0x558b87095920 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7c4801f040 tx=0x7f7c48009310 comp rx=0 tx=0).stop 2024-05-27T09:45:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.099+0000 7f7c70409b80 1 -- 172.21.15.44:0/1009705192 shutdown_connections 2024-05-27T09:45:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.099+0000 7f7c70409b80 1 --2- 172.21.15.44:0/1009705192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b8706b680 0x558b87070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.100+0000 7f7c70409b80 1 --2- 172.21.15.44:0/1009705192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b86eebfe0 0x558b870953e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.100+0000 7f7c70409b80 1 --2- 172.21.15.44:0/1009705192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86fd7b90 0x558b87095920 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.100+0000 7f7c70409b80 1 -- 172.21.15.44:0/1009705192 >> 172.21.15.44:0/1009705192 conn(0x558b86f85530 msgr2=0x558b8707d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:24.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.103+0000 7f7c70409b80 1 -- 172.21.15.44:0/1009705192 shutdown_connections 2024-05-27T09:45:24.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.103+0000 7f7c70409b80 1 -- 172.21.15.44:0/1009705192 wait complete. 2024-05-27T09:45:24.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.104+0000 7f7c70409b80 1 -- start start 2024-05-27T09:45:24.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.105+0000 7f7c70409b80 1 Processor -- start 2024-05-27T09:45:24.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.106+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86eebfe0 0x558b8708a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.106+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86fd7b90 0x558b870883c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.107+0000 7f7c5aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86eebfe0 0x558b8708a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.107+0000 7f7c5aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86eebfe0 0x558b8708a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53268/0 (socket says 172.21.15.44:53268) 2024-05-27T09:45:24.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.107+0000 7f7c70409b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b8706b680 0x558b870889c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.107+0000 7f7c5a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86fd7b90 0x558b870883c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.107+0000 7f7c5a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86fd7b90 0x558b870883c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54452/0 (socket says 172.21.15.44:54452) 2024-05-27T09:45:24.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.107+0000 7f7c5aa00700 1 -- 172.21.15.44:0/2983984268 learned_addr learned my addr 172.21.15.44:0/2983984268 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:24.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.107+0000 7f7c70409b80 1 -- 172.21.15.44:0/2983984268 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c1b3a10 con 0x558b8706b680 2024-05-27T09:45:24.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.107+0000 7f7c70409b80 1 -- 172.21.15.44:0/2983984268 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c1b4e80 con 0x558b86fd7b90 2024-05-27T09:45:24.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.107+0000 7f7c70409b80 1 -- 172.21.15.44:0/2983984268 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c1b40e0 con 0x558b86eebfe0 2024-05-27T09:45:24.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.108+0000 7f7c70409b80 10 client.0 _ll_register_callbacks cb 0x558b86f32e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:24.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.108+0000 7f7c5b400700 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b8706b680 0x558b870889c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.108+0000 7f7c5aa00700 1 -- 172.21.15.44:0/2983984268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86fd7b90 msgr2=0x558b870883c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.108+0000 7f7c5aa00700 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86fd7b90 0x558b870883c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.108+0000 7f7c5aa00700 1 -- 172.21.15.44:0/2983984268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b8706b680 msgr2=0x558b870889c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.108+0000 7f7c5aa00700 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b8706b680 0x558b870889c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.108+0000 7f7c5aa00700 1 -- 172.21.15.44:0/2983984268 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b870629e0 con 0x558b86eebfe0 2024-05-27T09:45:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.109+0000 7f7c5aa00700 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86eebfe0 0x558b8708a030 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x558b87090380 tx=0x7f7c4c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f3761761ce293e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:24.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.109+0000 7f7c44c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:45:24.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.110+0000 7f7c44c00700 1 -- 172.21.15.44:0/2983984268 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c004e40 con 0x558b86eebfe0 2024-05-27T09:45:24.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.110+0000 7f7c44c00700 1 -- 172.21.15.44:0/2983984268 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f7c4c004020 con 0x558b86eebfe0 2024-05-27T09:45:24.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.110+0000 7f7c70409b80 1 -- 172.21.15.44:0/2983984268 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c480b0e50 con 0x558b86eebfe0 2024-05-27T09:45:24.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.111+0000 7f7c44c00700 1 -- 172.21.15.44:0/2983984268 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7c4c004220 con 0x558b86eebfe0 2024-05-27T09:45:24.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.111+0000 7f7c70409b80 10 client.27127 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:24.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.111+0000 7f7c70409b80 1 -- 172.21.15.44:0/2983984268 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558b870f90c0 con 0x558b86eebfe0 2024-05-27T09:45:24.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.112+0000 7f7c44c00700 1 -- 172.21.15.44:0/2983984268 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c480b0e50 con 0x558b86eebfe0 2024-05-27T09:45:24.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.112+0000 7f7c44c00700 1 -- 172.21.15.44:0/2983984268 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f7c4c01f040 con 0x558b86eebfe0 2024-05-27T09:45:24.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.112+0000 7f7c70409b80 10 client.27127 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:24.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.112+0000 7f7c70409b80 10 client.27127 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:45:24.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.112+0000 7f7c70409b80 20 client.27127 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:24.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.112+0000 7f7c70409b80 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558b870f98a0 0x558b870fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:24.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.112+0000 7f7c70409b80 4 client.27127 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.95, "await_for": 45.0}] 2024-05-27T09:45:24.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.112+0000 7f7c70409b80 1 -- 172.21.15.44:0/2983984268 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.95, "await_for": 45.0}) -- 0x558b8705e8b0 con 0x558b870f98a0 2024-05-27T09:45:24.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.112+0000 7f7c5a000700 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558b870f98a0 0x558b870fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:24.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.114+0000 7f7c5a000700 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558b870f98a0 0x558b870fbcc0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=498596031c6237e1 server_cookie=2941c4dc14c02936 in_seq=0 out_seq=0 2024-05-27T09:45:24.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:24.114+0000 7f7c44c00700 10 client.27127 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:24 smithi189 ceph-mon[36671]: pgmap v317: 97 pgs: 97 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 574 KiB/s wr, 608 op/s 2024-05-27T09:45:24.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:24 smithi044 ceph-mon[31618]: pgmap v317: 97 pgs: 97 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 574 KiB/s wr, 608 op/s 2024-05-27T09:45:24.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:24 smithi050 ceph-mon[36371]: pgmap v317: 97 pgs: 97 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 574 KiB/s wr, 608 op/s 2024-05-27T09:45:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.331+0000 7f38ef200640 1 -- 172.21.15.44:0/1005678120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38f0104220 msgr2=0x7f38f01105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.331+0000 7f38ef200640 1 --2- 172.21.15.44:0/1005678120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38f0104220 0x7f38f01105f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009860 tx=0x7f38e802a340 comp rx=0 tx=0).stop 2024-05-27T09:45:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.331+0000 7f38ef200640 1 -- 172.21.15.44:0/1005678120 shutdown_connections 2024-05-27T09:45:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.331+0000 7f38ef200640 1 --2- 172.21.15.44:0/1005678120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38f0104220 0x7f38f01105f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.331+0000 7f38ef200640 1 --2- 172.21.15.44:0/1005678120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0109f70 0x7f38f0103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.331+0000 7f38ef200640 1 --2- 172.21.15.44:0/1005678120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38f0109ad0 0x7f38f0103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.331+0000 7f38ef200640 1 -- 172.21.15.44:0/1005678120 >> 172.21.15.44:0/1005678120 conn(0x7f38f0079100 msgr2=0x7f38f0079540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.331+0000 7f38ef200640 1 -- 172.21.15.44:0/1005678120 shutdown_connections 2024-05-27T09:45:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.331+0000 7f38ef200640 1 -- 172.21.15.44:0/1005678120 wait complete. 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.332+0000 7f38ef200640 1 Processor -- start 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.332+0000 7f38ef200640 1 -- start start 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.332+0000 7f38ef200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38f0104220 0x7f38f019a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.332+0000 7f38ef200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38f0109ad0 0x7f38f019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ef200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0109f70 0x7f38f019ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ef200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f38f0113cc0 con 0x7f38f0109f70 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ef200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f38f0113b40 con 0x7f38f0104220 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ef200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38f0113e40 con 0x7f38f0109ad0 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ede00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38f0104220 0x7f38f019a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ed400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38f0109ad0 0x7f38f019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ede00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38f0104220 0x7f38f019a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54468/0 (socket says 172.21.15.44:54468) 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ede00640 1 -- 172.21.15.44:0/282372559 learned_addr learned my addr 172.21.15.44:0/282372559 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ede00640 1 -- 172.21.15.44:0/282372559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38f0109ad0 msgr2=0x7f38f019ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ede00640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38f0109ad0 0x7f38f019ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ede00640 1 -- 172.21.15.44:0/282372559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0109f70 msgr2=0x7f38f019ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ee800640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0109f70 0x7f38f019ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ede00640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0109f70 0x7f38f019ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.333+0000 7f38ede00640 1 -- 172.21.15.44:0/282372559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f019f480 con 0x7f38f0104220 2024-05-27T09:45:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.334+0000 7f38ede00640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38f0104220 0x7f38f019a620 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f38e000e9a0 tx=0x7f38e000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.334+0000 7f38e6a00640 1 -- 172.21.15.44:0/282372559 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e000cd80 con 0x7f38f0104220 2024-05-27T09:45:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.334+0000 7f38e6a00640 1 -- 172.21.15.44:0/282372559 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e00035f0 con 0x7f38f0104220 2024-05-27T09:45:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.334+0000 7f38e6a00640 1 -- 172.21.15.44:0/282372559 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e0010690 con 0x7f38f0104220 2024-05-27T09:45:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.334+0000 7f38ef200640 1 -- 172.21.15.44:0/282372559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f019f770 con 0x7f38f0104220 2024-05-27T09:45:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.334+0000 7f38ef200640 1 -- 172.21.15.44:0/282372559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f01a70f0 con 0x7f38f0104220 2024-05-27T09:45:26.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:26 smithi189 ceph-mon[36671]: pgmap v318: 97 pgs: 97 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 483 KiB/s wr, 512 op/s 2024-05-27T09:45:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.337+0000 7f38fc1b7740 1 Processor -- start 2024-05-27T09:45:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38fc1b7740 1 -- start start 2024-05-27T09:45:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 0x5584de8eca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de8ecfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de5294d0 0x5584de8ed4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5584de85db90 con 0x5584de5294d0 2024-05-27T09:45:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5584de84d730 con 0x5584de54a1b0 2024-05-27T09:45:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5584de851770 con 0x5584de5212f0 2024-05-27T09:45:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38e6a00640 1 -- 172.21.15.44:0/282372559 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f38e0004000 con 0x7f38f0104220 2024-05-27T09:45:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38ee800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 0x5584de8eca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38ee800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 0x5584de8eca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54470/0 (socket says 172.21.15.44:54470) 2024-05-27T09:45:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38ee800640 1 -- 172.21.15.44:0/670687465 learned_addr learned my addr 172.21.15.44:0/670687465 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38ed400640 1 --2- 172.21.15.44:0/670687465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de5294d0 0x5584de8ed4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.338+0000 7f38ede00640 1 --2- 172.21.15.44:0/670687465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de8ecfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.339+0000 7f38ee800640 1 -- 172.21.15.44:0/670687465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 msgr2=0x5584de8ecfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.339+0000 7f38ee800640 1 --2- 172.21.15.44:0/670687465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de8ecfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.339+0000 7f38ee800640 1 -- 172.21.15.44:0/670687465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de5294d0 msgr2=0x5584de8ed4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.339+0000 7f38ee800640 1 --2- 172.21.15.44:0/670687465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de5294d0 0x5584de8ed4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.339+0000 7f38ee800640 1 -- 172.21.15.44:0/670687465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584de7e68b0 con 0x5584de54a1b0 2024-05-27T09:45:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.339+0000 7f38ed400640 1 --2- 172.21.15.44:0/670687465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de5294d0 0x5584de8ed4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:45:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.339+0000 7f38ee800640 1 --2- 172.21.15.44:0/670687465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 0x5584de8eca60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f38e802a860 tx=0x7f38e8005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=676b6e1fde69a5ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.339+0000 7f38df400640 1 -- 172.21.15.44:0/670687465 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e8040020 con 0x5584de54a1b0 2024-05-27T09:45:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.339+0000 7f38df400640 1 -- 172.21.15.44:0/670687465 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e8038040 con 0x5584de54a1b0 2024-05-27T09:45:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.339+0000 7f38df400640 1 -- 172.21.15.44:0/670687465 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e8037070 con 0x5584de54a1b0 2024-05-27T09:45:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38e6a00640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f38d00b2950 0x7f38d00b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38e6a00640 1 -- 172.21.15.44:0/282372559 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f38e00e29a0 con 0x7f38f0104220 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38fc1b7740 1 -- 172.21.15.44:0/670687465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 msgr2=0x5584de8eca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/670687465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 0x5584de8eca60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f38e802a860 tx=0x7f38e8005d80 comp rx=0 tx=0).stop 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38ede00640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f38d00b2950 0x7f38d00b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38fc1b7740 1 -- 172.21.15.44:0/670687465 shutdown_connections 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/670687465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de5294d0 0x5584de8ed4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/670687465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de8ecfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/670687465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 0x5584de8eca60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38fc1b7740 1 -- 172.21.15.44:0/670687465 >> 172.21.15.44:0/670687465 conn(0x5584de53e300 msgr2=0x5584de8e9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38fc1b7740 1 -- 172.21.15.44:0/670687465 shutdown_connections 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.340+0000 7f38fc1b7740 1 -- 172.21.15.44:0/670687465 wait complete. 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.341+0000 7f38ede00640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f38d00b2950 0x7f38d00b4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f38e00e01a0 tx=0x7f38e0000bc0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.341+0000 7f38fc1b7740 1 -- start start 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.341+0000 7f38fc1b7740 1 Processor -- start 2024-05-27T09:45:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de9010f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5294d0 0x5584de901630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de54a1b0 0x5584de8fb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5584de533790 con 0x5584de54a1b0 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5584de7f5bc0 con 0x5584de5294d0 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5584de8f9b30 con 0x5584de5212f0 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38ed400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5294d0 0x5584de901630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38fc1b7740 10 client.0 _ll_register_callbacks cb 0x5584de7d3400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38ed400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5294d0 0x5584de901630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54476/0 (socket says 172.21.15.44:54476) 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38ed400640 1 -- 172.21.15.44:0/1609356974 learned_addr learned my addr 172.21.15.44:0/1609356974 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38ee800640 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de9010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.342+0000 7f38ee800640 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de54a1b0 0x5584de8fb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.343+0000 7f38ed400640 1 -- 172.21.15.44:0/1609356974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 msgr2=0x5584de9010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.343+0000 7f38ed400640 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de9010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.343+0000 7f38ed400640 1 -- 172.21.15.44:0/1609356974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de54a1b0 msgr2=0x5584de8fb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.343+0000 7f38ed400640 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de54a1b0 0x5584de8fb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.343+0000 7f38ed400640 1 -- 172.21.15.44:0/1609356974 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584de562210 con 0x5584de5294d0 2024-05-27T09:45:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.343+0000 7f38ed400640 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5294d0 0x5584de901630 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f38d800eb20 tx=0x7f38d800d3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3dff13e57322163 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.343+0000 7f38d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.343+0000 7f38d7e00640 1 -- 172.21.15.44:0/1609356974 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d801b020 con 0x5584de5294d0 2024-05-27T09:45:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.343+0000 7f38fc1b7740 1 -- 172.21.15.44:0/1609356974 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38e8038040 con 0x5584de5294d0 2024-05-27T09:45:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.343+0000 7f38d7e00640 1 -- 172.21.15.44:0/1609356974 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f38d80175f0 con 0x5584de5294d0 2024-05-27T09:45:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.344+0000 7f38d7e00640 1 -- 172.21.15.44:0/1609356974 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f38d8026050 con 0x5584de5294d0 2024-05-27T09:45:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.344+0000 7f38fc1b7740 10 client.27155 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.344+0000 7f38fc1b7740 1 -- 172.21.15.44:0/1609356974 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5584de62e620 con 0x5584de5294d0 2024-05-27T09:45:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.344+0000 7f38d7e00640 1 -- 172.21.15.44:0/1609356974 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e8038040 con 0x5584de5294d0 2024-05-27T09:45:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.344+0000 7f38d7e00640 1 -- 172.21.15.44:0/1609356974 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f38d800b040 con 0x5584de5294d0 2024-05-27T09:45:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.344+0000 7f38fc1b7740 10 client.27155 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.344+0000 7f38fc1b7740 10 client.27155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.344+0000 7f38fc1b7740 20 client.27155 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.344+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5584de508d80 0x5584de90d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.344+0000 7f38fc1b7740 4 client.27155 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.344+0000 7f38fc1b7740 1 -- 172.21.15.44:0/1609356974 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584de85fae0 con 0x5584de508d80 2024-05-27T09:45:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.345+0000 7f38ee800640 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5584de508d80 0x5584de90d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.345+0000 7f38ee800640 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5584de508d80 0x5584de90d310 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86dcbb036f832f64 server_cookie=1e00d2c81819e7e3 in_seq=0 out_seq=0 2024-05-27T09:45:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.345+0000 7f38d7e00640 10 client.27155 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:26.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:26 smithi044 ceph-mon[31618]: pgmap v318: 97 pgs: 97 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 483 KiB/s wr, 512 op/s 2024-05-27T09:45:26.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:26 smithi050 ceph-mon[36371]: pgmap v318: 97 pgs: 97 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 483 KiB/s wr, 512 op/s 2024-05-27T09:45:26.731 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:45:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.749+0000 7f38d7e00640 1 -- 172.21.15.44:0/1609356974 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5584de85fae0 con 0x5584de508d80 2024-05-27T09:45:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.749+0000 7f38d7e00640 10 client.27155 handle_command_reply: tid=0 2024-05-27T09:45:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.749+0000 7f38fc1b7740 1 client.27155 shutdown 2024-05-27T09:45:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.750+0000 7f38fc1b7740 1 -- 172.21.15.44:0/1609356974 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5294d0 msgr2=0x5584de901630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.750+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5294d0 0x5584de901630 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f38d800eb20 tx=0x7f38d800d3e0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.750+0000 7f38fc1b7740 1 -- 172.21.15.44:0/1609356974 shutdown_connections 2024-05-27T09:45:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.750+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5584de508d80 0x5584de90d310 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.750+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de54a1b0 0x5584de8fb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.750+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5294d0 0x5584de901630 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.750+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/1609356974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de9010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.750+0000 7f38d7e00640 0 client.27155 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.750+0000 7f38fc1b7740 1 -- 172.21.15.44:0/1609356974 >> 172.21.15.44:0/1609356974 conn(0x5584de53e300 msgr2=0x5584de8ec890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.750+0000 7f38fc1b7740 1 -- 172.21.15.44:0/1609356974 shutdown_connections 2024-05-27T09:45:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.750+0000 7f38fc1b7740 1 -- 172.21.15.44:0/1609356974 wait complete. 2024-05-27T09:45:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.751+0000 7f38fc1b7740 20 client.27155 trim_cache size 0 max 16384 2024-05-27T09:45:26.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.792+0000 7f38fc1b7740 1 Processor -- start 2024-05-27T09:45:26.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.793+0000 7f38fc1b7740 1 -- start start 2024-05-27T09:45:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.793+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5212f0 0x5584de8f67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.793+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5294d0 0x5584de8ecf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.793+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de54a1b0 0x5584de8f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.793+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f38f01affb0 con 0x5584de54a1b0 2024-05-27T09:45:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.794+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f38f01ad850 con 0x5584de5212f0 2024-05-27T09:45:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.794+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38f01ab6c0 con 0x5584de5294d0 2024-05-27T09:45:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.794+0000 7f38ed400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5294d0 0x5584de8ecf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.794+0000 7f38ed400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5294d0 0x5584de8ecf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53296/0 (socket says 172.21.15.44:53296) 2024-05-27T09:45:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.794+0000 7f38ee800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5212f0 0x5584de8f67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.794+0000 7f38ed400640 1 -- 172.21.15.44:0/713956237 learned_addr learned my addr 172.21.15.44:0/713956237 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.794+0000 7f38ee800640 1 --2- 172.21.15.44:0/713956237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de54a1b0 0x5584de8f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.794+0000 7f38ed400640 1 -- 172.21.15.44:0/713956237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5212f0 msgr2=0x5584de8f67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.795+0000 7f38ed400640 1 --2- 172.21.15.44:0/713956237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5212f0 0x5584de8f67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.795+0000 7f38ed400640 1 -- 172.21.15.44:0/713956237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de54a1b0 msgr2=0x5584de8f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.795+0000 7f38ed400640 1 --2- 172.21.15.44:0/713956237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de54a1b0 0x5584de8f4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.795+0000 7f38ed400640 1 -- 172.21.15.44:0/713956237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584de8fa460 con 0x5584de5294d0 2024-05-27T09:45:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.795+0000 7f38ee800640 1 --2- 172.21.15.44:0/713956237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5212f0 0x5584de8f67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.795+0000 7f38ed400640 1 --2- 172.21.15.44:0/713956237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5294d0 0x5584de8ecf00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f38d801f9d0 tx=0x7f38d80059f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65e3d5c00a3eba42 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.795+0000 7f38dea00640 1 -- 172.21.15.44:0/713956237 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d802f070 con 0x5584de5294d0 2024-05-27T09:45:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.795+0000 7f38dea00640 1 -- 172.21.15.44:0/713956237 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38d8009630 con 0x5584de5294d0 2024-05-27T09:45:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.795+0000 7f38dea00640 1 -- 172.21.15.44:0/713956237 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d801b070 con 0x5584de5294d0 2024-05-27T09:45:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.797+0000 7f38fc1b7740 1 -- 172.21.15.44:0/713956237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5294d0 msgr2=0x5584de8ecf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.797+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/713956237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5294d0 0x5584de8ecf00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f38d801f9d0 tx=0x7f38d80059f0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.797+0000 7f38fc1b7740 1 -- 172.21.15.44:0/713956237 shutdown_connections 2024-05-27T09:45:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.797+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/713956237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de54a1b0 0x5584de8f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.797+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/713956237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5294d0 0x5584de8ecf00 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.797+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/713956237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de5212f0 0x5584de8f67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.797+0000 7f38fc1b7740 1 -- 172.21.15.44:0/713956237 >> 172.21.15.44:0/713956237 conn(0x5584de53e300 msgr2=0x5584de8ec7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.797+0000 7f38fc1b7740 1 -- 172.21.15.44:0/713956237 shutdown_connections 2024-05-27T09:45:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.798+0000 7f38fc1b7740 1 -- 172.21.15.44:0/713956237 wait complete. 2024-05-27T09:45:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.799+0000 7f38fc1b7740 1 -- start start 2024-05-27T09:45:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.799+0000 7f38fc1b7740 1 Processor -- start 2024-05-27T09:45:26.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.800+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de907630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.800+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de5294d0 0x5584de907b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.800+0000 7f38fc1b7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 0x5584de9029e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.800+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f38f01abd10 con 0x5584de5294d0 2024-05-27T09:45:26.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.800+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f38f01ad180 con 0x5584de54a1b0 2024-05-27T09:45:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.800+0000 7f38fc1b7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38f01ac3e0 con 0x5584de5212f0 2024-05-27T09:45:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38ee800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de907630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38fc1b7740 10 client.0 _ll_register_callbacks cb 0x7f38d80099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38ee800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de907630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53308/0 (socket says 172.21.15.44:53308) 2024-05-27T09:45:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38ee800640 1 -- 172.21.15.44:0/482587978 learned_addr learned my addr 172.21.15.44:0/482587978 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38ed400640 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de5294d0 0x5584de907b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38ee800640 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 0x5584de9029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38ee800640 1 -- 172.21.15.44:0/482587978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 msgr2=0x5584de9029e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38ee800640 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 0x5584de9029e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38ee800640 1 -- 172.21.15.44:0/482587978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de5294d0 msgr2=0x5584de907b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38ee800640 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de5294d0 0x5584de907b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38ee800640 1 -- 172.21.15.44:0/482587978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584de903170 con 0x5584de5212f0 2024-05-27T09:45:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38ee800640 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de907630 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009860 tx=0x7f38e8005d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f6d8273df6adc7c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.801+0000 7f38d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:45:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.802+0000 7f38d7e00640 1 -- 172.21.15.44:0/482587978 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e8040040 con 0x5584de5212f0 2024-05-27T09:45:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.802+0000 7f38d7e00640 1 -- 172.21.15.44:0/482587978 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f38e8053e00 con 0x5584de5212f0 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.802+0000 7f38fc1b7740 1 -- 172.21.15.44:0/482587978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38d8009630 con 0x5584de5212f0 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.802+0000 7f38d7e00640 1 -- 172.21.15.44:0/482587978 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f38e8037040 con 0x5584de5212f0 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.802+0000 7f38fc1b7740 10 client.27145 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.802+0000 7f38fc1b7740 1 -- 172.21.15.44:0/482587978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5584de975410 con 0x5584de5212f0 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.802+0000 7f38d7e00640 1 -- 172.21.15.44:0/482587978 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38d8009630 con 0x5584de5212f0 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.803+0000 7f38d7e00640 1 -- 172.21.15.44:0/482587978 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f38e80046c0 con 0x5584de5212f0 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.803+0000 7f38fc1b7740 10 client.27145 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.803+0000 7f38fc1b7740 10 client.27145 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.803+0000 7f38fc1b7740 20 client.27145 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.803+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5584de4de1c0 0x5584de977760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.803+0000 7f38fc1b7740 4 client.27145 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.803+0000 7f38fc1b7740 1 -- 172.21.15.44:0/482587978 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5584de8592c0 con 0x5584de4de1c0 2024-05-27T09:45:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.803+0000 7f38ed400640 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5584de4de1c0 0x5584de977760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.804+0000 7f38ed400640 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5584de4de1c0 0x5584de977760 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcc5757b9c363f43 server_cookie=5f8dd05e309fff6e in_seq=0 out_seq=0 2024-05-27T09:45:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:26.804+0000 7f38d7e00640 10 client.27145 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:27.212 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:27.346 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.345+0000 7f38d7e00640 1 -- 172.21.15.44:0/482587978 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5584de8592c0 con 0x5584de4de1c0 2024-05-27T09:45:27.346 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.345+0000 7f38d7e00640 10 client.27145 handle_command_reply: tid=0 2024-05-27T09:45:27.346 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.345+0000 7f38fc1b7740 1 client.27145 shutdown 2024-05-27T09:45:27.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.346+0000 7f38fc1b7740 1 -- 172.21.15.44:0/482587978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 msgr2=0x5584de907630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:27.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.346+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de907630 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009860 tx=0x7f38e8005d80 comp rx=0 tx=0).stop 2024-05-27T09:45:27.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.346+0000 7f38fc1b7740 1 -- 172.21.15.44:0/482587978 shutdown_connections 2024-05-27T09:45:27.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.346+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5584de4de1c0 0x5584de977760 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:27.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.346+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5584de54a1b0 0x5584de9029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:27.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.346+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584de5294d0 0x5584de907b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:27.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.346+0000 7f38fc1b7740 1 --2- 172.21.15.44:0/482587978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584de5212f0 0x5584de907630 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:27.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.346+0000 7f38fc1b7740 1 -- 172.21.15.44:0/482587978 >> 172.21.15.44:0/482587978 conn(0x5584de53e300 msgr2=0x5584de911ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:27.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.346+0000 7f38d7e00640 0 client.27145 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:27.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.347+0000 7f38fc1b7740 1 -- 172.21.15.44:0/482587978 shutdown_connections 2024-05-27T09:45:27.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.347+0000 7f38fc1b7740 1 -- 172.21.15.44:0/482587978 wait complete. 2024-05-27T09:45:27.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.347+0000 7f38fc1b7740 20 client.27145 trim_cache size 0 max 16384 2024-05-27T09:45:27.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:45:27.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:45:27.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:45:27.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.348+0000 7f38ef200640 1 -- 172.21.15.44:0/282372559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f38d00b2950 msgr2=0x7f38d00b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:27.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.348+0000 7f38ef200640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f38d00b2950 0x7f38d00b4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f38e00e01a0 tx=0x7f38e0000bc0 comp rx=0 tx=0).stop 2024-05-27T09:45:27.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.348+0000 7f38ef200640 1 -- 172.21.15.44:0/282372559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38f0104220 msgr2=0x7f38f019a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:27.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.348+0000 7f38ef200640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38f0104220 0x7f38f019a620 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f38e000e9a0 tx=0x7f38e000ee60 comp rx=0 tx=0).stop 2024-05-27T09:45:27.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.349+0000 7f38ef200640 1 -- 172.21.15.44:0/282372559 shutdown_connections 2024-05-27T09:45:27.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.349+0000 7f38ef200640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f38d00b2950 0x7f38d00b4e60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:27.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.349+0000 7f38ef200640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0109f70 0x7f38f019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:27.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.349+0000 7f38ef200640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38f0109ad0 0x7f38f019ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:27.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.349+0000 7f38ef200640 1 --2- 172.21.15.44:0/282372559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38f0104220 0x7f38f019a620 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:27.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.349+0000 7f38ef200640 1 -- 172.21.15.44:0/282372559 >> 172.21.15.44:0/282372559 conn(0x7f38f0079100 msgr2=0x7f38f0100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:27.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.350+0000 7f38ef200640 1 -- 172.21.15.44:0/282372559 shutdown_connections 2024-05-27T09:45:27.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:27.350+0000 7f38ef200640 1 -- 172.21.15.44:0/282372559 wait complete. 2024-05-27T09:45:28.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:28 smithi189 ceph-mon[36671]: pgmap v319: 97 pgs: 97 active+clean; 284 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 665 op/s 2024-05-27T09:45:28.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:28 smithi044 ceph-mon[31618]: pgmap v319: 97 pgs: 97 active+clean; 284 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 665 op/s 2024-05-27T09:45:28.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:28 smithi050 ceph-mon[36371]: pgmap v319: 97 pgs: 97 active+clean; 284 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 665 op/s 2024-05-27T09:45:29.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.353+0000 7faec3400700 1 -- 172.21.15.44:0/954905596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faebc109860 msgr2=0x7faebc1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:29.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.353+0000 7faec3400700 1 --2- 172.21.15.44:0/954905596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faebc109860 0x7faebc1111c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7faeb400b0d0 tx=0x7faeb400b3e0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.353+0000 7faec3400700 1 -- 172.21.15.44:0/954905596 shutdown_connections 2024-05-27T09:45:29.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.353+0000 7faec3400700 1 --2- 172.21.15.44:0/954905596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faebc109860 0x7faebc1111c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.353+0000 7faec3400700 1 --2- 172.21.15.44:0/954905596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc1045e0 0x7faebc1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.353+0000 7faec3400700 1 --2- 172.21.15.44:0/954905596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faebc108f10 0x7faebc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.353+0000 7faec3400700 1 -- 172.21.15.44:0/954905596 >> 172.21.15.44:0/954905596 conn(0x7faebc100030 msgr2=0x7faebc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:29.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.353+0000 7faec3400700 1 -- 172.21.15.44:0/954905596 shutdown_connections 2024-05-27T09:45:29.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.353+0000 7faec3400700 1 -- 172.21.15.44:0/954905596 wait complete. 2024-05-27T09:45:29.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.354+0000 7faec3400700 1 Processor -- start 2024-05-27T09:45:29.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.355+0000 7faec3400700 1 -- start start 2024-05-27T09:45:29.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.355+0000 7faec3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faebc1045e0 0x7faebc19d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.355+0000 7faec3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faebc108f10 0x7faebc19d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.355+0000 7faec3400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc109860 0x7faebc1a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.355+0000 7faec3400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1144e0 con 0x7faebc108f10 2024-05-27T09:45:29.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.355+0000 7faec3400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faebc114360 con 0x7faebc109860 2024-05-27T09:45:29.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.355+0000 7faec3400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faebc114660 con 0x7faebc1045e0 2024-05-27T09:45:29.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.356+0000 7faebbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faebc108f10 0x7faebc19d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.356+0000 7faec0a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faebc1045e0 0x7faebc19d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.356+0000 7faebbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faebc108f10 0x7faebc19d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55894/0 (socket says 172.21.15.44:55894) 2024-05-27T09:45:29.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.356+0000 7faebbe00700 1 -- 172.21.15.44:0/2190883412 learned_addr learned my addr 172.21.15.44:0/2190883412 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:29.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.356+0000 7faec1400700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc109860 0x7faebc1a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.356+0000 7faec0a00700 1 -- 172.21.15.44:0/2190883412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc109860 msgr2=0x7faebc1a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:29.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.356+0000 7faec0a00700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc109860 0x7faebc1a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.356+0000 7faec0a00700 1 -- 172.21.15.44:0/2190883412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faebc108f10 msgr2=0x7faebc19d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:29.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.356+0000 7faec0a00700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faebc108f10 0x7faebc19d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.356+0000 7faec0a00700 1 -- 172.21.15.44:0/2190883412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faebc1a1f40 con 0x7faebc1045e0 2024-05-27T09:45:29.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.356+0000 7faec0a00700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faebc1045e0 0x7faebc19d070 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7faeb000e9f0 tx=0x7faeb000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:29.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.357+0000 7faeb9600700 1 -- 172.21.15.44:0/2190883412 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb000c9c0 con 0x7faebc1045e0 2024-05-27T09:45:29.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.357+0000 7faeb9600700 1 -- 172.21.15.44:0/2190883412 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeb000cb60 con 0x7faebc1045e0 2024-05-27T09:45:29.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.357+0000 7faeb9600700 1 -- 172.21.15.44:0/2190883412 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb0018680 con 0x7faebc1045e0 2024-05-27T09:45:29.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.357+0000 7faec3400700 1 -- 172.21.15.44:0/2190883412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faebc1a2230 con 0x7faebc1045e0 2024-05-27T09:45:29.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.357+0000 7faec3400700 1 -- 172.21.15.44:0/2190883412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faebc1a9b50 con 0x7faebc1045e0 2024-05-27T09:45:29.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.361+0000 7faeb9600700 1 -- 172.21.15.44:0/2190883412 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7faeb0010430 con 0x7faebc1045e0 2024-05-27T09:45:29.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.362+0000 7faed1f7db80 1 Processor -- start 2024-05-27T09:45:29.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.362+0000 7faed1f7db80 1 -- start start 2024-05-27T09:45:29.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.362+0000 7faeb9600700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7faea40b2a70 0x7faea40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.362+0000 7faeb9600700 1 -- 172.21.15.44:0/2190883412 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7faeb00e2760 con 0x7faebc1045e0 2024-05-27T09:45:29.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 0x5642c5f5a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faec1400700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7faea40b2a70 0x7faea40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f5ad30 0x5642c5f54870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f54db0 0x5642c5f551c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faed1f7db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5642c5e82e40 con 0x5642c5dd5e40 2024-05-27T09:45:29.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faed1f7db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5642c5e729e0 con 0x5642c5f5ad30 2024-05-27T09:45:29.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faed1f7db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5642c5e76a20 con 0x5642c5f54db0 2024-05-27T09:45:29.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faec0a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 0x5642c5f5a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faec0a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 0x5642c5f5a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55906/0 (socket says 172.21.15.44:55906) 2024-05-27T09:45:29.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faec0a00700 1 -- 172.21.15.44:0/4036256827 learned_addr learned my addr 172.21.15.44:0/4036256827 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:29.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faec1400700 1 --2- 172.21.15.44:0/4036256827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f54db0 0x5642c5f551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.363+0000 7faebbe00700 1 --2- 172.21.15.44:0/4036256827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f5ad30 0x5642c5f54870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.364+0000 7faebbe00700 1 -- 172.21.15.44:0/4036256827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f54db0 msgr2=0x5642c5f551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:29.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.364+0000 7faebbe00700 1 --2- 172.21.15.44:0/4036256827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f54db0 0x5642c5f551c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.364+0000 7faebbe00700 1 -- 172.21.15.44:0/4036256827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 msgr2=0x5642c5f5a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:29.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.364+0000 7faebbe00700 1 --2- 172.21.15.44:0/4036256827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 0x5642c5f5a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.364+0000 7faebbe00700 1 -- 172.21.15.44:0/4036256827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642c5e3b420 con 0x5642c5f5ad30 2024-05-27T09:45:29.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.364+0000 7faec0a00700 1 --2- 172.21.15.44:0/4036256827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 0x5642c5f5a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:29.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.364+0000 7faec1400700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7faea40b2a70 0x7faea40b4f80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009fd0 tx=0x7faeb40093c0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:29.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.364+0000 7faebbe00700 1 --2- 172.21.15.44:0/4036256827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f5ad30 0x5642c5f54870 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7faea8005c50 tx=0x7faea8007ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e349463b32223954 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:29.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.364+0000 7faeae000700 1 -- 172.21.15.44:0/4036256827 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8004220 con 0x5642c5f5ad30 2024-05-27T09:45:29.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.364+0000 7faeae000700 1 -- 172.21.15.44:0/4036256827 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faea80043c0 con 0x5642c5f5ad30 2024-05-27T09:45:29.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.364+0000 7faeae000700 1 -- 172.21.15.44:0/4036256827 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8003d20 con 0x5642c5f5ad30 2024-05-27T09:45:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.365+0000 7faed1f7db80 1 -- 172.21.15.44:0/4036256827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f5ad30 msgr2=0x5642c5f54870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.365+0000 7faed1f7db80 1 --2- 172.21.15.44:0/4036256827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f5ad30 0x5642c5f54870 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7faea8005c50 tx=0x7faea8007ee0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.366+0000 7faed1f7db80 1 -- 172.21.15.44:0/4036256827 shutdown_connections 2024-05-27T09:45:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.366+0000 7faed1f7db80 1 --2- 172.21.15.44:0/4036256827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f54db0 0x5642c5f551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.366+0000 7faed1f7db80 1 --2- 172.21.15.44:0/4036256827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 0x5642c5f5a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.366+0000 7faed1f7db80 1 --2- 172.21.15.44:0/4036256827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f5ad30 0x5642c5f54870 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.366+0000 7faed1f7db80 1 -- 172.21.15.44:0/4036256827 >> 172.21.15.44:0/4036256827 conn(0x5642c5e35fc0 msgr2=0x5642c5f4f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.366+0000 7faed1f7db80 1 -- 172.21.15.44:0/4036256827 shutdown_connections 2024-05-27T09:45:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.366+0000 7faed1f7db80 1 -- 172.21.15.44:0/4036256827 wait complete. 2024-05-27T09:45:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.366+0000 7faed1f7db80 1 -- start start 2024-05-27T09:45:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.367+0000 7faed1f7db80 1 Processor -- start 2024-05-27T09:45:29.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.367+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5dd5e40 0x5642c5f5e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.367+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 0x5642c5f5ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.367+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5f5ad30 0x5642c5f5ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.367+0000 7faed1f7db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5642c5d78fa0 con 0x5642c5f5ad30 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.367+0000 7faed1f7db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5642c5e13760 con 0x5642c5f54db0 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.367+0000 7faed1f7db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5642c5f61800 con 0x5642c5dd5e40 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.367+0000 7faed1f7db80 10 client.0 _ll_register_callbacks cb 0x5642c5e1cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.367+0000 7faec0a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5dd5e40 0x5642c5f5e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.367+0000 7faec0a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5dd5e40 0x5642c5f5e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53342/0 (socket says 172.21.15.44:53342) 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.367+0000 7faec0a00700 1 -- 172.21.15.44:0/2442875049 learned_addr learned my addr 172.21.15.44:0/2442875049 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faebbe00700 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 0x5642c5f5ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faec0a00700 1 -- 172.21.15.44:0/2442875049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 msgr2=0x5642c5f5ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faec0a00700 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 0x5642c5f5ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faec1400700 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5f5ad30 0x5642c5f5ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faec0a00700 1 -- 172.21.15.44:0/2442875049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5f5ad30 msgr2=0x5642c5f5ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faec0a00700 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5f5ad30 0x5642c5f5ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faec0a00700 1 -- 172.21.15.44:0/2442875049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642c5dc28d0 con 0x5642c5dd5e40 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faebbe00700 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 0x5642c5f5ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faec0a00700 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5dd5e40 0x5642c5f5e510 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7faeb00a6c50 tx=0x7faeb00a9f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a143376702ca6b61 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faea2a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faea2a00700 1 -- 172.21.15.44:0/2442875049 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb00aa0e0 con 0x5642c5dd5e40 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faea2a00700 1 -- 172.21.15.44:0/2442875049 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7faeb00cc2d0 con 0x5642c5dd5e40 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.368+0000 7faed1f7db80 1 -- 172.21.15.44:0/2442875049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faea80043c0 con 0x5642c5dd5e40 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.369+0000 7faea2a00700 1 -- 172.21.15.44:0/2442875049 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7faeb00a99e0 con 0x5642c5dd5e40 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.369+0000 7faed1f7db80 10 client.27157 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.369+0000 7faed1f7db80 1 -- 172.21.15.44:0/2442875049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5642c5e43060 con 0x5642c5dd5e40 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.369+0000 7faea2a00700 1 -- 172.21.15.44:0/2442875049 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faea80043c0 con 0x5642c5dd5e40 2024-05-27T09:45:29.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.369+0000 7faea2a00700 1 -- 172.21.15.44:0/2442875049 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7faeb00a5350 con 0x5642c5dd5e40 2024-05-27T09:45:29.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.369+0000 7faed1f7db80 10 client.27157 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:29.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.369+0000 7faed1f7db80 10 client.27157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:29.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.369+0000 7faed1f7db80 20 client.27157 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:29.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.369+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5f6e760 0x5642c5f70b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.369+0000 7faed1f7db80 4 client.27157 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:29.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.369+0000 7faed1f7db80 1 -- 172.21.15.44:0/2442875049 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642c5e84d90 con 0x5642c5f6e760 2024-05-27T09:45:29.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.370+0000 7faebbe00700 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5f6e760 0x5642c5f70b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.370+0000 7faebbe00700 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5f6e760 0x5642c5f70b80 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8287d04f3b18321 server_cookie=b46571d88e977850 in_seq=0 out_seq=0 2024-05-27T09:45:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.370+0000 7faea2a00700 10 client.27157 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.371+0000 7faea2a00700 1 -- 172.21.15.44:0/2442875049 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5642c5e84d90 con 0x5642c5f6e760 2024-05-27T09:45:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.371+0000 7faea2a00700 10 client.27157 handle_command_reply: tid=0 2024-05-27T09:45:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.371+0000 7faed1f7db80 1 client.27157 shutdown 2024-05-27T09:45:29.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faed1f7db80 1 -- 172.21.15.44:0/2442875049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5dd5e40 msgr2=0x5642c5f5e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5dd5e40 0x5642c5f5e510 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7faeb00a6c50 tx=0x7faeb00a9f20 comp rx=0 tx=0).stop 2024-05-27T09:45:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faed1f7db80 1 -- 172.21.15.44:0/2442875049 shutdown_connections 2024-05-27T09:45:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5f6e760 0x5642c5f70b80 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5f5ad30 0x5642c5f5ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5dd5e40 0x5642c5f5e510 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2442875049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 0x5642c5f5ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faea2a00700 0 client.27157 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faed1f7db80 1 -- 172.21.15.44:0/2442875049 >> 172.21.15.44:0/2442875049 conn(0x5642c5e35fc0 msgr2=0x5642c5f4f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faed1f7db80 1 -- 172.21.15.44:0/2442875049 shutdown_connections 2024-05-27T09:45:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faed1f7db80 1 -- 172.21.15.44:0/2442875049 wait complete. 2024-05-27T09:45:29.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.372+0000 7faed1f7db80 20 client.27157 trim_cache size 0 max 16384 2024-05-27T09:45:29.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.395+0000 7faed1f7db80 1 Processor -- start 2024-05-27T09:45:29.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.395+0000 7faed1f7db80 1 -- start start 2024-05-27T09:45:29.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5dd5e40 0x5642c5f54870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f54db0 0x5642c5f5d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5f5ad30 0x5642c5f60980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faed1f7db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1b39f0 con 0x5642c5f5ad30 2024-05-27T09:45:29.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faed1f7db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1b1250 con 0x5642c5dd5e40 2024-05-27T09:45:29.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faed1f7db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1af0c0 con 0x5642c5f54db0 2024-05-27T09:45:29.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faebbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f54db0 0x5642c5f5d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faec0a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5dd5e40 0x5642c5f54870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faebbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f54db0 0x5642c5f5d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53346/0 (socket says 172.21.15.44:53346) 2024-05-27T09:45:29.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faec0a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5dd5e40 0x5642c5f54870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:54542/0 (socket says 172.21.15.44:54542) 2024-05-27T09:45:29.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faebbe00700 1 -- 172.21.15.44:0/2464823224 learned_addr learned my addr 172.21.15.44:0/2464823224 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:29.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.396+0000 7faec1400700 1 --2- 172.21.15.44:0/2464823224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5f5ad30 0x5642c5f60980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.397+0000 7faec0a00700 1 -- 172.21.15.44:0/2464823224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f54db0 msgr2=0x5642c5f5d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:29.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.397+0000 7faec0a00700 1 --2- 172.21.15.44:0/2464823224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f54db0 0x5642c5f5d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.397+0000 7faec0a00700 1 -- 172.21.15.44:0/2464823224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5f5ad30 msgr2=0x5642c5f60980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:29.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.397+0000 7faec0a00700 1 --2- 172.21.15.44:0/2464823224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5f5ad30 0x5642c5f60980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.397+0000 7faec0a00700 1 -- 172.21.15.44:0/2464823224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642c5f5f300 con 0x5642c5dd5e40 2024-05-27T09:45:29.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.397+0000 7faec0a00700 1 --2- 172.21.15.44:0/2464823224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5dd5e40 0x5642c5f54870 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7faeb00a5a60 tx=0x7faeb0009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d85c07dfa2b84f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:29.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.397+0000 7faead600700 1 -- 172.21.15.44:0/2464823224 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb00add90 con 0x5642c5dd5e40 2024-05-27T09:45:29.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.397+0000 7faead600700 1 -- 172.21.15.44:0/2464823224 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeb00b2840 con 0x5642c5dd5e40 2024-05-27T09:45:29.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.397+0000 7faead600700 1 -- 172.21.15.44:0/2464823224 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb00a8c40 con 0x5642c5dd5e40 2024-05-27T09:45:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.398+0000 7faed1f7db80 1 -- 172.21.15.44:0/2464823224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5dd5e40 msgr2=0x5642c5f54870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.398+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2464823224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5dd5e40 0x5642c5f54870 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7faeb00a5a60 tx=0x7faeb0009790 comp rx=0 tx=0).stop 2024-05-27T09:45:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.398+0000 7faed1f7db80 1 -- 172.21.15.44:0/2464823224 shutdown_connections 2024-05-27T09:45:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.398+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2464823224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5f5ad30 0x5642c5f60980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.398+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2464823224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5dd5e40 0x5642c5f54870 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.398+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2464823224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f54db0 0x5642c5f5d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.398+0000 7faed1f7db80 1 -- 172.21.15.44:0/2464823224 >> 172.21.15.44:0/2464823224 conn(0x5642c5e35fc0 msgr2=0x5642c5f58880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.398+0000 7faed1f7db80 1 -- 172.21.15.44:0/2464823224 shutdown_connections 2024-05-27T09:45:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.398+0000 7faed1f7db80 1 -- 172.21.15.44:0/2464823224 wait complete. 2024-05-27T09:45:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.399+0000 7faed1f7db80 1 -- start start 2024-05-27T09:45:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.399+0000 7faed1f7db80 1 Processor -- start 2024-05-27T09:45:29.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 0x5642c5f66f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 0x5642c5f759b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faed1f7db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f5ad30 0x5642c5f75ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faed1f7db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1af710 con 0x5642c5dd5e40 2024-05-27T09:45:29.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faed1f7db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1b0b80 con 0x5642c5f54db0 2024-05-27T09:45:29.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faed1f7db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1afde0 con 0x5642c5f5ad30 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faed1f7db80 10 client.0 _ll_register_callbacks cb 0x5642c5e1cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faec0a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 0x5642c5f66f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faec1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f5ad30 0x5642c5f75ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faec1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f5ad30 0x5642c5f75ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:53358/0 (socket says 172.21.15.44:53358) 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faec1400700 1 -- 172.21.15.44:0/2794080318 learned_addr learned my addr 172.21.15.44:0/2794080318 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.400+0000 7faebbe00700 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 0x5642c5f759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faebbe00700 1 -- 172.21.15.44:0/2794080318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f5ad30 msgr2=0x5642c5f75ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faebbe00700 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f5ad30 0x5642c5f75ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faebbe00700 1 -- 172.21.15.44:0/2794080318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 msgr2=0x5642c5f66f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faebbe00700 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 0x5642c5f66f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faebbe00700 1 -- 172.21.15.44:0/2794080318 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642c5f76430 con 0x5642c5f54db0 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faec0a00700 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 0x5642c5f66f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:29.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faec1400700 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f5ad30 0x5642c5f75ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:29.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faebbe00700 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 0x5642c5f759b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7faea8007bd0 tx=0x7faea8010dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c217d3bbcc3f7e47 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:29.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faea2a00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:29.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faea2a00700 1 -- 172.21.15.44:0/2794080318 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8005310 con 0x5642c5f54db0 2024-05-27T09:45:29.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faed1f7db80 1 -- 172.21.15.44:0/2794080318 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faeb00b2840 con 0x5642c5f54db0 2024-05-27T09:45:29.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.401+0000 7faea2a00700 1 -- 172.21.15.44:0/2794080318 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7faea80054b0 con 0x5642c5f54db0 2024-05-27T09:45:29.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.402+0000 7faea2a00700 1 -- 172.21.15.44:0/2794080318 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7faea80056b0 con 0x5642c5f54db0 2024-05-27T09:45:29.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.402+0000 7faed1f7db80 10 client.27179 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:29.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.402+0000 7faed1f7db80 1 -- 172.21.15.44:0/2794080318 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5642c5fd8ed0 con 0x5642c5f54db0 2024-05-27T09:45:29.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.402+0000 7faea2a00700 1 -- 172.21.15.44:0/2794080318 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeb00b2840 con 0x5642c5f54db0 2024-05-27T09:45:29.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.402+0000 7faea2a00700 1 -- 172.21.15.44:0/2794080318 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7faea80276f0 con 0x5642c5f54db0 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.402+0000 7faed1f7db80 10 client.27179 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.402+0000 7faed1f7db80 10 client.27179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.402+0000 7faed1f7db80 20 client.27179 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.402+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5fd9420 0x5642c5fdb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.402+0000 7faed1f7db80 4 client.27179 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.402+0000 7faed1f7db80 1 -- 172.21.15.44:0/2794080318 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5642c5e7e570 con 0x5642c5fd9420 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.403+0000 7faec0a00700 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5fd9420 0x5642c5fdb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.403+0000 7faec0a00700 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5fd9420 0x5642c5fdb840 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8de5a8368978e059 server_cookie=af6c1bec878cd8fe in_seq=0 out_seq=0 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.403+0000 7faea2a00700 10 client.27179 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.403+0000 7faea2a00700 1 -- 172.21.15.44:0/2794080318 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5642c5e7e570 con 0x5642c5fd9420 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.403+0000 7faea2a00700 10 client.27179 handle_command_reply: tid=0 2024-05-27T09:45:29.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 1 client.27179 shutdown 2024-05-27T09:45:29.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 1 -- 172.21.15.44:0/2794080318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 msgr2=0x5642c5f759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:29.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 0x5642c5f759b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7faea8007bd0 tx=0x7faea8010dd0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 1 -- 172.21.15.44:0/2794080318 shutdown_connections 2024-05-27T09:45:29.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5fd9420 0x5642c5fdb840 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5f5ad30 0x5642c5f75ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5dd5e40 0x5642c5f66f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 1 --2- 172.21.15.44:0/2794080318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5f54db0 0x5642c5f759b0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faea2a00700 0 client.27179 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:29.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 1 -- 172.21.15.44:0/2794080318 >> 172.21.15.44:0/2794080318 conn(0x5642c5e35fc0 msgr2=0x5642c5f5e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:29.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 1 -- 172.21.15.44:0/2794080318 shutdown_connections 2024-05-27T09:45:29.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 1 -- 172.21.15.44:0/2794080318 wait complete. 2024-05-27T09:45:29.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.404+0000 7faed1f7db80 20 client.27179 trim_cache size 0 max 16384 2024-05-27T09:45:29.407 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:45:29.407 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:45:29.408 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:45:29.408 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:45:29.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.405+0000 7faec3400700 1 -- 172.21.15.44:0/2190883412 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7faea40b2a70 msgr2=0x7faea40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:29.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.405+0000 7faec3400700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7faea40b2a70 0x7faea40b4f80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009fd0 tx=0x7faeb40093c0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.405+0000 7faec3400700 1 -- 172.21.15.44:0/2190883412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faebc1045e0 msgr2=0x7faebc19d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:29.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.405+0000 7faec3400700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faebc1045e0 0x7faebc19d070 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7faeb000e9f0 tx=0x7faeb000ed00 comp rx=0 tx=0).stop 2024-05-27T09:45:29.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.405+0000 7faec3400700 1 -- 172.21.15.44:0/2190883412 shutdown_connections 2024-05-27T09:45:29.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.405+0000 7faec3400700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7faea40b2a70 0x7faea40b4f80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.405+0000 7faec3400700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc109860 0x7faebc1a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.405+0000 7faec3400700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faebc1045e0 0x7faebc19d070 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.405+0000 7faec3400700 1 --2- 172.21.15.44:0/2190883412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faebc108f10 0x7faebc19d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:29.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.405+0000 7faec3400700 1 -- 172.21.15.44:0/2190883412 >> 172.21.15.44:0/2190883412 conn(0x7faebc100030 msgr2=0x7faebc10a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:29.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.405+0000 7faec3400700 1 -- 172.21.15.44:0/2190883412 shutdown_connections 2024-05-27T09:45:29.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:29.406+0000 7faec3400700 1 -- 172.21.15.44:0/2190883412 wait complete. 2024-05-27T09:45:30.228 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:45:30.228 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:45:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:30 smithi189 ceph-mon[36671]: pgmap v320: 97 pgs: 97 active+clean; 284 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 165 op/s 2024-05-27T09:45:30.344 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:30 smithi044 ceph-mon[31618]: pgmap v320: 97 pgs: 97 active+clean; 284 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 165 op/s 2024-05-27T09:45:30.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:30 smithi050 ceph-mon[36371]: pgmap v320: 97 pgs: 97 active+clean; 284 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 165 op/s 2024-05-27T09:45:30.705 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:32.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:32 smithi044 ceph-mon[31618]: pgmap v321: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-05-27T09:45:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:32 smithi189 ceph-mon[36671]: pgmap v321: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-05-27T09:45:32.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:32 smithi050 ceph-mon[36371]: pgmap v321: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-05-27T09:45:33.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.355+0000 7f5ad8400700 1 -- 172.21.15.44:0/2962482497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad0110ca0 msgr2=0x7f5ad01130c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:33.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.355+0000 7f5ad8400700 1 --2- 172.21.15.44:0/2962482497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad0110ca0 0x7f5ad01130c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800b0a0 tx=0x7f5ac800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.355+0000 7f5ad8400700 1 -- 172.21.15.44:0/2962482497 shutdown_connections 2024-05-27T09:45:33.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.355+0000 7f5ad8400700 1 --2- 172.21.15.44:0/2962482497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad0110ca0 0x7f5ad01130c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.355+0000 7f5ad8400700 1 --2- 172.21.15.44:0/2962482497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ad01030f0 0x7f5ad0103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.355+0000 7f5ad8400700 1 --2- 172.21.15.44:0/2962482497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ad0103a40 0x7f5ad0110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.355+0000 7f5ad8400700 1 -- 172.21.15.44:0/2962482497 >> 172.21.15.44:0/2962482497 conn(0x7f5ad00fedb0 msgr2=0x7f5ad01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:33.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.356+0000 7f5ad8400700 1 -- 172.21.15.44:0/2962482497 shutdown_connections 2024-05-27T09:45:33.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.356+0000 7f5ad8400700 1 -- 172.21.15.44:0/2962482497 wait complete. 2024-05-27T09:45:33.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.357+0000 7f5ad8400700 1 Processor -- start 2024-05-27T09:45:33.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.357+0000 7f5ad8400700 1 -- start start 2024-05-27T09:45:33.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.357+0000 7f5ad8400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad01030f0 0x7f5ad019d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:33.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.357+0000 7f5ad8400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ad0103a40 0x7f5ad019d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:33.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.357+0000 7f5ad8400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ad0110ca0 0x7f5ad01a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:33.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.357+0000 7f5ad8400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad01166b0 con 0x7f5ad0110ca0 2024-05-27T09:45:33.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.357+0000 7f5ad8400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad0116530 con 0x7f5ad0103a40 2024-05-27T09:45:33.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.357+0000 7f5ad8400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad0116830 con 0x7f5ad01030f0 2024-05-27T09:45:33.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.358+0000 7f5ad5000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ad0103a40 0x7f5ad019d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:33.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.358+0000 7f5ad5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad01030f0 0x7f5ad019d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:33.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.358+0000 7f5ad5000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ad0103a40 0x7f5ad019d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39360/0 (socket says 172.21.15.44:39360) 2024-05-27T09:45:33.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.358+0000 7f5ad5000700 1 -- 172.21.15.44:0/2810522930 learned_addr learned my addr 172.21.15.44:0/2810522930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:33.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.358+0000 7f5ad6400700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ad0110ca0 0x7f5ad01a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:33.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.358+0000 7f5ad5a00700 1 -- 172.21.15.44:0/2810522930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ad0103a40 msgr2=0x7f5ad019d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:33.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.358+0000 7f5ad5a00700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ad0103a40 0x7f5ad019d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.358+0000 7f5ad5a00700 1 -- 172.21.15.44:0/2810522930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ad0110ca0 msgr2=0x7f5ad01a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:33.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.358+0000 7f5ad5a00700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ad0110ca0 0x7f5ad01a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.358+0000 7f5ad5a00700 1 -- 172.21.15.44:0/2810522930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad01a1ef0 con 0x7f5ad01030f0 2024-05-27T09:45:33.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.359+0000 7f5ad5a00700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad01030f0 0x7f5ad019d020 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400d950 tx=0x7f5ac400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:33.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.359+0000 7f5ace000700 1 -- 172.21.15.44:0/2810522930 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4016070 con 0x7f5ad01030f0 2024-05-27T09:45:33.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.359+0000 7f5ace000700 1 -- 172.21.15.44:0/2810522930 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac400fd40 con 0x7f5ad01030f0 2024-05-27T09:45:33.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.359+0000 7f5ace000700 1 -- 172.21.15.44:0/2810522930 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac400e3b0 con 0x7f5ad01030f0 2024-05-27T09:45:33.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.359+0000 7f5ad8400700 1 -- 172.21.15.44:0/2810522930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad01a21e0 con 0x7f5ad01030f0 2024-05-27T09:45:33.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.359+0000 7f5ad8400700 1 -- 172.21.15.44:0/2810522930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad01a9b00 con 0x7f5ad01030f0 2024-05-27T09:45:33.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.360+0000 7f5ad8400700 1 -- 172.21.15.44:0/2810522930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad00512b0 con 0x7f5ad01030f0 2024-05-27T09:45:33.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.363+0000 7f5ace000700 1 -- 172.21.15.44:0/2810522930 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5ac4019070 con 0x7f5ad01030f0 2024-05-27T09:45:33.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.364+0000 7f5ace000700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ab80b2a70 0x7f5ab80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:33.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.364+0000 7f5ace000700 1 -- 172.21.15.44:0/2810522930 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f5ac40e1ff0 con 0x7f5ad01030f0 2024-05-27T09:45:33.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.365+0000 7f5ad5000700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ab80b2a70 0x7f5ab80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:33.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.365+0000 7f5ad5000700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ab80b2a70 0x7f5ab80b4f80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5ad010a6f0 tx=0x7f5abc00b4a0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:33.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.368+0000 7f5ace000700 1 -- 172.21.15.44:0/2810522930 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5ac40a9990 con 0x7f5ad01030f0 2024-05-27T09:45:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.653+0000 7f5ad8400700 1 -- 172.21.15.44:0/2810522930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ad019e380 con 0x7f5ad01030f0 2024-05-27T09:45:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.655+0000 7f5ace000700 1 -- 172.21.15.44:0/2810522930 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11866 (secure 0 0 0) 0x7f5ac40a9290 con 0x7f5ad01030f0 2024-05-27T09:45:33.656 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:45:33.656 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:44:44.299923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:45:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 -- 172.21.15.44:0/2810522930 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ab80b2a70 msgr2=0x7f5ab80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ab80b2a70 0x7f5ab80b4f80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5ad010a6f0 tx=0x7f5abc00b4a0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 -- 172.21.15.44:0/2810522930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad01030f0 msgr2=0x7f5ad019d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad01030f0 0x7f5ad019d020 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400d950 tx=0x7f5ac400dc60 comp rx=0 tx=0).stop 2024-05-27T09:45:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 -- 172.21.15.44:0/2810522930 shutdown_connections 2024-05-27T09:45:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5ab80b2a70 0x7f5ab80b4f80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ad0110ca0 0x7f5ad01a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad01030f0 0x7f5ad019d020 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 --2- 172.21.15.44:0/2810522930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ad0103a40 0x7f5ad019d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 -- 172.21.15.44:0/2810522930 >> 172.21.15.44:0/2810522930 conn(0x7f5ad00fedb0 msgr2=0x7f5ad01018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 -- 172.21.15.44:0/2810522930 shutdown_connections 2024-05-27T09:45:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.658+0000 7f5ad8400700 1 -- 172.21.15.44:0/2810522930 wait complete. 2024-05-27T09:45:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 33 2024-05-27T09:45:33.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.843+0000 7f7c44c00700 1 -- 172.21.15.44:0/2983984268 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x558b8705e8b0 con 0x558b870f98a0 2024-05-27T09:45:33.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.843+0000 7f7c44c00700 10 client.27127 handle_command_reply: tid=0 2024-05-27T09:45:33.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.843+0000 7f7c70409b80 1 client.27127 shutdown 2024-05-27T09:45:33.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.844+0000 7f7c70409b80 1 -- 172.21.15.44:0/2983984268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86eebfe0 msgr2=0x558b8708a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:33.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.844+0000 7f7c70409b80 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86eebfe0 0x558b8708a030 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x558b87090380 tx=0x7f7c4c004750 comp rx=0 tx=0).stop 2024-05-27T09:45:33.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.844+0000 7f7c70409b80 1 -- 172.21.15.44:0/2983984268 shutdown_connections 2024-05-27T09:45:33.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.844+0000 7f7c70409b80 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558b870f98a0 0x558b870fbcc0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.844+0000 7f7c70409b80 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558b8706b680 0x558b870889c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.844+0000 7f7c70409b80 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b86eebfe0 0x558b8708a030 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.844+0000 7f7c70409b80 1 --2- 172.21.15.44:0/2983984268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558b86fd7b90 0x558b870883c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.844+0000 7f7c70409b80 1 -- 172.21.15.44:0/2983984268 >> 172.21.15.44:0/2983984268 conn(0x558b86f85530 msgr2=0x558b8707d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.844+0000 7f7c44c00700 0 client.27127 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.844+0000 7f7c70409b80 1 -- 172.21.15.44:0/2983984268 shutdown_connections 2024-05-27T09:45:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.844+0000 7f7c70409b80 1 -- 172.21.15.44:0/2983984268 wait complete. 2024-05-27T09:45:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.845+0000 7f7c70409b80 20 client.27127 trim_cache size 0 max 16384 2024-05-27T09:45:33.847 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:45:33.847 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 33, 2024-05-27T09:45:33.847 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:45:33.847 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 23, 2024-05-27T09:45:33.847 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:45:33.847 INFO:teuthology.orchestra.run.smithi044.stdout: "2a69c329": { 2024-05-27T09:45:33.847 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 23, 2024-05-27T09:45:33.847 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:45:33.847 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:45:33.847 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 138.9, 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:33.848 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:33.849 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:33.849 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:45:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 -- 172.21.15.44:0/330645355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c400b2b40 msgr2=0x7f7c400b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c400b2b40 0x7f7c400b5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00af60 tx=0x7f7c4c00a2b0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 -- 172.21.15.44:0/330645355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5c110640 msgr2=0x7f7c5c1abaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5c110640 0x7f7c5c1abaa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7c48000c00 tx=0x7f7c48005f10 comp rx=0 tx=0).stop 2024-05-27T09:45:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 -- 172.21.15.44:0/330645355 shutdown_connections 2024-05-27T09:45:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7c400b2b40 0x7f7c400b5050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c5c110f90 0x7f7c5c1a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c5c10c820 0x7f7c5c1ab560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 --2- 172.21.15.44:0/330645355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5c110640 0x7f7c5c1abaa0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:33.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 -- 172.21.15.44:0/330645355 >> 172.21.15.44:0/330645355 conn(0x7f7c5c06f980 msgr2=0x7f7c5c10b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:33.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 -- 172.21.15.44:0/330645355 shutdown_connections 2024-05-27T09:45:33.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:33.846+0000 7f7c61800700 1 -- 172.21.15.44:0/330645355 wait complete. 2024-05-27T09:45:34.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:34 smithi189 ceph-mon[36671]: pgmap v322: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-05-27T09:45:34.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:34 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2810522930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:34.344 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:34 smithi044 ceph-mon[31618]: pgmap v322: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-05-27T09:45:34.344 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:34 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2810522930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:34.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:34 smithi050 ceph-mon[36371]: pgmap v322: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-05-27T09:45:34.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:34 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2810522930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:34.695 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:45:35.046 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '2a69c329', quiesce took 17.4 seconds. Will release after: 1.5500000000000007 2024-05-27T09:45:35.171 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:36.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:36 smithi189 ceph-mon[36671]: pgmap v323: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-05-27T09:45:36.341 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:36 smithi044 ceph-mon[31618]: pgmap v323: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-05-27T09:45:36.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:36 smithi050 ceph-mon[36371]: pgmap v323: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-05-27T09:45:36.596 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:45:37.354 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:37.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.414+0000 7f75c5e00700 1 -- 172.21.15.44:0/3027417544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c0108f10 msgr2=0x7f75c0109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:37.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.414+0000 7f75c5e00700 1 --2- 172.21.15.44:0/3027417544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c0108f10 0x7f75c0109320 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009920 tx=0x7f75a8009c30 comp rx=0 tx=0).stop 2024-05-27T09:45:37.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.415+0000 7f75c5e00700 1 -- 172.21.15.44:0/3027417544 shutdown_connections 2024-05-27T09:45:37.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.415+0000 7f75c5e00700 1 --2- 172.21.15.44:0/3027417544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0109860 0x7f75c01111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.415+0000 7f75c5e00700 1 --2- 172.21.15.44:0/3027417544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c01045e0 0x7f75c01049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.415+0000 7f75c5e00700 1 --2- 172.21.15.44:0/3027417544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c0108f10 0x7f75c0109320 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.415+0000 7f75c5e00700 1 -- 172.21.15.44:0/3027417544 >> 172.21.15.44:0/3027417544 conn(0x7f75c0100030 msgr2=0x7f75c0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:37.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.415+0000 7f75c5e00700 1 -- 172.21.15.44:0/3027417544 shutdown_connections 2024-05-27T09:45:37.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.415+0000 7f75c5e00700 1 -- 172.21.15.44:0/3027417544 wait complete. 2024-05-27T09:45:37.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.416+0000 7f75c5e00700 1 Processor -- start 2024-05-27T09:45:37.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.416+0000 7f75c5e00700 1 -- start start 2024-05-27T09:45:37.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.417+0000 7f75c5e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c01045e0 0x7f75c01a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.417+0000 7f75c5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c0108f10 0x7f75c01a2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.417+0000 7f75c5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c0109860 0x7f75c01ad220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.417+0000 7f75c5e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01144c0 con 0x7f75c0108f10 2024-05-27T09:45:37.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.417+0000 7f75c5e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f75c0114340 con 0x7f75c01045e0 2024-05-27T09:45:37.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.417+0000 7f75c5e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75c0114640 con 0x7f75c0109860 2024-05-27T09:45:37.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.417+0000 7f75b6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c0108f10 0x7f75c01a2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.417+0000 7f75b6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c0108f10 0x7f75c01a2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58910/0 (socket says 172.21.15.44:58910) 2024-05-27T09:45:37.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.417+0000 7f75b6000700 1 -- 172.21.15.44:0/947808310 learned_addr learned my addr 172.21.15.44:0/947808310 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:37.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.417+0000 7f75bea00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c01045e0 0x7f75c01a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.417+0000 7f75bf400700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c0109860 0x7f75c01ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.418+0000 7f75bea00700 1 -- 172.21.15.44:0/947808310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c0109860 msgr2=0x7f75c01ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:37.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.418+0000 7f75bea00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c0109860 0x7f75c01ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.418+0000 7f75bea00700 1 -- 172.21.15.44:0/947808310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c0108f10 msgr2=0x7f75c01a2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:37.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.418+0000 7f75bea00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c0108f10 0x7f75c01a2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.418+0000 7f75bea00700 1 -- 172.21.15.44:0/947808310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c01ad760 con 0x7f75c01045e0 2024-05-27T09:45:37.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.418+0000 7f75bea00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c01045e0 0x7f75c01a21a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f75b000eb90 tx=0x7f75b000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:37.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.419+0000 7f75c4a00700 1 -- 172.21.15.44:0/947808310 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b000cc30 con 0x7f75c01045e0 2024-05-27T09:45:37.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.419+0000 7f75c4a00700 1 -- 172.21.15.44:0/947808310 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75b00044d0 con 0x7f75c01045e0 2024-05-27T09:45:37.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.419+0000 7f75c5e00700 1 -- 172.21.15.44:0/947808310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c01ad9f0 con 0x7f75c01045e0 2024-05-27T09:45:37.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.419+0000 7f75c4a00700 1 -- 172.21.15.44:0/947808310 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0010430 con 0x7f75c01045e0 2024-05-27T09:45:37.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.419+0000 7f75c5e00700 1 -- 172.21.15.44:0/947808310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c01adf50 con 0x7f75c01045e0 2024-05-27T09:45:37.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.423+0000 7f75c4a00700 1 -- 172.21.15.44:0/947808310 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f75b00105d0 con 0x7f75c01045e0 2024-05-27T09:45:37.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.424+0000 7f75d49b9b80 1 Processor -- start 2024-05-27T09:45:37.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.424+0000 7f75d49b9b80 1 -- start start 2024-05-27T09:45:37.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.424+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfde330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.424+0000 7f75c4a00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75a40b2a70 0x7f75a40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.424+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436de5fe40 0x56436dfde870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.425+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436dfdee70 0x56436dfeb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.425+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56436df0ce40 con 0x56436df4b9f0 2024-05-27T09:45:37.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.425+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56436defc9e0 con 0x56436dfdee70 2024-05-27T09:45:37.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.425+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56436df00a20 con 0x56436de5fe40 2024-05-27T09:45:37.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.425+0000 7f75c4a00700 1 -- 172.21.15.44:0/947808310 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f75b00e28d0 con 0x7f75c01045e0 2024-05-27T09:45:37.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.425+0000 7f75bf400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfde330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.425+0000 7f75b6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436de5fe40 0x56436dfde870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.425+0000 7f75bea00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75a40b2a70 0x7f75a40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.425+0000 7f75bf400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfde330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58912/0 (socket says 172.21.15.44:58912) 2024-05-27T09:45:37.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.425+0000 7f75bf400700 1 -- 172.21.15.44:0/1653835412 learned_addr learned my addr 172.21.15.44:0/1653835412 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:37.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.426+0000 7f75bf400700 1 -- 172.21.15.44:0/1653835412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436de5fe40 msgr2=0x56436dfde870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:37.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.426+0000 7f75bf400700 1 --2- 172.21.15.44:0/1653835412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436de5fe40 0x56436dfde870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.426+0000 7f75bf400700 1 -- 172.21.15.44:0/1653835412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436dfdee70 msgr2=0x56436dfeb110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:45:37.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.426+0000 7f75bf400700 1 --2- 172.21.15.44:0/1653835412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436dfdee70 0x56436dfeb110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.426+0000 7f75bf400700 1 -- 172.21.15.44:0/1653835412 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56436dec5420 con 0x56436df4b9f0 2024-05-27T09:45:37.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.426+0000 7f75b6000700 1 --2- 172.21.15.44:0/1653835412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436de5fe40 0x56436dfde870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:45:37.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.427+0000 7f75bea00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75a40b2a70 0x7f75a40b4f80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f75b000c5b0 tx=0x7f75b0014000 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:37.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.427+0000 7f75bf400700 1 --2- 172.21.15.44:0/1653835412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfde330 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f75b8011890 tx=0x7f75b8009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9cddaba6c726b5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:37.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.427+0000 7f75b4c00700 1 -- 172.21.15.44:0/1653835412 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8003a00 con 0x56436df4b9f0 2024-05-27T09:45:37.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.427+0000 7f75b4c00700 1 -- 172.21.15.44:0/1653835412 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75b801a070 con 0x56436df4b9f0 2024-05-27T09:45:37.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.427+0000 7f75b4c00700 1 -- 172.21.15.44:0/1653835412 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8007760 con 0x56436df4b9f0 2024-05-27T09:45:37.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.429+0000 7f75d49b9b80 1 -- 172.21.15.44:0/1653835412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 msgr2=0x56436dfde330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:37.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.429+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/1653835412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfde330 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f75b8011890 tx=0x7f75b8009920 comp rx=0 tx=0).stop 2024-05-27T09:45:37.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.429+0000 7f75d49b9b80 1 -- 172.21.15.44:0/1653835412 shutdown_connections 2024-05-27T09:45:37.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.429+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/1653835412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436dfdee70 0x56436dfeb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.429+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/1653835412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfde330 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.429+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/1653835412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436de5fe40 0x56436dfde870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.429+0000 7f75d49b9b80 1 -- 172.21.15.44:0/1653835412 >> 172.21.15.44:0/1653835412 conn(0x56436def9390 msgr2=0x56436dfda4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:37.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.429+0000 7f75d49b9b80 1 -- 172.21.15.44:0/1653835412 shutdown_connections 2024-05-27T09:45:37.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.429+0000 7f75d49b9b80 1 -- 172.21.15.44:0/1653835412 wait complete. 2024-05-27T09:45:37.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.430+0000 7f75d49b9b80 1 -- start start 2024-05-27T09:45:37.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.430+0000 7f75d49b9b80 1 Processor -- start 2024-05-27T09:45:37.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.431+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 0x56436dfe8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.431+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 0x56436dfe8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.431+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dfe90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.431+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56436de02fa0 con 0x56436de5fe40 2024-05-27T09:45:37.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.431+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56436de9d760 con 0x56436df4b9f0 2024-05-27T09:45:37.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.431+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56436dfebb20 con 0x56436dfdee70 2024-05-27T09:45:37.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.431+0000 7f75d49b9b80 10 client.0 _ll_register_callbacks cb 0x56436dea6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.431+0000 7f75b6000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 0x56436dfe8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.431+0000 7f75b6000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 0x56436dfe8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39390/0 (socket says 172.21.15.44:39390) 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.431+0000 7f75b6000700 1 -- 172.21.15.44:0/64761629 learned_addr learned my addr 172.21.15.44:0/64761629 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.431+0000 7f75bf400700 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dfe90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.432+0000 7f75b6000700 1 -- 172.21.15.44:0/64761629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 msgr2=0x56436dfe90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.432+0000 7f75bf400700 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 0x56436dfe8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.432+0000 7f75b6000700 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dfe90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.432+0000 7f75b6000700 1 -- 172.21.15.44:0/64761629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 msgr2=0x56436dfe8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.432+0000 7f75b6000700 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 0x56436dfe8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.432+0000 7f75b6000700 1 -- 172.21.15.44:0/64761629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56436de4c8d0 con 0x56436df4b9f0 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.432+0000 7f75bf400700 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 0x56436dfe8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:45:37.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.432+0000 7f75b6000700 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 0x56436dfe8b90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f75a8000c00 tx=0x7f75a800f670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68b76d3d21ed7e4a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:37.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.432+0000 7f75ad600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:37.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.432+0000 7f75ad600700 1 -- 172.21.15.44:0/64761629 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8005040 con 0x56436df4b9f0 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.433+0000 7f75ad600700 1 -- 172.21.15.44:0/64761629 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f75a80051e0 con 0x56436df4b9f0 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.433+0000 7f75d49b9b80 1 -- 172.21.15.44:0/64761629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75b801a070 con 0x56436df4b9f0 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.433+0000 7f75ad600700 1 -- 172.21.15.44:0/64761629 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f75a8030050 con 0x56436df4b9f0 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.433+0000 7f75d49b9b80 10 client.27194 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.433+0000 7f75d49b9b80 1 -- 172.21.15.44:0/64761629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56436decd060 con 0x56436df4b9f0 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.433+0000 7f75ad600700 1 -- 172.21.15.44:0/64761629 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75b801a070 con 0x56436df4b9f0 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.433+0000 7f75ad600700 1 -- 172.21.15.44:0/64761629 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f75a8003c50 con 0x56436df4b9f0 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.434+0000 7f75d49b9b80 10 client.27194 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.434+0000 7f75d49b9b80 10 client.27194 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.434+0000 7f75d49b9b80 20 client.27194 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.434+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56436dff8af0 0x56436dffaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.434+0000 7f75d49b9b80 4 client.27194 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:37.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.434+0000 7f75d49b9b80 1 -- 172.21.15.44:0/64761629 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56436df0ed90 con 0x56436dff8af0 2024-05-27T09:45:37.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.434+0000 7f75bf400700 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56436dff8af0 0x56436dffaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.435+0000 7f75bf400700 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56436dff8af0 0x56436dffaf50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3687603f0962ca8 server_cookie=76c672d66222b795 in_seq=0 out_seq=0 2024-05-27T09:45:37.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.435+0000 7f75ad600700 10 client.27194 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:45:37.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75ad600700 1 -- 172.21.15.44:0/64761629 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56436df0ed90 con 0x56436dff8af0 2024-05-27T09:45:37.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75ad600700 10 client.27194 handle_command_reply: tid=0 2024-05-27T09:45:37.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75d49b9b80 1 client.27194 shutdown 2024-05-27T09:45:37.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75d49b9b80 1 -- 172.21.15.44:0/64761629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 msgr2=0x56436dfe8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:37.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 0x56436dfe8b90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f75a8000c00 tx=0x7f75a800f670 comp rx=0 tx=0).stop 2024-05-27T09:45:37.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75d49b9b80 1 -- 172.21.15.44:0/64761629 shutdown_connections 2024-05-27T09:45:37.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dfe90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56436dff8af0 0x56436dffaf50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 0x56436dfe8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/64761629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 0x56436dfe8b90 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75d49b9b80 1 -- 172.21.15.44:0/64761629 >> 172.21.15.44:0/64761629 conn(0x56436def9390 msgr2=0x56436dfde310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:37.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75ad600700 0 client.27194 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:45:37.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.436+0000 7f75d49b9b80 1 -- 172.21.15.44:0/64761629 shutdown_connections 2024-05-27T09:45:37.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.437+0000 7f75d49b9b80 1 -- 172.21.15.44:0/64761629 wait complete. 2024-05-27T09:45:37.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.437+0000 7f75d49b9b80 20 client.27194 trim_cache size 0 max 16384 2024-05-27T09:45:37.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.453+0000 7f75d49b9b80 1 Processor -- start 2024-05-27T09:45:37.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.453+0000 7f75d49b9b80 1 -- start start 2024-05-27T09:45:37.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.454+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436de5fe40 0x56436dfe3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.454+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfda4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.454+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dff4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.454+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01b7d70 con 0x56436df4b9f0 2024-05-27T09:45:37.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.454+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01b55d0 con 0x56436de5fe40 2024-05-27T09:45:37.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.454+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01b3440 con 0x56436dfdee70 2024-05-27T09:45:37.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.454+0000 7f75b6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfda4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.454+0000 7f75b6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfda4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58934/0 (socket says 172.21.15.44:58934) 2024-05-27T09:45:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.454+0000 7f75b6000700 1 -- 172.21.15.44:0/1553813987 learned_addr learned my addr 172.21.15.44:0/1553813987 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.454+0000 7f75bf400700 1 --2- 172.21.15.44:0/1553813987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dff4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.454+0000 7f75bf400700 1 --2- 172.21.15.44:0/1553813987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436de5fe40 0x56436dfe3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.455+0000 7f75b6000700 1 -- 172.21.15.44:0/1553813987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 msgr2=0x56436dff4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.455+0000 7f75b6000700 1 --2- 172.21.15.44:0/1553813987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dff4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.455+0000 7f75b6000700 1 -- 172.21.15.44:0/1553813987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436de5fe40 msgr2=0x56436dfe3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.455+0000 7f75b6000700 1 --2- 172.21.15.44:0/1553813987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436de5fe40 0x56436dfe3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.455+0000 7f75b6000700 1 -- 172.21.15.44:0/1553813987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56436dfe6bc0 con 0x56436df4b9f0 2024-05-27T09:45:37.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.455+0000 7f75bf400700 1 --2- 172.21.15.44:0/1553813987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dff4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:37.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.455+0000 7f75b6000700 1 --2- 172.21.15.44:0/1553813987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfda4f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f75a8005a70 tx=0x7f75a8005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfef902e120a4b03 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:37.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.455+0000 7f75afe00700 1 -- 172.21.15.44:0/1553813987 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a803a070 con 0x56436df4b9f0 2024-05-27T09:45:37.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.456+0000 7f75afe00700 1 -- 172.21.15.44:0/1553813987 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75a800bc70 con 0x56436df4b9f0 2024-05-27T09:45:37.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.456+0000 7f75afe00700 1 -- 172.21.15.44:0/1553813987 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a80038e0 con 0x56436df4b9f0 2024-05-27T09:45:37.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.456+0000 7f75d49b9b80 1 -- 172.21.15.44:0/1553813987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 msgr2=0x56436dfda4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:37.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.456+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/1553813987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfda4f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f75a8005a70 tx=0x7f75a8005ea0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.456+0000 7f75d49b9b80 1 -- 172.21.15.44:0/1553813987 shutdown_connections 2024-05-27T09:45:37.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.456+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/1553813987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dff4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.456+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/1553813987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436de5fe40 0x56436dfe3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.456+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/1553813987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436df4b9f0 0x56436dfda4f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.456+0000 7f75d49b9b80 1 -- 172.21.15.44:0/1553813987 >> 172.21.15.44:0/1553813987 conn(0x56436def9390 msgr2=0x56436dfe9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:37.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.457+0000 7f75d49b9b80 1 -- 172.21.15.44:0/1553813987 shutdown_connections 2024-05-27T09:45:37.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.457+0000 7f75d49b9b80 1 -- 172.21.15.44:0/1553813987 wait complete. 2024-05-27T09:45:37.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.457+0000 7f75d49b9b80 1 -- start start 2024-05-27T09:45:37.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.457+0000 7f75d49b9b80 1 Processor -- start 2024-05-27T09:45:37.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.458+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 0x56436dff1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.458+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 0x56436dff1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.458+0000 7f75d49b9b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dfe2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.458+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01b3a90 con 0x56436de5fe40 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.458+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01b4f00 con 0x56436df4b9f0 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.458+0000 7f75d49b9b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01b4160 con 0x56436dfdee70 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.458+0000 7f75d49b9b80 10 client.0 _ll_register_callbacks cb 0x56436dea6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.458+0000 7f75bf400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 0x56436dff1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.458+0000 7f75b6000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 0x56436dff1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.458+0000 7f75bf400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 0x56436dff1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58946/0 (socket says 172.21.15.44:58946) 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.458+0000 7f75bf400700 1 -- 172.21.15.44:0/3016887767 learned_addr learned my addr 172.21.15.44:0/3016887767 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.459+0000 7f75bf400700 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dfe2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.459+0000 7f75b6000700 1 -- 172.21.15.44:0/3016887767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 msgr2=0x56436dfe2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.459+0000 7f75b6000700 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dfe2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.459+0000 7f75b6000700 1 -- 172.21.15.44:0/3016887767 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 msgr2=0x56436dff1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.459+0000 7f75b6000700 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 0x56436dff1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.459+0000 7f75b6000700 1 -- 172.21.15.44:0/3016887767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56436dfe3290 con 0x56436df4b9f0 2024-05-27T09:45:37.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.459+0000 7f75bf400700 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 0x56436dff1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:37.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.459+0000 7f75b6000700 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 0x56436dff1640 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009840 tx=0x7f75a800bbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85295745a59e1eb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:37.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.459+0000 7f75ad600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:37.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.459+0000 7f75ad600700 1 -- 172.21.15.44:0/3016887767 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a803a040 con 0x56436df4b9f0 2024-05-27T09:45:37.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.459+0000 7f75d49b9b80 1 -- 172.21.15.44:0/3016887767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75a800bc70 con 0x56436df4b9f0 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.460+0000 7f75ad600700 1 -- 172.21.15.44:0/3016887767 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f75a80050d0 con 0x56436df4b9f0 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.460+0000 7f75ad600700 1 -- 172.21.15.44:0/3016887767 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f75a8005880 con 0x56436df4b9f0 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.460+0000 7f75d49b9b80 10 client.27200 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.460+0000 7f75d49b9b80 1 -- 172.21.15.44:0/3016887767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56436dff0960 con 0x56436df4b9f0 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.460+0000 7f75ad600700 1 -- 172.21.15.44:0/3016887767 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75a800bc70 con 0x56436df4b9f0 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.461+0000 7f75ad600700 1 -- 172.21.15.44:0/3016887767 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f75a8048480 con 0x56436df4b9f0 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.461+0000 7f75d49b9b80 10 client.27200 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.461+0000 7f75d49b9b80 10 client.27200 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.461+0000 7f75d49b9b80 20 client.27200 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.461+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56436e063760 0x56436e065b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.461+0000 7f75d49b9b80 4 client.27200 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.461+0000 7f75d49b9b80 1 -- 172.21.15.44:0/3016887767 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56436df08570 con 0x56436e063760 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.461+0000 7f75bf400700 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56436e063760 0x56436e065b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:37.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75bf400700 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56436e063760 0x56436e065b80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7255bba46bcf2893 server_cookie=d63f6c29e2b22e65 in_seq=0 out_seq=0 2024-05-27T09:45:37.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75ad600700 10 client.27200 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:45:37.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75ad600700 1 -- 172.21.15.44:0/3016887767 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56436df08570 con 0x56436e063760 2024-05-27T09:45:37.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75ad600700 10 client.27200 handle_command_reply: tid=0 2024-05-27T09:45:37.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75d49b9b80 1 client.27200 shutdown 2024-05-27T09:45:37.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75d49b9b80 1 -- 172.21.15.44:0/3016887767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 msgr2=0x56436dff1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:37.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 0x56436dff1640 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009840 tx=0x7f75a800bbd0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75d49b9b80 1 -- 172.21.15.44:0/3016887767 shutdown_connections 2024-05-27T09:45:37.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56436dfdee70 0x56436dfe2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x56436e063760 0x56436e065b80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56436de5fe40 0x56436dff1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75d49b9b80 1 --2- 172.21.15.44:0/3016887767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56436df4b9f0 0x56436dff1640 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75ad600700 0 client.27200 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:45:37.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75d49b9b80 1 -- 172.21.15.44:0/3016887767 >> 172.21.15.44:0/3016887767 conn(0x56436def9390 msgr2=0x56436dfdfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:37.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.462+0000 7f75d49b9b80 1 -- 172.21.15.44:0/3016887767 shutdown_connections 2024-05-27T09:45:37.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75d49b9b80 1 -- 172.21.15.44:0/3016887767 wait complete. 2024-05-27T09:45:37.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75d49b9b80 20 client.27200 trim_cache size 0 max 16384 2024-05-27T09:45:37.466 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:45:37.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75c5e00700 1 -- 172.21.15.44:0/947808310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75a40b2a70 msgr2=0x7f75a40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:37.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75c5e00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75a40b2a70 0x7f75a40b4f80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f75b000c5b0 tx=0x7f75b0014000 comp rx=0 tx=0).stop 2024-05-27T09:45:37.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75c5e00700 1 -- 172.21.15.44:0/947808310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c01045e0 msgr2=0x7f75c01a21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:37.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75c5e00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c01045e0 0x7f75c01a21a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f75b000eb90 tx=0x7f75b000eea0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75c5e00700 1 -- 172.21.15.44:0/947808310 shutdown_connections 2024-05-27T09:45:37.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75c5e00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f75a40b2a70 0x7f75a40b4f80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75c5e00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c0109860 0x7f75c01ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75c5e00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c01045e0 0x7f75c01a21a0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75c5e00700 1 --2- 172.21.15.44:0/947808310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75c0108f10 0x7f75c01a2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:37.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.463+0000 7f75c5e00700 1 -- 172.21.15.44:0/947808310 >> 172.21.15.44:0/947808310 conn(0x7f75c0100030 msgr2=0x7f75c010a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:37.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.464+0000 7f75c5e00700 1 -- 172.21.15.44:0/947808310 shutdown_connections 2024-05-27T09:45:37.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:37.464+0000 7f75c5e00700 1 -- 172.21.15.44:0/947808310 wait complete. 2024-05-27T09:45:38.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:38 smithi044 ceph-mon[31618]: pgmap v324: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-05-27T09:45:38.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:38 smithi050 ceph-mon[36371]: pgmap v324: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-05-27T09:45:38.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:38 smithi189 ceph-mon[36671]: pgmap v324: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-05-27T09:45:39.436 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:45:40.123 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:40.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:40 smithi044 ceph-mon[31618]: pgmap v325: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 325 KiB/s wr, 59 op/s 2024-05-27T09:45:40.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.208+0000 7f6e66e00700 1 -- 172.21.15.44:0/2431032620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e60108f10 msgr2=0x7f6e60109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.208+0000 7f6e66e00700 1 --2- 172.21.15.44:0/2431032620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e60108f10 0x7f6e60109320 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6e480099c0 tx=0x7f6e48009cd0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.208+0000 7f6e66e00700 1 -- 172.21.15.44:0/2431032620 shutdown_connections 2024-05-27T09:45:40.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.208+0000 7f6e66e00700 1 --2- 172.21.15.44:0/2431032620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e60109860 0x7f6e601111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.208+0000 7f6e66e00700 1 --2- 172.21.15.44:0/2431032620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e601045e0 0x7f6e601049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.208+0000 7f6e66e00700 1 --2- 172.21.15.44:0/2431032620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e60108f10 0x7f6e60109320 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.208+0000 7f6e66e00700 1 -- 172.21.15.44:0/2431032620 >> 172.21.15.44:0/2431032620 conn(0x7f6e60100030 msgr2=0x7f6e60102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:40.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.208+0000 7f6e66e00700 1 -- 172.21.15.44:0/2431032620 shutdown_connections 2024-05-27T09:45:40.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.208+0000 7f6e66e00700 1 -- 172.21.15.44:0/2431032620 wait complete. 2024-05-27T09:45:40.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.210+0000 7f6e66e00700 1 Processor -- start 2024-05-27T09:45:40.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.210+0000 7f6e66e00700 1 -- start start 2024-05-27T09:45:40.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.210+0000 7f6e66e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e601045e0 0x7f6e60075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.210+0000 7f6e66e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e60108f10 0x7f6e60075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.210+0000 7f6e66e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e60109860 0x7f6e60076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.210+0000 7f6e66e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6e601144c0 con 0x7f6e601045e0 2024-05-27T09:45:40.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.210+0000 7f6e66e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6e60114340 con 0x7f6e60109860 2024-05-27T09:45:40.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.210+0000 7f6e66e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6e60114640 con 0x7f6e60108f10 2024-05-27T09:45:40.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e57400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e60108f10 0x7f6e60075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e5fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e601045e0 0x7f6e60075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e57400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e60108f10 0x7f6e60075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:57600/0 (socket says 172.21.15.44:57600) 2024-05-27T09:45:40.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e5fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e601045e0 0x7f6e60075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58964/0 (socket says 172.21.15.44:58964) 2024-05-27T09:45:40.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e57400700 1 -- 172.21.15.44:0/3626230984 learned_addr learned my addr 172.21.15.44:0/3626230984 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:40.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e64e00700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e60109860 0x7f6e60076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e57400700 1 -- 172.21.15.44:0/3626230984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e60109860 msgr2=0x7f6e60076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e57400700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e60109860 0x7f6e60076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e57400700 1 -- 172.21.15.44:0/3626230984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e601045e0 msgr2=0x7f6e60075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e57400700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e601045e0 0x7f6e60075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e57400700 1 -- 172.21.15.44:0/3626230984 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e601a9210 con 0x7f6e60108f10 2024-05-27T09:45:40.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e64e00700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e60109860 0x7f6e60076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:40.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.211+0000 7f6e5fe00700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e601045e0 0x7f6e60075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:40.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.212+0000 7f6e57400700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e60108f10 0x7f6e60075ff0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6e48005f50 tx=0x7f6e48005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:40.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.212+0000 7f6e5d600700 1 -- 172.21.15.44:0/3626230984 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e48026070 con 0x7f6e60108f10 2024-05-27T09:45:40.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.212+0000 7f6e66e00700 1 -- 172.21.15.44:0/3626230984 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e601a94a0 con 0x7f6e60108f10 2024-05-27T09:45:40.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.212+0000 7f6e5d600700 1 -- 172.21.15.44:0/3626230984 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e48003c70 con 0x7f6e60108f10 2024-05-27T09:45:40.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.212+0000 7f6e5d600700 1 -- 172.21.15.44:0/3626230984 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4800f960 con 0x7f6e60108f10 2024-05-27T09:45:40.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.212+0000 7f6e66e00700 1 -- 172.21.15.44:0/3626230984 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e601a9950 con 0x7f6e60108f10 2024-05-27T09:45:40.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.213+0000 7f6e66e00700 1 -- 172.21.15.44:0/3626230984 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e600512b0 con 0x7f6e60108f10 2024-05-27T09:45:40.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.216+0000 7f6e5d600700 1 -- 172.21.15.44:0/3626230984 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6e48029070 con 0x7f6e60108f10 2024-05-27T09:45:40.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.218+0000 7f6e5d600700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6e440b2a70 0x7f6e440b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.218+0000 7f6e5d600700 1 -- 172.21.15.44:0/3626230984 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f6e480e3550 con 0x7f6e60108f10 2024-05-27T09:45:40.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.218+0000 7f6e5fe00700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6e440b2a70 0x7f6e440b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.219+0000 7f6e5fe00700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6e440b2a70 0x7f6e440b4f80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6e50011490 tx=0x7f6e50009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:40.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.219+0000 7f6e5d600700 1 -- 172.21.15.44:0/3626230984 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6e48030080 con 0x7f6e60108f10 2024-05-27T09:45:40.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.540+0000 7f6e66e00700 1 -- 172.21.15.44:0/3626230984 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e6002e5e0 con 0x7f6e60108f10 2024-05-27T09:45:40.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.541+0000 7f6e5d600700 1 -- 172.21.15.44:0/3626230984 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11866 (secure 0 0 0) 0x7f6e48013070 con 0x7f6e60108f10 2024-05-27T09:45:40.542 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:45:40.542 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:44:44.299923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24535]},"id":1}]} 2024-05-27T09:45:40.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:40 smithi050 ceph-mon[36371]: pgmap v325: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 325 KiB/s wr, 59 op/s 2024-05-27T09:45:40.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.544+0000 7f6e66e00700 1 -- 172.21.15.44:0/3626230984 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6e440b2a70 msgr2=0x7f6e440b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.544+0000 7f6e66e00700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6e440b2a70 0x7f6e440b4f80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6e50011490 tx=0x7f6e50009260 comp rx=0 tx=0).stop 2024-05-27T09:45:40.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.544+0000 7f6e66e00700 1 -- 172.21.15.44:0/3626230984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e60108f10 msgr2=0x7f6e60075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.544+0000 7f6e66e00700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e60108f10 0x7f6e60075ff0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6e48005f50 tx=0x7f6e48005280 comp rx=0 tx=0).stop 2024-05-27T09:45:40.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.544+0000 7f6e66e00700 1 -- 172.21.15.44:0/3626230984 shutdown_connections 2024-05-27T09:45:40.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.544+0000 7f6e66e00700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f6e440b2a70 0x7f6e440b4f80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.544+0000 7f6e66e00700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e60109860 0x7f6e60076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.544+0000 7f6e66e00700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e601045e0 0x7f6e60075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.544+0000 7f6e66e00700 1 --2- 172.21.15.44:0/3626230984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e60108f10 0x7f6e60075ff0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.544+0000 7f6e66e00700 1 -- 172.21.15.44:0/3626230984 >> 172.21.15.44:0/3626230984 conn(0x7f6e60100030 msgr2=0x7f6e6010a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:40.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.544+0000 7f6e66e00700 1 -- 172.21.15.44:0/3626230984 shutdown_connections 2024-05-27T09:45:40.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:40.545+0000 7f6e66e00700 1 -- 172.21.15.44:0/3626230984 wait complete. 2024-05-27T09:45:40.546 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 33 2024-05-27T09:45:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.583+0000 7fa9b7000640 1 -- 172.21.15.44:0/3453694960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b010b7f0 msgr2=0x7fa9b010dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.583+0000 7fa9b7000640 1 --2- 172.21.15.44:0/3453694960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b010b7f0 0x7fa9b010dc10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009900 tx=0x7fa99c02a3f0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.583+0000 7fa9b7000640 1 -- 172.21.15.44:0/3453694960 shutdown_connections 2024-05-27T09:45:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.583+0000 7fa9b7000640 1 --2- 172.21.15.44:0/3453694960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9b010e1e0 0x7fa9b01106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.583+0000 7fa9b7000640 1 --2- 172.21.15.44:0/3453694960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b010b7f0 0x7fa9b010dc10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.583+0000 7fa9b7000640 1 --2- 172.21.15.44:0/3453694960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b0104d90 0x7fa9b01051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.583+0000 7fa9b7000640 1 -- 172.21.15.44:0/3453694960 >> 172.21.15.44:0/3453694960 conn(0x7fa9b00782d0 msgr2=0x7fa9b0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.583+0000 7fa9b7000640 1 -- 172.21.15.44:0/3453694960 shutdown_connections 2024-05-27T09:45:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.583+0000 7fa9b7000640 1 -- 172.21.15.44:0/3453694960 wait complete. 2024-05-27T09:45:40.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.585+0000 7fa9b7000640 1 Processor -- start 2024-05-27T09:45:40.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.585+0000 7fa9b7000640 1 -- start start 2024-05-27T09:45:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.585+0000 7fa9b7000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9b0104d90 0x7fa9b0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.585+0000 7fa9b7000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b010b7f0 0x7fa9b0198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.585+0000 7fa9b7000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b010e1e0 0x7fa9b019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.585+0000 7fa9b7000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b0111c80 con 0x7fa9b0104d90 2024-05-27T09:45:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9b7000640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b0111b00 con 0x7fa9b010b7f0 2024-05-27T09:45:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9b7000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b0111e00 con 0x7fa9b010e1e0 2024-05-27T09:45:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9afe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9b0104d90 0x7fa9b0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9af400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b010b7f0 0x7fa9b0198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9afe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9b0104d90 0x7fa9b0198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58978/0 (socket says 172.21.15.44:58978) 2024-05-27T09:45:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9af400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b010b7f0 0x7fa9b0198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39438/0 (socket says 172.21.15.44:39438) 2024-05-27T09:45:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9afe00640 1 -- 172.21.15.44:0/947887221 learned_addr learned my addr 172.21.15.44:0/947887221 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9b5000640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b010e1e0 0x7fa9b019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9af400640 1 -- 172.21.15.44:0/947887221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b010e1e0 msgr2=0x7fa9b019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9af400640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b010e1e0 0x7fa9b019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9af400640 1 -- 172.21.15.44:0/947887221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9b0104d90 msgr2=0x7fa9b0198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9af400640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9b0104d90 0x7fa9b0198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.586+0000 7fa9af400640 1 -- 172.21.15.44:0/947887221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b019d4e0 con 0x7fa9b010b7f0 2024-05-27T09:45:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.587+0000 7fa9afe00640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9b0104d90 0x7fa9b0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.587+0000 7fa9af400640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b010b7f0 0x7fa9b0198be0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa99c02a9e0 tx=0x7fa99c003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.587+0000 7fa9acc00640 1 -- 172.21.15.44:0/947887221 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c03f070 con 0x7fa9b010b7f0 2024-05-27T09:45:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.587+0000 7fa9acc00640 1 -- 172.21.15.44:0/947887221 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa99c031b50 con 0x7fa9b010b7f0 2024-05-27T09:45:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.587+0000 7fa9b7000640 1 -- 172.21.15.44:0/947887221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b019d770 con 0x7fa9b010b7f0 2024-05-27T09:45:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.587+0000 7fa9acc00640 1 -- 172.21.15.44:0/947887221 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c035030 con 0x7fa9b010b7f0 2024-05-27T09:45:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.588+0000 7fa9b7000640 1 -- 172.21.15.44:0/947887221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b019dc30 con 0x7fa9b010b7f0 2024-05-27T09:45:40.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:40 smithi189 ceph-mon[36671]: pgmap v325: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 325 KiB/s wr, 59 op/s 2024-05-27T09:45:40.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.592+0000 7fa9be021740 1 Processor -- start 2024-05-27T09:45:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.592+0000 7fa9be021740 1 -- start start 2024-05-27T09:45:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.593+0000 7fa9be021740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eb0d1b0 0x55662eeaf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.593+0000 7fa9be021740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 0x55662eeafda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.593+0000 7fa9be021740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eeb02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.593+0000 7fa9be021740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55662ee20b90 con 0x55662eb0d1b0 2024-05-27T09:45:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.593+0000 7fa9be021740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55662ee10730 con 0x55662eaec4d0 2024-05-27T09:45:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.593+0000 7fa9be021740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55662ee14770 con 0x55662eae42f0 2024-05-27T09:45:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.593+0000 7fa9b5000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eb0d1b0 0x55662eeaf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.593+0000 7fa9afe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 0x55662eeafda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.593+0000 7fa9b5000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eb0d1b0 0x55662eeaf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58990/0 (socket says 172.21.15.44:58990) 2024-05-27T09:45:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.593+0000 7fa9b5000640 1 -- 172.21.15.44:0/2831339568 learned_addr learned my addr 172.21.15.44:0/2831339568 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.594+0000 7fa9afe00640 1 -- 172.21.15.44:0/2831339568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 msgr2=0x55662eeb02e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:45:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.594+0000 7fa9afe00640 1 --2- 172.21.15.44:0/2831339568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eeb02e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.594+0000 7fa9afe00640 1 -- 172.21.15.44:0/2831339568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eb0d1b0 msgr2=0x55662eeaf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.594+0000 7fa9afe00640 1 --2- 172.21.15.44:0/2831339568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eb0d1b0 0x55662eeaf860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.594+0000 7fa9afe00640 1 -- 172.21.15.44:0/2831339568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55662eda98b0 con 0x55662eae42f0 2024-05-27T09:45:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.594+0000 7fa9afe00640 1 --2- 172.21.15.44:0/2831339568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 0x55662eeafda0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0011510 tx=0x7fa9a00119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de8eec73f8532a4f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.594+0000 7fa9acc00640 1 -- 172.21.15.44:0/947887221 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa99c042070 con 0x7fa9b010b7f0 2024-05-27T09:45:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.597+0000 7fa9a5600640 1 -- 172.21.15.44:0/2831339568 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a001e070 con 0x55662eae42f0 2024-05-27T09:45:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.597+0000 7fa9a5600640 1 -- 172.21.15.44:0/2831339568 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a000fa70 con 0x55662eae42f0 2024-05-27T09:45:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.597+0000 7fa9a5600640 1 -- 172.21.15.44:0/2831339568 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0019b00 con 0x55662eae42f0 2024-05-27T09:45:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.597+0000 7fa9acc00640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa9940b29a0 0x7fa9940b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.597+0000 7fa9acc00640 1 -- 172.21.15.44:0/947887221 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fa99c102c80 con 0x7fa9b010b7f0 2024-05-27T09:45:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.598+0000 7fa9be021740 1 -- 172.21.15.44:0/2831339568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 msgr2=0x55662eeafda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.598+0000 7fa9be021740 1 --2- 172.21.15.44:0/2831339568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 0x55662eeafda0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0011510 tx=0x7fa9a00119d0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.598+0000 7fa9b5000640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa9940b29a0 0x7fa9940b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.598+0000 7fa9be021740 1 -- 172.21.15.44:0/2831339568 shutdown_connections 2024-05-27T09:45:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.598+0000 7fa9be021740 1 --2- 172.21.15.44:0/2831339568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eeb02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.598+0000 7fa9be021740 1 --2- 172.21.15.44:0/2831339568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 0x55662eeafda0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.598+0000 7fa9be021740 1 --2- 172.21.15.44:0/2831339568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eb0d1b0 0x55662eeaf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.598+0000 7fa9be021740 1 -- 172.21.15.44:0/2831339568 >> 172.21.15.44:0/2831339568 conn(0x55662eb01300 msgr2=0x55662eeac010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.598+0000 7fa9be021740 1 -- 172.21.15.44:0/2831339568 shutdown_connections 2024-05-27T09:45:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.598+0000 7fa9be021740 1 -- 172.21.15.44:0/2831339568 wait complete. 2024-05-27T09:45:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.599+0000 7fa9b5000640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa9940b29a0 0x7fa9940b4eb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800d700 tx=0x7fa9a800d130 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.599+0000 7fa9be021740 1 -- start start 2024-05-27T09:45:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.599+0000 7fa9be021740 1 Processor -- start 2024-05-27T09:45:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.600+0000 7fa9be021740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eae42f0 0x55662eeb9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.600+0000 7fa9be021740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eeba1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.600+0000 7fa9be021740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 0x55662eec5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.600+0000 7fa9be021740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55662eaf6790 con 0x55662eae42f0 2024-05-27T09:45:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.600+0000 7fa9be021740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55662edb8bc0 con 0x55662eaec4d0 2024-05-27T09:45:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.600+0000 7fa9be021740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55662eebc990 con 0x55662eb0d1b0 2024-05-27T09:45:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.600+0000 7fa9be021740 10 client.0 _ll_register_callbacks cb 0x55662ed96400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.600+0000 7fa9afe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eae42f0 0x55662eeb9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.600+0000 7fa9afe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eae42f0 0x55662eeb9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59006/0 (socket says 172.21.15.44:59006) 2024-05-27T09:45:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.600+0000 7fa9afe00640 1 -- 172.21.15.44:0/2551438978 learned_addr learned my addr 172.21.15.44:0/2551438978 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.600+0000 7fa9af400640 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eeba1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.601+0000 7fa9b5000640 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 0x55662eec5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.601+0000 7fa9afe00640 1 -- 172.21.15.44:0/2551438978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 msgr2=0x55662eec5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.601+0000 7fa9afe00640 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 0x55662eec5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.601+0000 7fa9afe00640 1 -- 172.21.15.44:0/2551438978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 msgr2=0x55662eeba1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.601+0000 7fa9afe00640 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eeba1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.601+0000 7fa9afe00640 1 -- 172.21.15.44:0/2551438978 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55662eb25210 con 0x55662eae42f0 2024-05-27T09:45:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.601+0000 7fa9af400640 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eeba1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:45:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.601+0000 7fa9afe00640 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eae42f0 0x55662eeb9cb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa9a001b000 tx=0x7fa9a0019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c98f75a6acc24d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.602+0000 7fa99a000640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.602+0000 7fa99a000640 1 -- 172.21.15.44:0/2551438978 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a001e070 con 0x55662eae42f0 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.602+0000 7fa9be021740 1 -- 172.21.15.44:0/2551438978 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9a000fa70 con 0x55662eae42f0 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.602+0000 7fa99a000640 1 -- 172.21.15.44:0/2551438978 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fa9a0019880 con 0x55662eae42f0 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.602+0000 7fa99a000640 1 -- 172.21.15.44:0/2551438978 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa9a0031050 con 0x55662eae42f0 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.602+0000 7fa9be021740 10 client.16998 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.602+0000 7fa9be021740 1 -- 172.21.15.44:0/2551438978 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55662ebf1620 con 0x55662eae42f0 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.603+0000 7fa99a000640 1 -- 172.21.15.44:0/2551438978 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a000fa70 con 0x55662eae42f0 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.603+0000 7fa99a000640 1 -- 172.21.15.44:0/2551438978 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fa9a00046c0 con 0x55662eae42f0 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.603+0000 7fa9be021740 10 client.16998 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.603+0000 7fa9be021740 10 client.16998 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.604+0000 7fa9be021740 20 client.16998 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.604+0000 7fa9be021740 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662eacbd80 0x55662eecbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.604+0000 7fa9be021740 4 client.16998 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.604+0000 7fa9be021740 1 -- 172.21.15.44:0/2551438978 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55662ee22ae0 con 0x55662eacbd80 2024-05-27T09:45:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.604+0000 7fa9af400640 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662eacbd80 0x55662eecbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.605+0000 7fa9af400640 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662eacbd80 0x55662eecbbd0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8554170c121c21c6 server_cookie=a22be2b3d5b415ec in_seq=0 out_seq=0 2024-05-27T09:45:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.605+0000 7fa99a000640 10 client.16998 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.606+0000 7fa99a000640 1 -- 172.21.15.44:0/2551438978 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55662ee22ae0 con 0x55662eacbd80 2024-05-27T09:45:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.606+0000 7fa99a000640 10 client.16998 handle_command_reply: tid=0 2024-05-27T09:45:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.606+0000 7fa9be021740 1 client.16998 shutdown 2024-05-27T09:45:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.606+0000 7fa9be021740 1 -- 172.21.15.44:0/2551438978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eae42f0 msgr2=0x55662eeb9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.606+0000 7fa9be021740 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eae42f0 0x55662eeb9cb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa9a001b000 tx=0x7fa9a0019640 comp rx=0 tx=0).stop 2024-05-27T09:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.606+0000 7fa9be021740 1 -- 172.21.15.44:0/2551438978 shutdown_connections 2024-05-27T09:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.606+0000 7fa9be021740 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662eacbd80 0x55662eecbbd0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.606+0000 7fa9be021740 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 0x55662eec5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.606+0000 7fa9be021740 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eeba1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.606+0000 7fa9be021740 1 --2- 172.21.15.44:0/2551438978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eae42f0 0x55662eeb9cb0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.607+0000 7fa9be021740 1 -- 172.21.15.44:0/2551438978 >> 172.21.15.44:0/2551438978 conn(0x55662eb01300 msgr2=0x55662eeb1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.606+0000 7fa99a000640 0 client.16998 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.607+0000 7fa9be021740 1 -- 172.21.15.44:0/2551438978 shutdown_connections 2024-05-27T09:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.607+0000 7fa9be021740 1 -- 172.21.15.44:0/2551438978 wait complete. 2024-05-27T09:45:40.609 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.607+0000 7fa9be021740 20 client.16998 trim_cache size 0 max 16384 2024-05-27T09:45:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.632+0000 7fa9be021740 1 Processor -- start 2024-05-27T09:45:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.632+0000 7fa9be021740 1 -- start start 2024-05-27T09:45:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.633+0000 7fa9be021740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eae42f0 0x55662eeb01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.633+0000 7fa9be021740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eaec4d0 0x55662eec5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.633+0000 7fa9be021740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 0x55662eec5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.633+0000 7fa9be021740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01b01f0 con 0x55662eaec4d0 2024-05-27T09:45:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.633+0000 7fa9be021740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01ada90 con 0x55662eae42f0 2024-05-27T09:45:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.633+0000 7fa9be021740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01ab900 con 0x55662eb0d1b0 2024-05-27T09:45:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.633+0000 7fa9b5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 0x55662eec5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.633+0000 7fa9af400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eaec4d0 0x55662eec5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.633+0000 7fa9b5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 0x55662eec5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:57652/0 (socket says 172.21.15.44:57652) 2024-05-27T09:45:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.633+0000 7fa9b5000640 1 -- 172.21.15.44:0/3743963887 learned_addr learned my addr 172.21.15.44:0/3743963887 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.634+0000 7fa9afe00640 1 --2- 172.21.15.44:0/3743963887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eae42f0 0x55662eeb01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.634+0000 7fa9b5000640 1 -- 172.21.15.44:0/3743963887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eae42f0 msgr2=0x55662eeb01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.634+0000 7fa9b5000640 1 --2- 172.21.15.44:0/3743963887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eae42f0 0x55662eeb01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.634+0000 7fa9b5000640 1 -- 172.21.15.44:0/3743963887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eaec4d0 msgr2=0x55662eec5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.634+0000 7fa9b5000640 1 --2- 172.21.15.44:0/3743963887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eaec4d0 0x55662eec5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.634+0000 7fa9b5000640 1 -- 172.21.15.44:0/3743963887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55662eeb7ff0 con 0x55662eb0d1b0 2024-05-27T09:45:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.634+0000 7fa9afe00640 1 --2- 172.21.15.44:0/3743963887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eae42f0 0x55662eeb01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:45:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.634+0000 7fa9b5000640 1 --2- 172.21.15.44:0/3743963887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 0x55662eec5fa0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80196c0 tx=0x7fa9a801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=527c45576f692d00 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.635+0000 7fa9a4c00640 1 -- 172.21.15.44:0/3743963887 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a801aac0 con 0x55662eb0d1b0 2024-05-27T09:45:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.635+0000 7fa9a4c00640 1 -- 172.21.15.44:0/3743963887 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a8020040 con 0x55662eb0d1b0 2024-05-27T09:45:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.635+0000 7fa9a4c00640 1 -- 172.21.15.44:0/3743963887 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8027380 con 0x55662eb0d1b0 2024-05-27T09:45:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.636+0000 7fa9be021740 1 -- 172.21.15.44:0/3743963887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 msgr2=0x55662eec5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.636+0000 7fa9be021740 1 --2- 172.21.15.44:0/3743963887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 0x55662eec5fa0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80196c0 tx=0x7fa9a801a000 comp rx=0 tx=0).stop 2024-05-27T09:45:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.636+0000 7fa9be021740 1 -- 172.21.15.44:0/3743963887 shutdown_connections 2024-05-27T09:45:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.636+0000 7fa9be021740 1 --2- 172.21.15.44:0/3743963887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eb0d1b0 0x55662eec5fa0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.636+0000 7fa9be021740 1 --2- 172.21.15.44:0/3743963887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eaec4d0 0x55662eec5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.636+0000 7fa9be021740 1 --2- 172.21.15.44:0/3743963887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eae42f0 0x55662eeb01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.636+0000 7fa9be021740 1 -- 172.21.15.44:0/3743963887 >> 172.21.15.44:0/3743963887 conn(0x55662eb01300 msgr2=0x55662eebbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.636+0000 7fa9be021740 1 -- 172.21.15.44:0/3743963887 shutdown_connections 2024-05-27T09:45:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.636+0000 7fa9be021740 1 -- 172.21.15.44:0/3743963887 wait complete. 2024-05-27T09:45:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.637+0000 7fa9be021740 1 -- start start 2024-05-27T09:45:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.637+0000 7fa9be021740 1 Processor -- start 2024-05-27T09:45:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.637+0000 7fa9be021740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 0x55662eebcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.637+0000 7fa9be021740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eebd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.637+0000 7fa9be021740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eb0d1b0 0x55662eebd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.637+0000 7fa9be021740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01abf50 con 0x55662eb0d1b0 2024-05-27T09:45:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.637+0000 7fa9be021740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01ad3c0 con 0x55662eaec4d0 2024-05-27T09:45:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.637+0000 7fa9be021740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01ac620 con 0x55662eae42f0 2024-05-27T09:45:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.638+0000 7fa9be021740 10 client.0 _ll_register_callbacks cb 0x7fa9a001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.638+0000 7fa9afe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 0x55662eebcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.638+0000 7fa9afe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 0x55662eebcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:57666/0 (socket says 172.21.15.44:57666) 2024-05-27T09:45:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.638+0000 7fa9afe00640 1 -- 172.21.15.44:0/2267614114 learned_addr learned my addr 172.21.15.44:0/2267614114 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.638+0000 7fa9b5000640 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eb0d1b0 0x55662eebd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.638+0000 7fa9af400640 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eebd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.639+0000 7fa9af400640 1 -- 172.21.15.44:0/2267614114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 msgr2=0x55662eebcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.639+0000 7fa9af400640 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 0x55662eebcc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.639+0000 7fa9af400640 1 -- 172.21.15.44:0/2267614114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eb0d1b0 msgr2=0x55662eebd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.639+0000 7fa9af400640 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eb0d1b0 0x55662eebd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.639+0000 7fa9af400640 1 -- 172.21.15.44:0/2267614114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55662ef320c0 con 0x55662eaec4d0 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.639+0000 7fa9af400640 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eebd150 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa99c1003d0 tx=0x7fa99c03b6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8862fe5a14fa704 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.639+0000 7fa99a000640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.639+0000 7fa99a000640 1 -- 172.21.15.44:0/2267614114 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c049040 con 0x55662eaec4d0 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.639+0000 7fa99a000640 1 -- 172.21.15.44:0/2267614114 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fa99c031d00 con 0x55662eaec4d0 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.639+0000 7fa9be021740 1 -- 172.21.15.44:0/2267614114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9a8020040 con 0x55662eaec4d0 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.640+0000 7fa99a000640 1 -- 172.21.15.44:0/2267614114 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa99c0d2dc0 con 0x55662eaec4d0 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.640+0000 7fa9be021740 10 client.27218 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.640+0000 7fa9be021740 1 -- 172.21.15.44:0/2267614114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55662ef331f0 con 0x55662eaec4d0 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.640+0000 7fa99a000640 1 -- 172.21.15.44:0/2267614114 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa99c100650 con 0x55662eaec4d0 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.640+0000 7fa99a000640 1 -- 172.21.15.44:0/2267614114 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fa99c0c9270 con 0x55662eaec4d0 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.640+0000 7fa9be021740 10 client.27218 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.640+0000 7fa9be021740 10 client.27218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.640+0000 7fa9be021740 20 client.27218 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.640+0000 7fa9be021740 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662eaa11c0 0x55662ef35930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.640+0000 7fa9be021740 4 client.27218 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:45:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.640+0000 7fa9be021740 1 -- 172.21.15.44:0/2267614114 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fa9a8020040 con 0x55662eaa11c0 2024-05-27T09:45:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.641+0000 7fa9afe00640 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662eaa11c0 0x55662ef35930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.641+0000 7fa99a000640 10 client.27218 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.641+0000 7fa9afe00640 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662eaa11c0 0x55662ef35930 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55169575a703b257 server_cookie=cf3c974b68f2c363 in_seq=0 out_seq=0 2024-05-27T09:45:40.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa99a000640 1 -- 172.21.15.44:0/2267614114 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7fa9a8020040 con 0x55662eaa11c0 2024-05-27T09:45:40.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa99a000640 10 client.27218 handle_command_reply: tid=0 2024-05-27T09:45:40.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa9be021740 1 client.27218 shutdown 2024-05-27T09:45:40.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa9be021740 1 -- 172.21.15.44:0/2267614114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 msgr2=0x55662eebd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa9be021740 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eebd150 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa99c1003d0 tx=0x7fa99c03b6b0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa9be021740 1 -- 172.21.15.44:0/2267614114 shutdown_connections 2024-05-27T09:45:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa9be021740 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55662eaa11c0 0x55662ef35930 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa9be021740 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55662eb0d1b0 0x55662eebd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa9be021740 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55662eaec4d0 0x55662eebd150 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa99a000640 0 client.27218 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa9be021740 1 --2- 172.21.15.44:0/2267614114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55662eae42f0 0x55662eebcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.679+0000 7fa9be021740 1 -- 172.21.15.44:0/2267614114 >> 172.21.15.44:0/2267614114 conn(0x55662eb01300 msgr2=0x55662eebb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.680+0000 7fa9be021740 1 -- 172.21.15.44:0/2267614114 shutdown_connections 2024-05-27T09:45:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.680+0000 7fa9be021740 1 -- 172.21.15.44:0/2267614114 wait complete. 2024-05-27T09:45:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.680+0000 7fa9be021740 20 client.27218 trim_cache size 0 max 16384 2024-05-27T09:45:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:45:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:45:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:45:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.680+0000 7fa9b7000640 1 -- 172.21.15.44:0/947887221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa9940b29a0 msgr2=0x7fa9940b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.680+0000 7fa9b7000640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa9940b29a0 0x7fa9940b4eb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800d700 tx=0x7fa9a800d130 comp rx=0 tx=0).stop 2024-05-27T09:45:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.680+0000 7fa9b7000640 1 -- 172.21.15.44:0/947887221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b010b7f0 msgr2=0x7fa9b0198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.680+0000 7fa9b7000640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b010b7f0 0x7fa9b0198be0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa99c02a9e0 tx=0x7fa99c003c00 comp rx=0 tx=0).stop 2024-05-27T09:45:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.681+0000 7fa9b7000640 1 -- 172.21.15.44:0/947887221 shutdown_connections 2024-05-27T09:45:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.681+0000 7fa9b7000640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b010e1e0 0x7fa9b019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.681+0000 7fa9b7000640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b010b7f0 0x7fa9b0198be0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.681+0000 7fa9b7000640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa9940b29a0 0x7fa9940b4eb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.681+0000 7fa9b7000640 1 --2- 172.21.15.44:0/947887221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa9b0104d90 0x7fa9b0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.681+0000 7fa9b7000640 1 -- 172.21.15.44:0/947887221 >> 172.21.15.44:0/947887221 conn(0x7fa9b00782d0 msgr2=0x7fa9b01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.681+0000 7fa9b7000640 1 -- 172.21.15.44:0/947887221 shutdown_connections 2024-05-27T09:45:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:40.681+0000 7fa9b7000640 1 -- 172.21.15.44:0/947887221 wait complete. 2024-05-27T09:45:41.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:41 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3626230984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:41.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:41 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3626230984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:41.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:41 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3626230984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:45:42.150 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db --set-id 2a69c329 --release --await' 2024-05-27T09:45:42.150 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db --set-id 2a69c329 --release --await 2024-05-27T09:45:42.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:42 smithi044 ceph-mon[31618]: pgmap v326: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 332 KiB/s wr, 64 op/s 2024-05-27T09:45:42.444 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:42 smithi050 ceph-mon[36371]: pgmap v326: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 332 KiB/s wr, 64 op/s 2024-05-27T09:45:42.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:42 smithi189 ceph-mon[36671]: pgmap v326: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 332 KiB/s wr, 64 op/s 2024-05-27T09:45:42.722 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:42.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.787+0000 7fa2c9200700 1 -- 172.21.15.44:0/2545690396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2c4109860 msgr2=0x7fa2c41111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:42.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.787+0000 7fa2c9200700 1 --2- 172.21.15.44:0/2545690396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2c4109860 0x7fa2c41111c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00b0a0 tx=0x7fa2bc00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.787+0000 7fa2c9200700 1 -- 172.21.15.44:0/2545690396 shutdown_connections 2024-05-27T09:45:42.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.787+0000 7fa2c9200700 1 --2- 172.21.15.44:0/2545690396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2c4109860 0x7fa2c41111c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.787+0000 7fa2c9200700 1 --2- 172.21.15.44:0/2545690396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c41045e0 0x7fa2c41049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.787+0000 7fa2c9200700 1 --2- 172.21.15.44:0/2545690396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2c4108f10 0x7fa2c4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.787+0000 7fa2c9200700 1 -- 172.21.15.44:0/2545690396 >> 172.21.15.44:0/2545690396 conn(0x7fa2c4100030 msgr2=0x7fa2c4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:42.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.787+0000 7fa2c9200700 1 -- 172.21.15.44:0/2545690396 shutdown_connections 2024-05-27T09:45:42.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.787+0000 7fa2c9200700 1 -- 172.21.15.44:0/2545690396 wait complete. 2024-05-27T09:45:42.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.789+0000 7fa2c9200700 1 Processor -- start 2024-05-27T09:45:42.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.789+0000 7fa2c9200700 1 -- start start 2024-05-27T09:45:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.790+0000 7fa2c9200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2c41045e0 0x7fa2c419d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.790+0000 7fa2c9200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c4108f10 0x7fa2c419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.790+0000 7fa2c9200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2c4109860 0x7fa2c41a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.790+0000 7fa2c9200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c4114500 con 0x7fa2c41045e0 2024-05-27T09:45:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.790+0000 7fa2c9200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c4114380 con 0x7fa2c4109860 2024-05-27T09:45:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.790+0000 7fa2c9200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c4114680 con 0x7fa2c4108f10 2024-05-27T09:45:42.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.790+0000 7fa2c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2c41045e0 0x7fa2c419d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.790+0000 7fa2c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2c41045e0 0x7fa2c419d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59054/0 (socket says 172.21.15.44:59054) 2024-05-27T09:45:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.790+0000 7fa2c1e00700 1 -- 172.21.15.44:0/3316363067 learned_addr learned my addr 172.21.15.44:0/3316363067 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.790+0000 7fa2c1400700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c4108f10 0x7fa2c419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.790+0000 7fa2c2800700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2c4109860 0x7fa2c41a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.791+0000 7fa2c1e00700 1 -- 172.21.15.44:0/3316363067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c4108f10 msgr2=0x7fa2c419d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.791+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c4108f10 0x7fa2c419d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.791+0000 7fa2c1e00700 1 -- 172.21.15.44:0/3316363067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2c4109860 msgr2=0x7fa2c41a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.791+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2c4109860 0x7fa2c41a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.791+0000 7fa2c1e00700 1 -- 172.21.15.44:0/3316363067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2c41a1f20 con 0x7fa2c41045e0 2024-05-27T09:45:42.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.791+0000 7fa2c1400700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c4108f10 0x7fa2c419d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:42.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.791+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2c41045e0 0x7fa2c419d050 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400e9c0 tx=0x7fa2b400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:42.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.792+0000 7fa2baa00700 1 -- 172.21.15.44:0/3316363067 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b400cbd0 con 0x7fa2c41045e0 2024-05-27T09:45:42.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.792+0000 7fa2baa00700 1 -- 172.21.15.44:0/3316363067 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b400ee60 con 0x7fa2c41045e0 2024-05-27T09:45:42.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.792+0000 7fa2c9200700 1 -- 172.21.15.44:0/3316363067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2c41a2210 con 0x7fa2c41045e0 2024-05-27T09:45:42.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.792+0000 7fa2baa00700 1 -- 172.21.15.44:0/3316363067 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b40055c0 con 0x7fa2c41045e0 2024-05-27T09:45:42.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.792+0000 7fa2c9200700 1 -- 172.21.15.44:0/3316363067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2c41a9b30 con 0x7fa2c41045e0 2024-05-27T09:45:42.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.796+0000 7fa2baa00700 1 -- 172.21.15.44:0/3316363067 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa2b4021440 con 0x7fa2c41045e0 2024-05-27T09:45:42.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.798+0000 7fa2baa00700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2a80b29f0 0x7fa2a80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.798+0000 7fa2baa00700 1 -- 172.21.15.44:0/3316363067 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fa2b40e2910 con 0x7fa2c41045e0 2024-05-27T09:45:42.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.798+0000 7fa2c1400700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2a80b29f0 0x7fa2a80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.799+0000 7fa2c1400700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2a80b29f0 0x7fa2a80b4f00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4110560 tx=0x7fa2b0009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:42.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.799+0000 7fa2d7d4ab80 1 Processor -- start 2024-05-27T09:45:42.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.799+0000 7fa2d7d4ab80 1 -- start start 2024-05-27T09:45:42.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.800+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa642c0e40 0x55aa644457f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.800+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa64445d30 0x55aa6443f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.800+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa6443fdb0 0x55aa644401c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.800+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55aa6436de40 con 0x55aa6443fdb0 2024-05-27T09:45:42.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.800+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55aa6435d9e0 con 0x55aa642c0e40 2024-05-27T09:45:42.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.800+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55aa64361a20 con 0x55aa64445d30 2024-05-27T09:45:42.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.800+0000 7fa2c1e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa642c0e40 0x55aa644457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.800+0000 7fa2c1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa64445d30 0x55aa6443f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.801+0000 7fa2c1e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa642c0e40 0x55aa644457f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39506/0 (socket says 172.21.15.44:39506) 2024-05-27T09:45:42.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.801+0000 7fa2c1e00700 1 -- 172.21.15.44:0/1676366070 learned_addr learned my addr 172.21.15.44:0/1676366070 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:42.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.801+0000 7fa2c2800700 1 --2- 172.21.15.44:0/1676366070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa6443fdb0 0x55aa644401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.801+0000 7fa2c1e00700 1 -- 172.21.15.44:0/1676366070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa64445d30 msgr2=0x55aa6443f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:42.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.801+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/1676366070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa64445d30 0x55aa6443f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.801+0000 7fa2c1e00700 1 -- 172.21.15.44:0/1676366070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa6443fdb0 msgr2=0x55aa644401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:42.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.801+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/1676366070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa6443fdb0 0x55aa644401c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.801+0000 7fa2c1e00700 1 -- 172.21.15.44:0/1676366070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa64326420 con 0x55aa642c0e40 2024-05-27T09:45:42.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.801+0000 7fa2c2800700 1 --2- 172.21.15.44:0/1676366070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa6443fdb0 0x55aa644401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:45:42.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.801+0000 7fa2c1400700 1 --2- 172.21.15.44:0/1676366070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa64445d30 0x55aa6443f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:42.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.801+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/1676366070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa642c0e40 0x55aa644457f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40a9b20 tx=0x7fa2b40a9580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e363a8eaba428eaa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:42.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.802+0000 7fa2af400700 1 -- 172.21.15.44:0/1676366070 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b40a9ff0 con 0x55aa642c0e40 2024-05-27T09:45:42.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.802+0000 7fa2af400700 1 -- 172.21.15.44:0/1676366070 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b40a5b30 con 0x55aa642c0e40 2024-05-27T09:45:42.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.802+0000 7fa2af400700 1 -- 172.21.15.44:0/1676366070 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b40b2760 con 0x55aa642c0e40 2024-05-27T09:45:42.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.803+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1676366070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa642c0e40 msgr2=0x55aa644457f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:42.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.803+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1676366070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa642c0e40 0x55aa644457f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40a9b20 tx=0x7fa2b40a9580 comp rx=0 tx=0).stop 2024-05-27T09:45:42.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.803+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1676366070 shutdown_connections 2024-05-27T09:45:42.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.803+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1676366070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa6443fdb0 0x55aa644401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.803+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1676366070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa642c0e40 0x55aa644457f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.803+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1676366070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa64445d30 0x55aa6443f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.803+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1676366070 >> 172.21.15.44:0/1676366070 conn(0x55aa64320fc0 msgr2=0x55aa6443a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:42.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.803+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1676366070 shutdown_connections 2024-05-27T09:45:42.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.803+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1676366070 wait complete. 2024-05-27T09:45:42.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.804+0000 7fa2d7d4ab80 1 -- start start 2024-05-27T09:45:42.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.804+0000 7fa2d7d4ab80 1 Processor -- start 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.804+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa644495f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.804+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa6443fdb0 0x55aa64449b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.804+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa64445d30 0x55aa6444a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.804+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55aa64263fa0 con 0x55aa642c0e40 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.804+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55aa642fe760 con 0x55aa6443fdb0 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.804+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55aa6444c800 con 0x55aa64445d30 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.805+0000 7fa2d7d4ab80 10 client.0 _ll_register_callbacks cb 0x55aa64307c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.805+0000 7fa2c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa644495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.805+0000 7fa2c1400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa6443fdb0 0x55aa64449b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.805+0000 7fa2c1400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa6443fdb0 0x55aa64449b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39512/0 (socket says 172.21.15.44:39512) 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.805+0000 7fa2c1400700 1 -- 172.21.15.44:0/1896482334 learned_addr learned my addr 172.21.15.44:0/1896482334 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:42.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.805+0000 7fa2c1400700 1 -- 172.21.15.44:0/1896482334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa64445d30 msgr2=0x55aa6444a070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T09:45:42.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.805+0000 7fa2c1400700 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa64445d30 0x55aa6444a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.805+0000 7fa2c1400700 1 -- 172.21.15.44:0/1896482334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 msgr2=0x55aa644495f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:42.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.805+0000 7fa2c1400700 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa644495f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.805+0000 7fa2c1400700 1 -- 172.21.15.44:0/1896482334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa642ad8d0 con 0x55aa6443fdb0 2024-05-27T09:45:42.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.805+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa644495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:42.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.806+0000 7fa2c1400700 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa6443fdb0 0x55aa64449b30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000ec70 tx=0x7fa2b00047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=619cd1f676c73f7a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:42.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.806+0000 7fa2a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:42.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.806+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1896482334 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0004520 con 0x55aa6443fdb0 2024-05-27T09:45:42.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.806+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1896482334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2b40a5b30 con 0x55aa6443fdb0 2024-05-27T09:45:42.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.806+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1896482334 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fa2b001f8c0 con 0x55aa6443fdb0 2024-05-27T09:45:42.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.806+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1896482334 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa2b001fac0 con 0x55aa6443fdb0 2024-05-27T09:45:42.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.807+0000 7fa2d7d4ab80 10 client.27230 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:42.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.807+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1896482334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55aa6432e060 con 0x55aa6443fdb0 2024-05-27T09:45:42.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.807+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1896482334 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b40a5b30 con 0x55aa6443fdb0 2024-05-27T09:45:42.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.807+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1896482334 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fa2b0026020 con 0x55aa6443fdb0 2024-05-27T09:45:42.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.807+0000 7fa2d7d4ab80 10 client.27230 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:42.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.807+0000 7fa2d7d4ab80 10 client.27230 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:45:42.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.807+0000 7fa2d7d4ab80 20 client.27230 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:42.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.807+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55aa64459790 0x55aa6445bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.807+0000 7fa2d7d4ab80 4 client.27230 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:42.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.807+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1896482334 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa6436fd90 con 0x55aa64459790 2024-05-27T09:45:42.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.808+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55aa64459790 0x55aa6445bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.808+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55aa64459790 0x55aa6445bbf0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=91e79690aeffd879 server_cookie=2ded29e823008778 in_seq=0 out_seq=0 2024-05-27T09:45:42.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.808+0000 7fa2a3e00700 10 client.27230 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:45:42.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.809+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1896482334 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55aa6436fd90 con 0x55aa64459790 2024-05-27T09:45:42.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.809+0000 7fa2a3e00700 10 client.27230 handle_command_reply: tid=0 2024-05-27T09:45:42.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.809+0000 7fa2d7d4ab80 1 client.27230 shutdown 2024-05-27T09:45:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.809+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1896482334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa6443fdb0 msgr2=0x55aa64449b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.809+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa6443fdb0 0x55aa64449b30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000ec70 tx=0x7fa2b00047c0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.810+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1896482334 shutdown_connections 2024-05-27T09:45:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.810+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa64445d30 0x55aa6444a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.810+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55aa64459790 0x55aa6445bbf0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.810+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa644495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.810+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1896482334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa6443fdb0 0x55aa64449b30 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.810+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1896482334 >> 172.21.15.44:0/1896482334 conn(0x55aa64320fc0 msgr2=0x55aa6443a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.810+0000 7fa2a3e00700 0 client.27230 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:45:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.810+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1896482334 shutdown_connections 2024-05-27T09:45:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.810+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1896482334 wait complete. 2024-05-27T09:45:42.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.810+0000 7fa2d7d4ab80 20 client.27230 trim_cache size 0 max 16384 2024-05-27T09:45:42.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.827+0000 7fa2d7d4ab80 1 Processor -- start 2024-05-27T09:45:42.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.827+0000 7fa2d7d4ab80 1 -- start start 2024-05-27T09:45:42.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.827+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa6443f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.827+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa6443fdb0 0x55aa64444dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.827+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 0x55aa64449fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.827+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c41af650 con 0x55aa642c0e40 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.827+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c41aceb0 con 0x55aa64445d30 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.827+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c410f390 con 0x55aa6443fdb0 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa6443f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa6443f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59074/0 (socket says 172.21.15.44:59074) 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c1e00700 1 -- 172.21.15.44:0/3628301145 learned_addr learned my addr 172.21.15.44:0/3628301145 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c1400700 1 --2- 172.21.15.44:0/3628301145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa6443fdb0 0x55aa64444dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c2800700 1 --2- 172.21.15.44:0/3628301145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 0x55aa64449fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c1e00700 1 -- 172.21.15.44:0/3628301145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa6443fdb0 msgr2=0x55aa64444dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:42.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/3628301145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa6443fdb0 0x55aa64444dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c1e00700 1 -- 172.21.15.44:0/3628301145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 msgr2=0x55aa64449fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:42.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/3628301145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 0x55aa64449fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c1e00700 1 -- 172.21.15.44:0/3628301145 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa64447b60 con 0x55aa642c0e40 2024-05-27T09:45:42.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c2800700 1 --2- 172.21.15.44:0/3628301145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 0x55aa64449fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:45:42.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.828+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/3628301145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa6443f870 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400c5b0 tx=0x7fa2b40aa290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9224cb5714186a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:42.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.829+0000 7fa2aea00700 1 -- 172.21.15.44:0/3628301145 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b40b2760 con 0x55aa642c0e40 2024-05-27T09:45:42.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.829+0000 7fa2aea00700 1 -- 172.21.15.44:0/3628301145 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b40c2040 con 0x55aa642c0e40 2024-05-27T09:45:42.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.829+0000 7fa2aea00700 1 -- 172.21.15.44:0/3628301145 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b40a8a60 con 0x55aa642c0e40 2024-05-27T09:45:42.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.830+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/3628301145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 msgr2=0x55aa6443f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:42.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.830+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/3628301145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa6443f870 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400c5b0 tx=0x7fa2b40aa290 comp rx=0 tx=0).stop 2024-05-27T09:45:42.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.830+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/3628301145 shutdown_connections 2024-05-27T09:45:42.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.830+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/3628301145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 0x55aa64449fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.830+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/3628301145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa6443f870 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.830+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/3628301145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa6443fdb0 0x55aa64444dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.830+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/3628301145 >> 172.21.15.44:0/3628301145 conn(0x55aa64320fc0 msgr2=0x55aa64443210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:42.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.830+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/3628301145 shutdown_connections 2024-05-27T09:45:42.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.830+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/3628301145 wait complete. 2024-05-27T09:45:42.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.831+0000 7fa2d7d4ab80 1 -- start start 2024-05-27T09:45:42.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.831+0000 7fa2d7d4ab80 1 Processor -- start 2024-05-27T09:45:42.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.831+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa64451d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.831+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa6443fdb0 0x55aa64452260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.831+0000 7fa2d7d4ab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 0x55aa64460aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c41ab390 con 0x55aa642c0e40 2024-05-27T09:45:42.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c41ac7e0 con 0x55aa64445d30 2024-05-27T09:45:42.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2d7d4ab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c41aba40 con 0x55aa6443fdb0 2024-05-27T09:45:42.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2d7d4ab80 10 client.0 _ll_register_callbacks cb 0x55aa6445c0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:42.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa64451d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2c1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa64451d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59078/0 (socket says 172.21.15.44:59078) 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2c1e00700 1 -- 172.21.15.44:0/1381885108 learned_addr learned my addr 172.21.15.44:0/1381885108 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2c2800700 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 0x55aa64460aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2c1400700 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa6443fdb0 0x55aa64452260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2c2800700 1 -- 172.21.15.44:0/1381885108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa6443fdb0 msgr2=0x55aa64452260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2c2800700 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa6443fdb0 0x55aa64452260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2c2800700 1 -- 172.21.15.44:0/1381885108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 msgr2=0x55aa64451d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2c2800700 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa64451d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.832+0000 7fa2c2800700 1 -- 172.21.15.44:0/1381885108 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa64461210 con 0x55aa64445d30 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.833+0000 7fa2c2800700 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 0x55aa64460aa0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc009fd0 tx=0x7fa2bc021d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd5f7f1d9baaeeb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.833+0000 7fa2a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.833+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1381885108 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc030060 con 0x55aa64445d30 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.833+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1381885108 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fa2bc00bcc0 con 0x55aa64445d30 2024-05-27T09:45:42.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.833+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1381885108 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2b40c2040 con 0x55aa64445d30 2024-05-27T09:45:42.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.833+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1381885108 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa2bc021e60 con 0x55aa64445d30 2024-05-27T09:45:42.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.833+0000 7fa2d7d4ab80 10 client.27236 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:42.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.833+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1381885108 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55aa644c3d40 con 0x55aa64445d30 2024-05-27T09:45:42.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.834+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1381885108 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b40c2040 con 0x55aa64445d30 2024-05-27T09:45:42.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.834+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1381885108 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fa2bc011070 con 0x55aa64445d30 2024-05-27T09:45:42.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.834+0000 7fa2d7d4ab80 10 client.27236 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:42.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.834+0000 7fa2d7d4ab80 10 client.27236 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.834+0000 7fa2d7d4ab80 20 client.27236 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.834+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55aa644c4450 0x55aa644c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.834+0000 7fa2d7d4ab80 4 client.27236 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.834+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1381885108 --> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aa64369570 con 0x55aa644c4450 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.835+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55aa644c4450 0x55aa644c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.835+0000 7fa2c1e00700 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55aa644c4450 0x55aa644c6870 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7bedd66e877fe5f server_cookie=7e73d902d58b0dd2 in_seq=0 out_seq=0 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.835+0000 7fa2a3e00700 10 client.27236 ms_handle_connect on v2:172.21.15.189:6838/4109192600 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.835+0000 7fa2a3e00700 1 -- 172.21.15.44:0/1381885108 <== mds.1 v2:172.21.15.189:6838/4109192600 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aa64369570 con 0x55aa644c4450 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.835+0000 7fa2a3e00700 10 client.27236 handle_command_reply: tid=0 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.835+0000 7fa2d7d4ab80 1 client.27236 shutdown 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1381885108 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 msgr2=0x55aa64460aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 0x55aa64460aa0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc009fd0 tx=0x7fa2bc021d70 comp rx=0 tx=0).stop 2024-05-27T09:45:42.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1381885108 shutdown_connections 2024-05-27T09:45:42.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55aa64445d30 0x55aa64460aa0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.189:6838/4109192600,v1:172.21.15.189:6839/4109192600] conn(0x55aa644c4450 0x55aa644c6870 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aa642c0e40 0x55aa64451d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2d7d4ab80 1 --2- 172.21.15.44:0/1381885108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa6443fdb0 0x55aa64452260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2a3e00700 0 client.27236 ms_handle_reset on v2:172.21.15.189:6838/4109192600 2024-05-27T09:45:42.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1381885108 >> 172.21.15.44:0/1381885108 conn(0x55aa64320fc0 msgr2=0x55aa64444dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:42.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1381885108 shutdown_connections 2024-05-27T09:45:42.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2d7d4ab80 1 -- 172.21.15.44:0/1381885108 wait complete. 2024-05-27T09:45:42.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.836+0000 7fa2d7d4ab80 20 client.27236 trim_cache size 0 max 16384 2024-05-27T09:45:42.839 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:45:42.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 -- 172.21.15.44:0/3316363067 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2a80b29f0 msgr2=0x7fa2a80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:42.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2a80b29f0 0x7fa2a80b4f00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4110560 tx=0x7fa2b0009260 comp rx=0 tx=0).stop 2024-05-27T09:45:42.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 -- 172.21.15.44:0/3316363067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2c41045e0 msgr2=0x7fa2c419d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:42.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2c41045e0 0x7fa2c419d050 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400e9c0 tx=0x7fa2b400ecd0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 -- 172.21.15.44:0/3316363067 shutdown_connections 2024-05-27T09:45:42.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa2a80b29f0 0x7fa2a80b4f00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2c4109860 0x7fa2c41a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2c41045e0 0x7fa2c419d050 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 --2- 172.21.15.44:0/3316363067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c4108f10 0x7fa2c419d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:42.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 -- 172.21.15.44:0/3316363067 >> 172.21.15.44:0/3316363067 conn(0x7fa2c4100030 msgr2=0x7fa2c4110e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:42.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 -- 172.21.15.44:0/3316363067 shutdown_connections 2024-05-27T09:45:42.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:42.837+0000 7fa2c9200700 1 -- 172.21.15.44:0/3316363067 wait complete. 2024-05-27T09:45:44.229 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:45:44.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:44 smithi044 ceph-mon[31618]: pgmap v327: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-05-27T09:45:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:44 smithi050 ceph-mon[36371]: pgmap v327: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-05-27T09:45:44.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:44 smithi189 ceph-mon[36671]: pgmap v327: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-05-27T09:45:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:46.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:46 smithi044 ceph-mon[31618]: pgmap v328: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-05-27T09:45:46.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:46 smithi050 ceph-mon[36371]: pgmap v328: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-05-27T09:45:46.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:46 smithi189 ceph-mon[36671]: pgmap v328: 97 pgs: 97 active+clean; 279 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-05-27T09:45:46.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.526+0000 7fc4ae600700 1 -- 172.21.15.44:0/4154273048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4a8109840 msgr2=0x7fc4a81111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:46.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.526+0000 7fc4ae600700 1 --2- 172.21.15.44:0/4154273048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4a8109840 0x7fc4a81111a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000b0d0 tx=0x7fc4a000b3e0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.526+0000 7fc4ae600700 1 -- 172.21.15.44:0/4154273048 shutdown_connections 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.526+0000 7fc4ae600700 1 --2- 172.21.15.44:0/4154273048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4a8109840 0x7fc4a81111a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.526+0000 7fc4ae600700 1 --2- 172.21.15.44:0/4154273048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a81045c0 0x7fc4a81049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.526+0000 7fc4ae600700 1 --2- 172.21.15.44:0/4154273048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a8108ef0 0x7fc4a8109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.526+0000 7fc4ae600700 1 -- 172.21.15.44:0/4154273048 >> 172.21.15.44:0/4154273048 conn(0x7fc4a80ffe90 msgr2=0x7fc4a81022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.526+0000 7fc4ae600700 1 -- 172.21.15.44:0/4154273048 shutdown_connections 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.526+0000 7fc4ae600700 1 -- 172.21.15.44:0/4154273048 wait complete. 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.527+0000 7fc4ae600700 1 Processor -- start 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.527+0000 7fc4ae600700 1 -- start start 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.528+0000 7fc4ae600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a81045c0 0x7fc4a819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.528+0000 7fc4ae600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4a8108ef0 0x7fc4a819d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.528+0000 7fc4ae600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8109840 0x7fc4a81a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.528+0000 7fc4ae600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a81144c0 con 0x7fc4a8108ef0 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.528+0000 7fc4ae600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a8114340 con 0x7fc4a81045c0 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.528+0000 7fc4ae600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a8114640 con 0x7fc4a8109840 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.528+0000 7fc49ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4a8108ef0 0x7fc4a819d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.528+0000 7fc4a7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a81045c0 0x7fc4a819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.528+0000 7fc49ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4a8108ef0 0x7fc4a819d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57906/0 (socket says 172.21.15.44:57906) 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.528+0000 7fc49ea00700 1 -- 172.21.15.44:0/1879778685 learned_addr learned my addr 172.21.15.44:0/1879778685 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.528+0000 7fc4a7e00700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8109840 0x7fc4a81a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.529+0000 7fc49ea00700 1 -- 172.21.15.44:0/1879778685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8109840 msgr2=0x7fc4a81a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.529+0000 7fc49ea00700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8109840 0x7fc4a81a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.529+0000 7fc49ea00700 1 -- 172.21.15.44:0/1879778685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a81045c0 msgr2=0x7fc4a819cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.529+0000 7fc49ea00700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a81045c0 0x7fc4a819cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.529+0000 7fc49ea00700 1 -- 172.21.15.44:0/1879778685 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a81a1ec0 con 0x7fc4a8108ef0 2024-05-27T09:45:46.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.529+0000 7fc49ea00700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4a8108ef0 0x7fc4a819d590 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc49000e970 tx=0x7fc49000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.530+0000 7fc4a4c00700 1 -- 172.21.15.44:0/1879778685 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc490004a30 con 0x7fc4a8108ef0 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.530+0000 7fc4ae600700 1 -- 172.21.15.44:0/1879778685 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a81a21b0 con 0x7fc4a8108ef0 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.530+0000 7fc4a4c00700 1 -- 172.21.15.44:0/1879778685 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc49000cc90 con 0x7fc4a8108ef0 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.530+0000 7fc4a4c00700 1 -- 172.21.15.44:0/1879778685 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc490005610 con 0x7fc4a8108ef0 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.530+0000 7fc4ae600700 1 -- 172.21.15.44:0/1879778685 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a81a9ad0 con 0x7fc4a8108ef0 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.534+0000 7fc4a4c00700 1 -- 172.21.15.44:0/1879778685 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc49000c7e0 con 0x7fc4a8108ef0 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.535+0000 7fc4bd11eb80 1 Processor -- start 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.535+0000 7fc4bd11eb80 1 -- start start 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.535+0000 7fc4a4c00700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4880b2a20 0x7fc4880b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.535+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb1e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.535+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 0x55d96eb1ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.535+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96eb185e0 0x55d96eb189f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.535+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d96ea45f40 con 0x55d96e998f40 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.535+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d96ea35ae0 con 0x55d96eb185e0 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.535+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d96ea39b20 con 0x55d96ea84af0 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.535+0000 7fc4a4c00700 1 -- 172.21.15.44:0/1879778685 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fc4900e27a0 con 0x7fc4a8108ef0 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc4a7e00700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4880b2a20 0x7fc4880b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc49ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 0x55d96eb1ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc4a7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb1e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc4a7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb1e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:33936/0 (socket says 172.21.15.44:33936) 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc4a7400700 1 -- 172.21.15.44:0/2971721810 learned_addr learned my addr 172.21.15.44:0/2971721810 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc49ea00700 1 --2- 172.21.15.44:0/2971721810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 0x55d96eb1ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57908/0 (socket says 172.21.15.44:57908) 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc4a7e00700 1 --2- 172.21.15.44:0/2971721810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96eb185e0 0x55d96eb189f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc4a7400700 1 -- 172.21.15.44:0/2971721810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96eb185e0 msgr2=0x55d96eb189f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:46.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc4a7400700 1 --2- 172.21.15.44:0/2971721810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96eb185e0 0x55d96eb189f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc4a7400700 1 -- 172.21.15.44:0/2971721810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 msgr2=0x55d96eb1ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc4a7400700 1 --2- 172.21.15.44:0/2971721810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 0x55d96eb1ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.536+0000 7fc4a7400700 1 -- 172.21.15.44:0/2971721810 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d96e9fe520 con 0x55d96ea84af0 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.537+0000 7fc4a7400700 1 --2- 172.21.15.44:0/2971721810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb1e530 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc498009d80 tx=0x7fc498011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=618cf89fd5fb28e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.537+0000 7fc4a7e00700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4880b2a20 0x7fc4880b4f30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009900 tx=0x7fc4a0006020 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.537+0000 7fc49cc00700 1 -- 172.21.15.44:0/2971721810 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498004230 con 0x55d96ea84af0 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.537+0000 7fc49cc00700 1 -- 172.21.15.44:0/2971721810 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4980043d0 con 0x55d96ea84af0 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.537+0000 7fc49cc00700 1 -- 172.21.15.44:0/2971721810 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498019920 con 0x55d96ea84af0 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.538+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2971721810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 msgr2=0x55d96eb1e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.538+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2971721810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb1e530 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc498009d80 tx=0x7fc498011720 comp rx=0 tx=0).stop 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.538+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2971721810 shutdown_connections 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.538+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2971721810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96eb185e0 0x55d96eb189f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.538+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2971721810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb1e530 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.538+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2971721810 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 0x55d96eb1ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.538+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2971721810 >> 172.21.15.44:0/2971721810 conn(0x55d96ea32490 msgr2=0x55d96eb13650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:46.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.538+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2971721810 shutdown_connections 2024-05-27T09:45:46.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.538+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2971721810 wait complete. 2024-05-27T09:45:46.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.539+0000 7fc4bd11eb80 1 -- start start 2024-05-27T09:45:46.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.539+0000 7fc4bd11eb80 1 Processor -- start 2024-05-27T09:45:46.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 0x55d96eb222c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb22800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96eb185e0 0x55d96eb22d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d96e93c0a0 con 0x55d96e998f40 2024-05-27T09:45:46.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d96e9d6860 con 0x55d96eb185e0 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d96eb193d0 con 0x55d96ea84af0 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc4bd11eb80 10 client.0 _ll_register_callbacks cb 0x55d96e9dfd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc4a7400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 0x55d96eb222c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc4a7400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 0x55d96eb222c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57922/0 (socket says 172.21.15.44:57922) 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc4a7400700 1 -- 172.21.15.44:0/2974216505 learned_addr learned my addr 172.21.15.44:0/2974216505 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc49ea00700 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb22800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.540+0000 7fc4a7e00700 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96eb185e0 0x55d96eb22d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.541+0000 7fc4a7400700 1 -- 172.21.15.44:0/2974216505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 msgr2=0x55d96eb22800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.541+0000 7fc4a7400700 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb22800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.541+0000 7fc4a7400700 1 -- 172.21.15.44:0/2974216505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96eb185e0 msgr2=0x55d96eb22d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.541+0000 7fc4a7400700 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96eb185e0 0x55d96eb22d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.541+0000 7fc4a7400700 1 -- 172.21.15.44:0/2974216505 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d96e9859d0 con 0x55d96e998f40 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.541+0000 7fc4a7400700 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 0x55d96eb222c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc498000c00 tx=0x7fc4980039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3044246cd27f048 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.541+0000 7fc495600700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.541+0000 7fc495600700 1 -- 172.21.15.44:0/2974216505 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498027070 con 0x55d96e998f40 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.541+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2974216505 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4980043d0 con 0x55d96e998f40 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.541+0000 7fc495600700 1 -- 172.21.15.44:0/2974216505 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fc498005450 con 0x55d96e998f40 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.542+0000 7fc495600700 1 -- 172.21.15.44:0/2974216505 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc498033050 con 0x55d96e998f40 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.542+0000 7fc4bd11eb80 10 client.17034 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.542+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2974216505 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d96ea06160 con 0x55d96e998f40 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.542+0000 7fc495600700 1 -- 172.21.15.44:0/2974216505 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4980043d0 con 0x55d96e998f40 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.542+0000 7fc495600700 1 -- 172.21.15.44:0/2974216505 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fc498019560 con 0x55d96e998f40 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.543+0000 7fc4bd11eb80 10 client.17034 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:46.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.543+0000 7fc4bd11eb80 10 client.17034 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.543+0000 7fc4bd11eb80 20 client.17034 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.543+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d96eb31f10 0x55d96eb34330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.543+0000 7fc4bd11eb80 4 client.17034 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.543+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2974216505 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d96ea47e90 con 0x55d96eb31f10 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.543+0000 7fc49ea00700 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d96eb31f10 0x55d96eb34330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.544+0000 7fc49ea00700 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d96eb31f10 0x55d96eb34330 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8745e924e0856859 server_cookie=d728ab2c815cad8d in_seq=0 out_seq=0 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.544+0000 7fc495600700 10 client.17034 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc495600700 1 -- 172.21.15.44:0/2974216505 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d96ea47e90 con 0x55d96eb31f10 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc495600700 10 client.17034 handle_command_reply: tid=0 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc4bd11eb80 1 client.17034 shutdown 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2974216505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 msgr2=0x55d96eb222c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 0x55d96eb222c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc498000c00 tx=0x7fc4980039e0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2974216505 shutdown_connections 2024-05-27T09:45:46.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d96eb31f10 0x55d96eb34330 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96eb185e0 0x55d96eb22d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96e998f40 0x55d96eb222c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2974216505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb22800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc495600700 0 client.17034 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2974216505 >> 172.21.15.44:0/2974216505 conn(0x55d96ea32490 msgr2=0x55d96eb129d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2974216505 shutdown_connections 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.545+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2974216505 wait complete. 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.546+0000 7fc4bd11eb80 20 client.17034 trim_cache size 0 max 16384 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.568+0000 7fc4bd11eb80 1 Processor -- start 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.568+0000 7fc4bd11eb80 1 -- start start 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.568+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 0x55d96eb227b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.568+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb22cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.568+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96eb185e0 0x55d96eb12fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.568+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a81b3a10 con 0x55d96eb185e0 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.568+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a81b1270 con 0x55d96e998f40 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.568+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a81af0e0 con 0x55d96ea84af0 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.569+0000 7fc4a7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96eb185e0 0x55d96eb12fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.569+0000 7fc4a7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 0x55d96eb227b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.569+0000 7fc49ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb22cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.569+0000 7fc4a7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 0x55d96eb227b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34662/0 (socket says 172.21.15.44:34662) 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.569+0000 7fc4a7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96eb185e0 0x55d96eb12fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57924/0 (socket says 172.21.15.44:57924) 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.569+0000 7fc4a7400700 1 -- 172.21.15.44:0/2634215352 learned_addr learned my addr 172.21.15.44:0/2634215352 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.569+0000 7fc4a7400700 1 -- 172.21.15.44:0/2634215352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 msgr2=0x55d96eb22cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.569+0000 7fc4a7400700 1 --2- 172.21.15.44:0/2634215352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb22cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.569+0000 7fc4a7400700 1 -- 172.21.15.44:0/2634215352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96eb185e0 msgr2=0x55d96eb12fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.569+0000 7fc4a7400700 1 --2- 172.21.15.44:0/2634215352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96eb185e0 0x55d96eb12fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.569+0000 7fc4a7400700 1 -- 172.21.15.44:0/2634215352 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d96eb0f940 con 0x55d96e998f40 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.570+0000 7fc4a7400700 1 --2- 172.21.15.44:0/2634215352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 0x55d96eb227b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc498011760 tx=0x7fc498019d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9337e485281a7fff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.570+0000 7fc497e00700 1 -- 172.21.15.44:0/2634215352 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498027070 con 0x55d96e998f40 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.570+0000 7fc497e00700 1 -- 172.21.15.44:0/2634215352 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc498003d00 con 0x55d96e998f40 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.570+0000 7fc497e00700 1 -- 172.21.15.44:0/2634215352 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4980236e0 con 0x55d96e998f40 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.571+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2634215352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 msgr2=0x55d96eb227b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.571+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2634215352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 0x55d96eb227b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc498011760 tx=0x7fc498019d60 comp rx=0 tx=0).stop 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.571+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2634215352 shutdown_connections 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.571+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2634215352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96eb185e0 0x55d96eb12fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.571+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2634215352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 0x55d96eb227b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.571+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/2634215352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96ea84af0 0x55d96eb22cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.571+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2634215352 >> 172.21.15.44:0/2634215352 conn(0x55d96ea32490 msgr2=0x55d96eb220a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.571+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2634215352 shutdown_connections 2024-05-27T09:45:46.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.571+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/2634215352 wait complete. 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.572+0000 7fc4bd11eb80 1 -- start start 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.572+0000 7fc4bd11eb80 1 Processor -- start 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 0x55d96eb2a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96ea84af0 0x55d96eb2a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc4bd11eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96eb185e0 0x55d96eb273d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a81af730 con 0x55d96ea84af0 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a81b0ba0 con 0x55d96e998f40 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc4bd11eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a81afe00 con 0x55d96eb185e0 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc4bd11eb80 10 client.0 _ll_register_callbacks cb 0x55d96e9dfd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc49ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96ea84af0 0x55d96eb2a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc4a7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96eb185e0 0x55d96eb273d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc4a7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 0x55d96eb2a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc49ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96ea84af0 0x55d96eb2a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57926/0 (socket says 172.21.15.44:57926) 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc4a7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96eb185e0 0x55d96eb273d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:33974/0 (socket says 172.21.15.44:33974) 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.573+0000 7fc49ea00700 1 -- 172.21.15.44:0/1051892498 learned_addr learned my addr 172.21.15.44:0/1051892498 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc4a7400700 1 -- 172.21.15.44:0/1051892498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96eb185e0 msgr2=0x55d96eb273d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc4a7400700 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96eb185e0 0x55d96eb273d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc4a7400700 1 -- 172.21.15.44:0/1051892498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96ea84af0 msgr2=0x55d96eb2a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc4a7400700 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96ea84af0 0x55d96eb2a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc4a7400700 1 -- 172.21.15.44:0/1051892498 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d96eb27910 con 0x55d96e998f40 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc49ea00700 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96ea84af0 0x55d96eb2a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc4a7400700 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 0x55d96eb2a400 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc498011760 tx=0x7fc498019630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b5e3e1ba308b31 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc495600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc495600700 1 -- 172.21.15.44:0/1051892498 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498027040 con 0x55d96e998f40 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/1051892498 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc498003d00 con 0x55d96e998f40 2024-05-27T09:45:46.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc495600700 1 -- 172.21.15.44:0/1051892498 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fc498023460 con 0x55d96e998f40 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.574+0000 7fc495600700 1 -- 172.21.15.44:0/1051892498 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc498005460 con 0x55d96e998f40 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.575+0000 7fc4bd11eb80 10 client.27254 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.575+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/1051892498 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d96eb9cf50 con 0x55d96e998f40 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.575+0000 7fc495600700 1 -- 172.21.15.44:0/1051892498 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc498003d00 con 0x55d96e998f40 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.575+0000 7fc495600700 1 -- 172.21.15.44:0/1051892498 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fc498023cc0 con 0x55d96e998f40 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.575+0000 7fc4bd11eb80 10 client.27254 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.575+0000 7fc4bd11eb80 10 client.27254 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.575+0000 7fc4bd11eb80 20 client.27254 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.575+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d96eb9d540 0x55d96eb9f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.575+0000 7fc4bd11eb80 4 client.27254 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "2a69c329", "release": true, "await": true}] 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.575+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/1051892498 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "set_id": "2a69c329", "release": true, "await": true}) -- 0x55d96eb0b810 con 0x55d96eb9d540 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.576+0000 7fc49ea00700 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d96eb9d540 0x55d96eb9f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:46.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.576+0000 7fc49ea00700 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d96eb9d540 0x55d96eb9f960 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5890935895324286 server_cookie=47741e0992f0db23 in_seq=0 out_seq=0 2024-05-27T09:45:46.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:46.576+0000 7fc495600700 10 client.27254 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:47.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.006+0000 7fc495600700 1 -- 172.21.15.44:0/1051892498 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d96eb0b810 con 0x55d96eb9d540 2024-05-27T09:45:47.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.006+0000 7fc495600700 10 client.27254 handle_command_reply: tid=0 2024-05-27T09:45:47.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.006+0000 7fc4bd11eb80 1 client.27254 shutdown 2024-05-27T09:45:47.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.007+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/1051892498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 msgr2=0x55d96eb2a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.007+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 0x55d96eb2a400 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc498011760 tx=0x7fc498019630 comp rx=0 tx=0).stop 2024-05-27T09:45:47.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.007+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/1051892498 shutdown_connections 2024-05-27T09:45:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.007+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55d96eb9d540 0x55d96eb9f960 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.007+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d96eb185e0 0x55d96eb273d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.007+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d96e998f40 0x55d96eb2a400 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.007+0000 7fc4bd11eb80 1 --2- 172.21.15.44:0/1051892498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d96ea84af0 0x55d96eb2a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.007+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/1051892498 >> 172.21.15.44:0/1051892498 conn(0x55d96ea32490 msgr2=0x55d96eb22ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.007+0000 7fc495600700 0 client.27254 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.008+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/1051892498 shutdown_connections 2024-05-27T09:45:47.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.008+0000 7fc4bd11eb80 1 -- 172.21.15.44:0/1051892498 wait complete. 2024-05-27T09:45:47.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.009+0000 7fc4bd11eb80 20 client.27254 trim_cache size 0 max 16384 2024-05-27T09:45:47.010 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:45:47.010 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 33, 2024-05-27T09:45:47.010 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:45:47.010 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 25, 2024-05-27T09:45:47.010 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "2a69c329": { 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 25, 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 138.9, 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:45:47.011 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:45:47.012 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:45:47.012 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:45:47.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:47.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:47.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:47.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:47.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:47.012 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.010+0000 7fc4ae600700 1 -- 172.21.15.44:0/1879778685 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4880b2a20 msgr2=0x7fc4880b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.010+0000 7fc4ae600700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4880b2a20 0x7fc4880b4f30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009900 tx=0x7fc4a0006020 comp rx=0 tx=0).stop 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.010+0000 7fc4ae600700 1 -- 172.21.15.44:0/1879778685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4a8108ef0 msgr2=0x7fc4a819d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.010+0000 7fc4ae600700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4a8108ef0 0x7fc4a819d590 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc49000e970 tx=0x7fc49000ec80 comp rx=0 tx=0).stop 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.010+0000 7fc4ae600700 1 -- 172.21.15.44:0/1879778685 shutdown_connections 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.010+0000 7fc4ae600700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc4880b2a20 0x7fc4880b4f30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.010+0000 7fc4ae600700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8109840 0x7fc4a81a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.010+0000 7fc4ae600700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a81045c0 0x7fc4a819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.010+0000 7fc4ae600700 1 --2- 172.21.15.44:0/1879778685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4a8108ef0 0x7fc4a819d590 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.010+0000 7fc4ae600700 1 -- 172.21.15.44:0/1879778685 >> 172.21.15.44:0/1879778685 conn(0x7fc4a80ffe90 msgr2=0x7fc4a8110e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.011+0000 7fc4ae600700 1 -- 172.21.15.44:0/1879778685 shutdown_connections 2024-05-27T09:45:47.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.011+0000 7fc4ae600700 1 -- 172.21.15.44:0/1879778685 wait complete. 2024-05-27T09:45:47.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.647+0000 7fb512a00700 1 -- 172.21.15.44:0/2292531614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb50c110960 msgr2=0x7fb50c110d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.647+0000 7fb512a00700 1 --2- 172.21.15.44:0/2292531614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb50c110960 0x7fb50c110d70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009990 tx=0x7fb4fc009ca0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.647+0000 7fb512a00700 1 -- 172.21.15.44:0/2292531614 shutdown_connections 2024-05-27T09:45:47.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.647+0000 7fb512a00700 1 --2- 172.21.15.44:0/2292531614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c10b2d0 0x7fb50c117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.647+0000 7fb512a00700 1 --2- 172.21.15.44:0/2292531614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb50c110960 0x7fb50c110d70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.647+0000 7fb512a00700 1 --2- 172.21.15.44:0/2292531614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb50c10a980 0x7fb50c10ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.647+0000 7fb512a00700 1 -- 172.21.15.44:0/2292531614 >> 172.21.15.44:0/2292531614 conn(0x7fb50c077c70 msgr2=0x7fb50c0780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:47.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.648+0000 7fb512a00700 1 -- 172.21.15.44:0/2292531614 shutdown_connections 2024-05-27T09:45:47.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.648+0000 7fb512a00700 1 -- 172.21.15.44:0/2292531614 wait complete. 2024-05-27T09:45:47.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.649+0000 7fb512a00700 1 Processor -- start 2024-05-27T09:45:47.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.650+0000 7fb512a00700 1 -- start start 2024-05-27T09:45:47.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.650+0000 7fb512a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c10a980 0x7fb50c1a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.650+0000 7fb512a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb50c10b2d0 0x7fb50c1a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.650+0000 7fb512a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb50c110960 0x7fb50c1a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.650+0000 7fb512a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c11aac0 con 0x7fb50c10a980 2024-05-27T09:45:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.650+0000 7fb512a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c11a940 con 0x7fb50c10b2d0 2024-05-27T09:45:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.650+0000 7fb512a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c11ac40 con 0x7fb50c110960 2024-05-27T09:45:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.651+0000 7fb50b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb50c10b2d0 0x7fb50c1a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.651+0000 7fb50b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb50c10b2d0 0x7fb50c1a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34688/0 (socket says 172.21.15.44:34688) 2024-05-27T09:45:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.651+0000 7fb50b400700 1 -- 172.21.15.44:0/2802761690 learned_addr learned my addr 172.21.15.44:0/2802761690 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.651+0000 7fb510a00700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb50c110960 0x7fb50c1a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.651+0000 7fb50be00700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c10a980 0x7fb50c1a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.651+0000 7fb50b400700 1 -- 172.21.15.44:0/2802761690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb50c110960 msgr2=0x7fb50c1a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.651+0000 7fb50b400700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb50c110960 0x7fb50c1a5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.651+0000 7fb50b400700 1 -- 172.21.15.44:0/2802761690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c10a980 msgr2=0x7fb50c1a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.651+0000 7fb50b400700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c10a980 0x7fb50c1a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.651+0000 7fb50b400700 1 -- 172.21.15.44:0/2802761690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50c1a6340 con 0x7fb50c10b2d0 2024-05-27T09:45:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.652+0000 7fb50b400700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb50c10b2d0 0x7fb50c1a1a10 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800e970 tx=0x7fb4f800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.652+0000 7fb508c00700 1 -- 172.21.15.44:0/2802761690 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8004a30 con 0x7fb50c10b2d0 2024-05-27T09:45:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.652+0000 7fb508c00700 1 -- 172.21.15.44:0/2802761690 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4f800cc90 con 0x7fb50c10b2d0 2024-05-27T09:45:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.652+0000 7fb512a00700 1 -- 172.21.15.44:0/2802761690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb50c1a6630 con 0x7fb50c10b2d0 2024-05-27T09:45:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.652+0000 7fb508c00700 1 -- 172.21.15.44:0/2802761690 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8005610 con 0x7fb50c10b2d0 2024-05-27T09:45:47.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.652+0000 7fb512a00700 1 -- 172.21.15.44:0/2802761690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb50c1adf50 con 0x7fb50c10b2d0 2024-05-27T09:45:47.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.657+0000 7fb508c00700 1 -- 172.21.15.44:0/2802761690 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb4f800c7e0 con 0x7fb50c10b2d0 2024-05-27T09:45:47.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.657+0000 7fb521488b80 1 Processor -- start 2024-05-27T09:45:47.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.658+0000 7fb521488b80 1 -- start start 2024-05-27T09:45:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.658+0000 7fb521488b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 0x5576772567f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.658+0000 7fb521488b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677256d30 0x557677250870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.658+0000 7fb521488b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557677250db0 0x5576772511c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.658+0000 7fb521488b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55767717ee40 con 0x5576770d1e40 2024-05-27T09:45:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.658+0000 7fb521488b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55767716e9e0 con 0x557677250db0 2024-05-27T09:45:47.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.658+0000 7fb521488b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557677172a20 con 0x557677256d30 2024-05-27T09:45:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.658+0000 7fb510a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 0x5576772567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.658+0000 7fb50be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677256d30 0x557677250870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.658+0000 7fb510a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 0x5576772567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57946/0 (socket says 172.21.15.44:57946) 2024-05-27T09:45:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.659+0000 7fb510a00700 1 -- 172.21.15.44:0/339521296 learned_addr learned my addr 172.21.15.44:0/339521296 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.659+0000 7fb508c00700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb4f00b29f0 0x7fb4f00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.659+0000 7fb50b400700 1 --2- 172.21.15.44:0/339521296 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557677250db0 0x5576772511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.659+0000 7fb510a00700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb4f00b29f0 0x7fb4f00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.659+0000 7fb508c00700 1 -- 172.21.15.44:0/2802761690 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fb4f80e2850 con 0x7fb50c10b2d0 2024-05-27T09:45:47.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.659+0000 7fb50be00700 1 -- 172.21.15.44:0/339521296 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557677250db0 msgr2=0x5576772511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.659+0000 7fb50be00700 1 --2- 172.21.15.44:0/339521296 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557677250db0 0x5576772511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.659+0000 7fb50be00700 1 -- 172.21.15.44:0/339521296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 msgr2=0x5576772567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.659+0000 7fb50be00700 1 --2- 172.21.15.44:0/339521296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 0x5576772567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.659+0000 7fb50be00700 1 -- 172.21.15.44:0/339521296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557677137420 con 0x557677256d30 2024-05-27T09:45:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.659+0000 7fb510a00700 1 --2- 172.21.15.44:0/339521296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 0x5576772567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.660+0000 7fb50be00700 1 --2- 172.21.15.44:0/339521296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677256d30 0x557677250870 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0098e0 tx=0x7fb4fc0096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=308aed461f941d53 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.660+0000 7fb501600700 1 -- 172.21.15.44:0/339521296 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc027070 con 0x557677256d30 2024-05-27T09:45:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.660+0000 7fb501600700 1 -- 172.21.15.44:0/339521296 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4fc0056b0 con 0x557677256d30 2024-05-27T09:45:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.660+0000 7fb510a00700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb4f00b29f0 0x7fb4f00b4f00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x557677251b30 tx=0x7fb504004b40 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:47.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.660+0000 7fb501600700 1 -- 172.21.15.44:0/339521296 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc003b80 con 0x557677256d30 2024-05-27T09:45:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.661+0000 7fb521488b80 1 -- 172.21.15.44:0/339521296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677256d30 msgr2=0x557677250870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.661+0000 7fb521488b80 1 --2- 172.21.15.44:0/339521296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677256d30 0x557677250870 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0098e0 tx=0x7fb4fc0096a0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.661+0000 7fb521488b80 1 -- 172.21.15.44:0/339521296 shutdown_connections 2024-05-27T09:45:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.661+0000 7fb521488b80 1 --2- 172.21.15.44:0/339521296 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557677250db0 0x5576772511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.661+0000 7fb521488b80 1 --2- 172.21.15.44:0/339521296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 0x5576772567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.661+0000 7fb521488b80 1 --2- 172.21.15.44:0/339521296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677256d30 0x557677250870 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.661+0000 7fb521488b80 1 -- 172.21.15.44:0/339521296 >> 172.21.15.44:0/339521296 conn(0x557677131fc0 msgr2=0x55767724b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:47.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.661+0000 7fb521488b80 1 -- 172.21.15.44:0/339521296 shutdown_connections 2024-05-27T09:45:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.662+0000 7fb521488b80 1 -- 172.21.15.44:0/339521296 wait complete. 2024-05-27T09:45:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.662+0000 7fb521488b80 1 -- start start 2024-05-27T09:45:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.662+0000 7fb521488b80 1 Processor -- start 2024-05-27T09:45:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.663+0000 7fb521488b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 0x55767725a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.663+0000 7fb521488b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.663+0000 7fb521488b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557677256d30 0x55767725af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.663+0000 7fb521488b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557677074fa0 con 0x5576770d1e40 2024-05-27T09:45:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.663+0000 7fb521488b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55767710f760 con 0x557677256d30 2024-05-27T09:45:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.663+0000 7fb521488b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55767725d7a0 con 0x557677250db0 2024-05-27T09:45:47.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.663+0000 7fb521488b80 10 client.0 _ll_register_callbacks cb 0x557677118c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:47.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb50b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb50b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34014/0 (socket says 172.21.15.44:34014) 2024-05-27T09:45:47.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb50b400700 1 -- 172.21.15.44:0/3747240008 learned_addr learned my addr 172.21.15.44:0/3747240008 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:47.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb510a00700 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557677256d30 0x55767725af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb50be00700 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 0x55767725a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb50b400700 1 -- 172.21.15.44:0/3747240008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557677256d30 msgr2=0x55767725af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb50b400700 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557677256d30 0x55767725af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb50b400700 1 -- 172.21.15.44:0/3747240008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 msgr2=0x55767725a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb50b400700 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 0x55767725a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb510a00700 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557677256d30 0x55767725af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:45:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb50b400700 1 -- 172.21.15.44:0/3747240008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576770be8d0 con 0x557677250db0 2024-05-27T09:45:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb50b400700 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725aa50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb4f80a5b50 tx=0x7fb4f8004e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d171839221272587 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb4f6000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:45:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.664+0000 7fb4f6000700 1 -- 172.21.15.44:0/3747240008 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f800ce40 con 0x557677250db0 2024-05-27T09:45:47.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb4f6000700 1 -- 172.21.15.44:0/3747240008 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fb4f80aae00 con 0x557677250db0 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb521488b80 1 -- 172.21.15.44:0/3747240008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4fc0056b0 con 0x557677250db0 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb4f6000700 1 -- 172.21.15.44:0/3747240008 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb4f80a95e0 con 0x557677250db0 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb521488b80 10 client.27256 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb521488b80 1 -- 172.21.15.44:0/3747240008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55767713f060 con 0x557677250db0 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb4f6000700 1 -- 172.21.15.44:0/3747240008 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4fc0056b0 con 0x557677250db0 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb4f6000700 1 -- 172.21.15.44:0/3747240008 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fb4f80b24a0 con 0x557677250db0 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb521488b80 10 client.27256 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb521488b80 10 client.27256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb521488b80 20 client.27256 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb521488b80 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55767726a600 0x55767726ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb521488b80 4 client.27256 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.665+0000 7fb521488b80 1 -- 172.21.15.44:0/3747240008 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557677180d90 con 0x55767726a600 2024-05-27T09:45:47.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.666+0000 7fb50be00700 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55767726a600 0x55767726ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.666+0000 7fb50be00700 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55767726a600 0x55767726ca20 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d69de7dbd74e063a server_cookie=d5d350c1f16c09e6 in_seq=0 out_seq=0 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.666+0000 7fb4f6000700 10 client.27256 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb4f6000700 1 -- 172.21.15.44:0/3747240008 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557677180d90 con 0x55767726a600 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb4f6000700 10 client.27256 handle_command_reply: tid=0 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb521488b80 1 client.27256 shutdown 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb521488b80 1 -- 172.21.15.44:0/3747240008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 msgr2=0x55767725aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb521488b80 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725aa50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb4f80a5b50 tx=0x7fb4f8004e90 comp rx=0 tx=0).stop 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb521488b80 1 -- 172.21.15.44:0/3747240008 shutdown_connections 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb521488b80 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55767726a600 0x55767726ca20 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb521488b80 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557677256d30 0x55767725af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb521488b80 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5576770d1e40 0x55767725a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb521488b80 1 --2- 172.21.15.44:0/3747240008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725aa50 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb4f6000700 0 client.27256 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.667+0000 7fb521488b80 1 -- 172.21.15.44:0/3747240008 >> 172.21.15.44:0/3747240008 conn(0x557677131fc0 msgr2=0x55767724b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.668+0000 7fb521488b80 1 -- 172.21.15.44:0/3747240008 shutdown_connections 2024-05-27T09:45:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.668+0000 7fb521488b80 1 -- 172.21.15.44:0/3747240008 wait complete. 2024-05-27T09:45:47.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.668+0000 7fb521488b80 20 client.27256 trim_cache size 0 max 16384 2024-05-27T09:45:47.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.685+0000 7fb521488b80 1 Processor -- start 2024-05-27T09:45:47.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.685+0000 7fb521488b80 1 -- start start 2024-05-27T09:45:47.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.685+0000 7fb521488b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5576770d1e40 0x557677250870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.685+0000 7fb521488b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.685+0000 7fb521488b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557677256d30 0x55767725c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.685+0000 7fb521488b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c1b7d50 con 0x557677256d30 2024-05-27T09:45:47.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.685+0000 7fb521488b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c1b55b0 con 0x5576770d1e40 2024-05-27T09:45:47.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.685+0000 7fb521488b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c1b3420 con 0x557677250db0 2024-05-27T09:45:47.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb50b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb510a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557677256d30 0x55767725c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb50be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5576770d1e40 0x557677250870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb50b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34016/0 (socket says 172.21.15.44:34016) 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb50b400700 1 -- 172.21.15.44:0/591202999 learned_addr learned my addr 172.21.15.44:0/591202999 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb50b400700 1 -- 172.21.15.44:0/591202999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5576770d1e40 msgr2=0x557677250870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb50b400700 1 --2- 172.21.15.44:0/591202999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5576770d1e40 0x557677250870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb50b400700 1 -- 172.21.15.44:0/591202999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557677256d30 msgr2=0x55767725c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb50b400700 1 --2- 172.21.15.44:0/591202999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557677256d30 0x55767725c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb50b400700 1 -- 172.21.15.44:0/591202999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557677254f90 con 0x557677250db0 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb50be00700 1 --2- 172.21.15.44:0/591202999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5576770d1e40 0x557677250870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb50b400700 1 --2- 172.21.15.44:0/591202999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725af00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8004800 tx=0x7fb4f80a9550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a82e8a906e70ee1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.686+0000 7fb500c00700 1 -- 172.21.15.44:0/591202999 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f800ce40 con 0x557677250db0 2024-05-27T09:45:47.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.687+0000 7fb500c00700 1 -- 172.21.15.44:0/591202999 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4f80c2020 con 0x557677250db0 2024-05-27T09:45:47.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.687+0000 7fb500c00700 1 -- 172.21.15.44:0/591202999 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f80ade80 con 0x557677250db0 2024-05-27T09:45:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.687+0000 7fb521488b80 1 -- 172.21.15.44:0/591202999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 msgr2=0x55767725af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.687+0000 7fb521488b80 1 --2- 172.21.15.44:0/591202999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725af00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8004800 tx=0x7fb4f80a9550 comp rx=0 tx=0).stop 2024-05-27T09:45:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.688+0000 7fb521488b80 1 -- 172.21.15.44:0/591202999 shutdown_connections 2024-05-27T09:45:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.688+0000 7fb521488b80 1 --2- 172.21.15.44:0/591202999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557677256d30 0x55767725c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.688+0000 7fb521488b80 1 --2- 172.21.15.44:0/591202999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5576770d1e40 0x557677250870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.688+0000 7fb521488b80 1 --2- 172.21.15.44:0/591202999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x55767725af00 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.688+0000 7fb521488b80 1 -- 172.21.15.44:0/591202999 >> 172.21.15.44:0/591202999 conn(0x557677131fc0 msgr2=0x557677254880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.688+0000 7fb521488b80 1 -- 172.21.15.44:0/591202999 shutdown_connections 2024-05-27T09:45:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.688+0000 7fb521488b80 1 -- 172.21.15.44:0/591202999 wait complete. 2024-05-27T09:45:47.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.688+0000 7fb521488b80 1 -- start start 2024-05-27T09:45:47.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.689+0000 7fb521488b80 1 Processor -- start 2024-05-27T09:45:47.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.689+0000 7fb521488b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5576770d1e40 0x557677262c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.689+0000 7fb521488b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x557677263190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.689+0000 7fb521488b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557677256d30 0x557677271610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.689+0000 7fb521488b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c1b3a70 con 0x557677256d30 2024-05-27T09:45:47.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.689+0000 7fb521488b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c1b4ee0 con 0x5576770d1e40 2024-05-27T09:45:47.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.689+0000 7fb521488b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c1b4140 con 0x557677250db0 2024-05-27T09:45:47.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.689+0000 7fb521488b80 10 client.0 _ll_register_callbacks cb 0x557677118c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.689+0000 7fb50b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x557677263190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.689+0000 7fb50b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x557677263190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34018/0 (socket says 172.21.15.44:34018) 2024-05-27T09:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.689+0000 7fb50b400700 1 -- 172.21.15.44:0/3691944694 learned_addr learned my addr 172.21.15.44:0/3691944694 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.690+0000 7fb510a00700 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557677256d30 0x557677271610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.690+0000 7fb50be00700 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5576770d1e40 0x557677262c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.690+0000 7fb50b400700 1 -- 172.21.15.44:0/3691944694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5576770d1e40 msgr2=0x557677262c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.690+0000 7fb50b400700 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5576770d1e40 0x557677262c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.690+0000 7fb50b400700 1 -- 172.21.15.44:0/3691944694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557677256d30 msgr2=0x557677271610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.690+0000 7fb50b400700 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557677256d30 0x557677271610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.690+0000 7fb50b400700 1 -- 172.21.15.44:0/3691944694 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557677271c50 con 0x557677250db0 2024-05-27T09:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.690+0000 7fb510a00700 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557677256d30 0x557677271610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.690+0000 7fb50b400700 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x557677263190 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8000c00 tx=0x7fb4f806f4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28ba03fa75cf02fc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.690+0000 7fb4f6000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.690+0000 7fb4f6000700 1 -- 172.21.15.44:0/3691944694 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f80c9070 con 0x557677250db0 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb4f6000700 1 -- 172.21.15.44:0/3691944694 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fb4f80aae00 con 0x557677250db0 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb521488b80 1 -- 172.21.15.44:0/3691944694 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4f80c2020 con 0x557677250db0 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb4f6000700 1 -- 172.21.15.44:0/3691944694 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb4f800ce40 con 0x557677250db0 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb521488b80 10 client.27262 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb521488b80 1 -- 172.21.15.44:0/3691944694 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5576772d4990 con 0x557677250db0 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb4f6000700 1 -- 172.21.15.44:0/3691944694 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4f80c2020 con 0x557677250db0 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb4f6000700 1 -- 172.21.15.44:0/3691944694 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fb4f80b2b80 con 0x557677250db0 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb521488b80 10 client.27262 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb521488b80 10 client.27262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb521488b80 20 client.27262 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb521488b80 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5576772d4e30 0x5576772d7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb521488b80 4 client.27262 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.691+0000 7fb521488b80 1 -- 172.21.15.44:0/3691944694 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55767717a570 con 0x5576772d4e30 2024-05-27T09:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.692+0000 7fb50be00700 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5576772d4e30 0x5576772d7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.692+0000 7fb50be00700 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5576772d4e30 0x5576772d7250 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0139b548e2a7970 server_cookie=680f96678e7a2367 in_seq=0 out_seq=0 2024-05-27T09:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.692+0000 7fb4f6000700 10 client.27262 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.692+0000 7fb4f6000700 1 -- 172.21.15.44:0/3691944694 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55767717a570 con 0x5576772d4e30 2024-05-27T09:45:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.692+0000 7fb4f6000700 10 client.27262 handle_command_reply: tid=0 2024-05-27T09:45:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.693+0000 7fb521488b80 1 client.27262 shutdown 2024-05-27T09:45:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.693+0000 7fb521488b80 1 -- 172.21.15.44:0/3691944694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 msgr2=0x557677263190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.693+0000 7fb521488b80 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x557677263190 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8000c00 tx=0x7fb4f806f4a0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.693+0000 7fb521488b80 1 -- 172.21.15.44:0/3691944694 shutdown_connections 2024-05-27T09:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.693+0000 7fb521488b80 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5576772d4e30 0x5576772d7250 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.693+0000 7fb521488b80 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557677256d30 0x557677271610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.693+0000 7fb521488b80 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5576770d1e40 0x557677262c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.693+0000 7fb521488b80 1 --2- 172.21.15.44:0/3691944694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557677250db0 0x557677263190 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.693+0000 7fb521488b80 1 -- 172.21.15.44:0/3691944694 >> 172.21.15.44:0/3691944694 conn(0x557677131fc0 msgr2=0x557677254880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.693+0000 7fb4f6000700 0 client.27262 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.694+0000 7fb521488b80 1 -- 172.21.15.44:0/3691944694 shutdown_connections 2024-05-27T09:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.694+0000 7fb521488b80 1 -- 172.21.15.44:0/3691944694 wait complete. 2024-05-27T09:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.695+0000 7fb521488b80 20 client.27262 trim_cache size 0 max 16384 2024-05-27T09:45:47.697 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:45:47.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.695+0000 7fb512a00700 1 -- 172.21.15.44:0/2802761690 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb4f00b29f0 msgr2=0x7fb4f00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.695+0000 7fb512a00700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb4f00b29f0 0x7fb4f00b4f00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x557677251b30 tx=0x7fb504004b40 comp rx=0 tx=0).stop 2024-05-27T09:45:47.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.695+0000 7fb512a00700 1 -- 172.21.15.44:0/2802761690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb50c10b2d0 msgr2=0x7fb50c1a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:47.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.695+0000 7fb512a00700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb50c10b2d0 0x7fb50c1a1a10 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800e970 tx=0x7fb4f800ec80 comp rx=0 tx=0).stop 2024-05-27T09:45:47.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.695+0000 7fb512a00700 1 -- 172.21.15.44:0/2802761690 shutdown_connections 2024-05-27T09:45:47.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.695+0000 7fb512a00700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb4f00b29f0 0x7fb4f00b4f00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.695+0000 7fb512a00700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb50c110960 0x7fb50c1a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.695+0000 7fb512a00700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb50c10a980 0x7fb50c1a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.695+0000 7fb512a00700 1 --2- 172.21.15.44:0/2802761690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb50c10b2d0 0x7fb50c1a1a10 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.695+0000 7fb512a00700 1 -- 172.21.15.44:0/2802761690 >> 172.21.15.44:0/2802761690 conn(0x7fb50c077c70 msgr2=0x7fb50c07bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.696+0000 7fb512a00700 1 -- 172.21.15.44:0/2802761690 shutdown_connections 2024-05-27T09:45:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:47.696+0000 7fb512a00700 1 -- 172.21.15.44:0/2802761690 wait complete. 2024-05-27T09:45:47.954 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '2a69c329', total seconds elapsed: 30.3 2024-05-27T09:45:48.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:48 smithi044 ceph-mon[31618]: pgmap v329: 97 pgs: 97 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 204 KiB/s wr, 23 op/s 2024-05-27T09:45:48.389 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:48 smithi189 ceph-mon[36671]: pgmap v329: 97 pgs: 97 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 204 KiB/s wr, 23 op/s 2024-05-27T09:45:48.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:48 smithi050 ceph-mon[36371]: pgmap v329: 97 pgs: 97 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 204 KiB/s wr, 23 op/s 2024-05-27T09:45:49.785 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:45:49.786 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:45:50.245 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:50.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:50 smithi044 ceph-mon[31618]: pgmap v330: 97 pgs: 97 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 173 KiB/s wr, 15 op/s 2024-05-27T09:45:50.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:50 smithi050 ceph-mon[36371]: pgmap v330: 97 pgs: 97 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 173 KiB/s wr, 15 op/s 2024-05-27T09:45:50.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:50 smithi189 ceph-mon[36671]: pgmap v330: 97 pgs: 97 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 173 KiB/s wr, 15 op/s 2024-05-27T09:45:52.405 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:52 smithi044 ceph-mon[31618]: pgmap v331: 97 pgs: 97 active+clean; 282 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 621 KiB/s wr, 19 op/s 2024-05-27T09:45:52.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:52 smithi050 ceph-mon[36371]: pgmap v331: 97 pgs: 97 active+clean; 282 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 621 KiB/s wr, 19 op/s 2024-05-27T09:45:52.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:52 smithi189 ceph-mon[36671]: pgmap v331: 97 pgs: 97 active+clean; 282 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 621 KiB/s wr, 19 op/s 2024-05-27T09:45:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.661+0000 7f4a44600700 1 -- 172.21.15.44:0/2802831503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3c075d20 msgr2=0x7f4a3c076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.661+0000 7f4a44600700 1 --2- 172.21.15.44:0/2802831503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3c075d20 0x7f4a3c076130 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4a300098f0 tx=0x7f4a30009c00 comp rx=0 tx=0).stop 2024-05-27T09:45:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.661+0000 7f4a44600700 1 -- 172.21.15.44:0/2802831503 shutdown_connections 2024-05-27T09:45:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.661+0000 7f4a44600700 1 --2- 172.21.15.44:0/2802831503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a3c110b90 0x7f4a3c112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.661+0000 7f4a44600700 1 --2- 172.21.15.44:0/2802831503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3c075d20 0x7f4a3c076130 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.661+0000 7f4a44600700 1 --2- 172.21.15.44:0/2802831503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a3c076670 0x7f4a3c110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.661+0000 7f4a44600700 1 -- 172.21.15.44:0/2802831503 >> 172.21.15.44:0/2802831503 conn(0x7f4a3c100030 msgr2=0x7f4a3c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.661+0000 7f4a44600700 1 -- 172.21.15.44:0/2802831503 shutdown_connections 2024-05-27T09:45:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.661+0000 7f4a44600700 1 -- 172.21.15.44:0/2802831503 wait complete. 2024-05-27T09:45:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.663+0000 7f4a44600700 1 Processor -- start 2024-05-27T09:45:52.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.663+0000 7f4a44600700 1 -- start start 2024-05-27T09:45:52.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.663+0000 7f4a44600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a3c075d20 0x7f4a3c19d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.663+0000 7f4a44600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3c076670 0x7f4a3c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.663+0000 7f4a44600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a3c110b90 0x7f4a3c1a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.663+0000 7f4a44600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1166d0 con 0x7f4a3c110b90 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.663+0000 7f4a44600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c116550 con 0x7f4a3c075d20 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.663+0000 7f4a44600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c116850 con 0x7f4a3c076670 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.664+0000 7f4a41c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a3c075d20 0x7f4a3c19d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.664+0000 7f4a41200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3c076670 0x7f4a3c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.664+0000 7f4a41c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a3c075d20 0x7f4a3c19d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34736/0 (socket says 172.21.15.44:34736) 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.664+0000 7f4a41c00700 1 -- 172.21.15.44:0/3373794949 learned_addr learned my addr 172.21.15.44:0/3373794949 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.664+0000 7f4a42600700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a3c110b90 0x7f4a3c1a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.664+0000 7f4a41200700 1 -- 172.21.15.44:0/3373794949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a3c075d20 msgr2=0x7f4a3c19d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.664+0000 7f4a41200700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a3c075d20 0x7f4a3c19d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.664+0000 7f4a41200700 1 -- 172.21.15.44:0/3373794949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a3c110b90 msgr2=0x7f4a3c1a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:52.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.664+0000 7f4a41200700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a3c110b90 0x7f4a3c1a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.664+0000 7f4a41200700 1 -- 172.21.15.44:0/3373794949 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3c1a1f30 con 0x7f4a3c076670 2024-05-27T09:45:52.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.664+0000 7f4a42600700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a3c110b90 0x7f4a3c1a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:45:52.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.665+0000 7f4a41200700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3c076670 0x7f4a3c19d600 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800e9f0 tx=0x7f4a2800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:52.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.665+0000 7f4a3a000700 1 -- 172.21.15.44:0/3373794949 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2800cbd0 con 0x7f4a3c076670 2024-05-27T09:45:52.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.665+0000 7f4a3a000700 1 -- 172.21.15.44:0/3373794949 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a2800cd70 con 0x7f4a3c076670 2024-05-27T09:45:52.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.665+0000 7f4a3a000700 1 -- 172.21.15.44:0/3373794949 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28018680 con 0x7f4a3c076670 2024-05-27T09:45:52.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.665+0000 7f4a44600700 1 -- 172.21.15.44:0/3373794949 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3c1a2220 con 0x7f4a3c076670 2024-05-27T09:45:52.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.666+0000 7f4a44600700 1 -- 172.21.15.44:0/3373794949 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3c1a9b40 con 0x7f4a3c076670 2024-05-27T09:45:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.670+0000 7f4a3a000700 1 -- 172.21.15.44:0/3373794949 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4a28010430 con 0x7f4a3c076670 2024-05-27T09:45:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.670+0000 7f4a53056b80 1 Processor -- start 2024-05-27T09:45:52.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.670+0000 7f4a53056b80 1 -- start start 2024-05-27T09:45:52.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 0x5642c5a11870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a11db0 0x5642c5a0b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a0be30 0x5642c5a0c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a53056b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5642c5939ec0 con 0x5642c5a0be30 2024-05-27T09:45:52.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a53056b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5642c5929a60 con 0x5642c5a11db0 2024-05-27T09:45:52.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a53056b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5642c592daa0 con 0x5642c588cec0 2024-05-27T09:45:52.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a41c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a11db0 0x5642c5a0b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a41c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a11db0 0x5642c5a0b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34740/0 (socket says 172.21.15.44:34740) 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a42600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 0x5642c5a11870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a41c00700 1 -- 172.21.15.44:0/3301970899 learned_addr learned my addr 172.21.15.44:0/3301970899 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a41200700 1 --2- 172.21.15.44:0/3301970899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a0be30 0x5642c5a0c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a41c00700 1 -- 172.21.15.44:0/3301970899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 msgr2=0x5642c5a11870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a41c00700 1 --2- 172.21.15.44:0/3301970899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 0x5642c5a11870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a41c00700 1 -- 172.21.15.44:0/3301970899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a0be30 msgr2=0x5642c5a0c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a41c00700 1 --2- 172.21.15.44:0/3301970899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a0be30 0x5642c5a0c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.671+0000 7f4a41c00700 1 -- 172.21.15.44:0/3301970899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642c58f24a0 con 0x5642c5a11db0 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.672+0000 7f4a41c00700 1 --2- 172.21.15.44:0/3301970899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a11db0 0x5642c5a0b8f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009230 tx=0x7f4a30009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b03b28fb371aafd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.672+0000 7f4a2ea00700 1 -- 172.21.15.44:0/3301970899 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30026070 con 0x5642c5a11db0 2024-05-27T09:45:52.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.672+0000 7f4a2ea00700 1 -- 172.21.15.44:0/3301970899 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a3000f910 con 0x5642c5a11db0 2024-05-27T09:45:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.672+0000 7f4a2ea00700 1 -- 172.21.15.44:0/3301970899 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a300038e0 con 0x5642c5a11db0 2024-05-27T09:45:52.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.672+0000 7f4a3a000700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4a240b2a70 0x7f4a240b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.672+0000 7f4a3a000700 1 -- 172.21.15.44:0/3373794949 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f4a280e27a0 con 0x7f4a3c076670 2024-05-27T09:45:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.673+0000 7f4a53056b80 1 -- 172.21.15.44:0/3301970899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a11db0 msgr2=0x5642c5a0b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.673+0000 7f4a53056b80 1 --2- 172.21.15.44:0/3301970899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a11db0 0x5642c5a0b8f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009230 tx=0x7f4a30009610 comp rx=0 tx=0).stop 2024-05-27T09:45:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.673+0000 7f4a42600700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4a240b2a70 0x7f4a240b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.673+0000 7f4a53056b80 1 -- 172.21.15.44:0/3301970899 shutdown_connections 2024-05-27T09:45:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.673+0000 7f4a53056b80 1 --2- 172.21.15.44:0/3301970899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a0be30 0x5642c5a0c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.673+0000 7f4a53056b80 1 --2- 172.21.15.44:0/3301970899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 0x5642c5a11870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.673+0000 7f4a53056b80 1 --2- 172.21.15.44:0/3301970899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a11db0 0x5642c5a0b8f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.673+0000 7f4a53056b80 1 -- 172.21.15.44:0/3301970899 >> 172.21.15.44:0/3301970899 conn(0x5642c58ed040 msgr2=0x5642c5a06950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:52.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.674+0000 7f4a53056b80 1 -- 172.21.15.44:0/3301970899 shutdown_connections 2024-05-27T09:45:52.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.674+0000 7f4a53056b80 1 -- 172.21.15.44:0/3301970899 wait complete. 2024-05-27T09:45:52.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.674+0000 7f4a42600700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4a240b2a70 0x7f4a240b4f80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4a34009fd0 tx=0x7f4a34009040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:52.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.674+0000 7f4a53056b80 1 -- start start 2024-05-27T09:45:52.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.675+0000 7f4a53056b80 1 Processor -- start 2024-05-27T09:45:52.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.675+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 0x5642c5a15550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.675+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a0be30 0x5642c5a15a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.675+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a11db0 0x5642c5a15fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.675+0000 7f4a53056b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5642c5830020 con 0x5642c5a11db0 2024-05-27T09:45:52.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.675+0000 7f4a53056b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5642c58ca7e0 con 0x5642c5a0be30 2024-05-27T09:45:52.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a53056b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5642c5a18820 con 0x5642c588cec0 2024-05-27T09:45:52.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a53056b80 10 client.0 _ll_register_callbacks cb 0x5642c58d3cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:52.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a41c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 0x5642c5a15550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a41c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 0x5642c5a15550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34050/0 (socket says 172.21.15.44:34050) 2024-05-27T09:45:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a41c00700 1 -- 172.21.15.44:0/4236289331 learned_addr learned my addr 172.21.15.44:0/4236289331 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a41200700 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a0be30 0x5642c5a15a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a42600700 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a11db0 0x5642c5a15fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a41c00700 1 -- 172.21.15.44:0/4236289331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a0be30 msgr2=0x5642c5a15a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:52.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a41c00700 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a0be30 0x5642c5a15a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a41c00700 1 -- 172.21.15.44:0/4236289331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a11db0 msgr2=0x5642c5a15fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a41c00700 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a11db0 0x5642c5a15fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.676+0000 7f4a41c00700 1 -- 172.21.15.44:0/4236289331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642c5879950 con 0x5642c588cec0 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.677+0000 7f4a41c00700 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 0x5642c5a15550 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4a30005620 tx=0x7f4a3000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=983905192e19ed8f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.677+0000 7f4a23400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.677+0000 7f4a23400700 1 -- 172.21.15.44:0/4236289331 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30026040 con 0x5642c588cec0 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.677+0000 7f4a23400700 1 -- 172.21.15.44:0/4236289331 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f4a30005070 con 0x5642c588cec0 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.677+0000 7f4a53056b80 1 -- 172.21.15.44:0/4236289331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a3000f910 con 0x5642c588cec0 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.677+0000 7f4a23400700 1 -- 172.21.15.44:0/4236289331 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4a30037050 con 0x5642c588cec0 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.677+0000 7f4a53056b80 10 client.27274 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.677+0000 7f4a53056b80 1 -- 172.21.15.44:0/4236289331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5642c58fa0e0 con 0x5642c588cec0 2024-05-27T09:45:52.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.678+0000 7f4a23400700 1 -- 172.21.15.44:0/4236289331 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a3000f910 con 0x5642c588cec0 2024-05-27T09:45:52.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.678+0000 7f4a23400700 1 -- 172.21.15.44:0/4236289331 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f4a30032030 con 0x5642c588cec0 2024-05-27T09:45:52.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.678+0000 7f4a53056b80 10 client.27274 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:52.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.678+0000 7f4a53056b80 10 client.27274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:52.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.678+0000 7f4a53056b80 20 client.27274 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.678+0000 7f4a53056b80 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5a2dd80 0x5642c5a301a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.678+0000 7f4a53056b80 4 client.27274 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.678+0000 7f4a53056b80 1 -- 172.21.15.44:0/4236289331 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642c593be10 con 0x5642c5a2dd80 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.678+0000 7f4a41200700 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5a2dd80 0x5642c5a301a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.679+0000 7f4a23400700 10 client.27274 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.679+0000 7f4a41200700 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5a2dd80 0x5642c5a301a0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bd8d49ce17c2760 server_cookie=c2acbb7229dcdd21 in_seq=0 out_seq=0 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.679+0000 7f4a23400700 1 -- 172.21.15.44:0/4236289331 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5642c593be10 con 0x5642c5a2dd80 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.679+0000 7f4a23400700 10 client.27274 handle_command_reply: tid=0 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.680+0000 7f4a53056b80 1 client.27274 shutdown 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.680+0000 7f4a53056b80 1 -- 172.21.15.44:0/4236289331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 msgr2=0x5642c5a15550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.680+0000 7f4a53056b80 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 0x5642c5a15550 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4a30005620 tx=0x7f4a3000b840 comp rx=0 tx=0).stop 2024-05-27T09:45:52.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.680+0000 7f4a53056b80 1 -- 172.21.15.44:0/4236289331 shutdown_connections 2024-05-27T09:45:52.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.680+0000 7f4a53056b80 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5a2dd80 0x5642c5a301a0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.680+0000 7f4a23400700 0 client.27274 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:52.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.680+0000 7f4a53056b80 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a11db0 0x5642c5a15fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.680+0000 7f4a53056b80 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c588cec0 0x5642c5a15550 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.680+0000 7f4a53056b80 1 --2- 172.21.15.44:0/4236289331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c5a0be30 0x5642c5a15a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.680+0000 7f4a53056b80 1 -- 172.21.15.44:0/4236289331 >> 172.21.15.44:0/4236289331 conn(0x5642c58ed040 msgr2=0x5642c5a06600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:52.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.680+0000 7f4a53056b80 1 -- 172.21.15.44:0/4236289331 shutdown_connections 2024-05-27T09:45:52.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.681+0000 7f4a53056b80 1 -- 172.21.15.44:0/4236289331 wait complete. 2024-05-27T09:45:52.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.681+0000 7f4a53056b80 20 client.27274 trim_cache size 0 max 16384 2024-05-27T09:45:52.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.705+0000 7f4a53056b80 1 Processor -- start 2024-05-27T09:45:52.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.705+0000 7f4a53056b80 1 -- start start 2024-05-27T09:45:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a0b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a0be30 0x5642c5a2a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5a11db0 0x5642c5a179f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a53056b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1af490 con 0x5642c5a0be30 2024-05-27T09:45:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a53056b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1accf0 con 0x5642c588cec0 2024-05-27T09:45:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a53056b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c108230 con 0x5642c5a11db0 2024-05-27T09:45:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a41c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a0b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a41c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a0b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34762/0 (socket says 172.21.15.44:34762) 2024-05-27T09:45:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a41200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a0be30 0x5642c5a2a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a41c00700 1 -- 172.21.15.44:0/2203631457 learned_addr learned my addr 172.21.15.44:0/2203631457 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a42600700 1 --2- 172.21.15.44:0/2203631457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5a11db0 0x5642c5a179f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a41c00700 1 -- 172.21.15.44:0/2203631457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5a11db0 msgr2=0x5642c5a179f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a41c00700 1 --2- 172.21.15.44:0/2203631457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5a11db0 0x5642c5a179f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a41c00700 1 -- 172.21.15.44:0/2203631457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a0be30 msgr2=0x5642c5a2a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a41c00700 1 --2- 172.21.15.44:0/2203631457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a0be30 0x5642c5a2a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.706+0000 7f4a41c00700 1 -- 172.21.15.44:0/2203631457 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642c5a16610 con 0x5642c588cec0 2024-05-27T09:45:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.707+0000 7f4a41c00700 1 --2- 172.21.15.44:0/2203631457 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a0b8f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4a300059d0 tx=0x7f4a30005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66e9509ea3b41b1f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.707+0000 7f4a2e000700 1 -- 172.21.15.44:0/2203631457 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30026040 con 0x5642c588cec0 2024-05-27T09:45:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.707+0000 7f4a2e000700 1 -- 172.21.15.44:0/2203631457 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a30003680 con 0x5642c588cec0 2024-05-27T09:45:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.707+0000 7f4a2e000700 1 -- 172.21.15.44:0/2203631457 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3000b960 con 0x5642c588cec0 2024-05-27T09:45:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.708+0000 7f4a53056b80 1 -- 172.21.15.44:0/2203631457 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 msgr2=0x5642c5a0b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.708+0000 7f4a53056b80 1 --2- 172.21.15.44:0/2203631457 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a0b8f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4a300059d0 tx=0x7f4a30005800 comp rx=0 tx=0).stop 2024-05-27T09:45:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.708+0000 7f4a53056b80 1 -- 172.21.15.44:0/2203631457 shutdown_connections 2024-05-27T09:45:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.708+0000 7f4a53056b80 1 --2- 172.21.15.44:0/2203631457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5a11db0 0x5642c5a179f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.708+0000 7f4a53056b80 1 --2- 172.21.15.44:0/2203631457 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a0b8f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.708+0000 7f4a53056b80 1 --2- 172.21.15.44:0/2203631457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a0be30 0x5642c5a2a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.708+0000 7f4a53056b80 1 -- 172.21.15.44:0/2203631457 >> 172.21.15.44:0/2203631457 conn(0x5642c58ed040 msgr2=0x5642c5a15330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.708+0000 7f4a53056b80 1 -- 172.21.15.44:0/2203631457 shutdown_connections 2024-05-27T09:45:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.708+0000 7f4a53056b80 1 -- 172.21.15.44:0/2203631457 wait complete. 2024-05-27T09:45:52.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.709+0000 7f4a53056b80 1 -- start start 2024-05-27T09:45:52.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.709+0000 7f4a53056b80 1 Processor -- start 2024-05-27T09:45:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a31780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5a0be30 0x5642c5a31cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a53056b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a11db0 0x5642c5a915c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a53056b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1ab1b0 con 0x5642c5a11db0 2024-05-27T09:45:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a53056b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1ac620 con 0x5642c588cec0 2024-05-27T09:45:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a53056b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1ab880 con 0x5642c5a0be30 2024-05-27T09:45:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a53056b80 10 client.0 _ll_register_callbacks cb 0x5642c58d3cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a41c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a31780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a41c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a31780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:34774/0 (socket says 172.21.15.44:34774) 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a41c00700 1 -- 172.21.15.44:0/3241717378 learned_addr learned my addr 172.21.15.44:0/3241717378 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a41200700 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5a0be30 0x5642c5a31cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a42600700 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a11db0 0x5642c5a915c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a41c00700 1 -- 172.21.15.44:0/3241717378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5a0be30 msgr2=0x5642c5a31cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a41c00700 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5a0be30 0x5642c5a31cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a41c00700 1 -- 172.21.15.44:0/3241717378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a11db0 msgr2=0x5642c5a915c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a41c00700 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a11db0 0x5642c5a915c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.710+0000 7f4a41c00700 1 -- 172.21.15.44:0/3241717378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642c5a91b30 con 0x5642c588cec0 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.711+0000 7f4a41c00700 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a31780 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009810 tx=0x7f4a3000b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b359059265b9f10 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.711+0000 7f4a23400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.711+0000 7f4a23400700 1 -- 172.21.15.44:0/3241717378 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3002a490 con 0x5642c588cec0 2024-05-27T09:45:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.711+0000 7f4a53056b80 1 -- 172.21.15.44:0/3241717378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a30003680 con 0x5642c588cec0 2024-05-27T09:45:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.711+0000 7f4a23400700 1 -- 172.21.15.44:0/3241717378 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7f4a3000bd70 con 0x5642c588cec0 2024-05-27T09:45:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.711+0000 7f4a23400700 1 -- 172.21.15.44:0/3241717378 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4a300050c0 con 0x5642c588cec0 2024-05-27T09:45:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.711+0000 7f4a53056b80 10 client.27287 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.711+0000 7f4a53056b80 1 -- 172.21.15.44:0/3241717378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5642c5a98e70 con 0x5642c588cec0 2024-05-27T09:45:52.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.711+0000 7f4a23400700 1 -- 172.21.15.44:0/3241717378 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a30003680 con 0x5642c588cec0 2024-05-27T09:45:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.712+0000 7f4a23400700 1 -- 172.21.15.44:0/3241717378 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f4a30041020 con 0x5642c588cec0 2024-05-27T09:45:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.712+0000 7f4a53056b80 10 client.27287 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.712+0000 7f4a53056b80 10 client.27287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.712+0000 7f4a53056b80 20 client.27287 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.712+0000 7f4a53056b80 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5a2dd80 0x5642c5a9b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.712+0000 7f4a53056b80 4 client.27287 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.712+0000 7f4a53056b80 1 -- 172.21.15.44:0/3241717378 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5642c59355f0 con 0x5642c5a2dd80 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.712+0000 7f4a41200700 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5a2dd80 0x5642c5a9b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.713+0000 7f4a23400700 10 client.27287 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.713+0000 7f4a41200700 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5a2dd80 0x5642c5a9b480 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8721fd9ff49e8668 server_cookie=8788c24b28f6e13f in_seq=0 out_seq=0 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a23400700 1 -- 172.21.15.44:0/3241717378 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5642c59355f0 con 0x5642c5a2dd80 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a23400700 10 client.27287 handle_command_reply: tid=0 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a53056b80 1 client.27287 shutdown 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a53056b80 1 -- 172.21.15.44:0/3241717378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 msgr2=0x5642c5a31780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a53056b80 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a31780 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009810 tx=0x7f4a3000b840 comp rx=0 tx=0).stop 2024-05-27T09:45:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a53056b80 1 -- 172.21.15.44:0/3241717378 shutdown_connections 2024-05-27T09:45:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a53056b80 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5642c5a2dd80 0x5642c5a9b480 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a53056b80 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5642c5a11db0 0x5642c5a915c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a53056b80 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5642c588cec0 0x5642c5a31780 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a53056b80 1 --2- 172.21.15.44:0/3241717378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642c5a0be30 0x5642c5a31cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a23400700 0 client.27287 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.714+0000 7f4a53056b80 1 -- 172.21.15.44:0/3241717378 >> 172.21.15.44:0/3241717378 conn(0x5642c58ed040 msgr2=0x5642c5a12c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.715+0000 7f4a53056b80 1 -- 172.21.15.44:0/3241717378 shutdown_connections 2024-05-27T09:45:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.715+0000 7f4a53056b80 1 -- 172.21.15.44:0/3241717378 wait complete. 2024-05-27T09:45:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.715+0000 7f4a53056b80 20 client.27287 trim_cache size 0 max 16384 2024-05-27T09:45:52.717 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:45:52.718 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:45:52.718 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "5348fc89-9539-4fcf-9fdc-6e30fd88b5c6", 2024-05-27T09:45:52.718 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:45:52.718 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:45:52.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 -- 172.21.15.44:0/3373794949 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4a240b2a70 msgr2=0x7f4a240b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:52.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4a240b2a70 0x7f4a240b4f80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4a34009fd0 tx=0x7f4a34009040 comp rx=0 tx=0).stop 2024-05-27T09:45:52.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 -- 172.21.15.44:0/3373794949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3c076670 msgr2=0x7f4a3c19d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:52.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3c076670 0x7f4a3c19d600 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800e9f0 tx=0x7f4a2800ed00 comp rx=0 tx=0).stop 2024-05-27T09:45:52.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 -- 172.21.15.44:0/3373794949 shutdown_connections 2024-05-27T09:45:52.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f4a240b2a70 0x7f4a240b4f80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a3c110b90 0x7f4a3c1a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a3c075d20 0x7f4a3c19d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 --2- 172.21.15.44:0/3373794949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3c076670 0x7f4a3c19d600 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:52.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 -- 172.21.15.44:0/3373794949 >> 172.21.15.44:0/3373794949 conn(0x7f4a3c100030 msgr2=0x7f4a3c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:52.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 -- 172.21.15.44:0/3373794949 shutdown_connections 2024-05-27T09:45:52.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:52.716+0000 7f4a44600700 1 -- 172.21.15.44:0/3373794949 wait complete. 2024-05-27T09:45:53.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:53 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:45:53.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:53 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:45:53.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:53 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:45:53.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:53 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:45:53.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:53 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:45:53.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:53 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:45:53.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:53 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:45:53.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:53 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:45:53.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:53 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:45:53.916 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:45:53.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.928+0000 7efe27400640 1 -- 172.21.15.44:0/594474552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe20102a00 msgr2=0x7efe20102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:53.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.928+0000 7efe27400640 1 --2- 172.21.15.44:0/594474552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe20102a00 0x7efe20102e10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7efe14009860 tx=0x7efe1402a340 comp rx=0 tx=0).stop 2024-05-27T09:45:53.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.929+0000 7efe27400640 1 -- 172.21.15.44:0/594474552 shutdown_connections 2024-05-27T09:45:53.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.929+0000 7efe27400640 1 --2- 172.21.15.44:0/594474552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe201105a0 0x7efe201129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.929+0000 7efe27400640 1 --2- 172.21.15.44:0/594474552 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe2010dc40 0x7efe20110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.929+0000 7efe27400640 1 --2- 172.21.15.44:0/594474552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe20102a00 0x7efe20102e10 secure :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7efe14009860 tx=0x7efe1402a340 comp rx=0 tx=0).stop 2024-05-27T09:45:53.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.929+0000 7efe27400640 1 -- 172.21.15.44:0/594474552 >> 172.21.15.44:0/594474552 conn(0x7efe200782d0 msgr2=0x7efe20078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:53.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.929+0000 7efe27400640 1 -- 172.21.15.44:0/594474552 shutdown_connections 2024-05-27T09:45:53.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.929+0000 7efe27400640 1 -- 172.21.15.44:0/594474552 wait complete. 2024-05-27T09:45:53.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.931+0000 7efe27400640 1 Processor -- start 2024-05-27T09:45:53.931 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.931+0000 7efe27400640 1 -- start start 2024-05-27T09:45:53.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.932+0000 7efe27400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe2010dc40 0x7efe2010c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.932 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.932+0000 7efe27400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe201105a0 0x7efe2010cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.932+0000 7efe27400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe201b34c0 0x7efe201b58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.932+0000 7efe27400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efe201058a0 con 0x7efe2010dc40 2024-05-27T09:45:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.932+0000 7efe27400640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efe20105720 con 0x7efe201b34c0 2024-05-27T09:45:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.932+0000 7efe27400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe20105a20 con 0x7efe201105a0 2024-05-27T09:45:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.932+0000 7efe1fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe201105a0 0x7efe2010cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.932+0000 7efe1fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe201105a0 0x7efe2010cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42052/0 (socket says 172.21.15.44:42052) 2024-05-27T09:45:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.932+0000 7efe1fe00640 1 -- 172.21.15.44:0/3484681833 learned_addr learned my addr 172.21.15.44:0/3484681833 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.932+0000 7efe25400640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe201b34c0 0x7efe201b58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.933+0000 7efe1fe00640 1 -- 172.21.15.44:0/3484681833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe201b34c0 msgr2=0x7efe201b58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.933+0000 7efe1fe00640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe201b34c0 0x7efe201b58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.933+0000 7efe1fe00640 1 -- 172.21.15.44:0/3484681833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe2010dc40 msgr2=0x7efe2010c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.933+0000 7efe24a00640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe2010dc40 0x7efe2010c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.933+0000 7efe1fe00640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe2010dc40 0x7efe2010c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.933+0000 7efe1fe00640 1 -- 172.21.15.44:0/3484681833 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe201b5e20 con 0x7efe201105a0 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.933+0000 7efe24a00640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe2010dc40 0x7efe2010c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.933+0000 7efe1fe00640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe201105a0 0x7efe2010cc90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00e9c0 tx=0x7efe0c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.933+0000 7efe1d600640 1 -- 172.21.15.44:0/3484681833 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c00cd80 con 0x7efe201105a0 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.934+0000 7efe1d600640 1 -- 172.21.15.44:0/3484681833 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe0c0054d0 con 0x7efe201105a0 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.934+0000 7efe1d600640 1 -- 172.21.15.44:0/3484681833 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c010430 con 0x7efe201105a0 2024-05-27T09:45:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.934+0000 7efe27400640 1 -- 172.21.15.44:0/3484681833 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe201b5ff0 con 0x7efe201105a0 2024-05-27T09:45:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.934+0000 7efe27400640 1 -- 172.21.15.44:0/3484681833 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe201b64c0 con 0x7efe201105a0 2024-05-27T09:45:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.939+0000 7efe2e42d740 1 Processor -- start 2024-05-27T09:45:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.939+0000 7efe2e42d740 1 -- start start 2024-05-27T09:45:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502831b0 0x560950625860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56095025a2f0 0x560950625da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x5609506262e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe2e42d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560950596b90 con 0x5609502624d0 2024-05-27T09:45:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe2e42d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560950586730 con 0x56095025a2f0 2024-05-27T09:45:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe2e42d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56095058a770 con 0x5609502831b0 2024-05-27T09:45:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe25400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502831b0 0x560950625860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe25400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502831b0 0x560950625860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42062/0 (socket says 172.21.15.44:42062) 2024-05-27T09:45:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe25400640 1 -- 172.21.15.44:0/2568695665 learned_addr learned my addr 172.21.15.44:0/2568695665 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe25400640 1 -- 172.21.15.44:0/2568695665 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56095025a2f0 msgr2=0x560950625da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe25400640 1 --2- 172.21.15.44:0/2568695665 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56095025a2f0 0x560950625da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe25400640 1 -- 172.21.15.44:0/2568695665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 msgr2=0x5609506262e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:45:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe25400640 1 --2- 172.21.15.44:0/2568695665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x5609506262e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.940+0000 7efe25400640 1 -- 172.21.15.44:0/2568695665 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56095051f8b0 con 0x5609502831b0 2024-05-27T09:45:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.941+0000 7efe1d600640 1 -- 172.21.15.44:0/3484681833 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7efe0c0105d0 con 0x7efe201105a0 2024-05-27T09:45:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.941+0000 7efe25400640 1 --2- 172.21.15.44:0/2568695665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502831b0 0x560950625860 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efe1800e480 tx=0x7efe180095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70e36b0501823a7c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.941+0000 7efe11c00640 1 -- 172.21.15.44:0/2568695665 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18003970 con 0x5609502831b0 2024-05-27T09:45:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.941+0000 7efe11c00640 1 -- 172.21.15.44:0/2568695665 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe1800d7d0 con 0x5609502831b0 2024-05-27T09:45:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.941+0000 7efe11c00640 1 -- 172.21.15.44:0/2568695665 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18004390 con 0x5609502831b0 2024-05-27T09:45:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.942+0000 7efe2e42d740 1 -- 172.21.15.44:0/2568695665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502831b0 msgr2=0x560950625860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.942+0000 7efe2e42d740 1 --2- 172.21.15.44:0/2568695665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502831b0 0x560950625860 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efe1800e480 tx=0x7efe180095c0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.943+0000 7efe2e42d740 1 -- 172.21.15.44:0/2568695665 shutdown_connections 2024-05-27T09:45:53.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.943+0000 7efe2e42d740 1 --2- 172.21.15.44:0/2568695665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x5609506262e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.943+0000 7efe2e42d740 1 --2- 172.21.15.44:0/2568695665 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56095025a2f0 0x560950625da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.943+0000 7efe2e42d740 1 --2- 172.21.15.44:0/2568695665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502831b0 0x560950625860 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.943+0000 7efe2e42d740 1 -- 172.21.15.44:0/2568695665 >> 172.21.15.44:0/2568695665 conn(0x560950277300 msgr2=0x560950622010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:53.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.943+0000 7efe1d600640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe080b2aa0 0x7efe080b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.944 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.944+0000 7efe24a00640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe080b2aa0 0x7efe080b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.944+0000 7efe1d600640 1 -- 172.21.15.44:0/3484681833 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7efe0c0e2a00 con 0x7efe201105a0 2024-05-27T09:45:53.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.944+0000 7efe2e42d740 1 -- 172.21.15.44:0/2568695665 shutdown_connections 2024-05-27T09:45:53.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.944+0000 7efe2e42d740 1 -- 172.21.15.44:0/2568695665 wait complete. 2024-05-27T09:45:53.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.945+0000 7efe24a00640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe080b2aa0 0x7efe080b4fb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efe14009830 tx=0x7efe14005f10 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:53.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.945+0000 7efe2e42d740 1 -- start start 2024-05-27T09:45:53.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.945+0000 7efe2e42d740 1 Processor -- start 2024-05-27T09:45:53.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.946+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 0x56095062fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.946+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x56095062ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.946+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x56095063bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.946+0000 7efe2e42d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56095026c790 con 0x5609502624d0 2024-05-27T09:45:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.946+0000 7efe2e42d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56095052ebc0 con 0x5609502831b0 2024-05-27T09:45:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.946+0000 7efe2e42d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560950632990 con 0x56095025a2f0 2024-05-27T09:45:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.946+0000 7efe2e42d740 10 client.0 _ll_register_callbacks cb 0x56095050c400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.947+0000 7efe24a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 0x56095062fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.947+0000 7efe25400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x56095063bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.947+0000 7efe24a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 0x56095062fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42076/0 (socket says 172.21.15.44:42076) 2024-05-27T09:45:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.947+0000 7efe24a00640 1 -- 172.21.15.44:0/2136393241 learned_addr learned my addr 172.21.15.44:0/2136393241 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.947+0000 7efe1fe00640 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x56095062ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.947+0000 7efe24a00640 1 -- 172.21.15.44:0/2136393241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 msgr2=0x56095063bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.947+0000 7efe24a00640 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x56095063bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.947+0000 7efe24a00640 1 -- 172.21.15.44:0/2136393241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 msgr2=0x56095062ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.947+0000 7efe24a00640 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x56095062ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.947+0000 7efe24a00640 1 -- 172.21.15.44:0/2136393241 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56095029b210 con 0x56095025a2f0 2024-05-27T09:45:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.948+0000 7efe24a00640 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 0x56095062fa70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efe14009860 tx=0x7efe1402a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5094b5ec6415f51e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.948+0000 7efe06000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:45:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.948+0000 7efe06000640 1 -- 172.21.15.44:0/2136393241 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe14043040 con 0x56095025a2f0 2024-05-27T09:45:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.948+0000 7efe06000640 1 -- 172.21.15.44:0/2136393241 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7efe14004980 con 0x56095025a2f0 2024-05-27T09:45:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.949+0000 7efe2e42d740 1 -- 172.21.15.44:0/2136393241 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe1800d7d0 con 0x56095025a2f0 2024-05-27T09:45:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.949+0000 7efe06000640 1 -- 172.21.15.44:0/2136393241 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7efe14037040 con 0x56095025a2f0 2024-05-27T09:45:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.949+0000 7efe2e42d740 10 client.27292 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.949+0000 7efe2e42d740 1 -- 172.21.15.44:0/2136393241 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560950367620 con 0x56095025a2f0 2024-05-27T09:45:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.949+0000 7efe06000640 1 -- 172.21.15.44:0/2136393241 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe1800d7d0 con 0x56095025a2f0 2024-05-27T09:45:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.950+0000 7efe06000640 1 -- 172.21.15.44:0/2136393241 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7efe1403a030 con 0x56095025a2f0 2024-05-27T09:45:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.950+0000 7efe2e42d740 10 client.27292 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.950+0000 7efe2e42d740 10 client.27292 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.950+0000 7efe2e42d740 20 client.27292 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.950+0000 7efe2e42d740 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560950241d80 0x560950641cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.950+0000 7efe2e42d740 4 client.27292 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.950+0000 7efe2e42d740 1 -- 172.21.15.44:0/2136393241 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560950598ae0 con 0x560950241d80 2024-05-27T09:45:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.950+0000 7efe1fe00640 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560950241d80 0x560950641cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.953+0000 7efe1fe00640 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560950241d80 0x560950641cd0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4455c1d022f8c2d1 server_cookie=6994953d8d5350d2 in_seq=0 out_seq=0 2024-05-27T09:45:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.953+0000 7efe06000640 10 client.27292 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.954+0000 7efe06000640 1 -- 172.21.15.44:0/2136393241 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560950598ae0 con 0x560950241d80 2024-05-27T09:45:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.954+0000 7efe06000640 10 client.27292 handle_command_reply: tid=0 2024-05-27T09:45:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.954+0000 7efe2e42d740 1 client.27292 shutdown 2024-05-27T09:45:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe2e42d740 1 -- 172.21.15.44:0/2136393241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 msgr2=0x56095062fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe2e42d740 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 0x56095062fa70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efe14009860 tx=0x7efe1402a340 comp rx=0 tx=0).stop 2024-05-27T09:45:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe2e42d740 1 -- 172.21.15.44:0/2136393241 shutdown_connections 2024-05-27T09:45:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe2e42d740 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560950241d80 0x560950641cd0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe2e42d740 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x56095063bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe2e42d740 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x56095062ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe2e42d740 1 --2- 172.21.15.44:0/2136393241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 0x56095062fa70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe2e42d740 1 -- 172.21.15.44:0/2136393241 >> 172.21.15.44:0/2136393241 conn(0x560950277300 msgr2=0x560950627580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe06000640 0 client.27292 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe2e42d740 1 -- 172.21.15.44:0/2136393241 shutdown_connections 2024-05-27T09:45:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe2e42d740 1 -- 172.21.15.44:0/2136393241 wait complete. 2024-05-27T09:45:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.955+0000 7efe2e42d740 20 client.27292 trim_cache size 0 max 16384 2024-05-27T09:45:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.979+0000 7efe2e42d740 1 Processor -- start 2024-05-27T09:45:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.979+0000 7efe2e42d740 1 -- start start 2024-05-27T09:45:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.980+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 0x56095063ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.980+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x56095063bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.980+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506295c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.980+0000 7efe2e42d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efe201c1980 con 0x5609502624d0 2024-05-27T09:45:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.980+0000 7efe2e42d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efe201bf220 con 0x5609502831b0 2024-05-27T09:45:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.980+0000 7efe2e42d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe201bd090 con 0x56095025a2f0 2024-05-27T09:45:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.980+0000 7efe24a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 0x56095063ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.981+0000 7efe25400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506295c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.981+0000 7efe25400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506295c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47546/0 (socket says 172.21.15.44:47546) 2024-05-27T09:45:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.981+0000 7efe25400640 1 -- 172.21.15.44:0/3580852411 learned_addr learned my addr 172.21.15.44:0/3580852411 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.981+0000 7efe1fe00640 1 --2- 172.21.15.44:0/3580852411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x56095063bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.981+0000 7efe25400640 1 -- 172.21.15.44:0/3580852411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 msgr2=0x56095063ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.981+0000 7efe25400640 1 --2- 172.21.15.44:0/3580852411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 0x56095063ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.981+0000 7efe25400640 1 -- 172.21.15.44:0/3580852411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 msgr2=0x56095063bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.981+0000 7efe25400640 1 --2- 172.21.15.44:0/3580852411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x56095063bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.981+0000 7efe25400640 1 -- 172.21.15.44:0/3580852411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560950622c00 con 0x5609502831b0 2024-05-27T09:45:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.981+0000 7efe1fe00640 1 --2- 172.21.15.44:0/3580852411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x56095063bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:45:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.981+0000 7efe24a00640 1 --2- 172.21.15.44:0/3580852411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 0x56095063ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:45:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.982+0000 7efe25400640 1 --2- 172.21.15.44:0/3580852411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506295c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efe1800e480 tx=0x7efe1800d7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75f928c899e624b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.982+0000 7efe11200640 1 -- 172.21.15.44:0/3580852411 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18003970 con 0x5609502831b0 2024-05-27T09:45:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.982+0000 7efe11200640 1 -- 172.21.15.44:0/3580852411 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe18027030 con 0x5609502831b0 2024-05-27T09:45:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.982+0000 7efe11200640 1 -- 172.21.15.44:0/3580852411 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18005360 con 0x5609502831b0 2024-05-27T09:45:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.983+0000 7efe2e42d740 1 -- 172.21.15.44:0/3580852411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 msgr2=0x5609506295c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.983+0000 7efe2e42d740 1 --2- 172.21.15.44:0/3580852411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506295c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efe1800e480 tx=0x7efe1800d7d0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.983+0000 7efe2e42d740 1 -- 172.21.15.44:0/3580852411 shutdown_connections 2024-05-27T09:45:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.983+0000 7efe2e42d740 1 --2- 172.21.15.44:0/3580852411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506295c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.983+0000 7efe2e42d740 1 --2- 172.21.15.44:0/3580852411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5609502624d0 0x56095063bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.983+0000 7efe2e42d740 1 --2- 172.21.15.44:0/3580852411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56095025a2f0 0x56095063ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.983+0000 7efe2e42d740 1 -- 172.21.15.44:0/3580852411 >> 172.21.15.44:0/3580852411 conn(0x560950277300 msgr2=0x56095062fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.983+0000 7efe2e42d740 1 -- 172.21.15.44:0/3580852411 shutdown_connections 2024-05-27T09:45:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.983+0000 7efe2e42d740 1 -- 172.21.15.44:0/3580852411 wait complete. 2024-05-27T09:45:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.984+0000 7efe2e42d740 1 -- start start 2024-05-27T09:45:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.984+0000 7efe2e42d740 1 Processor -- start 2024-05-27T09:45:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.984+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56095025a2f0 0x560950633240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502624d0 0x560950633780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe2e42d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506a2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe2e42d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efe201bd6e0 con 0x56095025a2f0 2024-05-27T09:45:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe2e42d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efe201beb50 con 0x5609502831b0 2024-05-27T09:45:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe2e42d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe201bddb0 con 0x5609502624d0 2024-05-27T09:45:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe2e42d740 10 client.0 _ll_register_callbacks cb 0x7efe14040040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe25400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe25400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506a2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47548/0 (socket says 172.21.15.44:47548) 2024-05-27T09:45:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe25400640 1 -- 172.21.15.44:0/885701098 learned_addr learned my addr 172.21.15.44:0/885701098 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe1fe00640 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502624d0 0x560950633780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe25400640 1 -- 172.21.15.44:0/885701098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502624d0 msgr2=0x560950633780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe25400640 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502624d0 0x560950633780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe25400640 1 -- 172.21.15.44:0/885701098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56095025a2f0 msgr2=0x560950633240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe25400640 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56095025a2f0 0x560950633240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.985+0000 7efe25400640 1 -- 172.21.15.44:0/885701098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609506a2860 con 0x5609502831b0 2024-05-27T09:45:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.986+0000 7efe25400640 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506a2320 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efe18005b10 tx=0x7efe18003be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f531cc8f5c28511 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.986+0000 7efe06000640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:45:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.986+0000 7efe06000640 1 -- 172.21.15.44:0/885701098 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1800dad0 con 0x5609502831b0 2024-05-27T09:45:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.986+0000 7efe06000640 1 -- 172.21.15.44:0/885701098 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7efe1801c070 con 0x5609502831b0 2024-05-27T09:45:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.987+0000 7efe2e42d740 1 -- 172.21.15.44:0/885701098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe18027030 con 0x5609502831b0 2024-05-27T09:45:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.987+0000 7efe06000640 1 -- 172.21.15.44:0/885701098 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7efe1800dc70 con 0x5609502831b0 2024-05-27T09:45:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.988+0000 7efe2e42d740 10 client.27302 fetch_fsmap learned FSMap version 33 2024-05-27T09:45:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.988+0000 7efe2e42d740 1 -- 172.21.15.44:0/885701098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5609506a9a20 con 0x5609502831b0 2024-05-27T09:45:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.988+0000 7efe06000640 1 -- 172.21.15.44:0/885701098 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe18027030 con 0x5609502831b0 2024-05-27T09:45:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.988+0000 7efe06000640 1 -- 172.21.15.44:0/885701098 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7efe1800b040 con 0x5609502831b0 2024-05-27T09:45:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.988+0000 7efe2e42d740 10 client.27302 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T09:45:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.988+0000 7efe2e42d740 10 client.27302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.988+0000 7efe2e42d740 20 client.27302 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.988+0000 7efe2e42d740 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5609502171c0 0x5609506abf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.988+0000 7efe2e42d740 4 client.27302 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:45:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.988+0000 7efe2e42d740 1 -- 172.21.15.44:0/885701098 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5609505922c0 con 0x5609502171c0 2024-05-27T09:45:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.989+0000 7efe24a00640 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5609502171c0 0x5609506abf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:53.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.989+0000 7efe24a00640 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5609502171c0 0x5609506abf20 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72ab1f06b6594ae4 server_cookie=93c0742bb1167bd2 in_seq=0 out_seq=0 2024-05-27T09:45:53.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:53.989+0000 7efe06000640 10 client.27302 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:54.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.376+0000 7efe06000640 1 -- 172.21.15.44:0/885701098 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5609505922c0 con 0x5609502171c0 2024-05-27T09:45:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.376+0000 7efe06000640 10 client.27302 handle_command_reply: tid=0 2024-05-27T09:45:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.376+0000 7efe2e42d740 1 client.27302 shutdown 2024-05-27T09:45:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.376+0000 7efe2e42d740 1 -- 172.21.15.44:0/885701098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 msgr2=0x5609506a2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.376+0000 7efe2e42d740 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506a2320 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efe18005b10 tx=0x7efe18003be0 comp rx=0 tx=0).stop 2024-05-27T09:45:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.377+0000 7efe2e42d740 1 -- 172.21.15.44:0/885701098 shutdown_connections 2024-05-27T09:45:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.377+0000 7efe2e42d740 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5609502171c0 0x5609506abf20 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.377+0000 7efe2e42d740 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5609502831b0 0x5609506a2320 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.377+0000 7efe2e42d740 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609502624d0 0x560950633780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.377+0000 7efe2e42d740 1 --2- 172.21.15.44:0/885701098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56095025a2f0 0x560950633240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.377+0000 7efe2e42d740 1 -- 172.21.15.44:0/885701098 >> 172.21.15.44:0/885701098 conn(0x560950277300 msgr2=0x560950631120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.377+0000 7efe06000640 0 client.27302 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.377+0000 7efe2e42d740 1 -- 172.21.15.44:0/885701098 shutdown_connections 2024-05-27T09:45:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.378+0000 7efe2e42d740 1 -- 172.21.15.44:0/885701098 wait complete. 2024-05-27T09:45:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.378+0000 7efe2e42d740 20 client.27302 trim_cache size 0 max 16384 2024-05-27T09:45:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:45:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:45:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:45:54.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.380+0000 7efe27400640 1 -- 172.21.15.44:0/3484681833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe080b2aa0 msgr2=0x7efe080b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:54.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.381+0000 7efe27400640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe080b2aa0 0x7efe080b4fb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efe14009830 tx=0x7efe14005f10 comp rx=0 tx=0).stop 2024-05-27T09:45:54.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.381+0000 7efe27400640 1 -- 172.21.15.44:0/3484681833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe201105a0 msgr2=0x7efe2010cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:54.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.381+0000 7efe27400640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe201105a0 0x7efe2010cc90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00e9c0 tx=0x7efe0c00ee80 comp rx=0 tx=0).stop 2024-05-27T09:45:54.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.382+0000 7efe27400640 1 -- 172.21.15.44:0/3484681833 shutdown_connections 2024-05-27T09:45:54.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.382+0000 7efe27400640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe201b34c0 0x7efe201b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:54.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.382+0000 7efe27400640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe201105a0 0x7efe2010cc90 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:54.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.382+0000 7efe27400640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efe080b2aa0 0x7efe080b4fb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:54.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.382+0000 7efe27400640 1 --2- 172.21.15.44:0/3484681833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe2010dc40 0x7efe2010c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:54.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.382+0000 7efe27400640 1 -- 172.21.15.44:0/3484681833 >> 172.21.15.44:0/3484681833 conn(0x7efe200782d0 msgr2=0x7efe201012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:54.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.383+0000 7efe27400640 1 -- 172.21.15.44:0/3484681833 shutdown_connections 2024-05-27T09:45:54.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:45:54.385+0000 7efe27400640 1 -- 172.21.15.44:0/3484681833 wait complete. 2024-05-27T09:45:54.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:54 smithi044 ceph-mon[31618]: pgmap v332: 97 pgs: 97 active+clean; 282 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 614 KiB/s wr, 14 op/s 2024-05-27T09:45:54.522 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:45:54.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:54 smithi050 ceph-mon[36371]: pgmap v332: 97 pgs: 97 active+clean; 282 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 614 KiB/s wr, 14 op/s 2024-05-27T09:45:54.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:54 smithi189 ceph-mon[36671]: pgmap v332: 97 pgs: 97 active+clean; 282 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 614 KiB/s wr, 14 op/s 2024-05-27T09:45:55.409 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:55 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:45:55.410 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:55 smithi044 ceph-mon[31618]: pgmap v333: 97 pgs: 97 active+clean; 282 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 614 KiB/s wr, 14 op/s 2024-05-27T09:45:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:55 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:45:55.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:55 smithi050 ceph-mon[36371]: pgmap v333: 97 pgs: 97 active+clean; 282 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 614 KiB/s wr, 14 op/s 2024-05-27T09:45:55.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:55 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:45:55.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:55 smithi189 ceph-mon[36671]: pgmap v333: 97 pgs: 97 active+clean; 282 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 614 KiB/s wr, 14 op/s 2024-05-27T09:45:56.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:56 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:45:56.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:56 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:45:56.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.571+0000 7fc3f5a00700 1 -- 172.21.15.44:0/4127631962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f0106ec0 msgr2=0x7fc3f0113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:56.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.571+0000 7fc3f5a00700 1 --2- 172.21.15.44:0/4127631962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f0106ec0 0x7fc3f0113350 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800b0a0 tx=0x7fc3e800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.571+0000 7fc3f5a00700 1 -- 172.21.15.44:0/4127631962 shutdown_connections 2024-05-27T09:45:56.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.571+0000 7fc3f5a00700 1 --2- 172.21.15.44:0/4127631962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f0106ec0 0x7fc3f0113350 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.571+0000 7fc3f5a00700 1 --2- 172.21.15.44:0/4127631962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3f010c5b0 0x7fc3f010c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.571+0000 7fc3f5a00700 1 --2- 172.21.15.44:0/4127631962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f0106570 0x7fc3f0106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.571+0000 7fc3f5a00700 1 -- 172.21.15.44:0/4127631962 >> 172.21.15.44:0/4127631962 conn(0x7fc3f0102070 msgr2=0x7fc3f01044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:56.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.571+0000 7fc3f5a00700 1 -- 172.21.15.44:0/4127631962 shutdown_connections 2024-05-27T09:45:56.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.571+0000 7fc3f5a00700 1 -- 172.21.15.44:0/4127631962 wait complete. 2024-05-27T09:45:56.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.572+0000 7fc3f5a00700 1 Processor -- start 2024-05-27T09:45:56.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.573+0000 7fc3f5a00700 1 -- start start 2024-05-27T09:45:56.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.573+0000 7fc3f5a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3f0106570 0x7fc3f019d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.573+0000 7fc3f5a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f0106ec0 0x7fc3f019d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.573+0000 7fc3f5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f010c5b0 0x7fc3f01a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.574+0000 7fc3f5a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f0116730 con 0x7fc3f0106570 2024-05-27T09:45:56.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.574+0000 7fc3f5a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f01165b0 con 0x7fc3f0106ec0 2024-05-27T09:45:56.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.574+0000 7fc3f5a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f01168b0 con 0x7fc3f010c5b0 2024-05-27T09:45:56.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.574+0000 7fc3ef400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f010c5b0 0x7fc3f01a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.574+0000 7fc3ef400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f010c5b0 0x7fc3f01a1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42108/0 (socket says 172.21.15.44:42108) 2024-05-27T09:45:56.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.574+0000 7fc3ef400700 1 -- 172.21.15.44:0/1131951215 learned_addr learned my addr 172.21.15.44:0/1131951215 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:56.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.574+0000 7fc3ee000700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f0106ec0 0x7fc3f019d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.574+0000 7fc3eea00700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3f0106570 0x7fc3f019d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.574+0000 7fc3ef400700 1 -- 172.21.15.44:0/1131951215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f0106ec0 msgr2=0x7fc3f019d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:56.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.574+0000 7fc3ef400700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f0106ec0 0x7fc3f019d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.574+0000 7fc3ef400700 1 -- 172.21.15.44:0/1131951215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3f0106570 msgr2=0x7fc3f019d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:56.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.575+0000 7fc3ef400700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3f0106570 0x7fc3f019d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.575+0000 7fc3ef400700 1 -- 172.21.15.44:0/1131951215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f01a1ef0 con 0x7fc3f010c5b0 2024-05-27T09:45:56.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.575+0000 7fc3eea00700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3f0106570 0x7fc3f019d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:45:56.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.575+0000 7fc3ef400700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f010c5b0 0x7fc3f01a1850 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800b070 tx=0x7fc3e8009c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:56.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.575+0000 7fc3e7400700 1 -- 172.21.15.44:0/1131951215 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8021070 con 0x7fc3f010c5b0 2024-05-27T09:45:56.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.575+0000 7fc3e7400700 1 -- 172.21.15.44:0/1131951215 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3e8007e30 con 0x7fc3f010c5b0 2024-05-27T09:45:56.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.575+0000 7fc3e7400700 1 -- 172.21.15.44:0/1131951215 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e80046a0 con 0x7fc3f010c5b0 2024-05-27T09:45:56.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.575+0000 7fc3f5a00700 1 -- 172.21.15.44:0/1131951215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f01a2180 con 0x7fc3f010c5b0 2024-05-27T09:45:56.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.576+0000 7fc3f5a00700 1 -- 172.21.15.44:0/1131951215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f01a9b00 con 0x7fc3f010c5b0 2024-05-27T09:45:56.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.580+0000 7fc3e7400700 1 -- 172.21.15.44:0/1131951215 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc3e8007980 con 0x7fc3f010c5b0 2024-05-27T09:45:56.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.581+0000 7fc404472b80 1 Processor -- start 2024-05-27T09:45:56.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.582+0000 7fc404472b80 1 -- start start 2024-05-27T09:45:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.582+0000 7fc404472b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.582+0000 7fc404472b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 0x55bf7c1c3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.582+0000 7fc404472b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.582+0000 7fc404472b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bf7c0f1e40 con 0x55bf7c1309f0 2024-05-27T09:45:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.582+0000 7fc404472b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bf7c0e19e0 con 0x55bf7c044e40 2024-05-27T09:45:56.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.582+0000 7fc404472b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bf7c0e5a20 con 0x55bf7c1c3e70 2024-05-27T09:45:56.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.582+0000 7fc3eea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 0x55bf7c1c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.582+0000 7fc3ee000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.582+0000 7fc3eea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 0x55bf7c1c3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47588/0 (socket says 172.21.15.44:47588) 2024-05-27T09:45:56.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.582+0000 7fc3eea00700 1 -- 172.21.15.44:0/3427284239 learned_addr learned my addr 172.21.15.44:0/3427284239 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:56.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.583+0000 7fc3ee000700 1 --2- 172.21.15.44:0/3427284239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.583+0000 7fc3eea00700 1 -- 172.21.15.44:0/3427284239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 msgr2=0x55bf7c1d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:56.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.583+0000 7fc3eea00700 1 --2- 172.21.15.44:0/3427284239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1d0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.583+0000 7fc3eea00700 1 -- 172.21.15.44:0/3427284239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 msgr2=0x55bf7c1c3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:56.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.583+0000 7fc3eea00700 1 --2- 172.21.15.44:0/3427284239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.583+0000 7fc3eea00700 1 -- 172.21.15.44:0/3427284239 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf7c0aa420 con 0x55bf7c044e40 2024-05-27T09:45:56.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.583+0000 7fc3eea00700 1 --2- 172.21.15.44:0/3427284239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 0x55bf7c1c3870 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009c90 tx=0x7fc3e0010370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a841f4a8635f0f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:56.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.583+0000 7fc3e7400700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc3d40b29f0 0x7fc3d40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.583+0000 7fc3dbe00700 1 -- 172.21.15.44:0/3427284239 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0010b90 con 0x55bf7c044e40 2024-05-27T09:45:56.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.583+0000 7fc3dbe00700 1 -- 172.21.15.44:0/3427284239 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3e00044e0 con 0x55bf7c044e40 2024-05-27T09:45:56.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.583+0000 7fc3dbe00700 1 -- 172.21.15.44:0/3427284239 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0012450 con 0x55bf7c044e40 2024-05-27T09:45:56.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.584+0000 7fc3e7400700 1 -- 172.21.15.44:0/1131951215 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fc3e800f020 con 0x7fc3f010c5b0 2024-05-27T09:45:56.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.584+0000 7fc3ee000700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc3d40b29f0 0x7fc3d40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.584+0000 7fc404472b80 1 -- 172.21.15.44:0/3427284239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 msgr2=0x55bf7c1c3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:56.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.584+0000 7fc404472b80 1 --2- 172.21.15.44:0/3427284239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 0x55bf7c1c3870 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009c90 tx=0x7fc3e0010370 comp rx=0 tx=0).stop 2024-05-27T09:45:56.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.585+0000 7fc404472b80 1 -- 172.21.15.44:0/3427284239 shutdown_connections 2024-05-27T09:45:56.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.585+0000 7fc404472b80 1 --2- 172.21.15.44:0/3427284239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1d0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.585+0000 7fc404472b80 1 --2- 172.21.15.44:0/3427284239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.585+0000 7fc404472b80 1 --2- 172.21.15.44:0/3427284239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 0x55bf7c1c3870 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.585+0000 7fc404472b80 1 -- 172.21.15.44:0/3427284239 >> 172.21.15.44:0/3427284239 conn(0x55bf7c0de390 msgr2=0x55bf7c1bf4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:56.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.585+0000 7fc3ee000700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc3d40b29f0 0x7fc3d40b4f00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009fd0 tx=0x7fc3dc0058e0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:56.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.585+0000 7fc404472b80 1 -- 172.21.15.44:0/3427284239 shutdown_connections 2024-05-27T09:45:56.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.585+0000 7fc404472b80 1 -- 172.21.15.44:0/3427284239 wait complete. 2024-05-27T09:45:56.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:56 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:45:56.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.586+0000 7fc404472b80 1 -- start start 2024-05-27T09:45:56.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.586+0000 7fc404472b80 1 Processor -- start 2024-05-27T09:45:56.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc404472b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 0x55bf7c1cd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc404472b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1309f0 0x55bf7c1cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc404472b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc3eea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 0x55bf7c1cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc404472b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bf7bfe7fa0 con 0x55bf7c1c3e70 2024-05-27T09:45:56.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc404472b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bf7c082760 con 0x55bf7c1309f0 2024-05-27T09:45:56.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc404472b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bf7c1d0b80 con 0x55bf7c044e40 2024-05-27T09:45:56.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc3eea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 0x55bf7c1cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42114/0 (socket says 172.21.15.44:42114) 2024-05-27T09:45:56.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc3eea00700 1 -- 172.21.15.44:0/2378340379 learned_addr learned my addr 172.21.15.44:0/2378340379 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:56.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc404472b80 10 client.0 _ll_register_callbacks cb 0x55bf7c08bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:56.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc3ee000700 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1309f0 0x55bf7c1cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.589+0000 7fc3ef400700 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.590+0000 7fc3eea00700 1 -- 172.21.15.44:0/2378340379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1309f0 msgr2=0x55bf7c1cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.590+0000 7fc3eea00700 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1309f0 0x55bf7c1cdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.590+0000 7fc3eea00700 1 -- 172.21.15.44:0/2378340379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1c3e70 msgr2=0x55bf7c1ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.590+0000 7fc3eea00700 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.590+0000 7fc3eea00700 1 -- 172.21.15.44:0/2378340379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf7c0318d0 con 0x55bf7c044e40 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.590+0000 7fc3ee000700 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1309f0 0x55bf7c1cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.590+0000 7fc3eea00700 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 0x55bf7c1cd740 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00058e0 tx=0x7fc3e00038e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc817c9540315244 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.590+0000 7fc3d8c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.591+0000 7fc3d8c00700 1 -- 172.21.15.44:0/2378340379 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e002a060 con 0x55bf7c044e40 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.591+0000 7fc3d8c00700 1 -- 172.21.15.44:0/2378340379 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fc3e0003f30 con 0x55bf7c044e40 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.591+0000 7fc404472b80 1 -- 172.21.15.44:0/2378340379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3e00044e0 con 0x55bf7c044e40 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.591+0000 7fc3d8c00700 1 -- 172.21.15.44:0/2378340379 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc3e0004130 con 0x55bf7c044e40 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.591+0000 7fc404472b80 10 client.27307 fetch_fsmap learned FSMap version 34 2024-05-27T09:45:56.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.591+0000 7fc404472b80 1 -- 172.21.15.44:0/2378340379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55bf7c0b2060 con 0x55bf7c044e40 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.591+0000 7fc3d8c00700 1 -- 172.21.15.44:0/2378340379 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3e00044e0 con 0x55bf7c044e40 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.591+0000 7fc3d8c00700 1 -- 172.21.15.44:0/2378340379 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fc3e000f040 con 0x55bf7c044e40 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.592+0000 7fc404472b80 10 client.27307 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.592+0000 7fc404472b80 10 client.27307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.592+0000 7fc404472b80 20 client.27307 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.592+0000 7fc404472b80 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf7c1e6220 0x55bf7c1e8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.592+0000 7fc404472b80 4 client.27307 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.592+0000 7fc404472b80 1 -- 172.21.15.44:0/2378340379 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf7c0f3d90 con 0x55bf7c1e6220 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.592+0000 7fc3ee000700 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf7c1e6220 0x55bf7c1e8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.593+0000 7fc3ee000700 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf7c1e6220 0x55bf7c1e8640 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1796455b36ee38cc server_cookie=99cb420e7790ecfd in_seq=0 out_seq=0 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.593+0000 7fc3d8c00700 10 client.27307 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:56.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.593+0000 7fc3d8c00700 1 -- 172.21.15.44:0/2378340379 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bf7c0f3d90 con 0x55bf7c1e6220 2024-05-27T09:45:56.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.593+0000 7fc3d8c00700 10 client.27307 handle_command_reply: tid=0 2024-05-27T09:45:56.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.593+0000 7fc404472b80 1 client.27307 shutdown 2024-05-27T09:45:56.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc404472b80 1 -- 172.21.15.44:0/2378340379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 msgr2=0x55bf7c1cd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc404472b80 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 0x55bf7c1cd740 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00058e0 tx=0x7fc3e00038e0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc404472b80 1 -- 172.21.15.44:0/2378340379 shutdown_connections 2024-05-27T09:45:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc404472b80 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf7c1e6220 0x55bf7c1e8640 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc404472b80 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc404472b80 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 0x55bf7c1cd740 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc404472b80 1 --2- 172.21.15.44:0/2378340379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1309f0 0x55bf7c1cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc404472b80 1 -- 172.21.15.44:0/2378340379 >> 172.21.15.44:0/2378340379 conn(0x55bf7c0de390 msgr2=0x55bf7c1c33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc3d8c00700 0 client.27307 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc404472b80 1 -- 172.21.15.44:0/2378340379 shutdown_connections 2024-05-27T09:45:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc404472b80 1 -- 172.21.15.44:0/2378340379 wait complete. 2024-05-27T09:45:56.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.594+0000 7fc404472b80 20 client.27307 trim_cache size 0 max 16384 2024-05-27T09:45:56.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.622+0000 7fc404472b80 1 Processor -- start 2024-05-27T09:45:56.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.622+0000 7fc404472b80 1 -- start start 2024-05-27T09:45:56.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.622+0000 7fc404472b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 0x55bf7c1c3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.623+0000 7fc404472b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.623+0000 7fc404472b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1e2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.623+0000 7fc404472b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f01af530 con 0x55bf7c1309f0 2024-05-27T09:45:56.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.623+0000 7fc404472b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f01acd60 con 0x55bf7c1c3e70 2024-05-27T09:45:56.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.623+0000 7fc404472b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f007ce00 con 0x55bf7c044e40 2024-05-27T09:45:56.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.624+0000 7fc3ef400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1e2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.624+0000 7fc3ef400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1e2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47596/0 (socket says 172.21.15.44:47596) 2024-05-27T09:45:56.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.624+0000 7fc3ef400700 1 -- 172.21.15.44:0/208169491 learned_addr learned my addr 172.21.15.44:0/208169491 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:56.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.624+0000 7fc3ee000700 1 --2- 172.21.15.44:0/208169491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.625+0000 7fc3eea00700 1 --2- 172.21.15.44:0/208169491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 0x55bf7c1c3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.625+0000 7fc3ef400700 1 -- 172.21.15.44:0/208169491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 msgr2=0x55bf7c1c3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:56.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.626+0000 7fc3ef400700 1 --2- 172.21.15.44:0/208169491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 0x55bf7c1c3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.626+0000 7fc3ef400700 1 -- 172.21.15.44:0/208169491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 msgr2=0x55bf7c1c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:56.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.626+0000 7fc3ef400700 1 --2- 172.21.15.44:0/208169491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.626+0000 7fc3ef400700 1 -- 172.21.15.44:0/208169491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf7c1ce530 con 0x55bf7c1c3e70 2024-05-27T09:45:56.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.626+0000 7fc3eea00700 1 --2- 172.21.15.44:0/208169491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 0x55bf7c1c3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:45:56.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.627+0000 7fc3ef400700 1 --2- 172.21.15.44:0/208169491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1e2510 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80a8010 tx=0x7fc3e800b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d9da3a13a4d5137 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:56.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.627+0000 7fc3db400700 1 -- 172.21.15.44:0/208169491 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e80ae020 con 0x55bf7c1c3e70 2024-05-27T09:45:56.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.628+0000 7fc3db400700 1 -- 172.21.15.44:0/208169491 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3e8007c70 con 0x55bf7c1c3e70 2024-05-27T09:45:56.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.628+0000 7fc3db400700 1 -- 172.21.15.44:0/208169491 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e80b3b70 con 0x55bf7c1c3e70 2024-05-27T09:45:56.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.629+0000 7fc404472b80 1 -- 172.21.15.44:0/208169491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1c3e70 msgr2=0x55bf7c1e2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:56.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.629+0000 7fc404472b80 1 --2- 172.21.15.44:0/208169491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1e2510 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80a8010 tx=0x7fc3e800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.630+0000 7fc404472b80 1 -- 172.21.15.44:0/208169491 shutdown_connections 2024-05-27T09:45:56.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.630+0000 7fc404472b80 1 --2- 172.21.15.44:0/208169491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1e2510 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.630+0000 7fc404472b80 1 --2- 172.21.15.44:0/208169491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c044e40 0x55bf7c1c3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.630+0000 7fc404472b80 1 --2- 172.21.15.44:0/208169491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.630+0000 7fc404472b80 1 -- 172.21.15.44:0/208169491 >> 172.21.15.44:0/208169491 conn(0x55bf7c0de390 msgr2=0x55bf7c1c4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:56.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.631+0000 7fc404472b80 1 -- 172.21.15.44:0/208169491 shutdown_connections 2024-05-27T09:45:56.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.631+0000 7fc404472b80 1 -- 172.21.15.44:0/208169491 wait complete. 2024-05-27T09:45:56.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.632+0000 7fc404472b80 1 -- start start 2024-05-27T09:45:56.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.632+0000 7fc404472b80 1 Processor -- start 2024-05-27T09:45:56.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.633+0000 7fc404472b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 0x55bf7c1dea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.633+0000 7fc404472b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.634+0000 7fc404472b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1c8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.634+0000 7fc404472b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f01ab240 con 0x55bf7c1309f0 2024-05-27T09:45:56.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.634+0000 7fc404472b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f01ac690 con 0x55bf7c044e40 2024-05-27T09:45:56.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.634+0000 7fc404472b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f01ab8f0 con 0x55bf7c1c3e70 2024-05-27T09:45:56.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.634+0000 7fc404472b80 10 client.0 _ll_register_callbacks cb 0x55bf7c08bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:45:56.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.634+0000 7fc3ef400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.634+0000 7fc3ef400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42136/0 (socket says 172.21.15.44:42136) 2024-05-27T09:45:56.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.634+0000 7fc3ef400700 1 -- 172.21.15.44:0/3307563900 learned_addr learned my addr 172.21.15.44:0/3307563900 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:45:56.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.635+0000 7fc3eea00700 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 0x55bf7c1dea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.637+0000 7fc3ef400700 1 -- 172.21.15.44:0/3307563900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 msgr2=0x55bf7c1dea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:56.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.637+0000 7fc3ef400700 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 0x55bf7c1dea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.637+0000 7fc3ef400700 1 -- 172.21.15.44:0/3307563900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 msgr2=0x55bf7c1c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:45:56.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.637+0000 7fc3ee000700 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.637+0000 7fc3ef400700 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.638+0000 7fc3ef400700 1 -- 172.21.15.44:0/3307563900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf7c1c8590 con 0x55bf7c1c3e70 2024-05-27T09:45:56.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.639+0000 7fc3eea00700 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 0x55bf7c1dea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:45:56.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.639+0000 7fc3ef400700 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1c8050 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80a6e20 tx=0x7fc3e8011fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4c7dc588789c1b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:45:56.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.639+0000 7fc3d8c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:45:56.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.639+0000 7fc3d8c00700 1 -- 172.21.15.44:0/3307563900 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e80ae040 con 0x55bf7c1c3e70 2024-05-27T09:45:56.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.639+0000 7fc3ee000700 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:45:56.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.640+0000 7fc404472b80 1 -- 172.21.15.44:0/3307563900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bf7c0ed570 con 0x55bf7c1c3e70 2024-05-27T09:45:56.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.640+0000 7fc3d8c00700 1 -- 172.21.15.44:0/3307563900 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8521+0+0 (secure 0 0 0) 0x7fc3e80ab000 con 0x55bf7c1c3e70 2024-05-27T09:45:56.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.640+0000 7fc3d8c00700 1 -- 172.21.15.44:0/3307563900 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc3e80ab4a0 con 0x55bf7c1c3e70 2024-05-27T09:45:56.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.640+0000 7fc404472b80 10 client.27310 fetch_fsmap learned FSMap version 34 2024-05-27T09:45:56.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.640+0000 7fc404472b80 1 -- 172.21.15.44:0/3307563900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55bf7c1de2e0 con 0x55bf7c1c3e70 2024-05-27T09:45:56.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.641+0000 7fc3d8c00700 1 -- 172.21.15.44:0/3307563900 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55bf7c0ed570 con 0x55bf7c1c3e70 2024-05-27T09:45:56.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.642+0000 7fc3d8c00700 1 -- 172.21.15.44:0/3307563900 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fc3e8011070 con 0x55bf7c1c3e70 2024-05-27T09:45:56.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.642+0000 7fc404472b80 10 client.27310 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:45:56.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.642+0000 7fc404472b80 10 client.27310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:45:56.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.642+0000 7fc404472b80 20 client.27310 populate_metadata read hostname 'smithi044' 2024-05-27T09:45:56.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.642+0000 7fc404472b80 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf7c250d30 0x55bf7c253150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:45:56.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.642+0000 7fc404472b80 4 client.27310 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:45:56.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.642+0000 7fc404472b80 1 -- 172.21.15.44:0/3307563900 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bf7bf64250 con 0x55bf7c250d30 2024-05-27T09:45:56.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.648+0000 7fc3eea00700 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf7c250d30 0x55bf7c253150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:45:56.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.658+0000 7fc3eea00700 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf7c250d30 0x55bf7c253150 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68237a6fc1ea25cb server_cookie=1cc23b15b461b583 in_seq=0 out_seq=0 2024-05-27T09:45:56.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.658+0000 7fc3d8c00700 10 client.27310 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:56.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.660+0000 7fc3d8c00700 1 -- 172.21.15.44:0/3307563900 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bf7bf64250 con 0x55bf7c250d30 2024-05-27T09:45:56.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.660+0000 7fc3d8c00700 10 client.27310 handle_command_reply: tid=0 2024-05-27T09:45:56.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.660+0000 7fc404472b80 1 client.27310 shutdown 2024-05-27T09:45:56.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.665+0000 7fc404472b80 1 -- 172.21.15.44:0/3307563900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 msgr2=0x55bf7c1c8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:56.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.665+0000 7fc404472b80 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1c8050 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80a6e20 tx=0x7fc3e8011fc0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.666+0000 7fc404472b80 1 -- 172.21.15.44:0/3307563900 shutdown_connections 2024-05-27T09:45:56.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.666+0000 7fc404472b80 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bf7c250d30 0x55bf7c253150 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.666+0000 7fc404472b80 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf7c1c3e70 0x55bf7c1c8050 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.666+0000 7fc404472b80 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf7c044e40 0x55bf7c1dea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.666+0000 7fc404472b80 1 --2- 172.21.15.44:0/3307563900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bf7c1309f0 0x55bf7c1c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.666+0000 7fc404472b80 1 -- 172.21.15.44:0/3307563900 >> 172.21.15.44:0/3307563900 conn(0x55bf7c0de390 msgr2=0x55bf7c1c4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:56.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.666+0000 7fc3d8c00700 0 client.27310 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:45:56.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.667+0000 7fc404472b80 1 -- 172.21.15.44:0/3307563900 shutdown_connections 2024-05-27T09:45:56.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.667+0000 7fc404472b80 1 -- 172.21.15.44:0/3307563900 wait complete. 2024-05-27T09:45:56.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.667+0000 7fc404472b80 20 client.27310 trim_cache size 0 max 16384 2024-05-27T09:45:56.669 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:45:56.669 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:45:56.669 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:45:56.669 INFO:teuthology.orchestra.run.smithi044.stdout: "5348fc89-9539-4fcf-9fdc-6e30fd88b5c6": { 2024-05-27T09:45:56.670 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:45:56.670 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "5348fc89-9539-4fcf-9fdc-6e30fd88b5c6", 2024-05-27T09:45:56.670 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:45:56.670 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:56.670 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:45:56.670 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:45:56.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.669+0000 7fc3f5a00700 1 -- 172.21.15.44:0/1131951215 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc3d40b29f0 msgr2=0x7fc3d40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.669+0000 7fc3f5a00700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc3d40b29f0 0x7fc3d40b4f00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009fd0 tx=0x7fc3dc0058e0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.669+0000 7fc3f5a00700 1 -- 172.21.15.44:0/1131951215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f010c5b0 msgr2=0x7fc3f01a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:45:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.669+0000 7fc3f5a00700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f010c5b0 0x7fc3f01a1850 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800b070 tx=0x7fc3e8009c90 comp rx=0 tx=0).stop 2024-05-27T09:45:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.669+0000 7fc3f5a00700 1 -- 172.21.15.44:0/1131951215 shutdown_connections 2024-05-27T09:45:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.669+0000 7fc3f5a00700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc3d40b29f0 0x7fc3d40b4f00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.669+0000 7fc3f5a00700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f010c5b0 0x7fc3f01a1850 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.669+0000 7fc3f5a00700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3f0106570 0x7fc3f019d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.669+0000 7fc3f5a00700 1 --2- 172.21.15.44:0/1131951215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f0106ec0 0x7fc3f019d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:45:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.669+0000 7fc3f5a00700 1 -- 172.21.15.44:0/1131951215 >> 172.21.15.44:0/1131951215 conn(0x7fc3f0102070 msgr2=0x7fc3f01045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:45:56.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.670+0000 7fc3f5a00700 1 -- 172.21.15.44:0/1131951215 shutdown_connections 2024-05-27T09:45:56.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:45:56.671+0000 7fc3f5a00700 1 -- 172.21.15.44:0/1131951215 wait complete. 2024-05-27T09:45:57.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:57 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:45:57.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:57 smithi044 ceph-mon[31618]: pgmap v334: 97 pgs: 97 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 1.07k op/s 2024-05-27T09:45:57.522 INFO:tasks.cephfs.filesystem:scrub status for tag:5348fc89-9539-4fcf-9fdc-6e30fd88b5c6 - {'path': '/', 'tag': '5348fc89-9539-4fcf-9fdc-6e30fd88b5c6', 'options': 'recursive,force'} 2024-05-27T09:45:57.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:57 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:45:57.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:57 smithi050 ceph-mon[36371]: pgmap v334: 97 pgs: 97 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 1.07k op/s 2024-05-27T09:45:57.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:57 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:45:57.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:57 smithi189 ceph-mon[36671]: pgmap v334: 97 pgs: 97 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 1.07k op/s 2024-05-27T09:46:00.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:45:59 smithi189 ceph-mon[36671]: pgmap v335: 97 pgs: 97 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 1.06k op/s 2024-05-27T09:46:00.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:45:59 smithi044 ceph-mon[31618]: pgmap v335: 97 pgs: 97 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 1.06k op/s 2024-05-27T09:46:00.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:45:59 smithi050 ceph-mon[36371]: pgmap v335: 97 pgs: 97 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 1.06k op/s 2024-05-27T09:46:02.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:01 smithi189 ceph-mon[36671]: pgmap v336: 97 pgs: 97 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 1.57k op/s 2024-05-27T09:46:02.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:01 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:46:02.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:01 smithi044 ceph-mon[31618]: pgmap v336: 97 pgs: 97 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 1.57k op/s 2024-05-27T09:46:02.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:01 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:46:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:01 smithi050 ceph-mon[36371]: pgmap v336: 97 pgs: 97 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 1.57k op/s 2024-05-27T09:46:02.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:01 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:46:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: pgmap v337: 97 pgs: 97 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 1.57k op/s 2024-05-27T09:46:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T09:46:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:03 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: pgmap v337: 97 pgs: 97 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 1.57k op/s 2024-05-27T09:46:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T09:46:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:03 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: pgmap v337: 97 pgs: 97 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 1.57k op/s 2024-05-27T09:46:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T09:46:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:04.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:03 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:05.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:04 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-05-27T09:46:05.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:04 smithi189 ceph-mon[36671]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-05-27T09:46:05.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:04 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:05.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:04 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:05.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:04 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-05-27T09:46:05.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:04 smithi044 ceph-mon[31618]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-05-27T09:46:05.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:04 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:05.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:04 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:05.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:04 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-05-27T09:46:05.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:04 smithi050 ceph-mon[36371]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-05-27T09:46:05.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:04 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:05.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:04 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:06.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:05 smithi189 ceph-mon[36671]: pgmap v339: 97 pgs: 97 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 1.89k op/s 2024-05-27T09:46:06.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:05 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:05 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:05 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:05 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:05 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:05 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:05 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:05 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:05 smithi044 ceph-mon[31618]: pgmap v339: 97 pgs: 97 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 1.89k op/s 2024-05-27T09:46:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:05 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:05 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:05 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:05 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:05 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:05 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:05 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:05 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:05 smithi050 ceph-mon[36371]: pgmap v339: 97 pgs: 97 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 1.89k op/s 2024-05-27T09:46:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:05 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:05 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:05 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:05 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:05 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:05 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:05 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:06.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:05 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-05-27T09:46:07.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: Unable to set osd_memory_target on smithi044 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:46:07.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:46:07.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:06 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:07.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-05-27T09:46:07.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: Unable to set osd_memory_target on smithi044 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T09:46:07.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:07.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:07.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:46:07.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:46:07.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:06 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:07.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-05-27T09:46:07.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: Unable to set osd_memory_target on smithi044 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T09:46:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T09:46:07.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:46:07.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:46:07.339 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:06 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:46:07.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.617+0000 7feaa4c00640 1 -- 172.21.15.44:0/2705110476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feaa0109f50 msgr2=0x7feaa0103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:07.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.617+0000 7feaa4c00640 1 --2- 172.21.15.44:0/2705110476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feaa0109f50 0x7feaa0103be0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fea88009900 tx=0x7fea8802a3f0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.617+0000 7feaa4c00640 1 -- 172.21.15.44:0/2705110476 shutdown_connections 2024-05-27T09:46:07.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.617+0000 7feaa4c00640 1 --2- 172.21.15.44:0/2705110476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feaa0104250 0x7feaa01105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.617+0000 7feaa4c00640 1 --2- 172.21.15.44:0/2705110476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feaa0109f50 0x7feaa0103be0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.617+0000 7feaa4c00640 1 --2- 172.21.15.44:0/2705110476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaa0109ab0 0x7feaa01036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.617+0000 7feaa4c00640 1 -- 172.21.15.44:0/2705110476 >> 172.21.15.44:0/2705110476 conn(0x7feaa0079240 msgr2=0x7feaa0079680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:07.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.617+0000 7feaa4c00640 1 -- 172.21.15.44:0/2705110476 shutdown_connections 2024-05-27T09:46:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.617+0000 7feaa4c00640 1 -- 172.21.15.44:0/2705110476 wait complete. 2024-05-27T09:46:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7feaa4c00640 1 Processor -- start 2024-05-27T09:46:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7feaa4c00640 1 -- start start 2024-05-27T09:46:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7feaa4c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaa0104250 0x7feaa019a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7feaa4c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feaa0109ab0 0x7feaa019ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7feaa4c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feaa019f000 0x7feaa019f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7feaa4c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0113ca0 con 0x7feaa019f000 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7feaa4c00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0113b20 con 0x7feaa0109ab0 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7feaa4c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0113e20 con 0x7feaa0104250 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7fea9f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaa0104250 0x7feaa019a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7fea9fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feaa019f000 0x7feaa019f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7fea97e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feaa0109ab0 0x7feaa019ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7fea9f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaa0104250 0x7feaa019a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42986/0 (socket says 172.21.15.44:42986) 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.619+0000 7fea9f400640 1 -- 172.21.15.44:0/981528596 learned_addr learned my addr 172.21.15.44:0/981528596 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.620+0000 7fea9fe00640 1 -- 172.21.15.44:0/981528596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaa0104250 msgr2=0x7feaa019a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.620+0000 7fea9fe00640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaa0104250 0x7feaa019a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.620+0000 7fea9fe00640 1 -- 172.21.15.44:0/981528596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feaa0109ab0 msgr2=0x7feaa019ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.620+0000 7fea9fe00640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feaa0109ab0 0x7feaa019ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.620+0000 7fea9fe00640 1 -- 172.21.15.44:0/981528596 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa019fb10 con 0x7feaa019f000 2024-05-27T09:46:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.620+0000 7fea9f400640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaa0104250 0x7feaa019a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:46:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.620+0000 7fea9fe00640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feaa019f000 0x7feaa019f410 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fea98009fd0 tx=0x7fea9800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.621+0000 7fea9cc00640 1 -- 172.21.15.44:0/981528596 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9800ce20 con 0x7feaa019f000 2024-05-27T09:46:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.621+0000 7fea9cc00640 1 -- 172.21.15.44:0/981528596 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea98005020 con 0x7feaa019f000 2024-05-27T09:46:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.621+0000 7feaa4c00640 1 -- 172.21.15.44:0/981528596 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaa019fe00 con 0x7feaa019f000 2024-05-27T09:46:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.621+0000 7fea9cc00640 1 -- 172.21.15.44:0/981528596 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea98010690 con 0x7feaa019f000 2024-05-27T09:46:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.621+0000 7feaa4c00640 1 -- 172.21.15.44:0/981528596 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa01a75b0 con 0x7feaa019f000 2024-05-27T09:46:07.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.624+0000 7feaad1e7740 1 Processor -- start 2024-05-27T09:46:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.625+0000 7fea9cc00640 1 -- 172.21.15.44:0/981528596 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fea980054d0 con 0x7feaa019f000 2024-05-27T09:46:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.625+0000 7feaad1e7740 1 -- start start 2024-05-27T09:46:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.626+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d61fe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.626+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e332f0 0x5594d61fee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.626+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e3b4d0 0x5594d61ff3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.626+0000 7feaad1e7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5594d616fb90 con 0x5594d5e332f0 2024-05-27T09:46:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.626+0000 7feaad1e7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5594d615f730 con 0x5594d5e3b4d0 2024-05-27T09:46:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.626+0000 7feaad1e7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5594d6163770 con 0x5594d5e5c1b0 2024-05-27T09:46:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.626+0000 7fea97e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d61fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.626+0000 7fea97e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d61fe930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43000/0 (socket says 172.21.15.44:43000) 2024-05-27T09:46:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.626+0000 7fea97e00640 1 -- 172.21.15.44:0/1533916864 learned_addr learned my addr 172.21.15.44:0/1533916864 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.626+0000 7fea9f400640 1 --2- 172.21.15.44:0/1533916864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e332f0 0x5594d61fee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.626+0000 7fea97e00640 1 --2- 172.21.15.44:0/1533916864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e3b4d0 0x5594d61ff3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea97e00640 1 -- 172.21.15.44:0/1533916864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e3b4d0 msgr2=0x5594d61ff3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea97e00640 1 --2- 172.21.15.44:0/1533916864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e3b4d0 0x5594d61ff3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea97e00640 1 -- 172.21.15.44:0/1533916864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e332f0 msgr2=0x5594d61fee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea97e00640 1 --2- 172.21.15.44:0/1533916864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e332f0 0x5594d61fee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea97e00640 1 -- 172.21.15.44:0/1533916864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594d60f88b0 con 0x5594d5e5c1b0 2024-05-27T09:46:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea9f400640 1 --2- 172.21.15.44:0/1533916864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e332f0 0x5594d61fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:46:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea9cc00640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fea840b2950 0x7fea840b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea97e00640 1 --2- 172.21.15.44:0/1533916864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d61fe930 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fea88005940 tx=0x7fea88031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bd44886ac7c7769 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea9cc00640 1 -- 172.21.15.44:0/981528596 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8565+0+0 (secure 0 0 0) 0x7fea980e2d40 con 0x7feaa019f000 2024-05-27T09:46:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea8fe00640 1 -- 172.21.15.44:0/1533916864 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea88035040 con 0x5594d5e5c1b0 2024-05-27T09:46:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea9f400640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fea840b2950 0x7fea840b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea8fe00640 1 -- 172.21.15.44:0/1533916864 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea88005030 con 0x5594d5e5c1b0 2024-05-27T09:46:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.627+0000 7fea8fe00640 1 -- 172.21.15.44:0/1533916864 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea880044e0 con 0x5594d5e5c1b0 2024-05-27T09:46:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.629+0000 7feaad1e7740 1 -- 172.21.15.44:0/1533916864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 msgr2=0x5594d61fe930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.629+0000 7fea9f400640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fea840b2950 0x7fea840b4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x5594d61fc2c0 tx=0x7fea90005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.629+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1533916864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d61fe930 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fea88005940 tx=0x7fea88031400 comp rx=0 tx=0).stop 2024-05-27T09:46:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.629+0000 7feaad1e7740 1 -- 172.21.15.44:0/1533916864 shutdown_connections 2024-05-27T09:46:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.629+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1533916864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e3b4d0 0x5594d61ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.629+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1533916864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e332f0 0x5594d61fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.629+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1533916864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d61fe930 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.629+0000 7feaad1e7740 1 -- 172.21.15.44:0/1533916864 >> 172.21.15.44:0/1533916864 conn(0x5594d5e50300 msgr2=0x5594d61fb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.629+0000 7feaad1e7740 1 -- 172.21.15.44:0/1533916864 shutdown_connections 2024-05-27T09:46:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.629+0000 7feaad1e7740 1 -- 172.21.15.44:0/1533916864 wait complete. 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.630+0000 7feaad1e7740 1 -- start start 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.630+0000 7feaad1e7740 1 Processor -- start 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.630+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d620cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.630+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e3b4d0 0x5594d620d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.630+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e5c1b0 0x5594d620da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.630+0000 7feaad1e7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5594d5e45790 con 0x5594d5e5c1b0 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.630+0000 7feaad1e7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5594d6107bc0 con 0x5594d5e332f0 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.630+0000 7feaad1e7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5594d620fff0 con 0x5594d5e3b4d0 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7feaad1e7740 10 client.0 _ll_register_callbacks cb 0x5594d60e5400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea9f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d620cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea97e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e3b4d0 0x5594d620d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea97e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e3b4d0 0x5594d620d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43016/0 (socket says 172.21.15.44:43016) 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea97e00640 1 -- 172.21.15.44:0/2048436788 learned_addr learned my addr 172.21.15.44:0/2048436788 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea9f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d620cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:40544/0 (socket says 172.21.15.44:40544) 2024-05-27T09:46:07.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea9fe00640 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e5c1b0 0x5594d620da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea9f400640 1 -- 172.21.15.44:0/2048436788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e3b4d0 msgr2=0x5594d620d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:07.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea9f400640 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e3b4d0 0x5594d620d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea9f400640 1 -- 172.21.15.44:0/2048436788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e5c1b0 msgr2=0x5594d620da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:07.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea9f400640 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e5c1b0 0x5594d620da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea9f400640 1 -- 172.21.15.44:0/2048436788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594d5e74210 con 0x5594d5e332f0 2024-05-27T09:46:07.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.631+0000 7fea97e00640 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e3b4d0 0x5594d620d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:46:07.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.632+0000 7fea9f400640 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d620cfc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fea90009fd0 tx=0x7fea90004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f35a7615a296ee78 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:07.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.632+0000 7fea8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:46:07.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.632+0000 7fea8cc00640 1 -- 172.21.15.44:0/2048436788 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea90005020 con 0x5594d5e332f0 2024-05-27T09:46:07.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.632+0000 7fea8cc00640 1 -- 172.21.15.44:0/2048436788 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8565+0+0 (secure 0 0 0) 0x7fea90018030 con 0x5594d5e332f0 2024-05-27T09:46:07.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.632+0000 7feaad1e7740 1 -- 172.21.15.44:0/2048436788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea88005030 con 0x5594d5e332f0 2024-05-27T09:46:07.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.632+0000 7fea8cc00640 1 -- 172.21.15.44:0/2048436788 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fea900051c0 con 0x5594d5e332f0 2024-05-27T09:46:07.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.632+0000 7feaad1e7740 10 client.27329 fetch_fsmap learned FSMap version 34 2024-05-27T09:46:07.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.632+0000 7feaad1e7740 1 -- 172.21.15.44:0/2048436788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5594d5f40620 con 0x5594d5e332f0 2024-05-27T09:46:07.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.633+0000 7fea8cc00640 1 -- 172.21.15.44:0/2048436788 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea88005030 con 0x5594d5e332f0 2024-05-27T09:46:07.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.633+0000 7fea8cc00640 1 -- 172.21.15.44:0/2048436788 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fea9001e620 con 0x5594d5e332f0 2024-05-27T09:46:07.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.633+0000 7feaad1e7740 10 client.27329 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:46:07.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.633+0000 7feaad1e7740 10 client.27329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:46:07.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.633+0000 7feaad1e7740 20 client.27329 populate_metadata read hostname 'smithi044' 2024-05-27T09:46:07.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.633+0000 7feaad1e7740 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5594d5e1ad80 0x5594d621f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.633+0000 7feaad1e7740 4 client.27329 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:46:07.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.633+0000 7feaad1e7740 1 -- 172.21.15.44:0/2048436788 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594d6171ae0 con 0x5594d5e1ad80 2024-05-27T09:46:07.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.633+0000 7fea97e00640 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5594d5e1ad80 0x5594d621f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.634+0000 7fea97e00640 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5594d5e1ad80 0x5594d621f3e0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=befc65595cd4e977 server_cookie=a327a4e4f2d94c18 in_seq=0 out_seq=0 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.634+0000 7fea8cc00640 10 client.27329 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.635+0000 7fea8cc00640 1 -- 172.21.15.44:0/2048436788 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5594d6171ae0 con 0x5594d5e1ad80 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.635+0000 7fea8cc00640 10 client.27329 handle_command_reply: tid=0 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 1 client.27329 shutdown 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 1 -- 172.21.15.44:0/2048436788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 msgr2=0x5594d620cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d620cfc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fea90009fd0 tx=0x7fea90004730 comp rx=0 tx=0).stop 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 1 -- 172.21.15.44:0/2048436788 shutdown_connections 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5594d5e1ad80 0x5594d621f3e0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e5c1b0 0x5594d620da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e3b4d0 0x5594d620d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 1 --2- 172.21.15.44:0/2048436788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d620cfc0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 1 -- 172.21.15.44:0/2048436788 >> 172.21.15.44:0/2048436788 conn(0x5594d5e50300 msgr2=0x5594d62010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7fea8cc00640 0 client.27329 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 1 -- 172.21.15.44:0/2048436788 shutdown_connections 2024-05-27T09:46:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 1 -- 172.21.15.44:0/2048436788 wait complete. 2024-05-27T09:46:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.636+0000 7feaad1e7740 20 client.27329 trim_cache size 0 max 16384 2024-05-27T09:46:07.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.654+0000 7feaad1e7740 1 Processor -- start 2024-05-27T09:46:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.654+0000 7feaad1e7740 1 -- start start 2024-05-27T09:46:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.654+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e332f0 0x5594d620f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.654+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e3b4d0 0x5594d61ff220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.654+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d620cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.654+0000 7feaad1e7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01b0750 con 0x5594d5e332f0 2024-05-27T09:46:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.654+0000 7feaad1e7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01adfc0 con 0x5594d5e3b4d0 2024-05-27T09:46:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.654+0000 7feaad1e7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01abe50 con 0x5594d5e5c1b0 2024-05-27T09:46:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.655+0000 7fea9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d620cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.655+0000 7fea9f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e332f0 0x5594d620f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.655+0000 7fea97e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e3b4d0 0x5594d61ff220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.655+0000 7fea9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d620cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43022/0 (socket says 172.21.15.44:43022) 2024-05-27T09:46:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.655+0000 7fea9fe00640 1 -- 172.21.15.44:0/1940598777 learned_addr learned my addr 172.21.15.44:0/1940598777 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.655+0000 7fea97e00640 1 -- 172.21.15.44:0/1940598777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 msgr2=0x5594d620cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.655+0000 7fea97e00640 1 --2- 172.21.15.44:0/1940598777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d620cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.655+0000 7fea97e00640 1 -- 172.21.15.44:0/1940598777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e332f0 msgr2=0x5594d620f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.655+0000 7fea97e00640 1 --2- 172.21.15.44:0/1940598777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e332f0 0x5594d620f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.655+0000 7fea97e00640 1 -- 172.21.15.44:0/1940598777 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594d61f6230 con 0x5594d5e3b4d0 2024-05-27T09:46:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.655+0000 7fea97e00640 1 --2- 172.21.15.44:0/1940598777 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e3b4d0 0x5594d61ff220 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x5594d621fb80 tx=0x7fea88031a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2783407c95f10551 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.656+0000 7fea8f400640 1 -- 172.21.15.44:0/1940598777 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea88035040 con 0x5594d5e3b4d0 2024-05-27T09:46:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.656+0000 7fea8f400640 1 -- 172.21.15.44:0/1940598777 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea88041040 con 0x5594d5e3b4d0 2024-05-27T09:46:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.656+0000 7fea8f400640 1 -- 172.21.15.44:0/1940598777 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea880055f0 con 0x5594d5e3b4d0 2024-05-27T09:46:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.657+0000 7feaad1e7740 1 -- 172.21.15.44:0/1940598777 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e3b4d0 msgr2=0x5594d61ff220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.657+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1940598777 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e3b4d0 0x5594d61ff220 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x5594d621fb80 tx=0x7fea88031a10 comp rx=0 tx=0).stop 2024-05-27T09:46:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.657+0000 7feaad1e7740 1 -- 172.21.15.44:0/1940598777 shutdown_connections 2024-05-27T09:46:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.657+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1940598777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d620cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.657+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1940598777 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e3b4d0 0x5594d61ff220 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.657+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1940598777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e332f0 0x5594d620f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.657+0000 7feaad1e7740 1 -- 172.21.15.44:0/1940598777 >> 172.21.15.44:0/1940598777 conn(0x5594d5e50300 msgr2=0x5594d620c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.657+0000 7feaad1e7740 1 -- 172.21.15.44:0/1940598777 shutdown_connections 2024-05-27T09:46:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.657+0000 7feaad1e7740 1 -- 172.21.15.44:0/1940598777 wait complete. 2024-05-27T09:46:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.657+0000 7feaad1e7740 1 -- start start 2024-05-27T09:46:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.657+0000 7feaad1e7740 1 Processor -- start 2024-05-27T09:46:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d6206cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e3b4d0 0x5594d6207210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7feaad1e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d6207750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7feaad1e7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01ac4a0 con 0x5594d5e3b4d0 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7feaad1e7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01ad8f0 con 0x5594d5e332f0 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7feaad1e7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01acb50 con 0x5594d5e5c1b0 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7feaad1e7740 10 client.0 _ll_register_callbacks cb 0x7fea90009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7fea9f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d6206cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7fea9f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d6206cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:40560/0 (socket says 172.21.15.44:40560) 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7fea9f400640 1 -- 172.21.15.44:0/1744385083 learned_addr learned my addr 172.21.15.44:0/1744385083 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7fea9fe00640 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d6207750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.658+0000 7fea97e00640 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e3b4d0 0x5594d6207210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7fea9f400640 1 -- 172.21.15.44:0/1744385083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 msgr2=0x5594d6207750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7fea9f400640 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d6207750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7fea9f400640 1 -- 172.21.15.44:0/1744385083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e3b4d0 msgr2=0x5594d6207210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7fea9f400640 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e3b4d0 0x5594d6207210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7fea9f400640 1 -- 172.21.15.44:0/1744385083 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594d62142b0 con 0x5594d5e332f0 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7fea9f400640 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d6206cd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fea90009fa0 tx=0x7fea9001ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1cd5989ba30e796 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7fea8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7fea8cc00640 1 -- 172.21.15.44:0/1744385083 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9001e250 con 0x5594d5e332f0 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7feaad1e7740 1 -- 172.21.15.44:0/1744385083 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea88041040 con 0x5594d5e332f0 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7fea8cc00640 1 -- 172.21.15.44:0/1744385083 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8565+0+0 (secure 0 0 0) 0x7fea90005360 con 0x5594d5e332f0 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7fea8cc00640 1 -- 172.21.15.44:0/1744385083 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fea90005560 con 0x5594d5e332f0 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7feaad1e7740 10 client.27335 fetch_fsmap learned FSMap version 34 2024-05-27T09:46:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.659+0000 7feaad1e7740 1 -- 172.21.15.44:0/1744385083 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5594d62154e0 con 0x5594d5e332f0 2024-05-27T09:46:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.660+0000 7fea8cc00640 1 -- 172.21.15.44:0/1744385083 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea88041040 con 0x5594d5e332f0 2024-05-27T09:46:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.660+0000 7fea8cc00640 1 -- 172.21.15.44:0/1744385083 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fea900424d0 con 0x5594d5e332f0 2024-05-27T09:46:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.660+0000 7feaad1e7740 10 client.27335 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:46:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.660+0000 7feaad1e7740 10 client.27335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:46:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.660+0000 7feaad1e7740 20 client.27335 populate_metadata read hostname 'smithi044' 2024-05-27T09:46:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.660+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5594d5df01c0 0x5594d6289ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.660+0000 7feaad1e7740 4 client.27335 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:46:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.660+0000 7feaad1e7740 1 -- 172.21.15.44:0/1744385083 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5594d616b2c0 con 0x5594d5df01c0 2024-05-27T09:46:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.660+0000 7fea97e00640 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5594d5df01c0 0x5594d6289ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.661+0000 7fea97e00640 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5594d5df01c0 0x5594d6289ac0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb51032b1a0e0100 server_cookie=3ce259c1ed5cf9d4 in_seq=0 out_seq=0 2024-05-27T09:46:07.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.661+0000 7fea8cc00640 10 client.27335 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.979+0000 7fea8cc00640 1 -- 172.21.15.44:0/1744385083 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5594d616b2c0 con 0x5594d5df01c0 2024-05-27T09:46:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.979+0000 7fea8cc00640 10 client.27335 handle_command_reply: tid=0 2024-05-27T09:46:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7feaad1e7740 1 client.27335 shutdown 2024-05-27T09:46:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7feaad1e7740 1 -- 172.21.15.44:0/1744385083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 msgr2=0x5594d6206cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d6206cd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fea90009fa0 tx=0x7fea9001ef10 comp rx=0 tx=0).stop 2024-05-27T09:46:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7feaad1e7740 1 -- 172.21.15.44:0/1744385083 shutdown_connections 2024-05-27T09:46:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5594d5df01c0 0x5594d6289ac0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594d5e5c1b0 0x5594d6207750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7fea8cc00640 0 client.27335 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5594d5e3b4d0 0x5594d6207210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7feaad1e7740 1 --2- 172.21.15.44:0/1744385083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5594d5e332f0 0x5594d6206cd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7feaad1e7740 1 -- 172.21.15.44:0/1744385083 >> 172.21.15.44:0/1744385083 conn(0x5594d5e50300 msgr2=0x5594d620d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7feaad1e7740 1 -- 172.21.15.44:0/1744385083 shutdown_connections 2024-05-27T09:46:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.980+0000 7feaad1e7740 1 -- 172.21.15.44:0/1744385083 wait complete. 2024-05-27T09:46:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.981+0000 7feaad1e7740 20 client.27335 trim_cache size 0 max 16384 2024-05-27T09:46:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:46:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:46:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:46:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.981+0000 7feaa4c00640 1 -- 172.21.15.44:0/981528596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fea840b2950 msgr2=0x7fea840b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.981+0000 7feaa4c00640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fea840b2950 0x7fea840b4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x5594d61fc2c0 tx=0x7fea90005f90 comp rx=0 tx=0).stop 2024-05-27T09:46:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.981+0000 7feaa4c00640 1 -- 172.21.15.44:0/981528596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feaa019f000 msgr2=0x7feaa019f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.981+0000 7feaa4c00640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feaa019f000 0x7feaa019f410 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fea98009fd0 tx=0x7fea9800ef10 comp rx=0 tx=0).stop 2024-05-27T09:46:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.982+0000 7feaa4c00640 1 -- 172.21.15.44:0/981528596 shutdown_connections 2024-05-27T09:46:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.982+0000 7feaa4c00640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fea840b2950 0x7fea840b4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.982+0000 7feaa4c00640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feaa019f000 0x7feaa019f410 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.982+0000 7feaa4c00640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feaa0109ab0 0x7feaa019ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.982+0000 7feaa4c00640 1 --2- 172.21.15.44:0/981528596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaa0104250 0x7feaa019a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.982+0000 7feaa4c00640 1 -- 172.21.15.44:0/981528596 >> 172.21.15.44:0/981528596 conn(0x7feaa0079240 msgr2=0x7feaa0102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.982+0000 7feaa4c00640 1 -- 172.21.15.44:0/981528596 shutdown_connections 2024-05-27T09:46:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:07.982+0000 7feaa4c00640 1 -- 172.21.15.44:0/981528596 wait complete. 2024-05-27T09:46:08.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:07 smithi044 ceph-mon[31618]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-27T09:46:08.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:07 smithi044 ceph-mon[31618]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T09:46:08.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:07 smithi044 ceph-mon[31618]: pgmap v340: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 1.09k op/s 2024-05-27T09:46:08.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:07 smithi044 ceph-mon[31618]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T09:46:08.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:07 smithi050 ceph-mon[36371]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-27T09:46:08.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:07 smithi050 ceph-mon[36371]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T09:46:08.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:07 smithi050 ceph-mon[36371]: pgmap v340: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 1.09k op/s 2024-05-27T09:46:08.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:07 smithi050 ceph-mon[36371]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T09:46:08.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:07 smithi189 ceph-mon[36671]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-05-27T09:46:08.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:07 smithi189 ceph-mon[36671]: Unable to set osd_memory_target on smithi189 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-27T09:46:08.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:07 smithi189 ceph-mon[36671]: pgmap v340: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 1.09k op/s 2024-05-27T09:46:08.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:07 smithi189 ceph-mon[36671]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T09:46:10.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:09 smithi044 ceph-mon[31618]: pgmap v342: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 591 op/s 2024-05-27T09:46:10.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:09 smithi050 ceph-mon[36371]: pgmap v342: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 591 op/s 2024-05-27T09:46:10.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:09 smithi189 ceph-mon[36671]: pgmap v342: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 591 op/s 2024-05-27T09:46:12.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:11 smithi044 ceph-mon[31618]: pgmap v343: 97 pgs: 97 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.9 MiB/s wr, 1.20k op/s 2024-05-27T09:46:12.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:11 smithi050 ceph-mon[36371]: pgmap v343: 97 pgs: 97 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.9 MiB/s wr, 1.20k op/s 2024-05-27T09:46:12.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:11 smithi189 ceph-mon[36671]: pgmap v343: 97 pgs: 97 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.9 MiB/s wr, 1.20k op/s 2024-05-27T09:46:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:13 smithi044 ceph-mon[31618]: pgmap v344: 97 pgs: 97 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 963 op/s 2024-05-27T09:46:14.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:13 smithi050 ceph-mon[36371]: pgmap v344: 97 pgs: 97 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 963 op/s 2024-05-27T09:46:14.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:13 smithi189 ceph-mon[36671]: pgmap v344: 97 pgs: 97 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 963 op/s 2024-05-27T09:46:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:15 smithi050 ceph-mon[36371]: pgmap v345: 97 pgs: 97 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 956 op/s 2024-05-27T09:46:16.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:15 smithi189 ceph-mon[36671]: pgmap v345: 97 pgs: 97 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 956 op/s 2024-05-27T09:46:16.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:15 smithi044 ceph-mon[31618]: pgmap v345: 97 pgs: 97 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 956 op/s 2024-05-27T09:46:18.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:17 smithi050 ceph-mon[36371]: pgmap v346: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 706 op/s 2024-05-27T09:46:18.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:17 smithi189 ceph-mon[36671]: pgmap v346: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 706 op/s 2024-05-27T09:46:18.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:17 smithi044 ceph-mon[31618]: pgmap v346: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 706 op/s 2024-05-27T09:46:20.241 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:19 smithi044 ceph-mon[31618]: pgmap v347: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 649 op/s 2024-05-27T09:46:20.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:19 smithi050 ceph-mon[36371]: pgmap v347: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 649 op/s 2024-05-27T09:46:20.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:19 smithi189 ceph-mon[36671]: pgmap v347: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 649 op/s 2024-05-27T09:46:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.231+0000 7ff739800640 1 -- 172.21.15.44:0/253508696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff73410bcd0 msgr2=0x7ff734105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.231+0000 7ff739800640 1 --2- 172.21.15.44:0/253508696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff73410bcd0 0x7ff734105870 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009ca0 tx=0x7ff72c02a8c0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.232+0000 7ff739800640 1 -- 172.21.15.44:0/253508696 shutdown_connections 2024-05-27T09:46:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.232+0000 7ff739800640 1 --2- 172.21.15.44:0/253508696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff734106420 0x7ff7341127f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.232+0000 7ff739800640 1 --2- 172.21.15.44:0/253508696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff73410c170 0x7ff734105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.232+0000 7ff739800640 1 --2- 172.21.15.44:0/253508696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff73410bcd0 0x7ff734105870 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.232+0000 7ff739800640 1 -- 172.21.15.44:0/253508696 >> 172.21.15.44:0/253508696 conn(0x7ff7341013d0 msgr2=0x7ff734103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.233+0000 7ff739800640 1 -- 172.21.15.44:0/253508696 shutdown_connections 2024-05-27T09:46:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.233+0000 7ff739800640 1 -- 172.21.15.44:0/253508696 wait complete. 2024-05-27T09:46:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.235+0000 7ff739800640 1 Processor -- start 2024-05-27T09:46:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.235+0000 7ff739800640 1 -- start start 2024-05-27T09:46:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.235+0000 7ff739800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff734106420 0x7ff734078430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.235+0000 7ff739800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff73410bcd0 0x7ff734078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.235+0000 7ff739800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff73410c170 0x7ff734078eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.236+0000 7ff739800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff734115ef0 con 0x7ff734106420 2024-05-27T09:46:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.236+0000 7ff739800640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff734115d70 con 0x7ff73410c170 2024-05-27T09:46:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.236+0000 7ff739800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff734116070 con 0x7ff73410bcd0 2024-05-27T09:46:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.236+0000 7ff738e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff73410c170 0x7ff734078eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.236+0000 7ff733e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff734106420 0x7ff734078430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.236+0000 7ff738e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff73410c170 0x7ff734078eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:33712/0 (socket says 172.21.15.44:33712) 2024-05-27T09:46:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.236+0000 7ff738e00640 1 -- 172.21.15.44:0/2453062248 learned_addr learned my addr 172.21.15.44:0/2453062248 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.236+0000 7ff733400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff73410bcd0 0x7ff734078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.237+0000 7ff738e00640 1 -- 172.21.15.44:0/2453062248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff73410bcd0 msgr2=0x7ff734078970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.237+0000 7ff738e00640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff73410bcd0 0x7ff734078970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.237+0000 7ff738e00640 1 -- 172.21.15.44:0/2453062248 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff734106420 msgr2=0x7ff734078430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.238+0000 7ff738e00640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff734106420 0x7ff734078430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.238+0000 7ff738e00640 1 -- 172.21.15.44:0/2453062248 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7341a2600 con 0x7ff73410c170 2024-05-27T09:46:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.238+0000 7ff733e00640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff734106420 0x7ff734078430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:46:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.239+0000 7ff738e00640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff73410c170 0x7ff734078eb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff72400e970 tx=0x7ff72400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.240+0000 7ff730c00640 1 -- 172.21.15.44:0/2453062248 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72400ccc0 con 0x7ff73410c170 2024-05-27T09:46:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.240+0000 7ff730c00640 1 -- 172.21.15.44:0/2453062248 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff72400ce60 con 0x7ff73410c170 2024-05-27T09:46:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.240+0000 7ff739800640 1 -- 172.21.15.44:0/2453062248 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7341a28f0 con 0x7ff73410c170 2024-05-27T09:46:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.240+0000 7ff730c00640 1 -- 172.21.15.44:0/2453062248 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff724010430 con 0x7ff73410c170 2024-05-27T09:46:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.241+0000 7ff739800640 1 -- 172.21.15.44:0/2453062248 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7341a2df0 con 0x7ff73410c170 2024-05-27T09:46:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.248+0000 7ff741de9740 1 Processor -- start 2024-05-27T09:46:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.251+0000 7ff730c00640 1 -- 172.21.15.44:0/2453062248 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff724004ff0 con 0x7ff73410c170 2024-05-27T09:46:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.251+0000 7ff741de9740 1 -- start start 2024-05-27T09:46:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.252+0000 7ff741de9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e611b0 0x55ef22203a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.252+0000 7ff741de9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22203fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.252+0000 7ff741de9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 0x55ef222044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.252+0000 7ff741de9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ef22174b90 con 0x55ef21e404d0 2024-05-27T09:46:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.252+0000 7ff741de9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ef22164730 con 0x55ef21e382f0 2024-05-27T09:46:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.252+0000 7ff741de9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ef22168770 con 0x55ef21e611b0 2024-05-27T09:46:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.253+0000 7ff733e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22203fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.253+0000 7ff733e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22203fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:33728/0 (socket says 172.21.15.44:33728) 2024-05-27T09:46:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.253+0000 7ff733e00640 1 -- 172.21.15.44:0/649120873 learned_addr learned my addr 172.21.15.44:0/649120873 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.254+0000 7ff733400640 1 --2- 172.21.15.44:0/649120873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 0x55ef222044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.255+0000 7ff733400640 1 --2- 172.21.15.44:0/649120873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e611b0 0x55ef22203a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.256+0000 7ff730c00640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7180b29a0 0x7ff7180b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.256+0000 7ff730c00640 1 -- 172.21.15.44:0/2453062248 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7ff7240e2ab0 con 0x7ff73410c170 2024-05-27T09:46:21.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.257+0000 7ff733e00640 1 -- 172.21.15.44:0/649120873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e611b0 msgr2=0x55ef22203a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:21.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.257+0000 7ff733e00640 1 --2- 172.21.15.44:0/649120873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e611b0 0x55ef22203a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.257+0000 7ff733e00640 1 -- 172.21.15.44:0/649120873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 msgr2=0x55ef222044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.257+0000 7ff738e00640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7180b29a0 0x7ff7180b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.258+0000 7ff733e00640 1 --2- 172.21.15.44:0/649120873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 0x55ef222044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.259+0000 7ff733e00640 1 -- 172.21.15.44:0/649120873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef220fd8b0 con 0x55ef21e382f0 2024-05-27T09:46:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.259+0000 7ff733400640 1 --2- 172.21.15.44:0/649120873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 0x55ef222044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:46:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.259+0000 7ff733e00640 1 --2- 172.21.15.44:0/649120873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22203fa0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009dc0 tx=0x7ff72c0094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feb5914009d3d58d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.259+0000 7ff729600640 1 -- 172.21.15.44:0/649120873 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c031cf0 con 0x55ef21e382f0 2024-05-27T09:46:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.260+0000 7ff729600640 1 -- 172.21.15.44:0/649120873 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff72c036070 con 0x55ef21e382f0 2024-05-27T09:46:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.260+0000 7ff729600640 1 -- 172.21.15.44:0/649120873 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c0042a0 con 0x55ef21e382f0 2024-05-27T09:46:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.260+0000 7ff738e00640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7180b29a0 0x7ff7180b4eb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff72400c6a0 tx=0x7ff7240e2db0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.261+0000 7ff741de9740 1 -- 172.21.15.44:0/649120873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 msgr2=0x55ef22203fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.261+0000 7ff741de9740 1 --2- 172.21.15.44:0/649120873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22203fa0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009dc0 tx=0x7ff72c0094a0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.261+0000 7ff741de9740 1 -- 172.21.15.44:0/649120873 shutdown_connections 2024-05-27T09:46:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.261+0000 7ff741de9740 1 --2- 172.21.15.44:0/649120873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 0x55ef222044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.261+0000 7ff741de9740 1 --2- 172.21.15.44:0/649120873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22203fa0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.261+0000 7ff741de9740 1 --2- 172.21.15.44:0/649120873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e611b0 0x55ef22203a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.261+0000 7ff741de9740 1 -- 172.21.15.44:0/649120873 >> 172.21.15.44:0/649120873 conn(0x55ef21e55300 msgr2=0x55ef22200210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.262+0000 7ff741de9740 1 -- 172.21.15.44:0/649120873 shutdown_connections 2024-05-27T09:46:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.262+0000 7ff741de9740 1 -- 172.21.15.44:0/649120873 wait complete. 2024-05-27T09:46:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.263+0000 7ff741de9740 1 -- start start 2024-05-27T09:46:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.265+0000 7ff741de9740 1 Processor -- start 2024-05-27T09:46:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.267+0000 7ff741de9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e382f0 0x55ef2220dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.267+0000 7ff741de9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 0x55ef2220e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.267+0000 7ff741de9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e611b0 0x55ef22219e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.267+0000 7ff741de9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ef21e4a790 con 0x55ef21e382f0 2024-05-27T09:46:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.267+0000 7ff741de9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ef2210cbc0 con 0x55ef21e611b0 2024-05-27T09:46:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.267+0000 7ff741de9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ef22210b30 con 0x55ef21e404d0 2024-05-27T09:46:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.267+0000 7ff741de9740 10 client.0 _ll_register_callbacks cb 0x55ef220ea400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:46:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.269+0000 7ff733e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 0x55ef2220e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.269+0000 7ff733e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 0x55ef2220e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60318/0 (socket says 172.21.15.44:60318) 2024-05-27T09:46:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.269+0000 7ff733e00640 1 -- 172.21.15.44:0/1709491961 learned_addr learned my addr 172.21.15.44:0/1709491961 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.270+0000 7ff733400640 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e611b0 0x55ef22219e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.270+0000 7ff733400640 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e382f0 0x55ef2220dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.271+0000 7ff733e00640 1 -- 172.21.15.44:0/1709491961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e611b0 msgr2=0x55ef22219e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.271+0000 7ff733e00640 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e611b0 0x55ef22219e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.271+0000 7ff733e00640 1 -- 172.21.15.44:0/1709491961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e382f0 msgr2=0x55ef2220dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.271+0000 7ff733e00640 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e382f0 0x55ef2220dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.271+0000 7ff733e00640 1 -- 172.21.15.44:0/1709491961 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef21e79210 con 0x55ef21e404d0 2024-05-27T09:46:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.272+0000 7ff733e00640 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 0x55ef2220e150 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009d90 tx=0x7ff72c02f380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62f87652c060191c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.272+0000 7ff71e000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:46:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.274+0000 7ff71e000640 1 -- 172.21.15.44:0/1709491961 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c049040 con 0x55ef21e404d0 2024-05-27T09:46:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.274+0000 7ff71e000640 1 -- 172.21.15.44:0/1709491961 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7ff72c031cf0 con 0x55ef21e404d0 2024-05-27T09:46:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.274+0000 7ff741de9740 1 -- 172.21.15.44:0/1709491961 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff72c036070 con 0x55ef21e404d0 2024-05-27T09:46:21.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.275+0000 7ff71e000640 1 -- 172.21.15.44:0/1709491961 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff72c05b050 con 0x55ef21e404d0 2024-05-27T09:46:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.275+0000 7ff741de9740 10 client.27340 fetch_fsmap learned FSMap version 34 2024-05-27T09:46:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.275+0000 7ff741de9740 1 -- 172.21.15.44:0/1709491961 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ef21f45620 con 0x55ef21e404d0 2024-05-27T09:46:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.277+0000 7ff71e000640 1 -- 172.21.15.44:0/1709491961 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff72c036070 con 0x55ef21e404d0 2024-05-27T09:46:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.277+0000 7ff71e000640 1 -- 172.21.15.44:0/1709491961 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7ff72c04c070 con 0x55ef21e404d0 2024-05-27T09:46:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.277+0000 7ff741de9740 10 client.27340 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:46:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.277+0000 7ff741de9740 10 client.27340 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:46:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.277+0000 7ff741de9740 20 client.27340 populate_metadata read hostname 'smithi044' 2024-05-27T09:46:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.277+0000 7ff741de9740 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef21e1fd80 0x55ef2221fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.277+0000 7ff741de9740 4 client.27340 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:46:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.277+0000 7ff741de9740 1 -- 172.21.15.44:0/1709491961 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef22176ae0 con 0x55ef21e1fd80 2024-05-27T09:46:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.278+0000 7ff733400640 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef21e1fd80 0x55ef2221fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.278+0000 7ff71e000640 10 client.27340 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.279+0000 7ff733400640 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef21e1fd80 0x55ef2221fc80 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34aee934bbe641a9 server_cookie=a8933b6c11fa2b53 in_seq=0 out_seq=0 2024-05-27T09:46:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.281+0000 7ff71e000640 1 -- 172.21.15.44:0/1709491961 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ef22176ae0 con 0x55ef21e1fd80 2024-05-27T09:46:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.281+0000 7ff71e000640 10 client.27340 handle_command_reply: tid=0 2024-05-27T09:46:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.281+0000 7ff741de9740 1 client.27340 shutdown 2024-05-27T09:46:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.283+0000 7ff741de9740 1 -- 172.21.15.44:0/1709491961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 msgr2=0x55ef2220e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.283+0000 7ff741de9740 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 0x55ef2220e150 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009d90 tx=0x7ff72c02f380 comp rx=0 tx=0).stop 2024-05-27T09:46:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.283+0000 7ff741de9740 1 -- 172.21.15.44:0/1709491961 shutdown_connections 2024-05-27T09:46:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.284+0000 7ff741de9740 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef21e1fd80 0x55ef2221fc80 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.284+0000 7ff741de9740 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e611b0 0x55ef22219e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.284+0000 7ff71e000640 0 client.27340 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.284+0000 7ff741de9740 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 0x55ef2220e150 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.284+0000 7ff741de9740 1 --2- 172.21.15.44:0/1709491961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e382f0 0x55ef2220dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.284+0000 7ff741de9740 1 -- 172.21.15.44:0/1709491961 >> 172.21.15.44:0/1709491961 conn(0x55ef21e55300 msgr2=0x55ef22203890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.286+0000 7ff741de9740 1 -- 172.21.15.44:0/1709491961 shutdown_connections 2024-05-27T09:46:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.286+0000 7ff741de9740 1 -- 172.21.15.44:0/1709491961 wait complete. 2024-05-27T09:46:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.287+0000 7ff741de9740 20 client.27340 trim_cache size 0 max 16384 2024-05-27T09:46:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.325+0000 7ff741de9740 1 Processor -- start 2024-05-27T09:46:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.326+0000 7ff741de9740 1 -- start start 2024-05-27T09:46:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.326+0000 7ff741de9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e1fd80 0x55ef2220e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.326+0000 7ff741de9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.326+0000 7ff741de9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 0x55ef222231f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.326+0000 7ff741de9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7341c11a0 con 0x55ef21e404d0 2024-05-27T09:46:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.326+0000 7ff741de9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff7341bea10 con 0x55ef21e382f0 2024-05-27T09:46:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.326+0000 7ff741de9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7341bc880 con 0x55ef21e1fd80 2024-05-27T09:46:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.326+0000 7ff733e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.326+0000 7ff733e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:33748/0 (socket says 172.21.15.44:33748) 2024-05-27T09:46:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.326+0000 7ff733e00640 1 -- 172.21.15.44:0/2487548396 learned_addr learned my addr 172.21.15.44:0/2487548396 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.326+0000 7ff733400640 1 --2- 172.21.15.44:0/2487548396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e1fd80 0x55ef2220e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.327+0000 7ff733400640 1 --2- 172.21.15.44:0/2487548396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 0x55ef222231f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.327+0000 7ff733e00640 1 -- 172.21.15.44:0/2487548396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e1fd80 msgr2=0x55ef2220e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.327+0000 7ff733e00640 1 --2- 172.21.15.44:0/2487548396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e1fd80 0x55ef2220e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.327+0000 7ff733e00640 1 -- 172.21.15.44:0/2487548396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 msgr2=0x55ef222231f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.327+0000 7ff733e00640 1 --2- 172.21.15.44:0/2487548396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 0x55ef222231f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.327+0000 7ff733e00640 1 -- 172.21.15.44:0/2487548396 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef22204810 con 0x55ef21e382f0 2024-05-27T09:46:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.327+0000 7ff733400640 1 --2- 172.21.15.44:0/2487548396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 0x55ef222231f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:46:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.327+0000 7ff733e00640 1 --2- 172.21.15.44:0/2487548396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222cb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff72c006010 tx=0x7ff72c005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d48abb38edebb3fb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.328+0000 7ff728c00640 1 -- 172.21.15.44:0/2487548396 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c02fd00 con 0x55ef21e382f0 2024-05-27T09:46:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.328+0000 7ff728c00640 1 -- 172.21.15.44:0/2487548396 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff72c036040 con 0x55ef21e382f0 2024-05-27T09:46:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.328+0000 7ff728c00640 1 -- 172.21.15.44:0/2487548396 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c031870 con 0x55ef21e382f0 2024-05-27T09:46:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.329+0000 7ff741de9740 1 -- 172.21.15.44:0/2487548396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 msgr2=0x55ef22222cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.329+0000 7ff741de9740 1 --2- 172.21.15.44:0/2487548396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222cb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff72c006010 tx=0x7ff72c005de0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.329+0000 7ff741de9740 1 -- 172.21.15.44:0/2487548396 shutdown_connections 2024-05-27T09:46:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.329+0000 7ff741de9740 1 --2- 172.21.15.44:0/2487548396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e404d0 0x55ef222231f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.329+0000 7ff741de9740 1 --2- 172.21.15.44:0/2487548396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222cb0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.329+0000 7ff741de9740 1 --2- 172.21.15.44:0/2487548396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e1fd80 0x55ef2220e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.329+0000 7ff741de9740 1 -- 172.21.15.44:0/2487548396 >> 172.21.15.44:0/2487548396 conn(0x55ef21e55300 msgr2=0x55ef2220f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.329+0000 7ff741de9740 1 -- 172.21.15.44:0/2487548396 shutdown_connections 2024-05-27T09:46:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.329+0000 7ff741de9740 1 -- 172.21.15.44:0/2487548396 wait complete. 2024-05-27T09:46:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.330+0000 7ff741de9740 1 -- start start 2024-05-27T09:46:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.330+0000 7ff741de9740 1 Processor -- start 2024-05-27T09:46:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.330+0000 7ff741de9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e1fd80 0x55ef22222230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.330+0000 7ff741de9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.330+0000 7ff741de9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 0x55ef222860d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.330+0000 7ff741de9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7341bced0 con 0x55ef21e1fd80 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.330+0000 7ff741de9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff7341be340 con 0x55ef21e382f0 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.330+0000 7ff741de9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7341bd5a0 con 0x55ef21e404d0 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.330+0000 7ff741de9740 10 client.0 _ll_register_callbacks cb 0x7ff724005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e1fd80 0x55ef22222230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:33754/0 (socket says 172.21.15.44:33754) 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733e00640 1 -- 172.21.15.44:0/2722345753 learned_addr learned my addr 172.21.15.44:0/2722345753 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733400640 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 0x55ef222860d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733e00640 1 -- 172.21.15.44:0/2722345753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 msgr2=0x55ef222860d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733e00640 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 0x55ef222860d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733e00640 1 -- 172.21.15.44:0/2722345753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e1fd80 msgr2=0x55ef22222230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733e00640 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e1fd80 0x55ef22222230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733e00640 1 -- 172.21.15.44:0/2722345753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef22286640 con 0x55ef21e382f0 2024-05-27T09:46:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733400640 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e1fd80 0x55ef22222230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:46:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff733e00640 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222770 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff72c02fb10 tx=0x7ff72c02ff30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89d2cf2ab6f4fab3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.331+0000 7ff71e000640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:46:21.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff71e000640 1 -- 172.21.15.44:0/2722345753 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c031cf0 con 0x55ef21e382f0 2024-05-27T09:46:21.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff71e000640 1 -- 172.21.15.44:0/2722345753 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7ff72c0314f0 con 0x55ef21e382f0 2024-05-27T09:46:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff741de9740 1 -- 172.21.15.44:0/2722345753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff72c036040 con 0x55ef21e382f0 2024-05-27T09:46:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff71e000640 1 -- 172.21.15.44:0/2722345753 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff72c06e050 con 0x55ef21e382f0 2024-05-27T09:46:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff741de9740 10 client.27353 fetch_fsmap learned FSMap version 34 2024-05-27T09:46:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff741de9740 1 -- 172.21.15.44:0/2722345753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ef22287800 con 0x55ef21e382f0 2024-05-27T09:46:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff71e000640 1 -- 172.21.15.44:0/2722345753 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff72c036040 con 0x55ef21e382f0 2024-05-27T09:46:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff71e000640 1 -- 172.21.15.44:0/2722345753 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7ff72c04c040 con 0x55ef21e382f0 2024-05-27T09:46:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff741de9740 10 client.27353 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:46:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff741de9740 10 client.27353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:46:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff741de9740 20 client.27353 populate_metadata read hostname 'smithi044' 2024-05-27T09:46:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff741de9740 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef21e611b0 0x55ef22289e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff741de9740 4 client.27353 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:46:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.332+0000 7ff741de9740 1 -- 172.21.15.44:0/2722345753 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ef221702c0 con 0x55ef21e611b0 2024-05-27T09:46:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.333+0000 7ff733400640 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef21e611b0 0x55ef22289e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.333+0000 7ff71e000640 10 client.27353 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:21.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.333+0000 7ff733400640 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef21e611b0 0x55ef22289e80 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=860b436e82c55e7b server_cookie=4d2cf4a4d0307c75 in_seq=0 out_seq=0 2024-05-27T09:46:21.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.807+0000 7ff71e000640 1 -- 172.21.15.44:0/2722345753 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55ef221702c0 con 0x55ef21e611b0 2024-05-27T09:46:21.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.807+0000 7ff71e000640 10 client.27353 handle_command_reply: tid=0 2024-05-27T09:46:21.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.807+0000 7ff741de9740 1 client.27353 shutdown 2024-05-27T09:46:21.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.808+0000 7ff741de9740 1 -- 172.21.15.44:0/2722345753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 msgr2=0x55ef22222770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:21.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.808+0000 7ff741de9740 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222770 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff72c02fb10 tx=0x7ff72c02ff30 comp rx=0 tx=0).stop 2024-05-27T09:46:21.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.808+0000 7ff741de9740 1 -- 172.21.15.44:0/2722345753 shutdown_connections 2024-05-27T09:46:21.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.808+0000 7ff741de9740 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ef21e611b0 0x55ef22289e80 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.808+0000 7ff741de9740 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef21e404d0 0x55ef222860d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.808+0000 7ff741de9740 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ef21e382f0 0x55ef22222770 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.808+0000 7ff71e000640 0 client.27353 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:21.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.808+0000 7ff741de9740 1 --2- 172.21.15.44:0/2722345753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ef21e1fd80 0x55ef22222230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.808+0000 7ff741de9740 1 -- 172.21.15.44:0/2722345753 >> 172.21.15.44:0/2722345753 conn(0x55ef21e55300 msgr2=0x55ef2220db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.808+0000 7ff741de9740 1 -- 172.21.15.44:0/2722345753 shutdown_connections 2024-05-27T09:46:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.808+0000 7ff741de9740 1 -- 172.21.15.44:0/2722345753 wait complete. 2024-05-27T09:46:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.809+0000 7ff741de9740 20 client.27353 trim_cache size 0 max 16384 2024-05-27T09:46:21.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:46:21.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:46:21.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:46:21.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.809+0000 7ff739800640 1 -- 172.21.15.44:0/2453062248 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7180b29a0 msgr2=0x7ff7180b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:21.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.809+0000 7ff739800640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7180b29a0 0x7ff7180b4eb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff72400c6a0 tx=0x7ff7240e2db0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.809+0000 7ff739800640 1 -- 172.21.15.44:0/2453062248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff73410c170 msgr2=0x7ff734078eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.809+0000 7ff739800640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff73410c170 0x7ff734078eb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff72400e970 tx=0x7ff72400ed70 comp rx=0 tx=0).stop 2024-05-27T09:46:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.810+0000 7ff739800640 1 -- 172.21.15.44:0/2453062248 shutdown_connections 2024-05-27T09:46:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.810+0000 7ff739800640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff73410c170 0x7ff734078eb0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.810+0000 7ff739800640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff73410bcd0 0x7ff734078970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.810+0000 7ff739800640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff7180b29a0 0x7ff7180b4eb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.810+0000 7ff739800640 1 --2- 172.21.15.44:0/2453062248 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff734106420 0x7ff734078430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.810+0000 7ff739800640 1 -- 172.21.15.44:0/2453062248 >> 172.21.15.44:0/2453062248 conn(0x7ff7341013d0 msgr2=0x7ff7341017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.810+0000 7ff739800640 1 -- 172.21.15.44:0/2453062248 shutdown_connections 2024-05-27T09:46:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:21.810+0000 7ff739800640 1 -- 172.21.15.44:0/2453062248 wait complete. 2024-05-27T09:46:22.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:21 smithi050 ceph-mon[36371]: pgmap v348: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.5 MiB/s wr, 933 op/s 2024-05-27T09:46:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:21 smithi189 ceph-mon[36671]: pgmap v348: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.5 MiB/s wr, 933 op/s 2024-05-27T09:46:22.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:21 smithi044 ceph-mon[31618]: pgmap v348: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.5 MiB/s wr, 933 op/s 2024-05-27T09:46:24.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:23 smithi050 ceph-mon[36371]: pgmap v349: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 530 op/s 2024-05-27T09:46:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:23 smithi189 ceph-mon[36671]: pgmap v349: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 530 op/s 2024-05-27T09:46:24.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:23 smithi044 ceph-mon[31618]: pgmap v349: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 530 op/s 2024-05-27T09:46:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:25 smithi050 ceph-mon[36371]: pgmap v350: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 530 op/s 2024-05-27T09:46:26.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:25 smithi189 ceph-mon[36671]: pgmap v350: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 530 op/s 2024-05-27T09:46:26.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:25 smithi044 ceph-mon[31618]: pgmap v350: 97 pgs: 97 active+clean; 317 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 530 op/s 2024-05-27T09:46:27.523 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:46:27.999 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:46:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:28 smithi050 ceph-mon[36371]: pgmap v351: 97 pgs: 97 active+clean; 283 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.8 MiB/s wr, 884 op/s 2024-05-27T09:46:28.323 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:27 smithi044 ceph-mon[31618]: pgmap v351: 97 pgs: 97 active+clean; 283 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.8 MiB/s wr, 884 op/s 2024-05-27T09:46:28.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:27 smithi189 ceph-mon[36671]: pgmap v351: 97 pgs: 97 active+clean; 283 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.8 MiB/s wr, 884 op/s 2024-05-27T09:46:30.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:30 smithi044 ceph-mon[31618]: pgmap v352: 97 pgs: 97 active+clean; 283 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.1 MiB/s wr, 698 op/s 2024-05-27T09:46:30.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:30 smithi050 ceph-mon[36371]: pgmap v352: 97 pgs: 97 active+clean; 283 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.1 MiB/s wr, 698 op/s 2024-05-27T09:46:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:30 smithi189 ceph-mon[36671]: pgmap v352: 97 pgs: 97 active+clean; 283 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.1 MiB/s wr, 698 op/s 2024-05-27T09:46:30.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.614+0000 7f3acac00700 1 -- 172.21.15.44:0/3211740634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac41043f0 msgr2=0x7f3ac4104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:30.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.614+0000 7f3acac00700 1 --2- 172.21.15.44:0/3211740634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac41043f0 0x7f3ac4104800 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3ab80099c0 tx=0x7f3ab8009cd0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.614+0000 7f3acac00700 1 -- 172.21.15.44:0/3211740634 shutdown_connections 2024-05-27T09:46:30.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.614+0000 7f3acac00700 1 --2- 172.21.15.44:0/3211740634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ac4109670 0x7f3ac4110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.614+0000 7f3acac00700 1 --2- 172.21.15.44:0/3211740634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac41043f0 0x7f3ac4104800 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.614+0000 7f3acac00700 1 --2- 172.21.15.44:0/3211740634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ac4108d20 0x7f3ac4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.614+0000 7f3acac00700 1 -- 172.21.15.44:0/3211740634 >> 172.21.15.44:0/3211740634 conn(0x7f3ac4079d00 msgr2=0x7f3ac407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:30.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.614+0000 7f3acac00700 1 -- 172.21.15.44:0/3211740634 shutdown_connections 2024-05-27T09:46:30.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.614+0000 7f3acac00700 1 -- 172.21.15.44:0/3211740634 wait complete. 2024-05-27T09:46:30.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.615+0000 7f3acac00700 1 Processor -- start 2024-05-27T09:46:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.616+0000 7f3acac00700 1 -- start start 2024-05-27T09:46:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.616+0000 7f3acac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac41043f0 0x7f3ac41a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.616+0000 7f3acac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ac4108d20 0x7f3ac41a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.616+0000 7f3acac00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ac4109670 0x7f3ac419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.616+0000 7f3acac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac41142f0 con 0x7f3ac4108d20 2024-05-27T09:46:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.616+0000 7f3acac00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac4114170 con 0x7f3ac4109670 2024-05-27T09:46:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.616+0000 7f3acac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac4114470 con 0x7f3ac41043f0 2024-05-27T09:46:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.616+0000 7f3ac9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac41043f0 0x7f3ac41a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.616+0000 7f3ac9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac41043f0 0x7f3ac41a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:56446/0 (socket says 172.21.15.44:56446) 2024-05-27T09:46:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.616+0000 7f3ac9800700 1 -- 172.21.15.44:0/2729909208 learned_addr learned my addr 172.21.15.44:0/2729909208 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:30.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.616+0000 7f3ac8e00700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ac4108d20 0x7f3ac41a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.617+0000 7f3aca200700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ac4109670 0x7f3ac419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.617+0000 7f3ac9800700 1 -- 172.21.15.44:0/2729909208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ac4109670 msgr2=0x7f3ac419ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.617+0000 7f3ac9800700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ac4109670 0x7f3ac419ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.617+0000 7f3ac9800700 1 -- 172.21.15.44:0/2729909208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ac4108d20 msgr2=0x7f3ac41a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.617+0000 7f3ac9800700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ac4108d20 0x7f3ac41a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.617+0000 7f3ac9800700 1 -- 172.21.15.44:0/2729909208 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac419b1c0 con 0x7f3ac41043f0 2024-05-27T09:46:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.617+0000 7f3ac8e00700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ac4108d20 0x7f3ac41a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:46:30.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.617+0000 7f3ac9800700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac41043f0 0x7f3ac41a08a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8006010 tx=0x7f3ab8005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.617+0000 7f3ac2000700 1 -- 172.21.15.44:0/2729909208 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8026070 con 0x7f3ac41043f0 2024-05-27T09:46:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.618+0000 7f3ac2000700 1 -- 172.21.15.44:0/2729909208 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ab8003c70 con 0x7f3ac41043f0 2024-05-27T09:46:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.618+0000 7f3acac00700 1 -- 172.21.15.44:0/2729909208 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac419b450 con 0x7f3ac41043f0 2024-05-27T09:46:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.618+0000 7f3acac00700 1 -- 172.21.15.44:0/2729909208 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac41a7730 con 0x7f3ac41043f0 2024-05-27T09:46:30.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.618+0000 7f3ac2000700 1 -- 172.21.15.44:0/2729909208 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab800fb40 con 0x7f3ac41043f0 2024-05-27T09:46:30.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.621+0000 7f3ac2000700 1 -- 172.21.15.44:0/2729909208 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3ab8029070 con 0x7f3ac41043f0 2024-05-27T09:46:30.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.623+0000 7f3adace3b80 1 Processor -- start 2024-05-27T09:46:30.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.623+0000 7f3adace3b80 1 -- start start 2024-05-27T09:46:30.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.623+0000 7f3ac2000700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3aac0b2a70 0x7f3aac0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.623+0000 7f3ac2000700 1 -- 172.21.15.44:0/2729909208 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7f3ab80e2ff0 con 0x7f3ac41043f0 2024-05-27T09:46:30.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.623+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10d939f0 0x563b10e27750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.623+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10ca7e40 0x563b10e27c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.623+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10e312c0 0x563b10e336e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.623+0000 7f3adace3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563b10d54e40 con 0x563b10ca7e40 2024-05-27T09:46:30.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.623+0000 7f3adace3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563b10d449e0 con 0x563b10e312c0 2024-05-27T09:46:30.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.623+0000 7f3adace3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563b10d48a20 con 0x563b10d939f0 2024-05-27T09:46:30.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.623+0000 7f3aca200700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3aac0b2a70 0x7f3aac0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.624+0000 7f3ac9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10d939f0 0x563b10e27750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.624+0000 7f3ac9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10d939f0 0x563b10e27750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:56462/0 (socket says 172.21.15.44:56462) 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.624+0000 7f3ac9800700 1 -- 172.21.15.44:0/3538376351 learned_addr learned my addr 172.21.15.44:0/3538376351 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.624+0000 7f3ac8e00700 1 --2- 172.21.15.44:0/3538376351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10ca7e40 0x563b10e27c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.624+0000 7f3aca200700 1 --2- 172.21.15.44:0/3538376351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10e312c0 0x563b10e336e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.624+0000 7f3ac9800700 1 -- 172.21.15.44:0/3538376351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10e312c0 msgr2=0x563b10e336e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.624+0000 7f3ac9800700 1 --2- 172.21.15.44:0/3538376351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10e312c0 0x563b10e336e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.624+0000 7f3ac9800700 1 -- 172.21.15.44:0/3538376351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10ca7e40 msgr2=0x563b10e27c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.624+0000 7f3ac9800700 1 --2- 172.21.15.44:0/3538376351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10ca7e40 0x563b10e27c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.624+0000 7f3ac9800700 1 -- 172.21.15.44:0/3538376351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b10d0d420 con 0x563b10d939f0 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.624+0000 7f3ac9800700 1 --2- 172.21.15.44:0/3538376351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10d939f0 0x563b10e27750 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3ab80aa290 tx=0x7f3ab80a9e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60616aa0b5ba9f00 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.625+0000 7f3ab6a00700 1 -- 172.21.15.44:0/3538376351 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab80aa300 con 0x563b10d939f0 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.625+0000 7f3aca200700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3aac0b2a70 0x7f3aac0b4f80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x563b10e29210 tx=0x7f3abc009920 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.625+0000 7f3ab6a00700 1 -- 172.21.15.44:0/3538376351 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ab80ae6b0 con 0x563b10d939f0 2024-05-27T09:46:30.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.625+0000 7f3ab6a00700 1 -- 172.21.15.44:0/3538376351 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab80a9560 con 0x563b10d939f0 2024-05-27T09:46:30.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.626+0000 7f3adace3b80 1 -- 172.21.15.44:0/3538376351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10d939f0 msgr2=0x563b10e27750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:30.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.626+0000 7f3adace3b80 1 --2- 172.21.15.44:0/3538376351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10d939f0 0x563b10e27750 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3ab80aa290 tx=0x7f3ab80a9e10 comp rx=0 tx=0).stop 2024-05-27T09:46:30.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.626+0000 7f3adace3b80 1 -- 172.21.15.44:0/3538376351 shutdown_connections 2024-05-27T09:46:30.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.626+0000 7f3adace3b80 1 --2- 172.21.15.44:0/3538376351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10e312c0 0x563b10e336e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.626+0000 7f3adace3b80 1 --2- 172.21.15.44:0/3538376351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10d939f0 0x563b10e27750 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.626+0000 7f3adace3b80 1 --2- 172.21.15.44:0/3538376351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10ca7e40 0x563b10e27c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.626+0000 7f3adace3b80 1 -- 172.21.15.44:0/3538376351 >> 172.21.15.44:0/3538376351 conn(0x563b10d41390 msgr2=0x563b10e22710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:30.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.626+0000 7f3adace3b80 1 -- 172.21.15.44:0/3538376351 shutdown_connections 2024-05-27T09:46:30.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.626+0000 7f3adace3b80 1 -- 172.21.15.44:0/3538376351 wait complete. 2024-05-27T09:46:30.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.627+0000 7f3adace3b80 1 -- start start 2024-05-27T09:46:30.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.627+0000 7f3adace3b80 1 Processor -- start 2024-05-27T09:46:30.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.628+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10ca7e40 0x563b10e2e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.628+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10d939f0 0x563b10e2e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.628+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10e312c0 0x563b10e2ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.628+0000 7f3adace3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563b10c4afa0 con 0x563b10e312c0 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.628+0000 7f3adace3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563b10ce5760 con 0x563b10d939f0 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.628+0000 7f3adace3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563b10e33c20 con 0x563b10ca7e40 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.628+0000 7f3adace3b80 10 client.0 _ll_register_callbacks cb 0x563b10ceec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.628+0000 7f3aca200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10e312c0 0x563b10e2ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.628+0000 7f3ac9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10ca7e40 0x563b10e2e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.629+0000 7f3ac9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10ca7e40 0x563b10e2e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:56468/0 (socket says 172.21.15.44:56468) 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.629+0000 7f3ac9800700 1 -- 172.21.15.44:0/1550954988 learned_addr learned my addr 172.21.15.44:0/1550954988 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.629+0000 7f3ac8e00700 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10d939f0 0x563b10e2e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.629+0000 7f3ac9800700 1 -- 172.21.15.44:0/1550954988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10d939f0 msgr2=0x563b10e2e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:30.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.629+0000 7f3ac9800700 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10d939f0 0x563b10e2e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.629+0000 7f3ac9800700 1 -- 172.21.15.44:0/1550954988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10e312c0 msgr2=0x563b10e2ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:30.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.629+0000 7f3ac9800700 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10e312c0 0x563b10e2ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.629+0000 7f3ac9800700 1 -- 172.21.15.44:0/1550954988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b10c948d0 con 0x563b10ca7e40 2024-05-27T09:46:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.629+0000 7f3ac9800700 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10ca7e40 0x563b10e2e270 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3ab80a9fb0 tx=0x7f3ab80aa790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34a4722d5df702f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.629+0000 7f3aab400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:46:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.630+0000 7f3aab400700 1 -- 172.21.15.44:0/1550954988 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab80cb300 con 0x563b10ca7e40 2024-05-27T09:46:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.630+0000 7f3aab400700 1 -- 172.21.15.44:0/1550954988 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7f3ab80039d0 con 0x563b10ca7e40 2024-05-27T09:46:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.630+0000 7f3adace3b80 1 -- 172.21.15.44:0/1550954988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ab80ae6b0 con 0x563b10ca7e40 2024-05-27T09:46:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.630+0000 7f3aab400700 1 -- 172.21.15.44:0/1550954988 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3ab80b30d0 con 0x563b10ca7e40 2024-05-27T09:46:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.630+0000 7f3adace3b80 10 client.27358 fetch_fsmap learned FSMap version 34 2024-05-27T09:46:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.630+0000 7f3adace3b80 1 -- 172.21.15.44:0/1550954988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563b10d15060 con 0x563b10ca7e40 2024-05-27T09:46:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.630+0000 7f3aab400700 1 -- 172.21.15.44:0/1550954988 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ab80ae6b0 con 0x563b10ca7e40 2024-05-27T09:46:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.630+0000 7f3aab400700 1 -- 172.21.15.44:0/1550954988 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f3ab800fce0 con 0x563b10ca7e40 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.631+0000 7f3adace3b80 10 client.27358 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.631+0000 7f3adace3b80 10 client.27358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.631+0000 7f3adace3b80 20 client.27358 populate_metadata read hostname 'smithi044' 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.631+0000 7f3adace3b80 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b10e40900 0x563b10e42d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.631+0000 7f3adace3b80 4 client.27358 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.631+0000 7f3adace3b80 1 -- 172.21.15.44:0/1550954988 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b10d56d90 con 0x563b10e40900 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.631+0000 7f3ac8e00700 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b10e40900 0x563b10e42d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.632+0000 7f3ac8e00700 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b10e40900 0x563b10e42d20 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d30ce13498f99ac server_cookie=5d2d0b86fbecd032 in_seq=0 out_seq=0 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.632+0000 7f3aab400700 10 client.27358 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3aab400700 1 -- 172.21.15.44:0/1550954988 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563b10d56d90 con 0x563b10e40900 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3aab400700 10 client.27358 handle_command_reply: tid=0 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3adace3b80 1 client.27358 shutdown 2024-05-27T09:46:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3adace3b80 1 -- 172.21.15.44:0/1550954988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10ca7e40 msgr2=0x563b10e2e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3adace3b80 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10ca7e40 0x563b10e2e270 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3ab80a9fb0 tx=0x7f3ab80aa790 comp rx=0 tx=0).stop 2024-05-27T09:46:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3adace3b80 1 -- 172.21.15.44:0/1550954988 shutdown_connections 2024-05-27T09:46:30.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3adace3b80 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b10e40900 0x563b10e42d20 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3adace3b80 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10e312c0 0x563b10e2ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3adace3b80 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10ca7e40 0x563b10e2e270 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3adace3b80 1 --2- 172.21.15.44:0/1550954988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10d939f0 0x563b10e2e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3adace3b80 1 -- 172.21.15.44:0/1550954988 >> 172.21.15.44:0/1550954988 conn(0x563b10d41390 msgr2=0x563b10e22300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.633+0000 7f3aab400700 0 client.27358 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.634+0000 7f3adace3b80 1 -- 172.21.15.44:0/1550954988 shutdown_connections 2024-05-27T09:46:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.634+0000 7f3adace3b80 1 -- 172.21.15.44:0/1550954988 wait complete. 2024-05-27T09:46:30.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.634+0000 7f3adace3b80 20 client.27358 trim_cache size 0 max 16384 2024-05-27T09:46:30.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3adace3b80 1 Processor -- start 2024-05-27T09:46:30.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3adace3b80 1 -- start start 2024-05-27T09:46:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e22740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10d939f0 0x563b10e3ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10e312c0 0x563b10e26b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3adace3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa400feb0 con 0x563b10d939f0 2024-05-27T09:46:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3adace3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa400d6e0 con 0x563b10ca7e40 2024-05-27T09:46:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3adace3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa400b550 con 0x563b10e312c0 2024-05-27T09:46:30.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3ac9800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e22740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3aca200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10e312c0 0x563b10e26b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3ac9800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e22740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:40510/0 (socket says 172.21.15.44:40510) 2024-05-27T09:46:30.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.657+0000 7f3ac9800700 1 -- 172.21.15.44:0/1850445159 learned_addr learned my addr 172.21.15.44:0/1850445159 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:30.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.658+0000 7f3ac8e00700 1 --2- 172.21.15.44:0/1850445159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10d939f0 0x563b10e3ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.658+0000 7f3ac9800700 1 -- 172.21.15.44:0/1850445159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10e312c0 msgr2=0x563b10e26b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:30.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.658+0000 7f3ac9800700 1 --2- 172.21.15.44:0/1850445159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10e312c0 0x563b10e26b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.658+0000 7f3ac9800700 1 -- 172.21.15.44:0/1850445159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10d939f0 msgr2=0x563b10e3ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.658+0000 7f3ac9800700 1 --2- 172.21.15.44:0/1850445159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10d939f0 0x563b10e3ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.658+0000 7f3ac9800700 1 -- 172.21.15.44:0/1850445159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b10e239a0 con 0x563b10ca7e40 2024-05-27T09:46:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.658+0000 7f3ac9800700 1 --2- 172.21.15.44:0/1850445159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e22740 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3ab80a6380 tx=0x7f3ab8005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b85297f88ba6e197 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.658+0000 7f3ab6000700 1 -- 172.21.15.44:0/1850445159 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab80ae6b0 con 0x563b10ca7e40 2024-05-27T09:46:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.658+0000 7f3ab6000700 1 -- 172.21.15.44:0/1850445159 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ab80c9070 con 0x563b10ca7e40 2024-05-27T09:46:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.658+0000 7f3ab6000700 1 -- 172.21.15.44:0/1850445159 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab80ae170 con 0x563b10ca7e40 2024-05-27T09:46:30.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.659+0000 7f3adace3b80 1 -- 172.21.15.44:0/1850445159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 msgr2=0x563b10e22740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.659+0000 7f3adace3b80 1 --2- 172.21.15.44:0/1850445159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e22740 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3ab80a6380 tx=0x7f3ab8005840 comp rx=0 tx=0).stop 2024-05-27T09:46:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.659+0000 7f3adace3b80 1 -- 172.21.15.44:0/1850445159 shutdown_connections 2024-05-27T09:46:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.659+0000 7f3adace3b80 1 --2- 172.21.15.44:0/1850445159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10e312c0 0x563b10e26b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.659+0000 7f3adace3b80 1 --2- 172.21.15.44:0/1850445159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e22740 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.659+0000 7f3adace3b80 1 --2- 172.21.15.44:0/1850445159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10d939f0 0x563b10e3ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.659+0000 7f3adace3b80 1 -- 172.21.15.44:0/1850445159 >> 172.21.15.44:0/1850445159 conn(0x563b10d41390 msgr2=0x563b10e2ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.660+0000 7f3adace3b80 1 -- 172.21.15.44:0/1850445159 shutdown_connections 2024-05-27T09:46:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.660+0000 7f3adace3b80 1 -- 172.21.15.44:0/1850445159 wait complete. 2024-05-27T09:46:30.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.660+0000 7f3adace3b80 1 -- start start 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.660+0000 7f3adace3b80 1 Processor -- start 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.661+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e461d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.661+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10d939f0 0x563b10e43550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.661+0000 7f3adace3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10e312c0 0x563b10e43a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.661+0000 7f3adace3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa400bba0 con 0x563b10d939f0 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.661+0000 7f3adace3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa400d010 con 0x563b10ca7e40 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.661+0000 7f3adace3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa400c270 con 0x563b10e312c0 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.661+0000 7f3adace3b80 10 client.0 _ll_register_callbacks cb 0x563b10ceec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.661+0000 7f3ac9800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.661+0000 7f3ac9800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e461d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:40524/0 (socket says 172.21.15.44:40524) 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.661+0000 7f3ac9800700 1 -- 172.21.15.44:0/2178367194 learned_addr learned my addr 172.21.15.44:0/2178367194 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3ac8e00700 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10d939f0 0x563b10e43550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3aca200700 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10e312c0 0x563b10e43a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3ac9800700 1 -- 172.21.15.44:0/2178367194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10e312c0 msgr2=0x563b10e43a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3ac9800700 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10e312c0 0x563b10e43a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3ac9800700 1 -- 172.21.15.44:0/2178367194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10d939f0 msgr2=0x563b10e43550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3ac9800700 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10d939f0 0x563b10e43550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3ac9800700 1 -- 172.21.15.44:0/2178367194 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b10e43fd0 con 0x563b10ca7e40 2024-05-27T09:46:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3aca200700 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10e312c0 0x563b10e43a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:46:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3ac8e00700 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10d939f0 0x563b10e43550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:46:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3ac9800700 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e461d0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8005910 tx=0x7f3ab8003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec3ab267d2468e83 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3aab400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:46:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3aab400700 1 -- 172.21.15.44:0/2178367194 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab80a5fd0 con 0x563b10ca7e40 2024-05-27T09:46:30.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.662+0000 7f3aab400700 1 -- 172.21.15.44:0/2178367194 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7f3ab80a9770 con 0x563b10ca7e40 2024-05-27T09:46:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3adace3b80 1 -- 172.21.15.44:0/2178367194 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ab80c9070 con 0x563b10ca7e40 2024-05-27T09:46:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3aab400700 1 -- 172.21.15.44:0/2178367194 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3ab80aa300 con 0x563b10ca7e40 2024-05-27T09:46:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3adace3b80 10 client.27371 fetch_fsmap learned FSMap version 34 2024-05-27T09:46:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3adace3b80 1 -- 172.21.15.44:0/2178367194 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563b10e43fd0 con 0x563b10ca7e40 2024-05-27T09:46:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3aab400700 1 -- 172.21.15.44:0/2178367194 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ab80c9070 con 0x563b10ca7e40 2024-05-27T09:46:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3aab400700 1 -- 172.21.15.44:0/2178367194 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f3ab800fce0 con 0x563b10ca7e40 2024-05-27T09:46:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3adace3b80 10 client.27371 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:46:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3adace3b80 10 client.27371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:46:30.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3adace3b80 20 client.27371 populate_metadata read hostname 'smithi044' 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3adace3b80 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b10eab040 0x563b10ead460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3adace3b80 4 client.27371 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.663+0000 7f3adace3b80 1 -- 172.21.15.44:0/2178367194 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563b10d50570 con 0x563b10eab040 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.664+0000 7f3ac8e00700 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b10eab040 0x563b10ead460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.664+0000 7f3aab400700 10 client.27371 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.664+0000 7f3ac8e00700 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b10eab040 0x563b10ead460 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cdb2375ba7f6a6d server_cookie=e4454ff9f3f9c88d in_seq=0 out_seq=0 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.664+0000 7f3aab400700 1 -- 172.21.15.44:0/2178367194 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563b10d50570 con 0x563b10eab040 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.664+0000 7f3aab400700 10 client.27371 handle_command_reply: tid=0 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.664+0000 7f3adace3b80 1 client.27371 shutdown 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3adace3b80 1 -- 172.21.15.44:0/2178367194 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 msgr2=0x563b10e461d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3adace3b80 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e461d0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8005910 tx=0x7f3ab8003c00 comp rx=0 tx=0).stop 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3adace3b80 1 -- 172.21.15.44:0/2178367194 shutdown_connections 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3adace3b80 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b10eab040 0x563b10ead460 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3adace3b80 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b10e312c0 0x563b10e43a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3adace3b80 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b10ca7e40 0x563b10e461d0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3adace3b80 1 --2- 172.21.15.44:0/2178367194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b10d939f0 0x563b10e43550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3aab400700 0 client.27371 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:30.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3adace3b80 1 -- 172.21.15.44:0/2178367194 >> 172.21.15.44:0/2178367194 conn(0x563b10d41390 msgr2=0x563b10e268b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:30.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3adace3b80 1 -- 172.21.15.44:0/2178367194 shutdown_connections 2024-05-27T09:46:30.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3adace3b80 1 -- 172.21.15.44:0/2178367194 wait complete. 2024-05-27T09:46:30.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.665+0000 7f3adace3b80 20 client.27371 trim_cache size 0 max 16384 2024-05-27T09:46:30.668 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:46:30.668 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:46:30.668 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:46:30.668 INFO:teuthology.orchestra.run.smithi044.stdout: "5348fc89-9539-4fcf-9fdc-6e30fd88b5c6": { 2024-05-27T09:46:30.669 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:46:30.669 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "5348fc89-9539-4fcf-9fdc-6e30fd88b5c6", 2024-05-27T09:46:30.669 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:46:30.669 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:46:30.669 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:46:30.669 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:46:30.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.666+0000 7f3acac00700 1 -- 172.21.15.44:0/2729909208 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3aac0b2a70 msgr2=0x7f3aac0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:30.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.666+0000 7f3acac00700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3aac0b2a70 0x7f3aac0b4f80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x563b10e29210 tx=0x7f3abc009920 comp rx=0 tx=0).stop 2024-05-27T09:46:30.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.666+0000 7f3acac00700 1 -- 172.21.15.44:0/2729909208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac41043f0 msgr2=0x7f3ac41a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:30.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.666+0000 7f3acac00700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac41043f0 0x7f3ac41a08a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8006010 tx=0x7f3ab8005280 comp rx=0 tx=0).stop 2024-05-27T09:46:30.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.666+0000 7f3acac00700 1 -- 172.21.15.44:0/2729909208 shutdown_connections 2024-05-27T09:46:30.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.666+0000 7f3acac00700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3aac0b2a70 0x7f3aac0b4f80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.666+0000 7f3acac00700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ac4109670 0x7f3ac419ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.666+0000 7f3acac00700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac41043f0 0x7f3ac41a08a0 secure :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8006010 tx=0x7f3ab8005280 comp rx=0 tx=0).stop 2024-05-27T09:46:30.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.666+0000 7f3acac00700 1 --2- 172.21.15.44:0/2729909208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ac4108d20 0x7f3ac41a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:30.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.666+0000 7f3acac00700 1 -- 172.21.15.44:0/2729909208 >> 172.21.15.44:0/2729909208 conn(0x7f3ac4079d00 msgr2=0x7f3ac4102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:30.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.667+0000 7f3acac00700 1 -- 172.21.15.44:0/2729909208 shutdown_connections 2024-05-27T09:46:30.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:46:30.667+0000 7f3acac00700 1 -- 172.21.15.44:0/2729909208 wait complete. 2024-05-27T09:46:31.417 INFO:tasks.cephfs.filesystem:scrub status for tag:5348fc89-9539-4fcf-9fdc-6e30fd88b5c6 - {'path': '/', 'tag': '5348fc89-9539-4fcf-9fdc-6e30fd88b5c6', 'options': 'recursive,force'} 2024-05-27T09:46:32.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:32 smithi050 ceph-mon[36371]: pgmap v353: 97 pgs: 97 active+clean; 299 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.7 MiB/s wr, 955 op/s 2024-05-27T09:46:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:32 smithi189 ceph-mon[36671]: pgmap v353: 97 pgs: 97 active+clean; 299 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.7 MiB/s wr, 955 op/s 2024-05-27T09:46:32.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:32 smithi044 ceph-mon[31618]: pgmap v353: 97 pgs: 97 active+clean; 299 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.7 MiB/s wr, 955 op/s 2024-05-27T09:46:34.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:34 smithi050 ceph-mon[36371]: pgmap v354: 97 pgs: 97 active+clean; 299 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 610 op/s 2024-05-27T09:46:34.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:34 smithi189 ceph-mon[36671]: pgmap v354: 97 pgs: 97 active+clean; 299 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 610 op/s 2024-05-27T09:46:34.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:34 smithi044 ceph-mon[31618]: pgmap v354: 97 pgs: 97 active+clean; 299 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 610 op/s 2024-05-27T09:46:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.027+0000 7fb0b5e00640 1 -- 172.21.15.44:0/3193272098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b01043b0 msgr2=0x7fb0b0110780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.027+0000 7fb0b5e00640 1 --2- 172.21.15.44:0/3193272098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b01043b0 0x7fb0b0110780 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009900 tx=0x7fb0a802a3f0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.027+0000 7fb0b5e00640 1 -- 172.21.15.44:0/3193272098 shutdown_connections 2024-05-27T09:46:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.027+0000 7fb0b5e00640 1 --2- 172.21.15.44:0/3193272098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b01043b0 0x7fb0b0110780 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.027+0000 7fb0b5e00640 1 --2- 172.21.15.44:0/3193272098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0b010a100 0x7fb0b0103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.027+0000 7fb0b5e00640 1 --2- 172.21.15.44:0/3193272098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b0109c60 0x7fb0b0103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.027+0000 7fb0b5e00640 1 -- 172.21.15.44:0/3193272098 >> 172.21.15.44:0/3193272098 conn(0x7fb0b00ff320 msgr2=0x7fb0b0101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.027+0000 7fb0b5e00640 1 -- 172.21.15.44:0/3193272098 shutdown_connections 2024-05-27T09:46:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.027+0000 7fb0b5e00640 1 -- 172.21.15.44:0/3193272098 wait complete. 2024-05-27T09:46:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.028+0000 7fb0b5e00640 1 Processor -- start 2024-05-27T09:46:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.028+0000 7fb0b5e00640 1 -- start start 2024-05-27T09:46:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0b5e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0b01043b0 0x7fb0b01a2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0b5e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b0109c60 0x7fb0b01a2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0b5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b010a100 0x7fb0b019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0b5e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b0113ea0 con 0x7fb0b01043b0 2024-05-27T09:46:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0b5e00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b0113d20 con 0x7fb0b0109c60 2024-05-27T09:46:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0b5e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b0114020 con 0x7fb0b010a100 2024-05-27T09:46:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0aea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0b01043b0 0x7fb0b01a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0af400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b010a100 0x7fb0b019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0a6000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b0109c60 0x7fb0b01a2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0af400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b010a100 0x7fb0b019c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:36856/0 (socket says 172.21.15.44:36856) 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0af400640 1 -- 172.21.15.44:0/4149083989 learned_addr learned my addr 172.21.15.44:0/4149083989 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.029+0000 7fb0a6000640 1 -- 172.21.15.44:0/4149083989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b010a100 msgr2=0x7fb0b019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.030+0000 7fb0a6000640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b010a100 0x7fb0b019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.030+0000 7fb0a6000640 1 -- 172.21.15.44:0/4149083989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0b01043b0 msgr2=0x7fb0b01a2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.030+0000 7fb0a6000640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0b01043b0 0x7fb0b01a2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.030+0000 7fb0a6000640 1 -- 172.21.15.44:0/4149083989 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b019cf40 con 0x7fb0b0109c60 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.030+0000 7fb0aea00640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0b01043b0 0x7fb0b01a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.030+0000 7fb0a6000640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b0109c60 0x7fb0b01a2b70 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb0b006b420 tx=0x7fb09800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.030+0000 7fb0b4a00640 1 -- 172.21.15.44:0/4149083989 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09800cd00 con 0x7fb0b0109c60 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.030+0000 7fb0b5e00640 1 -- 172.21.15.44:0/4149083989 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b019d230 con 0x7fb0b0109c60 2024-05-27T09:46:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.030+0000 7fb0b4a00640 1 -- 172.21.15.44:0/4149083989 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0980108e0 con 0x7fb0b0109c60 2024-05-27T09:46:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.030+0000 7fb0b4a00640 1 -- 172.21.15.44:0/4149083989 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0980188c0 con 0x7fb0b0109c60 2024-05-27T09:46:35.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.030+0000 7fb0b5e00640 1 -- 172.21.15.44:0/4149083989 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b01a9480 con 0x7fb0b0109c60 2024-05-27T09:46:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.033+0000 7fb0bcf63740 1 Processor -- start 2024-05-27T09:46:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0bcf63740 1 -- start start 2024-05-27T09:46:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0b4a00640 1 -- 172.21.15.44:0/4149083989 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb09801f020 con 0x7fb0b0109c60 2024-05-27T09:46:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 0x55cb73349860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb73349da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb7334a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cb732bab90 con 0x55cb72f864d0 2024-05-27T09:46:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cb732aa730 con 0x55cb72f7e2f0 2024-05-27T09:46:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cb732ae770 con 0x55cb72fa71b0 2024-05-27T09:46:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0af400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 0x55cb73349860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0aea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb73349da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0af400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 0x55cb73349860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:36870/0 (socket says 172.21.15.44:36870) 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.034+0000 7fb0af400640 1 -- 172.21.15.44:0/3223561440 learned_addr learned my addr 172.21.15.44:0/3223561440 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.035+0000 7fb0a6000640 1 --2- 172.21.15.44:0/3223561440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb7334a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.035+0000 7fb0af400640 1 -- 172.21.15.44:0/3223561440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 msgr2=0x55cb73349da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.035+0000 7fb0af400640 1 --2- 172.21.15.44:0/3223561440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb73349da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.035+0000 7fb0af400640 1 -- 172.21.15.44:0/3223561440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 msgr2=0x55cb7334a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.035+0000 7fb0af400640 1 --2- 172.21.15.44:0/3223561440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb7334a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.035+0000 7fb0af400640 1 -- 172.21.15.44:0/3223561440 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb732438b0 con 0x55cb72fa71b0 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.035+0000 7fb0a6000640 1 --2- 172.21.15.44:0/3223561440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb7334a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.035+0000 7fb0af400640 1 --2- 172.21.15.44:0/3223561440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 0x55cb73349860 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8004080 tx=0x7fb0a80040b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=727f1bbca3f137ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.035+0000 7fb09fe00640 1 -- 172.21.15.44:0/3223561440 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8040070 con 0x55cb72fa71b0 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.036+0000 7fb09fe00640 1 -- 172.21.15.44:0/3223561440 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a80054d0 con 0x55cb72fa71b0 2024-05-27T09:46:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.036+0000 7fb09fe00640 1 -- 172.21.15.44:0/3223561440 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8035040 con 0x55cb72fa71b0 2024-05-27T09:46:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.036+0000 7fb0b4a00640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0940b2950 0x7fb0940b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.036+0000 7fb0b4a00640 1 -- 172.21.15.44:0/4149083989 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fb0980e2660 con 0x7fb0b0109c60 2024-05-27T09:46:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.036+0000 7fb0aea00640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0940b2950 0x7fb0940b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.037+0000 7fb0bcf63740 1 -- 172.21.15.44:0/3223561440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 msgr2=0x55cb73349860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.037+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/3223561440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 0x55cb73349860 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8004080 tx=0x7fb0a80040b0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.037+0000 7fb0aea00640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0940b2950 0x7fb0940b4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55cb7334ac10 tx=0x7fb0a0009500 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.037+0000 7fb0bcf63740 1 -- 172.21.15.44:0/3223561440 shutdown_connections 2024-05-27T09:46:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.037+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/3223561440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb7334a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.037+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/3223561440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb73349da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.037+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/3223561440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 0x55cb73349860 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.037+0000 7fb0bcf63740 1 -- 172.21.15.44:0/3223561440 >> 172.21.15.44:0/3223561440 conn(0x55cb72f9b300 msgr2=0x55cb73346010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.037+0000 7fb0bcf63740 1 -- 172.21.15.44:0/3223561440 shutdown_connections 2024-05-27T09:46:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.037+0000 7fb0bcf63740 1 -- 172.21.15.44:0/3223561440 wait complete. 2024-05-27T09:46:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.038+0000 7fb0bcf63740 1 -- start start 2024-05-27T09:46:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.039+0000 7fb0bcf63740 1 Processor -- start 2024-05-27T09:46:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.039+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f7e2f0 0x55cb73353a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.039+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb73353fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.039+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72fa71b0 0x55cb7335fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cb72f90790 con 0x55cb72f864d0 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cb73252bc0 con 0x55cb72fa71b0 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cb73356930 con 0x55cb72f7e2f0 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0bcf63740 10 client.0 _ll_register_callbacks cb 0x55cb73230400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0aea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f7e2f0 0x55cb73353a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0af400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72fa71b0 0x55cb7335fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0aea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f7e2f0 0x55cb73353a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:36884/0 (socket says 172.21.15.44:36884) 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0aea00640 1 -- 172.21.15.44:0/4146000764 learned_addr learned my addr 172.21.15.44:0/4146000764 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0a6000640 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb73353fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0af400640 1 -- 172.21.15.44:0/4146000764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f7e2f0 msgr2=0x55cb73353a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0af400640 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f7e2f0 0x55cb73353a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0af400640 1 -- 172.21.15.44:0/4146000764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 msgr2=0x55cb73353fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0af400640 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb73353fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.040+0000 7fb0af400640 1 -- 172.21.15.44:0/4146000764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb72fbf210 con 0x55cb72fa71b0 2024-05-27T09:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.041+0000 7fb0af400640 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72fa71b0 0x55cb7335fc60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8005ec0 tx=0x7fb0a8004bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=147a42f1fb99869c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.041+0000 7fb09cc00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.041+0000 7fb09cc00640 1 -- 172.21.15.44:0/4146000764 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8035070 con 0x55cb72fa71b0 2024-05-27T09:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.041+0000 7fb0bcf63740 1 -- 172.21.15.44:0/4146000764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0a80054d0 con 0x55cb72fa71b0 2024-05-27T09:46:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.041+0000 7fb09cc00640 1 -- 172.21.15.44:0/4146000764 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fb0a8005000 con 0x55cb72fa71b0 2024-05-27T09:46:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.041+0000 7fb09cc00640 1 -- 172.21.15.44:0/4146000764 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb0a8040040 con 0x55cb72fa71b0 2024-05-27T09:46:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.042+0000 7fb0bcf63740 10 client.27383 fetch_fsmap learned FSMap version 34 2024-05-27T09:46:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.042+0000 7fb0bcf63740 1 -- 172.21.15.44:0/4146000764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cb7308b620 con 0x55cb72fa71b0 2024-05-27T09:46:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.042+0000 7fb09cc00640 1 -- 172.21.15.44:0/4146000764 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a80054d0 con 0x55cb72fa71b0 2024-05-27T09:46:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.042+0000 7fb09cc00640 1 -- 172.21.15.44:0/4146000764 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fb0a80314a0 con 0x55cb72fa71b0 2024-05-27T09:46:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.042+0000 7fb0bcf63740 10 client.27383 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:46:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.042+0000 7fb0bcf63740 10 client.27383 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:46:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.042+0000 7fb0bcf63740 20 client.27383 populate_metadata read hostname 'smithi044' 2024-05-27T09:46:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.042+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cb72f65d80 0x55cb73365aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.042+0000 7fb0bcf63740 4 client.27383 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.042+0000 7fb0bcf63740 1 -- 172.21.15.44:0/4146000764 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb732bcae0 con 0x55cb72f65d80 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.043+0000 7fb0aea00640 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cb72f65d80 0x55cb73365aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.043+0000 7fb09cc00640 10 client.27383 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.043+0000 7fb0aea00640 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cb72f65d80 0x55cb73365aa0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77a2171986ae7cd4 server_cookie=7ca084e92a6bb014 in_seq=0 out_seq=0 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb09cc00640 1 -- 172.21.15.44:0/4146000764 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cb732bcae0 con 0x55cb72f65d80 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb09cc00640 10 client.27383 handle_command_reply: tid=0 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb0bcf63740 1 client.27383 shutdown 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb0bcf63740 1 -- 172.21.15.44:0/4146000764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72fa71b0 msgr2=0x55cb7335fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72fa71b0 0x55cb7335fc60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8005ec0 tx=0x7fb0a8004bc0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb0bcf63740 1 -- 172.21.15.44:0/4146000764 shutdown_connections 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cb72f65d80 0x55cb73365aa0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72fa71b0 0x55cb7335fc60 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb73353fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/4146000764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f7e2f0 0x55cb73353a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb0bcf63740 1 -- 172.21.15.44:0/4146000764 >> 172.21.15.44:0/4146000764 conn(0x55cb72f9b300 msgr2=0x55cb73349690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.044+0000 7fb09cc00640 0 client.27383 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.045+0000 7fb0bcf63740 1 -- 172.21.15.44:0/4146000764 shutdown_connections 2024-05-27T09:46:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.045+0000 7fb0bcf63740 1 -- 172.21.15.44:0/4146000764 wait complete. 2024-05-27T09:46:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.045+0000 7fb0bcf63740 20 client.27383 trim_cache size 0 max 16384 2024-05-27T09:46:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.062+0000 7fb0bcf63740 1 Processor -- start 2024-05-27T09:46:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.062+0000 7fb0bcf63740 1 -- start start 2024-05-27T09:46:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb7334f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb73353fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 0x55cb7335fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01b4600 con 0x55cb72f864d0 2024-05-27T09:46:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01b1e70 con 0x55cb72f7e2f0 2024-05-27T09:46:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01afce0 con 0x55cb72fa71b0 2024-05-27T09:46:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0aea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb7334f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0aea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb7334f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:59994/0 (socket says 172.21.15.44:59994) 2024-05-27T09:46:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0aea00640 1 -- 172.21.15.44:0/3188118503 learned_addr learned my addr 172.21.15.44:0/3188118503 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0a6000640 1 --2- 172.21.15.44:0/3188118503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb73353fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0af400640 1 --2- 172.21.15.44:0/3188118503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 0x55cb7335fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0aea00640 1 -- 172.21.15.44:0/3188118503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 msgr2=0x55cb7335fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0aea00640 1 --2- 172.21.15.44:0/3188118503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 0x55cb7335fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0aea00640 1 -- 172.21.15.44:0/3188118503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 msgr2=0x55cb73353fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.063+0000 7fb0aea00640 1 --2- 172.21.15.44:0/3188118503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb73353fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.064+0000 7fb0aea00640 1 -- 172.21.15.44:0/3188118503 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb7334dfb0 con 0x55cb72f7e2f0 2024-05-27T09:46:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.064+0000 7fb0aea00640 1 --2- 172.21.15.44:0/3188118503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb7334f240 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0104940 tx=0x7fb0a001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1defed9fedf3f09c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.064+0000 7fb09f400640 1 -- 172.21.15.44:0/3188118503 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a002b4e0 con 0x55cb72f7e2f0 2024-05-27T09:46:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.064+0000 7fb09f400640 1 -- 172.21.15.44:0/3188118503 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a0030070 con 0x55cb72f7e2f0 2024-05-27T09:46:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.064+0000 7fb09f400640 1 -- 172.21.15.44:0/3188118503 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0011cd0 con 0x55cb72f7e2f0 2024-05-27T09:46:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.065+0000 7fb0bcf63740 1 -- 172.21.15.44:0/3188118503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 msgr2=0x55cb7334f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.065+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/3188118503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb7334f240 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0104940 tx=0x7fb0a001a000 comp rx=0 tx=0).stop 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.065+0000 7fb0bcf63740 1 -- 172.21.15.44:0/3188118503 shutdown_connections 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.065+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/3188118503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72fa71b0 0x55cb7335fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.065+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/3188118503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72f864d0 0x55cb73353fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.065+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/3188118503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb7334f240 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.065+0000 7fb0bcf63740 1 -- 172.21.15.44:0/3188118503 >> 172.21.15.44:0/3188118503 conn(0x55cb72f9b300 msgr2=0x55cb73355180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.065+0000 7fb0bcf63740 1 -- 172.21.15.44:0/3188118503 shutdown_connections 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.065+0000 7fb0bcf63740 1 -- 172.21.15.44:0/3188118503 wait complete. 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.066+0000 7fb0bcf63740 1 -- start start 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.066+0000 7fb0bcf63740 1 Processor -- start 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.066+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb73365990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.066+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f864d0 0x55cb73365ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.066+0000 7fb0bcf63740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72fa71b0 0x55cb73366410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.066+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01b0330 con 0x55cb72fa71b0 2024-05-27T09:46:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.066+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01b17a0 con 0x55cb72f7e2f0 2024-05-27T09:46:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.066+0000 7fb0bcf63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01b0a00 con 0x55cb72f864d0 2024-05-27T09:46:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0bcf63740 10 client.0 _ll_register_callbacks cb 0x7fb0a0009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:46:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0aea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb73365990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f864d0 0x55cb73365ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0aea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb73365990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:59996/0 (socket says 172.21.15.44:59996) 2024-05-27T09:46:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0aea00640 1 -- 172.21.15.44:0/689733467 learned_addr learned my addr 172.21.15.44:0/689733467 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0af400640 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72fa71b0 0x55cb73366410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0a6000640 1 -- 172.21.15.44:0/689733467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 msgr2=0x55cb73365990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0a6000640 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb73365990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0a6000640 1 -- 172.21.15.44:0/689733467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72fa71b0 msgr2=0x55cb73366410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0a6000640 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72fa71b0 0x55cb73366410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0a6000640 1 -- 172.21.15.44:0/689733467 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb733cbfb0 con 0x55cb72f864d0 2024-05-27T09:46:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb0a6000640 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f864d0 0x55cb73365ed0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb0980dfd50 tx=0x7fb0980a92e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2489008caae2b6e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.067+0000 7fb09cc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:46:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb09cc00640 1 -- 172.21.15.44:0/689733467 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0980a9fe0 con 0x55cb72f864d0 2024-05-27T09:46:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb09cc00640 1 -- 172.21.15.44:0/689733467 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fb0980a5ed0 con 0x55cb72f864d0 2024-05-27T09:46:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb0bcf63740 1 -- 172.21.15.44:0/689733467 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0a0030070 con 0x55cb72f864d0 2024-05-27T09:46:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb09cc00640 1 -- 172.21.15.44:0/689733467 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb0980a98e0 con 0x55cb72f864d0 2024-05-27T09:46:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb0bcf63740 10 client.27379 fetch_fsmap learned FSMap version 34 2024-05-27T09:46:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb0bcf63740 1 -- 172.21.15.44:0/689733467 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cb733cd220 con 0x55cb72f864d0 2024-05-27T09:46:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb09cc00640 1 -- 172.21.15.44:0/689733467 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a0030070 con 0x55cb72f864d0 2024-05-27T09:46:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb09cc00640 1 -- 172.21.15.44:0/689733467 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fb0980dffd0 con 0x55cb72f864d0 2024-05-27T09:46:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb0bcf63740 10 client.27379 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:46:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb0bcf63740 10 client.27379 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:46:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb0bcf63740 20 client.27379 populate_metadata read hostname 'smithi044' 2024-05-27T09:46:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.068+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cb72f3b1c0 0x55cb733cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.069+0000 7fb0bcf63740 4 client.27379 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:46:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.069+0000 7fb0bcf63740 1 -- 172.21.15.44:0/689733467 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cb732b62c0 con 0x55cb72f3b1c0 2024-05-27T09:46:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.069+0000 7fb0aea00640 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cb72f3b1c0 0x55cb733cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.069+0000 7fb0aea00640 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cb72f3b1c0 0x55cb733cf9b0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=334daefb14ffbd7a server_cookie=a03e7581c831ed7e in_seq=0 out_seq=0 2024-05-27T09:46:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.069+0000 7fb09cc00640 10 client.27379 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:35.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.508+0000 7fb09cc00640 1 -- 172.21.15.44:0/689733467 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55cb732b62c0 con 0x55cb72f3b1c0 2024-05-27T09:46:35.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.508+0000 7fb09cc00640 10 client.27379 handle_command_reply: tid=0 2024-05-27T09:46:35.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.508+0000 7fb0bcf63740 1 client.27379 shutdown 2024-05-27T09:46:35.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.509+0000 7fb0bcf63740 1 -- 172.21.15.44:0/689733467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f864d0 msgr2=0x55cb73365ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:35.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.509+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f864d0 0x55cb73365ed0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb0980dfd50 tx=0x7fb0980a92e0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.510+0000 7fb0bcf63740 1 -- 172.21.15.44:0/689733467 shutdown_connections 2024-05-27T09:46:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.510+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cb72f3b1c0 0x55cb733cf9b0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.510+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cb72fa71b0 0x55cb73366410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.510+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb72f864d0 0x55cb73365ed0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.510+0000 7fb0bcf63740 1 --2- 172.21.15.44:0/689733467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cb72f7e2f0 0x55cb73365990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.510+0000 7fb0bcf63740 1 -- 172.21.15.44:0/689733467 >> 172.21.15.44:0/689733467 conn(0x55cb72f9b300 msgr2=0x55cb733539d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.510+0000 7fb09cc00640 0 client.27379 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.510+0000 7fb0bcf63740 1 -- 172.21.15.44:0/689733467 shutdown_connections 2024-05-27T09:46:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.510+0000 7fb0bcf63740 1 -- 172.21.15.44:0/689733467 wait complete. 2024-05-27T09:46:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.511+0000 7fb0bcf63740 20 client.27379 trim_cache size 0 max 16384 2024-05-27T09:46:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:46:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:46:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:46:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.512+0000 7fb0b5e00640 1 -- 172.21.15.44:0/4149083989 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0940b2950 msgr2=0x7fb0940b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.512+0000 7fb0b5e00640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0940b2950 0x7fb0940b4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55cb7334ac10 tx=0x7fb0a0009500 comp rx=0 tx=0).stop 2024-05-27T09:46:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.512+0000 7fb0b5e00640 1 -- 172.21.15.44:0/4149083989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b0109c60 msgr2=0x7fb0b01a2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.512+0000 7fb0b5e00640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b0109c60 0x7fb0b01a2b70 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb0b006b420 tx=0x7fb09800ee50 comp rx=0 tx=0).stop 2024-05-27T09:46:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.512+0000 7fb0b5e00640 1 -- 172.21.15.44:0/4149083989 shutdown_connections 2024-05-27T09:46:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.512+0000 7fb0b5e00640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b010a100 0x7fb0b019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.512+0000 7fb0b5e00640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b0109c60 0x7fb0b01a2b70 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.512+0000 7fb0b5e00640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0940b2950 0x7fb0940b4e60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.512+0000 7fb0b5e00640 1 --2- 172.21.15.44:0/4149083989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0b01043b0 0x7fb0b01a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.512+0000 7fb0b5e00640 1 -- 172.21.15.44:0/4149083989 >> 172.21.15.44:0/4149083989 conn(0x7fb0b00ff320 msgr2=0x7fb0b0106a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.513+0000 7fb0b5e00640 1 -- 172.21.15.44:0/4149083989 shutdown_connections 2024-05-27T09:46:35.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:35.513+0000 7fb0b5e00640 1 -- 172.21.15.44:0/4149083989 wait complete. 2024-05-27T09:46:36.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:36 smithi050 ceph-mon[36371]: pgmap v355: 97 pgs: 97 active+clean; 299 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 610 op/s 2024-05-27T09:46:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:36 smithi189 ceph-mon[36671]: pgmap v355: 97 pgs: 97 active+clean; 299 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 610 op/s 2024-05-27T09:46:36.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:36 smithi044 ceph-mon[31618]: pgmap v355: 97 pgs: 97 active+clean; 299 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 610 op/s 2024-05-27T09:46:38.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:38 smithi044 ceph-mon[31618]: pgmap v356: 97 pgs: 97 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.7 MiB/s wr, 943 op/s 2024-05-27T09:46:38.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:38 smithi050 ceph-mon[36371]: pgmap v356: 97 pgs: 97 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.7 MiB/s wr, 943 op/s 2024-05-27T09:46:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:38 smithi189 ceph-mon[36671]: pgmap v356: 97 pgs: 97 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.7 MiB/s wr, 943 op/s 2024-05-27T09:46:40.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:40 smithi050 ceph-mon[36371]: pgmap v357: 97 pgs: 97 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 589 op/s 2024-05-27T09:46:40.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:40 smithi189 ceph-mon[36671]: pgmap v357: 97 pgs: 97 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 589 op/s 2024-05-27T09:46:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:40 smithi044 ceph-mon[31618]: pgmap v357: 97 pgs: 97 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 589 op/s 2024-05-27T09:46:42.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:42 smithi050 ceph-mon[36371]: pgmap v358: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 938 op/s 2024-05-27T09:46:42.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:42 smithi189 ceph-mon[36671]: pgmap v358: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 938 op/s 2024-05-27T09:46:42.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:42 smithi044 ceph-mon[31618]: pgmap v358: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 938 op/s 2024-05-27T09:46:44.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:44 smithi050 ceph-mon[36371]: pgmap v359: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 682 op/s 2024-05-27T09:46:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:44 smithi189 ceph-mon[36671]: pgmap v359: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 682 op/s 2024-05-27T09:46:44.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:44 smithi044 ceph-mon[31618]: pgmap v359: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 682 op/s 2024-05-27T09:46:46.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:46 smithi050 ceph-mon[36371]: pgmap v360: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 682 op/s 2024-05-27T09:46:46.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:46 smithi189 ceph-mon[36671]: pgmap v360: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 682 op/s 2024-05-27T09:46:46.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:46 smithi044 ceph-mon[31618]: pgmap v360: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 682 op/s 2024-05-27T09:46:48.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:48 smithi050 ceph-mon[36371]: pgmap v361: 97 pgs: 97 active+clean; 297 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 1.13k op/s 2024-05-27T09:46:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:48 smithi189 ceph-mon[36671]: pgmap v361: 97 pgs: 97 active+clean; 297 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 1.13k op/s 2024-05-27T09:46:48.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:48 smithi044 ceph-mon[31618]: pgmap v361: 97 pgs: 97 active+clean; 297 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 1.13k op/s 2024-05-27T09:46:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.720+0000 7f0a8ee00640 1 -- 172.21.15.44:0/2685412895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a88109c90 msgr2=0x7f0a88103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.720+0000 7f0a8ee00640 1 --2- 172.21.15.44:0/2685412895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a88109c90 0x7f0a88103830 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0a78009900 tx=0x7f0a7802a3f0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.720+0000 7f0a8ee00640 1 -- 172.21.15.44:0/2685412895 shutdown_connections 2024-05-27T09:46:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.720+0000 7f0a8ee00640 1 --2- 172.21.15.44:0/2685412895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a881043e0 0x7f0a881107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.720+0000 7f0a8ee00640 1 --2- 172.21.15.44:0/2685412895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a8810a130 0x7f0a88103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.720+0000 7f0a8ee00640 1 --2- 172.21.15.44:0/2685412895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a88109c90 0x7f0a88103830 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.720+0000 7f0a8ee00640 1 -- 172.21.15.44:0/2685412895 >> 172.21.15.44:0/2685412895 conn(0x7f0a88079390 msgr2=0x7f0a880797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.721+0000 7f0a8ee00640 1 -- 172.21.15.44:0/2685412895 shutdown_connections 2024-05-27T09:46:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.721+0000 7f0a8ee00640 1 -- 172.21.15.44:0/2685412895 wait complete. 2024-05-27T09:46:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.722+0000 7f0a8ee00640 1 Processor -- start 2024-05-27T09:46:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.722+0000 7f0a8ee00640 1 -- start start 2024-05-27T09:46:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.722+0000 7f0a8ee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a881043e0 0x7f0a8819d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.722+0000 7f0a8ee00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a88109c90 0x7f0a8819dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.722+0000 7f0a8ee00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a8810a130 0x7f0a881a8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.722+0000 7f0a8ee00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a88113eb0 con 0x7f0a88109c90 2024-05-27T09:46:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.722+0000 7f0a8ee00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0a88113d30 con 0x7f0a8810a130 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.722+0000 7f0a8ee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a88114030 con 0x7f0a881043e0 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.722+0000 7f0a87e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a881043e0 0x7f0a8819d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.722+0000 7f0a87e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a881043e0 0x7f0a8819d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47446/0 (socket says 172.21.15.44:47446) 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.722+0000 7f0a87e00640 1 -- 172.21.15.44:0/899993434 learned_addr learned my addr 172.21.15.44:0/899993434 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.723+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a8810a130 0x7f0a881a8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.723+0000 7f0a87400640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a88109c90 0x7f0a8819dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.723+0000 7f0a87e00640 1 -- 172.21.15.44:0/899993434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a8810a130 msgr2=0x7f0a881a8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.723+0000 7f0a87e00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a8810a130 0x7f0a881a8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.723+0000 7f0a87e00640 1 -- 172.21.15.44:0/899993434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a88109c90 msgr2=0x7f0a8819dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.723+0000 7f0a87e00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a88109c90 0x7f0a8819dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.723+0000 7f0a87e00640 1 -- 172.21.15.44:0/899993434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a881a8d50 con 0x7f0a881043e0 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.723+0000 7f0a87e00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a881043e0 0x7f0a8819d790 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0a78005ab0 tx=0x7f0a78003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.723+0000 7f0a84c00640 1 -- 172.21.15.44:0/899993434 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7803f070 con 0x7f0a881043e0 2024-05-27T09:46:48.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.724+0000 7f0a84c00640 1 -- 172.21.15.44:0/899993434 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a78031b50 con 0x7f0a881043e0 2024-05-27T09:46:48.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.724+0000 7f0a8ee00640 1 -- 172.21.15.44:0/899993434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a881a8f20 con 0x7f0a881043e0 2024-05-27T09:46:48.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.724+0000 7f0a8ee00640 1 -- 172.21.15.44:0/899993434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a881a9330 con 0x7f0a881043e0 2024-05-27T09:46:48.726 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.724+0000 7f0a84c00640 1 -- 172.21.15.44:0/899993434 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78035030 con 0x7f0a881043e0 2024-05-27T09:46:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.727+0000 7f0a95df5740 1 Processor -- start 2024-05-27T09:46:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.727+0000 7f0a95df5740 1 -- start start 2024-05-27T09:46:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.727+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 0x555f2a2ed860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.727+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f222f0 0x555f2a2edda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.727+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f2a4d0 0x555f2a2ee2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.727+0000 7f0a95df5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555f2a25eb90 con 0x555f29f222f0 2024-05-27T09:46:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.727+0000 7f0a95df5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x555f2a24e730 con 0x555f29f4b1b0 2024-05-27T09:46:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.727+0000 7f0a95df5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555f2a252770 con 0x555f29f2a4d0 2024-05-27T09:46:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a84c00640 1 -- 172.21.15.44:0/899993434 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0a78042070 con 0x7f0a881043e0 2024-05-27T09:46:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a8ce00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 0x555f2a2ed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a87400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f2a4d0 0x555f2a2ee2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a8ce00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 0x555f2a2ed860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:57044/0 (socket says 172.21.15.44:57044) 2024-05-27T09:46:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a8ce00640 1 -- 172.21.15.44:0/3814973524 learned_addr learned my addr 172.21.15.44:0/3814973524 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a87e00640 1 --2- 172.21.15.44:0/3814973524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f222f0 0x555f2a2edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a8ce00640 1 -- 172.21.15.44:0/3814973524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f2a4d0 msgr2=0x555f2a2ee2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3814973524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f2a4d0 0x555f2a2ee2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a8ce00640 1 -- 172.21.15.44:0/3814973524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f222f0 msgr2=0x555f2a2edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3814973524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f222f0 0x555f2a2edda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a8ce00640 1 -- 172.21.15.44:0/3814973524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f2a1e78b0 con 0x555f29f4b1b0 2024-05-27T09:46:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.728+0000 7f0a87400640 1 --2- 172.21.15.44:0/3814973524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f2a4d0 0x555f2a2ee2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:46:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.729+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3814973524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 0x555f2a2ed860 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0a80011440 tx=0x7f0a80011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=405d33284c1f8a2e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.729+0000 7f0a7d600640 1 -- 172.21.15.44:0/3814973524 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80003c30 con 0x555f29f4b1b0 2024-05-27T09:46:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.729+0000 7f0a7d600640 1 -- 172.21.15.44:0/3814973524 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a80003dd0 con 0x555f29f4b1b0 2024-05-27T09:46:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.729+0000 7f0a7d600640 1 -- 172.21.15.44:0/3814973524 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8000fae0 con 0x555f29f4b1b0 2024-05-27T09:46:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.729+0000 7f0a84c00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a6c0b2950 0x7f0a6c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.730+0000 7f0a84c00640 1 -- 172.21.15.44:0/899993434 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7f0a781028a0 con 0x7f0a881043e0 2024-05-27T09:46:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.730+0000 7f0a87e00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a6c0b2950 0x7f0a6c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.730+0000 7f0a95df5740 1 -- 172.21.15.44:0/3814973524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 msgr2=0x555f2a2ed860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.730+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3814973524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 0x555f2a2ed860 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0a80011440 tx=0x7f0a80011900 comp rx=0 tx=0).stop 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.730+0000 7f0a95df5740 1 -- 172.21.15.44:0/3814973524 shutdown_connections 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.730+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3814973524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f2a4d0 0x555f2a2ee2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.730+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3814973524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f222f0 0x555f2a2edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.730+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3814973524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 0x555f2a2ed860 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.730+0000 7f0a95df5740 1 -- 172.21.15.44:0/3814973524 >> 172.21.15.44:0/3814973524 conn(0x555f29f3f300 msgr2=0x555f2a2ea010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.730+0000 7f0a95df5740 1 -- 172.21.15.44:0/3814973524 shutdown_connections 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.730+0000 7f0a87e00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a6c0b2950 0x7f0a6c0b4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0a781000a0 tx=0x7f0a78031f70 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.731+0000 7f0a95df5740 1 -- 172.21.15.44:0/3814973524 wait complete. 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.731+0000 7f0a95df5740 1 -- start start 2024-05-27T09:46:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.732+0000 7f0a95df5740 1 Processor -- start 2024-05-27T09:46:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.732+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f222f0 0x555f2a2f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.732+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a2f7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.732+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 0x555f2a303cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.732+0000 7f0a95df5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555f29f34790 con 0x555f29f2a4d0 2024-05-27T09:46:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.732+0000 7f0a95df5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x555f2a1f6bc0 con 0x555f29f4b1b0 2024-05-27T09:46:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.732+0000 7f0a95df5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555f2a2fa990 con 0x555f29f222f0 2024-05-27T09:46:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a95df5740 10 client.0 _ll_register_callbacks cb 0x555f2a1d4400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:46:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a8ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f222f0 0x555f2a2f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a87400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a2f7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a87400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a2f7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57642/0 (socket says 172.21.15.44:57642) 2024-05-27T09:46:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a87400640 1 -- 172.21.15.44:0/3126484480 learned_addr learned my addr 172.21.15.44:0/3126484480 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 0x555f2a303cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a87400640 1 -- 172.21.15.44:0/3126484480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f222f0 msgr2=0x555f2a2f79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a87400640 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f222f0 0x555f2a2f79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a87400640 1 -- 172.21.15.44:0/3126484480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 msgr2=0x555f2a303cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a87400640 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 0x555f2a303cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a87400640 1 -- 172.21.15.44:0/3126484480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f29f63210 con 0x555f29f2a4d0 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f222f0 0x555f2a2f79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a87400640 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a2f7f30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0a74012700 tx=0x7f0a74012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f5714725edd397 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.733+0000 7f0a72000640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.734+0000 7f0a72000640 1 -- 172.21.15.44:0/3126484480 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74004030 con 0x555f29f2a4d0 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.734+0000 7f0a72000640 1 -- 172.21.15.44:0/3126484480 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7f0a740041d0 con 0x555f29f2a4d0 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.734+0000 7f0a95df5740 1 -- 172.21.15.44:0/3126484480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a80003dd0 con 0x555f29f2a4d0 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.734+0000 7f0a72000640 1 -- 172.21.15.44:0/3126484480 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0a740043d0 con 0x555f29f2a4d0 2024-05-27T09:46:48.737 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.734+0000 7f0a95df5740 10 client.17169 fetch_fsmap learned FSMap version 34 2024-05-27T09:46:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.734+0000 7f0a95df5740 1 -- 172.21.15.44:0/3126484480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555f2a02f620 con 0x555f29f2a4d0 2024-05-27T09:46:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.734+0000 7f0a72000640 1 -- 172.21.15.44:0/3126484480 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a80003dd0 con 0x555f29f2a4d0 2024-05-27T09:46:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.735+0000 7f0a72000640 1 -- 172.21.15.44:0/3126484480 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f0a74003b30 con 0x555f29f2a4d0 2024-05-27T09:46:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.735+0000 7f0a95df5740 10 client.17169 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:46:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.735+0000 7f0a95df5740 10 client.17169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:46:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.735+0000 7f0a95df5740 20 client.17169 populate_metadata read hostname 'smithi044' 2024-05-27T09:46:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.735+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x555f29f09d80 0x555f2a309c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.735+0000 7f0a95df5740 4 client.17169 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:46:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.735+0000 7f0a95df5740 1 -- 172.21.15.44:0/3126484480 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f2a260ae0 con 0x555f29f09d80 2024-05-27T09:46:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.735+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x555f29f09d80 0x555f2a309c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.736+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x555f29f09d80 0x555f2a309c60 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b108d91e33abdf74 server_cookie=d76a99f7519d957f in_seq=0 out_seq=0 2024-05-27T09:46:48.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.736+0000 7f0a72000640 10 client.17169 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:48.739 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a72000640 1 -- 172.21.15.44:0/3126484480 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555f2a260ae0 con 0x555f29f09d80 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a72000640 10 client.17169 handle_command_reply: tid=0 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a95df5740 1 client.17169 shutdown 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a95df5740 1 -- 172.21.15.44:0/3126484480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 msgr2=0x555f2a2f7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a2f7f30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0a74012700 tx=0x7f0a74012bc0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a95df5740 1 -- 172.21.15.44:0/3126484480 shutdown_connections 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x555f29f09d80 0x555f2a309c60 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f4b1b0 0x555f2a303cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a2f7f30 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a72000640 0 client.17169 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3126484480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f222f0 0x555f2a2f79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a95df5740 1 -- 172.21.15.44:0/3126484480 >> 172.21.15.44:0/3126484480 conn(0x555f29f3f300 msgr2=0x555f2a2ed690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:48.740 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.738+0000 7f0a95df5740 1 -- 172.21.15.44:0/3126484480 shutdown_connections 2024-05-27T09:46:48.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.739+0000 7f0a95df5740 1 -- 172.21.15.44:0/3126484480 wait complete. 2024-05-27T09:46:48.741 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.739+0000 7f0a95df5740 20 client.17169 trim_cache size 0 max 16384 2024-05-27T09:46:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.759+0000 7f0a95df5740 1 Processor -- start 2024-05-27T09:46:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.759+0000 7f0a95df5740 1 -- start start 2024-05-27T09:46:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.759+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 0x555f2a303a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.759+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 0x555f2a303f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.759+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a30f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.759+0000 7f0a95df5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a881b47f0 con 0x555f29f2a4d0 2024-05-27T09:46:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.759+0000 7f0a95df5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0a881b2060 con 0x555f29f222f0 2024-05-27T09:46:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.759+0000 7f0a95df5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a881afed0 con 0x555f29f09d80 2024-05-27T09:46:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.759+0000 7f0a87400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 0x555f2a303f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a87400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 0x555f2a303f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:57052/0 (socket says 172.21.15.44:57052) 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a87400640 1 -- 172.21.15.44:0/3560097760 learned_addr learned my addr 172.21.15.44:0/3560097760 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3560097760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 0x555f2a303a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3560097760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a30f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a87400640 1 -- 172.21.15.44:0/3560097760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 msgr2=0x555f2a303a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a87400640 1 --2- 172.21.15.44:0/3560097760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 0x555f2a303a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a87400640 1 -- 172.21.15.44:0/3560097760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 msgr2=0x555f2a30f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a87400640 1 --2- 172.21.15.44:0/3560097760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a30f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a87400640 1 -- 172.21.15.44:0/3560097760 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f2a30a1c0 con 0x555f29f222f0 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3560097760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 0x555f2a303a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a87400640 1 --2- 172.21.15.44:0/3560097760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 0x555f2a303f70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0a74000c00 tx=0x7f0a7400f850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f06891b880cf5191 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.760+0000 7f0a7cc00640 1 -- 172.21.15.44:0/3560097760 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74029040 con 0x555f29f222f0 2024-05-27T09:46:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.761+0000 7f0a7cc00640 1 -- 172.21.15.44:0/3560097760 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a74005700 con 0x555f29f222f0 2024-05-27T09:46:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.761+0000 7f0a7cc00640 1 -- 172.21.15.44:0/3560097760 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74003ba0 con 0x555f29f222f0 2024-05-27T09:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.761+0000 7f0a95df5740 1 -- 172.21.15.44:0/3560097760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 msgr2=0x555f2a303f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.761+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3560097760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 0x555f2a303f70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0a74000c00 tx=0x7f0a7400f850 comp rx=0 tx=0).stop 2024-05-27T09:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.762+0000 7f0a95df5740 1 -- 172.21.15.44:0/3560097760 shutdown_connections 2024-05-27T09:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.762+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3560097760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a30f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.762+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3560097760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 0x555f2a303f70 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.762+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3560097760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 0x555f2a303a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.762+0000 7f0a95df5740 1 -- 172.21.15.44:0/3560097760 >> 172.21.15.44:0/3560097760 conn(0x555f29f3f300 msgr2=0x555f2a2f7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.762+0000 7f0a95df5740 1 -- 172.21.15.44:0/3560097760 shutdown_connections 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.762+0000 7f0a95df5740 1 -- 172.21.15.44:0/3560097760 wait complete. 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.762+0000 7f0a95df5740 1 -- start start 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.762+0000 7f0a95df5740 1 Processor -- start 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.763+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 0x555f2a300100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.763+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 0x555f2a2fd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.763+0000 7f0a95df5740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a2fd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.763+0000 7f0a95df5740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a881b0520 con 0x555f29f2a4d0 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.763+0000 7f0a95df5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0a881b1990 con 0x555f29f222f0 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.763+0000 7f0a95df5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a881b0bf0 con 0x555f29f09d80 2024-05-27T09:46:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.763+0000 7f0a95df5740 10 client.0 _ll_register_callbacks cb 0x7f0a74009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:46:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.763+0000 7f0a8ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 0x555f2a300100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.763+0000 7f0a8ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 0x555f2a300100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47484/0 (socket says 172.21.15.44:47484) 2024-05-27T09:46:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.763+0000 7f0a8ce00640 1 -- 172.21.15.44:0/3557384506 learned_addr learned my addr 172.21.15.44:0/3557384506 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:46:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.764+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a2fd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.764+0000 7f0a87400640 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 0x555f2a2fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.764+0000 7f0a87400640 1 -- 172.21.15.44:0/3557384506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 msgr2=0x555f2a300100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.764+0000 7f0a87400640 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 0x555f2a300100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.764+0000 7f0a87400640 1 -- 172.21.15.44:0/3557384506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 msgr2=0x555f2a2fd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.764+0000 7f0a87400640 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a2fd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.764+0000 7f0a87400640 1 -- 172.21.15.44:0/3557384506 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f2a2fdef0 con 0x555f29f222f0 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.764+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 0x555f2a300100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.764+0000 7f0a87400640 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 0x555f2a2fd470 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0a7401c480 tx=0x7f0a74005fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4e5681c4e99c68c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.764+0000 7f0a72000640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.765+0000 7f0a72000640 1 -- 172.21.15.44:0/3557384506 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74029030 con 0x555f29f222f0 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.765+0000 7f0a72000640 1 -- 172.21.15.44:0/3557384506 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7f0a74005240 con 0x555f29f222f0 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.765+0000 7f0a95df5740 1 -- 172.21.15.44:0/3557384506 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a74005700 con 0x555f29f222f0 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.765+0000 7f0a72000640 1 -- 172.21.15.44:0/3557384506 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0a74005460 con 0x555f29f222f0 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.765+0000 7f0a95df5740 10 client.27407 fetch_fsmap learned FSMap version 34 2024-05-27T09:46:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.765+0000 7f0a95df5740 1 -- 172.21.15.44:0/3557384506 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555f2a371f90 con 0x555f29f222f0 2024-05-27T09:46:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.766+0000 7f0a72000640 1 -- 172.21.15.44:0/3557384506 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a74005700 con 0x555f29f222f0 2024-05-27T09:46:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.766+0000 7f0a72000640 1 -- 172.21.15.44:0/3557384506 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f0a7401c5d0 con 0x555f29f222f0 2024-05-27T09:46:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.766+0000 7f0a95df5740 10 client.27407 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:46:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.766+0000 7f0a95df5740 10 client.27407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:46:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.766+0000 7f0a95df5740 20 client.27407 populate_metadata read hostname 'smithi044' 2024-05-27T09:46:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.766+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x555f29f4b1b0 0x555f2a374650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:46:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.766+0000 7f0a95df5740 4 client.27407 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:46:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.766+0000 7f0a95df5740 1 -- 172.21.15.44:0/3557384506 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555f2a25a2c0 con 0x555f29f4b1b0 2024-05-27T09:46:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.766+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x555f29f4b1b0 0x555f2a374650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:46:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.767+0000 7f0a72000640 10 client.27407 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:48.767+0000 7f0a8ce00640 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x555f29f4b1b0 0x555f2a374650 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74b65bf08dc464fb server_cookie=5f16bdd09fb6398d in_seq=0 out_seq=0 2024-05-27T09:46:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a72000640 1 -- 172.21.15.44:0/3557384506 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x555f2a25a2c0 con 0x555f29f4b1b0 2024-05-27T09:46:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a72000640 10 client.27407 handle_command_reply: tid=0 2024-05-27T09:46:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a95df5740 1 client.27407 shutdown 2024-05-27T09:46:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a95df5740 1 -- 172.21.15.44:0/3557384506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 msgr2=0x555f2a2fd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 0x555f2a2fd470 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0a7401c480 tx=0x7f0a74005fa0 comp rx=0 tx=0).stop 2024-05-27T09:46:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a95df5740 1 -- 172.21.15.44:0/3557384506 shutdown_connections 2024-05-27T09:46:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x555f29f4b1b0 0x555f2a374650 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f29f2a4d0 0x555f2a2fd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555f29f222f0 0x555f2a2fd470 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a95df5740 1 --2- 172.21.15.44:0/3557384506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f29f09d80 0x555f2a300100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a72000640 0 client.27407 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:46:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.379+0000 7f0a95df5740 1 -- 172.21.15.44:0/3557384506 >> 172.21.15.44:0/3557384506 conn(0x555f29f3f300 msgr2=0x555f2a2f2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.380+0000 7f0a95df5740 1 -- 172.21.15.44:0/3557384506 shutdown_connections 2024-05-27T09:46:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.380+0000 7f0a95df5740 1 -- 172.21.15.44:0/3557384506 wait complete. 2024-05-27T09:46:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.380+0000 7f0a95df5740 20 client.27407 trim_cache size 0 max 16384 2024-05-27T09:46:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:46:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:46:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:46:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 -- 172.21.15.44:0/899993434 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a6c0b2950 msgr2=0x7f0a6c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a6c0b2950 0x7f0a6c0b4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0a781000a0 tx=0x7f0a78031f70 comp rx=0 tx=0).stop 2024-05-27T09:46:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 -- 172.21.15.44:0/899993434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a881043e0 msgr2=0x7f0a8819d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:46:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a881043e0 0x7f0a8819d790 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0a78005ab0 tx=0x7f0a78003b90 comp rx=0 tx=0).stop 2024-05-27T09:46:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 -- 172.21.15.44:0/899993434 shutdown_connections 2024-05-27T09:46:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a8810a130 0x7f0a881a8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a6c0b2950 0x7f0a6c0b4e60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a88109c90 0x7f0a8819dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 --2- 172.21.15.44:0/899993434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a881043e0 0x7f0a8819d790 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:46:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 -- 172.21.15.44:0/899993434 >> 172.21.15.44:0/899993434 conn(0x7f0a88079390 msgr2=0x7f0a881023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:46:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 -- 172.21.15.44:0/899993434 shutdown_connections 2024-05-27T09:46:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:46:49.381+0000 7f0a8ee00640 1 -- 172.21.15.44:0/899993434 wait complete. 2024-05-27T09:46:50.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:50 smithi050 ceph-mon[36371]: pgmap v362: 97 pgs: 97 active+clean; 297 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 793 op/s 2024-05-27T09:46:50.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:50 smithi189 ceph-mon[36671]: pgmap v362: 97 pgs: 97 active+clean; 297 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 793 op/s 2024-05-27T09:46:50.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:50 smithi044 ceph-mon[31618]: pgmap v362: 97 pgs: 97 active+clean; 297 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 793 op/s 2024-05-27T09:46:52.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:52 smithi050 ceph-mon[36371]: pgmap v363: 97 pgs: 97 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 1.30k op/s 2024-05-27T09:46:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:52 smithi189 ceph-mon[36671]: pgmap v363: 97 pgs: 97 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 1.30k op/s 2024-05-27T09:46:52.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:52 smithi044 ceph-mon[31618]: pgmap v363: 97 pgs: 97 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 1.30k op/s 2024-05-27T09:46:54.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:54 smithi189 ceph-mon[36671]: pgmap v364: 97 pgs: 97 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 948 op/s 2024-05-27T09:46:54.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:54 smithi044 ceph-mon[31618]: pgmap v364: 97 pgs: 97 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 948 op/s 2024-05-27T09:46:54.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:54 smithi050 ceph-mon[36371]: pgmap v364: 97 pgs: 97 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 948 op/s 2024-05-27T09:46:56.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:56 smithi189 ceph-mon[36671]: pgmap v365: 97 pgs: 97 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 948 op/s 2024-05-27T09:46:56.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:56 smithi044 ceph-mon[31618]: pgmap v365: 97 pgs: 97 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 948 op/s 2024-05-27T09:46:56.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:56 smithi050 ceph-mon[36371]: pgmap v365: 97 pgs: 97 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 948 op/s 2024-05-27T09:46:58.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:46:58 smithi189 ceph-mon[36671]: pgmap v366: 97 pgs: 97 active+clean; 315 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 1.32k op/s 2024-05-27T09:46:58.341 INFO:tasks.quiescer.fs.[cephfs]:Total duration (44.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-05-27T09:46:58.341 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.3 2024-05-27T09:46:58.341 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:46:58.398 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:46:58 smithi044 ceph-mon[31618]: pgmap v366: 97 pgs: 97 active+clean; 315 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 1.32k op/s 2024-05-27T09:46:58.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:46:58 smithi050 ceph-mon[36371]: pgmap v366: 97 pgs: 97 active+clean; 315 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 1.32k op/s 2024-05-27T09:46:58.845 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:00.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:00 smithi189 ceph-mon[36671]: pgmap v367: 97 pgs: 97 active+clean; 315 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 872 op/s 2024-05-27T09:47:00.426 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:00 smithi044 ceph-mon[31618]: pgmap v367: 97 pgs: 97 active+clean; 315 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 872 op/s 2024-05-27T09:47:00.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:00 smithi050 ceph-mon[36371]: pgmap v367: 97 pgs: 97 active+clean; 315 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 872 op/s 2024-05-27T09:47:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.023+0000 7f2967e00700 1 -- 172.21.15.44:0/3929550655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2968109670 msgr2=0x7f2968110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.023+0000 7f2967e00700 1 --2- 172.21.15.44:0/3929550655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2968109670 0x7f2968110fd0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2960009f90 tx=0x7f296000b330 comp rx=0 tx=0).stop 2024-05-27T09:47:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.023+0000 7f2967e00700 1 -- 172.21.15.44:0/3929550655 shutdown_connections 2024-05-27T09:47:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.023+0000 7f2967e00700 1 --2- 172.21.15.44:0/3929550655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2968109670 0x7f2968110fd0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.023+0000 7f2967e00700 1 --2- 172.21.15.44:0/3929550655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29681043f0 0x7f2968104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.023+0000 7f2967e00700 1 --2- 172.21.15.44:0/3929550655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2968108d20 0x7f2968109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.023+0000 7f2967e00700 1 -- 172.21.15.44:0/3929550655 >> 172.21.15.44:0/3929550655 conn(0x7f2968079d00 msgr2=0x7f296807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.023+0000 7f2967e00700 1 -- 172.21.15.44:0/3929550655 shutdown_connections 2024-05-27T09:47:01.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.023+0000 7f2967e00700 1 -- 172.21.15.44:0/3929550655 wait complete. 2024-05-27T09:47:01.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.024+0000 7f2967e00700 1 Processor -- start 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.024+0000 7f2967e00700 1 -- start start 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.025+0000 7f2967e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29681043f0 0x7f29681a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.025+0000 7f2967e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968108d20 0x7f29681a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.025+0000 7f2967e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2968109670 0x7f296819aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.025+0000 7f2967e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f29681142f0 con 0x7f29681043f0 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.025+0000 7f2967e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2968114170 con 0x7f2968109670 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.025+0000 7f2967e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2968114470 con 0x7f2968108d20 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.025+0000 7f2966a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29681043f0 0x7f29681a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.025+0000 7f2966000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968108d20 0x7f29681a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.025+0000 7f2966a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29681043f0 0x7f29681a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54304/0 (socket says 172.21.15.44:54304) 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.025+0000 7f2966a00700 1 -- 172.21.15.44:0/3716344090 learned_addr learned my addr 172.21.15.44:0/3716344090 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.025+0000 7f2967400700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2968109670 0x7f296819aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.026+0000 7f2966000700 1 -- 172.21.15.44:0/3716344090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2968109670 msgr2=0x7f296819aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.026+0000 7f2966000700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2968109670 0x7f296819aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.026+0000 7f2966000700 1 -- 172.21.15.44:0/3716344090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29681043f0 msgr2=0x7f29681a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.026+0000 7f2966000700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29681043f0 0x7f29681a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.026+0000 7f2966000700 1 -- 172.21.15.44:0/3716344090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f296819b100 con 0x7f2968108d20 2024-05-27T09:47:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.026+0000 7f2966000700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968108d20 0x7f29681a0d70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2954005fd0 tx=0x7f295400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.026+0000 7f295f400700 1 -- 172.21.15.44:0/3716344090 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295400cbd0 con 0x7f2968108d20 2024-05-27T09:47:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.026+0000 7f2967e00700 1 -- 172.21.15.44:0/3716344090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f296819b3f0 con 0x7f2968108d20 2024-05-27T09:47:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.026+0000 7f295f400700 1 -- 172.21.15.44:0/3716344090 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29540044d0 con 0x7f2968108d20 2024-05-27T09:47:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.026+0000 7f295f400700 1 -- 172.21.15.44:0/3716344090 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2954010430 con 0x7f2968108d20 2024-05-27T09:47:01.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.027+0000 7f2967e00700 1 -- 172.21.15.44:0/3716344090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29681a76c0 con 0x7f2968108d20 2024-05-27T09:47:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.028+0000 7f2967e00700 1 -- 172.21.15.44:0/3716344090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29680512b0 con 0x7f2968108d20 2024-05-27T09:47:01.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.031+0000 7f295f400700 1 -- 172.21.15.44:0/3716344090 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2954003eb0 con 0x7f2968108d20 2024-05-27T09:47:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.032+0000 7f295f400700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f294c0b29f0 0x7f294c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.032+0000 7f295f400700 1 -- 172.21.15.44:0/3716344090 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7f29540e27e0 con 0x7f2968108d20 2024-05-27T09:47:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.032+0000 7f2966a00700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f294c0b29f0 0x7f294c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:01.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.033+0000 7f2966a00700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f294c0b29f0 0x7f294c0b4f00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f296819bb90 tx=0x7f2958009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:01.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.037+0000 7f295f400700 1 -- 172.21.15.44:0/3716344090 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f29540aa160 con 0x7f2968108d20 2024-05-27T09:47:01.418 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:47:01.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.423+0000 7f2967e00700 1 -- 172.21.15.44:0/3716344090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f296819bf80 con 0x7f2968108d20 2024-05-27T09:47:01.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.424+0000 7f295f400700 1 -- 172.21.15.44:0/3716344090 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11868 (secure 0 0 0) 0x7f29540a9a60 con 0x7f2968108d20 2024-05-27T09:47:01.425 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:47:01.425 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:45:53.426075+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:47:01.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.427+0000 7f2967e00700 1 -- 172.21.15.44:0/3716344090 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f294c0b29f0 msgr2=0x7f294c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:01.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.427+0000 7f2967e00700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f294c0b29f0 0x7f294c0b4f00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f296819bb90 tx=0x7f2958009260 comp rx=0 tx=0).stop 2024-05-27T09:47:01.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.427+0000 7f2967e00700 1 -- 172.21.15.44:0/3716344090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968108d20 msgr2=0x7f29681a0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:01.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.427+0000 7f2967e00700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968108d20 0x7f29681a0d70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2954005fd0 tx=0x7f295400ee70 comp rx=0 tx=0).stop 2024-05-27T09:47:01.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.428+0000 7f2967e00700 1 -- 172.21.15.44:0/3716344090 shutdown_connections 2024-05-27T09:47:01.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.428+0000 7f2967e00700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f294c0b29f0 0x7f294c0b4f00 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f296819bb90 tx=0x7f2958009260 comp rx=0 tx=0).stop 2024-05-27T09:47:01.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.428+0000 7f2967e00700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2968109670 0x7f296819aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:01.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.428+0000 7f2967e00700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29681043f0 0x7f29681a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:01.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.428+0000 7f2967e00700 1 --2- 172.21.15.44:0/3716344090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968108d20 0x7f29681a0d70 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:01.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.428+0000 7f2967e00700 1 -- 172.21.15.44:0/3716344090 >> 172.21.15.44:0/3716344090 conn(0x7f2968079d00 msgr2=0x7f29681015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:01.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.428+0000 7f2967e00700 1 -- 172.21.15.44:0/3716344090 shutdown_connections 2024-05-27T09:47:01.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:01.428+0000 7f2967e00700 1 -- 172.21.15.44:0/3716344090 wait complete. 2024-05-27T09:47:01.430 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 34 2024-05-27T09:47:02.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:02 smithi044 ceph-mon[31618]: pgmap v368: 97 pgs: 97 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 1.08k op/s 2024-05-27T09:47:02.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:02 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3716344090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:02.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:02 smithi189 ceph-mon[36671]: pgmap v368: 97 pgs: 97 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 1.08k op/s 2024-05-27T09:47:02.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:02 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3716344090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:02.450 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:02.483 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db / --timeout 90 --expiration 142.3 --await-for 45.0' 2024-05-27T09:47:02.483 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db / --timeout 90 --expiration 142.3 --await-for 45.0 2024-05-27T09:47:02.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:02 smithi050 ceph-mon[36371]: pgmap v368: 97 pgs: 97 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 1.08k op/s 2024-05-27T09:47:02.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:02 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3716344090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.617+0000 7fb996600640 1 -- 172.21.15.44:0/3822981467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99010dc40 msgr2=0x7fb990110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.617+0000 7fb996600640 1 --2- 172.21.15.44:0/3822981467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99010dc40 0x7fb990110060 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb97c009900 tx=0x7fb97c02a410 comp rx=0 tx=0).stop 2024-05-27T09:47:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.617+0000 7fb996600640 1 -- 172.21.15.44:0/3822981467 shutdown_connections 2024-05-27T09:47:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.617+0000 7fb996600640 1 --2- 172.21.15.44:0/3822981467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9901105a0 0x7fb9901129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.617+0000 7fb996600640 1 --2- 172.21.15.44:0/3822981467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb99010dc40 0x7fb990110060 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.617+0000 7fb996600640 1 --2- 172.21.15.44:0/3822981467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb990102a00 0x7fb990102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.617+0000 7fb996600640 1 -- 172.21.15.44:0/3822981467 >> 172.21.15.44:0/3822981467 conn(0x7fb9900782d0 msgr2=0x7fb990078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.617+0000 7fb996600640 1 -- 172.21.15.44:0/3822981467 shutdown_connections 2024-05-27T09:47:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.617+0000 7fb996600640 1 -- 172.21.15.44:0/3822981467 wait complete. 2024-05-27T09:47:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.619+0000 7fb996600640 1 Processor -- start 2024-05-27T09:47:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.619+0000 7fb996600640 1 -- start start 2024-05-27T09:47:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.620+0000 7fb996600640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb990102a00 0x7fb99010ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.620+0000 7fb996600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99010dc40 0x7fb99010b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.621+0000 7fb996600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9901105a0 0x7fb99010b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.621+0000 7fb996600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb990116090 con 0x7fb99010dc40 2024-05-27T09:47:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.621+0000 7fb996600640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb990115f10 con 0x7fb990102a00 2024-05-27T09:47:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.621+0000 7fb996600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb990116210 con 0x7fb9901105a0 2024-05-27T09:47:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.621+0000 7fb98f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb990102a00 0x7fb99010ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.621+0000 7fb98f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb990102a00 0x7fb99010ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60116/0 (socket says 172.21.15.44:60116) 2024-05-27T09:47:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.622+0000 7fb98f400640 1 -- 172.21.15.44:0/205631221 learned_addr learned my addr 172.21.15.44:0/205631221 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.622+0000 7fb98ea00640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99010dc40 0x7fb99010b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.622+0000 7fb98ea00640 1 -- 172.21.15.44:0/205631221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9901105a0 msgr2=0x7fb99010b610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T09:47:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.622+0000 7fb98ea00640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9901105a0 0x7fb99010b610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.622+0000 7fb98ea00640 1 -- 172.21.15.44:0/205631221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb990102a00 msgr2=0x7fb99010ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.622+0000 7fb98ea00640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb990102a00 0x7fb99010ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.622+0000 7fb98ea00640 1 -- 172.21.15.44:0/205631221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9901acfe0 con 0x7fb99010dc40 2024-05-27T09:47:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.623+0000 7fb98ea00640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99010dc40 0x7fb99010b0d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb97c02a930 tx=0x7fb97c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.624+0000 7fb987e00640 1 -- 172.21.15.44:0/205631221 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c03f070 con 0x7fb99010dc40 2024-05-27T09:47:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.624+0000 7fb987e00640 1 -- 172.21.15.44:0/205631221 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb97c031b60 con 0x7fb99010dc40 2024-05-27T09:47:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.624+0000 7fb996600640 1 -- 172.21.15.44:0/205631221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9901ad270 con 0x7fb99010dc40 2024-05-27T09:47:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.624+0000 7fb987e00640 1 -- 172.21.15.44:0/205631221 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c03f070 con 0x7fb99010dc40 2024-05-27T09:47:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.624+0000 7fb996600640 1 -- 172.21.15.44:0/205631221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9901ad730 con 0x7fb99010dc40 2024-05-27T09:47:02.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.629+0000 7fb987e00640 1 -- 172.21.15.44:0/205631221 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb97c042070 con 0x7fb99010dc40 2024-05-27T09:47:02.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.629+0000 7fb99d56a740 1 Processor -- start 2024-05-27T09:47:02.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.629+0000 7fb99d56a740 1 -- start start 2024-05-27T09:47:02.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.629+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.629+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.629+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.629+0000 7fb99d56a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b515b90 con 0x55ae0b1e14d0 2024-05-27T09:47:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.629+0000 7fb99d56a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b505730 con 0x55ae0b1d92f0 2024-05-27T09:47:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.629+0000 7fb99d56a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b509770 con 0x55ae0b2021b0 2024-05-27T09:47:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.630+0000 7fb98f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.630+0000 7fb98f400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5a4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60124/0 (socket says 172.21.15.44:60124) 2024-05-27T09:47:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.630+0000 7fb98f400640 1 -- 172.21.15.44:0/2433001740 learned_addr learned my addr 172.21.15.44:0/2433001740 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.630+0000 7fb98fe00640 1 --2- 172.21.15.44:0/2433001740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.630+0000 7fb987e00640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb9740b2c00 0x7fb9740b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.630+0000 7fb987e00640 1 -- 172.21.15.44:0/205631221 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fb97c102520 con 0x7fb99010dc40 2024-05-27T09:47:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.630+0000 7fb98f400640 1 -- 172.21.15.44:0/2433001740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b2021b0 msgr2=0x55ae0b5a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.630+0000 7fb98f400640 1 --2- 172.21.15.44:0/2433001740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.630+0000 7fb98f400640 1 -- 172.21.15.44:0/2433001740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 msgr2=0x55ae0b5a52e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:47:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.630+0000 7fb98f400640 1 --2- 172.21.15.44:0/2433001740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a52e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.630+0000 7fb98f400640 1 -- 172.21.15.44:0/2433001740 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae0b49e8b0 con 0x55ae0b1d92f0 2024-05-27T09:47:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.631+0000 7fb98fe00640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb9740b2c00 0x7fb9740b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.631+0000 7fb98f400640 1 --2- 172.21.15.44:0/2433001740 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5a4da0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb980009d60 tx=0x7fb9800117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=433f3bd1cbe4caa2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.632+0000 7fb984c00640 1 -- 172.21.15.44:0/2433001740 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb980012450 con 0x55ae0b1d92f0 2024-05-27T09:47:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.632+0000 7fb98fe00640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb9740b2c00 0x7fb9740b5110 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55ae0b5a19d0 tx=0x7fb988004a40 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.632+0000 7fb984c00640 1 -- 172.21.15.44:0/2433001740 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb980012aa0 con 0x55ae0b1d92f0 2024-05-27T09:47:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.632+0000 7fb984c00640 1 -- 172.21.15.44:0/2433001740 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98000fb40 con 0x55ae0b1d92f0 2024-05-27T09:47:02.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.633+0000 7fb99d56a740 1 -- 172.21.15.44:0/2433001740 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 msgr2=0x55ae0b5a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:02.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.633+0000 7fb99d56a740 1 --2- 172.21.15.44:0/2433001740 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5a4da0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb980009d60 tx=0x7fb9800117b0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.633+0000 7fb99d56a740 1 -- 172.21.15.44:0/2433001740 shutdown_connections 2024-05-27T09:47:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.633+0000 7fb99d56a740 1 --2- 172.21.15.44:0/2433001740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.633+0000 7fb99d56a740 1 --2- 172.21.15.44:0/2433001740 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5a4da0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.633+0000 7fb99d56a740 1 --2- 172.21.15.44:0/2433001740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.633+0000 7fb99d56a740 1 -- 172.21.15.44:0/2433001740 >> 172.21.15.44:0/2433001740 conn(0x55ae0b1f6300 msgr2=0x55ae0b5a1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.633+0000 7fb99d56a740 1 -- 172.21.15.44:0/2433001740 shutdown_connections 2024-05-27T09:47:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.633+0000 7fb99d56a740 1 -- 172.21.15.44:0/2433001740 wait complete. 2024-05-27T09:47:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.634+0000 7fb99d56a740 1 -- start start 2024-05-27T09:47:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.634+0000 7fb99d56a740 1 Processor -- start 2024-05-27T09:47:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.635+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.635+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.635+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b2021b0 0x55ae0b5bacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.635+0000 7fb98ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.635+0000 7fb98ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5af200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54350/0 (socket says 172.21.15.44:54350) 2024-05-27T09:47:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.635+0000 7fb98f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.635+0000 7fb98f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5aecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47944/0 (socket says 172.21.15.44:47944) 2024-05-27T09:47:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.635+0000 7fb98ea00640 1 -- 172.21.15.44:0/3503308565 learned_addr learned my addr 172.21.15.44:0/3503308565 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.635+0000 7fb98fe00640 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b2021b0 0x55ae0b5bacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.635+0000 7fb99d56a740 1 -- 172.21.15.44:0/3503308565 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b1eb790 con 0x55ae0b1e14d0 2024-05-27T09:47:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.635+0000 7fb99d56a740 1 -- 172.21.15.44:0/3503308565 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b4adbc0 con 0x55ae0b2021b0 2024-05-27T09:47:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.636+0000 7fb99d56a740 1 -- 172.21.15.44:0/3503308565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ae0b5b1990 con 0x55ae0b1d92f0 2024-05-27T09:47:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.636+0000 7fb99d56a740 10 client.0 _ll_register_callbacks cb 0x55ae0b48b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.636+0000 7fb98fe00640 1 -- 172.21.15.44:0/3503308565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 msgr2=0x55ae0b5aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.636+0000 7fb98fe00640 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.636+0000 7fb98fe00640 1 -- 172.21.15.44:0/3503308565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 msgr2=0x55ae0b5af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.636+0000 7fb98fe00640 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.636+0000 7fb98fe00640 1 -- 172.21.15.44:0/3503308565 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae0b21a210 con 0x55ae0b2021b0 2024-05-27T09:47:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.636+0000 7fb98fe00640 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b2021b0 0x55ae0b5bacc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb990110e80 tx=0x7fb988004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec5efb886e2f538d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.636+0000 7fb979600640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:47:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.637+0000 7fb979600640 1 -- 172.21.15.44:0/3503308565 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb988018070 con 0x55ae0b2021b0 2024-05-27T09:47:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.637+0000 7fb99d56a740 1 -- 172.21.15.44:0/3503308565 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb980012aa0 con 0x55ae0b2021b0 2024-05-27T09:47:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.637+0000 7fb979600640 1 -- 172.21.15.44:0/3503308565 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fb9880109b0 con 0x55ae0b2021b0 2024-05-27T09:47:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.637+0000 7fb979600640 1 -- 172.21.15.44:0/3503308565 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb988010bb0 con 0x55ae0b2021b0 2024-05-27T09:47:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.637+0000 7fb99d56a740 10 client.27425 fetch_fsmap learned FSMap version 34 2024-05-27T09:47:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.637+0000 7fb99d56a740 1 -- 172.21.15.44:0/3503308565 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ae0b2e6620 con 0x55ae0b2021b0 2024-05-27T09:47:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.638+0000 7fb979600640 1 -- 172.21.15.44:0/3503308565 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb980012aa0 con 0x55ae0b2021b0 2024-05-27T09:47:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.638+0000 7fb979600640 1 -- 172.21.15.44:0/3503308565 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fb988021c10 con 0x55ae0b2021b0 2024-05-27T09:47:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.638+0000 7fb99d56a740 10 client.27425 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:47:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.638+0000 7fb99d56a740 10 client.27425 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.638+0000 7fb99d56a740 20 client.27425 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.638+0000 7fb99d56a740 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae0b1c0d80 0x55ae0b5c0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.638+0000 7fb99d56a740 4 client.27425 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.638+0000 7fb99d56a740 1 -- 172.21.15.44:0/3503308565 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae0b517ae0 con 0x55ae0b1c0d80 2024-05-27T09:47:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.638+0000 7fb98f400640 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae0b1c0d80 0x55ae0b5c0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.639+0000 7fb98f400640 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae0b1c0d80 0x55ae0b5c0d70 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba0747347f8d3eaf server_cookie=7b7f2d2146dcf71b in_seq=0 out_seq=0 2024-05-27T09:47:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.639+0000 7fb979600640 10 client.27425 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb979600640 1 -- 172.21.15.44:0/3503308565 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ae0b517ae0 con 0x55ae0b1c0d80 2024-05-27T09:47:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb979600640 10 client.27425 handle_command_reply: tid=0 2024-05-27T09:47:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb99d56a740 1 client.27425 shutdown 2024-05-27T09:47:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb99d56a740 1 -- 172.21.15.44:0/3503308565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b2021b0 msgr2=0x55ae0b5bacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb99d56a740 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b2021b0 0x55ae0b5bacc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb990110e80 tx=0x7fb988004ab0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb99d56a740 1 -- 172.21.15.44:0/3503308565 shutdown_connections 2024-05-27T09:47:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb99d56a740 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae0b1c0d80 0x55ae0b5c0d70 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb99d56a740 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b2021b0 0x55ae0b5bacc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb99d56a740 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb99d56a740 1 --2- 172.21.15.44:0/3503308565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb979600640 0 client.27425 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb99d56a740 1 -- 172.21.15.44:0/3503308565 >> 172.21.15.44:0/3503308565 conn(0x55ae0b1f6300 msgr2=0x55ae0b5a6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:02.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb99d56a740 1 -- 172.21.15.44:0/3503308565 shutdown_connections 2024-05-27T09:47:02.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.640+0000 7fb99d56a740 1 -- 172.21.15.44:0/3503308565 wait complete. 2024-05-27T09:47:02.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.641+0000 7fb99d56a740 20 client.27425 trim_cache size 0 max 16384 2024-05-27T09:47:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.657+0000 7fb99d56a740 1 Processor -- start 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.657+0000 7fb99d56a740 1 -- start start 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.658+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.658+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.658+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.658+0000 7fb99d56a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb9901b8aa0 con 0x55ae0b1e14d0 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.658+0000 7fb99d56a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb9901b6310 con 0x55ae0b2021b0 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.658+0000 7fb99d56a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9901b4180 con 0x55ae0b1d92f0 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.658+0000 7fb98f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.658+0000 7fb98fe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.658+0000 7fb98f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5af6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47946/0 (socket says 172.21.15.44:47946) 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.658+0000 7fb98ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.658+0000 7fb98f400640 1 -- 172.21.15.44:0/965060445 learned_addr learned my addr 172.21.15.44:0/965060445 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.659+0000 7fb98f400640 1 -- 172.21.15.44:0/965060445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b2021b0 msgr2=0x55ae0b5a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.659+0000 7fb98f400640 1 --2- 172.21.15.44:0/965060445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.659+0000 7fb98f400640 1 -- 172.21.15.44:0/965060445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 msgr2=0x55ae0b5a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.659+0000 7fb98f400640 1 --2- 172.21.15.44:0/965060445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.659+0000 7fb98f400640 1 -- 172.21.15.44:0/965060445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae0b5a1c00 con 0x55ae0b1d92f0 2024-05-27T09:47:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.659+0000 7fb98f400640 1 --2- 172.21.15.44:0/965060445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5af6f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb98000f6f0 tx=0x7fb98000f720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca644f00d56f77bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.659+0000 7fb97be00640 1 -- 172.21.15.44:0/965060445 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb980005040 con 0x55ae0b1d92f0 2024-05-27T09:47:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.659+0000 7fb97be00640 1 -- 172.21.15.44:0/965060445 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb980005690 con 0x55ae0b1d92f0 2024-05-27T09:47:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.659+0000 7fb97be00640 1 -- 172.21.15.44:0/965060445 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb980004680 con 0x55ae0b1d92f0 2024-05-27T09:47:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.660+0000 7fb99d56a740 1 -- 172.21.15.44:0/965060445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 msgr2=0x55ae0b5af6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.660+0000 7fb99d56a740 1 --2- 172.21.15.44:0/965060445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5af6f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb98000f6f0 tx=0x7fb98000f720 comp rx=0 tx=0).stop 2024-05-27T09:47:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.660+0000 7fb99d56a740 1 -- 172.21.15.44:0/965060445 shutdown_connections 2024-05-27T09:47:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.660+0000 7fb99d56a740 1 --2- 172.21.15.44:0/965060445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.660+0000 7fb99d56a740 1 --2- 172.21.15.44:0/965060445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.660+0000 7fb99d56a740 1 --2- 172.21.15.44:0/965060445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5af6f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.660+0000 7fb99d56a740 1 -- 172.21.15.44:0/965060445 >> 172.21.15.44:0/965060445 conn(0x55ae0b1f6300 msgr2=0x55ae0b5a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.661+0000 7fb99d56a740 1 -- 172.21.15.44:0/965060445 shutdown_connections 2024-05-27T09:47:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.661+0000 7fb99d56a740 1 -- 172.21.15.44:0/965060445 wait complete. 2024-05-27T09:47:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.661+0000 7fb99d56a740 1 -- start start 2024-05-27T09:47:02.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.661+0000 7fb99d56a740 1 Processor -- start 2024-05-27T09:47:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb99d56a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb99d56a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb9901b47d0 con 0x55ae0b2021b0 2024-05-27T09:47:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb99d56a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb9901b5c40 con 0x55ae0b1d92f0 2024-05-27T09:47:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb99d56a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9901b4ea0 con 0x55ae0b1e14d0 2024-05-27T09:47:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb99d56a740 10 client.0 _ll_register_callbacks cb 0x7fb988009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb98ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb98ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47962/0 (socket says 172.21.15.44:47962) 2024-05-27T09:47:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb98ea00640 1 -- 172.21.15.44:0/2775482162 learned_addr learned my addr 172.21.15.44:0/2775482162 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb98f400640 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.662+0000 7fb98fe00640 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb98ea00640 1 -- 172.21.15.44:0/2775482162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 msgr2=0x55ae0b5a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb98ea00640 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb98ea00640 1 -- 172.21.15.44:0/2775482162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b2021b0 msgr2=0x55ae0b5a9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb98ea00640 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb98ea00640 1 -- 172.21.15.44:0/2775482162 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae0b5b5e20 con 0x55ae0b1e14d0 2024-05-27T09:47:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb98f400640 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:47:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb98fe00640 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:47:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb98ea00640 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a8fe0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0c9730 tx=0x7fb97c0c9630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b29a979abc46e833 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb979600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb979600640 1 -- 172.21.15.44:0/2775482162 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c049070 con 0x55ae0b1e14d0 2024-05-27T09:47:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb99d56a740 1 -- 172.21.15.44:0/2775482162 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ae0b5112c0 con 0x55ae0b1e14d0 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.663+0000 7fb979600640 1 -- 172.21.15.44:0/2775482162 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fb97c0c9a00 con 0x55ae0b1e14d0 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.664+0000 7fb979600640 1 -- 172.21.15.44:0/2775482162 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb97c1000a0 con 0x55ae0b1e14d0 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.664+0000 7fb99d56a740 10 client.27418 fetch_fsmap learned FSMap version 34 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.664+0000 7fb99d56a740 1 -- 172.21.15.44:0/2775482162 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ae0b5b70c0 con 0x55ae0b1e14d0 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.664+0000 7fb979600640 1 -- 172.21.15.44:0/2775482162 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ae0b5112c0 con 0x55ae0b1e14d0 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.664+0000 7fb979600640 1 -- 172.21.15.44:0/2775482162 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fb97c04c060 con 0x55ae0b1e14d0 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.664+0000 7fb99d56a740 10 client.27418 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.664+0000 7fb99d56a740 10 client.27418 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.664+0000 7fb99d56a740 20 client.27418 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.664+0000 7fb99d56a740 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae0b1961c0 0x55ae0b62b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.664+0000 7fb99d56a740 4 client.27418 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:47:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.664+0000 7fb99d56a740 1 -- 172.21.15.44:0/2775482162 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ae0b48b4b0 con 0x55ae0b1961c0 2024-05-27T09:47:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.665+0000 7fb98f400640 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae0b1961c0 0x55ae0b62b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.665+0000 7fb98f400640 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae0b1961c0 0x55ae0b62b480 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19fad5a4ddd58cff server_cookie=832e0bf1fb6e9c0e in_seq=0 out_seq=0 2024-05-27T09:47:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:02.665+0000 7fb979600640 10 client.27418 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:04.287 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:04 smithi044 ceph-mon[31618]: pgmap v369: 97 pgs: 97 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 577 op/s 2024-05-27T09:47:04.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:04 smithi189 ceph-mon[36671]: pgmap v369: 97 pgs: 97 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 577 op/s 2024-05-27T09:47:04.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:04 smithi050 ceph-mon[36371]: pgmap v369: 97 pgs: 97 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 577 op/s 2024-05-27T09:47:05.058 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.057+0000 7fb979600640 1 -- 172.21.15.44:0/2775482162 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55ae0b48b4b0 con 0x55ae0b1961c0 2024-05-27T09:47:05.058 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.057+0000 7fb979600640 10 client.27418 handle_command_reply: tid=0 2024-05-27T09:47:05.058 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.057+0000 7fb99d56a740 1 client.27418 shutdown 2024-05-27T09:47:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.058+0000 7fb99d56a740 1 -- 172.21.15.44:0/2775482162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1e14d0 msgr2=0x55ae0b5a8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.058+0000 7fb99d56a740 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a8fe0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0c9730 tx=0x7fb97c0c9630 comp rx=0 tx=0).stop 2024-05-27T09:47:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.058+0000 7fb99d56a740 1 -- 172.21.15.44:0/2775482162 shutdown_connections 2024-05-27T09:47:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.058+0000 7fb99d56a740 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ae0b1961c0 0x55ae0b62b480 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.058+0000 7fb99d56a740 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ae0b2021b0 0x55ae0b5a9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.058+0000 7fb99d56a740 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae0b1e14d0 0x55ae0b5a8fe0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.058+0000 7fb99d56a740 1 --2- 172.21.15.44:0/2775482162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ae0b1d92f0 0x55ae0b5a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.058+0000 7fb99d56a740 1 -- 172.21.15.44:0/2775482162 >> 172.21.15.44:0/2775482162 conn(0x55ae0b1f6300 msgr2=0x55ae0b5a5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.058+0000 7fb979600640 0 client.27418 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.058+0000 7fb99d56a740 1 -- 172.21.15.44:0/2775482162 shutdown_connections 2024-05-27T09:47:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.059+0000 7fb99d56a740 1 -- 172.21.15.44:0/2775482162 wait complete. 2024-05-27T09:47:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.059+0000 7fb99d56a740 20 client.27418 trim_cache size 0 max 16384 2024-05-27T09:47:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:47:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:47:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:47:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.061+0000 7fb996600640 1 -- 172.21.15.44:0/205631221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb9740b2c00 msgr2=0x7fb9740b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.061+0000 7fb996600640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb9740b2c00 0x7fb9740b5110 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55ae0b5a19d0 tx=0x7fb988004a40 comp rx=0 tx=0).stop 2024-05-27T09:47:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.061+0000 7fb996600640 1 -- 172.21.15.44:0/205631221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99010dc40 msgr2=0x7fb99010b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.061+0000 7fb996600640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99010dc40 0x7fb99010b0d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb97c02a930 tx=0x7fb97c003c10 comp rx=0 tx=0).stop 2024-05-27T09:47:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.061+0000 7fb996600640 1 -- 172.21.15.44:0/205631221 shutdown_connections 2024-05-27T09:47:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.061+0000 7fb996600640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9901105a0 0x7fb99010b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.061+0000 7fb996600640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb9740b2c00 0x7fb9740b5110 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.061+0000 7fb996600640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99010dc40 0x7fb99010b0d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.061+0000 7fb996600640 1 --2- 172.21.15.44:0/205631221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb990102a00 0x7fb99010ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.061+0000 7fb996600640 1 -- 172.21.15.44:0/205631221 >> 172.21.15.44:0/205631221 conn(0x7fb9900782d0 msgr2=0x7fb9901012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.061+0000 7fb996600640 1 -- 172.21.15.44:0/205631221 shutdown_connections 2024-05-27T09:47:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:05.062+0000 7fb996600640 1 -- 172.21.15.44:0/205631221 wait complete. 2024-05-27T09:47:05.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.283+0000 7fef34c00700 1 -- 172.21.15.44:0/2638499192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef301045e0 msgr2=0x7fef301049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.283+0000 7fef34c00700 1 --2- 172.21.15.44:0/2638499192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef301045e0 0x7fef301049f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fef200099c0 tx=0x7fef20009cd0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.283+0000 7fef34c00700 1 -- 172.21.15.44:0/2638499192 shutdown_connections 2024-05-27T09:47:05.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.283+0000 7fef34c00700 1 --2- 172.21.15.44:0/2638499192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef30109860 0x7fef301111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.283+0000 7fef34c00700 1 --2- 172.21.15.44:0/2638499192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef301045e0 0x7fef301049f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.283+0000 7fef34c00700 1 --2- 172.21.15.44:0/2638499192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef30108f10 0x7fef30109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.283+0000 7fef34c00700 1 -- 172.21.15.44:0/2638499192 >> 172.21.15.44:0/2638499192 conn(0x7fef30100030 msgr2=0x7fef30102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:05.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.283+0000 7fef34c00700 1 -- 172.21.15.44:0/2638499192 shutdown_connections 2024-05-27T09:47:05.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.283+0000 7fef34c00700 1 -- 172.21.15.44:0/2638499192 wait complete. 2024-05-27T09:47:05.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.285+0000 7fef34c00700 1 Processor -- start 2024-05-27T09:47:05.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.285+0000 7fef34c00700 1 -- start start 2024-05-27T09:47:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.286+0000 7fef34c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef301045e0 0x7fef3019ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.286+0000 7fef34c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30108f10 0x7fef3019b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.286+0000 7fef34c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef30109860 0x7fef3019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.286+0000 7fef34c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fef30114500 con 0x7fef301045e0 2024-05-27T09:47:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.286+0000 7fef34c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fef30114380 con 0x7fef30109860 2024-05-27T09:47:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.286+0000 7fef34c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fef30114680 con 0x7fef30108f10 2024-05-27T09:47:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.286+0000 7fef2d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30108f10 0x7fef3019b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.286+0000 7fef2de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef301045e0 0x7fef3019ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.286+0000 7fef2d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30108f10 0x7fef3019b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59520/0 (socket says 172.21.15.44:59520) 2024-05-27T09:47:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.286+0000 7fef2d400700 1 -- 172.21.15.44:0/3802841982 learned_addr learned my addr 172.21.15.44:0/3802841982 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.286+0000 7fef2e800700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef30109860 0x7fef3019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.287+0000 7fef2d400700 1 -- 172.21.15.44:0/3802841982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef30109860 msgr2=0x7fef3019f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.287+0000 7fef2d400700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef30109860 0x7fef3019f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.287+0000 7fef2d400700 1 -- 172.21.15.44:0/3802841982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef301045e0 msgr2=0x7fef3019ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.287+0000 7fef2d400700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef301045e0 0x7fef3019ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.287+0000 7fef2d400700 1 -- 172.21.15.44:0/3802841982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3019fd20 con 0x7fef30108f10 2024-05-27T09:47:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.287+0000 7fef2e800700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef30109860 0x7fef3019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.287+0000 7fef2d400700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30108f10 0x7fef3019b3f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00e9f0 tx=0x7fef1c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.287+0000 7fef26a00700 1 -- 172.21.15.44:0/3802841982 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c004a30 con 0x7fef30108f10 2024-05-27T09:47:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.287+0000 7fef26a00700 1 -- 172.21.15.44:0/3802841982 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef1c00cc70 con 0x7fef30108f10 2024-05-27T09:47:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.287+0000 7fef26a00700 1 -- 172.21.15.44:0/3802841982 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c005530 con 0x7fef30108f10 2024-05-27T09:47:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.287+0000 7fef34c00700 1 -- 172.21.15.44:0/3802841982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef301a0010 con 0x7fef30108f10 2024-05-27T09:47:05.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.288+0000 7fef34c00700 1 -- 172.21.15.44:0/3802841982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef301a7930 con 0x7fef30108f10 2024-05-27T09:47:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.292+0000 7fef26a00700 1 -- 172.21.15.44:0/3802841982 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fef1c004bd0 con 0x7fef30108f10 2024-05-27T09:47:05.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.292+0000 7fef437adb80 1 Processor -- start 2024-05-27T09:47:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.292+0000 7fef437adb80 1 -- start start 2024-05-27T09:47:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef437adb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef437adb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151d2140 0x55cf151cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef437adb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf151cc1c0 0x55cf151cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef437adb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cf150f5e40 con 0x55cf151cc1c0 2024-05-27T09:47:05.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef437adb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cf150e59e0 con 0x55cf151d2140 2024-05-27T09:47:05.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef437adb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cf150e9a20 con 0x55cf15048e40 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef2e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef2e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151d1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59536/0 (socket says 172.21.15.44:59536) 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef2e800700 1 -- 172.21.15.44:0/2754441798 learned_addr learned my addr 172.21.15.44:0/2754441798 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef2de00700 1 --2- 172.21.15.44:0/2754441798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151d2140 0x55cf151cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef2d400700 1 --2- 172.21.15.44:0/2754441798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf151cc1c0 0x55cf151cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef2e800700 1 -- 172.21.15.44:0/2754441798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151d2140 msgr2=0x55cf151cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef2e800700 1 --2- 172.21.15.44:0/2754441798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151d2140 0x55cf151cbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef2e800700 1 -- 172.21.15.44:0/2754441798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf151cc1c0 msgr2=0x55cf151cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef2e800700 1 --2- 172.21.15.44:0/2754441798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf151cc1c0 0x55cf151cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.293+0000 7fef2e800700 1 -- 172.21.15.44:0/2754441798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf150ae420 con 0x55cf15048e40 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.294+0000 7fef26a00700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fef140b2a70 0x7fef140b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.294+0000 7fef26a00700 1 -- 172.21.15.44:0/3802841982 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fef1c0e27f0 con 0x7fef30108f10 2024-05-27T09:47:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.294+0000 7fef2e800700 1 --2- 172.21.15.44:0/2754441798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151d1c00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fef28003eb0 tx=0x7fef28009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25eeb7559ed2696c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.294+0000 7fef2de00700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fef140b2a70 0x7fef140b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.295+0000 7fef1b400700 1 -- 172.21.15.44:0/2754441798 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28019020 con 0x55cf15048e40 2024-05-27T09:47:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.295+0000 7fef1b400700 1 -- 172.21.15.44:0/2754441798 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef280092d0 con 0x55cf15048e40 2024-05-27T09:47:05.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.295+0000 7fef1b400700 1 -- 172.21.15.44:0/2754441798 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2800e870 con 0x55cf15048e40 2024-05-27T09:47:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.296+0000 7fef437adb80 1 -- 172.21.15.44:0/2754441798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 msgr2=0x55cf151d1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.296+0000 7fef2de00700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fef140b2a70 0x7fef140b4f80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fef20003760 tx=0x7fef2001a040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.296+0000 7fef437adb80 1 --2- 172.21.15.44:0/2754441798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151d1c00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fef28003eb0 tx=0x7fef28009920 comp rx=0 tx=0).stop 2024-05-27T09:47:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.296+0000 7fef437adb80 1 -- 172.21.15.44:0/2754441798 shutdown_connections 2024-05-27T09:47:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.296+0000 7fef437adb80 1 --2- 172.21.15.44:0/2754441798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf151cc1c0 0x55cf151cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.296+0000 7fef437adb80 1 --2- 172.21.15.44:0/2754441798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151d1c00 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.296+0000 7fef437adb80 1 --2- 172.21.15.44:0/2754441798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151d2140 0x55cf151cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.296+0000 7fef437adb80 1 -- 172.21.15.44:0/2754441798 >> 172.21.15.44:0/2754441798 conn(0x55cf150a8fc0 msgr2=0x55cf151c28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.296+0000 7fef437adb80 1 -- 172.21.15.44:0/2754441798 shutdown_connections 2024-05-27T09:47:05.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.296+0000 7fef437adb80 1 -- 172.21.15.44:0/2754441798 wait complete. 2024-05-27T09:47:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.297+0000 7fef437adb80 1 -- start start 2024-05-27T09:47:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.297+0000 7fef437adb80 1 Processor -- start 2024-05-27T09:47:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef437adb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf15048e40 0x55cf151d5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef437adb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf151d5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef437adb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf151d2140 0x55cf151d63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef437adb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cf14febfa0 con 0x55cf15048e40 2024-05-27T09:47:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef437adb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cf15086760 con 0x55cf151cc1c0 2024-05-27T09:47:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef437adb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cf151d8c10 con 0x55cf151d2140 2024-05-27T09:47:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef437adb80 10 client.0 _ll_register_callbacks cb 0x55cf1508fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef2de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf15048e40 0x55cf151d5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef2e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf151d2140 0x55cf151d63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef2e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf151d2140 0x55cf151d63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59540/0 (socket says 172.21.15.44:59540) 2024-05-27T09:47:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef2d400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf151d5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.298+0000 7fef2e800700 1 -- 172.21.15.44:0/3675686051 learned_addr learned my addr 172.21.15.44:0/3675686051 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.299+0000 7fef2e800700 1 -- 172.21.15.44:0/3675686051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 msgr2=0x55cf151d5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.299+0000 7fef2e800700 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf151d5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.299+0000 7fef2e800700 1 -- 172.21.15.44:0/3675686051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf15048e40 msgr2=0x55cf151d5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.299+0000 7fef2e800700 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf15048e40 0x55cf151d5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.299+0000 7fef2e800700 1 -- 172.21.15.44:0/3675686051 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf150358d0 con 0x55cf151d2140 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.299+0000 7fef2d400700 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf151d5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.299+0000 7fef2e800700 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf151d2140 0x55cf151d63d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fef28000f80 tx=0x7fef28007c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9d50d5e3d7004a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.299+0000 7fef0fe00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.299+0000 7fef0fe00700 1 -- 172.21.15.44:0/3675686051 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28019030 con 0x55cf151d2140 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.299+0000 7fef0fe00700 1 -- 172.21.15.44:0/3675686051 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fef28004ce0 con 0x55cf151d2140 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.299+0000 7fef437adb80 1 -- 172.21.15.44:0/3675686051 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef280092d0 con 0x55cf151d2140 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.300+0000 7fef0fe00700 1 -- 172.21.15.44:0/3675686051 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fef2800e4f0 con 0x55cf151d2140 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.300+0000 7fef437adb80 10 client.27430 fetch_fsmap learned FSMap version 34 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.300+0000 7fef437adb80 1 -- 172.21.15.44:0/3675686051 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cf150b6060 con 0x55cf151d2140 2024-05-27T09:47:05.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.300+0000 7fef0fe00700 1 -- 172.21.15.44:0/3675686051 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef280092d0 con 0x55cf151d2140 2024-05-27T09:47:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.300+0000 7fef0fe00700 1 -- 172.21.15.44:0/3675686051 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fef2802f5f0 con 0x55cf151d2140 2024-05-27T09:47:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.300+0000 7fef437adb80 10 client.27430 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:47:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.300+0000 7fef437adb80 10 client.27430 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.300+0000 7fef437adb80 20 client.27430 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.300+0000 7fef437adb80 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf151e5a40 0x55cf151e7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.300+0000 7fef437adb80 4 client.27430 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.300+0000 7fef437adb80 1 -- 172.21.15.44:0/3675686051 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf150f7d90 con 0x55cf151e5a40 2024-05-27T09:47:05.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.301+0000 7fef2de00700 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf151e5a40 0x55cf151e7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.301+0000 7fef2de00700 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf151e5a40 0x55cf151e7eb0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6a75a1f4f81a8b0 server_cookie=61ae5b08f4fd5665 in_seq=0 out_seq=0 2024-05-27T09:47:05.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.302+0000 7fef0fe00700 10 client.27430 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.302+0000 7fef0fe00700 1 -- 172.21.15.44:0/3675686051 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cf150f7d90 con 0x55cf151e5a40 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.302+0000 7fef0fe00700 10 client.27430 handle_command_reply: tid=0 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.302+0000 7fef437adb80 1 client.27430 shutdown 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.302+0000 7fef437adb80 1 -- 172.21.15.44:0/3675686051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf151d2140 msgr2=0x55cf151d63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.302+0000 7fef437adb80 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf151d2140 0x55cf151d63d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fef28000f80 tx=0x7fef28007c90 comp rx=0 tx=0).stop 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.303+0000 7fef437adb80 1 -- 172.21.15.44:0/3675686051 shutdown_connections 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.303+0000 7fef437adb80 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf151e5a40 0x55cf151e7eb0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.303+0000 7fef437adb80 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf151d2140 0x55cf151d63d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.303+0000 7fef437adb80 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf15048e40 0x55cf151d5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.303+0000 7fef0fe00700 0 client.27430 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.303+0000 7fef437adb80 1 --2- 172.21.15.44:0/3675686051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf151d5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.303+0000 7fef437adb80 1 -- 172.21.15.44:0/3675686051 >> 172.21.15.44:0/3675686051 conn(0x55cf150a8fc0 msgr2=0x55cf151c2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:05.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.303+0000 7fef437adb80 1 -- 172.21.15.44:0/3675686051 shutdown_connections 2024-05-27T09:47:05.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.303+0000 7fef437adb80 1 -- 172.21.15.44:0/3675686051 wait complete. 2024-05-27T09:47:05.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.303+0000 7fef437adb80 20 client.27430 trim_cache size 0 max 16384 2024-05-27T09:47:05.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.327+0000 7fef437adb80 1 Processor -- start 2024-05-27T09:47:05.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.327+0000 7fef437adb80 1 -- start start 2024-05-27T09:47:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.328+0000 7fef437adb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf150a8fc0 0x55cf151cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.328+0000 7fef437adb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151e1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.328+0000 7fef437adb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf151d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.328+0000 7fef437adb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fef301af620 con 0x55cf150a8fc0 2024-05-27T09:47:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.328+0000 7fef437adb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fef301ace50 con 0x55cf151cc1c0 2024-05-27T09:47:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.328+0000 7fef437adb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fef301aacc0 con 0x55cf15048e40 2024-05-27T09:47:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.328+0000 7fef2d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151e1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.328+0000 7fef2d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151e1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59550/0 (socket says 172.21.15.44:59550) 2024-05-27T09:47:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.328+0000 7fef2d400700 1 -- 172.21.15.44:0/1212054975 learned_addr learned my addr 172.21.15.44:0/1212054975 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:05.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.328+0000 7fef2e800700 1 --2- 172.21.15.44:0/1212054975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf151d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.328+0000 7fef2de00700 1 --2- 172.21.15.44:0/1212054975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf150a8fc0 0x55cf151cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.329+0000 7fef2d400700 1 -- 172.21.15.44:0/1212054975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 msgr2=0x55cf151d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.329+0000 7fef2d400700 1 --2- 172.21.15.44:0/1212054975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf151d7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.329+0000 7fef2d400700 1 -- 172.21.15.44:0/1212054975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf150a8fc0 msgr2=0x55cf151cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.329+0000 7fef2d400700 1 --2- 172.21.15.44:0/1212054975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf150a8fc0 0x55cf151cbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.329+0000 7fef2d400700 1 -- 172.21.15.44:0/1212054975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf151d6a10 con 0x55cf15048e40 2024-05-27T09:47:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.329+0000 7fef2d400700 1 --2- 172.21.15.44:0/1212054975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151e1eb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fef1c0a5af0 tx=0x7fef1c08ff90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=678ee09c0c3e7390 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.329+0000 7fef1aa00700 1 -- 172.21.15.44:0/1212054975 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c0e0370 con 0x55cf15048e40 2024-05-27T09:47:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.329+0000 7fef1aa00700 1 -- 172.21.15.44:0/1212054975 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef1c005730 con 0x55cf15048e40 2024-05-27T09:47:05.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.329+0000 7fef1aa00700 1 -- 172.21.15.44:0/1212054975 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c0a9a70 con 0x55cf15048e40 2024-05-27T09:47:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.330+0000 7fef437adb80 1 -- 172.21.15.44:0/1212054975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 msgr2=0x55cf151e1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.330+0000 7fef437adb80 1 --2- 172.21.15.44:0/1212054975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151e1eb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fef1c0a5af0 tx=0x7fef1c08ff90 comp rx=0 tx=0).stop 2024-05-27T09:47:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.330+0000 7fef437adb80 1 -- 172.21.15.44:0/1212054975 shutdown_connections 2024-05-27T09:47:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.330+0000 7fef437adb80 1 --2- 172.21.15.44:0/1212054975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf151d7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.330+0000 7fef437adb80 1 --2- 172.21.15.44:0/1212054975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf150a8fc0 0x55cf151cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.330+0000 7fef437adb80 1 --2- 172.21.15.44:0/1212054975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151e1eb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.330+0000 7fef437adb80 1 -- 172.21.15.44:0/1212054975 >> 172.21.15.44:0/1212054975 conn(0x55cf151d1390 msgr2=0x55cf151d2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.331+0000 7fef437adb80 1 -- 172.21.15.44:0/1212054975 shutdown_connections 2024-05-27T09:47:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.331+0000 7fef437adb80 1 -- 172.21.15.44:0/1212054975 wait complete. 2024-05-27T09:47:05.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.331+0000 7fef437adb80 1 -- start start 2024-05-27T09:47:05.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.331+0000 7fef437adb80 1 Processor -- start 2024-05-27T09:47:05.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.332+0000 7fef437adb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151eba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.332+0000 7fef437adb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf150a8fc0 0x55cf151ebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.332+0000 7fef437adb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf1524f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.332+0000 7fef437adb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fef301ab310 con 0x55cf150a8fc0 2024-05-27T09:47:05.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.332+0000 7fef437adb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fef301ac780 con 0x55cf151cc1c0 2024-05-27T09:47:05.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.332+0000 7fef437adb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fef301ab9e0 con 0x55cf15048e40 2024-05-27T09:47:05.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.332+0000 7fef437adb80 10 client.0 _ll_register_callbacks cb 0x55cf1508fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:05.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.332+0000 7fef2de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.332+0000 7fef2de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151eba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59556/0 (socket says 172.21.15.44:59556) 2024-05-27T09:47:05.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.332+0000 7fef2de00700 1 -- 172.21.15.44:0/1195915084 learned_addr learned my addr 172.21.15.44:0/1195915084 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:05.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.332+0000 7fef2d400700 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf150a8fc0 0x55cf151ebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef2de00700 1 -- 172.21.15.44:0/1195915084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 msgr2=0x55cf1524f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:05.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef2e800700 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf1524f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef2de00700 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf1524f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef2de00700 1 -- 172.21.15.44:0/1195915084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf150a8fc0 msgr2=0x55cf151ebfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:05.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef2de00700 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf150a8fc0 0x55cf151ebfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef2de00700 1 -- 172.21.15.44:0/1195915084 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf151ec8a0 con 0x55cf15048e40 2024-05-27T09:47:05.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef2e800700 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf1524f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:47:05.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef2de00700 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151eba80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fef200099c0 tx=0x7fef20005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2dd774ba81376b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:05.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef0fe00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:05.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef0fe00700 1 -- 172.21.15.44:0/1195915084 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef20004dc0 con 0x55cf15048e40 2024-05-27T09:47:05.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef0fe00700 1 -- 172.21.15.44:0/1195915084 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fef2000b790 con 0x55cf15048e40 2024-05-27T09:47:05.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef437adb80 1 -- 172.21.15.44:0/1195915084 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef1c005730 con 0x55cf15048e40 2024-05-27T09:47:05.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.333+0000 7fef0fe00700 1 -- 172.21.15.44:0/1195915084 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fef2000b990 con 0x55cf15048e40 2024-05-27T09:47:05.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.334+0000 7fef437adb80 10 client.27436 fetch_fsmap learned FSMap version 34 2024-05-27T09:47:05.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.334+0000 7fef437adb80 1 -- 172.21.15.44:0/1195915084 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cf152509b0 con 0x55cf15048e40 2024-05-27T09:47:05.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.334+0000 7fef0fe00700 1 -- 172.21.15.44:0/1195915084 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef1c005730 con 0x55cf15048e40 2024-05-27T09:47:05.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.334+0000 7fef0fe00700 1 -- 172.21.15.44:0/1195915084 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fef20024470 con 0x55cf15048e40 2024-05-27T09:47:05.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.334+0000 7fef437adb80 10 client.27436 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:47:05.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.334+0000 7fef437adb80 10 client.27436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:05.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.334+0000 7fef437adb80 20 client.27436 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:05.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.334+0000 7fef437adb80 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf151d2140 0x55cf152531e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.334+0000 7fef437adb80 4 client.27436 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.334+0000 7fef437adb80 1 -- 172.21.15.44:0/1195915084 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cf150f1570 con 0x55cf151d2140 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.335+0000 7fef2d400700 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf151d2140 0x55cf152531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.335+0000 7fef0fe00700 10 client.27436 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.335+0000 7fef2d400700 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf151d2140 0x55cf152531e0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34a41a9f3113e2b server_cookie=70a27fc6b65d3ea8 in_seq=0 out_seq=0 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.335+0000 7fef0fe00700 1 -- 172.21.15.44:0/1195915084 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cf150f1570 con 0x55cf151d2140 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.335+0000 7fef0fe00700 10 client.27436 handle_command_reply: tid=0 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.335+0000 7fef437adb80 1 client.27436 shutdown 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.335+0000 7fef437adb80 1 -- 172.21.15.44:0/1195915084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 msgr2=0x55cf151eba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.335+0000 7fef437adb80 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151eba80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fef200099c0 tx=0x7fef20005040 comp rx=0 tx=0).stop 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.336+0000 7fef437adb80 1 -- 172.21.15.44:0/1195915084 shutdown_connections 2024-05-27T09:47:05.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.336+0000 7fef437adb80 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cf151d2140 0x55cf152531e0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.336+0000 7fef437adb80 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf151cc1c0 0x55cf1524f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.336+0000 7fef437adb80 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf15048e40 0x55cf151eba80 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.336+0000 7fef437adb80 1 --2- 172.21.15.44:0/1195915084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cf150a8fc0 0x55cf151ebfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.336+0000 7fef437adb80 1 -- 172.21.15.44:0/1195915084 >> 172.21.15.44:0/1195915084 conn(0x55cf151d1390 msgr2=0x55cf151d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.336+0000 7fef0fe00700 0 client.27436 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.336+0000 7fef437adb80 1 -- 172.21.15.44:0/1195915084 shutdown_connections 2024-05-27T09:47:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.336+0000 7fef437adb80 1 -- 172.21.15.44:0/1195915084 wait complete. 2024-05-27T09:47:05.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.336+0000 7fef437adb80 20 client.27436 trim_cache size 0 max 16384 2024-05-27T09:47:05.339 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:47:05.339 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:47:05.340 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:47:05.340 INFO:teuthology.orchestra.run.smithi044.stdout: "5348fc89-9539-4fcf-9fdc-6e30fd88b5c6": { 2024-05-27T09:47:05.340 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:47:05.340 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "5348fc89-9539-4fcf-9fdc-6e30fd88b5c6", 2024-05-27T09:47:05.340 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:47:05.340 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:05.340 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:05.340 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:47:05.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 -- 172.21.15.44:0/3802841982 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fef140b2a70 msgr2=0x7fef140b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fef140b2a70 0x7fef140b4f80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fef20003760 tx=0x7fef2001a040 comp rx=0 tx=0).stop 2024-05-27T09:47:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 -- 172.21.15.44:0/3802841982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30108f10 msgr2=0x7fef3019b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30108f10 0x7fef3019b3f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00e9f0 tx=0x7fef1c00ed00 comp rx=0 tx=0).stop 2024-05-27T09:47:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 -- 172.21.15.44:0/3802841982 shutdown_connections 2024-05-27T09:47:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fef140b2a70 0x7fef140b4f80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef30109860 0x7fef3019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef301045e0 0x7fef3019ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 --2- 172.21.15.44:0/3802841982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30108f10 0x7fef3019b3f0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 -- 172.21.15.44:0/3802841982 >> 172.21.15.44:0/3802841982 conn(0x7fef30100030 msgr2=0x7fef30108180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 -- 172.21.15.44:0/3802841982 shutdown_connections 2024-05-27T09:47:05.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:05.337+0000 7fef34c00700 1 -- 172.21.15.44:0/3802841982 wait complete. 2024-05-27T09:47:06.184 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:06 smithi044 ceph-mon[31618]: pgmap v370: 97 pgs: 97 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 577 op/s 2024-05-27T09:47:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:06 smithi189 ceph-mon[36671]: pgmap v370: 97 pgs: 97 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 577 op/s 2024-05-27T09:47:06.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.534+0000 7fdc89a00700 1 -- 172.21.15.44:0/1288710422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc84108f10 msgr2=0x7fdc84109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:06.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.534+0000 7fdc89a00700 1 --2- 172.21.15.44:0/1288710422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc84108f10 0x7fdc84109320 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009920 tx=0x7fdc70009c30 comp rx=0 tx=0).stop 2024-05-27T09:47:06.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.534+0000 7fdc89a00700 1 -- 172.21.15.44:0/1288710422 shutdown_connections 2024-05-27T09:47:06.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.534+0000 7fdc89a00700 1 --2- 172.21.15.44:0/1288710422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc84109860 0x7fdc841111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.534+0000 7fdc89a00700 1 --2- 172.21.15.44:0/1288710422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc841045e0 0x7fdc841049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.534+0000 7fdc89a00700 1 --2- 172.21.15.44:0/1288710422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc84108f10 0x7fdc84109320 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.534+0000 7fdc89a00700 1 -- 172.21.15.44:0/1288710422 >> 172.21.15.44:0/1288710422 conn(0x7fdc84100030 msgr2=0x7fdc84102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:06.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.534+0000 7fdc89a00700 1 -- 172.21.15.44:0/1288710422 shutdown_connections 2024-05-27T09:47:06.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.535+0000 7fdc89a00700 1 -- 172.21.15.44:0/1288710422 wait complete. 2024-05-27T09:47:06.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.536+0000 7fdc89a00700 1 Processor -- start 2024-05-27T09:47:06.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.536+0000 7fdc89a00700 1 -- start start 2024-05-27T09:47:06.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc89a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc841045e0 0x7fdc8419af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc89a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc84108f10 0x7fdc8419b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc89a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc84109860 0x7fdc8419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc89a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841144c0 con 0x7fdc84108f10 2024-05-27T09:47:06.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc89a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdc84114340 con 0x7fdc84109860 2024-05-27T09:47:06.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc89a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdc84114640 con 0x7fdc841045e0 2024-05-27T09:47:06.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc841045e0 0x7fdc8419af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc841045e0 0x7fdc8419af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59584/0 (socket says 172.21.15.44:59584) 2024-05-27T09:47:06.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc82a00700 1 -- 172.21.15.44:0/2357454514 learned_addr learned my addr 172.21.15.44:0/2357454514 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:06.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc82000700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc84108f10 0x7fdc8419b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc83400700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc84109860 0x7fdc8419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc82a00700 1 -- 172.21.15.44:0/2357454514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc84109860 msgr2=0x7fdc8419f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:06.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc82a00700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc84109860 0x7fdc8419f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc82a00700 1 -- 172.21.15.44:0/2357454514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc84108f10 msgr2=0x7fdc8419b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:06.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc82a00700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc84108f10 0x7fdc8419b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.537+0000 7fdc82a00700 1 -- 172.21.15.44:0/2357454514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc8419fe60 con 0x7fdc841045e0 2024-05-27T09:47:06.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.538+0000 7fdc82a00700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc841045e0 0x7fdc8419af90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400e9c0 tx=0x7fdc7400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:06.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.538+0000 7fdc7b400700 1 -- 172.21.15.44:0/2357454514 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc74004a30 con 0x7fdc841045e0 2024-05-27T09:47:06.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.538+0000 7fdc7b400700 1 -- 172.21.15.44:0/2357454514 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc7400cc70 con 0x7fdc841045e0 2024-05-27T09:47:06.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.538+0000 7fdc7b400700 1 -- 172.21.15.44:0/2357454514 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc74005490 con 0x7fdc841045e0 2024-05-27T09:47:06.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.538+0000 7fdc89a00700 1 -- 172.21.15.44:0/2357454514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc841a0150 con 0x7fdc841045e0 2024-05-27T09:47:06.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.538+0000 7fdc89a00700 1 -- 172.21.15.44:0/2357454514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc841a7a70 con 0x7fdc841045e0 2024-05-27T09:47:06.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.543+0000 7fdc7b400700 1 -- 172.21.15.44:0/2357454514 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fdc74004bd0 con 0x7fdc841045e0 2024-05-27T09:47:06.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.543+0000 7fdc98518b80 1 Processor -- start 2024-05-27T09:47:06.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.543+0000 7fdc98518b80 1 -- start start 2024-05-27T09:47:06.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:06 smithi050 ceph-mon[36371]: pgmap v370: 97 pgs: 97 active+clean; 298 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 577 op/s 2024-05-27T09:47:06.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.544+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee3d1b80 0x55bfee46b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.544+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee2e5fd0 0x55bfee46bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.544+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee465670 0x55bfee465a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.544+0000 7fdc98518b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bfee392fd0 con 0x55bfee2e5fd0 2024-05-27T09:47:06.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.544+0000 7fdc98518b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bfee382b70 con 0x55bfee3d1b80 2024-05-27T09:47:06.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.544+0000 7fdc98518b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bfee386bb0 con 0x55bfee465670 2024-05-27T09:47:06.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.544+0000 7fdc83400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee3d1b80 0x55bfee46b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.544+0000 7fdc83400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee3d1b80 0x55bfee46b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60460/0 (socket says 172.21.15.44:60460) 2024-05-27T09:47:06.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.544+0000 7fdc83400700 1 -- 172.21.15.44:0/3498360436 learned_addr learned my addr 172.21.15.44:0/3498360436 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:06.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.544+0000 7fdc82a00700 1 --2- 172.21.15.44:0/3498360436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee2e5fd0 0x55bfee46bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.544+0000 7fdc82000700 1 --2- 172.21.15.44:0/3498360436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee465670 0x55bfee465a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.545+0000 7fdc82000700 1 -- 172.21.15.44:0/3498360436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee3d1b80 msgr2=0x55bfee46b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:06.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.545+0000 7fdc82000700 1 --2- 172.21.15.44:0/3498360436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee3d1b80 0x55bfee46b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.545+0000 7fdc82000700 1 -- 172.21.15.44:0/3498360436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee2e5fd0 msgr2=0x55bfee46bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:06.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.545+0000 7fdc82000700 1 --2- 172.21.15.44:0/3498360436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee2e5fd0 0x55bfee46bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.545+0000 7fdc82000700 1 -- 172.21.15.44:0/3498360436 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfee34b5b0 con 0x55bfee465670 2024-05-27T09:47:06.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.545+0000 7fdc83400700 1 --2- 172.21.15.44:0/3498360436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee3d1b80 0x55bfee46b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:06.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.545+0000 7fdc82000700 1 --2- 172.21.15.44:0/3498360436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee465670 0x55bfee465a80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdc70000c00 tx=0x7fdc7000bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce512caaad558ce7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:06.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.545+0000 7fdc7b400700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fdc680b29f0 0x7fdc680b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.545+0000 7fdc6fe00700 1 -- 172.21.15.44:0/3498360436 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7000fe40 con 0x55bfee465670 2024-05-27T09:47:06.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.546+0000 7fdc7b400700 1 -- 172.21.15.44:0/2357454514 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fdc740e26f0 con 0x7fdc841045e0 2024-05-27T09:47:06.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.546+0000 7fdc6fe00700 1 -- 172.21.15.44:0/3498360436 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc7000f910 con 0x55bfee465670 2024-05-27T09:47:06.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.546+0000 7fdc6fe00700 1 -- 172.21.15.44:0/3498360436 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc700038e0 con 0x55bfee465670 2024-05-27T09:47:06.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.546+0000 7fdc83400700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fdc680b29f0 0x7fdc680b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.547+0000 7fdc98518b80 1 -- 172.21.15.44:0/3498360436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee465670 msgr2=0x55bfee465a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:06.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.547+0000 7fdc98518b80 1 --2- 172.21.15.44:0/3498360436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee465670 0x55bfee465a80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdc70000c00 tx=0x7fdc7000bf60 comp rx=0 tx=0).stop 2024-05-27T09:47:06.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.547+0000 7fdc83400700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fdc680b29f0 0x7fdc680b4f00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c007610 tx=0x7fdc7c00b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:06.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.547+0000 7fdc98518b80 1 -- 172.21.15.44:0/3498360436 shutdown_connections 2024-05-27T09:47:06.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.547+0000 7fdc98518b80 1 --2- 172.21.15.44:0/3498360436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee465670 0x55bfee465a80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.547+0000 7fdc98518b80 1 --2- 172.21.15.44:0/3498360436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee3d1b80 0x55bfee46b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.547+0000 7fdc98518b80 1 --2- 172.21.15.44:0/3498360436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee2e5fd0 0x55bfee46bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.547+0000 7fdc98518b80 1 -- 172.21.15.44:0/3498360436 >> 172.21.15.44:0/3498360436 conn(0x55bfee37f520 msgr2=0x55bfee4606e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:06.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.547+0000 7fdc98518b80 1 -- 172.21.15.44:0/3498360436 shutdown_connections 2024-05-27T09:47:06.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.547+0000 7fdc98518b80 1 -- 172.21.15.44:0/3498360436 wait complete. 2024-05-27T09:47:06.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.548+0000 7fdc98518b80 1 -- start start 2024-05-27T09:47:06.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.548+0000 7fdc98518b80 1 Processor -- start 2024-05-27T09:47:06.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.548+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee2e5fd0 0x55bfee46f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.548+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee3d1b80 0x55bfee46f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.548+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee465670 0x55bfee46fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.548+0000 7fdc98518b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bfee289130 con 0x55bfee465670 2024-05-27T09:47:06.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.548+0000 7fdc98518b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bfee3238f0 con 0x55bfee2e5fd0 2024-05-27T09:47:06.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.548+0000 7fdc98518b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bfee466400 con 0x55bfee3d1b80 2024-05-27T09:47:06.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc98518b80 10 client.0 _ll_register_callbacks cb 0x55bfee32ce00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:06.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc82a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee2e5fd0 0x55bfee46f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc82a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee2e5fd0 0x55bfee46f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60462/0 (socket says 172.21.15.44:60462) 2024-05-27T09:47:06.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc82a00700 1 -- 172.21.15.44:0/4003993967 learned_addr learned my addr 172.21.15.44:0/4003993967 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:06.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc82000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee3d1b80 0x55bfee46f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc83400700 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee465670 0x55bfee46fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc82000700 1 -- 172.21.15.44:0/4003993967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee2e5fd0 msgr2=0x55bfee46f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:06.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc82000700 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee2e5fd0 0x55bfee46f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc82000700 1 -- 172.21.15.44:0/4003993967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee465670 msgr2=0x55bfee46fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:06.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc82000700 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee465670 0x55bfee46fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc82000700 1 -- 172.21.15.44:0/4003993967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfee2d2a60 con 0x55bfee3d1b80 2024-05-27T09:47:06.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.549+0000 7fdc82a00700 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee2e5fd0 0x55bfee46f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:47:06.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.550+0000 7fdc82000700 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee3d1b80 0x55bfee46f890 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009c70 tx=0x7fdc70005980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bac22a4fd7c9f7e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:06.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.550+0000 7fdc6cc00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:06.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.550+0000 7fdc6cc00700 1 -- 172.21.15.44:0/4003993967 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc70032070 con 0x55bfee3d1b80 2024-05-27T09:47:06.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.550+0000 7fdc98518b80 1 -- 172.21.15.44:0/4003993967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc7000f910 con 0x55bfee3d1b80 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.550+0000 7fdc6cc00700 1 -- 172.21.15.44:0/4003993967 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fdc7002b8c0 con 0x55bfee3d1b80 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.550+0000 7fdc6cc00700 1 -- 172.21.15.44:0/4003993967 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fdc7003e050 con 0x55bfee3d1b80 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.550+0000 7fdc98518b80 10 client.27448 fetch_fsmap learned FSMap version 34 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.550+0000 7fdc98518b80 1 -- 172.21.15.44:0/4003993967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55bfee3531f0 con 0x55bfee3d1b80 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.550+0000 7fdc6cc00700 1 -- 172.21.15.44:0/4003993967 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc7000f910 con 0x55bfee3d1b80 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.551+0000 7fdc6cc00700 1 -- 172.21.15.44:0/4003993967 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fdc70020070 con 0x55bfee3d1b80 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.551+0000 7fdc98518b80 10 client.27448 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.551+0000 7fdc98518b80 10 client.27448 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.551+0000 7fdc98518b80 20 client.27448 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.551+0000 7fdc98518b80 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bfee47eef0 0x55bfee481310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.551+0000 7fdc98518b80 4 client.27448 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.551+0000 7fdc98518b80 1 -- 172.21.15.44:0/4003993967 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfee394f20 con 0x55bfee47eef0 2024-05-27T09:47:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.551+0000 7fdc82a00700 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bfee47eef0 0x55bfee481310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.552+0000 7fdc82a00700 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bfee47eef0 0x55bfee481310 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66935171894450c5 server_cookie=3563dda30949342e in_seq=0 out_seq=0 2024-05-27T09:47:06.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.552+0000 7fdc6cc00700 10 client.27448 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:06.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc6cc00700 1 -- 172.21.15.44:0/4003993967 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bfee394f20 con 0x55bfee47eef0 2024-05-27T09:47:06.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc6cc00700 10 client.27448 handle_command_reply: tid=0 2024-05-27T09:47:06.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc98518b80 1 client.27448 shutdown 2024-05-27T09:47:06.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc98518b80 1 -- 172.21.15.44:0/4003993967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee3d1b80 msgr2=0x55bfee46f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:06.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc98518b80 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee3d1b80 0x55bfee46f890 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009c70 tx=0x7fdc70005980 comp rx=0 tx=0).stop 2024-05-27T09:47:06.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc98518b80 1 -- 172.21.15.44:0/4003993967 shutdown_connections 2024-05-27T09:47:06.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc98518b80 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bfee47eef0 0x55bfee481310 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc98518b80 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee465670 0x55bfee46fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc98518b80 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee2e5fd0 0x55bfee46f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc98518b80 1 --2- 172.21.15.44:0/4003993967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee3d1b80 0x55bfee46f890 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc98518b80 1 -- 172.21.15.44:0/4003993967 >> 172.21.15.44:0/4003993967 conn(0x55bfee37f520 msgr2=0x55bfee45fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:06.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.553+0000 7fdc6cc00700 0 client.27448 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:06.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.554+0000 7fdc98518b80 1 -- 172.21.15.44:0/4003993967 shutdown_connections 2024-05-27T09:47:06.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.554+0000 7fdc98518b80 1 -- 172.21.15.44:0/4003993967 wait complete. 2024-05-27T09:47:06.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.554+0000 7fdc98518b80 20 client.27448 trim_cache size 0 max 16384 2024-05-27T09:47:06.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.575+0000 7fdc98518b80 1 Processor -- start 2024-05-27T09:47:06.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.576+0000 7fdc98518b80 1 -- start start 2024-05-27T09:47:06.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.576+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee47b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.576+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee465670 0x55bfee464cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.576+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee46fad0 0x55bfee460040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.576+0000 7fdc98518b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841af680 con 0x55bfee465670 2024-05-27T09:47:06.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.576+0000 7fdc98518b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841aceb0 con 0x55bfee46fad0 2024-05-27T09:47:06.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.576+0000 7fdc98518b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841aad20 con 0x55bfee2e5fd0 2024-05-27T09:47:06.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.576+0000 7fdc82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee47b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.576+0000 7fdc82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee47b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59612/0 (socket says 172.21.15.44:59612) 2024-05-27T09:47:06.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.576+0000 7fdc82a00700 1 -- 172.21.15.44:0/505062979 learned_addr learned my addr 172.21.15.44:0/505062979 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:06.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.576+0000 7fdc82000700 1 --2- 172.21.15.44:0/505062979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee465670 0x55bfee464cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.577+0000 7fdc83400700 1 --2- 172.21.15.44:0/505062979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee46fad0 0x55bfee460040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.577+0000 7fdc82a00700 1 -- 172.21.15.44:0/505062979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee46fad0 msgr2=0x55bfee460040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:06.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.577+0000 7fdc82a00700 1 --2- 172.21.15.44:0/505062979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee46fad0 0x55bfee460040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.577+0000 7fdc82a00700 1 -- 172.21.15.44:0/505062979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee465670 msgr2=0x55bfee464cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:06.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.577+0000 7fdc82a00700 1 --2- 172.21.15.44:0/505062979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee465670 0x55bfee464cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.577+0000 7fdc82a00700 1 -- 172.21.15.44:0/505062979 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfee465f20 con 0x55bfee2e5fd0 2024-05-27T09:47:06.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.577+0000 7fdc82a00700 1 --2- 172.21.15.44:0/505062979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee47b1a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdc74000c00 tx=0x7fdc740e4bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a222ac189c5126 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:06.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.578+0000 7fdc6f400700 1 -- 172.21.15.44:0/505062979 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7400ce20 con 0x55bfee2e5fd0 2024-05-27T09:47:06.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.578+0000 7fdc6f400700 1 -- 172.21.15.44:0/505062979 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc740b2170 con 0x55bfee2e5fd0 2024-05-27T09:47:06.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.578+0000 7fdc6f400700 1 -- 172.21.15.44:0/505062979 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc740b2830 con 0x55bfee2e5fd0 2024-05-27T09:47:06.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.579+0000 7fdc98518b80 1 -- 172.21.15.44:0/505062979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 msgr2=0x55bfee47b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:06.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.579+0000 7fdc98518b80 1 --2- 172.21.15.44:0/505062979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee47b1a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdc74000c00 tx=0x7fdc740e4bd0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.579+0000 7fdc98518b80 1 -- 172.21.15.44:0/505062979 shutdown_connections 2024-05-27T09:47:06.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.579+0000 7fdc98518b80 1 --2- 172.21.15.44:0/505062979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee46fad0 0x55bfee460040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.579+0000 7fdc98518b80 1 --2- 172.21.15.44:0/505062979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee47b1a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.579+0000 7fdc98518b80 1 --2- 172.21.15.44:0/505062979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee465670 0x55bfee464cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.579+0000 7fdc98518b80 1 -- 172.21.15.44:0/505062979 >> 172.21.15.44:0/505062979 conn(0x55bfee37f520 msgr2=0x55bfee471380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:06.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.579+0000 7fdc98518b80 1 -- 172.21.15.44:0/505062979 shutdown_connections 2024-05-27T09:47:06.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.579+0000 7fdc98518b80 1 -- 172.21.15.44:0/505062979 wait complete. 2024-05-27T09:47:06.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.579+0000 7fdc98518b80 1 -- start start 2024-05-27T09:47:06.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.580+0000 7fdc98518b80 1 Processor -- start 2024-05-27T09:47:06.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.580+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee484350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.580+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee465670 0x55bfee4826e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.580+0000 7fdc98518b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee46fad0 0x55bfee482c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.580+0000 7fdc98518b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841ab370 con 0x55bfee46fad0 2024-05-27T09:47:06.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.580+0000 7fdc98518b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841ac7e0 con 0x55bfee465670 2024-05-27T09:47:06.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.580+0000 7fdc98518b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841aba40 con 0x55bfee2e5fd0 2024-05-27T09:47:06.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.580+0000 7fdc98518b80 10 client.0 _ll_register_callbacks cb 0x55bfee32ce00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.580+0000 7fdc82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee484350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.580+0000 7fdc82a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee484350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:59616/0 (socket says 172.21.15.44:59616) 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.580+0000 7fdc82a00700 1 -- 172.21.15.44:0/3049480642 learned_addr learned my addr 172.21.15.44:0/3049480642 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.581+0000 7fdc83400700 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee46fad0 0x55bfee482c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.581+0000 7fdc82000700 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee465670 0x55bfee4826e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.581+0000 7fdc82a00700 1 -- 172.21.15.44:0/3049480642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee465670 msgr2=0x55bfee4826e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.581+0000 7fdc82a00700 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee465670 0x55bfee4826e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.581+0000 7fdc82a00700 1 -- 172.21.15.44:0/3049480642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee46fad0 msgr2=0x55bfee482c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.581+0000 7fdc82a00700 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee46fad0 0x55bfee482c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.581+0000 7fdc82a00700 1 -- 172.21.15.44:0/3049480642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfee45c9d0 con 0x55bfee2e5fd0 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.581+0000 7fdc82a00700 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee484350 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fdc740a9960 tx=0x7fdc74005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c537fb6f27dbd5df server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.581+0000 7fdc6cc00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:06.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.581+0000 7fdc6cc00700 1 -- 172.21.15.44:0/3049480642 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc740e02d0 con 0x55bfee2e5fd0 2024-05-27T09:47:06.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.581+0000 7fdc6cc00700 1 -- 172.21.15.44:0/3049480642 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8521+0+0 (secure 0 0 0) 0x7fdc740a9c30 con 0x55bfee2e5fd0 2024-05-27T09:47:06.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc98518b80 1 -- 172.21.15.44:0/3049480642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bfee4588a0 con 0x55bfee2e5fd0 2024-05-27T09:47:06.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc6cc00700 1 -- 172.21.15.44:0/3049480642 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fdc7408d080 con 0x55bfee2e5fd0 2024-05-27T09:47:06.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc98518b80 10 client.27454 fetch_fsmap learned FSMap version 34 2024-05-27T09:47:06.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc98518b80 1 -- 172.21.15.44:0/3049480642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55bfee4ea190 con 0x55bfee2e5fd0 2024-05-27T09:47:06.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc6cc00700 1 -- 172.21.15.44:0/3049480642 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55bfee4588a0 con 0x55bfee2e5fd0 2024-05-27T09:47:06.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc6cc00700 1 -- 172.21.15.44:0/3049480642 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fdc740cc080 con 0x55bfee2e5fd0 2024-05-27T09:47:06.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc98518b80 10 client.27454 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:47:06.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc98518b80 10 client.27454 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:47:06.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc98518b80 20 client.27454 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:06.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc98518b80 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bfee4ea7d0 0x55bfee4ecbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:06.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc98518b80 4 client.27454 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.3, "await_for": 45.0}] 2024-05-27T09:47:06.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.582+0000 7fdc98518b80 1 -- 172.21.15.44:0/3049480642 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.3, "await_for": 45.0}) -- 0x55bfee38e700 con 0x55bfee4ea7d0 2024-05-27T09:47:06.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.583+0000 7fdc82000700 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bfee4ea7d0 0x55bfee4ecbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:06.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.583+0000 7fdc6cc00700 10 client.27454 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:06.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:06.583+0000 7fdc82000700 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bfee4ea7d0 0x55bfee4ecbf0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d699132c5d12bcfa server_cookie=f32493177fdb56d5 in_seq=0 out_seq=0 2024-05-27T09:47:06.813 INFO:tasks.cephfs.filesystem:scrub status for tag:5348fc89-9539-4fcf-9fdc-6e30fd88b5c6 - {'path': '/', 'tag': '5348fc89-9539-4fcf-9fdc-6e30fd88b5c6', 'options': 'recursive,force'} 2024-05-27T09:47:07.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:07 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:47:07.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:07 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:47:07.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:07 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:47:08.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:08 smithi189 ceph-mon[36671]: pgmap v371: 97 pgs: 97 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 1.07k op/s 2024-05-27T09:47:08.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:08 smithi189 ceph-mon[36671]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T09:47:08.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:08 smithi044 ceph-mon[31618]: pgmap v371: 97 pgs: 97 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 1.07k op/s 2024-05-27T09:47:08.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:08 smithi044 ceph-mon[31618]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T09:47:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:08 smithi050 ceph-mon[36371]: pgmap v371: 97 pgs: 97 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 1.07k op/s 2024-05-27T09:47:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:08 smithi050 ceph-mon[36371]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T09:47:09.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:09 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:47:09.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:09 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:47:09.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:09 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:47:09.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:09 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:47:09.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:09 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:47:09.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:09 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:47:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:09 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:47:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:09 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:47:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:09 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:47:10.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:10 smithi044 ceph-mon[31618]: pgmap v373: 97 pgs: 97 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 846 op/s 2024-05-27T09:47:10.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:10 smithi050 ceph-mon[36371]: pgmap v373: 97 pgs: 97 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 846 op/s 2024-05-27T09:47:10.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:10 smithi189 ceph-mon[36671]: pgmap v373: 97 pgs: 97 active+clean; 305 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 846 op/s 2024-05-27T09:47:12.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.306+0000 7fdc6cc00700 1 -- 172.21.15.44:0/3049480642 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55bfee38e700 con 0x55bfee4ea7d0 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.306+0000 7fdc6cc00700 10 client.27454 handle_command_reply: tid=0 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.306+0000 7fdc98518b80 1 client.27454 shutdown 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc98518b80 1 -- 172.21.15.44:0/3049480642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 msgr2=0x55bfee484350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc98518b80 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee484350 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fdc740a9960 tx=0x7fdc74005840 comp rx=0 tx=0).stop 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc98518b80 1 -- 172.21.15.44:0/3049480642 shutdown_connections 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc98518b80 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55bfee4ea7d0 0x55bfee4ecbf0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc98518b80 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bfee46fad0 0x55bfee482c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc98518b80 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfee2e5fd0 0x55bfee484350 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc98518b80 1 --2- 172.21.15.44:0/3049480642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bfee465670 0x55bfee4826e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc6cc00700 0 client.27454 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc98518b80 1 -- 172.21.15.44:0/3049480642 >> 172.21.15.44:0/3049480642 conn(0x55bfee37f520 msgr2=0x55bfee471380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc98518b80 1 -- 172.21.15.44:0/3049480642 shutdown_connections 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc98518b80 1 -- 172.21.15.44:0/3049480642 wait complete. 2024-05-27T09:47:12.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.307+0000 7fdc98518b80 20 client.27454 trim_cache size 0 max 16384 2024-05-27T09:47:12.310 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:47:12.310 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 34, 2024-05-27T09:47:12.310 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:47:12.310 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 27, 2024-05-27T09:47:12.310 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:47:12.310 INFO:teuthology.orchestra.run.smithi044.stdout: "b3fe0bae": { 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 27, 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 142.3, 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:47:12.311 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:47:12.312 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:12.312 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:12.312 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:12.312 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:12.312 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:12.312 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:47:12.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 -- 172.21.15.44:0/2357454514 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fdc680b29f0 msgr2=0x7fdc680b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:12.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fdc680b29f0 0x7fdc680b4f00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c007610 tx=0x7fdc7c00b040 comp rx=0 tx=0).stop 2024-05-27T09:47:12.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 -- 172.21.15.44:0/2357454514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc841045e0 msgr2=0x7fdc8419af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:12.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc841045e0 0x7fdc8419af90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400e9c0 tx=0x7fdc7400ecd0 comp rx=0 tx=0).stop 2024-05-27T09:47:12.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 -- 172.21.15.44:0/2357454514 shutdown_connections 2024-05-27T09:47:12.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fdc680b29f0 0x7fdc680b4f00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:12.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc84109860 0x7fdc8419f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:12.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc841045e0 0x7fdc8419af90 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:12.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 --2- 172.21.15.44:0/2357454514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc84108f10 0x7fdc8419b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:12.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 -- 172.21.15.44:0/2357454514 >> 172.21.15.44:0/2357454514 conn(0x7fdc84100030 msgr2=0x7fdc84102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:12.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 -- 172.21.15.44:0/2357454514 shutdown_connections 2024-05-27T09:47:12.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:12.309+0000 7fdc89a00700 1 -- 172.21.15.44:0/2357454514 wait complete. 2024-05-27T09:47:12.345 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:12 smithi050 ceph-mon[36371]: pgmap v374: 97 pgs: 97 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 738 op/s 2024-05-27T09:47:12.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:12 smithi044 ceph-mon[31618]: pgmap v374: 97 pgs: 97 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 738 op/s 2024-05-27T09:47:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:12 smithi189 ceph-mon[36671]: pgmap v374: 97 pgs: 97 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 738 op/s 2024-05-27T09:47:13.276 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b3fe0bae', quiesce took 14.9 seconds. Will release after: 7.4 2024-05-27T09:47:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:13 smithi044 ceph-mon[31618]: pgmap v375: 97 pgs: 97 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 738 op/s 2024-05-27T09:47:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:13 smithi044 ceph-mon[31618]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T09:47:14.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:13 smithi050 ceph-mon[36371]: pgmap v375: 97 pgs: 97 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 738 op/s 2024-05-27T09:47:14.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:13 smithi050 ceph-mon[36371]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T09:47:14.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:13 smithi189 ceph-mon[36671]: pgmap v375: 97 pgs: 97 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 738 op/s 2024-05-27T09:47:14.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:13 smithi189 ceph-mon[36671]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T09:47:14.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3868-ga6507388/rpm/el8/BUILD/ceph-19.0.0-3868-ga6507388/src/mds/MDCache.cc: In function 'void MDCache::remove_inode(CInode*)' thread 7f6aa9fcf700 time 2024-05-27T09:47:14.064342+0000 2024-05-27T09:47:14.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3868-ga6507388/rpm/el8/BUILD/ceph-19.0.0-3868-ga6507388/src/mds/MDCache.cc: 360: FAILED ceph_assert(o->get_num_ref() == 0) 2024-05-27T09:47:14.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev) 2024-05-27T09:47:14.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6abb011ba6] 2024-05-27T09:47:14.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6abb011dd1] 2024-05-27T09:47:14.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 3: (MDCache::remove_inode(CInode*)+0x588) [0x55d368cec744] 2024-05-27T09:47:14.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 4: (StrayManager::_purge_stray_logged(CDentry*, unsigned long, boost::intrusive_ptr&)+0x375) [0x55d368db3f45] 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 5: (virtual thunk to C_PurgeStrayLogged::finish(int)+0x27) [0x55d368db9aca] 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 6: (MDSContext::complete(int)+0x83) [0x55d368f5f079] 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 7: (MDSIOContextBase::complete(int)+0x75f) [0x55d368f5fa25] 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 8: (MDSLogContextBase::complete(int)+0x46) [0x55d368f5fb5e] 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 9: (Finisher::finisher_thread_entry()+0x5f9) [0x7f6abaf84ddd] 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 10: (Finisher::FinisherThread::entry()+0x11) [0x55d368b6c621] 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 11: (Thread::entry_wrapper()+0x43) [0x7f6abafe072d] 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 12: (Thread::_entry_func(void*)+0xd) [0x7f6abafe0749] 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f6ab9b841ca] 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 14: clone() 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: *** Caught signal (Aborted) ** 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: in thread 7f6aa9fcf700 thread_name:MR_Finisher 2024-05-27T09:47:14.339 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.066+0000 7f6aa9fcf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3868-ga6507388/rpm/el8/BUILD/ceph-19.0.0-3868-ga6507388/src/mds/MDCache.cc: In function 'void MDCache::remove_inode(CInode*)' thread 7f6aa9fcf700 time 2024-05-27T09:47:14.064342+0000 2024-05-27T09:47:14.340 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3868-ga6507388/rpm/el8/BUILD/ceph-19.0.0-3868-ga6507388/src/mds/MDCache.cc: 360: FAILED ceph_assert(o->get_num_ref() == 0) 2024-05-27T09:47:14.341 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.341 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev) 2024-05-27T09:47:14.341 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6abb011ba6] 2024-05-27T09:47:14.341 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6abb011dd1] 2024-05-27T09:47:14.341 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 3: (MDCache::remove_inode(CInode*)+0x588) [0x55d368cec744] 2024-05-27T09:47:14.341 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 4: (StrayManager::_purge_stray_logged(CDentry*, unsigned long, boost::intrusive_ptr&)+0x375) [0x55d368db3f45] 2024-05-27T09:47:14.341 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 5: (virtual thunk to C_PurgeStrayLogged::finish(int)+0x27) [0x55d368db9aca] 2024-05-27T09:47:14.341 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 6: (MDSContext::complete(int)+0x83) [0x55d368f5f079] 2024-05-27T09:47:14.341 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 7: (MDSIOContextBase::complete(int)+0x75f) [0x55d368f5fa25] 2024-05-27T09:47:14.341 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 8: (MDSLogContextBase::complete(int)+0x46) [0x55d368f5fb5e] 2024-05-27T09:47:14.341 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 9: (Finisher::finisher_thread_entry()+0x5f9) [0x7f6abaf84ddd] 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 10: (Finisher::FinisherThread::entry()+0x11) [0x55d368b6c621] 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 11: (Thread::entry_wrapper()+0x43) [0x7f6abafe072d] 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 12: (Thread::_entry_func(void*)+0xd) [0x7f6abafe0749] 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f6ab9b841ca] 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 14: clone() 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev) 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ab9b8ed20] 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2: gsignal() 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 3: abort() 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6abb011ced] 2024-05-27T09:47:14.342 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6abb011dd1] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 6: (MDCache::remove_inode(CInode*)+0x588) [0x55d368cec744] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 7: (StrayManager::_purge_stray_logged(CDentry*, unsigned long, boost::intrusive_ptr&)+0x375) [0x55d368db3f45] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 8: (virtual thunk to C_PurgeStrayLogged::finish(int)+0x27) [0x55d368db9aca] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 9: (MDSContext::complete(int)+0x83) [0x55d368f5f079] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 10: (MDSIOContextBase::complete(int)+0x75f) [0x55d368f5fa25] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 11: (MDSLogContextBase::complete(int)+0x46) [0x55d368f5fb5e] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 12: (Finisher::finisher_thread_entry()+0x5f9) [0x7f6abaf84ddd] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 13: (Finisher::FinisherThread::entry()+0x11) [0x55d368b6c621] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 14: (Thread::entry_wrapper()+0x43) [0x7f6abafe072d] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 15: (Thread::_entry_func(void*)+0xd) [0x7f6abafe0749] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f6ab9b841ca] 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 17: clone() 2024-05-27T09:47:14.343 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.069+0000 7f6aa9fcf700 -1 *** Caught signal (Aborted) ** 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: in thread 7f6aa9fcf700 thread_name:MR_Finisher 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev) 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ab9b8ed20] 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2: gsignal() 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 3: abort() 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6abb011ced] 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6abb011dd1] 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 6: (MDCache::remove_inode(CInode*)+0x588) [0x55d368cec744] 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 7: (StrayManager::_purge_stray_logged(CDentry*, unsigned long, boost::intrusive_ptr&)+0x375) [0x55d368db3f45] 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 8: (virtual thunk to C_PurgeStrayLogged::finish(int)+0x27) [0x55d368db9aca] 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 9: (MDSContext::complete(int)+0x83) [0x55d368f5f079] 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 10: (MDSIOContextBase::complete(int)+0x75f) [0x55d368f5fa25] 2024-05-27T09:47:14.344 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 11: (MDSLogContextBase::complete(int)+0x46) [0x55d368f5fb5e] 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 12: (Finisher::finisher_thread_entry()+0x5f9) [0x7f6abaf84ddd] 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 13: (Finisher::FinisherThread::entry()+0x11) [0x55d368b6c621] 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 14: (Thread::entry_wrapper()+0x43) [0x7f6abafe072d] 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 15: (Thread::_entry_func(void*)+0xd) [0x7f6abafe0749] 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f6ab9b841ca] 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 17: clone() 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: -122> 2024-05-27T09:47:14.066+0000 7f6aa9fcf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3868-ga6507388/rpm/el8/BUILD/ceph-19.0.0-3868-ga6507388/src/mds/MDCache.cc: In function 'void MDCache::remove_inode(CInode*)' thread 7f6aa9fcf700 time 2024-05-27T09:47:14.064342+0000 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3868-ga6507388/rpm/el8/BUILD/ceph-19.0.0-3868-ga6507388/src/mds/MDCache.cc: 360: FAILED ceph_assert(o->get_num_ref() == 0) 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev) 2024-05-27T09:47:14.345 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6abb011ba6] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6abb011dd1] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 3: (MDCache::remove_inode(CInode*)+0x588) [0x55d368cec744] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 4: (StrayManager::_purge_stray_logged(CDentry*, unsigned long, boost::intrusive_ptr&)+0x375) [0x55d368db3f45] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 5: (virtual thunk to C_PurgeStrayLogged::finish(int)+0x27) [0x55d368db9aca] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 6: (MDSContext::complete(int)+0x83) [0x55d368f5f079] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 7: (MDSIOContextBase::complete(int)+0x75f) [0x55d368f5fa25] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 8: (MDSLogContextBase::complete(int)+0x46) [0x55d368f5fb5e] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 9: (Finisher::finisher_thread_entry()+0x5f9) [0x7f6abaf84ddd] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 10: (Finisher::FinisherThread::entry()+0x11) [0x55d368b6c621] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 11: (Thread::entry_wrapper()+0x43) [0x7f6abafe072d] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 12: (Thread::_entry_func(void*)+0xd) [0x7f6abafe0749] 2024-05-27T09:47:14.346 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f6ab9b841ca] 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 14: clone() 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 0> 2024-05-27T09:47:14.069+0000 7f6aa9fcf700 -1 *** Caught signal (Aborted) ** 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: in thread 7f6aa9fcf700 thread_name:MR_Finisher 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev) 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ab9b8ed20] 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2: gsignal() 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 3: abort() 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6abb011ced] 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6abb011dd1] 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 6: (MDCache::remove_inode(CInode*)+0x588) [0x55d368cec744] 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 7: (StrayManager::_purge_stray_logged(CDentry*, unsigned long, boost::intrusive_ptr&)+0x375) [0x55d368db3f45] 2024-05-27T09:47:14.347 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 8: (virtual thunk to C_PurgeStrayLogged::finish(int)+0x27) [0x55d368db9aca] 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 9: (MDSContext::complete(int)+0x83) [0x55d368f5f079] 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 10: (MDSIOContextBase::complete(int)+0x75f) [0x55d368f5fa25] 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 11: (MDSLogContextBase::complete(int)+0x46) [0x55d368f5fb5e] 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 12: (Finisher::finisher_thread_entry()+0x5f9) [0x7f6abaf84ddd] 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 13: (Finisher::FinisherThread::entry()+0x11) [0x55d368b6c621] 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 14: (Thread::entry_wrapper()+0x43) [0x7f6abafe072d] 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 15: (Thread::_entry_func(void*)+0xd) [0x7f6abafe0749] 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f6ab9b841ca] 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 17: clone() 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: -308> 2024-05-27T09:47:14.066+0000 7f6aa9fcf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3868-ga6507388/rpm/el8/BUILD/ceph-19.0.0-3868-ga6507388/src/mds/MDCache.cc: In function 'void MDCache::remove_inode(CInode*)' thread 7f6aa9fcf700 time 2024-05-27T09:47:14.064342+0000 2024-05-27T09:47:14.348 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3868-ga6507388/rpm/el8/BUILD/ceph-19.0.0-3868-ga6507388/src/mds/MDCache.cc: 360: FAILED ceph_assert(o->get_num_ref() == 0) 2024-05-27T09:47:14.349 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.349 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev) 2024-05-27T09:47:14.349 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6abb011ba6] 2024-05-27T09:47:14.349 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6abb011dd1] 2024-05-27T09:47:14.349 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 3: (MDCache::remove_inode(CInode*)+0x588) [0x55d368cec744] 2024-05-27T09:47:14.349 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 4: (StrayManager::_purge_stray_logged(CDentry*, unsigned long, boost::intrusive_ptr&)+0x375) [0x55d368db3f45] 2024-05-27T09:47:14.349 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 5: (virtual thunk to C_PurgeStrayLogged::finish(int)+0x27) [0x55d368db9aca] 2024-05-27T09:47:14.349 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 6: (MDSContext::complete(int)+0x83) [0x55d368f5f079] 2024-05-27T09:47:14.349 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 7: (MDSIOContextBase::complete(int)+0x75f) [0x55d368f5fa25] 2024-05-27T09:47:14.349 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 8: (MDSLogContextBase::complete(int)+0x46) [0x55d368f5fb5e] 2024-05-27T09:47:14.349 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 9: (Finisher::finisher_thread_entry()+0x5f9) [0x7f6abaf84ddd] 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 10: (Finisher::FinisherThread::entry()+0x11) [0x55d368b6c621] 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 11: (Thread::entry_wrapper()+0x43) [0x7f6abafe072d] 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 12: (Thread::_entry_func(void*)+0xd) [0x7f6abafe0749] 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 13: /lib64/libpthread.so.0(+0x81ca) [0x7f6ab9b841ca] 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 14: clone() 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: -186> 2024-05-27T09:47:14.069+0000 7f6aa9fcf700 -1 *** Caught signal (Aborted) ** 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: in thread 7f6aa9fcf700 thread_name:MR_Finisher 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: ceph version 19.0.0-3868-ga6507388 (a6507388dd5057528934822c0163b0c347ef1d5d) squid (dev) 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6ab9b8ed20] 2024-05-27T09:47:14.350 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2: gsignal() 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 3: abort() 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6abb011ced] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6abb011dd1] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 6: (MDCache::remove_inode(CInode*)+0x588) [0x55d368cec744] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 7: (StrayManager::_purge_stray_logged(CDentry*, unsigned long, boost::intrusive_ptr&)+0x375) [0x55d368db3f45] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 8: (virtual thunk to C_PurgeStrayLogged::finish(int)+0x27) [0x55d368db9aca] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 9: (MDSContext::complete(int)+0x83) [0x55d368f5f079] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 10: (MDSIOContextBase::complete(int)+0x75f) [0x55d368f5fa25] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 11: (MDSLogContextBase::complete(int)+0x46) [0x55d368f5fb5e] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 12: (Finisher::finisher_thread_entry()+0x5f9) [0x7f6abaf84ddd] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 13: (Finisher::FinisherThread::entry()+0x11) [0x55d368b6c621] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 14: (Thread::entry_wrapper()+0x43) [0x7f6abafe072d] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 15: (Thread::_entry_func(void*)+0xd) [0x7f6abafe0749] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f6ab9b841ca] 2024-05-27T09:47:14.351 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 17: clone() 2024-05-27T09:47:14.352 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T09:47:14.352 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[67858]: 2024-05-27T09:47:14.352 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 podman[74066]: 2024-05-27 09:47:14.147246451 +0000 UTC m=+0.019120768 container died 05084cdaaf8ecba1b4f72d59ce45a1f33201ef66927b997adc959422eedf1e89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0) 2024-05-27T09:47:15.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 podman[74066]: 2024-05-27 09:47:14.877194144 +0000 UTC m=+0.749068483 container remove 05084cdaaf8ecba1b4f72d59ce45a1f33201ef66927b997adc959422eedf1e89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-27T09:47:15.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:14 smithi189 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-05-27T09:47:15.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:15 smithi189 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-05-27T09:47:15.338 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:15 smithi189 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.i.service: Consumed 7min 49.970s CPU time. 2024-05-27T09:47:16.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:15 smithi044 ceph-mon[31618]: pgmap v377: 97 pgs: 97 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 937 KiB/s wr, 178 op/s 2024-05-27T09:47:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:15 smithi050 ceph-mon[36371]: pgmap v377: 97 pgs: 97 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 937 KiB/s wr, 178 op/s 2024-05-27T09:47:16.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:15 smithi189 ceph-mon[36671]: pgmap v377: 97 pgs: 97 active+clean; 312 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 937 KiB/s wr, 178 op/s 2024-05-27T09:47:18.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:17 smithi044 ceph-mon[31618]: pgmap v378: 97 pgs: 97 active+clean; 316 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.1 MiB/s wr, 1.03k op/s 2024-05-27T09:47:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.291+0000 7f5047a00640 1 -- 172.21.15.44:0/376013880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f504010a120 msgr2=0x7f5040103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.291+0000 7f5047a00640 1 --2- 172.21.15.44:0/376013880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f504010a120 0x7f5040103d60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5030009900 tx=0x7f503002a3f0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.291+0000 7f5047a00640 1 -- 172.21.15.44:0/376013880 shutdown_connections 2024-05-27T09:47:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.291+0000 7f5047a00640 1 --2- 172.21.15.44:0/376013880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50401043d0 0x7f50401107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.291+0000 7f5047a00640 1 --2- 172.21.15.44:0/376013880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f504010a120 0x7f5040103d60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.291+0000 7f5047a00640 1 --2- 172.21.15.44:0/376013880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5040109c80 0x7f5040103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.291+0000 7f5047a00640 1 -- 172.21.15.44:0/376013880 >> 172.21.15.44:0/376013880 conn(0x7f50400ff340 msgr2=0x7f5040101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.291+0000 7f5047a00640 1 -- 172.21.15.44:0/376013880 shutdown_connections 2024-05-27T09:47:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.291+0000 7f5047a00640 1 -- 172.21.15.44:0/376013880 wait complete. 2024-05-27T09:47:18.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.292+0000 7f5047a00640 1 Processor -- start 2024-05-27T09:47:18.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5047a00640 1 -- start start 2024-05-27T09:47:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5047a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50401043d0 0x7f50401a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5047a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5040109c80 0x7f50401a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5047a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f504010a120 0x7f504019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5047a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5040113ea0 con 0x7f50401043d0 2024-05-27T09:47:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5047a00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5040113d20 con 0x7f5040109c80 2024-05-27T09:47:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5047a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5040114020 con 0x7f504010a120 2024-05-27T09:47:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5045000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50401043d0 0x7f50401a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5037e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5040109c80 0x7f50401a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5045000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50401043d0 0x7f50401a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41178/0 (socket says 172.21.15.44:41178) 2024-05-27T09:47:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5045000640 1 -- 172.21.15.44:0/1167079355 learned_addr learned my addr 172.21.15.44:0/1167079355 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.293+0000 7f5045a00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f504010a120 0x7f504019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:17 smithi050 ceph-mon[36371]: pgmap v378: 97 pgs: 97 active+clean; 316 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.1 MiB/s wr, 1.03k op/s 2024-05-27T09:47:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.294+0000 7f5037e00640 1 -- 172.21.15.44:0/1167079355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f504010a120 msgr2=0x7f504019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.294+0000 7f5037e00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f504010a120 0x7f504019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.294+0000 7f5037e00640 1 -- 172.21.15.44:0/1167079355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50401043d0 msgr2=0x7f50401a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.294+0000 7f5037e00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50401043d0 0x7f50401a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.294+0000 7f5037e00640 1 -- 172.21.15.44:0/1167079355 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504019ae90 con 0x7f5040109c80 2024-05-27T09:47:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.294+0000 7f5045000640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50401043d0 0x7f50401a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.294+0000 7f5037e00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5040109c80 0x7f50401a0b20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5030009230 tx=0x7f5030003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.294+0000 7f503e000640 1 -- 172.21.15.44:0/1167079355 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503003f070 con 0x7f5040109c80 2024-05-27T09:47:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.295+0000 7f5047a00640 1 -- 172.21.15.44:0/1167079355 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f504019b0c0 con 0x7f5040109c80 2024-05-27T09:47:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.295+0000 7f503e000640 1 -- 172.21.15.44:0/1167079355 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5030031b50 con 0x7f5040109c80 2024-05-27T09:47:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.295+0000 7f503e000640 1 -- 172.21.15.44:0/1167079355 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030035030 con 0x7f5040109c80 2024-05-27T09:47:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.295+0000 7f5047a00640 1 -- 172.21.15.44:0/1167079355 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f504019b5d0 con 0x7f5040109c80 2024-05-27T09:47:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.298+0000 7f504e9f8740 1 Processor -- start 2024-05-27T09:47:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.298+0000 7f504e9f8740 1 -- start start 2024-05-27T09:47:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.298+0000 7f503e000640 1 -- 172.21.15.44:0/1167079355 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5030042070 con 0x7f5040109c80 2024-05-27T09:47:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be53802730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 0x55be53802c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be5343f4d0 0x55be538031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f504e9f8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55be53773b90 con 0x55be534601b0 2024-05-27T09:47:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f504e9f8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55be53763730 con 0x55be5343f4d0 2024-05-27T09:47:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f504e9f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55be53767770 con 0x55be534372f0 2024-05-27T09:47:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f5045a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be53802730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f5045000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 0x55be53802c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f5045a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be53802730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41182/0 (socket says 172.21.15.44:41182) 2024-05-27T09:47:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f5045000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 0x55be53802c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:42498/0 (socket says 172.21.15.44:42498) 2024-05-27T09:47:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f5045a00640 1 -- 172.21.15.44:0/42749118 learned_addr learned my addr 172.21.15.44:0/42749118 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f5037e00640 1 --2- 172.21.15.44:0/42749118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be5343f4d0 0x55be538031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f5045000640 1 -- 172.21.15.44:0/42749118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be5343f4d0 msgr2=0x55be538031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f5045000640 1 --2- 172.21.15.44:0/42749118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be5343f4d0 0x55be538031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f5045000640 1 -- 172.21.15.44:0/42749118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 msgr2=0x55be53802730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.299+0000 7f5045000640 1 --2- 172.21.15.44:0/42749118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be53802730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.300+0000 7f5045000640 1 -- 172.21.15.44:0/42749118 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be536fc8b0 con 0x55be534372f0 2024-05-27T09:47:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.300+0000 7f5045000640 1 --2- 172.21.15.44:0/42749118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 0x55be53802c70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5028011470 tx=0x7f5028011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74a8f5dbaea86581 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.300+0000 7f503e000640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f50200b2950 0x7f50200b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.300+0000 7f5036000640 1 -- 172.21.15.44:0/42749118 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028004020 con 0x55be534372f0 2024-05-27T09:47:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.301+0000 7f503e000640 1 -- 172.21.15.44:0/1167079355 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8521+0+0 (secure 0 0 0) 0x7f5030102b90 con 0x7f5040109c80 2024-05-27T09:47:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.301+0000 7f5036000640 1 -- 172.21.15.44:0/42749118 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50280054d0 con 0x55be534372f0 2024-05-27T09:47:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.301+0000 7f5036000640 1 -- 172.21.15.44:0/42749118 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028004bc0 con 0x55be534372f0 2024-05-27T09:47:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.301+0000 7f5045a00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f50200b2950 0x7f50200b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.302+0000 7f504e9f8740 1 -- 172.21.15.44:0/42749118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 msgr2=0x55be53802c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.302+0000 7f5045a00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f50200b2950 0x7f50200b4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f50380116b0 tx=0x7f5038009500 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.302+0000 7f504e9f8740 1 --2- 172.21.15.44:0/42749118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 0x55be53802c70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5028011470 tx=0x7f5028011930 comp rx=0 tx=0).stop 2024-05-27T09:47:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.302+0000 7f504e9f8740 1 -- 172.21.15.44:0/42749118 shutdown_connections 2024-05-27T09:47:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.302+0000 7f504e9f8740 1 --2- 172.21.15.44:0/42749118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be5343f4d0 0x55be538031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.302+0000 7f504e9f8740 1 --2- 172.21.15.44:0/42749118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 0x55be53802c70 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.302+0000 7f504e9f8740 1 --2- 172.21.15.44:0/42749118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be53802730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.302+0000 7f504e9f8740 1 -- 172.21.15.44:0/42749118 >> 172.21.15.44:0/42749118 conn(0x55be53454300 msgr2=0x55be537ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.302+0000 7f504e9f8740 1 -- 172.21.15.44:0/42749118 shutdown_connections 2024-05-27T09:47:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.302+0000 7f504e9f8740 1 -- 172.21.15.44:0/42749118 wait complete. 2024-05-27T09:47:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.303+0000 7f504e9f8740 1 -- start start 2024-05-27T09:47:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.303+0000 7f504e9f8740 1 Processor -- start 2024-05-27T09:47:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.303+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 0x55be53810d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.303+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be5343f4d0 0x55be53811260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.303+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be538117a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.303+0000 7f504e9f8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55be53449790 con 0x55be534601b0 2024-05-27T09:47:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.303+0000 7f504e9f8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55be5370bbc0 con 0x55be534372f0 2024-05-27T09:47:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.303+0000 7f504e9f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55be53813e50 con 0x55be5343f4d0 2024-05-27T09:47:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f504e9f8740 10 client.0 _ll_register_callbacks cb 0x55be536e9400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f5045000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 0x55be53810d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f5045000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 0x55be53810d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:33666/0 (socket says 172.21.15.44:33666) 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f5045000640 1 -- 172.21.15.44:0/648676244 learned_addr learned my addr 172.21.15.44:0/648676244 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f5045a00640 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be538117a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f5037e00640 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be5343f4d0 0x55be53811260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f5045000640 1 -- 172.21.15.44:0/648676244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be5343f4d0 msgr2=0x55be53811260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f5045000640 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be5343f4d0 0x55be53811260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f5045000640 1 -- 172.21.15.44:0/648676244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 msgr2=0x55be538117a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f5045000640 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be538117a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f5045000640 1 -- 172.21.15.44:0/648676244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be53478210 con 0x55be534372f0 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.304+0000 7f5045a00640 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be538117a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.305+0000 7f5045000640 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 0x55be53810d20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5028000c00 tx=0x7f502800fcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f821c3e70105a66e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.305+0000 7f502ea00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.305+0000 7f502ea00640 1 -- 172.21.15.44:0/648676244 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028026020 con 0x55be534372f0 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.305+0000 7f502ea00640 1 -- 172.21.15.44:0/648676244 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8521+0+0 (secure 0 0 0) 0x7f5028004940 con 0x55be534372f0 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.305+0000 7f504e9f8740 1 -- 172.21.15.44:0/648676244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50280054d0 con 0x55be534372f0 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.306+0000 7f502ea00640 1 -- 172.21.15.44:0/648676244 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f5028031050 con 0x55be534372f0 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.306+0000 7f504e9f8740 10 client.27476 fetch_fsmap learned FSMap version 34 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.306+0000 7f504e9f8740 1 -- 172.21.15.44:0/648676244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55be53544620 con 0x55be534372f0 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.306+0000 7f502ea00640 1 -- 172.21.15.44:0/648676244 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50280054d0 con 0x55be534372f0 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.306+0000 7f502ea00640 1 -- 172.21.15.44:0/648676244 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f5028022410 con 0x55be534372f0 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.306+0000 7f504e9f8740 10 client.27476 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.306+0000 7f504e9f8740 10 client.27476 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.306+0000 7f504e9f8740 20 client.27476 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.306+0000 7f504e9f8740 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55be5341ed80 0x55be53827360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.306+0000 7f504e9f8740 4 client.27476 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:18.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.306+0000 7f504e9f8740 1 -- 172.21.15.44:0/648676244 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be53775ae0 con 0x55be5341ed80 2024-05-27T09:47:18.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.307+0000 7f5037e00640 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55be5341ed80 0x55be53827360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.307+0000 7f5037e00640 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55be5341ed80 0x55be53827360 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52568dee17a72bda server_cookie=6e4b848886a7e256 in_seq=0 out_seq=0 2024-05-27T09:47:18.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.307+0000 7f502ea00640 10 client.27476 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.308+0000 7f502ea00640 1 -- 172.21.15.44:0/648676244 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55be53775ae0 con 0x55be5341ed80 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.308+0000 7f502ea00640 10 client.27476 handle_command_reply: tid=0 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.308+0000 7f504e9f8740 1 client.27476 shutdown 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.308+0000 7f504e9f8740 1 -- 172.21.15.44:0/648676244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 msgr2=0x55be53810d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.308+0000 7f504e9f8740 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 0x55be53810d20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5028000c00 tx=0x7f502800fcf0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.309+0000 7f504e9f8740 1 -- 172.21.15.44:0/648676244 shutdown_connections 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.309+0000 7f504e9f8740 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55be5341ed80 0x55be53827360 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.309+0000 7f504e9f8740 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be538117a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.309+0000 7f504e9f8740 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be5343f4d0 0x55be53811260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.309+0000 7f504e9f8740 1 --2- 172.21.15.44:0/648676244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 0x55be53810d20 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.309+0000 7f504e9f8740 1 -- 172.21.15.44:0/648676244 >> 172.21.15.44:0/648676244 conn(0x55be53454300 msgr2=0x55be538025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.309+0000 7f502ea00640 0 client.27476 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:18.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.309+0000 7f504e9f8740 1 -- 172.21.15.44:0/648676244 shutdown_connections 2024-05-27T09:47:18.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.309+0000 7f504e9f8740 1 -- 172.21.15.44:0/648676244 wait complete. 2024-05-27T09:47:18.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.309+0000 7f504e9f8740 20 client.27476 trim_cache size 0 max 16384 2024-05-27T09:47:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.326+0000 7f504e9f8740 1 Processor -- start 2024-05-27T09:47:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.326+0000 7f504e9f8740 1 -- start start 2024-05-27T09:47:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.326+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 0x55be53810020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.326+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be5343f4d0 0x55be538115c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.326+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be5382ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.326+0000 7f504e9f8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f50401b02c0 con 0x55be534601b0 2024-05-27T09:47:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.326+0000 7f504e9f8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f50401adb30 con 0x55be534372f0 2024-05-27T09:47:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.326+0000 7f504e9f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f50401ab9c0 con 0x55be5343f4d0 2024-05-27T09:47:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.327+0000 7f5045a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be5382ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.327+0000 7f5045000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 0x55be53810020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.327+0000 7f5045a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be5382ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41192/0 (socket says 172.21.15.44:41192) 2024-05-27T09:47:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.327+0000 7f5045a00640 1 -- 172.21.15.44:0/3573770694 learned_addr learned my addr 172.21.15.44:0/3573770694 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.327+0000 7f5037e00640 1 --2- 172.21.15.44:0/3573770694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be5343f4d0 0x55be538115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.327+0000 7f5045000640 1 -- 172.21.15.44:0/3573770694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be5343f4d0 msgr2=0x55be538115c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.327+0000 7f5045000640 1 --2- 172.21.15.44:0/3573770694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be5343f4d0 0x55be538115c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.327+0000 7f5045000640 1 -- 172.21.15.44:0/3573770694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 msgr2=0x55be5382ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.327+0000 7f5045000640 1 --2- 172.21.15.44:0/3573770694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be5382ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.327+0000 7f5045000640 1 -- 172.21.15.44:0/3573770694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be53811b00 con 0x55be534372f0 2024-05-27T09:47:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.327+0000 7f5045000640 1 --2- 172.21.15.44:0/3573770694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 0x55be53810020 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f50280229a0 tx=0x7f5028005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f64e490f421093d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.328+0000 7f5035600640 1 -- 172.21.15.44:0/3573770694 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028026040 con 0x55be534372f0 2024-05-27T09:47:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.328+0000 7f5035600640 1 -- 172.21.15.44:0/3573770694 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50280054e0 con 0x55be534372f0 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.328+0000 7f5035600640 1 -- 172.21.15.44:0/3573770694 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502801d040 con 0x55be534372f0 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.328+0000 7f504e9f8740 1 -- 172.21.15.44:0/3573770694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 msgr2=0x55be53810020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.328+0000 7f504e9f8740 1 --2- 172.21.15.44:0/3573770694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 0x55be53810020 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f50280229a0 tx=0x7f5028005bb0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.328+0000 7f504e9f8740 1 -- 172.21.15.44:0/3573770694 shutdown_connections 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.328+0000 7f504e9f8740 1 --2- 172.21.15.44:0/3573770694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be534601b0 0x55be5382ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.328+0000 7f504e9f8740 1 --2- 172.21.15.44:0/3573770694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be5343f4d0 0x55be538115c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.328+0000 7f504e9f8740 1 --2- 172.21.15.44:0/3573770694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534372f0 0x55be53810020 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.328+0000 7f504e9f8740 1 -- 172.21.15.44:0/3573770694 >> 172.21.15.44:0/3573770694 conn(0x55be53454300 msgr2=0x55be53812440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.328+0000 7f504e9f8740 1 -- 172.21.15.44:0/3573770694 shutdown_connections 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.329+0000 7f504e9f8740 1 -- 172.21.15.44:0/3573770694 wait complete. 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.329+0000 7f504e9f8740 1 -- start start 2024-05-27T09:47:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.329+0000 7f504e9f8740 1 Processor -- start 2024-05-27T09:47:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 0x55be5381ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5343f4d0 0x55be5381f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f504e9f8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534601b0 0x55be5381f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f504e9f8740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f50401ac010 con 0x55be5343f4d0 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f504e9f8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f50401ad460 con 0x55be534601b0 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f504e9f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f50401ac6c0 con 0x55be534372f0 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f504e9f8740 10 client.0 _ll_register_callbacks cb 0x7f5038009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f5045a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534601b0 0x55be5381f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f5045000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 0x55be5381ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f5045a00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534601b0 0x55be5381f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:33692/0 (socket says 172.21.15.44:33692) 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f5045a00640 1 -- 172.21.15.44:0/3058492462 learned_addr learned my addr 172.21.15.44:0/3058492462 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f5037e00640 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5343f4d0 0x55be5381f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f5045000640 1 -- 172.21.15.44:0/3058492462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534601b0 msgr2=0x55be5381f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f5045000640 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534601b0 0x55be5381f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f5045000640 1 -- 172.21.15.44:0/3058492462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5343f4d0 msgr2=0x55be5381f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f5045000640 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5343f4d0 0x55be5381f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f5045000640 1 -- 172.21.15.44:0/3058492462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be5388db10 con 0x55be534372f0 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.330+0000 7f5045a00640 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534601b0 0x55be5381f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.331+0000 7f5045000640 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 0x55be5381ebc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5028022e90 tx=0x7f5028005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fc723dd16ed25f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.331+0000 7f502ea00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.331+0000 7f502ea00640 1 -- 172.21.15.44:0/3058492462 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028039070 con 0x55be534372f0 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.331+0000 7f502ea00640 1 -- 172.21.15.44:0/3058492462 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8521+0+0 (secure 0 0 0) 0x7f5028011020 con 0x55be534372f0 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.331+0000 7f504e9f8740 1 -- 172.21.15.44:0/3058492462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50280054e0 con 0x55be534372f0 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.331+0000 7f502ea00640 1 -- 172.21.15.44:0/3058492462 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f502801d040 con 0x55be534372f0 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.331+0000 7f504e9f8740 10 client.27472 fetch_fsmap learned FSMap version 34 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.331+0000 7f504e9f8740 1 -- 172.21.15.44:0/3058492462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55be5388eda0 con 0x55be534372f0 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.332+0000 7f502ea00640 1 -- 172.21.15.44:0/3058492462 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50280054e0 con 0x55be534372f0 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.332+0000 7f502ea00640 1 -- 172.21.15.44:0/3058492462 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f502803c020 con 0x55be534372f0 2024-05-27T09:47:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.332+0000 7f504e9f8740 10 client.27472 fetch_fsmap finished waiting for FSMap version 34 2024-05-27T09:47:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.332+0000 7f504e9f8740 10 client.27472 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.332+0000 7f504e9f8740 20 client.27472 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.332+0000 7f504e9f8740 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55be533f41c0 0x55be53891440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.332+0000 7f504e9f8740 4 client.27472 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:47:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.332+0000 7f504e9f8740 1 -- 172.21.15.44:0/3058492462 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55be5376f2c0 con 0x55be533f41c0 2024-05-27T09:47:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.332+0000 7f5037e00640 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55be533f41c0 0x55be53891440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.333+0000 7f5037e00640 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55be533f41c0 0x55be53891440 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=374cbe5c419a609 server_cookie=376a108b4b22b29 in_seq=0 out_seq=0 2024-05-27T09:47:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:18.333+0000 7f502ea00640 10 client.27472 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:18.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:17 smithi189 ceph-mon[36671]: pgmap v378: 97 pgs: 97 active+clean; 316 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.1 MiB/s wr, 1.03k op/s 2024-05-27T09:47:20.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:19 smithi044 ceph-mon[31618]: pgmap v379: 97 pgs: 97 active+clean; 316 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.1 MiB/s wr, 999 op/s 2024-05-27T09:47:20.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:19 smithi050 ceph-mon[36371]: pgmap v379: 97 pgs: 97 active+clean; 316 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.1 MiB/s wr, 999 op/s 2024-05-27T09:47:20.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:19 smithi189 ceph-mon[36671]: pgmap v379: 97 pgs: 97 active+clean; 316 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.1 MiB/s wr, 999 op/s 2024-05-27T09:47:20.677 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:47:21.148 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:22.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:21 smithi044 ceph-mon[31618]: pgmap v380: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 470 KiB/s wr, 908 op/s 2024-05-27T09:47:22.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:21 smithi050 ceph-mon[36371]: pgmap v380: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 470 KiB/s wr, 908 op/s 2024-05-27T09:47:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:21 smithi189 ceph-mon[36671]: pgmap v380: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 470 KiB/s wr, 908 op/s 2024-05-27T09:47:23.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.790+0000 7ff315a00700 1 -- 172.21.15.44:0/4160720468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff310109670 msgr2=0x7ff310110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:23.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.790+0000 7ff315a00700 1 --2- 172.21.15.44:0/4160720468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff310109670 0x7ff310110fd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff308009fc0 tx=0x7ff30800b330 comp rx=0 tx=0).stop 2024-05-27T09:47:23.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.791+0000 7ff315a00700 1 -- 172.21.15.44:0/4160720468 shutdown_connections 2024-05-27T09:47:23.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.791+0000 7ff315a00700 1 --2- 172.21.15.44:0/4160720468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff310109670 0x7ff310110fd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:23.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.791+0000 7ff315a00700 1 --2- 172.21.15.44:0/4160720468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3101043f0 0x7ff310104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.791+0000 7ff315a00700 1 --2- 172.21.15.44:0/4160720468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff310108d20 0x7ff310109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.791+0000 7ff315a00700 1 -- 172.21.15.44:0/4160720468 >> 172.21.15.44:0/4160720468 conn(0x7ff310079d00 msgr2=0x7ff31007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.791+0000 7ff315a00700 1 -- 172.21.15.44:0/4160720468 shutdown_connections 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.791+0000 7ff315a00700 1 -- 172.21.15.44:0/4160720468 wait complete. 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.792+0000 7ff315a00700 1 Processor -- start 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.793+0000 7ff315a00700 1 -- start start 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.793+0000 7ff315a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3101043f0 0x7ff31019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.793+0000 7ff315a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff310108d20 0x7ff31019d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.793+0000 7ff315a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff310109670 0x7ff3101a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.793+0000 7ff315a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff3101142a0 con 0x7ff310109670 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.793+0000 7ff315a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff310114120 con 0x7ff3101043f0 2024-05-27T09:47:23.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.793+0000 7ff315a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff310114420 con 0x7ff310108d20 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff315000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff310109670 0x7ff3101a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff30fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3101043f0 0x7ff31019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff315000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff310109670 0x7ff3101a1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34742/0 (socket says 172.21.15.44:34742) 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff30fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3101043f0 0x7ff31019cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:32836/0 (socket says 172.21.15.44:32836) 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff315000700 1 -- 172.21.15.44:0/2120470408 learned_addr learned my addr 172.21.15.44:0/2120470408 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff30f400700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff310108d20 0x7ff31019d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff30fe00700 1 -- 172.21.15.44:0/2120470408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff310108d20 msgr2=0x7ff31019d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff30fe00700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff310108d20 0x7ff31019d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff30fe00700 1 -- 172.21.15.44:0/2120470408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff310109670 msgr2=0x7ff3101a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff30fe00700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff310109670 0x7ff3101a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff30fe00700 1 -- 172.21.15.44:0/2120470408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3101a1cc0 con 0x7ff3101043f0 2024-05-27T09:47:23.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.794+0000 7ff30fe00700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3101043f0 0x7ff31019cdf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff30000eb90 tx=0x7ff30000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.795+0000 7ff30cc00700 1 -- 172.21.15.44:0/2120470408 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30000cc90 con 0x7ff3101043f0 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.795+0000 7ff30cc00700 1 -- 172.21.15.44:0/2120470408 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff30000ce30 con 0x7ff3101043f0 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.795+0000 7ff30cc00700 1 -- 172.21.15.44:0/2120470408 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff300010430 con 0x7ff3101043f0 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.795+0000 7ff315a00700 1 -- 172.21.15.44:0/2120470408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3101a1fb0 con 0x7ff3101043f0 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.795+0000 7ff315a00700 1 -- 172.21.15.44:0/2120470408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3101a98d0 con 0x7ff3101043f0 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.796+0000 7ff315a00700 1 -- 172.21.15.44:0/2120470408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3101058a0 con 0x7ff3101043f0 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.799+0000 7ff30cc00700 1 -- 172.21.15.44:0/2120470408 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff300003ef0 con 0x7ff3101043f0 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.801+0000 7ff30cc00700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2f40b29f0 0x7ff2f40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.801+0000 7ff30cc00700 1 -- 172.21.15.44:0/2120470408 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8521+0+0 (secure 0 0 0) 0x7ff3000e2a40 con 0x7ff3101043f0 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.801+0000 7ff30f400700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2f40b29f0 0x7ff2f40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.802+0000 7ff30f400700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2f40b29f0 0x7ff2f40b4f00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff3101094a0 tx=0x7ff2fc009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:23.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:23.803+0000 7ff30cc00700 1 -- 172.21.15.44:0/2120470408 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff3000aa450 con 0x7ff3101043f0 2024-05-27T09:47:24.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.092+0000 7ff315a00700 1 -- 172.21.15.44:0/2120470408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3100512b0 con 0x7ff3101043f0 2024-05-27T09:47:24.094 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:23 smithi044 ceph-mon[31618]: pgmap v381: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 470 KiB/s wr, 908 op/s 2024-05-27T09:47:24.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.093+0000 7ff30cc00700 1 -- 172.21.15.44:0/2120470408 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11868 (secure 0 0 0) 0x7ff3000a9d50 con 0x7ff3101043f0 2024-05-27T09:47:24.095 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:47:24.095 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:45:53.426075+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24535,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.189:6839/4109192600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4109192600},{"type":"v1","addr":"172.21.15.189:6839","nonce":4109192600}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24535,24542]},"id":1}]} 2024-05-27T09:47:24.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.097+0000 7ff315a00700 1 -- 172.21.15.44:0/2120470408 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2f40b29f0 msgr2=0x7ff2f40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:24.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.097+0000 7ff315a00700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2f40b29f0 0x7ff2f40b4f00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff3101094a0 tx=0x7ff2fc009260 comp rx=0 tx=0).stop 2024-05-27T09:47:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.097+0000 7ff315a00700 1 -- 172.21.15.44:0/2120470408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3101043f0 msgr2=0x7ff31019cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.097+0000 7ff315a00700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3101043f0 0x7ff31019cdf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff30000eb90 tx=0x7ff30000eea0 comp rx=0 tx=0).stop 2024-05-27T09:47:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.097+0000 7ff315a00700 1 -- 172.21.15.44:0/2120470408 shutdown_connections 2024-05-27T09:47:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.097+0000 7ff315a00700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff2f40b29f0 0x7ff2f40b4f00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.097+0000 7ff315a00700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff310109670 0x7ff3101a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.097+0000 7ff315a00700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3101043f0 0x7ff31019cdf0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.097+0000 7ff315a00700 1 --2- 172.21.15.44:0/2120470408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff310108d20 0x7ff31019d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.097+0000 7ff315a00700 1 -- 172.21.15.44:0/2120470408 >> 172.21.15.44:0/2120470408 conn(0x7ff310079d00 msgr2=0x7ff310101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.098+0000 7ff315a00700 1 -- 172.21.15.44:0/2120470408 shutdown_connections 2024-05-27T09:47:24.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:24.098+0000 7ff315a00700 1 -- 172.21.15.44:0/2120470408 wait complete. 2024-05-27T09:47:24.101 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 34 2024-05-27T09:47:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:23 smithi050 ceph-mon[36371]: pgmap v381: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 470 KiB/s wr, 908 op/s 2024-05-27T09:47:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:23 smithi189 ceph-mon[36671]: pgmap v381: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 470 KiB/s wr, 908 op/s 2024-05-27T09:47:24.950 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db --set-id b3fe0bae --release --await' 2024-05-27T09:47:24.950 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db --set-id b3fe0bae --release --await 2024-05-27T09:47:25.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:24 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2120470408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:25.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:24 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2120470408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:25.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:24 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2120470408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:25.426 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:25.594 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-05-27T09:47:25.594 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 systemd[1]: Stopped Ceph mds.i for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:47:25.595 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.i.service: Consumed 7min 49.970s CPU time. 2024-05-27T09:47:25.595 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 systemd[1]: Starting Ceph mds.i for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:47:25.595 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 podman[74158]: 2024-05-27 09:47:25.477577052 +0000 UTC m=+0.017466985 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:47:25.858 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 podman[74158]: 2024-05-27 09:47:25.594551756 +0000 UTC m=+0.134441670 container create b0bc0afa505177c0523859016e6665eb04d04d9e8fb53ef64b151753b0b78b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738) 2024-05-27T09:47:25.858 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 podman[74158]: 2024-05-27 09:47:25.822244898 +0000 UTC m=+0.362134817 container init b0bc0afa505177c0523859016e6665eb04d04d9e8fb53ef64b151753b0b78b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-05-27T09:47:25.858 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 podman[74158]: 2024-05-27 09:47:25.824839752 +0000 UTC m=+0.364729673 container start b0bc0afa505177c0523859016e6665eb04d04d9e8fb53ef64b151753b0b78b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-27T09:47:26.204 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 ceph-mds[74177]: -- 172.21.15.189:0/2552378146 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56519384ba40 con 0x565193893a80 2024-05-27T09:47:26.204 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[74173]: starting mds.i at 2024-05-27T09:47:26.204 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 bash[74158]: b0bc0afa505177c0523859016e6665eb04d04d9e8fb53ef64b151753b0b78b6a 2024-05-27T09:47:26.204 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:47:25 smithi189 systemd[1]: Started Ceph mds.i for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:47:26.205 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:25 smithi189 ceph-mon[36671]: pgmap v382: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 396 KiB/s wr, 765 op/s 2024-05-27T09:47:26.205 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:25 smithi189 ceph-mon[36671]: daemon mds.i restarted 2024-05-27T09:47:26.205 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:25 smithi189 ceph-mon[36671]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T09:47:26.205 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:25 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:47:26.205 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:25 smithi189 ceph-mon[36671]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T09:47:26.205 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:25 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6838/2373131774,v1:172.21.15.189:6839/2373131774] up:boot 2024-05-27T09:47:26.205 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:25 smithi189 ceph-mon[36671]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-05-27T09:47:26.205 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:25 smithi189 ceph-mon[36671]: fsmap cephfs:2/3 {0=b=up:active,2=c=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-05-27T09:47:26.205 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:25 smithi189 ceph-mon[36671]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:26.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:25 smithi044 ceph-mon[31618]: pgmap v382: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 396 KiB/s wr, 765 op/s 2024-05-27T09:47:26.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:25 smithi044 ceph-mon[31618]: daemon mds.i restarted 2024-05-27T09:47:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:25 smithi044 ceph-mon[31618]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T09:47:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:25 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:47:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:25 smithi044 ceph-mon[31618]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T09:47:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:25 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6838/2373131774,v1:172.21.15.189:6839/2373131774] up:boot 2024-05-27T09:47:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:25 smithi044 ceph-mon[31618]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-05-27T09:47:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:25 smithi044 ceph-mon[31618]: fsmap cephfs:2/3 {0=b=up:active,2=c=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-05-27T09:47:26.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:25 smithi044 ceph-mon[31618]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:25 smithi050 ceph-mon[36371]: pgmap v382: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 396 KiB/s wr, 765 op/s 2024-05-27T09:47:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:25 smithi050 ceph-mon[36371]: daemon mds.i restarted 2024-05-27T09:47:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:25 smithi050 ceph-mon[36371]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T09:47:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:25 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:47:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:25 smithi050 ceph-mon[36371]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T09:47:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:25 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6838/2373131774,v1:172.21.15.189:6839/2373131774] up:boot 2024-05-27T09:47:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:25 smithi050 ceph-mon[36371]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-05-27T09:47:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:25 smithi050 ceph-mon[36371]: fsmap cephfs:2/3 {0=b=up:active,2=c=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-05-27T09:47:26.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:25 smithi050 ceph-mon[36371]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:27 smithi044 ceph-mon[31618]: pgmap v384: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 130 KiB/s wr, 55 op/s 2024-05-27T09:47:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:27 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:resolve 2024-05-27T09:47:28.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:27 smithi044 ceph-mon[31618]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:28.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.264+0000 7f218bc00700 1 -- 172.21.15.44:0/1962357884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2184110c40 msgr2=0x7f2184113060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:28.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.264+0000 7f218bc00700 1 --2- 172.21.15.44:0/1962357884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2184110c40 0x7f2184113060 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f217c0098f0 tx=0x7f217c009c00 comp rx=0 tx=0).stop 2024-05-27T09:47:28.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.264+0000 7f218bc00700 1 -- 172.21.15.44:0/1962357884 shutdown_connections 2024-05-27T09:47:28.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.264+0000 7f218bc00700 1 --2- 172.21.15.44:0/1962357884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2184110c40 0x7f2184113060 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.264+0000 7f218bc00700 1 --2- 172.21.15.44:0/1962357884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2184103090 0x7f21841034a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.264+0000 7f218bc00700 1 --2- 172.21.15.44:0/1962357884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2184103a70 0x7f2184110700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.264+0000 7f218bc00700 1 -- 172.21.15.44:0/1962357884 >> 172.21.15.44:0/1962357884 conn(0x7f21840fef80 msgr2=0x7f21841013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:28.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.265+0000 7f218bc00700 1 -- 172.21.15.44:0/1962357884 shutdown_connections 2024-05-27T09:47:28.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.265+0000 7f218bc00700 1 -- 172.21.15.44:0/1962357884 wait complete. 2024-05-27T09:47:28.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.266+0000 7f218bc00700 1 Processor -- start 2024-05-27T09:47:28.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.266+0000 7f218bc00700 1 -- start start 2024-05-27T09:47:28.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.266+0000 7f218bc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2184103090 0x7f218419d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.266+0000 7f218bc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2184103a70 0x7f218419d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f218bc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2184110c40 0x7f21841a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f218bc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f21841166d0 con 0x7f2184110c40 2024-05-27T09:47:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f218bc00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2184116550 con 0x7f2184103090 2024-05-27T09:47:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f218bc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2184116850 con 0x7f2184103a70 2024-05-27T09:47:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f2189200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2184103090 0x7f218419d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f2189200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2184103090 0x7f218419d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:32856/0 (socket says 172.21.15.44:32856) 2024-05-27T09:47:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f2189200700 1 -- 172.21.15.44:0/3582328516 learned_addr learned my addr 172.21.15.44:0/3582328516 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f2189c00700 1 --2- 172.21.15.44:0/3582328516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2184110c40 0x7f21841a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f217be00700 1 --2- 172.21.15.44:0/3582328516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2184103a70 0x7f218419d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f2189200700 1 -- 172.21.15.44:0/3582328516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2184103a70 msgr2=0x7f218419d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f2189200700 1 --2- 172.21.15.44:0/3582328516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2184103a70 0x7f218419d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f2189200700 1 -- 172.21.15.44:0/3582328516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2184110c40 msgr2=0x7f21841a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f2189200700 1 --2- 172.21.15.44:0/3582328516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2184110c40 0x7f21841a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f2189200700 1 -- 172.21.15.44:0/3582328516 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21841a1ef0 con 0x7f2184103090 2024-05-27T09:47:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.267+0000 7f2189c00700 1 --2- 172.21.15.44:0/3582328516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2184110c40 0x7f21841a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:28.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.268+0000 7f2189200700 1 --2- 172.21.15.44:0/3582328516 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2184103090 0x7f218419d020 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f217400e3c0 tx=0x7f217400e6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.268+0000 7f2182000700 1 -- 172.21.15.44:0/3582328516 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217400ee40 con 0x7f2184103090 2024-05-27T09:47:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.268+0000 7f218bc00700 1 -- 172.21.15.44:0/3582328516 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21841a2180 con 0x7f2184103090 2024-05-27T09:47:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.268+0000 7f2182000700 1 -- 172.21.15.44:0/3582328516 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2174003a00 con 0x7f2184103090 2024-05-27T09:47:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.268+0000 7f2182000700 1 -- 172.21.15.44:0/3582328516 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174004720 con 0x7f2184103090 2024-05-27T09:47:28.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.268+0000 7f218bc00700 1 -- 172.21.15.44:0/3582328516 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21841a9b00 con 0x7f2184103090 2024-05-27T09:47:28.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.272+0000 7f2182000700 1 -- 172.21.15.44:0/3582328516 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f217401b030 con 0x7f2184103090 2024-05-27T09:47:28.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.273+0000 7f219a788b80 1 Processor -- start 2024-05-27T09:47:28.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.273+0000 7f219a788b80 1 -- start start 2024-05-27T09:47:28.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f219a788b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c175eeaf0 0x558c17688530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f219a788b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c17502f40 0x558c17688a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f219a788b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c176825e0 0x558c176829f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f219a788b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558c175aff40 con 0x558c17502f40 2024-05-27T09:47:28.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f219a788b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558c1759fae0 con 0x558c176825e0 2024-05-27T09:47:28.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f219a788b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558c175a3b20 con 0x558c175eeaf0 2024-05-27T09:47:28.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f2189200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c17502f40 0x558c17688a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f2189c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c175eeaf0 0x558c17688530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f2189200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c17502f40 0x558c17688a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34752/0 (socket says 172.21.15.44:34752) 2024-05-27T09:47:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f2189200700 1 -- 172.21.15.44:0/547845164 learned_addr learned my addr 172.21.15.44:0/547845164 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f217be00700 1 --2- 172.21.15.44:0/547845164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c176825e0 0x558c176829f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.274+0000 7f2182000700 1 --2- 172.21.15.44:0/3582328516 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f21680b29f0 0x7f21680b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.275+0000 7f2182000700 1 -- 172.21.15.44:0/3582328516 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f21740e1fb0 con 0x7f2184103090 2024-05-27T09:47:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.275+0000 7f2189c00700 1 --2- 172.21.15.44:0/3582328516 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f21680b29f0 0x7f21680b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.275+0000 7f2189200700 1 -- 172.21.15.44:0/547845164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c175eeaf0 msgr2=0x558c17688530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.275+0000 7f2189200700 1 --2- 172.21.15.44:0/547845164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c175eeaf0 0x558c17688530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.275+0000 7f2189200700 1 -- 172.21.15.44:0/547845164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c176825e0 msgr2=0x558c176829f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.275+0000 7f2189200700 1 --2- 172.21.15.44:0/547845164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c176825e0 0x558c176829f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.275+0000 7f2189200700 1 -- 172.21.15.44:0/547845164 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c17568520 con 0x558c17502f40 2024-05-27T09:47:28.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.275+0000 7f217be00700 1 --2- 172.21.15.44:0/547845164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c176825e0 0x558c176829f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.275+0000 7f2189200700 1 --2- 172.21.15.44:0/547845164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c17502f40 0x558c17688a90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f21740a6500 tx=0x7f21740a5280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80095237c7f10b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.276+0000 7f217a000700 1 -- 172.21.15.44:0/547845164 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21740a9060 con 0x558c17502f40 2024-05-27T09:47:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.276+0000 7f217a000700 1 -- 172.21.15.44:0/547845164 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21740a8da0 con 0x558c17502f40 2024-05-27T09:47:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.276+0000 7f217a000700 1 -- 172.21.15.44:0/547845164 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21740a8a00 con 0x558c17502f40 2024-05-27T09:47:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.277+0000 7f219a788b80 1 -- 172.21.15.44:0/547845164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c17502f40 msgr2=0x558c17688a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.277+0000 7f219a788b80 1 --2- 172.21.15.44:0/547845164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c17502f40 0x558c17688a90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f21740a6500 tx=0x7f21740a5280 comp rx=0 tx=0).stop 2024-05-27T09:47:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.277+0000 7f2189c00700 1 --2- 172.21.15.44:0/3582328516 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f21680b29f0 0x7f21680b4f00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f217c009fd0 tx=0x7f217c009c00 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.277+0000 7f219a788b80 1 -- 172.21.15.44:0/547845164 shutdown_connections 2024-05-27T09:47:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.277+0000 7f219a788b80 1 --2- 172.21.15.44:0/547845164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c176825e0 0x558c176829f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.277+0000 7f219a788b80 1 --2- 172.21.15.44:0/547845164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c175eeaf0 0x558c17688530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.277+0000 7f219a788b80 1 --2- 172.21.15.44:0/547845164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c17502f40 0x558c17688a90 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.277+0000 7f219a788b80 1 -- 172.21.15.44:0/547845164 >> 172.21.15.44:0/547845164 conn(0x558c1759c490 msgr2=0x558c1767d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:28.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.277+0000 7f219a788b80 1 -- 172.21.15.44:0/547845164 shutdown_connections 2024-05-27T09:47:28.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.277+0000 7f219a788b80 1 -- 172.21.15.44:0/547845164 wait complete. 2024-05-27T09:47:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.278+0000 7f219a788b80 1 -- start start 2024-05-27T09:47:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.278+0000 7f219a788b80 1 Processor -- start 2024-05-27T09:47:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f219a788b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c17502f40 0x558c1768c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f219a788b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c175eeaf0 0x558c1768c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f219a788b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c176825e0 0x558c1768cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f219a788b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558c174a60a0 con 0x558c176825e0 2024-05-27T09:47:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f219a788b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558c17540860 con 0x558c175eeaf0 2024-05-27T09:47:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f219a788b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558c176833d0 con 0x558c17502f40 2024-05-27T09:47:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f219a788b80 10 client.0 _ll_register_callbacks cb 0x558c17549d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f217be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c175eeaf0 0x558c1768c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f217be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c175eeaf0 0x558c1768c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:32872/0 (socket says 172.21.15.44:32872) 2024-05-27T09:47:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f217be00700 1 -- 172.21.15.44:0/265383640 learned_addr learned my addr 172.21.15.44:0/265383640 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f2189200700 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c17502f40 0x558c1768c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f2189c00700 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c176825e0 0x558c1768cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.279+0000 7f217be00700 1 -- 172.21.15.44:0/265383640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c17502f40 msgr2=0x558c1768c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:28.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.280+0000 7f217be00700 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c17502f40 0x558c1768c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.280+0000 7f217be00700 1 -- 172.21.15.44:0/265383640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c176825e0 msgr2=0x558c1768cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:28.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.280+0000 7f217be00700 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c176825e0 0x558c1768cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.280+0000 7f217be00700 1 -- 172.21.15.44:0/265383640 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c174ef9d0 con 0x558c175eeaf0 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.280+0000 7f217be00700 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c175eeaf0 0x558c1768c7e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f216c012700 tx=0x7f216c012a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea6c20ff1827c94e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.280+0000 7f2172a00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.280+0000 7f2172a00700 1 -- 172.21.15.44:0/265383640 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c004050 con 0x558c175eeaf0 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.280+0000 7f219a788b80 1 -- 172.21.15.44:0/265383640 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21740a8da0 con 0x558c175eeaf0 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.280+0000 7f2172a00700 1 -- 172.21.15.44:0/265383640 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f216c0041f0 con 0x558c175eeaf0 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.281+0000 7f2172a00700 1 -- 172.21.15.44:0/265383640 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f216c0043f0 con 0x558c175eeaf0 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.281+0000 7f219a788b80 10 client.27503 fetch_fsmap learned FSMap version 38 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.281+0000 7f219a788b80 1 -- 172.21.15.44:0/265383640 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x558c17570160 con 0x558c175eeaf0 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.281+0000 7f2172a00700 1 -- 172.21.15.44:0/265383640 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21740a8da0 con 0x558c175eeaf0 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.282+0000 7f2172a00700 1 -- 172.21.15.44:0/265383640 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f216c010730 con 0x558c175eeaf0 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.282+0000 7f219a788b80 10 client.27503 fetch_fsmap finished waiting for FSMap version 38 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.282+0000 7f219a788b80 10 client.27503 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:47:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.282+0000 7f219a788b80 20 client.27503 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:28.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.282+0000 7f219a788b80 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558c1769c040 0x558c1769e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.282+0000 7f219a788b80 4 client.27503 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:28.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.282+0000 7f219a788b80 1 -- 172.21.15.44:0/265383640 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c175b1e90 con 0x558c1769c040 2024-05-27T09:47:28.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.282+0000 7f2189200700 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558c1769c040 0x558c1769e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.283+0000 7f2189200700 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558c1769c040 0x558c1769e460 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38af25030291f94f server_cookie=eed29d364f581c5d in_seq=0 out_seq=0 2024-05-27T09:47:28.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.283+0000 7f2172a00700 10 client.27503 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:28.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f2172a00700 1 -- 172.21.15.44:0/265383640 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558c175b1e90 con 0x558c1769c040 2024-05-27T09:47:28.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f2172a00700 10 client.27503 handle_command_reply: tid=0 2024-05-27T09:47:28.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f219a788b80 1 client.27503 shutdown 2024-05-27T09:47:28.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f219a788b80 1 -- 172.21.15.44:0/265383640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c175eeaf0 msgr2=0x558c1768c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f219a788b80 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c175eeaf0 0x558c1768c7e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f216c012700 tx=0x7f216c012a10 comp rx=0 tx=0).stop 2024-05-27T09:47:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f219a788b80 1 -- 172.21.15.44:0/265383640 shutdown_connections 2024-05-27T09:47:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f219a788b80 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558c1769c040 0x558c1769e460 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f219a788b80 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c176825e0 0x558c1768cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f219a788b80 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c17502f40 0x558c1768c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f219a788b80 1 --2- 172.21.15.44:0/265383640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c175eeaf0 0x558c1768c7e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f2172a00700 0 client.27503 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.284+0000 7f219a788b80 1 -- 172.21.15.44:0/265383640 >> 172.21.15.44:0/265383640 conn(0x558c1759c490 msgr2=0x558c176815d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.285+0000 7f219a788b80 1 -- 172.21.15.44:0/265383640 shutdown_connections 2024-05-27T09:47:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.285+0000 7f219a788b80 1 -- 172.21.15.44:0/265383640 wait complete. 2024-05-27T09:47:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.285+0000 7f219a788b80 20 client.27503 trim_cache size 0 max 16384 2024-05-27T09:47:28.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:27 smithi050 ceph-mon[36371]: pgmap v384: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 130 KiB/s wr, 55 op/s 2024-05-27T09:47:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:27 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:resolve 2024-05-27T09:47:28.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:27 smithi050 ceph-mon[36371]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:28.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.307+0000 7f219a788b80 1 Processor -- start 2024-05-27T09:47:28.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.307+0000 7f219a788b80 1 -- start start 2024-05-27T09:47:28.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.308+0000 7f219a788b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c1768e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.308+0000 7f219a788b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c175eeaf0 0x558c17698210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.308+0000 7f219a788b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c176825e0 0x558c1767cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.308+0000 7f219a788b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f21841b1a20 con 0x558c176825e0 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.308+0000 7f219a788b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f21841af1c0 con 0x558c17502f40 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.308+0000 7f219a788b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f21841b1350 con 0x558c175eeaf0 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.308+0000 7f2189200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c1768e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.308+0000 7f2189200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c1768e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:32878/0 (socket says 172.21.15.44:32878) 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.308+0000 7f2189200700 1 -- 172.21.15.44:0/63273680 learned_addr learned my addr 172.21.15.44:0/63273680 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.308+0000 7f2189c00700 1 --2- 172.21.15.44:0/63273680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c176825e0 0x558c1767cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.309+0000 7f217be00700 1 --2- 172.21.15.44:0/63273680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c175eeaf0 0x558c17698210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.309+0000 7f2189200700 1 -- 172.21.15.44:0/63273680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c175eeaf0 msgr2=0x558c17698210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.309+0000 7f2189200700 1 --2- 172.21.15.44:0/63273680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c175eeaf0 0x558c17698210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.309+0000 7f2189200700 1 -- 172.21.15.44:0/63273680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c176825e0 msgr2=0x558c1767cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.309+0000 7f2189200700 1 --2- 172.21.15.44:0/63273680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c176825e0 0x558c1767cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.309+0000 7f2189200700 1 -- 172.21.15.44:0/63273680 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c17699ac0 con 0x558c17502f40 2024-05-27T09:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.309+0000 7f217be00700 1 --2- 172.21.15.44:0/63273680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c175eeaf0 0x558c17698210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:47:28.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.309+0000 7f2189200700 1 --2- 172.21.15.44:0/63273680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c1768e380 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2174009fd0 tx=0x7f217400eb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dbac50f8ad8d27 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:28.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.310+0000 7f2179600700 1 -- 172.21.15.44:0/63273680 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21740a8be0 con 0x558c17502f40 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.310+0000 7f2179600700 1 -- 172.21.15.44:0/63273680 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21740a8da0 con 0x558c17502f40 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.310+0000 7f2179600700 1 -- 172.21.15.44:0/63273680 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21740e5310 con 0x558c17502f40 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.310+0000 7f219a788b80 1 -- 172.21.15.44:0/63273680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 msgr2=0x558c1768e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.310+0000 7f219a788b80 1 --2- 172.21.15.44:0/63273680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c1768e380 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2174009fd0 tx=0x7f217400eb50 comp rx=0 tx=0).stop 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.311+0000 7f219a788b80 1 -- 172.21.15.44:0/63273680 shutdown_connections 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.311+0000 7f219a788b80 1 --2- 172.21.15.44:0/63273680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c176825e0 0x558c1767cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.311+0000 7f219a788b80 1 --2- 172.21.15.44:0/63273680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c1768e380 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.311+0000 7f219a788b80 1 --2- 172.21.15.44:0/63273680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c175eeaf0 0x558c17698210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.311+0000 7f219a788b80 1 -- 172.21.15.44:0/63273680 >> 172.21.15.44:0/63273680 conn(0x558c1759c490 msgr2=0x558c17681c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.311+0000 7f219a788b80 1 -- 172.21.15.44:0/63273680 shutdown_connections 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.311+0000 7f219a788b80 1 -- 172.21.15.44:0/63273680 wait complete. 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.311+0000 7f219a788b80 1 -- start start 2024-05-27T09:47:28.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.311+0000 7f219a788b80 1 Processor -- start 2024-05-27T09:47:28.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.312+0000 7f219a788b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c17694a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.312+0000 7f219a788b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c175eeaf0 0x558c176913d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.312+0000 7f219a788b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c176825e0 0x558c17691910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.312+0000 7f219a788b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f21841b0c80 con 0x558c175eeaf0 2024-05-27T09:47:28.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.312+0000 7f219a788b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f21841af810 con 0x558c17502f40 2024-05-27T09:47:28.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.312+0000 7f219a788b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f21841afee0 con 0x558c176825e0 2024-05-27T09:47:28.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.312+0000 7f219a788b80 10 client.0 _ll_register_callbacks cb 0x558c17549d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:28.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.312+0000 7f2189200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c17694a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.312+0000 7f217be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c175eeaf0 0x558c176913d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.312+0000 7f2189200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c17694a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:32890/0 (socket says 172.21.15.44:32890) 2024-05-27T09:47:28.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.312+0000 7f2189200700 1 -- 172.21.15.44:0/1684466674 learned_addr learned my addr 172.21.15.44:0/1684466674 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:28.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.313+0000 7f2189c00700 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c176825e0 0x558c17691910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.313+0000 7f217be00700 1 -- 172.21.15.44:0/1684466674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c176825e0 msgr2=0x558c17691910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:28.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.313+0000 7f217be00700 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c176825e0 0x558c17691910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.313+0000 7f217be00700 1 -- 172.21.15.44:0/1684466674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 msgr2=0x558c17694a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:28.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.313+0000 7f217be00700 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c17694a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.313+0000 7f217be00700 1 -- 172.21.15.44:0/1684466674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c1768d090 con 0x558c175eeaf0 2024-05-27T09:47:28.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.313+0000 7f2189200700 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c17694a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:28.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.313+0000 7f217be00700 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c175eeaf0 0x558c176913d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f216c010b50 tx=0x7f216c010760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7006d579e138de6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:28.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.313+0000 7f2172a00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:47:28.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.314+0000 7f2172a00700 1 -- 172.21.15.44:0/1684466674 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c028070 con 0x558c175eeaf0 2024-05-27T09:47:28.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.314+0000 7f2172a00700 1 -- 172.21.15.44:0/1684466674 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f216c003890 con 0x558c175eeaf0 2024-05-27T09:47:28.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.314+0000 7f219a788b80 1 -- 172.21.15.44:0/1684466674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21740a8da0 con 0x558c175eeaf0 2024-05-27T09:47:28.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.314+0000 7f2172a00700 1 -- 172.21.15.44:0/1684466674 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f216c03a050 con 0x558c175eeaf0 2024-05-27T09:47:28.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.314+0000 7f219a788b80 10 client.17274 fetch_fsmap learned FSMap version 38 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.314+0000 7f219a788b80 1 -- 172.21.15.44:0/1684466674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x558c1770ff80 con 0x558c175eeaf0 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.314+0000 7f2172a00700 1 -- 172.21.15.44:0/1684466674 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21740a8da0 con 0x558c175eeaf0 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.315+0000 7f2172a00700 1 -- 172.21.15.44:0/1684466674 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f216c035030 con 0x558c175eeaf0 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.315+0000 7f219a788b80 10 client.17274 fetch_fsmap finished waiting for FSMap version 38 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.315+0000 7f219a788b80 10 client.17274 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.315+0000 7f219a788b80 20 client.17274 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.315+0000 7f219a788b80 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558c17710680 0x558c17712aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.315+0000 7f219a788b80 4 client.17274 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b3fe0bae", "release": true, "await": true}] 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.315+0000 7f219a788b80 1 -- 172.21.15.44:0/1684466674 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b3fe0bae", "release": true, "await": true}) -- 0x558c17675810 con 0x558c17710680 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.315+0000 7f2189200700 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558c17710680 0x558c17712aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f2189200700 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558c17710680 0x558c17712aa0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2358dc66b0b5976 server_cookie=8c844162cbb2a9a6 in_seq=0 out_seq=0 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f2172a00700 10 client.17274 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f2172a00700 1 -- 172.21.15.44:0/1684466674 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: -1 ) ==== 8+0+591 (crc 0 0 0) 0x558c17675810 con 0x558c17710680 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f2172a00700 10 client.17274 handle_command_reply: tid=0 2024-05-27T09:47:28.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f219a788b80 1 client.17274 shutdown 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f219a788b80 1 -- 172.21.15.44:0/1684466674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c175eeaf0 msgr2=0x558c176913d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f219a788b80 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c175eeaf0 0x558c176913d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f216c010b50 tx=0x7f216c010760 comp rx=0 tx=0).stop 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f219a788b80 1 -- 172.21.15.44:0/1684466674 shutdown_connections 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f219a788b80 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558c17710680 0x558c17712aa0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f219a788b80 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c176825e0 0x558c17691910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f219a788b80 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558c17502f40 0x558c17694a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f219a788b80 1 --2- 172.21.15.44:0/1684466674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c175eeaf0 0x558c176913d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f219a788b80 1 -- 172.21.15.44:0/1684466674 >> 172.21.15.44:0/1684466674 conn(0x558c1759c490 msgr2=0x558c1768d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.316+0000 7f2172a00700 0 client.17274 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.317+0000 7f219a788b80 1 -- 172.21.15.44:0/1684466674 shutdown_connections 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.317+0000 7f219a788b80 1 -- 172.21.15.44:0/1684466674 wait complete. 2024-05-27T09:47:28.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:28.317+0000 7f219a788b80 20 client.17274 trim_cache size 0 max 16384 2024-05-27T09:47:28.321 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:47:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 38, 2024-05-27T09:47:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:47:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 28, 2024-05-27T09:47:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:47:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "b3fe0bae": { 2024-05-27T09:47:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 28, 2024-05-27T09:47:28.321 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 2.4, 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "CANCELED", 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 142.3, 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": true, 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 13.6 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:28.322 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:28.323 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:28.323 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:28.323 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:28.323 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:47:28.323 INFO:teuthology.orchestra.run.smithi044.stderr:Error EPERM: 2024-05-27T09:47:28.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:27 smithi189 ceph-mon[36671]: pgmap v384: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 130 KiB/s wr, 55 op/s 2024-05-27T09:47:28.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:27 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:resolve 2024-05-27T09:47:28.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:27 smithi189 ceph-mon[36671]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.070+0000 7f502ea00640 1 -- 172.21.15.44:0/3058492462 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55be5376f2c0 con 0x55be533f41c0 2024-05-27T09:47:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.070+0000 7f502ea00640 10 client.27472 handle_command_reply: tid=0 2024-05-27T09:47:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.070+0000 7f504e9f8740 1 client.27472 shutdown 2024-05-27T09:47:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.071+0000 7f504e9f8740 1 -- 172.21.15.44:0/3058492462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 msgr2=0x55be5381ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.071+0000 7f504e9f8740 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 0x55be5381ebc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5028022e90 tx=0x7f5028005f90 comp rx=0 tx=0).stop 2024-05-27T09:47:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.071+0000 7f504e9f8740 1 -- 172.21.15.44:0/3058492462 shutdown_connections 2024-05-27T09:47:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.071+0000 7f504e9f8740 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55be533f41c0 0x55be53891440 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.071+0000 7f504e9f8740 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55be534601b0 0x55be5381f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.071+0000 7f504e9f8740 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55be5343f4d0 0x55be5381f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.071+0000 7f504e9f8740 1 --2- 172.21.15.44:0/3058492462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be534372f0 0x55be5381ebc0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.071+0000 7f504e9f8740 1 -- 172.21.15.44:0/3058492462 >> 172.21.15.44:0/3058492462 conn(0x55be53454300 msgr2=0x55be5380fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.071+0000 7f502ea00640 0 client.27472 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.072+0000 7f504e9f8740 1 -- 172.21.15.44:0/3058492462 shutdown_connections 2024-05-27T09:47:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.072+0000 7f504e9f8740 1 -- 172.21.15.44:0/3058492462 wait complete. 2024-05-27T09:47:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.072+0000 7f504e9f8740 20 client.27472 trim_cache size 0 max 16384 2024-05-27T09:47:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:47:29.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:47:29.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:47:29.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 -- 172.21.15.44:0/1167079355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f50200b2950 msgr2=0x7f50200b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:29.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f50200b2950 0x7f50200b4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f50380116b0 tx=0x7f5038009500 comp rx=0 tx=0).stop 2024-05-27T09:47:29.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 -- 172.21.15.44:0/1167079355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5040109c80 msgr2=0x7f50401a0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:29.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5040109c80 0x7f50401a0b20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5030009230 tx=0x7f5030003b90 comp rx=0 tx=0).stop 2024-05-27T09:47:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 -- 172.21.15.44:0/1167079355 shutdown_connections 2024-05-27T09:47:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f504010a120 0x7f504019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5040109c80 0x7f50401a0b20 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f50200b2950 0x7f50200b4e60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 --2- 172.21.15.44:0/1167079355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f50401043d0 0x7f50401a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 -- 172.21.15.44:0/1167079355 >> 172.21.15.44:0/1167079355 conn(0x7f50400ff340 msgr2=0x7f50400ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 -- 172.21.15.44:0/1167079355 shutdown_connections 2024-05-27T09:47:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:29.074+0000 7f5047a00640 1 -- 172.21.15.44:0/1167079355 wait complete. 2024-05-27T09:47:29.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:28 smithi044 ceph-mon[31618]: reconnect by client.25325 172.21.15.44:0/2218285324 after 0.00199995 2024-05-27T09:47:29.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:28 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:reconnect 2024-05-27T09:47:29.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:28 smithi044 ceph-mon[31618]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:29.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:28 smithi044 ceph-mon[31618]: reconnect by client.25070 v1:172.21.15.44:0/1227851225 after 0.0109997 2024-05-27T09:47:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:28 smithi050 ceph-mon[36371]: reconnect by client.25325 172.21.15.44:0/2218285324 after 0.00199995 2024-05-27T09:47:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:28 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:reconnect 2024-05-27T09:47:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:28 smithi050 ceph-mon[36371]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:29.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:28 smithi050 ceph-mon[36371]: reconnect by client.25070 v1:172.21.15.44:0/1227851225 after 0.0109997 2024-05-27T09:47:29.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:28 smithi189 ceph-mon[36671]: reconnect by client.25325 172.21.15.44:0/2218285324 after 0.00199995 2024-05-27T09:47:29.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:28 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:reconnect 2024-05-27T09:47:29.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:28 smithi189 ceph-mon[36671]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:29.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:28 smithi189 ceph-mon[36671]: reconnect by client.25070 v1:172.21.15.44:0/1227851225 after 0.0109997 2024-05-27T09:47:29.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:47:29.458 WARNING:tasks.quiescer.fs.[cephfs]:Quiesce set got cancelled (cap = 10).Won't raise an error since this could be a failover, will wait for the next quiesce attempt 2024-05-27T09:47:29.458 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.3 2024-05-27T09:47:29.458 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:47:29.940 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:30.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:29 smithi044 ceph-mon[31618]: pgmap v385: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 130 KiB/s wr, 55 op/s 2024-05-27T09:47:30.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:29 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:rejoin 2024-05-27T09:47:30.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:29 smithi044 ceph-mon[31618]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:30.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:29 smithi044 ceph-mon[31618]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T09:47:30.293 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:47:30 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b[66721]: 2024-05-27T09:47:30.128+0000 7fc6ea134700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-05-27T09:47:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:29 smithi050 ceph-mon[36371]: pgmap v385: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 130 KiB/s wr, 55 op/s 2024-05-27T09:47:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:29 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:rejoin 2024-05-27T09:47:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:29 smithi050 ceph-mon[36371]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:30.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:29 smithi050 ceph-mon[36371]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T09:47:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:29 smithi189 ceph-mon[36671]: pgmap v385: 97 pgs: 97 active+clean; 317 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 130 KiB/s wr, 55 op/s 2024-05-27T09:47:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:29 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:rejoin 2024-05-27T09:47:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:29 smithi189 ceph-mon[36671]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:29 smithi189 ceph-mon[36671]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T09:47:31.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:30 smithi044 ceph-mon[31618]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T09:47:31.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:30 smithi044 ceph-mon[31618]: Cluster is now healthy 2024-05-27T09:47:31.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:30 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:active 2024-05-27T09:47:31.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:30 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:31.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:30 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:47:31.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:30 smithi050 ceph-mon[36371]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T09:47:31.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:30 smithi050 ceph-mon[36371]: Cluster is now healthy 2024-05-27T09:47:31.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:30 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:active 2024-05-27T09:47:31.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:30 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:31.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:30 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:47:31.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:30 smithi189 ceph-mon[36671]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T09:47:31.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:30 smithi189 ceph-mon[36671]: Cluster is now healthy 2024-05-27T09:47:31.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:30 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] up:active 2024-05-27T09:47:31.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:30 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T09:47:31.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:30 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:47:32.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:32 smithi050 ceph-mon[36371]: pgmap v386: 97 pgs: 97 active+clean; 313 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 16 KiB/s wr, 21 op/s 2024-05-27T09:47:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:32 smithi189 ceph-mon[36671]: pgmap v386: 97 pgs: 97 active+clean; 313 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 16 KiB/s wr, 21 op/s 2024-05-27T09:47:32.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.344+0000 7efc4f200700 1 -- 172.21.15.44:0/1960217188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc48105510 msgr2=0x7efc48105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:32.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.344+0000 7efc4f200700 1 --2- 172.21.15.44:0/1960217188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc48105510 0x7efc48105920 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7efc380098f0 tx=0x7efc38009c00 comp rx=0 tx=0).stop 2024-05-27T09:47:32.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.344+0000 7efc4f200700 1 -- 172.21.15.44:0/1960217188 shutdown_connections 2024-05-27T09:47:32.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.344+0000 7efc4f200700 1 --2- 172.21.15.44:0/1960217188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc4810e8f0 0x7efc48110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.344+0000 7efc4f200700 1 --2- 172.21.15.44:0/1960217188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc48105510 0x7efc48105920 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.344+0000 7efc4f200700 1 --2- 172.21.15.44:0/1960217188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc48105e60 0x7efc4810e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.344+0000 7efc4f200700 1 -- 172.21.15.44:0/1960217188 >> 172.21.15.44:0/1960217188 conn(0x7efc480fedb0 msgr2=0x7efc481011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:32.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.344+0000 7efc4f200700 1 -- 172.21.15.44:0/1960217188 shutdown_connections 2024-05-27T09:47:32.347 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:32 smithi044 ceph-mon[31618]: pgmap v386: 97 pgs: 97 active+clean; 313 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 16 KiB/s wr, 21 op/s 2024-05-27T09:47:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.344+0000 7efc4f200700 1 -- 172.21.15.44:0/1960217188 wait complete. 2024-05-27T09:47:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.345+0000 7efc4f200700 1 Processor -- start 2024-05-27T09:47:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.345+0000 7efc4f200700 1 -- start start 2024-05-27T09:47:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.346+0000 7efc4f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc48105510 0x7efc4819d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.346+0000 7efc4f200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc48105e60 0x7efc4819d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.346+0000 7efc4f200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc4810e8f0 0x7efc481a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.346+0000 7efc4f200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efc48114470 con 0x7efc48105e60 2024-05-27T09:47:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.346+0000 7efc4f200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efc481142f0 con 0x7efc4810e8f0 2024-05-27T09:47:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.346+0000 7efc4f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efc481145f0 con 0x7efc48105510 2024-05-27T09:47:32.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.346+0000 7efc47e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc48105510 0x7efc4819d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:32.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.346+0000 7efc47e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc48105510 0x7efc4819d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51434/0 (socket says 172.21.15.44:51434) 2024-05-27T09:47:32.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.346+0000 7efc47e00700 1 -- 172.21.15.44:0/189220870 learned_addr learned my addr 172.21.15.44:0/189220870 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:32.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.346+0000 7efc47400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc48105e60 0x7efc4819d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:32.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.347+0000 7efc4d200700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc4810e8f0 0x7efc481a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:32.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.347+0000 7efc47e00700 1 -- 172.21.15.44:0/189220870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc4810e8f0 msgr2=0x7efc481a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:32.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.347+0000 7efc47e00700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc4810e8f0 0x7efc481a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:32.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.347+0000 7efc47e00700 1 -- 172.21.15.44:0/189220870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc48105e60 msgr2=0x7efc4819d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:32.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.347+0000 7efc47e00700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc48105e60 0x7efc4819d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:32.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.347+0000 7efc47e00700 1 -- 172.21.15.44:0/189220870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc481a1f30 con 0x7efc48105510 2024-05-27T09:47:32.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.347+0000 7efc47400700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc48105e60 0x7efc4819d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:47:32.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.347+0000 7efc47e00700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc48105510 0x7efc4819d060 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efc38009230 tx=0x7efc3800fb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:32.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.347+0000 7efc44c00700 1 -- 172.21.15.44:0/189220870 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc38005220 con 0x7efc48105510 2024-05-27T09:47:32.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.347+0000 7efc44c00700 1 -- 172.21.15.44:0/189220870 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc3800b560 con 0x7efc48105510 2024-05-27T09:47:32.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.347+0000 7efc4f200700 1 -- 172.21.15.44:0/189220870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc481a21c0 con 0x7efc48105510 2024-05-27T09:47:32.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.348+0000 7efc44c00700 1 -- 172.21.15.44:0/189220870 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc38003b40 con 0x7efc48105510 2024-05-27T09:47:32.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.348+0000 7efc4f200700 1 -- 172.21.15.44:0/189220870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc481a26a0 con 0x7efc48105510 2024-05-27T09:47:32.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.349+0000 7efc4f200700 1 -- 172.21.15.44:0/189220870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc480512b0 con 0x7efc48105510 2024-05-27T09:47:32.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.352+0000 7efc44c00700 1 -- 172.21.15.44:0/189220870 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7efc38003d00 con 0x7efc48105510 2024-05-27T09:47:32.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.354+0000 7efc44c00700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efc2c0b29f0 0x7efc2c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:32.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.354+0000 7efc44c00700 1 -- 172.21.15.44:0/189220870 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7efc380e36f0 con 0x7efc48105510 2024-05-27T09:47:32.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.354+0000 7efc47400700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efc2c0b29f0 0x7efc2c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:32.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.355+0000 7efc47400700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efc2c0b29f0 0x7efc2c0b4f00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efc48102270 tx=0x7efc34009310 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:32.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.358+0000 7efc44c00700 1 -- 172.21.15.44:0/189220870 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efc38013030 con 0x7efc48105510 2024-05-27T09:47:32.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.654+0000 7efc4f200700 1 -- 172.21.15.44:0/189220870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc481a9ba0 con 0x7efc48105510 2024-05-27T09:47:32.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.654+0000 7efc44c00700 1 -- 172.21.15.44:0/189220870 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11868 (secure 0 0 0) 0x7efc380ab0d0 con 0x7efc48105510 2024-05-27T09:47:32.656 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:47:32.656 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:47:31.318532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24542,24503,24520]},"id":1}]} 2024-05-27T09:47:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.657+0000 7efc4f200700 1 -- 172.21.15.44:0/189220870 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efc2c0b29f0 msgr2=0x7efc2c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.657+0000 7efc4f200700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efc2c0b29f0 0x7efc2c0b4f00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efc48102270 tx=0x7efc34009310 comp rx=0 tx=0).stop 2024-05-27T09:47:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.657+0000 7efc4f200700 1 -- 172.21.15.44:0/189220870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc48105510 msgr2=0x7efc4819d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.657+0000 7efc4f200700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc48105510 0x7efc4819d060 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efc38009230 tx=0x7efc3800fb20 comp rx=0 tx=0).stop 2024-05-27T09:47:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.658+0000 7efc4f200700 1 -- 172.21.15.44:0/189220870 shutdown_connections 2024-05-27T09:47:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.658+0000 7efc4f200700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7efc2c0b29f0 0x7efc2c0b4f00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.658+0000 7efc4f200700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc4810e8f0 0x7efc481a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.658+0000 7efc4f200700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc48105510 0x7efc4819d060 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.658+0000 7efc4f200700 1 --2- 172.21.15.44:0/189220870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efc48105e60 0x7efc4819d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.658+0000 7efc4f200700 1 -- 172.21.15.44:0/189220870 >> 172.21.15.44:0/189220870 conn(0x7efc480fedb0 msgr2=0x7efc480ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.658+0000 7efc4f200700 1 -- 172.21.15.44:0/189220870 shutdown_connections 2024-05-27T09:47:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:32.658+0000 7efc4f200700 1 -- 172.21.15.44:0/189220870 wait complete. 2024-05-27T09:47:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 42 2024-05-27T09:47:33.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:33 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:47:33.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:33 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/189220870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:33.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:33 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:47:33.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:33 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/189220870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:33.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:33 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:47:33.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:33 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/189220870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:33.625 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db / --timeout 90 --expiration 142.3 --await-for 45.0' 2024-05-27T09:47:33.625 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db / --timeout 90 --expiration 142.3 --await-for 45.0 2024-05-27T09:47:34.045 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:34.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:34 smithi050 ceph-mon[36371]: pgmap v387: 97 pgs: 97 active+clean; 313 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 16 KiB/s wr, 21 op/s 2024-05-27T09:47:34.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:34 smithi189 ceph-mon[36671]: pgmap v387: 97 pgs: 97 active+clean; 313 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 16 KiB/s wr, 21 op/s 2024-05-27T09:47:34.343 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:34 smithi044 ceph-mon[31618]: pgmap v387: 97 pgs: 97 active+clean; 313 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 16 KiB/s wr, 21 op/s 2024-05-27T09:47:36.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:36 smithi050 ceph-mon[36371]: pgmap v388: 97 pgs: 97 active+clean; 313 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 16 KiB/s wr, 21 op/s 2024-05-27T09:47:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:36 smithi189 ceph-mon[36671]: pgmap v388: 97 pgs: 97 active+clean; 313 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 16 KiB/s wr, 21 op/s 2024-05-27T09:47:36.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:36 smithi044 ceph-mon[31618]: pgmap v388: 97 pgs: 97 active+clean; 313 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 16 KiB/s wr, 21 op/s 2024-05-27T09:47:36.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.384+0000 7fedc3c00700 1 -- 172.21.15.44:0/1856577719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedbc10e8f0 msgr2=0x7fedbc110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:36.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.384+0000 7fedc3c00700 1 --2- 172.21.15.44:0/1856577719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedbc10e8f0 0x7fedbc110db0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009990 tx=0x7fedb4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.385+0000 7fedc3c00700 1 -- 172.21.15.44:0/1856577719 shutdown_connections 2024-05-27T09:47:36.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.385+0000 7fedc3c00700 1 --2- 172.21.15.44:0/1856577719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedbc10e8f0 0x7fedbc110db0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.385+0000 7fedc3c00700 1 --2- 172.21.15.44:0/1856577719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc105510 0x7fedbc105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.385+0000 7fedc3c00700 1 --2- 172.21.15.44:0/1856577719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedbc105e60 0x7fedbc10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.385+0000 7fedc3c00700 1 -- 172.21.15.44:0/1856577719 >> 172.21.15.44:0/1856577719 conn(0x7fedbc0fedb0 msgr2=0x7fedbc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:36.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.385+0000 7fedc3c00700 1 -- 172.21.15.44:0/1856577719 shutdown_connections 2024-05-27T09:47:36.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.386+0000 7fedc3c00700 1 -- 172.21.15.44:0/1856577719 wait complete. 2024-05-27T09:47:36.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.388+0000 7fedc3c00700 1 Processor -- start 2024-05-27T09:47:36.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.388+0000 7fedc3c00700 1 -- start start 2024-05-27T09:47:36.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.389+0000 7fedc3c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedbc105510 0x7fedbc19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.389+0000 7fedc3c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc105e60 0x7fedbc19d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.389+0000 7fedc3c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedbc10e8f0 0x7fedbc1a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.389+0000 7fedc3c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc114480 con 0x7fedbc105510 2024-05-27T09:47:36.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.389+0000 7fedc3c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc114300 con 0x7fedbc10e8f0 2024-05-27T09:47:36.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.389+0000 7fedc3c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc114600 con 0x7fedbc105e60 2024-05-27T09:47:36.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.389+0000 7fedb3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc105e60 0x7fedbc19d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.389+0000 7fedb3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc105e60 0x7fedbc19d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45478/0 (socket says 172.21.15.44:45478) 2024-05-27T09:47:36.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.389+0000 7fedb3e00700 1 -- 172.21.15.44:0/2849887161 learned_addr learned my addr 172.21.15.44:0/2849887161 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:36.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.390+0000 7fedc1c00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedbc10e8f0 0x7fedbc1a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.390+0000 7fedb3e00700 1 -- 172.21.15.44:0/2849887161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedbc10e8f0 msgr2=0x7fedbc1a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:36.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.391+0000 7fedc1200700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedbc105510 0x7fedbc19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.391+0000 7fedb3e00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedbc10e8f0 0x7fedbc1a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.391+0000 7fedb3e00700 1 -- 172.21.15.44:0/2849887161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedbc105510 msgr2=0x7fedbc19cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:36.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.391+0000 7fedb3e00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedbc105510 0x7fedbc19cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.391+0000 7fedb3e00700 1 -- 172.21.15.44:0/2849887161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedbc1a1e80 con 0x7fedbc105e60 2024-05-27T09:47:36.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.391+0000 7fedc1c00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedbc10e8f0 0x7fedbc1a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:47:36.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.392+0000 7fedb3e00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc105e60 0x7fedbc19d550 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feda80060b0 tx=0x7feda8007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:36.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.392+0000 7fedc1200700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedbc105510 0x7fedbc19cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:47:36.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.392+0000 7fedba000700 1 -- 172.21.15.44:0/2849887161 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda801a070 con 0x7fedbc105e60 2024-05-27T09:47:36.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.392+0000 7fedba000700 1 -- 172.21.15.44:0/2849887161 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda800de20 con 0x7fedbc105e60 2024-05-27T09:47:36.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.392+0000 7fedba000700 1 -- 172.21.15.44:0/2849887161 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda8014700 con 0x7fedbc105e60 2024-05-27T09:47:36.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.393+0000 7fedc3c00700 1 -- 172.21.15.44:0/2849887161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedbc1a2170 con 0x7fedbc105e60 2024-05-27T09:47:36.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.393+0000 7fedc3c00700 1 -- 172.21.15.44:0/2849887161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedbc1a9a90 con 0x7fedbc105e60 2024-05-27T09:47:36.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.397+0000 7fedba000700 1 -- 172.21.15.44:0/2849887161 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7feda801d070 con 0x7fedbc105e60 2024-05-27T09:47:36.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.398+0000 7fedd2795b80 1 Processor -- start 2024-05-27T09:47:36.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.398+0000 7fedd2795b80 1 -- start start 2024-05-27T09:47:36.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.399+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c3ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.399+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 0x55e292c34f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.399+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c354d0 0x55e292c358e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.399+0000 7fedd2795b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e292b62fd0 con 0x55e292c3b380 2024-05-27T09:47:36.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.399+0000 7fedd2795b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e292b52b70 con 0x55e292c354d0 2024-05-27T09:47:36.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.399+0000 7fedd2795b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e292b56bb0 con 0x55e292ab5fd0 2024-05-27T09:47:36.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.399+0000 7fedba000700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fed980b2a70 0x7fed980b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.399+0000 7fedc1c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c3ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.399+0000 7fedb3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c354d0 0x55e292c358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.399+0000 7fedc1c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c3ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45492/0 (socket says 172.21.15.44:45492) 2024-05-27T09:47:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.399+0000 7fedc1c00700 1 -- 172.21.15.44:0/924359 learned_addr learned my addr 172.21.15.44:0/924359 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.400+0000 7fedba000700 1 -- 172.21.15.44:0/2849887161 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7feda80e1ff0 con 0x7fedbc105e60 2024-05-27T09:47:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.400+0000 7fedc1c00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fed980b2a70 0x7fed980b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.400+0000 7fedb3e00700 1 -- 172.21.15.44:0/924359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 msgr2=0x55e292c3ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.400+0000 7fedb3e00700 1 --2- 172.21.15.44:0/924359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c3ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.400+0000 7fedb3e00700 1 -- 172.21.15.44:0/924359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 msgr2=0x55e292c34f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.400+0000 7fedb3e00700 1 --2- 172.21.15.44:0/924359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 0x55e292c34f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.400+0000 7fedb3e00700 1 -- 172.21.15.44:0/924359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e292b1b5b0 con 0x55e292c354d0 2024-05-27T09:47:36.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.400+0000 7fedb3e00700 1 --2- 172.21.15.44:0/924359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c354d0 0x55e292c358e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7feda80df710 tx=0x7feda800dfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbc600ddd9b14c18 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:36.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.400+0000 7fedb2000700 1 -- 172.21.15.44:0/924359 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda80dfe60 con 0x55e292c354d0 2024-05-27T09:47:36.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.401+0000 7fedb2000700 1 -- 172.21.15.44:0/924359 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda80a99a0 con 0x55e292c354d0 2024-05-27T09:47:36.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.401+0000 7fedb2000700 1 -- 172.21.15.44:0/924359 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda80e40c0 con 0x55e292c354d0 2024-05-27T09:47:36.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.401+0000 7fedd2795b80 1 -- 172.21.15.44:0/924359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c354d0 msgr2=0x55e292c358e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:36.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.401+0000 7fedd2795b80 1 --2- 172.21.15.44:0/924359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c354d0 0x55e292c358e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7feda80df710 tx=0x7feda800dfd0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.401+0000 7fedc1c00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fed980b2a70 0x7fed980b4f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fedb400b2e0 tx=0x7fedb4009ca0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:36.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.402+0000 7fedd2795b80 1 -- 172.21.15.44:0/924359 shutdown_connections 2024-05-27T09:47:36.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.402+0000 7fedd2795b80 1 --2- 172.21.15.44:0/924359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c354d0 0x55e292c358e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.402+0000 7fedd2795b80 1 --2- 172.21.15.44:0/924359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c3ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.402+0000 7fedd2795b80 1 --2- 172.21.15.44:0/924359 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 0x55e292c34f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.402+0000 7fedd2795b80 1 -- 172.21.15.44:0/924359 >> 172.21.15.44:0/924359 conn(0x55e292b16150 msgr2=0x55e292c30190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:36.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.402+0000 7fedd2795b80 1 -- 172.21.15.44:0/924359 shutdown_connections 2024-05-27T09:47:36.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.402+0000 7fedd2795b80 1 -- 172.21.15.44:0/924359 wait complete. 2024-05-27T09:47:36.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.402+0000 7fedd2795b80 1 -- start start 2024-05-27T09:47:36.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.403+0000 7fedd2795b80 1 Processor -- start 2024-05-27T09:47:36.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.403+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c3eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.403+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c354d0 0x55e292c3f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 0x55e292c3f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedd2795b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e292a59130 con 0x55e292c3b380 2024-05-27T09:47:36.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedd2795b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e292af38f0 con 0x55e292c354d0 2024-05-27T09:47:36.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedd2795b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e292c41e50 con 0x55e292ab5fd0 2024-05-27T09:47:36.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedd2795b80 10 client.0 _ll_register_callbacks cb 0x55e292afce00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:36.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedc1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c3eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedc1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c3eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45500/0 (socket says 172.21.15.44:45500) 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedc1200700 1 -- 172.21.15.44:0/1722265290 learned_addr learned my addr 172.21.15.44:0/1722265290 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedc1c00700 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 0x55e292c3f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedb3e00700 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c354d0 0x55e292c3f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedc1200700 1 -- 172.21.15.44:0/1722265290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c354d0 msgr2=0x55e292c3f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedc1200700 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c354d0 0x55e292c3f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedc1200700 1 -- 172.21.15.44:0/1722265290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 msgr2=0x55e292c3f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedc1200700 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 0x55e292c3f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.404+0000 7fedc1200700 1 -- 172.21.15.44:0/1722265290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e292aa2a60 con 0x55e292ab5fd0 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.405+0000 7fedc1200700 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c3eac0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7feda00105d0 tx=0x7feda00108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=493cf8bb5ee92a6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.405+0000 7fedaee00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:36.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.405+0000 7fedaee00700 1 -- 172.21.15.44:0/1722265290 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0012860 con 0x55e292ab5fd0 2024-05-27T09:47:36.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.405+0000 7fedd2795b80 1 -- 172.21.15.44:0/1722265290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feda80a99a0 con 0x55e292ab5fd0 2024-05-27T09:47:36.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.405+0000 7fedaee00700 1 -- 172.21.15.44:0/1722265290 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7feda0012a00 con 0x55e292ab5fd0 2024-05-27T09:47:36.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.406+0000 7fedaee00700 1 -- 172.21.15.44:0/1722265290 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7feda0012c00 con 0x55e292ab5fd0 2024-05-27T09:47:36.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.406+0000 7fedd2795b80 10 client.27520 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:36.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.406+0000 7fedd2795b80 1 -- 172.21.15.44:0/1722265290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e292b231f0 con 0x55e292ab5fd0 2024-05-27T09:47:36.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.406+0000 7fedaee00700 1 -- 172.21.15.44:0/1722265290 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda80a99a0 con 0x55e292ab5fd0 2024-05-27T09:47:36.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.406+0000 7fedaee00700 1 -- 172.21.15.44:0/1722265290 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7feda000f040 con 0x55e292ab5fd0 2024-05-27T09:47:36.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.407+0000 7fedd2795b80 10 client.27520 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:36.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.407+0000 7fedd2795b80 10 client.27520 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:47:36.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.407+0000 7fedd2795b80 20 client.27520 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:36.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.407+0000 7fedd2795b80 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e292c4ecb0 0x55e292c51130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.407+0000 7fedd2795b80 4 client.27520 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:36.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.407+0000 7fedd2795b80 1 -- 172.21.15.44:0/1722265290 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e292b64f20 con 0x55e292c4ecb0 2024-05-27T09:47:36.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.407+0000 7fedb3e00700 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e292c4ecb0 0x55e292c51130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.408+0000 7fedaee00700 10 client.27520 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.408+0000 7fedb3e00700 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e292c4ecb0 0x55e292c51130 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58e31473e51ca74e server_cookie=a222cb53ac1d3423 in_seq=0 out_seq=0 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.408+0000 7fedaee00700 1 -- 172.21.15.44:0/1722265290 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e292b64f20 con 0x55e292c4ecb0 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.408+0000 7fedaee00700 10 client.27520 handle_command_reply: tid=0 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 1 client.27520 shutdown 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 1 -- 172.21.15.44:0/1722265290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 msgr2=0x55e292c3eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c3eac0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7feda00105d0 tx=0x7feda00108e0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 1 -- 172.21.15.44:0/1722265290 shutdown_connections 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e292c4ecb0 0x55e292c51130 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 0x55e292c3f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c3eac0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 1 --2- 172.21.15.44:0/1722265290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c354d0 0x55e292c3f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 1 -- 172.21.15.44:0/1722265290 >> 172.21.15.44:0/1722265290 conn(0x55e292b16150 msgr2=0x55e292c33da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:36.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedaee00700 0 client.27520 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:36.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 1 -- 172.21.15.44:0/1722265290 shutdown_connections 2024-05-27T09:47:36.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 1 -- 172.21.15.44:0/1722265290 wait complete. 2024-05-27T09:47:36.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.409+0000 7fedd2795b80 20 client.27520 trim_cache size 0 max 16384 2024-05-27T09:47:36.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.432+0000 7fedd2795b80 1 Processor -- start 2024-05-27T09:47:36.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.432+0000 7fedd2795b80 1 -- start start 2024-05-27T09:47:36.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.432+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292ab5fd0 0x55e292c4aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.432+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292c354d0 0x55e292c344b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 0x55e292c568f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedd2795b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1b3b70 con 0x55e292c3b380 2024-05-27T09:47:36.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedd2795b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1b13d0 con 0x55e292ab5fd0 2024-05-27T09:47:36.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedd2795b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1af240 con 0x55e292c354d0 2024-05-27T09:47:36.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedc1200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292ab5fd0 0x55e292c4aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedc1200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292ab5fd0 0x55e292c4aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:44188/0 (socket says 172.21.15.44:44188) 2024-05-27T09:47:36.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedc1200700 1 -- 172.21.15.44:0/3864867386 learned_addr learned my addr 172.21.15.44:0/3864867386 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:36.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedb3e00700 1 --2- 172.21.15.44:0/3864867386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292c354d0 0x55e292c344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedc1c00700 1 --2- 172.21.15.44:0/3864867386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 0x55e292c568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedc1200700 1 -- 172.21.15.44:0/3864867386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292c354d0 msgr2=0x55e292c344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:36.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedc1200700 1 --2- 172.21.15.44:0/3864867386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292c354d0 0x55e292c344b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedc1200700 1 -- 172.21.15.44:0/3864867386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 msgr2=0x55e292c568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:36.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedc1200700 1 --2- 172.21.15.44:0/3864867386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 0x55e292c568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.433+0000 7fedc1200700 1 -- 172.21.15.44:0/3864867386 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e292c3f8b0 con 0x55e292ab5fd0 2024-05-27T09:47:36.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.434+0000 7fedc1200700 1 --2- 172.21.15.44:0/3864867386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292ab5fd0 0x55e292c4aeb0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7feda0019f60 tx=0x7feda0010cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81edf57b5369375f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:36.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.434+0000 7fedb1600700 1 -- 172.21.15.44:0/3864867386 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0030020 con 0x55e292ab5fd0 2024-05-27T09:47:36.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.434+0000 7fedb1600700 1 -- 172.21.15.44:0/3864867386 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda00110e0 con 0x55e292ab5fd0 2024-05-27T09:47:36.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.434+0000 7fedb1600700 1 -- 172.21.15.44:0/3864867386 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0028040 con 0x55e292ab5fd0 2024-05-27T09:47:36.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.435+0000 7fedd2795b80 1 -- 172.21.15.44:0/3864867386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292ab5fd0 msgr2=0x55e292c4aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:36.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.435+0000 7fedd2795b80 1 --2- 172.21.15.44:0/3864867386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292ab5fd0 0x55e292c4aeb0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7feda0019f60 tx=0x7feda0010cf0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.435+0000 7fedd2795b80 1 -- 172.21.15.44:0/3864867386 shutdown_connections 2024-05-27T09:47:36.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.435+0000 7fedd2795b80 1 --2- 172.21.15.44:0/3864867386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c3b380 0x55e292c568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.435+0000 7fedd2795b80 1 --2- 172.21.15.44:0/3864867386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292ab5fd0 0x55e292c4aeb0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.435+0000 7fedd2795b80 1 --2- 172.21.15.44:0/3864867386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292c354d0 0x55e292c344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.435+0000 7fedd2795b80 1 -- 172.21.15.44:0/3864867386 >> 172.21.15.44:0/3864867386 conn(0x55e292b16150 msgr2=0x55e292c33da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:36.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.435+0000 7fedd2795b80 1 -- 172.21.15.44:0/3864867386 shutdown_connections 2024-05-27T09:47:36.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.435+0000 7fedd2795b80 1 -- 172.21.15.44:0/3864867386 wait complete. 2024-05-27T09:47:36.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.436+0000 7fedd2795b80 1 -- start start 2024-05-27T09:47:36.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.436+0000 7fedd2795b80 1 Processor -- start 2024-05-27T09:47:36.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.436+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c470a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.436+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c354d0 0x55e292c47600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedd2795b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c3b380 0x55e292c44070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedd2795b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1af890 con 0x55e292c354d0 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedd2795b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1b0d00 con 0x55e292c3b380 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedd2795b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1aff60 con 0x55e292ab5fd0 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedd2795b80 10 client.0 _ll_register_callbacks cb 0x55e292afce00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedc1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedc1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c470a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45510/0 (socket says 172.21.15.44:45510) 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedc1200700 1 -- 172.21.15.44:0/2898932629 learned_addr learned my addr 172.21.15.44:0/2898932629 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedc1c00700 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c3b380 0x55e292c44070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedb3e00700 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c354d0 0x55e292c47600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedc1200700 1 -- 172.21.15.44:0/2898932629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c3b380 msgr2=0x55e292c44070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedc1200700 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c3b380 0x55e292c44070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedc1200700 1 -- 172.21.15.44:0/2898932629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c354d0 msgr2=0x55e292c47600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:36.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedc1200700 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c354d0 0x55e292c47600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:36.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.437+0000 7fedc1200700 1 -- 172.21.15.44:0/2898932629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e292c44660 con 0x55e292ab5fd0 2024-05-27T09:47:36.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.438+0000 7fedc1200700 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c470a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7feda001e000 tx=0x7feda0009de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d85f86bced931811 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:36.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.438+0000 7fedaee00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:36.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.438+0000 7fedaee00700 1 -- 172.21.15.44:0/2898932629 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0030030 con 0x55e292ab5fd0 2024-05-27T09:47:36.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.438+0000 7fedaee00700 1 -- 172.21.15.44:0/2898932629 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7feda0012660 con 0x55e292ab5fd0 2024-05-27T09:47:36.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.438+0000 7fedd2795b80 1 -- 172.21.15.44:0/2898932629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feda00110e0 con 0x55e292ab5fd0 2024-05-27T09:47:36.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.438+0000 7fedaee00700 1 -- 172.21.15.44:0/2898932629 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7feda0028040 con 0x55e292ab5fd0 2024-05-27T09:47:36.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.438+0000 7fedd2795b80 10 client.27526 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:36.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.438+0000 7fedd2795b80 1 -- 172.21.15.44:0/2898932629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e292cb9fd0 con 0x55e292ab5fd0 2024-05-27T09:47:36.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.439+0000 7fedaee00700 1 -- 172.21.15.44:0/2898932629 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda00110e0 con 0x55e292ab5fd0 2024-05-27T09:47:36.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.439+0000 7fedaee00700 1 -- 172.21.15.44:0/2898932629 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7feda001b480 con 0x55e292ab5fd0 2024-05-27T09:47:36.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.439+0000 7fedd2795b80 10 client.27526 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:36.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.439+0000 7fedd2795b80 10 client.27526 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:47:36.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.439+0000 7fedd2795b80 20 client.27526 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:36.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.439+0000 7fedd2795b80 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e292cba5d0 0x55e292cbc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:36.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.439+0000 7fedd2795b80 4 client.27526 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.3, "await_for": 45.0}] 2024-05-27T09:47:36.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.439+0000 7fedd2795b80 1 -- 172.21.15.44:0/2898932629 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.3, "await_for": 45.0}) -- 0x55e292b5e700 con 0x55e292cba5d0 2024-05-27T09:47:36.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.439+0000 7fedb3e00700 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e292cba5d0 0x55e292cbc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:36.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.440+0000 7fedaee00700 10 client.27526 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:36.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:36.440+0000 7fedb3e00700 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e292cba5d0 0x55e292cbc9f0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80937688204a2abe server_cookie=762c685ae23e7cc4 in_seq=0 out_seq=0 2024-05-27T09:47:36.815 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:47:37.244 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:38.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:38 smithi050 ceph-mon[36371]: pgmap v389: 97 pgs: 97 active+clean; 296 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.6 MiB/s wr, 401 op/s 2024-05-27T09:47:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:38 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:47:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:38 smithi189 ceph-mon[36671]: pgmap v389: 97 pgs: 97 active+clean; 296 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.6 MiB/s wr, 401 op/s 2024-05-27T09:47:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:38 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:47:38.414 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:38 smithi044 ceph-mon[31618]: pgmap v389: 97 pgs: 97 active+clean; 296 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.6 MiB/s wr, 401 op/s 2024-05-27T09:47:38.415 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:38 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:47:39.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.407+0000 7f01d1400700 1 -- 172.21.15.44:0/1389757527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01cc108f10 msgr2=0x7f01cc109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:39.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.407+0000 7f01d1400700 1 --2- 172.21.15.44:0/1389757527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01cc108f10 0x7f01cc109320 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009990 tx=0x7f01b4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.407+0000 7f01d1400700 1 -- 172.21.15.44:0/1389757527 shutdown_connections 2024-05-27T09:47:39.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.407+0000 7f01d1400700 1 --2- 172.21.15.44:0/1389757527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01cc109860 0x7f01cc1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.407+0000 7f01d1400700 1 --2- 172.21.15.44:0/1389757527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01cc1045e0 0x7f01cc1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.407+0000 7f01d1400700 1 --2- 172.21.15.44:0/1389757527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01cc108f10 0x7f01cc109320 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.407+0000 7f01d1400700 1 -- 172.21.15.44:0/1389757527 >> 172.21.15.44:0/1389757527 conn(0x7f01cc100030 msgr2=0x7f01cc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:39.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.407+0000 7f01d1400700 1 -- 172.21.15.44:0/1389757527 shutdown_connections 2024-05-27T09:47:39.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.408+0000 7f01d1400700 1 -- 172.21.15.44:0/1389757527 wait complete. 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.409+0000 7f01d1400700 1 Processor -- start 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.409+0000 7f01d1400700 1 -- start start 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.409+0000 7f01d1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01cc1045e0 0x7f01cc075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.409+0000 7f01d1400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01cc108f10 0x7f01cc075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.409+0000 7f01d1400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01cc109860 0x7f01cc076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.409+0000 7f01d1400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc1144c0 con 0x7f01cc108f10 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.409+0000 7f01d1400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc114340 con 0x7f01cc109860 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.409+0000 7f01d1400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc114640 con 0x7f01cc1045e0 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.410+0000 7f01ca800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01cc1045e0 0x7f01cc075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.410+0000 7f01cb200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01cc109860 0x7f01cc076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.410+0000 7f01ca800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01cc1045e0 0x7f01cc075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45528/0 (socket says 172.21.15.44:45528) 2024-05-27T09:47:39.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.410+0000 7f01ca800700 1 -- 172.21.15.44:0/919500230 learned_addr learned my addr 172.21.15.44:0/919500230 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:39.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.410+0000 7f01c3e00700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01cc108f10 0x7f01cc075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.410+0000 7f01ca800700 1 -- 172.21.15.44:0/919500230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01cc109860 msgr2=0x7f01cc076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:39.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.410+0000 7f01ca800700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01cc109860 0x7f01cc076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.410+0000 7f01ca800700 1 -- 172.21.15.44:0/919500230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01cc108f10 msgr2=0x7f01cc075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:39.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.410+0000 7f01ca800700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01cc108f10 0x7f01cc075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.410+0000 7f01ca800700 1 -- 172.21.15.44:0/919500230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01cc1a9210 con 0x7f01cc1045e0 2024-05-27T09:47:39.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.410+0000 7f01ca800700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01cc1045e0 0x7f01cc075ab0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00eb30 tx=0x7f01bc00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:39.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.411+0000 7f01c3400700 1 -- 172.21.15.44:0/919500230 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc00cbd0 con 0x7f01cc1045e0 2024-05-27T09:47:39.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.411+0000 7f01c3400700 1 -- 172.21.15.44:0/919500230 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01bc0044d0 con 0x7f01cc1045e0 2024-05-27T09:47:39.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.411+0000 7f01d1400700 1 -- 172.21.15.44:0/919500230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01cc1a9500 con 0x7f01cc1045e0 2024-05-27T09:47:39.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.411+0000 7f01c3400700 1 -- 172.21.15.44:0/919500230 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc003e30 con 0x7f01cc1045e0 2024-05-27T09:47:39.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.411+0000 7f01d1400700 1 -- 172.21.15.44:0/919500230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01cc1a9a40 con 0x7f01cc1045e0 2024-05-27T09:47:39.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.415+0000 7f01c3400700 1 -- 172.21.15.44:0/919500230 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f01bc004000 con 0x7f01cc1045e0 2024-05-27T09:47:39.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.416+0000 7f01dff7db80 1 Processor -- start 2024-05-27T09:47:39.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.416+0000 7f01c3400700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01b00b29f0 0x7f01b00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.416+0000 7f01dff7db80 1 -- start start 2024-05-27T09:47:39.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.416+0000 7f01c3400700 1 -- 172.21.15.44:0/919500230 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f01bc0e2a40 con 0x7f01cc1045e0 2024-05-27T09:47:39.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.417+0000 7f01cb200700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01b00b29f0 0x7f01b00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.417+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082a2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.417+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082a2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.417+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x5597082af110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.417+0000 7f01dff7db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5597081d0e40 con 0x5597082a2e70 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.417+0000 7f01dff7db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5597081c09e0 con 0x559708123e40 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.417+0000 7f01dff7db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5597081c4a20 con 0x55970820f9f0 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.417+0000 7f01ca800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.417+0000 7f01c3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082a2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.417+0000 7f01ca800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082a2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45532/0 (socket says 172.21.15.44:45532) 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.417+0000 7f01ca800700 1 -- 172.21.15.44:0/4156773812 learned_addr learned my addr 172.21.15.44:0/4156773812 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.418+0000 7f01cb200700 1 --2- 172.21.15.44:0/4156773812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x5597082af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.418+0000 7f01ca800700 1 -- 172.21.15.44:0/4156773812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 msgr2=0x5597082a2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.418+0000 7f01ca800700 1 --2- 172.21.15.44:0/4156773812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082a2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.418+0000 7f01ca800700 1 -- 172.21.15.44:0/4156773812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 msgr2=0x5597082af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:39.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.418+0000 7f01ca800700 1 --2- 172.21.15.44:0/4156773812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x5597082af110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.418+0000 7f01ca800700 1 -- 172.21.15.44:0/4156773812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559708189420 con 0x55970820f9f0 2024-05-27T09:47:39.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.418+0000 7f01cb200700 1 --2- 172.21.15.44:0/4156773812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x5597082af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:47:39.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.418+0000 7f01cb200700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01b00b29f0 0x7f01b00b4f00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f01c400f590 tx=0x7f01c4009920 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:39.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.418+0000 7f01ca800700 1 --2- 172.21.15.44:0/4156773812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082a2330 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00c5b0 tx=0x7f01bc0a5c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ab2f02b28a1c73e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:39.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.418+0000 7f01bbe00700 1 -- 172.21.15.44:0/4156773812 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc0a9400 con 0x55970820f9f0 2024-05-27T09:47:39.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.419+0000 7f01bbe00700 1 -- 172.21.15.44:0/4156773812 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01bc0c1020 con 0x55970820f9f0 2024-05-27T09:47:39.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.419+0000 7f01bbe00700 1 -- 172.21.15.44:0/4156773812 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc0a5420 con 0x55970820f9f0 2024-05-27T09:47:39.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.420+0000 7f01dff7db80 1 -- 172.21.15.44:0/4156773812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 msgr2=0x5597082a2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:39.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.420+0000 7f01dff7db80 1 --2- 172.21.15.44:0/4156773812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082a2330 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00c5b0 tx=0x7f01bc0a5c80 comp rx=0 tx=0).stop 2024-05-27T09:47:39.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.420+0000 7f01dff7db80 1 -- 172.21.15.44:0/4156773812 shutdown_connections 2024-05-27T09:47:39.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.420+0000 7f01dff7db80 1 --2- 172.21.15.44:0/4156773812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x5597082af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.420+0000 7f01dff7db80 1 --2- 172.21.15.44:0/4156773812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082a2330 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.420+0000 7f01dff7db80 1 --2- 172.21.15.44:0/4156773812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082a2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.420+0000 7f01dff7db80 1 -- 172.21.15.44:0/4156773812 >> 172.21.15.44:0/4156773812 conn(0x5597081bd390 msgr2=0x55970829e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:39.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.420+0000 7f01dff7db80 1 -- 172.21.15.44:0/4156773812 shutdown_connections 2024-05-27T09:47:39.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.420+0000 7f01dff7db80 1 -- 172.21.15.44:0/4156773812 wait complete. 2024-05-27T09:47:39.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.421+0000 7f01dff7db80 1 -- start start 2024-05-27T09:47:39.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.421+0000 7f01dff7db80 1 Processor -- start 2024-05-27T09:47:39.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55970820f9f0 0x5597082acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5597082a2e70 0x5597082ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01dff7db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5597080c6fa0 con 0x55970820f9f0 2024-05-27T09:47:39.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01dff7db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559708161760 con 0x559708123e40 2024-05-27T09:47:39.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01dff7db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5597082afb80 con 0x5597082a2e70 2024-05-27T09:47:39.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01dff7db80 10 client.0 _ll_register_callbacks cb 0x55970816ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:39.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01ca800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01ca800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082ac6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:44248/0 (socket says 172.21.15.44:44248) 2024-05-27T09:47:39.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01ca800700 1 -- 172.21.15.44:0/3046101021 learned_addr learned my addr 172.21.15.44:0/3046101021 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:39.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01c3e00700 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55970820f9f0 0x5597082acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.422+0000 7f01cb200700 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5597082a2e70 0x5597082ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.423+0000 7f01c3e00700 1 -- 172.21.15.44:0/3046101021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5597082a2e70 msgr2=0x5597082ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:39.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.423+0000 7f01c3e00700 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5597082a2e70 0x5597082ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.423+0000 7f01c3e00700 1 -- 172.21.15.44:0/3046101021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 msgr2=0x5597082ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:39.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.423+0000 7f01c3e00700 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.423+0000 7f01c3e00700 1 -- 172.21.15.44:0/3046101021 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597081108d0 con 0x55970820f9f0 2024-05-27T09:47:39.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.424+0000 7f01c3e00700 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55970820f9f0 0x5597082acbf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f01b4000c00 tx=0x7f01b4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac00f0ee33922c6b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:39.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.424+0000 7f01b8c00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:47:39.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.425+0000 7f01b8c00700 1 -- 172.21.15.44:0/3046101021 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b400bdd0 con 0x55970820f9f0 2024-05-27T09:47:39.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.425+0000 7f01b8c00700 1 -- 172.21.15.44:0/3046101021 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f01b4003680 con 0x55970820f9f0 2024-05-27T09:47:39.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.425+0000 7f01dff7db80 1 -- 172.21.15.44:0/3046101021 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01bc0c1020 con 0x55970820f9f0 2024-05-27T09:47:39.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.425+0000 7f01b8c00700 1 -- 172.21.15.44:0/3046101021 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f01b400b610 con 0x55970820f9f0 2024-05-27T09:47:39.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.425+0000 7f01dff7db80 10 client.17298 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:39.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.425+0000 7f01dff7db80 1 -- 172.21.15.44:0/3046101021 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x559708191060 con 0x55970820f9f0 2024-05-27T09:47:39.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.426+0000 7f01b8c00700 1 -- 172.21.15.44:0/3046101021 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01bc0c1020 con 0x55970820f9f0 2024-05-27T09:47:39.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.426+0000 7f01b8c00700 1 -- 172.21.15.44:0/3046101021 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f01b4023a10 con 0x55970820f9f0 2024-05-27T09:47:39.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.426+0000 7f01dff7db80 10 client.17298 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:39.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.426+0000 7f01dff7db80 10 client.17298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:39.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.426+0000 7f01dff7db80 20 client.17298 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:39.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.426+0000 7f01dff7db80 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5597082bcb00 0x5597082bef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.426+0000 7f01dff7db80 4 client.17298 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:39.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.426+0000 7f01dff7db80 1 -- 172.21.15.44:0/3046101021 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597081d2d90 con 0x5597082bcb00 2024-05-27T09:47:39.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.427+0000 7f01ca800700 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5597082bcb00 0x5597082bef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.427+0000 7f01ca800700 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5597082bcb00 0x5597082bef20 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ebcd8f4c59de011 server_cookie=63cd7dac00b70656 in_seq=0 out_seq=0 2024-05-27T09:47:39.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.427+0000 7f01b8c00700 10 client.17298 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:39.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.428+0000 7f01b8c00700 1 -- 172.21.15.44:0/3046101021 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5597081d2d90 con 0x5597082bcb00 2024-05-27T09:47:39.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.428+0000 7f01b8c00700 10 client.17298 handle_command_reply: tid=0 2024-05-27T09:47:39.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.428+0000 7f01dff7db80 1 client.17298 shutdown 2024-05-27T09:47:39.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01dff7db80 1 -- 172.21.15.44:0/3046101021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55970820f9f0 msgr2=0x5597082acbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:39.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01dff7db80 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55970820f9f0 0x5597082acbf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f01b4000c00 tx=0x7f01b4005800 comp rx=0 tx=0).stop 2024-05-27T09:47:39.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01dff7db80 1 -- 172.21.15.44:0/3046101021 shutdown_connections 2024-05-27T09:47:39.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01dff7db80 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5597082bcb00 0x5597082bef20 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01dff7db80 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5597082a2e70 0x5597082ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01dff7db80 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01dff7db80 1 --2- 172.21.15.44:0/3046101021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55970820f9f0 0x5597082acbf0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01dff7db80 1 -- 172.21.15.44:0/3046101021 >> 172.21.15.44:0/3046101021 conn(0x5597081bd390 msgr2=0x5597082a2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:39.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01b8c00700 0 client.17298 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:39.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01dff7db80 1 -- 172.21.15.44:0/3046101021 shutdown_connections 2024-05-27T09:47:39.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01dff7db80 1 -- 172.21.15.44:0/3046101021 wait complete. 2024-05-27T09:47:39.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.429+0000 7f01dff7db80 20 client.17298 trim_cache size 0 max 16384 2024-05-27T09:47:39.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.453+0000 7f01dff7db80 1 Processor -- start 2024-05-27T09:47:39.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.453+0000 7f01dff7db80 1 -- start start 2024-05-27T09:47:39.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082a6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082a7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x55970829e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01dff7db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc1af670 con 0x5597082a2e70 2024-05-27T09:47:39.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01dff7db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc1aced0 con 0x559708123e40 2024-05-27T09:47:39.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01dff7db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc19edf0 con 0x55970820f9f0 2024-05-27T09:47:39.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01ca800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01ca800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082a6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:44256/0 (socket says 172.21.15.44:44256) 2024-05-27T09:47:39.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01c3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082a7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01ca800700 1 -- 172.21.15.44:0/2409664385 learned_addr learned my addr 172.21.15.44:0/2409664385 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:39.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01cb200700 1 --2- 172.21.15.44:0/2409664385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x55970829e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01ca800700 1 -- 172.21.15.44:0/2409664385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 msgr2=0x5597082a7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:39.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01ca800700 1 --2- 172.21.15.44:0/2409664385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082a7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01ca800700 1 -- 172.21.15.44:0/2409664385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 msgr2=0x55970829e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:39.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01ca800700 1 --2- 172.21.15.44:0/2409664385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x55970829e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01ca800700 1 -- 172.21.15.44:0/2409664385 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597082adb10 con 0x559708123e40 2024-05-27T09:47:39.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.454+0000 7f01cb200700 1 --2- 172.21.15.44:0/2409664385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x55970829e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:47:39.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.455+0000 7f01ca800700 1 --2- 172.21.15.44:0/2409664385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082a6910 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0a98c0 tx=0x7f01bc0e06a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97d95d479f92f478 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:39.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.455+0000 7f01bb400700 1 -- 172.21.15.44:0/2409664385 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc0b2440 con 0x559708123e40 2024-05-27T09:47:39.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.455+0000 7f01bb400700 1 -- 172.21.15.44:0/2409664385 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01bc08c030 con 0x559708123e40 2024-05-27T09:47:39.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.455+0000 7f01bb400700 1 -- 172.21.15.44:0/2409664385 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc0ae040 con 0x559708123e40 2024-05-27T09:47:39.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.456+0000 7f01dff7db80 1 -- 172.21.15.44:0/2409664385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 msgr2=0x5597082a6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:39.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.456+0000 7f01dff7db80 1 --2- 172.21.15.44:0/2409664385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082a6910 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0a98c0 tx=0x7f01bc0e06a0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.456+0000 7f01dff7db80 1 -- 172.21.15.44:0/2409664385 shutdown_connections 2024-05-27T09:47:39.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.456+0000 7f01dff7db80 1 --2- 172.21.15.44:0/2409664385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x55970829e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.456+0000 7f01dff7db80 1 --2- 172.21.15.44:0/2409664385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082a6910 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.456+0000 7f01dff7db80 1 --2- 172.21.15.44:0/2409664385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082a7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.456+0000 7f01dff7db80 1 -- 172.21.15.44:0/2409664385 >> 172.21.15.44:0/2409664385 conn(0x5597081bd390 msgr2=0x5597082add60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:39.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.456+0000 7f01dff7db80 1 -- 172.21.15.44:0/2409664385 shutdown_connections 2024-05-27T09:47:39.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.456+0000 7f01dff7db80 1 -- 172.21.15.44:0/2409664385 wait complete. 2024-05-27T09:47:39.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.457+0000 7f01dff7db80 1 -- start start 2024-05-27T09:47:39.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.457+0000 7f01dff7db80 1 Processor -- start 2024-05-27T09:47:39.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.457+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.457+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082b0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.457+0000 7f01dff7db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x5597082b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.457+0000 7f01dff7db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc1ab390 con 0x5597082a2e70 2024-05-27T09:47:39.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.457+0000 7f01dff7db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc1ac800 con 0x559708123e40 2024-05-27T09:47:39.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.457+0000 7f01dff7db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc1aba60 con 0x55970820f9f0 2024-05-27T09:47:39.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.458+0000 7f01dff7db80 10 client.0 _ll_register_callbacks cb 0x55970816ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.458+0000 7f01c3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.458+0000 7f01c3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082b0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45562/0 (socket says 172.21.15.44:45562) 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.458+0000 7f01c3e00700 1 -- 172.21.15.44:0/1070758864 learned_addr learned my addr 172.21.15.44:0/1070758864 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.458+0000 7f01ca800700 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.458+0000 7f01cb200700 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x5597082b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.458+0000 7f01c3e00700 1 -- 172.21.15.44:0/1070758864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 msgr2=0x5597082b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.458+0000 7f01c3e00700 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082b55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.458+0000 7f01c3e00700 1 -- 172.21.15.44:0/1070758864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 msgr2=0x5597082b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.458+0000 7f01c3e00700 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x5597082b0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.458+0000 7f01c3e00700 1 -- 172.21.15.44:0/1070758864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597082b16c0 con 0x55970820f9f0 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.459+0000 7f01c3e00700 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082b0940 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f01b40055a0 tx=0x7f01b4005900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c4ca17b84c990f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:39.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.459+0000 7f01b8c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:39.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.459+0000 7f01b8c00700 1 -- 172.21.15.44:0/1070758864 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b4023670 con 0x55970820f9f0 2024-05-27T09:47:39.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.459+0000 7f01dff7db80 1 -- 172.21.15.44:0/1070758864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01bc08c030 con 0x55970820f9f0 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.459+0000 7f01b8c00700 1 -- 172.21.15.44:0/1070758864 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f01b4023810 con 0x55970820f9f0 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.459+0000 7f01b8c00700 1 -- 172.21.15.44:0/1070758864 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f01b400b610 con 0x55970820f9f0 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.459+0000 7f01dff7db80 10 client.27544 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.459+0000 7f01dff7db80 1 -- 172.21.15.44:0/1070758864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x559708327550 con 0x55970820f9f0 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.460+0000 7f01b8c00700 1 -- 172.21.15.44:0/1070758864 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01bc08c030 con 0x55970820f9f0 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.460+0000 7f01b8c00700 1 -- 172.21.15.44:0/1070758864 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f01b403d070 con 0x55970820f9f0 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.460+0000 7f01dff7db80 10 client.27544 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.460+0000 7f01dff7db80 10 client.27544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.460+0000 7f01dff7db80 20 client.27544 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.460+0000 7f01dff7db80 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559708327c50 0x55970832a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.460+0000 7f01dff7db80 4 client.27544 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.460+0000 7f01dff7db80 1 -- 172.21.15.44:0/1070758864 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5597081cc570 con 0x559708327c50 2024-05-27T09:47:39.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.461+0000 7f01ca800700 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559708327c50 0x55970832a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:39.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.461+0000 7f01ca800700 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559708327c50 0x55970832a070 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b1ef4d3ae3b9ec1 server_cookie=3cc79bbb3664b858 in_seq=0 out_seq=0 2024-05-27T09:47:39.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.461+0000 7f01b8c00700 10 client.27544 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:39.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.461+0000 7f01b8c00700 1 -- 172.21.15.44:0/1070758864 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5597081cc570 con 0x559708327c50 2024-05-27T09:47:39.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.461+0000 7f01b8c00700 10 client.27544 handle_command_reply: tid=0 2024-05-27T09:47:39.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.461+0000 7f01dff7db80 1 client.27544 shutdown 2024-05-27T09:47:39.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01dff7db80 1 -- 172.21.15.44:0/1070758864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 msgr2=0x5597082b0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:39.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01dff7db80 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082b0940 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f01b40055a0 tx=0x7f01b4005900 comp rx=0 tx=0).stop 2024-05-27T09:47:39.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01dff7db80 1 -- 172.21.15.44:0/1070758864 shutdown_connections 2024-05-27T09:47:39.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01dff7db80 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559708327c50 0x55970832a070 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01dff7db80 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5597082a2e70 0x5597082b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01dff7db80 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559708123e40 0x5597082b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01dff7db80 1 --2- 172.21.15.44:0/1070758864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55970820f9f0 0x5597082b0940 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01dff7db80 1 -- 172.21.15.44:0/1070758864 >> 172.21.15.44:0/1070758864 conn(0x5597081bd390 msgr2=0x5597082acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:39.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01b8c00700 0 client.27544 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:39.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01dff7db80 1 -- 172.21.15.44:0/1070758864 shutdown_connections 2024-05-27T09:47:39.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01dff7db80 1 -- 172.21.15.44:0/1070758864 wait complete. 2024-05-27T09:47:39.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.462+0000 7f01dff7db80 20 client.27544 trim_cache size 0 max 16384 2024-05-27T09:47:39.465 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:47:39.466 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:47:39.466 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:47:39.466 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:47:39.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 -- 172.21.15.44:0/919500230 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01b00b29f0 msgr2=0x7f01b00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:39.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01b00b29f0 0x7f01b00b4f00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f01c400f590 tx=0x7f01c4009920 comp rx=0 tx=0).stop 2024-05-27T09:47:39.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 -- 172.21.15.44:0/919500230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01cc1045e0 msgr2=0x7f01cc075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:39.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01cc1045e0 0x7f01cc075ab0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00eb30 tx=0x7f01bc00ee40 comp rx=0 tx=0).stop 2024-05-27T09:47:39.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 -- 172.21.15.44:0/919500230 shutdown_connections 2024-05-27T09:47:39.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f01b00b29f0 0x7f01b00b4f00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01cc109860 0x7f01cc076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01cc1045e0 0x7f01cc075ab0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 --2- 172.21.15.44:0/919500230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01cc108f10 0x7f01cc075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:39.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 -- 172.21.15.44:0/919500230 >> 172.21.15.44:0/919500230 conn(0x7f01cc100030 msgr2=0x7f01cc108650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:39.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 -- 172.21.15.44:0/919500230 shutdown_connections 2024-05-27T09:47:39.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:39.463+0000 7f01d1400700 1 -- 172.21.15.44:0/919500230 wait complete. 2024-05-27T09:47:40.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:40 smithi044 ceph-mon[31618]: pgmap v390: 97 pgs: 97 active+clean; 296 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.5 MiB/s wr, 364 op/s 2024-05-27T09:47:40.191 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:47:40.191 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:47:40.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:40 smithi189 ceph-mon[36671]: pgmap v390: 97 pgs: 97 active+clean; 296 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.5 MiB/s wr, 364 op/s 2024-05-27T09:47:40.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:40 smithi050 ceph-mon[36371]: pgmap v390: 97 pgs: 97 active+clean; 296 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.5 MiB/s wr, 364 op/s 2024-05-27T09:47:40.711 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.316+0000 7f6112200640 1 -- 172.21.15.44:0/914847961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f610c10c140 msgr2=0x7f610c105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.316+0000 7f6112200640 1 --2- 172.21.15.44:0/914847961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f610c10c140 0x7f610c105d80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009900 tx=0x7f60f802a410 comp rx=0 tx=0).stop 2024-05-27T09:47:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.316+0000 7f6112200640 1 -- 172.21.15.44:0/914847961 shutdown_connections 2024-05-27T09:47:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.316+0000 7f6112200640 1 --2- 172.21.15.44:0/914847961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f610c1063f0 0x7f610c1127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.316+0000 7f6112200640 1 --2- 172.21.15.44:0/914847961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f610c10c140 0x7f610c105d80 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.316+0000 7f6112200640 1 --2- 172.21.15.44:0/914847961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f610c10bca0 0x7f610c105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.316+0000 7f6112200640 1 -- 172.21.15.44:0/914847961 >> 172.21.15.44:0/914847961 conn(0x7f610c101300 msgr2=0x7f610c103770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.316+0000 7f6112200640 1 -- 172.21.15.44:0/914847961 shutdown_connections 2024-05-27T09:47:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.316+0000 7f6112200640 1 -- 172.21.15.44:0/914847961 wait complete. 2024-05-27T09:47:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.317+0000 7f6112200640 1 Processor -- start 2024-05-27T09:47:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f6112200640 1 -- start start 2024-05-27T09:47:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f6112200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f610c1063f0 0x7f610c1a25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f6112200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f610c10bca0 0x7f610c1a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f6112200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f610c10c140 0x7f610c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f6112200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f610c115ec0 con 0x7f610c1063f0 2024-05-27T09:47:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f6112200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f610c115d40 con 0x7f610c10bca0 2024-05-27T09:47:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f6112200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f610c116040 con 0x7f610c10c140 2024-05-27T09:47:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f6110e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f610c1063f0 0x7f610c1a25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f610be00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f610c10bca0 0x7f610c1a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f610be00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f610c10bca0 0x7f610c1a2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:44272/0 (socket says 172.21.15.44:44272) 2024-05-27T09:47:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f6111800640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f610c10c140 0x7f610c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.318+0000 7f610be00640 1 -- 172.21.15.44:0/3238416101 learned_addr learned my addr 172.21.15.44:0/3238416101 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.319+0000 7f6110e00640 1 -- 172.21.15.44:0/3238416101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f610c10c140 msgr2=0x7f610c19c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.319+0000 7f6110e00640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f610c10c140 0x7f610c19c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.319+0000 7f6110e00640 1 -- 172.21.15.44:0/3238416101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f610c10bca0 msgr2=0x7f610c1a2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.319+0000 7f6110e00640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f610c10bca0 0x7f610c1a2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.319+0000 7f6110e00640 1 -- 172.21.15.44:0/3238416101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610c19ce50 con 0x7f610c1063f0 2024-05-27T09:47:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.319+0000 7f6110e00640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f610c1063f0 0x7f610c1a25a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f610000e9c0 tx=0x7f610000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.319+0000 7f6109600640 1 -- 172.21.15.44:0/3238416101 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610000ccc0 con 0x7f610c1063f0 2024-05-27T09:47:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.320+0000 7f6109600640 1 -- 172.21.15.44:0/3238416101 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61000108e0 con 0x7f610c1063f0 2024-05-27T09:47:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.320+0000 7f6112200640 1 -- 172.21.15.44:0/3238416101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f610c19d0e0 con 0x7f610c1063f0 2024-05-27T09:47:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.320+0000 7f6109600640 1 -- 172.21.15.44:0/3238416101 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61000187c0 con 0x7f610c1063f0 2024-05-27T09:47:42.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.320+0000 7f6112200640 1 -- 172.21.15.44:0/3238416101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f610c1a9340 con 0x7f610c1063f0 2024-05-27T09:47:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.323+0000 7f611a81c740 1 Processor -- start 2024-05-27T09:47:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.324+0000 7f611a81c740 1 -- start start 2024-05-27T09:47:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.324+0000 7f6109600640 1 -- 172.21.15.44:0/3238416101 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6100010430 con 0x7f610c1063f0 2024-05-27T09:47:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.324+0000 7f611a81c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 0x55e66ff93a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.324+0000 7f611a81c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff93fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.324+0000 7f611a81c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 0x55e66ff944e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.324+0000 7f611a81c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e66ff04b90 con 0x55e66fbc82f0 2024-05-27T09:47:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.324+0000 7f611a81c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e66fef4730 con 0x55e66fbd04d0 2024-05-27T09:47:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.324+0000 7f611a81c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e66fef8770 con 0x55e66fbf11b0 2024-05-27T09:47:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.324+0000 7f6111800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 0x55e66ff93a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.325+0000 7f6111800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 0x55e66ff93a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45598/0 (socket says 172.21.15.44:45598) 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.325+0000 7f610be00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 0x55e66ff944e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.325+0000 7f6111800640 1 -- 172.21.15.44:0/42095825 learned_addr learned my addr 172.21.15.44:0/42095825 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.325+0000 7f6110e00640 1 --2- 172.21.15.44:0/42095825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff93fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.325+0000 7f6111800640 1 -- 172.21.15.44:0/42095825 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 msgr2=0x55e66ff944e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.325+0000 7f6111800640 1 --2- 172.21.15.44:0/42095825 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 0x55e66ff944e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.325+0000 7f6111800640 1 -- 172.21.15.44:0/42095825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 msgr2=0x55e66ff93fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.325+0000 7f6111800640 1 --2- 172.21.15.44:0/42095825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff93fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.325+0000 7f6111800640 1 -- 172.21.15.44:0/42095825 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e66fe8d8b0 con 0x55e66fbf11b0 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.325+0000 7f6111800640 1 --2- 172.21.15.44:0/42095825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 0x55e66ff93a60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6104009fd0 tx=0x7f61040118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dbb7b68a858d7ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.325+0000 7f60fe000640 1 -- 172.21.15.44:0/42095825 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6104004d20 con 0x55e66fbf11b0 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.326+0000 7f60fe000640 1 -- 172.21.15.44:0/42095825 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6104004050 con 0x55e66fbf11b0 2024-05-27T09:47:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.326+0000 7f60fe000640 1 -- 172.21.15.44:0/42095825 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6104005280 con 0x55e66fbf11b0 2024-05-27T09:47:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.326+0000 7f6109600640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60f40b2950 0x7f60f40b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.326+0000 7f6110e00640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60f40b2950 0x7f60f40b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.326+0000 7f6109600640 1 -- 172.21.15.44:0/3238416101 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f61000e2980 con 0x7f610c1063f0 2024-05-27T09:47:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.327+0000 7f611a81c740 1 -- 172.21.15.44:0/42095825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 msgr2=0x55e66ff93a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.327+0000 7f611a81c740 1 --2- 172.21.15.44:0/42095825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 0x55e66ff93a60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6104009fd0 tx=0x7f61040118d0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.327+0000 7f611a81c740 1 -- 172.21.15.44:0/42095825 shutdown_connections 2024-05-27T09:47:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.327+0000 7f611a81c740 1 --2- 172.21.15.44:0/42095825 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 0x55e66ff944e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.327+0000 7f611a81c740 1 --2- 172.21.15.44:0/42095825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff93fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.327+0000 7f611a81c740 1 --2- 172.21.15.44:0/42095825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 0x55e66ff93a60 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.327+0000 7f611a81c740 1 -- 172.21.15.44:0/42095825 >> 172.21.15.44:0/42095825 conn(0x55e66fbe5300 msgr2=0x55e66ff90210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.327+0000 7f611a81c740 1 -- 172.21.15.44:0/42095825 shutdown_connections 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.327+0000 7f611a81c740 1 -- 172.21.15.44:0/42095825 wait complete. 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.327+0000 7f6110e00640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60f40b2950 0x7f60f40b4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x55e66ff91550 tx=0x7f610000ce60 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.328+0000 7f611a81c740 1 -- start start 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.328+0000 7f611a81c740 1 Processor -- start 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.328+0000 7f611a81c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff9deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.328+0000 7f611a81c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ff9e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.328+0000 7f611a81c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbf11b0 0x55e66ffa9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.328+0000 7f611a81c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e66fbda790 con 0x55e66fbc82f0 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.328+0000 7f611a81c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e66fe9cbc0 con 0x55e66fbf11b0 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.328+0000 7f611a81c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e66ffa0b90 con 0x55e66fbd04d0 2024-05-27T09:47:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.329+0000 7f611a81c740 10 client.0 _ll_register_callbacks cb 0x55e66fe7a400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.329+0000 7f610be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ff9e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.329+0000 7f6111800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff9deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.329+0000 7f610be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ff9e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45612/0 (socket says 172.21.15.44:45612) 2024-05-27T09:47:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.329+0000 7f610be00640 1 -- 172.21.15.44:0/3413042045 learned_addr learned my addr 172.21.15.44:0/3413042045 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.329+0000 7f6111800640 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbf11b0 0x55e66ffa9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.329+0000 7f610be00640 1 -- 172.21.15.44:0/3413042045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbf11b0 msgr2=0x55e66ffa9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.329+0000 7f610be00640 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbf11b0 0x55e66ffa9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.329+0000 7f610be00640 1 -- 172.21.15.44:0/3413042045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 msgr2=0x55e66ff9deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.329+0000 7f610be00640 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff9deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.329+0000 7f610be00640 1 -- 172.21.15.44:0/3413042045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e66fc09210 con 0x55e66fbd04d0 2024-05-27T09:47:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.330+0000 7f610be00640 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ff9e3f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f60f802a930 tx=0x7f60f80043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e99e5ed41251a70d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.330+0000 7f60f2a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.330+0000 7f60f2a00640 1 -- 172.21.15.44:0/3413042045 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8035040 con 0x55e66fbd04d0 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.330+0000 7f611a81c740 1 -- 172.21.15.44:0/3413042045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6104004050 con 0x55e66fbd04d0 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.330+0000 7f60f2a00640 1 -- 172.21.15.44:0/3413042045 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f60f8005220 con 0x55e66fbd04d0 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.330+0000 7f60f2a00640 1 -- 172.21.15.44:0/3413042045 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f60f8005420 con 0x55e66fbd04d0 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.330+0000 7f611a81c740 10 client.27556 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.330+0000 7f611a81c740 1 -- 172.21.15.44:0/3413042045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e66fcd5620 con 0x55e66fbd04d0 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.331+0000 7f60f2a00640 1 -- 172.21.15.44:0/3413042045 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6104004050 con 0x55e66fbd04d0 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.331+0000 7f60f2a00640 1 -- 172.21.15.44:0/3413042045 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f60f804c070 con 0x55e66fbd04d0 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.331+0000 7f611a81c740 10 client.27556 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.331+0000 7f611a81c740 10 client.27556 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.331+0000 7f611a81c740 20 client.27556 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.331+0000 7f611a81c740 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e66fbafd80 0x55e66ffafeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.331+0000 7f611a81c740 4 client.27556 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.331+0000 7f611a81c740 1 -- 172.21.15.44:0/3413042045 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e66ff06ae0 con 0x55e66fbafd80 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.331+0000 7f6111800640 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e66fbafd80 0x55e66ffafeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.332+0000 7f6111800640 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e66fbafd80 0x55e66ffafeb0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db88ebb706363993 server_cookie=d3821503345e46f5 in_seq=0 out_seq=0 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.332+0000 7f60f2a00640 10 client.27556 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f60f2a00640 1 -- 172.21.15.44:0/3413042045 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e66ff06ae0 con 0x55e66fbafd80 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f60f2a00640 10 client.27556 handle_command_reply: tid=0 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f611a81c740 1 client.27556 shutdown 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f611a81c740 1 -- 172.21.15.44:0/3413042045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 msgr2=0x55e66ff9e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f611a81c740 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ff9e3f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f60f802a930 tx=0x7f60f80043b0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f611a81c740 1 -- 172.21.15.44:0/3413042045 shutdown_connections 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f611a81c740 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e66fbafd80 0x55e66ffafeb0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f611a81c740 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbf11b0 0x55e66ffa9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f611a81c740 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ff9e3f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f611a81c740 1 --2- 172.21.15.44:0/3413042045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff9deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f611a81c740 1 -- 172.21.15.44:0/3413042045 >> 172.21.15.44:0/3413042045 conn(0x55e66fbe5300 msgr2=0x55e66ff93890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.333+0000 7f60f2a00640 0 client.27556 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.334+0000 7f611a81c740 1 -- 172.21.15.44:0/3413042045 shutdown_connections 2024-05-27T09:47:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.334+0000 7f611a81c740 1 -- 172.21.15.44:0/3413042045 wait complete. 2024-05-27T09:47:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.334+0000 7f611a81c740 20 client.27556 trim_cache size 0 max 16384 2024-05-27T09:47:42.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:42 smithi189 ceph-mon[36671]: pgmap v391: 97 pgs: 97 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 2.2 MiB/s wr, 489 op/s 2024-05-27T09:47:42.345 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:42 smithi050 ceph-mon[36371]: pgmap v391: 97 pgs: 97 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 2.2 MiB/s wr, 489 op/s 2024-05-27T09:47:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.351+0000 7f611a81c740 1 Processor -- start 2024-05-27T09:47:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.351+0000 7f611a81c740 1 -- start start 2024-05-27T09:47:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.351+0000 7f611a81c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff93f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.351+0000 7f611a81c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 0x55e66ffa9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.351+0000 7f611a81c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 0x55e66ffaa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.351+0000 7f611a81c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1b44d0 con 0x55e66fbc82f0 2024-05-27T09:47:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.351+0000 7f611a81c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1b1d70 con 0x55e66fbd04d0 2024-05-27T09:47:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.351+0000 7f611a81c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1afbc0 con 0x55e66fbf11b0 2024-05-27T09:47:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.351+0000 7f6111800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff93f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.352+0000 7f610be00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 0x55e66ffa9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.352+0000 7f610be00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 0x55e66ffa9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:44308/0 (socket says 172.21.15.44:44308) 2024-05-27T09:47:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.352+0000 7f610be00640 1 -- 172.21.15.44:0/3209891022 learned_addr learned my addr 172.21.15.44:0/3209891022 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.352+0000 7f6111800640 1 --2- 172.21.15.44:0/3209891022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 0x55e66ffaa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.352+0000 7f610be00640 1 -- 172.21.15.44:0/3209891022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 msgr2=0x55e66ffaa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.352+0000 7f610be00640 1 --2- 172.21.15.44:0/3209891022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 0x55e66ffaa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.352+0000 7f610be00640 1 -- 172.21.15.44:0/3209891022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 msgr2=0x55e66ff93f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.352+0000 7f610be00640 1 --2- 172.21.15.44:0/3209891022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff93f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.352+0000 7f610be00640 1 -- 172.21.15.44:0/3209891022 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e66ff91360 con 0x55e66fbd04d0 2024-05-27T09:47:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.352+0000 7f6111800640 1 --2- 172.21.15.44:0/3209891022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 0x55e66ffaa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:47:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.352+0000 7f610be00640 1 --2- 172.21.15.44:0/3209891022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 0x55e66ffa9c30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f60f8005ab0 tx=0x7f60f802a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16bf78668df59a1d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.353+0000 7f60fd600640 1 -- 172.21.15.44:0/3209891022 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8035070 con 0x55e66fbd04d0 2024-05-27T09:47:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.353+0000 7f60fd600640 1 -- 172.21.15.44:0/3209891022 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60f80035f0 con 0x55e66fbd04d0 2024-05-27T09:47:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.353+0000 7f60fd600640 1 -- 172.21.15.44:0/3209891022 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8052070 con 0x55e66fbd04d0 2024-05-27T09:47:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.353+0000 7f611a81c740 1 -- 172.21.15.44:0/3209891022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 msgr2=0x55e66ffa9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.353+0000 7f611a81c740 1 --2- 172.21.15.44:0/3209891022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 0x55e66ffa9c30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f60f8005ab0 tx=0x7f60f802a410 comp rx=0 tx=0).stop 2024-05-27T09:47:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.353+0000 7f611a81c740 1 -- 172.21.15.44:0/3209891022 shutdown_connections 2024-05-27T09:47:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.353+0000 7f611a81c740 1 --2- 172.21.15.44:0/3209891022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbf11b0 0x55e66ffaa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.353+0000 7f611a81c740 1 --2- 172.21.15.44:0/3209891022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbd04d0 0x55e66ffa9c30 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.354+0000 7f611a81c740 1 --2- 172.21.15.44:0/3209891022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ff93f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.354+0000 7f611a81c740 1 -- 172.21.15.44:0/3209891022 >> 172.21.15.44:0/3209891022 conn(0x55e66fbe5300 msgr2=0x55e66ff937f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.354+0000 7f611a81c740 1 -- 172.21.15.44:0/3209891022 shutdown_connections 2024-05-27T09:47:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.354+0000 7f611a81c740 1 -- 172.21.15.44:0/3209891022 wait complete. 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.354+0000 7f611a81c740 1 -- start start 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.354+0000 7f611a81c740 1 Processor -- start 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f611a81c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ffa2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f611a81c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ffa33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f611a81c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbf11b0 0x55e66ffa3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f611a81c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1b0230 con 0x55e66fbc82f0 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f611a81c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1b16a0 con 0x55e66fbf11b0 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f611a81c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1b0900 con 0x55e66fbd04d0 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f611a81c740 10 client.0 _ll_register_callbacks cb 0x7f60f8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f610be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ffa33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f6111800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ffa2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f610be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ffa33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:45624/0 (socket says 172.21.15.44:45624) 2024-05-27T09:47:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f610be00640 1 -- 172.21.15.44:0/763693625 learned_addr learned my addr 172.21.15.44:0/763693625 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.355+0000 7f6111800640 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbf11b0 0x55e66ffa3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.356+0000 7f610be00640 1 -- 172.21.15.44:0/763693625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbf11b0 msgr2=0x55e66ffa3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.356+0000 7f610be00640 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbf11b0 0x55e66ffa3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.356+0000 7f610be00640 1 -- 172.21.15.44:0/763693625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 msgr2=0x55e66ffa2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.356+0000 7f610be00640 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ffa2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.356+0000 7f610be00640 1 -- 172.21.15.44:0/763693625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e670016760 con 0x55e66fbd04d0 2024-05-27T09:47:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.356+0000 7f6111800640 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbf11b0 0x55e66ffa3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:47:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.356+0000 7f610be00640 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ffa33a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f60f8031b60 tx=0x7f60f80045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad82e0a4a0cbdc38 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.356+0000 7f60f2a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.356+0000 7f60f2a00640 1 -- 172.21.15.44:0/763693625 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8035070 con 0x55e66fbd04d0 2024-05-27T09:47:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.356+0000 7f60f2a00640 1 -- 172.21.15.44:0/763693625 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f60f804e460 con 0x55e66fbd04d0 2024-05-27T09:47:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.356+0000 7f611a81c740 1 -- 172.21.15.44:0/763693625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60f80035f0 con 0x55e66fbd04d0 2024-05-27T09:47:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f60f2a00640 1 -- 172.21.15.44:0/763693625 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f60f8064050 con 0x55e66fbd04d0 2024-05-27T09:47:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f611a81c740 10 client.27562 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f611a81c740 1 -- 172.21.15.44:0/763693625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e6700179f0 con 0x55e66fbd04d0 2024-05-27T09:47:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f60f2a00640 1 -- 172.21.15.44:0/763693625 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60f80035f0 con 0x55e66fbd04d0 2024-05-27T09:47:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f60f2a00640 1 -- 172.21.15.44:0/763693625 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f60f804c040 con 0x55e66fbd04d0 2024-05-27T09:47:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f611a81c740 10 client.27562 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f611a81c740 10 client.27562 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f611a81c740 20 client.27562 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f611a81c740 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e66fb851c0 0x55e67001a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f611a81c740 4 client.27562 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:47:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f611a81c740 1 -- 172.21.15.44:0/763693625 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e66ff002c0 con 0x55e66fb851c0 2024-05-27T09:47:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.357+0000 7f6111800640 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e66fb851c0 0x55e67001a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.358+0000 7f6111800640 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e66fb851c0 0x55e67001a000 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b79cc318cb28f9c2 server_cookie=484a1b9151887ba9 in_seq=0 out_seq=0 2024-05-27T09:47:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.358+0000 7f60f2a00640 10 client.27562 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:42.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:42 smithi044 ceph-mon[31618]: pgmap v391: 97 pgs: 97 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 2.2 MiB/s wr, 489 op/s 2024-05-27T09:47:42.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.480+0000 7fedaee00700 1 -- 172.21.15.44:0/2898932629 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e292b5e700 con 0x55e292cba5d0 2024-05-27T09:47:42.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.480+0000 7fedaee00700 10 client.27526 handle_command_reply: tid=0 2024-05-27T09:47:42.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.480+0000 7fedd2795b80 1 client.27526 shutdown 2024-05-27T09:47:42.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedd2795b80 1 -- 172.21.15.44:0/2898932629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 msgr2=0x55e292c470a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedd2795b80 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c470a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7feda001e000 tx=0x7feda0009de0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedd2795b80 1 -- 172.21.15.44:0/2898932629 shutdown_connections 2024-05-27T09:47:42.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedd2795b80 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e292cba5d0 0x55e292cbc9f0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedd2795b80 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e292c3b380 0x55e292c44070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedd2795b80 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e292ab5fd0 0x55e292c470a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedd2795b80 1 --2- 172.21.15.44:0/2898932629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e292c354d0 0x55e292c47600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedaee00700 0 client.27526 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:42.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedd2795b80 1 -- 172.21.15.44:0/2898932629 >> 172.21.15.44:0/2898932629 conn(0x55e292b16150 msgr2=0x55e292c38ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:42.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedd2795b80 1 -- 172.21.15.44:0/2898932629 shutdown_connections 2024-05-27T09:47:42.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedd2795b80 1 -- 172.21.15.44:0/2898932629 wait complete. 2024-05-27T09:47:42.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.481+0000 7fedd2795b80 20 client.27526 trim_cache size 0 max 16384 2024-05-27T09:47:42.483 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 42, 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 30, 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "8d7cdbbb": { 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 30, 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 142.3, 2024-05-27T09:47:42.484 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:47:42.485 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:47:42.485 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:47:42.485 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:47:42.485 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:47:42.485 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:47:42.485 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:42.485 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:42.485 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:42.485 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:42.485 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:47:42.485 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:47:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.482+0000 7fedc3c00700 1 -- 172.21.15.44:0/2849887161 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fed980b2a70 msgr2=0x7fed980b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.482+0000 7fedc3c00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fed980b2a70 0x7fed980b4f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fedb400b2e0 tx=0x7fedb4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.482+0000 7fedc3c00700 1 -- 172.21.15.44:0/2849887161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc105e60 msgr2=0x7fedbc19d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.482+0000 7fedc3c00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc105e60 0x7fedbc19d550 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feda80060b0 tx=0x7feda8007f90 comp rx=0 tx=0).stop 2024-05-27T09:47:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.482+0000 7fedc3c00700 1 -- 172.21.15.44:0/2849887161 shutdown_connections 2024-05-27T09:47:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.482+0000 7fedc3c00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fed980b2a70 0x7fed980b4f80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.482+0000 7fedc3c00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedbc10e8f0 0x7fedbc1a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.482+0000 7fedc3c00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedbc105510 0x7fedbc19cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.482+0000 7fedc3c00700 1 --2- 172.21.15.44:0/2849887161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc105e60 0x7fedbc19d550 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.482+0000 7fedc3c00700 1 -- 172.21.15.44:0/2849887161 >> 172.21.15.44:0/2849887161 conn(0x7fedbc0fedb0 msgr2=0x7fedbc103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.483+0000 7fedc3c00700 1 -- 172.21.15.44:0/2849887161 shutdown_connections 2024-05-27T09:47:42.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:42.483+0000 7fedc3c00700 1 -- 172.21.15.44:0/2849887161 wait complete. 2024-05-27T09:47:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.546+0000 7f60f2a00640 1 -- 172.21.15.44:0/763693625 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e66ff002c0 con 0x55e66fb851c0 2024-05-27T09:47:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.546+0000 7f60f2a00640 10 client.27562 handle_command_reply: tid=0 2024-05-27T09:47:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.546+0000 7f611a81c740 1 client.27562 shutdown 2024-05-27T09:47:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.546+0000 7f611a81c740 1 -- 172.21.15.44:0/763693625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 msgr2=0x55e66ffa33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.546+0000 7f611a81c740 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ffa33a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f60f8031b60 tx=0x7f60f80045e0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.547+0000 7f611a81c740 1 -- 172.21.15.44:0/763693625 shutdown_connections 2024-05-27T09:47:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.547+0000 7f611a81c740 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e66fb851c0 0x55e67001a000 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.547+0000 7f611a81c740 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e66fbf11b0 0x55e66ffa3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.547+0000 7f611a81c740 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e66fbd04d0 0x55e66ffa33a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.547+0000 7f611a81c740 1 --2- 172.21.15.44:0/763693625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e66fbc82f0 0x55e66ffa2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.547+0000 7f611a81c740 1 -- 172.21.15.44:0/763693625 >> 172.21.15.44:0/763693625 conn(0x55e66fbe5300 msgr2=0x55e66ff9fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.547+0000 7f60f2a00640 0 client.27562 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.547+0000 7f611a81c740 1 -- 172.21.15.44:0/763693625 shutdown_connections 2024-05-27T09:47:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.548+0000 7f611a81c740 1 -- 172.21.15.44:0/763693625 wait complete. 2024-05-27T09:47:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.548+0000 7f611a81c740 20 client.27562 trim_cache size 0 max 16384 2024-05-27T09:47:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:47:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:47:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:47:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.550+0000 7f6112200640 1 -- 172.21.15.44:0/3238416101 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60f40b2950 msgr2=0x7f60f40b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.550+0000 7f6112200640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60f40b2950 0x7f60f40b4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x55e66ff91550 tx=0x7f610000ce60 comp rx=0 tx=0).stop 2024-05-27T09:47:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.550+0000 7f6112200640 1 -- 172.21.15.44:0/3238416101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f610c1063f0 msgr2=0x7f610c1a25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.550+0000 7f6112200640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f610c1063f0 0x7f610c1a25a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f610000e9c0 tx=0x7f610000ee80 comp rx=0 tx=0).stop 2024-05-27T09:47:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.551+0000 7f6112200640 1 -- 172.21.15.44:0/3238416101 shutdown_connections 2024-05-27T09:47:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.551+0000 7f6112200640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f610c10c140 0x7f610c19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.551+0000 7f6112200640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f610c10bca0 0x7f610c1a2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.551+0000 7f6112200640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f60f40b2950 0x7f60f40b4e60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.551+0000 7f6112200640 1 --2- 172.21.15.44:0/3238416101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f610c1063f0 0x7f610c1a25a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.551+0000 7f6112200640 1 -- 172.21.15.44:0/3238416101 >> 172.21.15.44:0/3238416101 conn(0x7f610c101300 msgr2=0x7f610c1107b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:42.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.551+0000 7f6112200640 1 -- 172.21.15.44:0/3238416101 shutdown_connections 2024-05-27T09:47:42.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:42.551+0000 7f6112200640 1 -- 172.21.15.44:0/3238416101 wait complete. 2024-05-27T09:47:43.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.095+0000 7fe8e6e00700 1 -- 172.21.15.44:0/3550513270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e01034b0 msgr2=0x7fe8e01038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:43.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.095+0000 7fe8e6e00700 1 --2- 172.21.15.44:0/3550513270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e01034b0 0x7fe8e01038c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00099c0 tx=0x7fe8d0009cd0 comp rx=0 tx=0).stop 2024-05-27T09:47:43.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.095+0000 7fe8e6e00700 1 -- 172.21.15.44:0/3550513270 shutdown_connections 2024-05-27T09:47:43.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.095+0000 7fe8e6e00700 1 --2- 172.21.15.44:0/3550513270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8e0108730 0x7fe8e0110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:43.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.095+0000 7fe8e6e00700 1 --2- 172.21.15.44:0/3550513270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e01034b0 0x7fe8e01038c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:43.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.095+0000 7fe8e6e00700 1 --2- 172.21.15.44:0/3550513270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8e0107de0 0x7fe8e01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:43.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.095+0000 7fe8e6e00700 1 -- 172.21.15.44:0/3550513270 >> 172.21.15.44:0/3550513270 conn(0x7fe8e00fedb0 msgr2=0x7fe8e01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:43.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.095+0000 7fe8e6e00700 1 -- 172.21.15.44:0/3550513270 shutdown_connections 2024-05-27T09:47:43.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.095+0000 7fe8e6e00700 1 -- 172.21.15.44:0/3550513270 wait complete. 2024-05-27T09:47:43.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.096+0000 7fe8e6e00700 1 Processor -- start 2024-05-27T09:47:43.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.096+0000 7fe8e6e00700 1 -- start start 2024-05-27T09:47:43.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.097+0000 7fe8e6e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8e01034b0 0x7fe8e019d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:43.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.097+0000 7fe8e6e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8e0107de0 0x7fe8e019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:43.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.097+0000 7fe8e6e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e0108730 0x7fe8e01a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:43.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.097+0000 7fe8e6e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e01133b0 con 0x7fe8e0108730 2024-05-27T09:47:43.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.097+0000 7fe8e6e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e0113230 con 0x7fe8e0107de0 2024-05-27T09:47:43.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.097+0000 7fe8e6e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e0113530 con 0x7fe8e01034b0 2024-05-27T09:47:43.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.097+0000 7fe8dfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8e01034b0 0x7fe8e019d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:43.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.097+0000 7fe8df400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8e0107de0 0x7fe8e019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:43.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.097+0000 7fe8dfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8e01034b0 0x7fe8e019d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:37182/0 (socket says 172.21.15.44:37182) 2024-05-27T09:47:43.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.097+0000 7fe8dfe00700 1 -- 172.21.15.44:0/529152527 learned_addr learned my addr 172.21.15.44:0/529152527 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:43.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.098+0000 7fe8e4e00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e0108730 0x7fe8e01a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:43.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.098+0000 7fe8dfe00700 1 -- 172.21.15.44:0/529152527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8e0107de0 msgr2=0x7fe8e019d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.098+0000 7fe8dfe00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8e0107de0 0x7fe8e019d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.098+0000 7fe8dfe00700 1 -- 172.21.15.44:0/529152527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e0108730 msgr2=0x7fe8e01a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.098+0000 7fe8dfe00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e0108730 0x7fe8e01a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.098+0000 7fe8dfe00700 1 -- 172.21.15.44:0/529152527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e01a1f50 con 0x7fe8e01034b0 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.098+0000 7fe8e4e00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e0108730 0x7fe8e01a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.098+0000 7fe8df400700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8e0107de0 0x7fe8e019d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.098+0000 7fe8dfe00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8e01034b0 0x7fe8e019d080 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0006010 tx=0x7fe8d0005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.099+0000 7fe8dcc00700 1 -- 172.21.15.44:0/529152527 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0026070 con 0x7fe8e01034b0 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.099+0000 7fe8e6e00700 1 -- 172.21.15.44:0/529152527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e01a21e0 con 0x7fe8e01034b0 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.099+0000 7fe8dcc00700 1 -- 172.21.15.44:0/529152527 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8d0003c70 con 0x7fe8e01034b0 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.099+0000 7fe8dcc00700 1 -- 172.21.15.44:0/529152527 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d000fb40 con 0x7fe8e01034b0 2024-05-27T09:47:43.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.099+0000 7fe8e6e00700 1 -- 172.21.15.44:0/529152527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e01a9b60 con 0x7fe8e01034b0 2024-05-27T09:47:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.100+0000 7fe8e6e00700 1 -- 172.21.15.44:0/529152527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8e0104960 con 0x7fe8e01034b0 2024-05-27T09:47:43.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.102+0000 7fe8dcc00700 1 -- 172.21.15.44:0/529152527 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fe8d0029070 con 0x7fe8e01034b0 2024-05-27T09:47:43.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.104+0000 7fe8dcc00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe8c40b2a70 0x7fe8c40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:43.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.104+0000 7fe8dcc00700 1 -- 172.21.15.44:0/529152527 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7fe8d00e2f00 con 0x7fe8e01034b0 2024-05-27T09:47:43.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.104+0000 7fe8df400700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe8c40b2a70 0x7fe8c40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:43.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.105+0000 7fe8df400700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe8c40b2a70 0x7fe8c40b4f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe8e006c590 tx=0x7fe8cc009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:43.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.108+0000 7fe8dcc00700 1 -- 172.21.15.44:0/529152527 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe8d0013070 con 0x7fe8e01034b0 2024-05-27T09:47:43.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.399+0000 7fe8e6e00700 1 -- 172.21.15.44:0/529152527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8e019e3e0 con 0x7fe8e01034b0 2024-05-27T09:47:43.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.400+0000 7fe8dcc00700 1 -- 172.21.15.44:0/529152527 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11868 (secure 0 0 0) 0x7fe8d0003e20 con 0x7fe8e01034b0 2024-05-27T09:47:43.402 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:47:43.402 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:47:31.318532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24542,24503,24520]},"id":1}]} 2024-05-27T09:47:43.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.404+0000 7fe8e6e00700 1 -- 172.21.15.44:0/529152527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe8c40b2a70 msgr2=0x7fe8c40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:43.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.404+0000 7fe8e6e00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe8c40b2a70 0x7fe8c40b4f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe8e006c590 tx=0x7fe8cc009260 comp rx=0 tx=0).stop 2024-05-27T09:47:43.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.404+0000 7fe8e6e00700 1 -- 172.21.15.44:0/529152527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8e01034b0 msgr2=0x7fe8e019d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:43.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.404+0000 7fe8e6e00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8e01034b0 0x7fe8e019d080 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0006010 tx=0x7fe8d0005280 comp rx=0 tx=0).stop 2024-05-27T09:47:43.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.404+0000 7fe8e6e00700 1 -- 172.21.15.44:0/529152527 shutdown_connections 2024-05-27T09:47:43.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.404+0000 7fe8e6e00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe8c40b2a70 0x7fe8c40b4f80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:43.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.404+0000 7fe8e6e00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8e0108730 0x7fe8e01a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:43.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.404+0000 7fe8e6e00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8e01034b0 0x7fe8e019d080 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:43.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.404+0000 7fe8e6e00700 1 --2- 172.21.15.44:0/529152527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8e0107de0 0x7fe8e019d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:43.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.404+0000 7fe8e6e00700 1 -- 172.21.15.44:0/529152527 >> 172.21.15.44:0/529152527 conn(0x7fe8e00fedb0 msgr2=0x7fe8e01096f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:43.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.404+0000 7fe8e6e00700 1 -- 172.21.15.44:0/529152527 shutdown_connections 2024-05-27T09:47:43.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:43.405+0000 7fe8e6e00700 1 -- 172.21.15.44:0/529152527 wait complete. 2024-05-27T09:47:43.407 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 42 2024-05-27T09:47:43.532 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '8d7cdbbb', quiesce took 14.1 seconds. Will release after: 8.200000000000001 2024-05-27T09:47:44.192 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:47:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:44 smithi189 ceph-mon[36671]: pgmap v392: 97 pgs: 97 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 2.2 MiB/s wr, 474 op/s 2024-05-27T09:47:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:44 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/529152527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:44.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:44 smithi044 ceph-mon[31618]: pgmap v392: 97 pgs: 97 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 2.2 MiB/s wr, 474 op/s 2024-05-27T09:47:44.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:44 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/529152527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:44 smithi050 ceph-mon[36371]: pgmap v392: 97 pgs: 97 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 2.2 MiB/s wr, 474 op/s 2024-05-27T09:47:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:44 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/529152527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:44.676 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:46.200 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:46 smithi044 ceph-mon[31618]: pgmap v393: 97 pgs: 97 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 2.2 MiB/s wr, 474 op/s 2024-05-27T09:47:46.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:46 smithi189 ceph-mon[36671]: pgmap v393: 97 pgs: 97 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 2.2 MiB/s wr, 474 op/s 2024-05-27T09:47:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:46 smithi050 ceph-mon[36371]: pgmap v393: 97 pgs: 97 active+clean; 304 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 2.2 MiB/s wr, 474 op/s 2024-05-27T09:47:46.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.804+0000 7f2690800700 1 -- 172.21.15.44:0/3911258645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688115360 msgr2=0x7f2688117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:46.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.804+0000 7f2690800700 1 --2- 172.21.15.44:0/3911258645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688115360 0x7f2688117780 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f268400b0a0 tx=0x7f268400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.804+0000 7f2690800700 1 -- 172.21.15.44:0/3911258645 shutdown_connections 2024-05-27T09:47:46.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.804+0000 7f2690800700 1 --2- 172.21.15.44:0/3911258645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688115360 0x7f2688117780 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.804+0000 7f2690800700 1 --2- 172.21.15.44:0/3911258645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2688075b50 0x7f2688075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.804+0000 7f2690800700 1 --2- 172.21.15.44:0/3911258645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26880764a0 0x7f2688114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.804+0000 7f2690800700 1 -- 172.21.15.44:0/3911258645 >> 172.21.15.44:0/3911258645 conn(0x7f2688100030 msgr2=0x7f2688102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:46.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.805+0000 7f2690800700 1 -- 172.21.15.44:0/3911258645 shutdown_connections 2024-05-27T09:47:46.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.805+0000 7f2690800700 1 -- 172.21.15.44:0/3911258645 wait complete. 2024-05-27T09:47:46.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.806+0000 7f2690800700 1 Processor -- start 2024-05-27T09:47:46.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.806+0000 7f2690800700 1 -- start start 2024-05-27T09:47:46.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.807+0000 7f2690800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2688075b50 0x7f26881a1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.807+0000 7f2690800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26880764a0 0x7f26881a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.807+0000 7f2690800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688115360 0x7f26881a5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.807+0000 7f2690800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f268811aac0 con 0x7f26880764a0 2024-05-27T09:47:46.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.807+0000 7f2690800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f268811a940 con 0x7f2688075b50 2024-05-27T09:47:46.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.807+0000 7f2690800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f268811ac40 con 0x7f2688115360 2024-05-27T09:47:46.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.807+0000 7f268d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26880764a0 0x7f26881a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.807+0000 7f268de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2688075b50 0x7f26881a1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.807+0000 7f268d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26880764a0 0x7f26881a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41812/0 (socket says 172.21.15.44:41812) 2024-05-27T09:47:46.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.807+0000 7f268d400700 1 -- 172.21.15.44:0/1188952031 learned_addr learned my addr 172.21.15.44:0/1188952031 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:46.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.807+0000 7f268e800700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688115360 0x7f26881a5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.808+0000 7f268de00700 1 -- 172.21.15.44:0/1188952031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688115360 msgr2=0x7f26881a5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.808+0000 7f268de00700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688115360 0x7f26881a5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.808+0000 7f268de00700 1 -- 172.21.15.44:0/1188952031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26880764a0 msgr2=0x7f26881a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:46.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.808+0000 7f268de00700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26880764a0 0x7f26881a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.808+0000 7f268de00700 1 -- 172.21.15.44:0/1188952031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26881a6260 con 0x7f2688075b50 2024-05-27T09:47:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.808+0000 7f268e800700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688115360 0x7f26881a5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.808+0000 7f268d400700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26880764a0 0x7f26881a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.808+0000 7f268de00700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2688075b50 0x7f26881a1390 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f267c00e9f0 tx=0x7f267c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.808+0000 7f2682a00700 1 -- 172.21.15.44:0/1188952031 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c00c9c0 con 0x7f2688075b50 2024-05-27T09:47:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.808+0000 7f2682a00700 1 -- 172.21.15.44:0/1188952031 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f267c00cb60 con 0x7f2688075b50 2024-05-27T09:47:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.808+0000 7f2690800700 1 -- 172.21.15.44:0/1188952031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26881a6550 con 0x7f2688075b50 2024-05-27T09:47:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.809+0000 7f2682a00700 1 -- 172.21.15.44:0/1188952031 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c018680 con 0x7f2688075b50 2024-05-27T09:47:46.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.809+0000 7f2690800700 1 -- 172.21.15.44:0/1188952031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26881ade70 con 0x7f2688075b50 2024-05-27T09:47:46.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.812+0000 7f2682a00700 1 -- 172.21.15.44:0/1188952031 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f267c010430 con 0x7f2688075b50 2024-05-27T09:47:46.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.813+0000 7f269f445b80 1 Processor -- start 2024-05-27T09:47:46.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f269f445b80 1 -- start start 2024-05-27T09:47:46.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f269f445b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d31ae40 0x55895d49f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f269f445b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d49fd30 0x55895d499870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f269f445b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d49a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f269f445b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55895d3c7e40 con 0x55895d49fd30 2024-05-27T09:47:46.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f269f445b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55895d3b79e0 con 0x55895d31ae40 2024-05-27T09:47:46.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f269f445b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55895d3bba20 con 0x55895d499db0 2024-05-27T09:47:46.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f2682a00700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26700b29f0 0x7f26700b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f268d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d49a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f268e800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d31ae40 0x55895d49f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.815+0000 7f268e800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d31ae40 0x55895d49f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36798/0 (socket says 172.21.15.44:36798) 2024-05-27T09:47:46.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.815+0000 7f268e800700 1 -- 172.21.15.44:0/1523073964 learned_addr learned my addr 172.21.15.44:0/1523073964 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:46.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f268d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d49a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:37216/0 (socket says 172.21.15.44:37216) 2024-05-27T09:47:46.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.815+0000 7f2682a00700 1 -- 172.21.15.44:0/1188952031 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f267c0e2800 con 0x7f2688075b50 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.814+0000 7f268de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d49fd30 0x55895d499870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.815+0000 7f268e800700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26700b29f0 0x7f26700b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.815+0000 7f268d400700 1 -- 172.21.15.44:0/1523073964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d31ae40 msgr2=0x55895d49f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.815+0000 7f268d400700 1 --2- 172.21.15.44:0/1523073964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d31ae40 0x55895d49f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.815+0000 7f268d400700 1 -- 172.21.15.44:0/1523073964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d49fd30 msgr2=0x55895d499870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.815+0000 7f268d400700 1 --2- 172.21.15.44:0/1523073964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d49fd30 0x55895d499870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.815+0000 7f268d400700 1 -- 172.21.15.44:0/1523073964 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895d380420 con 0x55895d499db0 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.815+0000 7f268de00700 1 --2- 172.21.15.44:0/1523073964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d49fd30 0x55895d499870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.815+0000 7f268d400700 1 --2- 172.21.15.44:0/1523073964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d49a1c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f26881034d0 tx=0x7f26780117a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=623a407033d19d60 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.816+0000 7f268e800700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26700b29f0 0x7f26700b4f00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55895d49ab30 tx=0x7f2684006210 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.816+0000 7f2677400700 1 -- 172.21.15.44:0/1523073964 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678004230 con 0x55895d499db0 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.816+0000 7f2677400700 1 -- 172.21.15.44:0/1523073964 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26780043d0 con 0x55895d499db0 2024-05-27T09:47:46.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.816+0000 7f2677400700 1 -- 172.21.15.44:0/1523073964 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267800e9a0 con 0x55895d499db0 2024-05-27T09:47:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.817+0000 7f269f445b80 1 -- 172.21.15.44:0/1523073964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 msgr2=0x55895d49a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.817+0000 7f269f445b80 1 --2- 172.21.15.44:0/1523073964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d49a1c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f26881034d0 tx=0x7f26780117a0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.817+0000 7f269f445b80 1 -- 172.21.15.44:0/1523073964 shutdown_connections 2024-05-27T09:47:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.817+0000 7f269f445b80 1 --2- 172.21.15.44:0/1523073964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d49a1c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.817+0000 7f269f445b80 1 --2- 172.21.15.44:0/1523073964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d31ae40 0x55895d49f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.817+0000 7f269f445b80 1 --2- 172.21.15.44:0/1523073964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d49fd30 0x55895d499870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.817+0000 7f269f445b80 1 -- 172.21.15.44:0/1523073964 >> 172.21.15.44:0/1523073964 conn(0x55895d37afc0 msgr2=0x55895d4948d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.817+0000 7f269f445b80 1 -- 172.21.15.44:0/1523073964 shutdown_connections 2024-05-27T09:47:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.817+0000 7f269f445b80 1 -- 172.21.15.44:0/1523073964 wait complete. 2024-05-27T09:47:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.818+0000 7f269f445b80 1 -- start start 2024-05-27T09:47:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.818+0000 7f269f445b80 1 Processor -- start 2024-05-27T09:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.818+0000 7f269f445b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 0x55895d4a3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.819+0000 7f269f445b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d499db0 0x55895d4a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.819+0000 7f269f445b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 0x55895d4a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.819+0000 7f269f445b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55895d2bdfa0 con 0x55895d499db0 2024-05-27T09:47:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.819+0000 7f269f445b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55895d358760 con 0x55895d49fd30 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.819+0000 7f269f445b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55895d4a6800 con 0x55895d31ae40 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.819+0000 7f269f445b80 10 client.0 _ll_register_callbacks cb 0x55895d361c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.819+0000 7f268d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d499db0 0x55895d4a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.819+0000 7f268e800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 0x55895d4a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.819+0000 7f268de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 0x55895d4a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.819+0000 7f268e800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 0x55895d4a3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36808/0 (socket says 172.21.15.44:36808) 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.819+0000 7f268e800700 1 -- 172.21.15.44:0/60710377 learned_addr learned my addr 172.21.15.44:0/60710377 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f268de00700 1 -- 172.21.15.44:0/60710377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 msgr2=0x55895d4a3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f268de00700 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 0x55895d4a3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f268de00700 1 -- 172.21.15.44:0/60710377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d499db0 msgr2=0x55895d4a3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f268de00700 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d499db0 0x55895d4a3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f268de00700 1 -- 172.21.15.44:0/60710377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895d3078d0 con 0x55895d31ae40 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f268d400700 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d499db0 0x55895d4a3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f268e800700 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 0x55895d4a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:46.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f268de00700 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 0x55895d4a3510 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f267c0dfec0 tx=0x7f267c0732f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d664d0680fcce277 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:46.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f266be00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f266be00700 1 -- 172.21.15.44:0/60710377 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c0a5040 con 0x55895d31ae40 2024-05-27T09:47:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f269f445b80 1 -- 172.21.15.44:0/60710377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26780043d0 con 0x55895d31ae40 2024-05-27T09:47:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f266be00700 1 -- 172.21.15.44:0/60710377 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f267c0a9ec0 con 0x55895d31ae40 2024-05-27T09:47:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.820+0000 7f266be00700 1 -- 172.21.15.44:0/60710377 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f267c0a9560 con 0x55895d31ae40 2024-05-27T09:47:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.821+0000 7f269f445b80 10 client.27580 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.821+0000 7f269f445b80 1 -- 172.21.15.44:0/60710377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55895d388060 con 0x55895d31ae40 2024-05-27T09:47:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.821+0000 7f266be00700 1 -- 172.21.15.44:0/60710377 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26780043d0 con 0x55895d31ae40 2024-05-27T09:47:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.821+0000 7f266be00700 1 -- 172.21.15.44:0/60710377 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f267c0b2420 con 0x55895d31ae40 2024-05-27T09:47:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.821+0000 7f269f445b80 10 client.27580 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.821+0000 7f269f445b80 10 client.27580 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T09:47:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.821+0000 7f269f445b80 20 client.27580 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:46.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.821+0000 7f269f445b80 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55895d4b3740 0x55895d4b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.821+0000 7f269f445b80 4 client.27580 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:46.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.821+0000 7f269f445b80 1 -- 172.21.15.44:0/60710377 --> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55895d3c9d90 con 0x55895d4b3740 2024-05-27T09:47:46.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.822+0000 7f268d400700 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55895d4b3740 0x55895d4b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.822+0000 7f268d400700 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55895d4b3740 0x55895d4b5bd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e22b5f43924cce24 server_cookie=8cd611df4108a377 in_seq=0 out_seq=0 2024-05-27T09:47:46.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.822+0000 7f266be00700 10 client.27580 ms_handle_connect on v2:172.21.15.50:6836/3540954143 2024-05-27T09:47:46.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.823+0000 7f266be00700 1 -- 172.21.15.44:0/60710377 <== mds.1 v2:172.21.15.50:6836/3540954143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55895d3c9d90 con 0x55895d4b3740 2024-05-27T09:47:46.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.823+0000 7f266be00700 10 client.27580 handle_command_reply: tid=0 2024-05-27T09:47:46.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.823+0000 7f269f445b80 1 client.27580 shutdown 2024-05-27T09:47:46.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f269f445b80 1 -- 172.21.15.44:0/60710377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 msgr2=0x55895d4a3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:46.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f269f445b80 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 0x55895d4a3510 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f267c0dfec0 tx=0x7f267c0732f0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f269f445b80 1 -- 172.21.15.44:0/60710377 shutdown_connections 2024-05-27T09:47:46.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f269f445b80 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55895d4b3740 0x55895d4b5bd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f269f445b80 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 0x55895d4a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f269f445b80 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 0x55895d4a3510 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f269f445b80 1 --2- 172.21.15.44:0/60710377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d499db0 0x55895d4a3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f269f445b80 1 -- 172.21.15.44:0/60710377 >> 172.21.15.44:0/60710377 conn(0x55895d37afc0 msgr2=0x55895d494580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:46.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f266be00700 0 client.27580 ms_handle_reset on v2:172.21.15.50:6836/3540954143 2024-05-27T09:47:46.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f269f445b80 1 -- 172.21.15.44:0/60710377 shutdown_connections 2024-05-27T09:47:46.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f269f445b80 1 -- 172.21.15.44:0/60710377 wait complete. 2024-05-27T09:47:46.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.824+0000 7f269f445b80 20 client.27580 trim_cache size 0 max 16384 2024-05-27T09:47:46.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.841+0000 7f269f445b80 1 Processor -- start 2024-05-27T09:47:46.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.841+0000 7f269f445b80 1 -- start start 2024-05-27T09:47:46.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.842+0000 7f269f445b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 0x55895d499870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.842+0000 7f269f445b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d499db0 0x55895d49db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.842+0000 7f269f445b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d49fd30 0x55895d4af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.842+0000 7f269f445b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26881b7de0 con 0x55895d49fd30 2024-05-27T09:47:46.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.842+0000 7f269f445b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f26881b5640 con 0x55895d499db0 2024-05-27T09:47:46.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.842+0000 7f269f445b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f26881b3490 con 0x55895d31ae40 2024-05-27T09:47:46.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.842+0000 7f268de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 0x55895d499870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.842+0000 7f268e800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d49fd30 0x55895d4af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.842+0000 7f268de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 0x55895d499870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:37224/0 (socket says 172.21.15.44:37224) 2024-05-27T09:47:46.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.842+0000 7f268de00700 1 -- 172.21.15.44:0/468283752 learned_addr learned my addr 172.21.15.44:0/468283752 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:46.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.842+0000 7f268d400700 1 --2- 172.21.15.44:0/468283752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d499db0 0x55895d49db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.843+0000 7f268de00700 1 -- 172.21.15.44:0/468283752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d499db0 msgr2=0x55895d49db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:46.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.843+0000 7f268de00700 1 --2- 172.21.15.44:0/468283752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d499db0 0x55895d49db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.843+0000 7f268de00700 1 -- 172.21.15.44:0/468283752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d49fd30 msgr2=0x55895d4af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:46.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.843+0000 7f268de00700 1 --2- 172.21.15.44:0/468283752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d49fd30 0x55895d4af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.843+0000 7f268de00700 1 -- 172.21.15.44:0/468283752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895d49a650 con 0x55895d31ae40 2024-05-27T09:47:46.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.843+0000 7f268de00700 1 --2- 172.21.15.44:0/468283752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 0x55895d499870 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f267c0e2bc0 tx=0x7f267c0a9950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88ccd6ea5d327727 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:46.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.843+0000 7f2676a00700 1 -- 172.21.15.44:0/468283752 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c0aa150 con 0x55895d31ae40 2024-05-27T09:47:46.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.843+0000 7f2676a00700 1 -- 172.21.15.44:0/468283752 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f267c0b2040 con 0x55895d31ae40 2024-05-27T09:47:46.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.843+0000 7f2676a00700 1 -- 172.21.15.44:0/468283752 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c0cc0b0 con 0x55895d31ae40 2024-05-27T09:47:46.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.844+0000 7f269f445b80 1 -- 172.21.15.44:0/468283752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 msgr2=0x55895d499870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:46.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.844+0000 7f269f445b80 1 --2- 172.21.15.44:0/468283752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 0x55895d499870 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f267c0e2bc0 tx=0x7f267c0a9950 comp rx=0 tx=0).stop 2024-05-27T09:47:46.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.844+0000 7f269f445b80 1 -- 172.21.15.44:0/468283752 shutdown_connections 2024-05-27T09:47:46.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.844+0000 7f269f445b80 1 --2- 172.21.15.44:0/468283752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d49fd30 0x55895d4af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.844+0000 7f269f445b80 1 --2- 172.21.15.44:0/468283752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d31ae40 0x55895d499870 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.844+0000 7f269f445b80 1 --2- 172.21.15.44:0/468283752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d499db0 0x55895d49db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.844+0000 7f269f445b80 1 -- 172.21.15.44:0/468283752 >> 172.21.15.44:0/468283752 conn(0x55895d37afc0 msgr2=0x55895d4a1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:46.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.845+0000 7f269f445b80 1 -- 172.21.15.44:0/468283752 shutdown_connections 2024-05-27T09:47:46.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.845+0000 7f269f445b80 1 -- 172.21.15.44:0/468283752 wait complete. 2024-05-27T09:47:46.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.845+0000 7f269f445b80 1 -- start start 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.845+0000 7f269f445b80 1 Processor -- start 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f269f445b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d31ae40 0x55895d4abda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f269f445b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d4ac300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f269f445b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 0x55895d4baac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f269f445b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26881b3b00 con 0x55895d31ae40 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f269f445b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f26881b4f70 con 0x55895d49fd30 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f269f445b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f26881b41d0 con 0x55895d499db0 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f269f445b80 10 client.0 _ll_register_callbacks cb 0x55895d361c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f268de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d31ae40 0x55895d4abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f268d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d4ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f268d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d4ac300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:37230/0 (socket says 172.21.15.44:37230) 2024-05-27T09:47:46.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f268de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d31ae40 0x55895d4abda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41854/0 (socket says 172.21.15.44:41854) 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f268d400700 1 -- 172.21.15.44:0/1139524464 learned_addr learned my addr 172.21.15.44:0/1139524464 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.846+0000 7f268e800700 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 0x55895d4baac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.847+0000 7f268d400700 1 -- 172.21.15.44:0/1139524464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 msgr2=0x55895d4baac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.847+0000 7f268d400700 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 0x55895d4baac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.847+0000 7f268d400700 1 -- 172.21.15.44:0/1139524464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d31ae40 msgr2=0x55895d4abda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.847+0000 7f268d400700 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d31ae40 0x55895d4abda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.847+0000 7f268d400700 1 -- 172.21.15.44:0/1139524464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895d4a4300 con 0x55895d499db0 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.847+0000 7f268d400700 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d4ac300 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2678011af0 tx=0x7f2678011270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5672e6b9c97662 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.847+0000 7f266be00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.847+0000 7f266be00700 1 -- 172.21.15.44:0/1139524464 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678003680 con 0x55895d499db0 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.847+0000 7f266be00700 1 -- 172.21.15.44:0/1139524464 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f2678003820 con 0x55895d499db0 2024-05-27T09:47:46.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.847+0000 7f269f445b80 1 -- 172.21.15.44:0/1139524464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f267c0b2040 con 0x55895d499db0 2024-05-27T09:47:46.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.848+0000 7f266be00700 1 -- 172.21.15.44:0/1139524464 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f267803b050 con 0x55895d499db0 2024-05-27T09:47:46.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.848+0000 7f269f445b80 10 client.27586 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.848+0000 7f269f445b80 1 -- 172.21.15.44:0/1139524464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55895d51dc10 con 0x55895d499db0 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.848+0000 7f266be00700 1 -- 172.21.15.44:0/1139524464 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f267c0b2040 con 0x55895d499db0 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.848+0000 7f266be00700 1 -- 172.21.15.44:0/1139524464 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f267802b850 con 0x55895d499db0 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.848+0000 7f269f445b80 10 client.27586 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.848+0000 7f269f445b80 10 client.27586 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.848+0000 7f269f445b80 20 client.27586 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.849+0000 7f269f445b80 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55895d51e1b0 0x55895d5205d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.849+0000 7f269f445b80 4 client.27586 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.849+0000 7f269f445b80 1 -- 172.21.15.44:0/1139524464 --> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55895d3c3570 con 0x55895d51e1b0 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.849+0000 7f268de00700 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55895d51e1b0 0x55895d5205d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f268de00700 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55895d51e1b0 0x55895d5205d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e96b85f37eaed9b server_cookie=18cd7ca4522e9025 in_seq=0 out_seq=0 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f266be00700 10 client.27586 ms_handle_connect on v2:172.21.15.50:6836/3540954143 2024-05-27T09:47:46.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f266be00700 1 -- 172.21.15.44:0/1139524464 <== mds.1 v2:172.21.15.50:6836/3540954143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55895d3c3570 con 0x55895d51e1b0 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f266be00700 10 client.27586 handle_command_reply: tid=0 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f269f445b80 1 client.27586 shutdown 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f269f445b80 1 -- 172.21.15.44:0/1139524464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 msgr2=0x55895d4ac300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f269f445b80 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d4ac300 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2678011af0 tx=0x7f2678011270 comp rx=0 tx=0).stop 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f269f445b80 1 -- 172.21.15.44:0/1139524464 shutdown_connections 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f269f445b80 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55895d51e1b0 0x55895d5205d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f269f445b80 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895d49fd30 0x55895d4baac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f269f445b80 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895d31ae40 0x55895d4abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f269f445b80 1 --2- 172.21.15.44:0/1139524464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d499db0 0x55895d4ac300 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f266be00700 0 client.27586 ms_handle_reset on v2:172.21.15.50:6836/3540954143 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f269f445b80 1 -- 172.21.15.44:0/1139524464 >> 172.21.15.44:0/1139524464 conn(0x55895d37afc0 msgr2=0x55895d49d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f269f445b80 1 -- 172.21.15.44:0/1139524464 shutdown_connections 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.850+0000 7f269f445b80 1 -- 172.21.15.44:0/1139524464 wait complete. 2024-05-27T09:47:46.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.851+0000 7f269f445b80 20 client.27586 trim_cache size 0 max 16384 2024-05-27T09:47:46.857 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:47:46.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.851+0000 7f2690800700 1 -- 172.21.15.44:0/1188952031 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26700b29f0 msgr2=0x7f26700b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:46.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.851+0000 7f2690800700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26700b29f0 0x7f26700b4f00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55895d49ab30 tx=0x7f2684006210 comp rx=0 tx=0).stop 2024-05-27T09:47:46.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.851+0000 7f2690800700 1 -- 172.21.15.44:0/1188952031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2688075b50 msgr2=0x7f26881a1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:46.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.851+0000 7f2690800700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2688075b50 0x7f26881a1390 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f267c00e9f0 tx=0x7f267c00ed00 comp rx=0 tx=0).stop 2024-05-27T09:47:46.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.851+0000 7f2690800700 1 -- 172.21.15.44:0/1188952031 shutdown_connections 2024-05-27T09:47:46.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.851+0000 7f2690800700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26700b29f0 0x7f26700b4f00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.851+0000 7f2690800700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688115360 0x7f26881a5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.851+0000 7f2690800700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2688075b50 0x7f26881a1390 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.851+0000 7f2690800700 1 --2- 172.21.15.44:0/1188952031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26880764a0 0x7f26881a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:46.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.851+0000 7f2690800700 1 -- 172.21.15.44:0/1188952031 >> 172.21.15.44:0/1188952031 conn(0x7f2688100030 msgr2=0x7f26881005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:46.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.852+0000 7f2690800700 1 -- 172.21.15.44:0/1188952031 shutdown_connections 2024-05-27T09:47:46.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:46.852+0000 7f2690800700 1 -- 172.21.15.44:0/1188952031 wait complete. 2024-05-27T09:47:47.600 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:47:48.059 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:48.332 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:48 smithi044 ceph-mon[31618]: pgmap v394: 97 pgs: 97 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 2.6 MiB/s wr, 586 op/s 2024-05-27T09:47:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:48 smithi189 ceph-mon[36671]: pgmap v394: 97 pgs: 97 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 2.6 MiB/s wr, 586 op/s 2024-05-27T09:47:48.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:48 smithi050 ceph-mon[36371]: pgmap v394: 97 pgs: 97 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 2.6 MiB/s wr, 586 op/s 2024-05-27T09:47:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.309+0000 7f5e57000700 1 -- 172.21.15.44:0/739548647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e5010a980 msgr2=0x7f5e5010ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.309+0000 7f5e57000700 1 --2- 172.21.15.44:0/739548647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e5010a980 0x7f5e5010ad90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009990 tx=0x7f5e38009ca0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.309+0000 7f5e57000700 1 -- 172.21.15.44:0/739548647 shutdown_connections 2024-05-27T09:47:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.309+0000 7f5e57000700 1 --2- 172.21.15.44:0/739548647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e5010b2d0 0x7f5e50117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.309+0000 7f5e57000700 1 --2- 172.21.15.44:0/739548647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e50110960 0x7f5e50110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.309+0000 7f5e57000700 1 --2- 172.21.15.44:0/739548647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e5010a980 0x7f5e5010ad90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.309+0000 7f5e57000700 1 -- 172.21.15.44:0/739548647 >> 172.21.15.44:0/739548647 conn(0x7f5e50077c70 msgr2=0x7f5e500780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.309+0000 7f5e57000700 1 -- 172.21.15.44:0/739548647 shutdown_connections 2024-05-27T09:47:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.309+0000 7f5e57000700 1 -- 172.21.15.44:0/739548647 wait complete. 2024-05-27T09:47:50.311 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:50 smithi044 ceph-mon[31618]: pgmap v395: 97 pgs: 97 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 236 op/s 2024-05-27T09:47:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.311+0000 7f5e57000700 1 Processor -- start 2024-05-27T09:47:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.311+0000 7f5e57000700 1 -- start start 2024-05-27T09:47:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.311+0000 7f5e57000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e5010a980 0x7f5e501a14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e57000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e5010b2d0 0x7f5e501a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e57000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e50110960 0x7f5e501a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e57000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5011aad0 con 0x7f5e50110960 2024-05-27T09:47:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e57000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5011a950 con 0x7f5e5010b2d0 2024-05-27T09:47:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e57000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5011ac50 con 0x7f5e5010a980 2024-05-27T09:47:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e47400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e5010b2d0 0x7f5e501a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e4fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e5010a980 0x7f5e501a14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e47400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e5010b2d0 0x7f5e501a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36850/0 (socket says 172.21.15.44:36850) 2024-05-27T09:47:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e47400700 1 -- 172.21.15.44:0/2398441566 learned_addr learned my addr 172.21.15.44:0/2398441566 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e55000700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e50110960 0x7f5e501a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e47400700 1 -- 172.21.15.44:0/2398441566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e5010a980 msgr2=0x7f5e501a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.312+0000 7f5e47400700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e5010a980 0x7f5e501a14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.313+0000 7f5e47400700 1 -- 172.21.15.44:0/2398441566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e50110960 msgr2=0x7f5e501a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.313+0000 7f5e47400700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e50110960 0x7f5e501a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.313+0000 7f5e47400700 1 -- 172.21.15.44:0/2398441566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e501a6370 con 0x7f5e5010b2d0 2024-05-27T09:47:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.313+0000 7f5e4fe00700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e5010a980 0x7f5e501a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.313+0000 7f5e47400700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e5010b2d0 0x7f5e501a1a40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e38000c00 tx=0x7f5e380057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.313+0000 7f5e4d600700 1 -- 172.21.15.44:0/2398441566 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e38027070 con 0x7f5e5010b2d0 2024-05-27T09:47:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.313+0000 7f5e57000700 1 -- 172.21.15.44:0/2398441566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e501a6600 con 0x7f5e5010b2d0 2024-05-27T09:47:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.313+0000 7f5e4d600700 1 -- 172.21.15.44:0/2398441566 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e38022470 con 0x7f5e5010b2d0 2024-05-27T09:47:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.313+0000 7f5e4d600700 1 -- 172.21.15.44:0/2398441566 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e38017650 con 0x7f5e5010b2d0 2024-05-27T09:47:50.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.313+0000 7f5e57000700 1 -- 172.21.15.44:0/2398441566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e501a6ae0 con 0x7f5e5010b2d0 2024-05-27T09:47:50.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.317+0000 7f5e4d600700 1 -- 172.21.15.44:0/2398441566 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5e3802a070 con 0x7f5e5010b2d0 2024-05-27T09:47:50.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.319+0000 7f5e65bb4b80 1 Processor -- start 2024-05-27T09:47:50.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.319+0000 7f5e65bb4b80 1 -- start start 2024-05-27T09:47:50.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.319+0000 7f5e4d600700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5e340b29f0 0x7f5e340b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.319+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x5639343747f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.319+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x56393436e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.319+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x56393436f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.319+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56393429ce40 con 0x5639341efe40 2024-05-27T09:47:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.319+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56393428c9e0 con 0x563934374d30 2024-05-27T09:47:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.319+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563934290a20 con 0x56393436edb0 2024-05-27T09:47:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.319+0000 7f5e4d600700 1 -- 172.21.15.44:0/2398441566 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f5e380e3670 con 0x7f5e5010b2d0 2024-05-27T09:47:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.320+0000 7f5e55000700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5e340b29f0 0x7f5e340b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.320+0000 7f5e4fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x5639343747f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.320+0000 7f5e4fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x5639343747f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41884/0 (socket says 172.21.15.44:41884) 2024-05-27T09:47:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.320+0000 7f5e4fe00700 1 -- 172.21.15.44:0/2863841892 learned_addr learned my addr 172.21.15.44:0/2863841892 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:50.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.320+0000 7f5e55000700 1 --2- 172.21.15.44:0/2863841892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x56393436f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.320+0000 7f5e4fe00700 1 -- 172.21.15.44:0/2863841892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 msgr2=0x56393436f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:50.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.320+0000 7f5e4fe00700 1 --2- 172.21.15.44:0/2863841892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x56393436f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.320+0000 7f5e4fe00700 1 -- 172.21.15.44:0/2863841892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 msgr2=0x56393436e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:50.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.320+0000 7f5e4fe00700 1 --2- 172.21.15.44:0/2863841892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x56393436e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.320+0000 7f5e4fe00700 1 -- 172.21.15.44:0/2863841892 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563934255420 con 0x5639341efe40 2024-05-27T09:47:50.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.321+0000 7f5e55000700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5e340b29f0 0x7f5e340b4f00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009ec0 tx=0x7f5e480134a0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:50.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.321+0000 7f5e4fe00700 1 --2- 172.21.15.44:0/2863841892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x5639343747f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5e400114f0 tx=0x7f5e40011800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d036d4e6414c25 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:50.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.321+0000 7f5e45600700 1 -- 172.21.15.44:0/2863841892 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40004230 con 0x5639341efe40 2024-05-27T09:47:50.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.321+0000 7f5e45600700 1 -- 172.21.15.44:0/2863841892 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e400043d0 con 0x5639341efe40 2024-05-27T09:47:50.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.321+0000 7f5e45600700 1 -- 172.21.15.44:0/2863841892 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4000e910 con 0x5639341efe40 2024-05-27T09:47:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.322+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2863841892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 msgr2=0x5639343747f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.322+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/2863841892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x5639343747f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5e400114f0 tx=0x7f5e40011800 comp rx=0 tx=0).stop 2024-05-27T09:47:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.322+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2863841892 shutdown_connections 2024-05-27T09:47:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.322+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/2863841892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x56393436f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.322+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/2863841892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x5639343747f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.322+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/2863841892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x56393436e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.322+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2863841892 >> 172.21.15.44:0/2863841892 conn(0x56393424ffc0 msgr2=0x5639343698d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.322+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2863841892 shutdown_connections 2024-05-27T09:47:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.323+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2863841892 wait complete. 2024-05-27T09:47:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.323+0000 7f5e65bb4b80 1 -- start start 2024-05-27T09:47:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.323+0000 7f5e65bb4b80 1 Processor -- start 2024-05-27T09:47:50.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639341efe40 0x5639343784a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56393436edb0 0x5639343789e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x563934378f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563934192fa0 con 0x56393436edb0 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56393422d760 con 0x563934374d30 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56393437b800 con 0x5639341efe40 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e65bb4b80 10 client.0 _ll_register_callbacks cb 0x563934236c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e4fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639341efe40 0x5639343784a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e4fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639341efe40 0x5639343784a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:37264/0 (socket says 172.21.15.44:37264) 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e4fe00700 1 -- 172.21.15.44:0/2070690348 learned_addr learned my addr 172.21.15.44:0/2070690348 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e55000700 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x563934378f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.324+0000 7f5e47400700 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56393436edb0 0x5639343789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.325+0000 7f5e4fe00700 1 -- 172.21.15.44:0/2070690348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 msgr2=0x563934378f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.325+0000 7f5e4fe00700 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x563934378f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.325+0000 7f5e4fe00700 1 -- 172.21.15.44:0/2070690348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56393436edb0 msgr2=0x5639343789e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.325+0000 7f5e4fe00700 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56393436edb0 0x5639343789e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.325+0000 7f5e4fe00700 1 -- 172.21.15.44:0/2070690348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639341dc8d0 con 0x5639341efe40 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.325+0000 7f5e55000700 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x563934378f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.325+0000 7f5e4fe00700 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639341efe40 0x5639343784a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5e4001b040 tx=0x7f5e400038b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a45087dfb4d15ad7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.325+0000 7f5e3e000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.326+0000 7f5e3e000700 1 -- 172.21.15.44:0/2070690348 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40005240 con 0x5639341efe40 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.326+0000 7f5e3e000700 1 -- 172.21.15.44:0/2070690348 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f5e400053e0 con 0x5639341efe40 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.326+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2070690348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e400043d0 con 0x5639341efe40 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.326+0000 7f5e3e000700 1 -- 172.21.15.44:0/2070690348 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f5e400055e0 con 0x5639341efe40 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.326+0000 7f5e65bb4b80 10 client.27598 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.326+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2070690348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56393425d060 con 0x5639341efe40 2024-05-27T09:47:50.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.326+0000 7f5e3e000700 1 -- 172.21.15.44:0/2070690348 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e400043d0 con 0x5639341efe40 2024-05-27T09:47:50.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.327+0000 7f5e3e000700 1 -- 172.21.15.44:0/2070690348 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f5e4000e5a0 con 0x5639341efe40 2024-05-27T09:47:50.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.327+0000 7f5e65bb4b80 10 client.27598 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:50.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.327+0000 7f5e65bb4b80 10 client.27598 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:47:50.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.327+0000 7f5e65bb4b80 20 client.27598 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:50.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.327+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5639343885a0 0x56393438a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.327+0000 7f5e65bb4b80 4 client.27598 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.327+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2070690348 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56393429ed90 con 0x5639343885a0 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.327+0000 7f5e47400700 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5639343885a0 0x56393438a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.328+0000 7f5e47400700 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5639343885a0 0x56393438a9c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a75ff8e97e2028df server_cookie=3c799c26360f97f6 in_seq=0 out_seq=0 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.328+0000 7f5e3e000700 10 client.27598 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e3e000700 1 -- 172.21.15.44:0/2070690348 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56393429ed90 con 0x5639343885a0 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e3e000700 10 client.27598 handle_command_reply: tid=0 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e65bb4b80 1 client.27598 shutdown 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2070690348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639341efe40 msgr2=0x5639343784a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639341efe40 0x5639343784a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5e4001b040 tx=0x7f5e400038b0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2070690348 shutdown_connections 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x563934378f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639341efe40 0x5639343784a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5639343885a0 0x56393438a9c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/2070690348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56393436edb0 0x5639343789e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e3e000700 0 client.27598 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:47:50.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.329+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2070690348 >> 172.21.15.44:0/2070690348 conn(0x56393424ffc0 msgr2=0x56393436f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:50.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.330+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2070690348 shutdown_connections 2024-05-27T09:47:50.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.330+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/2070690348 wait complete. 2024-05-27T09:47:50.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.330+0000 7f5e65bb4b80 20 client.27598 trim_cache size 0 max 16384 2024-05-27T09:47:50.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:50 smithi189 ceph-mon[36671]: pgmap v395: 97 pgs: 97 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 236 op/s 2024-05-27T09:47:50.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.347+0000 7f5e65bb4b80 1 Processor -- start 2024-05-27T09:47:50.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.347+0000 7f5e65bb4b80 1 -- start start 2024-05-27T09:47:50.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.347+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x56393436e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x56393438a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x563934384860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5e501b7d40 con 0x5639341efe40 2024-05-27T09:47:50.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5e501b55a0 con 0x563934374d30 2024-05-27T09:47:50.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5e501b3410 con 0x56393436edb0 2024-05-27T09:47:50.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e55000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x563934384860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e4fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x56393436e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e55000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x563934384860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36870/0 (socket says 172.21.15.44:36870) 2024-05-27T09:47:50.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e55000700 1 -- 172.21.15.44:0/815106829 learned_addr learned my addr 172.21.15.44:0/815106829 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:50.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e47400700 1 --2- 172.21.15.44:0/815106829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x56393438a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e55000700 1 -- 172.21.15.44:0/815106829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 msgr2=0x56393438a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:50.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e55000700 1 --2- 172.21.15.44:0/815106829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x56393438a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e55000700 1 -- 172.21.15.44:0/815106829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 msgr2=0x56393436e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:50.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e55000700 1 --2- 172.21.15.44:0/815106829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x56393436e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e55000700 1 -- 172.21.15.44:0/815106829 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56393436f800 con 0x563934374d30 2024-05-27T09:47:50.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.348+0000 7f5e4fe00700 1 --2- 172.21.15.44:0/815106829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x56393436e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:50.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.349+0000 7f5e55000700 1 --2- 172.21.15.44:0/815106829 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x563934384860 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5e480136e0 tx=0x7f5e4801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1349a2444185113 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:50.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.349+0000 7f5e44c00700 1 -- 172.21.15.44:0/815106829 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4802d070 con 0x563934374d30 2024-05-27T09:47:50.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.349+0000 7f5e44c00700 1 -- 172.21.15.44:0/815106829 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e48008dd0 con 0x563934374d30 2024-05-27T09:47:50.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.349+0000 7f5e44c00700 1 -- 172.21.15.44:0/815106829 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e48021070 con 0x563934374d30 2024-05-27T09:47:50.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.350+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/815106829 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 msgr2=0x563934384860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:50.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.350+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/815106829 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x563934384860 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5e480136e0 tx=0x7f5e4801d000 comp rx=0 tx=0).stop 2024-05-27T09:47:50.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.350+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/815106829 shutdown_connections 2024-05-27T09:47:50.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.350+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/815106829 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x563934384860 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.350+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/815106829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x56393436e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.350+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/815106829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x56393438a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.350+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/815106829 >> 172.21.15.44:0/815106829 conn(0x56393424ffc0 msgr2=0x563934372140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:50.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.350+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/815106829 shutdown_connections 2024-05-27T09:47:50.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.350+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/815106829 wait complete. 2024-05-27T09:47:50.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.351+0000 7f5e65bb4b80 1 -- start start 2024-05-27T09:47:50.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.351+0000 7f5e65bb4b80 1 Processor -- start 2024-05-27T09:47:50.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x563934380a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x563934380f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e65bb4b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x56393438f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5e501b3a60 con 0x5639341efe40 2024-05-27T09:47:50.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5e501b4ed0 con 0x563934374d30 2024-05-27T09:47:50.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e65bb4b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5e501b4130 con 0x56393436edb0 2024-05-27T09:47:50.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e65bb4b80 10 client.0 _ll_register_callbacks cb 0x563934236c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:50.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e4fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x563934380a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e4fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x563934380a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41916/0 (socket says 172.21.15.44:41916) 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e55000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x56393438f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e47400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x563934380f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.352+0000 7f5e4fe00700 1 -- 172.21.15.44:0/3623273678 learned_addr learned my addr 172.21.15.44:0/3623273678 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e4fe00700 1 -- 172.21.15.44:0/3623273678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 msgr2=0x563934380f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e4fe00700 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x563934380f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e4fe00700 1 -- 172.21.15.44:0/3623273678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 msgr2=0x56393438f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e4fe00700 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x56393438f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e4fe00700 1 -- 172.21.15.44:0/3623273678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563934381670 con 0x5639341efe40 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e47400700 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x563934380f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e55000700 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x56393438f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e4fe00700 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x563934380a50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5e40000c00 tx=0x7f5e40009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23971de49dfb395b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:50.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e3e000700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:47:50.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e3e000700 1 -- 172.21.15.44:0/3623273678 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40036070 con 0x5639341efe40 2024-05-27T09:47:50.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/3623273678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e48008dd0 con 0x5639341efe40 2024-05-27T09:47:50.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.353+0000 7f5e3e000700 1 -- 172.21.15.44:0/3623273678 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f5e40022500 con 0x5639341efe40 2024-05-27T09:47:50.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.354+0000 7f5e3e000700 1 -- 172.21.15.44:0/3623273678 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f5e40022700 con 0x5639341efe40 2024-05-27T09:47:50.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.354+0000 7f5e65bb4b80 10 client.17361 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:50.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.354+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/3623273678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5639343f24b0 con 0x5639341efe40 2024-05-27T09:47:50.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.354+0000 7f5e3e000700 1 -- 172.21.15.44:0/3623273678 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e48008dd0 con 0x5639341efe40 2024-05-27T09:47:50.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.354+0000 7f5e3e000700 1 -- 172.21.15.44:0/3623273678 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f5e400425f0 con 0x5639341efe40 2024-05-27T09:47:50.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.354+0000 7f5e65bb4b80 10 client.17361 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:50.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.354+0000 7f5e65bb4b80 10 client.17361 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:47:50.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.354+0000 7f5e65bb4b80 20 client.17361 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:50.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.354+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5639343f2a90 0x5639343f4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:50.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.354+0000 7f5e65bb4b80 4 client.17361 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:47:50.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.354+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/3623273678 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563934298570 con 0x5639343f2a90 2024-05-27T09:47:50.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.355+0000 7f5e47400700 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5639343f2a90 0x5639343f4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:50.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.355+0000 7f5e47400700 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5639343f2a90 0x5639343f4eb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=70f4b9e4e1dfa74e server_cookie=15ee641fd425a966 in_seq=0 out_seq=0 2024-05-27T09:47:50.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.355+0000 7f5e3e000700 10 client.17361 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e3e000700 1 -- 172.21.15.44:0/3623273678 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563934298570 con 0x5639343f2a90 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e3e000700 10 client.17361 handle_command_reply: tid=0 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 1 client.17361 shutdown 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/3623273678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 msgr2=0x563934380a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x563934380a50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5e40000c00 tx=0x7f5e40009290 comp rx=0 tx=0).stop 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/3623273678 shutdown_connections 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563934374d30 0x56393438f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x5639343f2a90 0x5639343f4eb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5639341efe40 0x563934380a50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 1 --2- 172.21.15.44:0/3623273678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56393436edb0 0x563934380f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/3623273678 >> 172.21.15.44:0/3623273678 conn(0x56393424ffc0 msgr2=0x563934373d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e3e000700 0 client.17361 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/3623273678 shutdown_connections 2024-05-27T09:47:50.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 1 -- 172.21.15.44:0/3623273678 wait complete. 2024-05-27T09:47:50.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.356+0000 7f5e65bb4b80 20 client.17361 trim_cache size 0 max 16384 2024-05-27T09:47:50.360 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:47:50.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.357+0000 7f5e57000700 1 -- 172.21.15.44:0/2398441566 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5e340b29f0 msgr2=0x7f5e340b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:50.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.357+0000 7f5e57000700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5e340b29f0 0x7f5e340b4f00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009ec0 tx=0x7f5e480134a0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.357+0000 7f5e57000700 1 -- 172.21.15.44:0/2398441566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e5010b2d0 msgr2=0x7f5e501a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:50.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.357+0000 7f5e57000700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e5010b2d0 0x7f5e501a1a40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e38000c00 tx=0x7f5e380057c0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.357+0000 7f5e57000700 1 -- 172.21.15.44:0/2398441566 shutdown_connections 2024-05-27T09:47:50.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.357+0000 7f5e57000700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5e340b29f0 0x7f5e340b4f00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.357+0000 7f5e57000700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e50110960 0x7f5e501a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.357+0000 7f5e57000700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e5010a980 0x7f5e501a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.357+0000 7f5e57000700 1 --2- 172.21.15.44:0/2398441566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e5010b2d0 0x7f5e501a1a40 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:50.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.357+0000 7f5e57000700 1 -- 172.21.15.44:0/2398441566 >> 172.21.15.44:0/2398441566 conn(0x7f5e50077c70 msgr2=0x7f5e5007b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:50.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.358+0000 7f5e57000700 1 -- 172.21.15.44:0/2398441566 shutdown_connections 2024-05-27T09:47:50.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:50.358+0000 7f5e57000700 1 -- 172.21.15.44:0/2398441566 wait complete. 2024-05-27T09:47:50.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:50 smithi050 ceph-mon[36371]: pgmap v395: 97 pgs: 97 active+clean; 309 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 236 op/s 2024-05-27T09:47:51.177 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:47:51.659 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:51.734 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:47:52.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:52 smithi189 ceph-mon[36671]: pgmap v396: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-05-27T09:47:52.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:52 smithi044 ceph-mon[31618]: pgmap v396: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-05-27T09:47:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:52 smithi050 ceph-mon[36371]: pgmap v396: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-05-27T09:47:52.902 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:54.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:54 smithi189 ceph-mon[36671]: pgmap v397: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 502 KiB/s wr, 116 op/s 2024-05-27T09:47:54.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:54 smithi044 ceph-mon[31618]: pgmap v397: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 502 KiB/s wr, 116 op/s 2024-05-27T09:47:54.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:54 smithi050 ceph-mon[36371]: pgmap v397: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 502 KiB/s wr, 116 op/s 2024-05-27T09:47:54.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.492+0000 7fb8b2e00700 1 -- 172.21.15.44:0/681288112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8ac10b100 msgr2=0x7fb8ac117610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:54.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.492+0000 7fb8b2e00700 1 --2- 172.21.15.44:0/681288112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8ac10b100 0x7fb8ac117610 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400b0a0 tx=0x7fb8a400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.493+0000 7fb8b2e00700 1 -- 172.21.15.44:0/681288112 shutdown_connections 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.493+0000 7fb8b2e00700 1 --2- 172.21.15.44:0/681288112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8ac10b100 0x7fb8ac117610 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.493+0000 7fb8b2e00700 1 --2- 172.21.15.44:0/681288112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ac10f6b0 0x7fb8ac10fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.493+0000 7fb8b2e00700 1 --2- 172.21.15.44:0/681288112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac10a7b0 0x7fb8ac10abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.493+0000 7fb8b2e00700 1 -- 172.21.15.44:0/681288112 >> 172.21.15.44:0/681288112 conn(0x7fb8ac076af0 msgr2=0x7fb8ac076f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.493+0000 7fb8b2e00700 1 -- 172.21.15.44:0/681288112 shutdown_connections 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.493+0000 7fb8b2e00700 1 -- 172.21.15.44:0/681288112 wait complete. 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.494+0000 7fb8b2e00700 1 Processor -- start 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.494+0000 7fb8b2e00700 1 -- start start 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.495+0000 7fb8b2e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac10a7b0 0x7fb8ac1a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.495+0000 7fb8b2e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8ac10b100 0x7fb8ac1a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.495+0000 7fb8b2e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ac10f6b0 0x7fb8ac1a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.495+0000 7fb8b2e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac11a8a0 con 0x7fb8ac10b100 2024-05-27T09:47:54.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.495+0000 7fb8b2e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac11a720 con 0x7fb8ac10a7b0 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.495+0000 7fb8b2e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac11aa20 con 0x7fb8ac10f6b0 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.495+0000 7fb8b1000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8ac10b100 0x7fb8ac1a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.495+0000 7fb8b1000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8ac10b100 0x7fb8ac1a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40470/0 (socket says 172.21.15.44:40470) 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.495+0000 7fb8b1000700 1 -- 172.21.15.44:0/4003328806 learned_addr learned my addr 172.21.15.44:0/4003328806 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.495+0000 7fb8b2400700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ac10f6b0 0x7fb8ac1a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.495+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac10a7b0 0x7fb8ac1a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.496+0000 7fb8b1000700 1 -- 172.21.15.44:0/4003328806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ac10f6b0 msgr2=0x7fb8ac1a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.496+0000 7fb8b1000700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ac10f6b0 0x7fb8ac1a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.496+0000 7fb8b1000700 1 -- 172.21.15.44:0/4003328806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac10a7b0 msgr2=0x7fb8ac1a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.496+0000 7fb8b1000700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac10a7b0 0x7fb8ac1a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.496+0000 7fb8b1000700 1 -- 172.21.15.44:0/4003328806 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ac1a6020 con 0x7fb8ac10b100 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.496+0000 7fb8b1000700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8ac10b100 0x7fb8ac1a16f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00e9a0 tx=0x7fb89c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.497+0000 7fb8aa000700 1 -- 172.21.15.44:0/4003328806 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c00ca10 con 0x7fb8ac10b100 2024-05-27T09:47:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.497+0000 7fb8aa000700 1 -- 172.21.15.44:0/4003328806 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb89c00cbb0 con 0x7fb8ac10b100 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.497+0000 7fb8b2e00700 1 -- 172.21.15.44:0/4003328806 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ac1a6310 con 0x7fb8ac10b100 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.497+0000 7fb8aa000700 1 -- 172.21.15.44:0/4003328806 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c018680 con 0x7fb8ac10b100 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.497+0000 7fb8b2e00700 1 -- 172.21.15.44:0/4003328806 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8ac1adc30 con 0x7fb8ac10b100 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.503+0000 7fb8aa000700 1 -- 172.21.15.44:0/4003328806 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb89c010430 con 0x7fb8ac10b100 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.503+0000 7fb8c2e8db80 1 Processor -- start 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.503+0000 7fb8c2e8db80 1 -- start start 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.504+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 0x55ffbd05ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.504+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbd05d410 0x55ffbd067ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.504+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d820 0x55ffbd06a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.504+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ffbcf8be40 con 0x55ffbd05d820 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.504+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ffbcf7b9e0 con 0x55ffbcedee40 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.504+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ffbcf7fa20 con 0x55ffbd05d410 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.504+0000 7fb8b2400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 0x55ffbd05ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.504+0000 7fb8b2400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 0x55ffbd05ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52832/0 (socket says 172.21.15.44:52832) 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.504+0000 7fb8b2400700 1 -- 172.21.15.44:0/204637025 learned_addr learned my addr 172.21.15.44:0/204637025 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.504+0000 7fb8b1a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbd05d410 0x55ffbd067ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.504+0000 7fb8b1000700 1 --2- 172.21.15.44:0/204637025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d820 0x55ffbd06a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.505+0000 7fb8b2400700 1 -- 172.21.15.44:0/204637025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbd05d410 msgr2=0x55ffbd067ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.505+0000 7fb8b2400700 1 --2- 172.21.15.44:0/204637025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbd05d410 0x55ffbd067ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.505+0000 7fb8b2400700 1 -- 172.21.15.44:0/204637025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d820 msgr2=0x55ffbd06a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.505+0000 7fb8b2400700 1 --2- 172.21.15.44:0/204637025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d820 0x55ffbd06a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.505+0000 7fb8b2400700 1 -- 172.21.15.44:0/204637025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffbcf44420 con 0x55ffbcedee40 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.505+0000 7fb8aa000700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb8940b29f0 0x7fb8940b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.505+0000 7fb8aa000700 1 -- 172.21.15.44:0/4003328806 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7fb89c0e2800 con 0x7fb8ac10b100 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.505+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb8940b29f0 0x7fb8940b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.505+0000 7fb8b2400700 1 --2- 172.21.15.44:0/204637025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 0x55ffbd05ced0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4015000 tx=0x7fb8a4009d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfb74016b0c71d7e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.506+0000 7fb89aa00700 1 -- 172.21.15.44:0/204637025 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400bcb0 con 0x55ffbcedee40 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.506+0000 7fb89aa00700 1 -- 172.21.15.44:0/204637025 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8a4009460 con 0x55ffbcedee40 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.506+0000 7fb89aa00700 1 -- 172.21.15.44:0/204637025 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4005040 con 0x55ffbcedee40 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.507+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/204637025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 msgr2=0x55ffbd05ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.507+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb8940b29f0 0x7fb8940b4f00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55ffbd05e430 tx=0x7fb8a0009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.507+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/204637025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 0x55ffbd05ced0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4015000 tx=0x7fb8a4009d90 comp rx=0 tx=0).stop 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.507+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/204637025 shutdown_connections 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.507+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/204637025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d820 0x55ffbd06a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.507+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/204637025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 0x55ffbd05ced0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.507+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/204637025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbd05d410 0x55ffbd067ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.507+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/204637025 >> 172.21.15.44:0/204637025 conn(0x55ffbcf3efc0 msgr2=0x55ffbd058d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.507+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/204637025 shutdown_connections 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.507+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/204637025 wait complete. 2024-05-27T09:47:54.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.507+0000 7fb8c2e8db80 1 -- start start 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.508+0000 7fb8c2e8db80 1 Processor -- start 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.508+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbcedee40 0x55ffbd067630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.508+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd067b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.508+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbd05d820 0x55ffbd0680b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.508+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ffbce81fa0 con 0x55ffbd05d410 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.508+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ffbcf1c760 con 0x55ffbd05d820 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.508+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ffbd06a930 con 0x55ffbcedee40 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.508+0000 7fb8c2e8db80 10 client.0 _ll_register_callbacks cb 0x55ffbcf25c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb8b1000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd067b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb8b1000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd067b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40488/0 (socket says 172.21.15.44:40488) 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb8b1000700 1 -- 172.21.15.44:0/3596017509 learned_addr learned my addr 172.21.15.44:0/3596017509 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:54.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbcedee40 0x55ffbd067630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb8b2400700 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbd05d820 0x55ffbd0680b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb8b1000700 1 -- 172.21.15.44:0/3596017509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbcedee40 msgr2=0x55ffbd067630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb8b1000700 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbcedee40 0x55ffbd067630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb8b1000700 1 -- 172.21.15.44:0/3596017509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbd05d820 msgr2=0x55ffbd0680b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb8b1000700 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbd05d820 0x55ffbd0680b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb8b1000700 1 -- 172.21.15.44:0/3596017509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffbcecb8d0 con 0x55ffbd05d410 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb8b1000700 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd067b70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb89c000c00 tx=0x7fb89c0a9f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de4425091ec15bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.509+0000 7fb88f400700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.510+0000 7fb88f400700 1 -- 172.21.15.44:0/3596017509 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c0a5040 con 0x55ffbd05d410 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.510+0000 7fb88f400700 1 -- 172.21.15.44:0/3596017509 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7fb89c0cc300 con 0x55ffbd05d410 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.510+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/3596017509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8a4009460 con 0x55ffbd05d410 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.510+0000 7fb88f400700 1 -- 172.21.15.44:0/3596017509 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fb89c0a51e0 con 0x55ffbd05d410 2024-05-27T09:47:54.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.510+0000 7fb8c2e8db80 10 client.17373 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.510+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/3596017509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55ffbcf4c060 con 0x55ffbd05d410 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.510+0000 7fb88f400700 1 -- 172.21.15.44:0/3596017509 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8a4009460 con 0x55ffbd05d410 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.511+0000 7fb88f400700 1 -- 172.21.15.44:0/3596017509 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7fb89c010720 con 0x55ffbd05d410 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.511+0000 7fb8c2e8db80 10 client.17373 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.511+0000 7fb8c2e8db80 10 client.17373 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.511+0000 7fb8c2e8db80 20 client.17373 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.511+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ffbd077840 0x55ffbd079c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.511+0000 7fb8c2e8db80 4 client.17373 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.511+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/3596017509 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffbcf8dd90 con 0x55ffbd077840 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.511+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ffbd077840 0x55ffbd079c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.512+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ffbd077840 0x55ffbd079c60 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58f46e3dcb12f9c4 server_cookie=d002d3d21339168f in_seq=0 out_seq=0 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.512+0000 7fb88f400700 10 client.17373 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb88f400700 1 -- 172.21.15.44:0/3596017509 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ffbcf8dd90 con 0x55ffbd077840 2024-05-27T09:47:54.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb88f400700 10 client.17373 handle_command_reply: tid=0 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 1 client.17373 shutdown 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/3596017509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 msgr2=0x55ffbd067b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd067b70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb89c000c00 tx=0x7fb89c0a9f90 comp rx=0 tx=0).stop 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/3596017509 shutdown_connections 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ffbd077840 0x55ffbd079c60 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbd05d820 0x55ffbd0680b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbcedee40 0x55ffbd067630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/3596017509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd067b70 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb88f400700 0 client.17373 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/3596017509 >> 172.21.15.44:0/3596017509 conn(0x55ffbcf3efc0 msgr2=0x55ffbd057ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/3596017509 shutdown_connections 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/3596017509 wait complete. 2024-05-27T09:47:54.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.513+0000 7fb8c2e8db80 20 client.17373 trim_cache size 0 max 16384 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.531+0000 7fb8c2e8db80 1 Processor -- start 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.531+0000 7fb8c2e8db80 1 -- start start 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 0x55ffbd062e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd069ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbd05d820 0x55ffbd066590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac1b7b50 con 0x55ffbd05d410 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac1b53b0 con 0x55ffbcedee40 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac1b3200 con 0x55ffbd05d820 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8b1000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd069ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8b1a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 0x55ffbd062e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8b1000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd069ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40492/0 (socket says 172.21.15.44:40492) 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8b1000700 1 -- 172.21.15.44:0/371183884 learned_addr learned my addr 172.21.15.44:0/371183884 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:54.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8b2400700 1 --2- 172.21.15.44:0/371183884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbd05d820 0x55ffbd066590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8b1a00700 1 -- 172.21.15.44:0/371183884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbd05d820 msgr2=0x55ffbd066590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/371183884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbd05d820 0x55ffbd066590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8b1a00700 1 -- 172.21.15.44:0/371183884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 msgr2=0x55ffbd069ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/371183884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd069ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8b1a00700 1 -- 172.21.15.44:0/371183884 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffbd0531d0 con 0x55ffbcedee40 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.532+0000 7fb8b2400700 1 --2- 172.21.15.44:0/371183884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbd05d820 0x55ffbd066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.533+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/371183884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 0x55ffbd062e30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x55ffbd07a350 tx=0x7fb8a001b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdd2dd97e65c398d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.533+0000 7fb89a000700 1 -- 172.21.15.44:0/371183884 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0003a60 con 0x55ffbcedee40 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.533+0000 7fb89a000700 1 -- 172.21.15.44:0/371183884 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8a0003c00 con 0x55ffbcedee40 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.533+0000 7fb89a000700 1 -- 172.21.15.44:0/371183884 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a001e750 con 0x55ffbcedee40 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.534+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/371183884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 msgr2=0x55ffbd062e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.534+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/371183884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 0x55ffbd062e30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x55ffbd07a350 tx=0x7fb8a001b000 comp rx=0 tx=0).stop 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.534+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/371183884 shutdown_connections 2024-05-27T09:47:54.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.534+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/371183884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbd05d820 0x55ffbd066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.534+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/371183884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbcedee40 0x55ffbd062e30 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.534+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/371183884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd069ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.534+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/371183884 >> 172.21.15.44:0/371183884 conn(0x55ffbcf3efc0 msgr2=0x55ffbd067e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.534+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/371183884 shutdown_connections 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.534+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/371183884 wait complete. 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.535+0000 7fb8c2e8db80 1 -- start start 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.535+0000 7fb8c2e8db80 1 Processor -- start 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbcedee40 0x55ffbd070470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd06b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8c2e8db80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbd05d820 0x55ffbd06be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac1b3870 con 0x55ffbd05d410 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac1b4ce0 con 0x55ffbd05d820 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8c2e8db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac1b3f40 con 0x55ffbcedee40 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8c2e8db80 10 client.0 _ll_register_callbacks cb 0x55ffbcf25c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:54.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8b1000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd06b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8b1000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd06b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40494/0 (socket says 172.21.15.44:40494) 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8b1000700 1 -- 172.21.15.44:0/1090531969 learned_addr learned my addr 172.21.15.44:0/1090531969 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbcedee40 0x55ffbd070470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8b2400700 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbd05d820 0x55ffbd06be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8b1000700 1 -- 172.21.15.44:0/1090531969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbcedee40 msgr2=0x55ffbd070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8b1000700 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbcedee40 0x55ffbd070470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8b1000700 1 -- 172.21.15.44:0/1090531969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbd05d820 msgr2=0x55ffbd06be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8b1000700 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbd05d820 0x55ffbd06be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.536+0000 7fb8b1000700 1 -- 172.21.15.44:0/1090531969 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffbd05a350 con 0x55ffbd05d410 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.537+0000 7fb8b1000700 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd06b7d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009230 tx=0x7fb89c0a8fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c59bd2ffdc194ca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.537+0000 7fb88f400700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.537+0000 7fb88f400700 1 -- 172.21.15.44:0/1090531969 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c0a5040 con 0x55ffbd05d410 2024-05-27T09:47:54.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.537+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/1090531969 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8a0003c00 con 0x55ffbd05d410 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.537+0000 7fb88f400700 1 -- 172.21.15.44:0/1090531969 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7fb89c0a6040 con 0x55ffbd05d410 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.537+0000 7fb88f400700 1 -- 172.21.15.44:0/1090531969 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fb89c0b21f0 con 0x55ffbd05d410 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.537+0000 7fb8c2e8db80 10 client.17379 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.537+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/1090531969 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55ffbd0e2170 con 0x55ffbd05d410 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.538+0000 7fb88f400700 1 -- 172.21.15.44:0/1090531969 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8a0003c00 con 0x55ffbd05d410 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.538+0000 7fb88f400700 1 -- 172.21.15.44:0/1090531969 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7fb89c06f1a0 con 0x55ffbd05d410 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.538+0000 7fb8c2e8db80 10 client.17379 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.538+0000 7fb8c2e8db80 10 client.17379 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.538+0000 7fb8c2e8db80 20 client.17379 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.538+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ffbd0e2760 0x55ffbd0e4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.538+0000 7fb8c2e8db80 4 client.17379 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.538+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/1090531969 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ffbcf87570 con 0x55ffbd0e2760 2024-05-27T09:47:54.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.538+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ffbd0e2760 0x55ffbd0e4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb8b1a00700 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ffbd0e2760 0x55ffbd0e4b80 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e82d5ef754b998b0 server_cookie=6497a10dab4025d4 in_seq=0 out_seq=0 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb88f400700 10 client.17379 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb88f400700 1 -- 172.21.15.44:0/1090531969 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ffbcf87570 con 0x55ffbd0e2760 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb88f400700 10 client.17379 handle_command_reply: tid=0 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb8c2e8db80 1 client.17379 shutdown 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/1090531969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 msgr2=0x55ffbd06b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd06b7d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009230 tx=0x7fb89c0a8fe0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/1090531969 shutdown_connections 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ffbd0e2760 0x55ffbd0e4b80 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffbd05d820 0x55ffbd06be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ffbcedee40 0x55ffbd070470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb8c2e8db80 1 --2- 172.21.15.44:0/1090531969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ffbd05d410 0x55ffbd06b7d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/1090531969 >> 172.21.15.44:0/1090531969 conn(0x55ffbcf3efc0 msgr2=0x55ffbd067e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.539+0000 7fb88f400700 0 client.17379 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.540+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/1090531969 shutdown_connections 2024-05-27T09:47:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.540+0000 7fb8c2e8db80 1 -- 172.21.15.44:0/1090531969 wait complete. 2024-05-27T09:47:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.540+0000 7fb8c2e8db80 20 client.17379 trim_cache size 0 max 16384 2024-05-27T09:47:54.900 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:47:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.540+0000 7fb8b2e00700 1 -- 172.21.15.44:0/4003328806 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb8940b29f0 msgr2=0x7fb8940b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.541+0000 7fb8b2e00700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb8940b29f0 0x7fb8940b4f00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55ffbd05e430 tx=0x7fb8a0009260 comp rx=0 tx=0).stop 2024-05-27T09:47:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.541+0000 7fb8b2e00700 1 -- 172.21.15.44:0/4003328806 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8ac10b100 msgr2=0x7fb8ac1a16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.541+0000 7fb8b2e00700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8ac10b100 0x7fb8ac1a16f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00e9a0 tx=0x7fb89c00ecb0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.541+0000 7fb8b2e00700 1 -- 172.21.15.44:0/4003328806 shutdown_connections 2024-05-27T09:47:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.541+0000 7fb8b2e00700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb8940b29f0 0x7fb8940b4f00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.541+0000 7fb8b2e00700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ac10f6b0 0x7fb8ac1a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.541+0000 7fb8b2e00700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac10a7b0 0x7fb8ac1a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.541+0000 7fb8b2e00700 1 --2- 172.21.15.44:0/4003328806 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8ac10b100 0x7fb8ac1a16f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.541+0000 7fb8b2e00700 1 -- 172.21.15.44:0/4003328806 >> 172.21.15.44:0/4003328806 conn(0x7fb8ac076af0 msgr2=0x7fb8ac1087b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.541+0000 7fb8b2e00700 1 -- 172.21.15.44:0/4003328806 shutdown_connections 2024-05-27T09:47:54.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:54.541+0000 7fb8b2e00700 1 -- 172.21.15.44:0/4003328806 wait complete. 2024-05-27T09:47:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.815+0000 7fbf5da00640 1 -- 172.21.15.44:0/3193713730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf58110470 msgr2=0x7fbf581128d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.815+0000 7fbf5da00640 1 --2- 172.21.15.44:0/3193713730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf58110470 0x7fbf581128d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009ca0 tx=0x7fbf5002a950 comp rx=0 tx=0).stop 2024-05-27T09:47:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.815+0000 7fbf5da00640 1 -- 172.21.15.44:0/3193713730 shutdown_connections 2024-05-27T09:47:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.815+0000 7fbf5da00640 1 --2- 172.21.15.44:0/3193713730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf58110470 0x7fbf581128d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.815+0000 7fbf5da00640 1 --2- 172.21.15.44:0/3193713730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf5810db10 0x7fbf5810ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.815+0000 7fbf5da00640 1 --2- 172.21.15.44:0/3193713730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58076770 0x7fbf58076b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.815+0000 7fbf5da00640 1 -- 172.21.15.44:0/3193713730 >> 172.21.15.44:0/3193713730 conn(0x7fbf58079290 msgr2=0x7fbf580796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.815+0000 7fbf5da00640 1 -- 172.21.15.44:0/3193713730 shutdown_connections 2024-05-27T09:47:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.815+0000 7fbf5da00640 1 -- 172.21.15.44:0/3193713730 wait complete. 2024-05-27T09:47:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.817+0000 7fbf5da00640 1 Processor -- start 2024-05-27T09:47:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.817+0000 7fbf5da00640 1 -- start start 2024-05-27T09:47:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.818+0000 7fbf5da00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf58076770 0x7fbf581a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.818+0000 7fbf5da00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5810db10 0x7fbf581a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.818+0000 7fbf5da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58110470 0x7fbf5819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.818+0000 7fbf5da00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbf58116080 con 0x7fbf58076770 2024-05-27T09:47:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.818+0000 7fbf5da00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbf58115f00 con 0x7fbf5810db10 2024-05-27T09:47:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.818+0000 7fbf5da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbf58116200 con 0x7fbf58110470 2024-05-27T09:47:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.818+0000 7fbf56a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf58076770 0x7fbf581a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.818+0000 7fbf56a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf58076770 0x7fbf581a26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40522/0 (socket says 172.21.15.44:40522) 2024-05-27T09:47:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.818+0000 7fbf56a00640 1 -- 172.21.15.44:0/2003452068 learned_addr learned my addr 172.21.15.44:0/2003452068 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.818+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5810db10 0x7fbf581a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.818+0000 7fbf57400640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58110470 0x7fbf5819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.819+0000 7fbf56a00640 1 -- 172.21.15.44:0/2003452068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58110470 msgr2=0x7fbf5819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.819+0000 7fbf56a00640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58110470 0x7fbf5819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.819+0000 7fbf56a00640 1 -- 172.21.15.44:0/2003452068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5810db10 msgr2=0x7fbf581a2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.819+0000 7fbf56a00640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5810db10 0x7fbf581a2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.819+0000 7fbf56a00640 1 -- 172.21.15.44:0/2003452068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5819cfa0 con 0x7fbf58076770 2024-05-27T09:47:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.819+0000 7fbf56a00640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf58076770 0x7fbf581a26f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800e9f0 tx=0x7fbf4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.820+0000 7fbf4f400640 1 -- 172.21.15.44:0/2003452068 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4800cd80 con 0x7fbf58076770 2024-05-27T09:47:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.820+0000 7fbf5da00640 1 -- 172.21.15.44:0/2003452068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf5819d290 con 0x7fbf58076770 2024-05-27T09:47:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.820+0000 7fbf4f400640 1 -- 172.21.15.44:0/2003452068 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf480054d0 con 0x7fbf58076770 2024-05-27T09:47:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.820+0000 7fbf4f400640 1 -- 172.21.15.44:0/2003452068 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf48010690 con 0x7fbf58076770 2024-05-27T09:47:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.820+0000 7fbf5da00640 1 -- 172.21.15.44:0/2003452068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf581a9570 con 0x7fbf58076770 2024-05-27T09:47:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.825+0000 7fbf649f3740 1 Processor -- start 2024-05-27T09:47:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.825+0000 7fbf649f3740 1 -- start start 2024-05-27T09:47:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.825+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc4e1b0 0x56553fff0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.825+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc252f0 0x56553fff0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.825+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc2d4d0 0x56553fff12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.825+0000 7fbf649f3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56553ff61b90 con 0x56553fc4e1b0 2024-05-27T09:47:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.825+0000 7fbf649f3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56553ff51730 con 0x56553fc252f0 2024-05-27T09:47:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.825+0000 7fbf649f3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56553ff55770 con 0x56553fc2d4d0 2024-05-27T09:47:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.826+0000 7fbf57400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc4e1b0 0x56553fff0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.826+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc2d4d0 0x56553fff12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.826+0000 7fbf57400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc4e1b0 0x56553fff0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40524/0 (socket says 172.21.15.44:40524) 2024-05-27T09:47:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.826+0000 7fbf57400640 1 -- 172.21.15.44:0/3285057654 learned_addr learned my addr 172.21.15.44:0/3285057654 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.826+0000 7fbf4f400640 1 -- 172.21.15.44:0/2003452068 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fbf48004e90 con 0x7fbf58076770 2024-05-27T09:47:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.826+0000 7fbf4fe00640 1 -- 172.21.15.44:0/3285057654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc252f0 msgr2=0x56553fff0da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T09:47:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.826+0000 7fbf4fe00640 1 --2- 172.21.15.44:0/3285057654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc252f0 0x56553fff0da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.826+0000 7fbf4fe00640 1 -- 172.21.15.44:0/3285057654 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc4e1b0 msgr2=0x56553fff0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.826+0000 7fbf4fe00640 1 --2- 172.21.15.44:0/3285057654 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc4e1b0 0x56553fff0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.826+0000 7fbf4fe00640 1 -- 172.21.15.44:0/3285057654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56553feea8b0 con 0x56553fc2d4d0 2024-05-27T09:47:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.827+0000 7fbf4fe00640 1 --2- 172.21.15.44:0/3285057654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc2d4d0 0x56553fff12e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbf58079970 tx=0x7fbf40011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd5956dd734a4a89 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.827+0000 7fbf47e00640 1 -- 172.21.15.44:0/3285057654 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf40004050 con 0x56553fc2d4d0 2024-05-27T09:47:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.827+0000 7fbf47e00640 1 -- 172.21.15.44:0/3285057654 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf400054d0 con 0x56553fc2d4d0 2024-05-27T09:47:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.827+0000 7fbf47e00640 1 -- 172.21.15.44:0/3285057654 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf40004ba0 con 0x56553fc2d4d0 2024-05-27T09:47:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.828+0000 7fbf4f400640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbf3c0b2950 0x7fbf3c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.829+0000 7fbf4f400640 1 -- 172.21.15.44:0/2003452068 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7fbf480e2ad0 con 0x7fbf58076770 2024-05-27T09:47:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.829+0000 7fbf649f3740 1 -- 172.21.15.44:0/3285057654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc2d4d0 msgr2=0x56553fff12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.829+0000 7fbf649f3740 1 --2- 172.21.15.44:0/3285057654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc2d4d0 0x56553fff12e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbf58079970 tx=0x7fbf40011810 comp rx=0 tx=0).stop 2024-05-27T09:47:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.829+0000 7fbf57400640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbf3c0b2950 0x7fbf3c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.829+0000 7fbf649f3740 1 -- 172.21.15.44:0/3285057654 shutdown_connections 2024-05-27T09:47:55.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.829+0000 7fbf649f3740 1 --2- 172.21.15.44:0/3285057654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc2d4d0 0x56553fff12e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.829+0000 7fbf649f3740 1 --2- 172.21.15.44:0/3285057654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc252f0 0x56553fff0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.829+0000 7fbf649f3740 1 --2- 172.21.15.44:0/3285057654 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc4e1b0 0x56553fff0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.829+0000 7fbf649f3740 1 -- 172.21.15.44:0/3285057654 >> 172.21.15.44:0/3285057654 conn(0x56553fc42300 msgr2=0x56553ffed010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.830+0000 7fbf57400640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbf3c0b2950 0x7fbf3c0b4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbf500094e0 tx=0x7fbf50005d80 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.830+0000 7fbf649f3740 1 -- 172.21.15.44:0/3285057654 shutdown_connections 2024-05-27T09:47:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.830+0000 7fbf649f3740 1 -- 172.21.15.44:0/3285057654 wait complete. 2024-05-27T09:47:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.831+0000 7fbf649f3740 1 -- start start 2024-05-27T09:47:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.831+0000 7fbf649f3740 1 Processor -- start 2024-05-27T09:47:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.831+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc252f0 0x56553fffa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 0x56553fffaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc4e1b0 0x565540006cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf649f3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56553fc37790 con 0x56553fc4e1b0 2024-05-27T09:47:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf649f3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56553fef9bc0 con 0x56553fc2d4d0 2024-05-27T09:47:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf649f3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56553fffd990 con 0x56553fc252f0 2024-05-27T09:47:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf649f3740 10 client.0 _ll_register_callbacks cb 0x56553fed7400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf56a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc252f0 0x56553fffa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf57400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc4e1b0 0x565540006cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf56a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc252f0 0x56553fffa9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39226/0 (socket says 172.21.15.44:39226) 2024-05-27T09:47:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf56a00640 1 -- 172.21.15.44:0/1076867434 learned_addr learned my addr 172.21.15.44:0/1076867434 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf4fe00640 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 0x56553fffaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf56a00640 1 -- 172.21.15.44:0/1076867434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 msgr2=0x56553fffaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:55.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf56a00640 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 0x56553fffaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf56a00640 1 -- 172.21.15.44:0/1076867434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc4e1b0 msgr2=0x565540006cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:55.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf56a00640 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc4e1b0 0x565540006cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.832+0000 7fbf56a00640 1 -- 172.21.15.44:0/1076867434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56553fc66210 con 0x56553fc252f0 2024-05-27T09:47:55.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.833+0000 7fbf56a00640 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc252f0 0x56553fffa9a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800e9c0 tx=0x7fbf480048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=323282aa5b77bb93 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:55.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.833+0000 7fbf44c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:55.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.833+0000 7fbf44c00640 1 -- 172.21.15.44:0/1076867434 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf480c0030 con 0x56553fc252f0 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.833+0000 7fbf649f3740 1 -- 172.21.15.44:0/1076867434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf400054d0 con 0x56553fc252f0 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.833+0000 7fbf44c00640 1 -- 172.21.15.44:0/1076867434 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7fbf480a6310 con 0x56553fc252f0 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.833+0000 7fbf44c00640 1 -- 172.21.15.44:0/1076867434 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fbf480e0620 con 0x56553fc252f0 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.833+0000 7fbf649f3740 10 client.27634 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.833+0000 7fbf649f3740 1 -- 172.21.15.44:0/1076867434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56553fd32620 con 0x56553fc252f0 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.834+0000 7fbf44c00640 1 -- 172.21.15.44:0/1076867434 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf400054d0 con 0x56553fc252f0 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.834+0000 7fbf44c00640 1 -- 172.21.15.44:0/1076867434 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7fbf480a5690 con 0x56553fc252f0 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.834+0000 7fbf649f3740 10 client.27634 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.834+0000 7fbf649f3740 10 client.27634 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.834+0000 7fbf649f3740 20 client.27634 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.834+0000 7fbf649f3740 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56553fc0cd80 0x56554000cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.834+0000 7fbf649f3740 4 client.27634 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.834+0000 7fbf649f3740 1 -- 172.21.15.44:0/1076867434 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56553ff63ae0 con 0x56553fc0cd80 2024-05-27T09:47:55.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.835+0000 7fbf4fe00640 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56553fc0cd80 0x56554000cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.835+0000 7fbf4fe00640 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56553fc0cd80 0x56554000cc00 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9be8c3312d55b0c server_cookie=2c9d9f318e09dc1 in_seq=0 out_seq=0 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.835+0000 7fbf44c00640 10 client.27634 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf44c00640 1 -- 172.21.15.44:0/1076867434 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56553ff63ae0 con 0x56553fc0cd80 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf44c00640 10 client.27634 handle_command_reply: tid=0 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf649f3740 1 client.27634 shutdown 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf649f3740 1 -- 172.21.15.44:0/1076867434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc252f0 msgr2=0x56553fffa9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf649f3740 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc252f0 0x56553fffa9a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800e9c0 tx=0x7fbf480048f0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf649f3740 1 -- 172.21.15.44:0/1076867434 shutdown_connections 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf649f3740 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56553fc0cd80 0x56554000cc00 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf649f3740 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc4e1b0 0x565540006cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf649f3740 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 0x56553fffaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf44c00640 0 client.27634 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf649f3740 1 --2- 172.21.15.44:0/1076867434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc252f0 0x56553fffa9a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.836+0000 7fbf649f3740 1 -- 172.21.15.44:0/1076867434 >> 172.21.15.44:0/1076867434 conn(0x56553fc42300 msgr2=0x56553fff2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.837+0000 7fbf649f3740 1 -- 172.21.15.44:0/1076867434 shutdown_connections 2024-05-27T09:47:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.837+0000 7fbf649f3740 1 -- 172.21.15.44:0/1076867434 wait complete. 2024-05-27T09:47:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.837+0000 7fbf649f3740 20 client.27634 trim_cache size 0 max 16384 2024-05-27T09:47:55.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.854+0000 7fbf649f3740 1 Processor -- start 2024-05-27T09:47:55.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.854+0000 7fbf649f3740 1 -- start start 2024-05-27T09:47:55.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.855+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc252f0 0x56553fffcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.855+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 0x56553ffec330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.855+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 0x56553fffa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.855+0000 7fbf649f3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbf581b46e0 con 0x56553fc252f0 2024-05-27T09:47:55.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.855+0000 7fbf649f3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbf581b1f80 con 0x56553fc2d4d0 2024-05-27T09:47:55.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.855+0000 7fbf649f3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbf581afdd0 con 0x56553fc4e1b0 2024-05-27T09:47:55.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.855+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 0x56553ffec330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.855+0000 7fbf57400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 0x56553fffa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.855+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 0x56553ffec330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52906/0 (socket says 172.21.15.44:52906) 2024-05-27T09:47:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.855+0000 7fbf4fe00640 1 -- 172.21.15.44:0/1922592177 learned_addr learned my addr 172.21.15.44:0/1922592177 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.855+0000 7fbf56a00640 1 --2- 172.21.15.44:0/1922592177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc252f0 0x56553fffcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.856+0000 7fbf4fe00640 1 -- 172.21.15.44:0/1922592177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 msgr2=0x56553fffa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.856+0000 7fbf4fe00640 1 --2- 172.21.15.44:0/1922592177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 0x56553fffa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.856+0000 7fbf4fe00640 1 -- 172.21.15.44:0/1922592177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc252f0 msgr2=0x56553fffcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.856+0000 7fbf4fe00640 1 --2- 172.21.15.44:0/1922592177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc252f0 0x56553fffcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.856+0000 7fbf4fe00640 1 -- 172.21.15.44:0/1922592177 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56554000cf30 con 0x56553fc2d4d0 2024-05-27T09:47:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.856+0000 7fbf57400640 1 --2- 172.21.15.44:0/1922592177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 0x56553fffa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.856+0000 7fbf4fe00640 1 --2- 172.21.15.44:0/1922592177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 0x56553ffec330 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fbf40014040 tx=0x7fbf40005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb65b0b854d6dff5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.856+0000 7fbf47400640 1 -- 172.21.15.44:0/1922592177 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf40004250 con 0x56553fc2d4d0 2024-05-27T09:47:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.857+0000 7fbf47400640 1 -- 172.21.15.44:0/1922592177 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf400043f0 con 0x56553fc2d4d0 2024-05-27T09:47:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.857+0000 7fbf47400640 1 -- 172.21.15.44:0/1922592177 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf400039a0 con 0x56553fc2d4d0 2024-05-27T09:47:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.857+0000 7fbf649f3740 1 -- 172.21.15.44:0/1922592177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 msgr2=0x56553ffec330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.857+0000 7fbf649f3740 1 --2- 172.21.15.44:0/1922592177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 0x56553ffec330 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fbf40014040 tx=0x7fbf40005f90 comp rx=0 tx=0).stop 2024-05-27T09:47:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.858+0000 7fbf649f3740 1 -- 172.21.15.44:0/1922592177 shutdown_connections 2024-05-27T09:47:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.858+0000 7fbf649f3740 1 --2- 172.21.15.44:0/1922592177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 0x56553fffa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.858+0000 7fbf649f3740 1 --2- 172.21.15.44:0/1922592177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc2d4d0 0x56553ffec330 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.858+0000 7fbf649f3740 1 --2- 172.21.15.44:0/1922592177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc252f0 0x56553fffcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.858+0000 7fbf649f3740 1 -- 172.21.15.44:0/1922592177 >> 172.21.15.44:0/1922592177 conn(0x56553fc42300 msgr2=0x56553fffa230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:55.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.858+0000 7fbf649f3740 1 -- 172.21.15.44:0/1922592177 shutdown_connections 2024-05-27T09:47:55.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.858+0000 7fbf649f3740 1 -- 172.21.15.44:0/1922592177 wait complete. 2024-05-27T09:47:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.859+0000 7fbf649f3740 1 -- start start 2024-05-27T09:47:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.859+0000 7fbf649f3740 1 Processor -- start 2024-05-27T09:47:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.859+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc252f0 0x56553fff4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.859+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc2d4d0 0x56553fff5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.859+0000 7fbf649f3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 0x565540001c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.859+0000 7fbf649f3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbf581b0440 con 0x56553fc2d4d0 2024-05-27T09:47:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.859+0000 7fbf649f3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbf581b18b0 con 0x56553fc252f0 2024-05-27T09:47:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.859+0000 7fbf649f3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbf581b0b10 con 0x56553fc4e1b0 2024-05-27T09:47:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.859+0000 7fbf649f3740 10 client.0 _ll_register_callbacks cb 0x7fbf4800cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:47:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc2d4d0 0x56553fff5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf57400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 0x565540001c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc2d4d0 0x56553fff5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40550/0 (socket says 172.21.15.44:40550) 2024-05-27T09:47:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf4fe00640 1 -- 172.21.15.44:0/3904347362 learned_addr learned my addr 172.21.15.44:0/3904347362 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf56a00640 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc252f0 0x56553fff4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf57400640 1 -- 172.21.15.44:0/3904347362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc252f0 msgr2=0x56553fff4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:55.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf57400640 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc252f0 0x56553fff4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf57400640 1 -- 172.21.15.44:0/3904347362 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc2d4d0 msgr2=0x56553fff5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf57400640 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc2d4d0 0x56553fff5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf57400640 1 -- 172.21.15.44:0/3904347362 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565540002160 con 0x56553fc4e1b0 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf56a00640 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc252f0 0x56553fff4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.860+0000 7fbf4fe00640 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc2d4d0 0x56553fff5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.861+0000 7fbf57400640 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 0x565540001c20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009a60 tx=0x7fbf5003c050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6933d697f007ba5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.861+0000 7fbf44c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.861+0000 7fbf44c00640 1 -- 172.21.15.44:0/3904347362 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf50034040 con 0x56553fc4e1b0 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.861+0000 7fbf649f3740 1 -- 172.21.15.44:0/3904347362 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf400043f0 con 0x56553fc4e1b0 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.861+0000 7fbf44c00640 1 -- 172.21.15.44:0/3904347362 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7fbf50004510 con 0x56553fc4e1b0 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.861+0000 7fbf44c00640 1 -- 172.21.15.44:0/3904347362 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fbf50061050 con 0x56553fc4e1b0 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.861+0000 7fbf649f3740 10 client.27640 fetch_fsmap learned FSMap version 42 2024-05-27T09:47:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.861+0000 7fbf649f3740 1 -- 172.21.15.44:0/3904347362 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x565540003400 con 0x56553fc4e1b0 2024-05-27T09:47:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.862+0000 7fbf44c00640 1 -- 172.21.15.44:0/3904347362 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf400043f0 con 0x56553fc4e1b0 2024-05-27T09:47:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.862+0000 7fbf44c00640 1 -- 172.21.15.44:0/3904347362 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7fbf5003a430 con 0x56553fc4e1b0 2024-05-27T09:47:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.862+0000 7fbf649f3740 10 client.27640 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:47:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.862+0000 7fbf649f3740 10 client.27640 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:47:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.862+0000 7fbf649f3740 20 client.27640 populate_metadata read hostname 'smithi044' 2024-05-27T09:47:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.862+0000 7fbf649f3740 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56553fbe21c0 0x565540077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.862+0000 7fbf649f3740 4 client.27640 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:47:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.862+0000 7fbf649f3740 1 -- 172.21.15.44:0/3904347362 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56553ff5d2c0 con 0x56553fbe21c0 2024-05-27T09:47:55.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.862+0000 7fbf56a00640 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56553fbe21c0 0x565540077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:55.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.863+0000 7fbf56a00640 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56553fbe21c0 0x565540077230 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17beddbb88a364b5 server_cookie=1ab53d36f6b733f2 in_seq=0 out_seq=0 2024-05-27T09:47:55.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.863+0000 7fbf44c00640 10 client.27640 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf44c00640 1 -- 172.21.15.44:0/3904347362 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56553ff5d2c0 con 0x56553fbe21c0 2024-05-27T09:47:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf44c00640 10 client.27640 handle_command_reply: tid=0 2024-05-27T09:47:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 1 client.27640 shutdown 2024-05-27T09:47:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 1 -- 172.21.15.44:0/3904347362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 msgr2=0x565540001c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 0x565540001c20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009a60 tx=0x7fbf5003c050 comp rx=0 tx=0).stop 2024-05-27T09:47:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 1 -- 172.21.15.44:0/3904347362 shutdown_connections 2024-05-27T09:47:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56553fbe21c0 0x565540077230 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553fc4e1b0 0x565540001c20 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56553fc2d4d0 0x56553fff5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 1 --2- 172.21.15.44:0/3904347362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56553fc252f0 0x56553fff4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf44c00640 0 client.27640 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:47:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 1 -- 172.21.15.44:0/3904347362 >> 172.21.15.44:0/3904347362 conn(0x56553fc42300 msgr2=0x56553fffac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 1 -- 172.21.15.44:0/3904347362 shutdown_connections 2024-05-27T09:47:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 1 -- 172.21.15.44:0/3904347362 wait complete. 2024-05-27T09:47:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.948+0000 7fbf649f3740 20 client.27640 trim_cache size 0 max 16384 2024-05-27T09:47:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:47:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:47:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:47:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 -- 172.21.15.44:0/2003452068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbf3c0b2950 msgr2=0x7fbf3c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbf3c0b2950 0x7fbf3c0b4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbf500094e0 tx=0x7fbf50005d80 comp rx=0 tx=0).stop 2024-05-27T09:47:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 -- 172.21.15.44:0/2003452068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf58076770 msgr2=0x7fbf581a26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf58076770 0x7fbf581a26f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800e9f0 tx=0x7fbf4800edf0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 -- 172.21.15.44:0/2003452068 shutdown_connections 2024-05-27T09:47:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58110470 0x7fbf5819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5810db10 0x7fbf581a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbf3c0b2950 0x7fbf3c0b4e60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 --2- 172.21.15.44:0/2003452068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf58076770 0x7fbf581a26f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 -- 172.21.15.44:0/2003452068 >> 172.21.15.44:0/2003452068 conn(0x7fbf58079290 msgr2=0x7fbf581023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 -- 172.21.15.44:0/2003452068 shutdown_connections 2024-05-27T09:47:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:47:55.949+0000 7fbf5da00640 1 -- 172.21.15.44:0/2003452068 wait complete. 2024-05-27T09:47:56.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:56 smithi044 ceph-mon[31618]: pgmap v398: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 502 KiB/s wr, 116 op/s 2024-05-27T09:47:56.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.228+0000 7f363ce00700 1 -- 172.21.15.44:0/2107395697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3638105dc0 msgr2=0x7f363810e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:56.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.228+0000 7f363ce00700 1 --2- 172.21.15.44:0/2107395697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3638105dc0 0x7f363810e310 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f36200098f0 tx=0x7f3620009c00 comp rx=0 tx=0).stop 2024-05-27T09:47:56.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.228+0000 7f363ce00700 1 -- 172.21.15.44:0/2107395697 shutdown_connections 2024-05-27T09:47:56.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.228+0000 7f363ce00700 1 --2- 172.21.15.44:0/2107395697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f363810e8e0 0x7f3638110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:56.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.228+0000 7f363ce00700 1 --2- 172.21.15.44:0/2107395697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3638105470 0x7f3638105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:56.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.228+0000 7f363ce00700 1 --2- 172.21.15.44:0/2107395697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3638105dc0 0x7f363810e310 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:56.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.228+0000 7f363ce00700 1 -- 172.21.15.44:0/2107395697 >> 172.21.15.44:0/2107395697 conn(0x7f36380fef20 msgr2=0x7f3638101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:56.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.228+0000 7f363ce00700 1 -- 172.21.15.44:0/2107395697 shutdown_connections 2024-05-27T09:47:56.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.228+0000 7f363ce00700 1 -- 172.21.15.44:0/2107395697 wait complete. 2024-05-27T09:47:56.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.229+0000 7f363ce00700 1 Processor -- start 2024-05-27T09:47:56.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.230+0000 7f363ce00700 1 -- start start 2024-05-27T09:47:56.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.230+0000 7f363ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3638105470 0x7f363819ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:56.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.230+0000 7f363ce00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3638105dc0 0x7f363819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:56.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.230+0000 7f363ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f363810e8e0 0x7f363819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:56.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.230+0000 7f363ce00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3638114490 con 0x7f363810e8e0 2024-05-27T09:47:56.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.230+0000 7f363ce00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3638114310 con 0x7f3638105dc0 2024-05-27T09:47:56.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.230+0000 7f363ce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3638114610 con 0x7f3638105470 2024-05-27T09:47:56.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3635e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3638105470 0x7f363819ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:56.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3636800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f363810e8e0 0x7f363819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:56.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3635400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3638105dc0 0x7f363819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:56.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3635e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3638105470 0x7f363819ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39256/0 (socket says 172.21.15.44:39256) 2024-05-27T09:47:56.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3635e00700 1 -- 172.21.15.44:0/2914770226 learned_addr learned my addr 172.21.15.44:0/2914770226 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:47:56.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3635400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3638105dc0 0x7f363819b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52936/0 (socket says 172.21.15.44:52936) 2024-05-27T09:47:56.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3635400700 1 -- 172.21.15.44:0/2914770226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3638105470 msgr2=0x7f363819ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:56.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3635400700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3638105470 0x7f363819ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:56.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3635400700 1 -- 172.21.15.44:0/2914770226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f363810e8e0 msgr2=0x7f363819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:56.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3635400700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f363810e8e0 0x7f363819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:56.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3635400700 1 -- 172.21.15.44:0/2914770226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363819fcf0 con 0x7f3638105dc0 2024-05-27T09:47:56.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.231+0000 7f3635400700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3638105dc0 0x7f363819b3c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f362000c010 tx=0x7f362000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:56.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.232+0000 7f362ea00700 1 -- 172.21.15.44:0/2914770226 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36200050c0 con 0x7f3638105dc0 2024-05-27T09:47:56.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.232+0000 7f363ce00700 1 -- 172.21.15.44:0/2914770226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f363819ff80 con 0x7f3638105dc0 2024-05-27T09:47:56.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.232+0000 7f362ea00700 1 -- 172.21.15.44:0/2914770226 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f362000b560 con 0x7f3638105dc0 2024-05-27T09:47:56.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.232+0000 7f362ea00700 1 -- 172.21.15.44:0/2914770226 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620003a00 con 0x7f3638105dc0 2024-05-27T09:47:56.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.232+0000 7f363ce00700 1 -- 172.21.15.44:0/2914770226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36381a7900 con 0x7f3638105dc0 2024-05-27T09:47:56.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.233+0000 7f363ce00700 1 -- 172.21.15.44:0/2914770226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f363806c910 con 0x7f3638105dc0 2024-05-27T09:47:56.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.235+0000 7f362ea00700 1 -- 172.21.15.44:0/2914770226 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f362000b710 con 0x7f3638105dc0 2024-05-27T09:47:56.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.237+0000 7f362ea00700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f361c0b29f0 0x7f361c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:47:56.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.237+0000 7f362ea00700 1 -- 172.21.15.44:0/2914770226 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f36200e3450 con 0x7f3638105dc0 2024-05-27T09:47:56.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.237+0000 7f3635e00700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f361c0b29f0 0x7f361c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:47:56.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.238+0000 7f3635e00700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f361c0b29f0 0x7f361c0b4f00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3628005950 tx=0x7f3628009310 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:47:56.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.242+0000 7f362ea00700 1 -- 172.21.15.44:0/2914770226 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3620013030 con 0x7f3638105dc0 2024-05-27T09:47:56.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:56 smithi189 ceph-mon[36671]: pgmap v398: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 502 KiB/s wr, 116 op/s 2024-05-27T09:47:56.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.528+0000 7f363ce00700 1 -- 172.21.15.44:0/2914770226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f363819c180 con 0x7f3638105dc0 2024-05-27T09:47:56.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.530+0000 7f362ea00700 1 -- 172.21.15.44:0/2914770226 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11868 (secure 0 0 0) 0x7f36200aae30 con 0x7f3638105dc0 2024-05-27T09:47:56.531 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:47:56.531 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:47:31.318532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24542,24503,24520]},"id":1}]} 2024-05-27T09:47:56.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.534+0000 7f363ce00700 1 -- 172.21.15.44:0/2914770226 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f361c0b29f0 msgr2=0x7f361c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:56.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.534+0000 7f363ce00700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f361c0b29f0 0x7f361c0b4f00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3628005950 tx=0x7f3628009310 comp rx=0 tx=0).stop 2024-05-27T09:47:56.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.534+0000 7f363ce00700 1 -- 172.21.15.44:0/2914770226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3638105dc0 msgr2=0x7f363819b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:47:56.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.534+0000 7f363ce00700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3638105dc0 0x7f363819b3c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f362000c010 tx=0x7f362000fa00 comp rx=0 tx=0).stop 2024-05-27T09:47:56.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.535+0000 7f363ce00700 1 -- 172.21.15.44:0/2914770226 shutdown_connections 2024-05-27T09:47:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.535+0000 7f363ce00700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f361c0b29f0 0x7f361c0b4f00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.535+0000 7f363ce00700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f363810e8e0 0x7f363819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.535+0000 7f363ce00700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3638105470 0x7f363819ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.535+0000 7f363ce00700 1 --2- 172.21.15.44:0/2914770226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3638105dc0 0x7f363819b3c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:47:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.535+0000 7f363ce00700 1 -- 172.21.15.44:0/2914770226 >> 172.21.15.44:0/2914770226 conn(0x7f36380fef20 msgr2=0x7f36380ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:47:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.535+0000 7f363ce00700 1 -- 172.21.15.44:0/2914770226 shutdown_connections 2024-05-27T09:47:56.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:47:56.535+0000 7f363ce00700 1 -- 172.21.15.44:0/2914770226 wait complete. 2024-05-27T09:47:56.537 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 42 2024-05-27T09:47:56.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:56 smithi050 ceph-mon[36371]: pgmap v398: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 502 KiB/s wr, 116 op/s 2024-05-27T09:47:56.671 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:47:56.671 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:47:57.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:57 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2914770226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:57.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:57 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2914770226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:57.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:57 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2914770226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:47:57.845 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db --set-id 8d7cdbbb --release --await' 2024-05-27T09:47:57.845 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db --set-id 8d7cdbbb --release --await 2024-05-27T09:47:57.895 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:47:58.303 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:47:58 smithi044 ceph-mon[31618]: pgmap v399: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 519 KiB/s wr, 121 op/s 2024-05-27T09:47:58.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:47:58 smithi050 ceph-mon[36371]: pgmap v399: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 519 KiB/s wr, 121 op/s 2024-05-27T09:47:58.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:47:58 smithi189 ceph-mon[36671]: pgmap v399: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 519 KiB/s wr, 121 op/s 2024-05-27T09:47:59.427 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:48:00.271 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:00 smithi044 ceph-mon[31618]: pgmap v400: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2024-05-27T09:48:00.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:00 smithi050 ceph-mon[36371]: pgmap v400: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2024-05-27T09:48:00.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:00 smithi189 ceph-mon[36671]: pgmap v400: 97 pgs: 97 active+clean; 310 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2024-05-27T09:48:00.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.432+0000 7f40a5e00700 1 -- 172.21.15.44:0/108871447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40a0108f10 msgr2=0x7f40a0109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.432+0000 7f40a5e00700 1 --2- 172.21.15.44:0/108871447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40a0108f10 0x7f40a0109320 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f408800b0d0 tx=0x7f408800b3e0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.432+0000 7f40a5e00700 1 -- 172.21.15.44:0/108871447 shutdown_connections 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.432+0000 7f40a5e00700 1 --2- 172.21.15.44:0/108871447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a0109860 0x7f40a01111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.432+0000 7f40a5e00700 1 --2- 172.21.15.44:0/108871447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40a01045e0 0x7f40a01049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.432+0000 7f40a5e00700 1 --2- 172.21.15.44:0/108871447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40a0108f10 0x7f40a0109320 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.432+0000 7f40a5e00700 1 -- 172.21.15.44:0/108871447 >> 172.21.15.44:0/108871447 conn(0x7f40a0100030 msgr2=0x7f40a0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.432+0000 7f40a5e00700 1 -- 172.21.15.44:0/108871447 shutdown_connections 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.432+0000 7f40a5e00700 1 -- 172.21.15.44:0/108871447 wait complete. 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.434+0000 7f40a5e00700 1 Processor -- start 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.434+0000 7f40a5e00700 1 -- start start 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.434+0000 7f40a5e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40a01045e0 0x7f40a019afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.435+0000 7f40a5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40a0109860 0x7f40a019b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.435+0000 7f40a5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a019f750 0x7f40a019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.435+0000 7f40a5e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f40a0114470 con 0x7f40a0109860 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.435+0000 7f40a5e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01142f0 con 0x7f40a01045e0 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.435+0000 7f40a5e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01145f0 con 0x7f40a019f750 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.435+0000 7f409ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40a01045e0 0x7f40a019afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.435+0000 7f4096000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40a0109860 0x7f40a019b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.435+0000 7f409ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40a01045e0 0x7f40a019afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:52944/0 (socket says 172.21.15.44:52944) 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.435+0000 7f409ea00700 1 -- 172.21.15.44:0/22007141 learned_addr learned my addr 172.21.15.44:0/22007141 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.435+0000 7f409f400700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a019f750 0x7f40a019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.435+0000 7f409ea00700 1 -- 172.21.15.44:0/22007141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a019f750 msgr2=0x7f40a019fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.436+0000 7f409ea00700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a019f750 0x7f40a019fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.436+0000 7f409ea00700 1 -- 172.21.15.44:0/22007141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40a0109860 msgr2=0x7f40a019b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.436+0000 7f409ea00700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40a0109860 0x7f40a019b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.436+0000 7f409ea00700 1 -- 172.21.15.44:0/22007141 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a01a02a0 con 0x7f40a01045e0 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.436+0000 7f409ea00700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40a01045e0 0x7f40a019afe0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f409000e9a0 tx=0x7f409000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.436+0000 7f40a4a00700 1 -- 172.21.15.44:0/22007141 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409000ca10 con 0x7f40a01045e0 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.436+0000 7f40a4a00700 1 -- 172.21.15.44:0/22007141 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f409000cbb0 con 0x7f40a01045e0 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.436+0000 7f40a5e00700 1 -- 172.21.15.44:0/22007141 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a01a0590 con 0x7f40a01045e0 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.437+0000 7f40a4a00700 1 -- 172.21.15.44:0/22007141 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4090018680 con 0x7f40a01045e0 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.437+0000 7f40a5e00700 1 -- 172.21.15.44:0/22007141 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a01a7d90 con 0x7f40a01045e0 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.441+0000 7f40a4a00700 1 -- 172.21.15.44:0/22007141 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4090010430 con 0x7f40a01045e0 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.441+0000 7f40b4993b80 1 Processor -- start 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.442+0000 7f40b4993b80 1 -- start start 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.442+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 0x562d2bd4ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.442+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 0x562d2bd44d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.442+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd45650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.442+0000 7f40b4993b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562d2bc6eec0 con 0x562d2bd4b160 2024-05-27T09:48:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.442+0000 7f40b4993b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562d2bc5ea60 con 0x562d2bd45240 2024-05-27T09:48:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.442+0000 7f40b4993b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562d2bc62aa0 con 0x562d2bbc1ec0 2024-05-27T09:48:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.442+0000 7f409ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 0x562d2bd44d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.442+0000 7f409f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 0x562d2bd4ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f409ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 0x562d2bd44d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40606/0 (socket says 172.21.15.44:40606) 2024-05-27T09:48:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f409ea00700 1 -- 172.21.15.44:0/3934164735 learned_addr learned my addr 172.21.15.44:0/3934164735 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f4096000700 1 --2- 172.21.15.44:0/3934164735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd45650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f40a4a00700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40840b2a70 0x7f40840b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f40a4a00700 1 -- 172.21.15.44:0/22007141 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f40900e27a0 con 0x7f40a01045e0 2024-05-27T09:48:00.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f409f400700 1 -- 172.21.15.44:0/3934164735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 msgr2=0x562d2bd45650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:00.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f409f400700 1 --2- 172.21.15.44:0/3934164735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd45650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f409f400700 1 -- 172.21.15.44:0/3934164735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 msgr2=0x562d2bd44d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:00.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f409f400700 1 --2- 172.21.15.44:0/3934164735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 0x562d2bd44d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f409f400700 1 -- 172.21.15.44:0/3934164735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d2bc274a0 con 0x562d2bbc1ec0 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f4096000700 1 --2- 172.21.15.44:0/3934164735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd45650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f409ea00700 1 --2- 172.21.15.44:0/3934164735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 0x562d2bd44d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.443+0000 7f409f400700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40840b2a70 0x7f40840b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.444+0000 7f409f400700 1 --2- 172.21.15.44:0/3934164735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 0x562d2bd4ac20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4098000b60 tx=0x7f409800b970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ade505e6764ea36 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.444+0000 7f4094c00700 1 -- 172.21.15.44:0/3934164735 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098024070 con 0x562d2bbc1ec0 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.444+0000 7f4094c00700 1 -- 172.21.15.44:0/3934164735 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f409801e480 con 0x562d2bbc1ec0 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.444+0000 7f4094c00700 1 -- 172.21.15.44:0/3934164735 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098011680 con 0x562d2bbc1ec0 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.444+0000 7f409f400700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40840b2a70 0x7f40840b4f80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4098021000 tx=0x7f409801e000 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.445+0000 7f40b4993b80 1 -- 172.21.15.44:0/3934164735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 msgr2=0x562d2bd4ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.445+0000 7f40b4993b80 1 --2- 172.21.15.44:0/3934164735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 0x562d2bd4ac20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4098000b60 tx=0x7f409800b970 comp rx=0 tx=0).stop 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.445+0000 7f40b4993b80 1 -- 172.21.15.44:0/3934164735 shutdown_connections 2024-05-27T09:48:00.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.445+0000 7f40b4993b80 1 --2- 172.21.15.44:0/3934164735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd45650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.445+0000 7f40b4993b80 1 --2- 172.21.15.44:0/3934164735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 0x562d2bd4ac20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.445+0000 7f40b4993b80 1 --2- 172.21.15.44:0/3934164735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 0x562d2bd44d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.445+0000 7f40b4993b80 1 -- 172.21.15.44:0/3934164735 >> 172.21.15.44:0/3934164735 conn(0x562d2bc22040 msgr2=0x562d2bd3b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.445+0000 7f40b4993b80 1 -- 172.21.15.44:0/3934164735 shutdown_connections 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.446+0000 7f40b4993b80 1 -- 172.21.15.44:0/3934164735 wait complete. 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.446+0000 7f40b4993b80 1 -- start start 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.446+0000 7f40b4993b80 1 Processor -- start 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 0x562d2bd4e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd4ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 0x562d2bd4f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f40b4993b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562d2bb65020 con 0x562d2bd4b160 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f40b4993b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562d2bbff7e0 con 0x562d2bd45240 2024-05-27T09:48:00.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f40b4993b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562d2bd51c40 con 0x562d2bbc1ec0 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f40b4993b80 10 client.0 _ll_register_callbacks cb 0x562d2bc08cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f409ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 0x562d2bd4e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f409ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 0x562d2bd4e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39300/0 (socket says 172.21.15.44:39300) 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f409ea00700 1 -- 172.21.15.44:0/1859834681 learned_addr learned my addr 172.21.15.44:0/1859834681 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f409f400700 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 0x562d2bd4f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.447+0000 7f4096000700 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd4ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.448+0000 7f409ea00700 1 -- 172.21.15.44:0/1859834681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 msgr2=0x562d2bd4ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.448+0000 7f409ea00700 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd4ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.448+0000 7f409ea00700 1 -- 172.21.15.44:0/1859834681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 msgr2=0x562d2bd4f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.448+0000 7f409ea00700 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 0x562d2bd4f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.448+0000 7f409ea00700 1 -- 172.21.15.44:0/1859834681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d2bbae950 con 0x562d2bbc1ec0 2024-05-27T09:48:00.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.448+0000 7f409ea00700 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 0x562d2bd4e940 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f409000c2d0 tx=0x7f40900e03c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d2e5a7067b1cd9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.448+0000 7f408d600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.448+0000 7f408d600700 1 -- 172.21.15.44:0/1859834681 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40900a97c0 con 0x562d2bbc1ec0 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.448+0000 7f408d600700 1 -- 172.21.15.44:0/1859834681 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f40900b25c0 con 0x562d2bbc1ec0 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.448+0000 7f40b4993b80 1 -- 172.21.15.44:0/1859834681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f409801e480 con 0x562d2bbc1ec0 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.449+0000 7f408d600700 1 -- 172.21.15.44:0/1859834681 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f40900a9500 con 0x562d2bbc1ec0 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.449+0000 7f40b4993b80 10 client.27658 fetch_fsmap learned FSMap version 42 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.449+0000 7f40b4993b80 1 -- 172.21.15.44:0/1859834681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562d2bc2f0e0 con 0x562d2bbc1ec0 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.449+0000 7f408d600700 1 -- 172.21.15.44:0/1859834681 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f409801e480 con 0x562d2bbc1ec0 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.449+0000 7f408d600700 1 -- 172.21.15.44:0/1859834681 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f40900e2b90 con 0x562d2bbc1ec0 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.449+0000 7f40b4993b80 10 client.27658 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.449+0000 7f40b4993b80 10 client.27658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.449+0000 7f40b4993b80 20 client.27658 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.449+0000 7f40b4993b80 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d2bd5ead0 0x562d2bd60ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.449+0000 7f40b4993b80 4 client.27658 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.449+0000 7f40b4993b80 1 -- 172.21.15.44:0/1859834681 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d2bc70e10 con 0x562d2bd5ead0 2024-05-27T09:48:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.450+0000 7f4096000700 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d2bd5ead0 0x562d2bd60ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.450+0000 7f4096000700 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d2bd5ead0 0x562d2bd60ef0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75a5fc33f20b43f4 server_cookie=8f9cef1ed1bc5ad1 in_seq=0 out_seq=0 2024-05-27T09:48:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.450+0000 7f408d600700 10 client.27658 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.451+0000 7f408d600700 1 -- 172.21.15.44:0/1859834681 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562d2bc70e10 con 0x562d2bd5ead0 2024-05-27T09:48:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.451+0000 7f408d600700 10 client.27658 handle_command_reply: tid=0 2024-05-27T09:48:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.451+0000 7f40b4993b80 1 client.27658 shutdown 2024-05-27T09:48:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.451+0000 7f40b4993b80 1 -- 172.21.15.44:0/1859834681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 msgr2=0x562d2bd4e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.451+0000 7f40b4993b80 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 0x562d2bd4e940 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f409000c2d0 tx=0x7f40900e03c0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.452+0000 7f40b4993b80 1 -- 172.21.15.44:0/1859834681 shutdown_connections 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.452+0000 7f40b4993b80 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d2bd5ead0 0x562d2bd60ef0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.452+0000 7f40b4993b80 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bd4b160 0x562d2bd4f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.452+0000 7f40b4993b80 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bbc1ec0 0x562d2bd4e940 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.452+0000 7f408d600700 0 client.27658 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.452+0000 7f40b4993b80 1 --2- 172.21.15.44:0/1859834681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd4ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.452+0000 7f40b4993b80 1 -- 172.21.15.44:0/1859834681 >> 172.21.15.44:0/1859834681 conn(0x562d2bc22040 msgr2=0x562d2bd3b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.452+0000 7f40b4993b80 1 -- 172.21.15.44:0/1859834681 shutdown_connections 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.452+0000 7f40b4993b80 1 -- 172.21.15.44:0/1859834681 wait complete. 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.452+0000 7f40b4993b80 20 client.27658 trim_cache size 0 max 16384 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.475+0000 7f40b4993b80 1 Processor -- start 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.475+0000 7f40b4993b80 1 -- start start 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.476+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd5ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.476+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd48cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.476+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bd4b160 0x562d2bd44910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.476+0000 7f40b4993b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01afca0 con 0x562d2bbc1ec0 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.476+0000 7f40b4993b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01ad500 con 0x562d2bd45240 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.476+0000 7f40b4993b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01ab350 con 0x562d2bd4b160 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.476+0000 7f409f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bd4b160 0x562d2bd44910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.476+0000 7f409ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd5ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.476+0000 7f4096000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd48cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.476+0000 7f409ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd5ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40634/0 (socket says 172.21.15.44:40634) 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.476+0000 7f409ea00700 1 -- 172.21.15.44:0/666455118 learned_addr learned my addr 172.21.15.44:0/666455118 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.477+0000 7f4096000700 1 -- 172.21.15.44:0/666455118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bd4b160 msgr2=0x562d2bd44910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.477+0000 7f4096000700 1 --2- 172.21.15.44:0/666455118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bd4b160 0x562d2bd44910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.477+0000 7f4096000700 1 -- 172.21.15.44:0/666455118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 msgr2=0x562d2bd5ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.477+0000 7f4096000700 1 --2- 172.21.15.44:0/666455118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd5ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.477+0000 7f4096000700 1 -- 172.21.15.44:0/666455118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d2bd5c5a0 con 0x562d2bd45240 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.477+0000 7f409ea00700 1 --2- 172.21.15.44:0/666455118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd5ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.477+0000 7f4096000700 1 --2- 172.21.15.44:0/666455118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd48cf0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f40a0105410 tx=0x7f408800bd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb7d0f98b1249abc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.477+0000 7f408fe00700 1 -- 172.21.15.44:0/666455118 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408800be10 con 0x562d2bd45240 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.477+0000 7f408fe00700 1 -- 172.21.15.44:0/666455118 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40880094e0 con 0x562d2bd45240 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.477+0000 7f408fe00700 1 -- 172.21.15.44:0/666455118 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4088007d20 con 0x562d2bd45240 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.478+0000 7f40b4993b80 1 -- 172.21.15.44:0/666455118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 msgr2=0x562d2bd48cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.478+0000 7f40b4993b80 1 --2- 172.21.15.44:0/666455118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd48cf0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f40a0105410 tx=0x7f408800bd20 comp rx=0 tx=0).stop 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.478+0000 7f40b4993b80 1 -- 172.21.15.44:0/666455118 shutdown_connections 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.478+0000 7f40b4993b80 1 --2- 172.21.15.44:0/666455118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bd4b160 0x562d2bd44910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.478+0000 7f40b4993b80 1 --2- 172.21.15.44:0/666455118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd5ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.478+0000 7f40b4993b80 1 --2- 172.21.15.44:0/666455118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd48cf0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.478+0000 7f40b4993b80 1 -- 172.21.15.44:0/666455118 >> 172.21.15.44:0/666455118 conn(0x562d2bc22040 msgr2=0x562d2bd485e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.478+0000 7f40b4993b80 1 -- 172.21.15.44:0/666455118 shutdown_connections 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.478+0000 7f40b4993b80 1 -- 172.21.15.44:0/666455118 wait complete. 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.479+0000 7f40b4993b80 1 -- start start 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.479+0000 7f40b4993b80 1 Processor -- start 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.479+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd56ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.479+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd57410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.479+0000 7f40b4993b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bd4b160 0x562d2bd42b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.479+0000 7f40b4993b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01ab9c0 con 0x562d2bbc1ec0 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.479+0000 7f40b4993b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01ace30 con 0x562d2bd45240 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.479+0000 7f40b4993b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01ac090 con 0x562d2bd4b160 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f40b4993b80 10 client.0 _ll_register_callbacks cb 0x562d2bc08cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f409ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd56ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f409ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd56ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40644/0 (socket says 172.21.15.44:40644) 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f409ea00700 1 -- 172.21.15.44:0/830844848 learned_addr learned my addr 172.21.15.44:0/830844848 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f4096000700 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd57410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f409f400700 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bd4b160 0x562d2bd42b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f409ea00700 1 -- 172.21.15.44:0/830844848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bd4b160 msgr2=0x562d2bd42b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f409ea00700 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bd4b160 0x562d2bd42b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f409ea00700 1 -- 172.21.15.44:0/830844848 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 msgr2=0x562d2bd57410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f409ea00700 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd57410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f409ea00700 1 -- 172.21.15.44:0/830844848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d2bd4f730 con 0x562d2bbc1ec0 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f409f400700 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bd4b160 0x562d2bd42b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.480+0000 7f4096000700 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd57410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.481+0000 7f409ea00700 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd56ed0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f409000c2d0 tx=0x7f40900aa0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf35164eb000cd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.481+0000 7f408d600700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.481+0000 7f408d600700 1 -- 172.21.15.44:0/830844848 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40900b2850 con 0x562d2bbc1ec0 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.481+0000 7f408d600700 1 -- 172.21.15.44:0/830844848 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f40900e2a20 con 0x562d2bbc1ec0 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.481+0000 7f40b4993b80 1 -- 172.21.15.44:0/830844848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562d2bc6a5f0 con 0x562d2bbc1ec0 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.481+0000 7f408d600700 1 -- 172.21.15.44:0/830844848 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f40900a99f0 con 0x562d2bbc1ec0 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.481+0000 7f40b4993b80 10 client.17421 fetch_fsmap learned FSMap version 42 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.481+0000 7f40b4993b80 1 -- 172.21.15.44:0/830844848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562d2bdce480 con 0x562d2bbc1ec0 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.481+0000 7f408d600700 1 -- 172.21.15.44:0/830844848 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562d2bc6a5f0 con 0x562d2bbc1ec0 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.482+0000 7f408d600700 1 -- 172.21.15.44:0/830844848 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f40900a86c0 con 0x562d2bbc1ec0 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.482+0000 7f40b4993b80 10 client.17421 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.482+0000 7f40b4993b80 10 client.17421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.482+0000 7f40b4993b80 20 client.17421 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.482+0000 7f40b4993b80 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d2bdceb80 0x562d2bdd0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.482+0000 7f40b4993b80 4 client.17421 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:48:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.482+0000 7f40b4993b80 1 -- 172.21.15.44:0/830844848 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562d2bae12d0 con 0x562d2bdceb80 2024-05-27T09:48:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.482+0000 7f4096000700 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d2bdceb80 0x562d2bdd0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.483+0000 7f4096000700 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d2bdceb80 0x562d2bdd0fa0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebbb4a5dfb355d1c server_cookie=badf29dfb6f287a0 in_seq=0 out_seq=0 2024-05-27T09:48:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.483+0000 7f408d600700 10 client.17421 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f408d600700 1 -- 172.21.15.44:0/830844848 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562d2bae12d0 con 0x562d2bdceb80 2024-05-27T09:48:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f408d600700 10 client.17421 handle_command_reply: tid=0 2024-05-27T09:48:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f40b4993b80 1 client.17421 shutdown 2024-05-27T09:48:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f40b4993b80 1 -- 172.21.15.44:0/830844848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 msgr2=0x562d2bd56ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f40b4993b80 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd56ed0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f409000c2d0 tx=0x7f40900aa0f0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f40b4993b80 1 -- 172.21.15.44:0/830844848 shutdown_connections 2024-05-27T09:48:00.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f40b4993b80 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x562d2bdceb80 0x562d2bdd0fa0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f40b4993b80 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d2bd4b160 0x562d2bd42b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f40b4993b80 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562d2bbc1ec0 0x562d2bd56ed0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f40b4993b80 1 --2- 172.21.15.44:0/830844848 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562d2bd45240 0x562d2bd57410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f40b4993b80 1 -- 172.21.15.44:0/830844848 >> 172.21.15.44:0/830844848 conn(0x562d2bc22040 msgr2=0x562d2bd4e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:00.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.484+0000 7f408d600700 0 client.17421 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:00.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.485+0000 7f40b4993b80 1 -- 172.21.15.44:0/830844848 shutdown_connections 2024-05-27T09:48:00.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.485+0000 7f40b4993b80 1 -- 172.21.15.44:0/830844848 wait complete. 2024-05-27T09:48:00.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.485+0000 7f40b4993b80 20 client.17421 trim_cache size 0 max 16384 2024-05-27T09:48:00.634 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:48:00.635 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:48:00.635 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "c9cadb11-5c59-4338-b772-13e508320e23", 2024-05-27T09:48:00.635 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:48:00.635 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:48:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 -- 172.21.15.44:0/22007141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40840b2a70 msgr2=0x7f40840b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40840b2a70 0x7f40840b4f80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4098021000 tx=0x7f409801e000 comp rx=0 tx=0).stop 2024-05-27T09:48:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 -- 172.21.15.44:0/22007141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40a01045e0 msgr2=0x7f40a019afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40a01045e0 0x7f40a019afe0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f409000e9a0 tx=0x7f409000ecb0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 -- 172.21.15.44:0/22007141 shutdown_connections 2024-05-27T09:48:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f40840b2a70 0x7f40840b4f80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a019f750 0x7f40a019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40a01045e0 0x7f40a019afe0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 --2- 172.21.15.44:0/22007141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40a0109860 0x7f40a019b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 -- 172.21.15.44:0/22007141 >> 172.21.15.44:0/22007141 conn(0x7f40a0100030 msgr2=0x7f40a0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 -- 172.21.15.44:0/22007141 shutdown_connections 2024-05-27T09:48:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:00.486+0000 7f40a5e00700 1 -- 172.21.15.44:0/22007141 wait complete. 2024-05-27T09:48:01.431 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:01 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:48:01.431 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:01 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:48:01.431 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:01 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:48:01.431 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:01 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:48:01.493 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:48:01.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:01 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:48:01.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:01 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:48:01.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:01 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:48:01.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:01 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:48:01.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:01 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:48:01.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:01 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:48:01.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:01 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:48:01.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:01 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:48:02.082 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:48:02.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:02 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:48:02.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:02 smithi044 ceph-mon[31618]: pgmap v401: 97 pgs: 97 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 143 KiB/s wr, 56 op/s 2024-05-27T09:48:02.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:02 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:48:02.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:02 smithi050 ceph-mon[36371]: pgmap v401: 97 pgs: 97 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 143 KiB/s wr, 56 op/s 2024-05-27T09:48:02.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:02 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:48:02.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:02 smithi189 ceph-mon[36671]: pgmap v401: 97 pgs: 97 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 143 KiB/s wr, 56 op/s 2024-05-27T09:48:03.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.254+0000 7f2356600700 1 -- 172.21.15.44:0/2971566148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2350107df0 msgr2=0x7f2350108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:03.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.255+0000 7f2356600700 1 --2- 172.21.15.44:0/2971566148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2350107df0 0x7f2350108200 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f233c009920 tx=0x7f233c009c30 comp rx=0 tx=0).stop 2024-05-27T09:48:03.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.255+0000 7f2356600700 1 -- 172.21.15.44:0/2971566148 shutdown_connections 2024-05-27T09:48:03.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.255+0000 7f2356600700 1 --2- 172.21.15.44:0/2971566148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2350108740 0x7f23501100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.255+0000 7f2356600700 1 --2- 172.21.15.44:0/2971566148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23501034c0 0x7f23501038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.255+0000 7f2356600700 1 --2- 172.21.15.44:0/2971566148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2350107df0 0x7f2350108200 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.255+0000 7f2356600700 1 -- 172.21.15.44:0/2971566148 >> 172.21.15.44:0/2971566148 conn(0x7f23500fef10 msgr2=0x7f2350101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:03.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.255+0000 7f2356600700 1 -- 172.21.15.44:0/2971566148 shutdown_connections 2024-05-27T09:48:03.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.255+0000 7f2356600700 1 -- 172.21.15.44:0/2971566148 wait complete. 2024-05-27T09:48:03.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.256+0000 7f2356600700 1 Processor -- start 2024-05-27T09:48:03.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.256+0000 7f2356600700 1 -- start start 2024-05-27T09:48:03.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.257+0000 7f2356600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23501034c0 0x7f23500ff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.257+0000 7f2356600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2350107df0 0x7f23500ff930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.257+0000 7f2356600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2350108740 0x7f23500ffe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.257+0000 7f2356600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f23501133a0 con 0x7f2350107df0 2024-05-27T09:48:03.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.257+0000 7f2356600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2350113220 con 0x7f23501034c0 2024-05-27T09:48:03.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.257+0000 7f2356600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2350113520 con 0x7f2350108740 2024-05-27T09:48:03.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.257+0000 7f234ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2350107df0 0x7f23500ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.257+0000 7f234f400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23501034c0 0x7f23500ff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.257+0000 7f234f400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23501034c0 0x7f23500ff3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39974/0 (socket says 172.21.15.44:39974) 2024-05-27T09:48:03.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.257+0000 7f234f400700 1 -- 172.21.15.44:0/894594068 learned_addr learned my addr 172.21.15.44:0/894594068 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:03.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.257+0000 7f234fe00700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2350108740 0x7f23500ffe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.258+0000 7f234f400700 1 -- 172.21.15.44:0/894594068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2350108740 msgr2=0x7f23500ffe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:03.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.258+0000 7f234f400700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2350108740 0x7f23500ffe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.258+0000 7f234f400700 1 -- 172.21.15.44:0/894594068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2350107df0 msgr2=0x7f23500ff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:03.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.258+0000 7f234f400700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2350107df0 0x7f23500ff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.258+0000 7f234f400700 1 -- 172.21.15.44:0/894594068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23501a92a0 con 0x7f23501034c0 2024-05-27T09:48:03.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.258+0000 7f234f400700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23501034c0 0x7f23500ff3f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f234000e9c0 tx=0x7f234000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:03.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.258+0000 7f2347e00700 1 -- 172.21.15.44:0/894594068 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2340004a30 con 0x7f23501034c0 2024-05-27T09:48:03.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.259+0000 7f2347e00700 1 -- 172.21.15.44:0/894594068 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f234000cc70 con 0x7f23501034c0 2024-05-27T09:48:03.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.259+0000 7f2356600700 1 -- 172.21.15.44:0/894594068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23501a9590 con 0x7f23501034c0 2024-05-27T09:48:03.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.259+0000 7f2356600700 1 -- 172.21.15.44:0/894594068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23501a9aa0 con 0x7f23501034c0 2024-05-27T09:48:03.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.259+0000 7f2347e00700 1 -- 172.21.15.44:0/894594068 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2340005490 con 0x7f23501034c0 2024-05-27T09:48:03.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.262+0000 7f2347e00700 1 -- 172.21.15.44:0/894594068 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2340021440 con 0x7f23501034c0 2024-05-27T09:48:03.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.264+0000 7f236517fb80 1 Processor -- start 2024-05-27T09:48:03.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.264+0000 7f236517fb80 1 -- start start 2024-05-27T09:48:03.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.264+0000 7f2347e00700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23340b29f0 0x7f23340b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.264+0000 7f2347e00700 1 -- 172.21.15.44:0/894594068 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f23400e26e0 con 0x7f23501034c0 2024-05-27T09:48:03.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.264+0000 7f236517fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf62db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f236517fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf632f0 0x55895bf5cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f236517fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf5d440 0x55895bf5d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f236517fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55895be8af40 con 0x55895bf5d440 2024-05-27T09:48:03.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f236517fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55895be7aae0 con 0x55895bf632f0 2024-05-27T09:48:03.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f236517fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55895be7eb20 con 0x55895bdddf40 2024-05-27T09:48:03.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f234fe00700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23340b29f0 0x7f23340b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f234ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf632f0 0x55895bf5cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f234f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf62db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f234ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf632f0 0x55895bf5cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39986/0 (socket says 172.21.15.44:39986) 2024-05-27T09:48:03.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f234ea00700 1 -- 172.21.15.44:0/1254248289 learned_addr learned my addr 172.21.15.44:0/1254248289 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:03.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f234f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf62db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38420/0 (socket says 172.21.15.44:38420) 2024-05-27T09:48:03.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f234f400700 1 -- 172.21.15.44:0/1254248289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf632f0 msgr2=0x55895bf5cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:03.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.265+0000 7f234f400700 1 --2- 172.21.15.44:0/1254248289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf632f0 0x55895bf5cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.266+0000 7f234f400700 1 -- 172.21.15.44:0/1254248289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf5d440 msgr2=0x55895bf5d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:03.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.266+0000 7f234f400700 1 --2- 172.21.15.44:0/1254248289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf5d440 0x55895bf5d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.266+0000 7f234f400700 1 -- 172.21.15.44:0/1254248289 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895be43520 con 0x55895bdddf40 2024-05-27T09:48:03.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.266+0000 7f234f400700 1 --2- 172.21.15.44:0/1254248289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf62db0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f23400dfda0 tx=0x7f23400a9170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e0e59d928240bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:03.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.266+0000 7f2344c00700 1 -- 172.21.15.44:0/1254248289 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23400a9440 con 0x55895bdddf40 2024-05-27T09:48:03.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.266+0000 7f234fe00700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23340b29f0 0x7f23340b4f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f234800f5e0 tx=0x7f2348009920 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:03.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.266+0000 7f2344c00700 1 -- 172.21.15.44:0/1254248289 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f234000ce20 con 0x55895bdddf40 2024-05-27T09:48:03.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.266+0000 7f2344c00700 1 -- 172.21.15.44:0/1254248289 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23400b2790 con 0x55895bdddf40 2024-05-27T09:48:03.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.267+0000 7f236517fb80 1 -- 172.21.15.44:0/1254248289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 msgr2=0x55895bf62db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:03.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.267+0000 7f236517fb80 1 --2- 172.21.15.44:0/1254248289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf62db0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f23400dfda0 tx=0x7f23400a9170 comp rx=0 tx=0).stop 2024-05-27T09:48:03.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.267+0000 7f236517fb80 1 -- 172.21.15.44:0/1254248289 shutdown_connections 2024-05-27T09:48:03.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.267+0000 7f236517fb80 1 --2- 172.21.15.44:0/1254248289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf5d440 0x55895bf5d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.267+0000 7f236517fb80 1 --2- 172.21.15.44:0/1254248289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf62db0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.267+0000 7f236517fb80 1 --2- 172.21.15.44:0/1254248289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf632f0 0x55895bf5cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.267+0000 7f236517fb80 1 -- 172.21.15.44:0/1254248289 >> 172.21.15.44:0/1254248289 conn(0x55895be3e0c0 msgr2=0x55895bf58100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:03.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.268+0000 7f236517fb80 1 -- 172.21.15.44:0/1254248289 shutdown_connections 2024-05-27T09:48:03.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.268+0000 7f236517fb80 1 -- 172.21.15.44:0/1254248289 wait complete. 2024-05-27T09:48:03.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.268+0000 7f236517fb80 1 -- start start 2024-05-27T09:48:03.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.269+0000 7f236517fb80 1 Processor -- start 2024-05-27T09:48:03.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.269+0000 7f236517fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bdddf40 0x55895bf66a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.269+0000 7f236517fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf5d440 0x55895bf66f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.269+0000 7f236517fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf632f0 0x55895bf674c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.269+0000 7f236517fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55895bd810a0 con 0x55895bf632f0 2024-05-27T09:48:03.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.269+0000 7f236517fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55895be1b860 con 0x55895bdddf40 2024-05-27T09:48:03.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.269+0000 7f236517fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55895bf69dc0 con 0x55895bf5d440 2024-05-27T09:48:03.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.269+0000 7f236517fb80 10 client.0 _ll_register_callbacks cb 0x55895be24d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:03.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.270+0000 7f234f400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bdddf40 0x55895bf66a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.270+0000 7f234ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf5d440 0x55895bf66f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.270+0000 7f234ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf5d440 0x55895bf66f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38428/0 (socket says 172.21.15.44:38428) 2024-05-27T09:48:03.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.270+0000 7f234ea00700 1 -- 172.21.15.44:0/2171543006 learned_addr learned my addr 172.21.15.44:0/2171543006 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:03.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.270+0000 7f234fe00700 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf632f0 0x55895bf674c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.270+0000 7f234f400700 1 -- 172.21.15.44:0/2171543006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf5d440 msgr2=0x55895bf66f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:03.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.270+0000 7f234f400700 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf5d440 0x55895bf66f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.270+0000 7f234f400700 1 -- 172.21.15.44:0/2171543006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf632f0 msgr2=0x55895bf674c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:03.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.270+0000 7f234f400700 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf632f0 0x55895bf674c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.270+0000 7f234f400700 1 -- 172.21.15.44:0/2171543006 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895bdca9d0 con 0x55895bdddf40 2024-05-27T09:48:03.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.271+0000 7f234ea00700 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf5d440 0x55895bf66f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:03.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.271+0000 7f234fe00700 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf632f0 0x55895bf674c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:03.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.271+0000 7f234f400700 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bdddf40 0x55895bf66a40 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f23400dfdd0 tx=0x7f23400cae10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce717ff0c14a7144 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:03.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.271+0000 7f2339600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:03.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.271+0000 7f2339600700 1 -- 172.21.15.44:0/2171543006 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23400aa030 con 0x55895bdddf40 2024-05-27T09:48:03.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.271+0000 7f236517fb80 1 -- 172.21.15.44:0/2171543006 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f234000ce20 con 0x55895bdddf40 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.271+0000 7f2339600700 1 -- 172.21.15.44:0/2171543006 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f23400a9b90 con 0x55895bdddf40 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.271+0000 7f2339600700 1 -- 172.21.15.44:0/2171543006 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f23400b2790 con 0x55895bdddf40 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.272+0000 7f236517fb80 10 client.27674 fetch_fsmap learned FSMap version 42 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.272+0000 7f236517fb80 1 -- 172.21.15.44:0/2171543006 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55895be4b160 con 0x55895bdddf40 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.272+0000 7f2339600700 1 -- 172.21.15.44:0/2171543006 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f234000ce20 con 0x55895bdddf40 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.272+0000 7f2339600700 1 -- 172.21.15.44:0/2171543006 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f23400c7070 con 0x55895bdddf40 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.272+0000 7f236517fb80 10 client.27674 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.272+0000 7f236517fb80 10 client.27674 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.272+0000 7f236517fb80 20 client.27674 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.272+0000 7f236517fb80 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55895bf76a70 0x55895bf78e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.272+0000 7f236517fb80 4 client.27674 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:03.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.272+0000 7f236517fb80 1 -- 172.21.15.44:0/2171543006 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55895be8ce90 con 0x55895bf76a70 2024-05-27T09:48:03.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.272+0000 7f234ea00700 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55895bf76a70 0x55895bf78e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.273+0000 7f234ea00700 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55895bf76a70 0x55895bf78e90 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90559f97896d07e3 server_cookie=567f39141afd1b1a in_seq=0 out_seq=0 2024-05-27T09:48:03.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.273+0000 7f2339600700 10 client.27674 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:03.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f2339600700 1 -- 172.21.15.44:0/2171543006 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55895be8ce90 con 0x55895bf76a70 2024-05-27T09:48:03.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f2339600700 10 client.27674 handle_command_reply: tid=0 2024-05-27T09:48:03.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f236517fb80 1 client.27674 shutdown 2024-05-27T09:48:03.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f236517fb80 1 -- 172.21.15.44:0/2171543006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bdddf40 msgr2=0x55895bf66a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:03.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f236517fb80 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bdddf40 0x55895bf66a40 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f23400dfdd0 tx=0x7f23400cae10 comp rx=0 tx=0).stop 2024-05-27T09:48:03.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f236517fb80 1 -- 172.21.15.44:0/2171543006 shutdown_connections 2024-05-27T09:48:03.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f236517fb80 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55895bf76a70 0x55895bf78e90 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f236517fb80 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf632f0 0x55895bf674c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f236517fb80 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bdddf40 0x55895bf66a40 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f236517fb80 1 --2- 172.21.15.44:0/2171543006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf5d440 0x55895bf66f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f236517fb80 1 -- 172.21.15.44:0/2171543006 >> 172.21.15.44:0/2171543006 conn(0x55895be3e0c0 msgr2=0x55895bf5bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:03.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.274+0000 7f2339600700 0 client.27674 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:03.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.275+0000 7f236517fb80 1 -- 172.21.15.44:0/2171543006 shutdown_connections 2024-05-27T09:48:03.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.275+0000 7f236517fb80 1 -- 172.21.15.44:0/2171543006 wait complete. 2024-05-27T09:48:03.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.275+0000 7f236517fb80 20 client.27674 trim_cache size 0 max 16384 2024-05-27T09:48:03.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.299+0000 7f236517fb80 1 Processor -- start 2024-05-27T09:48:03.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.299+0000 7f236517fb80 1 -- start start 2024-05-27T09:48:03.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.299+0000 7f236517fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf73060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.299+0000 7f236517fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf5d440 0x55895bf5c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.299+0000 7f236517fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf632f0 0x55895bf7beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f236517fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f233000ffa0 con 0x55895bf5d440 2024-05-27T09:48:03.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f236517fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f233000d800 con 0x55895bf632f0 2024-05-27T09:48:03.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f236517fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f233000b650 con 0x55895bdddf40 2024-05-27T09:48:03.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f234ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf5d440 0x55895bf5c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f234f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf73060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f234ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf5d440 0x55895bf5c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48564/0 (socket says 172.21.15.44:48564) 2024-05-27T09:48:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f234f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf73060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38440/0 (socket says 172.21.15.44:38440) 2024-05-27T09:48:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f234ea00700 1 -- 172.21.15.44:0/2952620673 learned_addr learned my addr 172.21.15.44:0/2952620673 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f234fe00700 1 --2- 172.21.15.44:0/2952620673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf632f0 0x55895bf7beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f234f400700 1 -- 172.21.15.44:0/2952620673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf632f0 msgr2=0x55895bf7beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f234f400700 1 --2- 172.21.15.44:0/2952620673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf632f0 0x55895bf7beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f234f400700 1 -- 172.21.15.44:0/2952620673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf5d440 msgr2=0x55895bf5c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f234f400700 1 --2- 172.21.15.44:0/2952620673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf5d440 0x55895bf5c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.300+0000 7f234f400700 1 -- 172.21.15.44:0/2952620673 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895bf58c40 con 0x55895bdddf40 2024-05-27T09:48:03.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.301+0000 7f234f400700 1 --2- 172.21.15.44:0/2952620673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf73060 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f23400a6ba0 tx=0x7f23400b21a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdddc034fb7811fa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:03.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.301+0000 7f233be00700 1 -- 172.21.15.44:0/2952620673 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23400702f0 con 0x55895bdddf40 2024-05-27T09:48:03.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.301+0000 7f233be00700 1 -- 172.21.15.44:0/2952620673 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23400aa030 con 0x55895bdddf40 2024-05-27T09:48:03.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.301+0000 7f233be00700 1 -- 172.21.15.44:0/2952620673 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23400a57f0 con 0x55895bdddf40 2024-05-27T09:48:03.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.302+0000 7f236517fb80 1 -- 172.21.15.44:0/2952620673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 msgr2=0x55895bf73060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:03.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.302+0000 7f236517fb80 1 --2- 172.21.15.44:0/2952620673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf73060 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f23400a6ba0 tx=0x7f23400b21a0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.302+0000 7f236517fb80 1 -- 172.21.15.44:0/2952620673 shutdown_connections 2024-05-27T09:48:03.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.302+0000 7f236517fb80 1 --2- 172.21.15.44:0/2952620673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf632f0 0x55895bf7beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.302+0000 7f236517fb80 1 --2- 172.21.15.44:0/2952620673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bdddf40 0x55895bf73060 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.302+0000 7f236517fb80 1 --2- 172.21.15.44:0/2952620673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bf5d440 0x55895bf5c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.302+0000 7f236517fb80 1 -- 172.21.15.44:0/2952620673 >> 172.21.15.44:0/2952620673 conn(0x55895be3e0c0 msgr2=0x55895bf60e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:03.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.302+0000 7f236517fb80 1 -- 172.21.15.44:0/2952620673 shutdown_connections 2024-05-27T09:48:03.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.302+0000 7f236517fb80 1 -- 172.21.15.44:0/2952620673 wait complete. 2024-05-27T09:48:03.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.303+0000 7f236517fb80 1 -- start start 2024-05-27T09:48:03.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.303+0000 7f236517fb80 1 Processor -- start 2024-05-27T09:48:03.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f236517fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bdddf40 0x55895bf7ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f236517fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf5d440 0x55895bf7b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f236517fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf632f0 0x55895bf7b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f236517fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f233000bcc0 con 0x55895bdddf40 2024-05-27T09:48:03.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f236517fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f233000d130 con 0x55895bf5d440 2024-05-27T09:48:03.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f236517fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f233000c390 con 0x55895bf632f0 2024-05-27T09:48:03.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f236517fb80 10 client.0 _ll_register_callbacks cb 0x55895be24d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:03.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f234f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bdddf40 0x55895bf7ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f234f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bdddf40 0x55895bf7ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48580/0 (socket says 172.21.15.44:48580) 2024-05-27T09:48:03.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f234f400700 1 -- 172.21.15.44:0/828558515 learned_addr learned my addr 172.21.15.44:0/828558515 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:03.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f234ea00700 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf5d440 0x55895bf7b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.304+0000 7f234fe00700 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf632f0 0x55895bf7b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.305+0000 7f234ea00700 1 -- 172.21.15.44:0/828558515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf632f0 msgr2=0x55895bf7b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:03.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.305+0000 7f234ea00700 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf632f0 0x55895bf7b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.305+0000 7f234ea00700 1 -- 172.21.15.44:0/828558515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bdddf40 msgr2=0x55895bf7ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:03.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.305+0000 7f234ea00700 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bdddf40 0x55895bf7ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.305+0000 7f234ea00700 1 -- 172.21.15.44:0/828558515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895bfe12f0 con 0x55895bf5d440 2024-05-27T09:48:03.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.305+0000 7f234ea00700 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf5d440 0x55895bf7b050 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f233c009920 tx=0x7f233c00f8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63470de70ea8e7da server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:03.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.305+0000 7f2339600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:03.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.306+0000 7f2339600700 1 -- 172.21.15.44:0/828558515 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c005520 con 0x55895bf5d440 2024-05-27T09:48:03.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.306+0000 7f236517fb80 1 -- 172.21.15.44:0/828558515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23400aa030 con 0x55895bf5d440 2024-05-27T09:48:03.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.306+0000 7f2339600700 1 -- 172.21.15.44:0/828558515 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f233c003680 con 0x55895bf5d440 2024-05-27T09:48:03.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.306+0000 7f2339600700 1 -- 172.21.15.44:0/828558515 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f233c0056c0 con 0x55895bf5d440 2024-05-27T09:48:03.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.306+0000 7f236517fb80 10 client.27680 fetch_fsmap learned FSMap version 42 2024-05-27T09:48:03.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.306+0000 7f236517fb80 1 -- 172.21.15.44:0/828558515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55895bfe24b0 con 0x55895bf5d440 2024-05-27T09:48:03.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.306+0000 7f2339600700 1 -- 172.21.15.44:0/828558515 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23400aa030 con 0x55895bf5d440 2024-05-27T09:48:03.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.306+0000 7f2339600700 1 -- 172.21.15.44:0/828558515 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f233c036070 con 0x55895bf5d440 2024-05-27T09:48:03.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.306+0000 7f236517fb80 10 client.27680 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:48:03.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.307+0000 7f236517fb80 10 client.27680 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:48:03.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.307+0000 7f236517fb80 20 client.27680 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:03.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.307+0000 7f236517fb80 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55895bfe2c10 0x55895bfe5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:03.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.307+0000 7f236517fb80 4 client.27680 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "8d7cdbbb", "release": true, "await": true}] 2024-05-27T09:48:03.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.307+0000 7f236517fb80 1 -- 172.21.15.44:0/828558515 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "set_id": "8d7cdbbb", "release": true, "await": true}) -- 0x55895be86670 con 0x55895bfe2c10 2024-05-27T09:48:03.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.307+0000 7f234f400700 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55895bfe2c10 0x55895bfe5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:03.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.307+0000 7f234f400700 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55895bfe2c10 0x55895bfe5030 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=896a1f7b86cba4b2 server_cookie=9c47b5235e6df29c in_seq=0 out_seq=0 2024-05-27T09:48:03.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.307+0000 7f2339600700 10 client.27680 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:03.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f2339600700 1 -- 172.21.15.44:0/828558515 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55895be86670 con 0x55895bfe2c10 2024-05-27T09:48:03.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f2339600700 10 client.27680 handle_command_reply: tid=0 2024-05-27T09:48:03.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f236517fb80 1 client.27680 shutdown 2024-05-27T09:48:03.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f236517fb80 1 -- 172.21.15.44:0/828558515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf5d440 msgr2=0x55895bf7b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:03.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f236517fb80 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf5d440 0x55895bf7b050 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f233c009920 tx=0x7f233c00f8c0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f236517fb80 1 -- 172.21.15.44:0/828558515 shutdown_connections 2024-05-27T09:48:03.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f236517fb80 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55895bfe2c10 0x55895bfe5030 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f236517fb80 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895bf632f0 0x55895bf7b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f236517fb80 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55895bdddf40 0x55895bf7ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f2339600700 0 client.27680 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:03.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f236517fb80 1 --2- 172.21.15.44:0/828558515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55895bf5d440 0x55895bf7b050 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f236517fb80 1 -- 172.21.15.44:0/828558515 >> 172.21.15.44:0/828558515 conn(0x55895be3e0c0 msgr2=0x55895bf60e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:03.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f236517fb80 1 -- 172.21.15.44:0/828558515 shutdown_connections 2024-05-27T09:48:03.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.312+0000 7f236517fb80 1 -- 172.21.15.44:0/828558515 wait complete. 2024-05-27T09:48:03.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.313+0000 7f236517fb80 20 client.27680 trim_cache size 0 max 16384 2024-05-27T09:48:03.314 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:48:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 42, 2024-05-27T09:48:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:48:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 32, 2024-05-27T09:48:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "8d7cdbbb": { 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 32, 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 142.3, 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:48:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:48:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:48:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:48:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:48:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:48:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:48:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:48:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:48:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:48:03.316 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:48:03.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.313+0000 7f2356600700 1 -- 172.21.15.44:0/894594068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23340b29f0 msgr2=0x7f23340b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:03.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.313+0000 7f2356600700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23340b29f0 0x7f23340b4f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f234800f5e0 tx=0x7f2348009920 comp rx=0 tx=0).stop 2024-05-27T09:48:03.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.313+0000 7f2356600700 1 -- 172.21.15.44:0/894594068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23501034c0 msgr2=0x7f23500ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:03.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.313+0000 7f2356600700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23501034c0 0x7f23500ff3f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f234000e9c0 tx=0x7f234000ecd0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.314+0000 7f2356600700 1 -- 172.21.15.44:0/894594068 shutdown_connections 2024-05-27T09:48:03.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.314+0000 7f2356600700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f23340b29f0 0x7f23340b4f00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.314+0000 7f2356600700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2350108740 0x7f23500ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.314+0000 7f2356600700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23501034c0 0x7f23500ff3f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.314+0000 7f2356600700 1 --2- 172.21.15.44:0/894594068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2350107df0 0x7f23500ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:03.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.314+0000 7f2356600700 1 -- 172.21.15.44:0/894594068 >> 172.21.15.44:0/894594068 conn(0x7f23500fef10 msgr2=0x7f23501075b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:03.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.314+0000 7f2356600700 1 -- 172.21.15.44:0/894594068 shutdown_connections 2024-05-27T09:48:03.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:03.314+0000 7f2356600700 1 -- 172.21.15.44:0/894594068 wait complete. 2024-05-27T09:48:04.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:04 smithi044 ceph-mon[31618]: pgmap v402: 97 pgs: 97 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 116 KiB/s wr, 51 op/s 2024-05-27T09:48:04.386 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:04 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:48:04.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:04 smithi050 ceph-mon[36371]: pgmap v402: 97 pgs: 97 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 116 KiB/s wr, 51 op/s 2024-05-27T09:48:04.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:04 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:48:04.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:04 smithi189 ceph-mon[36671]: pgmap v402: 97 pgs: 97 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 116 KiB/s wr, 51 op/s 2024-05-27T09:48:04.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:04 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:48:04.774 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '8d7cdbbb', total seconds elapsed: 35.3 2024-05-27T09:48:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.438+0000 7f0603200700 1 -- 172.21.15.44:0/2581102720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05fc105da0 msgr2=0x7f05fc10e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.438+0000 7f0603200700 1 --2- 172.21.15.44:0/2581102720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05fc105da0 0x7f05fc10e2f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009990 tx=0x7f05e4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.438+0000 7f0603200700 1 -- 172.21.15.44:0/2581102720 shutdown_connections 2024-05-27T09:48:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.438+0000 7f0603200700 1 --2- 172.21.15.44:0/2581102720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05fc10e8c0 0x7f05fc110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.438+0000 7f0603200700 1 --2- 172.21.15.44:0/2581102720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05fc105450 0x7f05fc105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.438+0000 7f0603200700 1 --2- 172.21.15.44:0/2581102720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05fc105da0 0x7f05fc10e2f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.438+0000 7f0603200700 1 -- 172.21.15.44:0/2581102720 >> 172.21.15.44:0/2581102720 conn(0x7f05fc0fedb0 msgr2=0x7f05fc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.438+0000 7f0603200700 1 -- 172.21.15.44:0/2581102720 shutdown_connections 2024-05-27T09:48:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.438+0000 7f0603200700 1 -- 172.21.15.44:0/2581102720 wait complete. 2024-05-27T09:48:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.439+0000 7f0603200700 1 Processor -- start 2024-05-27T09:48:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.440+0000 7f0603200700 1 -- start start 2024-05-27T09:48:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.440+0000 7f0603200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05fc105450 0x7f05fc19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.440+0000 7f0603200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05fc105da0 0x7f05fc19b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.440+0000 7f0603200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05fc10e8c0 0x7f05fc19f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.440+0000 7f0603200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc114470 con 0x7f05fc105da0 2024-05-27T09:48:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.440+0000 7f0603200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc1142f0 con 0x7f05fc10e8c0 2024-05-27T09:48:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.440+0000 7f0603200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc1145f0 con 0x7f05fc105450 2024-05-27T09:48:05.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.440+0000 7f0601200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05fc10e8c0 0x7f05fc19f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.441+0000 7f05fbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05fc105450 0x7f05fc19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.441+0000 7f0601200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05fc10e8c0 0x7f05fc19f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:40028/0 (socket says 172.21.15.44:40028) 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.441+0000 7f0601200700 1 -- 172.21.15.44:0/28205112 learned_addr learned my addr 172.21.15.44:0/28205112 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.441+0000 7f05f3e00700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05fc105da0 0x7f05fc19b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.441+0000 7f05fbe00700 1 -- 172.21.15.44:0/28205112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05fc10e8c0 msgr2=0x7f05fc19f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.441+0000 7f05fbe00700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05fc10e8c0 0x7f05fc19f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.441+0000 7f05fbe00700 1 -- 172.21.15.44:0/28205112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05fc105da0 msgr2=0x7f05fc19b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.441+0000 7f05fbe00700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05fc105da0 0x7f05fc19b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.441+0000 7f05fbe00700 1 -- 172.21.15.44:0/28205112 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05fc19fc90 con 0x7f05fc105450 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.441+0000 7f05f3e00700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05fc105da0 0x7f05fc19b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.442+0000 7f05fbe00700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05fc105450 0x7f05fc19adc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00e9f0 tx=0x7f05ec00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.442+0000 7f05f9600700 1 -- 172.21.15.44:0/28205112 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec00c9c0 con 0x7f05fc105450 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.442+0000 7f0603200700 1 -- 172.21.15.44:0/28205112 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05fc19ff80 con 0x7f05fc105450 2024-05-27T09:48:05.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.442+0000 7f05f9600700 1 -- 172.21.15.44:0/28205112 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05ec00cb60 con 0x7f05fc105450 2024-05-27T09:48:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.442+0000 7f05f9600700 1 -- 172.21.15.44:0/28205112 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec018680 con 0x7f05fc105450 2024-05-27T09:48:05.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.442+0000 7f0603200700 1 -- 172.21.15.44:0/28205112 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05fc1a78a0 con 0x7f05fc105450 2024-05-27T09:48:05.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.448+0000 7f05f9600700 1 -- 172.21.15.44:0/28205112 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f05ec010430 con 0x7f05fc105450 2024-05-27T09:48:05.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.449+0000 7f0611ceeb80 1 Processor -- start 2024-05-27T09:48:05.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.449+0000 7f0611ceeb80 1 -- start start 2024-05-27T09:48:05.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e427f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e3c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e3d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55add1d6ae40 con 0x55add1e42d30 2024-05-27T09:48:05.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55add1d5a9e0 con 0x55add1e3cdb0 2024-05-27T09:48:05.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55add1d5ea20 con 0x55add1cbde40 2024-05-27T09:48:05.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f05f9600700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05e00b29f0 0x7f05e00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f0601200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f05fbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e3c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f05f3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e3d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f0601200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e427f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38478/0 (socket says 172.21.15.44:38478) 2024-05-27T09:48:05.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.450+0000 7f0601200700 1 -- 172.21.15.44:0/3048748798 learned_addr learned my addr 172.21.15.44:0/3048748798 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:05.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.451+0000 7f05f3e00700 1 -- 172.21.15.44:0/3048748798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 msgr2=0x55add1e427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:05.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.451+0000 7f05f3e00700 1 --2- 172.21.15.44:0/3048748798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e427f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.451+0000 7f05f3e00700 1 -- 172.21.15.44:0/3048748798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 msgr2=0x55add1e3c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:05.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.451+0000 7f05f9600700 1 -- 172.21.15.44:0/28205112 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f05ec0e2810 con 0x7f05fc105450 2024-05-27T09:48:05.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.451+0000 7f0601200700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05e00b29f0 0x7f05e00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.451+0000 7f05f3e00700 1 --2- 172.21.15.44:0/3048748798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e3c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.451+0000 7f05f3e00700 1 -- 172.21.15.44:0/3048748798 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55add1d23420 con 0x55add1e3cdb0 2024-05-27T09:48:05.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.451+0000 7f05fbe00700 1 --2- 172.21.15.44:0/3048748798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e3c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:05.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.451+0000 7f05f3e00700 1 --2- 172.21.15.44:0/3048748798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e3d1c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f05e4006010 tx=0x7f05e40096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c2d7a284aaa22e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:05.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.452+0000 7f0601200700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05e00b29f0 0x7f05e00b4f00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55add1e3db30 tx=0x7f05f4004b40 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:05.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.452+0000 7f05f1600700 1 -- 172.21.15.44:0/3048748798 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e4005270 con 0x55add1e3cdb0 2024-05-27T09:48:05.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.452+0000 7f05f1600700 1 -- 172.21.15.44:0/3048748798 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05e4005410 con 0x55add1e3cdb0 2024-05-27T09:48:05.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.452+0000 7f05f1600700 1 -- 172.21.15.44:0/3048748798 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e400f480 con 0x55add1e3cdb0 2024-05-27T09:48:05.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.453+0000 7f0611ceeb80 1 -- 172.21.15.44:0/3048748798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 msgr2=0x55add1e3d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:05.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.453+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/3048748798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e3d1c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f05e4006010 tx=0x7f05e40096a0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.453+0000 7f0611ceeb80 1 -- 172.21.15.44:0/3048748798 shutdown_connections 2024-05-27T09:48:05.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.453+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/3048748798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e3d1c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.453+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/3048748798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.453+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/3048748798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e3c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.453+0000 7f0611ceeb80 1 -- 172.21.15.44:0/3048748798 >> 172.21.15.44:0/3048748798 conn(0x55add1d1dfc0 msgr2=0x55add1e378d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:05.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.453+0000 7f0611ceeb80 1 -- 172.21.15.44:0/3048748798 shutdown_connections 2024-05-27T09:48:05.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.453+0000 7f0611ceeb80 1 -- 172.21.15.44:0/3048748798 wait complete. 2024-05-27T09:48:05.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.454+0000 7f0611ceeb80 1 -- start start 2024-05-27T09:48:05.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.454+0000 7f0611ceeb80 1 Processor -- start 2024-05-27T09:48:05.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e46480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e469c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e46f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55add1c60fa0 con 0x55add1e42d30 2024-05-27T09:48:05.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55add1cfb760 con 0x55add1e3cdb0 2024-05-27T09:48:05.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55add1e4db50 con 0x55add1cbde40 2024-05-27T09:48:05.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f0611ceeb80 10 client.0 _ll_register_callbacks cb 0x55add1d04c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:05.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f05f3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f05f3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e469c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:40040/0 (socket says 172.21.15.44:40040) 2024-05-27T09:48:05.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f05f3e00700 1 -- 172.21.15.44:0/1154689732 learned_addr learned my addr 172.21.15.44:0/1154689732 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:05.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f05fbe00700 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e46480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.455+0000 7f0601200700 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e46f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.456+0000 7f05f3e00700 1 -- 172.21.15.44:0/1154689732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 msgr2=0x55add1e46480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:05.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.456+0000 7f05f3e00700 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e46480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.456+0000 7f05f3e00700 1 -- 172.21.15.44:0/1154689732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 msgr2=0x55add1e46f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:05.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.456+0000 7f05f3e00700 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e46f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.456+0000 7f05f3e00700 1 -- 172.21.15.44:0/1154689732 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55add1caa8d0 con 0x55add1e3cdb0 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.456+0000 7f05fbe00700 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e46480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.456+0000 7f05f3e00700 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e469c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f05e4005a80 tx=0x7f05e40224e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb46f168700e1597 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.456+0000 7f05ea000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.457+0000 7f05ea000700 1 -- 172.21.15.44:0/1154689732 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e4031060 con 0x55add1e3cdb0 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.457+0000 7f05ea000700 1 -- 172.21.15.44:0/1154689732 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f05e40227b0 con 0x55add1e3cdb0 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.457+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1154689732 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05e4005410 con 0x55add1e3cdb0 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.457+0000 7f05ea000700 1 -- 172.21.15.44:0/1154689732 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f05e40229b0 con 0x55add1e3cdb0 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.457+0000 7f0611ceeb80 10 client.27692 fetch_fsmap learned FSMap version 42 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.457+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1154689732 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55add1d2b060 con 0x55add1e3cdb0 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.457+0000 7f05ea000700 1 -- 172.21.15.44:0/1154689732 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05e4005410 con 0x55add1e3cdb0 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.458+0000 7f05ea000700 1 -- 172.21.15.44:0/1154689732 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f05e400f460 con 0x55add1e3cdb0 2024-05-27T09:48:05.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.458+0000 7f0611ceeb80 10 client.27692 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:48:05.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.458+0000 7f0611ceeb80 10 client.27692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:05.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.458+0000 7f0611ceeb80 20 client.27692 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:05.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.458+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55add1e5a850 0x55add1e5cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.458+0000 7f0611ceeb80 4 client.27692 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:05.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.458+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1154689732 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55add1d6cd90 con 0x55add1e5a850 2024-05-27T09:48:05.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.458+0000 7f05fbe00700 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55add1e5a850 0x55add1e5cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.459+0000 7f05fbe00700 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55add1e5a850 0x55add1e5cc70 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce6c2f9a87a53663 server_cookie=7b75cb77f72cff92 in_seq=0 out_seq=0 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.459+0000 7f05ea000700 10 client.27692 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f05ea000700 1 -- 172.21.15.44:0/1154689732 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55add1d6cd90 con 0x55add1e5a850 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f05ea000700 10 client.27692 handle_command_reply: tid=0 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f0611ceeb80 1 client.27692 shutdown 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1154689732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 msgr2=0x55add1e469c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e469c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f05e4005a80 tx=0x7f05e40224e0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1154689732 shutdown_connections 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55add1e5a850 0x55add1e5cc70 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e46f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e46480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1154689732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e469c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1154689732 >> 172.21.15.44:0/1154689732 conn(0x55add1d1dfc0 msgr2=0x55add1e37580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:05.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.460+0000 7f05ea000700 0 client.27692 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:05.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.461+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1154689732 shutdown_connections 2024-05-27T09:48:05.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.461+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1154689732 wait complete. 2024-05-27T09:48:05.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.461+0000 7f0611ceeb80 20 client.27692 trim_cache size 0 max 16384 2024-05-27T09:48:05.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.486+0000 7f0611ceeb80 1 Processor -- start 2024-05-27T09:48:05.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.486+0000 7f0611ceeb80 1 -- start start 2024-05-27T09:48:05.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.486+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e3c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.486+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e56db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.486+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e48920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.486+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc1af510 con 0x55add1e42d30 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.486+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc1acd70 con 0x55add1e3cdb0 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.486+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc1aabc0 con 0x55add1cbde40 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f05f3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e56db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f05f3e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e56db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:40042/0 (socket says 172.21.15.44:40042) 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f05f3e00700 1 -- 172.21.15.44:0/1013496166 learned_addr learned my addr 172.21.15.44:0/1013496166 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f0601200700 1 --2- 172.21.15.44:0/1013496166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e48920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f05fbe00700 1 --2- 172.21.15.44:0/1013496166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e3c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f05f3e00700 1 -- 172.21.15.44:0/1013496166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 msgr2=0x55add1e3c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f05f3e00700 1 --2- 172.21.15.44:0/1013496166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e3c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f05f3e00700 1 -- 172.21.15.44:0/1013496166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 msgr2=0x55add1e48920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:05.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f05f3e00700 1 --2- 172.21.15.44:0/1013496166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e48920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f05f3e00700 1 -- 172.21.15.44:0/1013496166 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55add1e47270 con 0x55add1e3cdb0 2024-05-27T09:48:05.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f0601200700 1 --2- 172.21.15.44:0/1013496166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e48920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.487+0000 7f05f3e00700 1 --2- 172.21.15.44:0/1013496166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e56db0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f05e400c010 tx=0x7f05e400b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=677245dfe0cb4375 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.488+0000 7f05f0c00700 1 -- 172.21.15.44:0/1013496166 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e4031070 con 0x55add1e3cdb0 2024-05-27T09:48:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.488+0000 7f05f0c00700 1 -- 172.21.15.44:0/1013496166 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05e4005410 con 0x55add1e3cdb0 2024-05-27T09:48:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.488+0000 7f05f0c00700 1 -- 172.21.15.44:0/1013496166 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e40038e0 con 0x55add1e3cdb0 2024-05-27T09:48:05.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.489+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1013496166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 msgr2=0x55add1e56db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:05.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.489+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1013496166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e56db0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f05e400c010 tx=0x7f05e400b260 comp rx=0 tx=0).stop 2024-05-27T09:48:05.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.489+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1013496166 shutdown_connections 2024-05-27T09:48:05.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.489+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1013496166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1e42d30 0x55add1e48920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.489+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1013496166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1cbde40 0x55add1e3c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.489+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1013496166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e3cdb0 0x55add1e56db0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.489+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1013496166 >> 172.21.15.44:0/1013496166 conn(0x55add1d1dfc0 msgr2=0x55add1e407b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:05.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.489+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1013496166 shutdown_connections 2024-05-27T09:48:05.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.489+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1013496166 wait complete. 2024-05-27T09:48:05.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.490+0000 7f0611ceeb80 1 -- start start 2024-05-27T09:48:05.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.490+0000 7f0611ceeb80 1 Processor -- start 2024-05-27T09:48:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.490+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1cbde40 0x55add1e47f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.490+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1e3cdb0 0x55add1e48450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.490+0000 7f0611ceeb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e42d30 0x55add1ebd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.490+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc1ab230 con 0x55add1cbde40 2024-05-27T09:48:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.490+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc1ac6a0 con 0x55add1e42d30 2024-05-27T09:48:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.490+0000 7f0611ceeb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc1ab900 con 0x55add1e3cdb0 2024-05-27T09:48:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.490+0000 7f0611ceeb80 10 client.0 _ll_register_callbacks cb 0x55add1d04c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.491+0000 7f05fbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1cbde40 0x55add1e47f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.491+0000 7f05fbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1cbde40 0x55add1e47f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48654/0 (socket says 172.21.15.44:48654) 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.491+0000 7f05fbe00700 1 -- 172.21.15.44:0/1593867389 learned_addr learned my addr 172.21.15.44:0/1593867389 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.491+0000 7f05f3e00700 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1e3cdb0 0x55add1e48450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.491+0000 7f05fbe00700 1 -- 172.21.15.44:0/1593867389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1e3cdb0 msgr2=0x55add1e48450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.491+0000 7f05fbe00700 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1e3cdb0 0x55add1e48450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.491+0000 7f05fbe00700 1 -- 172.21.15.44:0/1593867389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e42d30 msgr2=0x55add1ebd0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.491+0000 7f05fbe00700 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e42d30 0x55add1ebd0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.491+0000 7f05fbe00700 1 -- 172.21.15.44:0/1593867389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55add1ebd6b0 con 0x55add1cbde40 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.492+0000 7f05fbe00700 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1cbde40 0x55add1e47f10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0dff30 tx=0x7f05ec0a5240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae1d595959752d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.492+0000 7f05ea000700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.492+0000 7f05ea000700 1 -- 172.21.15.44:0/1593867389 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec010920 con 0x55add1cbde40 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.493+0000 7f05ea000700 1 -- 172.21.15.44:0/1593867389 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8563+0+0 (secure 0 0 0) 0x7f05ec0a9cc0 con 0x55add1cbde40 2024-05-27T09:48:05.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.493+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1593867389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05e4005410 con 0x55add1cbde40 2024-05-27T09:48:05.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.493+0000 7f05ea000700 1 -- 172.21.15.44:0/1593867389 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f05ec0aa160 con 0x55add1cbde40 2024-05-27T09:48:05.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.493+0000 7f0611ceeb80 10 client.17454 fetch_fsmap learned FSMap version 42 2024-05-27T09:48:05.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.493+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1593867389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55add1ec4a30 con 0x55add1cbde40 2024-05-27T09:48:05.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.493+0000 7f05ea000700 1 -- 172.21.15.44:0/1593867389 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05e4005410 con 0x55add1cbde40 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.493+0000 7f05ea000700 1 -- 172.21.15.44:0/1593867389 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f05ec0b2430 con 0x55add1cbde40 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.494+0000 7f0611ceeb80 10 client.17454 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.494+0000 7f0611ceeb80 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.494+0000 7f0611ceeb80 20 client.17454 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.494+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55add1ec50f0 0x55add1ec7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.494+0000 7f0611ceeb80 4 client.17454 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.494+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1593867389 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55add1d66570 con 0x55add1ec50f0 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.494+0000 7f05f3e00700 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55add1ec50f0 0x55add1ec7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f05f3e00700 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55add1ec50f0 0x55add1ec7530 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52bb960267fdffaf server_cookie=a47eede1006d9bf6 in_seq=0 out_seq=0 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f05ea000700 10 client.17454 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f05ea000700 1 -- 172.21.15.44:0/1593867389 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55add1d66570 con 0x55add1ec50f0 2024-05-27T09:48:05.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f05ea000700 10 client.17454 handle_command_reply: tid=0 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f0611ceeb80 1 client.17454 shutdown 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1593867389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1cbde40 msgr2=0x55add1e47f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1cbde40 0x55add1e47f10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0dff30 tx=0x7f05ec0a5240 comp rx=0 tx=0).stop 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1593867389 shutdown_connections 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55add1ec50f0 0x55add1ec7530 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55add1e42d30 0x55add1ebd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55add1cbde40 0x55add1e47f10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f0611ceeb80 1 --2- 172.21.15.44:0/1593867389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55add1e3cdb0 0x55add1e48450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1593867389 >> 172.21.15.44:0/1593867389 conn(0x55add1d1dfc0 msgr2=0x55add1e44c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f05ea000700 0 client.17454 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1593867389 shutdown_connections 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.495+0000 7f0611ceeb80 1 -- 172.21.15.44:0/1593867389 wait complete. 2024-05-27T09:48:05.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.496+0000 7f0611ceeb80 20 client.17454 trim_cache size 0 max 16384 2024-05-27T09:48:05.499 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:48:05.499 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:48:05.499 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:48:05.499 INFO:teuthology.orchestra.run.smithi044.stdout: "c9cadb11-5c59-4338-b772-13e508320e23": { 2024-05-27T09:48:05.499 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:48:05.499 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "c9cadb11-5c59-4338-b772-13e508320e23", 2024-05-27T09:48:05.499 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:48:05.499 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:48:05.499 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:48:05.500 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:48:05.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.496+0000 7f0603200700 1 -- 172.21.15.44:0/28205112 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05e00b29f0 msgr2=0x7f05e00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:05.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.496+0000 7f0603200700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05e00b29f0 0x7f05e00b4f00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55add1e3db30 tx=0x7f05f4004b40 comp rx=0 tx=0).stop 2024-05-27T09:48:05.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.496+0000 7f0603200700 1 -- 172.21.15.44:0/28205112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05fc105450 msgr2=0x7f05fc19adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:05.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.496+0000 7f0603200700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05fc105450 0x7f05fc19adc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00e9f0 tx=0x7f05ec00ed00 comp rx=0 tx=0).stop 2024-05-27T09:48:05.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.497+0000 7f0603200700 1 -- 172.21.15.44:0/28205112 shutdown_connections 2024-05-27T09:48:05.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.497+0000 7f0603200700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f05e00b29f0 0x7f05e00b4f00 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.497+0000 7f0603200700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05fc10e8c0 0x7f05fc19f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.497+0000 7f0603200700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05fc105450 0x7f05fc19adc0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.497+0000 7f0603200700 1 --2- 172.21.15.44:0/28205112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f05fc105da0 0x7f05fc19b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:05.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.497+0000 7f0603200700 1 -- 172.21.15.44:0/28205112 >> 172.21.15.44:0/28205112 conn(0x7f05fc0fedb0 msgr2=0x7f05fc100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:05.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.497+0000 7f0603200700 1 -- 172.21.15.44:0/28205112 shutdown_connections 2024-05-27T09:48:05.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:05.497+0000 7f0603200700 1 -- 172.21.15.44:0/28205112 wait complete. 2024-05-27T09:48:06.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:06 smithi044 ceph-mon[31618]: pgmap v403: 97 pgs: 97 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 116 KiB/s wr, 51 op/s 2024-05-27T09:48:06.387 INFO:tasks.cephfs.filesystem:scrub status for tag:c9cadb11-5c59-4338-b772-13e508320e23 - {'path': '/', 'tag': 'c9cadb11-5c59-4338-b772-13e508320e23', 'options': 'recursive,force'} 2024-05-27T09:48:06.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:06 smithi050 ceph-mon[36371]: pgmap v403: 97 pgs: 97 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 116 KiB/s wr, 51 op/s 2024-05-27T09:48:06.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:06 smithi189 ceph-mon[36671]: pgmap v403: 97 pgs: 97 active+clean; 311 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 116 KiB/s wr, 51 op/s 2024-05-27T09:48:08.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:08 smithi044 ceph-mon[31618]: pgmap v404: 97 pgs: 97 active+clean; 316 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 562 KiB/s wr, 1.11k op/s 2024-05-27T09:48:08.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:08 smithi044 ceph-mon[31618]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T09:48:08.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:08 smithi050 ceph-mon[36371]: pgmap v404: 97 pgs: 97 active+clean; 316 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 562 KiB/s wr, 1.11k op/s 2024-05-27T09:48:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:08 smithi050 ceph-mon[36371]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T09:48:08.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:08 smithi189 ceph-mon[36671]: pgmap v404: 97 pgs: 97 active+clean; 316 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 562 KiB/s wr, 1.11k op/s 2024-05-27T09:48:08.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:08 smithi189 ceph-mon[36671]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T09:48:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.302+0000 7f928ce00640 1 -- 172.21.15.44:0/51225423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9288112900 msgr2=0x7f9288118ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.302+0000 7f928ce00640 1 --2- 172.21.15.44:0/51225423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9288112900 0x7f9288118ec0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9280009be0 tx=0x7f928002a6d0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.302+0000 7f928ce00640 1 -- 172.21.15.44:0/51225423 shutdown_connections 2024-05-27T09:48:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.302+0000 7f928ce00640 1 --2- 172.21.15.44:0/51225423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9288112900 0x7f9288118ec0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.302+0000 7f928ce00640 1 --2- 172.21.15.44:0/51225423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9288111e60 0x7f9288112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.302+0000 7f928ce00640 1 --2- 172.21.15.44:0/51225423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f928810a7c0 0x7f928810abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.302+0000 7f928ce00640 1 -- 172.21.15.44:0/51225423 >> 172.21.15.44:0/51225423 conn(0x7f928806eda0 msgr2=0x7f928806f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.302+0000 7f928ce00640 1 -- 172.21.15.44:0/51225423 shutdown_connections 2024-05-27T09:48:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.302+0000 7f928ce00640 1 -- 172.21.15.44:0/51225423 wait complete. 2024-05-27T09:48:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.305+0000 7f928ce00640 1 Processor -- start 2024-05-27T09:48:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.305+0000 7f928ce00640 1 -- start start 2024-05-27T09:48:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.306+0000 7f928ce00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f928810a7c0 0x7f92881a9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.306+0000 7f928ce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9288111e60 0x7f92881a9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.306+0000 7f928ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9288112900 0x7f92881ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.306+0000 7f928ce00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9288122c70 con 0x7f9288111e60 2024-05-27T09:48:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.306+0000 7f928ce00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9288122af0 con 0x7f928810a7c0 2024-05-27T09:48:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.306+0000 7f928ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9288122df0 con 0x7f9288112900 2024-05-27T09:48:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.307+0000 7f9286800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9288112900 0x7f92881ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.307+0000 7f9286800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9288112900 0x7f92881ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38524/0 (socket says 172.21.15.44:38524) 2024-05-27T09:48:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.307+0000 7f9286800640 1 -- 172.21.15.44:0/3113945710 learned_addr learned my addr 172.21.15.44:0/3113945710 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.308+0000 7f9285e00640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f928810a7c0 0x7f92881a9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.308+0000 7f9286800640 1 -- 172.21.15.44:0/3113945710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f928810a7c0 msgr2=0x7f92881a9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.308+0000 7f9286800640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f928810a7c0 0x7f92881a9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.308+0000 7f9285400640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9288111e60 0x7f92881a9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.308+0000 7f9286800640 1 -- 172.21.15.44:0/3113945710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9288111e60 msgr2=0x7f92881a9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.308+0000 7f9286800640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9288111e60 0x7f92881a9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.308+0000 7f9286800640 1 -- 172.21.15.44:0/3113945710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92881ae490 con 0x7f9288112900 2024-05-27T09:48:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.309+0000 7f9286800640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9288112900 0x7f92881ade50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9280003c10 tx=0x7f9280003c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.309+0000 7f927ea00640 1 -- 172.21.15.44:0/3113945710 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280040020 con 0x7f9288112900 2024-05-27T09:48:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.310+0000 7f927ea00640 1 -- 172.21.15.44:0/3113945710 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9280003e20 con 0x7f9288112900 2024-05-27T09:48:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.310+0000 7f928ce00640 1 -- 172.21.15.44:0/3113945710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92881ae720 con 0x7f9288112900 2024-05-27T09:48:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.310+0000 7f928ce00640 1 -- 172.21.15.44:0/3113945710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92881aec60 con 0x7f9288112900 2024-05-27T09:48:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.310+0000 7f927ea00640 1 -- 172.21.15.44:0/3113945710 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280040020 con 0x7f9288112900 2024-05-27T09:48:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.315+0000 7f9293f71740 1 Processor -- start 2024-05-27T09:48:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.319+0000 7f927ea00640 1 -- 172.21.15.44:0/3113945710 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f9280043020 con 0x7f9288112900 2024-05-27T09:48:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.320+0000 7f9293f71740 1 -- start start 2024-05-27T09:48:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.320+0000 7f9293f71740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f27860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.320+0000 7f9293f71740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b5c2f0 0x55b466f27da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.320+0000 7f9293f71740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b644d0 0x55b466f282e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.320+0000 7f9293f71740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b466e98b90 con 0x55b466b5c2f0 2024-05-27T09:48:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.320+0000 7f9293f71740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b466e88730 con 0x55b466b644d0 2024-05-27T09:48:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.320+0000 7f9293f71740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b466e8c770 con 0x55b466b851b0 2024-05-27T09:48:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.321+0000 7f9285400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f27860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.321+0000 7f9285400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f27860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38536/0 (socket says 172.21.15.44:38536) 2024-05-27T09:48:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.321+0000 7f9285400640 1 -- 172.21.15.44:0/165436898 learned_addr learned my addr 172.21.15.44:0/165436898 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.321+0000 7f9285400640 1 --2- 172.21.15.44:0/165436898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b644d0 0x55b466f282e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.322+0000 7f9285400640 1 -- 172.21.15.44:0/165436898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b644d0 msgr2=0x55b466f282e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.322+0000 7f9285400640 1 --2- 172.21.15.44:0/165436898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b644d0 0x55b466f282e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.322+0000 7f9285400640 1 -- 172.21.15.44:0/165436898 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b5c2f0 msgr2=0x55b466f27da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.322+0000 7f9285400640 1 --2- 172.21.15.44:0/165436898 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b5c2f0 0x55b466f27da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.322+0000 7f9285400640 1 -- 172.21.15.44:0/165436898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b466e218b0 con 0x55b466b851b0 2024-05-27T09:48:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.322+0000 7f9285400640 1 --2- 172.21.15.44:0/165436898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f27860 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f92740058c0 tx=0x7f9274009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe3e1cbac6abd5fb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.323+0000 7f9273400640 1 -- 172.21.15.44:0/165436898 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92740150f0 con 0x55b466b851b0 2024-05-27T09:48:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.323+0000 7f9273400640 1 -- 172.21.15.44:0/165436898 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9274015290 con 0x55b466b851b0 2024-05-27T09:48:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.323+0000 7f9273400640 1 -- 172.21.15.44:0/165436898 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92740047a0 con 0x55b466b851b0 2024-05-27T09:48:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.324+0000 7f927ea00640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f926c0b2c10 0x7f926c0b5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.324+0000 7f927ea00640 1 -- 172.21.15.44:0/3113945710 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8651+0+0 (secure 0 0 0) 0x7f9280102da0 con 0x7f9288112900 2024-05-27T09:48:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.325+0000 7f9293f71740 1 -- 172.21.15.44:0/165436898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 msgr2=0x55b466f27860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.325+0000 7f9285e00640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f926c0b2c10 0x7f926c0b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.326+0000 7f9293f71740 1 --2- 172.21.15.44:0/165436898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f27860 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f92740058c0 tx=0x7f9274009650 comp rx=0 tx=0).stop 2024-05-27T09:48:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.327+0000 7f9293f71740 1 -- 172.21.15.44:0/165436898 shutdown_connections 2024-05-27T09:48:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.327+0000 7f9293f71740 1 --2- 172.21.15.44:0/165436898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b644d0 0x55b466f282e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.327+0000 7f9293f71740 1 --2- 172.21.15.44:0/165436898 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b5c2f0 0x55b466f27da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.327+0000 7f9293f71740 1 --2- 172.21.15.44:0/165436898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f27860 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.327+0000 7f9293f71740 1 -- 172.21.15.44:0/165436898 >> 172.21.15.44:0/165436898 conn(0x55b466b79300 msgr2=0x55b466f24010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.327+0000 7f9285e00640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f926c0b2c10 0x7f926c0b5120 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55b466f28c10 tx=0x7f927800d670 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.329+0000 7f9293f71740 1 -- 172.21.15.44:0/165436898 shutdown_connections 2024-05-27T09:48:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.330+0000 7f9293f71740 1 -- 172.21.15.44:0/165436898 wait complete. 2024-05-27T09:48:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.330+0000 7f9293f71740 1 -- start start 2024-05-27T09:48:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.331+0000 7f9293f71740 1 Processor -- start 2024-05-27T09:48:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.333+0000 7f9293f71740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b5c2f0 0x55b466f319f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.334+0000 7f9293f71740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 0x55b466f31f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.334+0000 7f9293f71740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f3dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.335+0000 7f9286800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f3dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.335+0000 7f9286800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f3dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38548/0 (socket says 172.21.15.44:38548) 2024-05-27T09:48:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.335+0000 7f9286800640 1 -- 172.21.15.44:0/103002660 learned_addr learned my addr 172.21.15.44:0/103002660 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.335+0000 7f9293f71740 1 -- 172.21.15.44:0/103002660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b466b6e790 con 0x55b466b644d0 2024-05-27T09:48:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.335+0000 7f9293f71740 1 -- 172.21.15.44:0/103002660 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b466e30bc0 con 0x55b466b5c2f0 2024-05-27T09:48:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.335+0000 7f9293f71740 1 -- 172.21.15.44:0/103002660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b466f34930 con 0x55b466b851b0 2024-05-27T09:48:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.335+0000 7f9293f71740 10 client.0 _ll_register_callbacks cb 0x55b466e0e400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.336+0000 7f9285e00640 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b5c2f0 0x55b466f319f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.336+0000 7f9285400640 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 0x55b466f31f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.337+0000 7f9286800640 1 -- 172.21.15.44:0/103002660 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b5c2f0 msgr2=0x55b466f319f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.337+0000 7f9286800640 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b5c2f0 0x55b466f319f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.337+0000 7f9286800640 1 -- 172.21.15.44:0/103002660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 msgr2=0x55b466f31f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.337+0000 7f9286800640 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 0x55b466f31f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.337+0000 7f9286800640 1 -- 172.21.15.44:0/103002660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b466b9d210 con 0x55b466b851b0 2024-05-27T09:48:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.337+0000 7f9285400640 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 0x55b466f31f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:48:09.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.338+0000 7f9286800640 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f3dcf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9280100a70 tx=0x7f92800ca6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65208622600bdbcd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:09.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.338+0000 7f9267e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:48:09.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.338+0000 7f9267e00640 1 -- 172.21.15.44:0/103002660 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92800ca110 con 0x55b466b851b0 2024-05-27T09:48:09.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.338+0000 7f9267e00640 1 -- 172.21.15.44:0/103002660 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8651+0+0 (secure 0 0 0) 0x7f9280100dd0 con 0x55b466b851b0 2024-05-27T09:48:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.340+0000 7f9293f71740 1 -- 172.21.15.44:0/103002660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9274015290 con 0x55b466b851b0 2024-05-27T09:48:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.340+0000 7f9267e00640 1 -- 172.21.15.44:0/103002660 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f92800ce4c0 con 0x55b466b851b0 2024-05-27T09:48:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.340+0000 7f9293f71740 10 client.27712 fetch_fsmap learned FSMap version 42 2024-05-27T09:48:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.340+0000 7f9293f71740 1 -- 172.21.15.44:0/103002660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55b466c69620 con 0x55b466b851b0 2024-05-27T09:48:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.341+0000 7f9267e00640 1 -- 172.21.15.44:0/103002660 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9274015290 con 0x55b466b851b0 2024-05-27T09:48:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.342+0000 7f9267e00640 1 -- 172.21.15.44:0/103002660 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f92800c5a80 con 0x55b466b851b0 2024-05-27T09:48:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.342+0000 7f9293f71740 10 client.27712 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:48:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.342+0000 7f9293f71740 10 client.27712 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.342+0000 7f9293f71740 20 client.27712 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.342+0000 7f9293f71740 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b466b43d80 0x55b466f43da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.342+0000 7f9293f71740 4 client.27712 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.342+0000 7f9293f71740 1 -- 172.21.15.44:0/103002660 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b466e9aae0 con 0x55b466b43d80 2024-05-27T09:48:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.343+0000 7f9285e00640 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b466b43d80 0x55b466f43da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.344+0000 7f9285e00640 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b466b43d80 0x55b466f43da0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4df43e6b048c227d server_cookie=a26bc11e16fae077 in_seq=0 out_seq=0 2024-05-27T09:48:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.344+0000 7f9267e00640 10 client.27712 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.345+0000 7f9267e00640 1 -- 172.21.15.44:0/103002660 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b466e9aae0 con 0x55b466b43d80 2024-05-27T09:48:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.345+0000 7f9267e00640 10 client.27712 handle_command_reply: tid=0 2024-05-27T09:48:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.346+0000 7f9293f71740 1 client.27712 shutdown 2024-05-27T09:48:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.347+0000 7f9293f71740 1 -- 172.21.15.44:0/103002660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 msgr2=0x55b466f3dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.347+0000 7f9293f71740 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f3dcf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9280100a70 tx=0x7f92800ca6b0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.347+0000 7f9293f71740 1 -- 172.21.15.44:0/103002660 shutdown_connections 2024-05-27T09:48:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.347+0000 7f9293f71740 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b466b43d80 0x55b466f43da0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.347+0000 7f9293f71740 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b851b0 0x55b466f3dcf0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.347+0000 7f9293f71740 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 0x55b466f31f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.347+0000 7f9293f71740 1 --2- 172.21.15.44:0/103002660 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b5c2f0 0x55b466f319f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.347+0000 7f9293f71740 1 -- 172.21.15.44:0/103002660 >> 172.21.15.44:0/103002660 conn(0x55b466b79300 msgr2=0x55b466f27690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.347+0000 7f9267e00640 0 client.27712 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.348+0000 7f9293f71740 1 -- 172.21.15.44:0/103002660 shutdown_connections 2024-05-27T09:48:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.348+0000 7f9293f71740 1 -- 172.21.15.44:0/103002660 wait complete. 2024-05-27T09:48:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.348+0000 7f9293f71740 20 client.27712 trim_cache size 0 max 16384 2024-05-27T09:48:09.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.379+0000 7f9293f71740 1 Processor -- start 2024-05-27T09:48:09.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.380+0000 7f9293f71740 1 -- start start 2024-05-27T09:48:09.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.380+0000 7f9293f71740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b5c2f0 0x55b466f34120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.380+0000 7f9293f71740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b644d0 0x55b466f3da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.380+0000 7f9293f71740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b851b0 0x55b466f3dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.381+0000 7f9293f71740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9268011040 con 0x55b466b5c2f0 2024-05-27T09:48:09.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.381+0000 7f9293f71740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f926800e8e0 con 0x55b466b851b0 2024-05-27T09:48:09.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.381+0000 7f9293f71740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f926800c750 con 0x55b466b644d0 2024-05-27T09:48:09.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.381+0000 7f9285400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b644d0 0x55b466f3da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.382+0000 7f9285400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b644d0 0x55b466f3da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38560/0 (socket says 172.21.15.44:38560) 2024-05-27T09:48:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.382+0000 7f9285400640 1 -- 172.21.15.44:0/701160946 learned_addr learned my addr 172.21.15.44:0/701160946 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.382+0000 7f9285e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b5c2f0 0x55b466f34120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.382+0000 7f9286800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b851b0 0x55b466f3dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.382+0000 7f9285400640 1 -- 172.21.15.44:0/701160946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b851b0 msgr2=0x55b466f3dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.382+0000 7f9285400640 1 --2- 172.21.15.44:0/701160946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b851b0 0x55b466f3dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.382+0000 7f9285400640 1 -- 172.21.15.44:0/701160946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b5c2f0 msgr2=0x55b466f34120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.382+0000 7f9285400640 1 --2- 172.21.15.44:0/701160946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b5c2f0 0x55b466f34120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.382+0000 7f9285400640 1 -- 172.21.15.44:0/701160946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b466f28c40 con 0x55b466b644d0 2024-05-27T09:48:09.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.383+0000 7f9285400640 1 --2- 172.21.15.44:0/701160946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b644d0 0x55b466f3da60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9274005890 tx=0x7f9274009cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f730971f51649d4c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:09.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.383+0000 7f9272a00640 1 -- 172.21.15.44:0/701160946 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9274023020 con 0x55b466b644d0 2024-05-27T09:48:09.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.383+0000 7f9272a00640 1 -- 172.21.15.44:0/701160946 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92740054f0 con 0x55b466b644d0 2024-05-27T09:48:09.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.383+0000 7f9272a00640 1 -- 172.21.15.44:0/701160946 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927400ec40 con 0x55b466b644d0 2024-05-27T09:48:09.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.385+0000 7f9293f71740 1 -- 172.21.15.44:0/701160946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b644d0 msgr2=0x55b466f3da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:09.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.385+0000 7f9293f71740 1 --2- 172.21.15.44:0/701160946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b644d0 0x55b466f3da60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9274005890 tx=0x7f9274009cf0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.385+0000 7f9293f71740 1 -- 172.21.15.44:0/701160946 shutdown_connections 2024-05-27T09:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.385+0000 7f9293f71740 1 --2- 172.21.15.44:0/701160946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b851b0 0x55b466f3dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.385+0000 7f9293f71740 1 --2- 172.21.15.44:0/701160946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b644d0 0x55b466f3da60 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.385+0000 7f9293f71740 1 --2- 172.21.15.44:0/701160946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b5c2f0 0x55b466f34120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.385+0000 7f9293f71740 1 -- 172.21.15.44:0/701160946 >> 172.21.15.44:0/701160946 conn(0x55b466b79300 msgr2=0x55b466f31cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.385+0000 7f9293f71740 1 -- 172.21.15.44:0/701160946 shutdown_connections 2024-05-27T09:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.385+0000 7f9293f71740 1 -- 172.21.15.44:0/701160946 wait complete. 2024-05-27T09:48:09.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.386+0000 7f9293f71740 1 -- start start 2024-05-27T09:48:09.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.386+0000 7f9293f71740 1 Processor -- start 2024-05-27T09:48:09.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.387+0000 7f9293f71740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b5c2f0 0x55b466f39bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.387+0000 7f9293f71740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 0x55b466f3a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.387+0000 7f9293f71740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b851b0 0x55b466f370f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.387+0000 7f9293f71740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f926800cda0 con 0x55b466b644d0 2024-05-27T09:48:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.387+0000 7f9293f71740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f926800e210 con 0x55b466b851b0 2024-05-27T09:48:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.387+0000 7f9293f71740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f926800d470 con 0x55b466b5c2f0 2024-05-27T09:48:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.387+0000 7f9293f71740 10 client.0 _ll_register_callbacks cb 0x7f9280009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.388+0000 7f9285e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b5c2f0 0x55b466f39bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.388+0000 7f9285e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b5c2f0 0x55b466f39bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38572/0 (socket says 172.21.15.44:38572) 2024-05-27T09:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.388+0000 7f9285e00640 1 -- 172.21.15.44:0/4202305681 learned_addr learned my addr 172.21.15.44:0/4202305681 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.388+0000 7f9286800640 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b851b0 0x55b466f370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.388+0000 7f9285400640 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 0x55b466f3a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.389+0000 7f9285e00640 1 -- 172.21.15.44:0/4202305681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b851b0 msgr2=0x55b466f370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.389+0000 7f9285e00640 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b851b0 0x55b466f370f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.389+0000 7f9285e00640 1 -- 172.21.15.44:0/4202305681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 msgr2=0x55b466f3a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.389+0000 7f9285e00640 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 0x55b466f3a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.389+0000 7f9285e00640 1 -- 172.21.15.44:0/4202305681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b466f37760 con 0x55b466b5c2f0 2024-05-27T09:48:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.389+0000 7f9285400640 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 0x55b466f3a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:48:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.390+0000 7f9285e00640 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b5c2f0 0x55b466f39bf0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f927800d5c0 tx=0x7f927800f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18f45c2441c9ab4d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.390+0000 7f9267e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:48:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.390+0000 7f9267e00640 1 -- 172.21.15.44:0/4202305681 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278034060 con 0x55b466b5c2f0 2024-05-27T09:48:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.390+0000 7f9267e00640 1 -- 172.21.15.44:0/4202305681 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8651+0+0 (secure 0 0 0) 0x7f9278018820 con 0x55b466b5c2f0 2024-05-27T09:48:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.390+0000 7f9293f71740 1 -- 172.21.15.44:0/4202305681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92740054f0 con 0x55b466b5c2f0 2024-05-27T09:48:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.391+0000 7f9267e00640 1 -- 172.21.15.44:0/4202305681 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f9278018a20 con 0x55b466b5c2f0 2024-05-27T09:48:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.391+0000 7f9293f71740 10 client.27718 fetch_fsmap learned FSMap version 42 2024-05-27T09:48:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.391+0000 7f9293f71740 1 -- 172.21.15.44:0/4202305681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55b466fabe50 con 0x55b466b5c2f0 2024-05-27T09:48:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.391+0000 7f9267e00640 1 -- 172.21.15.44:0/4202305681 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92740054f0 con 0x55b466b5c2f0 2024-05-27T09:48:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.392+0000 7f9267e00640 1 -- 172.21.15.44:0/4202305681 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f9278024420 con 0x55b466b5c2f0 2024-05-27T09:48:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.393+0000 7f9293f71740 10 client.27718 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T09:48:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.393+0000 7f9293f71740 10 client.27718 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.393+0000 7f9293f71740 20 client.27718 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.393+0000 7f9293f71740 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b466b191c0 0x55b466fae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:09.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.393+0000 7f9293f71740 4 client.27718 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.393+0000 7f9293f71740 1 -- 172.21.15.44:0/4202305681 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b466e942c0 con 0x55b466b191c0 2024-05-27T09:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.393+0000 7f9285400640 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b466b191c0 0x55b466fae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.395+0000 7f9285400640 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b466b191c0 0x55b466fae5c0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13f720fdbb51d98e server_cookie=2959eb33775a051a in_seq=0 out_seq=0 2024-05-27T09:48:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:09.395+0000 7f9267e00640 10 client.27718 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:09.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:09 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:48:09.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:09 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:48:09.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:09 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:48:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9267e00640 1 -- 172.21.15.44:0/4202305681 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b466e942c0 con 0x55b466b191c0 2024-05-27T09:48:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9267e00640 10 client.27718 handle_command_reply: tid=0 2024-05-27T09:48:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9293f71740 1 client.27718 shutdown 2024-05-27T09:48:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9293f71740 1 -- 172.21.15.44:0/4202305681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b5c2f0 msgr2=0x55b466f39bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9293f71740 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b5c2f0 0x55b466f39bf0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f927800d5c0 tx=0x7f927800f040 comp rx=0 tx=0).stop 2024-05-27T09:48:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9293f71740 1 -- 172.21.15.44:0/4202305681 shutdown_connections 2024-05-27T09:48:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9293f71740 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b466b191c0 0x55b466fae5c0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9293f71740 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b466b851b0 0x55b466f370f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9293f71740 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b466b644d0 0x55b466f3a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9293f71740 1 --2- 172.21.15.44:0/4202305681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b466b5c2f0 0x55b466f39bf0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9293f71740 1 -- 172.21.15.44:0/4202305681 >> 172.21.15.44:0/4202305681 conn(0x55b466b79300 msgr2=0x55b466f31190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.047+0000 7f9267e00640 0 client.27718 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.048+0000 7f9293f71740 1 -- 172.21.15.44:0/4202305681 shutdown_connections 2024-05-27T09:48:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.048+0000 7f9293f71740 1 -- 172.21.15.44:0/4202305681 wait complete. 2024-05-27T09:48:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.048+0000 7f9293f71740 20 client.27718 trim_cache size 0 max 16384 2024-05-27T09:48:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:48:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:48:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:48:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.050+0000 7f928ce00640 1 -- 172.21.15.44:0/3113945710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f926c0b2c10 msgr2=0x7f926c0b5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.050+0000 7f928ce00640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f926c0b2c10 0x7f926c0b5120 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55b466f28c10 tx=0x7f927800d670 comp rx=0 tx=0).stop 2024-05-27T09:48:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.050+0000 7f928ce00640 1 -- 172.21.15.44:0/3113945710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9288112900 msgr2=0x7f92881ade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.050+0000 7f928ce00640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9288112900 0x7f92881ade50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9280003c10 tx=0x7f9280003c40 comp rx=0 tx=0).stop 2024-05-27T09:48:10.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.051+0000 7f928ce00640 1 -- 172.21.15.44:0/3113945710 shutdown_connections 2024-05-27T09:48:10.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.051+0000 7f928ce00640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9288112900 0x7f92881ade50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:10.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.051+0000 7f928ce00640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f926c0b2c10 0x7f926c0b5120 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:10.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.051+0000 7f928ce00640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9288111e60 0x7f92881a9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:10.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.051+0000 7f928ce00640 1 --2- 172.21.15.44:0/3113945710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f928810a7c0 0x7f92881a9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:10.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.051+0000 7f928ce00640 1 -- 172.21.15.44:0/3113945710 >> 172.21.15.44:0/3113945710 conn(0x7f928806eda0 msgr2=0x7f9288073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.053+0000 7f928ce00640 1 -- 172.21.15.44:0/3113945710 shutdown_connections 2024-05-27T09:48:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:10.053+0000 7f928ce00640 1 -- 172.21.15.44:0/3113945710 wait complete. 2024-05-27T09:48:10.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:10 smithi044 ceph-mon[31618]: pgmap v406: 97 pgs: 97 active+clean; 316 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 655 KiB/s wr, 1.33k op/s 2024-05-27T09:48:10.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:10 smithi050 ceph-mon[36371]: pgmap v406: 97 pgs: 97 active+clean; 316 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 655 KiB/s wr, 1.33k op/s 2024-05-27T09:48:10.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:10 smithi189 ceph-mon[36671]: pgmap v406: 97 pgs: 97 active+clean; 316 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 655 KiB/s wr, 1.33k op/s 2024-05-27T09:48:11.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:11 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:48:11.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:11 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:48:11.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:11 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:48:11.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:11 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:48:11.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:11 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:48:11.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:11 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:48:11.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:11 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:48:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:11 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:48:11.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:11 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:48:12.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:12 smithi044 ceph-mon[31618]: pgmap v407: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 1.54k op/s 2024-05-27T09:48:12.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:12 smithi050 ceph-mon[36371]: pgmap v407: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 1.54k op/s 2024-05-27T09:48:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:12 smithi189 ceph-mon[36671]: pgmap v407: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 1.54k op/s 2024-05-27T09:48:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:13 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:48:13.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:13 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:48:13.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:13 smithi044 ceph-mon[31618]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T09:48:13.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:13 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:48:13.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:13 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:48:13.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:13 smithi050 ceph-mon[36371]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T09:48:13.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:13 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:48:13.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:13 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:48:13.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:13 smithi189 ceph-mon[36671]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T09:48:14.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:14 smithi044 ceph-mon[31618]: pgmap v408: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 1.54k op/s 2024-05-27T09:48:14.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:14 smithi050 ceph-mon[36371]: pgmap v408: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 1.54k op/s 2024-05-27T09:48:14.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:14 smithi189 ceph-mon[36671]: pgmap v408: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 1.54k op/s 2024-05-27T09:48:15.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:15 smithi044 ceph-mon[31618]: pgmap v410: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 334 op/s 2024-05-27T09:48:15.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:15 smithi050 ceph-mon[36371]: pgmap v410: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 334 op/s 2024-05-27T09:48:15.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:15 smithi189 ceph-mon[36671]: pgmap v410: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 334 op/s 2024-05-27T09:48:18.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:17 smithi044 ceph-mon[31618]: pgmap v411: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.3 MiB/s wr, 735 op/s 2024-05-27T09:48:18.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:17 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:48:18.290 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:17 smithi189 ceph-mon[36671]: pgmap v411: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.3 MiB/s wr, 735 op/s 2024-05-27T09:48:18.290 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:17 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:48:18.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:17 smithi050 ceph-mon[36371]: pgmap v411: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.3 MiB/s wr, 735 op/s 2024-05-27T09:48:18.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:17 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:48:20.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:19 smithi044 ceph-mon[31618]: pgmap v412: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 710 op/s 2024-05-27T09:48:20.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:19 smithi050 ceph-mon[36371]: pgmap v412: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 710 op/s 2024-05-27T09:48:20.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:19 smithi189 ceph-mon[36671]: pgmap v412: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 710 op/s 2024-05-27T09:48:22.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:21 smithi044 ceph-mon[31618]: pgmap v413: 97 pgs: 97 active+clean; 315 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 715 op/s 2024-05-27T09:48:22.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:21 smithi050 ceph-mon[36371]: pgmap v413: 97 pgs: 97 active+clean; 315 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 715 op/s 2024-05-27T09:48:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:21 smithi189 ceph-mon[36671]: pgmap v413: 97 pgs: 97 active+clean; 315 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 715 op/s 2024-05-27T09:48:23.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.303+0000 7f8397600640 1 -- 172.21.15.44:0/297113231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f839010dc40 msgr2=0x7f8390110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:23.305 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.303+0000 7f8397600640 1 --2- 172.21.15.44:0/297113231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f839010dc40 0x7f8390110060 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8384009900 tx=0x7f838402a3f0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.304+0000 7f8397600640 1 -- 172.21.15.44:0/297113231 shutdown_connections 2024-05-27T09:48:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.304+0000 7f8397600640 1 --2- 172.21.15.44:0/297113231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83901105a0 0x7f83901129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.304+0000 7f8397600640 1 --2- 172.21.15.44:0/297113231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f839010dc40 0x7f8390110060 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.304+0000 7f8397600640 1 --2- 172.21.15.44:0/297113231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8390102a00 0x7f8390102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.304+0000 7f8397600640 1 -- 172.21.15.44:0/297113231 >> 172.21.15.44:0/297113231 conn(0x7f83900782d0 msgr2=0x7f8390078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.307+0000 7f8397600640 1 -- 172.21.15.44:0/297113231 shutdown_connections 2024-05-27T09:48:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.307+0000 7f8397600640 1 -- 172.21.15.44:0/297113231 wait complete. 2024-05-27T09:48:23.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.309+0000 7f8397600640 1 Processor -- start 2024-05-27T09:48:23.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.309+0000 7f8397600640 1 -- start start 2024-05-27T09:48:23.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.309+0000 7f8397600640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8390102a00 0x7f839010c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.310+0000 7f8397600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839010dc40 0x7f839010c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.310+0000 7f8397600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83901105a0 0x7f83901a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.310+0000 7f8397600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f83901055f0 con 0x7f83901105a0 2024-05-27T09:48:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.310+0000 7f8397600640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8390105470 con 0x7f8390102a00 2024-05-27T09:48:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.310+0000 7f8397600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8390105770 con 0x7f839010dc40 2024-05-27T09:48:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.310+0000 7f838fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839010dc40 0x7f839010c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.310+0000 7f838fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839010dc40 0x7f839010c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:46258/0 (socket says 172.21.15.44:46258) 2024-05-27T09:48:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.310+0000 7f838fe00640 1 -- 172.21.15.44:0/2848645209 learned_addr learned my addr 172.21.15.44:0/2848645209 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.310+0000 7f8394c00640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8390102a00 0x7f839010c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.311+0000 7f838fe00640 1 -- 172.21.15.44:0/2848645209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8390102a00 msgr2=0x7f839010c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.311+0000 7f838fe00640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8390102a00 0x7f839010c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.311+0000 7f838fe00640 1 -- 172.21.15.44:0/2848645209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83901105a0 msgr2=0x7f83901a6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.311+0000 7f838fe00640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83901105a0 0x7f83901a6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.311+0000 7f838fe00640 1 -- 172.21.15.44:0/2848645209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83901a6e50 con 0x7f839010dc40 2024-05-27T09:48:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.311+0000 7f838fe00640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839010dc40 0x7f839010c8b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8384005ab0 tx=0x7f8384003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.313+0000 7f838d600640 1 -- 172.21.15.44:0/2848645209 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838403f070 con 0x7f839010dc40 2024-05-27T09:48:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.313+0000 7f838d600640 1 -- 172.21.15.44:0/2848645209 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8384031b50 con 0x7f839010dc40 2024-05-27T09:48:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.313+0000 7f838d600640 1 -- 172.21.15.44:0/2848645209 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8384035030 con 0x7f839010dc40 2024-05-27T09:48:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.313+0000 7f8397600640 1 -- 172.21.15.44:0/2848645209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83901a70e0 con 0x7f839010dc40 2024-05-27T09:48:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.314+0000 7f8397600640 1 -- 172.21.15.44:0/2848645209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83901a7580 con 0x7f839010dc40 2024-05-27T09:48:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.319+0000 7f839e58f740 1 Processor -- start 2024-05-27T09:48:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.319+0000 7f839e58f740 1 -- start start 2024-05-27T09:48:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.320+0000 7f839e58f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176821b0 0x55dd17a24860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.320+0000 7f839e58f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a24da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.320+0000 7f839e58f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a252e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.320+0000 7f839e58f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55dd17995b90 con 0x55dd176614d0 2024-05-27T09:48:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.320+0000 7f839e58f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55dd17985730 con 0x55dd176821b0 2024-05-27T09:48:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.320+0000 7f839e58f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dd17989770 con 0x55dd176592f0 2024-05-27T09:48:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.321+0000 7f8394c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a24da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.321+0000 7f8394c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a24da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:46274/0 (socket says 172.21.15.44:46274) 2024-05-27T09:48:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.321+0000 7f8394c00640 1 -- 172.21.15.44:0/2044236725 learned_addr learned my addr 172.21.15.44:0/2044236725 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.321+0000 7f8395600640 1 --2- 172.21.15.44:0/2044236725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176821b0 0x55dd17a24860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.321+0000 7f8394c00640 1 -- 172.21.15.44:0/2044236725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176821b0 msgr2=0x55dd17a24860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.321+0000 7f8394c00640 1 --2- 172.21.15.44:0/2044236725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176821b0 0x55dd17a24860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.321+0000 7f8394c00640 1 -- 172.21.15.44:0/2044236725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 msgr2=0x55dd17a252e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:48:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.321+0000 7f8394c00640 1 --2- 172.21.15.44:0/2044236725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a252e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.321+0000 7f8394c00640 1 -- 172.21.15.44:0/2044236725 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd1791e8b0 con 0x55dd176592f0 2024-05-27T09:48:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.322+0000 7f8394c00640 1 --2- 172.21.15.44:0/2044236725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a24da0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f837c00bcb0 tx=0x7f837c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9e2ca60c920eee7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.324+0000 7f8381c00640 1 -- 172.21.15.44:0/2044236725 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c01c480 con 0x55dd176592f0 2024-05-27T09:48:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.324+0000 7f8381c00640 1 -- 172.21.15.44:0/2044236725 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f837c01cad0 con 0x55dd176592f0 2024-05-27T09:48:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.324+0000 7f8381c00640 1 -- 172.21.15.44:0/2044236725 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c011cd0 con 0x55dd176592f0 2024-05-27T09:48:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.325+0000 7f838d600640 1 -- 172.21.15.44:0/2848645209 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f8384042070 con 0x7f839010dc40 2024-05-27T09:48:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.326+0000 7f839e58f740 1 -- 172.21.15.44:0/2044236725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 msgr2=0x55dd17a24da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.326+0000 7f839e58f740 1 --2- 172.21.15.44:0/2044236725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a24da0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f837c00bcb0 tx=0x7f837c0107a0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.328+0000 7f838d600640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f83780b2b20 0x7f83780b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.329+0000 7f838d600640 1 -- 172.21.15.44:0/2848645209 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f8384102cb0 con 0x7f839010dc40 2024-05-27T09:48:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.329+0000 7f839e58f740 1 -- 172.21.15.44:0/2044236725 shutdown_connections 2024-05-27T09:48:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.329+0000 7f839e58f740 1 --2- 172.21.15.44:0/2044236725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a252e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.329+0000 7f839e58f740 1 --2- 172.21.15.44:0/2044236725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a24da0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.329+0000 7f839e58f740 1 --2- 172.21.15.44:0/2044236725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176821b0 0x55dd17a24860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.329+0000 7f839e58f740 1 -- 172.21.15.44:0/2044236725 >> 172.21.15.44:0/2044236725 conn(0x55dd17676300 msgr2=0x55dd17a21010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.330+0000 7f8395600640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f83780b2b20 0x7f83780b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.331+0000 7f839e58f740 1 -- 172.21.15.44:0/2044236725 shutdown_connections 2024-05-27T09:48:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.331+0000 7f839e58f740 1 -- 172.21.15.44:0/2044236725 wait complete. 2024-05-27T09:48:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.333+0000 7f8395600640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f83780b2b20 0x7f83780b5030 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8388005c40 tx=0x7f8388005bd0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.334+0000 7f839e58f740 1 -- start start 2024-05-27T09:48:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.334+0000 7f839e58f740 1 Processor -- start 2024-05-27T09:48:23.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.335+0000 7f839e58f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a2ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.335+0000 7f839e58f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a2f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.335+0000 7f839e58f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176821b0 0x55dd17a3acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.335+0000 7f839e58f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55dd1766b790 con 0x55dd176614d0 2024-05-27T09:48:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.335+0000 7f839e58f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55dd1792dbc0 con 0x55dd176821b0 2024-05-27T09:48:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.335+0000 7f839e58f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dd17a31990 con 0x55dd176592f0 2024-05-27T09:48:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.335+0000 7f839e58f740 10 client.0 _ll_register_callbacks cb 0x55dd1790b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.336+0000 7f8394c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a2ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.336+0000 7f8394c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a2ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:46280/0 (socket says 172.21.15.44:46280) 2024-05-27T09:48:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.336+0000 7f8395600640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176821b0 0x55dd17a3acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.336+0000 7f8394c00640 1 -- 172.21.15.44:0/4181810548 learned_addr learned my addr 172.21.15.44:0/4181810548 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:23.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.336+0000 7f838fe00640 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a2f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.337+0000 7f8395600640 1 -- 172.21.15.44:0/4181810548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 msgr2=0x55dd17a2ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:23.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.337+0000 7f8395600640 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a2ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.338+0000 7f8395600640 1 -- 172.21.15.44:0/4181810548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 msgr2=0x55dd17a2f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:23.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.338+0000 7f8395600640 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a2f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.338+0000 7f8395600640 1 -- 172.21.15.44:0/4181810548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd1769a210 con 0x55dd176821b0 2024-05-27T09:48:23.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.338+0000 7f838fe00640 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a2f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:23.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.338+0000 7f8394c00640 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a2ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:23.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.340+0000 7f8395600640 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176821b0 0x55dd17a3acc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8388009fd0 tx=0x7f8388004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=570f4dbe7f365477 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:23.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.340+0000 7f8376000640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:23.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.341+0000 7f8376000640 1 -- 172.21.15.44:0/4181810548 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838800ed20 con 0x55dd176821b0 2024-05-27T09:48:23.342 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.341+0000 7f8376000640 1 -- 172.21.15.44:0/4181810548 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f838801e420 con 0x55dd176821b0 2024-05-27T09:48:23.342 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.341+0000 7f839e58f740 1 -- 172.21.15.44:0/4181810548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f837c01cad0 con 0x55dd176821b0 2024-05-27T09:48:23.342 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.342+0000 7f8376000640 1 -- 172.21.15.44:0/4181810548 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f838801e660 con 0x55dd176821b0 2024-05-27T09:48:23.342 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.342+0000 7f839e58f740 10 client.27725 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.342+0000 7f839e58f740 1 -- 172.21.15.44:0/4181810548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55dd17766620 con 0x55dd176821b0 2024-05-27T09:48:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.342+0000 7f8376000640 1 -- 172.21.15.44:0/4181810548 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f837c01cad0 con 0x55dd176821b0 2024-05-27T09:48:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.343+0000 7f8376000640 1 -- 172.21.15.44:0/4181810548 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f8388030880 con 0x55dd176821b0 2024-05-27T09:48:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.343+0000 7f839e58f740 10 client.27725 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.343+0000 7f839e58f740 10 client.27725 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.343+0000 7f839e58f740 20 client.27725 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.343+0000 7f839e58f740 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dd17640d80 0x55dd17a40d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.343+0000 7f839e58f740 4 client.27725 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.343+0000 7f839e58f740 1 -- 172.21.15.44:0/4181810548 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd17997ae0 con 0x55dd17640d80 2024-05-27T09:48:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.343+0000 7f8394c00640 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dd17640d80 0x55dd17a40d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.344+0000 7f8376000640 10 client.27725 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.344+0000 7f8394c00640 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dd17640d80 0x55dd17a40d70 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f836a9193858be9d server_cookie=13e04cd106257d74 in_seq=0 out_seq=0 2024-05-27T09:48:23.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.347+0000 7f8376000640 1 -- 172.21.15.44:0/4181810548 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dd17997ae0 con 0x55dd17640d80 2024-05-27T09:48:23.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.347+0000 7f8376000640 10 client.27725 handle_command_reply: tid=0 2024-05-27T09:48:23.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.347+0000 7f839e58f740 1 client.27725 shutdown 2024-05-27T09:48:23.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.348+0000 7f839e58f740 1 -- 172.21.15.44:0/4181810548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176821b0 msgr2=0x55dd17a3acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:23.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.348+0000 7f839e58f740 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176821b0 0x55dd17a3acc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8388009fd0 tx=0x7f8388004a40 comp rx=0 tx=0).stop 2024-05-27T09:48:23.348 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.348+0000 7f839e58f740 1 -- 172.21.15.44:0/4181810548 shutdown_connections 2024-05-27T09:48:23.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.348+0000 7f839e58f740 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dd17640d80 0x55dd17a40d70 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.348+0000 7f839e58f740 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176821b0 0x55dd17a3acc0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.348+0000 7f839e58f740 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a2f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.348+0000 7f8376000640 0 client.27725 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:23.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.348+0000 7f839e58f740 1 --2- 172.21.15.44:0/4181810548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a2ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.348+0000 7f839e58f740 1 -- 172.21.15.44:0/4181810548 >> 172.21.15.44:0/4181810548 conn(0x55dd17676300 msgr2=0x55dd17a26580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:23.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.349+0000 7f839e58f740 1 -- 172.21.15.44:0/4181810548 shutdown_connections 2024-05-27T09:48:23.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.350+0000 7f839e58f740 1 -- 172.21.15.44:0/4181810548 wait complete. 2024-05-27T09:48:23.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.350+0000 7f839e58f740 20 client.27725 trim_cache size 0 max 16384 2024-05-27T09:48:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.395+0000 7f839e58f740 1 Processor -- start 2024-05-27T09:48:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.395+0000 7f839e58f740 1 -- start start 2024-05-27T09:48:23.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.396+0000 7f839e58f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176592f0 0x55dd17a3aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.396+0000 7f839e58f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a3af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.396+0000 7f839e58f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176821b0 0x55dd17a285c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.396+0000 7f839e58f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f83901c1490 con 0x55dd176614d0 2024-05-27T09:48:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.396+0000 7f839e58f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f83901bed00 con 0x55dd176592f0 2024-05-27T09:48:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.396+0000 7f839e58f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f839010d6a0 con 0x55dd176821b0 2024-05-27T09:48:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.396+0000 7f8394c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176592f0 0x55dd17a3aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.397+0000 7f8394c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176592f0 0x55dd17a3aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:49150/0 (socket says 172.21.15.44:49150) 2024-05-27T09:48:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.397+0000 7f8394c00640 1 -- 172.21.15.44:0/3082252978 learned_addr learned my addr 172.21.15.44:0/3082252978 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.397+0000 7f8395600640 1 --2- 172.21.15.44:0/3082252978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176821b0 0x55dd17a285c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.397+0000 7f8394c00640 1 -- 172.21.15.44:0/3082252978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176821b0 msgr2=0x55dd17a285c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.397+0000 7f8394c00640 1 --2- 172.21.15.44:0/3082252978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176821b0 0x55dd17a285c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.397+0000 7f8394c00640 1 -- 172.21.15.44:0/3082252978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 msgr2=0x55dd17a3af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.398+0000 7f838fe00640 1 --2- 172.21.15.44:0/3082252978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a3af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.398+0000 7f8394c00640 1 --2- 172.21.15.44:0/3082252978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a3af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.398+0000 7f8394c00640 1 -- 172.21.15.44:0/3082252978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd17a21c00 con 0x55dd176592f0 2024-05-27T09:48:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.399+0000 7f8395600640 1 --2- 172.21.15.44:0/3082252978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176821b0 0x55dd17a285c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:48:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.399+0000 7f838fe00640 1 --2- 172.21.15.44:0/3082252978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a3af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:48:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.400+0000 7f8394c00640 1 --2- 172.21.15.44:0/3082252978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176592f0 0x55dd17a3aa30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x55dd17a22350 tx=0x7f837c011770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d69c28c7f21781e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.402+0000 7f8381200640 1 -- 172.21.15.44:0/3082252978 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c011890 con 0x55dd176592f0 2024-05-27T09:48:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.402+0000 7f8381200640 1 -- 172.21.15.44:0/3082252978 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f837c005040 con 0x55dd176592f0 2024-05-27T09:48:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.402+0000 7f8381200640 1 -- 172.21.15.44:0/3082252978 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c01c6e0 con 0x55dd176592f0 2024-05-27T09:48:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.405+0000 7f839e58f740 1 -- 172.21.15.44:0/3082252978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176592f0 msgr2=0x55dd17a3aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.405+0000 7f839e58f740 1 --2- 172.21.15.44:0/3082252978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176592f0 0x55dd17a3aa30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x55dd17a22350 tx=0x7f837c011770 comp rx=0 tx=0).stop 2024-05-27T09:48:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.406+0000 7f839e58f740 1 -- 172.21.15.44:0/3082252978 shutdown_connections 2024-05-27T09:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.406+0000 7f839e58f740 1 --2- 172.21.15.44:0/3082252978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176821b0 0x55dd17a285c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.406+0000 7f839e58f740 1 --2- 172.21.15.44:0/3082252978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176614d0 0x55dd17a3af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.406+0000 7f839e58f740 1 --2- 172.21.15.44:0/3082252978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176592f0 0x55dd17a3aa30 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.406+0000 7f839e58f740 1 -- 172.21.15.44:0/3082252978 >> 172.21.15.44:0/3082252978 conn(0x55dd17676300 msgr2=0x55dd17a30c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.407+0000 7f839e58f740 1 -- 172.21.15.44:0/3082252978 shutdown_connections 2024-05-27T09:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.407+0000 7f839e58f740 1 -- 172.21.15.44:0/3082252978 wait complete. 2024-05-27T09:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.409+0000 7f839e58f740 1 -- start start 2024-05-27T09:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.409+0000 7f839e58f740 1 Processor -- start 2024-05-27T09:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.410+0000 7f839e58f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a31be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.411+0000 7f839e58f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176614d0 0x55dd17a32120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.411+0000 7f8394c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a31be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.411+0000 7f8394c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a31be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:46308/0 (socket says 172.21.15.44:46308) 2024-05-27T09:48:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.411+0000 7f8394c00640 1 -- 172.21.15.44:0/465213940 learned_addr learned my addr 172.21.15.44:0/465213940 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.412+0000 7f839e58f740 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176821b0 0x55dd17a32660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.412+0000 7f839e58f740 1 -- 172.21.15.44:0/465213940 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f839011c2d0 con 0x55dd176821b0 2024-05-27T09:48:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.412+0000 7f839e58f740 1 -- 172.21.15.44:0/465213940 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f839011d740 con 0x55dd176614d0 2024-05-27T09:48:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.412+0000 7f839e58f740 1 -- 172.21.15.44:0/465213940 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f839011c9a0 con 0x55dd176592f0 2024-05-27T09:48:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.412+0000 7f839e58f740 10 client.0 _ll_register_callbacks cb 0x7f8388009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.413+0000 7f8394c00640 1 -- 172.21.15.44:0/465213940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176614d0 msgr2=0x55dd17a32120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.413+0000 7f8394c00640 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176614d0 0x55dd17a32120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.413+0000 7f8394c00640 1 -- 172.21.15.44:0/465213940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176821b0 msgr2=0x55dd17a32660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.414+0000 7f8395600640 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176821b0 0x55dd17a32660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.414+0000 7f8394c00640 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176821b0 0x55dd17a32660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.414+0000 7f8394c00640 1 -- 172.21.15.44:0/465213940 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd17aa7430 con 0x55dd176592f0 2024-05-27T09:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.414+0000 7f8395600640 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176821b0 0x55dd17a32660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:48:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.415+0000 7f8394c00640 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a31be0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f837c00bcb0 tx=0x7f837c00bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c097a02342c98928 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.415+0000 7f8376000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:48:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.415+0000 7f8376000640 1 -- 172.21.15.44:0/465213940 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c0054b0 con 0x55dd176592f0 2024-05-27T09:48:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.416+0000 7f8376000640 1 -- 172.21.15.44:0/465213940 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f837c005650 con 0x55dd176592f0 2024-05-27T09:48:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.416+0000 7f839e58f740 1 -- 172.21.15.44:0/465213940 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f837c005040 con 0x55dd176592f0 2024-05-27T09:48:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.416+0000 7f8376000640 1 -- 172.21.15.44:0/465213940 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f837c011810 con 0x55dd176592f0 2024-05-27T09:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.416+0000 7f839e58f740 10 client.27733 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.416+0000 7f839e58f740 1 -- 172.21.15.44:0/465213940 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55dd17aa8500 con 0x55dd176592f0 2024-05-27T09:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.417+0000 7f8376000640 1 -- 172.21.15.44:0/465213940 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f837c005040 con 0x55dd176592f0 2024-05-27T09:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.417+0000 7f8376000640 1 -- 172.21.15.44:0/465213940 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f837c003680 con 0x55dd176592f0 2024-05-27T09:48:23.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.417+0000 7f839e58f740 10 client.27733 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:23.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.417+0000 7f839e58f740 10 client.27733 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:23.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.417+0000 7f839e58f740 20 client.27733 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:23.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.417+0000 7f839e58f740 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dd176161c0 0x55dd17aaa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:23.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.417+0000 7f839e58f740 4 client.27733 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:48:23.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.417+0000 7f839e58f740 1 -- 172.21.15.44:0/465213940 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd179912c0 con 0x55dd176161c0 2024-05-27T09:48:23.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.418+0000 7f838fe00640 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dd176161c0 0x55dd17aaa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:23.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.420+0000 7f838fe00640 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dd176161c0 0x55dd17aaa970 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c6ac8e9dbaf3709 server_cookie=55851eb4d732dda4 in_seq=0 out_seq=0 2024-05-27T09:48:23.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.420+0000 7f8376000640 10 client.27733 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.885+0000 7f8376000640 1 -- 172.21.15.44:0/465213940 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55dd179912c0 con 0x55dd176161c0 2024-05-27T09:48:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f8376000640 10 client.27733 handle_command_reply: tid=0 2024-05-27T09:48:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 1 client.27733 shutdown 2024-05-27T09:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 1 -- 172.21.15.44:0/465213940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 msgr2=0x55dd17a31be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a31be0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f837c00bcb0 tx=0x7f837c00bda0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 1 -- 172.21.15.44:0/465213940 shutdown_connections 2024-05-27T09:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55dd176161c0 0x55dd17aaa970 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd176821b0 0x55dd17a32660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f8376000640 0 client.27733 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd176614d0 0x55dd17a32120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 1 --2- 172.21.15.44:0/465213940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd176592f0 0x55dd17a31be0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 1 -- 172.21.15.44:0/465213940 >> 172.21.15.44:0/465213940 conn(0x55dd17676300 msgr2=0x55dd17a245f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 1 -- 172.21.15.44:0/465213940 shutdown_connections 2024-05-27T09:48:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 1 -- 172.21.15.44:0/465213940 wait complete. 2024-05-27T09:48:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.886+0000 7f839e58f740 20 client.27733 trim_cache size 0 max 16384 2024-05-27T09:48:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:48:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:48:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.887+0000 7f8397600640 1 -- 172.21.15.44:0/2848645209 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f83780b2b20 msgr2=0x7f83780b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.887+0000 7f8397600640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f83780b2b20 0x7f83780b5030 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8388005c40 tx=0x7f8388005bd0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.887+0000 7f8397600640 1 -- 172.21.15.44:0/2848645209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839010dc40 msgr2=0x7f839010c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.887+0000 7f8397600640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839010dc40 0x7f839010c8b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8384005ab0 tx=0x7f8384003b90 comp rx=0 tx=0).stop 2024-05-27T09:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.887+0000 7f8397600640 1 -- 172.21.15.44:0/2848645209 shutdown_connections 2024-05-27T09:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.887+0000 7f8397600640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f83780b2b20 0x7f83780b5030 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.887+0000 7f8397600640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83901105a0 0x7f83901a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.887+0000 7f8397600640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839010dc40 0x7f839010c8b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.887+0000 7f8397600640 1 --2- 172.21.15.44:0/2848645209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8390102a00 0x7f839010c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.887+0000 7f8397600640 1 -- 172.21.15.44:0/2848645209 >> 172.21.15.44:0/2848645209 conn(0x7f83900782d0 msgr2=0x7f83901012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.888+0000 7f8397600640 1 -- 172.21.15.44:0/2848645209 shutdown_connections 2024-05-27T09:48:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:23.888+0000 7f8397600640 1 -- 172.21.15.44:0/2848645209 wait complete. 2024-05-27T09:48:24.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:23 smithi044 ceph-mon[31618]: pgmap v414: 97 pgs: 97 active+clean; 315 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 715 op/s 2024-05-27T09:48:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:23 smithi050 ceph-mon[36371]: pgmap v414: 97 pgs: 97 active+clean; 315 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 715 op/s 2024-05-27T09:48:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:23 smithi189 ceph-mon[36671]: pgmap v414: 97 pgs: 97 active+clean; 315 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 715 op/s 2024-05-27T09:48:26.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:25 smithi044 ceph-mon[31618]: pgmap v415: 97 pgs: 97 active+clean; 315 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 602 op/s 2024-05-27T09:48:26.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:25 smithi050 ceph-mon[36371]: pgmap v415: 97 pgs: 97 active+clean; 315 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 602 op/s 2024-05-27T09:48:26.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:25 smithi189 ceph-mon[36671]: pgmap v415: 97 pgs: 97 active+clean; 315 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 602 op/s 2024-05-27T09:48:28.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:27 smithi044 ceph-mon[31618]: pgmap v416: 97 pgs: 97 active+clean; 325 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 879 op/s 2024-05-27T09:48:28.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:27 smithi050 ceph-mon[36371]: pgmap v416: 97 pgs: 97 active+clean; 325 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 879 op/s 2024-05-27T09:48:28.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:27 smithi189 ceph-mon[36671]: pgmap v416: 97 pgs: 97 active+clean; 325 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 879 op/s 2024-05-27T09:48:30.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:29 smithi044 ceph-mon[31618]: pgmap v417: 97 pgs: 97 active+clean; 325 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 510 op/s 2024-05-27T09:48:30.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:29 smithi050 ceph-mon[36371]: pgmap v417: 97 pgs: 97 active+clean; 325 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 510 op/s 2024-05-27T09:48:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:29 smithi189 ceph-mon[36671]: pgmap v417: 97 pgs: 97 active+clean; 325 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 510 op/s 2024-05-27T09:48:32.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:31 smithi044 ceph-mon[31618]: pgmap v418: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 645 op/s 2024-05-27T09:48:32.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:31 smithi050 ceph-mon[36371]: pgmap v418: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 645 op/s 2024-05-27T09:48:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:31 smithi189 ceph-mon[36671]: pgmap v418: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 645 op/s 2024-05-27T09:48:34.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:33 smithi044 ceph-mon[31618]: pgmap v419: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 418 op/s 2024-05-27T09:48:34.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:33 smithi050 ceph-mon[36371]: pgmap v419: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 418 op/s 2024-05-27T09:48:34.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:33 smithi189 ceph-mon[36671]: pgmap v419: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 418 op/s 2024-05-27T09:48:36.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:35 smithi044 ceph-mon[31618]: pgmap v420: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 418 op/s 2024-05-27T09:48:36.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:35 smithi050 ceph-mon[36371]: pgmap v420: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 418 op/s 2024-05-27T09:48:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:35 smithi189 ceph-mon[36671]: pgmap v420: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 418 op/s 2024-05-27T09:48:36.388 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:48:36.851 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:48:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.108+0000 7f5b0f000640 1 -- 172.21.15.44:0/2713219812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b081065e0 msgr2=0x7f5b081129b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.108+0000 7f5b0f000640 1 --2- 172.21.15.44:0/2713219812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b081065e0 0x7f5b081129b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009ca0 tx=0x7f5b0002a8c0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.109+0000 7f5b0f000640 1 -- 172.21.15.44:0/2713219812 shutdown_connections 2024-05-27T09:48:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.109+0000 7f5b0f000640 1 --2- 172.21.15.44:0/2713219812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b081065e0 0x7f5b081129b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.109+0000 7f5b0f000640 1 --2- 172.21.15.44:0/2713219812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b0810c330 0x7f5b08105f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.109+0000 7f5b0f000640 1 --2- 172.21.15.44:0/2713219812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0810be90 0x7f5b08105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.109+0000 7f5b0f000640 1 -- 172.21.15.44:0/2713219812 >> 172.21.15.44:0/2713219812 conn(0x7f5b08101530 msgr2=0x7f5b08103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.109+0000 7f5b0f000640 1 -- 172.21.15.44:0/2713219812 shutdown_connections 2024-05-27T09:48:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.109+0000 7f5b0f000640 1 -- 172.21.15.44:0/2713219812 wait complete. 2024-05-27T09:48:37.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.110+0000 7f5b0f000640 1 Processor -- start 2024-05-27T09:48:37.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.110+0000 7f5b0f000640 1 -- start start 2024-05-27T09:48:37.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0f000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b081065e0 0x7f5b081a2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0810be90 0x7f5b081a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0f000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b0810c330 0x7f5b0819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0f000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b081160f0 con 0x7f5b081065e0 2024-05-27T09:48:37.113 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0f000640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b08115f70 con 0x7f5b0810c330 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b08116270 con 0x7f5b0810be90 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b081065e0 0x7f5b081a2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0d000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b0810c330 0x7f5b0819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0d000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b0810c330 0x7f5b0819c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56504/0 (socket says 172.21.15.44:56504) 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0d000640 1 -- 172.21.15.44:0/1548328919 learned_addr learned my addr 172.21.15.44:0/1548328919 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b081065e0 0x7f5b081a2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58428/0 (socket says 172.21.15.44:58428) 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5affe00640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0810be90 0x7f5b081a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0d000640 1 -- 172.21.15.44:0/1548328919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0810be90 msgr2=0x7f5b081a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0d000640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0810be90 0x7f5b081a2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0d000640 1 -- 172.21.15.44:0/1548328919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b081065e0 msgr2=0x7f5b081a2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0d000640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b081065e0 0x7f5b081a2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.111+0000 7f5b0d000640 1 -- 172.21.15.44:0/1548328919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0819cf10 con 0x7f5b0810c330 2024-05-27T09:48:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.112+0000 7f5b0d000640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b0810c330 0x7f5b0819c870 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5b0002f540 tx=0x7f5b000316f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.112+0000 7f5b05600640 1 -- 172.21.15.44:0/1548328919 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00003b70 con 0x7f5b0810c330 2024-05-27T09:48:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.112+0000 7f5b05600640 1 -- 172.21.15.44:0/1548328919 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b00003d10 con 0x7f5b0810c330 2024-05-27T09:48:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.112+0000 7f5b05600640 1 -- 172.21.15.44:0/1548328919 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0002f990 con 0x7f5b0810c330 2024-05-27T09:48:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.112+0000 7f5b0f000640 1 -- 172.21.15.44:0/1548328919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0819d1a0 con 0x7f5b0810c330 2024-05-27T09:48:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.112+0000 7f5b0f000640 1 -- 172.21.15.44:0/1548328919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b0819d6a0 con 0x7f5b0810c330 2024-05-27T09:48:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.116+0000 7f5b15fdb740 1 Processor -- start 2024-05-27T09:48:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.116+0000 7f5b05600640 1 -- 172.21.15.44:0/1548328919 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5b0002fb30 con 0x7f5b0810c330 2024-05-27T09:48:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.116+0000 7f5b15fdb740 1 -- start start 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 0x55e5882e8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882e8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882e92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e588259b90 con 0x55e587f254d0 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e588249730 con 0x55e587f1d2f0 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e58824d770 con 0x55e587f461b0 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882e8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882e8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56518/0 (socket says 172.21.15.44:56518) 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b07e00640 1 -- 172.21.15.44:0/2356965667 learned_addr learned my addr 172.21.15.44:0/2356965667 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5affe00640 1 --2- 172.21.15.44:0/2356965667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882e92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5affe00640 1 --2- 172.21.15.44:0/2356965667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 0x55e5882e8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b07e00640 1 -- 172.21.15.44:0/2356965667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 msgr2=0x55e5882e8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b07e00640 1 --2- 172.21.15.44:0/2356965667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 0x55e5882e8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b07e00640 1 -- 172.21.15.44:0/2356965667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 msgr2=0x55e5882e92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b07e00640 1 --2- 172.21.15.44:0/2356965667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882e92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5b07e00640 1 -- 172.21.15.44:0/2356965667 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5881e28b0 con 0x55e587f1d2f0 2024-05-27T09:48:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.117+0000 7f5affe00640 1 --2- 172.21.15.44:0/2356965667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 0x55e5882e8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:37.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.118+0000 7f5b07e00640 1 --2- 172.21.15.44:0/2356965667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882e8da0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009c60 tx=0x7f5af8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2b01c59a56db908 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:37.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.118+0000 7f5afd600640 1 -- 172.21.15.44:0/2356965667 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af801e070 con 0x55e587f1d2f0 2024-05-27T09:48:37.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.118+0000 7f5afd600640 1 -- 172.21.15.44:0/2356965667 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5af8019ae0 con 0x55e587f1d2f0 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.118+0000 7f5afd600640 1 -- 172.21.15.44:0/2356965667 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af800ea10 con 0x55e587f1d2f0 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.118+0000 7f5b05600640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5aec0b29d0 0x7f5aec0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5b05600640 1 -- 172.21.15.44:0/1548328919 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f5b00103930 con 0x7f5b0810c330 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5affe00640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5aec0b29d0 0x7f5aec0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5b15fdb740 1 -- 172.21.15.44:0/2356965667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 msgr2=0x55e5882e8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/2356965667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882e8da0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009c60 tx=0x7f5af8011810 comp rx=0 tx=0).stop 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5b15fdb740 1 -- 172.21.15.44:0/2356965667 shutdown_connections 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/2356965667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882e92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/2356965667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882e8da0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/2356965667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 0x55e5882e8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5b15fdb740 1 -- 172.21.15.44:0/2356965667 >> 172.21.15.44:0/2356965667 conn(0x55e587f3a300 msgr2=0x55e5882e5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5b15fdb740 1 -- 172.21.15.44:0/2356965667 shutdown_connections 2024-05-27T09:48:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5affe00640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5aec0b29d0 0x7f5aec0b4ee0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x55e5882e59d0 tx=0x7f5af0004730 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.119+0000 7f5b15fdb740 1 -- 172.21.15.44:0/2356965667 wait complete. 2024-05-27T09:48:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.120+0000 7f5b15fdb740 1 -- start start 2024-05-27T09:48:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.120+0000 7f5b15fdb740 1 Processor -- start 2024-05-27T09:48:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.120+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f1d2f0 0x55e5882f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.120+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f254d0 0x55e5882f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f461b0 0x55e5882fecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e587f2f790 con 0x55e587f461b0 2024-05-27T09:48:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e5881f1bc0 con 0x55e587f254d0 2024-05-27T09:48:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e5882f5990 con 0x55e587f1d2f0 2024-05-27T09:48:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5b15fdb740 10 client.0 _ll_register_callbacks cb 0x55e5881cf400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5affe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f254d0 0x55e5882f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5affe00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f254d0 0x55e5882f2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56522/0 (socket says 172.21.15.44:56522) 2024-05-27T09:48:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5affe00640 1 -- 172.21.15.44:0/3613215255 learned_addr learned my addr 172.21.15.44:0/3613215255 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5b0d000640 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f461b0 0x55e5882fecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5b07e00640 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f1d2f0 0x55e5882f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5affe00640 1 -- 172.21.15.44:0/3613215255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f1d2f0 msgr2=0x55e5882f29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5affe00640 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f1d2f0 0x55e5882f29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5affe00640 1 -- 172.21.15.44:0/3613215255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f461b0 msgr2=0x55e5882fecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5affe00640 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f461b0 0x55e5882fecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.121+0000 7f5affe00640 1 -- 172.21.15.44:0/3613215255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e587f5e210 con 0x55e587f254d0 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.122+0000 7f5affe00640 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f254d0 0x55e5882f2f30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5af0004580 tx=0x7f5af0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80b99ce6b0f39db6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.122+0000 7f5af6000640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.122+0000 7f5af6000640 1 -- 172.21.15.44:0/3613215255 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0010640 con 0x55e587f254d0 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.122+0000 7f5b15fdb740 1 -- 172.21.15.44:0/3613215255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5af8019ae0 con 0x55e587f254d0 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.122+0000 7f5af6000640 1 -- 172.21.15.44:0/3613215255 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f5af00107e0 con 0x55e587f254d0 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.122+0000 7f5af6000640 1 -- 172.21.15.44:0/3613215255 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f5af00109e0 con 0x55e587f254d0 2024-05-27T09:48:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.122+0000 7f5b15fdb740 10 client.27740 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.122+0000 7f5b15fdb740 1 -- 172.21.15.44:0/3613215255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e58802a620 con 0x55e587f254d0 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.122+0000 7f5af6000640 1 -- 172.21.15.44:0/3613215255 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5af8019ae0 con 0x55e587f254d0 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.123+0000 7f5af6000640 1 -- 172.21.15.44:0/3613215255 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f5af001e030 con 0x55e587f254d0 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.123+0000 7f5b15fdb740 10 client.27740 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.123+0000 7f5b15fdb740 10 client.27740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.123+0000 7f5b15fdb740 20 client.27740 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.123+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e587f04d80 0x55e588304ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.123+0000 7f5b15fdb740 4 client.27740 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.123+0000 7f5b15fdb740 1 -- 172.21.15.44:0/3613215255 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e58825bae0 con 0x55e587f04d80 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.123+0000 7f5b07e00640 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e587f04d80 0x55e588304ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.124+0000 7f5b07e00640 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e587f04d80 0x55e588304ce0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=919bba1fc7448e27 server_cookie=43e98bf424219cca in_seq=0 out_seq=0 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.124+0000 7f5af6000640 10 client.27740 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.124+0000 7f5af6000640 1 -- 172.21.15.44:0/3613215255 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e58825bae0 con 0x55e587f04d80 2024-05-27T09:48:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.124+0000 7f5af6000640 10 client.27740 handle_command_reply: tid=0 2024-05-27T09:48:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.124+0000 7f5b15fdb740 1 client.27740 shutdown 2024-05-27T09:48:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5b15fdb740 1 -- 172.21.15.44:0/3613215255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f254d0 msgr2=0x55e5882f2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f254d0 0x55e5882f2f30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5af0004580 tx=0x7f5af0005f90 comp rx=0 tx=0).stop 2024-05-27T09:48:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5b15fdb740 1 -- 172.21.15.44:0/3613215255 shutdown_connections 2024-05-27T09:48:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e587f04d80 0x55e588304ce0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f461b0 0x55e5882fecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f254d0 0x55e5882f2f30 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/3613215255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f1d2f0 0x55e5882f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5af6000640 0 client.27740 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5b15fdb740 1 -- 172.21.15.44:0/3613215255 >> 172.21.15.44:0/3613215255 conn(0x55e587f3a300 msgr2=0x55e5882e8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5b15fdb740 1 -- 172.21.15.44:0/3613215255 shutdown_connections 2024-05-27T09:48:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5b15fdb740 1 -- 172.21.15.44:0/3613215255 wait complete. 2024-05-27T09:48:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.125+0000 7f5b15fdb740 20 client.27740 trim_cache size 0 max 16384 2024-05-27T09:48:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.144+0000 7f5b15fdb740 1 Processor -- start 2024-05-27T09:48:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.144+0000 7f5b15fdb740 1 -- start start 2024-05-27T09:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882f4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 0x55e58830a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b081b46f0 con 0x55e587f254d0 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b081b1f60 con 0x55e587f1d2f0 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b081afdd0 con 0x55e587f461b0 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882f2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56534/0 (socket says 172.21.15.44:56534) 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b07e00640 1 -- 172.21.15.44:0/721145697 learned_addr learned my addr 172.21.15.44:0/721145697 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5affe00640 1 --2- 172.21.15.44:0/721145697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882f4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b0d000640 1 --2- 172.21.15.44:0/721145697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 0x55e58830a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b07e00640 1 -- 172.21.15.44:0/721145697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 msgr2=0x55e58830a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b07e00640 1 --2- 172.21.15.44:0/721145697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 0x55e58830a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b07e00640 1 -- 172.21.15.44:0/721145697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 msgr2=0x55e5882f4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b07e00640 1 --2- 172.21.15.44:0/721145697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882f4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b07e00640 1 -- 172.21.15.44:0/721145697 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5882e9c40 con 0x55e587f1d2f0 2024-05-27T09:48:37.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.145+0000 7f5b0d000640 1 --2- 172.21.15.44:0/721145697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 0x55e58830a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:37.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.146+0000 7f5b07e00640 1 --2- 172.21.15.44:0/721145697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882f2f40 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009c60 tx=0x7f5af800e540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bce2839c02d5ae08 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:37.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.146+0000 7f5afcc00640 1 -- 172.21.15.44:0/721145697 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af801e040 con 0x55e587f1d2f0 2024-05-27T09:48:37.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.146+0000 7f5afcc00640 1 -- 172.21.15.44:0/721145697 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5af801f040 con 0x55e587f1d2f0 2024-05-27T09:48:37.149 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.146+0000 7f5afcc00640 1 -- 172.21.15.44:0/721145697 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af800e640 con 0x55e587f1d2f0 2024-05-27T09:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.147+0000 7f5b15fdb740 1 -- 172.21.15.44:0/721145697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 msgr2=0x55e5882f2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.147+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/721145697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882f2f40 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009c60 tx=0x7f5af800e540 comp rx=0 tx=0).stop 2024-05-27T09:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.147+0000 7f5b15fdb740 1 -- 172.21.15.44:0/721145697 shutdown_connections 2024-05-27T09:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.147+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/721145697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f461b0 0x55e58830a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.147+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/721145697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882f4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.147+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/721145697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f1d2f0 0x55e5882f2f40 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.147+0000 7f5b15fdb740 1 -- 172.21.15.44:0/721145697 >> 172.21.15.44:0/721145697 conn(0x55e587f3a300 msgr2=0x55e5882edc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.147+0000 7f5b15fdb740 1 -- 172.21.15.44:0/721145697 shutdown_connections 2024-05-27T09:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.147+0000 7f5b15fdb740 1 -- 172.21.15.44:0/721145697 wait complete. 2024-05-27T09:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.147+0000 7f5b15fdb740 1 -- start start 2024-05-27T09:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.148+0000 7f5b15fdb740 1 Processor -- start 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.148+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f1d2f0 0x55e5882fb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.148+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882f85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.148+0000 7f5b15fdb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f461b0 0x55e5882f8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.148+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b081b0420 con 0x55e587f254d0 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.148+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b081b1890 con 0x55e587f461b0 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.148+0000 7f5b15fdb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b081b0af0 con 0x55e587f1d2f0 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.148+0000 7f5b15fdb740 10 client.0 _ll_register_callbacks cb 0x7f5af0009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5affe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882f85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5affe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882f85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58464/0 (socket says 172.21.15.44:58464) 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5b0d000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f461b0 0x55e5882f8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5affe00640 1 -- 172.21.15.44:0/1214959986 learned_addr learned my addr 172.21.15.44:0/1214959986 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5b07e00640 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f1d2f0 0x55e5882fb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5affe00640 1 -- 172.21.15.44:0/1214959986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f1d2f0 msgr2=0x55e5882fb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5affe00640 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f1d2f0 0x55e5882fb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5affe00640 1 -- 172.21.15.44:0/1214959986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f461b0 msgr2=0x55e5882f8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5affe00640 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f461b0 0x55e5882f8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5affe00640 1 -- 172.21.15.44:0/1214959986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5882f9100 con 0x55e587f254d0 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5b0d000640 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f461b0 0x55e5882f8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5affe00640 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882f85b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5af002bcb0 tx=0x7f5af0010e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f24c901d39868b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.149+0000 7f5af6000640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.150+0000 7f5af6000640 1 -- 172.21.15.44:0/1214959986 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0004340 con 0x55e587f254d0 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.150+0000 7f5b15fdb740 1 -- 172.21.15.44:0/1214959986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5af801f040 con 0x55e587f254d0 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.150+0000 7f5af6000640 1 -- 172.21.15.44:0/1214959986 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f5af002b680 con 0x55e587f254d0 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.150+0000 7f5af6000640 1 -- 172.21.15.44:0/1214959986 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f5af0049050 con 0x55e587f254d0 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.150+0000 7f5b15fdb740 10 client.17484 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.150+0000 7f5b15fdb740 1 -- 172.21.15.44:0/1214959986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e58836d110 con 0x55e587f254d0 2024-05-27T09:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.151+0000 7f5af6000640 1 -- 172.21.15.44:0/1214959986 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5af801f040 con 0x55e587f254d0 2024-05-27T09:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.151+0000 7f5af6000640 1 -- 172.21.15.44:0/1214959986 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f5af001e070 con 0x55e587f254d0 2024-05-27T09:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.151+0000 7f5b15fdb740 10 client.17484 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.151+0000 7f5b15fdb740 10 client.17484 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.151+0000 7f5b15fdb740 20 client.17484 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.151+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e587eda1c0 0x55e58836f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.151+0000 7f5b15fdb740 4 client.17484 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.151+0000 7f5b15fdb740 1 -- 172.21.15.44:0/1214959986 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e5882552c0 con 0x55e587eda1c0 2024-05-27T09:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.151+0000 7f5b07e00640 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e587eda1c0 0x55e58836f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.152+0000 7f5b07e00640 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e587eda1c0 0x55e58836f8c0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f428bf3aeda644a3 server_cookie=3b2b8818613aa093 in_seq=0 out_seq=0 2024-05-27T09:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.152+0000 7f5af6000640 10 client.17484 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.416+0000 7f5af6000640 1 -- 172.21.15.44:0/1214959986 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e5882552c0 con 0x55e587eda1c0 2024-05-27T09:48:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.416+0000 7f5af6000640 10 client.17484 handle_command_reply: tid=0 2024-05-27T09:48:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.417+0000 7f5b15fdb740 1 client.17484 shutdown 2024-05-27T09:48:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.417+0000 7f5b15fdb740 1 -- 172.21.15.44:0/1214959986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 msgr2=0x55e5882f85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.417+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882f85b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5af002bcb0 tx=0x7f5af0010e30 comp rx=0 tx=0).stop 2024-05-27T09:48:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.417+0000 7f5b15fdb740 1 -- 172.21.15.44:0/1214959986 shutdown_connections 2024-05-27T09:48:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.417+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55e587eda1c0 0x55e58836f8c0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.417+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e587f461b0 0x55e5882f8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.417+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e587f254d0 0x55e5882f85b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.417+0000 7f5b15fdb740 1 --2- 172.21.15.44:0/1214959986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e587f1d2f0 0x55e5882fb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.417+0000 7f5af6000640 0 client.17484 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.417+0000 7f5b15fdb740 1 -- 172.21.15.44:0/1214959986 >> 172.21.15.44:0/1214959986 conn(0x55e587f3a300 msgr2=0x55e5882ed180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.418+0000 7f5b15fdb740 1 -- 172.21.15.44:0/1214959986 shutdown_connections 2024-05-27T09:48:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.418+0000 7f5b15fdb740 1 -- 172.21.15.44:0/1214959986 wait complete. 2024-05-27T09:48:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.418+0000 7f5b15fdb740 20 client.17484 trim_cache size 0 max 16384 2024-05-27T09:48:37.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:48:37.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:48:37.421 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:48:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 -- 172.21.15.44:0/1548328919 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5aec0b29d0 msgr2=0x7f5aec0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5aec0b29d0 0x7f5aec0b4ee0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x55e5882e59d0 tx=0x7f5af0004730 comp rx=0 tx=0).stop 2024-05-27T09:48:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 -- 172.21.15.44:0/1548328919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b0810c330 msgr2=0x7f5b0819c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b0810c330 0x7f5b0819c870 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5b0002f540 tx=0x7f5b000316f0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 -- 172.21.15.44:0/1548328919 shutdown_connections 2024-05-27T09:48:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b0810c330 0x7f5b0819c870 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0810be90 0x7f5b081a2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5aec0b29d0 0x7f5aec0b4ee0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 --2- 172.21.15.44:0/1548328919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b081065e0 0x7f5b081a2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 -- 172.21.15.44:0/1548328919 >> 172.21.15.44:0/1548328919 conn(0x7f5b08101530 msgr2=0x7f5b08108850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 -- 172.21.15.44:0/1548328919 shutdown_connections 2024-05-27T09:48:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:37.419+0000 7f5b0f000640 1 -- 172.21.15.44:0/1548328919 wait complete. 2024-05-27T09:48:38.219 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:37 smithi044 ceph-mon[31618]: pgmap v421: 97 pgs: 97 active+clean; 352 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 576 op/s 2024-05-27T09:48:38.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:37 smithi050 ceph-mon[36371]: pgmap v421: 97 pgs: 97 active+clean; 352 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 576 op/s 2024-05-27T09:48:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:37 smithi189 ceph-mon[36671]: pgmap v421: 97 pgs: 97 active+clean; 352 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 576 op/s 2024-05-27T09:48:39.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.103+0000 7fd509c00700 1 -- 172.21.15.44:0/828341860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd50410c820 msgr2=0x7fd50410cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:39.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.103+0000 7fd509c00700 1 --2- 172.21.15.44:0/828341860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd50410c820 0x7fd50410cc30 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4009990 tx=0x7fd4f4009ca0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.103+0000 7fd509c00700 1 -- 172.21.15.44:0/828341860 shutdown_connections 2024-05-27T09:48:39.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.103+0000 7fd509c00700 1 --2- 172.21.15.44:0/828341860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd504110f90 0x7fd5041188f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.103+0000 7fd509c00700 1 --2- 172.21.15.44:0/828341860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd50410c820 0x7fd50410cc30 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.103+0000 7fd509c00700 1 --2- 172.21.15.44:0/828341860 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd504110640 0x7fd504110a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.103+0000 7fd509c00700 1 -- 172.21.15.44:0/828341860 >> 172.21.15.44:0/828341860 conn(0x7fd50406f980 msgr2=0x7fd50406fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:39.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.103+0000 7fd509c00700 1 -- 172.21.15.44:0/828341860 shutdown_connections 2024-05-27T09:48:39.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.103+0000 7fd509c00700 1 -- 172.21.15.44:0/828341860 wait complete. 2024-05-27T09:48:39.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.105+0000 7fd509c00700 1 Processor -- start 2024-05-27T09:48:39.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.105+0000 7fd509c00700 1 -- start start 2024-05-27T09:48:39.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.106+0000 7fd509c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd50410c820 0x7fd50411b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.106+0000 7fd509c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd504110640 0x7fd50411bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.106+0000 7fd509c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd504110f90 0x7fd50411c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.106+0000 7fd509c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd504119ee0 con 0x7fd504110f90 2024-05-27T09:48:39.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.106+0000 7fd509c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd504119d60 con 0x7fd504110640 2024-05-27T09:48:39.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.106+0000 7fd509c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd50411a060 con 0x7fd50410c820 2024-05-27T09:48:39.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.107+0000 7fd502000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd504110640 0x7fd50411bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.107+0000 7fd502000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd504110640 0x7fd50411bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56560/0 (socket says 172.21.15.44:56560) 2024-05-27T09:48:39.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.107+0000 7fd502000700 1 -- 172.21.15.44:0/2270718008 learned_addr learned my addr 172.21.15.44:0/2270718008 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:39.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.108+0000 7fd502a00700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd50410c820 0x7fd50411b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.108+0000 7fd503400700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd504110f90 0x7fd50411c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.109+0000 7fd503400700 1 -- 172.21.15.44:0/2270718008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd50410c820 msgr2=0x7fd50411b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:39.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.109+0000 7fd503400700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd50410c820 0x7fd50411b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.109+0000 7fd503400700 1 -- 172.21.15.44:0/2270718008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd504110640 msgr2=0x7fd50411bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:39.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.109+0000 7fd503400700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd504110640 0x7fd50411bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.109+0000 7fd503400700 1 -- 172.21.15.44:0/2270718008 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5041be610 con 0x7fd504110f90 2024-05-27T09:48:39.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.110+0000 7fd503400700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd504110f90 0x7fd50411c410 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc00cc50 tx=0x7fd4fc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:39.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.110+0000 7fd4fb400700 1 -- 172.21.15.44:0/2270718008 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc016020 con 0x7fd504110f90 2024-05-27T09:48:39.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.111+0000 7fd4fb400700 1 -- 172.21.15.44:0/2270718008 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4fc012420 con 0x7fd504110f90 2024-05-27T09:48:39.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.111+0000 7fd509c00700 1 -- 172.21.15.44:0/2270718008 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5041be900 con 0x7fd504110f90 2024-05-27T09:48:39.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.111+0000 7fd4fb400700 1 -- 172.21.15.44:0/2270718008 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc0115a0 con 0x7fd504110f90 2024-05-27T09:48:39.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.111+0000 7fd509c00700 1 -- 172.21.15.44:0/2270718008 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5041bee10 con 0x7fd504110f90 2024-05-27T09:48:39.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.119+0000 7fd4fb400700 1 -- 172.21.15.44:0/2270718008 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd4fc019020 con 0x7fd504110f90 2024-05-27T09:48:39.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.120+0000 7fd51871fb80 1 Processor -- start 2024-05-27T09:48:39.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.120+0000 7fd51871fb80 1 -- start start 2024-05-27T09:48:39.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.121+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b046be40 0x55f7b05f07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.121+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f0d30 0x55f7b05ea870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.121+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b05eadb0 0x55f7b05eb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.121+0000 7fd51871fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f7b0518e40 con 0x55f7b05f0d30 2024-05-27T09:48:39.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.121+0000 7fd51871fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f7b05089e0 con 0x55f7b05eadb0 2024-05-27T09:48:39.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.121+0000 7fd51871fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f7b050ca20 con 0x55f7b046be40 2024-05-27T09:48:39.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.123+0000 7fd502a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f0d30 0x55f7b05ea870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.123+0000 7fd502a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f0d30 0x55f7b05ea870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58492/0 (socket says 172.21.15.44:58492) 2024-05-27T09:48:39.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.123+0000 7fd502a00700 1 -- 172.21.15.44:0/2443028555 learned_addr learned my addr 172.21.15.44:0/2443028555 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:39.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.123+0000 7fd502000700 1 --2- 172.21.15.44:0/2443028555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b046be40 0x55f7b05f07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.123+0000 7fd4fb400700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd4e80b2af0 0x7fd4e80b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.123+0000 7fd502a00700 1 -- 172.21.15.44:0/2443028555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b046be40 msgr2=0x55f7b05f07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:39.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.123+0000 7fd4fb400700 1 -- 172.21.15.44:0/2270718008 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7fd4fc0e22f0 con 0x7fd504110f90 2024-05-27T09:48:39.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.125+0000 7fd503400700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd4e80b2af0 0x7fd4e80b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.126+0000 7fd502a00700 1 --2- 172.21.15.44:0/2443028555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b046be40 0x55f7b05f07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.126+0000 7fd502a00700 1 -- 172.21.15.44:0/2443028555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b05eadb0 msgr2=0x55f7b05eb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:39.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.126+0000 7fd502a00700 1 --2- 172.21.15.44:0/2443028555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b05eadb0 0x55f7b05eb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.126+0000 7fd502a00700 1 -- 172.21.15.44:0/2443028555 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7b04d1420 con 0x55f7b05f0d30 2024-05-27T09:48:39.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.126+0000 7fd502000700 1 --2- 172.21.15.44:0/2443028555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b046be40 0x55f7b05f07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:48:39.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.127+0000 7fd502a00700 1 --2- 172.21.15.44:0/2443028555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f0d30 0x55f7b05ea870 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4015000 tx=0x7fd4f4003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04bad21af7af235 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:39.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.128+0000 7fd4efe00700 1 -- 172.21.15.44:0/2443028555 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4003d00 con 0x55f7b05f0d30 2024-05-27T09:48:39.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.128+0000 7fd503400700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd4e80b2af0 0x7fd4e80b5000 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc0dfb00 tx=0x7fd4fc00a500 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:39.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.128+0000 7fd4efe00700 1 -- 172.21.15.44:0/2443028555 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4f4005580 con 0x55f7b05f0d30 2024-05-27T09:48:39.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.128+0000 7fd4efe00700 1 -- 172.21.15.44:0/2443028555 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f400f460 con 0x55f7b05f0d30 2024-05-27T09:48:39.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.130+0000 7fd51871fb80 1 -- 172.21.15.44:0/2443028555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f0d30 msgr2=0x55f7b05ea870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:39.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.130+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2443028555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f0d30 0x55f7b05ea870 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4015000 tx=0x7fd4f4003730 comp rx=0 tx=0).stop 2024-05-27T09:48:39.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.130+0000 7fd51871fb80 1 -- 172.21.15.44:0/2443028555 shutdown_connections 2024-05-27T09:48:39.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.130+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2443028555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b05eadb0 0x55f7b05eb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.130+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2443028555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b046be40 0x55f7b05f07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.130+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2443028555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f0d30 0x55f7b05ea870 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.130+0000 7fd51871fb80 1 -- 172.21.15.44:0/2443028555 >> 172.21.15.44:0/2443028555 conn(0x55f7b04cbfc0 msgr2=0x55f7b05e58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:39.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.130+0000 7fd51871fb80 1 -- 172.21.15.44:0/2443028555 shutdown_connections 2024-05-27T09:48:39.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.130+0000 7fd51871fb80 1 -- 172.21.15.44:0/2443028555 wait complete. 2024-05-27T09:48:39.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.131+0000 7fd51871fb80 1 -- start start 2024-05-27T09:48:39.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.132+0000 7fd51871fb80 1 Processor -- start 2024-05-27T09:48:39.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.134+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b05f4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.134+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05eadb0 0x55f7b05f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.134+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05f50f0 0x55f7b0600b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.134+0000 7fd51871fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f7b040efa0 con 0x55f7b05eadb0 2024-05-27T09:48:39.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.134+0000 7fd51871fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f7b04a9760 con 0x55f7b046be40 2024-05-27T09:48:39.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.134+0000 7fd51871fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f7b05f77a0 con 0x55f7b05f50f0 2024-05-27T09:48:39.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.134+0000 7fd51871fb80 10 client.0 _ll_register_callbacks cb 0x55f7b04b2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:39.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.134+0000 7fd502000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b05f4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.134+0000 7fd502000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b05f4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56582/0 (socket says 172.21.15.44:56582) 2024-05-27T09:48:39.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.134+0000 7fd502000700 1 -- 172.21.15.44:0/2455012326 learned_addr learned my addr 172.21.15.44:0/2455012326 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:39.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.135+0000 7fd502000700 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05f50f0 0x55f7b0600b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.135+0000 7fd502a00700 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05eadb0 0x55f7b05f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.136+0000 7fd502a00700 1 -- 172.21.15.44:0/2455012326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05f50f0 msgr2=0x55f7b0600b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:39.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.137+0000 7fd502a00700 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05f50f0 0x55f7b0600b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.137+0000 7fd502a00700 1 -- 172.21.15.44:0/2455012326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 msgr2=0x55f7b05f4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:39.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.137+0000 7fd502a00700 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b05f4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.137+0000 7fd502a00700 1 -- 172.21.15.44:0/2455012326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7b04588d0 con 0x55f7b05eadb0 2024-05-27T09:48:39.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.138+0000 7fd502a00700 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05eadb0 0x55f7b05f4bb0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x55f7b05e6550 tx=0x7fd4f4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0c330919a664bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:39.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.141+0000 7fd4ecc00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:48:39.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.142+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2455012326 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4030060 con 0x55f7b05eadb0 2024-05-27T09:48:39.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.142+0000 7fd51871fb80 1 -- 172.21.15.44:0/2455012326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4f4005580 con 0x55f7b05eadb0 2024-05-27T09:48:39.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.143+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2455012326 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7fd4f400bd80 con 0x55f7b05eadb0 2024-05-27T09:48:39.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.144+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2455012326 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd4f4003b00 con 0x55f7b05eadb0 2024-05-27T09:48:39.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.144+0000 7fd51871fb80 10 client.17493 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:39.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.145+0000 7fd51871fb80 1 -- 172.21.15.44:0/2455012326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55f7b04d9060 con 0x55f7b05eadb0 2024-05-27T09:48:39.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.145+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2455012326 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4f4005580 con 0x55f7b05eadb0 2024-05-27T09:48:39.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.146+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2455012326 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fd4f400f8f0 con 0x55f7b05eadb0 2024-05-27T09:48:39.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.147+0000 7fd51871fb80 10 client.17493 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:39.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.147+0000 7fd51871fb80 10 client.17493 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:39.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.147+0000 7fd51871fb80 20 client.17493 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:39.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.147+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f7b0604b20 0x55f7b0606f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.147+0000 7fd51871fb80 4 client.17493 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:39.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.147+0000 7fd51871fb80 1 -- 172.21.15.44:0/2455012326 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7b051ad90 con 0x55f7b0604b20 2024-05-27T09:48:39.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.148+0000 7fd502000700 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f7b0604b20 0x55f7b0606f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.149+0000 7fd502000700 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f7b0604b20 0x55f7b0606f40 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eefd1f4b1c3233f5 server_cookie=9cccf33a33b7aacb in_seq=0 out_seq=0 2024-05-27T09:48:39.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.149+0000 7fd4ecc00700 10 client.17493 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:39.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.151+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2455012326 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f7b051ad90 con 0x55f7b0604b20 2024-05-27T09:48:39.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.151+0000 7fd4ecc00700 10 client.17493 handle_command_reply: tid=0 2024-05-27T09:48:39.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.151+0000 7fd51871fb80 1 client.17493 shutdown 2024-05-27T09:48:39.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd51871fb80 1 -- 172.21.15.44:0/2455012326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05eadb0 msgr2=0x55f7b05f4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:39.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05eadb0 0x55f7b05f4bb0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x55f7b05e6550 tx=0x7fd4f4009310 comp rx=0 tx=0).stop 2024-05-27T09:48:39.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd51871fb80 1 -- 172.21.15.44:0/2455012326 shutdown_connections 2024-05-27T09:48:39.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f7b0604b20 0x55f7b0606f40 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05f50f0 0x55f7b0600b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b05f4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2455012326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05eadb0 0x55f7b05f4bb0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd4ecc00700 0 client.17493 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:39.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd51871fb80 1 -- 172.21.15.44:0/2455012326 >> 172.21.15.44:0/2455012326 conn(0x55f7b04cbfc0 msgr2=0x55f7b05e5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:39.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd51871fb80 1 -- 172.21.15.44:0/2455012326 shutdown_connections 2024-05-27T09:48:39.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd51871fb80 1 -- 172.21.15.44:0/2455012326 wait complete. 2024-05-27T09:48:39.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.152+0000 7fd51871fb80 20 client.17493 trim_cache size 0 max 16384 2024-05-27T09:48:39.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.192+0000 7fd51871fb80 1 Processor -- start 2024-05-27T09:48:39.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.192+0000 7fd51871fb80 1 -- start start 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.192+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b05ea870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.192+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b05f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.192+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f50f0 0x55f7b0600a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.192+0000 7fd51871fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd5041c8de0 con 0x55f7b05f50f0 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.192+0000 7fd51871fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5041c6610 con 0x55f7b046be40 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd51871fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5041c4480 con 0x55f7b05eadb0 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd502000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b05ea870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd502a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b05f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd502000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b05ea870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56594/0 (socket says 172.21.15.44:56594) 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd502000700 1 -- 172.21.15.44:0/2247075397 learned_addr learned my addr 172.21.15.44:0/2247075397 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd502a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b05f39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43748/0 (socket says 172.21.15.44:43748) 2024-05-27T09:48:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd502000700 1 --2- 172.21.15.44:0/2247075397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f50f0 0x55f7b0600a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd502a00700 1 -- 172.21.15.44:0/2247075397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 msgr2=0x55f7b05ea870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd502a00700 1 --2- 172.21.15.44:0/2247075397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b05ea870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd502a00700 1 -- 172.21.15.44:0/2247075397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f50f0 msgr2=0x55f7b0600a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd502a00700 1 --2- 172.21.15.44:0/2247075397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f50f0 0x55f7b0600a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.193+0000 7fd502a00700 1 -- 172.21.15.44:0/2247075397 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7b05f1140 con 0x55f7b05eadb0 2024-05-27T09:48:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.194+0000 7fd502a00700 1 --2- 172.21.15.44:0/2247075397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b05f39b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4035040 tx=0x7fd4f4003730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea71691ccd95ba5a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.194+0000 7fd4ef400700 1 -- 172.21.15.44:0/2247075397 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4030070 con 0x55f7b05eadb0 2024-05-27T09:48:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.194+0000 7fd4ef400700 1 -- 172.21.15.44:0/2247075397 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4f4005580 con 0x55f7b05eadb0 2024-05-27T09:48:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.194+0000 7fd4ef400700 1 -- 172.21.15.44:0/2247075397 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f400f6c0 con 0x55f7b05eadb0 2024-05-27T09:48:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.195+0000 7fd51871fb80 1 -- 172.21.15.44:0/2247075397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 msgr2=0x55f7b05f39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.195+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2247075397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b05f39b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4035040 tx=0x7fd4f4003730 comp rx=0 tx=0).stop 2024-05-27T09:48:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.195+0000 7fd51871fb80 1 -- 172.21.15.44:0/2247075397 shutdown_connections 2024-05-27T09:48:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.195+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2247075397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f50f0 0x55f7b0600a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.195+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2247075397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b05ea870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.195+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2247075397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b05f39b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.195+0000 7fd51871fb80 1 -- 172.21.15.44:0/2247075397 >> 172.21.15.44:0/2247075397 conn(0x55f7b04cbfc0 msgr2=0x55f7b05ee1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.196+0000 7fd51871fb80 1 -- 172.21.15.44:0/2247075397 shutdown_connections 2024-05-27T09:48:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.196+0000 7fd51871fb80 1 -- 172.21.15.44:0/2247075397 wait complete. 2024-05-27T09:48:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.196+0000 7fd51871fb80 1 -- start start 2024-05-27T09:48:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.196+0000 7fd51871fb80 1 Processor -- start 2024-05-27T09:48:39.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.198+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b060a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.198+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b0607710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.198+0000 7fd51871fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f50f0 0x55f7b0607c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.198+0000 7fd51871fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd5041c4ad0 con 0x55f7b05f50f0 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.198+0000 7fd51871fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5041c5f40 con 0x55f7b046be40 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.198+0000 7fd51871fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5041c51a0 con 0x55f7b05eadb0 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.198+0000 7fd51871fb80 10 client.0 _ll_register_callbacks cb 0x55f7b04b2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.198+0000 7fd502000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b060a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd502a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b0607710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd502a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b0607710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43764/0 (socket says 172.21.15.44:43764) 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd502000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b060a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56606/0 (socket says 172.21.15.44:56606) 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd502a00700 1 -- 172.21.15.44:0/2174622190 learned_addr learned my addr 172.21.15.44:0/2174622190 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd502a00700 1 -- 172.21.15.44:0/2174622190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 msgr2=0x55f7b060a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd502a00700 1 --2- 172.21.15.44:0/2174622190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b060a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd502a00700 1 -- 172.21.15.44:0/2174622190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f50f0 msgr2=0x55f7b0607c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:39.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd502a00700 1 --2- 172.21.15.44:0/2174622190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f50f0 0x55f7b0607c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd502a00700 1 -- 172.21.15.44:0/2174622190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7b0608190 con 0x55f7b05eadb0 2024-05-27T09:48:39.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd502a00700 1 --2- 172.21.15.44:0/2174622190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b0607710 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4000c00 tx=0x7fd4f40224f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e427397405c50fc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd4ecc00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2174622190 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4030040 con 0x55f7b05eadb0 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2174622190 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7fd4f400f460 con 0x55f7b05eadb0 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd51871fb80 1 -- 172.21.15.44:0/2174622190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4f4005580 con 0x55f7b05eadb0 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2174622190 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd4f404a050 con 0x55f7b05eadb0 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd51871fb80 10 client.27763 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.199+0000 7fd51871fb80 1 -- 172.21.15.44:0/2174622190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55f7b0608190 con 0x55f7b05eadb0 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.200+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2174622190 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4f4005580 con 0x55f7b05eadb0 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.200+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2174622190 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fd4f4022b20 con 0x55f7b05eadb0 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.200+0000 7fd51871fb80 10 client.27763 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.200+0000 7fd51871fb80 10 client.27763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:39.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.200+0000 7fd51871fb80 20 client.27763 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:39.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.200+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2174622190 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f7b066f300 0x55f7b0671720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:39.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.200+0000 7fd51871fb80 4 client.27763 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:48:39.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.200+0000 7fd51871fb80 1 -- 172.21.15.44:0/2174622190 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f7b0514570 con 0x55f7b066f300 2024-05-27T09:48:39.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.200+0000 7fd502000700 1 --2- 172.21.15.44:0/2174622190 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f7b066f300 0x55f7b0671720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:39.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.201+0000 7fd502000700 1 --2- 172.21.15.44:0/2174622190 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f7b066f300 0x55f7b0671720 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5f47a2c133392bb server_cookie=74f48ebaf3bcb344 in_seq=0 out_seq=0 2024-05-27T09:48:39.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.201+0000 7fd4ecc00700 10 client.27763 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:39.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.201+0000 7fd4ecc00700 1 -- 172.21.15.44:0/2174622190 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f7b0514570 con 0x55f7b066f300 2024-05-27T09:48:39.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.201+0000 7fd4ecc00700 10 client.27763 handle_command_reply: tid=0 2024-05-27T09:48:39.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.201+0000 7fd51871fb80 1 client.27763 shutdown 2024-05-27T09:48:39.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd51871fb80 1 -- 172.21.15.44:0/2174622190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 msgr2=0x55f7b0607710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:39.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2174622190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b0607710 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4000c00 tx=0x7fd4f40224f0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd51871fb80 1 -- 172.21.15.44:0/2174622190 shutdown_connections 2024-05-27T09:48:39.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2174622190 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f7b066f300 0x55f7b0671720 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2174622190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f7b05f50f0 0x55f7b0607c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2174622190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7b046be40 0x55f7b060a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd51871fb80 1 --2- 172.21.15.44:0/2174622190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7b05eadb0 0x55f7b0607710 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd4ecc00700 0 client.27763 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:39.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd51871fb80 1 -- 172.21.15.44:0/2174622190 >> 172.21.15.44:0/2174622190 conn(0x55f7b04cbfc0 msgr2=0x55f7b05eec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:39.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd51871fb80 1 -- 172.21.15.44:0/2174622190 shutdown_connections 2024-05-27T09:48:39.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd51871fb80 1 -- 172.21.15.44:0/2174622190 wait complete. 2024-05-27T09:48:39.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.202+0000 7fd51871fb80 20 client.27763 trim_cache size 0 max 16384 2024-05-27T09:48:39.205 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:48:39.205 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:48:39.206 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:48:39.206 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:48:39.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 -- 172.21.15.44:0/2270718008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd4e80b2af0 msgr2=0x7fd4e80b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:39.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd4e80b2af0 0x7fd4e80b5000 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc0dfb00 tx=0x7fd4fc00a500 comp rx=0 tx=0).stop 2024-05-27T09:48:39.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 -- 172.21.15.44:0/2270718008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd504110f90 msgr2=0x7fd50411c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:39.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd504110f90 0x7fd50411c410 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc00cc50 tx=0x7fd4fc00cf60 comp rx=0 tx=0).stop 2024-05-27T09:48:39.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 -- 172.21.15.44:0/2270718008 shutdown_connections 2024-05-27T09:48:39.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd4e80b2af0 0x7fd4e80b5000 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd504110f90 0x7fd50411c410 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd50410c820 0x7fd50411b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 --2- 172.21.15.44:0/2270718008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd504110640 0x7fd50411bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:39.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 -- 172.21.15.44:0/2270718008 >> 172.21.15.44:0/2270718008 conn(0x7fd50406f980 msgr2=0x7fd50410e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:39.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 -- 172.21.15.44:0/2270718008 shutdown_connections 2024-05-27T09:48:39.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:39.204+0000 7fd509c00700 1 -- 172.21.15.44:0/2270718008 wait complete. 2024-05-27T09:48:39.932 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:48:39.933 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:48:40.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:39 smithi044 ceph-mon[31618]: pgmap v422: 97 pgs: 97 active+clean; 352 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 293 op/s 2024-05-27T09:48:40.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:39 smithi050 ceph-mon[36371]: pgmap v422: 97 pgs: 97 active+clean; 352 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 293 op/s 2024-05-27T09:48:40.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:39 smithi189 ceph-mon[36671]: pgmap v422: 97 pgs: 97 active+clean; 352 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 293 op/s 2024-05-27T09:48:40.427 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:48:42.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:41 smithi044 ceph-mon[31618]: pgmap v423: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 531 op/s 2024-05-27T09:48:42.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:41 smithi050 ceph-mon[36371]: pgmap v423: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 531 op/s 2024-05-27T09:48:42.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:41 smithi189 ceph-mon[36671]: pgmap v423: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 531 op/s 2024-05-27T09:48:42.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.763+0000 7f2529800700 1 -- 172.21.15.44:0/1113772430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f252410e8d0 msgr2=0x7f2524110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:42.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.763+0000 7f2529800700 1 --2- 172.21.15.44:0/1113772430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f252410e8d0 0x7f2524110db0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f251c00b0a0 tx=0x7f251c00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:48:42.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.764+0000 7f2529800700 1 -- 172.21.15.44:0/1113772430 shutdown_connections 2024-05-27T09:48:42.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.764+0000 7f2529800700 1 --2- 172.21.15.44:0/1113772430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f252410e8d0 0x7f2524110db0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:42.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.764+0000 7f2529800700 1 --2- 172.21.15.44:0/1113772430 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25241054f0 0x7f2524105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:42.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.764+0000 7f2529800700 1 --2- 172.21.15.44:0/1113772430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2524105e40 0x7f252410e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:42.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.764+0000 7f2529800700 1 -- 172.21.15.44:0/1113772430 >> 172.21.15.44:0/1113772430 conn(0x7f25240fedb0 msgr2=0x7f25241011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:42.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.764+0000 7f2529800700 1 -- 172.21.15.44:0/1113772430 shutdown_connections 2024-05-27T09:48:42.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.764+0000 7f2529800700 1 -- 172.21.15.44:0/1113772430 wait complete. 2024-05-27T09:48:42.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.765+0000 7f2529800700 1 Processor -- start 2024-05-27T09:48:42.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.766+0000 7f2529800700 1 -- start start 2024-05-27T09:48:42.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.766+0000 7f2529800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25241054f0 0x7f25241031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:42.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.766+0000 7f2529800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2524105e40 0x7f2524103720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:42.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.766+0000 7f2529800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f252410e8d0 0x7f2524103c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:42.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.766+0000 7f2529800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2524114480 con 0x7f25241054f0 2024-05-27T09:48:42.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.766+0000 7f2529800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2524114300 con 0x7f2524105e40 2024-05-27T09:48:42.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.766+0000 7f2529800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2524114600 con 0x7f252410e8d0 2024-05-27T09:48:42.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.766+0000 7f2522a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25241054f0 0x7f25241031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:42.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.766+0000 7f2522000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2524105e40 0x7f2524103720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:42.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.766+0000 7f2522000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2524105e40 0x7f2524103720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56626/0 (socket says 172.21.15.44:56626) 2024-05-27T09:48:42.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.766+0000 7f2522000700 1 -- 172.21.15.44:0/2718804225 learned_addr learned my addr 172.21.15.44:0/2718804225 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:42.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.767+0000 7f2523400700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f252410e8d0 0x7f2524103c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:42.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.767+0000 7f2522a00700 1 -- 172.21.15.44:0/2718804225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f252410e8d0 msgr2=0x7f2524103c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:42.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.767+0000 7f2522a00700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f252410e8d0 0x7f2524103c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:42.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.767+0000 7f2522a00700 1 -- 172.21.15.44:0/2718804225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2524105e40 msgr2=0x7f2524103720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:42.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.767+0000 7f2522a00700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2524105e40 0x7f2524103720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:42.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.767+0000 7f2522a00700 1 -- 172.21.15.44:0/2718804225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25241a9150 con 0x7f25241054f0 2024-05-27T09:48:42.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.767+0000 7f2522a00700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25241054f0 0x7f25241031e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f251400e9f0 tx=0x7f251400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:42.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.768+0000 7f251b400700 1 -- 172.21.15.44:0/2718804225 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251400c9c0 con 0x7f25241054f0 2024-05-27T09:48:42.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.768+0000 7f251b400700 1 -- 172.21.15.44:0/2718804225 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f251400cb60 con 0x7f25241054f0 2024-05-27T09:48:42.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.768+0000 7f2529800700 1 -- 172.21.15.44:0/2718804225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25241a9440 con 0x7f25241054f0 2024-05-27T09:48:42.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.768+0000 7f251b400700 1 -- 172.21.15.44:0/2718804225 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514018680 con 0x7f25241054f0 2024-05-27T09:48:42.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.768+0000 7f2529800700 1 -- 172.21.15.44:0/2718804225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25241a9980 con 0x7f25241054f0 2024-05-27T09:48:42.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.769+0000 7f2529800700 1 -- 172.21.15.44:0/2718804225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25240512b0 con 0x7f25241054f0 2024-05-27T09:48:42.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.772+0000 7f251b400700 1 -- 172.21.15.44:0/2718804225 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2514010430 con 0x7f25241054f0 2024-05-27T09:48:42.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.773+0000 7f251b400700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f25080b29f0 0x7f25080b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:42.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.773+0000 7f251b400700 1 -- 172.21.15.44:0/2718804225 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f25140e2ba0 con 0x7f25241054f0 2024-05-27T09:48:42.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.774+0000 7f2522000700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f25080b29f0 0x7f25080b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:42.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.775+0000 7f2522000700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f25080b29f0 0x7f25080b4f00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2524108390 tx=0x7f2510009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:42.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:42.779+0000 7f251b400700 1 -- 172.21.15.44:0/2718804225 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f25140aa4d0 con 0x7f25241054f0 2024-05-27T09:48:43.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.078+0000 7f2529800700 1 -- 172.21.15.44:0/2718804225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25241a9c80 con 0x7f25241054f0 2024-05-27T09:48:43.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.080+0000 7f251b400700 1 -- 172.21.15.44:0/2718804225 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11870 (secure 0 0 0) 0x7f25140a9dd0 con 0x7f25241054f0 2024-05-27T09:48:43.081 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:48:43.081 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:48:16.864365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24503]},"id":1}]} 2024-05-27T09:48:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.083+0000 7f2529800700 1 -- 172.21.15.44:0/2718804225 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f25080b29f0 msgr2=0x7f25080b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.083+0000 7f2529800700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f25080b29f0 0x7f25080b4f00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2524108390 tx=0x7f2510009260 comp rx=0 tx=0).stop 2024-05-27T09:48:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.083+0000 7f2529800700 1 -- 172.21.15.44:0/2718804225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25241054f0 msgr2=0x7f25241031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.083+0000 7f2529800700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25241054f0 0x7f25241031e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f251400e9f0 tx=0x7f251400ed00 comp rx=0 tx=0).stop 2024-05-27T09:48:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.084+0000 7f2529800700 1 -- 172.21.15.44:0/2718804225 shutdown_connections 2024-05-27T09:48:43.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.084+0000 7f2529800700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f25080b29f0 0x7f25080b4f00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:43.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.084+0000 7f2529800700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f252410e8d0 0x7f2524103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:43.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.084+0000 7f2529800700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25241054f0 0x7f25241031e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:43.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.084+0000 7f2529800700 1 --2- 172.21.15.44:0/2718804225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2524105e40 0x7f2524103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:43.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.084+0000 7f2529800700 1 -- 172.21.15.44:0/2718804225 >> 172.21.15.44:0/2718804225 conn(0x7f25240fedb0 msgr2=0x7f2524108470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:43.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.084+0000 7f2529800700 1 -- 172.21.15.44:0/2718804225 shutdown_connections 2024-05-27T09:48:43.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:43.084+0000 7f2529800700 1 -- 172.21.15.44:0/2718804225 wait complete. 2024-05-27T09:48:43.086 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 43 2024-05-27T09:48:44.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:43 smithi050 ceph-mon[36371]: pgmap v424: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 396 op/s 2024-05-27T09:48:44.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:43 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2718804225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:48:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:43 smithi189 ceph-mon[36671]: pgmap v424: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 396 op/s 2024-05-27T09:48:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:43 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2718804225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:48:44.353 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:43 smithi044 ceph-mon[31618]: pgmap v424: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 396 op/s 2024-05-27T09:48:44.353 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:43 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2718804225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:48:44.415 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:48:44.891 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:48:46.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:45 smithi044 ceph-mon[31618]: pgmap v425: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 396 op/s 2024-05-27T09:48:46.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:45 smithi050 ceph-mon[36371]: pgmap v425: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 396 op/s 2024-05-27T09:48:46.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:45 smithi189 ceph-mon[36671]: pgmap v425: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 396 op/s 2024-05-27T09:48:47.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.234+0000 7f3020e00700 1 -- 172.21.15.44:0/37922884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f301c108750 msgr2=0x7f301c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:47.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.234+0000 7f3020e00700 1 --2- 172.21.15.44:0/37922884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f301c108750 0x7f301c1100b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f301400b0a0 tx=0x7f301400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.234+0000 7f3020e00700 1 -- 172.21.15.44:0/37922884 shutdown_connections 2024-05-27T09:48:47.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.234+0000 7f3020e00700 1 --2- 172.21.15.44:0/37922884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f301c108750 0x7f301c1100b0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.234+0000 7f3020e00700 1 --2- 172.21.15.44:0/37922884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f301c1034d0 0x7f301c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.234+0000 7f3020e00700 1 --2- 172.21.15.44:0/37922884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f301c107e00 0x7f301c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.234+0000 7f3020e00700 1 -- 172.21.15.44:0/37922884 >> 172.21.15.44:0/37922884 conn(0x7f301c0fef20 msgr2=0x7f301c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:47.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.234+0000 7f3020e00700 1 -- 172.21.15.44:0/37922884 shutdown_connections 2024-05-27T09:48:47.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.234+0000 7f3020e00700 1 -- 172.21.15.44:0/37922884 wait complete. 2024-05-27T09:48:47.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.236+0000 7f3020e00700 1 Processor -- start 2024-05-27T09:48:47.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.237+0000 7f3020e00700 1 -- start start 2024-05-27T09:48:47.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.237+0000 7f3020e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f301c1034d0 0x7f301c19bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.237+0000 7f3020e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f301c107e00 0x7f301c19c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.237+0000 7f3020e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f301c108750 0x7f301c19c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.237+0000 7f3020e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1133d0 con 0x7f301c107e00 2024-05-27T09:48:47.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.237+0000 7f3020e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f301c113250 con 0x7f301c108750 2024-05-27T09:48:47.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.237+0000 7f3020e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f301c113550 con 0x7f301c1034d0 2024-05-27T09:48:47.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f301a800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f301c108750 0x7f301c19c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f301a800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f301c108750 0x7f301c19c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56044/0 (socket says 172.21.15.44:56044) 2024-05-27T09:48:47.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f301a800700 1 -- 172.21.15.44:0/869538378 learned_addr learned my addr 172.21.15.44:0/869538378 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:47.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f3019e00700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f301c1034d0 0x7f301c19bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f3019400700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f301c107e00 0x7f301c19c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f301a800700 1 -- 172.21.15.44:0/869538378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f301c1034d0 msgr2=0x7f301c19bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:47.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f301a800700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f301c1034d0 0x7f301c19bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f301a800700 1 -- 172.21.15.44:0/869538378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f301c107e00 msgr2=0x7f301c19c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:47.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f301a800700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f301c107e00 0x7f301c19c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f301a800700 1 -- 172.21.15.44:0/869538378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301c1a6fc0 con 0x7f301c108750 2024-05-27T09:48:47.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f3019400700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f301c107e00 0x7f301c19c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:47.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.238+0000 7f301a800700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f301c108750 0x7f301c19c710 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f301400b070 tx=0x7f3014004c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:47.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.239+0000 7f3012a00700 1 -- 172.21.15.44:0/869538378 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014021070 con 0x7f301c108750 2024-05-27T09:48:47.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.239+0000 7f3012a00700 1 -- 172.21.15.44:0/869538378 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3014007b80 con 0x7f301c108750 2024-05-27T09:48:47.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.239+0000 7f3020e00700 1 -- 172.21.15.44:0/869538378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f301c1a7250 con 0x7f301c108750 2024-05-27T09:48:47.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.239+0000 7f3012a00700 1 -- 172.21.15.44:0/869538378 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014005040 con 0x7f301c108750 2024-05-27T09:48:47.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.239+0000 7f3020e00700 1 -- 172.21.15.44:0/869538378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f301c1a7750 con 0x7f301c108750 2024-05-27T09:48:47.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.244+0000 7f3012a00700 1 -- 172.21.15.44:0/869538378 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3014024070 con 0x7f301c108750 2024-05-27T09:48:47.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.244+0000 7f302fa3fb80 1 Processor -- start 2024-05-27T09:48:47.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.244+0000 7f302fa3fb80 1 -- start start 2024-05-27T09:48:47.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.245+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe07ed9f0 0x55cbe088f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.245+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 0x55cbe088fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.245+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0880f30 0x55cbe08813e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.245+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cbe07aee40 con 0x55cbe07ed9f0 2024-05-27T09:48:47.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.245+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cbe079e9e0 con 0x55cbe0880f30 2024-05-27T09:48:47.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.245+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cbe07a2a20 con 0x55cbe0701e40 2024-05-27T09:48:47.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.245+0000 7f3019400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe07ed9f0 0x55cbe088f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.245+0000 7f3019400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe07ed9f0 0x55cbe088f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59480/0 (socket says 172.21.15.44:59480) 2024-05-27T09:48:47.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.245+0000 7f3019e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 0x55cbe088fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.246+0000 7f3019400700 1 -- 172.21.15.44:0/1207926651 learned_addr learned my addr 172.21.15.44:0/1207926651 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:47.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.246+0000 7f3019400700 1 --2- 172.21.15.44:0/1207926651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0880f30 0x55cbe08813e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.246+0000 7f3019e00700 1 -- 172.21.15.44:0/1207926651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0880f30 msgr2=0x55cbe08813e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.246+0000 7f3019e00700 1 --2- 172.21.15.44:0/1207926651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0880f30 0x55cbe08813e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.246+0000 7f3019e00700 1 -- 172.21.15.44:0/1207926651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe07ed9f0 msgr2=0x55cbe088f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.246+0000 7f3019e00700 1 --2- 172.21.15.44:0/1207926651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe07ed9f0 0x55cbe088f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.246+0000 7f3019e00700 1 -- 172.21.15.44:0/1207926651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbe0767420 con 0x55cbe0701e40 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.246+0000 7f3019400700 1 --2- 172.21.15.44:0/1207926651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe07ed9f0 0x55cbe088f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.246+0000 7f3012a00700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30000b2a70 0x7f30000b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.247+0000 7f3012a00700 1 -- 172.21.15.44:0/869538378 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f301400f020 con 0x7f301c108750 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.247+0000 7f3019e00700 1 --2- 172.21.15.44:0/1207926651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 0x55cbe088fe50 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f300c009cb0 tx=0x7f300c011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=205101bb4b4dd580 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.247+0000 7f3019400700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30000b2a70 0x7f30000b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.247+0000 7f3007400700 1 -- 172.21.15.44:0/1207926651 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c011d20 con 0x55cbe0701e40 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.247+0000 7f3007400700 1 -- 172.21.15.44:0/1207926651 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f300c004500 con 0x55cbe0701e40 2024-05-27T09:48:47.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.247+0000 7f3007400700 1 -- 172.21.15.44:0/1207926651 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c005280 con 0x55cbe0701e40 2024-05-27T09:48:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.248+0000 7f302fa3fb80 1 -- 172.21.15.44:0/1207926651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 msgr2=0x55cbe088fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.248+0000 7f3019400700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30000b2a70 0x7f30000b4f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x55cbe088f330 tx=0x7f3008009710 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.248+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/1207926651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 0x55cbe088fe50 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f300c009cb0 tx=0x7f300c011440 comp rx=0 tx=0).stop 2024-05-27T09:48:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.248+0000 7f302fa3fb80 1 -- 172.21.15.44:0/1207926651 shutdown_connections 2024-05-27T09:48:47.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.248+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/1207926651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0880f30 0x55cbe08813e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.248+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/1207926651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe07ed9f0 0x55cbe088f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.248+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/1207926651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 0x55cbe088fe50 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.248+0000 7f302fa3fb80 1 -- 172.21.15.44:0/1207926651 >> 172.21.15.44:0/1207926651 conn(0x55cbe079b390 msgr2=0x55cbe087c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:47.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.248+0000 7f302fa3fb80 1 -- 172.21.15.44:0/1207926651 shutdown_connections 2024-05-27T09:48:47.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.249+0000 7f302fa3fb80 1 -- 172.21.15.44:0/1207926651 wait complete. 2024-05-27T09:48:47.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.249+0000 7f302fa3fb80 1 -- start start 2024-05-27T09:48:47.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.250+0000 7f302fa3fb80 1 Processor -- start 2024-05-27T09:48:47.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.250+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe0893560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.250+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe07ed9f0 0x55cbe0893aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.250+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe0893fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.250+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cbe06a4fa0 con 0x55cbe0880f30 2024-05-27T09:48:47.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.250+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cbe073f760 con 0x55cbe0701e40 2024-05-27T09:48:47.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.250+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cbe0896500 con 0x55cbe07ed9f0 2024-05-27T09:48:47.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.250+0000 7f302fa3fb80 10 client.0 _ll_register_callbacks cb 0x55cbe0748c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:47.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f3019e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe0893560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f3019e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe0893560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56062/0 (socket says 172.21.15.44:56062) 2024-05-27T09:48:47.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f3019e00700 1 -- 172.21.15.44:0/3208240217 learned_addr learned my addr 172.21.15.44:0/3208240217 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:47.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f3019400700 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe07ed9f0 0x55cbe0893aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f301a800700 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe0893fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f3019e00700 1 -- 172.21.15.44:0/3208240217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe07ed9f0 msgr2=0x55cbe0893aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:47.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f3019e00700 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe07ed9f0 0x55cbe0893aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f3019e00700 1 -- 172.21.15.44:0/3208240217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 msgr2=0x55cbe0893fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:47.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f3019e00700 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe0893fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f3019e00700 1 -- 172.21.15.44:0/3208240217 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbe06ee8d0 con 0x55cbe0701e40 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f3019e00700 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe0893560 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f300c004bc0 tx=0x7f300c004c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c06d5fdc97aee11c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.251+0000 7f2ffbe00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.252+0000 7f2ffbe00700 1 -- 172.21.15.44:0/3208240217 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c004360 con 0x55cbe0701e40 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.252+0000 7f302fa3fb80 1 -- 172.21.15.44:0/3208240217 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f300c004500 con 0x55cbe0701e40 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.252+0000 7f2ffbe00700 1 -- 172.21.15.44:0/3208240217 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f300c011b20 con 0x55cbe0701e40 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.252+0000 7f2ffbe00700 1 -- 172.21.15.44:0/3208240217 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f300c030050 con 0x55cbe0701e40 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.252+0000 7f302fa3fb80 10 client.27779 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.252+0000 7f302fa3fb80 1 -- 172.21.15.44:0/3208240217 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55cbe076f060 con 0x55cbe0701e40 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.252+0000 7f2ffbe00700 1 -- 172.21.15.44:0/3208240217 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f300c004500 con 0x55cbe0701e40 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.253+0000 7f2ffbe00700 1 -- 172.21.15.44:0/3208240217 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f300c018070 con 0x55cbe0701e40 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.253+0000 7f302fa3fb80 10 client.27779 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.253+0000 7f302fa3fb80 10 client.27779 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T09:48:47.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.253+0000 7f302fa3fb80 20 client.27779 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:47.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.253+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55cbe08a3330 0x55cbe08a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.253+0000 7f302fa3fb80 4 client.27779 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:47.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.253+0000 7f302fa3fb80 1 -- 172.21.15.44:0/3208240217 --> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbe07b0d90 con 0x55cbe08a3330 2024-05-27T09:48:47.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.253+0000 7f3019400700 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55cbe08a3330 0x55cbe08a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.254+0000 7f3019400700 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55cbe08a3330 0x55cbe08a5750 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f53f1f673154d81e server_cookie=e29e3897f8b5de87 in_seq=0 out_seq=0 2024-05-27T09:48:47.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.254+0000 7f2ffbe00700 10 client.27779 ms_handle_connect on v2:172.21.15.50:6836/3540954143 2024-05-27T09:48:47.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f2ffbe00700 1 -- 172.21.15.44:0/3208240217 <== mds.1 v2:172.21.15.50:6836/3540954143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cbe07b0d90 con 0x55cbe08a3330 2024-05-27T09:48:47.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f2ffbe00700 10 client.27779 handle_command_reply: tid=0 2024-05-27T09:48:47.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 1 client.27779 shutdown 2024-05-27T09:48:47.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 1 -- 172.21.15.44:0/3208240217 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 msgr2=0x55cbe0893560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:47.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe0893560 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f300c004bc0 tx=0x7f300c004c20 comp rx=0 tx=0).stop 2024-05-27T09:48:47.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 1 -- 172.21.15.44:0/3208240217 shutdown_connections 2024-05-27T09:48:47.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe0893fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55cbe08a3330 0x55cbe08a5750 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe0893560 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/3208240217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe07ed9f0 0x55cbe0893aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 1 -- 172.21.15.44:0/3208240217 >> 172.21.15.44:0/3208240217 conn(0x55cbe079b390 msgr2=0x55cbe087fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:47.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f2ffbe00700 0 client.27779 ms_handle_reset on v2:172.21.15.50:6836/3540954143 2024-05-27T09:48:47.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 1 -- 172.21.15.44:0/3208240217 shutdown_connections 2024-05-27T09:48:47.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 1 -- 172.21.15.44:0/3208240217 wait complete. 2024-05-27T09:48:47.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.255+0000 7f302fa3fb80 20 client.27779 trim_cache size 0 max 16384 2024-05-27T09:48:47.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.273+0000 7f302fa3fb80 1 Processor -- start 2024-05-27T09:48:47.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.273+0000 7f302fa3fb80 1 -- start start 2024-05-27T09:48:47.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.273+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe08921c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.273+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe07ed9f0 0x55cbe088ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.273+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe089f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.273+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1af500 con 0x55cbe0880f30 2024-05-27T09:48:47.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.273+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1acd30 con 0x55cbe0701e40 2024-05-27T09:48:47.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.273+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1aaba0 con 0x55cbe07ed9f0 2024-05-27T09:48:47.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f3019e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe08921c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f3019e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe08921c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56070/0 (socket says 172.21.15.44:56070) 2024-05-27T09:48:47.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f3019e00700 1 -- 172.21.15.44:0/2369899400 learned_addr learned my addr 172.21.15.44:0/2369899400 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:47.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f3019400700 1 --2- 172.21.15.44:0/2369899400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe07ed9f0 0x55cbe088ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f301a800700 1 --2- 172.21.15.44:0/2369899400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe089f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f3019e00700 1 -- 172.21.15.44:0/2369899400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe07ed9f0 msgr2=0x55cbe088ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:47.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f3019e00700 1 --2- 172.21.15.44:0/2369899400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe07ed9f0 0x55cbe088ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f3019e00700 1 -- 172.21.15.44:0/2369899400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 msgr2=0x55cbe089f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:47.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f3019e00700 1 --2- 172.21.15.44:0/2369899400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe089f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f3019e00700 1 -- 172.21.15.44:0/2369899400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbe0878240 con 0x55cbe0701e40 2024-05-27T09:48:47.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f3019400700 1 --2- 172.21.15.44:0/2369899400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe07ed9f0 0x55cbe088ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:47.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.274+0000 7f3019e00700 1 --2- 172.21.15.44:0/2369899400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe08921c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f300c011480 tx=0x7f300c005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c842a2abfc944de server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:47.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.275+0000 7f3006a00700 1 -- 172.21.15.44:0/2369899400 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c003680 con 0x55cbe0701e40 2024-05-27T09:48:47.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.275+0000 7f3006a00700 1 -- 172.21.15.44:0/2369899400 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f300c027070 con 0x55cbe0701e40 2024-05-27T09:48:47.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.275+0000 7f3006a00700 1 -- 172.21.15.44:0/2369899400 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c004d90 con 0x55cbe0701e40 2024-05-27T09:48:47.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.276+0000 7f302fa3fb80 1 -- 172.21.15.44:0/2369899400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 msgr2=0x55cbe08921c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:47.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.276+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/2369899400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe08921c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f300c011480 tx=0x7f300c005a20 comp rx=0 tx=0).stop 2024-05-27T09:48:47.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.276+0000 7f302fa3fb80 1 -- 172.21.15.44:0/2369899400 shutdown_connections 2024-05-27T09:48:47.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.276+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/2369899400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe089f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.276+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/2369899400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe0701e40 0x55cbe08921c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.276+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/2369899400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe07ed9f0 0x55cbe088ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.276+0000 7f302fa3fb80 1 -- 172.21.15.44:0/2369899400 >> 172.21.15.44:0/2369899400 conn(0x55cbe079b390 msgr2=0x55cbe088e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.276+0000 7f302fa3fb80 1 -- 172.21.15.44:0/2369899400 shutdown_connections 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.276+0000 7f302fa3fb80 1 -- 172.21.15.44:0/2369899400 wait complete. 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.277+0000 7f302fa3fb80 1 -- start start 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.277+0000 7f302fa3fb80 1 Processor -- start 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.277+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 0x55cbe089b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.277+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe07ed9f0 0x55cbe089bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.277+0000 7f302fa3fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe088f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.277+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1ab1f0 con 0x55cbe0880f30 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.277+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1ac660 con 0x55cbe07ed9f0 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.277+0000 7f302fa3fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1ab8c0 con 0x55cbe0701e40 2024-05-27T09:48:47.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f302fa3fb80 10 client.0 _ll_register_callbacks cb 0x55cbe0748c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:47.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f3019400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe07ed9f0 0x55cbe089bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f3019400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe07ed9f0 0x55cbe089bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56086/0 (socket says 172.21.15.44:56086) 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f3019400700 1 -- 172.21.15.44:0/275611102 learned_addr learned my addr 172.21.15.44:0/275611102 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f301a800700 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe088f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f3019e00700 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 0x55cbe089b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f3019400700 1 -- 172.21.15.44:0/275611102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 msgr2=0x55cbe089b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f3019400700 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 0x55cbe089b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f3019400700 1 -- 172.21.15.44:0/275611102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 msgr2=0x55cbe088f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f3019400700 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe088f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f3019400700 1 -- 172.21.15.44:0/275611102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbe0891310 con 0x55cbe07ed9f0 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f3019e00700 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 0x55cbe089b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f3019400700 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe07ed9f0 0x55cbe089bc90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f30080096e0 tx=0x7f3008009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8545026ed65ea8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:47.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.278+0000 7f2ffbe00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.279+0000 7f2ffbe00700 1 -- 172.21.15.44:0/275611102 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008030020 con 0x55cbe07ed9f0 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.279+0000 7f2ffbe00700 1 -- 172.21.15.44:0/275611102 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f300802c420 con 0x55cbe07ed9f0 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.279+0000 7f302fa3fb80 1 -- 172.21.15.44:0/275611102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f300c027070 con 0x55cbe07ed9f0 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.279+0000 7f2ffbe00700 1 -- 172.21.15.44:0/275611102 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f300803b050 con 0x55cbe07ed9f0 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.279+0000 7f302fa3fb80 10 client.27785 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.279+0000 7f302fa3fb80 1 -- 172.21.15.44:0/275611102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55cbe090d030 con 0x55cbe07ed9f0 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.279+0000 7f2ffbe00700 1 -- 172.21.15.44:0/275611102 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f300c027070 con 0x55cbe07ed9f0 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.280+0000 7f2ffbe00700 1 -- 172.21.15.44:0/275611102 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f3008020920 con 0x55cbe07ed9f0 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.280+0000 7f302fa3fb80 10 client.27785 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.280+0000 7f302fa3fb80 10 client.27785 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.280+0000 7f302fa3fb80 20 client.27785 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.280+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55cbe090d7d0 0x55cbe090fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.280+0000 7f302fa3fb80 4 client.27785 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:48:47.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.280+0000 7f302fa3fb80 1 -- 172.21.15.44:0/275611102 --> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cbe07aa570 con 0x55cbe090d7d0 2024-05-27T09:48:47.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.280+0000 7f3019e00700 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55cbe090d7d0 0x55cbe090fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:47.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f3019e00700 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55cbe090d7d0 0x55cbe090fbf0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=77fe119e3276f6a5 server_cookie=fa3e39ee2da7aa8 in_seq=0 out_seq=0 2024-05-27T09:48:47.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f2ffbe00700 10 client.27785 ms_handle_connect on v2:172.21.15.50:6836/3540954143 2024-05-27T09:48:47.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f2ffbe00700 1 -- 172.21.15.44:0/275611102 <== mds.1 v2:172.21.15.50:6836/3540954143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cbe07aa570 con 0x55cbe090d7d0 2024-05-27T09:48:47.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f2ffbe00700 10 client.27785 handle_command_reply: tid=0 2024-05-27T09:48:47.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f302fa3fb80 1 client.27785 shutdown 2024-05-27T09:48:47.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f302fa3fb80 1 -- 172.21.15.44:0/275611102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe07ed9f0 msgr2=0x55cbe089bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:47.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe07ed9f0 0x55cbe089bc90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f30080096e0 tx=0x7f3008009260 comp rx=0 tx=0).stop 2024-05-27T09:48:47.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f302fa3fb80 1 -- 172.21.15.44:0/275611102 shutdown_connections 2024-05-27T09:48:47.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cbe0880f30 0x55cbe088f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cbe0701e40 0x55cbe089b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55cbe090d7d0 0x55cbe090fbf0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f302fa3fb80 1 --2- 172.21.15.44:0/275611102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cbe07ed9f0 0x55cbe089bc90 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f302fa3fb80 1 -- 172.21.15.44:0/275611102 >> 172.21.15.44:0/275611102 conn(0x55cbe079b390 msgr2=0x55cbe087fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:47.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.281+0000 7f2ffbe00700 0 client.27785 ms_handle_reset on v2:172.21.15.50:6836/3540954143 2024-05-27T09:48:47.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.282+0000 7f302fa3fb80 1 -- 172.21.15.44:0/275611102 shutdown_connections 2024-05-27T09:48:47.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.282+0000 7f302fa3fb80 1 -- 172.21.15.44:0/275611102 wait complete. 2024-05-27T09:48:47.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.282+0000 7f302fa3fb80 20 client.27785 trim_cache size 0 max 16384 2024-05-27T09:48:47.286 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:48:47.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.282+0000 7f3020e00700 1 -- 172.21.15.44:0/869538378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30000b2a70 msgr2=0x7f30000b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:47.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.282+0000 7f3020e00700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30000b2a70 0x7f30000b4f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x55cbe088f330 tx=0x7f3008009710 comp rx=0 tx=0).stop 2024-05-27T09:48:47.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.282+0000 7f3020e00700 1 -- 172.21.15.44:0/869538378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f301c108750 msgr2=0x7f301c19c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:47.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.283+0000 7f3020e00700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f301c108750 0x7f301c19c710 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f301400b070 tx=0x7f3014004c60 comp rx=0 tx=0).stop 2024-05-27T09:48:47.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.283+0000 7f3020e00700 1 -- 172.21.15.44:0/869538378 shutdown_connections 2024-05-27T09:48:47.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.283+0000 7f3020e00700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f30000b2a70 0x7f30000b4f80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.283+0000 7f3020e00700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f301c108750 0x7f301c19c710 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.283+0000 7f3020e00700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f301c1034d0 0x7f301c19bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.283+0000 7f3020e00700 1 --2- 172.21.15.44:0/869538378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f301c107e00 0x7f301c19c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:47.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.283+0000 7f3020e00700 1 -- 172.21.15.44:0/869538378 >> 172.21.15.44:0/869538378 conn(0x7f301c0fef20 msgr2=0x7f301c1096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:47.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.283+0000 7f3020e00700 1 -- 172.21.15.44:0/869538378 shutdown_connections 2024-05-27T09:48:47.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:47.283+0000 7f3020e00700 1 -- 172.21.15.44:0/869538378 wait complete. 2024-05-27T09:48:48.023 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:48:48.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:47 smithi050 ceph-mon[36371]: pgmap v426: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 506 op/s 2024-05-27T09:48:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:47 smithi189 ceph-mon[36671]: pgmap v426: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 506 op/s 2024-05-27T09:48:48.452 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:48:48.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:47 smithi044 ceph-mon[31618]: pgmap v426: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 506 op/s 2024-05-27T09:48:50.207 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:49 smithi044 ceph-mon[31618]: pgmap v427: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 347 op/s 2024-05-27T09:48:50.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:49 smithi050 ceph-mon[36371]: pgmap v427: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 347 op/s 2024-05-27T09:48:50.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:49 smithi189 ceph-mon[36671]: pgmap v427: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 347 op/s 2024-05-27T09:48:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.645+0000 7f1bf7e00640 1 -- 172.21.15.44:0/860973850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf01032e0 msgr2=0x7f1bf010f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.645+0000 7f1bf7e00640 1 --2- 172.21.15.44:0/860973850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf01032e0 0x7f1bf010f6b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009ca0 tx=0x7f1be802a8c0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.645+0000 7f1bf7e00640 1 -- 172.21.15.44:0/860973850 shutdown_connections 2024-05-27T09:48:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.645+0000 7f1bf7e00640 1 --2- 172.21.15.44:0/860973850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf01032e0 0x7f1bf010f6b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.645+0000 7f1bf7e00640 1 --2- 172.21.15.44:0/860973850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf0109030 0x7f1bf0102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.645+0000 7f1bf7e00640 1 --2- 172.21.15.44:0/860973850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bf0108b90 0x7f1bf0102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.645+0000 7f1bf7e00640 1 -- 172.21.15.44:0/860973850 >> 172.21.15.44:0/860973850 conn(0x7f1bf00782d0 msgr2=0x7f1bf0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.645+0000 7f1bf7e00640 1 -- 172.21.15.44:0/860973850 shutdown_connections 2024-05-27T09:48:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.645+0000 7f1bf7e00640 1 -- 172.21.15.44:0/860973850 wait complete. 2024-05-27T09:48:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.647+0000 7f1bf7e00640 1 Processor -- start 2024-05-27T09:48:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.647+0000 7f1bf7e00640 1 -- start start 2024-05-27T09:48:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.647+0000 7f1bf7e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf01032e0 0x7f1bf01a2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.647+0000 7f1bf7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf0108b90 0x7f1bf01a2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.647+0000 7f1bf7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bf0109030 0x7f1bf019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.647+0000 7f1bf7e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf0112dd0 con 0x7f1bf0109030 2024-05-27T09:48:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.647+0000 7f1bf7e00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf0112c50 con 0x7f1bf01032e0 2024-05-27T09:48:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.647+0000 7f1bf7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf0112f50 con 0x7f1bf0108b90 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.648+0000 7f1bf5400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf01032e0 0x7f1bf01a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.648+0000 7f1bf5400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf01032e0 0x7f1bf01a2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56098/0 (socket says 172.21.15.44:56098) 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.648+0000 7f1bf5400640 1 -- 172.21.15.44:0/1711794180 learned_addr learned my addr 172.21.15.44:0/1711794180 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.648+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf0108b90 0x7f1bf01a2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.648+0000 7f1bf5e00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bf0109030 0x7f1bf019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.648+0000 7f1bf4a00640 1 -- 172.21.15.44:0/1711794180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf01032e0 msgr2=0x7f1bf01a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.648+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf01032e0 0x7f1bf01a2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.648+0000 7f1bf4a00640 1 -- 172.21.15.44:0/1711794180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bf0109030 msgr2=0x7f1bf019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.648+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bf0109030 0x7f1bf019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.648+0000 7f1bf4a00640 1 -- 172.21.15.44:0/1711794180 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf019cf30 con 0x7f1bf0108b90 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.648+0000 7f1bf5e00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bf0109030 0x7f1bf019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:50.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.649+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf0108b90 0x7f1bf01a2bb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1bf006c640 tx=0x7f1bdc00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.649+0000 7f1bee000640 1 -- 172.21.15.44:0/1711794180 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc00cc40 con 0x7f1bf0108b90 2024-05-27T09:48:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.649+0000 7f1bee000640 1 -- 172.21.15.44:0/1711794180 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bdc00cde0 con 0x7f1bf0108b90 2024-05-27T09:48:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.649+0000 7f1bee000640 1 -- 172.21.15.44:0/1711794180 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc010430 con 0x7f1bf0108b90 2024-05-27T09:48:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.649+0000 7f1bf7e00640 1 -- 172.21.15.44:0/1711794180 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf019d220 con 0x7f1bf0108b90 2024-05-27T09:48:50.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.649+0000 7f1bf7e00640 1 -- 172.21.15.44:0/1711794180 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf01a94f0 con 0x7f1bf0108b90 2024-05-27T09:48:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.653+0000 7f1bfedea740 1 Processor -- start 2024-05-27T09:48:50.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.653+0000 7f1bfedea740 1 -- start start 2024-05-27T09:48:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f0849b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bee000640 1 -- 172.21.15.44:0/1711794180 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f1bdc003ed0 con 0x7f1bf0108b90 2024-05-27T09:48:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f0849bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f0849c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bfedea740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558f0840cb90 con 0x558f080d02f0 2024-05-27T09:48:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bfedea740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558f083fc730 con 0x558f080f91b0 2024-05-27T09:48:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bfedea740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558f08400770 con 0x558f080d84d0 2024-05-27T09:48:50.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bf5e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f0849b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bf5e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f0849b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56114/0 (socket says 172.21.15.44:56114) 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bf5e00640 1 -- 172.21.15.44:0/2191915076 learned_addr learned my addr 172.21.15.44:0/2191915076 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/2191915076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f0849c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.654+0000 7f1bf5400640 1 --2- 172.21.15.44:0/2191915076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f0849bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.655+0000 7f1bf5e00640 1 -- 172.21.15.44:0/2191915076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 msgr2=0x558f0849c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.655+0000 7f1bf5e00640 1 --2- 172.21.15.44:0/2191915076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f0849c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.655+0000 7f1bf5e00640 1 -- 172.21.15.44:0/2191915076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 msgr2=0x558f0849bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.655+0000 7f1bf5e00640 1 --2- 172.21.15.44:0/2191915076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f0849bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.655+0000 7f1bf5e00640 1 -- 172.21.15.44:0/2191915076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f083958b0 con 0x558f080f91b0 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.655+0000 7f1bf5400640 1 --2- 172.21.15.44:0/2191915076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f0849bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.655+0000 7f1bf5e00640 1 --2- 172.21.15.44:0/2191915076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f0849b860 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009870 tx=0x7f1be80098a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=282b94a369a94e44 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.655+0000 7f1be2600640 1 -- 172.21.15.44:0/2191915076 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8031d70 con 0x558f080f91b0 2024-05-27T09:48:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.655+0000 7f1be2600640 1 -- 172.21.15.44:0/2191915076 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be8036040 con 0x558f080f91b0 2024-05-27T09:48:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.655+0000 7f1be2600640 1 -- 172.21.15.44:0/2191915076 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be80042f0 con 0x558f080f91b0 2024-05-27T09:48:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.656+0000 7f1bee000640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1bd80b29d0 0x7f1bd80b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.656+0000 7f1bee000640 1 -- 172.21.15.44:0/1711794180 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f1bdc0e2a20 con 0x7f1bf0108b90 2024-05-27T09:48:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.656+0000 7f1bf5400640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1bd80b29d0 0x7f1bd80b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.657+0000 7f1bfedea740 1 -- 172.21.15.44:0/2191915076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 msgr2=0x558f0849b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.657+0000 7f1bf5400640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1bd80b29d0 0x7f1bd80b4ee0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x558f0849cc10 tx=0x7f1be4005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.657+0000 7f1bfedea740 1 --2- 172.21.15.44:0/2191915076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f0849b860 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009870 tx=0x7f1be80098a0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.657+0000 7f1bfedea740 1 -- 172.21.15.44:0/2191915076 shutdown_connections 2024-05-27T09:48:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.657+0000 7f1bfedea740 1 --2- 172.21.15.44:0/2191915076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f0849c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.657+0000 7f1bfedea740 1 --2- 172.21.15.44:0/2191915076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f0849bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.657+0000 7f1bfedea740 1 --2- 172.21.15.44:0/2191915076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f0849b860 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.657+0000 7f1bfedea740 1 -- 172.21.15.44:0/2191915076 >> 172.21.15.44:0/2191915076 conn(0x558f080ed300 msgr2=0x558f08498010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:50.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.657+0000 7f1bfedea740 1 -- 172.21.15.44:0/2191915076 shutdown_connections 2024-05-27T09:48:50.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.657+0000 7f1bfedea740 1 -- 172.21.15.44:0/2191915076 wait complete. 2024-05-27T09:48:50.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.658+0000 7f1bfedea740 1 -- start start 2024-05-27T09:48:50.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.659+0000 7f1bfedea740 1 Processor -- start 2024-05-27T09:48:50.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.659+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f084a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.659+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f084a5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.659+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f084b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bfedea740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558f080e2790 con 0x558f080d02f0 2024-05-27T09:48:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bfedea740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558f083a4bc0 con 0x558f080f91b0 2024-05-27T09:48:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bfedea740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558f084a8930 con 0x558f080d84d0 2024-05-27T09:48:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bfedea740 10 client.0 _ll_register_callbacks cb 0x558f08382400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bf5400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f084a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bf4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f084a5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bf4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f084a5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49776/0 (socket says 172.21.15.44:49776) 2024-05-27T09:48:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bf4a00640 1 -- 172.21.15.44:0/1388683698 learned_addr learned my addr 172.21.15.44:0/1388683698 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bf5e00640 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f084b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bf5e00640 1 -- 172.21.15.44:0/1388683698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 msgr2=0x558f084a5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bf5e00640 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f084a5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bf5e00640 1 -- 172.21.15.44:0/1388683698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 msgr2=0x558f084a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.660+0000 7f1bf5e00640 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f084a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.661+0000 7f1bf5e00640 1 -- 172.21.15.44:0/1388683698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f08111210 con 0x558f080f91b0 2024-05-27T09:48:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.661+0000 7f1bf5e00640 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f084b1c60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009ca0 tx=0x7f1be802f3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78e2ecad10b6b540 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.661+0000 7f1bd6a00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.661+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1388683698 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8052020 con 0x558f080f91b0 2024-05-27T09:48:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.661+0000 7f1bfedea740 1 -- 172.21.15.44:0/1388683698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1be8036040 con 0x558f080f91b0 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.661+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1388683698 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f1be8003970 con 0x558f080f91b0 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.662+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1388683698 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1be805d050 con 0x558f080f91b0 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.662+0000 7f1bfedea740 10 client.27797 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.662+0000 7f1bfedea740 1 -- 172.21.15.44:0/1388683698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x558f081dd620 con 0x558f080f91b0 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.662+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1388683698 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be8036040 con 0x558f080f91b0 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.662+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1388683698 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f1be804d030 con 0x558f080f91b0 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.663+0000 7f1bfedea740 10 client.27797 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.663+0000 7f1bfedea740 10 client.27797 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.663+0000 7f1bfedea740 20 client.27797 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.663+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f080b7d80 0x558f084b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.663+0000 7f1bfedea740 4 client.27797 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.663+0000 7f1bfedea740 1 -- 172.21.15.44:0/1388683698 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f0840eae0 con 0x558f080b7d80 2024-05-27T09:48:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.663+0000 7f1bf5400640 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f080b7d80 0x558f084b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.664+0000 7f1bd6a00640 10 client.27797 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.664+0000 7f1bf5400640 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f080b7d80 0x558f084b7a30 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25bdbb117f92b192 server_cookie=15b997a640406191 in_seq=0 out_seq=0 2024-05-27T09:48:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.665+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1388683698 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558f0840eae0 con 0x558f080b7d80 2024-05-27T09:48:50.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.665+0000 7f1bd6a00640 10 client.27797 handle_command_reply: tid=0 2024-05-27T09:48:50.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.665+0000 7f1bfedea740 1 client.27797 shutdown 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.665+0000 7f1bfedea740 1 -- 172.21.15.44:0/1388683698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 msgr2=0x558f084b1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.665+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f084b1c60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009ca0 tx=0x7f1be802f3a0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.666+0000 7f1bfedea740 1 -- 172.21.15.44:0/1388683698 shutdown_connections 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.666+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f080b7d80 0x558f084b7a30 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.666+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f084b1c60 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.666+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f084a5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.666+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1388683698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f084a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.666+0000 7f1bd6a00640 0 client.27797 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.666+0000 7f1bfedea740 1 -- 172.21.15.44:0/1388683698 >> 172.21.15.44:0/1388683698 conn(0x558f080ed300 msgr2=0x558f0849b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.666+0000 7f1bfedea740 1 -- 172.21.15.44:0/1388683698 shutdown_connections 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.666+0000 7f1bfedea740 1 -- 172.21.15.44:0/1388683698 wait complete. 2024-05-27T09:48:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.666+0000 7f1bfedea740 20 client.27797 trim_cache size 0 max 16384 2024-05-27T09:48:50.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.687+0000 7f1bfedea740 1 Processor -- start 2024-05-27T09:48:50.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bfedea740 1 -- start start 2024-05-27T09:48:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f084a1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f08497330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f084b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bfedea740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01b4780 con 0x558f080d02f0 2024-05-27T09:48:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bfedea740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01b1ff0 con 0x558f080f91b0 2024-05-27T09:48:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bfedea740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01afe60 con 0x558f080d84d0 2024-05-27T09:48:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bf4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f08497330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bf5e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f084b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bf5400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f084a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bf4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f08497330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49788/0 (socket says 172.21.15.44:49788) 2024-05-27T09:48:50.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.688+0000 7f1bf4a00640 1 -- 172.21.15.44:0/1021784987 learned_addr learned my addr 172.21.15.44:0/1021784987 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:50.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.689+0000 7f1bf4a00640 1 -- 172.21.15.44:0/1021784987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 msgr2=0x558f084b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.689+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/1021784987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f084b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.689+0000 7f1bf4a00640 1 -- 172.21.15.44:0/1021784987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 msgr2=0x558f084a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.689+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/1021784987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f084a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.689+0000 7f1bf4a00640 1 -- 172.21.15.44:0/1021784987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f0849d0b0 con 0x558f080d84d0 2024-05-27T09:48:50.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.689+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/1021784987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f08497330 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00cfd0 tx=0x7f1bdc009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5b65425f3c72047 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.689+0000 7f1be1c00640 1 -- 172.21.15.44:0/1021784987 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc0a9a80 con 0x558f080d84d0 2024-05-27T09:48:50.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.689+0000 7f1be1c00640 1 -- 172.21.15.44:0/1021784987 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bdc0eb070 con 0x558f080d84d0 2024-05-27T09:48:50.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.689+0000 7f1be1c00640 1 -- 172.21.15.44:0/1021784987 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc0a5440 con 0x558f080d84d0 2024-05-27T09:48:50.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.690+0000 7f1bfedea740 1 -- 172.21.15.44:0/1021784987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 msgr2=0x558f08497330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.690+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1021784987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f08497330 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00cfd0 tx=0x7f1bdc009310 comp rx=0 tx=0).stop 2024-05-27T09:48:50.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.691+0000 7f1bfedea740 1 -- 172.21.15.44:0/1021784987 shutdown_connections 2024-05-27T09:48:50.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.691+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1021784987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080f91b0 0x558f084b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.691+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1021784987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080d84d0 0x558f08497330 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.691+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1021784987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d02f0 0x558f084a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.691+0000 7f1bfedea740 1 -- 172.21.15.44:0/1021784987 >> 172.21.15.44:0/1021784987 conn(0x558f080ed300 msgr2=0x558f084a7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:50.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.691+0000 7f1bfedea740 1 -- 172.21.15.44:0/1021784987 shutdown_connections 2024-05-27T09:48:50.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.691+0000 7f1bfedea740 1 -- 172.21.15.44:0/1021784987 wait complete. 2024-05-27T09:48:50.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.691+0000 7f1bfedea740 1 -- start start 2024-05-27T09:48:50.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.691+0000 7f1bfedea740 1 Processor -- start 2024-05-27T09:48:50.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.692+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080d02f0 0x558f084b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.692+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d84d0 0x558f084b8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.692+0000 7f1bfedea740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080f91b0 0x558f084b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.692+0000 7f1bfedea740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01b04b0 con 0x558f080d84d0 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.692+0000 7f1bfedea740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01b1920 con 0x558f080d02f0 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.692+0000 7f1bfedea740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01b0b80 con 0x558f080f91b0 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.692+0000 7f1bfedea740 10 client.0 _ll_register_callbacks cb 0x7f1bdc0e7040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.692+0000 7f1bf5400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080d02f0 0x558f084b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.692+0000 7f1bf5400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080d02f0 0x558f084b89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56136/0 (socket says 172.21.15.44:56136) 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.692+0000 7f1bf5400640 1 -- 172.21.15.44:0/1107532354 learned_addr learned my addr 172.21.15.44:0/1107532354 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.692+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d84d0 0x558f084b8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.693+0000 7f1bf5e00640 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080f91b0 0x558f084b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.693+0000 7f1bf5400640 1 -- 172.21.15.44:0/1107532354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080f91b0 msgr2=0x558f084b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.693+0000 7f1bf5400640 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080f91b0 0x558f084b9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.693+0000 7f1bf5400640 1 -- 172.21.15.44:0/1107532354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d84d0 msgr2=0x558f084b8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.693+0000 7f1bf5400640 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d84d0 0x558f084b8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.693+0000 7f1bf5400640 1 -- 172.21.15.44:0/1107532354 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f0849cc40 con 0x558f080d02f0 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.693+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d84d0 0x558f084b8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.693+0000 7f1bf5400640 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080d02f0 0x558f084b89d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x558f08499350 tx=0x7f1be40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19330927e6707910 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.693+0000 7f1bd6a00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.694+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1107532354 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4004020 con 0x558f080d02f0 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.694+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1107532354 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f1be40041c0 con 0x558f080d02f0 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.694+0000 7f1bfedea740 1 -- 172.21.15.44:0/1107532354 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bdc0eb070 con 0x558f080d02f0 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.694+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1107532354 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1be40043c0 con 0x558f080d02f0 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.694+0000 7f1bfedea740 10 client.27803 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:50.696 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.694+0000 7f1bfedea740 1 -- 172.21.15.44:0/1107532354 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x558f08520100 con 0x558f080d02f0 2024-05-27T09:48:50.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.694+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1107532354 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bdc0eb070 con 0x558f080d02f0 2024-05-27T09:48:50.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.695+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1107532354 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f1be4032070 con 0x558f080d02f0 2024-05-27T09:48:50.697 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.695+0000 7f1bfedea740 10 client.27803 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:50.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.695+0000 7f1bfedea740 10 client.27803 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:50.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.695+0000 7f1bfedea740 20 client.27803 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:50.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.695+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f0808d1c0 0x558f085228c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.695+0000 7f1bfedea740 4 client.27803 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:48:50.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.695+0000 7f1bfedea740 1 -- 172.21.15.44:0/1107532354 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558f084082c0 con 0x558f0808d1c0 2024-05-27T09:48:50.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.695+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f0808d1c0 0x558f085228c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.696+0000 7f1bf4a00640 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f0808d1c0 0x558f085228c0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d26ad8767faa9e14 server_cookie=a2316c8b2cd734bd in_seq=0 out_seq=0 2024-05-27T09:48:50.698 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:50.696+0000 7f1bd6a00640 10 client.27803 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.889+0000 7f3e06400700 1 -- 172.21.15.44:0/1166567237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e00106eb0 msgr2=0x7f3e00113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.889+0000 7f3e06400700 1 --2- 172.21.15.44:0/1166567237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e00106eb0 0x7f3e00113340 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009f90 tx=0x7f3df800b330 comp rx=0 tx=0).stop 2024-05-27T09:48:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.889+0000 7f3e06400700 1 -- 172.21.15.44:0/1166567237 shutdown_connections 2024-05-27T09:48:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.889+0000 7f3e06400700 1 --2- 172.21.15.44:0/1166567237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e00106eb0 0x7f3e00113340 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.889+0000 7f3e06400700 1 --2- 172.21.15.44:0/1166567237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e0010c5a0 0x7f3e0010c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.889+0000 7f3e06400700 1 --2- 172.21.15.44:0/1166567237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e00106560 0x7f3e00106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.889+0000 7f3e06400700 1 -- 172.21.15.44:0/1166567237 >> 172.21.15.44:0/1166567237 conn(0x7f3e00102080 msgr2=0x7f3e001044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.890+0000 7f3e06400700 1 -- 172.21.15.44:0/1166567237 shutdown_connections 2024-05-27T09:48:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.890+0000 7f3e06400700 1 -- 172.21.15.44:0/1166567237 wait complete. 2024-05-27T09:48:50.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.892+0000 7f3e06400700 1 Processor -- start 2024-05-27T09:48:50.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.892+0000 7f3e06400700 1 -- start start 2024-05-27T09:48:50.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.893+0000 7f3e06400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e00106560 0x7f3e00077cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.893+0000 7f3e06400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e00106eb0 0x7f3e00078200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.893+0000 7f3e06400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e0010c5a0 0x7f3e00078740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.893+0000 7f3e06400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001166d0 con 0x7f3e00106eb0 2024-05-27T09:48:50.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.893+0000 7f3e06400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3e00116550 con 0x7f3e0010c5a0 2024-05-27T09:48:50.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.893+0000 7f3e06400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e00116850 con 0x7f3e00106560 2024-05-27T09:48:50.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.893+0000 7f3dffe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e0010c5a0 0x7f3e00078740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.894+0000 7f3dffe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e0010c5a0 0x7f3e00078740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56140/0 (socket says 172.21.15.44:56140) 2024-05-27T09:48:50.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.894+0000 7f3dffe00700 1 -- 172.21.15.44:0/2523522620 learned_addr learned my addr 172.21.15.44:0/2523522620 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:50.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.894+0000 7f3dff400700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e00106560 0x7f3e00077cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.894+0000 7f3dffe00700 1 -- 172.21.15.44:0/2523522620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e00106560 msgr2=0x7f3e00077cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.894+0000 7f3dffe00700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e00106560 0x7f3e00077cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.894+0000 7f3dffe00700 1 -- 172.21.15.44:0/2523522620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e00106eb0 msgr2=0x7f3e00078200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.894+0000 7f3dfea00700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e00106eb0 0x7f3e00078200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.894+0000 7f3dffe00700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e00106eb0 0x7f3e00078200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.894+0000 7f3dffe00700 1 -- 172.21.15.44:0/2523522620 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e001ad500 con 0x7f3e0010c5a0 2024-05-27T09:48:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.895+0000 7f3dfea00700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e00106eb0 0x7f3e00078200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:48:50.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.896+0000 7f3dffe00700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e0010c5a0 0x7f3e00078740 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009790 tx=0x7f3df800cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.897+0000 7f3df7e00700 1 -- 172.21.15.44:0/2523522620 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8021070 con 0x7f3e0010c5a0 2024-05-27T09:48:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.897+0000 7f3df7e00700 1 -- 172.21.15.44:0/2523522620 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3df8007ac0 con 0x7f3e0010c5a0 2024-05-27T09:48:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.897+0000 7f3df7e00700 1 -- 172.21.15.44:0/2523522620 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df80052a0 con 0x7f3e0010c5a0 2024-05-27T09:48:50.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.897+0000 7f3e06400700 1 -- 172.21.15.44:0/2523522620 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e001ad730 con 0x7f3e0010c5a0 2024-05-27T09:48:50.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.897+0000 7f3e06400700 1 -- 172.21.15.44:0/2523522620 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e001adc30 con 0x7f3e0010c5a0 2024-05-27T09:48:50.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.905+0000 7f3e14e89b80 1 Processor -- start 2024-05-27T09:48:50.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.906+0000 7f3df7e00700 1 -- 172.21.15.44:0/2523522620 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3df8024070 con 0x7f3e0010c5a0 2024-05-27T09:48:50.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.907+0000 7f3e14e89b80 1 -- start start 2024-05-27T09:48:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.910+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda33ce40 0x560fda4c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.910+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4c1d30 0x560fda4bb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.910+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4bc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.910+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560fda3e9e40 con 0x560fda4c1d30 2024-05-27T09:48:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.910+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560fda3d99e0 con 0x560fda33ce40 2024-05-27T09:48:50.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.910+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560fda3dda20 con 0x560fda4bbdb0 2024-05-27T09:48:50.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.911+0000 7f3dfea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4bc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.911+0000 7f3dfea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4bc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49828/0 (socket says 172.21.15.44:49828) 2024-05-27T09:48:50.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.911+0000 7f3dfea00700 1 -- 172.21.15.44:0/2941800514 learned_addr learned my addr 172.21.15.44:0/2941800514 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:50.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.911+0000 7f3dfea00700 1 --2- 172.21.15.44:0/2941800514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda33ce40 0x560fda4c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.911+0000 7f3dff400700 1 --2- 172.21.15.44:0/2941800514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4c1d30 0x560fda4bb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.912+0000 7f3dff400700 1 -- 172.21.15.44:0/2941800514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 msgr2=0x560fda4bc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.912+0000 7f3df7e00700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3de40b2af0 0x7f3de40b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.912+0000 7f3dff400700 1 --2- 172.21.15.44:0/2941800514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4bc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.912+0000 7f3dff400700 1 -- 172.21.15.44:0/2941800514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda33ce40 msgr2=0x560fda4c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.912+0000 7f3dff400700 1 --2- 172.21.15.44:0/2941800514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda33ce40 0x560fda4c17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.913+0000 7f3dff400700 1 -- 172.21.15.44:0/2941800514 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fda3a2420 con 0x560fda4c1d30 2024-05-27T09:48:50.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.913+0000 7f3dfea00700 1 --2- 172.21.15.44:0/2941800514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4bc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:50.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.913+0000 7f3df7e00700 1 -- 172.21.15.44:0/2523522620 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f3df800f020 con 0x7f3e0010c5a0 2024-05-27T09:48:50.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.913+0000 7f3dffe00700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3de40b2af0 0x7f3de40b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.914+0000 7f3dff400700 1 --2- 172.21.15.44:0/2941800514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4c1d30 0x560fda4bb870 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3df0005fd0 tx=0x7f3df0011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c80cae2a50d2d82 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.915+0000 7f3df4c00700 1 -- 172.21.15.44:0/2941800514 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df0004250 con 0x560fda4c1d30 2024-05-27T09:48:50.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.915+0000 7f3dffe00700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3de40b2af0 0x7f3de40b5000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3df80a6df0 tx=0x7f3df80cbf90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.915+0000 7f3df4c00700 1 -- 172.21.15.44:0/2941800514 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3df00043f0 con 0x560fda4c1d30 2024-05-27T09:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.917+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2941800514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4c1d30 msgr2=0x560fda4bb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.917+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/2941800514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4c1d30 0x560fda4bb870 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3df0005fd0 tx=0x7f3df0011720 comp rx=0 tx=0).stop 2024-05-27T09:48:50.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.917+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2941800514 shutdown_connections 2024-05-27T09:48:50.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.917+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/2941800514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4bc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.917+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/2941800514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda33ce40 0x560fda4c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.917+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/2941800514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4c1d30 0x560fda4bb870 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.917+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2941800514 >> 172.21.15.44:0/2941800514 conn(0x560fda39cfc0 msgr2=0x560fda4b68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:50.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.917+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2941800514 shutdown_connections 2024-05-27T09:48:50.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.917+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2941800514 wait complete. 2024-05-27T09:48:50.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.918+0000 7f3e14e89b80 1 -- start start 2024-05-27T09:48:50.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.918+0000 7f3e14e89b80 1 Processor -- start 2024-05-27T09:48:50.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.920+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda33ce40 0x560fda4c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.920+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4bbdb0 0x560fda4c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.920+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.920+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560fda2dffa0 con 0x560fda4bbdb0 2024-05-27T09:48:50.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.920+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560fda37a760 con 0x560fda4c1d30 2024-05-27T09:48:50.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.920+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560fda4c8800 con 0x560fda33ce40 2024-05-27T09:48:50.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.920+0000 7f3e14e89b80 10 client.0 _ll_register_callbacks cb 0x560fda383c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:50.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.921+0000 7f3dff400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4bbdb0 0x560fda4c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.921+0000 7f3dff400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4bbdb0 0x560fda4c58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59600/0 (socket says 172.21.15.44:59600) 2024-05-27T09:48:50.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.922+0000 7f3dff400700 1 -- 172.21.15.44:0/1365820743 learned_addr learned my addr 172.21.15.44:0/1365820743 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:50.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.921+0000 7f3dfea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda33ce40 0x560fda4c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.922+0000 7f3dfea00700 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.923+0000 7f3dfea00700 1 -- 172.21.15.44:0/1365820743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 msgr2=0x560fda4c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.923+0000 7f3dfea00700 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4c5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.923+0000 7f3dfea00700 1 -- 172.21.15.44:0/1365820743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4bbdb0 msgr2=0x560fda4c58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.923+0000 7f3dfea00700 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4bbdb0 0x560fda4c58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.923+0000 7f3dfea00700 1 -- 172.21.15.44:0/1365820743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fda3298d0 con 0x560fda33ce40 2024-05-27T09:48:50.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.923+0000 7f3dfea00700 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda33ce40 0x560fda4c53b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x560fda4bcb30 tx=0x7f3dec0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3bc95d93535070d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.923+0000 7f3de9600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:48:50.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.924+0000 7f3de9600700 1 -- 172.21.15.44:0/1365820743 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec00fe10 con 0x560fda33ce40 2024-05-27T09:48:50.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.924+0000 7f3de9600700 1 -- 172.21.15.44:0/1365820743 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f3dec011440 con 0x560fda33ce40 2024-05-27T09:48:50.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.924+0000 7f3e14e89b80 1 -- 172.21.15.44:0/1365820743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3df00043f0 con 0x560fda33ce40 2024-05-27T09:48:50.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.924+0000 7f3de9600700 1 -- 172.21.15.44:0/1365820743 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3dec011640 con 0x560fda33ce40 2024-05-27T09:48:50.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.924+0000 7f3e14e89b80 10 client.27817 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:50.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.924+0000 7f3e14e89b80 1 -- 172.21.15.44:0/1365820743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x560fda3aa060 con 0x560fda33ce40 2024-05-27T09:48:50.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.925+0000 7f3de9600700 1 -- 172.21.15.44:0/1365820743 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3df00043f0 con 0x560fda33ce40 2024-05-27T09:48:50.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.926+0000 7f3de9600700 1 -- 172.21.15.44:0/1365820743 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f3dec021710 con 0x560fda33ce40 2024-05-27T09:48:50.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.926+0000 7f3e14e89b80 10 client.27817 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:50.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.926+0000 7f3e14e89b80 10 client.27817 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:48:50.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.926+0000 7f3e14e89b80 20 client.27817 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:50.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.926+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560fda4d56d0 0x560fda4d7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.926+0000 7f3e14e89b80 4 client.27817 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:50.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.926+0000 7f3e14e89b80 1 -- 172.21.15.44:0/1365820743 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fda3ebd90 con 0x560fda4d56d0 2024-05-27T09:48:50.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.926+0000 7f3dff400700 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560fda4d56d0 0x560fda4d7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.927+0000 7f3dff400700 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560fda4d56d0 0x560fda4d7b50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9af6efda28208f29 server_cookie=9626791117151779 in_seq=0 out_seq=0 2024-05-27T09:48:50.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.927+0000 7f3de9600700 10 client.27817 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:48:50.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.929+0000 7f3de9600700 1 -- 172.21.15.44:0/1365820743 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560fda3ebd90 con 0x560fda4d56d0 2024-05-27T09:48:50.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.929+0000 7f3de9600700 10 client.27817 handle_command_reply: tid=0 2024-05-27T09:48:50.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.929+0000 7f3e14e89b80 1 client.27817 shutdown 2024-05-27T09:48:50.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.930+0000 7f3e14e89b80 1 -- 172.21.15.44:0/1365820743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda33ce40 msgr2=0x560fda4c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.930+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda33ce40 0x560fda4c53b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x560fda4bcb30 tx=0x7f3dec0047c0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.930+0000 7f3e14e89b80 1 -- 172.21.15.44:0/1365820743 shutdown_connections 2024-05-27T09:48:50.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.930+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.930+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda33ce40 0x560fda4c53b0 secure :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0x560fda4bcb30 tx=0x7f3dec0047c0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.930+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560fda4d56d0 0x560fda4d7b50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.930+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/1365820743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4bbdb0 0x560fda4c58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.930+0000 7f3e14e89b80 1 -- 172.21.15.44:0/1365820743 >> 172.21.15.44:0/1365820743 conn(0x560fda39cfc0 msgr2=0x560fda4b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:50.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.930+0000 7f3de9600700 0 client.27817 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:48:50.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.931+0000 7f3e14e89b80 1 -- 172.21.15.44:0/1365820743 shutdown_connections 2024-05-27T09:48:50.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.931+0000 7f3e14e89b80 1 -- 172.21.15.44:0/1365820743 wait complete. 2024-05-27T09:48:50.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.931+0000 7f3e14e89b80 20 client.27817 trim_cache size 0 max 16384 2024-05-27T09:48:50.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.958+0000 7f3e14e89b80 1 Processor -- start 2024-05-27T09:48:50.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.958+0000 7f3e14e89b80 1 -- start start 2024-05-27T09:48:50.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4bb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 0x560fda4bf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001b7d00 con 0x560fda4b6b60 2024-05-27T09:48:50.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001b5530 con 0x560fda4c1d30 2024-05-27T09:48:50.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001b33a0 con 0x560fda4bbdb0 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3dfea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4bb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3dff400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3dfea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4bb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49858/0 (socket says 172.21.15.44:49858) 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3dfea00700 1 -- 172.21.15.44:0/3907495429 learned_addr learned my addr 172.21.15.44:0/3907495429 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3dff400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56166/0 (socket says 172.21.15.44:56166) 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3dfea00700 1 --2- 172.21.15.44:0/3907495429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 0x560fda4bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3dff400700 1 -- 172.21.15.44:0/3907495429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 msgr2=0x560fda4bb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3dff400700 1 --2- 172.21.15.44:0/3907495429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4bb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.959+0000 7f3dff400700 1 -- 172.21.15.44:0/3907495429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 msgr2=0x560fda4bf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.960+0000 7f3dff400700 1 --2- 172.21.15.44:0/3907495429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 0x560fda4bf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.960+0000 7f3dff400700 1 -- 172.21.15.44:0/3907495429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fda4bfe60 con 0x560fda4c1d30 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.960+0000 7f3dff400700 1 --2- 172.21.15.44:0/3907495429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4c7850 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3df0000c00 tx=0x7f3df0005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e250c253becca0ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.960+0000 7f3debe00700 1 -- 172.21.15.44:0/3907495429 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df0003db0 con 0x560fda4c1d30 2024-05-27T09:48:50.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.960+0000 7f3debe00700 1 -- 172.21.15.44:0/3907495429 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3df0019970 con 0x560fda4c1d30 2024-05-27T09:48:50.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.960+0000 7f3debe00700 1 -- 172.21.15.44:0/3907495429 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df0003870 con 0x560fda4c1d30 2024-05-27T09:48:50.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.961+0000 7f3e14e89b80 1 -- 172.21.15.44:0/3907495429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 msgr2=0x560fda4c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.961+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/3907495429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4c7850 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3df0000c00 tx=0x7f3df0005f10 comp rx=0 tx=0).stop 2024-05-27T09:48:50.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.961+0000 7f3e14e89b80 1 -- 172.21.15.44:0/3907495429 shutdown_connections 2024-05-27T09:48:50.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.961+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/3907495429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 0x560fda4bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.961+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/3907495429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4bb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.961+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/3907495429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4c7850 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.961+0000 7f3e14e89b80 1 -- 172.21.15.44:0/3907495429 >> 172.21.15.44:0/3907495429 conn(0x560fda39cfc0 msgr2=0x560fda4bf210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:50.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.962+0000 7f3e14e89b80 1 -- 172.21.15.44:0/3907495429 shutdown_connections 2024-05-27T09:48:50.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.962+0000 7f3e14e89b80 1 -- 172.21.15.44:0/3907495429 wait complete. 2024-05-27T09:48:50.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.962+0000 7f3e14e89b80 1 -- start start 2024-05-27T09:48:50.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.963+0000 7f3e14e89b80 1 Processor -- start 2024-05-27T09:48:50.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.963+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 0x560fda4ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.963+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4dca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.963+0000 7f3e14e89b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4dcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.963+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001b39f0 con 0x560fda4b6b60 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.963+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001b4e60 con 0x560fda4c1d30 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.963+0000 7f3e14e89b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001b40c0 con 0x560fda4bbdb0 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.963+0000 7f3e14e89b80 10 client.0 _ll_register_callbacks cb 0x560fda383c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.963+0000 7f3dfea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 0x560fda4ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3dfea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 0x560fda4ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59612/0 (socket says 172.21.15.44:59612) 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3dfea00700 1 -- 172.21.15.44:0/2874081768 learned_addr learned my addr 172.21.15.44:0/2874081768 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3dfea00700 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4dcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3dff400700 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4dca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3dfea00700 1 -- 172.21.15.44:0/2874081768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 msgr2=0x560fda4dca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3dfea00700 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4dca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3dfea00700 1 -- 172.21.15.44:0/2874081768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 msgr2=0x560fda4dcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:50.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3dfea00700 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4dcf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3dfea00700 1 -- 172.21.15.44:0/2874081768 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fda4dd590 con 0x560fda4b6b60 2024-05-27T09:48:50.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3dff400700 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4dca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:50.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3dfea00700 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 0x560fda4ce0e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3dec009590 tx=0x7f3dec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae78d9dc4682d84 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:50.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.964+0000 7f3de9600700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:48:50.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.965+0000 7f3de9600700 1 -- 172.21.15.44:0/2874081768 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec02d040 con 0x560fda4b6b60 2024-05-27T09:48:50.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.965+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2874081768 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3df0019970 con 0x560fda4b6b60 2024-05-27T09:48:50.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.965+0000 7f3de9600700 1 -- 172.21.15.44:0/2874081768 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f3dec021840 con 0x560fda4b6b60 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.965+0000 7f3de9600700 1 -- 172.21.15.44:0/2874081768 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3dec004030 con 0x560fda4b6b60 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.965+0000 7f3e14e89b80 10 client.17553 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.965+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2874081768 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x560fda53ffb0 con 0x560fda4b6b60 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.966+0000 7f3de9600700 1 -- 172.21.15.44:0/2874081768 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3df0019970 con 0x560fda4b6b60 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.966+0000 7f3de9600700 1 -- 172.21.15.44:0/2874081768 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f3dec033070 con 0x560fda4b6b60 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.966+0000 7f3e14e89b80 10 client.17553 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.966+0000 7f3e14e89b80 10 client.17553 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.966+0000 7f3e14e89b80 20 client.17553 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.966+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560fda540630 0x560fda542a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.966+0000 7f3e14e89b80 4 client.17553 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.966+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2874081768 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560fda3e5570 con 0x560fda540630 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.966+0000 7f3dff400700 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560fda540630 0x560fda542a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:50.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3dff400700 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560fda540630 0x560fda542a50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc362e01a138276c server_cookie=8f1238efae696207 in_seq=0 out_seq=0 2024-05-27T09:48:50.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3de9600700 10 client.17553 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:48:50.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3de9600700 1 -- 172.21.15.44:0/2874081768 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560fda3e5570 con 0x560fda540630 2024-05-27T09:48:50.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3de9600700 10 client.17553 handle_command_reply: tid=0 2024-05-27T09:48:50.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3e14e89b80 1 client.17553 shutdown 2024-05-27T09:48:50.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2874081768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 msgr2=0x560fda4ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 0x560fda4ce0e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3dec009590 tx=0x7f3dec009bb0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2874081768 shutdown_connections 2024-05-27T09:48:50.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560fda4c1d30 0x560fda4dcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x560fda540630 0x560fda542a50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560fda4b6b60 0x560fda4ce0e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3e14e89b80 1 --2- 172.21.15.44:0/2874081768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fda4bbdb0 0x560fda4dca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2874081768 >> 172.21.15.44:0/2874081768 conn(0x560fda39cfc0 msgr2=0x560fda4c0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:50.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.967+0000 7f3de9600700 0 client.17553 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:48:50.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.968+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2874081768 shutdown_connections 2024-05-27T09:48:50.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.968+0000 7f3e14e89b80 1 -- 172.21.15.44:0/2874081768 wait complete. 2024-05-27T09:48:50.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.968+0000 7f3e14e89b80 20 client.17553 trim_cache size 0 max 16384 2024-05-27T09:48:50.971 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:48:50.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 -- 172.21.15.44:0/2523522620 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3de40b2af0 msgr2=0x7f3de40b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3de40b2af0 0x7f3de40b5000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3df80a6df0 tx=0x7f3df80cbf90 comp rx=0 tx=0).stop 2024-05-27T09:48:50.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 -- 172.21.15.44:0/2523522620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e0010c5a0 msgr2=0x7f3e00078740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:50.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e0010c5a0 0x7f3e00078740 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009790 tx=0x7f3df800cc30 comp rx=0 tx=0).stop 2024-05-27T09:48:50.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 -- 172.21.15.44:0/2523522620 shutdown_connections 2024-05-27T09:48:50.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f3de40b2af0 0x7f3de40b5000 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e0010c5a0 0x7f3e00078740 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e00106560 0x7f3e00077cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 --2- 172.21.15.44:0/2523522620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e00106eb0 0x7f3e00078200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:50.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 -- 172.21.15.44:0/2523522620 >> 172.21.15.44:0/2523522620 conn(0x7f3e00102080 msgr2=0x7f3e001046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:50.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 -- 172.21.15.44:0/2523522620 shutdown_connections 2024-05-27T09:48:50.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:50.969+0000 7f3e06400700 1 -- 172.21.15.44:0/2523522620 wait complete. 2024-05-27T09:48:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bd6a00640 1 -- 172.21.15.44:0/1107532354 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x558f084082c0 con 0x558f0808d1c0 2024-05-27T09:48:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bd6a00640 10 client.27803 handle_command_reply: tid=0 2024-05-27T09:48:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bfedea740 1 client.27803 shutdown 2024-05-27T09:48:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bfedea740 1 -- 172.21.15.44:0/1107532354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080d02f0 msgr2=0x558f084b89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080d02f0 0x558f084b89d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x558f08499350 tx=0x7f1be40046c0 comp rx=0 tx=0).stop 2024-05-27T09:48:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bfedea740 1 -- 172.21.15.44:0/1107532354 shutdown_connections 2024-05-27T09:48:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558f0808d1c0 0x558f085228c0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f080f91b0 0x558f084b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558f080d84d0 0x558f084b8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bfedea740 1 --2- 172.21.15.44:0/1107532354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558f080d02f0 0x558f084b89d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bd6a00640 0 client.27803 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.031+0000 7f1bfedea740 1 -- 172.21.15.44:0/1107532354 >> 172.21.15.44:0/1107532354 conn(0x558f080ed300 msgr2=0x558f084a01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.032+0000 7f1bfedea740 1 -- 172.21.15.44:0/1107532354 shutdown_connections 2024-05-27T09:48:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.032+0000 7f1bfedea740 1 -- 172.21.15.44:0/1107532354 wait complete. 2024-05-27T09:48:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.032+0000 7f1bfedea740 20 client.27803 trim_cache size 0 max 16384 2024-05-27T09:48:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:48:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:48:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:48:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.032+0000 7f1bf7e00640 1 -- 172.21.15.44:0/1711794180 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1bd80b29d0 msgr2=0x7f1bd80b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.032+0000 7f1bf7e00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1bd80b29d0 0x7f1bd80b4ee0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x558f0849cc10 tx=0x7f1be4005f90 comp rx=0 tx=0).stop 2024-05-27T09:48:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.032+0000 7f1bf7e00640 1 -- 172.21.15.44:0/1711794180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf0108b90 msgr2=0x7f1bf01a2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.032+0000 7f1bf7e00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf0108b90 0x7f1bf01a2bb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1bf006c640 tx=0x7f1bdc00ee50 comp rx=0 tx=0).stop 2024-05-27T09:48:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.033+0000 7f1bf7e00640 1 -- 172.21.15.44:0/1711794180 shutdown_connections 2024-05-27T09:48:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.033+0000 7f1bf7e00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f1bd80b29d0 0x7f1bd80b4ee0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.033+0000 7f1bf7e00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bf0109030 0x7f1bf019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.033+0000 7f1bf7e00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf0108b90 0x7f1bf01a2bb0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.033+0000 7f1bf7e00640 1 --2- 172.21.15.44:0/1711794180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf01032e0 0x7f1bf01a2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.033+0000 7f1bf7e00640 1 -- 172.21.15.44:0/1711794180 >> 172.21.15.44:0/1711794180 conn(0x7f1bf00782d0 msgr2=0x7f1bf0100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.033+0000 7f1bf7e00640 1 -- 172.21.15.44:0/1711794180 shutdown_connections 2024-05-27T09:48:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:48:51.033+0000 7f1bf7e00640 1 -- 172.21.15.44:0/1711794180 wait complete. 2024-05-27T09:48:51.813 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:48:52.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:51 smithi044 ceph-mon[31618]: pgmap v428: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 415 op/s 2024-05-27T09:48:52.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:51 smithi050 ceph-mon[36371]: pgmap v428: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 415 op/s 2024-05-27T09:48:52.307 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:48:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:51 smithi189 ceph-mon[36671]: pgmap v428: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 415 op/s 2024-05-27T09:48:54.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:53 smithi044 ceph-mon[31618]: pgmap v429: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1001 KiB/s wr, 177 op/s 2024-05-27T09:48:54.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:53 smithi050 ceph-mon[36371]: pgmap v429: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1001 KiB/s wr, 177 op/s 2024-05-27T09:48:54.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:53 smithi189 ceph-mon[36671]: pgmap v429: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1001 KiB/s wr, 177 op/s 2024-05-27T09:48:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.531+0000 7fac38000700 1 -- 172.21.15.44:0/3308857198 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac30075af0 msgr2=0x7fac30075f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.531+0000 7fac38000700 1 --2- 172.21.15.44:0/3308857198 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac30075af0 0x7fac30075f00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fac24009990 tx=0x7fac24009ca0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.531+0000 7fac38000700 1 -- 172.21.15.44:0/3308857198 shutdown_connections 2024-05-27T09:48:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.531+0000 7fac38000700 1 --2- 172.21.15.44:0/3308857198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac30110c10 0x7fac30113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.531+0000 7fac38000700 1 --2- 172.21.15.44:0/3308857198 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac30075af0 0x7fac30075f00 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.531+0000 7fac38000700 1 --2- 172.21.15.44:0/3308857198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac30076440 0x7fac301106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.531+0000 7fac38000700 1 -- 172.21.15.44:0/3308857198 >> 172.21.15.44:0/3308857198 conn(0x7fac3007af60 msgr2=0x7fac3007b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.531+0000 7fac38000700 1 -- 172.21.15.44:0/3308857198 shutdown_connections 2024-05-27T09:48:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.532+0000 7fac38000700 1 -- 172.21.15.44:0/3308857198 wait complete. 2024-05-27T09:48:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.533+0000 7fac38000700 1 Processor -- start 2024-05-27T09:48:54.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.533+0000 7fac38000700 1 -- start start 2024-05-27T09:48:54.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.534+0000 7fac38000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac30075af0 0x7fac30104330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.534+0000 7fac38000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac30076440 0x7fac30104870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.534+0000 7fac38000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac30110c10 0x7fac30104db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.534+0000 7fac38000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fac301166d0 con 0x7fac30110c10 2024-05-27T09:48:54.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.534+0000 7fac38000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fac30116550 con 0x7fac30075af0 2024-05-27T09:48:54.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.534+0000 7fac38000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac30116850 con 0x7fac30076440 2024-05-27T09:48:54.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.534+0000 7fac35600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac30075af0 0x7fac30104330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.534+0000 7fac35600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac30075af0 0x7fac30104330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56236/0 (socket says 172.21.15.44:56236) 2024-05-27T09:48:54.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.534+0000 7fac35600700 1 -- 172.21.15.44:0/1375378156 learned_addr learned my addr 172.21.15.44:0/1375378156 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:54.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.534+0000 7fac34c00700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac30076440 0x7fac30104870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.534+0000 7fac36000700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac30110c10 0x7fac30104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.535+0000 7fac35600700 1 -- 172.21.15.44:0/1375378156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac30076440 msgr2=0x7fac30104870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:54.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.535+0000 7fac35600700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac30076440 0x7fac30104870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.535+0000 7fac35600700 1 -- 172.21.15.44:0/1375378156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac30110c10 msgr2=0x7fac30104db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:54.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.535+0000 7fac35600700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac30110c10 0x7fac30104db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.535+0000 7fac35600700 1 -- 172.21.15.44:0/1375378156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac301ad6e0 con 0x7fac30075af0 2024-05-27T09:48:54.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.535+0000 7fac35600700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac30075af0 0x7fac30104330 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fac24000c00 tx=0x7fac240057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:54.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.536+0000 7fac2e000700 1 -- 172.21.15.44:0/1375378156 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2400fad0 con 0x7fac30075af0 2024-05-27T09:48:54.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.536+0000 7fac38000700 1 -- 172.21.15.44:0/1375378156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac301ad910 con 0x7fac30075af0 2024-05-27T09:48:54.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.536+0000 7fac2e000700 1 -- 172.21.15.44:0/1375378156 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac2400fc70 con 0x7fac30075af0 2024-05-27T09:48:54.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.536+0000 7fac2e000700 1 -- 172.21.15.44:0/1375378156 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac24022e60 con 0x7fac30075af0 2024-05-27T09:48:54.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.536+0000 7fac38000700 1 -- 172.21.15.44:0/1375378156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac301add00 con 0x7fac30075af0 2024-05-27T09:48:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.540+0000 7fac2e000700 1 -- 172.21.15.44:0/1375378156 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fac24009d10 con 0x7fac30075af0 2024-05-27T09:48:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.541+0000 7fac46be3b80 1 Processor -- start 2024-05-27T09:48:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.541+0000 7fac46be3b80 1 -- start start 2024-05-27T09:48:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.542+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84fdede40 0x55b84ff727f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.542+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 0x55b84ff6c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.542+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff6cdb0 0x55b84ff6d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.542+0000 7fac46be3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b84fe9ae40 con 0x55b84ff6cdb0 2024-05-27T09:48:54.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.542+0000 7fac46be3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b84fe8a9e0 con 0x55b84ff72d30 2024-05-27T09:48:54.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.542+0000 7fac46be3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b84fe8ea20 con 0x55b84fdede40 2024-05-27T09:48:54.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.542+0000 7fac35600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 0x55b84ff6c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.542+0000 7fac2e000700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fac180b29f0 0x7fac180b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.542+0000 7fac35600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 0x55b84ff6c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56250/0 (socket says 172.21.15.44:56250) 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.542+0000 7fac35600700 1 -- 172.21.15.44:0/2823517970 learned_addr learned my addr 172.21.15.44:0/2823517970 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.542+0000 7fac34c00700 1 --2- 172.21.15.44:0/2823517970 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff6cdb0 0x55b84ff6d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.543+0000 7fac36000700 1 --2- 172.21.15.44:0/2823517970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84fdede40 0x55b84ff727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.543+0000 7fac2e000700 1 -- 172.21.15.44:0/1375378156 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7fac240e3620 con 0x7fac30075af0 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.543+0000 7fac35600700 1 -- 172.21.15.44:0/2823517970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84fdede40 msgr2=0x55b84ff727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.543+0000 7fac35600700 1 --2- 172.21.15.44:0/2823517970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84fdede40 0x55b84ff727f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.543+0000 7fac35600700 1 -- 172.21.15.44:0/2823517970 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff6cdb0 msgr2=0x55b84ff6d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.543+0000 7fac35600700 1 --2- 172.21.15.44:0/2823517970 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff6cdb0 0x55b84ff6d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.543+0000 7fac35600700 1 -- 172.21.15.44:0/2823517970 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b84fe53420 con 0x55b84ff72d30 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.543+0000 7fac35600700 1 --2- 172.21.15.44:0/2823517970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 0x55b84ff6c870 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fac24021000 tx=0x7fac24009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f2151b2ec72951a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.543+0000 7fac36000700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fac180b29f0 0x7fac180b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.544+0000 7fac22a00700 1 -- 172.21.15.44:0/2823517970 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac240aa3f0 con 0x55b84ff72d30 2024-05-27T09:48:54.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.544+0000 7fac22a00700 1 -- 172.21.15.44:0/2823517970 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac240aafe0 con 0x55b84ff72d30 2024-05-27T09:48:54.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.544+0000 7fac22a00700 1 -- 172.21.15.44:0/2823517970 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac240b1110 con 0x55b84ff72d30 2024-05-27T09:48:54.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.545+0000 7fac46be3b80 1 -- 172.21.15.44:0/2823517970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 msgr2=0x55b84ff6c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:54.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.545+0000 7fac36000700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fac180b29f0 0x7fac180b4f00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fac28003fd0 tx=0x7fac28004b40 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:54.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.545+0000 7fac46be3b80 1 --2- 172.21.15.44:0/2823517970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 0x55b84ff6c870 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fac24021000 tx=0x7fac24009ca0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.545+0000 7fac46be3b80 1 -- 172.21.15.44:0/2823517970 shutdown_connections 2024-05-27T09:48:54.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.545+0000 7fac46be3b80 1 --2- 172.21.15.44:0/2823517970 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff6cdb0 0x55b84ff6d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.545+0000 7fac46be3b80 1 --2- 172.21.15.44:0/2823517970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84fdede40 0x55b84ff727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.545+0000 7fac46be3b80 1 --2- 172.21.15.44:0/2823517970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 0x55b84ff6c870 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.545+0000 7fac46be3b80 1 -- 172.21.15.44:0/2823517970 >> 172.21.15.44:0/2823517970 conn(0x55b84fe4dfc0 msgr2=0x55b84ff678d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:54.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.545+0000 7fac46be3b80 1 -- 172.21.15.44:0/2823517970 shutdown_connections 2024-05-27T09:48:54.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.545+0000 7fac46be3b80 1 -- 172.21.15.44:0/2823517970 wait complete. 2024-05-27T09:48:54.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.546+0000 7fac46be3b80 1 -- start start 2024-05-27T09:48:54.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.546+0000 7fac46be3b80 1 Processor -- start 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff764d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff76a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff72d30 0x55b84ff76f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac46be3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b84fd90fa0 con 0x55b84ff72d30 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac46be3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b84fe2b760 con 0x55b84fdede40 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac46be3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b84ff795c0 con 0x55b84ff6cdb0 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac46be3b80 10 client.0 _ll_register_callbacks cb 0x55b84fe34c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac35600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff764d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac35600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff764d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56258/0 (socket says 172.21.15.44:56258) 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac35600700 1 -- 172.21.15.44:0/2005976338 learned_addr learned my addr 172.21.15.44:0/2005976338 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac36000700 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff72d30 0x55b84ff76f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac34c00700 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff76a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac35600700 1 -- 172.21.15.44:0/2005976338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 msgr2=0x55b84ff76a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac35600700 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff76a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac35600700 1 -- 172.21.15.44:0/2005976338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff72d30 msgr2=0x55b84ff76f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.547+0000 7fac35600700 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff72d30 0x55b84ff76f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.548+0000 7fac35600700 1 -- 172.21.15.44:0/2005976338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b84fdda8d0 con 0x55b84fdede40 2024-05-27T09:48:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.548+0000 7fac34c00700 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff76a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.548+0000 7fac35600700 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff764d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fac240e3910 tx=0x7fac240aa360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f13b33a1068c191e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.548+0000 7fac17400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.548+0000 7fac17400700 1 -- 172.21.15.44:0/2005976338 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac240aa3f0 con 0x55b84fdede40 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.548+0000 7fac17400700 1 -- 172.21.15.44:0/2005976338 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7fac240a6ed0 con 0x55b84fdede40 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.548+0000 7fac46be3b80 1 -- 172.21.15.44:0/2005976338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac240aafe0 con 0x55b84fdede40 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.548+0000 7fac17400700 1 -- 172.21.15.44:0/2005976338 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fac240aec90 con 0x55b84fdede40 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.548+0000 7fac46be3b80 10 client.27833 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.548+0000 7fac46be3b80 1 -- 172.21.15.44:0/2005976338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55b84fe5b060 con 0x55b84fdede40 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.549+0000 7fac17400700 1 -- 172.21.15.44:0/2005976338 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac240aafe0 con 0x55b84fdede40 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.549+0000 7fac17400700 1 -- 172.21.15.44:0/2005976338 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fac240cbb00 con 0x55b84fdede40 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.550+0000 7fac46be3b80 10 client.27833 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.550+0000 7fac46be3b80 10 client.27833 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.550+0000 7fac46be3b80 20 client.27833 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.550+0000 7fac46be3b80 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b84ff86290 0x55b84ff886b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.550+0000 7fac46be3b80 4 client.27833 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:54.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.550+0000 7fac46be3b80 1 -- 172.21.15.44:0/2005976338 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b84fe9cd90 con 0x55b84ff86290 2024-05-27T09:48:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.550+0000 7fac34c00700 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b84ff86290 0x55b84ff886b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.551+0000 7fac34c00700 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b84ff86290 0x55b84ff886b0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63abb17b2b74ed09 server_cookie=e59be84aa6f302b1 in_seq=0 out_seq=0 2024-05-27T09:48:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.551+0000 7fac17400700 10 client.27833 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.551+0000 7fac17400700 1 -- 172.21.15.44:0/2005976338 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b84fe9cd90 con 0x55b84ff86290 2024-05-27T09:48:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.551+0000 7fac17400700 10 client.27833 handle_command_reply: tid=0 2024-05-27T09:48:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 1 client.27833 shutdown 2024-05-27T09:48:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 1 -- 172.21.15.44:0/2005976338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 msgr2=0x55b84ff764d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff764d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fac240e3910 tx=0x7fac240aa360 comp rx=0 tx=0).stop 2024-05-27T09:48:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 1 -- 172.21.15.44:0/2005976338 shutdown_connections 2024-05-27T09:48:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b84ff86290 0x55b84ff886b0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff72d30 0x55b84ff76f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff764d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac17400700 0 client.27833 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 1 --2- 172.21.15.44:0/2005976338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff76a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 1 -- 172.21.15.44:0/2005976338 >> 172.21.15.44:0/2005976338 conn(0x55b84fe4dfc0 msgr2=0x55b84ff67580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 1 -- 172.21.15.44:0/2005976338 shutdown_connections 2024-05-27T09:48:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 1 -- 172.21.15.44:0/2005976338 wait complete. 2024-05-27T09:48:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.552+0000 7fac46be3b80 20 client.27833 trim_cache size 0 max 16384 2024-05-27T09:48:54.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.572+0000 7fac46be3b80 1 Processor -- start 2024-05-27T09:48:54.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.572+0000 7fac46be3b80 1 -- start start 2024-05-27T09:48:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.572+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84fdede40 0x55b84ff6c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.572+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff78970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.572+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 0x55b84ff82620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.572+0000 7fac46be3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fac301b7e30 con 0x55b84fdede40 2024-05-27T09:48:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.572+0000 7fac46be3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fac301b5660 con 0x55b84ff72d30 2024-05-27T09:48:54.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.572+0000 7fac46be3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac301b34d0 con 0x55b84ff6cdb0 2024-05-27T09:48:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac35600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84fdede40 0x55b84ff6c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac36000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 0x55b84ff82620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac35600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84fdede40 0x55b84ff6c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51678/0 (socket says 172.21.15.44:51678) 2024-05-27T09:48:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac35600700 1 -- 172.21.15.44:0/1838295487 learned_addr learned my addr 172.21.15.44:0/1838295487 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac34c00700 1 --2- 172.21.15.44:0/1838295487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff78970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac35600700 1 -- 172.21.15.44:0/1838295487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 msgr2=0x55b84ff78970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac35600700 1 --2- 172.21.15.44:0/1838295487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff78970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac35600700 1 -- 172.21.15.44:0/1838295487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 msgr2=0x55b84ff82620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac35600700 1 --2- 172.21.15.44:0/1838295487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 0x55b84ff82620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac35600700 1 -- 172.21.15.44:0/1838295487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b84ff772c0 con 0x55b84fdede40 2024-05-27T09:48:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac34c00700 1 --2- 172.21.15.44:0/1838295487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff78970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.573+0000 7fac36000700 1 --2- 172.21.15.44:0/1838295487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 0x55b84ff82620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.574+0000 7fac35600700 1 --2- 172.21.15.44:0/1838295487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84fdede40 0x55b84ff6c870 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fac240095c0 tx=0x7fac240a6050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74420723e4bf3be7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.574+0000 7fac22000700 1 -- 172.21.15.44:0/1838295487 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac240aa3f0 con 0x55b84fdede40 2024-05-27T09:48:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.574+0000 7fac22000700 1 -- 172.21.15.44:0/1838295487 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac240aafe0 con 0x55b84fdede40 2024-05-27T09:48:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.574+0000 7fac22000700 1 -- 172.21.15.44:0/1838295487 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac240a9b40 con 0x55b84fdede40 2024-05-27T09:48:54.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.575+0000 7fac46be3b80 1 -- 172.21.15.44:0/1838295487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84fdede40 msgr2=0x55b84ff6c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:54.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.575+0000 7fac46be3b80 1 --2- 172.21.15.44:0/1838295487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84fdede40 0x55b84ff6c870 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fac240095c0 tx=0x7fac240a6050 comp rx=0 tx=0).stop 2024-05-27T09:48:54.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.575+0000 7fac46be3b80 1 -- 172.21.15.44:0/1838295487 shutdown_connections 2024-05-27T09:48:54.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.575+0000 7fac46be3b80 1 --2- 172.21.15.44:0/1838295487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84ff72d30 0x55b84ff82620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.575+0000 7fac46be3b80 1 --2- 172.21.15.44:0/1838295487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84fdede40 0x55b84ff6c870 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.575+0000 7fac46be3b80 1 --2- 172.21.15.44:0/1838295487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff78970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.575+0000 7fac46be3b80 1 -- 172.21.15.44:0/1838295487 >> 172.21.15.44:0/1838295487 conn(0x55b84fe4dfc0 msgr2=0x55b84ff70140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.575+0000 7fac46be3b80 1 -- 172.21.15.44:0/1838295487 shutdown_connections 2024-05-27T09:48:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.575+0000 7fac46be3b80 1 -- 172.21.15.44:0/1838295487 wait complete. 2024-05-27T09:48:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.576+0000 7fac46be3b80 1 -- start start 2024-05-27T09:48:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.576+0000 7fac46be3b80 1 Processor -- start 2024-05-27T09:48:54.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.577+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff7ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.577+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff8d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.577+0000 7fac46be3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff72d30 0x55b84ff8d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.577+0000 7fac46be3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fac301b3b20 con 0x55b84ff72d30 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.577+0000 7fac46be3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fac301b4f90 con 0x55b84fdede40 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.577+0000 7fac46be3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac301b41f0 con 0x55b84ff6cdb0 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.577+0000 7fac46be3b80 10 client.0 _ll_register_callbacks cb 0x55b84fe34c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.577+0000 7fac35600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff7ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.577+0000 7fac35600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff7ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56270/0 (socket says 172.21.15.44:56270) 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.577+0000 7fac35600700 1 -- 172.21.15.44:0/3172895397 learned_addr learned my addr 172.21.15.44:0/3172895397 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.577+0000 7fac34c00700 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff8d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac36000700 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff72d30 0x55b84ff8d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac35600700 1 -- 172.21.15.44:0/3172895397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 msgr2=0x55b84ff8d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac35600700 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff8d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac35600700 1 -- 172.21.15.44:0/3172895397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff72d30 msgr2=0x55b84ff8d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:54.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac35600700 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff72d30 0x55b84ff8d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac35600700 1 -- 172.21.15.44:0/3172895397 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b84ff8dd30 con 0x55b84fdede40 2024-05-27T09:48:54.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac34c00700 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff8d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:48:54.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac36000700 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff72d30 0x55b84ff8d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:48:54.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac35600700 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff7ebe0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fac240aa5e0 tx=0x7fac240cd2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48080a706fabe681 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:54.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac17400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac17400700 1 -- 172.21.15.44:0/3172895397 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac240aa3f0 con 0x55b84fdede40 2024-05-27T09:48:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac46be3b80 1 -- 172.21.15.44:0/3172895397 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac240aafe0 con 0x55b84fdede40 2024-05-27T09:48:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.578+0000 7fac17400700 1 -- 172.21.15.44:0/3172895397 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7fac240a6ed0 con 0x55b84fdede40 2024-05-27T09:48:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.579+0000 7fac17400700 1 -- 172.21.15.44:0/3172895397 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fac240050b0 con 0x55b84fdede40 2024-05-27T09:48:54.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.579+0000 7fac46be3b80 10 client.27839 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.579+0000 7fac46be3b80 1 -- 172.21.15.44:0/3172895397 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55b84ff8dd30 con 0x55b84fdede40 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.579+0000 7fac17400700 1 -- 172.21.15.44:0/3172895397 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac240aafe0 con 0x55b84fdede40 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.580+0000 7fac17400700 1 -- 172.21.15.44:0/3172895397 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fac240c8050 con 0x55b84fdede40 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.580+0000 7fac46be3b80 10 client.27839 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.580+0000 7fac46be3b80 10 client.27839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.580+0000 7fac46be3b80 20 client.27839 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.580+0000 7fac46be3b80 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b84fff09a0 0x55b84fff2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.580+0000 7fac46be3b80 4 client.27839 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.580+0000 7fac46be3b80 1 -- 172.21.15.44:0/3172895397 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b84fe96570 con 0x55b84fff09a0 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.580+0000 7fac34c00700 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b84fff09a0 0x55b84fff2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac17400700 10 client.27839 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac34c00700 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b84fff09a0 0x55b84fff2dc0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f84b83806a28bd1 server_cookie=1fec150108fe3955 in_seq=0 out_seq=0 2024-05-27T09:48:54.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac17400700 1 -- 172.21.15.44:0/3172895397 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b84fe96570 con 0x55b84fff09a0 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac17400700 10 client.27839 handle_command_reply: tid=0 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac46be3b80 1 client.27839 shutdown 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac46be3b80 1 -- 172.21.15.44:0/3172895397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 msgr2=0x55b84ff7ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac46be3b80 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff7ebe0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fac240aa5e0 tx=0x7fac240cd2e0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac46be3b80 1 -- 172.21.15.44:0/3172895397 shutdown_connections 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac46be3b80 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55b84fff09a0 0x55b84fff2dc0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac46be3b80 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b84ff72d30 0x55b84ff8d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac46be3b80 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b84fdede40 0x55b84ff7ebe0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac17400700 0 client.27839 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac46be3b80 1 --2- 172.21.15.44:0/3172895397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b84ff6cdb0 0x55b84ff8d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac46be3b80 1 -- 172.21.15.44:0/3172895397 >> 172.21.15.44:0/3172895397 conn(0x55b84fe4dfc0 msgr2=0x55b84ff70140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac46be3b80 1 -- 172.21.15.44:0/3172895397 shutdown_connections 2024-05-27T09:48:54.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.581+0000 7fac46be3b80 1 -- 172.21.15.44:0/3172895397 wait complete. 2024-05-27T09:48:54.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.582+0000 7fac46be3b80 20 client.27839 trim_cache size 0 max 16384 2024-05-27T09:48:54.585 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:48:54.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.582+0000 7fac38000700 1 -- 172.21.15.44:0/1375378156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fac180b29f0 msgr2=0x7fac180b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:54.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.582+0000 7fac38000700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fac180b29f0 0x7fac180b4f00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fac28003fd0 tx=0x7fac28004b40 comp rx=0 tx=0).stop 2024-05-27T09:48:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.582+0000 7fac38000700 1 -- 172.21.15.44:0/1375378156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac30075af0 msgr2=0x7fac30104330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.582+0000 7fac38000700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac30075af0 0x7fac30104330 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fac24000c00 tx=0x7fac240057c0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.583+0000 7fac38000700 1 -- 172.21.15.44:0/1375378156 shutdown_connections 2024-05-27T09:48:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.583+0000 7fac38000700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fac180b29f0 0x7fac180b4f00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.583+0000 7fac38000700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac30110c10 0x7fac30104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.583+0000 7fac38000700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac30075af0 0x7fac30104330 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.583+0000 7fac38000700 1 --2- 172.21.15.44:0/1375378156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac30076440 0x7fac30104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.583+0000 7fac38000700 1 -- 172.21.15.44:0/1375378156 >> 172.21.15.44:0/1375378156 conn(0x7fac3007af60 msgr2=0x7fac30102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.583+0000 7fac38000700 1 -- 172.21.15.44:0/1375378156 shutdown_connections 2024-05-27T09:48:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:54.583+0000 7fac38000700 1 -- 172.21.15.44:0/1375378156 wait complete. 2024-05-27T09:48:56.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:55 smithi050 ceph-mon[36371]: pgmap v430: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1001 KiB/s wr, 177 op/s 2024-05-27T09:48:56.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:55 smithi189 ceph-mon[36671]: pgmap v430: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1001 KiB/s wr, 177 op/s 2024-05-27T09:48:56.353 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:48:56.353 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:48:56.425 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:55 smithi044 ceph-mon[31618]: pgmap v430: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1001 KiB/s wr, 177 op/s 2024-05-27T09:48:56.804 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:48:58.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:48:58 smithi050 ceph-mon[36371]: pgmap v431: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 327 op/s 2024-05-27T09:48:58.332 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:48:58 smithi044 ceph-mon[31618]: pgmap v431: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 327 op/s 2024-05-27T09:48:58.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:48:58 smithi189 ceph-mon[36671]: pgmap v431: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 327 op/s 2024-05-27T09:48:58.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.860+0000 7f8640800700 1 -- 172.21.15.44:0/1194532847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8638105e20 msgr2=0x7f863810e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:58.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.860+0000 7f8640800700 1 --2- 172.21.15.44:0/1194532847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8638105e20 0x7f863810e370 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8628009990 tx=0x7f8628009ca0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.860+0000 7f8640800700 1 -- 172.21.15.44:0/1194532847 shutdown_connections 2024-05-27T09:48:58.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.860+0000 7f8640800700 1 --2- 172.21.15.44:0/1194532847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f863810e900 0x7f8638110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.860+0000 7f8640800700 1 --2- 172.21.15.44:0/1194532847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86381054d0 0x7f86381058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.860+0000 7f8640800700 1 --2- 172.21.15.44:0/1194532847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8638105e20 0x7f863810e370 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.860+0000 7f8640800700 1 -- 172.21.15.44:0/1194532847 >> 172.21.15.44:0/1194532847 conn(0x7f86380fedb0 msgr2=0x7f86381011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:58.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.860+0000 7f8640800700 1 -- 172.21.15.44:0/1194532847 shutdown_connections 2024-05-27T09:48:58.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.860+0000 7f8640800700 1 -- 172.21.15.44:0/1194532847 wait complete. 2024-05-27T09:48:58.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.861+0000 7f8640800700 1 Processor -- start 2024-05-27T09:48:58.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.862+0000 7f8640800700 1 -- start start 2024-05-27T09:48:58.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.862+0000 7f8640800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86381054d0 0x7f863819adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.862+0000 7f8640800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8638105e20 0x7f863819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.862+0000 7f8640800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f863810e900 0x7f863819f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.862+0000 7f8640800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8638114480 con 0x7f8638105e20 2024-05-27T09:48:58.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.862+0000 7f8640800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8638114300 con 0x7f86381054d0 2024-05-27T09:48:58.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.862+0000 7f8640800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8638114600 con 0x7f863810e900 2024-05-27T09:48:58.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.862+0000 7f863de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86381054d0 0x7f863819adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.863+0000 7f863de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86381054d0 0x7f863819adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56274/0 (socket says 172.21.15.44:56274) 2024-05-27T09:48:58.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.863+0000 7f863de00700 1 -- 172.21.15.44:0/1469067476 learned_addr learned my addr 172.21.15.44:0/1469067476 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:58.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.863+0000 7f863d400700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8638105e20 0x7f863819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.863+0000 7f863e800700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f863810e900 0x7f863819f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.863+0000 7f863de00700 1 -- 172.21.15.44:0/1469067476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f863810e900 msgr2=0x7f863819f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:58.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.863+0000 7f863de00700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f863810e900 0x7f863819f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.863+0000 7f863de00700 1 -- 172.21.15.44:0/1469067476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8638105e20 msgr2=0x7f863819b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:58.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.863+0000 7f863de00700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8638105e20 0x7f863819b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.863+0000 7f863de00700 1 -- 172.21.15.44:0/1469067476 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863819fcc0 con 0x7f86381054d0 2024-05-27T09:48:58.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.864+0000 7f863de00700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86381054d0 0x7f863819adf0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f862c00e9a0 tx=0x7f862c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:58.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.864+0000 7f8632a00700 1 -- 172.21.15.44:0/1469067476 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c00ca10 con 0x7f86381054d0 2024-05-27T09:48:58.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.864+0000 7f8632a00700 1 -- 172.21.15.44:0/1469067476 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f862c00cbb0 con 0x7f86381054d0 2024-05-27T09:48:58.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.864+0000 7f8640800700 1 -- 172.21.15.44:0/1469067476 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f863819ffb0 con 0x7f86381054d0 2024-05-27T09:48:58.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.864+0000 7f8632a00700 1 -- 172.21.15.44:0/1469067476 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c018680 con 0x7f86381054d0 2024-05-27T09:48:58.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.864+0000 7f8640800700 1 -- 172.21.15.44:0/1469067476 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86381a78d0 con 0x7f86381054d0 2024-05-27T09:48:58.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.870+0000 7f8632a00700 1 -- 172.21.15.44:0/1469067476 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f862c010430 con 0x7f86381054d0 2024-05-27T09:48:58.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.871+0000 7f864f376b80 1 Processor -- start 2024-05-27T09:48:58.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.871+0000 7f864f376b80 1 -- start start 2024-05-27T09:48:58.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.871+0000 7f864f376b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd957a70 0x55cfbd9ea450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.871+0000 7f8632a00700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f86200b2a70 0x7f86200b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.871+0000 7f864f376b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 0x55cfbd9ea990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.871+0000 7f864f376b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd9f9590 0x55cfbd9fb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.871+0000 7f864f376b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cfbd918ec0 con 0x55cfbd86bec0 2024-05-27T09:48:58.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.871+0000 7f864f376b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cfbd908a60 con 0x55cfbd9f9590 2024-05-27T09:48:58.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.871+0000 7f864f376b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cfbd90caa0 con 0x55cfbd957a70 2024-05-27T09:48:58.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.872+0000 7f8632a00700 1 -- 172.21.15.44:0/1469067476 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f862c0e2760 con 0x7f86381054d0 2024-05-27T09:48:58.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.872+0000 7f863e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd957a70 0x55cfbd9ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.872+0000 7f863e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd957a70 0x55cfbd9ea450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60150/0 (socket says 172.21.15.44:60150) 2024-05-27T09:48:58.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.872+0000 7f863e800700 1 -- 172.21.15.44:0/2160236326 learned_addr learned my addr 172.21.15.44:0/2160236326 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:58.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.872+0000 7f863d400700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f86200b2a70 0x7f86200b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.872+0000 7f863de00700 1 --2- 172.21.15.44:0/2160236326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 0x55cfbd9ea990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.872+0000 7f863e800700 1 --2- 172.21.15.44:0/2160236326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd9f9590 0x55cfbd9fb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.873+0000 7f863de00700 1 -- 172.21.15.44:0/2160236326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd957a70 msgr2=0x55cfbd9ea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:58.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.873+0000 7f863de00700 1 --2- 172.21.15.44:0/2160236326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd957a70 0x55cfbd9ea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.873+0000 7f863de00700 1 -- 172.21.15.44:0/2160236326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd9f9590 msgr2=0x55cfbd9fb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:58.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.873+0000 7f863de00700 1 --2- 172.21.15.44:0/2160236326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd9f9590 0x55cfbd9fb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.873+0000 7f863de00700 1 -- 172.21.15.44:0/2160236326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfbd8d14a0 con 0x55cfbd86bec0 2024-05-27T09:48:58.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.873+0000 7f863e800700 1 --2- 172.21.15.44:0/2160236326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd9f9590 0x55cfbd9fb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:58.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.873+0000 7f863d400700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f86200b2a70 0x7f86200b4f80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8638102260 tx=0x7f8628005fd0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:58.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.873+0000 7f863de00700 1 --2- 172.21.15.44:0/2160236326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 0x55cfbd9ea990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f862c0dfe60 tx=0x7f862c0a98f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75141ab0a179c8db server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:58.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.873+0000 7f8627400700 1 -- 172.21.15.44:0/2160236326 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c0c0430 con 0x55cfbd86bec0 2024-05-27T09:48:58.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.873+0000 7f8627400700 1 -- 172.21.15.44:0/2160236326 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f862c0b2190 con 0x55cfbd86bec0 2024-05-27T09:48:58.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.873+0000 7f8627400700 1 -- 172.21.15.44:0/2160236326 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c0adda0 con 0x55cfbd86bec0 2024-05-27T09:48:58.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.874+0000 7f864f376b80 1 -- 172.21.15.44:0/2160236326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 msgr2=0x55cfbd9ea990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:58.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.874+0000 7f864f376b80 1 --2- 172.21.15.44:0/2160236326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 0x55cfbd9ea990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f862c0dfe60 tx=0x7f862c0a98f0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.875+0000 7f864f376b80 1 -- 172.21.15.44:0/2160236326 shutdown_connections 2024-05-27T09:48:58.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.875+0000 7f864f376b80 1 --2- 172.21.15.44:0/2160236326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd9f9590 0x55cfbd9fb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.875+0000 7f864f376b80 1 --2- 172.21.15.44:0/2160236326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd957a70 0x55cfbd9ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.875+0000 7f864f376b80 1 --2- 172.21.15.44:0/2160236326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 0x55cfbd9ea990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.875+0000 7f864f376b80 1 -- 172.21.15.44:0/2160236326 >> 172.21.15.44:0/2160236326 conn(0x55cfbd905410 msgr2=0x55cfbd9e6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:58.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.875+0000 7f864f376b80 1 -- 172.21.15.44:0/2160236326 shutdown_connections 2024-05-27T09:48:58.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.875+0000 7f864f376b80 1 -- 172.21.15.44:0/2160236326 wait complete. 2024-05-27T09:48:58.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.875+0000 7f864f376b80 1 -- start start 2024-05-27T09:48:58.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.876+0000 7f864f376b80 1 Processor -- start 2024-05-27T09:48:58.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.876+0000 7f864f376b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 0x55cfbd9f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.876+0000 7f864f376b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd957a70 0x55cfbd9f6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.876+0000 7f864f376b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd9f9590 0x55cfbd9f7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.876+0000 7f864f376b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55cfbd80f020 con 0x55cfbd86bec0 2024-05-27T09:48:58.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.876+0000 7f864f376b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cfbd8a97e0 con 0x55cfbd957a70 2024-05-27T09:48:58.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.876+0000 7f864f376b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cfbd9fc040 con 0x55cfbd9f9590 2024-05-27T09:48:58.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.876+0000 7f864f376b80 10 client.0 _ll_register_callbacks cb 0x55cfbd8b2cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.877+0000 7f863d400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd957a70 0x55cfbd9f6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.877+0000 7f863de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 0x55cfbd9f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.877+0000 7f863d400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd957a70 0x55cfbd9f6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56288/0 (socket says 172.21.15.44:56288) 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.877+0000 7f863d400700 1 -- 172.21.15.44:0/3941507769 learned_addr learned my addr 172.21.15.44:0/3941507769 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.877+0000 7f863e800700 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd9f9590 0x55cfbd9f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.877+0000 7f863d400700 1 -- 172.21.15.44:0/3941507769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd9f9590 msgr2=0x55cfbd9f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.877+0000 7f863d400700 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd9f9590 0x55cfbd9f7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.877+0000 7f863d400700 1 -- 172.21.15.44:0/3941507769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 msgr2=0x55cfbd9f65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.877+0000 7f863d400700 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 0x55cfbd9f65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.877+0000 7f863d400700 1 -- 172.21.15.44:0/3941507769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfbd858950 con 0x55cfbd957a70 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.877+0000 7f863de00700 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 0x55cfbd9f65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:58.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.878+0000 7f863d400700 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd957a70 0x55cfbd9f6b10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8628015000 tx=0x7f8628009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eef4c6fe54998850 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:58.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.878+0000 7f861be00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:58.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.878+0000 7f861be00700 1 -- 172.21.15.44:0/3941507769 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8628033060 con 0x55cfbd957a70 2024-05-27T09:48:58.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.878+0000 7f861be00700 1 -- 172.21.15.44:0/3941507769 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f8628003910 con 0x55cfbd957a70 2024-05-27T09:48:58.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.878+0000 7f864f376b80 1 -- 172.21.15.44:0/3941507769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cfbd91ae10 con 0x55cfbd957a70 2024-05-27T09:48:58.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.878+0000 7f861be00700 1 -- 172.21.15.44:0/3941507769 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f8628003b10 con 0x55cfbd957a70 2024-05-27T09:48:58.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.878+0000 7f864f376b80 10 client.27851 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:58.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.878+0000 7f864f376b80 1 -- 172.21.15.44:0/3941507769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55cfbd8d90e0 con 0x55cfbd957a70 2024-05-27T09:48:58.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.879+0000 7f861be00700 1 -- 172.21.15.44:0/3941507769 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55cfbd91ae10 con 0x55cfbd957a70 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.879+0000 7f861be00700 1 -- 172.21.15.44:0/3941507769 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f8628022070 con 0x55cfbd957a70 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.879+0000 7f864f376b80 10 client.27851 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.879+0000 7f864f376b80 10 client.27851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.879+0000 7f864f376b80 20 client.27851 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.879+0000 7f864f376b80 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cfbda08fd0 0x55cfbda0b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.879+0000 7f864f376b80 4 client.27851 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.879+0000 7f864f376b80 1 -- 172.21.15.44:0/3941507769 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfbd9145f0 con 0x55cfbda08fd0 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.880+0000 7f863de00700 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cfbda08fd0 0x55cfbda0b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.880+0000 7f863de00700 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cfbda08fd0 0x55cfbda0b3f0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f27cc99ef3426fd4 server_cookie=d74aea4a918f465c in_seq=0 out_seq=0 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.880+0000 7f861be00700 10 client.27851 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f861be00700 1 -- 172.21.15.44:0/3941507769 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cfbd9145f0 con 0x55cfbda08fd0 2024-05-27T09:48:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f861be00700 10 client.27851 handle_command_reply: tid=0 2024-05-27T09:48:58.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f864f376b80 1 client.27851 shutdown 2024-05-27T09:48:58.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f864f376b80 1 -- 172.21.15.44:0/3941507769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd957a70 msgr2=0x55cfbd9f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:58.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f864f376b80 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd957a70 0x55cfbd9f6b10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8628015000 tx=0x7f8628009ca0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f864f376b80 1 -- 172.21.15.44:0/3941507769 shutdown_connections 2024-05-27T09:48:58.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f864f376b80 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cfbda08fd0 0x55cfbda0b3f0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f864f376b80 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd9f9590 0x55cfbd9f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f864f376b80 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd86bec0 0x55cfbd9f65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f864f376b80 1 --2- 172.21.15.44:0/3941507769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd957a70 0x55cfbd9f6b10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f861be00700 0 client.27851 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:58.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.881+0000 7f864f376b80 1 -- 172.21.15.44:0/3941507769 >> 172.21.15.44:0/3941507769 conn(0x55cfbd905410 msgr2=0x55cfbd9ea450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:58.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.882+0000 7f864f376b80 1 -- 172.21.15.44:0/3941507769 shutdown_connections 2024-05-27T09:48:58.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.882+0000 7f864f376b80 1 -- 172.21.15.44:0/3941507769 wait complete. 2024-05-27T09:48:58.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.882+0000 7f864f376b80 20 client.27851 trim_cache size 0 max 16384 2024-05-27T09:48:58.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.907+0000 7f864f376b80 1 Processor -- start 2024-05-27T09:48:58.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.907+0000 7f864f376b80 1 -- start start 2024-05-27T09:48:58.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.907+0000 7f864f376b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd86bec0 0x55cfbd9e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.907+0000 7f864f376b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 0x55cfbda050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.907+0000 7f864f376b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd9f9590 0x55cfbd9ec700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.907+0000 7f864f376b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f86381af750 con 0x55cfbd957a70 2024-05-27T09:48:58.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.907+0000 7f864f376b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f86381acf80 con 0x55cfbd9f9590 2024-05-27T09:48:58.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.907+0000 7f864f376b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86381aadf0 con 0x55cfbd86bec0 2024-05-27T09:48:58.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.908+0000 7f863d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 0x55cfbda050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.908+0000 7f863de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd86bec0 0x55cfbd9e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.908+0000 7f863de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd86bec0 0x55cfbd9e6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60164/0 (socket says 172.21.15.44:60164) 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.908+0000 7f863de00700 1 -- 172.21.15.44:0/1470229628 learned_addr learned my addr 172.21.15.44:0/1470229628 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.908+0000 7f863e800700 1 --2- 172.21.15.44:0/1470229628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd9f9590 0x55cfbd9ec700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.908+0000 7f863de00700 1 -- 172.21.15.44:0/1470229628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd9f9590 msgr2=0x55cfbd9ec700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.908+0000 7f863de00700 1 --2- 172.21.15.44:0/1470229628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd9f9590 0x55cfbd9ec700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.908+0000 7f863de00700 1 -- 172.21.15.44:0/1470229628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 msgr2=0x55cfbda050e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.908+0000 7f863de00700 1 --2- 172.21.15.44:0/1470229628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 0x55cfbda050e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.908+0000 7f863de00700 1 -- 172.21.15.44:0/1470229628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfbd9f8e70 con 0x55cfbd86bec0 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.908+0000 7f863d400700 1 --2- 172.21.15.44:0/1470229628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 0x55cfbda050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.909+0000 7f863de00700 1 --2- 172.21.15.44:0/1470229628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd86bec0 0x55cfbd9e6570 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f862c0a9620 tx=0x7f862c00e8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a226e5ec46e8edbc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.909+0000 7f8626a00700 1 -- 172.21.15.44:0/1470229628 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c0c40d0 con 0x55cfbd86bec0 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.909+0000 7f8626a00700 1 -- 172.21.15.44:0/1470229628 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f862c0b2850 con 0x55cfbd86bec0 2024-05-27T09:48:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.909+0000 7f8626a00700 1 -- 172.21.15.44:0/1470229628 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c0a9e20 con 0x55cfbd86bec0 2024-05-27T09:48:58.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.910+0000 7f864f376b80 1 -- 172.21.15.44:0/1470229628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd86bec0 msgr2=0x55cfbd9e6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:58.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.910+0000 7f864f376b80 1 --2- 172.21.15.44:0/1470229628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd86bec0 0x55cfbd9e6570 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f862c0a9620 tx=0x7f862c00e8f0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.910+0000 7f864f376b80 1 -- 172.21.15.44:0/1470229628 shutdown_connections 2024-05-27T09:48:58.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.910+0000 7f864f376b80 1 --2- 172.21.15.44:0/1470229628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd9f9590 0x55cfbd9ec700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.910+0000 7f864f376b80 1 --2- 172.21.15.44:0/1470229628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd86bec0 0x55cfbd9e6570 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.910+0000 7f864f376b80 1 --2- 172.21.15.44:0/1470229628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 0x55cfbda050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.910+0000 7f864f376b80 1 -- 172.21.15.44:0/1470229628 >> 172.21.15.44:0/1470229628 conn(0x55cfbd905410 msgr2=0x55cfbd9ebff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:58.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.910+0000 7f864f376b80 1 -- 172.21.15.44:0/1470229628 shutdown_connections 2024-05-27T09:48:58.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.911+0000 7f864f376b80 1 -- 172.21.15.44:0/1470229628 wait complete. 2024-05-27T09:48:58.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.911+0000 7f864f376b80 1 -- start start 2024-05-27T09:48:58.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.911+0000 7f864f376b80 1 Processor -- start 2024-05-27T09:48:58.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f864f376b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd86bec0 0x55cfbda012e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f864f376b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 0x55cfbda01820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f864f376b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd9f9590 0x55cfbd9fc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f864f376b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f86381ab440 con 0x55cfbd957a70 2024-05-27T09:48:58.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f864f376b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f86381ac8b0 con 0x55cfbd86bec0 2024-05-27T09:48:58.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f864f376b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86381abb10 con 0x55cfbd9f9590 2024-05-27T09:48:58.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f864f376b80 10 client.0 _ll_register_callbacks cb 0x55cfbd8b2cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:48:58.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f863de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd86bec0 0x55cfbda012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f863d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 0x55cfbda01820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f863e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd9f9590 0x55cfbd9fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f863de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd86bec0 0x55cfbda012e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56304/0 (socket says 172.21.15.44:56304) 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.912+0000 7f863de00700 1 -- 172.21.15.44:0/2693831849 learned_addr learned my addr 172.21.15.44:0/2693831849 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.913+0000 7f863de00700 1 -- 172.21.15.44:0/2693831849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd9f9590 msgr2=0x55cfbd9fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.913+0000 7f863de00700 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd9f9590 0x55cfbd9fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.913+0000 7f863de00700 1 -- 172.21.15.44:0/2693831849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 msgr2=0x55cfbda01820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.913+0000 7f863de00700 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 0x55cfbda01820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.913+0000 7f863de00700 1 -- 172.21.15.44:0/2693831849 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfbd9fc7d0 con 0x55cfbd86bec0 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.913+0000 7f863d400700 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 0x55cfbda01820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.913+0000 7f863de00700 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd86bec0 0x55cfbda012e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f862c009230 tx=0x7f862c0c40d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e71dcdc50ef7097 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:48:58.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.913+0000 7f861be00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:48:58.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.914+0000 7f861be00700 1 -- 172.21.15.44:0/2693831849 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c08e070 con 0x55cfbd86bec0 2024-05-27T09:48:58.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.914+0000 7f861be00700 1 -- 172.21.15.44:0/2693831849 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8607+0+0 (secure 0 0 0) 0x7f862c0a9c50 con 0x55cfbd86bec0 2024-05-27T09:48:58.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.914+0000 7f864f376b80 1 -- 172.21.15.44:0/2693831849 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f862c0b2850 con 0x55cfbd86bec0 2024-05-27T09:48:58.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.914+0000 7f861be00700 1 -- 172.21.15.44:0/2693831849 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f862c0aa0f0 con 0x55cfbd86bec0 2024-05-27T09:48:58.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.914+0000 7f864f376b80 10 client.27857 fetch_fsmap learned FSMap version 43 2024-05-27T09:48:58.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.914+0000 7f864f376b80 1 -- 172.21.15.44:0/2693831849 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55cfbda741e0 con 0x55cfbd86bec0 2024-05-27T09:48:58.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.914+0000 7f861be00700 1 -- 172.21.15.44:0/2693831849 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f862c087080 con 0x55cfbd86bec0 2024-05-27T09:48:58.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.914+0000 7f861be00700 1 -- 172.21.15.44:0/2693831849 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f862c010720 con 0x55cfbd86bec0 2024-05-27T09:48:58.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.914+0000 7f864f376b80 10 client.27857 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:48:58.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.915+0000 7f864f376b80 10 client.27857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:48:58.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.915+0000 7f864f376b80 20 client.27857 populate_metadata read hostname 'smithi044' 2024-05-27T09:48:58.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.915+0000 7f864f376b80 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cfbda74400 0x55cfbda76820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:48:58.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.915+0000 7f864f376b80 4 client.27857 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.915+0000 7f864f376b80 1 -- 172.21.15.44:0/2693831849 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cfbd78b2d0 con 0x55cfbda74400 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.915+0000 7f863d400700 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cfbda74400 0x55cfbda76820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.915+0000 7f863d400700 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cfbda74400 0x55cfbda76820 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f17c6d797a009b76 server_cookie=456fa6f2b4b5a76b in_seq=0 out_seq=0 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.915+0000 7f861be00700 10 client.27857 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f861be00700 1 -- 172.21.15.44:0/2693831849 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cfbd78b2d0 con 0x55cfbda74400 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f861be00700 10 client.27857 handle_command_reply: tid=0 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 1 client.27857 shutdown 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 1 -- 172.21.15.44:0/2693831849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd86bec0 msgr2=0x55cfbda012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd86bec0 0x55cfbda012e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f862c009230 tx=0x7f862c0c40d0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 1 -- 172.21.15.44:0/2693831849 shutdown_connections 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55cfbda74400 0x55cfbda76820 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfbd9f9590 0x55cfbd9fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfbd86bec0 0x55cfbda012e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 1 --2- 172.21.15.44:0/2693831849 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55cfbd957a70 0x55cfbda01820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f861be00700 0 client.27857 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:48:58.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 1 -- 172.21.15.44:0/2693831849 >> 172.21.15.44:0/2693831849 conn(0x55cfbd905410 msgr2=0x55cfbd9f4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:58.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 1 -- 172.21.15.44:0/2693831849 shutdown_connections 2024-05-27T09:48:58.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 1 -- 172.21.15.44:0/2693831849 wait complete. 2024-05-27T09:48:58.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.917+0000 7f864f376b80 20 client.27857 trim_cache size 0 max 16384 2024-05-27T09:48:58.921 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:48:58.921 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:48:58.921 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "bafbe750-ac8d-41eb-a14b-0f2553ebad05", 2024-05-27T09:48:58.921 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:48:58.921 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.918+0000 7f8640800700 1 -- 172.21.15.44:0/1469067476 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f86200b2a70 msgr2=0x7f86200b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.918+0000 7f8640800700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f86200b2a70 0x7f86200b4f80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8638102260 tx=0x7f8628005fd0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.918+0000 7f8640800700 1 -- 172.21.15.44:0/1469067476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86381054d0 msgr2=0x7f863819adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.918+0000 7f8640800700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86381054d0 0x7f863819adf0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f862c00e9a0 tx=0x7f862c00ecb0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.918+0000 7f8640800700 1 -- 172.21.15.44:0/1469067476 shutdown_connections 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.918+0000 7f8640800700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f86200b2a70 0x7f86200b4f80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.918+0000 7f8640800700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f863810e900 0x7f863819f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.918+0000 7f8640800700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86381054d0 0x7f863819adf0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.918+0000 7f8640800700 1 --2- 172.21.15.44:0/1469067476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8638105e20 0x7f863819b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.918+0000 7f8640800700 1 -- 172.21.15.44:0/1469067476 >> 172.21.15.44:0/1469067476 conn(0x7f86380fedb0 msgr2=0x7f86380ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.919+0000 7f8640800700 1 -- 172.21.15.44:0/1469067476 shutdown_connections 2024-05-27T09:48:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:48:58.919+0000 7f8640800700 1 -- 172.21.15.44:0/1469067476 wait complete. 2024-05-27T09:48:59.794 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:49:00.260 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:49:00.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:00 smithi050 ceph-mon[36371]: pgmap v432: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 753 KiB/s wr, 217 op/s 2024-05-27T09:49:00.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:00 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:49:00.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:00 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:49:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:00 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:49:00.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:00 smithi189 ceph-mon[36671]: pgmap v432: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 753 KiB/s wr, 217 op/s 2024-05-27T09:49:00.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:00 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:49:00.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:00 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:49:00.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:00 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:49:00.410 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:00 smithi044 ceph-mon[31618]: pgmap v432: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 753 KiB/s wr, 217 op/s 2024-05-27T09:49:00.410 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:00 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:49:00.410 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:00 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:49:00.410 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:00 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:49:01.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:01 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:49:01.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:01 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:49:01.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:01 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:49:01.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:01 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:49:01.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:01 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:49:01.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:01 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:49:02.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:02 smithi044 ceph-mon[31618]: pgmap v433: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 287 op/s 2024-05-27T09:49:02.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:02 smithi050 ceph-mon[36371]: pgmap v433: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 287 op/s 2024-05-27T09:49:02.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:02 smithi189 ceph-mon[36671]: pgmap v433: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 287 op/s 2024-05-27T09:49:02.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.728+0000 7ff96b600700 1 -- 172.21.15.44:0/3634595168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9640764a0 msgr2=0x7ff964114e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:02.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.728+0000 7ff96b600700 1 --2- 172.21.15.44:0/3634595168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9640764a0 0x7ff964114e20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff950009990 tx=0x7ff950009ca0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.729+0000 7ff96b600700 1 -- 172.21.15.44:0/3634595168 shutdown_connections 2024-05-27T09:49:02.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.729+0000 7ff96b600700 1 --2- 172.21.15.44:0/3634595168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff964115360 0x7ff964117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.729+0000 7ff96b600700 1 --2- 172.21.15.44:0/3634595168 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff964075b50 0x7ff964075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.729+0000 7ff96b600700 1 --2- 172.21.15.44:0/3634595168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9640764a0 0x7ff964114e20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.729+0000 7ff96b600700 1 -- 172.21.15.44:0/3634595168 >> 172.21.15.44:0/3634595168 conn(0x7ff964100030 msgr2=0x7ff964102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:02.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.730+0000 7ff96b600700 1 -- 172.21.15.44:0/3634595168 shutdown_connections 2024-05-27T09:49:02.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.730+0000 7ff96b600700 1 -- 172.21.15.44:0/3634595168 wait complete. 2024-05-27T09:49:02.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.731+0000 7ff96b600700 1 Processor -- start 2024-05-27T09:49:02.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.732+0000 7ff96b600700 1 -- start start 2024-05-27T09:49:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.734+0000 7ff96b600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff964075b50 0x7ff964075350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.734+0000 7ff96b600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9640764a0 0x7ff96406e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.734+0000 7ff96b600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff964115360 0x7ff96406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.734+0000 7ff968c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff964075b50 0x7ff964075350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.734+0000 7ff968c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff964075b50 0x7ff964075350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56324/0 (socket says 172.21.15.44:56324) 2024-05-27T09:49:02.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.734+0000 7ff963e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9640764a0 0x7ff96406e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.735+0000 7ff969600700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff964115360 0x7ff96406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.735+0000 7ff968c00700 1 -- 172.21.15.44:0/3074163628 learned_addr learned my addr 172.21.15.44:0/3074163628 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:02.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.735+0000 7ff96b600700 1 -- 172.21.15.44:0/3074163628 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff96411ab00 con 0x7ff964115360 2024-05-27T09:49:02.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.735+0000 7ff96b600700 1 -- 172.21.15.44:0/3074163628 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff96411a980 con 0x7ff964075b50 2024-05-27T09:49:02.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.735+0000 7ff96b600700 1 -- 172.21.15.44:0/3074163628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff96411ac80 con 0x7ff9640764a0 2024-05-27T09:49:02.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.736+0000 7ff963e00700 1 -- 172.21.15.44:0/3074163628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff964075b50 msgr2=0x7ff964075350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:02.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.736+0000 7ff963e00700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff964075b50 0x7ff964075350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.736+0000 7ff963e00700 1 -- 172.21.15.44:0/3074163628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff964115360 msgr2=0x7ff96406ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:02.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.736+0000 7ff963e00700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff964115360 0x7ff96406ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.736+0000 7ff963e00700 1 -- 172.21.15.44:0/3074163628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96406f2b0 con 0x7ff9640764a0 2024-05-27T09:49:02.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.737+0000 7ff963e00700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9640764a0 0x7ff96406e480 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff9500098e0 tx=0x7ff950005af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:02.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.738+0000 7ff961600700 1 -- 172.21.15.44:0/3074163628 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950026070 con 0x7ff9640764a0 2024-05-27T09:49:02.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.738+0000 7ff961600700 1 -- 172.21.15.44:0/3074163628 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff950021410 con 0x7ff9640764a0 2024-05-27T09:49:02.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.738+0000 7ff961600700 1 -- 172.21.15.44:0/3074163628 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950017650 con 0x7ff9640764a0 2024-05-27T09:49:02.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.738+0000 7ff96b600700 1 -- 172.21.15.44:0/3074163628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff964072f40 con 0x7ff9640764a0 2024-05-27T09:49:02.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.738+0000 7ff96b600700 1 -- 172.21.15.44:0/3074163628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9640733d0 con 0x7ff9640764a0 2024-05-27T09:49:02.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.745+0000 7ff961600700 1 -- 172.21.15.44:0/3074163628 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff950029070 con 0x7ff9640764a0 2024-05-27T09:49:02.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.747+0000 7ff97a20eb80 1 Processor -- start 2024-05-27T09:49:02.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.747+0000 7ff97a20eb80 1 -- start start 2024-05-27T09:49:02.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.748+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d009f0 0x5593f2d93330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.748+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2d93870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.748+0000 7ff969600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d009f0 0x5593f2d93330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.748+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d93e70 0x5593f2da0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.748+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5593f2cc1e40 con 0x5593f2d009f0 2024-05-27T09:49:02.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.748+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5593f2cb19e0 con 0x5593f2c14e40 2024-05-27T09:49:02.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.748+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5593f2cb5a20 con 0x5593f2d93e70 2024-05-27T09:49:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.749+0000 7ff969600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d009f0 0x5593f2d93330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51762/0 (socket says 172.21.15.44:51762) 2024-05-27T09:49:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.749+0000 7ff969600700 1 -- 172.21.15.44:0/3542370947 learned_addr learned my addr 172.21.15.44:0/3542370947 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:02.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.749+0000 7ff961600700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff94c0b2c00 0x7ff94c0b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.749+0000 7ff968c00700 1 --2- 172.21.15.44:0/3542370947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2d93870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.749+0000 7ff961600700 1 -- 172.21.15.44:0/3074163628 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8651+0+0 (secure 0 0 0) 0x7ff9500e3520 con 0x7ff9640764a0 2024-05-27T09:49:02.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.750+0000 7ff963e00700 1 --2- 172.21.15.44:0/3542370947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d93e70 0x5593f2da0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.750+0000 7ff963e00700 1 -- 172.21.15.44:0/3542370947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 msgr2=0x5593f2d93870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:02.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.750+0000 7ff963e00700 1 --2- 172.21.15.44:0/3542370947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2d93870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.750+0000 7ff963e00700 1 -- 172.21.15.44:0/3542370947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d009f0 msgr2=0x5593f2d93330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:02.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.751+0000 7ff963e00700 1 --2- 172.21.15.44:0/3542370947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d009f0 0x5593f2d93330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.751+0000 7ff963e00700 1 -- 172.21.15.44:0/3542370947 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593f2c7a420 con 0x5593f2d93e70 2024-05-27T09:49:02.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.751+0000 7ff969600700 1 --2- 172.21.15.44:0/3542370947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d009f0 0x5593f2d93330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:02.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.751+0000 7ff963e00700 1 --2- 172.21.15.44:0/3542370947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d93e70 0x5593f2da0110 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff9500e0b90 tx=0x7ff9500aa3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dce557a8362a0348 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:02.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.751+0000 7ff969600700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff94c0b2c00 0x7ff94c0b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.752+0000 7ff956000700 1 -- 172.21.15.44:0/3542370947 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9500aa7b0 con 0x5593f2d93e70 2024-05-27T09:49:02.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.752+0000 7ff956000700 1 -- 172.21.15.44:0/3542370947 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9500b1e50 con 0x5593f2d93e70 2024-05-27T09:49:02.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.753+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3542370947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d93e70 msgr2=0x5593f2da0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:02.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.753+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/3542370947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d93e70 0x5593f2da0110 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff9500e0b90 tx=0x7ff9500aa3c0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.754+0000 7ff969600700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff94c0b2c00 0x7ff94c0b5110 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff95c003fd0 tx=0x7ff95c004ad0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:02.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.754+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3542370947 shutdown_connections 2024-05-27T09:49:02.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.754+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/3542370947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d93e70 0x5593f2da0110 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.754+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/3542370947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d009f0 0x5593f2d93330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.754+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/3542370947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2d93870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.754+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3542370947 >> 172.21.15.44:0/3542370947 conn(0x5593f2cae390 msgr2=0x5593f2d8f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:02.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.755+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3542370947 shutdown_connections 2024-05-27T09:49:02.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.756+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3542370947 wait complete. 2024-05-27T09:49:02.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.757+0000 7ff97a20eb80 1 -- start start 2024-05-27T09:49:02.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.757+0000 7ff97a20eb80 1 Processor -- start 2024-05-27T09:49:02.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.759+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2d9d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.759+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 0x5593f2d9db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.759+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d9e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.759+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5593f2bb7fa0 con 0x5593f2d93e70 2024-05-27T09:49:02.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.759+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5593f2c52760 con 0x5593f2c14e40 2024-05-27T09:49:02.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.759+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5593f2da0940 con 0x5593f2d009f0 2024-05-27T09:49:02.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.759+0000 7ff97a20eb80 10 client.0 _ll_register_callbacks cb 0x5593f2c5bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:02.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.760+0000 7ff963e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 0x5593f2d9db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.760+0000 7ff963e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 0x5593f2d9db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60212/0 (socket says 172.21.15.44:60212) 2024-05-27T09:49:02.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.760+0000 7ff963e00700 1 -- 172.21.15.44:0/399875961 learned_addr learned my addr 172.21.15.44:0/399875961 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.760+0000 7ff968c00700 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2d9d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.760+0000 7ff963e00700 1 -- 172.21.15.44:0/399875961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 msgr2=0x5593f2d9d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.761+0000 7ff963e00700 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2d9d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.761+0000 7ff963e00700 1 -- 172.21.15.44:0/399875961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 msgr2=0x5593f2d9e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.761+0000 7ff963e00700 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d9e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.761+0000 7ff963e00700 1 -- 172.21.15.44:0/399875961 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593f2c018d0 con 0x5593f2d009f0 2024-05-27T09:49:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.763+0000 7ff968c00700 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2d9d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:49:02.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.763+0000 7ff963e00700 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 0x5593f2d9db80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff9500aa740 tx=0x7ff950021c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8453e0372c325d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:02.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.763+0000 7ff94aa00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:49:02.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.763+0000 7ff94aa00700 1 -- 172.21.15.44:0/399875961 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9500c3020 con 0x5593f2d009f0 2024-05-27T09:49:02.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.764+0000 7ff94aa00700 1 -- 172.21.15.44:0/399875961 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8651+0+0 (secure 0 0 0) 0x7ff9500b3300 con 0x5593f2d009f0 2024-05-27T09:49:02.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.764+0000 7ff97a20eb80 1 -- 172.21.15.44:0/399875961 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9500b1e50 con 0x5593f2d009f0 2024-05-27T09:49:02.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.765+0000 7ff94aa00700 1 -- 172.21.15.44:0/399875961 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7ff9500aeb60 con 0x5593f2d009f0 2024-05-27T09:49:02.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.765+0000 7ff97a20eb80 10 client.27871 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:02.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.765+0000 7ff97a20eb80 1 -- 172.21.15.44:0/399875961 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5593f2c82060 con 0x5593f2d009f0 2024-05-27T09:49:02.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.765+0000 7ff94aa00700 1 -- 172.21.15.44:0/399875961 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9500b1e50 con 0x5593f2d009f0 2024-05-27T09:49:02.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.767+0000 7ff94aa00700 1 -- 172.21.15.44:0/399875961 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7ff9500a9a10 con 0x5593f2d009f0 2024-05-27T09:49:02.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.767+0000 7ff97a20eb80 10 client.27871 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:02.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.767+0000 7ff97a20eb80 10 client.27871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:02.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.767+0000 7ff97a20eb80 20 client.27871 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:02.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.767+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5593f2dad770 0x5593f2dafb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.767+0000 7ff97a20eb80 4 client.27871 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:02.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.767+0000 7ff97a20eb80 1 -- 172.21.15.44:0/399875961 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593f2cc3d90 con 0x5593f2dad770 2024-05-27T09:49:02.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.768+0000 7ff968c00700 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5593f2dad770 0x5593f2dafb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.769+0000 7ff968c00700 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5593f2dad770 0x5593f2dafb90 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7338c15359575b12 server_cookie=1d0fabdbeb79a10e in_seq=0 out_seq=0 2024-05-27T09:49:02.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.769+0000 7ff94aa00700 10 client.27871 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:02.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.770+0000 7ff94aa00700 1 -- 172.21.15.44:0/399875961 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5593f2cc3d90 con 0x5593f2dad770 2024-05-27T09:49:02.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.770+0000 7ff94aa00700 10 client.27871 handle_command_reply: tid=0 2024-05-27T09:49:02.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.770+0000 7ff97a20eb80 1 client.27871 shutdown 2024-05-27T09:49:02.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.771+0000 7ff97a20eb80 1 -- 172.21.15.44:0/399875961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 msgr2=0x5593f2d9db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:02.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.771+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 0x5593f2d9db80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff9500aa740 tx=0x7ff950021c80 comp rx=0 tx=0).stop 2024-05-27T09:49:02.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.772+0000 7ff97a20eb80 1 -- 172.21.15.44:0/399875961 shutdown_connections 2024-05-27T09:49:02.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.772+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5593f2dad770 0x5593f2dafb90 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.772+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d9e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.772+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2d9d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.772+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/399875961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 0x5593f2d9db80 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.772+0000 7ff97a20eb80 1 -- 172.21.15.44:0/399875961 >> 172.21.15.44:0/399875961 conn(0x5593f2cae390 msgr2=0x5593f2d93310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:02.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.772+0000 7ff94aa00700 0 client.27871 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:02.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.772+0000 7ff97a20eb80 1 -- 172.21.15.44:0/399875961 shutdown_connections 2024-05-27T09:49:02.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.773+0000 7ff97a20eb80 1 -- 172.21.15.44:0/399875961 wait complete. 2024-05-27T09:49:02.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.773+0000 7ff97a20eb80 20 client.27871 trim_cache size 0 max 16384 2024-05-27T09:49:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.817+0000 7ff97a20eb80 1 Processor -- start 2024-05-27T09:49:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.817+0000 7ff97a20eb80 1 -- start start 2024-05-27T09:49:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.817+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2c14e40 0x5593f2d98e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.818+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2d009f0 0x5593f2da9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.818+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d955e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.818+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff9641c07e0 con 0x5593f2d93e70 2024-05-27T09:49:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.818+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff9641be010 con 0x5593f2d009f0 2024-05-27T09:49:02.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.818+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff9641bbe60 con 0x5593f2c14e40 2024-05-27T09:49:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.818+0000 7ff968c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2c14e40 0x5593f2d98e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.818+0000 7ff968c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2c14e40 0x5593f2d98e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60226/0 (socket says 172.21.15.44:60226) 2024-05-27T09:49:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.818+0000 7ff968c00700 1 -- 172.21.15.44:0/81164151 learned_addr learned my addr 172.21.15.44:0/81164151 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.818+0000 7ff969600700 1 --2- 172.21.15.44:0/81164151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.819+0000 7ff968c00700 1 -- 172.21.15.44:0/81164151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2d009f0 msgr2=0x5593f2da9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.819+0000 7ff968c00700 1 --2- 172.21.15.44:0/81164151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2d009f0 0x5593f2da9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.819+0000 7ff968c00700 1 -- 172.21.15.44:0/81164151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 msgr2=0x5593f2d955e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.820+0000 7ff968c00700 1 --2- 172.21.15.44:0/81164151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d955e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.820+0000 7ff968c00700 1 -- 172.21.15.44:0/81164151 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593f2d8b240 con 0x5593f2c14e40 2024-05-27T09:49:02.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.820+0000 7ff969600700 1 --2- 172.21.15.44:0/81164151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:02.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.821+0000 7ff968c00700 1 --2- 172.21.15.44:0/81164151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2c14e40 0x5593f2d98e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff958000c00 tx=0x7ff9580097e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e238dd0d7d918896 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:02.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.821+0000 7ff955600700 1 -- 172.21.15.44:0/81164151 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff958004030 con 0x5593f2c14e40 2024-05-27T09:49:02.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.822+0000 7ff955600700 1 -- 172.21.15.44:0/81164151 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9580128c0 con 0x5593f2c14e40 2024-05-27T09:49:02.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.822+0000 7ff955600700 1 -- 172.21.15.44:0/81164151 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff958003b10 con 0x5593f2c14e40 2024-05-27T09:49:02.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.823+0000 7ff97a20eb80 1 -- 172.21.15.44:0/81164151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2c14e40 msgr2=0x5593f2d98e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:02.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.823+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/81164151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2c14e40 0x5593f2d98e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff958000c00 tx=0x7ff9580097e0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.823+0000 7ff97a20eb80 1 -- 172.21.15.44:0/81164151 shutdown_connections 2024-05-27T09:49:02.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.823+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/81164151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.823+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/81164151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2c14e40 0x5593f2d98e30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.823+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/81164151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2d009f0 0x5593f2da9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.824+0000 7ff97a20eb80 1 -- 172.21.15.44:0/81164151 >> 172.21.15.44:0/81164151 conn(0x5593f2cae390 msgr2=0x5593f2d94ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:02.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.826+0000 7ff97a20eb80 1 -- 172.21.15.44:0/81164151 shutdown_connections 2024-05-27T09:49:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.826+0000 7ff97a20eb80 1 -- 172.21.15.44:0/81164151 wait complete. 2024-05-27T09:49:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.827+0000 7ff97a20eb80 1 -- start start 2024-05-27T09:49:02.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.827+0000 7ff97a20eb80 1 Processor -- start 2024-05-27T09:49:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.829+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2da6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.829+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 0x5593f2d97bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.829+0000 7ff97a20eb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d98130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.830+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff9641bc4d0 con 0x5593f2d93e70 2024-05-27T09:49:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.830+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff9641bd940 con 0x5593f2c14e40 2024-05-27T09:49:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.830+0000 7ff97a20eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff9641bcba0 con 0x5593f2d009f0 2024-05-27T09:49:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.830+0000 7ff97a20eb80 10 client.0 _ll_register_callbacks cb 0x5593f2c5bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.830+0000 7ff968c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2da6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.830+0000 7ff968c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2da6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:56362/0 (socket says 172.21.15.44:56362) 2024-05-27T09:49:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.830+0000 7ff968c00700 1 -- 172.21.15.44:0/3699232977 learned_addr learned my addr 172.21.15.44:0/3699232977 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.830+0000 7ff963e00700 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 0x5593f2d97bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.831+0000 7ff969600700 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d98130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.832+0000 7ff963e00700 1 -- 172.21.15.44:0/3699232977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 msgr2=0x5593f2da6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.832+0000 7ff963e00700 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2da6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.832+0000 7ff963e00700 1 -- 172.21.15.44:0/3699232977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 msgr2=0x5593f2d98130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.832+0000 7ff963e00700 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d98130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.832+0000 7ff963e00700 1 -- 172.21.15.44:0/3699232977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593f2d9e430 con 0x5593f2d009f0 2024-05-27T09:49:02.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.835+0000 7ff963e00700 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 0x5593f2d97bf0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff9500ab9a0 tx=0x7ff9500bfcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe796e153272a9e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:02.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.835+0000 7ff94aa00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:49:02.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.835+0000 7ff94aa00700 1 -- 172.21.15.44:0/3699232977 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9500aeb60 con 0x5593f2d009f0 2024-05-27T09:49:02.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.837+0000 7ff94aa00700 1 -- 172.21.15.44:0/3699232977 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8651+0+0 (secure 0 0 0) 0x7ff9500a6da0 con 0x5593f2d009f0 2024-05-27T09:49:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.837+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3699232977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9580128c0 con 0x5593f2d009f0 2024-05-27T09:49:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.837+0000 7ff94aa00700 1 -- 172.21.15.44:0/3699232977 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7ff9500a8b80 con 0x5593f2d009f0 2024-05-27T09:49:02.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.838+0000 7ff97a20eb80 10 client.27877 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:02.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.838+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3699232977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5593f2d9e430 con 0x5593f2d009f0 2024-05-27T09:49:02.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.839+0000 7ff94aa00700 1 -- 172.21.15.44:0/3699232977 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9580128c0 con 0x5593f2d009f0 2024-05-27T09:49:02.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.839+0000 7ff94aa00700 1 -- 172.21.15.44:0/3699232977 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7ff9500a9a10 con 0x5593f2d009f0 2024-05-27T09:49:02.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.839+0000 7ff97a20eb80 10 client.27877 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:02.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.839+0000 7ff97a20eb80 10 client.27877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:02.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.839+0000 7ff97a20eb80 20 client.27877 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:02.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.839+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5593f2e18090 0x5593f2e1a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:02.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.839+0000 7ff97a20eb80 4 client.27877 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:49:02.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.839+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3699232977 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5593f2cbd570 con 0x5593f2e18090 2024-05-27T09:49:02.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.840+0000 7ff968c00700 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5593f2e18090 0x5593f2e1a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:02.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.842+0000 7ff94aa00700 10 client.27877 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.843+0000 7ff968c00700 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5593f2e18090 0x5593f2e1a4b0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c11ec2fc90b3452 server_cookie=8a55c86f625f582c in_seq=0 out_seq=0 2024-05-27T09:49:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.843+0000 7ff94aa00700 1 -- 172.21.15.44:0/3699232977 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5593f2cbd570 con 0x5593f2e18090 2024-05-27T09:49:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.843+0000 7ff94aa00700 10 client.27877 handle_command_reply: tid=0 2024-05-27T09:49:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.843+0000 7ff97a20eb80 1 client.27877 shutdown 2024-05-27T09:49:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.843+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3699232977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 msgr2=0x5593f2d97bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.843+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 0x5593f2d97bf0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff9500ab9a0 tx=0x7ff9500bfcc0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.844+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3699232977 shutdown_connections 2024-05-27T09:49:02.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.844+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5593f2e18090 0x5593f2e1a4b0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.844+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5593f2d93e70 0x5593f2d98130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.844+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5593f2c14e40 0x5593f2da6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.844+0000 7ff97a20eb80 1 --2- 172.21.15.44:0/3699232977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593f2d009f0 0x5593f2d97bf0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.844+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3699232977 >> 172.21.15.44:0/3699232977 conn(0x5593f2cae390 msgr2=0x5593f2d94ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:02.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.844+0000 7ff94aa00700 0 client.27877 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:02.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.844+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3699232977 shutdown_connections 2024-05-27T09:49:02.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.846+0000 7ff97a20eb80 1 -- 172.21.15.44:0/3699232977 wait complete. 2024-05-27T09:49:02.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.847+0000 7ff97a20eb80 20 client.27877 trim_cache size 0 max 16384 2024-05-27T09:49:02.847 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:49:02.847 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T09:49:02.847 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-05-27T09:49:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: "bafbe750-ac8d-41eb-a14b-0f2553ebad05": { 2024-05-27T09:49:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-05-27T09:49:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "bafbe750-ac8d-41eb-a14b-0f2553ebad05", 2024-05-27T09:49:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-05-27T09:49:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:02.848 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:49:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.848+0000 7ff96b600700 1 -- 172.21.15.44:0/3074163628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff94c0b2c00 msgr2=0x7ff94c0b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.848+0000 7ff96b600700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff94c0b2c00 0x7ff94c0b5110 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff95c003fd0 tx=0x7ff95c004ad0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.848+0000 7ff96b600700 1 -- 172.21.15.44:0/3074163628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9640764a0 msgr2=0x7ff96406e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.848+0000 7ff96b600700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9640764a0 0x7ff96406e480 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff9500098e0 tx=0x7ff950005af0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.848+0000 7ff96b600700 1 -- 172.21.15.44:0/3074163628 shutdown_connections 2024-05-27T09:49:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.848+0000 7ff96b600700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff94c0b2c00 0x7ff94c0b5110 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.848+0000 7ff96b600700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff964115360 0x7ff96406ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.848+0000 7ff96b600700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff964075b50 0x7ff964075350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.848+0000 7ff96b600700 1 --2- 172.21.15.44:0/3074163628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9640764a0 0x7ff96406e480 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.848+0000 7ff96b600700 1 -- 172.21.15.44:0/3074163628 >> 172.21.15.44:0/3074163628 conn(0x7ff964100030 msgr2=0x7ff964100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.849+0000 7ff96b600700 1 -- 172.21.15.44:0/3074163628 shutdown_connections 2024-05-27T09:49:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:02.849+0000 7ff96b600700 1 -- 172.21.15.44:0/3074163628 wait complete. 2024-05-27T09:49:03.292 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:03 smithi044 ceph-mon[31618]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T09:49:03.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:03 smithi050 ceph-mon[36371]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T09:49:03.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:03 smithi189 ceph-mon[36671]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T09:49:03.724 INFO:tasks.cephfs.filesystem:scrub status for tag:bafbe750-ac8d-41eb-a14b-0f2553ebad05 - {'path': '/', 'tag': 'bafbe750-ac8d-41eb-a14b-0f2553ebad05', 'options': 'recursive,force'} 2024-05-27T09:49:04.224 INFO:tasks.workunit.client.0.smithi044.stdout:cleaned up hierarchy 2024-05-27T09:49:04.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:04 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:49:04.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:04 smithi050 ceph-mon[36371]: pgmap v435: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.0 MiB/s wr, 263 op/s 2024-05-27T09:49:04.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:04 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:49:04.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:04 smithi189 ceph-mon[36671]: pgmap v435: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.0 MiB/s wr, 263 op/s 2024-05-27T09:49:04.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.361+0000 7ff8a7600640 1 -- 172.21.15.44:0/3330268652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a01029e0 msgr2=0x7ff8a0102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:04.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.361+0000 7ff8a7600640 1 --2- 172.21.15.44:0/3330268652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a01029e0 0x7ff8a0102df0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff894009900 tx=0x7ff89402a490 comp rx=0 tx=0).stop 2024-05-27T09:49:04.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.362+0000 7ff8a7600640 1 -- 172.21.15.44:0/3330268652 shutdown_connections 2024-05-27T09:49:04.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.362+0000 7ff8a7600640 1 --2- 172.21.15.44:0/3330268652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8a0110580 0x7ff8a01129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.362+0000 7ff8a7600640 1 --2- 172.21.15.44:0/3330268652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8a010dc20 0x7ff8a0110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.362+0000 7ff8a7600640 1 --2- 172.21.15.44:0/3330268652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a01029e0 0x7ff8a0102df0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.362 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.362+0000 7ff8a7600640 1 -- 172.21.15.44:0/3330268652 >> 172.21.15.44:0/3330268652 conn(0x7ff8a00782d0 msgr2=0x7ff8a0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.362+0000 7ff8a7600640 1 -- 172.21.15.44:0/3330268652 shutdown_connections 2024-05-27T09:49:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.362+0000 7ff8a7600640 1 -- 172.21.15.44:0/3330268652 wait complete. 2024-05-27T09:49:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.364+0000 7ff8a7600640 1 Processor -- start 2024-05-27T09:49:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.365+0000 7ff8a7600640 1 -- start start 2024-05-27T09:49:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.365+0000 7ff8a7600640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8a01029e0 0x7ff8a01a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.365+0000 7ff8a7600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8a010dc20 0x7ff8a01a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.365+0000 7ff8a7600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0110580 0x7ff8a01a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.365+0000 7ff8a7600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a0116120 con 0x7ff8a010dc20 2024-05-27T09:49:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.365+0000 7ff8a7600640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a0115fa0 con 0x7ff8a01029e0 2024-05-27T09:49:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.366+0000 7ff8a7600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a01162a0 con 0x7ff8a0110580 2024-05-27T09:49:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.366+0000 7ff8a5600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0110580 0x7ff8a01a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.366+0000 7ff8a5600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0110580 0x7ff8a01a1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38542/0 (socket says 172.21.15.44:38542) 2024-05-27T09:49:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.366+0000 7ff8a5600640 1 -- 172.21.15.44:0/1973118319 learned_addr learned my addr 172.21.15.44:0/1973118319 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.366+0000 7ff89fe00640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8a010dc20 0x7ff8a01a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.367+0000 7ff8a5600640 1 -- 172.21.15.44:0/1973118319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8a01029e0 msgr2=0x7ff8a01a3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.367+0000 7ff8a5600640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8a01029e0 0x7ff8a01a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.367+0000 7ff8a5600640 1 -- 172.21.15.44:0/1973118319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8a010dc20 msgr2=0x7ff8a01a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.367+0000 7ff8a5600640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8a010dc20 0x7ff8a01a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.367+0000 7ff8a5600640 1 -- 172.21.15.44:0/1973118319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a01a22e0 con 0x7ff8a0110580 2024-05-27T09:49:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.367+0000 7ff8a5600640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0110580 0x7ff8a01a1c30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff89800eb10 tx=0x7ff89800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.368+0000 7ff89d600640 1 -- 172.21.15.44:0/1973118319 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898004b20 con 0x7ff8a0110580 2024-05-27T09:49:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.368+0000 7ff89d600640 1 -- 172.21.15.44:0/1973118319 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff898004cc0 con 0x7ff8a0110580 2024-05-27T09:49:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.368+0000 7ff89d600640 1 -- 172.21.15.44:0/1973118319 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898010670 con 0x7ff8a0110580 2024-05-27T09:49:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.368+0000 7ff8a7600640 1 -- 172.21.15.44:0/1973118319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a010c830 con 0x7ff8a0110580 2024-05-27T09:49:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.368+0000 7ff8a7600640 1 -- 172.21.15.44:0/1973118319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a010ccd0 con 0x7ff8a0110580 2024-05-27T09:49:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.374+0000 7ff8ae569740 1 Processor -- start 2024-05-27T09:49:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.377+0000 7ff8ae569740 1 -- start start 2024-05-27T09:49:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.377+0000 7ff89d600640 1 -- 172.21.15.44:0/1973118319 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff898005020 con 0x7ff8a0110580 2024-05-27T09:49:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.378+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 0x5579d71ec860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.378+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e212f0 0x5579d71ecda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.378+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e294d0 0x5579d71ed2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.378+0000 7ff8ae569740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5579d715db90 con 0x5579d6e4a1b0 2024-05-27T09:49:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.378+0000 7ff8ae569740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5579d714d730 con 0x5579d6e294d0 2024-05-27T09:49:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.378+0000 7ff8ae569740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5579d7151770 con 0x5579d6e212f0 2024-05-27T09:49:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.379+0000 7ff89fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 0x5579d71ec860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.379+0000 7ff89fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 0x5579d71ec860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50538/0 (socket says 172.21.15.44:50538) 2024-05-27T09:49:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.379+0000 7ff89fe00640 1 -- 172.21.15.44:0/3310573148 learned_addr learned my addr 172.21.15.44:0/3310573148 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.379+0000 7ff89fe00640 1 --2- 172.21.15.44:0/3310573148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e294d0 0x5579d71ed2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.380+0000 7ff89fe00640 1 -- 172.21.15.44:0/3310573148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e212f0 msgr2=0x5579d71ecda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:49:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.380+0000 7ff89fe00640 1 --2- 172.21.15.44:0/3310573148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e212f0 0x5579d71ecda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.380+0000 7ff89fe00640 1 -- 172.21.15.44:0/3310573148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e294d0 msgr2=0x5579d71ed2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.380+0000 7ff89fe00640 1 --2- 172.21.15.44:0/3310573148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e294d0 0x5579d71ed2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.380+0000 7ff89fe00640 1 -- 172.21.15.44:0/3310573148 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579d70e68b0 con 0x5579d6e4a1b0 2024-05-27T09:49:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.381+0000 7ff89fe00640 1 --2- 172.21.15.44:0/3310573148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 0x5579d71ec860 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff88c00af20 tx=0x7ff88c00a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9c7783aa64e246 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.381+0000 7ff891c00640 1 -- 172.21.15.44:0/3310573148 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c01c070 con 0x5579d6e4a1b0 2024-05-27T09:49:04.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.381+0000 7ff891c00640 1 -- 172.21.15.44:0/3310573148 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff88c017ac0 con 0x5579d6e4a1b0 2024-05-27T09:49:04.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.381+0000 7ff891c00640 1 -- 172.21.15.44:0/3310573148 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c00f040 con 0x5579d6e4a1b0 2024-05-27T09:49:04.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.382+0000 7ff89d600640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff8880b2a20 0x7ff8880b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.383+0000 7ff8ae569740 1 -- 172.21.15.44:0/3310573148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 msgr2=0x5579d71ec860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:04.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.383+0000 7ff8ae569740 1 --2- 172.21.15.44:0/3310573148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 0x5579d71ec860 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff88c00af20 tx=0x7ff88c00a0d0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.383+0000 7ff89d600640 1 -- 172.21.15.44:0/1973118319 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8651+0+0 (secure 0 0 0) 0x7ff8980e2e20 con 0x7ff8a0110580 2024-05-27T09:49:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.384+0000 7ff8a4c00640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff8880b2a20 0x7ff8880b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.385+0000 7ff8ae569740 1 -- 172.21.15.44:0/3310573148 shutdown_connections 2024-05-27T09:49:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.385+0000 7ff8ae569740 1 --2- 172.21.15.44:0/3310573148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e294d0 0x5579d71ed2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.385+0000 7ff8ae569740 1 --2- 172.21.15.44:0/3310573148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e212f0 0x5579d71ecda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.385+0000 7ff8ae569740 1 --2- 172.21.15.44:0/3310573148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 0x5579d71ec860 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.385+0000 7ff8ae569740 1 -- 172.21.15.44:0/3310573148 >> 172.21.15.44:0/3310573148 conn(0x5579d6e3e300 msgr2=0x5579d71e9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.386+0000 7ff8ae569740 1 -- 172.21.15.44:0/3310573148 shutdown_connections 2024-05-27T09:49:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.386+0000 7ff8ae569740 1 -- 172.21.15.44:0/3310573148 wait complete. 2024-05-27T09:49:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.386+0000 7ff8a4c00640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff8880b2a20 0x7ff8880b4f30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff89402aa80 tx=0x7ff894000bc0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:04.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.388+0000 7ff8ae569740 1 -- start start 2024-05-27T09:49:04.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.388+0000 7ff8ae569740 1 Processor -- start 2024-05-27T09:49:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e212f0 0x5579d71f6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 0x5579d71f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 0x5579d7202cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff8ae569740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5579d6e33790 con 0x5579d6e212f0 2024-05-27T09:49:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff8ae569740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5579d70f5bc0 con 0x5579d6e4a1b0 2024-05-27T09:49:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff8ae569740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5579d71f9990 con 0x5579d6e294d0 2024-05-27T09:49:04.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff8ae569740 10 client.0 _ll_register_callbacks cb 0x5579d70d3400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:04.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff89fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 0x5579d71f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff89fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 0x5579d71f71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38574/0 (socket says 172.21.15.44:38574) 2024-05-27T09:49:04.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff89fe00640 1 -- 172.21.15.44:0/2362066334 learned_addr learned my addr 172.21.15.44:0/2362066334 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff8a5600640 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 0x5579d7202cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.392+0000 7ff8a4c00640 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e212f0 0x5579d71f6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.393+0000 7ff8a5600640 1 -- 172.21.15.44:0/2362066334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 msgr2=0x5579d71f71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:04.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.393+0000 7ff8a5600640 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 0x5579d71f71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.393+0000 7ff8a5600640 1 -- 172.21.15.44:0/2362066334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e212f0 msgr2=0x5579d71f6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.393+0000 7ff8a5600640 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e212f0 0x5579d71f6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.393+0000 7ff8a5600640 1 -- 172.21.15.44:0/2362066334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579d6e62210 con 0x5579d6e4a1b0 2024-05-27T09:49:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.394+0000 7ff886000640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:49:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.394+0000 7ff8a5600640 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 0x5579d7202cf0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff8980e0490 tx=0x7ff8980a9850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5f0704aa0fa6371 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.395+0000 7ff886000640 1 -- 172.21.15.44:0/2362066334 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8980c3020 con 0x5579d6e4a1b0 2024-05-27T09:49:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.395+0000 7ff8ae569740 1 -- 172.21.15.44:0/2362066334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff88c017ac0 con 0x5579d6e4a1b0 2024-05-27T09:49:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.395+0000 7ff886000640 1 -- 172.21.15.44:0/2362066334 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8651+0+0 (secure 0 0 0) 0x7ff8980e0df0 con 0x5579d6e4a1b0 2024-05-27T09:49:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.395+0000 7ff886000640 1 -- 172.21.15.44:0/2362066334 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7ff8980ae3d0 con 0x5579d6e4a1b0 2024-05-27T09:49:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.396+0000 7ff8ae569740 10 client.27878 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.396+0000 7ff8ae569740 1 -- 172.21.15.44:0/2362066334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5579d6f2e620 con 0x5579d6e4a1b0 2024-05-27T09:49:04.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.397+0000 7ff886000640 1 -- 172.21.15.44:0/2362066334 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff88c017ac0 con 0x5579d6e4a1b0 2024-05-27T09:49:04.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.397+0000 7ff886000640 1 -- 172.21.15.44:0/2362066334 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7ff8980a5990 con 0x5579d6e4a1b0 2024-05-27T09:49:04.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.397+0000 7ff8ae569740 10 client.27878 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:04.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.397+0000 7ff8ae569740 10 client.27878 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:04.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.397+0000 7ff8ae569740 20 client.27878 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.397+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5579d6e08d80 0x5579d7208de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.397+0000 7ff8ae569740 4 client.27878 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.397+0000 7ff8ae569740 1 -- 172.21.15.44:0/2362066334 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579d715fae0 con 0x5579d6e08d80 2024-05-27T09:49:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.398+0000 7ff8a4c00640 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5579d6e08d80 0x5579d7208de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.401+0000 7ff8a4c00640 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5579d6e08d80 0x5579d7208de0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daef4c17bd2141f server_cookie=3051f226669a677e in_seq=0 out_seq=0 2024-05-27T09:49:04.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.401+0000 7ff886000640 10 client.27878 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.402+0000 7ff886000640 1 -- 172.21.15.44:0/2362066334 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5579d715fae0 con 0x5579d6e08d80 2024-05-27T09:49:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.402+0000 7ff886000640 10 client.27878 handle_command_reply: tid=0 2024-05-27T09:49:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.402+0000 7ff8ae569740 1 client.27878 shutdown 2024-05-27T09:49:04.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.403+0000 7ff8ae569740 1 -- 172.21.15.44:0/2362066334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 msgr2=0x5579d7202cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:04.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.403+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 0x5579d7202cf0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff8980e0490 tx=0x7ff8980a9850 comp rx=0 tx=0).stop 2024-05-27T09:49:04.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.404+0000 7ff8ae569740 1 -- 172.21.15.44:0/2362066334 shutdown_connections 2024-05-27T09:49:04.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.404+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5579d6e08d80 0x5579d7208de0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.404+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 0x5579d7202cf0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.404+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 0x5579d71f71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.404+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2362066334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e212f0 0x5579d71f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.404+0000 7ff8ae569740 1 -- 172.21.15.44:0/2362066334 >> 172.21.15.44:0/2362066334 conn(0x5579d6e3e300 msgr2=0x5579d71ec630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.404+0000 7ff886000640 0 client.27878 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.404+0000 7ff8ae569740 1 -- 172.21.15.44:0/2362066334 shutdown_connections 2024-05-27T09:49:04.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.410+0000 7ff8ae569740 1 -- 172.21.15.44:0/2362066334 wait complete. 2024-05-27T09:49:04.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.410+0000 7ff8ae569740 20 client.27878 trim_cache size 0 max 16384 2024-05-27T09:49:04.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.447+0000 7ff8ae569740 1 Processor -- start 2024-05-27T09:49:04.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.448+0000 7ff8ae569740 1 -- start start 2024-05-27T09:49:04.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.448+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e212f0 0x5579d71f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.448+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e294d0 0x5579d71f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.448+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 0x5579d71f8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.448+0000 7ff8ae569740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a01c9930 con 0x5579d6e294d0 2024-05-27T09:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.448+0000 7ff8ae569740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a0074260 con 0x5579d6e4a1b0 2024-05-27T09:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.448+0000 7ff8ae569740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a00720d0 con 0x5579d6e212f0 2024-05-27T09:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.449+0000 7ff8a4c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e212f0 0x5579d71f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.449+0000 7ff8a5600640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 0x5579d71f8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.449+0000 7ff8a4c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e212f0 0x5579d71f22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38584/0 (socket says 172.21.15.44:38584) 2024-05-27T09:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.449+0000 7ff8a4c00640 1 -- 172.21.15.44:0/30131646 learned_addr learned my addr 172.21.15.44:0/30131646 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.449+0000 7ff89fe00640 1 --2- 172.21.15.44:0/30131646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e294d0 0x5579d71f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.449+0000 7ff8a4c00640 1 -- 172.21.15.44:0/30131646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 msgr2=0x5579d71f8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.449+0000 7ff8a4c00640 1 --2- 172.21.15.44:0/30131646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 0x5579d71f8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.449+0000 7ff8a4c00640 1 -- 172.21.15.44:0/30131646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e294d0 msgr2=0x5579d71f88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.449+0000 7ff8a4c00640 1 --2- 172.21.15.44:0/30131646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e294d0 0x5579d71f88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.449+0000 7ff8a4c00640 1 -- 172.21.15.44:0/30131646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579d71edc40 con 0x5579d6e212f0 2024-05-27T09:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.449+0000 7ff8a5600640 1 --2- 172.21.15.44:0/30131646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 0x5579d71f8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.450+0000 7ff8a4c00640 1 --2- 172.21.15.44:0/30131646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e212f0 0x5579d71f22e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff894009900 tx=0x7ff89403d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4a76caa450423d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.451+0000 7ff891200640 1 -- 172.21.15.44:0/30131646 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894035040 con 0x5579d6e212f0 2024-05-27T09:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.451+0000 7ff891200640 1 -- 172.21.15.44:0/30131646 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff894055070 con 0x5579d6e212f0 2024-05-27T09:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.451+0000 7ff891200640 1 -- 172.21.15.44:0/30131646 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894048950 con 0x5579d6e212f0 2024-05-27T09:49:04.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.454+0000 7ff8ae569740 1 -- 172.21.15.44:0/30131646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e212f0 msgr2=0x5579d71f22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:04.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.454+0000 7ff8ae569740 1 --2- 172.21.15.44:0/30131646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e212f0 0x5579d71f22e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff894009900 tx=0x7ff89403d000 comp rx=0 tx=0).stop 2024-05-27T09:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.455+0000 7ff8ae569740 1 -- 172.21.15.44:0/30131646 shutdown_connections 2024-05-27T09:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.455+0000 7ff8ae569740 1 --2- 172.21.15.44:0/30131646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e4a1b0 0x5579d71f8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.455+0000 7ff8ae569740 1 --2- 172.21.15.44:0/30131646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e294d0 0x5579d71f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.455+0000 7ff8ae569740 1 --2- 172.21.15.44:0/30131646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e212f0 0x5579d71f22e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.455+0000 7ff8ae569740 1 -- 172.21.15.44:0/30131646 >> 172.21.15.44:0/30131646 conn(0x5579d6e3e300 msgr2=0x5579d71f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.455+0000 7ff8ae569740 1 -- 172.21.15.44:0/30131646 shutdown_connections 2024-05-27T09:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.455+0000 7ff8ae569740 1 -- 172.21.15.44:0/30131646 wait complete. 2024-05-27T09:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.455+0000 7ff8ae569740 1 -- start start 2024-05-27T09:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.456+0000 7ff8ae569740 1 Processor -- start 2024-05-27T09:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.457+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e212f0 0x5579d71fc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.457+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 0x5579d71fc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.457+0000 7ff8ae569740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 0x5579d71fe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.457+0000 7ff8ae569740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a0072720 con 0x5579d6e4a1b0 2024-05-27T09:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.457+0000 7ff8ae569740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a0073b90 con 0x5579d6e212f0 2024-05-27T09:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.457+0000 7ff8ae569740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a0072df0 con 0x5579d6e294d0 2024-05-27T09:49:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.457+0000 7ff8ae569740 10 client.0 _ll_register_callbacks cb 0x7ff8980a9970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.457+0000 7ff89fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 0x5579d71fc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.458+0000 7ff89fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 0x5579d71fc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38586/0 (socket says 172.21.15.44:38586) 2024-05-27T09:49:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.458+0000 7ff89fe00640 1 -- 172.21.15.44:0/2708465429 learned_addr learned my addr 172.21.15.44:0/2708465429 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.458+0000 7ff8a4c00640 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e212f0 0x5579d71fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.458+0000 7ff8a5600640 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 0x5579d71fe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.459+0000 7ff89fe00640 1 -- 172.21.15.44:0/2708465429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e212f0 msgr2=0x5579d71fc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.459+0000 7ff89fe00640 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e212f0 0x5579d71fc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.459+0000 7ff89fe00640 1 -- 172.21.15.44:0/2708465429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 msgr2=0x5579d71fe9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.459+0000 7ff89fe00640 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 0x5579d71fe9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.459+0000 7ff89fe00640 1 -- 172.21.15.44:0/2708465429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579d71fef20 con 0x5579d6e294d0 2024-05-27T09:49:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.460+0000 7ff89fe00640 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 0x5579d71fc960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x5579d71e3020 tx=0x7ff88c00a700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e0c68ae092d0f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.460+0000 7ff886000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:49:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.460+0000 7ff886000640 1 -- 172.21.15.44:0/2708465429 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c01c040 con 0x5579d6e294d0 2024-05-27T09:49:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.461+0000 7ff886000640 1 -- 172.21.15.44:0/2708465429 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8651+0+0 (secure 0 0 0) 0x7ff88c00adf0 con 0x5579d6e294d0 2024-05-27T09:49:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.461+0000 7ff8ae569740 1 -- 172.21.15.44:0/2708465429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff894055070 con 0x5579d6e294d0 2024-05-27T09:49:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.461+0000 7ff886000640 1 -- 172.21.15.44:0/2708465429 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7ff88c016370 con 0x5579d6e294d0 2024-05-27T09:49:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.461+0000 7ff8ae569740 10 client.27892 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.462+0000 7ff8ae569740 1 -- 172.21.15.44:0/2708465429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5579d7270ed0 con 0x5579d6e294d0 2024-05-27T09:49:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.462+0000 7ff886000640 1 -- 172.21.15.44:0/2708465429 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff894055070 con 0x5579d6e294d0 2024-05-27T09:49:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.462+0000 7ff886000640 1 -- 172.21.15.44:0/2708465429 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7ff88c015040 con 0x5579d6e294d0 2024-05-27T09:49:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.463+0000 7ff8ae569740 10 client.27892 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.463+0000 7ff8ae569740 10 client.27892 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.463+0000 7ff8ae569740 20 client.27892 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.463+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5579d6dde1c0 0x5579d7273220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.463+0000 7ff8ae569740 4 client.27892 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:49:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.463+0000 7ff8ae569740 1 -- 172.21.15.44:0/2708465429 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5579d71592c0 con 0x5579d6dde1c0 2024-05-27T09:49:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.463+0000 7ff8a4c00640 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5579d6dde1c0 0x5579d7273220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.467+0000 7ff886000640 10 client.27892 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:04.467+0000 7ff8a4c00640 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5579d6dde1c0 0x5579d7273220 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5883a870f57bd9a server_cookie=f0501d86764a324 in_seq=0 out_seq=0 2024-05-27T09:49:04.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:04 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:49:04.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:04 smithi044 ceph-mon[31618]: pgmap v435: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.0 MiB/s wr, 263 op/s 2024-05-27T09:49:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.180+0000 7ff886000640 1 -- 172.21.15.44:0/2708465429 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5579d71592c0 con 0x5579d6dde1c0 2024-05-27T09:49:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.180+0000 7ff886000640 10 client.27892 handle_command_reply: tid=0 2024-05-27T09:49:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.181+0000 7ff8ae569740 1 client.27892 shutdown 2024-05-27T09:49:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.181+0000 7ff8ae569740 1 -- 172.21.15.44:0/2708465429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 msgr2=0x5579d71fc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.181+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 0x5579d71fc960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x5579d71e3020 tx=0x7ff88c00a700 comp rx=0 tx=0).stop 2024-05-27T09:49:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.181+0000 7ff8ae569740 1 -- 172.21.15.44:0/2708465429 shutdown_connections 2024-05-27T09:49:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.181+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5579d6dde1c0 0x5579d7273220 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.182+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5579d6e4a1b0 0x5579d71fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.182+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579d6e294d0 0x5579d71fc960 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.182+0000 7ff8ae569740 1 --2- 172.21.15.44:0/2708465429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579d6e212f0 0x5579d71fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.182+0000 7ff8ae569740 1 -- 172.21.15.44:0/2708465429 >> 172.21.15.44:0/2708465429 conn(0x5579d6e3e300 msgr2=0x5579d7201a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.182+0000 7ff886000640 0 client.27892 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.182+0000 7ff8ae569740 1 -- 172.21.15.44:0/2708465429 shutdown_connections 2024-05-27T09:49:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.182+0000 7ff8ae569740 1 -- 172.21.15.44:0/2708465429 wait complete. 2024-05-27T09:49:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.183+0000 7ff8ae569740 20 client.27892 trim_cache size 0 max 16384 2024-05-27T09:49:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:49:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:49:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:49:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.184+0000 7ff8a7600640 1 -- 172.21.15.44:0/1973118319 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff8880b2a20 msgr2=0x7ff8880b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.184+0000 7ff8a7600640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff8880b2a20 0x7ff8880b4f30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff89402aa80 tx=0x7ff894000bc0 comp rx=0 tx=0).stop 2024-05-27T09:49:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.184+0000 7ff8a7600640 1 -- 172.21.15.44:0/1973118319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0110580 msgr2=0x7ff8a01a1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.184+0000 7ff8a7600640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0110580 0x7ff8a01a1c30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff89800eb10 tx=0x7ff89800c2d0 comp rx=0 tx=0).stop 2024-05-27T09:49:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.184+0000 7ff8a7600640 1 -- 172.21.15.44:0/1973118319 shutdown_connections 2024-05-27T09:49:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.184+0000 7ff8a7600640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0110580 0x7ff8a01a1c30 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.184+0000 7ff8a7600640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff8880b2a20 0x7ff8880b4f30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.184+0000 7ff8a7600640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8a010dc20 0x7ff8a01a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.184+0000 7ff8a7600640 1 --2- 172.21.15.44:0/1973118319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8a01029e0 0x7ff8a01a3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.184+0000 7ff8a7600640 1 -- 172.21.15.44:0/1973118319 >> 172.21.15.44:0/1973118319 conn(0x7ff8a00782d0 msgr2=0x7ff8a00ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.185+0000 7ff8a7600640 1 -- 172.21.15.44:0/1973118319 shutdown_connections 2024-05-27T09:49:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:05.185+0000 7ff8a7600640 1 -- 172.21.15.44:0/1973118319 wait complete. 2024-05-27T09:49:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:06 smithi189 ceph-mon[36671]: pgmap v436: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.0 MiB/s wr, 263 op/s 2024-05-27T09:49:06.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:06 smithi044 ceph-mon[31618]: pgmap v436: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.0 MiB/s wr, 263 op/s 2024-05-27T09:49:06.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:06 smithi050 ceph-mon[36371]: pgmap v436: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.0 MiB/s wr, 263 op/s 2024-05-27T09:49:08.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:08 smithi189 ceph-mon[36671]: pgmap v437: 97 pgs: 22 active+clean+snaptrim, 33 active+clean, 42 active+clean+snaptrim_wait; 344 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.1 MiB/s wr, 544 op/s 2024-05-27T09:49:08.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:08 smithi044 ceph-mon[31618]: pgmap v437: 97 pgs: 22 active+clean+snaptrim, 33 active+clean, 42 active+clean+snaptrim_wait; 344 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.1 MiB/s wr, 544 op/s 2024-05-27T09:49:08.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:08 smithi050 ceph-mon[36371]: pgmap v437: 97 pgs: 22 active+clean+snaptrim, 33 active+clean, 42 active+clean+snaptrim_wait; 344 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.1 MiB/s wr, 544 op/s 2024-05-27T09:49:10.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:10 smithi044 ceph-mon[31618]: pgmap v438: 97 pgs: 22 active+clean+snaptrim, 33 active+clean, 42 active+clean+snaptrim_wait; 344 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.1 MiB/s wr, 544 op/s 2024-05-27T09:49:10.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:10 smithi189 ceph-mon[36671]: pgmap v438: 97 pgs: 22 active+clean+snaptrim, 33 active+clean, 42 active+clean+snaptrim_wait; 344 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.1 MiB/s wr, 544 op/s 2024-05-27T09:49:10.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:10 smithi050 ceph-mon[36371]: pgmap v438: 97 pgs: 22 active+clean+snaptrim, 33 active+clean, 42 active+clean+snaptrim_wait; 344 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.1 MiB/s wr, 544 op/s 2024-05-27T09:49:11.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:11 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:49:11.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:11 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:49:11.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:11 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:49:12.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:12 smithi044 ceph-mon[31618]: pgmap v439: 97 pgs: 13 active+clean+snaptrim, 59 active+clean, 25 active+clean+snaptrim_wait; 345 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 668 KiB/s wr, 494 op/s 2024-05-27T09:49:12.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:12 smithi189 ceph-mon[36671]: pgmap v439: 97 pgs: 13 active+clean+snaptrim, 59 active+clean, 25 active+clean+snaptrim_wait; 345 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 668 KiB/s wr, 494 op/s 2024-05-27T09:49:12.445 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:12 smithi050 ceph-mon[36371]: pgmap v439: 97 pgs: 13 active+clean+snaptrim, 59 active+clean, 25 active+clean+snaptrim_wait; 345 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 668 KiB/s wr, 494 op/s 2024-05-27T09:49:13.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:13 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:49:13.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:13 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:49:13.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:13 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:49:13.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:13 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:49:13.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:13 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:49:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:13 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:49:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:13 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:49:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:13 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:49:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:13 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:49:13.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:13 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:49:13.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:13 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:49:13.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:13 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:49:13.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:13 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:49:13.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:13 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:49:13.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:13 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:49:14.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:14 smithi189 ceph-mon[36671]: pgmap v440: 97 pgs: 13 active+clean+snaptrim, 59 active+clean, 25 active+clean+snaptrim_wait; 345 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 618 KiB/s wr, 457 op/s 2024-05-27T09:49:14.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:14 smithi044 ceph-mon[31618]: pgmap v440: 97 pgs: 13 active+clean+snaptrim, 59 active+clean, 25 active+clean+snaptrim_wait; 345 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 618 KiB/s wr, 457 op/s 2024-05-27T09:49:14.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:14 smithi050 ceph-mon[36371]: pgmap v440: 97 pgs: 13 active+clean+snaptrim, 59 active+clean, 25 active+clean+snaptrim_wait; 345 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 618 KiB/s wr, 457 op/s 2024-05-27T09:49:16.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:16 smithi044 ceph-mon[31618]: pgmap v441: 97 pgs: 13 active+clean+snaptrim, 59 active+clean, 25 active+clean+snaptrim_wait; 345 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 557 KiB/s wr, 411 op/s 2024-05-27T09:49:16.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:16 smithi050 ceph-mon[36371]: pgmap v441: 97 pgs: 13 active+clean+snaptrim, 59 active+clean, 25 active+clean+snaptrim_wait; 345 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 557 KiB/s wr, 411 op/s 2024-05-27T09:49:16.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:16 smithi189 ceph-mon[36671]: pgmap v441: 97 pgs: 13 active+clean+snaptrim, 59 active+clean, 25 active+clean+snaptrim_wait; 345 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 557 KiB/s wr, 411 op/s 2024-05-27T09:49:18.389 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:18 smithi189 ceph-mon[36671]: pgmap v442: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 562 KiB/s wr, 417 op/s 2024-05-27T09:49:18.389 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:18 smithi189 ceph-mon[36671]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T09:49:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.429+0000 7f2c84c00640 1 -- 172.21.15.44:0/3769864558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c80109ab0 msgr2=0x7f2c801036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.429+0000 7f2c84c00640 1 --2- 172.21.15.44:0/3769864558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c80109ab0 0x7f2c801036a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2c70009ca0 tx=0x7f2c7002a8b0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.429+0000 7f2c84c00640 1 -- 172.21.15.44:0/3769864558 shutdown_connections 2024-05-27T09:49:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.429+0000 7f2c84c00640 1 --2- 172.21.15.44:0/3769864558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c80104250 0x7f2c801105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.429+0000 7f2c84c00640 1 --2- 172.21.15.44:0/3769864558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c80109f50 0x7f2c80103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.429+0000 7f2c84c00640 1 --2- 172.21.15.44:0/3769864558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c80109ab0 0x7f2c801036a0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.429+0000 7f2c84c00640 1 -- 172.21.15.44:0/3769864558 >> 172.21.15.44:0/3769864558 conn(0x7f2c80079240 msgr2=0x7f2c80079680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.429+0000 7f2c84c00640 1 -- 172.21.15.44:0/3769864558 shutdown_connections 2024-05-27T09:49:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.429+0000 7f2c84c00640 1 -- 172.21.15.44:0/3769864558 wait complete. 2024-05-27T09:49:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.430+0000 7f2c84c00640 1 Processor -- start 2024-05-27T09:49:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.431+0000 7f2c84c00640 1 -- start start 2024-05-27T09:49:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.431+0000 7f2c84c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c80104250 0x7f2c8019d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.431+0000 7f2c84c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c80109ab0 0x7f2c8019dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.431+0000 7f2c84c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c80109f50 0x7f2c801a85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.431+0000 7f2c84c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2c80113ca0 con 0x7f2c80109ab0 2024-05-27T09:49:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.431+0000 7f2c84c00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2c80113b20 con 0x7f2c80109f50 2024-05-27T09:49:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.431+0000 7f2c84c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2c80113e20 con 0x7f2c80104250 2024-05-27T09:49:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.431+0000 7f2c7f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c80104250 0x7f2c8019d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.431+0000 7f2c7f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c80104250 0x7f2c8019d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35868/0 (socket says 172.21.15.44:35868) 2024-05-27T09:49:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.431+0000 7f2c7f400640 1 -- 172.21.15.44:0/3104390754 learned_addr learned my addr 172.21.15.44:0/3104390754 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.431+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c80109f50 0x7f2c801a85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.432+0000 7f2c7ea00640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c80109ab0 0x7f2c8019dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.432+0000 7f2c7f400640 1 -- 172.21.15.44:0/3104390754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c80109f50 msgr2=0x7f2c801a85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.432+0000 7f2c7f400640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c80109f50 0x7f2c801a85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.432+0000 7f2c7f400640 1 -- 172.21.15.44:0/3104390754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c80109ab0 msgr2=0x7f2c8019dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.432+0000 7f2c7f400640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c80109ab0 0x7f2c8019dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.432+0000 7f2c7f400640 1 -- 172.21.15.44:0/3104390754 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c801a8b20 con 0x7f2c80104250 2024-05-27T09:49:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.432+0000 7f2c7f400640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c80104250 0x7f2c8019d740 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2c70006010 tx=0x7f2c70030da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.433+0000 7f2c77e00640 1 -- 172.21.15.44:0/3104390754 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c70004040 con 0x7f2c80104250 2024-05-27T09:49:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.433+0000 7f2c77e00640 1 -- 172.21.15.44:0/3104390754 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c70040070 con 0x7f2c80104250 2024-05-27T09:49:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.433+0000 7f2c84c00640 1 -- 172.21.15.44:0/3104390754 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c801a8d50 con 0x7f2c80104250 2024-05-27T09:49:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.433+0000 7f2c77e00640 1 -- 172.21.15.44:0/3104390754 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c70045c30 con 0x7f2c80104250 2024-05-27T09:49:18.435 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.433+0000 7f2c84c00640 1 -- 172.21.15.44:0/3104390754 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c801a9210 con 0x7f2c80104250 2024-05-27T09:49:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.436+0000 7f2c8d1aa740 1 Processor -- start 2024-05-27T09:49:18.437 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c8d1aa740 1 -- start start 2024-05-27T09:49:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 0x5625a4124a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a4124fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.438 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 0x5625a41254e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5625a4095b90 con 0x5625a3d592f0 2024-05-27T09:49:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5625a4085730 con 0x5625a3d614d0 2024-05-27T09:49:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5625a4089770 con 0x5625a3d821b0 2024-05-27T09:49:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c77e00640 1 -- 172.21.15.44:0/3104390754 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2c700041e0 con 0x7f2c80104250 2024-05-27T09:49:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c7fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 0x5625a4124a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c7ea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 0x5625a41254e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c7fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 0x5625a4124a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35878/0 (socket says 172.21.15.44:35878) 2024-05-27T09:49:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c7ea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 0x5625a41254e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37260/0 (socket says 172.21.15.44:37260) 2024-05-27T09:49:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c7fe00640 1 -- 172.21.15.44:0/1913212139 learned_addr learned my addr 172.21.15.44:0/1913212139 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.437+0000 7f2c7f400640 1 --2- 172.21.15.44:0/1913212139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a4124fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.438+0000 7f2c7fe00640 1 -- 172.21.15.44:0/1913212139 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 msgr2=0x5625a41254e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:18.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.438+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/1913212139 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 0x5625a41254e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.438+0000 7f2c7fe00640 1 -- 172.21.15.44:0/1913212139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 msgr2=0x5625a4124fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:18.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.438+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/1913212139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a4124fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.438+0000 7f2c7fe00640 1 -- 172.21.15.44:0/1913212139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625a401e8b0 con 0x5625a3d821b0 2024-05-27T09:49:18.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.438+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/1913212139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 0x5625a4124a60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800b680 tx=0x7f2c7800bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eab24d53951fd783 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:18.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.438+0000 7f2c6fe00640 1 -- 172.21.15.44:0/1913212139 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78003bd0 con 0x5625a3d821b0 2024-05-27T09:49:18.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.439+0000 7f2c6fe00640 1 -- 172.21.15.44:0/1913212139 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c78015070 con 0x5625a3d821b0 2024-05-27T09:49:18.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.439+0000 7f2c6fe00640 1 -- 172.21.15.44:0/1913212139 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c780106a0 con 0x5625a3d821b0 2024-05-27T09:49:18.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.439+0000 7f2c77e00640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c640b2950 0x7f2c640b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.439+0000 7f2c77e00640 1 -- 172.21.15.44:0/3104390754 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f2c70103730 con 0x7f2c80104250 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.439+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/1913212139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 msgr2=0x5625a4124a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.439+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/1913212139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 0x5625a4124a60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800b680 tx=0x7f2c7800bb40 comp rx=0 tx=0).stop 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.439+0000 7f2c7f400640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c640b2950 0x7f2c640b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.440+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/1913212139 shutdown_connections 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.440+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/1913212139 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 0x5625a41254e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.440+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/1913212139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a4124fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.440+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/1913212139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 0x5625a4124a60 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.440+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/1913212139 >> 172.21.15.44:0/1913212139 conn(0x5625a3d76300 msgr2=0x5625a4121210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.440+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/1913212139 shutdown_connections 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.440+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/1913212139 wait complete. 2024-05-27T09:49:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.440+0000 7f2c7f400640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c640b2950 0x7f2c640b4e60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x5625a4122550 tx=0x7f2c700b0f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.441+0000 7f2c8d1aa740 1 -- start start 2024-05-27T09:49:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.441+0000 7f2c8d1aa740 1 Processor -- start 2024-05-27T09:49:18.445 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d592f0 0x5625a412ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d614d0 0x5625a412f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d821b0 0x5625a413aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5625a3d6b790 con 0x5625a3d614d0 2024-05-27T09:49:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5625a402dbc0 con 0x5625a3d821b0 2024-05-27T09:49:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5625a4131b90 con 0x5625a3d592f0 2024-05-27T09:49:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c8d1aa740 10 client.0 _ll_register_callbacks cb 0x5625a400b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c7ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d614d0 0x5625a412f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c7ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d614d0 0x5625a412f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37020/0 (socket says 172.21.15.44:37020) 2024-05-27T09:49:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c7ea00640 1 -- 172.21.15.44:0/2476942860 learned_addr learned my addr 172.21.15.44:0/2476942860 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d592f0 0x5625a412ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.442+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d821b0 0x5625a413aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.443+0000 7f2c7ea00640 1 -- 172.21.15.44:0/2476942860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d592f0 msgr2=0x5625a412ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.443+0000 7f2c7ea00640 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d592f0 0x5625a412ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.443+0000 7f2c7ea00640 1 -- 172.21.15.44:0/2476942860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d821b0 msgr2=0x5625a413aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.443+0000 7f2c7ea00640 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d821b0 0x5625a413aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.443+0000 7f2c7ea00640 1 -- 172.21.15.44:0/2476942860 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625a3d9a210 con 0x5625a3d614d0 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.443+0000 7f2c7ea00640 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d614d0 0x5625a412f110 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2c68012700 tx=0x7f2c68012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f7155db75d555b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.443+0000 7f2c6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.443+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2476942860 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6800f920 con 0x5625a3d614d0 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.443+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2476942860 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f2c6800fac0 con 0x5625a3d614d0 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.443+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2476942860 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c78015070 con 0x5625a3d614d0 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.444+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2476942860 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f2c6800fcc0 con 0x5625a3d614d0 2024-05-27T09:49:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.444+0000 7f2c8d1aa740 10 client.17625 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.444+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2476942860 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5625a3e66620 con 0x5625a3d614d0 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.444+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2476942860 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c78015070 con 0x5625a3d614d0 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.444+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2476942860 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f2c6801c420 con 0x5625a3d614d0 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.444+0000 7f2c8d1aa740 10 client.17625 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.444+0000 7f2c8d1aa740 10 client.17625 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.444+0000 7f2c8d1aa740 20 client.17625 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.444+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a3d40d80 0x5625a4140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.444+0000 7f2c8d1aa740 4 client.17625 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.444+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2476942860 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625a4097ae0 con 0x5625a3d40d80 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.445+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a3d40d80 0x5625a4140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.445+0000 7f2c6cc00640 10 client.17625 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.445+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a3d40d80 0x5625a4140ec0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59c6466869593664 server_cookie=95f4abc00c9f4adc in_seq=0 out_seq=0 2024-05-27T09:49:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.446+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2476942860 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5625a4097ae0 con 0x5625a3d40d80 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.446+0000 7f2c6cc00640 10 client.17625 handle_command_reply: tid=0 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.446+0000 7f2c8d1aa740 1 client.17625 shutdown 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.446+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2476942860 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d614d0 msgr2=0x5625a412f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.446+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d614d0 0x5625a412f110 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2c68012700 tx=0x7f2c68012bc0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.447+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2476942860 shutdown_connections 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.447+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a3d40d80 0x5625a4140ec0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.447+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d821b0 0x5625a413aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.447+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d614d0 0x5625a412f110 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.447+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2476942860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d592f0 0x5625a412ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.447+0000 7f2c6cc00640 0 client.17625 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.447+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2476942860 >> 172.21.15.44:0/2476942860 conn(0x5625a3d76300 msgr2=0x5625a4124890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.447+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2476942860 shutdown_connections 2024-05-27T09:49:18.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.447+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2476942860 wait complete. 2024-05-27T09:49:18.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.447+0000 7f2c8d1aa740 20 client.17625 trim_cache size 0 max 16384 2024-05-27T09:49:18.464 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.464+0000 7f2c8d1aa740 1 Processor -- start 2024-05-27T09:49:18.464 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.464+0000 7f2c8d1aa740 1 -- start start 2024-05-27T09:49:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.464+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a412a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.464+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 0x5625a412eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.464+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 0x5625a412f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.464+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801b4510 con 0x5625a3d592f0 2024-05-27T09:49:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.464+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801b1d80 con 0x5625a3d614d0 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.464+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801afbf0 con 0x5625a3d821b0 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a412a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7ea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 0x5625a412eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7ea00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 0x5625a412eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37278/0 (socket says 172.21.15.44:37278) 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7ea00640 1 -- 172.21.15.44:0/1121353932 learned_addr learned my addr 172.21.15.44:0/1121353932 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/1121353932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 0x5625a412f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7fe00640 1 -- 172.21.15.44:0/1121353932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 msgr2=0x5625a412f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/1121353932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 0x5625a412f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7fe00640 1 -- 172.21.15.44:0/1121353932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 msgr2=0x5625a412eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/1121353932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 0x5625a412eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7fe00640 1 -- 172.21.15.44:0/1121353932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625a412d120 con 0x5625a3d592f0 2024-05-27T09:49:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7ea00640 1 --2- 172.21.15.44:0/1121353932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 0x5625a412eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.465+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/1121353932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a412a3e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2c78004650 tx=0x7f2c78011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b04e257cf3cd2ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.466+0000 7f2c6f400640 1 -- 172.21.15.44:0/1121353932 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78011640 con 0x5625a3d592f0 2024-05-27T09:49:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.466+0000 7f2c6f400640 1 -- 172.21.15.44:0/1121353932 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c78015070 con 0x5625a3d592f0 2024-05-27T09:49:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.466+0000 7f2c6f400640 1 -- 172.21.15.44:0/1121353932 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78004970 con 0x5625a3d592f0 2024-05-27T09:49:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.467+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/1121353932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 msgr2=0x5625a412a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.467+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/1121353932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a412a3e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2c78004650 tx=0x7f2c78011c90 comp rx=0 tx=0).stop 2024-05-27T09:49:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.467+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/1121353932 shutdown_connections 2024-05-27T09:49:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.467+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/1121353932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d821b0 0x5625a412f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.467+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/1121353932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d614d0 0x5625a412eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.467+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/1121353932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a412a3e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.467+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/1121353932 >> 172.21.15.44:0/1121353932 conn(0x5625a3d76300 msgr2=0x5625a412e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.467+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/1121353932 shutdown_connections 2024-05-27T09:49:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.467+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/1121353932 wait complete. 2024-05-27T09:49:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.467+0000 7f2c8d1aa740 1 -- start start 2024-05-27T09:49:18.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.467+0000 7f2c8d1aa740 1 Processor -- start 2024-05-27T09:49:18.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.468+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a41373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.468+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d614d0 0x5625a4132740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.468+0000 7f2c8d1aa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d821b0 0x5625a4132c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.468+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801b0240 con 0x5625a3d592f0 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.468+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801b16b0 con 0x5625a3d821b0 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.468+0000 7f2c8d1aa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801b0910 con 0x5625a3d614d0 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.468+0000 7f2c8d1aa740 10 client.0 _ll_register_callbacks cb 0x7f2c68009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.468+0000 7f2c7fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a41373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.468+0000 7f2c7ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d614d0 0x5625a4132740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.468+0000 7f2c7fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a41373f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37044/0 (socket says 172.21.15.44:37044) 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.468+0000 7f2c7fe00640 1 -- 172.21.15.44:0/2826754278 learned_addr learned my addr 172.21.15.44:0/2826754278 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.469+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d821b0 0x5625a4132c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.469+0000 7f2c7ea00640 1 -- 172.21.15.44:0/2826754278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d821b0 msgr2=0x5625a4132c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.469+0000 7f2c7ea00640 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d821b0 0x5625a4132c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.469+0000 7f2c7ea00640 1 -- 172.21.15.44:0/2826754278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 msgr2=0x5625a41373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.469+0000 7f2c7ea00640 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a41373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.469+0000 7f2c7ea00640 1 -- 172.21.15.44:0/2826754278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625a4133270 con 0x5625a3d614d0 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.469+0000 7f2c7ea00640 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d614d0 0x5625a4132740 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2c68012420 tx=0x7f2c68009fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e031a3f6b54440db server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.469+0000 7f2c6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.469+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2826754278 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6801c420 con 0x5625a3d614d0 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.469+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2826754278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c78015070 con 0x5625a3d614d0 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.469+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2826754278 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f2c6801c5c0 con 0x5625a3d614d0 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2826754278 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f2c680050d0 con 0x5625a3d614d0 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c8d1aa740 10 client.27910 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2826754278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5625a41a9340 con 0x5625a3d614d0 2024-05-27T09:49:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2826754278 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c78015070 con 0x5625a3d614d0 2024-05-27T09:49:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2826754278 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f2c68003c00 con 0x5625a3d614d0 2024-05-27T09:49:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c8d1aa740 10 client.27910 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c8d1aa740 10 client.27910 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c8d1aa740 20 client.27910 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a3d161c0 0x5625a41aba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c8d1aa740 4 client.27910 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:49:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2826754278 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5625a40912c0 con 0x5625a3d161c0 2024-05-27T09:49:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.470+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a3d161c0 0x5625a41aba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.471+0000 7f2c7fe00640 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a3d161c0 0x5625a41aba20 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=868db60be9211908 server_cookie=6b57b3a9e84ba37e in_seq=0 out_seq=0 2024-05-27T09:49:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.471+0000 7f2c6cc00640 10 client.27910 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:18.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:18 smithi044 ceph-mon[31618]: pgmap v442: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 562 KiB/s wr, 417 op/s 2024-05-27T09:49:18.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:18 smithi044 ceph-mon[31618]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T09:49:18.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:18 smithi050 ceph-mon[36371]: pgmap v442: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 562 KiB/s wr, 417 op/s 2024-05-27T09:49:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:18 smithi050 ceph-mon[36371]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T09:49:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c6cc00640 1 -- 172.21.15.44:0/2826754278 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5625a40912c0 con 0x5625a3d161c0 2024-05-27T09:49:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c6cc00640 10 client.27910 handle_command_reply: tid=0 2024-05-27T09:49:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 1 client.27910 shutdown 2024-05-27T09:49:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2826754278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d614d0 msgr2=0x5625a4132740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d614d0 0x5625a4132740 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2c68012420 tx=0x7f2c68009fc0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2826754278 shutdown_connections 2024-05-27T09:49:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5625a3d161c0 0x5625a41aba20 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5625a3d821b0 0x5625a4132c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625a3d614d0 0x5625a4132740 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 1 --2- 172.21.15.44:0/2826754278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5625a3d592f0 0x5625a41373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c6cc00640 0 client.27910 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2826754278 >> 172.21.15.44:0/2826754278 conn(0x5625a3d76300 msgr2=0x5625a412e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2826754278 shutdown_connections 2024-05-27T09:49:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 1 -- 172.21.15.44:0/2826754278 wait complete. 2024-05-27T09:49:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.618+0000 7f2c8d1aa740 20 client.27910 trim_cache size 0 max 16384 2024-05-27T09:49:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:49:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:49:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:49:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 -- 172.21.15.44:0/3104390754 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c640b2950 msgr2=0x7f2c640b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c640b2950 0x7f2c640b4e60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x5625a4122550 tx=0x7f2c700b0f90 comp rx=0 tx=0).stop 2024-05-27T09:49:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 -- 172.21.15.44:0/3104390754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c80104250 msgr2=0x7f2c8019d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c80104250 0x7f2c8019d740 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2c70006010 tx=0x7f2c70030da0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 -- 172.21.15.44:0/3104390754 shutdown_connections 2024-05-27T09:49:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c80109f50 0x7f2c801a85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2c640b2950 0x7f2c640b4e60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c80109ab0 0x7f2c8019dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 --2- 172.21.15.44:0/3104390754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c80104250 0x7f2c8019d740 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 -- 172.21.15.44:0/3104390754 >> 172.21.15.44:0/3104390754 conn(0x7f2c80079240 msgr2=0x7f2c80102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 -- 172.21.15.44:0/3104390754 shutdown_connections 2024-05-27T09:49:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:18.619+0000 7f2c84c00640 1 -- 172.21.15.44:0/3104390754 wait complete. 2024-05-27T09:49:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:20 smithi044 ceph-mon[31618]: pgmap v444: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 75 KiB/s wr, 39 op/s 2024-05-27T09:49:20.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:20 smithi050 ceph-mon[36371]: pgmap v444: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 75 KiB/s wr, 39 op/s 2024-05-27T09:49:20.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:20 smithi189 ceph-mon[36671]: pgmap v444: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 75 KiB/s wr, 39 op/s 2024-05-27T09:49:22.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:22 smithi044 ceph-mon[31618]: pgmap v445: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 29 KiB/s wr, 124 op/s 2024-05-27T09:49:22.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:22 smithi050 ceph-mon[36371]: pgmap v445: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 29 KiB/s wr, 124 op/s 2024-05-27T09:49:22.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:22 smithi189 ceph-mon[36671]: pgmap v445: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 29 KiB/s wr, 124 op/s 2024-05-27T09:49:24.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:24 smithi044 ceph-mon[31618]: pgmap v446: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 29 KiB/s wr, 124 op/s 2024-05-27T09:49:24.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:24 smithi050 ceph-mon[36371]: pgmap v446: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 29 KiB/s wr, 124 op/s 2024-05-27T09:49:24.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:24 smithi189 ceph-mon[36671]: pgmap v446: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 29 KiB/s wr, 124 op/s 2024-05-27T09:49:24.783 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T09:49:24.783 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T09:49:24.901 INFO:tasks.workunit:Running workunit fs/misc/filelock_deadlock.py... 2024-05-27T09:49:24.901 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/filelock_deadlock.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6507388dd5057528934822c0163b0c347ef1d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_deadlock.py 2024-05-27T09:49:26.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:26 smithi044 ceph-mon[31618]: pgmap v447: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 29 KiB/s wr, 124 op/s 2024-05-27T09:49:26.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:26 smithi050 ceph-mon[36371]: pgmap v447: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 29 KiB/s wr, 124 op/s 2024-05-27T09:49:26.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:26 smithi189 ceph-mon[36671]: pgmap v447: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 29 KiB/s wr, 124 op/s 2024-05-27T09:49:27.605 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-05-27T09:49:27.606 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:49:28.065 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:49:28.406 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:28 smithi044 ceph-mon[31618]: pgmap v448: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 57 KiB/s wr, 132 op/s 2024-05-27T09:49:28.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:28 smithi050 ceph-mon[36371]: pgmap v448: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 57 KiB/s wr, 132 op/s 2024-05-27T09:49:28.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:28 smithi189 ceph-mon[36671]: pgmap v448: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 57 KiB/s wr, 132 op/s 2024-05-27T09:49:30.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.238+0000 7fd15b000700 1 -- 172.21.15.44:0/1594661599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd15410b2d0 msgr2=0x7fd154117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:30.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.238+0000 7fd15b000700 1 --2- 172.21.15.44:0/1594661599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd15410b2d0 0x7fd154117800 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00b0a0 tx=0x7fd14c00b3b0 comp rx=0 tx=0).stop 2024-05-27T09:49:30.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.238+0000 7fd15b000700 1 -- 172.21.15.44:0/1594661599 shutdown_connections 2024-05-27T09:49:30.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.238+0000 7fd15b000700 1 --2- 172.21.15.44:0/1594661599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd15410b2d0 0x7fd154117800 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:30.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.238+0000 7fd15b000700 1 --2- 172.21.15.44:0/1594661599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd154110960 0x7fd154110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:30.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.238+0000 7fd15b000700 1 --2- 172.21.15.44:0/1594661599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd15410a980 0x7fd15410ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:30.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.238+0000 7fd15b000700 1 -- 172.21.15.44:0/1594661599 >> 172.21.15.44:0/1594661599 conn(0x7fd154077c70 msgr2=0x7fd1540780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:30.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.238+0000 7fd15b000700 1 -- 172.21.15.44:0/1594661599 shutdown_connections 2024-05-27T09:49:30.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.239+0000 7fd15b000700 1 -- 172.21.15.44:0/1594661599 wait complete. 2024-05-27T09:49:30.242 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:30 smithi044 ceph-mon[31618]: pgmap v449: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 52 KiB/s wr, 121 op/s 2024-05-27T09:49:30.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.240+0000 7fd15b000700 1 Processor -- start 2024-05-27T09:49:30.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.240+0000 7fd15b000700 1 -- start start 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.241+0000 7fd15b000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd15410a980 0x7fd1541a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.241+0000 7fd15b000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd15410b2d0 0x7fd1541a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.241+0000 7fd15b000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd154110960 0x7fd1541a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.241+0000 7fd15b000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd15411aad0 con 0x7fd15410a980 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.241+0000 7fd15b000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd15411a950 con 0x7fd154110960 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.241+0000 7fd15b000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd15411ac50 con 0x7fd15410b2d0 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.241+0000 7fd153e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd15410a980 0x7fd1541a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.241+0000 7fd153e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd15410a980 0x7fd1541a1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37332/0 (socket says 172.21.15.44:37332) 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.241+0000 7fd153e00700 1 -- 172.21.15.44:0/1703451113 learned_addr learned my addr 172.21.15.44:0/1703451113 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.241+0000 7fd153400700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd15410b2d0 0x7fd1541a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.241+0000 7fd159000700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd154110960 0x7fd1541a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:30.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.242+0000 7fd153e00700 1 -- 172.21.15.44:0/1703451113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd15410b2d0 msgr2=0x7fd1541a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:30.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.242+0000 7fd153e00700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd15410b2d0 0x7fd1541a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:30.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.242+0000 7fd153e00700 1 -- 172.21.15.44:0/1703451113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd154110960 msgr2=0x7fd1541a5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:30.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.242+0000 7fd153e00700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd154110960 0x7fd1541a5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:30.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.242+0000 7fd153e00700 1 -- 172.21.15.44:0/1703451113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1541a62e0 con 0x7fd15410a980 2024-05-27T09:49:30.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.242+0000 7fd159000700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd154110960 0x7fd1541a5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:30.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.242+0000 7fd153400700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd15410b2d0 0x7fd1541a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:30.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.242+0000 7fd153e00700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd15410a980 0x7fd1541a1460 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd14400d950 tx=0x7fd14400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.242+0000 7fd150c00700 1 -- 172.21.15.44:0/1703451113 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144016070 con 0x7fd15410a980 2024-05-27T09:49:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.242+0000 7fd15b000700 1 -- 172.21.15.44:0/1703451113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1541a65d0 con 0x7fd15410a980 2024-05-27T09:49:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.243+0000 7fd150c00700 1 -- 172.21.15.44:0/1703451113 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd144004830 con 0x7fd15410a980 2024-05-27T09:49:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.243+0000 7fd150c00700 1 -- 172.21.15.44:0/1703451113 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144005520 con 0x7fd15410a980 2024-05-27T09:49:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.243+0000 7fd15b000700 1 -- 172.21.15.44:0/1703451113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1541adef0 con 0x7fd15410a980 2024-05-27T09:49:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.244+0000 7fd15b000700 1 -- 172.21.15.44:0/1703451113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1540512b0 con 0x7fd15410a980 2024-05-27T09:49:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.246+0000 7fd150c00700 1 -- 172.21.15.44:0/1703451113 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd144004220 con 0x7fd15410a980 2024-05-27T09:49:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.248+0000 7fd150c00700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1380b2a70 0x7fd1380b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:30.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.248+0000 7fd150c00700 1 -- 172.21.15.44:0/1703451113 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7fd1440e22f0 con 0x7fd15410a980 2024-05-27T09:49:30.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.248+0000 7fd153400700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1380b2a70 0x7fd1380b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:30.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.249+0000 7fd153400700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1380b2a70 0x7fd1380b4f80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd154109790 tx=0x7fd140009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:30.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.253+0000 7fd150c00700 1 -- 172.21.15.44:0/1703451113 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd14401f020 con 0x7fd15410a980 2024-05-27T09:49:30.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.542+0000 7fd15b000700 1 -- 172.21.15.44:0/1703451113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1541a2770 con 0x7fd15410a980 2024-05-27T09:49:30.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:30 smithi050 ceph-mon[36371]: pgmap v449: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 52 KiB/s wr, 121 op/s 2024-05-27T09:49:30.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.543+0000 7fd150c00700 1 -- 172.21.15.44:0/1703451113 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11870 (secure 0 0 0) 0x7fd1440a9c20 con 0x7fd15410a980 2024-05-27T09:49:30.545 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:49:30.545 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:48:16.864365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24503]},"id":1}]} 2024-05-27T09:49:30.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.546+0000 7fd15b000700 1 -- 172.21.15.44:0/1703451113 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1380b2a70 msgr2=0x7fd1380b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:30.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.546+0000 7fd15b000700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1380b2a70 0x7fd1380b4f80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd154109790 tx=0x7fd140009260 comp rx=0 tx=0).stop 2024-05-27T09:49:30.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.546+0000 7fd15b000700 1 -- 172.21.15.44:0/1703451113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd15410a980 msgr2=0x7fd1541a1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:30.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.546+0000 7fd15b000700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd15410a980 0x7fd1541a1460 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd14400d950 tx=0x7fd14400dc60 comp rx=0 tx=0).stop 2024-05-27T09:49:30.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.546+0000 7fd15b000700 1 -- 172.21.15.44:0/1703451113 shutdown_connections 2024-05-27T09:49:30.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.546+0000 7fd15b000700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd1380b2a70 0x7fd1380b4f80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:30.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.546+0000 7fd15b000700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd154110960 0x7fd1541a5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:30.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.546+0000 7fd15b000700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd15410a980 0x7fd1541a1460 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:30.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.546+0000 7fd15b000700 1 --2- 172.21.15.44:0/1703451113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd15410b2d0 0x7fd1541a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:30.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.546+0000 7fd15b000700 1 -- 172.21.15.44:0/1703451113 >> 172.21.15.44:0/1703451113 conn(0x7fd154077c70 msgr2=0x7fd15407b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:30.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.547+0000 7fd15b000700 1 -- 172.21.15.44:0/1703451113 shutdown_connections 2024-05-27T09:49:30.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:30.547+0000 7fd15b000700 1 -- 172.21.15.44:0/1703451113 wait complete. 2024-05-27T09:49:30.550 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 43 2024-05-27T09:49:30.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:30 smithi189 ceph-mon[36671]: pgmap v449: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 52 KiB/s wr, 121 op/s 2024-05-27T09:49:31.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:31 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1703451113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:49:31.466 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-05-27T09:49:31.466 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-05-27T09:49:31.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:31 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1703451113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:49:31.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:31 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1703451113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:49:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.847+0000 7f5718000640 1 -- 172.21.15.44:0/1399968080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5710112310 msgr2=0x7f5710114730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.847+0000 7f5718000640 1 --2- 172.21.15.44:0/1399968080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5710112310 0x7f5710114730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009900 tx=0x7f56fc02a480 comp rx=0 tx=0).stop 2024-05-27T09:49:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.847+0000 7f5718000640 1 -- 172.21.15.44:0/1399968080 shutdown_connections 2024-05-27T09:49:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.847+0000 7f5718000640 1 --2- 172.21.15.44:0/1399968080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5710114c70 0x7f5710117090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.847+0000 7f5718000640 1 --2- 172.21.15.44:0/1399968080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5710112310 0x7f5710114730 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.847+0000 7f5718000640 1 --2- 172.21.15.44:0/1399968080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5710075480 0x7f5710075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.847+0000 7f5718000640 1 -- 172.21.15.44:0/1399968080 >> 172.21.15.44:0/1399968080 conn(0x7f57100ff480 msgr2=0x7f57101018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.847+0000 7f5718000640 1 -- 172.21.15.44:0/1399968080 shutdown_connections 2024-05-27T09:49:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.847+0000 7f5718000640 1 -- 172.21.15.44:0/1399968080 wait complete. 2024-05-27T09:49:31.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.848+0000 7f5718000640 1 Processor -- start 2024-05-27T09:49:31.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.848+0000 7f5718000640 1 -- start start 2024-05-27T09:49:31.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.849+0000 7f5718000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5710075480 0x7f57101a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.849+0000 7f5718000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5710112310 0x7f57101a1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.849+0000 7f5718000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5710114c70 0x7f57101a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.849+0000 7f5718000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f571011a470 con 0x7f5710112310 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.849+0000 7f5718000640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f571011a2f0 con 0x7f5710114c70 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.849+0000 7f5718000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f571011a5f0 con 0x7f5710075480 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.849+0000 7f5714c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5710112310 0x7f57101a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.849+0000 7f5714c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5710112310 0x7f57101a1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37344/0 (socket says 172.21.15.44:37344) 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.849+0000 7f5714c00640 1 -- 172.21.15.44:0/3377320782 learned_addr learned my addr 172.21.15.44:0/3377320782 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.849+0000 7f5715600640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5710075480 0x7f57101a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.849+0000 7f5716000640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5710114c70 0x7f57101a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.850+0000 7f5714c00640 1 -- 172.21.15.44:0/3377320782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5710075480 msgr2=0x7f57101a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.850+0000 7f5714c00640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5710075480 0x7f57101a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.850+0000 7f5714c00640 1 -- 172.21.15.44:0/3377320782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5710114c70 msgr2=0x7f57101a5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.850+0000 7f5714c00640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5710114c70 0x7f57101a5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.850+0000 7f5714c00640 1 -- 172.21.15.44:0/3377320782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57101a5d50 con 0x7f5710112310 2024-05-27T09:49:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.850+0000 7f5714c00640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5710112310 0x7f57101a1450 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f56fc005940 tx=0x7f56fc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.850+0000 7f570e000640 1 -- 172.21.15.44:0/3377320782 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc03f030 con 0x7f5710112310 2024-05-27T09:49:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.851+0000 7f570e000640 1 -- 172.21.15.44:0/3377320782 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56fc0044c0 con 0x7f5710112310 2024-05-27T09:49:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.851+0000 7f5718000640 1 -- 172.21.15.44:0/3377320782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57101a5fe0 con 0x7f5710112310 2024-05-27T09:49:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.851+0000 7f570e000640 1 -- 172.21.15.44:0/3377320782 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc035070 con 0x7f5710112310 2024-05-27T09:49:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.851+0000 7f5718000640 1 -- 172.21.15.44:0/3377320782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57101a64d0 con 0x7f5710112310 2024-05-27T09:49:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.856+0000 7f571f00f740 1 Processor -- start 2024-05-27T09:49:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.856+0000 7f571f00f740 1 -- start start 2024-05-27T09:49:31.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.856+0000 7f571f00f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3d8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.856+0000 7f571f00f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3d8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.856+0000 7f571f00f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3d92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.856+0000 7f571f00f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559f1a349b90 con 0x559f1a00d2f0 2024-05-27T09:49:31.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.856+0000 7f571f00f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559f1a339730 con 0x559f1a0154d0 2024-05-27T09:49:31.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.856+0000 7f571f00f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559f1a33d770 con 0x559f1a0361b0 2024-05-27T09:49:31.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.856+0000 7f570e000640 1 -- 172.21.15.44:0/3377320782 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f56fc042030 con 0x7f5710112310 2024-05-27T09:49:31.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.857+0000 7f5715600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3d8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.857+0000 7f5714c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3d92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.857+0000 7f5714c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3d92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:60200/0 (socket says 172.21.15.44:60200) 2024-05-27T09:49:31.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.857+0000 7f5714c00640 1 -- 172.21.15.44:0/232387373 learned_addr learned my addr 172.21.15.44:0/232387373 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:31.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.857+0000 7f5716000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3d8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.857+0000 7f5715600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3d8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37360/0 (socket says 172.21.15.44:37360) 2024-05-27T09:49:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.858+0000 7f5715600640 1 -- 172.21.15.44:0/232387373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 msgr2=0x559f1a3d8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.858+0000 7f5715600640 1 --2- 172.21.15.44:0/232387373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3d8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.858+0000 7f5715600640 1 -- 172.21.15.44:0/232387373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 msgr2=0x559f1a3d92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.858+0000 7f5715600640 1 --2- 172.21.15.44:0/232387373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3d92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.858+0000 7f5715600640 1 -- 172.21.15.44:0/232387373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f1a2d28b0 con 0x559f1a00d2f0 2024-05-27T09:49:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.858+0000 7f5714c00640 1 --2- 172.21.15.44:0/232387373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3d92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.858+0000 7f5715600640 1 --2- 172.21.15.44:0/232387373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3d8da0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5704009c00 tx=0x7f5704011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be298573374d7e23 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.858+0000 7f5702600640 1 -- 172.21.15.44:0/232387373 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704004230 con 0x559f1a00d2f0 2024-05-27T09:49:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.858+0000 7f5702600640 1 -- 172.21.15.44:0/232387373 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57040043d0 con 0x559f1a00d2f0 2024-05-27T09:49:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.858+0000 7f5702600640 1 -- 172.21.15.44:0/232387373 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704004c30 con 0x559f1a00d2f0 2024-05-27T09:49:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.859+0000 7f570e000640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f56f80b2950 0x7f56f80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.859+0000 7f570e000640 1 -- 172.21.15.44:0/3377320782 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f56fc102990 con 0x7f5710112310 2024-05-27T09:49:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.859+0000 7f5716000640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f56f80b2950 0x7f56f80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.860+0000 7f5716000640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f56f80b2950 0x7f56f80b4e60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x559f1a3d59d0 tx=0x7f5708009500 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.860+0000 7f571f00f740 1 -- 172.21.15.44:0/232387373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 msgr2=0x559f1a3d8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.860+0000 7f571f00f740 1 --2- 172.21.15.44:0/232387373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3d8da0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5704009c00 tx=0x7f5704011810 comp rx=0 tx=0).stop 2024-05-27T09:49:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.860+0000 7f571f00f740 1 -- 172.21.15.44:0/232387373 shutdown_connections 2024-05-27T09:49:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.860+0000 7f571f00f740 1 --2- 172.21.15.44:0/232387373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3d92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.860+0000 7f571f00f740 1 --2- 172.21.15.44:0/232387373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3d8da0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.860+0000 7f571f00f740 1 --2- 172.21.15.44:0/232387373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3d8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.860+0000 7f571f00f740 1 -- 172.21.15.44:0/232387373 >> 172.21.15.44:0/232387373 conn(0x559f1a02a300 msgr2=0x559f1a3d5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.860+0000 7f571f00f740 1 -- 172.21.15.44:0/232387373 shutdown_connections 2024-05-27T09:49:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.860+0000 7f571f00f740 1 -- 172.21.15.44:0/232387373 wait complete. 2024-05-27T09:49:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.861+0000 7f571f00f740 1 -- start start 2024-05-27T09:49:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.861+0000 7f571f00f740 1 Processor -- start 2024-05-27T09:49:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f571f00f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f571f00f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3e2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f571f00f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3eecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f571f00f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559f1a01f790 con 0x559f1a00d2f0 2024-05-27T09:49:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f571f00f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559f1a2e1bc0 con 0x559f1a0154d0 2024-05-27T09:49:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f571f00f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559f1a3e5990 con 0x559f1a0361b0 2024-05-27T09:49:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f571f00f740 10 client.0 _ll_register_callbacks cb 0x559f1a2bf400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f5715600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f5714c00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3e2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f5715600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37370/0 (socket says 172.21.15.44:37370) 2024-05-27T09:49:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f5715600640 1 -- 172.21.15.44:0/2874432048 learned_addr learned my addr 172.21.15.44:0/2874432048 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f5716000640 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3eecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f5714c00640 1 -- 172.21.15.44:0/2874432048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 msgr2=0x559f1a3eecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f5714c00640 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3eecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f5714c00640 1 -- 172.21.15.44:0/2874432048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 msgr2=0x559f1a3e2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.862+0000 7f5714c00640 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.863+0000 7f5714c00640 1 -- 172.21.15.44:0/2874432048 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f1a04e210 con 0x559f1a0154d0 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.863+0000 7f5715600640 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.863+0000 7f5714c00640 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3e2ed0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0c6e90 tx=0x7f56fc005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea6ecee777ec0806 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.863+0000 7f56f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.863+0000 7f56f6a00640 1 -- 172.21.15.44:0/2874432048 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc049040 con 0x559f1a0154d0 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.863+0000 7f56f6a00640 1 -- 172.21.15.44:0/2874432048 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f56fc0d2800 con 0x559f1a0154d0 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.863+0000 7f571f00f740 1 -- 172.21.15.44:0/2874432048 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57040043d0 con 0x559f1a0154d0 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.863+0000 7f56f6a00640 1 -- 172.21.15.44:0/2874432048 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f56fc0c5ae0 con 0x559f1a0154d0 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.864+0000 7f571f00f740 10 client.27920 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.864+0000 7f571f00f740 1 -- 172.21.15.44:0/2874432048 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559f1a11a620 con 0x559f1a0154d0 2024-05-27T09:49:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.864+0000 7f56f6a00640 1 -- 172.21.15.44:0/2874432048 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57040043d0 con 0x559f1a0154d0 2024-05-27T09:49:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.864+0000 7f56f6a00640 1 -- 172.21.15.44:0/2874432048 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f56fc0c5590 con 0x559f1a0154d0 2024-05-27T09:49:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.864+0000 7f571f00f740 10 client.27920 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.864+0000 7f571f00f740 10 client.27920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.864+0000 7f571f00f740 20 client.27920 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.864+0000 7f571f00f740 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559f19ff4d80 0x559f1a3f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.864+0000 7f571f00f740 4 client.27920 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.864+0000 7f571f00f740 1 -- 172.21.15.44:0/2874432048 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f1a34bae0 con 0x559f19ff4d80 2024-05-27T09:49:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.865+0000 7f5715600640 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559f19ff4d80 0x559f1a3f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.866+0000 7f5715600640 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559f19ff4d80 0x559f1a3f4cd0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8884c82369001c3 server_cookie=232e560361690d40 in_seq=0 out_seq=0 2024-05-27T09:49:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.866+0000 7f56f6a00640 10 client.27920 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:31.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.866+0000 7f56f6a00640 1 -- 172.21.15.44:0/2874432048 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559f1a34bae0 con 0x559f19ff4d80 2024-05-27T09:49:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.866+0000 7f56f6a00640 10 client.27920 handle_command_reply: tid=0 2024-05-27T09:49:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 1 client.27920 shutdown 2024-05-27T09:49:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 1 -- 172.21.15.44:0/2874432048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 msgr2=0x559f1a3e2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3e2ed0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0c6e90 tx=0x7f56fc005c90 comp rx=0 tx=0).stop 2024-05-27T09:49:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 1 -- 172.21.15.44:0/2874432048 shutdown_connections 2024-05-27T09:49:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559f19ff4d80 0x559f1a3f4cd0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3eecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3e2ed0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 1 --2- 172.21.15.44:0/2874432048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f56f6a00640 0 client.27920 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 1 -- 172.21.15.44:0/2874432048 >> 172.21.15.44:0/2874432048 conn(0x559f1a02a300 msgr2=0x559f1a3d8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 1 -- 172.21.15.44:0/2874432048 shutdown_connections 2024-05-27T09:49:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 1 -- 172.21.15.44:0/2874432048 wait complete. 2024-05-27T09:49:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.867+0000 7f571f00f740 20 client.27920 trim_cache size 0 max 16384 2024-05-27T09:49:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.885+0000 7f571f00f740 1 Processor -- start 2024-05-27T09:49:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.885+0000 7f571f00f740 1 -- start start 2024-05-27T09:49:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.885+0000 7f571f00f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3d51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f571f00f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3d4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f571f00f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3e4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f571f00f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f57101b8a90 con 0x559f1a00d2f0 2024-05-27T09:49:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f571f00f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f57101b6300 con 0x559f1a0154d0 2024-05-27T09:49:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f571f00f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f57101b4170 con 0x559f1a0361b0 2024-05-27T09:49:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f5716000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3e4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f5715600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3d51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f5716000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3e4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:33470/0 (socket says 172.21.15.44:33470) 2024-05-27T09:49:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f5716000640 1 -- 172.21.15.44:0/1675177918 learned_addr learned my addr 172.21.15.44:0/1675177918 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f5714c00640 1 --2- 172.21.15.44:0/1675177918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3d4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f5716000640 1 -- 172.21.15.44:0/1675177918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 msgr2=0x559f1a3d4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f5716000640 1 --2- 172.21.15.44:0/1675177918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3d4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f5716000640 1 -- 172.21.15.44:0/1675177918 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 msgr2=0x559f1a3d51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f5716000640 1 --2- 172.21.15.44:0/1675177918 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3d51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f5716000640 1 -- 172.21.15.44:0/1675177918 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f1a3e3230 con 0x559f1a0361b0 2024-05-27T09:49:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.886+0000 7f5714c00640 1 --2- 172.21.15.44:0/1675177918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3d4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.887+0000 7f5716000640 1 --2- 172.21.15.44:0/1675177918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3e4750 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f570800f9a0 tx=0x7f5708017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58599765e26056b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.887+0000 7f5701c00640 1 -- 172.21.15.44:0/1675177918 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570801d4d0 con 0x559f1a0361b0 2024-05-27T09:49:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.887+0000 7f5701c00640 1 -- 172.21.15.44:0/1675177918 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f570802e070 con 0x559f1a0361b0 2024-05-27T09:49:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.887+0000 7f5701c00640 1 -- 172.21.15.44:0/1675177918 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57080299c0 con 0x559f1a0361b0 2024-05-27T09:49:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.888+0000 7f571f00f740 1 -- 172.21.15.44:0/1675177918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 msgr2=0x559f1a3e4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.888+0000 7f571f00f740 1 --2- 172.21.15.44:0/1675177918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3e4750 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f570800f9a0 tx=0x7f5708017000 comp rx=0 tx=0).stop 2024-05-27T09:49:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.888+0000 7f571f00f740 1 -- 172.21.15.44:0/1675177918 shutdown_connections 2024-05-27T09:49:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.888+0000 7f571f00f740 1 --2- 172.21.15.44:0/1675177918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a3e4750 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.888+0000 7f571f00f740 1 --2- 172.21.15.44:0/1675177918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3d4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.888+0000 7f571f00f740 1 --2- 172.21.15.44:0/1675177918 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3d51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.888+0000 7f571f00f740 1 -- 172.21.15.44:0/1675177918 >> 172.21.15.44:0/1675177918 conn(0x559f1a02a300 msgr2=0x559f1a3e4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.888+0000 7f571f00f740 1 -- 172.21.15.44:0/1675177918 shutdown_connections 2024-05-27T09:49:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.889+0000 7f571f00f740 1 -- 172.21.15.44:0/1675177918 wait complete. 2024-05-27T09:49:31.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.889+0000 7f571f00f740 1 -- start start 2024-05-27T09:49:31.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.889+0000 7f571f00f740 1 Processor -- start 2024-05-27T09:49:31.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f571f00f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f571f00f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3e6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f571f00f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a455320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f571f00f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f57101b47c0 con 0x559f1a00d2f0 2024-05-27T09:49:31.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f571f00f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f57101b5c30 con 0x559f1a0154d0 2024-05-27T09:49:31.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f571f00f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f57101b4e90 con 0x559f1a0361b0 2024-05-27T09:49:31.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f571f00f740 10 client.0 _ll_register_callbacks cb 0x7f56fc100540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:31.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f5715600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f5716000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a455320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f5715600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37402/0 (socket says 172.21.15.44:37402) 2024-05-27T09:49:31.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f5715600640 1 -- 172.21.15.44:0/3902748063 learned_addr learned my addr 172.21.15.44:0/3902748063 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:31.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.890+0000 7f5714c00640 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3e6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.891+0000 7f5714c00640 1 -- 172.21.15.44:0/3902748063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 msgr2=0x559f1a455320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:31.895 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.891+0000 7f5714c00640 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a455320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.891+0000 7f5714c00640 1 -- 172.21.15.44:0/3902748063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 msgr2=0x559f1a3e6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.891+0000 7f5714c00640 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.891+0000 7f5714c00640 1 -- 172.21.15.44:0/3902748063 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f1a4558e0 con 0x559f1a0154d0 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.891+0000 7f5715600640 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.891+0000 7f5714c00640 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3e6770 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f56fc02a4c0 tx=0x7f56fc009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bed62714c0050ac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.891+0000 7f56f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.891+0000 7f56f6a00640 1 -- 172.21.15.44:0/3902748063 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc0ea060 con 0x559f1a0154d0 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.891+0000 7f56f6a00640 1 -- 172.21.15.44:0/3902748063 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f56fc1061d0 con 0x559f1a0154d0 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.891+0000 7f571f00f740 1 -- 172.21.15.44:0/3902748063 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f570802e070 con 0x559f1a0154d0 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.892+0000 7f56f6a00640 1 -- 172.21.15.44:0/3902748063 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f56fc0ab050 con 0x559f1a0154d0 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.892+0000 7f571f00f740 10 client.27926 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.892+0000 7f571f00f740 1 -- 172.21.15.44:0/3902748063 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559f1a45cc20 con 0x559f1a0154d0 2024-05-27T09:49:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.892+0000 7f56f6a00640 1 -- 172.21.15.44:0/3902748063 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f570802e070 con 0x559f1a0154d0 2024-05-27T09:49:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.893+0000 7f56f6a00640 1 -- 172.21.15.44:0/3902748063 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f56fc031ac0 con 0x559f1a0154d0 2024-05-27T09:49:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.893+0000 7f571f00f740 10 client.27926 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.893+0000 7f571f00f740 10 client.27926 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.893+0000 7f571f00f740 20 client.27926 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.893+0000 7f571f00f740 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559f19fca1c0 0x559f1a45f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.893+0000 7f571f00f740 4 client.27926 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:49:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.893+0000 7f571f00f740 1 -- 172.21.15.44:0/3902748063 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559f1a3452c0 con 0x559f19fca1c0 2024-05-27T09:49:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.893+0000 7f5715600640 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559f19fca1c0 0x559f1a45f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.894+0000 7f56f6a00640 10 client.27926 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:31.894+0000 7f5715600640 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559f19fca1c0 0x559f1a45f280 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4816608ac9e01813 server_cookie=b82d5a7c726c660 in_seq=0 out_seq=0 2024-05-27T09:49:31.954 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:49:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.095+0000 7f56f6a00640 1 -- 172.21.15.44:0/3902748063 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x559f1a3452c0 con 0x559f19fca1c0 2024-05-27T09:49:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.095+0000 7f56f6a00640 10 client.27926 handle_command_reply: tid=0 2024-05-27T09:49:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.095+0000 7f571f00f740 1 client.27926 shutdown 2024-05-27T09:49:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f571f00f740 1 -- 172.21.15.44:0/3902748063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 msgr2=0x559f1a3e6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f571f00f740 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3e6770 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f56fc02a4c0 tx=0x7f56fc009620 comp rx=0 tx=0).stop 2024-05-27T09:49:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f571f00f740 1 -- 172.21.15.44:0/3902748063 shutdown_connections 2024-05-27T09:49:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f571f00f740 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559f19fca1c0 0x559f1a45f280 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f571f00f740 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f1a0361b0 0x559f1a455320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f571f00f740 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559f1a0154d0 0x559f1a3e6770 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f56f6a00640 0 client.27926 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f571f00f740 1 --2- 172.21.15.44:0/3902748063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559f1a00d2f0 0x559f1a3e6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f571f00f740 1 -- 172.21.15.44:0/3902748063 >> 172.21.15.44:0/3902748063 conn(0x559f1a02a300 msgr2=0x559f1a3e3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f571f00f740 1 -- 172.21.15.44:0/3902748063 shutdown_connections 2024-05-27T09:49:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f571f00f740 1 -- 172.21.15.44:0/3902748063 wait complete. 2024-05-27T09:49:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.096+0000 7f571f00f740 20 client.27926 trim_cache size 0 max 16384 2024-05-27T09:49:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:49:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:49:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:49:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 -- 172.21.15.44:0/3377320782 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f56f80b2950 msgr2=0x7f56f80b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f56f80b2950 0x7f56f80b4e60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x559f1a3d59d0 tx=0x7f5708009500 comp rx=0 tx=0).stop 2024-05-27T09:49:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 -- 172.21.15.44:0/3377320782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5710112310 msgr2=0x7f57101a1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5710112310 0x7f57101a1450 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f56fc005940 tx=0x7f56fc0040c0 comp rx=0 tx=0).stop 2024-05-27T09:49:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 -- 172.21.15.44:0/3377320782 shutdown_connections 2024-05-27T09:49:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5710114c70 0x7f57101a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f56f80b2950 0x7f56f80b4e60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5710112310 0x7f57101a1450 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 --2- 172.21.15.44:0/3377320782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5710075480 0x7f57101a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 -- 172.21.15.44:0/3377320782 >> 172.21.15.44:0/3377320782 conn(0x7f57100ff480 msgr2=0x7f5710111b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 -- 172.21.15.44:0/3377320782 shutdown_connections 2024-05-27T09:49:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:32.097+0000 7f5718000640 1 -- 172.21.15.44:0/3377320782 wait complete. 2024-05-27T09:49:32.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:32 smithi044 ceph-mon[31618]: pgmap v450: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 51 KiB/s wr, 113 op/s 2024-05-27T09:49:32.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:32 smithi050 ceph-mon[36371]: pgmap v450: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 51 KiB/s wr, 113 op/s 2024-05-27T09:49:32.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:32 smithi189 ceph-mon[36671]: pgmap v450: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 51 KiB/s wr, 113 op/s 2024-05-27T09:49:33.725 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:49:34.395 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:49:34.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.411+0000 7fbd2b600700 1 -- 172.21.15.44:0/241593978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24076e30 msgr2=0x7fbd24077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:34.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.411+0000 7fbd2b600700 1 --2- 172.21.15.44:0/241593978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24076e30 0x7fbd24077240 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbd18009990 tx=0x7fbd18009ca0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.411+0000 7fbd2b600700 1 -- 172.21.15.44:0/241593978 shutdown_connections 2024-05-27T09:49:34.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.411+0000 7fbd2b600700 1 --2- 172.21.15.44:0/241593978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd24110b90 0x7fbd24112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.411+0000 7fbd2b600700 1 --2- 172.21.15.44:0/241593978 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24076e30 0x7fbd24077240 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.411+0000 7fbd2b600700 1 --2- 172.21.15.44:0/241593978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd24077780 0x7fbd24110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.411+0000 7fbd2b600700 1 -- 172.21.15.44:0/241593978 >> 172.21.15.44:0/241593978 conn(0x7fbd240ffeb0 msgr2=0x7fbd24102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:34.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.411+0000 7fbd2b600700 1 -- 172.21.15.44:0/241593978 shutdown_connections 2024-05-27T09:49:34.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.411+0000 7fbd2b600700 1 -- 172.21.15.44:0/241593978 wait complete. 2024-05-27T09:49:34.413 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:34 smithi044 ceph-mon[31618]: pgmap v451: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-05-27T09:49:34.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.412+0000 7fbd2b600700 1 Processor -- start 2024-05-27T09:49:34.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.412+0000 7fbd2b600700 1 -- start start 2024-05-27T09:49:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.413+0000 7fbd2b600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24076e30 0x7fbd2419d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.413+0000 7fbd2b600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd24077780 0x7fbd2419d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.413+0000 7fbd2b600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd24110b90 0x7fbd241a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.413+0000 7fbd2b600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd241166d0 con 0x7fbd24076e30 2024-05-27T09:49:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.413+0000 7fbd2b600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbd24116550 con 0x7fbd24110b90 2024-05-27T09:49:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.413+0000 7fbd2b600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbd24116850 con 0x7fbd24077780 2024-05-27T09:49:34.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.413+0000 7fbd28c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24076e30 0x7fbd2419d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.413+0000 7fbd23e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd24077780 0x7fbd2419d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.414+0000 7fbd29600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd24110b90 0x7fbd241a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.414+0000 7fbd23e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd24077780 0x7fbd2419d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38944/0 (socket says 172.21.15.44:38944) 2024-05-27T09:49:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.414+0000 7fbd23e00700 1 -- 172.21.15.44:0/2262240537 learned_addr learned my addr 172.21.15.44:0/2262240537 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.414+0000 7fbd29600700 1 -- 172.21.15.44:0/2262240537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd24077780 msgr2=0x7fbd2419d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.414+0000 7fbd29600700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd24077780 0x7fbd2419d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.414+0000 7fbd29600700 1 -- 172.21.15.44:0/2262240537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24076e30 msgr2=0x7fbd2419d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.414+0000 7fbd29600700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24076e30 0x7fbd2419d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.414+0000 7fbd29600700 1 -- 172.21.15.44:0/2262240537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd241a1ef0 con 0x7fbd24110b90 2024-05-27T09:49:34.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.415+0000 7fbd29600700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd24110b90 0x7fbd241a1850 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00ef90 tx=0x7fbd1c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:34.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.415+0000 7fbd21600700 1 -- 172.21.15.44:0/2262240537 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0092e0 con 0x7fbd24110b90 2024-05-27T09:49:34.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.415+0000 7fbd21600700 1 -- 172.21.15.44:0/2262240537 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd1c004bb0 con 0x7fbd24110b90 2024-05-27T09:49:34.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.415+0000 7fbd21600700 1 -- 172.21.15.44:0/2262240537 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c004280 con 0x7fbd24110b90 2024-05-27T09:49:34.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.415+0000 7fbd2b600700 1 -- 172.21.15.44:0/2262240537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd241a2180 con 0x7fbd24110b90 2024-05-27T09:49:34.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.415+0000 7fbd2b600700 1 -- 172.21.15.44:0/2262240537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd241a2670 con 0x7fbd24110b90 2024-05-27T09:49:34.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.419+0000 7fbd21600700 1 -- 172.21.15.44:0/2262240537 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fbd1c01b030 con 0x7fbd24110b90 2024-05-27T09:49:34.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.420+0000 7fbd3a22cb80 1 Processor -- start 2024-05-27T09:49:34.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.420+0000 7fbd3a22cb80 1 -- start start 2024-05-27T09:49:34.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.421+0000 7fbd21600700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd0c0b2a70 0x7fbd0c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.421+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 0x560ee579fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.421+0000 7fbd21600700 1 -- 172.21.15.44:0/2262240537 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7fbd1c0e2200 con 0x7fbd24110b90 2024-05-27T09:49:34.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.421+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee57a0380 0x560ee5799f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.421+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 0x560ee579a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.421+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560ee56c7fd0 con 0x560ee57a0380 2024-05-27T09:49:34.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.421+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560ee56b7b70 con 0x560ee579a4d0 2024-05-27T09:49:34.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.421+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560ee56bbbb0 con 0x560ee561afd0 2024-05-27T09:49:34.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.421+0000 7fbd23e00700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd0c0b2a70 0x7fbd0c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.422+0000 7fbd29600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 0x560ee579a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.422+0000 7fbd28c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 0x560ee579fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.422+0000 7fbd28c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 0x560ee579fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38948/0 (socket says 172.21.15.44:38948) 2024-05-27T09:49:34.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.422+0000 7fbd29600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 0x560ee579a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47818/0 (socket says 172.21.15.44:47818) 2024-05-27T09:49:34.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.422+0000 7fbd28c00700 1 -- 172.21.15.44:0/638399299 learned_addr learned my addr 172.21.15.44:0/638399299 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:34.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.422+0000 7fbd23e00700 1 --2- 172.21.15.44:0/638399299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee57a0380 0x560ee5799f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.422+0000 7fbd29600700 1 -- 172.21.15.44:0/638399299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 msgr2=0x560ee579fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:34.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.422+0000 7fbd29600700 1 --2- 172.21.15.44:0/638399299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 0x560ee579fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.422+0000 7fbd29600700 1 -- 172.21.15.44:0/638399299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee57a0380 msgr2=0x560ee5799f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:34.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.422+0000 7fbd29600700 1 --2- 172.21.15.44:0/638399299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee57a0380 0x560ee5799f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.422+0000 7fbd29600700 1 -- 172.21.15.44:0/638399299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ee56805b0 con 0x560ee579a4d0 2024-05-27T09:49:34.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.423+0000 7fbd29600700 1 --2- 172.21.15.44:0/638399299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 0x560ee579a8e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0df8a0 tx=0x7fbd1c004d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f06d8c5f8e833adc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:34.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.423+0000 7fbd23e00700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd0c0b2a70 0x7fbd0c0b4f80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbd100046c0 tx=0x7fbd10003a10 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:34.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.423+0000 7fbd16000700 1 -- 172.21.15.44:0/638399299 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0a9bc0 con 0x560ee579a4d0 2024-05-27T09:49:34.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.423+0000 7fbd16000700 1 -- 172.21.15.44:0/638399299 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd1c0a94c0 con 0x560ee579a4d0 2024-05-27T09:49:34.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.423+0000 7fbd16000700 1 -- 172.21.15.44:0/638399299 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0a9720 con 0x560ee579a4d0 2024-05-27T09:49:34.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.424+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/638399299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 msgr2=0x560ee579a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:34.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.424+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/638399299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 0x560ee579a8e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0df8a0 tx=0x7fbd1c004d60 comp rx=0 tx=0).stop 2024-05-27T09:49:34.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.424+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/638399299 shutdown_connections 2024-05-27T09:49:34.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.424+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/638399299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 0x560ee579a8e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.424+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/638399299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 0x560ee579fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.424+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/638399299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee57a0380 0x560ee5799f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.424+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/638399299 >> 172.21.15.44:0/638399299 conn(0x560ee567b150 msgr2=0x560ee5795190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:34.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.425+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/638399299 shutdown_connections 2024-05-27T09:49:34.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.425+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/638399299 wait complete. 2024-05-27T09:49:34.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.425+0000 7fbd3a22cb80 1 -- start start 2024-05-27T09:49:34.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.426+0000 7fbd3a22cb80 1 Processor -- start 2024-05-27T09:49:34.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee561afd0 0x560ee57a3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee579a4d0 0x560ee57a4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee57a0380 0x560ee57a4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560ee55be130 con 0x560ee561afd0 2024-05-27T09:49:34.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560ee56588f0 con 0x560ee57a0380 2024-05-27T09:49:34.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560ee57a6e50 con 0x560ee579a4d0 2024-05-27T09:49:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd3a22cb80 10 client.0 _ll_register_callbacks cb 0x560ee5661e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd28c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee561afd0 0x560ee57a3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd28c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee561afd0 0x560ee57a3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44324/0 (socket says 172.21.15.44:44324) 2024-05-27T09:49:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd28c00700 1 -- 172.21.15.44:0/2693745575 learned_addr learned my addr 172.21.15.44:0/2693745575 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd29600700 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee57a0380 0x560ee57a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.427+0000 7fbd23e00700 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee579a4d0 0x560ee57a4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.428+0000 7fbd29600700 1 -- 172.21.15.44:0/2693745575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee579a4d0 msgr2=0x560ee57a4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:34.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.428+0000 7fbd29600700 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee579a4d0 0x560ee57a4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.428+0000 7fbd29600700 1 -- 172.21.15.44:0/2693745575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee561afd0 msgr2=0x560ee57a3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:34.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.428+0000 7fbd29600700 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee561afd0 0x560ee57a3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.428+0000 7fbd29600700 1 -- 172.21.15.44:0/2693745575 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ee5607a60 con 0x560ee57a0380 2024-05-27T09:49:34.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.428+0000 7fbd28c00700 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee561afd0 0x560ee57a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:34.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.428+0000 7fbd29600700 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee57a0380 0x560ee57a4570 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0a6730 tx=0x7fbd1c0b2010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e31aa4c8480afd3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:34.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.428+0000 7fbd0aa00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:49:34.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.428+0000 7fbd0aa00700 1 -- 172.21.15.44:0/2693745575 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0ad870 con 0x560ee57a0380 2024-05-27T09:49:34.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.429+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/2693745575 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd1c0a94c0 con 0x560ee57a0380 2024-05-27T09:49:34.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.429+0000 7fbd0aa00700 1 -- 172.21.15.44:0/2693745575 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7fbd1c0b0d70 con 0x560ee57a0380 2024-05-27T09:49:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.429+0000 7fbd0aa00700 1 -- 172.21.15.44:0/2693745575 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fbd1c0b2290 con 0x560ee57a0380 2024-05-27T09:49:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.429+0000 7fbd3a22cb80 10 client.27938 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.429+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/2693745575 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x560ee56881f0 con 0x560ee57a0380 2024-05-27T09:49:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.429+0000 7fbd0aa00700 1 -- 172.21.15.44:0/2693745575 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd1c0a94c0 con 0x560ee57a0380 2024-05-27T09:49:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.429+0000 7fbd0aa00700 1 -- 172.21.15.44:0/2693745575 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fbd1c083b80 con 0x560ee57a0380 2024-05-27T09:49:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.429+0000 7fbd3a22cb80 10 client.27938 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.429+0000 7fbd3a22cb80 10 client.27938 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:49:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.430+0000 7fbd3a22cb80 20 client.27938 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.430+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560ee57b3b20 0x560ee57b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.430+0000 7fbd3a22cb80 4 client.27938 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:34.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.430+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/2693745575 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ee56c9f20 con 0x560ee57b3b20 2024-05-27T09:49:34.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.430+0000 7fbd28c00700 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560ee57b3b20 0x560ee57b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.431+0000 7fbd28c00700 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560ee57b3b20 0x560ee57b5f40 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7addd25086bfc30b server_cookie=6d3ea03ba01a7c00 in_seq=0 out_seq=0 2024-05-27T09:49:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.431+0000 7fbd0aa00700 10 client.27938 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.431+0000 7fbd0aa00700 1 -- 172.21.15.44:0/2693745575 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560ee56c9f20 con 0x560ee57b3b20 2024-05-27T09:49:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.431+0000 7fbd0aa00700 10 client.27938 handle_command_reply: tid=0 2024-05-27T09:49:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd3a22cb80 1 client.27938 shutdown 2024-05-27T09:49:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/2693745575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee57a0380 msgr2=0x560ee57a4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee57a0380 0x560ee57a4570 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0a6730 tx=0x7fbd1c0b2010 comp rx=0 tx=0).stop 2024-05-27T09:49:34.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/2693745575 shutdown_connections 2024-05-27T09:49:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560ee57b3b20 0x560ee57b5f40 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee57a0380 0x560ee57a4570 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee561afd0 0x560ee57a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/2693745575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee579a4d0 0x560ee57a4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/2693745575 >> 172.21.15.44:0/2693745575 conn(0x560ee567b150 msgr2=0x560ee5798da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd0aa00700 0 client.27938 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/2693745575 shutdown_connections 2024-05-27T09:49:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.432+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/2693745575 wait complete. 2024-05-27T09:49:34.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.433+0000 7fbd3a22cb80 20 client.27938 trim_cache size 0 max 16384 2024-05-27T09:49:34.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.458+0000 7fbd3a22cb80 1 Processor -- start 2024-05-27T09:49:34.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.458+0000 7fbd3a22cb80 1 -- start start 2024-05-27T09:49:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.458+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 0x560ee57b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.458+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 0x560ee579f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.458+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee57a0380 0x560ee579e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.458+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd241b3b50 con 0x560ee57a0380 2024-05-27T09:49:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.458+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbd241b1380 con 0x560ee579a4d0 2024-05-27T09:49:34.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.458+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbd241af1f0 con 0x560ee561afd0 2024-05-27T09:49:34.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd28c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 0x560ee57b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd29600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee57a0380 0x560ee579e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd28c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 0x560ee57b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38970/0 (socket says 172.21.15.44:38970) 2024-05-27T09:49:34.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd28c00700 1 -- 172.21.15.44:0/3956747377 learned_addr learned my addr 172.21.15.44:0/3956747377 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:34.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd23e00700 1 --2- 172.21.15.44:0/3956747377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 0x560ee579f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd28c00700 1 -- 172.21.15.44:0/3956747377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 msgr2=0x560ee579f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:34.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd28c00700 1 --2- 172.21.15.44:0/3956747377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 0x560ee579f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd28c00700 1 -- 172.21.15.44:0/3956747377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee57a0380 msgr2=0x560ee579e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:34.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd28c00700 1 --2- 172.21.15.44:0/3956747377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee57a0380 0x560ee579e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd28c00700 1 -- 172.21.15.44:0/3956747377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ee57a48e0 con 0x560ee561afd0 2024-05-27T09:49:34.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd23e00700 1 --2- 172.21.15.44:0/3956747377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 0x560ee579f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:49:34.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.459+0000 7fbd28c00700 1 --2- 172.21.15.44:0/3956747377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 0x560ee57b00b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x560ee57a3370 tx=0x7fbd18005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e237d1bbf478c9a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:34.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.460+0000 7fbd15600700 1 -- 172.21.15.44:0/3956747377 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1800bc80 con 0x560ee561afd0 2024-05-27T09:49:34.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.460+0000 7fbd15600700 1 -- 172.21.15.44:0/3956747377 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd1800be20 con 0x560ee561afd0 2024-05-27T09:49:34.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.460+0000 7fbd15600700 1 -- 172.21.15.44:0/3956747377 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1800f9f0 con 0x560ee561afd0 2024-05-27T09:49:34.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.461+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/3956747377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 msgr2=0x560ee57b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:34.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.461+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/3956747377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 0x560ee57b00b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x560ee57a3370 tx=0x7fbd18005040 comp rx=0 tx=0).stop 2024-05-27T09:49:34.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.461+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/3956747377 shutdown_connections 2024-05-27T09:49:34.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.461+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/3956747377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee57a0380 0x560ee579e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.461+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/3956747377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee561afd0 0x560ee57b00b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.461+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/3956747377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee579a4d0 0x560ee579f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.461+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/3956747377 >> 172.21.15.44:0/3956747377 conn(0x560ee567b150 msgr2=0x560ee5799410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.461+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/3956747377 shutdown_connections 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.461+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/3956747377 wait complete. 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.462+0000 7fbd3a22cb80 1 -- start start 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.462+0000 7fbd3a22cb80 1 Processor -- start 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.462+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee561afd0 0x560ee57b7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.462+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee579a4d0 0x560ee57b7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.462+0000 7fbd3a22cb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee57a0380 0x560ee58173d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.462+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd241af840 con 0x560ee579a4d0 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.462+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbd241b0cb0 con 0x560ee561afd0 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.462+0000 7fbd3a22cb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbd241aff10 con 0x560ee57a0380 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd3a22cb80 10 client.0 _ll_register_callbacks cb 0x560ee5661e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:34.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd28c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee561afd0 0x560ee57b7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd23e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee579a4d0 0x560ee57b7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd28c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee561afd0 0x560ee57b7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47840/0 (socket says 172.21.15.44:47840) 2024-05-27T09:49:34.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd28c00700 1 -- 172.21.15.44:0/1331474903 learned_addr learned my addr 172.21.15.44:0/1331474903 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:34.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd29600700 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee57a0380 0x560ee58173d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd29600700 1 -- 172.21.15.44:0/1331474903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee561afd0 msgr2=0x560ee57b7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:34.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd29600700 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee561afd0 0x560ee57b7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd29600700 1 -- 172.21.15.44:0/1331474903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee579a4d0 msgr2=0x560ee57b7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:34.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd29600700 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee579a4d0 0x560ee57b7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd29600700 1 -- 172.21.15.44:0/1331474903 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ee5817910 con 0x560ee57a0380 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.463+0000 7fbd23e00700 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee579a4d0 0x560ee57b7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.464+0000 7fbd29600700 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee57a0380 0x560ee58173d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009510 tx=0x7fbd1c0a5380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63ff0b16ea6766d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.464+0000 7fbd0aa00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.465+0000 7fbd0aa00700 1 -- 172.21.15.44:0/1331474903 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0b2080 con 0x560ee57a0380 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.465+0000 7fbd0aa00700 1 -- 172.21.15.44:0/1331474903 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7fbd1c0a8930 con 0x560ee57a0380 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.465+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/1331474903 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd1800be20 con 0x560ee57a0380 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.465+0000 7fbd0aa00700 1 -- 172.21.15.44:0/1331474903 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fbd1c0a9bc0 con 0x560ee57a0380 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.465+0000 7fbd3a22cb80 10 client.27952 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.465+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/1331474903 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x560ee581ec20 con 0x560ee57a0380 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.465+0000 7fbd0aa00700 1 -- 172.21.15.44:0/1331474903 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd1800be20 con 0x560ee57a0380 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.465+0000 7fbd0aa00700 1 -- 172.21.15.44:0/1331474903 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fbd1c003a00 con 0x560ee57a0380 2024-05-27T09:49:34.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.466+0000 7fbd3a22cb80 10 client.27952 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:34.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.466+0000 7fbd3a22cb80 10 client.27952 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:49:34.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.466+0000 7fbd3a22cb80 20 client.27952 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:34.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.466+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560ee581f030 0x560ee5821450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:34.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.466+0000 7fbd3a22cb80 4 client.27952 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-05-27T09:49:34.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.466+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/1331474903 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x560ee56c3700 con 0x560ee581f030 2024-05-27T09:49:34.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.466+0000 7fbd28c00700 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560ee581f030 0x560ee5821450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:34.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.467+0000 7fbd28c00700 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560ee581f030 0x560ee5821450 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5c82da79b3d4448 server_cookie=cc9d37c518ff5051 in_seq=0 out_seq=0 2024-05-27T09:49:34.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.467+0000 7fbd0aa00700 10 client.27952 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:34.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.487+0000 7fbd0aa00700 1 -- 172.21.15.44:0/1331474903 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x560ee56c3700 con 0x560ee581f030 2024-05-27T09:49:34.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.487+0000 7fbd0aa00700 10 client.27952 handle_command_reply: tid=0 2024-05-27T09:49:34.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.487+0000 7fbd3a22cb80 1 client.27952 shutdown 2024-05-27T09:49:34.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.487+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/1331474903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee57a0380 msgr2=0x560ee58173d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:34.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.487+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee57a0380 0x560ee58173d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009510 tx=0x7fbd1c0a5380 comp rx=0 tx=0).stop 2024-05-27T09:49:34.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.487+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/1331474903 shutdown_connections 2024-05-27T09:49:34.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.487+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560ee581f030 0x560ee5821450 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.487+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ee57a0380 0x560ee58173d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.487+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560ee561afd0 0x560ee57b7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.487+0000 7fbd3a22cb80 1 --2- 172.21.15.44:0/1331474903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560ee579a4d0 0x560ee57b7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.488+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/1331474903 >> 172.21.15.44:0/1331474903 conn(0x560ee567b150 msgr2=0x560ee579d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:34.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.488+0000 7fbd0aa00700 0 client.27952 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:34.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.488+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/1331474903 shutdown_connections 2024-05-27T09:49:34.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.488+0000 7fbd3a22cb80 1 -- 172.21.15.44:0/1331474903 wait complete. 2024-05-27T09:49:34.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.488+0000 7fbd3a22cb80 20 client.27952 trim_cache size 0 max 16384 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 43, 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 34, 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout: "9fa86eb2": { 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 34, 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:49:34.491 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 130.0, 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:49:34.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 -- 172.21.15.44:0/2262240537 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd0c0b2a70 msgr2=0x7fbd0c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:34.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd0c0b2a70 0x7fbd0c0b4f80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbd100046c0 tx=0x7fbd10003a10 comp rx=0 tx=0).stop 2024-05-27T09:49:34.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 -- 172.21.15.44:0/2262240537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd24110b90 msgr2=0x7fbd241a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:34.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd24110b90 0x7fbd241a1850 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00ef90 tx=0x7fbd1c00c3a0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 -- 172.21.15.44:0/2262240537 shutdown_connections 2024-05-27T09:49:34.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fbd0c0b2a70 0x7fbd0c0b4f80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd24110b90 0x7fbd241a1850 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd24076e30 0x7fbd2419d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 --2- 172.21.15.44:0/2262240537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd24077780 0x7fbd2419d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:34.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 -- 172.21.15.44:0/2262240537 >> 172.21.15.44:0/2262240537 conn(0x7fbd240ffeb0 msgr2=0x7fbd241004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:34.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 -- 172.21.15.44:0/2262240537 shutdown_connections 2024-05-27T09:49:34.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:34.489+0000 7fbd2b600700 1 -- 172.21.15.44:0/2262240537 wait complete. 2024-05-27T09:49:34.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:34 smithi050 ceph-mon[36371]: pgmap v451: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-05-27T09:49:34.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:34 smithi189 ceph-mon[36671]: pgmap v451: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-05-27T09:49:35.054 INFO:tasks.workunit.client.0.smithi044.stdout:ok 2024-05-27T09:49:35.068 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T09:49:35.068 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T09:49:35.600 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '9fa86eb2', quiesce took 8.0 seconds. Will release after: 2.0 2024-05-27T09:49:36.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:36 smithi044 ceph-mon[31618]: pgmap v452: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-05-27T09:49:36.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:36 smithi050 ceph-mon[36371]: pgmap v452: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-05-27T09:49:36.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:36 smithi189 ceph-mon[36671]: pgmap v452: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 31 KiB/s wr, 15 op/s 2024-05-27T09:49:37.602 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:49:37.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.824+0000 7f8814e00700 1 -- 172.21.15.44:0/3282438880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f881010e8d0 msgr2=0x7f8810110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:37.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.824+0000 7f8814e00700 1 --2- 172.21.15.44:0/3282438880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f881010e8d0 0x7f8810110db0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f880800b0a0 tx=0x7f880800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.824+0000 7f8814e00700 1 -- 172.21.15.44:0/3282438880 shutdown_connections 2024-05-27T09:49:37.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.824+0000 7f8814e00700 1 --2- 172.21.15.44:0/3282438880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f881010e8d0 0x7f8810110db0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.824+0000 7f8814e00700 1 --2- 172.21.15.44:0/3282438880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88101054f0 0x7f8810105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.824+0000 7f8814e00700 1 --2- 172.21.15.44:0/3282438880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8810105e40 0x7f881010e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.824+0000 7f8814e00700 1 -- 172.21.15.44:0/3282438880 >> 172.21.15.44:0/3282438880 conn(0x7f88100fedb0 msgr2=0x7f88101011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:37.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.824+0000 7f8814e00700 1 -- 172.21.15.44:0/3282438880 shutdown_connections 2024-05-27T09:49:37.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.824+0000 7f8814e00700 1 -- 172.21.15.44:0/3282438880 wait complete. 2024-05-27T09:49:37.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.826+0000 7f8814e00700 1 Processor -- start 2024-05-27T09:49:37.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.827+0000 7f8814e00700 1 -- start start 2024-05-27T09:49:37.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.827+0000 7f8814e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88101054f0 0x7f881019ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.827+0000 7f8814e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8810105e40 0x7f881019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f8814e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881010e8d0 0x7f881019f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f8814e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8810114430 con 0x7f88101054f0 2024-05-27T09:49:37.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f8814e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f88101142b0 con 0x7f8810105e40 2024-05-27T09:49:37.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f8814e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f88101145b0 con 0x7f881010e8d0 2024-05-27T09:49:37.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f880de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88101054f0 0x7f881019ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f880de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88101054f0 0x7f881019ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44360/0 (socket says 172.21.15.44:44360) 2024-05-27T09:49:37.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f880de00700 1 -- 172.21.15.44:0/1995934648 learned_addr learned my addr 172.21.15.44:0/1995934648 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:37.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f880d400700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8810105e40 0x7f881019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f880e800700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881010e8d0 0x7f881019f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f880d400700 1 -- 172.21.15.44:0/1995934648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881010e8d0 msgr2=0x7f881019f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:37.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f880d400700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881010e8d0 0x7f881019f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f880d400700 1 -- 172.21.15.44:0/1995934648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88101054f0 msgr2=0x7f881019ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:37.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f880d400700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88101054f0 0x7f881019ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.828+0000 7f880d400700 1 -- 172.21.15.44:0/1995934648 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881019fc30 con 0x7f8810105e40 2024-05-27T09:49:37.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.829+0000 7f880d400700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8810105e40 0x7f881019b300 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00e9f0 tx=0x7f87fc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:37.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.829+0000 7f8806a00700 1 -- 172.21.15.44:0/1995934648 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc00c9c0 con 0x7f8810105e40 2024-05-27T09:49:37.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.829+0000 7f8806a00700 1 -- 172.21.15.44:0/1995934648 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87fc00cb60 con 0x7f8810105e40 2024-05-27T09:49:37.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.829+0000 7f8814e00700 1 -- 172.21.15.44:0/1995934648 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f881019ff20 con 0x7f8810105e40 2024-05-27T09:49:37.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.829+0000 7f8814e00700 1 -- 172.21.15.44:0/1995934648 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8810103110 con 0x7f8810105e40 2024-05-27T09:49:37.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.830+0000 7f8806a00700 1 -- 172.21.15.44:0/1995934648 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc018680 con 0x7f8810105e40 2024-05-27T09:49:37.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.836+0000 7f88238ceb80 1 Processor -- start 2024-05-27T09:49:37.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.836+0000 7f88238ceb80 1 -- start start 2024-05-27T09:49:37.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f8806a00700 1 -- 172.21.15.44:0/1995934648 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f87fc018820 con 0x7f8810105e40 2024-05-27T09:49:37.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 0x55663092d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630927870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556630927db0 0x5566309281c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f88238ceb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556630855e40 con 0x5566307a8e40 2024-05-27T09:49:37.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f88238ceb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5566308459e0 con 0x556630927db0 2024-05-27T09:49:37.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f88238ceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556630849a20 con 0x55663092dd30 2024-05-27T09:49:37.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f880e800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 0x55663092d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f880e800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 0x55663092d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44364/0 (socket says 172.21.15.44:44364) 2024-05-27T09:49:37.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f880e800700 1 -- 172.21.15.44:0/3578615629 learned_addr learned my addr 172.21.15.44:0/3578615629 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:37.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f880de00700 1 --2- 172.21.15.44:0/3578615629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630927870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.837+0000 7f880d400700 1 --2- 172.21.15.44:0/3578615629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556630927db0 0x5566309281c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.838+0000 7f880de00700 1 -- 172.21.15.44:0/3578615629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556630927db0 msgr2=0x5566309281c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:37.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.838+0000 7f880de00700 1 --2- 172.21.15.44:0/3578615629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556630927db0 0x5566309281c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.838+0000 7f880de00700 1 -- 172.21.15.44:0/3578615629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 msgr2=0x55663092d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:37.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.838+0000 7f880de00700 1 --2- 172.21.15.44:0/3578615629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 0x55663092d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.838+0000 7f880de00700 1 -- 172.21.15.44:0/3578615629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55663080e420 con 0x55663092dd30 2024-05-27T09:49:37.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.838+0000 7f880e800700 1 --2- 172.21.15.44:0/3578615629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 0x55663092d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:37.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.838+0000 7f880de00700 1 --2- 172.21.15.44:0/3578615629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630927870 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8800007ca0 tx=0x7f88000103d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e5e88264700c816 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:37.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.839+0000 7f87fb400700 1 -- 172.21.15.44:0/3578615629 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8800010d10 con 0x55663092dd30 2024-05-27T09:49:37.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.839+0000 7f87fb400700 1 -- 172.21.15.44:0/3578615629 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8800015070 con 0x55663092dd30 2024-05-27T09:49:37.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.839+0000 7f87fb400700 1 -- 172.21.15.44:0/3578615629 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880001a470 con 0x55663092dd30 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.840+0000 7f8806a00700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f87f40b2a70 0x7f87f40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.840+0000 7f88238ceb80 1 -- 172.21.15.44:0/3578615629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 msgr2=0x556630927870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.840+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3578615629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630927870 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8800007ca0 tx=0x7f88000103d0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.840+0000 7f8806a00700 1 -- 172.21.15.44:0/1995934648 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f87fc0e2680 con 0x7f8810105e40 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.840+0000 7f88238ceb80 1 -- 172.21.15.44:0/3578615629 shutdown_connections 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.840+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3578615629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556630927db0 0x5566309281c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.840+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3578615629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 0x55663092d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.840+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3578615629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630927870 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.840+0000 7f88238ceb80 1 -- 172.21.15.44:0/3578615629 >> 172.21.15.44:0/3578615629 conn(0x556630808fc0 msgr2=0x5566309228d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.840+0000 7f880e800700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f87f40b2a70 0x7f87f40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.840+0000 7f88238ceb80 1 -- 172.21.15.44:0/3578615629 shutdown_connections 2024-05-27T09:49:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.841+0000 7f88238ceb80 1 -- 172.21.15.44:0/3578615629 wait complete. 2024-05-27T09:49:37.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.841+0000 7f88238ceb80 1 -- start start 2024-05-27T09:49:37.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.841+0000 7f880e800700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f87f40b2a70 0x7f87f40b4f80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x556630928b30 tx=0x7f880800b720 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:37.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.841+0000 7f88238ceb80 1 Processor -- start 2024-05-27T09:49:37.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.842+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x556630931480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.842+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556630927db0 0x5566309319c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.842+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630931f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.842+0000 7f88238ceb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55663074bfa0 con 0x556630927db0 2024-05-27T09:49:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.842+0000 7f88238ceb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5566307e6760 con 0x5566307a8e40 2024-05-27T09:49:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.842+0000 7f88238ceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556630934800 con 0x55663092dd30 2024-05-27T09:49:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.842+0000 7f88238ceb80 10 client.0 _ll_register_callbacks cb 0x5566307efc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f880de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x556630931480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f880d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556630927db0 0x5566309319c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f880de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x556630931480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47890/0 (socket says 172.21.15.44:47890) 2024-05-27T09:49:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f880de00700 1 -- 172.21.15.44:0/3635628884 learned_addr learned my addr 172.21.15.44:0/3635628884 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f880e800700 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630931f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f880de00700 1 -- 172.21.15.44:0/3635628884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 msgr2=0x556630931f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f880de00700 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630931f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f880de00700 1 -- 172.21.15.44:0/3635628884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556630927db0 msgr2=0x5566309319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f880de00700 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556630927db0 0x5566309319c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f880de00700 1 -- 172.21.15.44:0/3635628884 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566307958d0 con 0x5566307a8e40 2024-05-27T09:49:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f880de00700 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x556630931480 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f88000078a0 tx=0x7f8800010d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eede0e16ad83020 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.843+0000 7f87efe00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:49:37.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.844+0000 7f87efe00700 1 -- 172.21.15.44:0/3635628884 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88000037c0 con 0x5566307a8e40 2024-05-27T09:49:37.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.844+0000 7f88238ceb80 1 -- 172.21.15.44:0/3635628884 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8800015070 con 0x5566307a8e40 2024-05-27T09:49:37.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.844+0000 7f87efe00700 1 -- 172.21.15.44:0/3635628884 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f8800003c90 con 0x5566307a8e40 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.844+0000 7f87efe00700 1 -- 172.21.15.44:0/3635628884 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f8800005020 con 0x5566307a8e40 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.844+0000 7f88238ceb80 10 client.27956 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.844+0000 7f88238ceb80 1 -- 172.21.15.44:0/3635628884 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x556630816060 con 0x5566307a8e40 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.844+0000 7f87efe00700 1 -- 172.21.15.44:0/3635628884 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8800015070 con 0x5566307a8e40 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.845+0000 7f87efe00700 1 -- 172.21.15.44:0/3635628884 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f8800004750 con 0x5566307a8e40 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.845+0000 7f88238ceb80 10 client.27956 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.845+0000 7f88238ceb80 10 client.27956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.845+0000 7f88238ceb80 20 client.27956 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.845+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5566309415c0 0x5566309439e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.845+0000 7f88238ceb80 4 client.27956 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.845+0000 7f88238ceb80 1 -- 172.21.15.44:0/3635628884 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556630857d90 con 0x5566309415c0 2024-05-27T09:49:37.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.845+0000 7f880d400700 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5566309415c0 0x5566309439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.846+0000 7f880d400700 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5566309415c0 0x5566309439e0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2be68856a8685cd5 server_cookie=8ec18921cd2067ce in_seq=0 out_seq=0 2024-05-27T09:49:37.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.846+0000 7f87efe00700 10 client.27956 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:37.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f87efe00700 1 -- 172.21.15.44:0/3635628884 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556630857d90 con 0x5566309415c0 2024-05-27T09:49:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f87efe00700 10 client.27956 handle_command_reply: tid=0 2024-05-27T09:49:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f88238ceb80 1 client.27956 shutdown 2024-05-27T09:49:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f88238ceb80 1 -- 172.21.15.44:0/3635628884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 msgr2=0x556630931480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x556630931480 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f88000078a0 tx=0x7f8800010d40 comp rx=0 tx=0).stop 2024-05-27T09:49:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f88238ceb80 1 -- 172.21.15.44:0/3635628884 shutdown_connections 2024-05-27T09:49:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5566309415c0 0x5566309439e0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630931f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x556630931480 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3635628884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556630927db0 0x5566309319c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f88238ceb80 1 -- 172.21.15.44:0/3635628884 >> 172.21.15.44:0/3635628884 conn(0x556630808fc0 msgr2=0x556630922580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:37.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f87efe00700 0 client.27956 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:37.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f88238ceb80 1 -- 172.21.15.44:0/3635628884 shutdown_connections 2024-05-27T09:49:37.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.847+0000 7f88238ceb80 1 -- 172.21.15.44:0/3635628884 wait complete. 2024-05-27T09:49:37.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.848+0000 7f88238ceb80 20 client.27956 trim_cache size 0 max 16384 2024-05-27T09:49:37.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.886+0000 7f88238ceb80 1 Processor -- start 2024-05-27T09:49:37.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.886+0000 7f88238ceb80 1 -- start start 2024-05-27T09:49:37.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.886+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 0x556630927870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.886+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556630927db0 0x55663093db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.886+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630930750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.886+0000 7f88238ceb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f87f0010000 con 0x5566307a8e40 2024-05-27T09:49:37.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.886+0000 7f88238ceb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f87f000d830 con 0x556630927db0 2024-05-27T09:49:37.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.886+0000 7f88238ceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87f000b6a0 con 0x55663092dd30 2024-05-27T09:49:37.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.887+0000 7f880de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 0x556630927870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.887+0000 7f880de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 0x556630927870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44396/0 (socket says 172.21.15.44:44396) 2024-05-27T09:49:37.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.887+0000 7f880de00700 1 -- 172.21.15.44:0/20558114 learned_addr learned my addr 172.21.15.44:0/20558114 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:37.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.887+0000 7f880d400700 1 --2- 172.21.15.44:0/20558114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556630927db0 0x55663093db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.887+0000 7f880e800700 1 --2- 172.21.15.44:0/20558114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630930750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.887+0000 7f880d400700 1 -- 172.21.15.44:0/20558114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 msgr2=0x556630930750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:37.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.887+0000 7f880d400700 1 --2- 172.21.15.44:0/20558114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630930750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.887+0000 7f880d400700 1 -- 172.21.15.44:0/20558114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 msgr2=0x556630927870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:37.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.887+0000 7f880d400700 1 --2- 172.21.15.44:0/20558114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 0x556630927870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.887+0000 7f880d400700 1 -- 172.21.15.44:0/20558114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556630924310 con 0x556630927db0 2024-05-27T09:49:37.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.888+0000 7f880d400700 1 --2- 172.21.15.44:0/20558114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556630927db0 0x55663093db00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0dfd20 tx=0x7f87fc0b2040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29decf059e585656 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:37.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.888+0000 7f87faa00700 1 -- 172.21.15.44:0/20558114 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc0a9020 con 0x556630927db0 2024-05-27T09:49:37.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.888+0000 7f87faa00700 1 -- 172.21.15.44:0/20558114 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87fc0a93c0 con 0x556630927db0 2024-05-27T09:49:37.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.888+0000 7f87faa00700 1 -- 172.21.15.44:0/20558114 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc018b10 con 0x556630927db0 2024-05-27T09:49:37.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.889+0000 7f88238ceb80 1 -- 172.21.15.44:0/20558114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556630927db0 msgr2=0x55663093db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:37.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.889+0000 7f88238ceb80 1 --2- 172.21.15.44:0/20558114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556630927db0 0x55663093db00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0dfd20 tx=0x7f87fc0b2040 comp rx=0 tx=0).stop 2024-05-27T09:49:37.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.889+0000 7f88238ceb80 1 -- 172.21.15.44:0/20558114 shutdown_connections 2024-05-27T09:49:37.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.889+0000 7f88238ceb80 1 --2- 172.21.15.44:0/20558114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55663092dd30 0x556630930750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.889+0000 7f88238ceb80 1 --2- 172.21.15.44:0/20558114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566307a8e40 0x556630927870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.889+0000 7f88238ceb80 1 --2- 172.21.15.44:0/20558114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556630927db0 0x55663093db00 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.889+0000 7f88238ceb80 1 -- 172.21.15.44:0/20558114 >> 172.21.15.44:0/20558114 conn(0x556630808fc0 msgr2=0x556630931260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:37.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.889+0000 7f88238ceb80 1 -- 172.21.15.44:0/20558114 shutdown_connections 2024-05-27T09:49:37.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.889+0000 7f88238ceb80 1 -- 172.21.15.44:0/20558114 wait complete. 2024-05-27T09:49:37.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.890+0000 7f88238ceb80 1 -- start start 2024-05-27T09:49:37.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.890+0000 7f88238ceb80 1 Processor -- start 2024-05-27T09:49:37.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.891+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x5566309446b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.891+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556630927db0 0x556630944bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.891+0000 7f88238ceb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55663092dd30 0x556630945130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.891+0000 7f88238ceb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f87f000bcf0 con 0x55663092dd30 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.891+0000 7f88238ceb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f87f000d160 con 0x5566307a8e40 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.891+0000 7f88238ceb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87f000c3c0 con 0x556630927db0 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.891+0000 7f88238ceb80 10 client.0 _ll_register_callbacks cb 0x5566307efc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x5566309446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556630927db0 0x556630944bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880e800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55663092dd30 0x556630945130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x5566309446b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47916/0 (socket says 172.21.15.44:47916) 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556630927db0 0x556630944bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:39030/0 (socket says 172.21.15.44:39030) 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880de00700 1 -- 172.21.15.44:0/3381568655 learned_addr learned my addr 172.21.15.44:0/3381568655 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:37.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880de00700 1 -- 172.21.15.44:0/3381568655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556630927db0 msgr2=0x556630944bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:37.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880de00700 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556630927db0 0x556630944bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880de00700 1 -- 172.21.15.44:0/3381568655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55663092dd30 msgr2=0x556630945130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:37.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880de00700 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55663092dd30 0x556630945130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880de00700 1 -- 172.21.15.44:0/3381568655 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566309b3640 con 0x5566307a8e40 2024-05-27T09:49:37.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880e800700 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55663092dd30 0x556630945130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:49:37.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.892+0000 7f880d400700 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556630927db0 0x556630944bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:37.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.893+0000 7f880de00700 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x5566309446b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f88000107a0 tx=0x7f88000075c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb30b13626c53c85 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:37.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.893+0000 7f87efe00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.893+0000 7f87efe00700 1 -- 172.21.15.44:0/3381568655 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88000053d0 con 0x5566307a8e40 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.893+0000 7f87efe00700 1 -- 172.21.15.44:0/3381568655 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f8800005570 con 0x5566307a8e40 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.893+0000 7f88238ceb80 1 -- 172.21.15.44:0/3381568655 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87fc0a93c0 con 0x5566307a8e40 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.893+0000 7f87efe00700 1 -- 172.21.15.44:0/3381568655 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f8800041050 con 0x5566307a8e40 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.893+0000 7f88238ceb80 10 client.27962 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.893+0000 7f88238ceb80 1 -- 172.21.15.44:0/3381568655 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5566309b4720 con 0x5566307a8e40 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.894+0000 7f87efe00700 1 -- 172.21.15.44:0/3381568655 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87fc0a93c0 con 0x5566307a8e40 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.894+0000 7f87efe00700 1 -- 172.21.15.44:0/3381568655 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f880003c030 con 0x5566307a8e40 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.894+0000 7f88238ceb80 10 client.27962 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.894+0000 7f88238ceb80 10 client.27962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.894+0000 7f88238ceb80 20 client.27962 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:37.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.894+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5566309b4d90 0x5566309b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:37.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.894+0000 7f88238ceb80 4 client.27962 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:49:37.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.894+0000 7f88238ceb80 1 -- 172.21.15.44:0/3381568655 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556630851570 con 0x5566309b4d90 2024-05-27T09:49:37.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.895+0000 7f880d400700 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5566309b4d90 0x5566309b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:37.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.895+0000 7f880d400700 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5566309b4d90 0x5566309b71b0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c95c1e9e58737cd server_cookie=4fdef31c2cdaf29e in_seq=0 out_seq=0 2024-05-27T09:49:37.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.895+0000 7f87efe00700 10 client.27962 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:37.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f87efe00700 1 -- 172.21.15.44:0/3381568655 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556630851570 con 0x5566309b4d90 2024-05-27T09:49:37.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f87efe00700 10 client.27962 handle_command_reply: tid=0 2024-05-27T09:49:37.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 1 client.27962 shutdown 2024-05-27T09:49:37.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 1 -- 172.21.15.44:0/3381568655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 msgr2=0x5566309446b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:37.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x5566309446b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f88000107a0 tx=0x7f88000075c0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 1 -- 172.21.15.44:0/3381568655 shutdown_connections 2024-05-27T09:49:37.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5566309b4d90 0x5566309b71b0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55663092dd30 0x556630945130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566307a8e40 0x5566309446b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 1 --2- 172.21.15.44:0/3381568655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556630927db0 0x556630944bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 1 -- 172.21.15.44:0/3381568655 >> 172.21.15.44:0/3381568655 conn(0x556630808fc0 msgr2=0x556630931c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:37.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f87efe00700 0 client.27962 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:37.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 1 -- 172.21.15.44:0/3381568655 shutdown_connections 2024-05-27T09:49:37.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 1 -- 172.21.15.44:0/3381568655 wait complete. 2024-05-27T09:49:37.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.896+0000 7f88238ceb80 20 client.27962 trim_cache size 0 max 16384 2024-05-27T09:49:37.900 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:49:37.901 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:49:37.901 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:49:37.901 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:49:37.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 -- 172.21.15.44:0/1995934648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f87f40b2a70 msgr2=0x7f87f40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:37.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f87f40b2a70 0x7f87f40b4f80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x556630928b30 tx=0x7f880800b720 comp rx=0 tx=0).stop 2024-05-27T09:49:37.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 -- 172.21.15.44:0/1995934648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8810105e40 msgr2=0x7f881019b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:37.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8810105e40 0x7f881019b300 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00e9f0 tx=0x7f87fc00ed00 comp rx=0 tx=0).stop 2024-05-27T09:49:37.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 -- 172.21.15.44:0/1995934648 shutdown_connections 2024-05-27T09:49:37.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f87f40b2a70 0x7f87f40b4f80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881010e8d0 0x7f881019f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88101054f0 0x7f881019ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 --2- 172.21.15.44:0/1995934648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8810105e40 0x7f881019b300 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:37.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 -- 172.21.15.44:0/1995934648 >> 172.21.15.44:0/1995934648 conn(0x7f88100fedb0 msgr2=0x7f8810108470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:37.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 -- 172.21.15.44:0/1995934648 shutdown_connections 2024-05-27T09:49:37.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:37.897+0000 7f8814e00700 1 -- 172.21.15.44:0/1995934648 wait complete. 2024-05-27T09:49:38.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:38 smithi044 ceph-mon[31618]: pgmap v453: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 60 KiB/s wr, 109 op/s 2024-05-27T09:49:38.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:38 smithi050 ceph-mon[36371]: pgmap v453: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 60 KiB/s wr, 109 op/s 2024-05-27T09:49:38.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:38 smithi189 ceph-mon[36671]: pgmap v453: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 60 KiB/s wr, 109 op/s 2024-05-27T09:49:38.807 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:49:38.807 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:49:38.837 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:49:40.266 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:49:40.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:40 smithi044 ceph-mon[31618]: pgmap v454: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 32 KiB/s wr, 98 op/s 2024-05-27T09:49:40.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:40 smithi050 ceph-mon[36371]: pgmap v454: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 32 KiB/s wr, 98 op/s 2024-05-27T09:49:40.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:40 smithi189 ceph-mon[36671]: pgmap v454: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 32 KiB/s wr, 98 op/s 2024-05-27T09:49:41.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.499+0000 7f73c0a00700 1 -- 172.21.15.44:0/313488079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73bc109840 msgr2=0x7f73bc1111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:41.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.499+0000 7f73c0a00700 1 --2- 172.21.15.44:0/313488079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73bc109840 0x7f73bc1111a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f73b400b0d0 tx=0x7f73b400b3e0 comp rx=0 tx=0).stop 2024-05-27T09:49:41.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.499+0000 7f73c0a00700 1 -- 172.21.15.44:0/313488079 shutdown_connections 2024-05-27T09:49:41.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.499+0000 7f73c0a00700 1 --2- 172.21.15.44:0/313488079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73bc109840 0x7f73bc1111a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:41.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.499+0000 7f73c0a00700 1 --2- 172.21.15.44:0/313488079 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73bc1045c0 0x7f73bc1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:41.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.499+0000 7f73c0a00700 1 --2- 172.21.15.44:0/313488079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73bc108ef0 0x7f73bc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:41.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.499+0000 7f73c0a00700 1 -- 172.21.15.44:0/313488079 >> 172.21.15.44:0/313488079 conn(0x7f73bc0ffe90 msgr2=0x7f73bc1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:41.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.500+0000 7f73c0a00700 1 -- 172.21.15.44:0/313488079 shutdown_connections 2024-05-27T09:49:41.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.500+0000 7f73c0a00700 1 -- 172.21.15.44:0/313488079 wait complete. 2024-05-27T09:49:41.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.501+0000 7f73c0a00700 1 Processor -- start 2024-05-27T09:49:41.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.501+0000 7f73c0a00700 1 -- start start 2024-05-27T09:49:41.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.501+0000 7f73c0a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73bc1045c0 0x7f73bc19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:41.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.501+0000 7f73c0a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73bc108ef0 0x7f73bc19d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:41.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.501+0000 7f73c0a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73bc109840 0x7f73bc1a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:41.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.501+0000 7f73c0a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc1144e0 con 0x7f73bc108ef0 2024-05-27T09:49:41.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.501+0000 7f73c0a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc114360 con 0x7f73bc1045c0 2024-05-27T09:49:41.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.501+0000 7f73c0a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc114660 con 0x7f73bc109840 2024-05-27T09:49:41.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.502+0000 7f73b9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73bc108ef0 0x7f73bc19d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:41.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.502+0000 7f73b9e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73bc1045c0 0x7f73bc19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:41.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.502+0000 7f73b9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73bc108ef0 0x7f73bc19d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44418/0 (socket says 172.21.15.44:44418) 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.502+0000 7f73b9400700 1 -- 172.21.15.44:0/1036816888 learned_addr learned my addr 172.21.15.44:0/1036816888 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.502+0000 7f73ba800700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73bc109840 0x7f73bc1a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.502+0000 7f73b9e00700 1 -- 172.21.15.44:0/1036816888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73bc109840 msgr2=0x7f73bc1a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.502+0000 7f73b9e00700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73bc109840 0x7f73bc1a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.502+0000 7f73b9e00700 1 -- 172.21.15.44:0/1036816888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73bc108ef0 msgr2=0x7f73bc19d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.502+0000 7f73b9e00700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73bc108ef0 0x7f73bc19d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.502+0000 7f73b9e00700 1 -- 172.21.15.44:0/1036816888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73bc1a1ee0 con 0x7f73bc1045c0 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.502+0000 7f73b9400700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73bc108ef0 0x7f73bc19d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.503+0000 7f73b9e00700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73bc1045c0 0x7f73bc19d010 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00e9f0 tx=0x7f73ac00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.503+0000 7f73b2a00700 1 -- 172.21.15.44:0/1036816888 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac00cb70 con 0x7f73bc1045c0 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.503+0000 7f73c0a00700 1 -- 172.21.15.44:0/1036816888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73bc1a21d0 con 0x7f73bc1045c0 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.503+0000 7f73b2a00700 1 -- 172.21.15.44:0/1036816888 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73ac00ee90 con 0x7f73bc1045c0 2024-05-27T09:49:41.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.503+0000 7f73b2a00700 1 -- 172.21.15.44:0/1036816888 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac003d40 con 0x7f73bc1045c0 2024-05-27T09:49:41.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.503+0000 7f73c0a00700 1 -- 172.21.15.44:0/1036816888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73bc1a9af0 con 0x7f73bc1045c0 2024-05-27T09:49:41.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.504+0000 7f73c0a00700 1 -- 172.21.15.44:0/1036816888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73bc0512b0 con 0x7f73bc1045c0 2024-05-27T09:49:41.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.507+0000 7f73b2a00700 1 -- 172.21.15.44:0/1036816888 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f73ac003ee0 con 0x7f73bc1045c0 2024-05-27T09:49:41.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.508+0000 7f73b2a00700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73a00b29f0 0x7f73a00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:41.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.508+0000 7f73b2a00700 1 -- 172.21.15.44:0/1036816888 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f73ac0e2a30 con 0x7f73bc1045c0 2024-05-27T09:49:41.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.509+0000 7f73b9400700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73a00b29f0 0x7f73a00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:41.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.509+0000 7f73b9400700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73a00b29f0 0x7f73a00b4f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f73bc10a7e0 tx=0x7f73a8009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:41.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.511+0000 7f73b2a00700 1 -- 172.21.15.44:0/1036816888 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f73ac054df0 con 0x7f73bc1045c0 2024-05-27T09:49:41.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.803+0000 7f73c0a00700 1 -- 172.21.15.44:0/1036816888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73bc19e370 con 0x7f73bc1045c0 2024-05-27T09:49:41.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.804+0000 7f73b2a00700 1 -- 172.21.15.44:0/1036816888 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11870 (secure 0 0 0) 0x7f73bc19e370 con 0x7f73bc1045c0 2024-05-27T09:49:41.806 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:49:41.807 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:48:16.864365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24503]},"id":1}]} 2024-05-27T09:49:41.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.806+0000 7f73c0a00700 1 -- 172.21.15.44:0/1036816888 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73a00b29f0 msgr2=0x7f73a00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:41.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.806+0000 7f73c0a00700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73a00b29f0 0x7f73a00b4f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f73bc10a7e0 tx=0x7f73a8009260 comp rx=0 tx=0).stop 2024-05-27T09:49:41.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.808+0000 7f73c0a00700 1 -- 172.21.15.44:0/1036816888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73bc1045c0 msgr2=0x7f73bc19d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:41.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.808+0000 7f73c0a00700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73bc1045c0 0x7f73bc19d010 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00e9f0 tx=0x7f73ac00ed00 comp rx=0 tx=0).stop 2024-05-27T09:49:41.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.808+0000 7f73c0a00700 1 -- 172.21.15.44:0/1036816888 shutdown_connections 2024-05-27T09:49:41.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.808+0000 7f73c0a00700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73a00b29f0 0x7f73a00b4f00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:41.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.808+0000 7f73c0a00700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73bc109840 0x7f73bc1a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:41.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.808+0000 7f73c0a00700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73bc1045c0 0x7f73bc19d010 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:41.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.808+0000 7f73c0a00700 1 --2- 172.21.15.44:0/1036816888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73bc108ef0 0x7f73bc19d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:41.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.808+0000 7f73c0a00700 1 -- 172.21.15.44:0/1036816888 >> 172.21.15.44:0/1036816888 conn(0x7f73bc0ffe90 msgr2=0x7f73bc110e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:41.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.808+0000 7f73c0a00700 1 -- 172.21.15.44:0/1036816888 shutdown_connections 2024-05-27T09:49:41.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:41.808+0000 7f73c0a00700 1 -- 172.21.15.44:0/1036816888 wait complete. 2024-05-27T09:49:41.811 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 43 2024-05-27T09:49:42.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:42 smithi044 ceph-mon[31618]: pgmap v455: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 KiB/s wr, 101 op/s 2024-05-27T09:49:42.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:42 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1036816888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:49:42.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:42 smithi050 ceph-mon[36371]: pgmap v455: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 KiB/s wr, 101 op/s 2024-05-27T09:49:42.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:42 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1036816888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:49:42.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:42 smithi189 ceph-mon[36671]: pgmap v455: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 KiB/s wr, 101 op/s 2024-05-27T09:49:42.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:42 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1036816888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:49:43.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.254+0000 7f93dae00700 1 -- 172.21.15.44:0/3052076627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93d4075d20 msgr2=0x7f93d4076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:43.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.254+0000 7f93dae00700 1 --2- 172.21.15.44:0/3052076627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93d4075d20 0x7f93d4076130 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009990 tx=0x7f93c8009ca0 comp rx=0 tx=0).stop 2024-05-27T09:49:43.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.254+0000 7f93dae00700 1 -- 172.21.15.44:0/3052076627 shutdown_connections 2024-05-27T09:49:43.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.254+0000 7f93dae00700 1 --2- 172.21.15.44:0/3052076627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93d41109a0 0x7f93d4112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:43.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.254+0000 7f93dae00700 1 --2- 172.21.15.44:0/3052076627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93d4075d20 0x7f93d4076130 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:43.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.254+0000 7f93dae00700 1 --2- 172.21.15.44:0/3052076627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93d4076670 0x7f93d4110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:43.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.254+0000 7f93dae00700 1 -- 172.21.15.44:0/3052076627 >> 172.21.15.44:0/3052076627 conn(0x7f93d4079d00 msgr2=0x7f93d407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:43.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.254+0000 7f93dae00700 1 -- 172.21.15.44:0/3052076627 shutdown_connections 2024-05-27T09:49:43.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.254+0000 7f93dae00700 1 -- 172.21.15.44:0/3052076627 wait complete. 2024-05-27T09:49:43.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.255+0000 7f93dae00700 1 Processor -- start 2024-05-27T09:49:43.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.255+0000 7f93dae00700 1 -- start start 2024-05-27T09:49:43.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.256+0000 7f93dae00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93d4075d20 0x7f93d419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:43.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.256+0000 7f93dae00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93d4076670 0x7f93d419d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:43.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.256+0000 7f93dae00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93d41109a0 0x7f93d41a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:43.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.256+0000 7f93dae00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f93d41164e0 con 0x7f93d4076670 2024-05-27T09:49:43.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.256+0000 7f93dae00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f93d4116360 con 0x7f93d41109a0 2024-05-27T09:49:43.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.256+0000 7f93dae00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93d4116660 con 0x7f93d4075d20 2024-05-27T09:49:43.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.256+0000 7f93d9000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93d4076670 0x7f93d419d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:43.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.256+0000 7f93d9a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93d4075d20 0x7f93d419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:43.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.256+0000 7f93d9a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93d4075d20 0x7f93d419cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51496/0 (socket says 172.21.15.44:51496) 2024-05-27T09:49:43.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.256+0000 7f93d9a00700 1 -- 172.21.15.44:0/1282165008 learned_addr learned my addr 172.21.15.44:0/1282165008 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:43.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.256+0000 7f93da400700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93d41109a0 0x7f93d41a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:43.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.257+0000 7f93d9a00700 1 -- 172.21.15.44:0/1282165008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93d41109a0 msgr2=0x7f93d41a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:43.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.257+0000 7f93d9a00700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93d41109a0 0x7f93d41a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:43.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.257+0000 7f93d9a00700 1 -- 172.21.15.44:0/1282165008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93d4076670 msgr2=0x7f93d419d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:43.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.257+0000 7f93d9a00700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93d4076670 0x7f93d419d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:43.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.257+0000 7f93d9a00700 1 -- 172.21.15.44:0/1282165008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d41a1de0 con 0x7f93d4075d20 2024-05-27T09:49:43.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.257+0000 7f93d9a00700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93d4075d20 0x7f93d419cf10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f93c8000c00 tx=0x7f93c80057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:43.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.257+0000 7f93d2000700 1 -- 172.21.15.44:0/1282165008 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8027070 con 0x7f93d4075d20 2024-05-27T09:49:43.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.258+0000 7f93d2000700 1 -- 172.21.15.44:0/1282165008 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93c8022470 con 0x7f93d4075d20 2024-05-27T09:49:43.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.258+0000 7f93dae00700 1 -- 172.21.15.44:0/1282165008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d41a2070 con 0x7f93d4075d20 2024-05-27T09:49:43.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.258+0000 7f93d2000700 1 -- 172.21.15.44:0/1282165008 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8017650 con 0x7f93d4075d20 2024-05-27T09:49:43.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.258+0000 7f93dae00700 1 -- 172.21.15.44:0/1282165008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d41a99f0 con 0x7f93d4075d20 2024-05-27T09:49:43.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.259+0000 7f93dae00700 1 -- 172.21.15.44:0/1282165008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d4077290 con 0x7f93d4075d20 2024-05-27T09:49:43.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.261+0000 7f93d2000700 1 -- 172.21.15.44:0/1282165008 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f93c802a070 con 0x7f93d4075d20 2024-05-27T09:49:43.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.263+0000 7f93d2000700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93bc0b2a70 0x7f93bc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:43.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.263+0000 7f93d2000700 1 -- 172.21.15.44:0/1282165008 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f93c80e3590 con 0x7f93d4075d20 2024-05-27T09:49:43.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.263+0000 7f93d9000700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93bc0b2a70 0x7f93bc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:43.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.264+0000 7f93d9000700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93bc0b2a70 0x7f93bc0b4f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f93d41124f0 tx=0x7f93c0009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:43.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.266+0000 7f93d2000700 1 -- 172.21.15.44:0/1282165008 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f93c8013070 con 0x7f93d4075d20 2024-05-27T09:49:43.544 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db --set-id 9fa86eb2 --release --await' 2024-05-27T09:49:43.545 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db --set-id 9fa86eb2 --release --await 2024-05-27T09:49:43.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.565+0000 7f93dae00700 1 -- 172.21.15.44:0/1282165008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93d419e270 con 0x7f93d4075d20 2024-05-27T09:49:43.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.566+0000 7f93d2000700 1 -- 172.21.15.44:0/1282165008 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11870 (secure 0 0 0) 0x7f93c80aaf50 con 0x7f93d4075d20 2024-05-27T09:49:43.575 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:49:43.576 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:48:16.864365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24503]},"id":1}]} 2024-05-27T09:49:43.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 -- 172.21.15.44:0/1282165008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93bc0b2a70 msgr2=0x7f93bc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:43.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93bc0b2a70 0x7f93bc0b4f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f93d41124f0 tx=0x7f93c0009260 comp rx=0 tx=0).stop 2024-05-27T09:49:43.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 -- 172.21.15.44:0/1282165008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93d4075d20 msgr2=0x7f93d419cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:43.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93d4075d20 0x7f93d419cf10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f93c8000c00 tx=0x7f93c80057c0 comp rx=0 tx=0).stop 2024-05-27T09:49:43.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 -- 172.21.15.44:0/1282165008 shutdown_connections 2024-05-27T09:49:43.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f93bc0b2a70 0x7f93bc0b4f80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:43.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93d41109a0 0x7f93d41a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:43.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93d4075d20 0x7f93d419cf10 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:43.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 --2- 172.21.15.44:0/1282165008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93d4076670 0x7f93d419d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:43.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 -- 172.21.15.44:0/1282165008 >> 172.21.15.44:0/1282165008 conn(0x7f93d4079d00 msgr2=0x7f93d41020b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:43.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 -- 172.21.15.44:0/1282165008 shutdown_connections 2024-05-27T09:49:43.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:43.570+0000 7f93dae00700 1 -- 172.21.15.44:0/1282165008 wait complete. 2024-05-27T09:49:43.577 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 43 2024-05-27T09:49:44.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:44 smithi044 ceph-mon[31618]: pgmap v456: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 KiB/s wr, 98 op/s 2024-05-27T09:49:44.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:44 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1282165008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:49:44.479 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:49:44.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:44 smithi050 ceph-mon[36371]: pgmap v456: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 KiB/s wr, 98 op/s 2024-05-27T09:49:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:44 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1282165008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:49:44.586 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:49:44.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:44 smithi189 ceph-mon[36671]: pgmap v456: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 KiB/s wr, 98 op/s 2024-05-27T09:49:44.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:44 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1282165008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:49:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.319+0000 7f2bad600640 1 -- 172.21.15.44:0/2314423921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba8112160 msgr2=0x7f2ba8114580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.319+0000 7f2bad600640 1 --2- 172.21.15.44:0/2314423921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba8112160 0x7f2ba8114580 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009860 tx=0x7f2b9002a3d0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.319+0000 7f2bad600640 1 -- 172.21.15.44:0/2314423921 shutdown_connections 2024-05-27T09:49:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.319+0000 7f2bad600640 1 --2- 172.21.15.44:0/2314423921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ba8114ac0 0x7f2ba8116ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.319+0000 7f2bad600640 1 --2- 172.21.15.44:0/2314423921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba8112160 0x7f2ba8114580 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.319+0000 7f2bad600640 1 --2- 172.21.15.44:0/2314423921 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ba8075480 0x7f2ba8075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.319+0000 7f2bad600640 1 -- 172.21.15.44:0/2314423921 >> 172.21.15.44:0/2314423921 conn(0x7f2ba8079240 msgr2=0x7f2ba8079680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.319+0000 7f2bad600640 1 -- 172.21.15.44:0/2314423921 shutdown_connections 2024-05-27T09:49:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.319+0000 7f2bad600640 1 -- 172.21.15.44:0/2314423921 wait complete. 2024-05-27T09:49:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.320+0000 7f2bad600640 1 Processor -- start 2024-05-27T09:49:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2bad600640 1 -- start start 2024-05-27T09:49:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2bad600640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ba8075480 0x7f2ba81a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2bad600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba8112160 0x7f2ba81a6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2bad600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ba8114ac0 0x7f2ba81a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2bad600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba811a2a0 con 0x7f2ba8114ac0 2024-05-27T09:49:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2bad600640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba811a120 con 0x7f2ba8075480 2024-05-27T09:49:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2bad600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba811a420 con 0x7f2ba8112160 2024-05-27T09:49:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2b9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba8112160 0x7f2ba81a6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2b9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba8112160 0x7f2ba81a6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51514/0 (socket says 172.21.15.44:51514) 2024-05-27T09:49:45.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2b9fe00640 1 -- 172.21.15.44:0/454060915 learned_addr learned my addr 172.21.15.44:0/454060915 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:45.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2bacc00640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ba8114ac0 0x7f2ba81a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.321+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ba8075480 0x7f2ba81a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.322+0000 7f2b9fe00640 1 -- 172.21.15.44:0/454060915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ba8075480 msgr2=0x7f2ba81a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:45.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.322+0000 7f2b9fe00640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ba8075480 0x7f2ba81a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.322+0000 7f2b9fe00640 1 -- 172.21.15.44:0/454060915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ba8114ac0 msgr2=0x7f2ba81a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:45.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.322+0000 7f2b9fe00640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ba8114ac0 0x7f2ba81a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.322+0000 7f2b9fe00640 1 -- 172.21.15.44:0/454060915 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba81a11a0 con 0x7f2ba8112160 2024-05-27T09:49:45.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.322+0000 7f2bacc00640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ba8114ac0 0x7f2ba81a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:45.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.322+0000 7f2b9fe00640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba8112160 0x7f2ba81a6e10 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2b90005940 tx=0x7f2b90030fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:45.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.322+0000 7f2ba5600640 1 -- 172.21.15.44:0/454060915 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b90040070 con 0x7f2ba8112160 2024-05-27T09:49:45.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.323+0000 7f2bad600640 1 -- 172.21.15.44:0/454060915 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba81a1430 con 0x7f2ba8112160 2024-05-27T09:49:45.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.323+0000 7f2ba5600640 1 -- 172.21.15.44:0/454060915 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b90005570 con 0x7f2ba8112160 2024-05-27T09:49:45.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.323+0000 7f2ba5600640 1 -- 172.21.15.44:0/454060915 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b900046c0 con 0x7f2ba8112160 2024-05-27T09:49:45.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.323+0000 7f2bad600640 1 -- 172.21.15.44:0/454060915 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba81ad770 con 0x7f2ba8112160 2024-05-27T09:49:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.326+0000 7f2bb5c4d740 1 Processor -- start 2024-05-27T09:49:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.326+0000 7f2bb5c4d740 1 -- start start 2024-05-27T09:49:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.327+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed8169a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.327+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 0x558ed8169fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.327+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7da64d0 0x558ed816a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.327+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558ed80dab90 con 0x558ed7d9e2f0 2024-05-27T09:49:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.327+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558ed80ca730 con 0x558ed7dc71b0 2024-05-27T09:49:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.327+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558ed80ce770 con 0x558ed7da64d0 2024-05-27T09:49:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.327+0000 7f2ba5600640 1 -- 172.21.15.44:0/454060915 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f2b90043070 con 0x7f2ba8112160 2024-05-27T09:49:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.327+0000 7f2bacc00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed8169a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.327+0000 7f2bacc00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed8169a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39740/0 (socket says 172.21.15.44:39740) 2024-05-27T09:49:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.327+0000 7f2bacc00640 1 -- 172.21.15.44:0/932085077 learned_addr learned my addr 172.21.15.44:0/932085077 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.327+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/932085077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 0x558ed8169fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.328+0000 7f2bacc00640 1 -- 172.21.15.44:0/932085077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7da64d0 msgr2=0x558ed816a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.328+0000 7f2bacc00640 1 --2- 172.21.15.44:0/932085077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7da64d0 0x558ed816a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.328+0000 7f2bacc00640 1 -- 172.21.15.44:0/932085077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 msgr2=0x558ed8169fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.328+0000 7f2bacc00640 1 --2- 172.21.15.44:0/932085077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 0x558ed8169fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.328+0000 7f2bacc00640 1 -- 172.21.15.44:0/932085077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ed80638b0 con 0x558ed7dc71b0 2024-05-27T09:49:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.328+0000 7f2bacc00640 1 --2- 172.21.15.44:0/932085077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed8169a60 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000e870 tx=0x7f2ba00095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ce9fd07b81eac09 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.328+0000 7f2b9d200640 1 -- 172.21.15.44:0/932085077 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba000ddd0 con 0x558ed7dc71b0 2024-05-27T09:49:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.328+0000 7f2b9d200640 1 -- 172.21.15.44:0/932085077 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ba001c070 con 0x558ed7dc71b0 2024-05-27T09:49:45.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.328+0000 7f2b9d200640 1 -- 172.21.15.44:0/932085077 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00165e0 con 0x558ed7dc71b0 2024-05-27T09:49:45.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/932085077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 msgr2=0x558ed8169a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/932085077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed8169a60 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000e870 tx=0x7f2ba00095c0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/932085077 shutdown_connections 2024-05-27T09:49:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/932085077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7da64d0 0x558ed816a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/932085077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 0x558ed8169fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/932085077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed8169a60 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/932085077 >> 172.21.15.44:0/932085077 conn(0x558ed7dbb300 msgr2=0x558ed8166210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/932085077 shutdown_connections 2024-05-27T09:49:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/932085077 wait complete. 2024-05-27T09:49:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2ba5600640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2b8c0b28e0 0x7f2b8c0b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2ba5600640 1 -- 172.21.15.44:0/454060915 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f2b90103160 con 0x7f2ba8112160 2024-05-27T09:49:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.330+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2b8c0b28e0 0x7f2b8c0b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.331+0000 7f2bb5c4d740 1 -- start start 2024-05-27T09:49:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.331+0000 7f2bb5c4d740 1 Processor -- start 2024-05-27T09:49:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.331+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2b8c0b28e0 0x7f2b8c0b4df0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x558ed816ae10 tx=0x7f2b98005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 0x558ed8173c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7da64d0 0x558ed8174170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed817fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558ed7db0790 con 0x558ed7d9e2f0 2024-05-27T09:49:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558ed8072bc0 con 0x558ed7dc71b0 2024-05-27T09:49:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558ed8176b90 con 0x558ed7da64d0 2024-05-27T09:49:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2bb5c4d740 10 client.0 _ll_register_callbacks cb 0x558ed8050400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2ba7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 0x558ed8173c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2ba7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 0x558ed8173c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54936/0 (socket says 172.21.15.44:54936) 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2ba7e00640 1 -- 172.21.15.44:0/2451869587 learned_addr learned my addr 172.21.15.44:0/2451869587 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2bacc00640 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed817fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2b9fe00640 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7da64d0 0x558ed8174170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2ba7e00640 1 -- 172.21.15.44:0/2451869587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7da64d0 msgr2=0x558ed8174170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7da64d0 0x558ed8174170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2ba7e00640 1 -- 172.21.15.44:0/2451869587 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 msgr2=0x558ed817fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed817fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.332+0000 7f2ba7e00640 1 -- 172.21.15.44:0/2451869587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ed7ddf210 con 0x558ed7d9e2f0 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.333+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 0x558ed8173c30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009fd0 tx=0x7f2b98004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e70dc6f0d81ed3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.333+0000 7f2b95600640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:49:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.333+0000 7f2b95600640 1 -- 172.21.15.44:0/2451869587 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b98029440 con 0x558ed7d9e2f0 2024-05-27T09:49:45.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.333+0000 7f2b95600640 1 -- 172.21.15.44:0/2451869587 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f2b9801f430 con 0x558ed7d9e2f0 2024-05-27T09:49:45.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.333+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/2451869587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ba001c070 con 0x558ed7d9e2f0 2024-05-27T09:49:45.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.333+0000 7f2b95600640 1 -- 172.21.15.44:0/2451869587 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f2b9801f630 con 0x558ed7d9e2f0 2024-05-27T09:49:45.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.334+0000 7f2bb5c4d740 10 client.17715 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:45.336 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.334+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/2451869587 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x558ed7eab620 con 0x558ed7d9e2f0 2024-05-27T09:49:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.334+0000 7f2b95600640 1 -- 172.21.15.44:0/2451869587 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ba001c070 con 0x558ed7d9e2f0 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.334+0000 7f2b95600640 1 -- 172.21.15.44:0/2451869587 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f2b98027070 con 0x558ed7d9e2f0 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.334+0000 7f2bb5c4d740 10 client.17715 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.334+0000 7f2bb5c4d740 10 client.17715 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.334+0000 7f2bb5c4d740 20 client.17715 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.334+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558ed7d85d80 0x558ed8185f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.334+0000 7f2bb5c4d740 4 client.17715 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.334+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/2451869587 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ed80dcae0 con 0x558ed7d85d80 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.335+0000 7f2b9fe00640 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558ed7d85d80 0x558ed8185f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.335+0000 7f2b9fe00640 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558ed7d85d80 0x558ed8185f30 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad6ab14f39956ce8 server_cookie=46200f5d2d225669 in_seq=0 out_seq=0 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.335+0000 7f2b95600640 10 client.17715 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2b95600640 1 -- 172.21.15.44:0/2451869587 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558ed80dcae0 con 0x558ed7d85d80 2024-05-27T09:49:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2b95600640 10 client.17715 handle_command_reply: tid=0 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2bb5c4d740 1 client.17715 shutdown 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/2451869587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 msgr2=0x558ed8173c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 0x558ed8173c30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009fd0 tx=0x7f2b98004730 comp rx=0 tx=0).stop 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/2451869587 shutdown_connections 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558ed7d85d80 0x558ed8185f30 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed817fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7da64d0 0x558ed8174170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/2451869587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7d9e2f0 0x558ed8173c30 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2b95600640 0 client.17715 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/2451869587 >> 172.21.15.44:0/2451869587 conn(0x558ed7dbb300 msgr2=0x558ed816b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.336+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/2451869587 shutdown_connections 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.337+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/2451869587 wait complete. 2024-05-27T09:49:45.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.337+0000 7f2bb5c4d740 20 client.17715 trim_cache size 0 max 16384 2024-05-27T09:49:45.354 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.354+0000 7f2bb5c4d740 1 Processor -- start 2024-05-27T09:49:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.354+0000 7f2bb5c4d740 1 -- start start 2024-05-27T09:49:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.354+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7d9e2f0 0x558ed816f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.354+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 0x558ed81736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.354+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed817fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.354+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba81b8a10 con 0x558ed7da64d0 2024-05-27T09:49:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.354+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba81b6280 con 0x558ed7dc71b0 2024-05-27T09:49:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.354+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba81b40f0 con 0x558ed7d9e2f0 2024-05-27T09:49:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.355+0000 7f2b9fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 0x558ed81736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.355+0000 7f2ba7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7d9e2f0 0x558ed816f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.355+0000 7f2bacc00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed817fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.355+0000 7f2b9fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 0x558ed81736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54940/0 (socket says 172.21.15.44:54940) 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.355+0000 7f2b9fe00640 1 -- 172.21.15.44:0/4195494798 learned_addr learned my addr 172.21.15.44:0/4195494798 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.355+0000 7f2ba7e00640 1 -- 172.21.15.44:0/4195494798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 msgr2=0x558ed817fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.355+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/4195494798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed817fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.355+0000 7f2ba7e00640 1 -- 172.21.15.44:0/4195494798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 msgr2=0x558ed81736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.355+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/4195494798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 0x558ed81736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.355+0000 7f2ba7e00640 1 -- 172.21.15.44:0/4195494798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ed8164610 con 0x558ed7d9e2f0 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.355+0000 7f2bacc00640 1 --2- 172.21.15.44:0/4195494798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed817fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.356+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/4195494798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7d9e2f0 0x558ed816f470 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2b98029de0 tx=0x7f2b98003e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c443857d48f1926 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.356+0000 7f2b97e00640 1 -- 172.21.15.44:0/4195494798 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b98029e60 con 0x558ed7d9e2f0 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.356+0000 7f2b97e00640 1 -- 172.21.15.44:0/4195494798 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b9801f7d0 con 0x558ed7d9e2f0 2024-05-27T09:49:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.356+0000 7f2b97e00640 1 -- 172.21.15.44:0/4195494798 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9801d620 con 0x558ed7d9e2f0 2024-05-27T09:49:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.357+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/4195494798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7d9e2f0 msgr2=0x558ed816f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.357+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/4195494798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7d9e2f0 0x558ed816f470 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2b98029de0 tx=0x7f2b98003e50 comp rx=0 tx=0).stop 2024-05-27T09:49:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.357+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/4195494798 shutdown_connections 2024-05-27T09:49:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.357+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/4195494798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed817fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.357+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/4195494798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 0x558ed81736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.357+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/4195494798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7d9e2f0 0x558ed816f470 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.357+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/4195494798 >> 172.21.15.44:0/4195494798 conn(0x558ed7dbb300 msgr2=0x558ed816a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.357+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/4195494798 shutdown_connections 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.357+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/4195494798 wait complete. 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.358+0000 7f2bb5c4d740 1 -- start start 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.358+0000 7f2bb5c4d740 1 Processor -- start 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.358+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7d9e2f0 0x558ed817be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.358+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 0x558ed817c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.358+0000 7f2bb5c4d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed8177410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.358+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba81b4740 con 0x558ed7da64d0 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.358+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba81b5bb0 con 0x558ed7dc71b0 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.358+0000 7f2bb5c4d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba81b4e10 con 0x558ed7d9e2f0 2024-05-27T09:49:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.358+0000 7f2bb5c4d740 10 client.0 _ll_register_callbacks cb 0x7f2b98009680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2b9fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 0x558ed817c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2b9fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 0x558ed817c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54952/0 (socket says 172.21.15.44:54952) 2024-05-27T09:49:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2b9fe00640 1 -- 172.21.15.44:0/737789225 learned_addr learned my addr 172.21.15.44:0/737789225 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7d9e2f0 0x558ed817be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2bacc00640 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed8177410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2b9fe00640 1 -- 172.21.15.44:0/737789225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7d9e2f0 msgr2=0x558ed817be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2b9fe00640 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7d9e2f0 0x558ed817be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2b9fe00640 1 -- 172.21.15.44:0/737789225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 msgr2=0x558ed8177410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2b9fe00640 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed8177410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2b9fe00640 1 -- 172.21.15.44:0/737789225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ed8177ad0 con 0x558ed7da64d0 2024-05-27T09:49:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2b9fe00640 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 0x558ed817c370 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2b900ca3d0 tx=0x7f2b9002a3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2310cd713bfcdcee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.359+0000 7f2b95600640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:49:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2b95600640 1 -- 172.21.15.44:0/737789225 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9004a040 con 0x558ed7da64d0 2024-05-27T09:49:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2b95600640 1 -- 172.21.15.44:0/737789225 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f2b90031400 con 0x558ed7da64d0 2024-05-27T09:49:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/737789225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b9801f7d0 con 0x558ed7da64d0 2024-05-27T09:49:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2b95600640 1 -- 172.21.15.44:0/737789225 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f2b900ca440 con 0x558ed7da64d0 2024-05-27T09:49:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2bb5c4d740 10 client.17721 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/737789225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x558ed81edf60 con 0x558ed7da64d0 2024-05-27T09:49:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2b95600640 1 -- 172.21.15.44:0/737789225 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b9801f7d0 con 0x558ed7da64d0 2024-05-27T09:49:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2b95600640 1 -- 172.21.15.44:0/737789225 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f2b90100af0 con 0x558ed7da64d0 2024-05-27T09:49:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2bb5c4d740 10 client.17721 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:45.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2bb5c4d740 10 client.17721 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:45.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2bb5c4d740 20 client.17721 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:45.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558ed7d5b1c0 0x558ed81f06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:45.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2bb5c4d740 4 client.17721 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:49:45.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.360+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/737789225 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558ed81745a0 con 0x558ed7d5b1c0 2024-05-27T09:49:45.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.361+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558ed7d5b1c0 0x558ed81f06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:45.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.361+0000 7f2ba7e00640 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558ed7d5b1c0 0x558ed81f06a0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5fbc9e1b31a6d50 server_cookie=8d668791d70542ca in_seq=0 out_seq=0 2024-05-27T09:49:45.364 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.361+0000 7f2b95600640 10 client.17721 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2b95600640 1 -- 172.21.15.44:0/737789225 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x558ed81745a0 con 0x558ed7d5b1c0 2024-05-27T09:49:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2b95600640 10 client.17721 handle_command_reply: tid=0 2024-05-27T09:49:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 1 client.17721 shutdown 2024-05-27T09:49:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/737789225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 msgr2=0x558ed817c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 0x558ed817c370 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2b900ca3d0 tx=0x7f2b9002a3d0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/737789225 shutdown_connections 2024-05-27T09:49:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x558ed7d5b1c0 0x558ed81f06a0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558ed7dc71b0 0x558ed8177410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558ed7da64d0 0x558ed817c370 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 1 --2- 172.21.15.44:0/737789225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ed7d9e2f0 0x558ed817be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/737789225 >> 172.21.15.44:0/737789225 conn(0x558ed7dbb300 msgr2=0x558ed8175080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2b95600640 0 client.17721 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/737789225 shutdown_connections 2024-05-27T09:49:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 1 -- 172.21.15.44:0/737789225 wait complete. 2024-05-27T09:49:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.506+0000 7f2bb5c4d740 20 client.17721 trim_cache size 0 max 16384 2024-05-27T09:49:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:49:45.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:49:45.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:49:45.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.507+0000 7f2bad600640 1 -- 172.21.15.44:0/454060915 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2b8c0b28e0 msgr2=0x7f2b8c0b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:45.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.507+0000 7f2bad600640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2b8c0b28e0 0x7f2b8c0b4df0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x558ed816ae10 tx=0x7f2b98005f90 comp rx=0 tx=0).stop 2024-05-27T09:49:45.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.507+0000 7f2bad600640 1 -- 172.21.15.44:0/454060915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba8112160 msgr2=0x7f2ba81a6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.507+0000 7f2bad600640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba8112160 0x7f2ba81a6e10 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2b90005940 tx=0x7f2b90030fa0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.507+0000 7f2bad600640 1 -- 172.21.15.44:0/454060915 shutdown_connections 2024-05-27T09:49:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.507+0000 7f2bad600640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f2b8c0b28e0 0x7f2b8c0b4df0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.507+0000 7f2bad600640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ba8114ac0 0x7f2ba81a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.507+0000 7f2bad600640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ba8112160 0x7f2ba81a6e10 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.507+0000 7f2bad600640 1 --2- 172.21.15.44:0/454060915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ba8075480 0x7f2ba81a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.507+0000 7f2bad600640 1 -- 172.21.15.44:0/454060915 >> 172.21.15.44:0/454060915 conn(0x7f2ba8079240 msgr2=0x7f2ba8112570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.507+0000 7f2bad600640 1 -- 172.21.15.44:0/454060915 shutdown_connections 2024-05-27T09:49:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:45.508+0000 7f2bad600640 1 -- 172.21.15.44:0/454060915 wait complete. 2024-05-27T09:49:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:49:46.455 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:46 smithi044 ceph-mon[31618]: pgmap v457: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 KiB/s wr, 98 op/s 2024-05-27T09:49:46.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:46 smithi050 ceph-mon[36371]: pgmap v457: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 KiB/s wr, 98 op/s 2024-05-27T09:49:46.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:46 smithi189 ceph-mon[36671]: pgmap v457: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 KiB/s wr, 98 op/s 2024-05-27T09:49:47.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.894+0000 7fcace800700 1 -- 172.21.15.44:0/837870054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcac8103260 msgr2=0x7fcac8103670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:47.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.894+0000 7fcace800700 1 --2- 172.21.15.44:0/837870054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcac8103260 0x7fcac8103670 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009990 tx=0x7fcab8009ca0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.894+0000 7fcace800700 1 -- 172.21.15.44:0/837870054 shutdown_connections 2024-05-27T09:49:47.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.894+0000 7fcace800700 1 --2- 172.21.15.44:0/837870054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac81086b0 0x7fcac8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.894+0000 7fcace800700 1 --2- 172.21.15.44:0/837870054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcac8103260 0x7fcac8103670 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.894+0000 7fcace800700 1 --2- 172.21.15.44:0/837870054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcac8103bb0 0x7fcac81080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.894+0000 7fcace800700 1 -- 172.21.15.44:0/837870054 >> 172.21.15.44:0/837870054 conn(0x7fcac80fef80 msgr2=0x7fcac81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:47.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.894+0000 7fcace800700 1 -- 172.21.15.44:0/837870054 shutdown_connections 2024-05-27T09:49:47.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.895+0000 7fcace800700 1 -- 172.21.15.44:0/837870054 wait complete. 2024-05-27T09:49:47.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.896+0000 7fcace800700 1 Processor -- start 2024-05-27T09:49:47.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.896+0000 7fcace800700 1 -- start start 2024-05-27T09:49:47.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.896+0000 7fcace800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcac8103260 0x7fcac81a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.896+0000 7fcace800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac8103bb0 0x7fcac81a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.896+0000 7fcace800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcac81086b0 0x7fcac81a5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.896+0000 7fcace800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcac811aa30 con 0x7fcac8103260 2024-05-27T09:49:47.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.896+0000 7fcace800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcac811a8b0 con 0x7fcac81086b0 2024-05-27T09:49:47.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.896+0000 7fcace800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcac811abb0 con 0x7fcac8103bb0 2024-05-27T09:49:47.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.897+0000 7fcac7400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcac8103260 0x7fcac81a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.897+0000 7fcac6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac8103bb0 0x7fcac81a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.897+0000 7fcac6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac8103bb0 0x7fcac81a1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51556/0 (socket says 172.21.15.44:51556) 2024-05-27T09:49:47.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.897+0000 7fcac6a00700 1 -- 172.21.15.44:0/678881555 learned_addr learned my addr 172.21.15.44:0/678881555 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:47.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.897+0000 7fcac7e00700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcac81086b0 0x7fcac81a5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.897+0000 7fcac6a00700 1 -- 172.21.15.44:0/678881555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcac81086b0 msgr2=0x7fcac81a5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:47.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.897+0000 7fcac6a00700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcac81086b0 0x7fcac81a5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.897+0000 7fcac6a00700 1 -- 172.21.15.44:0/678881555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcac8103260 msgr2=0x7fcac81a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:47.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.897+0000 7fcac6a00700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcac8103260 0x7fcac81a13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.897+0000 7fcac6a00700 1 -- 172.21.15.44:0/678881555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac81a6290 con 0x7fcac8103bb0 2024-05-27T09:49:47.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.898+0000 7fcac6a00700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac8103bb0 0x7fcac81a1960 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcab400eb30 tx=0x7fcab400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:47.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.898+0000 7fcabfe00700 1 -- 172.21.15.44:0/678881555 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab400cbd0 con 0x7fcac8103bb0 2024-05-27T09:49:47.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.898+0000 7fcace800700 1 -- 172.21.15.44:0/678881555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac81a6580 con 0x7fcac8103bb0 2024-05-27T09:49:47.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.898+0000 7fcabfe00700 1 -- 172.21.15.44:0/678881555 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcab40044d0 con 0x7fcac8103bb0 2024-05-27T09:49:47.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.898+0000 7fcabfe00700 1 -- 172.21.15.44:0/678881555 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab4003df0 con 0x7fcac8103bb0 2024-05-27T09:49:47.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.898+0000 7fcace800700 1 -- 172.21.15.44:0/678881555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac81adea0 con 0x7fcac8103bb0 2024-05-27T09:49:47.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.903+0000 7fcabfe00700 1 -- 172.21.15.44:0/678881555 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fcab4004000 con 0x7fcac8103bb0 2024-05-27T09:49:47.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.903+0000 7fcadd3e0b80 1 Processor -- start 2024-05-27T09:49:47.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.903+0000 7fcadd3e0b80 1 -- start start 2024-05-27T09:49:47.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.903+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 0x560e8e34f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e34fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e3495e0 0x560e8e3499f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560e8e276f40 con 0x560e8e2b5af0 2024-05-27T09:49:47.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560e8e266ae0 con 0x560e8e1c9f40 2024-05-27T09:49:47.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560e8e26ab20 con 0x560e8e3495e0 2024-05-27T09:49:47.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcac7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 0x560e8e34f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcac7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e34fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcac7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e34fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39798/0 (socket says 172.21.15.44:39798) 2024-05-27T09:49:47.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcac7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 0x560e8e34f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54976/0 (socket says 172.21.15.44:54976) 2024-05-27T09:49:47.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcac7400700 1 -- 172.21.15.44:0/2571232480 learned_addr learned my addr 172.21.15.44:0/2571232480 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:47.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcac6a00700 1 --2- 172.21.15.44:0/2571232480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e3495e0 0x560e8e3499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.904+0000 7fcabfe00700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcaac0b29f0 0x7fcaac0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.905+0000 7fcac7400700 1 -- 172.21.15.44:0/2571232480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e3495e0 msgr2=0x560e8e3499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:47.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.905+0000 7fcac7400700 1 --2- 172.21.15.44:0/2571232480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e3495e0 0x560e8e3499f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.905+0000 7fcac7400700 1 -- 172.21.15.44:0/2571232480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 msgr2=0x560e8e34f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:47.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.905+0000 7fcac7400700 1 --2- 172.21.15.44:0/2571232480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 0x560e8e34f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.905+0000 7fcac7400700 1 -- 172.21.15.44:0/2571232480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e8e22f520 con 0x560e8e1c9f40 2024-05-27T09:49:47.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.905+0000 7fcabfe00700 1 -- 172.21.15.44:0/678881555 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7fcab40e2920 con 0x7fcac8103bb0 2024-05-27T09:49:47.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.905+0000 7fcac7e00700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcaac0b29f0 0x7fcaac0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.905+0000 7fcac7400700 1 --2- 172.21.15.44:0/2571232480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e34fa90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009990 tx=0x7fcab80096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=124627b5ae99af5c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:47.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.906+0000 7fcabcc00700 1 -- 172.21.15.44:0/2571232480 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab8027070 con 0x560e8e1c9f40 2024-05-27T09:49:47.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.906+0000 7fcabcc00700 1 -- 172.21.15.44:0/2571232480 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcab80056b0 con 0x560e8e1c9f40 2024-05-27T09:49:47.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.906+0000 7fcabcc00700 1 -- 172.21.15.44:0/2571232480 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab8003c70 con 0x560e8e1c9f40 2024-05-27T09:49:47.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.907+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/2571232480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 msgr2=0x560e8e34fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:47.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.907+0000 7fcac7e00700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcaac0b29f0 0x7fcaac0b4f00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009fd0 tx=0x7fcac0009040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:47.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.907+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/2571232480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e34fa90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009990 tx=0x7fcab80096a0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.907+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/2571232480 shutdown_connections 2024-05-27T09:49:47.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.907+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/2571232480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e3495e0 0x560e8e3499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.907+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/2571232480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 0x560e8e34f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.907+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/2571232480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e34fa90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.907+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/2571232480 >> 172.21.15.44:0/2571232480 conn(0x560e8e263490 msgr2=0x560e8e344650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:47.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.907+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/2571232480 shutdown_connections 2024-05-27T09:49:47.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.907+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/2571232480 wait complete. 2024-05-27T09:49:47.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.907+0000 7fcadd3e0b80 1 -- start start 2024-05-27T09:49:47.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.908+0000 7fcadd3e0b80 1 Processor -- start 2024-05-27T09:49:47.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.908+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 0x560e8e353330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.908+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 0x560e8e353870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.908+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e3495e0 0x560e8e353db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.908+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560e8e16d0a0 con 0x560e8e2b5af0 2024-05-27T09:49:47.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.908+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560e8e207860 con 0x560e8e3495e0 2024-05-27T09:49:47.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.908+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560e8e34a370 con 0x560e8e1c9f40 2024-05-27T09:49:47.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.908+0000 7fcadd3e0b80 10 client.0 _ll_register_callbacks cb 0x560e8e210d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:47.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.909+0000 7fcac6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 0x560e8e353870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.909+0000 7fcac7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 0x560e8e353330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.909+0000 7fcac6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 0x560e8e353870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54990/0 (socket says 172.21.15.44:54990) 2024-05-27T09:49:47.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.909+0000 7fcac6a00700 1 -- 172.21.15.44:0/332118009 learned_addr learned my addr 172.21.15.44:0/332118009 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:47.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.909+0000 7fcac7e00700 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e3495e0 0x560e8e353db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.909+0000 7fcac6a00700 1 -- 172.21.15.44:0/332118009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 msgr2=0x560e8e353330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:47.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.909+0000 7fcac6a00700 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 0x560e8e353330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.909+0000 7fcac6a00700 1 -- 172.21.15.44:0/332118009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e3495e0 msgr2=0x560e8e353db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:47.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.909+0000 7fcac6a00700 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e3495e0 0x560e8e353db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.909+0000 7fcac6a00700 1 -- 172.21.15.44:0/332118009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e8e1b69d0 con 0x560e8e2b5af0 2024-05-27T09:49:47.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.909+0000 7fcac7400700 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 0x560e8e353330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:47.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.910+0000 7fcac6a00700 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 0x560e8e353870 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcab40e0080 tx=0x7fcab40e05a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7557f823efa7310 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:47.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.910+0000 7fcab1600700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:49:47.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.910+0000 7fcab1600700 1 -- 172.21.15.44:0/332118009 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab40cae60 con 0x560e8e2b5af0 2024-05-27T09:49:47.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.910+0000 7fcab1600700 1 -- 172.21.15.44:0/332118009 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7fcab40aa020 con 0x560e8e2b5af0 2024-05-27T09:49:47.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.910+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/332118009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcab80056b0 con 0x560e8e2b5af0 2024-05-27T09:49:47.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.910+0000 7fcab1600700 1 -- 172.21.15.44:0/332118009 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fcab40a5a70 con 0x560e8e2b5af0 2024-05-27T09:49:47.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.910+0000 7fcadd3e0b80 10 client.17733 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:47.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.910+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/332118009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x560e8e237160 con 0x560e8e2b5af0 2024-05-27T09:49:47.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.911+0000 7fcab1600700 1 -- 172.21.15.44:0/332118009 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcab80056b0 con 0x560e8e2b5af0 2024-05-27T09:49:47.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.911+0000 7fcab1600700 1 -- 172.21.15.44:0/332118009 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fcab40b2580 con 0x560e8e2b5af0 2024-05-27T09:49:47.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.911+0000 7fcadd3e0b80 10 client.17733 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:47.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.911+0000 7fcadd3e0b80 10 client.17733 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:49:47.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.911+0000 7fcadd3e0b80 20 client.17733 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:47.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.911+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e8e363090 0x560e8e3654b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.911+0000 7fcadd3e0b80 4 client.17733 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:47.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.911+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/332118009 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e8e278e90 con 0x560e8e363090 2024-05-27T09:49:47.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.912+0000 7fcac7400700 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e8e363090 0x560e8e3654b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.912+0000 7fcac7400700 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e8e363090 0x560e8e3654b0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dee4555ec5b3bf95 server_cookie=a5b7acece497b909 in_seq=0 out_seq=0 2024-05-27T09:49:47.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.912+0000 7fcab1600700 10 client.17733 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:47.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.913+0000 7fcab1600700 1 -- 172.21.15.44:0/332118009 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560e8e278e90 con 0x560e8e363090 2024-05-27T09:49:47.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.913+0000 7fcab1600700 10 client.17733 handle_command_reply: tid=0 2024-05-27T09:49:47.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.913+0000 7fcadd3e0b80 1 client.17733 shutdown 2024-05-27T09:49:47.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.913+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/332118009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 msgr2=0x560e8e353870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:47.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.913+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 0x560e8e353870 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcab40e0080 tx=0x7fcab40e05a0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.914+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/332118009 shutdown_connections 2024-05-27T09:49:47.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.914+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e8e363090 0x560e8e3654b0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.914+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e3495e0 0x560e8e353db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.914+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 0x560e8e353330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.914+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/332118009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e2b5af0 0x560e8e353870 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.914+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/332118009 >> 172.21.15.44:0/332118009 conn(0x560e8e263490 msgr2=0x560e8e3485d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:47.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.914+0000 7fcab1600700 0 client.17733 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:47.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.914+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/332118009 shutdown_connections 2024-05-27T09:49:47.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.914+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/332118009 wait complete. 2024-05-27T09:49:47.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.914+0000 7fcadd3e0b80 20 client.17733 trim_cache size 0 max 16384 2024-05-27T09:49:47.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.937+0000 7fcadd3e0b80 1 Processor -- start 2024-05-27T09:49:47.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.937+0000 7fcadd3e0b80 1 -- start start 2024-05-27T09:49:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.937+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e3522f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.937+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e2b5af0 0x560e8e3557d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.937+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 0x560e8e349000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.937+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcac81b7e50 con 0x560e8e3495e0 2024-05-27T09:49:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.937+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcac81b5680 con 0x560e8e1c9f40 2024-05-27T09:49:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.937+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcac81b34f0 con 0x560e8e2b5af0 2024-05-27T09:49:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.938+0000 7fcac7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e3522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.938+0000 7fcac7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 0x560e8e349000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.938+0000 7fcac7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e3522f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39824/0 (socket says 172.21.15.44:39824) 2024-05-27T09:49:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.938+0000 7fcac7400700 1 -- 172.21.15.44:0/852057185 learned_addr learned my addr 172.21.15.44:0/852057185 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.938+0000 7fcac6a00700 1 --2- 172.21.15.44:0/852057185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e2b5af0 0x560e8e3557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.938+0000 7fcac7400700 1 -- 172.21.15.44:0/852057185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e2b5af0 msgr2=0x560e8e3557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.938+0000 7fcac7400700 1 --2- 172.21.15.44:0/852057185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e2b5af0 0x560e8e3557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.938+0000 7fcac7400700 1 -- 172.21.15.44:0/852057185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 msgr2=0x560e8e349000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:47.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.938+0000 7fcac7400700 1 --2- 172.21.15.44:0/852057185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 0x560e8e349000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.938+0000 7fcac7400700 1 -- 172.21.15.44:0/852057185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e8e340940 con 0x560e8e1c9f40 2024-05-27T09:49:47.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.939+0000 7fcac7400700 1 --2- 172.21.15.44:0/852057185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e3522f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcab8003820 tx=0x7fcab8009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=343fc079b9ecca29 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:47.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.939+0000 7fcab3e00700 1 -- 172.21.15.44:0/852057185 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab8027040 con 0x560e8e1c9f40 2024-05-27T09:49:47.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.939+0000 7fcab3e00700 1 -- 172.21.15.44:0/852057185 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcab8003e00 con 0x560e8e1c9f40 2024-05-27T09:49:47.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.939+0000 7fcab3e00700 1 -- 172.21.15.44:0/852057185 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab80226e0 con 0x560e8e1c9f40 2024-05-27T09:49:47.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.940+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/852057185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 msgr2=0x560e8e3522f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:47.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.940+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/852057185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e3522f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcab8003820 tx=0x7fcab8009310 comp rx=0 tx=0).stop 2024-05-27T09:49:47.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.940+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/852057185 shutdown_connections 2024-05-27T09:49:47.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.940+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/852057185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 0x560e8e349000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.940+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/852057185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e1c9f40 0x560e8e3522f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.940+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/852057185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e2b5af0 0x560e8e3557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.940+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/852057185 >> 172.21.15.44:0/852057185 conn(0x560e8e263490 msgr2=0x560e8e353960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:47.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.940+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/852057185 shutdown_connections 2024-05-27T09:49:47.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.940+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/852057185 wait complete. 2024-05-27T09:49:47.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.941+0000 7fcadd3e0b80 1 -- start start 2024-05-27T09:49:47.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.941+0000 7fcadd3e0b80 1 Processor -- start 2024-05-27T09:49:47.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 0x560e8e35b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e2b5af0 0x560e8e36a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcadd3e0b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 0x560e8e36a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcac81b3b40 con 0x560e8e3495e0 2024-05-27T09:49:47.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcac81b4fb0 con 0x560e8e2b5af0 2024-05-27T09:49:47.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcadd3e0b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcac81b4210 con 0x560e8e1c9f40 2024-05-27T09:49:47.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcadd3e0b80 10 client.0 _ll_register_callbacks cb 0x560e8e210d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:47.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcac7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 0x560e8e36a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcac7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 0x560e8e36a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55018/0 (socket says 172.21.15.44:55018) 2024-05-27T09:49:47.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcac7e00700 1 -- 172.21.15.44:0/79182786 learned_addr learned my addr 172.21.15.44:0/79182786 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:47.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcac7400700 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 0x560e8e35b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcac6a00700 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e2b5af0 0x560e8e36a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcac7400700 1 -- 172.21.15.44:0/79182786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e2b5af0 msgr2=0x560e8e36a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:47.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcac7400700 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e2b5af0 0x560e8e36a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcac7400700 1 -- 172.21.15.44:0/79182786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 msgr2=0x560e8e36a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:47.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcac7400700 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 0x560e8e36a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcac7400700 1 -- 172.21.15.44:0/79182786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e8e36b040 con 0x560e8e1c9f40 2024-05-27T09:49:47.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.942+0000 7fcac7e00700 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 0x560e8e36a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:47.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.943+0000 7fcac7400700 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 0x560e8e35b8f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009ce0 tx=0x7fcab800be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7884a07bb4779bb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:47.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.943+0000 7fcab1600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:49:47.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.943+0000 7fcab1600700 1 -- 172.21.15.44:0/79182786 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab8027040 con 0x560e8e1c9f40 2024-05-27T09:49:47.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.943+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/79182786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcab8003e00 con 0x560e8e1c9f40 2024-05-27T09:49:47.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.943+0000 7fcab1600700 1 -- 172.21.15.44:0/79182786 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7fcab800b720 con 0x560e8e1c9f40 2024-05-27T09:49:47.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.943+0000 7fcab1600700 1 -- 172.21.15.44:0/79182786 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fcab8048050 con 0x560e8e1c9f40 2024-05-27T09:49:47.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.943+0000 7fcadd3e0b80 10 client.28015 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:47.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.943+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/79182786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x560e8e3cd980 con 0x560e8e1c9f40 2024-05-27T09:49:47.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.944+0000 7fcab1600700 1 -- 172.21.15.44:0/79182786 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcab8003e00 con 0x560e8e1c9f40 2024-05-27T09:49:47.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.944+0000 7fcab1600700 1 -- 172.21.15.44:0/79182786 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fcab8037020 con 0x560e8e1c9f40 2024-05-27T09:49:47.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.944+0000 7fcadd3e0b80 10 client.28015 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:47.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.944+0000 7fcadd3e0b80 10 client.28015 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:49:47.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.944+0000 7fcadd3e0b80 20 client.28015 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:47.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.944+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e8e3cdbb0 0x560e8e3cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:47.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.944+0000 7fcadd3e0b80 4 client.28015 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9fa86eb2", "release": true, "await": true}] 2024-05-27T09:49:47.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.944+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/79182786 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9fa86eb2", "release": true, "await": true}) -- 0x560e8e33c810 con 0x560e8e3cdbb0 2024-05-27T09:49:47.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.945+0000 7fcac6a00700 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e8e3cdbb0 0x560e8e3cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:47.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.945+0000 7fcac6a00700 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e8e3cdbb0 0x560e8e3cffd0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53e5b1e8ca710502 server_cookie=6e063bef7e7a9ad5 in_seq=0 out_seq=0 2024-05-27T09:49:47.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.945+0000 7fcab1600700 10 client.28015 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:47.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.948+0000 7fcab1600700 1 -- 172.21.15.44:0/79182786 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x560e8e33c810 con 0x560e8e3cdbb0 2024-05-27T09:49:47.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.948+0000 7fcab1600700 10 client.28015 handle_command_reply: tid=0 2024-05-27T09:49:47.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 1 client.28015 shutdown 2024-05-27T09:49:47.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/79182786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 msgr2=0x560e8e35b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:47.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 0x560e8e35b8f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009ce0 tx=0x7fcab800be20 comp rx=0 tx=0).stop 2024-05-27T09:49:47.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/79182786 shutdown_connections 2024-05-27T09:49:47.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e8e3cdbb0 0x560e8e3cffd0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e8e3495e0 0x560e8e36a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8e1c9f40 0x560e8e35b8f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 1 --2- 172.21.15.44:0/79182786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e8e2b5af0 0x560e8e36a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcab1600700 0 client.28015 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:47.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/79182786 >> 172.21.15.44:0/79182786 conn(0x560e8e263490 msgr2=0x560e8e34cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:47.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/79182786 shutdown_connections 2024-05-27T09:49:47.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 1 -- 172.21.15.44:0/79182786 wait complete. 2024-05-27T09:49:47.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.949+0000 7fcadd3e0b80 20 client.28015 trim_cache size 0 max 16384 2024-05-27T09:49:47.951 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:49:47.952 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 43, 2024-05-27T09:49:47.952 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:49:47.952 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 36, 2024-05-27T09:49:47.952 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:49:47.952 INFO:teuthology.orchestra.run.smithi044.stdout: "9fa86eb2": { 2024-05-27T09:49:47.952 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 36, 2024-05-27T09:49:47.952 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:49:47.952 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:49:47.952 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:49:47.952 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:49:47.952 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 130.0, 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RELEASED", 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:47.953 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:47.954 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:49:47.954 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:49:47.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.950+0000 7fcace800700 1 -- 172.21.15.44:0/678881555 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcaac0b29f0 msgr2=0x7fcaac0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:47.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.950+0000 7fcace800700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcaac0b29f0 0x7fcaac0b4f00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009fd0 tx=0x7fcac0009040 comp rx=0 tx=0).stop 2024-05-27T09:49:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.950+0000 7fcace800700 1 -- 172.21.15.44:0/678881555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac8103bb0 msgr2=0x7fcac81a1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.950+0000 7fcace800700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac8103bb0 0x7fcac81a1960 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcab400eb30 tx=0x7fcab400ee40 comp rx=0 tx=0).stop 2024-05-27T09:49:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.950+0000 7fcace800700 1 -- 172.21.15.44:0/678881555 shutdown_connections 2024-05-27T09:49:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.950+0000 7fcace800700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fcaac0b29f0 0x7fcaac0b4f00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.950+0000 7fcace800700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcac81086b0 0x7fcac81a5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.950+0000 7fcace800700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcac8103260 0x7fcac81a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.950+0000 7fcace800700 1 --2- 172.21.15.44:0/678881555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac8103bb0 0x7fcac81a1960 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.950+0000 7fcace800700 1 -- 172.21.15.44:0/678881555 >> 172.21.15.44:0/678881555 conn(0x7fcac80fef80 msgr2=0x7fcac8106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.950+0000 7fcace800700 1 -- 172.21.15.44:0/678881555 shutdown_connections 2024-05-27T09:49:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:47.951+0000 7fcace800700 1 -- 172.21.15.44:0/678881555 wait complete. 2024-05-27T09:49:47.977 INFO:tasks.workunit:Running workunit fs/misc/filelock_interrupt.py... 2024-05-27T09:49:47.978 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/filelock_interrupt.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6507388dd5057528934822c0163b0c347ef1d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_interrupt.py 2024-05-27T09:49:48.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:48 smithi044 ceph-mon[31618]: pgmap v458: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 37 KiB/s wr, 104 op/s 2024-05-27T09:49:48.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:48 smithi050 ceph-mon[36371]: pgmap v458: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 37 KiB/s wr, 104 op/s 2024-05-27T09:49:48.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:48 smithi189 ceph-mon[36671]: pgmap v458: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 37 KiB/s wr, 104 op/s 2024-05-27T09:49:48.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.757+0000 7f9e4be00700 1 -- 172.21.15.44:0/1735520208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e4c115030 msgr2=0x7f9e4c117450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:48.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.757+0000 7f9e4be00700 1 --2- 172.21.15.44:0/1735520208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e4c115030 0x7f9e4c117450 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400b0a0 tx=0x7f9e4400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.757+0000 7f9e4be00700 1 -- 172.21.15.44:0/1735520208 shutdown_connections 2024-05-27T09:49:48.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.757+0000 7f9e4be00700 1 --2- 172.21.15.44:0/1735520208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e4c115030 0x7f9e4c117450 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.757+0000 7f9e4be00700 1 --2- 172.21.15.44:0/1735520208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e4c077ce0 0x7f9e4c0780f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.757+0000 7f9e4be00700 1 --2- 172.21.15.44:0/1735520208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c078630 0x7f9e4c114af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.757+0000 7f9e4be00700 1 -- 172.21.15.44:0/1735520208 >> 172.21.15.44:0/1735520208 conn(0x7f9e4c101ec0 msgr2=0x7f9e4c1042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:48.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.757+0000 7f9e4be00700 1 -- 172.21.15.44:0/1735520208 shutdown_connections 2024-05-27T09:49:48.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.757+0000 7f9e4be00700 1 -- 172.21.15.44:0/1735520208 wait complete. 2024-05-27T09:49:48.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.759+0000 7f9e4be00700 1 Processor -- start 2024-05-27T09:49:48.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.759+0000 7f9e4be00700 1 -- start start 2024-05-27T09:49:48.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.760+0000 7f9e4be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e4c077ce0 0x7f9e4c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.760+0000 7f9e4be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e4c078630 0x7f9e4c1a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.760+0000 7f9e4be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c115030 0x7f9e4c1a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.760+0000 7f9e4be00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c11a920 con 0x7f9e4c115030 2024-05-27T09:49:48.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.760+0000 7f9e4be00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c11a7a0 con 0x7f9e4c077ce0 2024-05-27T09:49:48.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.760+0000 7f9e4be00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c11aaa0 con 0x7f9e4c078630 2024-05-27T09:49:48.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.760+0000 7f9e4aa00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e4c077ce0 0x7f9e4c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.760+0000 7f9e4a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e4c078630 0x7f9e4c1a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.760+0000 7f9e4aa00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e4c077ce0 0x7f9e4c1a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39848/0 (socket says 172.21.15.44:39848) 2024-05-27T09:49:48.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.760+0000 7f9e4aa00700 1 -- 172.21.15.44:0/351767438 learned_addr learned my addr 172.21.15.44:0/351767438 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:48.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.760+0000 7f9e4b400700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c115030 0x7f9e4c1a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.761+0000 7f9e4aa00700 1 -- 172.21.15.44:0/351767438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e4c078630 msgr2=0x7f9e4c1a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:48.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.761+0000 7f9e4aa00700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e4c078630 0x7f9e4c1a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.761+0000 7f9e4aa00700 1 -- 172.21.15.44:0/351767438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c115030 msgr2=0x7f9e4c1a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:48.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.761+0000 7f9e4aa00700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c115030 0x7f9e4c1a59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.761+0000 7f9e4aa00700 1 -- 172.21.15.44:0/351767438 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4c1a6060 con 0x7f9e4c077ce0 2024-05-27T09:49:48.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.761+0000 7f9e4b400700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c115030 0x7f9e4c1a59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:49:48.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.761+0000 7f9e4a000700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e4c078630 0x7f9e4c1a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:48.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.767+0000 7f9e4aa00700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e4c077ce0 0x7f9e4c1a1190 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c00e9a0 tx=0x7f9e3c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:48.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.767+0000 7f9e43400700 1 -- 172.21.15.44:0/351767438 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c00cbd0 con 0x7f9e4c077ce0 2024-05-27T09:49:48.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.767+0000 7f9e4be00700 1 -- 172.21.15.44:0/351767438 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e4c1a6350 con 0x7f9e4c077ce0 2024-05-27T09:49:48.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.767+0000 7f9e43400700 1 -- 172.21.15.44:0/351767438 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e3c00cd70 con 0x7f9e4c077ce0 2024-05-27T09:49:48.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.767+0000 7f9e43400700 1 -- 172.21.15.44:0/351767438 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c018680 con 0x7f9e4c077ce0 2024-05-27T09:49:48.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.767+0000 7f9e4be00700 1 -- 172.21.15.44:0/351767438 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e4c1adc70 con 0x7f9e4c077ce0 2024-05-27T09:49:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.773+0000 7f9e43400700 1 -- 172.21.15.44:0/351767438 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f9e3c018820 con 0x7f9e4c077ce0 2024-05-27T09:49:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.773+0000 7f9e60225b80 1 Processor -- start 2024-05-27T09:49:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.774+0000 7f9e60225b80 1 -- start start 2024-05-27T09:49:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.774+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b17ae40 0x55d13b2f8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.774+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9410 0x55d13b303ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.774+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b2f9820 0x55d13b306020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.774+0000 7f9e60225b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d13b227e40 con 0x55d13b2f9410 2024-05-27T09:49:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.774+0000 7f9e60225b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d13b2179e0 con 0x55d13b2f9820 2024-05-27T09:49:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.774+0000 7f9e60225b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d13b21ba20 con 0x55d13b17ae40 2024-05-27T09:49:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.775+0000 7f9e4aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9410 0x55d13b303ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.775+0000 7f9e4b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b17ae40 0x55d13b2f8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.775+0000 7f9e4b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b17ae40 0x55d13b2f8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51606/0 (socket says 172.21.15.44:51606) 2024-05-27T09:49:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.775+0000 7f9e4b400700 1 -- 172.21.15.44:0/2236761054 learned_addr learned my addr 172.21.15.44:0/2236761054 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.775+0000 7f9e4b400700 1 -- 172.21.15.44:0/2236761054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b2f9820 msgr2=0x55d13b306020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T09:49:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.775+0000 7f9e4b400700 1 --2- 172.21.15.44:0/2236761054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b2f9820 0x55d13b306020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.775+0000 7f9e4b400700 1 -- 172.21.15.44:0/2236761054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9410 msgr2=0x55d13b303ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.775+0000 7f9e4b400700 1 --2- 172.21.15.44:0/2236761054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9410 0x55d13b303ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.775+0000 7f9e4b400700 1 -- 172.21.15.44:0/2236761054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d13b1e0420 con 0x55d13b17ae40 2024-05-27T09:49:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.776+0000 7f9e43400700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9e300b2a70 0x7f9e300b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.776+0000 7f9e43400700 1 -- 172.21.15.44:0/351767438 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f9e3c0e2950 con 0x7f9e4c077ce0 2024-05-27T09:49:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.776+0000 7f9e4aa00700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9e300b2a70 0x7f9e300b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.777+0000 7f9e4aa00700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9e300b2a70 0x7f9e300b4f80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55d13b2f6050 tx=0x7f9e3c0e8530 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.780+0000 7f9e4b400700 1 --2- 172.21.15.44:0/2236761054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b17ae40 0x55d13b2f8ed0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9e44015000 tx=0x7f9e44009df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1469f6e49ff8a224 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.781+0000 7f9e37e00700 1 -- 172.21.15.44:0/2236761054 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4400bcb0 con 0x55d13b17ae40 2024-05-27T09:49:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.781+0000 7f9e37e00700 1 -- 172.21.15.44:0/2236761054 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e44009460 con 0x55d13b17ae40 2024-05-27T09:49:48.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.781+0000 7f9e37e00700 1 -- 172.21.15.44:0/2236761054 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e44005040 con 0x55d13b17ae40 2024-05-27T09:49:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.782+0000 7f9e60225b80 1 -- 172.21.15.44:0/2236761054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b17ae40 msgr2=0x55d13b2f8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.782+0000 7f9e60225b80 1 --2- 172.21.15.44:0/2236761054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b17ae40 0x55d13b2f8ed0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9e44015000 tx=0x7f9e44009df0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.782+0000 7f9e60225b80 1 -- 172.21.15.44:0/2236761054 shutdown_connections 2024-05-27T09:49:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.782+0000 7f9e60225b80 1 --2- 172.21.15.44:0/2236761054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b2f9820 0x55d13b306020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.782+0000 7f9e60225b80 1 --2- 172.21.15.44:0/2236761054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b17ae40 0x55d13b2f8ed0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.782+0000 7f9e60225b80 1 --2- 172.21.15.44:0/2236761054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9410 0x55d13b303ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.782+0000 7f9e60225b80 1 -- 172.21.15.44:0/2236761054 >> 172.21.15.44:0/2236761054 conn(0x55d13b1dafc0 msgr2=0x55d13b2f4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.782+0000 7f9e60225b80 1 -- 172.21.15.44:0/2236761054 shutdown_connections 2024-05-27T09:49:48.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.782+0000 7f9e60225b80 1 -- 172.21.15.44:0/2236761054 wait complete. 2024-05-27T09:49:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.783+0000 7f9e60225b80 1 -- start start 2024-05-27T09:49:48.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.783+0000 7f9e60225b80 1 Processor -- start 2024-05-27T09:49:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.784+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 0x55d13b3035c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.784+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 0x55d13b303b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.784+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9820 0x55d13b304040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.784+0000 7f9e60225b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d13b11dfa0 con 0x55d13b2f9820 2024-05-27T09:49:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.784+0000 7f9e60225b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d13b1b8760 con 0x55d13b17ae40 2024-05-27T09:49:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.784+0000 7f9e60225b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d13b306930 con 0x55d13b2f9410 2024-05-27T09:49:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.784+0000 7f9e60225b80 10 client.0 _ll_register_callbacks cb 0x55d13b1c1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.784+0000 7f9e4b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 0x55d13b3035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e4a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 0x55d13b303b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e4b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 0x55d13b3035c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39878/0 (socket says 172.21.15.44:39878) 2024-05-27T09:49:48.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e4b400700 1 -- 172.21.15.44:0/3991207522 learned_addr learned my addr 172.21.15.44:0/3991207522 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:48.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e4b400700 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9820 0x55d13b304040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e4b400700 1 -- 172.21.15.44:0/3991207522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 msgr2=0x55d13b303b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e4b400700 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 0x55d13b303b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e4b400700 1 -- 172.21.15.44:0/3991207522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9820 msgr2=0x55d13b304040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e4b400700 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9820 0x55d13b304040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e4b400700 1 -- 172.21.15.44:0/3991207522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d13b1678d0 con 0x55d13b17ae40 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e4a000700 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 0x55d13b303b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e4b400700 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 0x55d13b3035c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400bdd0 tx=0x7f9e44007b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3176b7a30c4a3427 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.785+0000 7f9e34c00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.786+0000 7f9e34c00700 1 -- 172.21.15.44:0/3991207522 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e440077d0 con 0x55d13b17ae40 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.786+0000 7f9e34c00700 1 -- 172.21.15.44:0/3991207522 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f9e44003a00 con 0x55d13b17ae40 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.786+0000 7f9e60225b80 1 -- 172.21.15.44:0/3991207522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e44009460 con 0x55d13b17ae40 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.786+0000 7f9e34c00700 1 -- 172.21.15.44:0/3991207522 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f9e44039050 con 0x55d13b17ae40 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.786+0000 7f9e60225b80 10 client.28019 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.786+0000 7f9e60225b80 1 -- 172.21.15.44:0/3991207522 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55d13b1e8060 con 0x55d13b17ae40 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.786+0000 7f9e34c00700 1 -- 172.21.15.44:0/3991207522 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e44009460 con 0x55d13b17ae40 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.787+0000 7f9e34c00700 1 -- 172.21.15.44:0/3991207522 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f9e44011070 con 0x55d13b17ae40 2024-05-27T09:49:48.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.787+0000 7f9e60225b80 10 client.28019 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:48.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.787+0000 7f9e60225b80 10 client.28019 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T09:49:48.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.787+0000 7f9e60225b80 20 client.28019 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:48.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.787+0000 7f9e60225b80 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55d13b313750 0x55d13b315b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.787+0000 7f9e60225b80 4 client.28019 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:48.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.787+0000 7f9e60225b80 1 -- 172.21.15.44:0/3991207522 --> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d13b229d90 con 0x55d13b313750 2024-05-27T09:49:48.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.787+0000 7f9e4a000700 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55d13b313750 0x55d13b315b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.788+0000 7f9e4a000700 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55d13b313750 0x55d13b315b70 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac95a1a474faa545 server_cookie=b1b0f853ec3d87bd in_seq=0 out_seq=0 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.788+0000 7f9e34c00700 10 client.28019 ms_handle_connect on v2:172.21.15.50:6836/3540954143 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.789+0000 7f9e34c00700 1 -- 172.21.15.44:0/3991207522 <== mds.1 v2:172.21.15.50:6836/3540954143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d13b229d90 con 0x55d13b313750 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.789+0000 7f9e34c00700 10 client.28019 handle_command_reply: tid=0 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.789+0000 7f9e60225b80 1 client.28019 shutdown 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.789+0000 7f9e60225b80 1 -- 172.21.15.44:0/3991207522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 msgr2=0x55d13b3035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.789+0000 7f9e60225b80 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 0x55d13b3035c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400bdd0 tx=0x7f9e44007b00 comp rx=0 tx=0).stop 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.790+0000 7f9e60225b80 1 -- 172.21.15.44:0/3991207522 shutdown_connections 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.790+0000 7f9e60225b80 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9820 0x55d13b304040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.790+0000 7f9e60225b80 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55d13b313750 0x55d13b315b70 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.790+0000 7f9e60225b80 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 0x55d13b3035c0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.790+0000 7f9e60225b80 1 --2- 172.21.15.44:0/3991207522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 0x55d13b303b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.790+0000 7f9e34c00700 0 client.28019 ms_handle_reset on v2:172.21.15.50:6836/3540954143 2024-05-27T09:49:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.790+0000 7f9e60225b80 1 -- 172.21.15.44:0/3991207522 >> 172.21.15.44:0/3991207522 conn(0x55d13b1dafc0 msgr2=0x55d13b2f3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.790+0000 7f9e60225b80 1 -- 172.21.15.44:0/3991207522 shutdown_connections 2024-05-27T09:49:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.790+0000 7f9e60225b80 1 -- 172.21.15.44:0/3991207522 wait complete. 2024-05-27T09:49:48.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.790+0000 7f9e60225b80 20 client.28019 trim_cache size 0 max 16384 2024-05-27T09:49:48.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.807+0000 7f9e60225b80 1 Processor -- start 2024-05-27T09:49:48.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.807+0000 7f9e60225b80 1 -- start start 2024-05-27T09:49:48.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.808+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b17ae40 0x55d13b2fd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.808+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b2f9410 0x55d13b2f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.808+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9820 0x55d13b2fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.808+0000 7f9e60225b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c1b7be0 con 0x55d13b17ae40 2024-05-27T09:49:48.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.808+0000 7f9e60225b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c1b5410 con 0x55d13b2f9410 2024-05-27T09:49:48.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.808+0000 7f9e60225b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c1b3280 con 0x55d13b2f9820 2024-05-27T09:49:48.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.808+0000 7f9e4b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b17ae40 0x55d13b2fd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.808+0000 7f9e4b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b17ae40 0x55d13b2fd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55068/0 (socket says 172.21.15.44:55068) 2024-05-27T09:49:48.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.808+0000 7f9e4b400700 1 -- 172.21.15.44:0/4102410618 learned_addr learned my addr 172.21.15.44:0/4102410618 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:48.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.808+0000 7f9e4a000700 1 --2- 172.21.15.44:0/4102410618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b2f9410 0x55d13b2f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.809+0000 7f9e4b400700 1 --2- 172.21.15.44:0/4102410618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9820 0x55d13b2fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.809+0000 7f9e4b400700 1 -- 172.21.15.44:0/4102410618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9820 msgr2=0x55d13b2fed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:48.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.809+0000 7f9e4b400700 1 --2- 172.21.15.44:0/4102410618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9820 0x55d13b2fed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.809+0000 7f9e4b400700 1 -- 172.21.15.44:0/4102410618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b2f9410 msgr2=0x55d13b2f4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:48.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.809+0000 7f9e4b400700 1 --2- 172.21.15.44:0/4102410618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b2f9410 0x55d13b2f4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.809+0000 7f9e4b400700 1 -- 172.21.15.44:0/4102410618 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d13b2f43b0 con 0x55d13b17ae40 2024-05-27T09:49:48.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.809+0000 7f9e4b400700 1 --2- 172.21.15.44:0/4102410618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b17ae40 0x55d13b2fd7a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400b3f0 tx=0x7f9e4400afe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f873e599200d87b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:48.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.809+0000 7f9e37400700 1 -- 172.21.15.44:0/4102410618 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e440077d0 con 0x55d13b17ae40 2024-05-27T09:49:48.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.809+0000 7f9e37400700 1 -- 172.21.15.44:0/4102410618 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e4402a030 con 0x55d13b17ae40 2024-05-27T09:49:48.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.810+0000 7f9e37400700 1 -- 172.21.15.44:0/4102410618 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e440050e0 con 0x55d13b17ae40 2024-05-27T09:49:48.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.810+0000 7f9e60225b80 1 -- 172.21.15.44:0/4102410618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b17ae40 msgr2=0x55d13b2fd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:48.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.810+0000 7f9e60225b80 1 --2- 172.21.15.44:0/4102410618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b17ae40 0x55d13b2fd7a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400b3f0 tx=0x7f9e4400afe0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.810+0000 7f9e60225b80 1 -- 172.21.15.44:0/4102410618 shutdown_connections 2024-05-27T09:49:48.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.810+0000 7f9e60225b80 1 --2- 172.21.15.44:0/4102410618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9820 0x55d13b2fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.810+0000 7f9e60225b80 1 --2- 172.21.15.44:0/4102410618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b17ae40 0x55d13b2fd7a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.810+0000 7f9e60225b80 1 --2- 172.21.15.44:0/4102410618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b2f9410 0x55d13b2f4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.811+0000 7f9e60225b80 1 -- 172.21.15.44:0/4102410618 >> 172.21.15.44:0/4102410618 conn(0x55d13b1dafc0 msgr2=0x55d13b3033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:48.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.811+0000 7f9e60225b80 1 -- 172.21.15.44:0/4102410618 shutdown_connections 2024-05-27T09:49:48.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.811+0000 7f9e60225b80 1 -- 172.21.15.44:0/4102410618 wait complete. 2024-05-27T09:49:48.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.811+0000 7f9e60225b80 1 -- start start 2024-05-27T09:49:48.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.811+0000 7f9e60225b80 1 Processor -- start 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 0x55d13b315ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 0x55d13b316220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e60225b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9820 0x55d13b37bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e60225b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c1b38d0 con 0x55d13b2f9820 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e60225b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c1b4d40 con 0x55d13b17ae40 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e60225b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c1b3fa0 con 0x55d13b2f9410 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e60225b80 10 client.0 _ll_register_callbacks cb 0x55d13b1c1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e4b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 0x55d13b315ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e4a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 0x55d13b316220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e4a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 0x55d13b316220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51632/0 (socket says 172.21.15.44:51632) 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e4a000700 1 -- 172.21.15.44:0/1123991684 learned_addr learned my addr 172.21.15.44:0/1123991684 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:48.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e4b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 0x55d13b315ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39906/0 (socket says 172.21.15.44:39906) 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.812+0000 7f9e4b400700 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9820 0x55d13b37bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.813+0000 7f9e4a000700 1 -- 172.21.15.44:0/1123991684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 msgr2=0x55d13b315ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.813+0000 7f9e4a000700 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 0x55d13b315ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.813+0000 7f9e4a000700 1 -- 172.21.15.44:0/1123991684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9820 msgr2=0x55d13b37bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.813+0000 7f9e4a000700 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9820 0x55d13b37bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.813+0000 7f9e4a000700 1 -- 172.21.15.44:0/1123991684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d13b316ad0 con 0x55d13b2f9410 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.813+0000 7f9e4a000700 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 0x55d13b316220 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x55d13b2f9cb0 tx=0x7f9e38009770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7052afa6c805b04 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.813+0000 7f9e34c00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.813+0000 7f9e34c00700 1 -- 172.21.15.44:0/1123991684 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e38004050 con 0x55d13b2f9410 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.814+0000 7f9e60225b80 1 -- 172.21.15.44:0/1123991684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e4402a030 con 0x55d13b2f9410 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.814+0000 7f9e34c00700 1 -- 172.21.15.44:0/1123991684 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f9e380041f0 con 0x55d13b2f9410 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.814+0000 7f9e34c00700 1 -- 172.21.15.44:0/1123991684 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f9e3802f050 con 0x55d13b2f9410 2024-05-27T09:49:48.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.814+0000 7f9e60225b80 10 client.28033 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:48.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.814+0000 7f9e60225b80 1 -- 172.21.15.44:0/1123991684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55d13b37d5e0 con 0x55d13b2f9410 2024-05-27T09:49:48.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.814+0000 7f9e34c00700 1 -- 172.21.15.44:0/1123991684 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e4402a030 con 0x55d13b2f9410 2024-05-27T09:49:48.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.814+0000 7f9e34c00700 1 -- 172.21.15.44:0/1123991684 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f9e38003680 con 0x55d13b2f9410 2024-05-27T09:49:48.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.814+0000 7f9e60225b80 10 client.28033 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:48.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.814+0000 7f9e60225b80 10 client.28033 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T09:49:48.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.814+0000 7f9e60225b80 20 client.28033 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:48.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.815+0000 7f9e60225b80 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55d13b37dd30 0x55d13b380150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:48.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.815+0000 7f9e60225b80 4 client.28033 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:49:48.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.815+0000 7f9e60225b80 1 -- 172.21.15.44:0/1123991684 --> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d13b223570 con 0x55d13b37dd30 2024-05-27T09:49:48.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.815+0000 7f9e4b400700 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55d13b37dd30 0x55d13b380150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:48.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.815+0000 7f9e4b400700 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55d13b37dd30 0x55d13b380150 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=626096e8dd2645dc server_cookie=530957ce48a368f7 in_seq=0 out_seq=0 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.815+0000 7f9e34c00700 10 client.28033 ms_handle_connect on v2:172.21.15.50:6836/3540954143 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e34c00700 1 -- 172.21.15.44:0/1123991684 <== mds.1 v2:172.21.15.50:6836/3540954143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d13b223570 con 0x55d13b37dd30 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e34c00700 10 client.28033 handle_command_reply: tid=0 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 1 client.28033 shutdown 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 1 -- 172.21.15.44:0/1123991684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 msgr2=0x55d13b316220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 0x55d13b316220 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x55d13b2f9cb0 tx=0x7f9e38009770 comp rx=0 tx=0).stop 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 1 -- 172.21.15.44:0/1123991684 shutdown_connections 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d13b2f9820 0x55d13b37bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55d13b37dd30 0x55d13b380150 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d13b17ae40 0x55d13b315ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 1 --2- 172.21.15.44:0/1123991684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d13b2f9410 0x55d13b316220 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 1 -- 172.21.15.44:0/1123991684 >> 172.21.15.44:0/1123991684 conn(0x55d13b1dafc0 msgr2=0x55d13b303dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e34c00700 0 client.28033 ms_handle_reset on v2:172.21.15.50:6836/3540954143 2024-05-27T09:49:48.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 1 -- 172.21.15.44:0/1123991684 shutdown_connections 2024-05-27T09:49:48.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 1 -- 172.21.15.44:0/1123991684 wait complete. 2024-05-27T09:49:48.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.816+0000 7f9e60225b80 20 client.28033 trim_cache size 0 max 16384 2024-05-27T09:49:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:49:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.817+0000 7f9e4be00700 1 -- 172.21.15.44:0/351767438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9e300b2a70 msgr2=0x7f9e300b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.817+0000 7f9e4be00700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9e300b2a70 0x7f9e300b4f80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55d13b2f6050 tx=0x7f9e3c0e8530 comp rx=0 tx=0).stop 2024-05-27T09:49:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.817+0000 7f9e4be00700 1 -- 172.21.15.44:0/351767438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e4c077ce0 msgr2=0x7f9e4c1a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.817+0000 7f9e4be00700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e4c077ce0 0x7f9e4c1a1190 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c00e9a0 tx=0x7f9e3c00ecb0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.817+0000 7f9e4be00700 1 -- 172.21.15.44:0/351767438 shutdown_connections 2024-05-27T09:49:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.817+0000 7f9e4be00700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f9e300b2a70 0x7f9e300b4f80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.817+0000 7f9e4be00700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e4c115030 0x7f9e4c1a59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.817+0000 7f9e4be00700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e4c077ce0 0x7f9e4c1a1190 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.817+0000 7f9e4be00700 1 --2- 172.21.15.44:0/351767438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e4c078630 0x7f9e4c1a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.817+0000 7f9e4be00700 1 -- 172.21.15.44:0/351767438 >> 172.21.15.44:0/351767438 conn(0x7f9e4c101ec0 msgr2=0x7f9e4c1042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:48.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.817+0000 7f9e4be00700 1 -- 172.21.15.44:0/351767438 shutdown_connections 2024-05-27T09:49:48.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:48.818+0000 7f9e4be00700 1 -- 172.21.15.44:0/351767438 wait complete. 2024-05-27T09:49:49.536 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '9fa86eb2', total seconds elapsed: 21.9 2024-05-27T09:49:49.961 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:49:50.432 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:49:50.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:50 smithi044 ceph-mon[31618]: pgmap v459: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.1 KiB/s wr, 9 op/s 2024-05-27T09:49:50.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:50 smithi050 ceph-mon[36371]: pgmap v459: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.1 KiB/s wr, 9 op/s 2024-05-27T09:49:50.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:50 smithi189 ceph-mon[36671]: pgmap v459: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.1 KiB/s wr, 9 op/s 2024-05-27T09:49:52.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:52 smithi044 ceph-mon[31618]: pgmap v460: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 98 op/s 2024-05-27T09:49:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:52 smithi050 ceph-mon[36371]: pgmap v460: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 98 op/s 2024-05-27T09:49:52.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:52 smithi189 ceph-mon[36671]: pgmap v460: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 98 op/s 2024-05-27T09:49:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.695+0000 7f48b2e00700 1 -- 172.21.15.44:0/3526833722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac115360 msgr2=0x7f48ac117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.695+0000 7f48b2e00700 1 --2- 172.21.15.44:0/3526833722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac115360 0x7f48ac117780 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f48a400b0a0 tx=0x7f48a400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.695+0000 7f48b2e00700 1 -- 172.21.15.44:0/3526833722 shutdown_connections 2024-05-27T09:49:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.695+0000 7f48b2e00700 1 --2- 172.21.15.44:0/3526833722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac115360 0x7f48ac117780 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.695+0000 7f48b2e00700 1 --2- 172.21.15.44:0/3526833722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48ac075b50 0x7f48ac075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.695+0000 7f48b2e00700 1 --2- 172.21.15.44:0/3526833722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac0764a0 0x7f48ac114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.695+0000 7f48b2e00700 1 -- 172.21.15.44:0/3526833722 >> 172.21.15.44:0/3526833722 conn(0x7f48ac100030 msgr2=0x7f48ac102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.695+0000 7f48b2e00700 1 -- 172.21.15.44:0/3526833722 shutdown_connections 2024-05-27T09:49:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.695+0000 7f48b2e00700 1 -- 172.21.15.44:0/3526833722 wait complete. 2024-05-27T09:49:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.696+0000 7f48b2e00700 1 Processor -- start 2024-05-27T09:49:52.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.697+0000 7f48b2e00700 1 -- start start 2024-05-27T09:49:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.697+0000 7f48b2e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48ac075b50 0x7f48ac1a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.697+0000 7f48b2e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac0764a0 0x7f48ac1a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48b2e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac115360 0x7f48ac1a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48b2e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac11aa60 con 0x7f48ac115360 2024-05-27T09:49:52.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48b2e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac11a8e0 con 0x7f48ac075b50 2024-05-27T09:49:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48b2e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac11abe0 con 0x7f48ac0764a0 2024-05-27T09:49:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48abe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48ac075b50 0x7f48ac1a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48abe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48ac075b50 0x7f48ac1a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:39924/0 (socket says 172.21.15.44:39924) 2024-05-27T09:49:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48abe00700 1 -- 172.21.15.44:0/1086709224 learned_addr learned my addr 172.21.15.44:0/1086709224 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48ab400700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac0764a0 0x7f48ac1a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48b0e00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac115360 0x7f48ac1a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48abe00700 1 -- 172.21.15.44:0/1086709224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac0764a0 msgr2=0x7f48ac1a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48abe00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac0764a0 0x7f48ac1a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48abe00700 1 -- 172.21.15.44:0/1086709224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac115360 msgr2=0x7f48ac1a5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48abe00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac115360 0x7f48ac1a5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.698+0000 7f48abe00700 1 -- 172.21.15.44:0/1086709224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ac1a6200 con 0x7f48ac075b50 2024-05-27T09:49:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.699+0000 7f48abe00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48ac075b50 0x7f48ac1a1330 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f489c00e9a0 tx=0x7f489c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.699+0000 7f48a8c00700 1 -- 172.21.15.44:0/1086709224 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c00ca10 con 0x7f48ac075b50 2024-05-27T09:49:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.699+0000 7f48a8c00700 1 -- 172.21.15.44:0/1086709224 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f489c00cbb0 con 0x7f48ac075b50 2024-05-27T09:49:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.699+0000 7f48b2e00700 1 -- 172.21.15.44:0/1086709224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48ac1a64f0 con 0x7f48ac075b50 2024-05-27T09:49:52.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.699+0000 7f48a8c00700 1 -- 172.21.15.44:0/1086709224 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c018680 con 0x7f48ac075b50 2024-05-27T09:49:52.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.700+0000 7f48b2e00700 1 -- 172.21.15.44:0/1086709224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48ac1ade10 con 0x7f48ac075b50 2024-05-27T09:49:52.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.703+0000 7f48a8c00700 1 -- 172.21.15.44:0/1086709224 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f489c010430 con 0x7f48ac075b50 2024-05-27T09:49:52.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.705+0000 7f48c1991b80 1 Processor -- start 2024-05-27T09:49:52.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.705+0000 7f48c1991b80 1 -- start start 2024-05-27T09:49:52.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8d87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8d2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 0x55e56f8d31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48c1991b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e56f800e40 con 0x55e56f8d2db0 2024-05-27T09:49:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48c1991b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e56f7f09e0 con 0x55e56f8d8d30 2024-05-27T09:49:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48c1991b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e56f7f4a20 con 0x55e56f753e40 2024-05-27T09:49:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48a8c00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f48900b29f0 0x7f48900b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48b0e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8d87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48a8c00700 1 -- 172.21.15.44:0/1086709224 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f489c0e2820 con 0x7f48ac075b50 2024-05-27T09:49:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48abe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48b0e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8d87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51646/0 (socket says 172.21.15.44:51646) 2024-05-27T09:49:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.706+0000 7f48b0e00700 1 -- 172.21.15.44:0/1989953627 learned_addr learned my addr 172.21.15.44:0/1989953627 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.707+0000 7f48b0e00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f48900b29f0 0x7f48900b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.707+0000 7f48ab400700 1 --2- 172.21.15.44:0/1989953627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 0x55e56f8d31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.707+0000 7f48abe00700 1 -- 172.21.15.44:0/1989953627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 msgr2=0x55e56f8d87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.707+0000 7f48abe00700 1 --2- 172.21.15.44:0/1989953627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8d87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.707+0000 7f48abe00700 1 -- 172.21.15.44:0/1989953627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 msgr2=0x55e56f8d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.707+0000 7f48abe00700 1 --2- 172.21.15.44:0/1989953627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 0x55e56f8d31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.707+0000 7f48abe00700 1 -- 172.21.15.44:0/1989953627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e56f7b9420 con 0x55e56f8d8d30 2024-05-27T09:49:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.708+0000 7f48abe00700 1 --2- 172.21.15.44:0/1989953627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8d2870 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f489c0dfec0 tx=0x7f489c0b2270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab78154e83a01144 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.708+0000 7f48b0e00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f48900b29f0 0x7f48900b4f00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009900 tx=0x7f48a400b720 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:52.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.708+0000 7f48a1600700 1 -- 172.21.15.44:0/1989953627 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c0a5050 con 0x55e56f8d8d30 2024-05-27T09:49:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.708+0000 7f48a1600700 1 -- 172.21.15.44:0/1989953627 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f489c0b2040 con 0x55e56f8d8d30 2024-05-27T09:49:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.708+0000 7f48a1600700 1 -- 172.21.15.44:0/1989953627 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c0b2420 con 0x55e56f8d8d30 2024-05-27T09:49:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.709+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989953627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 msgr2=0x55e56f8d2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.709+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1989953627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8d2870 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f489c0dfec0 tx=0x7f489c0b2270 comp rx=0 tx=0).stop 2024-05-27T09:49:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.709+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989953627 shutdown_connections 2024-05-27T09:49:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.709+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1989953627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 0x55e56f8d31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.709+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1989953627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8d87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.709+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1989953627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8d2870 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.709+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989953627 >> 172.21.15.44:0/1989953627 conn(0x55e56f7b3fc0 msgr2=0x55e56f8cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.709+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989953627 shutdown_connections 2024-05-27T09:49:52.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.709+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989953627 wait complete. 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.710+0000 7f48c1991b80 1 -- start start 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.710+0000 7f48c1991b80 1 Processor -- start 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.710+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8dc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.710+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 0x55e56f8dcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8dd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48c1991b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e56f6f6fa0 con 0x55e56f8d2db0 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48c1991b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e56f791760 con 0x55e56f8d8d30 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48c1991b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e56f8df7a0 con 0x55e56f753e40 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48c1991b80 10 client.0 _ll_register_callbacks cb 0x55e56f79ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48ab400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 0x55e56f8dcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48ab400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 0x55e56f8dcad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55116/0 (socket says 172.21.15.44:55116) 2024-05-27T09:49:52.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48ab400700 1 -- 172.21.15.44:0/1989550735 learned_addr learned my addr 172.21.15.44:0/1989550735 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48abe00700 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8dc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48b0e00700 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8dd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48ab400700 1 -- 172.21.15.44:0/1989550735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 msgr2=0x55e56f8dc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48ab400700 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8dc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48ab400700 1 -- 172.21.15.44:0/1989550735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 msgr2=0x55e56f8dd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48ab400700 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8dd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48ab400700 1 -- 172.21.15.44:0/1989550735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e56f7408d0 con 0x55e56f8d2db0 2024-05-27T09:49:52.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.711+0000 7f48abe00700 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8dc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.712+0000 7f48ab400700 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 0x55e56f8dcad0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f48980127a0 tx=0x7f4898012ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5a13a6c818d725 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.712+0000 7f4896000700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:49:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.712+0000 7f4896000700 1 -- 172.21.15.44:0/1989550735 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489800f480 con 0x55e56f8d2db0 2024-05-27T09:49:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.712+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989550735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f489c0b2040 con 0x55e56f8d2db0 2024-05-27T09:49:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.712+0000 7f4896000700 1 -- 172.21.15.44:0/1989550735 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f489800f620 con 0x55e56f8d2db0 2024-05-27T09:49:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.712+0000 7f4896000700 1 -- 172.21.15.44:0/1989550735 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f489800f820 con 0x55e56f8d2db0 2024-05-27T09:49:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.712+0000 7f48c1991b80 10 client.17769 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.712+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989550735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e56f7c1060 con 0x55e56f8d2db0 2024-05-27T09:49:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.713+0000 7f4896000700 1 -- 172.21.15.44:0/1989550735 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f489c0b2040 con 0x55e56f8d2db0 2024-05-27T09:49:52.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.713+0000 7f4896000700 1 -- 172.21.15.44:0/1989550735 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f4898010a00 con 0x55e56f8d2db0 2024-05-27T09:49:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.713+0000 7f48c1991b80 10 client.17769 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.713+0000 7f48c1991b80 10 client.17769 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:49:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.713+0000 7f48c1991b80 20 client.17769 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.713+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e56f8ec7e0 0x55e56f8eec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.713+0000 7f48c1991b80 4 client.17769 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.713+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989550735 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e56f802d90 con 0x55e56f8ec7e0 2024-05-27T09:49:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.714+0000 7f48abe00700 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e56f8ec7e0 0x55e56f8eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.714+0000 7f48abe00700 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e56f8ec7e0 0x55e56f8eec00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=23735cd2c7d93402 server_cookie=c68581ec3416ae57 in_seq=0 out_seq=0 2024-05-27T09:49:52.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.714+0000 7f4896000700 10 client.17769 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:49:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.715+0000 7f4896000700 1 -- 172.21.15.44:0/1989550735 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e56f802d90 con 0x55e56f8ec7e0 2024-05-27T09:49:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.715+0000 7f4896000700 10 client.17769 handle_command_reply: tid=0 2024-05-27T09:49:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.715+0000 7f48c1991b80 1 client.17769 shutdown 2024-05-27T09:49:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.715+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989550735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 msgr2=0x55e56f8dcad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.715+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 0x55e56f8dcad0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f48980127a0 tx=0x7f4898012ab0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.716+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989550735 shutdown_connections 2024-05-27T09:49:52.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.716+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8dd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.716+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8dc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.716+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e56f8ec7e0 0x55e56f8eec00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.716+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1989550735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8d2db0 0x55e56f8dcad0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.716+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989550735 >> 172.21.15.44:0/1989550735 conn(0x55e56f7b3fc0 msgr2=0x55e56f8cd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:52.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.716+0000 7f4896000700 0 client.17769 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:49:52.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.716+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989550735 shutdown_connections 2024-05-27T09:49:52.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.716+0000 7f48c1991b80 1 -- 172.21.15.44:0/1989550735 wait complete. 2024-05-27T09:49:52.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.716+0000 7f48c1991b80 20 client.17769 trim_cache size 0 max 16384 2024-05-27T09:49:52.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.732+0000 7f48c1991b80 1 Processor -- start 2024-05-27T09:49:52.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.733+0000 7f48c1991b80 1 -- start start 2024-05-27T09:49:52.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.733+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f753e40 0x55e56f8d2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.733+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f8d2db0 0x55e56f8db4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.733+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8dea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.733+0000 7f48c1991b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac1b7e70 con 0x55e56f753e40 2024-05-27T09:49:52.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.733+0000 7f48c1991b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac1b56a0 con 0x55e56f8d8d30 2024-05-27T09:49:52.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.733+0000 7f48c1991b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac1b3510 con 0x55e56f8d2db0 2024-05-27T09:49:52.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.733+0000 7f48abe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f753e40 0x55e56f8d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.733+0000 7f48abe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f753e40 0x55e56f8d2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55132/0 (socket says 172.21.15.44:55132) 2024-05-27T09:49:52.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.733+0000 7f48abe00700 1 -- 172.21.15.44:0/1737229110 learned_addr learned my addr 172.21.15.44:0/1737229110 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:52.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.733+0000 7f48ab400700 1 --2- 172.21.15.44:0/1737229110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f8d2db0 0x55e56f8db4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.734+0000 7f48b0e00700 1 --2- 172.21.15.44:0/1737229110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8dea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.734+0000 7f48ab400700 1 -- 172.21.15.44:0/1737229110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 msgr2=0x55e56f8dea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:52.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.734+0000 7f48ab400700 1 --2- 172.21.15.44:0/1737229110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8dea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.734+0000 7f48ab400700 1 -- 172.21.15.44:0/1737229110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f753e40 msgr2=0x55e56f8d2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:52.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.734+0000 7f48ab400700 1 --2- 172.21.15.44:0/1737229110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f753e40 0x55e56f8d2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.734+0000 7f48ab400700 1 -- 172.21.15.44:0/1737229110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e56f8dab00 con 0x55e56f8d2db0 2024-05-27T09:49:52.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.734+0000 7f48abe00700 1 --2- 172.21.15.44:0/1737229110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f753e40 0x55e56f8d2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:52.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.734+0000 7f48ab400700 1 --2- 172.21.15.44:0/1737229110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f8d2db0 0x55e56f8db4f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4898010ea0 tx=0x7f489800f630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d36aa8aeb129774a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:52.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.734+0000 7f48a0c00700 1 -- 172.21.15.44:0/1737229110 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48980122e0 con 0x55e56f8d2db0 2024-05-27T09:49:52.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.735+0000 7f48a0c00700 1 -- 172.21.15.44:0/1737229110 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f489800fe70 con 0x55e56f8d2db0 2024-05-27T09:49:52.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.735+0000 7f48a0c00700 1 -- 172.21.15.44:0/1737229110 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4898003970 con 0x55e56f8d2db0 2024-05-27T09:49:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.735+0000 7f48c1991b80 1 -- 172.21.15.44:0/1737229110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f8d2db0 msgr2=0x55e56f8db4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.735+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1737229110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f8d2db0 0x55e56f8db4f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4898010ea0 tx=0x7f489800f630 comp rx=0 tx=0).stop 2024-05-27T09:49:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.735+0000 7f48c1991b80 1 -- 172.21.15.44:0/1737229110 shutdown_connections 2024-05-27T09:49:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.735+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1737229110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8dea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.735+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1737229110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f753e40 0x55e56f8d2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.735+0000 7f48c1991b80 1 --2- 172.21.15.44:0/1737229110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f8d2db0 0x55e56f8db4f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.735+0000 7f48c1991b80 1 -- 172.21.15.44:0/1737229110 >> 172.21.15.44:0/1737229110 conn(0x55e56f7b3fc0 msgr2=0x55e56f8dc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.736+0000 7f48c1991b80 1 -- 172.21.15.44:0/1737229110 shutdown_connections 2024-05-27T09:49:52.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.736+0000 7f48c1991b80 1 -- 172.21.15.44:0/1737229110 wait complete. 2024-05-27T09:49:52.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.736+0000 7f48c1991b80 1 -- start start 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.736+0000 7f48c1991b80 1 Processor -- start 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8e4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8e5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48c1991b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8f3550 0x55e56f8f3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48c1991b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac1b3b60 con 0x55e56f8f3550 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48c1991b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac1b4fd0 con 0x55e56f8d8d30 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48c1991b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac1b4230 con 0x55e56f753e40 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48c1991b80 10 client.0 _ll_register_callbacks cb 0x55e56f79ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48abe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8e4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48abe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8e4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:51676/0 (socket says 172.21.15.44:51676) 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48abe00700 1 -- 172.21.15.44:0/489540624 learned_addr learned my addr 172.21.15.44:0/489540624 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:52.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48b0e00700 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8f3550 0x55e56f8f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.737+0000 7f48ab400700 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8e5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.738+0000 7f48abe00700 1 -- 172.21.15.44:0/489540624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 msgr2=0x55e56f8e5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:52.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.738+0000 7f48abe00700 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8e5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.738+0000 7f48abe00700 1 -- 172.21.15.44:0/489540624 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8f3550 msgr2=0x55e56f8f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:52.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.738+0000 7f48abe00700 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8f3550 0x55e56f8f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.738+0000 7f48abe00700 1 -- 172.21.15.44:0/489540624 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e56f8f3fd0 con 0x55e56f753e40 2024-05-27T09:49:52.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.738+0000 7f48abe00700 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8e4bf0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f489c0a5ad0 tx=0x7f489c00c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ee21d2f824e1c56 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:52.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.738+0000 7f4896000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:49:52.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.738+0000 7f4896000700 1 -- 172.21.15.44:0/489540624 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c0c3060 con 0x55e56f753e40 2024-05-27T09:49:52.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.738+0000 7f4896000700 1 -- 172.21.15.44:0/489540624 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f489c0b2630 con 0x55e56f753e40 2024-05-27T09:49:52.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.738+0000 7f48c1991b80 1 -- 172.21.15.44:0/489540624 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f489800fe70 con 0x55e56f753e40 2024-05-27T09:49:52.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.739+0000 7f4896000700 1 -- 172.21.15.44:0/489540624 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f489c0aa150 con 0x55e56f753e40 2024-05-27T09:49:52.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.739+0000 7f48c1991b80 10 client.28051 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:52.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.739+0000 7f48c1991b80 1 -- 172.21.15.44:0/489540624 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e56f956f40 con 0x55e56f753e40 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.739+0000 7f4896000700 1 -- 172.21.15.44:0/489540624 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f489800fe70 con 0x55e56f753e40 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.739+0000 7f4896000700 1 -- 172.21.15.44:0/489540624 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f489c06f1a0 con 0x55e56f753e40 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.739+0000 7f48c1991b80 10 client.28051 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.739+0000 7f48c1991b80 10 client.28051 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.739+0000 7f48c1991b80 20 client.28051 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.739+0000 7f48c1991b80 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e56f9571d0 0x55e56f9595f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.739+0000 7f48c1991b80 4 client.28051 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.739+0000 7f48c1991b80 1 -- 172.21.15.44:0/489540624 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e56f7fc570 con 0x55e56f9571d0 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.740+0000 7f48ab400700 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e56f9571d0 0x55e56f9595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.740+0000 7f48ab400700 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e56f9571d0 0x55e56f9595f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=37c4e7f1e19f2f0 server_cookie=d3c111ac01dd896a in_seq=0 out_seq=0 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.740+0000 7f4896000700 10 client.28051 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f4896000700 1 -- 172.21.15.44:0/489540624 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e56f7fc570 con 0x55e56f9571d0 2024-05-27T09:49:52.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f4896000700 10 client.28051 handle_command_reply: tid=0 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 1 client.28051 shutdown 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 1 -- 172.21.15.44:0/489540624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 msgr2=0x55e56f8e4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8e4bf0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f489c0a5ad0 tx=0x7f489c00c400 comp rx=0 tx=0).stop 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 1 -- 172.21.15.44:0/489540624 shutdown_connections 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55e56f9571d0 0x55e56f9595f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e56f8f3550 0x55e56f8f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e56f753e40 0x55e56f8e4bf0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f4896000700 0 client.28051 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 1 --2- 172.21.15.44:0/489540624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e56f8d8d30 0x55e56f8e5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 1 -- 172.21.15.44:0/489540624 >> 172.21.15.44:0/489540624 conn(0x55e56f7b3fc0 msgr2=0x55e56f8dc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 1 -- 172.21.15.44:0/489540624 shutdown_connections 2024-05-27T09:49:52.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 1 -- 172.21.15.44:0/489540624 wait complete. 2024-05-27T09:49:52.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.741+0000 7f48c1991b80 20 client.28051 trim_cache size 0 max 16384 2024-05-27T09:49:52.745 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:49:52.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 -- 172.21.15.44:0/1086709224 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f48900b29f0 msgr2=0x7f48900b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:52.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f48900b29f0 0x7f48900b4f00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009900 tx=0x7f48a400b720 comp rx=0 tx=0).stop 2024-05-27T09:49:52.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 -- 172.21.15.44:0/1086709224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48ac075b50 msgr2=0x7f48ac1a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:52.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48ac075b50 0x7f48ac1a1330 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f489c00e9a0 tx=0x7f489c00ecb0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 -- 172.21.15.44:0/1086709224 shutdown_connections 2024-05-27T09:49:52.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f48900b29f0 0x7f48900b4f00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48ac115360 0x7f48ac1a5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48ac075b50 0x7f48ac1a1330 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 --2- 172.21.15.44:0/1086709224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac0764a0 0x7f48ac1a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:52.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 -- 172.21.15.44:0/1086709224 >> 172.21.15.44:0/1086709224 conn(0x7f48ac100030 msgr2=0x7f48ac1005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:52.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 -- 172.21.15.44:0/1086709224 shutdown_connections 2024-05-27T09:49:52.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:52.742+0000 7f48b2e00700 1 -- 172.21.15.44:0/1086709224 wait complete. 2024-05-27T09:49:53.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:53 smithi044 ceph-mon[31618]: pgmap v461: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 95 op/s 2024-05-27T09:49:53.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:53 smithi050 ceph-mon[36371]: pgmap v461: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 95 op/s 2024-05-27T09:49:53.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:53 smithi189 ceph-mon[36671]: pgmap v461: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 95 op/s 2024-05-27T09:49:53.597 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:49:54.082 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:49:56.072 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:55 smithi044 ceph-mon[31618]: pgmap v462: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 95 op/s 2024-05-27T09:49:56.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:55 smithi050 ceph-mon[36371]: pgmap v462: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 95 op/s 2024-05-27T09:49:56.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.319+0000 7f5b55a00700 1 -- 172.21.15.44:0/3337370594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b50115360 msgr2=0x7f5b50117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:56.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.319+0000 7f5b55a00700 1 --2- 172.21.15.44:0/3337370594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b50115360 0x7f5b50117780 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f5b480098f0 tx=0x7f5b48009c00 comp rx=0 tx=0).stop 2024-05-27T09:49:56.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.319+0000 7f5b55a00700 1 -- 172.21.15.44:0/3337370594 shutdown_connections 2024-05-27T09:49:56.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.319+0000 7f5b55a00700 1 --2- 172.21.15.44:0/3337370594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b50115360 0x7f5b50117780 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.319+0000 7f5b55a00700 1 --2- 172.21.15.44:0/3337370594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b50075b50 0x7f5b50075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.319+0000 7f5b55a00700 1 --2- 172.21.15.44:0/3337370594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b500764a0 0x7f5b50114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.319+0000 7f5b55a00700 1 -- 172.21.15.44:0/3337370594 >> 172.21.15.44:0/3337370594 conn(0x7f5b50100030 msgr2=0x7f5b50102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:56.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.319+0000 7f5b55a00700 1 -- 172.21.15.44:0/3337370594 shutdown_connections 2024-05-27T09:49:56.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.319+0000 7f5b55a00700 1 -- 172.21.15.44:0/3337370594 wait complete. 2024-05-27T09:49:56.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.320+0000 7f5b55a00700 1 Processor -- start 2024-05-27T09:49:56.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.321+0000 7f5b55a00700 1 -- start start 2024-05-27T09:49:56.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.321+0000 7f5b55a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b50075b50 0x7f5b501a1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.321+0000 7f5b55a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b500764a0 0x7f5b501a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.321+0000 7f5b55a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b50115360 0x7f5b501a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.321+0000 7f5b55a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5011aac0 con 0x7f5b500764a0 2024-05-27T09:49:56.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.321+0000 7f5b55a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5011a940 con 0x7f5b50075b50 2024-05-27T09:49:56.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.321+0000 7f5b55a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5011ac40 con 0x7f5b50115360 2024-05-27T09:49:56.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.321+0000 7f5b46000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b500764a0 0x7f5b501a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.322+0000 7f5b4ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b50075b50 0x7f5b501a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.322+0000 7f5b46000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b500764a0 0x7f5b501a19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37654/0 (socket says 172.21.15.44:37654) 2024-05-27T09:49:56.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.322+0000 7f5b46000700 1 -- 172.21.15.44:0/4024200259 learned_addr learned my addr 172.21.15.44:0/4024200259 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:56.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.322+0000 7f5b4f400700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b50115360 0x7f5b501a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.322+0000 7f5b46000700 1 -- 172.21.15.44:0/4024200259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b50115360 msgr2=0x7f5b501a5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:56.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.322+0000 7f5b46000700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b50115360 0x7f5b501a5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.322+0000 7f5b46000700 1 -- 172.21.15.44:0/4024200259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b50075b50 msgr2=0x7f5b501a1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:56.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.322+0000 7f5b46000700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b50075b50 0x7f5b501a1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.322+0000 7f5b46000700 1 -- 172.21.15.44:0/4024200259 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b501a6320 con 0x7f5b500764a0 2024-05-27T09:49:56.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.323+0000 7f5b46000700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b500764a0 0x7f5b501a19f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800ef10 tx=0x7f5b3800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:56.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.323+0000 7f5b47e00700 1 -- 172.21.15.44:0/4024200259 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3800c9c0 con 0x7f5b500764a0 2024-05-27T09:49:56.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.323+0000 7f5b55a00700 1 -- 172.21.15.44:0/4024200259 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b501a65b0 con 0x7f5b500764a0 2024-05-27T09:49:56.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.323+0000 7f5b47e00700 1 -- 172.21.15.44:0/4024200259 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b38019070 con 0x7f5b500764a0 2024-05-27T09:49:56.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.323+0000 7f5b47e00700 1 -- 172.21.15.44:0/4024200259 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38014480 con 0x7f5b500764a0 2024-05-27T09:49:56.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.323+0000 7f5b55a00700 1 -- 172.21.15.44:0/4024200259 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b501adf30 con 0x7f5b500764a0 2024-05-27T09:49:56.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.328+0000 7f5b47e00700 1 -- 172.21.15.44:0/4024200259 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5b38012050 con 0x7f5b500764a0 2024-05-27T09:49:56.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.329+0000 7f5b644adb80 1 Processor -- start 2024-05-27T09:49:56.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.329+0000 7f5b644adb80 1 -- start start 2024-05-27T09:49:56.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 0x563b494df330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b49360e40 0x563b494df870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b494dfe70 0x563b494ec110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b644adb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563b4940de40 con 0x563b49360e40 2024-05-27T09:49:56.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b644adb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563b493fd9e0 con 0x563b494dfe70 2024-05-27T09:49:56.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b644adb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563b49401a20 con 0x563b4944c9f0 2024-05-27T09:49:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b4ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b49360e40 0x563b494df870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b4f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 0x563b494df330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b4ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b49360e40 0x563b494df870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37666/0 (socket says 172.21.15.44:37666) 2024-05-27T09:49:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b4ea00700 1 -- 172.21.15.44:0/1774822031 learned_addr learned my addr 172.21.15.44:0/1774822031 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b4f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 0x563b494df330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35458/0 (socket says 172.21.15.44:35458) 2024-05-27T09:49:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b47e00700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b340b2a70 0x7f5b340b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.330+0000 7f5b46000700 1 --2- 172.21.15.44:0/1774822031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b494dfe70 0x563b494ec110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.331+0000 7f5b47e00700 1 -- 172.21.15.44:0/4024200259 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f5b380e2510 con 0x7f5b500764a0 2024-05-27T09:49:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.331+0000 7f5b4f400700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b340b2a70 0x7f5b340b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.331+0000 7f5b46000700 1 -- 172.21.15.44:0/1774822031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 msgr2=0x563b494df330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:56.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.331+0000 7f5b46000700 1 --2- 172.21.15.44:0/1774822031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 0x563b494df330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.331+0000 7f5b46000700 1 -- 172.21.15.44:0/1774822031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b49360e40 msgr2=0x563b494df870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:56.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.331+0000 7f5b46000700 1 --2- 172.21.15.44:0/1774822031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b49360e40 0x563b494df870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.331+0000 7f5b46000700 1 -- 172.21.15.44:0/1774822031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b493c6420 con 0x563b494dfe70 2024-05-27T09:49:56.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.331+0000 7f5b4ea00700 1 --2- 172.21.15.44:0/1774822031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b49360e40 0x563b494df870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:56.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.332+0000 7f5b46000700 1 --2- 172.21.15.44:0/1774822031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b494dfe70 0x563b494ec110 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5b380dfbb0 tx=0x7f5b38077310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9ce265653387659 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:56.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.332+0000 7f5b4f400700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b340b2a70 0x7f5b340b4f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009fd0 tx=0x7f5b48009c00 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:56.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.332+0000 7f5b3fe00700 1 -- 172.21.15.44:0/1774822031 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b380a3e60 con 0x563b494dfe70 2024-05-27T09:49:56.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.332+0000 7f5b3fe00700 1 -- 172.21.15.44:0/1774822031 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b38077160 con 0x563b494dfe70 2024-05-27T09:49:56.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.332+0000 7f5b3fe00700 1 -- 172.21.15.44:0/1774822031 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3806ef00 con 0x563b494dfe70 2024-05-27T09:49:56.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.333+0000 7f5b644adb80 1 -- 172.21.15.44:0/1774822031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b494dfe70 msgr2=0x563b494ec110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:56.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.333+0000 7f5b644adb80 1 --2- 172.21.15.44:0/1774822031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b494dfe70 0x563b494ec110 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5b380dfbb0 tx=0x7f5b38077310 comp rx=0 tx=0).stop 2024-05-27T09:49:56.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.333+0000 7f5b644adb80 1 -- 172.21.15.44:0/1774822031 shutdown_connections 2024-05-27T09:49:56.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.333+0000 7f5b644adb80 1 --2- 172.21.15.44:0/1774822031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b494dfe70 0x563b494ec110 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.333+0000 7f5b644adb80 1 --2- 172.21.15.44:0/1774822031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 0x563b494df330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.333+0000 7f5b644adb80 1 --2- 172.21.15.44:0/1774822031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b49360e40 0x563b494df870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.333+0000 7f5b644adb80 1 -- 172.21.15.44:0/1774822031 >> 172.21.15.44:0/1774822031 conn(0x563b493fa390 msgr2=0x563b494db4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:56.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.333+0000 7f5b644adb80 1 -- 172.21.15.44:0/1774822031 shutdown_connections 2024-05-27T09:49:56.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.333+0000 7f5b644adb80 1 -- 172.21.15.44:0/1774822031 wait complete. 2024-05-27T09:49:56.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.334+0000 7f5b644adb80 1 -- start start 2024-05-27T09:49:56.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.334+0000 7f5b644adb80 1 Processor -- start 2024-05-27T09:49:56.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 0x563b494e96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 0x563b494e9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b494dfe70 0x563b494ea120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b644adb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563b49303fa0 con 0x563b494dfe70 2024-05-27T09:49:56.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b644adb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563b4939e760 con 0x563b49360e40 2024-05-27T09:49:56.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b644adb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563b494ecb20 con 0x563b4944c9f0 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b644adb80 10 client.0 _ll_register_callbacks cb 0x563b493a7c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b46000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 0x563b494e9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b4ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 0x563b494e96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b46000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 0x563b494e9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35470/0 (socket says 172.21.15.44:35470) 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b4f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b494dfe70 0x563b494ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.335+0000 7f5b46000700 1 -- 172.21.15.44:0/83679534 learned_addr learned my addr 172.21.15.44:0/83679534 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b46000700 1 -- 172.21.15.44:0/83679534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 msgr2=0x563b494e96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b46000700 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 0x563b494e96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b46000700 1 -- 172.21.15.44:0/83679534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b494dfe70 msgr2=0x563b494ea120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b46000700 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b494dfe70 0x563b494ea120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b46000700 1 -- 172.21.15.44:0/83679534 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b4934d8d0 con 0x563b4944c9f0 2024-05-27T09:49:56.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b4ea00700 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 0x563b494e96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:56.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:55 smithi189 ceph-mon[36671]: pgmap v462: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 95 op/s 2024-05-27T09:49:56.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b46000700 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 0x563b494e9be0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800cfd0 tx=0x7f5b380b2340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c095c12a4d31760 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:56.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b3cc00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:49:56.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b3cc00700 1 -- 172.21.15.44:0/83679534 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b380a3e60 con 0x563b4944c9f0 2024-05-27T09:49:56.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b644adb80 1 -- 172.21.15.44:0/83679534 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b38077160 con 0x563b4944c9f0 2024-05-27T09:49:56.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b3cc00700 1 -- 172.21.15.44:0/83679534 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f5b380a89a0 con 0x563b4944c9f0 2024-05-27T09:49:56.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.336+0000 7f5b3cc00700 1 -- 172.21.15.44:0/83679534 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f5b380912f0 con 0x563b4944c9f0 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.337+0000 7f5b644adb80 10 client.28063 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.337+0000 7f5b644adb80 1 -- 172.21.15.44:0/83679534 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x563b493ce060 con 0x563b4944c9f0 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.337+0000 7f5b3cc00700 1 -- 172.21.15.44:0/83679534 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b38077160 con 0x563b4944c9f0 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.337+0000 7f5b3cc00700 1 -- 172.21.15.44:0/83679534 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f5b38004ad0 con 0x563b4944c9f0 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.337+0000 7f5b644adb80 10 client.28063 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.337+0000 7f5b644adb80 10 client.28063 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.337+0000 7f5b644adb80 20 client.28063 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.337+0000 7f5b644adb80 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b494f9950 0x563b494fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.337+0000 7f5b644adb80 4 client.28063 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.337+0000 7f5b644adb80 1 -- 172.21.15.44:0/83679534 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b4940fd90 con 0x563b494f9950 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.338+0000 7f5b4ea00700 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b494f9950 0x563b494fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.338+0000 7f5b4ea00700 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b494f9950 0x563b494fbd70 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65aebd65cf6f6062 server_cookie=20aff6937e0b65f7 in_seq=0 out_seq=0 2024-05-27T09:49:56.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.338+0000 7f5b3cc00700 10 client.28063 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.339+0000 7f5b3cc00700 1 -- 172.21.15.44:0/83679534 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563b4940fd90 con 0x563b494f9950 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.339+0000 7f5b3cc00700 10 client.28063 handle_command_reply: tid=0 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.339+0000 7f5b644adb80 1 client.28063 shutdown 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.339+0000 7f5b644adb80 1 -- 172.21.15.44:0/83679534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 msgr2=0x563b494e9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.339+0000 7f5b644adb80 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 0x563b494e9be0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800cfd0 tx=0x7f5b380b2340 comp rx=0 tx=0).stop 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.340+0000 7f5b644adb80 1 -- 172.21.15.44:0/83679534 shutdown_connections 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.340+0000 7f5b644adb80 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b494f9950 0x563b494fbd70 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.340+0000 7f5b644adb80 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b494dfe70 0x563b494ea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.340+0000 7f5b644adb80 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 0x563b494e96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.340+0000 7f5b644adb80 1 --2- 172.21.15.44:0/83679534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b4944c9f0 0x563b494e9be0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.340+0000 7f5b3cc00700 0 client.28063 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.340+0000 7f5b644adb80 1 -- 172.21.15.44:0/83679534 >> 172.21.15.44:0/83679534 conn(0x563b493fa390 msgr2=0x563b494df310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:56.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.340+0000 7f5b644adb80 1 -- 172.21.15.44:0/83679534 shutdown_connections 2024-05-27T09:49:56.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.340+0000 7f5b644adb80 1 -- 172.21.15.44:0/83679534 wait complete. 2024-05-27T09:49:56.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.340+0000 7f5b644adb80 20 client.28063 trim_cache size 0 max 16384 2024-05-27T09:49:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.357+0000 7f5b644adb80 1 Processor -- start 2024-05-27T09:49:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.357+0000 7f5b644adb80 1 -- start start 2024-05-27T09:49:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.357+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b49360e40 0x563b494e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.357+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b4944c9f0 0x563b494df7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.357+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b494dfe70 0x563b494e2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.357+0000 7f5b644adb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b501b7f60 con 0x563b494dfe70 2024-05-27T09:49:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.357+0000 7f5b644adb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b501b5790 con 0x563b4944c9f0 2024-05-27T09:49:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.357+0000 7f5b644adb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b501b3600 con 0x563b49360e40 2024-05-27T09:49:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.358+0000 7f5b46000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b4944c9f0 0x563b494df7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.358+0000 7f5b46000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b4944c9f0 0x563b494df7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42422/0 (socket says 172.21.15.44:42422) 2024-05-27T09:49:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.358+0000 7f5b4f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b494dfe70 0x563b494e2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.358+0000 7f5b46000700 1 -- 172.21.15.44:0/1059313954 learned_addr learned my addr 172.21.15.44:0/1059313954 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.358+0000 7f5b4ea00700 1 --2- 172.21.15.44:0/1059313954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b49360e40 0x563b494e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.358+0000 7f5b46000700 1 -- 172.21.15.44:0/1059313954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b49360e40 msgr2=0x563b494e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.358+0000 7f5b46000700 1 --2- 172.21.15.44:0/1059313954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b49360e40 0x563b494e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.358+0000 7f5b46000700 1 -- 172.21.15.44:0/1059313954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b494dfe70 msgr2=0x563b494e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.358+0000 7f5b46000700 1 --2- 172.21.15.44:0/1059313954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b494dfe70 0x563b494e2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.358+0000 7f5b46000700 1 -- 172.21.15.44:0/1059313954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b494dc000 con 0x563b4944c9f0 2024-05-27T09:49:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.359+0000 7f5b46000700 1 --2- 172.21.15.44:0/1059313954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b4944c9f0 0x563b494df7c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800eee0 tx=0x7f5b380e2400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a42bc71f5b8f218c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.359+0000 7f5b3f400700 1 -- 172.21.15.44:0/1059313954 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b380a3e60 con 0x563b4944c9f0 2024-05-27T09:49:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.359+0000 7f5b3f400700 1 -- 172.21.15.44:0/1059313954 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b380a9e40 con 0x563b4944c9f0 2024-05-27T09:49:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.359+0000 7f5b3f400700 1 -- 172.21.15.44:0/1059313954 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3808fc60 con 0x563b4944c9f0 2024-05-27T09:49:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.360+0000 7f5b644adb80 1 -- 172.21.15.44:0/1059313954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b4944c9f0 msgr2=0x563b494df7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.360+0000 7f5b644adb80 1 --2- 172.21.15.44:0/1059313954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b4944c9f0 0x563b494df7c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800eee0 tx=0x7f5b380e2400 comp rx=0 tx=0).stop 2024-05-27T09:49:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.360+0000 7f5b644adb80 1 -- 172.21.15.44:0/1059313954 shutdown_connections 2024-05-27T09:49:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.360+0000 7f5b644adb80 1 --2- 172.21.15.44:0/1059313954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b494dfe70 0x563b494e2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.360+0000 7f5b644adb80 1 --2- 172.21.15.44:0/1059313954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b49360e40 0x563b494e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.360+0000 7f5b644adb80 1 --2- 172.21.15.44:0/1059313954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b4944c9f0 0x563b494df7c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.360+0000 7f5b644adb80 1 -- 172.21.15.44:0/1059313954 >> 172.21.15.44:0/1059313954 conn(0x563b493fa390 msgr2=0x563b494df0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.360+0000 7f5b644adb80 1 -- 172.21.15.44:0/1059313954 shutdown_connections 2024-05-27T09:49:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.360+0000 7f5b644adb80 1 -- 172.21.15.44:0/1059313954 wait complete. 2024-05-27T09:49:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.361+0000 7f5b644adb80 1 -- start start 2024-05-27T09:49:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.361+0000 7f5b644adb80 1 Processor -- start 2024-05-27T09:49:56.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 0x563b494fbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b4944c9f0 0x563b494fc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b644adb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b494dfe70 0x563b494fc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b644adb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b501b3c50 con 0x563b4944c9f0 2024-05-27T09:49:56.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b644adb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b501b50c0 con 0x563b49360e40 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b644adb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b501b4320 con 0x563b494dfe70 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b644adb80 10 client.0 _ll_register_callbacks cb 0x563b493a7c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b46000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b4944c9f0 0x563b494fc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b46000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b4944c9f0 0x563b494fc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37712/0 (socket says 172.21.15.44:37712) 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b46000700 1 -- 172.21.15.44:0/2941054436 learned_addr learned my addr 172.21.15.44:0/2941054436 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b4f400700 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b494dfe70 0x563b494fc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.362+0000 7f5b4ea00700 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 0x563b494fbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.363+0000 7f5b4ea00700 1 -- 172.21.15.44:0/2941054436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b494dfe70 msgr2=0x563b494fc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.363+0000 7f5b4ea00700 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b494dfe70 0x563b494fc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.363+0000 7f5b4ea00700 1 -- 172.21.15.44:0/2941054436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b4944c9f0 msgr2=0x563b494fc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.363+0000 7f5b4ea00700 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b4944c9f0 0x563b494fc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.363+0000 7f5b4ea00700 1 -- 172.21.15.44:0/2941054436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b49562200 con 0x563b49360e40 2024-05-27T09:49:56.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.363+0000 7f5b4f400700 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b494dfe70 0x563b494fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:56.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.363+0000 7f5b46000700 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b4944c9f0 0x563b494fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:56.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.363+0000 7f5b4ea00700 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 0x563b494fbc10 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5b40005920 tx=0x7f5b40012a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f04c40b9fb2dac63 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:56.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.363+0000 7f5b3cc00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:49:56.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.363+0000 7f5b3cc00700 1 -- 172.21.15.44:0/2941054436 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b40004a30 con 0x563b49360e40 2024-05-27T09:49:56.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.363+0000 7f5b644adb80 1 -- 172.21.15.44:0/2941054436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b380a9e40 con 0x563b49360e40 2024-05-27T09:49:56.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.364+0000 7f5b3cc00700 1 -- 172.21.15.44:0/2941054436 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f5b40004bd0 con 0x563b49360e40 2024-05-27T09:49:56.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.364+0000 7f5b3cc00700 1 -- 172.21.15.44:0/2941054436 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f5b40004dd0 con 0x563b49360e40 2024-05-27T09:49:56.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.364+0000 7f5b644adb80 10 client.28061 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:56.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.364+0000 7f5b644adb80 1 -- 172.21.15.44:0/2941054436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x563b495634c0 con 0x563b49360e40 2024-05-27T09:49:56.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.364+0000 7f5b3cc00700 1 -- 172.21.15.44:0/2941054436 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b380a9e40 con 0x563b49360e40 2024-05-27T09:49:56.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.364+0000 7f5b3cc00700 1 -- 172.21.15.44:0/2941054436 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f5b400108e0 con 0x563b49360e40 2024-05-27T09:49:56.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.365+0000 7f5b644adb80 10 client.28061 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:56.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.365+0000 7f5b644adb80 10 client.28061 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:56.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.365+0000 7f5b644adb80 20 client.28061 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.365+0000 7f5b644adb80 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b49563b70 0x563b49565f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.365+0000 7f5b644adb80 4 client.28061 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.365+0000 7f5b644adb80 1 -- 172.21.15.44:0/2941054436 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563b49409570 con 0x563b49563b70 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.365+0000 7f5b46000700 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b49563b70 0x563b49565f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b46000700 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b49563b70 0x563b49565f90 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=131fca96cbdc6ba server_cookie=7bae2a82e01b19fa in_seq=0 out_seq=0 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b3cc00700 10 client.28061 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b3cc00700 1 -- 172.21.15.44:0/2941054436 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563b49409570 con 0x563b49563b70 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b3cc00700 10 client.28061 handle_command_reply: tid=0 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b644adb80 1 client.28061 shutdown 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b644adb80 1 -- 172.21.15.44:0/2941054436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 msgr2=0x563b494fbc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b644adb80 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 0x563b494fbc10 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5b40005920 tx=0x7f5b40012a10 comp rx=0 tx=0).stop 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b644adb80 1 -- 172.21.15.44:0/2941054436 shutdown_connections 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b644adb80 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x563b49563b70 0x563b49565f90 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b644adb80 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b494dfe70 0x563b494fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b644adb80 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b49360e40 0x563b494fbc10 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b644adb80 1 --2- 172.21.15.44:0/2941054436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563b4944c9f0 0x563b494fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b644adb80 1 -- 172.21.15.44:0/2941054436 >> 172.21.15.44:0/2941054436 conn(0x563b493fa390 msgr2=0x563b494e9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:56.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.366+0000 7f5b3cc00700 0 client.28061 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:56.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.367+0000 7f5b644adb80 1 -- 172.21.15.44:0/2941054436 shutdown_connections 2024-05-27T09:49:56.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.367+0000 7f5b644adb80 1 -- 172.21.15.44:0/2941054436 wait complete. 2024-05-27T09:49:56.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.367+0000 7f5b644adb80 20 client.28061 trim_cache size 0 max 16384 2024-05-27T09:49:56.370 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:49:56.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.367+0000 7f5b55a00700 1 -- 172.21.15.44:0/4024200259 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b340b2a70 msgr2=0x7f5b340b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:56.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.367+0000 7f5b55a00700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b340b2a70 0x7f5b340b4f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009fd0 tx=0x7f5b48009c00 comp rx=0 tx=0).stop 2024-05-27T09:49:56.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.367+0000 7f5b55a00700 1 -- 172.21.15.44:0/4024200259 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b500764a0 msgr2=0x7f5b501a19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:56.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.367+0000 7f5b55a00700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b500764a0 0x7f5b501a19f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800ef10 tx=0x7f5b3800c3a0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.368+0000 7f5b55a00700 1 -- 172.21.15.44:0/4024200259 shutdown_connections 2024-05-27T09:49:56.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.368+0000 7f5b55a00700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b340b2a70 0x7f5b340b4f80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.368+0000 7f5b55a00700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b50115360 0x7f5b501a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.368+0000 7f5b55a00700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b50075b50 0x7f5b501a1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.368+0000 7f5b55a00700 1 --2- 172.21.15.44:0/4024200259 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b500764a0 0x7f5b501a19f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:56.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.368+0000 7f5b55a00700 1 -- 172.21.15.44:0/4024200259 >> 172.21.15.44:0/4024200259 conn(0x7f5b50100030 msgr2=0x7f5b50100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:56.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.368+0000 7f5b55a00700 1 -- 172.21.15.44:0/4024200259 shutdown_connections 2024-05-27T09:49:56.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:49:56.368+0000 7f5b55a00700 1 -- 172.21.15.44:0/4024200259 wait complete. 2024-05-27T09:49:58.184 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:49:58.184 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:49:58.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:57 smithi044 ceph-mon[31618]: pgmap v463: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 99 op/s 2024-05-27T09:49:58.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:57 smithi050 ceph-mon[36371]: pgmap v463: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 99 op/s 2024-05-27T09:49:58.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:57 smithi189 ceph-mon[36671]: pgmap v463: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 99 op/s 2024-05-27T09:49:58.676 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:49:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.754+0000 7f146a400640 1 -- 172.21.15.44:0/3827374509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14641043e0 msgr2=0x7f14641107b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.754+0000 7f146a400640 1 --2- 172.21.15.44:0/3827374509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14641043e0 0x7f14641107b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f145c009be0 tx=0x7f145c02a6d0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.754+0000 7f146a400640 1 -- 172.21.15.44:0/3827374509 shutdown_connections 2024-05-27T09:49:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.754+0000 7f146a400640 1 --2- 172.21.15.44:0/3827374509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14641043e0 0x7f14641107b0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.754+0000 7f146a400640 1 --2- 172.21.15.44:0/3827374509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f146410a130 0x7f1464103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.754+0000 7f146a400640 1 --2- 172.21.15.44:0/3827374509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464109c90 0x7f1464103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.754+0000 7f146a400640 1 -- 172.21.15.44:0/3827374509 >> 172.21.15.44:0/3827374509 conn(0x7f1464079390 msgr2=0x7f14640797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.754+0000 7f146a400640 1 -- 172.21.15.44:0/3827374509 shutdown_connections 2024-05-27T09:49:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.755+0000 7f146a400640 1 -- 172.21.15.44:0/3827374509 wait complete. 2024-05-27T09:49:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.756+0000 7f146a400640 1 Processor -- start 2024-05-27T09:49:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.756+0000 7f146a400640 1 -- start start 2024-05-27T09:49:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.756+0000 7f146a400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14641043e0 0x7f1464074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.756+0000 7f146a400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464109c90 0x7f1464075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.756+0000 7f146a400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f146410a130 0x7f1464075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.756+0000 7f146a400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1464113ea0 con 0x7f146410a130 2024-05-27T09:49:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.756+0000 7f146a400640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1464113d20 con 0x7f14641043e0 2024-05-27T09:49:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.756+0000 7f146a400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1464114020 con 0x7f1464109c90 2024-05-27T09:49:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f1463e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f146410a130 0x7f1464075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f1463400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14641043e0 0x7f1464074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f1463e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f146410a130 0x7f1464075a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37720/0 (socket says 172.21.15.44:37720) 2024-05-27T09:49:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f1463e00640 1 -- 172.21.15.44:0/1599370330 learned_addr learned my addr 172.21.15.44:0/1599370330 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f1463400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14641043e0 0x7f1464074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42444/0 (socket says 172.21.15.44:42444) 2024-05-27T09:49:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f145be00640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464109c90 0x7f1464075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f1463e00640 1 -- 172.21.15.44:0/1599370330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464109c90 msgr2=0x7f1464075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f1463e00640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464109c90 0x7f1464075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.759 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f1463e00640 1 -- 172.21.15.44:0/1599370330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14641043e0 msgr2=0x7f1464074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:58.759 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f1463e00640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14641043e0 0x7f1464074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.759 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f1463e00640 1 -- 172.21.15.44:0/1599370330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14641a8c40 con 0x7f146410a130 2024-05-27T09:49:58.759 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.757+0000 7f1463400640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14641043e0 0x7f1464074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.758+0000 7f1463e00640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f146410a130 0x7f1464075a40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f145c004af0 tx=0x7f145c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.758+0000 7f1460c00640 1 -- 172.21.15.44:0/1599370330 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c041020 con 0x7f146410a130 2024-05-27T09:49:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.758+0000 7f1460c00640 1 -- 172.21.15.44:0/1599370330 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f145c003e10 con 0x7f146410a130 2024-05-27T09:49:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.758+0000 7f146a400640 1 -- 172.21.15.44:0/1599370330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14641a8ed0 con 0x7f146410a130 2024-05-27T09:49:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.758+0000 7f1460c00640 1 -- 172.21.15.44:0/1599370330 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c036040 con 0x7f146410a130 2024-05-27T09:49:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.758+0000 7f146a400640 1 -- 172.21.15.44:0/1599370330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14641a9330 con 0x7f146410a130 2024-05-27T09:49:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.762+0000 7f14713f4740 1 Processor -- start 2024-05-27T09:49:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.762+0000 7f1460c00640 1 -- 172.21.15.44:0/1599370330 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f145c044020 con 0x7f146410a130 2024-05-27T09:49:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.762+0000 7f14713f4740 1 -- start start 2024-05-27T09:49:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.762+0000 7f14713f4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f501b0 0x5624e22f2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.762+0000 7f14713f4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f272f0 0x5624e22f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f14713f4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f2f4d0 0x5624e22f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f14713f4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5624e2263b90 con 0x5624e1f272f0 2024-05-27T09:49:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f14713f4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5624e2253730 con 0x5624e1f501b0 2024-05-27T09:49:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f14713f4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5624e2257770 con 0x5624e1f2f4d0 2024-05-27T09:49:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f1463400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f272f0 0x5624e22f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f145be00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f501b0 0x5624e22f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f1463400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f272f0 0x5624e22f2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37736/0 (socket says 172.21.15.44:37736) 2024-05-27T09:49:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f1463400640 1 -- 172.21.15.44:0/3015989260 learned_addr learned my addr 172.21.15.44:0/3015989260 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f145be00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f501b0 0x5624e22f2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42448/0 (socket says 172.21.15.44:42448) 2024-05-27T09:49:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f145be00640 1 --2- 172.21.15.44:0/3015989260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f2f4d0 0x5624e22f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f1463400640 1 -- 172.21.15.44:0/3015989260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f2f4d0 msgr2=0x5624e22f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f1463400640 1 --2- 172.21.15.44:0/3015989260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f2f4d0 0x5624e22f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f1463400640 1 -- 172.21.15.44:0/3015989260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f501b0 msgr2=0x5624e22f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f1463400640 1 --2- 172.21.15.44:0/3015989260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f501b0 0x5624e22f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f1463400640 1 -- 172.21.15.44:0/3015989260 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624e21ec8b0 con 0x5624e1f272f0 2024-05-27T09:49:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.763+0000 7f145be00640 1 --2- 172.21.15.44:0/3015989260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f501b0 0x5624e22f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.764+0000 7f1463400640 1 --2- 172.21.15.44:0/3015989260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f272f0 0x5624e22f2da0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f14540114f0 tx=0x7f14540119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad73b73c5a49d40 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.764+0000 7f1460c00640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14480b2950 0x7f14480b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.764+0000 7f1458c00640 1 -- 172.21.15.44:0/3015989260 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454011030 con 0x5624e1f272f0 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.764+0000 7f1460c00640 1 -- 172.21.15.44:0/1599370330 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f145c102f40 con 0x7f146410a130 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.764+0000 7f145be00640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14480b2950 0x7f14480b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.764+0000 7f1458c00640 1 -- 172.21.15.44:0/3015989260 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14540111d0 con 0x5624e1f272f0 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.764+0000 7f1458c00640 1 -- 172.21.15.44:0/3015989260 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145400f5c0 con 0x5624e1f272f0 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.765+0000 7f14713f4740 1 -- 172.21.15.44:0/3015989260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f272f0 msgr2=0x5624e22f2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.765+0000 7f14713f4740 1 --2- 172.21.15.44:0/3015989260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f272f0 0x5624e22f2da0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f14540114f0 tx=0x7f14540119b0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.765+0000 7f145be00640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14480b2950 0x7f14480b4e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x5624e22ef9d0 tx=0x7f144c009680 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.765+0000 7f14713f4740 1 -- 172.21.15.44:0/3015989260 shutdown_connections 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.765+0000 7f14713f4740 1 --2- 172.21.15.44:0/3015989260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f2f4d0 0x5624e22f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.765+0000 7f14713f4740 1 --2- 172.21.15.44:0/3015989260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f272f0 0x5624e22f2da0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.765+0000 7f14713f4740 1 --2- 172.21.15.44:0/3015989260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f501b0 0x5624e22f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.765+0000 7f14713f4740 1 -- 172.21.15.44:0/3015989260 >> 172.21.15.44:0/3015989260 conn(0x5624e1f44300 msgr2=0x5624e22ef010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.765+0000 7f14713f4740 1 -- 172.21.15.44:0/3015989260 shutdown_connections 2024-05-27T09:49:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.765+0000 7f14713f4740 1 -- 172.21.15.44:0/3015989260 wait complete. 2024-05-27T09:49:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.766+0000 7f14713f4740 1 -- start start 2024-05-27T09:49:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.766+0000 7f14713f4740 1 Processor -- start 2024-05-27T09:49:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f14713f4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f272f0 0x5624e22fca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f14713f4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 0x5624e22fcf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f14713f4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 0x5624e2308cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f14713f4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5624e1f39790 con 0x5624e1f501b0 2024-05-27T09:49:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f14713f4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5624e21fbbc0 con 0x5624e1f2f4d0 2024-05-27T09:49:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f14713f4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5624e22ff990 con 0x5624e1f272f0 2024-05-27T09:49:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f14713f4740 10 client.0 _ll_register_callbacks cb 0x5624e21d9400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f1463400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f272f0 0x5624e22fca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f1463e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 0x5624e2308cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f1463400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f272f0 0x5624e22fca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35530/0 (socket says 172.21.15.44:35530) 2024-05-27T09:49:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f1463400640 1 -- 172.21.15.44:0/1963658414 learned_addr learned my addr 172.21.15.44:0/1963658414 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f145be00640 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 0x5624e22fcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f1463400640 1 -- 172.21.15.44:0/1963658414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 msgr2=0x5624e22fcf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.767+0000 7f1463400640 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 0x5624e22fcf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.768+0000 7f1463400640 1 -- 172.21.15.44:0/1963658414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 msgr2=0x5624e2308cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.768+0000 7f1463400640 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 0x5624e2308cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.768+0000 7f1463400640 1 -- 172.21.15.44:0/1963658414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624e1f68210 con 0x5624e1f272f0 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.768+0000 7f145be00640 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 0x5624e22fcf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.768+0000 7f1463e00640 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 0x5624e2308cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.768+0000 7f1463400640 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f272f0 0x5624e22fca00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f145401b040 tx=0x7f1454005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fec99cbad19f67f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.768+0000 7f1451600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.768+0000 7f1451600640 1 -- 172.21.15.44:0/1963658414 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454027070 con 0x5624e1f272f0 2024-05-27T09:49:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.768+0000 7f14713f4740 1 -- 172.21.15.44:0/1963658414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14540111d0 con 0x5624e1f272f0 2024-05-27T09:49:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.768+0000 7f1451600640 1 -- 172.21.15.44:0/1963658414 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f14540046c0 con 0x5624e1f272f0 2024-05-27T09:49:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.769+0000 7f1451600640 1 -- 172.21.15.44:0/1963658414 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1454033050 con 0x5624e1f272f0 2024-05-27T09:49:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.769+0000 7f14713f4740 10 client.28081 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.769+0000 7f14713f4740 1 -- 172.21.15.44:0/1963658414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5624e2034620 con 0x5624e1f272f0 2024-05-27T09:49:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.769+0000 7f1451600640 1 -- 172.21.15.44:0/1963658414 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14540111d0 con 0x5624e1f272f0 2024-05-27T09:49:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.769+0000 7f1451600640 1 -- 172.21.15.44:0/1963658414 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f145400f8f0 con 0x5624e1f272f0 2024-05-27T09:49:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.769+0000 7f14713f4740 10 client.28081 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.769+0000 7f14713f4740 10 client.28081 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.769+0000 7f14713f4740 20 client.28081 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.769+0000 7f14713f4740 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624e1f0ed80 0x5624e230ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.769+0000 7f14713f4740 4 client.28081 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:49:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.769+0000 7f14713f4740 1 -- 172.21.15.44:0/1963658414 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624e2265ae0 con 0x5624e1f0ed80 2024-05-27T09:49:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.770+0000 7f145be00640 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624e1f0ed80 0x5624e230ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.771+0000 7f145be00640 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624e1f0ed80 0x5624e230ebb0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48d025530f459447 server_cookie=6e16570fc103d225 in_seq=0 out_seq=0 2024-05-27T09:49:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.771+0000 7f1451600640 10 client.28081 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.771+0000 7f1451600640 1 -- 172.21.15.44:0/1963658414 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5624e2265ae0 con 0x5624e1f0ed80 2024-05-27T09:49:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.771+0000 7f1451600640 10 client.28081 handle_command_reply: tid=0 2024-05-27T09:49:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.771+0000 7f14713f4740 1 client.28081 shutdown 2024-05-27T09:49:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f14713f4740 1 -- 172.21.15.44:0/1963658414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f272f0 msgr2=0x5624e22fca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f14713f4740 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f272f0 0x5624e22fca00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f145401b040 tx=0x7f1454005270 comp rx=0 tx=0).stop 2024-05-27T09:49:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f14713f4740 1 -- 172.21.15.44:0/1963658414 shutdown_connections 2024-05-27T09:49:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f14713f4740 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624e1f0ed80 0x5624e230ebb0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f14713f4740 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 0x5624e2308cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f14713f4740 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 0x5624e22fcf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f14713f4740 1 --2- 172.21.15.44:0/1963658414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f272f0 0x5624e22fca00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f14713f4740 1 -- 172.21.15.44:0/1963658414 >> 172.21.15.44:0/1963658414 conn(0x5624e1f44300 msgr2=0x5624e22f4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f1451600640 0 client.28081 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f14713f4740 1 -- 172.21.15.44:0/1963658414 shutdown_connections 2024-05-27T09:49:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f14713f4740 1 -- 172.21.15.44:0/1963658414 wait complete. 2024-05-27T09:49:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.772+0000 7f14713f4740 20 client.28081 trim_cache size 0 max 16384 2024-05-27T09:49:58.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.789+0000 7f14713f4740 1 Processor -- start 2024-05-27T09:49:58.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.789+0000 7f14713f4740 1 -- start start 2024-05-27T09:49:58.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.789+0000 7f14713f4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 0x5624e22f8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.789+0000 7f14713f4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f501b0 0x5624e2308a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f14713f4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1ee41c0 0x5624e2308f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f14713f4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f14641b46f0 con 0x5624e1ee41c0 2024-05-27T09:49:58.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f14713f4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f14641b1f60 con 0x5624e1f2f4d0 2024-05-27T09:49:58.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f14713f4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f14641afdd0 con 0x5624e1f501b0 2024-05-27T09:49:58.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f1463e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1ee41c0 0x5624e2308f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f1463e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1ee41c0 0x5624e2308f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37760/0 (socket says 172.21.15.44:37760) 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f1463e00640 1 -- 172.21.15.44:0/2175365372 learned_addr learned my addr 172.21.15.44:0/2175365372 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f145be00640 1 --2- 172.21.15.44:0/2175365372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f501b0 0x5624e2308a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f1463400640 1 --2- 172.21.15.44:0/2175365372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 0x5624e22f8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f1463e00640 1 -- 172.21.15.44:0/2175365372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f501b0 msgr2=0x5624e2308a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f1463e00640 1 --2- 172.21.15.44:0/2175365372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f501b0 0x5624e2308a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f1463e00640 1 -- 172.21.15.44:0/2175365372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 msgr2=0x5624e22f8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f1463e00640 1 --2- 172.21.15.44:0/2175365372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 0x5624e22f8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f1463e00640 1 -- 172.21.15.44:0/2175365372 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624e22efc00 con 0x5624e1ee41c0 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f1463400640 1 --2- 172.21.15.44:0/2175365372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 0x5624e22f8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.790+0000 7f145be00640 1 --2- 172.21.15.44:0/2175365372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f501b0 0x5624e2308a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.791+0000 7f1463e00640 1 --2- 172.21.15.44:0/2175365372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1ee41c0 0x5624e2308f70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f145c100750 tx=0x7f145c0c9d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ce197c8ca0cedb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:58.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.791+0000 7f1453e00640 1 -- 172.21.15.44:0/2175365372 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c0ca980 con 0x5624e1ee41c0 2024-05-27T09:49:58.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.791+0000 7f1453e00640 1 -- 172.21.15.44:0/2175365372 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f145c0e4070 con 0x5624e1ee41c0 2024-05-27T09:49:58.793 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.791+0000 7f1453e00640 1 -- 172.21.15.44:0/2175365372 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c0d2eb0 con 0x5624e1ee41c0 2024-05-27T09:49:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.792+0000 7f14713f4740 1 -- 172.21.15.44:0/2175365372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1ee41c0 msgr2=0x5624e2308f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.792+0000 7f14713f4740 1 --2- 172.21.15.44:0/2175365372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1ee41c0 0x5624e2308f70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f145c100750 tx=0x7f145c0c9d90 comp rx=0 tx=0).stop 2024-05-27T09:49:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.792+0000 7f14713f4740 1 -- 172.21.15.44:0/2175365372 shutdown_connections 2024-05-27T09:49:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.792+0000 7f14713f4740 1 --2- 172.21.15.44:0/2175365372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1ee41c0 0x5624e2308f70 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.792+0000 7f14713f4740 1 --2- 172.21.15.44:0/2175365372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f501b0 0x5624e2308a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.792+0000 7f14713f4740 1 --2- 172.21.15.44:0/2175365372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1f2f4d0 0x5624e22f8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.792+0000 7f14713f4740 1 -- 172.21.15.44:0/2175365372 >> 172.21.15.44:0/2175365372 conn(0x5624e1f44300 msgr2=0x5624e22fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.792+0000 7f14713f4740 1 -- 172.21.15.44:0/2175365372 shutdown_connections 2024-05-27T09:49:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.792+0000 7f14713f4740 1 -- 172.21.15.44:0/2175365372 wait complete. 2024-05-27T09:49:58.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.792+0000 7f14713f4740 1 -- start start 2024-05-27T09:49:58.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.793+0000 7f14713f4740 1 Processor -- start 2024-05-27T09:49:58.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.793+0000 7f14713f4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1ee41c0 0x5624e22ffbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.793+0000 7f14713f4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f2f4d0 0x5624e2300120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.793+0000 7f14713f4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 0x5624e2300660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.793+0000 7f14713f4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f14641b0420 con 0x5624e1f501b0 2024-05-27T09:49:58.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.793+0000 7f14713f4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f14641b1890 con 0x5624e1ee41c0 2024-05-27T09:49:58.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.793+0000 7f14713f4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f14641b0af0 con 0x5624e1f2f4d0 2024-05-27T09:49:58.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.793+0000 7f14713f4740 10 client.0 _ll_register_callbacks cb 0x7f144c009a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:49:58.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1463e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 0x5624e2300660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1463400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1ee41c0 0x5624e22ffbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.797 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f145be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f2f4d0 0x5624e2300120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1463e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 0x5624e2300660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37762/0 (socket says 172.21.15.44:37762) 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1463e00640 1 -- 172.21.15.44:0/3200437339 learned_addr learned my addr 172.21.15.44:0/3200437339 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1463e00640 1 -- 172.21.15.44:0/3200437339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f2f4d0 msgr2=0x5624e2300120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1463e00640 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f2f4d0 0x5624e2300120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1463e00640 1 -- 172.21.15.44:0/3200437339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1ee41c0 msgr2=0x5624e22ffbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1463e00640 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1ee41c0 0x5624e22ffbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1463e00640 1 -- 172.21.15.44:0/3200437339 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624e2375280 con 0x5624e1f501b0 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1463400640 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1ee41c0 0x5624e22ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1463e00640 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 0x5624e2300660 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f145c0c74f0 tx=0x7f145c0ce3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d84b50b7124780 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1451600640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.794+0000 7f1451600640 1 -- 172.21.15.44:0/3200437339 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c0e8020 con 0x5624e1f501b0 2024-05-27T09:49:58.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f1451600640 1 -- 172.21.15.44:0/3200437339 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f145c0e4d80 con 0x5624e1f501b0 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f14713f4740 1 -- 172.21.15.44:0/3200437339 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f145c0e4070 con 0x5624e1f501b0 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f1451600640 1 -- 172.21.15.44:0/3200437339 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f145c030d50 con 0x5624e1f501b0 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f14713f4740 10 client.17811 fetch_fsmap learned FSMap version 43 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f14713f4740 1 -- 172.21.15.44:0/3200437339 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5624e2376510 con 0x5624e1f501b0 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f1451600640 1 -- 172.21.15.44:0/3200437339 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f145c0e4070 con 0x5624e1f501b0 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f1451600640 1 -- 172.21.15.44:0/3200437339 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f145c0c94e0 con 0x5624e1f501b0 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f14713f4740 10 client.17811 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f14713f4740 10 client.17811 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f14713f4740 20 client.17811 populate_metadata read hostname 'smithi044' 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f14713f4740 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624e2376b80 0x5624e2378fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f14713f4740 4 client.17811 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:49:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.795+0000 7f14713f4740 1 -- 172.21.15.44:0/3200437339 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5624e225f2c0 con 0x5624e2376b80 2024-05-27T09:49:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.796+0000 7f1463400640 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624e2376b80 0x5624e2378fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:49:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.796+0000 7f1463400640 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624e2376b80 0x5624e2378fa0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=866aa9d06fbd1d31 server_cookie=e47ca02251fb6c55 in_seq=0 out_seq=0 2024-05-27T09:49:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.796+0000 7f1451600640 10 client.17811 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:58.817 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f1451600640 1 -- 172.21.15.44:0/3200437339 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5624e225f2c0 con 0x5624e2376b80 2024-05-27T09:49:58.817 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f1451600640 10 client.17811 handle_command_reply: tid=0 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f14713f4740 1 client.17811 shutdown 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f14713f4740 1 -- 172.21.15.44:0/3200437339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 msgr2=0x5624e2300660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f14713f4740 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 0x5624e2300660 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f145c0c74f0 tx=0x7f145c0ce3a0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f14713f4740 1 -- 172.21.15.44:0/3200437339 shutdown_connections 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f14713f4740 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5624e2376b80 0x5624e2378fa0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f14713f4740 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5624e1f501b0 0x5624e2300660 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f14713f4740 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624e1f2f4d0 0x5624e2300120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f14713f4740 1 --2- 172.21.15.44:0/3200437339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5624e1ee41c0 0x5624e22ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f1451600640 0 client.17811 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f14713f4740 1 -- 172.21.15.44:0/3200437339 >> 172.21.15.44:0/3200437339 conn(0x5624e1f44300 msgr2=0x5624e22f3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f14713f4740 1 -- 172.21.15.44:0/3200437339 shutdown_connections 2024-05-27T09:49:58.818 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.817+0000 7f14713f4740 1 -- 172.21.15.44:0/3200437339 wait complete. 2024-05-27T09:49:58.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.818+0000 7f14713f4740 20 client.17811 trim_cache size 0 max 16384 2024-05-27T09:49:58.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:49:58.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:49:58.819 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:49:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.818+0000 7f146a400640 1 -- 172.21.15.44:0/1599370330 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14480b2950 msgr2=0x7f14480b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.818+0000 7f146a400640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14480b2950 0x7f14480b4e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x5624e22ef9d0 tx=0x7f144c009680 comp rx=0 tx=0).stop 2024-05-27T09:49:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.818+0000 7f146a400640 1 -- 172.21.15.44:0/1599370330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f146410a130 msgr2=0x7f1464075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:49:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.818+0000 7f146a400640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f146410a130 0x7f1464075a40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f145c004af0 tx=0x7f145c003c20 comp rx=0 tx=0).stop 2024-05-27T09:49:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.818+0000 7f146a400640 1 -- 172.21.15.44:0/1599370330 shutdown_connections 2024-05-27T09:49:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.818+0000 7f146a400640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f14480b2950 0x7f14480b4e60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.818+0000 7f146a400640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f146410a130 0x7f1464075a40 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.818+0000 7f146a400640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464109c90 0x7f1464075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.818+0000 7f146a400640 1 --2- 172.21.15.44:0/1599370330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14641043e0 0x7f1464074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:49:58.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.819+0000 7f146a400640 1 -- 172.21.15.44:0/1599370330 >> 172.21.15.44:0/1599370330 conn(0x7f1464079390 msgr2=0x7f14641020c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:49:58.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.819+0000 7f146a400640 1 -- 172.21.15.44:0/1599370330 shutdown_connections 2024-05-27T09:49:58.821 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:49:58.819+0000 7f146a400640 1 -- 172.21.15.44:0/1599370330 wait complete. 2024-05-27T09:50:00.174 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:49:59 smithi044 ceph-mon[31618]: pgmap v464: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 93 op/s 2024-05-27T09:50:00.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:49:59 smithi050 ceph-mon[36371]: pgmap v464: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 93 op/s 2024-05-27T09:50:00.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:49:59 smithi189 ceph-mon[36671]: pgmap v464: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 93 op/s 2024-05-27T09:50:01.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:00 smithi044 ceph-mon[31618]: overall HEALTH_OK 2024-05-27T09:50:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.229+0000 7f26a0200700 1 -- 172.21.15.44:0/137868976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2698105e30 msgr2=0x7f269810e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.229+0000 7f26a0200700 1 --2- 172.21.15.44:0/137868976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2698105e30 0x7f269810e380 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f26840098f0 tx=0x7f2684009c00 comp rx=0 tx=0).stop 2024-05-27T09:50:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.229+0000 7f26a0200700 1 -- 172.21.15.44:0/137868976 shutdown_connections 2024-05-27T09:50:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.229+0000 7f26a0200700 1 --2- 172.21.15.44:0/137868976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f269810e910 0x7f2698110df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.229+0000 7f26a0200700 1 --2- 172.21.15.44:0/137868976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26981054e0 0x7f26981058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.229+0000 7f26a0200700 1 --2- 172.21.15.44:0/137868976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2698105e30 0x7f269810e380 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.229+0000 7f26a0200700 1 -- 172.21.15.44:0/137868976 >> 172.21.15.44:0/137868976 conn(0x7f2698078df0 msgr2=0x7f2698079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.229+0000 7f26a0200700 1 -- 172.21.15.44:0/137868976 shutdown_connections 2024-05-27T09:50:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.229+0000 7f26a0200700 1 -- 172.21.15.44:0/137868976 wait complete. 2024-05-27T09:50:01.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.230+0000 7f26a0200700 1 Processor -- start 2024-05-27T09:50:01.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.231+0000 7f26a0200700 1 -- start start 2024-05-27T09:50:01.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.231+0000 7f26a0200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26981054e0 0x7f269819aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.231+0000 7f26a0200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2698105e30 0x7f269819b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.231+0000 7f26a0200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269810e910 0x7f269819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.231+0000 7f26a0200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26981144c0 con 0x7f26981054e0 2024-05-27T09:50:01.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.231+0000 7f26a0200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2698114340 con 0x7f2698105e30 2024-05-27T09:50:01.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.231+0000 7f26a0200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2698114640 con 0x7f269810e910 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269d800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26981054e0 0x7f269819aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269ce00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2698105e30 0x7f269819b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269810e910 0x7f269819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269ce00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2698105e30 0x7f269819b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:42500/0 (socket says 172.21.15.44:42500) 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269810e910 0x7f269819f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35554/0 (socket says 172.21.15.44:35554) 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269ce00700 1 -- 172.21.15.44:0/3167809141 learned_addr learned my addr 172.21.15.44:0/3167809141 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269d800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26981054e0 0x7f269819aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37768/0 (socket says 172.21.15.44:37768) 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269e200700 1 -- 172.21.15.44:0/3167809141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2698105e30 msgr2=0x7f269819b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269e200700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2698105e30 0x7f269819b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269e200700 1 -- 172.21.15.44:0/3167809141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26981054e0 msgr2=0x7f269819aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269e200700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26981054e0 0x7f269819aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.232+0000 7f269e200700 1 -- 172.21.15.44:0/3167809141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269819fda0 con 0x7f269810e910 2024-05-27T09:50:01.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.233+0000 7f269e200700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269810e910 0x7f269819f700 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f269000efc0 tx=0x7f269000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:01.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.233+0000 7f2696000700 1 -- 172.21.15.44:0/3167809141 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26900092e0 con 0x7f269810e910 2024-05-27T09:50:01.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.233+0000 7f2696000700 1 -- 172.21.15.44:0/3167809141 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2690004bb0 con 0x7f269810e910 2024-05-27T09:50:01.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.233+0000 7f2696000700 1 -- 172.21.15.44:0/3167809141 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2690004020 con 0x7f269810e910 2024-05-27T09:50:01.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.233+0000 7f26a0200700 1 -- 172.21.15.44:0/3167809141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26981a0030 con 0x7f269810e910 2024-05-27T09:50:01.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.233+0000 7f26a0200700 1 -- 172.21.15.44:0/3167809141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26981a0520 con 0x7f269810e910 2024-05-27T09:50:01.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.237+0000 7f2696000700 1 -- 172.21.15.44:0/3167809141 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f269001b030 con 0x7f269810e910 2024-05-27T09:50:01.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.238+0000 7f26aed31b80 1 Processor -- start 2024-05-27T09:50:01.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.238+0000 7f26aed31b80 1 -- start start 2024-05-27T09:50:01.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.238+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f7163b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f7168f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f716ef0 0x564d3f723190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f26aed31b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564d3f644ec0 con 0x564d3f597ec0 2024-05-27T09:50:01.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f26aed31b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564d3f634a60 con 0x564d3f716ef0 2024-05-27T09:50:01.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f26aed31b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564d3f638aa0 con 0x564d3f683a70 2024-05-27T09:50:01.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f2696000700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26800b2a70 0x7f26800b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f2696000700 1 -- 172.21.15.44:0/3167809141 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f26900e1ff0 con 0x7f269810e910 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f269d800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f7168f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f269ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f7163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f269ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f7163b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35568/0 (socket says 172.21.15.44:35568) 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f269ce00700 1 -- 172.21.15.44:0/783823927 learned_addr learned my addr 172.21.15.44:0/783823927 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f269d800700 1 --2- 172.21.15.44:0/783823927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f7168f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37780/0 (socket says 172.21.15.44:37780) 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f269e200700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26800b2a70 0x7f26800b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.239+0000 7f269ce00700 1 --2- 172.21.15.44:0/783823927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f716ef0 0x564d3f723190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.240+0000 7f269d800700 1 -- 172.21.15.44:0/783823927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 msgr2=0x564d3f7163b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.240+0000 7f269d800700 1 --2- 172.21.15.44:0/783823927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f7163b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.240+0000 7f269d800700 1 -- 172.21.15.44:0/783823927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f716ef0 msgr2=0x564d3f723190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.240+0000 7f269d800700 1 --2- 172.21.15.44:0/783823927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f716ef0 0x564d3f723190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.240+0000 7f269d800700 1 -- 172.21.15.44:0/783823927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d3f5fd4a0 con 0x564d3f597ec0 2024-05-27T09:50:01.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.240+0000 7f269e200700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26800b2a70 0x7f26800b4f80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f26900df730 tx=0x7f2690009040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:01.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.240+0000 7f269d800700 1 --2- 172.21.15.44:0/783823927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f7168f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f268c007bf0 tx=0x7f268c007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38d16bcf96dfe96 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:01.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.241+0000 7f268aa00700 1 -- 172.21.15.44:0/783823927 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c004310 con 0x564d3f597ec0 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.241+0000 7f268aa00700 1 -- 172.21.15.44:0/783823927 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f268c015070 con 0x564d3f597ec0 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.241+0000 7f268aa00700 1 -- 172.21.15.44:0/783823927 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c004e20 con 0x564d3f597ec0 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.242+0000 7f26aed31b80 1 -- 172.21.15.44:0/783823927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 msgr2=0x564d3f7168f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.242+0000 7f26aed31b80 1 --2- 172.21.15.44:0/783823927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f7168f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f268c007bf0 tx=0x7f268c007f00 comp rx=0 tx=0).stop 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.242+0000 7f26aed31b80 1 -- 172.21.15.44:0/783823927 shutdown_connections 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.242+0000 7f26aed31b80 1 --2- 172.21.15.44:0/783823927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f716ef0 0x564d3f723190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.242+0000 7f26aed31b80 1 --2- 172.21.15.44:0/783823927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f7163b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.242+0000 7f26aed31b80 1 --2- 172.21.15.44:0/783823927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f7168f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.242+0000 7f26aed31b80 1 -- 172.21.15.44:0/783823927 >> 172.21.15.44:0/783823927 conn(0x564d3f631410 msgr2=0x564d3f712540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.242+0000 7f26aed31b80 1 -- 172.21.15.44:0/783823927 shutdown_connections 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.242+0000 7f26aed31b80 1 -- 172.21.15.44:0/783823927 wait complete. 2024-05-27T09:50:01.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.242+0000 7f26aed31b80 1 -- start start 2024-05-27T09:50:01.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.243+0000 7f26aed31b80 1 Processor -- start 2024-05-27T09:50:01.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.243+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f720750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.243+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f720c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.243+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f716ef0 0x564d3f7211d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.243+0000 7f26aed31b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564d3f53b020 con 0x564d3f597ec0 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.243+0000 7f26aed31b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564d3f5d57e0 con 0x564d3f716ef0 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.243+0000 7f26aed31b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564d3f723c00 con 0x564d3f683a70 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f26aed31b80 10 client.0 _ll_register_callbacks cb 0x564d3f5decf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f269ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f720750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f269d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f720c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f269ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f720750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37786/0 (socket says 172.21.15.44:37786) 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f269ce00700 1 -- 172.21.15.44:0/2848715870 learned_addr learned my addr 172.21.15.44:0/2848715870 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f269d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f720c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35584/0 (socket says 172.21.15.44:35584) 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f269ce00700 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f716ef0 0x564d3f7211d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f269d800700 1 -- 172.21.15.44:0/2848715870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f716ef0 msgr2=0x564d3f7211d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:01.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f269d800700 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f716ef0 0x564d3f7211d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f269d800700 1 -- 172.21.15.44:0/2848715870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 msgr2=0x564d3f720750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:01.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f269d800700 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f720750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.244+0000 7f269d800700 1 -- 172.21.15.44:0/2848715870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d3f584950 con 0x564d3f683a70 2024-05-27T09:50:01.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.245+0000 7f269d800700 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f720c90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f268c000c00 tx=0x7f268c0044d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3efbfe5b282f03f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:01.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.245+0000 7f267f400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:50:01.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.245+0000 7f267f400700 1 -- 172.21.15.44:0/2848715870 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c027060 con 0x564d3f683a70 2024-05-27T09:50:01.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.245+0000 7f267f400700 1 -- 172.21.15.44:0/2848715870 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f268c010c50 con 0x564d3f683a70 2024-05-27T09:50:01.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.245+0000 7f26aed31b80 1 -- 172.21.15.44:0/2848715870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f268c015070 con 0x564d3f683a70 2024-05-27T09:50:01.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.246+0000 7f267f400700 1 -- 172.21.15.44:0/2848715870 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f268c005030 con 0x564d3f683a70 2024-05-27T09:50:01.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.246+0000 7f26aed31b80 10 client.28099 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:01.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.246+0000 7f26aed31b80 1 -- 172.21.15.44:0/2848715870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x564d3f6050e0 con 0x564d3f683a70 2024-05-27T09:50:01.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.246+0000 7f267f400700 1 -- 172.21.15.44:0/2848715870 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f268c015070 con 0x564d3f683a70 2024-05-27T09:50:01.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.246+0000 7f267f400700 1 -- 172.21.15.44:0/2848715870 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f268c003680 con 0x564d3f683a70 2024-05-27T09:50:01.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.246+0000 7f26aed31b80 10 client.28099 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:01.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.246+0000 7f26aed31b80 10 client.28099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:01.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.246+0000 7f26aed31b80 20 client.28099 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:01.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.246+0000 7f26aed31b80 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564d3f730a60 0x564d3f732e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.246+0000 7f26aed31b80 4 client.28099 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:01.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.246+0000 7f26aed31b80 1 -- 172.21.15.44:0/2848715870 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d3f646e10 con 0x564d3f730a60 2024-05-27T09:50:01.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.247+0000 7f269ce00700 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564d3f730a60 0x564d3f732e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.247+0000 7f269ce00700 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564d3f730a60 0x564d3f732e80 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a93c390a3c9b55ac server_cookie=222022514eda04b7 in_seq=0 out_seq=0 2024-05-27T09:50:01.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.247+0000 7f267f400700 10 client.28099 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:01.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.248+0000 7f267f400700 1 -- 172.21.15.44:0/2848715870 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564d3f646e10 con 0x564d3f730a60 2024-05-27T09:50:01.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.248+0000 7f267f400700 10 client.28099 handle_command_reply: tid=0 2024-05-27T09:50:01.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.248+0000 7f26aed31b80 1 client.28099 shutdown 2024-05-27T09:50:01.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f26aed31b80 1 -- 172.21.15.44:0/2848715870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 msgr2=0x564d3f720c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:01.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f26aed31b80 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f720c90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f268c000c00 tx=0x7f268c0044d0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f26aed31b80 1 -- 172.21.15.44:0/2848715870 shutdown_connections 2024-05-27T09:50:01.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f26aed31b80 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564d3f730a60 0x564d3f732e80 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f26aed31b80 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f716ef0 0x564d3f7211d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f26aed31b80 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f720750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f26aed31b80 1 --2- 172.21.15.44:0/2848715870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f720c90 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f267f400700 0 client.28099 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:01.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f26aed31b80 1 -- 172.21.15.44:0/2848715870 >> 172.21.15.44:0/2848715870 conn(0x564d3f631410 msgr2=0x564d3f716390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:01.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f26aed31b80 1 -- 172.21.15.44:0/2848715870 shutdown_connections 2024-05-27T09:50:01.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f26aed31b80 1 -- 172.21.15.44:0/2848715870 wait complete. 2024-05-27T09:50:01.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.249+0000 7f26aed31b80 20 client.28099 trim_cache size 0 max 16384 2024-05-27T09:50:01.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.272+0000 7f26aed31b80 1 Processor -- start 2024-05-27T09:50:01.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.272+0000 7f26aed31b80 1 -- start start 2024-05-27T09:50:01.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f597ec0 0x564d3f71bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f71f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f716ef0 0x564d3f721140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f26aed31b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26981af760 con 0x564d3f716ef0 2024-05-27T09:50:01.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f26aed31b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f26981acf90 con 0x564d3f597ec0 2024-05-27T09:50:01.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f26aed31b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f26981aae00 con 0x564d3f683a70 2024-05-27T09:50:01.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f269d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f71f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f269d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f71f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35598/0 (socket says 172.21.15.44:35598) 2024-05-27T09:50:01.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f269d800700 1 -- 172.21.15.44:0/1660784122 learned_addr learned my addr 172.21.15.44:0/1660784122 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:01.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f269ce00700 1 --2- 172.21.15.44:0/1660784122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f597ec0 0x564d3f71bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f269d800700 1 -- 172.21.15.44:0/1660784122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f597ec0 msgr2=0x564d3f71bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:01.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f269d800700 1 --2- 172.21.15.44:0/1660784122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f597ec0 0x564d3f71bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f269d800700 1 -- 172.21.15.44:0/1660784122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f716ef0 msgr2=0x564d3f721140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:01.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f269d800700 1 --2- 172.21.15.44:0/1660784122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f716ef0 0x564d3f721140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f269d800700 1 -- 172.21.15.44:0/1660784122 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d3f717a30 con 0x564d3f683a70 2024-05-27T09:50:01.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.273+0000 7f269ce00700 1 --2- 172.21.15.44:0/1660784122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f597ec0 0x564d3f71bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:50:01.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.274+0000 7f269d800700 1 --2- 172.21.15.44:0/1660784122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f71f6b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f268c007870 tx=0x7f268c0106b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b9d479437186ee0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:01.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.274+0000 7f268a000700 1 -- 172.21.15.44:0/1660784122 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c027070 con 0x564d3f683a70 2024-05-27T09:50:01.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.274+0000 7f268a000700 1 -- 172.21.15.44:0/1660784122 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f268c015070 con 0x564d3f683a70 2024-05-27T09:50:01.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.274+0000 7f268a000700 1 -- 172.21.15.44:0/1660784122 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c0045f0 con 0x564d3f683a70 2024-05-27T09:50:01.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.275+0000 7f26aed31b80 1 -- 172.21.15.44:0/1660784122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 msgr2=0x564d3f71f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:01.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.275+0000 7f26aed31b80 1 --2- 172.21.15.44:0/1660784122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f71f6b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f268c007870 tx=0x7f268c0106b0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.275+0000 7f26aed31b80 1 -- 172.21.15.44:0/1660784122 shutdown_connections 2024-05-27T09:50:01.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.275+0000 7f26aed31b80 1 --2- 172.21.15.44:0/1660784122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f716ef0 0x564d3f721140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.275+0000 7f26aed31b80 1 --2- 172.21.15.44:0/1660784122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f597ec0 0x564d3f71bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.275+0000 7f26aed31b80 1 --2- 172.21.15.44:0/1660784122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f683a70 0x564d3f71f6b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.275+0000 7f26aed31b80 1 -- 172.21.15.44:0/1660784122 >> 172.21.15.44:0/1660784122 conn(0x564d3f631410 msgr2=0x564d3f72be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:01.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.275+0000 7f26aed31b80 1 -- 172.21.15.44:0/1660784122 shutdown_connections 2024-05-27T09:50:01.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.275+0000 7f26aed31b80 1 -- 172.21.15.44:0/1660784122 wait complete. 2024-05-27T09:50:01.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.276+0000 7f26aed31b80 1 -- start start 2024-05-27T09:50:01.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.276+0000 7f26aed31b80 1 Processor -- start 2024-05-27T09:50:01.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.276+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f7935b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.276+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f683a70 0x564d3f793af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f26aed31b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f716ef0 0x564d3f7324d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f26aed31b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26981ab450 con 0x564d3f597ec0 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f26aed31b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f26981ac8c0 con 0x564d3f683a70 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f26aed31b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f26981abb20 con 0x564d3f716ef0 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f26aed31b80 10 client.0 _ll_register_callbacks cb 0x564d3f5decf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f269ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f7935b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f269d800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f683a70 0x564d3f793af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f269ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f7935b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37802/0 (socket says 172.21.15.44:37802) 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f269ce00700 1 -- 172.21.15.44:0/337570879 learned_addr learned my addr 172.21.15.44:0/337570879 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f269ce00700 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f716ef0 0x564d3f7324d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f269d800700 1 -- 172.21.15.44:0/337570879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f716ef0 msgr2=0x564d3f7324d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f269d800700 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f716ef0 0x564d3f7324d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f269d800700 1 -- 172.21.15.44:0/337570879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 msgr2=0x564d3f7935b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:01.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f269d800700 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f7935b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.277+0000 7f269d800700 1 -- 172.21.15.44:0/337570879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d3f732b40 con 0x564d3f683a70 2024-05-27T09:50:01.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.278+0000 7f269d800700 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f683a70 0x564d3f793af0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f268c013040 tx=0x7f268c003c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=756b7a05f084186c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:01.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.278+0000 7f267f400700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:01.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.278+0000 7f267f400700 1 -- 172.21.15.44:0/337570879 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c027040 con 0x564d3f683a70 2024-05-27T09:50:01.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.278+0000 7f26aed31b80 1 -- 172.21.15.44:0/337570879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f268c015070 con 0x564d3f683a70 2024-05-27T09:50:01.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.278+0000 7f267f400700 1 -- 172.21.15.44:0/337570879 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8607+0+0 (secure 0 0 0) 0x7f268c0055e0 con 0x564d3f683a70 2024-05-27T09:50:01.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.279+0000 7f267f400700 1 -- 172.21.15.44:0/337570879 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f268c010dd0 con 0x564d3f683a70 2024-05-27T09:50:01.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.279+0000 7f26aed31b80 10 client.28094 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.279+0000 7f26aed31b80 1 -- 172.21.15.44:0/337570879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x564d3f79bd40 con 0x564d3f683a70 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.279+0000 7f267f400700 1 -- 172.21.15.44:0/337570879 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f268c015070 con 0x564d3f683a70 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.279+0000 7f267f400700 1 -- 172.21.15.44:0/337570879 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f268c0236d0 con 0x564d3f683a70 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.279+0000 7f26aed31b80 10 client.28094 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.279+0000 7f26aed31b80 10 client.28094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.280+0000 7f26aed31b80 20 client.28094 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.280+0000 7f26aed31b80 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564d3f79c390 0x564d3f79e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.280+0000 7f26aed31b80 4 client.28094 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.280+0000 7f26aed31b80 1 -- 172.21.15.44:0/337570879 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564d3f6405f0 con 0x564d3f79c390 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.280+0000 7f269ce00700 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564d3f79c390 0x564d3f79e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.280+0000 7f269ce00700 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564d3f79c390 0x564d3f79e7b0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f82b6326f04b6ef8 server_cookie=38ba5f17c489826d in_seq=0 out_seq=0 2024-05-27T09:50:01.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.280+0000 7f267f400700 10 client.28094 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:01.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f267f400700 1 -- 172.21.15.44:0/337570879 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564d3f6405f0 con 0x564d3f79c390 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f267f400700 10 client.28094 handle_command_reply: tid=0 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f26aed31b80 1 client.28094 shutdown 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f26aed31b80 1 -- 172.21.15.44:0/337570879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f683a70 msgr2=0x564d3f793af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f26aed31b80 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f683a70 0x564d3f793af0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f268c013040 tx=0x7f268c003c20 comp rx=0 tx=0).stop 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f26aed31b80 1 -- 172.21.15.44:0/337570879 shutdown_connections 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f26aed31b80 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564d3f79c390 0x564d3f79e7b0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f26aed31b80 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d3f716ef0 0x564d3f7324d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f26aed31b80 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d3f597ec0 0x564d3f7935b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f26aed31b80 1 --2- 172.21.15.44:0/337570879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564d3f683a70 0x564d3f793af0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f26aed31b80 1 -- 172.21.15.44:0/337570879 >> 172.21.15.44:0/337570879 conn(0x564d3f631410 msgr2=0x564d3f7186e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f267f400700 0 client.28094 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f26aed31b80 1 -- 172.21.15.44:0/337570879 shutdown_connections 2024-05-27T09:50:01.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.282+0000 7f26aed31b80 1 -- 172.21.15.44:0/337570879 wait complete. 2024-05-27T09:50:01.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.283+0000 7f26aed31b80 20 client.28094 trim_cache size 0 max 16384 2024-05-27T09:50:01.285 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:50:01.285 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:50:01.285 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "2ae01219-0f80-494a-bd79-d97ff8b29510", 2024-05-27T09:50:01.285 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:50:01.285 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:50:01.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.283+0000 7f26a0200700 1 -- 172.21.15.44:0/3167809141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26800b2a70 msgr2=0x7f26800b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:01.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.283+0000 7f26a0200700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26800b2a70 0x7f26800b4f80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f26900df730 tx=0x7f2690009040 comp rx=0 tx=0).stop 2024-05-27T09:50:01.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.284+0000 7f26a0200700 1 -- 172.21.15.44:0/3167809141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269810e910 msgr2=0x7f269819f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:01.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.284+0000 7f26a0200700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269810e910 0x7f269819f700 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f269000efc0 tx=0x7f269000c3a0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.284+0000 7f26a0200700 1 -- 172.21.15.44:0/3167809141 shutdown_connections 2024-05-27T09:50:01.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.284+0000 7f26a0200700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f26800b2a70 0x7f26800b4f80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.284+0000 7f26a0200700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269810e910 0x7f269819f700 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.284+0000 7f26a0200700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26981054e0 0x7f269819aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.284+0000 7f26a0200700 1 --2- 172.21.15.44:0/3167809141 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2698105e30 0x7f269819b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:01.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.284+0000 7f26a0200700 1 -- 172.21.15.44:0/3167809141 >> 172.21.15.44:0/3167809141 conn(0x7f2698078df0 msgr2=0x7f2698100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:01.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.284+0000 7f26a0200700 1 -- 172.21.15.44:0/3167809141 shutdown_connections 2024-05-27T09:50:01.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:01.284+0000 7f26a0200700 1 -- 172.21.15.44:0/3167809141 wait complete. 2024-05-27T09:50:01.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:00 smithi050 ceph-mon[36371]: overall HEALTH_OK 2024-05-27T09:50:01.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:00 smithi189 ceph-mon[36671]: overall HEALTH_OK 2024-05-27T09:50:02.008 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:50:02.123 INFO:tasks.workunit.client.0.smithi044.stdout:ok 2024-05-27T09:50:02.127 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T09:50:02.127 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T09:50:02.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:01 smithi044 ceph-mon[31618]: pgmap v465: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 99 op/s 2024-05-27T09:50:02.218 INFO:tasks.workunit:Running workunit fs/misc/general_vxattrs.sh... 2024-05-27T09:50:02.219 DEBUG:teuthology.orchestra.run.smithi044:workunit test fs/misc/general_vxattrs.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6507388dd5057528934822c0163b0c347ef1d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh 2024-05-27T09:50:02.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:01 smithi050 ceph-mon[36371]: pgmap v465: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 99 op/s 2024-05-27T09:50:02.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:01 smithi189 ceph-mon[36671]: pgmap v465: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 99 op/s 2024-05-27T09:50:02.521 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:03.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:02 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:50:03.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:02 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:03.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:02 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:50:03.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:02 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:03.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:02 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:50:03.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:02 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:03.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:02 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:50:03.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:02 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:03.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:02 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:50:03.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:02 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:03.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:02 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:50:03.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:02 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:03 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:50:04.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:03 smithi044 ceph-mon[31618]: pgmap v466: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-05-27T09:50:04.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:03 smithi044 ceph-mon[31618]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T09:50:04.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:03 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:50:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:03 smithi050 ceph-mon[36371]: pgmap v466: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-05-27T09:50:04.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:03 smithi050 ceph-mon[36371]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T09:50:04.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:03 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:50:04.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:03 smithi189 ceph-mon[36671]: pgmap v466: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-05-27T09:50:04.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:03 smithi189 ceph-mon[36671]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T09:50:05.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.048+0000 7f91cea00700 1 -- 172.21.15.44:0/3467734104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c810a920 msgr2=0x7f91c810ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:05.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.048+0000 7f91cea00700 1 --2- 172.21.15.44:0/3467734104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c810a920 0x7f91c810ad30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f91b80098f0 tx=0x7f91b8009c00 comp rx=0 tx=0).stop 2024-05-27T09:50:05.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.049+0000 7f91cea00700 1 -- 172.21.15.44:0/3467734104 shutdown_connections 2024-05-27T09:50:05.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.049+0000 7f91cea00700 1 --2- 172.21.15.44:0/3467734104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91c811d920 0x7f91c811fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.049+0000 7f91cea00700 1 --2- 172.21.15.44:0/3467734104 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c810a920 0x7f91c810ad30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.049+0000 7f91cea00700 1 --2- 172.21.15.44:0/3467734104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c810b270 0x7f91c811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.049+0000 7f91cea00700 1 -- 172.21.15.44:0/3467734104 >> 172.21.15.44:0/3467734104 conn(0x7f91c806f4c0 msgr2=0x7f91c806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:05.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.051+0000 7f91cea00700 1 -- 172.21.15.44:0/3467734104 shutdown_connections 2024-05-27T09:50:05.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.052+0000 7f91cea00700 1 -- 172.21.15.44:0/3467734104 wait complete. 2024-05-27T09:50:05.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.053+0000 7f91cea00700 1 Processor -- start 2024-05-27T09:50:05.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.054+0000 7f91cea00700 1 -- start start 2024-05-27T09:50:05.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.055+0000 7f91cea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c810a920 0x7f91c81af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.055+0000 7f91cea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91c810b270 0x7f91c81afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.055+0000 7f91cea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c811d920 0x7f91c81a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.055+0000 7f91cea00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f91c81232d0 con 0x7f91c811d920 2024-05-27T09:50:05.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.055+0000 7f91cea00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f91c8123150 con 0x7f91c810b270 2024-05-27T09:50:05.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.055+0000 7f91cea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f91c8123450 con 0x7f91c810a920 2024-05-27T09:50:05.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.055+0000 7f91c7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c810a920 0x7f91c81af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.055+0000 7f91c7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c810a920 0x7f91c81af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34896/0 (socket says 172.21.15.44:34896) 2024-05-27T09:50:05.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.055+0000 7f91c7e00700 1 -- 172.21.15.44:0/1348547407 learned_addr learned my addr 172.21.15.44:0/1348547407 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:05.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.056+0000 7f91c7400700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91c810b270 0x7f91c81afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.056+0000 7f91cca00700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c811d920 0x7f91c81a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.056+0000 7f91c7400700 1 -- 172.21.15.44:0/1348547407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c810a920 msgr2=0x7f91c81af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:05.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.056+0000 7f91c7400700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c810a920 0x7f91c81af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.056+0000 7f91c7400700 1 -- 172.21.15.44:0/1348547407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c811d920 msgr2=0x7f91c81a9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:05.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.057+0000 7f91c7400700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c811d920 0x7f91c81a9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.057+0000 7f91c7400700 1 -- 172.21.15.44:0/1348547407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c81aa250 con 0x7f91c810b270 2024-05-27T09:50:05.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.057+0000 7f91c7e00700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c810a920 0x7f91c81af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:05.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.057+0000 7f91c7400700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91c810b270 0x7f91c81afdf0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f91b400c970 tx=0x7f91b400cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:05.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.059+0000 7f91c4c00700 1 -- 172.21.15.44:0/1348547407 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b4004a30 con 0x7f91c810b270 2024-05-27T09:50:05.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.060+0000 7f91c4c00700 1 -- 172.21.15.44:0/1348547407 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91b40054d0 con 0x7f91c810b270 2024-05-27T09:50:05.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.060+0000 7f91c4c00700 1 -- 172.21.15.44:0/1348547407 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b400ebe0 con 0x7f91c810b270 2024-05-27T09:50:05.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.060+0000 7f91cea00700 1 -- 172.21.15.44:0/1348547407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91c81aa540 con 0x7f91c810b270 2024-05-27T09:50:05.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.060+0000 7f91cea00700 1 -- 172.21.15.44:0/1348547407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91c81b6740 con 0x7f91c810b270 2024-05-27T09:50:05.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.067+0000 7f91c4c00700 1 -- 172.21.15.44:0/1348547407 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f91b400f450 con 0x7f91c810b270 2024-05-27T09:50:05.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.069+0000 7f91dd5ffb80 1 Processor -- start 2024-05-27T09:50:05.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.069+0000 7f91dd5ffb80 1 -- start start 2024-05-27T09:50:05.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.070+0000 7f91c4c00700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91ac0b2a70 0x7f91ac0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.071+0000 7f91c4c00700 1 -- 172.21.15.44:0/1348547407 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f91b40e2410 con 0x7f91c810b270 2024-05-27T09:50:05.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.071+0000 7f91cca00700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91ac0b2a70 0x7f91ac0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.071+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 0x56522d1ac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.071+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d1ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.073+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1ace70 0x56522d1b9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.073+0000 7f91dd5ffb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56522d0dae40 con 0x56522d1ace70 2024-05-27T09:50:05.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.073+0000 7f91dd5ffb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56522d0ca9e0 con 0x56522d1199f0 2024-05-27T09:50:05.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.073+0000 7f91dd5ffb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56522d0cea20 con 0x56522d02de40 2024-05-27T09:50:05.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.074+0000 7f91c7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d1ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.074+0000 7f91c7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d1ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34906/0 (socket says 172.21.15.44:34906) 2024-05-27T09:50:05.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.074+0000 7f91c7400700 1 -- 172.21.15.44:0/1264978771 learned_addr learned my addr 172.21.15.44:0/1264978771 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:05.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.074+0000 7f91c7e00700 1 --2- 172.21.15.44:0/1264978771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 0x56522d1ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.074+0000 7f91c7400700 1 -- 172.21.15.44:0/1264978771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 msgr2=0x56522d1ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:05.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.074+0000 7f91c7400700 1 --2- 172.21.15.44:0/1264978771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 0x56522d1ac330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.074+0000 7f91c7400700 1 -- 172.21.15.44:0/1264978771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1ace70 msgr2=0x56522d1b9110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:50:05.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.075+0000 7f91cca00700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91ac0b2a70 0x7f91ac0b4f80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f91c8112b90 tx=0x7f91c000b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:05.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.075+0000 7f91c7400700 1 --2- 172.21.15.44:0/1264978771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1ace70 0x56522d1b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.075+0000 7f91c7400700 1 -- 172.21.15.44:0/1264978771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56522d093420 con 0x56522d02de40 2024-05-27T09:50:05.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.076+0000 7f91c7400700 1 --2- 172.21.15.44:0/1264978771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d1ac870 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f91b40a9600 tx=0x7f91b400e680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f7709cd026f6d05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:05.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.077+0000 7f91bd600700 1 -- 172.21.15.44:0/1264978771 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b401d040 con 0x56522d02de40 2024-05-27T09:50:05.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.079+0000 7f91bd600700 1 -- 172.21.15.44:0/1264978771 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91b40b24d0 con 0x56522d02de40 2024-05-27T09:50:05.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.079+0000 7f91bd600700 1 -- 172.21.15.44:0/1264978771 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b40910a0 con 0x56522d02de40 2024-05-27T09:50:05.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.080+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/1264978771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 msgr2=0x56522d1ac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:05.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.080+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/1264978771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d1ac870 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f91b40a9600 tx=0x7f91b400e680 comp rx=0 tx=0).stop 2024-05-27T09:50:05.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.081+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/1264978771 shutdown_connections 2024-05-27T09:50:05.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.081+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/1264978771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1ace70 0x56522d1b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.081+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/1264978771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 0x56522d1ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.081+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/1264978771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d1ac870 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.081+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/1264978771 >> 172.21.15.44:0/1264978771 conn(0x56522d0c7390 msgr2=0x56522d1a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:05.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.081+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/1264978771 shutdown_connections 2024-05-27T09:50:05.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.081+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/1264978771 wait complete. 2024-05-27T09:50:05.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.082+0000 7f91dd5ffb80 1 -- start start 2024-05-27T09:50:05.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.082+0000 7f91dd5ffb80 1 Processor -- start 2024-05-27T09:50:05.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.085+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d02de40 0x56522d1b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.086+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d1199f0 0x56522d1b6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.086+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 0x56522d1b7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.086+0000 7f91dd5ffb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56522cfd0fa0 con 0x56522d02de40 2024-05-27T09:50:05.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.086+0000 7f91dd5ffb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56522d06b760 con 0x56522d1ace70 2024-05-27T09:50:05.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.086+0000 7f91dd5ffb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56522d1b9b20 con 0x56522d1199f0 2024-05-27T09:50:05.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.086+0000 7f91dd5ffb80 10 client.0 _ll_register_callbacks cb 0x56522d074c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:05.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.086+0000 7f91cca00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 0x56522d1b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.086+0000 7f91cca00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 0x56522d1b7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:46836/0 (socket says 172.21.15.44:46836) 2024-05-27T09:50:05.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.086+0000 7f91cca00700 1 -- 172.21.15.44:0/3181154243 learned_addr learned my addr 172.21.15.44:0/3181154243 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:05.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.086+0000 7f91c7400700 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d1199f0 0x56522d1b6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.087+0000 7f91c7e00700 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d02de40 0x56522d1b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.087+0000 7f91cca00700 1 -- 172.21.15.44:0/3181154243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d1199f0 msgr2=0x56522d1b6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:05.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.087+0000 7f91cca00700 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d1199f0 0x56522d1b6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.087+0000 7f91cca00700 1 -- 172.21.15.44:0/3181154243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d02de40 msgr2=0x56522d1b66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:05.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.087+0000 7f91cca00700 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d02de40 0x56522d1b66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.088+0000 7f91cca00700 1 -- 172.21.15.44:0/3181154243 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56522d01a8d0 con 0x56522d1ace70 2024-05-27T09:50:05.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.088+0000 7f91c7e00700 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d02de40 0x56522d1b66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:50:05.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.088+0000 7f91cca00700 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 0x56522d1b7160 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f91c0017e10 tx=0x7f91c00099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bde138da23350e9f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:05.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.088+0000 7f91b2000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:05.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.089+0000 7f91b2000700 1 -- 172.21.15.44:0/3181154243 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c0025070 con 0x56522d1ace70 2024-05-27T09:50:05.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.089+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/3181154243 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91b40b24d0 con 0x56522d1ace70 2024-05-27T09:50:05.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.090+0000 7f91b2000700 1 -- 172.21.15.44:0/3181154243 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f91c0015db0 con 0x56522d1ace70 2024-05-27T09:50:05.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.091+0000 7f91b2000700 1 -- 172.21.15.44:0/3181154243 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f91c0030050 con 0x56522d1ace70 2024-05-27T09:50:05.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.091+0000 7f91dd5ffb80 10 client.28103 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:05.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.091+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/3181154243 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56522d09b060 con 0x56522d1ace70 2024-05-27T09:50:05.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.092+0000 7f91b2000700 1 -- 172.21.15.44:0/3181154243 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91b40b24d0 con 0x56522d1ace70 2024-05-27T09:50:05.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.092+0000 7f91b2000700 1 -- 172.21.15.44:0/3181154243 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f91c0020070 con 0x56522d1ace70 2024-05-27T09:50:05.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.092+0000 7f91dd5ffb80 10 client.28103 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:05.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.092+0000 7f91dd5ffb80 10 client.28103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:05.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.092+0000 7f91dd5ffb80 20 client.28103 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:05.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.092+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56522d1c6b00 0x56522d1c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.092+0000 7f91dd5ffb80 4 client.28103 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:05.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.092+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/3181154243 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56522d0dcd90 con 0x56522d1c6b00 2024-05-27T09:50:05.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.094+0000 7f91c7e00700 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56522d1c6b00 0x56522d1c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.095+0000 7f91c7e00700 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56522d1c6b00 0x56522d1c8fd0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65c5c12f1d069539 server_cookie=3ea134e218a9f484 in_seq=0 out_seq=0 2024-05-27T09:50:05.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.095+0000 7f91b2000700 10 client.28103 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:05.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.097+0000 7f91b2000700 1 -- 172.21.15.44:0/3181154243 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56522d0dcd90 con 0x56522d1c6b00 2024-05-27T09:50:05.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.097+0000 7f91b2000700 10 client.28103 handle_command_reply: tid=0 2024-05-27T09:50:05.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.097+0000 7f91dd5ffb80 1 client.28103 shutdown 2024-05-27T09:50:05.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.098+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/3181154243 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 msgr2=0x56522d1b7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:05.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.098+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 0x56522d1b7160 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f91c0017e10 tx=0x7f91c00099e0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.098+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/3181154243 shutdown_connections 2024-05-27T09:50:05.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.098+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56522d1c6b00 0x56522d1c8fd0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.098+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 0x56522d1b7160 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.098+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d02de40 0x56522d1b66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.098+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/3181154243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d1199f0 0x56522d1b6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.098+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/3181154243 >> 172.21.15.44:0/3181154243 conn(0x56522d0c7390 msgr2=0x56522d1ac350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:05.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.098+0000 7f91b2000700 0 client.28103 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:05.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.099+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/3181154243 shutdown_connections 2024-05-27T09:50:05.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.099+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/3181154243 wait complete. 2024-05-27T09:50:05.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.099+0000 7f91dd5ffb80 20 client.28103 trim_cache size 0 max 16384 2024-05-27T09:50:05.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.144+0000 7f91dd5ffb80 1 Processor -- start 2024-05-27T09:50:05.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.144+0000 7f91dd5ffb80 1 -- start start 2024-05-27T09:50:05.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.144+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d1b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.144+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1199f0 0x56522d1b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.145+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 0x56522d1c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.145+0000 7f91dd5ffb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f91c81c08d0 con 0x56522d1199f0 2024-05-27T09:50:05.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.145+0000 7f91dd5ffb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f91c81be100 con 0x56522d1ace70 2024-05-27T09:50:05.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.145+0000 7f91dd5ffb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f91c81bbf70 con 0x56522d02de40 2024-05-27T09:50:05.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.145+0000 7f91c7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d1b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.145+0000 7f91c7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d1b1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34918/0 (socket says 172.21.15.44:34918) 2024-05-27T09:50:05.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.145+0000 7f91c7e00700 1 -- 172.21.15.44:0/1003510644 learned_addr learned my addr 172.21.15.44:0/1003510644 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:05.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.145+0000 7f91c7400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1199f0 0x56522d1b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.146+0000 7f91cca00700 1 --2- 172.21.15.44:0/1003510644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 0x56522d1c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.146+0000 7f91cca00700 1 -- 172.21.15.44:0/1003510644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 msgr2=0x56522d1b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.146+0000 7f91cca00700 1 --2- 172.21.15.44:0/1003510644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d1b1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.146+0000 7f91cca00700 1 -- 172.21.15.44:0/1003510644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1199f0 msgr2=0x56522d1b87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.146+0000 7f91cca00700 1 --2- 172.21.15.44:0/1003510644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1199f0 0x56522d1b87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.146+0000 7f91cca00700 1 -- 172.21.15.44:0/1003510644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56522d1a4240 con 0x56522d1ace70 2024-05-27T09:50:05.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.147+0000 7f91c7400700 1 --2- 172.21.15.44:0/1003510644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1199f0 0x56522d1b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:05.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.147+0000 7f91cca00700 1 --2- 172.21.15.44:0/1003510644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 0x56522d1c2bc0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f91c000fc60 tx=0x7f91c00216d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=772c565ca3040eb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:05.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.151+0000 7f91bcc00700 1 -- 172.21.15.44:0/1003510644 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c0025040 con 0x56522d1ace70 2024-05-27T09:50:05.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.151+0000 7f91bcc00700 1 -- 172.21.15.44:0/1003510644 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91c001b070 con 0x56522d1ace70 2024-05-27T09:50:05.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.151+0000 7f91bcc00700 1 -- 172.21.15.44:0/1003510644 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c0009330 con 0x56522d1ace70 2024-05-27T09:50:05.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.153+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/1003510644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 msgr2=0x56522d1c2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.153+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/1003510644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 0x56522d1c2bc0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f91c000fc60 tx=0x7f91c00216d0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.154+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/1003510644 shutdown_connections 2024-05-27T09:50:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.154+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/1003510644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1ace70 0x56522d1c2bc0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.154+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/1003510644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d1b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.154+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/1003510644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1199f0 0x56522d1b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.154+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/1003510644 >> 172.21.15.44:0/1003510644 conn(0x56522d0c7390 msgr2=0x56522d1b6d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.154+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/1003510644 shutdown_connections 2024-05-27T09:50:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.155+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/1003510644 wait complete. 2024-05-27T09:50:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.155+0000 7f91dd5ffb80 1 -- start start 2024-05-27T09:50:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.156+0000 7f91dd5ffb80 1 Processor -- start 2024-05-27T09:50:05.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.160+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d22eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.160+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 0x56522d22f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.160+0000 7f91dd5ffb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1ace70 0x56522d1bee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.161+0000 7f91c7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d22eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.161+0000 7f91c7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d22eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34926/0 (socket says 172.21.15.44:34926) 2024-05-27T09:50:05.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.161+0000 7f91c7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 0x56522d22f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.161+0000 7f91c7400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 0x56522d22f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:46854/0 (socket says 172.21.15.44:46854) 2024-05-27T09:50:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.161+0000 7f91c7e00700 1 -- 172.21.15.44:0/2092840161 learned_addr learned my addr 172.21.15.44:0/2092840161 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.161+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/2092840161 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f91c81bc5c0 con 0x56522d1ace70 2024-05-27T09:50:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.161+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/2092840161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f91c81bda30 con 0x56522d1199f0 2024-05-27T09:50:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.161+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/2092840161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f91c81bcc90 con 0x56522d02de40 2024-05-27T09:50:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.161+0000 7f91cca00700 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1ace70 0x56522d1bee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.162+0000 7f91dd5ffb80 10 client.0 _ll_register_callbacks cb 0x56522d074c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.162+0000 7f91cca00700 1 -- 172.21.15.44:0/2092840161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 msgr2=0x56522d22eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.162+0000 7f91cca00700 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d22eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.163+0000 7f91cca00700 1 -- 172.21.15.44:0/2092840161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 msgr2=0x56522d22f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.163+0000 7f91cca00700 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 0x56522d22f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.163+0000 7f91cca00700 1 -- 172.21.15.44:0/2092840161 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56522d1bf5c0 con 0x56522d1ace70 2024-05-27T09:50:05.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.164+0000 7f91c7400700 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 0x56522d22f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:05.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.164+0000 7f91cca00700 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1ace70 0x56522d1bee20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f91c0008ee0 tx=0x7f91c0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae911e7eb8a3880 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:05.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.164+0000 7f91b2000700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:50:05.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.165+0000 7f91b2000700 1 -- 172.21.15.44:0/2092840161 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c0040020 con 0x56522d1ace70 2024-05-27T09:50:05.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.165+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/2092840161 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91c001b070 con 0x56522d1ace70 2024-05-27T09:50:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.166+0000 7f91b2000700 1 -- 172.21.15.44:0/2092840161 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f91c0015d10 con 0x56522d1ace70 2024-05-27T09:50:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.166+0000 7f91b2000700 1 -- 172.21.15.44:0/2092840161 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f91c0025040 con 0x56522d1ace70 2024-05-27T09:50:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.166+0000 7f91dd5ffb80 10 client.17838 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.167+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/2092840161 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56522d1b1810 con 0x56522d1ace70 2024-05-27T09:50:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.167+0000 7f91b2000700 1 -- 172.21.15.44:0/2092840161 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91c001b070 con 0x56522d1ace70 2024-05-27T09:50:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.167+0000 7f91b2000700 1 -- 172.21.15.44:0/2092840161 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f91c0020040 con 0x56522d1ace70 2024-05-27T09:50:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.168+0000 7f91dd5ffb80 10 client.17838 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.168+0000 7f91dd5ffb80 10 client.17838 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.168+0000 7f91dd5ffb80 20 client.17838 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.168+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56522d2429a0 0x56522d244dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:05.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.168+0000 7f91dd5ffb80 4 client.17838 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:50:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.168+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/2092840161 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56522d0d6570 con 0x56522d2429a0 2024-05-27T09:50:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.168+0000 7f91c7e00700 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56522d2429a0 0x56522d244dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.170+0000 7f91c7e00700 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56522d2429a0 0x56522d244dc0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9585f3d4c4acb96 server_cookie=f421f58772791c39 in_seq=0 out_seq=0 2024-05-27T09:50:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.170+0000 7f91b2000700 10 client.17838 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.170+0000 7f91b2000700 1 -- 172.21.15.44:0/2092840161 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56522d0d6570 con 0x56522d2429a0 2024-05-27T09:50:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.171+0000 7f91b2000700 10 client.17838 handle_command_reply: tid=0 2024-05-27T09:50:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.171+0000 7f91dd5ffb80 1 client.17838 shutdown 2024-05-27T09:50:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.171+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/2092840161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1ace70 msgr2=0x56522d1bee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.171+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1ace70 0x56522d1bee20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f91c0008ee0 tx=0x7f91c0005fd0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.171+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/2092840161 shutdown_connections 2024-05-27T09:50:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.171+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x56522d2429a0 0x56522d244dc0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.171+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56522d1ace70 0x56522d1bee20 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.172+0000 7f91b2000700 0 client.17838 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.172+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56522d02de40 0x56522d22eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.172+0000 7f91dd5ffb80 1 --2- 172.21.15.44:0/2092840161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56522d1199f0 0x56522d22f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.172+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/2092840161 >> 172.21.15.44:0/2092840161 conn(0x56522d0c7390 msgr2=0x56522d1aded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.172+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/2092840161 shutdown_connections 2024-05-27T09:50:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.172+0000 7f91dd5ffb80 1 -- 172.21.15.44:0/2092840161 wait complete. 2024-05-27T09:50:05.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.172+0000 7f91dd5ffb80 20 client.17838 trim_cache size 0 max 16384 2024-05-27T09:50:05.174 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:50:05.174 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:50:05.174 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:50:05.174 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:50:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.173+0000 7f91cea00700 1 -- 172.21.15.44:0/1348547407 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91ac0b2a70 msgr2=0x7f91ac0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.173+0000 7f91cea00700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91ac0b2a70 0x7f91ac0b4f80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f91c8112b90 tx=0x7f91c000b040 comp rx=0 tx=0).stop 2024-05-27T09:50:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.173+0000 7f91cea00700 1 -- 172.21.15.44:0/1348547407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91c810b270 msgr2=0x7f91c81afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.173+0000 7f91cea00700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91c810b270 0x7f91c81afdf0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f91b400c970 tx=0x7f91b400cc80 comp rx=0 tx=0).stop 2024-05-27T09:50:05.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.175+0000 7f91cea00700 1 -- 172.21.15.44:0/1348547407 shutdown_connections 2024-05-27T09:50:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.175+0000 7f91cea00700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f91ac0b2a70 0x7f91ac0b4f80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.175+0000 7f91cea00700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c811d920 0x7f91c81a9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.175+0000 7f91cea00700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c810a920 0x7f91c81af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.175+0000 7f91cea00700 1 --2- 172.21.15.44:0/1348547407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91c810b270 0x7f91c81afdf0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.175+0000 7f91cea00700 1 -- 172.21.15.44:0/1348547407 >> 172.21.15.44:0/1348547407 conn(0x7f91c806f4c0 msgr2=0x7f91c811b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.176+0000 7f91cea00700 1 -- 172.21.15.44:0/1348547407 shutdown_connections 2024-05-27T09:50:05.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:05.176+0000 7f91cea00700 1 -- 172.21.15.44:0/1348547407 wait complete. 2024-05-27T09:50:06.170 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:05 smithi044 ceph-mon[31618]: pgmap v468: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 13 op/s 2024-05-27T09:50:06.234 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:50:06.234 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:50:06.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:05 smithi050 ceph-mon[36371]: pgmap v468: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 13 op/s 2024-05-27T09:50:06.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:05 smithi189 ceph-mon[36671]: pgmap v468: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 13 op/s 2024-05-27T09:50:06.890 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:07.099 INFO:tasks.workunit.client.0.smithi044.stderr:+ mkdir -p dir 2024-05-27T09:50:07.106 INFO:tasks.workunit.client.0.smithi044.stderr:+ getfattr -n ceph.dir.pin dir 2024-05-27T09:50:07.106 INFO:tasks.workunit.client.0.smithi044.stderr:+ grep 'ceph.dir.pin="-1"' 2024-05-27T09:50:07.110 INFO:tasks.workunit.client.0.smithi044.stdout:ceph.dir.pin="-1" 2024-05-27T09:50:07.112 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n ceph.dir.pin dir 2024-05-27T09:50:07.112 INFO:tasks.workunit.client.0.smithi044.stderr:+ grep 'setfattr: dir: Invalid argument' 2024-05-27T09:50:07.115 INFO:tasks.workunit.client.0.smithi044.stdout:setfattr: dir: Invalid argument 2024-05-27T09:50:07.116 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -n ceph.dir.pin -v 1 dir 2024-05-27T09:50:07.123 INFO:tasks.workunit.client.0.smithi044.stderr:+ getfattr -n ceph.dir.pin dir 2024-05-27T09:50:07.124 INFO:tasks.workunit.client.0.smithi044.stderr:+ grep 'ceph.dir.pin="1"' 2024-05-27T09:50:07.131 INFO:tasks.workunit.client.0.smithi044.stdout:ceph.dir.pin="1" 2024-05-27T09:50:07.132 INFO:tasks.workunit.client.0.smithi044.stderr:+ setfattr -x ceph.dir.pin dir 2024-05-27T09:50:07.139 INFO:tasks.workunit.client.0.smithi044.stderr:+ getfattr -n ceph.dir.pin dir 2024-05-27T09:50:07.139 INFO:tasks.workunit.client.0.smithi044.stderr:+ grep 'ceph.dir.pin="-1"' 2024-05-27T09:50:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:50:07.147 INFO:tasks.workunit:Stopping ['fs/misc'] on client.0... 2024-05-27T09:50:07.147 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-27T09:50:08.108 INFO:tasks.quiescer.fs.[cephfs]:Total duration (45.0) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-05-27T09:50:08.108 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.5 2024-05-27T09:50:08.108 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:50:08.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:07 smithi044 ceph-mon[31618]: pgmap v469: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 72 KiB/s wr, 33 op/s 2024-05-27T09:50:08.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:07 smithi050 ceph-mon[36371]: pgmap v469: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 72 KiB/s wr, 33 op/s 2024-05-27T09:50:08.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:07 smithi189 ceph-mon[36671]: pgmap v469: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 72 KiB/s wr, 33 op/s 2024-05-27T09:50:09.010 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:10.086 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:09 smithi044 ceph-mon[31618]: pgmap v470: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 72 KiB/s wr, 33 op/s 2024-05-27T09:50:10.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:09 smithi050 ceph-mon[36371]: pgmap v470: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 72 KiB/s wr, 33 op/s 2024-05-27T09:50:10.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:09 smithi189 ceph-mon[36671]: pgmap v470: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 72 KiB/s wr, 33 op/s 2024-05-27T09:50:10.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.084+0000 7fb0d3800700 1 -- 172.21.15.44:0/3667156623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc103240 msgr2=0x7fb0cc103650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:10.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.085+0000 7fb0d3800700 1 --2- 172.21.15.44:0/3667156623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc103240 0x7fb0cc103650 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00098f0 tx=0x7fb0c0009c00 comp rx=0 tx=0).stop 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.085+0000 7fb0d3800700 1 -- 172.21.15.44:0/3667156623 shutdown_connections 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.085+0000 7fb0d3800700 1 --2- 172.21.15.44:0/3667156623 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0cc108690 0x7fb0cc108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.085+0000 7fb0d3800700 1 --2- 172.21.15.44:0/3667156623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc103240 0x7fb0cc103650 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.085+0000 7fb0d3800700 1 --2- 172.21.15.44:0/3667156623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc103b90 0x7fb0cc1080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.085+0000 7fb0d3800700 1 -- 172.21.15.44:0/3667156623 >> 172.21.15.44:0/3667156623 conn(0x7fb0cc0fedb0 msgr2=0x7fb0cc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.086+0000 7fb0d3800700 1 -- 172.21.15.44:0/3667156623 shutdown_connections 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.086+0000 7fb0d3800700 1 -- 172.21.15.44:0/3667156623 wait complete. 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.091+0000 7fb0d3800700 1 Processor -- start 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.091+0000 7fb0d3800700 1 -- start start 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.093+0000 7fb0d3800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc103240 0x7fb0cc117430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.093+0000 7fb0d3800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc103b90 0x7fb0cc117990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:10.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.093+0000 7fb0d3800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0cc108690 0x7fb0cc111700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:10.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.093+0000 7fb0d3800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc109a20 con 0x7fb0cc108690 2024-05-27T09:50:10.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.093+0000 7fb0d3800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc1098a0 con 0x7fb0cc103b90 2024-05-27T09:50:10.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.093+0000 7fb0d3800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc109ba0 con 0x7fb0cc103240 2024-05-27T09:50:10.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.093+0000 7fb0d0e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc103240 0x7fb0cc117430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:10.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.094+0000 7fb0d0e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc103240 0x7fb0cc117430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:34942/0 (socket says 172.21.15.44:34942) 2024-05-27T09:50:10.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.094+0000 7fb0d0e00700 1 -- 172.21.15.44:0/799567742 learned_addr learned my addr 172.21.15.44:0/799567742 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:10.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.094+0000 7fb0cbe00700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc103b90 0x7fb0cc117990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:10.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.094+0000 7fb0d0e00700 1 -- 172.21.15.44:0/799567742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc103b90 msgr2=0x7fb0cc117990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:10.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.094+0000 7fb0d0e00700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc103b90 0x7fb0cc117990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:10.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.095+0000 7fb0d0e00700 1 -- 172.21.15.44:0/799567742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0cc108690 msgr2=0x7fb0cc111700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:10.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.095+0000 7fb0d0e00700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0cc108690 0x7fb0cc111700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.095+0000 7fb0d0e00700 1 -- 172.21.15.44:0/799567742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0cc111f30 con 0x7fb0cc103240 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.095+0000 7fb0d0e00700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc103240 0x7fb0cc117430 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009230 tx=0x7fb0c0005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.096+0000 7fb0c9600700 1 -- 172.21.15.44:0/799567742 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0027070 con 0x7fb0cc103240 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.096+0000 7fb0c9600700 1 -- 172.21.15.44:0/799567742 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0c000b560 con 0x7fb0cc103240 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.096+0000 7fb0c9600700 1 -- 172.21.15.44:0/799567742 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0004b10 con 0x7fb0cc103240 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.097+0000 7fb0d3800700 1 -- 172.21.15.44:0/799567742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0cc1121c0 con 0x7fb0cc103240 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.097+0000 7fb0d3800700 1 -- 172.21.15.44:0/799567742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0cc1b65b0 con 0x7fb0cc103240 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.099+0000 7fb0d3800700 1 -- 172.21.15.44:0/799567742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0cc06bee0 con 0x7fb0cc103240 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.104+0000 7fb0c9600700 1 -- 172.21.15.44:0/799567742 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb0c002a070 con 0x7fb0cc103240 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.106+0000 7fb0c9600700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0b40b2b40 0x7fb0b40b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.107+0000 7fb0c9600700 1 -- 172.21.15.44:0/799567742 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7fb0c00e3030 con 0x7fb0cc103240 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.107+0000 7fb0cbe00700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0b40b2b40 0x7fb0b40b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:10.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.109+0000 7fb0cbe00700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0b40b2b40 0x7fb0b40b5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc108b90 tx=0x7fb0b8009310 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:10.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.109+0000 7fb0c9600700 1 -- 172.21.15.44:0/799567742 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb0c0013030 con 0x7fb0cc103240 2024-05-27T09:50:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.669+0000 7fb0d3800700 1 -- 172.21.15.44:0/799567742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0cc0512b0 con 0x7fb0cc103240 2024-05-27T09:50:10.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.670+0000 7fb0c9600700 1 -- 172.21.15.44:0/799567742 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11870 (secure 0 0 0) 0x7fb0cc0512b0 con 0x7fb0cc103240 2024-05-27T09:50:10.671 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:50:10.674 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:48:16.864365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24503]},"id":1}]} 2024-05-27T09:50:10.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.674+0000 7fb0d3800700 1 -- 172.21.15.44:0/799567742 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0b40b2b40 msgr2=0x7fb0b40b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.674+0000 7fb0d3800700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0b40b2b40 0x7fb0b40b5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc108b90 tx=0x7fb0b8009310 comp rx=0 tx=0).stop 2024-05-27T09:50:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.674+0000 7fb0d3800700 1 -- 172.21.15.44:0/799567742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc103240 msgr2=0x7fb0cc117430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.674+0000 7fb0d3800700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc103240 0x7fb0cc117430 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009230 tx=0x7fb0c0005040 comp rx=0 tx=0).stop 2024-05-27T09:50:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.675+0000 7fb0d3800700 1 -- 172.21.15.44:0/799567742 shutdown_connections 2024-05-27T09:50:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.675+0000 7fb0d3800700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb0b40b2b40 0x7fb0b40b5050 secure :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc108b90 tx=0x7fb0b8009310 comp rx=0 tx=0).stop 2024-05-27T09:50:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.675+0000 7fb0d3800700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0cc108690 0x7fb0cc111700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.675+0000 7fb0d3800700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc103240 0x7fb0cc117430 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.675+0000 7fb0d3800700 1 --2- 172.21.15.44:0/799567742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc103b90 0x7fb0cc117990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:10.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.675+0000 7fb0d3800700 1 -- 172.21.15.44:0/799567742 >> 172.21.15.44:0/799567742 conn(0x7fb0cc0fedb0 msgr2=0x7fb0cc105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.676+0000 7fb0d3800700 1 -- 172.21.15.44:0/799567742 shutdown_connections 2024-05-27T09:50:10.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:10.676+0000 7fb0d3800700 1 -- 172.21.15.44:0/799567742 wait complete. 2024-05-27T09:50:10.678 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 43 2024-05-27T09:50:11.243 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:10 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/799567742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:11.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:10 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/799567742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:11.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:10 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/799567742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.143+0000 7f5b84200640 1 -- 172.21.15.44:0/732955653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c109030 msgr2=0x7f5b7c102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.144+0000 7f5b84200640 1 --2- 172.21.15.44:0/732955653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c109030 0x7f5b7c102c70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009ca0 tx=0x7f5b7002a8c0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.144+0000 7f5b84200640 1 -- 172.21.15.44:0/732955653 shutdown_connections 2024-05-27T09:50:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.144+0000 7f5b84200640 1 --2- 172.21.15.44:0/732955653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b7c1032e0 0x7f5b7c06a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.144+0000 7f5b84200640 1 --2- 172.21.15.44:0/732955653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c109030 0x7f5b7c102c70 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.144+0000 7f5b84200640 1 --2- 172.21.15.44:0/732955653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b7c108b90 0x7f5b7c102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.144+0000 7f5b84200640 1 -- 172.21.15.44:0/732955653 >> 172.21.15.44:0/732955653 conn(0x7f5b7c0782d0 msgr2=0x7f5b7c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.146+0000 7f5b84200640 1 -- 172.21.15.44:0/732955653 shutdown_connections 2024-05-27T09:50:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.146+0000 7f5b84200640 1 -- 172.21.15.44:0/732955653 wait complete. 2024-05-27T09:50:12.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.148+0000 7f5b84200640 1 Processor -- start 2024-05-27T09:50:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.149+0000 7f5b84200640 1 -- start start 2024-05-27T09:50:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.150+0000 7f5b84200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c1032e0 0x7f5b7c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.150+0000 7f5b84200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b7c108b90 0x7f5b7c198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.150+0000 7f5b84200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b7c109030 0x7f5b7c19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.150+0000 7f5b84200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c111d20 con 0x7f5b7c109030 2024-05-27T09:50:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.150+0000 7f5b84200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c111ba0 con 0x7f5b7c108b90 2024-05-27T09:50:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.150+0000 7f5b84200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c111ea0 con 0x7f5b7c1032e0 2024-05-27T09:50:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.151+0000 7f5b82200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b7c109030 0x7f5b7c19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.151+0000 7f5b82200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b7c109030 0x7f5b7c19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53332/0 (socket says 172.21.15.44:53332) 2024-05-27T09:50:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.151+0000 7f5b82200640 1 -- 172.21.15.44:0/60295831 learned_addr learned my addr 172.21.15.44:0/60295831 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.151+0000 7f5b80e00640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b7c108b90 0x7f5b7c198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.151+0000 7f5b81800640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c1032e0 0x7f5b7c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.152+0000 7f5b81800640 1 -- 172.21.15.44:0/60295831 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b7c108b90 msgr2=0x7f5b7c198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.152+0000 7f5b81800640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b7c108b90 0x7f5b7c198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.152+0000 7f5b81800640 1 -- 172.21.15.44:0/60295831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b7c109030 msgr2=0x7f5b7c19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.152+0000 7f5b81800640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b7c109030 0x7f5b7c19cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.152+0000 7f5b81800640 1 -- 172.21.15.44:0/60295831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b7c19d5e0 con 0x7f5b7c1032e0 2024-05-27T09:50:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.152+0000 7f5b80e00640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b7c108b90 0x7f5b7c198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.152+0000 7f5b81800640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c1032e0 0x7f5b7c198780 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400b3c0 tx=0x7f5b7400b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.154+0000 7f5b7a000640 1 -- 172.21.15.44:0/60295831 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74011830 con 0x7f5b7c1032e0 2024-05-27T09:50:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.154+0000 7f5b7a000640 1 -- 172.21.15.44:0/60295831 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b74016070 con 0x7f5b7c1032e0 2024-05-27T09:50:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.154+0000 7f5b7a000640 1 -- 172.21.15.44:0/60295831 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74010e20 con 0x7f5b7c1032e0 2024-05-27T09:50:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.154+0000 7f5b84200640 1 -- 172.21.15.44:0/60295831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b7c19d8d0 con 0x7f5b7c1032e0 2024-05-27T09:50:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.154+0000 7f5b84200640 1 -- 172.21.15.44:0/60295831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b7c19dda0 con 0x7f5b7c1032e0 2024-05-27T09:50:12.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.160+0000 7f5b8b1fa740 1 Processor -- start 2024-05-27T09:50:12.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.160+0000 7f5b8b1fa740 1 -- start start 2024-05-27T09:50:12.160 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.160+0000 7f5b7a000640 1 -- 172.21.15.44:0/60295831 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5b740104a0 con 0x7f5b7c1032e0 2024-05-27T09:50:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.161+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 0x55f1903ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.163+0000 7f5b7a000640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b640b29d0 0x7f5b640b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.164+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 0x55f1903adda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.164+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffea4d0 0x55f1903ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.164+0000 7f5b8b1fa740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f19031eb90 con 0x55f18ffe22f0 2024-05-27T09:50:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.164+0000 7f5b8b1fa740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f19030e730 con 0x55f19000b1b0 2024-05-27T09:50:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.164+0000 7f5b8b1fa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f190312770 con 0x55f18ffea4d0 2024-05-27T09:50:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.164+0000 7f5b81800640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b640b29d0 0x7f5b640b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.164+0000 7f5b82200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 0x55f1903ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.164+0000 7f5b82200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 0x55f1903ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:46908/0 (socket says 172.21.15.44:46908) 2024-05-27T09:50:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.164+0000 7f5b82200640 1 -- 172.21.15.44:0/224221187 learned_addr learned my addr 172.21.15.44:0/224221187 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.165+0000 7f5b82200640 1 --2- 172.21.15.44:0/224221187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffea4d0 0x55f1903ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.165+0000 7f5b82200640 1 -- 172.21.15.44:0/224221187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffea4d0 msgr2=0x55f1903ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.165+0000 7f5b82200640 1 --2- 172.21.15.44:0/224221187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffea4d0 0x55f1903ae2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.165+0000 7f5b82200640 1 -- 172.21.15.44:0/224221187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 msgr2=0x55f1903adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.165+0000 7f5b80e00640 1 --2- 172.21.15.44:0/224221187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 0x55f1903adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.165+0000 7f5b82200640 1 --2- 172.21.15.44:0/224221187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 0x55f1903adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.165+0000 7f5b82200640 1 -- 172.21.15.44:0/224221187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1902a78b0 con 0x55f19000b1b0 2024-05-27T09:50:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.165+0000 7f5b80e00640 1 --2- 172.21.15.44:0/224221187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 0x55f1903adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:50:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.166+0000 7f5b82200640 1 --2- 172.21.15.44:0/224221187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 0x55f1903ad860 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f5b68010890 tx=0x7f5b68005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bfa4a4f0a613c5e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.166+0000 7f5b6ea00640 1 -- 172.21.15.44:0/224221187 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b68010e10 con 0x55f19000b1b0 2024-05-27T09:50:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.166+0000 7f5b6ea00640 1 -- 172.21.15.44:0/224221187 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b6800bc00 con 0x55f19000b1b0 2024-05-27T09:50:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.168+0000 7f5b6ea00640 1 -- 172.21.15.44:0/224221187 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6801f980 con 0x55f19000b1b0 2024-05-27T09:50:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.168+0000 7f5b7a000640 1 -- 172.21.15.44:0/60295831 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f5b740e1a50 con 0x7f5b7c1032e0 2024-05-27T09:50:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.169+0000 7f5b81800640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b640b29d0 0x7f5b640b4ee0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5b740a5fd0 tx=0x7f5b740a8b30 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.169+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/224221187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 msgr2=0x55f1903ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.169+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/224221187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 0x55f1903ad860 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f5b68010890 tx=0x7f5b68005f90 comp rx=0 tx=0).stop 2024-05-27T09:50:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.169+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/224221187 shutdown_connections 2024-05-27T09:50:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.169+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/224221187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffea4d0 0x55f1903ae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.169+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/224221187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 0x55f1903adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.169+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/224221187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 0x55f1903ad860 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.169+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/224221187 >> 172.21.15.44:0/224221187 conn(0x55f18ffff300 msgr2=0x55f1903aa010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.169+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/224221187 shutdown_connections 2024-05-27T09:50:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.169+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/224221187 wait complete. 2024-05-27T09:50:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.170+0000 7f5b8b1fa740 1 -- start start 2024-05-27T09:50:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.170+0000 7f5b8b1fa740 1 Processor -- start 2024-05-27T09:50:12.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.172+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 0x55f1903b79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.172 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.172+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffea4d0 0x55f1903b7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.173+0000 7f5b82200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 0x55f1903b79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.173+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f19000b1b0 0x55f1903c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.173+0000 7f5b8b1fa740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f18fff4790 con 0x55f18ffe22f0 2024-05-27T09:50:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.173+0000 7f5b82200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 0x55f1903b79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53340/0 (socket says 172.21.15.44:53340) 2024-05-27T09:50:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.174+0000 7f5b82200640 1 -- 172.21.15.44:0/1578136671 learned_addr learned my addr 172.21.15.44:0/1578136671 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:12.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.174+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/1578136671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f1902b6bc0 con 0x55f18ffea4d0 2024-05-27T09:50:12.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.174+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/1578136671 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f1903baa30 con 0x55f19000b1b0 2024-05-27T09:50:12.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.174+0000 7f5b8b1fa740 10 client.0 _ll_register_callbacks cb 0x55f190294400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:12.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.174+0000 7f5b80e00640 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffea4d0 0x55f1903b7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.175 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.175+0000 7f5b82200640 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f19000b1b0 0x55f1903c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.175+0000 7f5b80e00640 1 -- 172.21.15.44:0/1578136671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f19000b1b0 msgr2=0x55f1903c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.175+0000 7f5b80e00640 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f19000b1b0 0x55f1903c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.175+0000 7f5b80e00640 1 -- 172.21.15.44:0/1578136671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 msgr2=0x55f1903b79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.175+0000 7f5b80e00640 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 0x55f1903b79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.175+0000 7f5b80e00640 1 -- 172.21.15.44:0/1578136671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f190023210 con 0x55f18ffea4d0 2024-05-27T09:50:12.177 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.177+0000 7f5b80e00640 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffea4d0 0x55f1903b7ef0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5b7002ade0 tx=0x7f5b70009420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bf312dbf6eb6c82 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:12.177 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.177+0000 7f5b63400640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.178+0000 7f5b63400640 1 -- 172.21.15.44:0/1578136671 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7004a060 con 0x55f18ffea4d0 2024-05-27T09:50:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.179+0000 7f5b63400640 1 -- 172.21.15.44:0/1578136671 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f5b70003b80 con 0x55f18ffea4d0 2024-05-27T09:50:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.179+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/1578136671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b6800bc00 con 0x55f18ffea4d0 2024-05-27T09:50:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.180+0000 7f5b63400640 1 -- 172.21.15.44:0/1578136671 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f5b70032eb0 con 0x55f18ffea4d0 2024-05-27T09:50:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.180+0000 7f5b8b1fa740 10 client.28124 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.180+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/1578136671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55f1900ef620 con 0x55f18ffea4d0 2024-05-27T09:50:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.181+0000 7f5b63400640 1 -- 172.21.15.44:0/1578136671 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b70036050 con 0x55f18ffea4d0 2024-05-27T09:50:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.182+0000 7f5b63400640 1 -- 172.21.15.44:0/1578136671 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f5b7002f880 con 0x55f18ffea4d0 2024-05-27T09:50:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.182+0000 7f5b8b1fa740 10 client.28124 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.182+0000 7f5b8b1fa740 10 client.28124 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.182+0000 7f5b8b1fa740 20 client.28124 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.182+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f18ffc9d80 0x55f1903c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.182+0000 7f5b8b1fa740 4 client.28124 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.182+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/1578136671 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5b6800bc00 con 0x55f18ffc9d80 2024-05-27T09:50:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.183+0000 7f5b82200640 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f18ffc9d80 0x55f1903c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.183+0000 7f5b82200640 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f18ffc9d80 0x55f1903c9bc0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d59aab13e443982 server_cookie=b5d45a08b977694e in_seq=0 out_seq=0 2024-05-27T09:50:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.183+0000 7f5b63400640 10 client.28124 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.185+0000 7f5b63400640 1 -- 172.21.15.44:0/1578136671 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f5b6800bc00 con 0x55f18ffc9d80 2024-05-27T09:50:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.185+0000 7f5b63400640 10 client.28124 handle_command_reply: tid=0 2024-05-27T09:50:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.185+0000 7f5b8b1fa740 1 client.28124 shutdown 2024-05-27T09:50:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.186+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/1578136671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffea4d0 msgr2=0x55f1903b7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.186+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffea4d0 0x55f1903b7ef0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5b7002ade0 tx=0x7f5b70009420 comp rx=0 tx=0).stop 2024-05-27T09:50:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.187+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/1578136671 shutdown_connections 2024-05-27T09:50:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.187+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f18ffc9d80 0x55f1903c9bc0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.187+0000 7f5b63400640 0 client.28124 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.187+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f19000b1b0 0x55f1903c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.187+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffea4d0 0x55f1903b7ef0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.187+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/1578136671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffe22f0 0x55f1903b79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.187+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/1578136671 >> 172.21.15.44:0/1578136671 conn(0x55f18ffff300 msgr2=0x55f1903ad690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.188+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/1578136671 shutdown_connections 2024-05-27T09:50:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.188+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/1578136671 wait complete. 2024-05-27T09:50:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.188+0000 7f5b8b1fa740 20 client.28124 trim_cache size 0 max 16384 2024-05-27T09:50:12.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:11 smithi044 ceph-mon[31618]: pgmap v471: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 66 KiB/s wr, 29 op/s 2024-05-27T09:50:12.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.226+0000 7f5b8b1fa740 1 Processor -- start 2024-05-27T09:50:12.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.226+0000 7f5b8b1fa740 1 -- start start 2024-05-27T09:50:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.227+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffe22f0 0x55f1903c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.227+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 0x55f1903c4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.227+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 0x55f1903cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.228+0000 7f5b8b1fa740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1c1450 con 0x55f18ffea4d0 2024-05-27T09:50:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.228+0000 7f5b8b1fa740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1becc0 con 0x55f19000b1b0 2024-05-27T09:50:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.228+0000 7f5b8b1fa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1bcb30 con 0x55f18ffe22f0 2024-05-27T09:50:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.228+0000 7f5b82200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffe22f0 0x55f1903c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.228+0000 7f5b82200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffe22f0 0x55f1903c3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:35004/0 (socket says 172.21.15.44:35004) 2024-05-27T09:50:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.228+0000 7f5b82200640 1 -- 172.21.15.44:0/483441998 learned_addr learned my addr 172.21.15.44:0/483441998 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.229+0000 7f5b82200640 1 --2- 172.21.15.44:0/483441998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 0x55f1903cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.230+0000 7f5b80e00640 1 --2- 172.21.15.44:0/483441998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 0x55f1903c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.230+0000 7f5b82200640 1 -- 172.21.15.44:0/483441998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 msgr2=0x55f1903cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.231+0000 7f5b82200640 1 --2- 172.21.15.44:0/483441998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 0x55f1903cf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.231+0000 7f5b82200640 1 -- 172.21.15.44:0/483441998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 msgr2=0x55f1903c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.231+0000 7f5b82200640 1 --2- 172.21.15.44:0/483441998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 0x55f1903c4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.231+0000 7f5b82200640 1 -- 172.21.15.44:0/483441998 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1903aac00 con 0x55f18ffe22f0 2024-05-27T09:50:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.231+0000 7f5b80e00640 1 --2- 172.21.15.44:0/483441998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 0x55f1903c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.231+0000 7f5b82200640 1 --2- 172.21.15.44:0/483441998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffe22f0 0x55f1903c3b20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5b68011880 tx=0x7f5b68011dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e79d4a7af0983845 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.232+0000 7f5b6e000640 1 -- 172.21.15.44:0/483441998 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6801f4c0 con 0x55f18ffe22f0 2024-05-27T09:50:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.232+0000 7f5b6e000640 1 -- 172.21.15.44:0/483441998 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b6800bbd0 con 0x55f18ffe22f0 2024-05-27T09:50:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.232+0000 7f5b6e000640 1 -- 172.21.15.44:0/483441998 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b68005290 con 0x55f18ffe22f0 2024-05-27T09:50:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.233+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/483441998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffe22f0 msgr2=0x55f1903c3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.233+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/483441998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffe22f0 0x55f1903c3b20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5b68011880 tx=0x7f5b68011dc0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.233+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/483441998 shutdown_connections 2024-05-27T09:50:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.233+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/483441998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f19000b1b0 0x55f1903cf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.233+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/483441998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 0x55f1903c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.233+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/483441998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f18ffe22f0 0x55f1903c3b20 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.233+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/483441998 >> 172.21.15.44:0/483441998 conn(0x55f18ffff300 msgr2=0x55f1903b7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.234+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/483441998 shutdown_connections 2024-05-27T09:50:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.234+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/483441998 wait complete. 2024-05-27T09:50:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.234+0000 7f5b8b1fa740 1 -- start start 2024-05-27T09:50:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.235+0000 7f5b8b1fa740 1 Processor -- start 2024-05-27T09:50:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.235+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffe22f0 0x55f1903c0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.235+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 0x55f1903c05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.235+0000 7f5b8b1fa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f19000b1b0 0x55f1903bd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.235+0000 7f5b8b1fa740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1bd180 con 0x55f18ffea4d0 2024-05-27T09:50:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.235+0000 7f5b8b1fa740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1be5f0 con 0x55f18ffe22f0 2024-05-27T09:50:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.235+0000 7f5b8b1fa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1bd850 con 0x55f19000b1b0 2024-05-27T09:50:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.235+0000 7f5b8b1fa740 10 client.0 _ll_register_callbacks cb 0x7f5b70009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.236+0000 7f5b82200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffe22f0 0x55f1903c0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.236+0000 7f5b82200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffe22f0 0x55f1903c0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:46930/0 (socket says 172.21.15.44:46930) 2024-05-27T09:50:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.236+0000 7f5b82200640 1 -- 172.21.15.44:0/2325566090 learned_addr learned my addr 172.21.15.44:0/2325566090 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.236+0000 7f5b82200640 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f19000b1b0 0x55f1903bd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.237+0000 7f5b80e00640 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 0x55f1903c05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.237+0000 7f5b82200640 1 -- 172.21.15.44:0/2325566090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f19000b1b0 msgr2=0x55f1903bd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.237+0000 7f5b82200640 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f19000b1b0 0x55f1903bd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.237+0000 7f5b82200640 1 -- 172.21.15.44:0/2325566090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 msgr2=0x55f1903c05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:12.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.237+0000 7f5b82200640 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 0x55f1903c05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.237+0000 7f5b82200640 1 -- 172.21.15.44:0/2325566090 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1903bdc70 con 0x55f18ffe22f0 2024-05-27T09:50:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.237+0000 7f5b80e00640 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 0x55f1903c05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:50:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.238+0000 7f5b82200640 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffe22f0 0x55f1903c0060 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5b68005fd0 tx=0x7f5b680119f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f274f4fdef3a134 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.238+0000 7f5b63400640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.238+0000 7f5b63400640 1 -- 172.21.15.44:0/2325566090 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b68040060 con 0x55f18ffe22f0 2024-05-27T09:50:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.238+0000 7f5b63400640 1 -- 172.21.15.44:0/2325566090 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f5b6802d3c0 con 0x55f18ffe22f0 2024-05-27T09:50:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.238+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/2325566090 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b6800bbd0 con 0x55f18ffe22f0 2024-05-27T09:50:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.240+0000 7f5b63400640 1 -- 172.21.15.44:0/2325566090 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f5b6800ba10 con 0x55f18ffe22f0 2024-05-27T09:50:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.240+0000 7f5b8b1fa740 10 client.28130 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.240+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/2325566090 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55f190431d50 con 0x55f18ffe22f0 2024-05-27T09:50:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.244+0000 7f5b63400640 1 -- 172.21.15.44:0/2325566090 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b6800bbd0 con 0x55f18ffe22f0 2024-05-27T09:50:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.244+0000 7f5b63400640 1 -- 172.21.15.44:0/2325566090 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f5b6801d020 con 0x55f18ffe22f0 2024-05-27T09:50:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.245+0000 7f5b8b1fa740 10 client.28130 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.245+0000 7f5b8b1fa740 10 client.28130 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.245+0000 7f5b8b1fa740 20 client.28130 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.245+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f18ff9f1c0 0x55f190434140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.245+0000 7f5b8b1fa740 4 client.28130 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:50:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.245+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/2325566090 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f190320ae0 con 0x55f18ff9f1c0 2024-05-27T09:50:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.245+0000 7f5b80e00640 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f18ff9f1c0 0x55f190434140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.249+0000 7f5b80e00640 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f18ff9f1c0 0x55f190434140 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43e466020039ced1 server_cookie=4dee0a63d6218371 in_seq=0 out_seq=0 2024-05-27T09:50:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.249+0000 7f5b63400640 10 client.28130 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:12.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:11 smithi050 ceph-mon[36371]: pgmap v471: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 66 KiB/s wr, 29 op/s 2024-05-27T09:50:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.323+0000 7f5b63400640 1 -- 172.21.15.44:0/2325566090 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f190320ae0 con 0x55f18ff9f1c0 2024-05-27T09:50:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.323+0000 7f5b63400640 10 client.28130 handle_command_reply: tid=0 2024-05-27T09:50:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.323+0000 7f5b8b1fa740 1 client.28130 shutdown 2024-05-27T09:50:12.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.323+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/2325566090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffe22f0 msgr2=0x55f1903c0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:12.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.323+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffe22f0 0x55f1903c0060 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5b68005fd0 tx=0x7f5b680119f0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.324+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/2325566090 shutdown_connections 2024-05-27T09:50:12.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.324+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55f18ff9f1c0 0x55f190434140 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.324+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f19000b1b0 0x55f1903bd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.324+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f18ffea4d0 0x55f1903c05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.324+0000 7f5b8b1fa740 1 --2- 172.21.15.44:0/2325566090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f18ffe22f0 0x55f1903c0060 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.324+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/2325566090 >> 172.21.15.44:0/2325566090 conn(0x55f18ffff300 msgr2=0x55f1903b79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:12.325 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.325+0000 7f5b63400640 0 client.28130 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.325+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/2325566090 shutdown_connections 2024-05-27T09:50:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.325+0000 7f5b8b1fa740 1 -- 172.21.15.44:0/2325566090 wait complete. 2024-05-27T09:50:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.325+0000 7f5b8b1fa740 20 client.28130 trim_cache size 0 max 16384 2024-05-27T09:50:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:50:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:50:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:50:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.327+0000 7f5b84200640 1 -- 172.21.15.44:0/60295831 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b640b29d0 msgr2=0x7f5b640b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.327+0000 7f5b84200640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b640b29d0 0x7f5b640b4ee0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5b740a5fd0 tx=0x7f5b740a8b30 comp rx=0 tx=0).stop 2024-05-27T09:50:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.327+0000 7f5b84200640 1 -- 172.21.15.44:0/60295831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c1032e0 msgr2=0x7f5b7c198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.327+0000 7f5b84200640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c1032e0 0x7f5b7c198780 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400b3c0 tx=0x7f5b7400b880 comp rx=0 tx=0).stop 2024-05-27T09:50:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.327+0000 7f5b84200640 1 -- 172.21.15.44:0/60295831 shutdown_connections 2024-05-27T09:50:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.327+0000 7f5b84200640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b640b29d0 0x7f5b640b4ee0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.327+0000 7f5b84200640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b7c109030 0x7f5b7c19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.327+0000 7f5b84200640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b7c108b90 0x7f5b7c198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.327+0000 7f5b84200640 1 --2- 172.21.15.44:0/60295831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c1032e0 0x7f5b7c198780 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.327+0000 7f5b84200640 1 -- 172.21.15.44:0/60295831 >> 172.21.15.44:0/60295831 conn(0x7f5b7c0782d0 msgr2=0x7f5b7c1098b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.328+0000 7f5b84200640 1 -- 172.21.15.44:0/60295831 shutdown_connections 2024-05-27T09:50:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:12.328+0000 7f5b84200640 1 -- 172.21.15.44:0/60295831 wait complete. 2024-05-27T09:50:12.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:11 smithi189 ceph-mon[36671]: pgmap v471: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 66 KiB/s wr, 29 op/s 2024-05-27T09:50:12.447 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:50:13.289 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:13.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:13 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:50:13.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.301+0000 7f749b200700 1 -- 172.21.15.44:0/3316102597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74941045e0 msgr2=0x7f74941049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:13.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.301+0000 7f749b200700 1 --2- 172.21.15.44:0/3316102597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74941045e0 0x7f74941049f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f748c00b0d0 tx=0x7f748c00b3e0 comp rx=0 tx=0).stop 2024-05-27T09:50:13.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.301+0000 7f749b200700 1 -- 172.21.15.44:0/3316102597 shutdown_connections 2024-05-27T09:50:13.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.301+0000 7f749b200700 1 --2- 172.21.15.44:0/3316102597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7494109860 0x7f74941111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:13.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.301+0000 7f749b200700 1 --2- 172.21.15.44:0/3316102597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74941045e0 0x7f74941049f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:13.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.301+0000 7f749b200700 1 --2- 172.21.15.44:0/3316102597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7494108f10 0x7f7494109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:13.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.301+0000 7f749b200700 1 -- 172.21.15.44:0/3316102597 >> 172.21.15.44:0/3316102597 conn(0x7f7494100030 msgr2=0x7f7494102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:13.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.302+0000 7f749b200700 1 -- 172.21.15.44:0/3316102597 shutdown_connections 2024-05-27T09:50:13.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.302+0000 7f749b200700 1 -- 172.21.15.44:0/3316102597 wait complete. 2024-05-27T09:50:13.305 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:13 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:50:13.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.304+0000 7f749b200700 1 Processor -- start 2024-05-27T09:50:13.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.304+0000 7f749b200700 1 -- start start 2024-05-27T09:50:13.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.305+0000 7f749b200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74941045e0 0x7f749410f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:13.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.305+0000 7f749b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7494108f10 0x7f749410fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:13.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.306+0000 7f749b200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7494109860 0x7f74941ad270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:13.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.306+0000 7f749b200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7494112390 con 0x7f74941045e0 2024-05-27T09:50:13.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.306+0000 7f749b200700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7494112210 con 0x7f7494109860 2024-05-27T09:50:13.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.306+0000 7f749b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7494112510 con 0x7f7494108f10 2024-05-27T09:50:13.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.306+0000 7f7499200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7494109860 0x7f74941ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:13.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.306+0000 7f7499200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7494109860 0x7f74941ad270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:41078/0 (socket says 172.21.15.44:41078) 2024-05-27T09:50:13.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.306+0000 7f7499200700 1 -- 172.21.15.44:0/4233024124 learned_addr learned my addr 172.21.15.44:0/4233024124 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:13.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.306+0000 7f7493400700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7494108f10 0x7f749410fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:13.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.307+0000 7f7493e00700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74941045e0 0x7f749410f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:13.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.307+0000 7f7499200700 1 -- 172.21.15.44:0/4233024124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7494108f10 msgr2=0x7f749410fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:13.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.308+0000 7f7499200700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7494108f10 0x7f749410fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:13.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.308+0000 7f7499200700 1 -- 172.21.15.44:0/4233024124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74941045e0 msgr2=0x7f749410f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:13.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.308+0000 7f7499200700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74941045e0 0x7f749410f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:13.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.308+0000 7f7499200700 1 -- 172.21.15.44:0/4233024124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74941ad7b0 con 0x7f7494109860 2024-05-27T09:50:13.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.308+0000 7f7493e00700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74941045e0 0x7f749410f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:13.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.308+0000 7f7493400700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7494108f10 0x7f749410fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:50:13.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.308+0000 7f7499200700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7494109860 0x7f74941ad270 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f747c00e940 tx=0x7f747c00ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:13.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.309+0000 7f7490c00700 1 -- 172.21.15.44:0/4233024124 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c00cbd0 con 0x7f7494109860 2024-05-27T09:50:13.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.309+0000 7f749b200700 1 -- 172.21.15.44:0/4233024124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74941ad980 con 0x7f7494109860 2024-05-27T09:50:13.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.309+0000 7f749b200700 1 -- 172.21.15.44:0/4233024124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74941adec0 con 0x7f7494109860 2024-05-27T09:50:13.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.310+0000 7f7490c00700 1 -- 172.21.15.44:0/4233024124 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f747c00ee90 con 0x7f7494109860 2024-05-27T09:50:13.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.310+0000 7f7490c00700 1 -- 172.21.15.44:0/4233024124 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c005590 con 0x7f7494109860 2024-05-27T09:50:13.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.311+0000 7f749b200700 1 -- 172.21.15.44:0/4233024124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7470005330 con 0x7f7494109860 2024-05-27T09:50:13.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.318+0000 7f7490c00700 1 -- 172.21.15.44:0/4233024124 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f747c0479e0 con 0x7f7494109860 2024-05-27T09:50:13.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.323+0000 7f7490c00700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74780b2aa0 0x7f74780b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:13.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.324+0000 7f7490c00700 1 -- 172.21.15.44:0/4233024124 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f747c0e2790 con 0x7f7494109860 2024-05-27T09:50:13.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.324+0000 7f7493e00700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74780b2aa0 0x7f74780b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:13.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.325+0000 7f7490c00700 1 -- 172.21.15.44:0/4233024124 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f747c0aa090 con 0x7f7494109860 2024-05-27T09:50:13.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.329+0000 7f7493e00700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74780b2aa0 0x7f74780b4fb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f748c009e60 tx=0x7f748c00b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:13.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:13 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:50:13.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.878+0000 7f749b200700 1 -- 172.21.15.44:0/4233024124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7470005620 con 0x7f7494109860 2024-05-27T09:50:13.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.879+0000 7f7490c00700 1 -- 172.21.15.44:0/4233024124 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11870 (secure 0 0 0) 0x7f747c0a9990 con 0x7f7494109860 2024-05-27T09:50:13.886 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:50:13.886 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:48:16.864365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24503]},"id":1}]} 2024-05-27T09:50:13.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.887+0000 7f749b200700 1 -- 172.21.15.44:0/4233024124 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74780b2aa0 msgr2=0x7f74780b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:13.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.887+0000 7f749b200700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74780b2aa0 0x7f74780b4fb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f748c009e60 tx=0x7f748c00b040 comp rx=0 tx=0).stop 2024-05-27T09:50:13.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.887+0000 7f749b200700 1 -- 172.21.15.44:0/4233024124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7494109860 msgr2=0x7f74941ad270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:13.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.887+0000 7f749b200700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7494109860 0x7f74941ad270 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f747c00e940 tx=0x7f747c00ec50 comp rx=0 tx=0).stop 2024-05-27T09:50:13.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.887+0000 7f749b200700 1 -- 172.21.15.44:0/4233024124 shutdown_connections 2024-05-27T09:50:13.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.887+0000 7f749b200700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74780b2aa0 0x7f74780b4fb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:13.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.887+0000 7f749b200700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7494109860 0x7f74941ad270 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:13.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.887+0000 7f749b200700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74941045e0 0x7f749410f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:13.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.887+0000 7f749b200700 1 --2- 172.21.15.44:0/4233024124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7494108f10 0x7f749410fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:13.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.887+0000 7f749b200700 1 -- 172.21.15.44:0/4233024124 >> 172.21.15.44:0/4233024124 conn(0x7f7494100030 msgr2=0x7f74941076a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:13.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.889+0000 7f749b200700 1 -- 172.21.15.44:0/4233024124 shutdown_connections 2024-05-27T09:50:13.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:13.889+0000 7f749b200700 1 -- 172.21.15.44:0/4233024124 wait complete. 2024-05-27T09:50:13.893 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 43 2024-05-27T09:50:14.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:14 smithi044 ceph-mon[31618]: pgmap v472: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 66 KiB/s wr, 29 op/s 2024-05-27T09:50:14.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:14 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4233024124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:14.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:14 smithi050 ceph-mon[36371]: pgmap v472: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 66 KiB/s wr, 29 op/s 2024-05-27T09:50:14.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:14 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/4233024124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:14.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:14 smithi189 ceph-mon[36671]: pgmap v472: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 66 KiB/s wr, 29 op/s 2024-05-27T09:50:14.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:14 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/4233024124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:14.916 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:50:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:50:14.618+0000 7f49fc600700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-27T09:50:14.920 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db / --timeout 90 --expiration 142.5 --await-for 45.0' 2024-05-27T09:50:14.921 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db / --timeout 90 --expiration 142.5 --await-for 45.0 2024-05-27T09:50:15.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:15 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:50:15.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:15 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:50:15.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:15 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:50:15.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:15 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:50:15.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:15 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:50:15.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:15 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:50:15.352 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6507388dd5057528934822c0163b0c347ef1d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2024-05-27T09:50:15.525 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e2b4eb7192c3476a8ab4bbf2d943712e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6507388dd5057528934822c0163b0c347ef1d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2024-05-27T09:50:15.527 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-05-27T09:50:15.539 INFO:tasks.quiescer:joining Quiescers 2024-05-27T09:50:15.540 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-05-27T09:50:15.540 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:50:16.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:16 smithi050 ceph-mon[36371]: pgmap v473: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 55 KiB/s wr, 25 op/s 2024-05-27T09:50:16.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:16 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:50:16.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:16 smithi189 ceph-mon[36671]: pgmap v473: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 55 KiB/s wr, 25 op/s 2024-05-27T09:50:16.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:16 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:50:16.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:16 smithi044 ceph-mon[31618]: pgmap v473: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 55 KiB/s wr, 25 op/s 2024-05-27T09:50:16.476 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:16 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:50:16.716 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:16.764 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:18.219 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:18 smithi044 ceph-mon[31618]: pgmap v474: 97 pgs: 19 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 43 active+clean; 342 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 65 KiB/s wr, 32 op/s 2024-05-27T09:50:18.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:18 smithi050 ceph-mon[36371]: pgmap v474: 97 pgs: 19 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 43 active+clean; 342 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 65 KiB/s wr, 32 op/s 2024-05-27T09:50:18.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:18 smithi189 ceph-mon[36671]: pgmap v474: 97 pgs: 19 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 43 active+clean; 342 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 65 KiB/s wr, 32 op/s 2024-05-27T09:50:18.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.217+0000 7f10a6400700 1 -- 172.21.15.44:0/4019210705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a0108550 msgr2=0x7f10a010feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:18.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.217+0000 7f10a6400700 1 --2- 172.21.15.44:0/4019210705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a0108550 0x7f10a010feb0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f1098009f90 tx=0x7f109800b330 comp rx=0 tx=0).stop 2024-05-27T09:50:18.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.219+0000 7f10a6400700 1 -- 172.21.15.44:0/4019210705 shutdown_connections 2024-05-27T09:50:18.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.219+0000 7f10a6400700 1 --2- 172.21.15.44:0/4019210705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a0108550 0x7f10a010feb0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.219+0000 7f10a6400700 1 --2- 172.21.15.44:0/4019210705 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10a01032d0 0x7f10a01036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.219+0000 7f10a6400700 1 --2- 172.21.15.44:0/4019210705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a0107c00 0x7f10a0108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.219+0000 7f10a6400700 1 -- 172.21.15.44:0/4019210705 >> 172.21.15.44:0/4019210705 conn(0x7f10a0078ab0 msgr2=0x7f10a0078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:18.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.219+0000 7f10a6400700 1 -- 172.21.15.44:0/4019210705 shutdown_connections 2024-05-27T09:50:18.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.219+0000 7f10a6400700 1 -- 172.21.15.44:0/4019210705 wait complete. 2024-05-27T09:50:18.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.220+0000 7f10a6400700 1 Processor -- start 2024-05-27T09:50:18.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.220+0000 7f10a6400700 1 -- start start 2024-05-27T09:50:18.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.221+0000 7f10a6400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a01032d0 0x7f10a019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.221+0000 7f10a6400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a0107c00 0x7f10a019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.221+0000 7f10a6400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10a0108550 0x7f10a01a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.221+0000 7f10a6400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f10a0112130 con 0x7f10a0108550 2024-05-27T09:50:18.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.221+0000 7f10a6400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f10a0111fb0 con 0x7f10a0107c00 2024-05-27T09:50:18.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.221+0000 7f10a6400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01122b0 con 0x7f10a01032d0 2024-05-27T09:50:18.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.221+0000 7f109fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a0107c00 0x7f10a019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.221+0000 7f109fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a0107c00 0x7f10a019d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:41084/0 (socket says 172.21.15.44:41084) 2024-05-27T09:50:18.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.221+0000 7f109fe00700 1 -- 172.21.15.44:0/3075227493 learned_addr learned my addr 172.21.15.44:0/3075227493 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:18.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.222+0000 7f10a5000700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a01032d0 0x7f10a019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.222+0000 7f10a5a00700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10a0108550 0x7f10a01a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.222+0000 7f109fe00700 1 -- 172.21.15.44:0/3075227493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a01032d0 msgr2=0x7f10a019ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:18.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.222+0000 7f109fe00700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a01032d0 0x7f10a019ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.222+0000 7f109fe00700 1 -- 172.21.15.44:0/3075227493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10a0108550 msgr2=0x7f10a01a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:18.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.222+0000 7f109fe00700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10a0108550 0x7f10a01a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.222+0000 7f109fe00700 1 -- 172.21.15.44:0/3075227493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a01a1d80 con 0x7f10a0107c00 2024-05-27T09:50:18.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.222+0000 7f10a5a00700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10a0108550 0x7f10a01a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:18.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.222+0000 7f10a5000700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a01032d0 0x7f10a019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:18.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.222+0000 7f109fe00700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a0107c00 0x7f10a019d450 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f108c00b550 tx=0x7f108c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:18.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.223+0000 7f109d600700 1 -- 172.21.15.44:0/3075227493 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c015020 con 0x7f10a0107c00 2024-05-27T09:50:18.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.223+0000 7f109d600700 1 -- 172.21.15.44:0/3075227493 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f108c011ce0 con 0x7f10a0107c00 2024-05-27T09:50:18.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.223+0000 7f10a6400700 1 -- 172.21.15.44:0/3075227493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10a01a2070 con 0x7f10a0107c00 2024-05-27T09:50:18.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.223+0000 7f10a6400700 1 -- 172.21.15.44:0/3075227493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10a006d1d0 con 0x7f10a0107c00 2024-05-27T09:50:18.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.224+0000 7f109d600700 1 -- 172.21.15.44:0/3075227493 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c010d60 con 0x7f10a0107c00 2024-05-27T09:50:18.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.229+0000 7f10b64f7b80 1 Processor -- start 2024-05-27T09:50:18.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.229+0000 7f10b64f7b80 1 -- start start 2024-05-27T09:50:18.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.229+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c8f69f0 0x55969c98a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.229+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 0x55969c98ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.229+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c9942c0 0x55969c9966e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.229+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55969c8b7e40 con 0x55969c9942c0 2024-05-27T09:50:18.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.229+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55969c8a79e0 con 0x55969c8f69f0 2024-05-27T09:50:18.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.229+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55969c8aba20 con 0x55969c80ae40 2024-05-27T09:50:18.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.229+0000 7f10a5a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c8f69f0 0x55969c98a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.230+0000 7f10a5a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c8f69f0 0x55969c98a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:41090/0 (socket says 172.21.15.44:41090) 2024-05-27T09:50:18.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.230+0000 7f10a5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 0x55969c98ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.230+0000 7f10a5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 0x55969c98ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:40330/0 (socket says 172.21.15.44:40330) 2024-05-27T09:50:18.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.230+0000 7f10a5000700 1 -- 172.21.15.44:0/2796812464 learned_addr learned my addr 172.21.15.44:0/2796812464 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:18.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.230+0000 7f10a5000700 1 -- 172.21.15.44:0/2796812464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c8f69f0 msgr2=0x55969c98a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:18.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.230+0000 7f10a5000700 1 --2- 172.21.15.44:0/2796812464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c8f69f0 0x55969c98a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.230+0000 7f10a5000700 1 -- 172.21.15.44:0/2796812464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c9942c0 msgr2=0x55969c9966e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T09:50:18.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.230+0000 7f10a5000700 1 --2- 172.21.15.44:0/2796812464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c9942c0 0x55969c9966e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.230+0000 7f10a5000700 1 -- 172.21.15.44:0/2796812464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55969c870420 con 0x55969c80ae40 2024-05-27T09:50:18.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.231+0000 7f10a5000700 1 --2- 172.21.15.44:0/2796812464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 0x55969c98ac90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f10940115a0 tx=0x7f10940118b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=196c44c1c07b81cf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:18.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.231+0000 7f1092000700 1 -- 172.21.15.44:0/2796812464 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094004230 con 0x55969c80ae40 2024-05-27T09:50:18.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.231+0000 7f1092000700 1 -- 172.21.15.44:0/2796812464 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10940043d0 con 0x55969c80ae40 2024-05-27T09:50:18.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.231+0000 7f1092000700 1 -- 172.21.15.44:0/2796812464 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094019940 con 0x55969c80ae40 2024-05-27T09:50:18.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.232+0000 7f10b64f7b80 1 -- 172.21.15.44:0/2796812464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 msgr2=0x55969c98ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:18.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.232+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/2796812464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 0x55969c98ac90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f10940115a0 tx=0x7f10940118b0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.234+0000 7f10b64f7b80 1 -- 172.21.15.44:0/2796812464 shutdown_connections 2024-05-27T09:50:18.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.234+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/2796812464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c9942c0 0x55969c9966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.234+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/2796812464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c8f69f0 0x55969c98a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.234+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/2796812464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 0x55969c98ac90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.234+0000 7f10b64f7b80 1 -- 172.21.15.44:0/2796812464 >> 172.21.15.44:0/2796812464 conn(0x55969c8a4390 msgr2=0x55969c985710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:18.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.242+0000 7f109d600700 1 -- 172.21.15.44:0/3075227493 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f108c018020 con 0x7f10a0107c00 2024-05-27T09:50:18.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.243+0000 7f10b64f7b80 1 -- 172.21.15.44:0/2796812464 shutdown_connections 2024-05-27T09:50:18.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.243+0000 7f10b64f7b80 1 -- 172.21.15.44:0/2796812464 wait complete. 2024-05-27T09:50:18.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.244+0000 7f10b64f7b80 1 -- start start 2024-05-27T09:50:18.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.245+0000 7f109d600700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10880b2980 0x7f10880b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.246+0000 7f10a5a00700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10880b2980 0x7f10880b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.246+0000 7f10b64f7b80 1 Processor -- start 2024-05-27T09:50:18.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.247+0000 7f109d600700 1 -- 172.21.15.44:0/3075227493 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f108c0e1b90 con 0x7f10a0107c00 2024-05-27T09:50:18.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.248+0000 7f10a5a00700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10880b2980 0x7f10880b4e90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55969c986a90 tx=0x7f1098009b40 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:18.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.248+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c80ae40 0x55969c991280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.248+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c8f69f0 0x55969c9917c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.248+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c991d00 0x55969c9a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.248+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55969c7adfa0 con 0x55969c80ae40 2024-05-27T09:50:18.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.248+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55969c848760 con 0x55969c8f69f0 2024-05-27T09:50:18.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.248+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55969c996e00 con 0x55969c991d00 2024-05-27T09:50:18.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.249+0000 7f10b64f7b80 10 client.0 _ll_register_callbacks cb 0x55969c851c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:18.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.249+0000 7f10a5000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c80ae40 0x55969c991280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.249+0000 7f10a5000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c80ae40 0x55969c991280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38316/0 (socket says 172.21.15.44:38316) 2024-05-27T09:50:18.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.249+0000 7f10a5000700 1 -- 172.21.15.44:0/4107150324 learned_addr learned my addr 172.21.15.44:0/4107150324 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:18.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.249+0000 7f10a5a00700 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c991d00 0x55969c9a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.249+0000 7f109fe00700 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c8f69f0 0x55969c9917c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.251+0000 7f10a5a00700 1 -- 172.21.15.44:0/4107150324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c8f69f0 msgr2=0x55969c9917c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:18.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.251+0000 7f10a5a00700 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c8f69f0 0x55969c9917c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.251+0000 7f10a5a00700 1 -- 172.21.15.44:0/4107150324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c80ae40 msgr2=0x55969c991280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:18.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.251+0000 7f10a5a00700 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c80ae40 0x55969c991280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.251+0000 7f10a5a00700 1 -- 172.21.15.44:0/4107150324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55969c7f78d0 con 0x55969c991d00 2024-05-27T09:50:18.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.253+0000 7f10a5a00700 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c991d00 0x55969c9a01c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1098009840 tx=0x7f1098009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13c50f130dad6a6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:18.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.253+0000 7f1086a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:50:18.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.253+0000 7f1086a00700 1 -- 172.21.15.44:0/4107150324 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109802d070 con 0x55969c991d00 2024-05-27T09:50:18.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.254+0000 7f1086a00700 1 -- 172.21.15.44:0/4107150324 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f1098007d70 con 0x55969c991d00 2024-05-27T09:50:18.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.254+0000 7f10b64f7b80 1 -- 172.21.15.44:0/4107150324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10940043d0 con 0x55969c991d00 2024-05-27T09:50:18.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.254+0000 7f1086a00700 1 -- 172.21.15.44:0/4107150324 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1098023070 con 0x55969c991d00 2024-05-27T09:50:18.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.255+0000 7f10b64f7b80 10 client.28156 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:18.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.255+0000 7f10b64f7b80 1 -- 172.21.15.44:0/4107150324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55969c878060 con 0x55969c991d00 2024-05-27T09:50:18.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.255+0000 7f1086a00700 1 -- 172.21.15.44:0/4107150324 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10940043d0 con 0x55969c991d00 2024-05-27T09:50:18.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.255+0000 7f1086a00700 1 -- 172.21.15.44:0/4107150324 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f1098011070 con 0x55969c991d00 2024-05-27T09:50:18.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.256+0000 7f10b64f7b80 10 client.28156 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:18.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.256+0000 7f10b64f7b80 10 client.28156 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T09:50:18.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.256+0000 7f10b64f7b80 20 client.28156 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:18.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.256+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55969c9a41b0 0x55969c9a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.256+0000 7f10b64f7b80 4 client.28156 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:18.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.256+0000 7f10b64f7b80 1 -- 172.21.15.44:0/4107150324 --> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55969c8b9d90 con 0x55969c9a41b0 2024-05-27T09:50:18.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.256+0000 7f10a5000700 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55969c9a41b0 0x55969c9a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.257+0000 7f1086a00700 10 client.28156 ms_handle_connect on v2:172.21.15.50:6836/3540954143 2024-05-27T09:50:18.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.257+0000 7f10a5000700 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55969c9a41b0 0x55969c9a65f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f743647e75eb97d5 server_cookie=1eb52f151777fdde in_seq=0 out_seq=0 2024-05-27T09:50:18.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.259+0000 7f1086a00700 1 -- 172.21.15.44:0/4107150324 <== mds.1 v2:172.21.15.50:6836/3540954143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55969c8b9d90 con 0x55969c9a41b0 2024-05-27T09:50:18.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.259+0000 7f1086a00700 10 client.28156 handle_command_reply: tid=0 2024-05-27T09:50:18.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.259+0000 7f10b64f7b80 1 client.28156 shutdown 2024-05-27T09:50:18.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.261+0000 7f10b64f7b80 1 -- 172.21.15.44:0/4107150324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c991d00 msgr2=0x55969c9a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:18.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.261+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c991d00 0x55969c9a01c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1098009840 tx=0x7f1098009920 comp rx=0 tx=0).stop 2024-05-27T09:50:18.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.261+0000 7f10b64f7b80 1 -- 172.21.15.44:0/4107150324 shutdown_connections 2024-05-27T09:50:18.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.261+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c991d00 0x55969c9a01c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.261+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c80ae40 0x55969c991280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.261+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55969c9a41b0 0x55969c9a65f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.261+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/4107150324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c8f69f0 0x55969c9917c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.261+0000 7f10b64f7b80 1 -- 172.21.15.44:0/4107150324 >> 172.21.15.44:0/4107150324 conn(0x55969c8a4390 msgr2=0x55969c98a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:18.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.261+0000 7f1086a00700 0 client.28156 ms_handle_reset on v2:172.21.15.50:6836/3540954143 2024-05-27T09:50:18.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.262+0000 7f10b64f7b80 1 -- 172.21.15.44:0/4107150324 shutdown_connections 2024-05-27T09:50:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.262+0000 7f10b64f7b80 1 -- 172.21.15.44:0/4107150324 wait complete. 2024-05-27T09:50:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.262+0000 7f10b64f7b80 20 client.28156 trim_cache size 0 max 16384 2024-05-27T09:50:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.298+0000 7f10b64f7b80 1 Processor -- start 2024-05-27T09:50:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.300+0000 7f10b64f7b80 1 -- start start 2024-05-27T09:50:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.301+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 0x55969c9901e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.301+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c8f69f0 0x55969c98c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.301+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c991d00 0x55969c99ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.301+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f10800101a0 con 0x55969c8f69f0 2024-05-27T09:50:18.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.301+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f108000d9d0 con 0x55969c991d00 2024-05-27T09:50:18.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.301+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f108000b840 con 0x55969c80ae40 2024-05-27T09:50:18.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.302+0000 7f10a5a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c991d00 0x55969c99ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.302+0000 7f10a5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 0x55969c9901e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.302+0000 7f10a5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 0x55969c9901e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:40362/0 (socket says 172.21.15.44:40362) 2024-05-27T09:50:18.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.302+0000 7f10a5000700 1 -- 172.21.15.44:0/774745290 learned_addr learned my addr 172.21.15.44:0/774745290 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:18.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.303+0000 7f109fe00700 1 --2- 172.21.15.44:0/774745290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c8f69f0 0x55969c98c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.303+0000 7f10a5a00700 1 -- 172.21.15.44:0/774745290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 msgr2=0x55969c9901e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.303+0000 7f10a5a00700 1 --2- 172.21.15.44:0/774745290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 0x55969c9901e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.303+0000 7f10a5a00700 1 -- 172.21.15.44:0/774745290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c8f69f0 msgr2=0x55969c98c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.303+0000 7f10a5a00700 1 --2- 172.21.15.44:0/774745290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c8f69f0 0x55969c98c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.304+0000 7f10a5a00700 1 -- 172.21.15.44:0/774745290 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55969c98f7f0 con 0x55969c991d00 2024-05-27T09:50:18.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.305+0000 7f10a5a00700 1 --2- 172.21.15.44:0/774745290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c991d00 0x55969c99ff30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f109801d5b0 tx=0x7f109800cfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1938c8f7594119a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.306+0000 7f1091600700 1 -- 172.21.15.44:0/774745290 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109802d040 con 0x55969c991d00 2024-05-27T09:50:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.306+0000 7f1091600700 1 -- 172.21.15.44:0/774745290 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f109801dcd0 con 0x55969c991d00 2024-05-27T09:50:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.306+0000 7f1091600700 1 -- 172.21.15.44:0/774745290 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098029420 con 0x55969c991d00 2024-05-27T09:50:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.307+0000 7f10b64f7b80 1 -- 172.21.15.44:0/774745290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c991d00 msgr2=0x55969c99ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.307+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/774745290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c991d00 0x55969c99ff30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f109801d5b0 tx=0x7f109800cfd0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.307+0000 7f10b64f7b80 1 -- 172.21.15.44:0/774745290 shutdown_connections 2024-05-27T09:50:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.307+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/774745290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c991d00 0x55969c99ff30 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.307+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/774745290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c80ae40 0x55969c9901e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.307+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/774745290 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c8f69f0 0x55969c98c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.307+0000 7f10b64f7b80 1 -- 172.21.15.44:0/774745290 >> 172.21.15.44:0/774745290 conn(0x55969c8a4390 msgr2=0x55969c98c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.308+0000 7f10b64f7b80 1 -- 172.21.15.44:0/774745290 shutdown_connections 2024-05-27T09:50:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.308+0000 7f10b64f7b80 1 -- 172.21.15.44:0/774745290 wait complete. 2024-05-27T09:50:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.308+0000 7f10b64f7b80 1 -- start start 2024-05-27T09:50:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.309+0000 7f10b64f7b80 1 Processor -- start 2024-05-27T09:50:18.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.310+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c80ae40 0x55969c99c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.310+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c8f69f0 0x55969c997a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.310+0000 7f10b64f7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c991d00 0x55969c997f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.310+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f108000be90 con 0x55969c991d00 2024-05-27T09:50:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.310+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f108000d300 con 0x55969c80ae40 2024-05-27T09:50:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.310+0000 7f10b64f7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f108000c560 con 0x55969c8f69f0 2024-05-27T09:50:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.310+0000 7f10b64f7b80 10 client.0 _ll_register_callbacks cb 0x55969c851c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.311+0000 7f10a5000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c80ae40 0x55969c99c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.311+0000 7f109fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c8f69f0 0x55969c997a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.311+0000 7f109fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c8f69f0 0x55969c997a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:40366/0 (socket says 172.21.15.44:40366) 2024-05-27T09:50:18.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.311+0000 7f109fe00700 1 -- 172.21.15.44:0/3632485447 learned_addr learned my addr 172.21.15.44:0/3632485447 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:18.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.313+0000 7f109fe00700 1 -- 172.21.15.44:0/3632485447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c80ae40 msgr2=0x55969c99c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:18.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.313+0000 7f109fe00700 1 --2- 172.21.15.44:0/3632485447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c80ae40 0x55969c99c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.313+0000 7f109fe00700 1 -- 172.21.15.44:0/3632485447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c991d00 msgr2=0x55969c997f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:18.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.313+0000 7f109fe00700 1 --2- 172.21.15.44:0/3632485447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c991d00 0x55969c997f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.313+0000 7f109fe00700 1 -- 172.21.15.44:0/3632485447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55969c998520 con 0x55969c8f69f0 2024-05-27T09:50:18.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.313+0000 7f109fe00700 1 --2- 172.21.15.44:0/3632485447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c8f69f0 0x55969c997a20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f108c006e20 tx=0x7f108c0a9440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cbb24e371fa6991 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:18.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.314+0000 7f1086a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:50:18.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.314+0000 7f1086a00700 1 -- 172.21.15.44:0/3632485447 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c0a9600 con 0x55969c8f69f0 2024-05-27T09:50:18.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.314+0000 7f1086a00700 1 -- 172.21.15.44:0/3632485447 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f108c010590 con 0x55969c8f69f0 2024-05-27T09:50:18.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.314+0000 7f10b64f7b80 1 -- 172.21.15.44:0/3632485447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f109801dcd0 con 0x55969c8f69f0 2024-05-27T09:50:18.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.315+0000 7f1086a00700 1 -- 172.21.15.44:0/3632485447 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f108c0a8f00 con 0x55969c8f69f0 2024-05-27T09:50:18.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.315+0000 7f10b64f7b80 10 client.28162 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.315+0000 7f10b64f7b80 1 -- 172.21.15.44:0/3632485447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55969ca17280 con 0x55969c8f69f0 2024-05-27T09:50:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.316+0000 7f1086a00700 1 -- 172.21.15.44:0/3632485447 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f109801dcd0 con 0x55969c8f69f0 2024-05-27T09:50:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.316+0000 7f1086a00700 1 -- 172.21.15.44:0/3632485447 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f108c0a4870 con 0x55969c8f69f0 2024-05-27T09:50:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.317+0000 7f10b64f7b80 10 client.28162 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.317+0000 7f10b64f7b80 10 client.28162 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T09:50:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.317+0000 7f10b64f7b80 20 client.28162 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.317+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/3632485447 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55969ca179c0 0x55969ca19de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:18.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.317+0000 7f10b64f7b80 4 client.28162 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:50:18.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.317+0000 7f10b64f7b80 1 -- 172.21.15.44:0/3632485447 --> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55969c8b3570 con 0x55969ca179c0 2024-05-27T09:50:18.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.318+0000 7f10a5000700 1 --2- 172.21.15.44:0/3632485447 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55969ca179c0 0x55969ca19de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:18.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.318+0000 7f10a5000700 1 --2- 172.21.15.44:0/3632485447 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55969ca179c0 0x55969ca19de0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf5a974d2b9f3660 server_cookie=883cd18d80d252b1 in_seq=0 out_seq=0 2024-05-27T09:50:18.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.318+0000 7f1086a00700 10 client.28162 ms_handle_connect on v2:172.21.15.50:6836/3540954143 2024-05-27T09:50:18.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.319+0000 7f1086a00700 1 -- 172.21.15.44:0/3632485447 <== mds.1 v2:172.21.15.50:6836/3540954143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55969c8b3570 con 0x55969ca179c0 2024-05-27T09:50:18.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.319+0000 7f1086a00700 10 client.28162 handle_command_reply: tid=0 2024-05-27T09:50:18.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.319+0000 7f10b64f7b80 1 client.28162 shutdown 2024-05-27T09:50:18.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.320+0000 7f10b64f7b80 1 -- 172.21.15.44:0/3632485447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c8f69f0 msgr2=0x55969c997a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:18.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.320+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/3632485447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c8f69f0 0x55969c997a20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f108c006e20 tx=0x7f108c0a9440 comp rx=0 tx=0).stop 2024-05-27T09:50:18.422 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:50:18.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.320+0000 7f10b64f7b80 1 -- 172.21.15.44:0/3632485447 shutdown_connections 2024-05-27T09:50:18.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.320+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/3632485447 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55969c991d00 0x55969c997f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.320+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/3632485447 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x55969ca179c0 0x55969ca19de0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.320+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/3632485447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55969c80ae40 0x55969c99c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.320+0000 7f10b64f7b80 1 --2- 172.21.15.44:0/3632485447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55969c8f69f0 0x55969c997a20 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.320+0000 7f1086a00700 0 client.28162 ms_handle_reset on v2:172.21.15.50:6836/3540954143 2024-05-27T09:50:18.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.320+0000 7f10b64f7b80 1 -- 172.21.15.44:0/3632485447 >> 172.21.15.44:0/3632485447 conn(0x55969c8a4390 msgr2=0x55969c989120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:18.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.322+0000 7f10b64f7b80 1 -- 172.21.15.44:0/3632485447 shutdown_connections 2024-05-27T09:50:18.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.322+0000 7f10b64f7b80 1 -- 172.21.15.44:0/3632485447 wait complete. 2024-05-27T09:50:18.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.322+0000 7f10b64f7b80 20 client.28162 trim_cache size 0 max 16384 2024-05-27T09:50:18.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.323+0000 7f10a6400700 1 -- 172.21.15.44:0/3075227493 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10880b2980 msgr2=0x7f10880b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:18.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.323+0000 7f10a6400700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10880b2980 0x7f10880b4e90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55969c986a90 tx=0x7f1098009b40 comp rx=0 tx=0).stop 2024-05-27T09:50:18.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.323+0000 7f10a6400700 1 -- 172.21.15.44:0/3075227493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a0107c00 msgr2=0x7f10a019d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:18.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.323+0000 7f10a6400700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a0107c00 0x7f10a019d450 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f108c00b550 tx=0x7f108c00b910 comp rx=0 tx=0).stop 2024-05-27T09:50:18.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.323+0000 7f10a6400700 1 -- 172.21.15.44:0/3075227493 shutdown_connections 2024-05-27T09:50:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.323+0000 7f10a6400700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f10880b2980 0x7f10880b4e90 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.323+0000 7f10a6400700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10a0108550 0x7f10a01a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.323+0000 7f10a6400700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a01032d0 0x7f10a019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.323+0000 7f10a6400700 1 --2- 172.21.15.44:0/3075227493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a0107c00 0x7f10a019d450 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.323+0000 7f10a6400700 1 -- 172.21.15.44:0/3075227493 >> 172.21.15.44:0/3075227493 conn(0x7f10a0078ab0 msgr2=0x7f10a0102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.325+0000 7f10a6400700 1 -- 172.21.15.44:0/3075227493 shutdown_connections 2024-05-27T09:50:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:18.325+0000 7f10a6400700 1 -- 172.21.15.44:0/3075227493 wait complete. 2024-05-27T09:50:20.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:20 smithi044 ceph-mon[31618]: pgmap v475: 97 pgs: 19 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 43 active+clean; 342 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-05-27T09:50:20.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:20 smithi189 ceph-mon[36671]: pgmap v475: 97 pgs: 19 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 43 active+clean; 342 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-05-27T09:50:20.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:20 smithi050 ceph-mon[36371]: pgmap v475: 97 pgs: 19 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 43 active+clean; 342 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-05-27T09:50:20.875 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:50:22.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.066+0000 7fa120600700 1 -- 172.21.15.44:0/269125174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa11810b270 msgr2=0x7fa11811d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.066+0000 7fa120600700 1 --2- 172.21.15.44:0/269125174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa11810b270 0x7fa11811d3e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa104009990 tx=0x7fa104009ca0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.066+0000 7fa120600700 1 -- 172.21.15.44:0/269125174 shutdown_connections 2024-05-27T09:50:22.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.066+0000 7fa120600700 1 --2- 172.21.15.44:0/269125174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa11811d920 0x7fa11811fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.066+0000 7fa120600700 1 --2- 172.21.15.44:0/269125174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11810a920 0x7fa11810ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.067+0000 7fa120600700 1 --2- 172.21.15.44:0/269125174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa11810b270 0x7fa11811d3e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.067+0000 7fa120600700 1 -- 172.21.15.44:0/269125174 >> 172.21.15.44:0/269125174 conn(0x7fa11806f4c0 msgr2=0x7fa11806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:22.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.067+0000 7fa120600700 1 -- 172.21.15.44:0/269125174 shutdown_connections 2024-05-27T09:50:22.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.067+0000 7fa120600700 1 -- 172.21.15.44:0/269125174 wait complete. 2024-05-27T09:50:22.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.069+0000 7fa120600700 1 Processor -- start 2024-05-27T09:50:22.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.070+0000 7fa120600700 1 -- start start 2024-05-27T09:50:22.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.072+0000 7fa120600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa11810a920 0x7fa1181af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.072+0000 7fa120600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11810b270 0x7fa1181afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.072+0000 7fa120600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa11811d920 0x7fa1181a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.072+0000 7fa120600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181233a0 con 0x7fa11810a920 2024-05-27T09:50:22.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.072+0000 7fa120600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa118123220 con 0x7fa11811d920 2024-05-27T09:50:22.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.072+0000 7fa120600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa118123520 con 0x7fa11810b270 2024-05-27T09:50:22.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.073+0000 7fa11d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11810b270 0x7fa1181afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.073+0000 7fa11d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11810b270 0x7fa1181afef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:40388/0 (socket says 172.21.15.44:40388) 2024-05-27T09:50:22.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.073+0000 7fa11d200700 1 -- 172.21.15.44:0/2791414641 learned_addr learned my addr 172.21.15.44:0/2791414641 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:22.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.074+0000 7fa11dc00700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa11810a920 0x7fa1181af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.073+0000 7fa11e600700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa11811d920 0x7fa1181a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.074+0000 7fa11dc00700 1 -- 172.21.15.44:0/2791414641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11810b270 msgr2=0x7fa1181afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.074+0000 7fa11dc00700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11810b270 0x7fa1181afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.074+0000 7fa11dc00700 1 -- 172.21.15.44:0/2791414641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa11811d920 msgr2=0x7fa1181a9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.074+0000 7fa11dc00700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa11811d920 0x7fa1181a9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.074+0000 7fa11dc00700 1 -- 172.21.15.44:0/2791414641 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1181aa1e0 con 0x7fa11810a920 2024-05-27T09:50:22.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.075+0000 7fa11d200700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11810b270 0x7fa1181afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:22.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.075+0000 7fa11dc00700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa11810a920 0x7fa1181af9b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa110007d60 tx=0x7fa11000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:22.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.075+0000 7fa116000700 1 -- 172.21.15.44:0/2791414641 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa110016020 con 0x7fa11810a920 2024-05-27T09:50:22.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.075+0000 7fa116000700 1 -- 172.21.15.44:0/2791414641 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11000ad40 con 0x7fa11810a920 2024-05-27T09:50:22.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.076+0000 7fa116000700 1 -- 172.21.15.44:0/2791414641 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1100115e0 con 0x7fa11810a920 2024-05-27T09:50:22.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.077+0000 7fa120600700 1 -- 172.21.15.44:0/2791414641 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1181aa4d0 con 0x7fa11810a920 2024-05-27T09:50:22.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.078+0000 7fa120600700 1 -- 172.21.15.44:0/2791414641 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1181b6760 con 0x7fa11810a920 2024-05-27T09:50:22.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.081+0000 7fa116000700 1 -- 172.21.15.44:0/2791414641 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa110019020 con 0x7fa11810a920 2024-05-27T09:50:22.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.088+0000 7fa116000700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa1000b2b80 0x7fa1000b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.088+0000 7fa116000700 1 -- 172.21.15.44:0/2791414641 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7fa1100e1bc0 con 0x7fa11810a920 2024-05-27T09:50:22.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.089+0000 7fa11d200700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa1000b2b80 0x7fa1000b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.091+0000 7fa11d200700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa1000b2b80 0x7fa1000b5090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa104003800 tx=0x7fa10401a040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:22.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.092+0000 7fa12f0c8b80 1 Processor -- start 2024-05-27T09:50:22.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.092+0000 7fa12f0c8b80 1 -- start start 2024-05-27T09:50:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.093+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5ec3b80 0x564ca5f5d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.093+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5dd7fd0 0x564ca5f5db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.093+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 0x564ca5f57a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.093+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564ca5e84fd0 con 0x564ca5ec3b80 2024-05-27T09:50:22.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.093+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564ca5e74b70 con 0x564ca5f57670 2024-05-27T09:50:22.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.093+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564ca5e78bb0 con 0x564ca5dd7fd0 2024-05-27T09:50:22.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.093+0000 7fa11e600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 0x564ca5f57a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.093+0000 7fa11e600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 0x564ca5f57a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:41148/0 (socket says 172.21.15.44:41148) 2024-05-27T09:50:22.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.093+0000 7fa11e600700 1 -- 172.21.15.44:0/224317914 learned_addr learned my addr 172.21.15.44:0/224317914 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.094+0000 7fa11dc00700 1 --2- 172.21.15.44:0/224317914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5ec3b80 0x564ca5f5d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.094+0000 7fa11d200700 1 --2- 172.21.15.44:0/224317914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5dd7fd0 0x564ca5f5db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.094+0000 7fa11e600700 1 -- 172.21.15.44:0/224317914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5dd7fd0 msgr2=0x564ca5f5db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.094+0000 7fa11e600700 1 --2- 172.21.15.44:0/224317914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5dd7fd0 0x564ca5f5db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.094+0000 7fa11e600700 1 -- 172.21.15.44:0/224317914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5ec3b80 msgr2=0x564ca5f5d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:22.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.094+0000 7fa11e600700 1 --2- 172.21.15.44:0/224317914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5ec3b80 0x564ca5f5d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.094+0000 7fa11e600700 1 -- 172.21.15.44:0/224317914 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ca5e3d5b0 con 0x564ca5f57670 2024-05-27T09:50:22.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.094+0000 7fa11e600700 1 --2- 172.21.15.44:0/224317914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 0x564ca5f57a80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009a40 tx=0x7fa10c009d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6e39ecf181341bf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:22.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.095+0000 7fa10aa00700 1 -- 172.21.15.44:0/224317914 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c017070 con 0x564ca5f57670 2024-05-27T09:50:22.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.095+0000 7fa10aa00700 1 -- 172.21.15.44:0/224317914 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa10c00ed80 con 0x564ca5f57670 2024-05-27T09:50:22.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.095+0000 7fa10aa00700 1 -- 172.21.15.44:0/224317914 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c0123d0 con 0x564ca5f57670 2024-05-27T09:50:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.097+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/224317914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 msgr2=0x564ca5f57a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.097+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/224317914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 0x564ca5f57a80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009a40 tx=0x7fa10c009d50 comp rx=0 tx=0).stop 2024-05-27T09:50:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.099+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/224317914 shutdown_connections 2024-05-27T09:50:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.099+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/224317914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 0x564ca5f57a80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.099+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/224317914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5ec3b80 0x564ca5f5d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.099+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/224317914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5dd7fd0 0x564ca5f5db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.099+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/224317914 >> 172.21.15.44:0/224317914 conn(0x564ca5e71520 msgr2=0x564ca5f526e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.099+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/224317914 shutdown_connections 2024-05-27T09:50:22.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.099+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/224317914 wait complete. 2024-05-27T09:50:22.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.100+0000 7fa12f0c8b80 1 -- start start 2024-05-27T09:50:22.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.101+0000 7fa12f0c8b80 1 Processor -- start 2024-05-27T09:50:22.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.102+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5dd7fd0 0x564ca5f61350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.102+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5ec3b80 0x564ca5f61890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.102+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5f57670 0x564ca5f61dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.102+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564ca5d7b130 con 0x564ca5ec3b80 2024-05-27T09:50:22.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.102+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564ca5e158f0 con 0x564ca5dd7fd0 2024-05-27T09:50:22.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.102+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564ca5f58460 con 0x564ca5f57670 2024-05-27T09:50:22.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.102+0000 7fa12f0c8b80 10 client.0 _ll_register_callbacks cb 0x564ca5e1ee00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:22.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.103+0000 7fa11e600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5f57670 0x564ca5f61dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.103+0000 7fa11e600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5f57670 0x564ca5f61dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:40406/0 (socket says 172.21.15.44:40406) 2024-05-27T09:50:22.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.103+0000 7fa11e600700 1 -- 172.21.15.44:0/1941456536 learned_addr learned my addr 172.21.15.44:0/1941456536 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:22.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.103+0000 7fa11dc00700 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5dd7fd0 0x564ca5f61350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.103+0000 7fa11e600700 1 -- 172.21.15.44:0/1941456536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5dd7fd0 msgr2=0x564ca5f61350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:22.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.103+0000 7fa11e600700 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5dd7fd0 0x564ca5f61350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.103+0000 7fa11e600700 1 -- 172.21.15.44:0/1941456536 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5ec3b80 msgr2=0x564ca5f61890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:22.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.103+0000 7fa11e600700 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5ec3b80 0x564ca5f61890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.103+0000 7fa11e600700 1 -- 172.21.15.44:0/1941456536 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ca5dc4a60 con 0x564ca5f57670 2024-05-27T09:50:22.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.103+0000 7fa11e600700 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5f57670 0x564ca5f61dd0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa10c006af0 tx=0x7fa10c0136a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3f27d095a176b24 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:22.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.103+0000 7fa0ff400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:50:22.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.104+0000 7fa0ff400700 1 -- 172.21.15.44:0/1941456536 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c017040 con 0x564ca5f57670 2024-05-27T09:50:22.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.104+0000 7fa0ff400700 1 -- 172.21.15.44:0/1941456536 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7fa10c01c450 con 0x564ca5f57670 2024-05-27T09:50:22.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.104+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1941456536 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa10c00ed80 con 0x564ca5f57670 2024-05-27T09:50:22.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.104+0000 7fa0ff400700 1 -- 172.21.15.44:0/1941456536 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa10c01c670 con 0x564ca5f57670 2024-05-27T09:50:22.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.105+0000 7fa12f0c8b80 10 client.28171 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:22.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.105+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1941456536 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x564ca5e451f0 con 0x564ca5f57670 2024-05-27T09:50:22.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.105+0000 7fa0ff400700 1 -- 172.21.15.44:0/1941456536 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa10c00ed80 con 0x564ca5f57670 2024-05-27T09:50:22.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.105+0000 7fa0ff400700 1 -- 172.21.15.44:0/1941456536 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fa10c023070 con 0x564ca5f57670 2024-05-27T09:50:22.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.105+0000 7fa12f0c8b80 10 client.28171 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:22.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.105+0000 7fa12f0c8b80 10 client.28171 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:50:22.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.106+0000 7fa12f0c8b80 20 client.28171 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:22.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.106+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564ca5f70f80 0x564ca5f733a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.106+0000 7fa12f0c8b80 4 client.28171 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:22.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.106+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1941456536 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ca5e86f20 con 0x564ca5f70f80 2024-05-27T09:50:22.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.107+0000 7fa11dc00700 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564ca5f70f80 0x564ca5f733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.109+0000 7fa11dc00700 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564ca5f70f80 0x564ca5f733a0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8adb76837117e67d server_cookie=e78fb3ca788e57b2 in_seq=0 out_seq=0 2024-05-27T09:50:22.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.109+0000 7fa0ff400700 10 client.28171 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:22.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.110+0000 7fa0ff400700 1 -- 172.21.15.44:0/1941456536 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564ca5e86f20 con 0x564ca5f70f80 2024-05-27T09:50:22.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.110+0000 7fa0ff400700 10 client.28171 handle_command_reply: tid=0 2024-05-27T09:50:22.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.110+0000 7fa12f0c8b80 1 client.28171 shutdown 2024-05-27T09:50:22.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.111+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1941456536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5f57670 msgr2=0x564ca5f61dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.111+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5f57670 0x564ca5f61dd0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa10c006af0 tx=0x7fa10c0136a0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.111+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1941456536 shutdown_connections 2024-05-27T09:50:22.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.111+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564ca5f70f80 0x564ca5f733a0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.111+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5f57670 0x564ca5f61dd0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.111+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5dd7fd0 0x564ca5f61350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.111+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/1941456536 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5ec3b80 0x564ca5f61890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.111+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1941456536 >> 172.21.15.44:0/1941456536 conn(0x564ca5e71520 msgr2=0x564ca5f56660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:22.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.111+0000 7fa0ff400700 0 client.28171 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:22.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.111+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1941456536 shutdown_connections 2024-05-27T09:50:22.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.112+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1941456536 wait complete. 2024-05-27T09:50:22.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.114+0000 7fa12f0c8b80 20 client.28171 trim_cache size 0 max 16384 2024-05-27T09:50:22.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.159+0000 7fa12f0c8b80 1 Processor -- start 2024-05-27T09:50:22.160 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:22 smithi044 ceph-mon[31618]: pgmap v476: 97 pgs: 13 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 61 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-05-27T09:50:22.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.159+0000 7fa12f0c8b80 1 -- start start 2024-05-27T09:50:22.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.159+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5dd7fd0 0x564ca5f61d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.159+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5ec3b80 0x564ca5f6d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.159+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5f57670 0x564ca5f52040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.159+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181c0800 con 0x564ca5f57670 2024-05-27T09:50:22.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.159+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181be030 con 0x564ca5ec3b80 2024-05-27T09:50:22.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.159+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181bbea0 con 0x564ca5dd7fd0 2024-05-27T09:50:22.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.160+0000 7fa11d200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5ec3b80 0x564ca5f6d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.160+0000 7fa11d200700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5ec3b80 0x564ca5f6d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:41162/0 (socket says 172.21.15.44:41162) 2024-05-27T09:50:22.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.160+0000 7fa11d200700 1 -- 172.21.15.44:0/1789838760 learned_addr learned my addr 172.21.15.44:0/1789838760 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.160+0000 7fa11dc00700 1 --2- 172.21.15.44:0/1789838760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5dd7fd0 0x564ca5f61d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.160+0000 7fa11d200700 1 -- 172.21.15.44:0/1789838760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5dd7fd0 msgr2=0x564ca5f61d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.160+0000 7fa11d200700 1 --2- 172.21.15.44:0/1789838760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5dd7fd0 0x564ca5f61d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.160+0000 7fa11d200700 1 -- 172.21.15.44:0/1789838760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5f57670 msgr2=0x564ca5f52040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.160+0000 7fa11d200700 1 --2- 172.21.15.44:0/1789838760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5f57670 0x564ca5f52040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.161+0000 7fa11d200700 1 -- 172.21.15.44:0/1789838760 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ca5f5cf90 con 0x564ca5ec3b80 2024-05-27T09:50:22.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.161+0000 7fa11d200700 1 --2- 172.21.15.44:0/1789838760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5ec3b80 0x564ca5f6d4a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa1040098e0 tx=0x7fa10401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7643093ffd1e2e0d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:22.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.164+0000 7fa10a000700 1 -- 172.21.15.44:0/1789838760 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10400f9c0 con 0x564ca5ec3b80 2024-05-27T09:50:22.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.164+0000 7fa10a000700 1 -- 172.21.15.44:0/1789838760 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa104033070 con 0x564ca5ec3b80 2024-05-27T09:50:22.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.164+0000 7fa10a000700 1 -- 172.21.15.44:0/1789838760 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104022de0 con 0x564ca5ec3b80 2024-05-27T09:50:22.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.165+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1789838760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5ec3b80 msgr2=0x564ca5f6d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.165+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/1789838760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5ec3b80 0x564ca5f6d4a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa1040098e0 tx=0x7fa10401d000 comp rx=0 tx=0).stop 2024-05-27T09:50:22.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.169+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1789838760 shutdown_connections 2024-05-27T09:50:22.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.169+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/1789838760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5f57670 0x564ca5f52040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.169+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/1789838760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5dd7fd0 0x564ca5f61d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.169+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/1789838760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5ec3b80 0x564ca5f6d4a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.169+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1789838760 >> 172.21.15.44:0/1789838760 conn(0x564ca5e71520 msgr2=0x564ca5f56cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:22.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.169+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1789838760 shutdown_connections 2024-05-27T09:50:22.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.169+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/1789838760 wait complete. 2024-05-27T09:50:22.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.170+0000 7fa12f0c8b80 1 -- start start 2024-05-27T09:50:22.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.171+0000 7fa12f0c8b80 1 Processor -- start 2024-05-27T09:50:22.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.172+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5dd7fd0 0x564ca5f75430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.173+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5ec3b80 0x564ca5f75970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.173+0000 7fa12f0c8b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 0x564ca5fd57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.173+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181bc4f0 con 0x564ca5dd7fd0 2024-05-27T09:50:22.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.173+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181bd960 con 0x564ca5f57670 2024-05-27T09:50:22.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.173+0000 7fa12f0c8b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181bcbc0 con 0x564ca5ec3b80 2024-05-27T09:50:22.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.174+0000 7fa11dc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5dd7fd0 0x564ca5f75430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.174+0000 7fa11dc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5dd7fd0 0x564ca5f75430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38394/0 (socket says 172.21.15.44:38394) 2024-05-27T09:50:22.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.174+0000 7fa11dc00700 1 -- 172.21.15.44:0/748364210 learned_addr learned my addr 172.21.15.44:0/748364210 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:22.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.174+0000 7fa11d200700 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5ec3b80 0x564ca5f75970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.174+0000 7fa11e600700 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 0x564ca5fd57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.174+0000 7fa11d200700 1 -- 172.21.15.44:0/748364210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 msgr2=0x564ca5fd57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:22.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.174+0000 7fa11d200700 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 0x564ca5fd57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.174+0000 7fa11d200700 1 -- 172.21.15.44:0/748364210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5dd7fd0 msgr2=0x564ca5f75430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:22.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.174+0000 7fa11d200700 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5dd7fd0 0x564ca5f75430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.174+0000 7fa11d200700 1 -- 172.21.15.44:0/748364210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ca5f76190 con 0x564ca5ec3b80 2024-05-27T09:50:22.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.175+0000 7fa11d200700 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5ec3b80 0x564ca5f75970 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa10400b5c0 tx=0x7fa104005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f63ebc2ec872414f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:22.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.175+0000 7fa12f0c8b80 10 client.0 _ll_register_callbacks cb 0x564ca5e1ee00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:22.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.175+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/748364210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa104033070 con 0x564ca5ec3b80 2024-05-27T09:50:22.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.175+0000 7fa0ff400700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:50:22.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.176+0000 7fa0ff400700 1 -- 172.21.15.44:0/748364210 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104045050 con 0x564ca5ec3b80 2024-05-27T09:50:22.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.176+0000 7fa0ff400700 1 -- 172.21.15.44:0/748364210 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7fa10400fbd0 con 0x564ca5ec3b80 2024-05-27T09:50:22.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.176+0000 7fa0ff400700 1 -- 172.21.15.44:0/748364210 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa10403b070 con 0x564ca5ec3b80 2024-05-27T09:50:22.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.177+0000 7fa12f0c8b80 10 client.28177 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:22.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.177+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/748364210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=43}) -- 0x564ca5fd6af0 con 0x564ca5ec3b80 2024-05-27T09:50:22.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.177+0000 7fa0ff400700 1 -- 172.21.15.44:0/748364210 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fa104040440 con 0x564ca5ec3b80 2024-05-27T09:50:22.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.178+0000 7fa12f0c8b80 10 client.28177 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:22.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.178+0000 7fa12f0c8b80 10 client.28177 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:50:22.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.178+0000 7fa12f0c8b80 20 client.28177 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:22.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.178+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564ca5fd7090 0x564ca5ff0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.178+0000 7fa12f0c8b80 4 client.28177 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.5, "await_for": 45.0}] 2024-05-27T09:50:22.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.178+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/748364210 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.5, "await_for": 45.0}) -- 0x564ca5f4a8a0 con 0x564ca5fd7090 2024-05-27T09:50:22.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.178+0000 7fa11dc00700 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564ca5fd7090 0x564ca5ff0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.179+0000 7fa11dc00700 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564ca5fd7090 0x564ca5ff0810 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbf14408c9911393 server_cookie=b6b1481f8729c031 in_seq=0 out_seq=0 2024-05-27T09:50:22.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.179+0000 7fa0ff400700 10 client.28177 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:22.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.195+0000 7fa0ff400700 1 -- 172.21.15.44:0/748364210 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x564ca5f4a8a0 con 0x564ca5fd7090 2024-05-27T09:50:22.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.196+0000 7fa0ff400700 10 client.28177 handle_command_reply: tid=0 2024-05-27T09:50:22.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.196+0000 7fa12f0c8b80 1 client.28177 shutdown 2024-05-27T09:50:22.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.196+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/748364210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5ec3b80 msgr2=0x564ca5f75970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.196+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5ec3b80 0x564ca5f75970 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa10400b5c0 tx=0x7fa104005fd0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.197+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/748364210 shutdown_connections 2024-05-27T09:50:22.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.197+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x564ca5fd7090 0x564ca5ff0810 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.197+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564ca5f57670 0x564ca5fd57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.197+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564ca5dd7fd0 0x564ca5f75430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.197+0000 7fa12f0c8b80 1 --2- 172.21.15.44:0/748364210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ca5ec3b80 0x564ca5f75970 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.197+0000 7fa0ff400700 0 client.28177 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:22.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.197+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/748364210 >> 172.21.15.44:0/748364210 conn(0x564ca5e71520 msgr2=0x564ca5f61b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:22.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.197+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/748364210 shutdown_connections 2024-05-27T09:50:22.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.197+0000 7fa12f0c8b80 1 -- 172.21.15.44:0/748364210 wait complete. 2024-05-27T09:50:22.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.198+0000 7fa12f0c8b80 20 client.28177 trim_cache size 0 max 16384 2024-05-27T09:50:22.200 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:50:22.200 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 43, 2024-05-27T09:50:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:50:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 38, 2024-05-27T09:50:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": { 2024-05-27T09:50:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "573513ce": { 2024-05-27T09:50:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "version": 38, 2024-05-27T09:50:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "age_ref": 0.0, 2024-05-27T09:50:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:50:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:50:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:50:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:50:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: "timeout": 90.0, 2024-05-27T09:50:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: "expiration": 142.5, 2024-05-27T09:50:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: "members": { 2024-05-27T09:50:22.203 INFO:teuthology.orchestra.run.smithi044.stdout: "file:/": { 2024-05-27T09:50:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "excluded": false, 2024-05-27T09:50:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "state": { 2024-05-27T09:50:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "QUIESCED", 2024-05-27T09:50:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "age": 0.0 2024-05-27T09:50:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:50:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:50:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:50:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:50:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:50:22.207 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:50:22.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.199+0000 7fa120600700 1 -- 172.21.15.44:0/2791414641 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa1000b2b80 msgr2=0x7fa1000b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.199+0000 7fa120600700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa1000b2b80 0x7fa1000b5090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa104003800 tx=0x7fa10401a040 comp rx=0 tx=0).stop 2024-05-27T09:50:22.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.199+0000 7fa120600700 1 -- 172.21.15.44:0/2791414641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa11810a920 msgr2=0x7fa1181af9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.199+0000 7fa120600700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa11810a920 0x7fa1181af9b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa110007d60 tx=0x7fa11000a340 comp rx=0 tx=0).stop 2024-05-27T09:50:22.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.199+0000 7fa120600700 1 -- 172.21.15.44:0/2791414641 shutdown_connections 2024-05-27T09:50:22.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.199+0000 7fa120600700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa1000b2b80 0x7fa1000b5090 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.199+0000 7fa120600700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa11811d920 0x7fa1181a9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.199+0000 7fa120600700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa11810a920 0x7fa1181af9b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.199+0000 7fa120600700 1 --2- 172.21.15.44:0/2791414641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11810b270 0x7fa1181afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.199+0000 7fa120600700 1 -- 172.21.15.44:0/2791414641 >> 172.21.15.44:0/2791414641 conn(0x7fa11806f4c0 msgr2=0x7fa11811b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:22.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.200+0000 7fa120600700 1 -- 172.21.15.44:0/2791414641 shutdown_connections 2024-05-27T09:50:22.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.200+0000 7fa120600700 1 -- 172.21.15.44:0/2791414641 wait complete. 2024-05-27T09:50:22.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:22 smithi189 ceph-mon[36671]: pgmap v476: 97 pgs: 13 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 61 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-05-27T09:50:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.424+0000 7f7fc7600700 1 -- 172.21.15.44:0/1964529021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc010b270 msgr2=0x7f7fc011d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.424+0000 7f7fc7600700 1 --2- 172.21.15.44:0/1964529021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc010b270 0x7f7fc011d3e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009990 tx=0x7f7fa8009ca0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.424+0000 7f7fc7600700 1 -- 172.21.15.44:0/1964529021 shutdown_connections 2024-05-27T09:50:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.424+0000 7f7fc7600700 1 --2- 172.21.15.44:0/1964529021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc011d920 0x7f7fc011fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.424+0000 7f7fc7600700 1 --2- 172.21.15.44:0/1964529021 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fc010a920 0x7f7fc010ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.424+0000 7f7fc7600700 1 --2- 172.21.15.44:0/1964529021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc010b270 0x7f7fc011d3e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.425+0000 7f7fc7600700 1 -- 172.21.15.44:0/1964529021 >> 172.21.15.44:0/1964529021 conn(0x7f7fc006f4c0 msgr2=0x7f7fc006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:22.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.425+0000 7f7fc7600700 1 -- 172.21.15.44:0/1964529021 shutdown_connections 2024-05-27T09:50:22.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.425+0000 7f7fc7600700 1 -- 172.21.15.44:0/1964529021 wait complete. 2024-05-27T09:50:22.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.427+0000 7f7fc7600700 1 Processor -- start 2024-05-27T09:50:22.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.428+0000 7f7fc7600700 1 -- start start 2024-05-27T09:50:22.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.429+0000 7f7fc7600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc010a920 0x7f7fc01af8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.430+0000 7f7fc7600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc010b270 0x7f7fc01afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.430+0000 7f7fc7600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fc011d920 0x7f7fc01a9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.430+0000 7f7fc4c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc010a920 0x7f7fc01af8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.431+0000 7f7fbfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc010b270 0x7f7fc01afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.431+0000 7f7fbfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc010b270 0x7f7fc01afe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:40438/0 (socket says 172.21.15.44:40438) 2024-05-27T09:50:22.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.431+0000 7f7fbfe00700 1 -- 172.21.15.44:0/779509834 learned_addr learned my addr 172.21.15.44:0/779509834 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:22.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.431+0000 7f7fc5600700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fc011d920 0x7f7fc01a9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.432+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc01232e0 con 0x7f7fc011d920 2024-05-27T09:50:22.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.432+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0123160 con 0x7f7fc010a920 2024-05-27T09:50:22.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.432+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0123460 con 0x7f7fc010b270 2024-05-27T09:50:22.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.433+0000 7f7fc4c00700 1 -- 172.21.15.44:0/779509834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc010b270 msgr2=0x7f7fc01afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.433+0000 7f7fc4c00700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc010b270 0x7f7fc01afe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.433+0000 7f7fc4c00700 1 -- 172.21.15.44:0/779509834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fc011d920 msgr2=0x7f7fc01a9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.433+0000 7f7fc4c00700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fc011d920 0x7f7fc01a9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.433+0000 7f7fc4c00700 1 -- 172.21.15.44:0/779509834 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc01aa1c0 con 0x7f7fc010a920 2024-05-27T09:50:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.433+0000 7f7fc5600700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fc011d920 0x7f7fc01a9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:22.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.436+0000 7f7fc4c00700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc010a920 0x7f7fc01af8a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800ada0 tx=0x7f7fb800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.436+0000 7f7fbd600700 1 -- 172.21.15.44:0/779509834 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8017020 con 0x7f7fc010a920 2024-05-27T09:50:22.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.436+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc01aa4b0 con 0x7f7fc010a920 2024-05-27T09:50:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.436+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc01b6750 con 0x7f7fc010a920 2024-05-27T09:50:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.437+0000 7f7fbd600700 1 -- 172.21.15.44:0/779509834 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb800ce30 con 0x7f7fc010a920 2024-05-27T09:50:22.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.437+0000 7f7fbd600700 1 -- 172.21.15.44:0/779509834 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8005280 con 0x7f7fc010a920 2024-05-27T09:50:22.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.439+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f9c005360 con 0x7f7fc010a920 2024-05-27T09:50:22.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.442+0000 7f7fbd600700 1 -- 172.21.15.44:0/779509834 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f7fb80054e0 con 0x7f7fc010a920 2024-05-27T09:50:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.445+0000 7f7fbd600700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7fa40b2af0 0x7f7fa40b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.445+0000 7f7fbd600700 1 -- 172.21.15.44:0/779509834 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f7fb80e1c10 con 0x7f7fc010a920 2024-05-27T09:50:22.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.446+0000 7f7fbfe00700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7fa40b2af0 0x7f7fa40b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:22.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.448+0000 7f7fbfe00700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7fa40b2af0 0x7f7fa40b5000 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0120140 tx=0x7f7fa800b280 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:22.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.448+0000 7f7fbd600700 1 -- 172.21.15.44:0/779509834 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7fb8021020 con 0x7f7fc010a920 2024-05-27T09:50:22.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:22 smithi050 ceph-mon[36371]: pgmap v476: 97 pgs: 13 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 61 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-05-27T09:50:22.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.928+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f9c005920 con 0x7f7fc010a920 2024-05-27T09:50:22.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.930+0000 7f7fbd600700 1 -- 172.21.15.44:0/779509834 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11870 (secure 0 0 0) 0x7f7fb8004ad0 con 0x7f7fc010a920 2024-05-27T09:50:22.931 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:50:22.931 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:48:16.864365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24503]},"id":1}]} 2024-05-27T09:50:22.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.934+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7fa40b2af0 msgr2=0x7f7fa40b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.934+0000 7f7fc7600700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7fa40b2af0 0x7f7fa40b5000 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0120140 tx=0x7f7fa800b280 comp rx=0 tx=0).stop 2024-05-27T09:50:22.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.934+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc010a920 msgr2=0x7f7fc01af8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:22.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.934+0000 7f7fc7600700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc010a920 0x7f7fc01af8a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800ada0 tx=0x7f7fb800c330 comp rx=0 tx=0).stop 2024-05-27T09:50:22.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.934+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 shutdown_connections 2024-05-27T09:50:22.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.934+0000 7f7fc7600700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7fa40b2af0 0x7f7fa40b5000 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.934+0000 7f7fc7600700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fc011d920 0x7f7fc01a9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.934+0000 7f7fc7600700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc010a920 0x7f7fc01af8a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.934+0000 7f7fc7600700 1 --2- 172.21.15.44:0/779509834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc010b270 0x7f7fc01afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.934+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 >> 172.21.15.44:0/779509834 conn(0x7f7fc006f4c0 msgr2=0x7f7fc011b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.934+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 shutdown_connections 2024-05-27T09:50:22.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:22.935+0000 7f7fc7600700 1 -- 172.21.15.44:0/779509834 wait complete. 2024-05-27T09:50:22.942 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 43 2024-05-27T09:50:23.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:23 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/779509834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:23.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:23 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/779509834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:23.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:23 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/779509834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:23.992 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '573513ce', quiesce took 15.9 seconds. Will release after: 6.6 2024-05-27T09:50:24.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:24 smithi044 ceph-mon[31618]: pgmap v477: 97 pgs: 13 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 61 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.6 KiB/s wr, 9 op/s 2024-05-27T09:50:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:24 smithi189 ceph-mon[36671]: pgmap v477: 97 pgs: 13 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 61 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.6 KiB/s wr, 9 op/s 2024-05-27T09:50:24.449 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:24.490 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24542 quiesce db --cancel --all' 2024-05-27T09:50:24.491 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24542 quiesce db --cancel --all 2024-05-27T09:50:24.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:24 smithi050 ceph-mon[36371]: pgmap v477: 97 pgs: 13 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 61 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.6 KiB/s wr, 9 op/s 2024-05-27T09:50:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.629+0000 7f0c1b200640 1 -- 172.21.15.44:0/1127234613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c116100 msgr2=0x7f0c1c118520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.629+0000 7f0c1b200640 1 --2- 172.21.15.44:0/1127234613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c116100 0x7f0c1c118520 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009860 tx=0x7f0c0802a3d0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.629+0000 7f0c1b200640 1 -- 172.21.15.44:0/1127234613 shutdown_connections 2024-05-27T09:50:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.629+0000 7f0c1b200640 1 --2- 172.21.15.44:0/1127234613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c1c118a60 0x7f0c1c11aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.629+0000 7f0c1b200640 1 --2- 172.21.15.44:0/1127234613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c116100 0x7f0c1c118520 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.629+0000 7f0c1b200640 1 --2- 172.21.15.44:0/1127234613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c1c10a2c0 0x7f0c1c10a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.629+0000 7f0c1b200640 1 -- 172.21.15.44:0/1127234613 >> 172.21.15.44:0/1127234613 conn(0x7f0c1c06f800 msgr2=0x7f0c1c06fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.629+0000 7f0c1b200640 1 -- 172.21.15.44:0/1127234613 shutdown_connections 2024-05-27T09:50:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.629+0000 7f0c1b200640 1 -- 172.21.15.44:0/1127234613 wait complete. 2024-05-27T09:50:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.631+0000 7f0c1b200640 1 Processor -- start 2024-05-27T09:50:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.631+0000 7f0c1b200640 1 -- start start 2024-05-27T09:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.632+0000 7f0c1b200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c1c10a2c0 0x7f0c1c113080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.632+0000 7f0c1b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c116100 0x7f0c1c1135c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.632+0000 7f0c1b200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c1c118a60 0x7f0c1c113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.632+0000 7f0c1b200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c11e6a0 con 0x7f0c1c10a2c0 2024-05-27T09:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.632+0000 7f0c19400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c116100 0x7f0c1c1135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.632+0000 7f0c1b200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c11e520 con 0x7f0c1c118a60 2024-05-27T09:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.632+0000 7f0c19400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c116100 0x7f0c1c1135c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44748/0 (socket says 172.21.15.44:44748) 2024-05-27T09:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.632+0000 7f0c19400640 1 -- 172.21.15.44:0/2983295814 learned_addr learned my addr 172.21.15.44:0/2983295814 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.632+0000 7f0c1a800640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c1c118a60 0x7f0c1c113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.632+0000 7f0c1b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c11e820 con 0x7f0c1c116100 2024-05-27T09:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.633+0000 7f0c1a800640 1 -- 172.21.15.44:0/2983295814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c116100 msgr2=0x7f0c1c1135c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.633+0000 7f0c1a800640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c116100 0x7f0c1c1135c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.633+0000 7f0c1a800640 1 -- 172.21.15.44:0/2983295814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c1c10a2c0 msgr2=0x7f0c1c113080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.633+0000 7f0c1a800640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c1c10a2c0 0x7f0c1c113080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.633+0000 7f0c1a800640 1 -- 172.21.15.44:0/2983295814 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1c1aae00 con 0x7f0c1c118a60 2024-05-27T09:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.633+0000 7f0c19400640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c116100 0x7f0c1c1135c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.633+0000 7f0c1a800640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c1c118a60 0x7f0c1c113b00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0c14009fd0 tx=0x7f0c1400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.634+0000 7f0c12a00640 1 -- 172.21.15.44:0/2983295814 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1400ce20 con 0x7f0c1c118a60 2024-05-27T09:50:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.634+0000 7f0c12a00640 1 -- 172.21.15.44:0/2983295814 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c14005020 con 0x7f0c1c118a60 2024-05-27T09:50:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.634+0000 7f0c12a00640 1 -- 172.21.15.44:0/2983295814 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c14010690 con 0x7f0c1c118a60 2024-05-27T09:50:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.634+0000 7f0c1b200640 1 -- 172.21.15.44:0/2983295814 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1c1ab0f0 con 0x7f0c1c118a60 2024-05-27T09:50:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.634+0000 7f0c1b200640 1 -- 172.21.15.44:0/2983295814 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c1c1ab5f0 con 0x7f0c1c118a60 2024-05-27T09:50:25.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.638+0000 7f0c12a00640 1 -- 172.21.15.44:0/2983295814 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0c140054d0 con 0x7f0c1c118a60 2024-05-27T09:50:25.638 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.638+0000 7f0c27c24740 1 Processor -- start 2024-05-27T09:50:25.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.638+0000 7f0c27c24740 1 -- start start 2024-05-27T09:50:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.639+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e29a1a1b0 0x560e29dbca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.639+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dbcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.639+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dbd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.639+0000 7f0c27c24740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560e29d2db90 con 0x560e299f12f0 2024-05-27T09:50:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.639+0000 7f0c27c24740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560e29d1d730 con 0x560e299f94d0 2024-05-27T09:50:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.639+0000 7f0c27c24740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560e29d21770 con 0x560e29a1a1b0 2024-05-27T09:50:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.639+0000 7f0c19400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dbd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.639+0000 7f0c19400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dbd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:59090/0 (socket says 172.21.15.44:59090) 2024-05-27T09:50:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.639+0000 7f0c19400640 1 -- 172.21.15.44:0/901739528 learned_addr learned my addr 172.21.15.44:0/901739528 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.639+0000 7f0c19400640 1 --2- 172.21.15.44:0/901739528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e29a1a1b0 0x560e29dbca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.640+0000 7f0c19e00640 1 --2- 172.21.15.44:0/901739528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dbcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.641+0000 7f0c19400640 1 -- 172.21.15.44:0/901739528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e29a1a1b0 msgr2=0x560e29dbca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.641+0000 7f0c19400640 1 --2- 172.21.15.44:0/901739528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e29a1a1b0 0x560e29dbca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.641+0000 7f0c19400640 1 -- 172.21.15.44:0/901739528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 msgr2=0x560e29dbcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.641+0000 7f0c19400640 1 --2- 172.21.15.44:0/901739528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dbcfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.641+0000 7f0c19400640 1 -- 172.21.15.44:0/901739528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e29cb68b0 con 0x560e299f94d0 2024-05-27T09:50:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.641+0000 7f0c19400640 1 --2- 172.21.15.44:0/901739528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dbd4e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0c080095f0 tx=0x7f0c08005e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c957f3b7352676ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.642+0000 7f0c07000640 1 -- 172.21.15.44:0/901739528 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c08030bd0 con 0x560e299f94d0 2024-05-27T09:50:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.642+0000 7f0c07000640 1 -- 172.21.15.44:0/901739528 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c08030d70 con 0x560e299f94d0 2024-05-27T09:50:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.642+0000 7f0c07000640 1 -- 172.21.15.44:0/901739528 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c08003ca0 con 0x560e299f94d0 2024-05-27T09:50:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.642+0000 7f0c12a00640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c000b2a50 0x7f0c000b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.643+0000 7f0c27c24740 1 -- 172.21.15.44:0/901739528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 msgr2=0x560e29dbd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.643+0000 7f0c27c24740 1 --2- 172.21.15.44:0/901739528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dbd4e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0c080095f0 tx=0x7f0c08005e10 comp rx=0 tx=0).stop 2024-05-27T09:50:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.643+0000 7f0c12a00640 1 -- 172.21.15.44:0/2983295814 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f0c140e2db0 con 0x7f0c1c118a60 2024-05-27T09:50:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.643+0000 7f0c27c24740 1 -- 172.21.15.44:0/901739528 shutdown_connections 2024-05-27T09:50:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.643+0000 7f0c27c24740 1 --2- 172.21.15.44:0/901739528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dbd4e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.643+0000 7f0c27c24740 1 --2- 172.21.15.44:0/901739528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dbcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.643+0000 7f0c27c24740 1 --2- 172.21.15.44:0/901739528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e29a1a1b0 0x560e29dbca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.643+0000 7f0c19e00640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c000b2a50 0x7f0c000b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.643+0000 7f0c27c24740 1 -- 172.21.15.44:0/901739528 >> 172.21.15.44:0/901739528 conn(0x560e29a0e300 msgr2=0x560e29db9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.643+0000 7f0c27c24740 1 -- 172.21.15.44:0/901739528 shutdown_connections 2024-05-27T09:50:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.643+0000 7f0c27c24740 1 -- 172.21.15.44:0/901739528 wait complete. 2024-05-27T09:50:25.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.644+0000 7f0c19e00640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c000b2a50 0x7f0c000b4f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c005ab0 tx=0x7f0c0c005a20 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:25.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.644+0000 7f0c27c24740 1 -- start start 2024-05-27T09:50:25.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.644+0000 7f0c27c24740 1 Processor -- start 2024-05-27T09:50:25.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.645+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dc6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.645+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 0x560e29dc71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.645+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 0x560e29dd2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.645+0000 7f0c27c24740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560e29a03790 con 0x560e299f12f0 2024-05-27T09:50:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.645+0000 7f0c27c24740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560e29cc5bc0 con 0x560e29a1a1b0 2024-05-27T09:50:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.645+0000 7f0c27c24740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560e29dc9b30 con 0x560e299f94d0 2024-05-27T09:50:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c1a800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 0x560e29dd2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c19e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dc6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c1a800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 0x560e29dd2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:59106/0 (socket says 172.21.15.44:59106) 2024-05-27T09:50:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c1a800640 1 -- 172.21.15.44:0/694971287 learned_addr learned my addr 172.21.15.44:0/694971287 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c19400640 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 0x560e29dc71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c27c24740 10 client.0 _ll_register_callbacks cb 0x560e29ca3400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c1a800640 1 -- 172.21.15.44:0/694971287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 msgr2=0x560e29dc71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c1a800640 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 0x560e29dc71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c1a800640 1 -- 172.21.15.44:0/694971287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 msgr2=0x560e29dc6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c1a800640 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dc6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c1a800640 1 -- 172.21.15.44:0/694971287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e29a32210 con 0x560e29a1a1b0 2024-05-27T09:50:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c19e00640 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dc6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.646+0000 7f0c19400640 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 0x560e29dc71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.647+0000 7f0c1a800640 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 0x560e29dd2e90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0c140a6030 tx=0x7f0c140a9bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a62e6c62cae4739f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.647+0000 7f0bfb400640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.647+0000 7f0bfb400640 1 -- 172.21.15.44:0/694971287 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c140c3020 con 0x560e29a1a1b0 2024-05-27T09:50:25.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.647+0000 7f0bfb400640 1 -- 172.21.15.44:0/694971287 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f0c140b2980 con 0x560e29a1a1b0 2024-05-27T09:50:25.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.647+0000 7f0c27c24740 1 -- 172.21.15.44:0/694971287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c08030d70 con 0x560e29a1a1b0 2024-05-27T09:50:25.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.647+0000 7f0bfb400640 1 -- 172.21.15.44:0/694971287 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0c140e08b0 con 0x560e29a1a1b0 2024-05-27T09:50:25.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.647+0000 7f0c27c24740 10 client.28187 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:25.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.647+0000 7f0c27c24740 1 -- 172.21.15.44:0/694971287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x560e29afe620 con 0x560e29a1a1b0 2024-05-27T09:50:25.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.648+0000 7f0bfb400640 1 -- 172.21.15.44:0/694971287 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c08030d70 con 0x560e29a1a1b0 2024-05-27T09:50:25.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.648+0000 7f0bfb400640 1 -- 172.21.15.44:0/694971287 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f0c140a5920 con 0x560e29a1a1b0 2024-05-27T09:50:25.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.648+0000 7f0c27c24740 10 client.28187 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:25.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.648+0000 7f0c27c24740 10 client.28187 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:25.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.648+0000 7f0c27c24740 20 client.28187 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:25.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.648+0000 7f0c27c24740 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e299d8d80 0x560e29dd8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.648+0000 7f0c27c24740 4 client.28187 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:25.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.648+0000 7f0c27c24740 1 -- 172.21.15.44:0/694971287 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e29d2fae0 con 0x560e299d8d80 2024-05-27T09:50:25.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.649+0000 7f0c19e00640 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e299d8d80 0x560e29dd8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.649+0000 7f0c19e00640 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e299d8d80 0x560e29dd8f50 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ecb07f9cbc7aab1 server_cookie=24299e5b4010ffb2 in_seq=0 out_seq=0 2024-05-27T09:50:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.649+0000 7f0bfb400640 10 client.28187 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.650+0000 7f0bfb400640 1 -- 172.21.15.44:0/694971287 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560e29d2fae0 con 0x560e299d8d80 2024-05-27T09:50:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.650+0000 7f0bfb400640 10 client.28187 handle_command_reply: tid=0 2024-05-27T09:50:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.650+0000 7f0c27c24740 1 client.28187 shutdown 2024-05-27T09:50:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.650+0000 7f0c27c24740 1 -- 172.21.15.44:0/694971287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 msgr2=0x560e29dd2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.650+0000 7f0c27c24740 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 0x560e29dd2e90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0c140a6030 tx=0x7f0c140a9bf0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.651+0000 7f0c27c24740 1 -- 172.21.15.44:0/694971287 shutdown_connections 2024-05-27T09:50:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.651+0000 7f0c27c24740 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e299d8d80 0x560e29dd8f50 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.651+0000 7f0c27c24740 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 0x560e29dd2e90 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.651+0000 7f0c27c24740 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 0x560e29dc71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.651+0000 7f0c27c24740 1 --2- 172.21.15.44:0/694971287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dc6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.651+0000 7f0c27c24740 1 -- 172.21.15.44:0/694971287 >> 172.21.15.44:0/694971287 conn(0x560e29a0e300 msgr2=0x560e29dbe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.651+0000 7f0bfb400640 0 client.28187 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.651+0000 7f0c27c24740 1 -- 172.21.15.44:0/694971287 shutdown_connections 2024-05-27T09:50:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.651+0000 7f0c27c24740 1 -- 172.21.15.44:0/694971287 wait complete. 2024-05-27T09:50:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.651+0000 7f0c27c24740 20 client.28187 trim_cache size 0 max 16384 2024-05-27T09:50:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.678+0000 7f0c27c24740 1 Processor -- start 2024-05-27T09:50:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.678+0000 7f0c27c24740 1 -- start start 2024-05-27T09:50:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.678+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f12f0 0x560e29dd2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.678+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dd3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.678+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e29a1a1b0 0x560e29dde750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.678+0000 7f0c27c24740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c1c9af0 con 0x560e29a1a1b0 2024-05-27T09:50:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.678+0000 7f0c27c24740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c1c7360 con 0x560e299f94d0 2024-05-27T09:50:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.678+0000 7f0c27c24740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c1c51d0 con 0x560e299f12f0 2024-05-27T09:50:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.679+0000 7f0c19400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dd3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.679+0000 7f0c19e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f12f0 0x560e29dd2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.679+0000 7f0c19400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dd3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:59116/0 (socket says 172.21.15.44:59116) 2024-05-27T09:50:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.679+0000 7f0c19400640 1 -- 172.21.15.44:0/1447143459 learned_addr learned my addr 172.21.15.44:0/1447143459 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.679+0000 7f0c19e00640 1 -- 172.21.15.44:0/1447143459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 msgr2=0x560e29dd3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.679+0000 7f0c1a800640 1 --2- 172.21.15.44:0/1447143459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e29a1a1b0 0x560e29dde750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.679+0000 7f0c19e00640 1 --2- 172.21.15.44:0/1447143459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dd3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.679+0000 7f0c19e00640 1 -- 172.21.15.44:0/1447143459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e29a1a1b0 msgr2=0x560e29dde750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.679+0000 7f0c19e00640 1 --2- 172.21.15.44:0/1447143459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e29a1a1b0 0x560e29dde750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.679+0000 7f0c19e00640 1 -- 172.21.15.44:0/1447143459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e29dd3680 con 0x560e299f12f0 2024-05-27T09:50:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.680+0000 7f0c1a800640 1 --2- 172.21.15.44:0/1447143459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e29a1a1b0 0x560e29dde750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:50:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.680+0000 7f0c19400640 1 --2- 172.21.15.44:0/1447143459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dd3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.680+0000 7f0c19e00640 1 --2- 172.21.15.44:0/1447143459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f12f0 0x560e29dd2c00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x560e29dbde10 tx=0x7f0c0c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38839824ace5a6f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.680+0000 7f0c06600640 1 -- 172.21.15.44:0/1447143459 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c02e070 con 0x560e299f12f0 2024-05-27T09:50:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.681+0000 7f0c06600640 1 -- 172.21.15.44:0/1447143459 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c0c0211d0 con 0x560e299f12f0 2024-05-27T09:50:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.681+0000 7f0c06600640 1 -- 172.21.15.44:0/1447143459 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c0285c0 con 0x560e299f12f0 2024-05-27T09:50:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.681+0000 7f0c27c24740 1 -- 172.21.15.44:0/1447143459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f12f0 msgr2=0x560e29dd2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.681+0000 7f0c27c24740 1 --2- 172.21.15.44:0/1447143459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f12f0 0x560e29dd2c00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x560e29dbde10 tx=0x7f0c0c014000 comp rx=0 tx=0).stop 2024-05-27T09:50:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.682+0000 7f0c27c24740 1 -- 172.21.15.44:0/1447143459 shutdown_connections 2024-05-27T09:50:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.682+0000 7f0c27c24740 1 --2- 172.21.15.44:0/1447143459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e29a1a1b0 0x560e29dde750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.682+0000 7f0c27c24740 1 --2- 172.21.15.44:0/1447143459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e299f94d0 0x560e29dd3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.682+0000 7f0c27c24740 1 --2- 172.21.15.44:0/1447143459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f12f0 0x560e29dd2c00 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.682+0000 7f0c27c24740 1 -- 172.21.15.44:0/1447143459 >> 172.21.15.44:0/1447143459 conn(0x560e29a0e300 msgr2=0x560e29dc8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.682+0000 7f0c27c24740 1 -- 172.21.15.44:0/1447143459 shutdown_connections 2024-05-27T09:50:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.682+0000 7f0c27c24740 1 -- 172.21.15.44:0/1447143459 wait complete. 2024-05-27T09:50:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.683+0000 7f0c27c24740 1 -- start start 2024-05-27T09:50:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.683+0000 7f0c27c24740 1 Processor -- start 2024-05-27T09:50:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dcf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 0x560e29dcf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c27c24740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 0x560e29dca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c27c24740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c1c5820 con 0x560e299f12f0 2024-05-27T09:50:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c27c24740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c1c6c90 con 0x560e29a1a1b0 2024-05-27T09:50:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c27c24740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c1c5ef0 con 0x560e299f94d0 2024-05-27T09:50:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c27c24740 10 client.0 _ll_register_callbacks cb 0x7f0c14005c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c19e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dcf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c19e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dcf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56862/0 (socket says 172.21.15.44:56862) 2024-05-27T09:50:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c19400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 0x560e29dcf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c1a800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 0x560e29dca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c19400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 0x560e29dcf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44788/0 (socket says 172.21.15.44:44788) 2024-05-27T09:50:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.684+0000 7f0c19400640 1 -- 172.21.15.44:0/412787307 learned_addr learned my addr 172.21.15.44:0/412787307 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.685+0000 7f0c19400640 1 -- 172.21.15.44:0/412787307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 msgr2=0x560e29dca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.685+0000 7f0c19400640 1 --2- 172.21.15.44:0/412787307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 0x560e29dca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.685+0000 7f0c19400640 1 -- 172.21.15.44:0/412787307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 msgr2=0x560e29dcf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.685+0000 7f0c19400640 1 --2- 172.21.15.44:0/412787307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dcf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.685+0000 7f0c19400640 1 -- 172.21.15.44:0/412787307 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e29dcabc0 con 0x560e299f94d0 2024-05-27T09:50:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.686+0000 7f0c19400640 1 --2- 172.21.15.44:0/412787307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 0x560e29dcf590 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0c0802a990 tx=0x7f0c08004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63fded44d7d2521c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.686+0000 7f0bfb400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:50:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.686+0000 7f0bfb400640 1 -- 172.21.15.44:0/412787307 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c08058060 con 0x560e299f94d0 2024-05-27T09:50:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.686+0000 7f0c27c24740 1 -- 172.21.15.44:0/412787307 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c0c0211d0 con 0x560e299f94d0 2024-05-27T09:50:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.686+0000 7f0bfb400640 1 -- 172.21.15.44:0/412787307 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f0c08032920 con 0x560e299f94d0 2024-05-27T09:50:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.687+0000 7f0bfb400640 1 -- 172.21.15.44:0/412787307 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0c08032b20 con 0x560e299f94d0 2024-05-27T09:50:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.687+0000 7f0c27c24740 10 client.28201 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.687+0000 7f0c27c24740 1 -- 172.21.15.44:0/412787307 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x560e29e40fe0 con 0x560e299f94d0 2024-05-27T09:50:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.687+0000 7f0bfb400640 1 -- 172.21.15.44:0/412787307 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c0c0211d0 con 0x560e299f94d0 2024-05-27T09:50:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.687+0000 7f0bfb400640 1 -- 172.21.15.44:0/412787307 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f0c080536e0 con 0x560e299f94d0 2024-05-27T09:50:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.687+0000 7f0c27c24740 10 client.28201 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.687+0000 7f0c27c24740 10 client.28201 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.687+0000 7f0c27c24740 20 client.28201 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.687+0000 7f0c27c24740 1 --2- 172.21.15.44:0/412787307 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e299ae1c0 0x560e29e431c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.687+0000 7f0c27c24740 4 client.28201 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T09:50:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.687+0000 7f0c27c24740 1 -- 172.21.15.44:0/412787307 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560e29d292c0 con 0x560e299ae1c0 2024-05-27T09:50:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.688+0000 7f0c19e00640 1 --2- 172.21.15.44:0/412787307 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e299ae1c0 0x560e29e431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.689+0000 7f0c19e00640 1 --2- 172.21.15.44:0/412787307 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e299ae1c0 0x560e29e431c0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bba309b62b1bffea server_cookie=acc4271150b3fdf7 in_seq=0 out_seq=0 2024-05-27T09:50:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.689+0000 7f0bfb400640 10 client.28201 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:25.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.706+0000 7f0bfb400640 1 -- 172.21.15.44:0/412787307 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x560e29d292c0 con 0x560e299ae1c0 2024-05-27T09:50:25.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.707+0000 7f0bfb400640 10 client.28201 handle_command_reply: tid=0 2024-05-27T09:50:25.707 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.707+0000 7f0c27c24740 1 client.28201 shutdown 2024-05-27T09:50:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.707+0000 7f0c27c24740 1 -- 172.21.15.44:0/412787307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 msgr2=0x560e29dcf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.707+0000 7f0c27c24740 1 --2- 172.21.15.44:0/412787307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 0x560e29dcf590 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0c0802a990 tx=0x7f0c08004270 comp rx=0 tx=0).stop 2024-05-27T09:50:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.707+0000 7f0c27c24740 1 -- 172.21.15.44:0/412787307 shutdown_connections 2024-05-27T09:50:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.707+0000 7f0c27c24740 1 --2- 172.21.15.44:0/412787307 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x560e299ae1c0 0x560e29e431c0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.707+0000 7f0c27c24740 1 --2- 172.21.15.44:0/412787307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560e29a1a1b0 0x560e29dca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.707+0000 7f0c27c24740 1 --2- 172.21.15.44:0/412787307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e299f94d0 0x560e29dcf590 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.707+0000 7f0c27c24740 1 --2- 172.21.15.44:0/412787307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e299f12f0 0x560e29dcf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.708+0000 7f0c27c24740 1 -- 172.21.15.44:0/412787307 >> 172.21.15.44:0/412787307 conn(0x560e29a0e300 msgr2=0x560e29dc13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.708+0000 7f0bfb400640 0 client.28201 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.708+0000 7f0c27c24740 1 -- 172.21.15.44:0/412787307 shutdown_connections 2024-05-27T09:50:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.709+0000 7f0c27c24740 1 -- 172.21.15.44:0/412787307 wait complete. 2024-05-27T09:50:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.709+0000 7f0c27c24740 20 client.28201 trim_cache size 0 max 16384 2024-05-27T09:50:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-05-27T09:50:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-05-27T09:50:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-05-27T09:50:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 -- 172.21.15.44:0/2983295814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c000b2a50 msgr2=0x7f0c000b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c000b2a50 0x7f0c000b4f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c005ab0 tx=0x7f0c0c005a20 comp rx=0 tx=0).stop 2024-05-27T09:50:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 -- 172.21.15.44:0/2983295814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c1c118a60 msgr2=0x7f0c1c113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c1c118a60 0x7f0c1c113b00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0c14009fd0 tx=0x7f0c1400ef10 comp rx=0 tx=0).stop 2024-05-27T09:50:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 -- 172.21.15.44:0/2983295814 shutdown_connections 2024-05-27T09:50:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c1c118a60 0x7f0c1c113b00 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c116100 0x7f0c1c1135c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0c000b2a50 0x7f0c000b4f60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 --2- 172.21.15.44:0/2983295814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c1c10a2c0 0x7f0c1c113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 -- 172.21.15.44:0/2983295814 >> 172.21.15.44:0/2983295814 conn(0x7f0c1c06f800 msgr2=0x7f0c1c116510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 -- 172.21.15.44:0/2983295814 shutdown_connections 2024-05-27T09:50:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-05-27T09:50:25.710+0000 7f0c1b200640 1 -- 172.21.15.44:0/2983295814 wait complete. 2024-05-27T09:50:25.833 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:26.070 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:26 smithi044 ceph-mon[31618]: pgmap v478: 97 pgs: 13 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 61 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.6 KiB/s wr, 9 op/s 2024-05-27T09:50:26.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:26 smithi189 ceph-mon[36671]: pgmap v478: 97 pgs: 13 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 61 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.6 KiB/s wr, 9 op/s 2024-05-27T09:50:26.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:26 smithi050 ceph-mon[36371]: pgmap v478: 97 pgs: 13 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 61 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.6 KiB/s wr, 9 op/s 2024-05-27T09:50:27.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.619+0000 7fe33a000700 1 -- 172.21.15.44:0/434623528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe33410b890 msgr2=0x7fe3341131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:27.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.619+0000 7fe33a000700 1 --2- 172.21.15.44:0/434623528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe33410b890 0x7fe3341131f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe32c009fc0 tx=0x7fe32c00b330 comp rx=0 tx=0).stop 2024-05-27T09:50:27.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.619+0000 7fe33a000700 1 -- 172.21.15.44:0/434623528 shutdown_connections 2024-05-27T09:50:27.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.619+0000 7fe33a000700 1 --2- 172.21.15.44:0/434623528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe33410b890 0x7fe3341131f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.619+0000 7fe33a000700 1 --2- 172.21.15.44:0/434623528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe334106610 0x7fe334106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.619+0000 7fe33a000700 1 --2- 172.21.15.44:0/434623528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe33410af40 0x7fe33410b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.619+0000 7fe33a000700 1 -- 172.21.15.44:0/434623528 >> 172.21.15.44:0/434623528 conn(0x7fe334102020 msgr2=0x7fe334104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.620+0000 7fe33a000700 1 -- 172.21.15.44:0/434623528 shutdown_connections 2024-05-27T09:50:27.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.620+0000 7fe33a000700 1 -- 172.21.15.44:0/434623528 wait complete. 2024-05-27T09:50:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.621+0000 7fe33a000700 1 Processor -- start 2024-05-27T09:50:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.621+0000 7fe33a000700 1 -- start start 2024-05-27T09:50:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.622+0000 7fe33a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe334106610 0x7fe33419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.622+0000 7fe33a000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe33410af40 0x7fe33419d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.622+0000 7fe33a000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe33410b890 0x7fe3341a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.622+0000 7fe33a000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe3341164c0 con 0x7fe33410af40 2024-05-27T09:50:27.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.622+0000 7fe33a000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe334116340 con 0x7fe33410b890 2024-05-27T09:50:27.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.622+0000 7fe33a000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe334116640 con 0x7fe334106610 2024-05-27T09:50:27.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.622+0000 7fe338c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe334106610 0x7fe33419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.622+0000 7fe338c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe334106610 0x7fe33419cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:44800/0 (socket says 172.21.15.44:44800) 2024-05-27T09:50:27.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.622+0000 7fe338c00700 1 -- 172.21.15.44:0/230927833 learned_addr learned my addr 172.21.15.44:0/230927833 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:27.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.622+0000 7fe339600700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe33410b890 0x7fe3341a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.622+0000 7fe333e00700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe33410af40 0x7fe33419d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.623+0000 7fe339600700 1 -- 172.21.15.44:0/230927833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe334106610 msgr2=0x7fe33419cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.623+0000 7fe339600700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe334106610 0x7fe33419cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.623+0000 7fe339600700 1 -- 172.21.15.44:0/230927833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe33410af40 msgr2=0x7fe33419d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.623+0000 7fe339600700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe33410af40 0x7fe33419d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.623+0000 7fe339600700 1 -- 172.21.15.44:0/230927833 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3341a1c80 con 0x7fe33410b890 2024-05-27T09:50:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.623+0000 7fe339600700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe33410b890 0x7fe3341a15e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe32c015000 tx=0x7fe32c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.624+0000 7fe331600700 1 -- 172.21.15.44:0/230927833 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c021070 con 0x7fe33410b890 2024-05-27T09:50:27.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.624+0000 7fe331600700 1 -- 172.21.15.44:0/230927833 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe32c007ac0 con 0x7fe33410b890 2024-05-27T09:50:27.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.624+0000 7fe331600700 1 -- 172.21.15.44:0/230927833 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c005040 con 0x7fe33410b890 2024-05-27T09:50:27.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.624+0000 7fe33a000700 1 -- 172.21.15.44:0/230927833 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3341a1f10 con 0x7fe33410b890 2024-05-27T09:50:27.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.624+0000 7fe33a000700 1 -- 172.21.15.44:0/230927833 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3341a9890 con 0x7fe33410b890 2024-05-27T09:50:27.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.628+0000 7fe331600700 1 -- 172.21.15.44:0/230927833 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fe32c024070 con 0x7fe33410b890 2024-05-27T09:50:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.629+0000 7fe34a1eab80 1 Processor -- start 2024-05-27T09:50:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.629+0000 7fe34a1eab80 1 -- start start 2024-05-27T09:50:27.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.629+0000 7fe331600700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe31c0b2a70 0x7fe31c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.629+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d2a9e40 0x557e3d427ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.629+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d428410 0x557e3d432ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.629+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d435020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.629+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557e3d356e40 con 0x557e3d2a9e40 2024-05-27T09:50:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.629+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557e3d3469e0 con 0x557e3d428820 2024-05-27T09:50:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.629+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557e3d34aa20 con 0x557e3d428410 2024-05-27T09:50:27.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.630+0000 7fe331600700 1 -- 172.21.15.44:0/230927833 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7fe32c00f020 con 0x7fe33410b890 2024-05-27T09:50:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.630+0000 7fe333e00700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe31c0b2a70 0x7fe31c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.630+0000 7fe338c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d2a9e40 0x557e3d427ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.630+0000 7fe338c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d2a9e40 0x557e3d427ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56894/0 (socket says 172.21.15.44:56894) 2024-05-27T09:50:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.630+0000 7fe338c00700 1 -- 172.21.15.44:0/3795377428 learned_addr learned my addr 172.21.15.44:0/3795377428 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.630+0000 7fe339600700 1 --2- 172.21.15.44:0/3795377428 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d435020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.630+0000 7fe333e00700 1 --2- 172.21.15.44:0/3795377428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d428410 0x557e3d432ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.630+0000 7fe339600700 1 -- 172.21.15.44:0/3795377428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d428410 msgr2=0x557e3d432ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.630+0000 7fe339600700 1 --2- 172.21.15.44:0/3795377428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d428410 0x557e3d432ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.630+0000 7fe339600700 1 -- 172.21.15.44:0/3795377428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d2a9e40 msgr2=0x557e3d427ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.630+0000 7fe339600700 1 --2- 172.21.15.44:0/3795377428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d2a9e40 0x557e3d427ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.631+0000 7fe339600700 1 -- 172.21.15.44:0/3795377428 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e3d30f420 con 0x557e3d428820 2024-05-27T09:50:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.631+0000 7fe333e00700 1 --2- 172.21.15.44:0/3795377428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d428410 0x557e3d432ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.631+0000 7fe338c00700 1 --2- 172.21.15.44:0/3795377428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d2a9e40 0x557e3d427ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.631+0000 7fe339600700 1 --2- 172.21.15.44:0/3795377428 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d435020 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0e1010 tx=0x7fe32c0a63c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b49f696dc90a8a43 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.631+0000 7fe326000700 1 -- 172.21.15.44:0/3795377428 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c0b1040 con 0x557e3d428820 2024-05-27T09:50:27.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.631+0000 7fe326000700 1 -- 172.21.15.44:0/3795377428 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe32c0cbe50 con 0x557e3d428820 2024-05-27T09:50:27.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.631+0000 7fe326000700 1 -- 172.21.15.44:0/3795377428 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c0a6ec0 con 0x557e3d428820 2024-05-27T09:50:27.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.632+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3795377428 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 msgr2=0x557e3d435020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:27.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.632+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/3795377428 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d435020 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0e1010 tx=0x7fe32c0a63c0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.632+0000 7fe333e00700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe31c0b2a70 0x7fe31c0b4f80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x557e3d429430 tx=0x7fe320009930 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:27.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.632+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3795377428 shutdown_connections 2024-05-27T09:50:27.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.632+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/3795377428 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d435020 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.632+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/3795377428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d2a9e40 0x557e3d427ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.632+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/3795377428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d428410 0x557e3d432ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.632+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3795377428 >> 172.21.15.44:0/3795377428 conn(0x557e3d309fc0 msgr2=0x557e3d423d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:27.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.632+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3795377428 shutdown_connections 2024-05-27T09:50:27.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.632+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3795377428 wait complete. 2024-05-27T09:50:27.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.633+0000 7fe34a1eab80 1 -- start start 2024-05-27T09:50:27.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.633+0000 7fe34a1eab80 1 Processor -- start 2024-05-27T09:50:27.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.634+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d432610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.634+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 0x557e3d432b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.634+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d433090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.634+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557e3d24cfa0 con 0x557e3d428410 2024-05-27T09:50:27.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.634+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557e3d2e7760 con 0x557e3d428820 2024-05-27T09:50:27.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.634+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557e3d435750 con 0x557e3d2a9e40 2024-05-27T09:50:27.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.634+0000 7fe34a1eab80 10 client.0 _ll_register_callbacks cb 0x557e3d2f0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:27.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.634+0000 7fe333e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 0x557e3d432b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe338c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d432610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe333e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 0x557e3d432b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56910/0 (socket says 172.21.15.44:56910) 2024-05-27T09:50:27.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe333e00700 1 -- 172.21.15.44:0/3551427896 learned_addr learned my addr 172.21.15.44:0/3551427896 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:27.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe339600700 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d433090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe339600700 1 -- 172.21.15.44:0/3551427896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 msgr2=0x557e3d432610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:27.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe339600700 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d432610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe339600700 1 -- 172.21.15.44:0/3551427896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 msgr2=0x557e3d432b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:27.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe339600700 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 0x557e3d432b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe339600700 1 -- 172.21.15.44:0/3551427896 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e3d2968d0 con 0x557e3d428820 2024-05-27T09:50:27.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe338c00700 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d432610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:27.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe333e00700 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 0x557e3d432b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:27.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe339600700 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d433090 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0aaa70 tx=0x7fe32c0ab0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=348e00525b37088e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:27.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.635+0000 7fe31aa00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:27.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.636+0000 7fe31aa00700 1 -- 172.21.15.44:0/3551427896 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c0b1040 con 0x557e3d428820 2024-05-27T09:50:27.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.636+0000 7fe31aa00700 1 -- 172.21.15.44:0/3551427896 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7fe32c0a7220 con 0x557e3d428820 2024-05-27T09:50:27.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.636+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3551427896 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe32c0cbe50 con 0x557e3d428820 2024-05-27T09:50:27.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.636+0000 7fe31aa00700 1 -- 172.21.15.44:0/3551427896 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fe32c0aac30 con 0x557e3d428820 2024-05-27T09:50:27.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.636+0000 7fe34a1eab80 10 client.28205 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:27.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.636+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3551427896 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x557e3d317060 con 0x557e3d428820 2024-05-27T09:50:27.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.636+0000 7fe31aa00700 1 -- 172.21.15.44:0/3551427896 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe32c0cbe50 con 0x557e3d428820 2024-05-27T09:50:27.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.636+0000 7fe31aa00700 1 -- 172.21.15.44:0/3551427896 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fe32c086070 con 0x557e3d428820 2024-05-27T09:50:27.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.637+0000 7fe34a1eab80 10 client.28205 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:27.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.637+0000 7fe34a1eab80 10 client.28205 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:50:27.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.637+0000 7fe34a1eab80 20 client.28205 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:27.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.637+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557e3d442530 0x557e3d444950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.637+0000 7fe34a1eab80 4 client.28205 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:27.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.637+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3551427896 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e3d358d90 con 0x557e3d442530 2024-05-27T09:50:27.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.637+0000 7fe338c00700 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557e3d442530 0x557e3d444950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.638+0000 7fe338c00700 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557e3d442530 0x557e3d444950 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e1a6cd759566f62 server_cookie=1c1294479f401343 in_seq=0 out_seq=0 2024-05-27T09:50:27.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.638+0000 7fe31aa00700 10 client.28205 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:50:27.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe31aa00700 1 -- 172.21.15.44:0/3551427896 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557e3d358d90 con 0x557e3d442530 2024-05-27T09:50:27.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe31aa00700 10 client.28205 handle_command_reply: tid=0 2024-05-27T09:50:27.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 1 client.28205 shutdown 2024-05-27T09:50:27.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3551427896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 msgr2=0x557e3d433090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:27.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d433090 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0aaa70 tx=0x7fe32c0ab0c0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3551427896 shutdown_connections 2024-05-27T09:50:27.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d433090 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d432610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557e3d442530 0x557e3d444950 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/3551427896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 0x557e3d432b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3551427896 >> 172.21.15.44:0/3551427896 conn(0x557e3d309fc0 msgr2=0x557e3d422ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:27.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe31aa00700 0 client.28205 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:50:27.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3551427896 shutdown_connections 2024-05-27T09:50:27.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 1 -- 172.21.15.44:0/3551427896 wait complete. 2024-05-27T09:50:27.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.639+0000 7fe34a1eab80 20 client.28205 trim_cache size 0 max 16384 2024-05-27T09:50:27.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.657+0000 7fe34a1eab80 1 Processor -- start 2024-05-27T09:50:27.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.657+0000 7fe34a1eab80 1 -- start start 2024-05-27T09:50:27.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.658+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d42de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.658+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428410 0x557e3d431570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.658+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428820 0x557e3d434ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.658+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe3341b3710 con 0x557e3d428820 2024-05-27T09:50:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.658+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe3341b0f40 con 0x557e3d428410 2024-05-27T09:50:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.658+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3341aed90 con 0x557e3d2a9e40 2024-05-27T09:50:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.658+0000 7fe333e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428410 0x557e3d431570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.658+0000 7fe338c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d42de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.658+0000 7fe339600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428820 0x557e3d434ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.658+0000 7fe333e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428410 0x557e3d431570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:59184/0 (socket says 172.21.15.44:59184) 2024-05-27T09:50:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.658+0000 7fe333e00700 1 -- 172.21.15.44:0/2214129625 learned_addr learned my addr 172.21.15.44:0/2214129625 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.659+0000 7fe333e00700 1 -- 172.21.15.44:0/2214129625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 msgr2=0x557e3d42de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.659+0000 7fe333e00700 1 --2- 172.21.15.44:0/2214129625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d42de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.659+0000 7fe333e00700 1 -- 172.21.15.44:0/2214129625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428820 msgr2=0x557e3d434ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.659+0000 7fe333e00700 1 --2- 172.21.15.44:0/2214129625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428820 0x557e3d434ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.659+0000 7fe333e00700 1 -- 172.21.15.44:0/2214129625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e3d430bb0 con 0x557e3d428410 2024-05-27T09:50:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.659+0000 7fe338c00700 1 --2- 172.21.15.44:0/2214129625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d42de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.659+0000 7fe339600700 1 --2- 172.21.15.44:0/2214129625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428820 0x557e3d434ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.659+0000 7fe333e00700 1 --2- 172.21.15.44:0/2214129625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428410 0x557e3d431570 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe320011b70 tx=0x7fe320015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b37e6aa3131f016c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.660+0000 7fe325600700 1 -- 172.21.15.44:0/2214129625 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320022a80 con 0x557e3d428410 2024-05-27T09:50:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.660+0000 7fe325600700 1 -- 172.21.15.44:0/2214129625 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe320022c20 con 0x557e3d428410 2024-05-27T09:50:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.660+0000 7fe325600700 1 -- 172.21.15.44:0/2214129625 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3200103d0 con 0x557e3d428410 2024-05-27T09:50:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.661+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2214129625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428410 msgr2=0x557e3d431570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.661+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/2214129625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428410 0x557e3d431570 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe320011b70 tx=0x7fe320015040 comp rx=0 tx=0).stop 2024-05-27T09:50:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.661+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2214129625 shutdown_connections 2024-05-27T09:50:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.661+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/2214129625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428820 0x557e3d434ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.661+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/2214129625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d42de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.661+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/2214129625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428410 0x557e3d431570 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.661+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2214129625 >> 172.21.15.44:0/2214129625 conn(0x557e3d309fc0 msgr2=0x557e3d43d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.661+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2214129625 shutdown_connections 2024-05-27T09:50:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.661+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2214129625 wait complete. 2024-05-27T09:50:27.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.661+0000 7fe34a1eab80 1 -- start start 2024-05-27T09:50:27.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.662+0000 7fe34a1eab80 1 Processor -- start 2024-05-27T09:50:27.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.662+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d444f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.662+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 0x557e3d445440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.662+0000 7fe34a1eab80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d4a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.662+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe3341af400 con 0x557e3d428410 2024-05-27T09:50:27.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.662+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe3341b0870 con 0x557e3d428820 2024-05-27T09:50:27.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.662+0000 7fe34a1eab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3341afad0 con 0x557e3d2a9e40 2024-05-27T09:50:27.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.662+0000 7fe34a1eab80 10 client.0 _ll_register_callbacks cb 0x557e3d2f0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:27.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe339600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d4a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe339600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d4a4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:59190/0 (socket says 172.21.15.44:59190) 2024-05-27T09:50:27.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe339600700 1 -- 172.21.15.44:0/2889682484 learned_addr learned my addr 172.21.15.44:0/2889682484 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:27.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe333e00700 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 0x557e3d445440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe338c00700 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d444f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe339600700 1 -- 172.21.15.44:0/2889682484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 msgr2=0x557e3d444f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe339600700 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d444f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe339600700 1 -- 172.21.15.44:0/2889682484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 msgr2=0x557e3d445440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe339600700 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 0x557e3d445440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe339600700 1 -- 172.21.15.44:0/2889682484 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e3d4a52d0 con 0x557e3d428820 2024-05-27T09:50:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe333e00700 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 0x557e3d445440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe339600700 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d4a4d60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe32c004fd0 tx=0x7fe32c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=932f2b8aaeecd80d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.663+0000 7fe31aa00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe31aa00700 1 -- 172.21.15.44:0/2889682484 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c0b1040 con 0x557e3d428820 2024-05-27T09:50:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2889682484 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe320022c20 con 0x557e3d428820 2024-05-27T09:50:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe31aa00700 1 -- 172.21.15.44:0/2889682484 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7fe32c009bd0 con 0x557e3d428820 2024-05-27T09:50:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe31aa00700 1 -- 172.21.15.44:0/2889682484 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fe32c08c070 con 0x557e3d428820 2024-05-27T09:50:27.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe34a1eab80 10 client.28211 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2889682484 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x557e3d4a52d0 con 0x557e3d428820 2024-05-27T09:50:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe31aa00700 1 -- 172.21.15.44:0/2889682484 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe320022c20 con 0x557e3d428820 2024-05-27T09:50:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe31aa00700 1 -- 172.21.15.44:0/2889682484 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7fe32c086070 con 0x557e3d428820 2024-05-27T09:50:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe34a1eab80 10 client.28211 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe34a1eab80 10 client.28211 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:50:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe34a1eab80 20 client.28211 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557e3d4ac7e0 0x557e3d4aec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:27.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe34a1eab80 4 client.28211 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:50:27.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.664+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2889682484 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557e3d352570 con 0x557e3d4ac7e0 2024-05-27T09:50:27.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.665+0000 7fe338c00700 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557e3d4ac7e0 0x557e3d4aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:27.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.665+0000 7fe338c00700 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557e3d4ac7e0 0x557e3d4aec00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6d1a6db48fc8cbd5 server_cookie=8c6a1f1ee49c3789 in_seq=0 out_seq=0 2024-05-27T09:50:27.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.665+0000 7fe31aa00700 10 client.28211 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:50:27.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe31aa00700 1 -- 172.21.15.44:0/2889682484 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557e3d352570 con 0x557e3d4ac7e0 2024-05-27T09:50:27.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe31aa00700 10 client.28211 handle_command_reply: tid=0 2024-05-27T09:50:27.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 1 client.28211 shutdown 2024-05-27T09:50:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2889682484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 msgr2=0x557e3d4a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d4a4d60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe32c004fd0 tx=0x7fe32c00b330 comp rx=0 tx=0).stop 2024-05-27T09:50:27.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2889682484 shutdown_connections 2024-05-27T09:50:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557e3d428820 0x557e3d4a4d60 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e3d2a9e40 0x557e3d444f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x557e3d4ac7e0 0x557e3d4aec00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 1 --2- 172.21.15.44:0/2889682484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557e3d428410 0x557e3d445440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2889682484 >> 172.21.15.44:0/2889682484 conn(0x557e3d309fc0 msgr2=0x557e3d43ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe31aa00700 0 client.28211 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:50:27.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2889682484 shutdown_connections 2024-05-27T09:50:27.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 1 -- 172.21.15.44:0/2889682484 wait complete. 2024-05-27T09:50:27.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.666+0000 7fe34a1eab80 20 client.28211 trim_cache size 0 max 16384 2024-05-27T09:50:27.674 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:50:27.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 -- 172.21.15.44:0/230927833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe31c0b2a70 msgr2=0x7fe31c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:27.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe31c0b2a70 0x7fe31c0b4f80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x557e3d429430 tx=0x7fe320009930 comp rx=0 tx=0).stop 2024-05-27T09:50:27.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 -- 172.21.15.44:0/230927833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe33410b890 msgr2=0x7fe3341a15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:27.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe33410b890 0x7fe3341a15e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe32c015000 tx=0x7fe32c00ccb0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 -- 172.21.15.44:0/230927833 shutdown_connections 2024-05-27T09:50:27.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe31c0b2a70 0x7fe31c0b4f80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe33410b890 0x7fe3341a15e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe334106610 0x7fe33419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 --2- 172.21.15.44:0/230927833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe33410af40 0x7fe33419d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:27.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 -- 172.21.15.44:0/230927833 >> 172.21.15.44:0/230927833 conn(0x7fe334102020 msgr2=0x7fe33410a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:27.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 -- 172.21.15.44:0/230927833 shutdown_connections 2024-05-27T09:50:27.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:27.667+0000 7fe33a000700 1 -- 172.21.15.44:0/230927833 wait complete. 2024-05-27T09:50:28.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:28 smithi189 ceph-mon[36671]: pgmap v479: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 342 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-05-27T09:50:28.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:28 smithi044 ceph-mon[31618]: pgmap v479: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 342 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-05-27T09:50:28.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:28 smithi050 ceph-mon[36371]: pgmap v479: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 342 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-05-27T09:50:28.746 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:50:29.428 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:30.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:30 smithi189 ceph-mon[36671]: pgmap v480: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 342 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-05-27T09:50:30.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:30 smithi044 ceph-mon[31618]: pgmap v480: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 342 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-05-27T09:50:30.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:30 smithi050 ceph-mon[36371]: pgmap v480: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 342 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-05-27T09:50:30.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.272+0000 7f37d8800700 1 -- 172.21.15.44:0/2931758574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37d010a980 msgr2=0x7f37d010ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:30.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.272+0000 7f37d8800700 1 --2- 172.21.15.44:0/2931758574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37d010a980 0x7f37d010ad90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f37c0009990 tx=0x7f37c0009ca0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.272+0000 7f37d8800700 1 -- 172.21.15.44:0/2931758574 shutdown_connections 2024-05-27T09:50:30.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.272+0000 7f37d8800700 1 --2- 172.21.15.44:0/2931758574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37d010b2d0 0x7f37d0117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.272+0000 7f37d8800700 1 --2- 172.21.15.44:0/2931758574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d0110960 0x7f37d0110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.272+0000 7f37d8800700 1 --2- 172.21.15.44:0/2931758574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37d010a980 0x7f37d010ad90 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.272+0000 7f37d8800700 1 -- 172.21.15.44:0/2931758574 >> 172.21.15.44:0/2931758574 conn(0x7f37d0077c70 msgr2=0x7f37d00780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:30.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.272+0000 7f37d8800700 1 -- 172.21.15.44:0/2931758574 shutdown_connections 2024-05-27T09:50:30.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.272+0000 7f37d8800700 1 -- 172.21.15.44:0/2931758574 wait complete. 2024-05-27T09:50:30.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.273+0000 7f37d8800700 1 Processor -- start 2024-05-27T09:50:30.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.274+0000 7f37d8800700 1 -- start start 2024-05-27T09:50:30.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.274+0000 7f37d8800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37d010a980 0x7f37d01a1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.274+0000 7f37d8800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d010b2d0 0x7f37d01a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.274+0000 7f37d8800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37d0110960 0x7f37d01a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.274+0000 7f37d8800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f37d011aa80 con 0x7f37d010a980 2024-05-27T09:50:30.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.274+0000 7f37d8800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f37d011a900 con 0x7f37d010b2d0 2024-05-27T09:50:30.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.274+0000 7f37d8800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f37d011ac00 con 0x7f37d0110960 2024-05-27T09:50:30.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d5400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d010b2d0 0x7f37d01a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.274+0000 7f37d5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37d010a980 0x7f37d01a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d5400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d010b2d0 0x7f37d01a19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:59202/0 (socket says 172.21.15.44:59202) 2024-05-27T09:50:30.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37d010a980 0x7f37d01a1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56944/0 (socket says 172.21.15.44:56944) 2024-05-27T09:50:30.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d5400700 1 -- 172.21.15.44:0/3502801335 learned_addr learned my addr 172.21.15.44:0/3502801335 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:30.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d6800700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37d0110960 0x7f37d01a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d5400700 1 -- 172.21.15.44:0/3502801335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37d0110960 msgr2=0x7f37d01a5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:30.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d5400700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37d0110960 0x7f37d01a5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d5400700 1 -- 172.21.15.44:0/3502801335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37d010a980 msgr2=0x7f37d01a1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:30.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d5400700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37d010a980 0x7f37d01a1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d5400700 1 -- 172.21.15.44:0/3502801335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d01a6320 con 0x7f37d010b2d0 2024-05-27T09:50:30.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d5e00700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37d010a980 0x7f37d01a1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:30.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.275+0000 7f37d5400700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d010b2d0 0x7f37d01a19f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f37c0000c00 tx=0x7f37c000bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:30.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.276+0000 7f37caa00700 1 -- 172.21.15.44:0/3502801335 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c0027070 con 0x7f37d010b2d0 2024-05-27T09:50:30.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.276+0000 7f37caa00700 1 -- 172.21.15.44:0/3502801335 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37c0022470 con 0x7f37d010b2d0 2024-05-27T09:50:30.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.276+0000 7f37d8800700 1 -- 172.21.15.44:0/3502801335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d01a65b0 con 0x7f37d010b2d0 2024-05-27T09:50:30.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.276+0000 7f37caa00700 1 -- 172.21.15.44:0/3502801335 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c0017650 con 0x7f37d010b2d0 2024-05-27T09:50:30.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.276+0000 7f37d8800700 1 -- 172.21.15.44:0/3502801335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d01adf30 con 0x7f37d010b2d0 2024-05-27T09:50:30.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.280+0000 7f37caa00700 1 -- 172.21.15.44:0/3502801335 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f37c002a070 con 0x7f37d010b2d0 2024-05-27T09:50:30.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.281+0000 7f37e72acb80 1 Processor -- start 2024-05-27T09:50:30.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.282+0000 7f37e72acb80 1 -- start start 2024-05-27T09:50:30.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.282+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec847c9eb0 0x55ec8494ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.282+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec8494f260 0x55ec84948e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.282+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec849497c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.282+0000 7f37e72acb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ec84876eb0 con 0x55ec849493b0 2024-05-27T09:50:30.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.282+0000 7f37e72acb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ec84866a50 con 0x55ec8494f260 2024-05-27T09:50:30.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.282+0000 7f37e72acb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ec8486aa90 con 0x55ec847c9eb0 2024-05-27T09:50:30.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.282+0000 7f37caa00700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37b80b2a70 0x7f37b80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37caa00700 1 -- 172.21.15.44:0/3502801335 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f37c00e32d0 con 0x7f37d010b2d0 2024-05-27T09:50:30.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37d5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec849497c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37d5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec849497c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56950/0 (socket says 172.21.15.44:56950) 2024-05-27T09:50:30.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37d5e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec8494f260 0x55ec84948e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37d5400700 1 -- 172.21.15.44:0/2853919287 learned_addr learned my addr 172.21.15.44:0/2853919287 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:30.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37d6800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec847c9eb0 0x55ec8494ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37d6800700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37b80b2a70 0x7f37b80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37d5400700 1 -- 172.21.15.44:0/2853919287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec847c9eb0 msgr2=0x55ec8494ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:30.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37d5400700 1 --2- 172.21.15.44:0/2853919287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec847c9eb0 0x55ec8494ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37d5400700 1 -- 172.21.15.44:0/2853919287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec8494f260 msgr2=0x55ec84948e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:30.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37d5400700 1 --2- 172.21.15.44:0/2853919287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec8494f260 0x55ec84948e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.283+0000 7f37d5400700 1 -- 172.21.15.44:0/2853919287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec8482f490 con 0x55ec849493b0 2024-05-27T09:50:30.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.284+0000 7f37d5400700 1 --2- 172.21.15.44:0/2853919287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec849497c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f37c00098e0 tx=0x7f37c0015d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e5322ab0efef60 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:30.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.284+0000 7f37d6800700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37b80b2a70 0x7f37b80b4f80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55ec8494a060 tx=0x7f37cc00b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:30.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.284+0000 7f37bf400700 1 -- 172.21.15.44:0/2853919287 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c00aa070 con 0x55ec849493b0 2024-05-27T09:50:30.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.284+0000 7f37bf400700 1 -- 172.21.15.44:0/2853919287 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37c00aac60 con 0x55ec849493b0 2024-05-27T09:50:30.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.284+0000 7f37bf400700 1 -- 172.21.15.44:0/2853919287 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c0022940 con 0x55ec849493b0 2024-05-27T09:50:30.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.285+0000 7f37e72acb80 1 -- 172.21.15.44:0/2853919287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 msgr2=0x55ec849497c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:30.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.285+0000 7f37e72acb80 1 --2- 172.21.15.44:0/2853919287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec849497c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f37c00098e0 tx=0x7f37c0015d80 comp rx=0 tx=0).stop 2024-05-27T09:50:30.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.285+0000 7f37e72acb80 1 -- 172.21.15.44:0/2853919287 shutdown_connections 2024-05-27T09:50:30.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.285+0000 7f37e72acb80 1 --2- 172.21.15.44:0/2853919287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec849497c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.285+0000 7f37e72acb80 1 --2- 172.21.15.44:0/2853919287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec847c9eb0 0x55ec8494ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.285+0000 7f37e72acb80 1 --2- 172.21.15.44:0/2853919287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec8494f260 0x55ec84948e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.285+0000 7f37e72acb80 1 -- 172.21.15.44:0/2853919287 >> 172.21.15.44:0/2853919287 conn(0x55ec8482a030 msgr2=0x55ec84944070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:30.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.285+0000 7f37e72acb80 1 -- 172.21.15.44:0/2853919287 shutdown_connections 2024-05-27T09:50:30.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.285+0000 7f37e72acb80 1 -- 172.21.15.44:0/2853919287 wait complete. 2024-05-27T09:50:30.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.286+0000 7f37e72acb80 1 -- start start 2024-05-27T09:50:30.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.286+0000 7f37e72acb80 1 Processor -- start 2024-05-27T09:50:30.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec847c9eb0 0x55ec84952950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec84952e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec8494f260 0x55ec849533d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37e72acb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ec8476d010 con 0x55ec849493b0 2024-05-27T09:50:30.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37e72acb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ec848077d0 con 0x55ec847c9eb0 2024-05-27T09:50:30.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37e72acb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ec84955af0 con 0x55ec8494f260 2024-05-27T09:50:30.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37e72acb80 10 client.0 _ll_register_callbacks cb 0x55ec84810ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:30.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37d5e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec847c9eb0 0x55ec84952950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37d5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec84952e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37d5e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec847c9eb0 0x55ec84952950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:59220/0 (socket says 172.21.15.44:59220) 2024-05-27T09:50:30.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37d5e00700 1 -- 172.21.15.44:0/1013307559 learned_addr learned my addr 172.21.15.44:0/1013307559 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:30.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.287+0000 7f37d6800700 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec8494f260 0x55ec849533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.288+0000 7f37d5e00700 1 -- 172.21.15.44:0/1013307559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec8494f260 msgr2=0x55ec849533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:30.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.288+0000 7f37d5e00700 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec8494f260 0x55ec849533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.288+0000 7f37d5e00700 1 -- 172.21.15.44:0/1013307559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 msgr2=0x55ec84952e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:30.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.288+0000 7f37d5e00700 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec84952e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.288+0000 7f37d5e00700 1 -- 172.21.15.44:0/1013307559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec847b6940 con 0x55ec847c9eb0 2024-05-27T09:50:30.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.288+0000 7f37d5400700 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec84952e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:30.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.288+0000 7f37d5e00700 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec847c9eb0 0x55ec84952950 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f37c4012d20 tx=0x7f37c400f6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a0d74b36a9dfff3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:30.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.288+0000 7f37b3e00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:30.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.288+0000 7f37b3e00700 1 -- 172.21.15.44:0/1013307559 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c400fb80 con 0x55ec847c9eb0 2024-05-27T09:50:30.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.288+0000 7f37b3e00700 1 -- 172.21.15.44:0/1013307559 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f37c400fd20 con 0x55ec847c9eb0 2024-05-27T09:50:30.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.288+0000 7f37e72acb80 1 -- 172.21.15.44:0/1013307559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37c00aac60 con 0x55ec847c9eb0 2024-05-27T09:50:30.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.289+0000 7f37b3e00700 1 -- 172.21.15.44:0/1013307559 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f37c4004050 con 0x55ec847c9eb0 2024-05-27T09:50:30.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.289+0000 7f37e72acb80 10 client.28223 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:30.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.289+0000 7f37e72acb80 1 -- 172.21.15.44:0/1013307559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55ec848370d0 con 0x55ec847c9eb0 2024-05-27T09:50:30.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.289+0000 7f37b3e00700 1 -- 172.21.15.44:0/1013307559 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37c00aac60 con 0x55ec847c9eb0 2024-05-27T09:50:30.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.289+0000 7f37b3e00700 1 -- 172.21.15.44:0/1013307559 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f37c4022070 con 0x55ec847c9eb0 2024-05-27T09:50:30.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.289+0000 7f37e72acb80 10 client.28223 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:30.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.289+0000 7f37e72acb80 10 client.28223 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:50:30.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.289+0000 7f37e72acb80 20 client.28223 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:30.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.289+0000 7f37e72acb80 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ec84962850 0x55ec84964cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.289+0000 7f37e72acb80 4 client.28223 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:30.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.289+0000 7f37e72acb80 1 -- 172.21.15.44:0/1013307559 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec84878e00 con 0x55ec84962850 2024-05-27T09:50:30.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.290+0000 7f37d5400700 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ec84962850 0x55ec84964cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.290+0000 7f37d5400700 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ec84962850 0x55ec84964cd0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbabca38e827114b server_cookie=ff711a3f3723ed03 in_seq=0 out_seq=0 2024-05-27T09:50:30.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.290+0000 7f37b3e00700 10 client.28223 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:30.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.291+0000 7f37b3e00700 1 -- 172.21.15.44:0/1013307559 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ec84878e00 con 0x55ec84962850 2024-05-27T09:50:30.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.291+0000 7f37b3e00700 10 client.28223 handle_command_reply: tid=0 2024-05-27T09:50:30.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.291+0000 7f37e72acb80 1 client.28223 shutdown 2024-05-27T09:50:30.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37e72acb80 1 -- 172.21.15.44:0/1013307559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec847c9eb0 msgr2=0x55ec84952950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:30.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37e72acb80 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec847c9eb0 0x55ec84952950 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f37c4012d20 tx=0x7f37c400f6a0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37e72acb80 1 -- 172.21.15.44:0/1013307559 shutdown_connections 2024-05-27T09:50:30.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37e72acb80 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ec84962850 0x55ec84964cd0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37e72acb80 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec8494f260 0x55ec849533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37b3e00700 0 client.28223 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:30.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37e72acb80 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec847c9eb0 0x55ec84952950 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37e72acb80 1 --2- 172.21.15.44:0/1013307559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec84952e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37e72acb80 1 -- 172.21.15.44:0/1013307559 >> 172.21.15.44:0/1013307559 conn(0x55ec8482a030 msgr2=0x55ec84947c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:30.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37e72acb80 1 -- 172.21.15.44:0/1013307559 shutdown_connections 2024-05-27T09:50:30.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37e72acb80 1 -- 172.21.15.44:0/1013307559 wait complete. 2024-05-27T09:50:30.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.292+0000 7f37e72acb80 20 client.28223 trim_cache size 0 max 16384 2024-05-27T09:50:30.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.315+0000 7f37e72acb80 1 Processor -- start 2024-05-27T09:50:30.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.315+0000 7f37e72acb80 1 -- start start 2024-05-27T09:50:30.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.316+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec847c9eb0 0x55ec849486b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.316+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec849493b0 0x55ec8494e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.316+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec8494f260 0x55ec8495ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.316+0000 7f37e72acb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f37d01b7fb0 con 0x55ec847c9eb0 2024-05-27T09:50:30.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.316+0000 7f37e72acb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f37d01b57e0 con 0x55ec849493b0 2024-05-27T09:50:30.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.316+0000 7f37e72acb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f37d01b3650 con 0x55ec8494f260 2024-05-27T09:50:30.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.316+0000 7f37d5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec847c9eb0 0x55ec849486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.316+0000 7f37d5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec847c9eb0 0x55ec849486b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56966/0 (socket says 172.21.15.44:56966) 2024-05-27T09:50:30.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.316+0000 7f37d5e00700 1 -- 172.21.15.44:0/572900982 learned_addr learned my addr 172.21.15.44:0/572900982 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:30.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.316+0000 7f37d6800700 1 --2- 172.21.15.44:0/572900982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec8494f260 0x55ec8495ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.316+0000 7f37d5400700 1 --2- 172.21.15.44:0/572900982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec849493b0 0x55ec8494e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.317+0000 7f37d5e00700 1 -- 172.21.15.44:0/572900982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec8494f260 msgr2=0x55ec8495ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:30.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.317+0000 7f37d5e00700 1 --2- 172.21.15.44:0/572900982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec8494f260 0x55ec8495ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.317+0000 7f37d5e00700 1 -- 172.21.15.44:0/572900982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec849493b0 msgr2=0x55ec8494e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:30.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.317+0000 7f37d5e00700 1 --2- 172.21.15.44:0/572900982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec849493b0 0x55ec8494e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.317+0000 7f37d5e00700 1 -- 172.21.15.44:0/572900982 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec84950ee0 con 0x55ec847c9eb0 2024-05-27T09:50:30.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.317+0000 7f37d5e00700 1 --2- 172.21.15.44:0/572900982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec847c9eb0 0x55ec849486b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f37c4004020 tx=0x7f37c4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ec366bce23c2ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:30.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.317+0000 7f37bea00700 1 -- 172.21.15.44:0/572900982 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c4031070 con 0x55ec847c9eb0 2024-05-27T09:50:30.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.317+0000 7f37bea00700 1 -- 172.21.15.44:0/572900982 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37c4005260 con 0x55ec847c9eb0 2024-05-27T09:50:30.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.317+0000 7f37bea00700 1 -- 172.21.15.44:0/572900982 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c4028040 con 0x55ec847c9eb0 2024-05-27T09:50:30.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.318+0000 7f37e72acb80 1 -- 172.21.15.44:0/572900982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec847c9eb0 msgr2=0x55ec849486b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:30.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.318+0000 7f37e72acb80 1 --2- 172.21.15.44:0/572900982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec847c9eb0 0x55ec849486b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f37c4004020 tx=0x7f37c4010740 comp rx=0 tx=0).stop 2024-05-27T09:50:30.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.318+0000 7f37e72acb80 1 -- 172.21.15.44:0/572900982 shutdown_connections 2024-05-27T09:50:30.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.318+0000 7f37e72acb80 1 --2- 172.21.15.44:0/572900982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec8494f260 0x55ec8495ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.318+0000 7f37e72acb80 1 --2- 172.21.15.44:0/572900982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec847c9eb0 0x55ec849486b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.318+0000 7f37e72acb80 1 --2- 172.21.15.44:0/572900982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec849493b0 0x55ec8494e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.318+0000 7f37e72acb80 1 -- 172.21.15.44:0/572900982 >> 172.21.15.44:0/572900982 conn(0x55ec8482a030 msgr2=0x55ec8494cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:30.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.318+0000 7f37e72acb80 1 -- 172.21.15.44:0/572900982 shutdown_connections 2024-05-27T09:50:30.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.318+0000 7f37e72acb80 1 -- 172.21.15.44:0/572900982 wait complete. 2024-05-27T09:50:30.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.319+0000 7f37e72acb80 1 -- start start 2024-05-27T09:50:30.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.319+0000 7f37e72acb80 1 Processor -- start 2024-05-27T09:50:30.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec847c9eb0 0x55ec8495b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec84957c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37e72acb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec8494f260 0x55ec84958180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37e72acb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f37d01b3ca0 con 0x55ec849493b0 2024-05-27T09:50:30.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37e72acb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f37d01b5110 con 0x55ec8494f260 2024-05-27T09:50:30.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37e72acb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f37d01b4370 con 0x55ec847c9eb0 2024-05-27T09:50:30.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37e72acb80 10 client.0 _ll_register_callbacks cb 0x55ec84810ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:30.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37d5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec84957c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37d5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec84957c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56974/0 (socket says 172.21.15.44:56974) 2024-05-27T09:50:30.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37d5400700 1 -- 172.21.15.44:0/718023204 learned_addr learned my addr 172.21.15.44:0/718023204 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:30.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37d5e00700 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec847c9eb0 0x55ec8495b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.320+0000 7f37d6800700 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec8494f260 0x55ec84958180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37d5400700 1 -- 172.21.15.44:0/718023204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec847c9eb0 msgr2=0x55ec8495b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:30.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37d5400700 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec847c9eb0 0x55ec8495b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37d5400700 1 -- 172.21.15.44:0/718023204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec8494f260 msgr2=0x55ec84958180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:30.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37d5400700 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec8494f260 0x55ec84958180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37d5400700 1 -- 172.21.15.44:0/718023204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec849586f0 con 0x55ec849493b0 2024-05-27T09:50:30.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37d6800700 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec8494f260 0x55ec84958180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:30.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37d5400700 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec84957c40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x55ec8494f7d0 tx=0x7f37c0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a76f9da8873328 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:30.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37b3e00700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:50:30.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37b3e00700 1 -- 172.21.15.44:0/718023204 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c00e0dd0 con 0x55ec849493b0 2024-05-27T09:50:30.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37b3e00700 1 -- 172.21.15.44:0/718023204 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f37c00a6b50 con 0x55ec849493b0 2024-05-27T09:50:30.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37e72acb80 1 -- 172.21.15.44:0/718023204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37c4005260 con 0x55ec849493b0 2024-05-27T09:50:30.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37b3e00700 1 -- 172.21.15.44:0/718023204 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f37c00aac60 con 0x55ec849493b0 2024-05-27T09:50:30.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37e72acb80 10 client.17925 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:30.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.321+0000 7f37e72acb80 1 -- 172.21.15.44:0/718023204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55ec849ce0b0 con 0x55ec849493b0 2024-05-27T09:50:30.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.322+0000 7f37b3e00700 1 -- 172.21.15.44:0/718023204 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37c4005260 con 0x55ec849493b0 2024-05-27T09:50:30.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.322+0000 7f37b3e00700 1 -- 172.21.15.44:0/718023204 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f37c00cb2f0 con 0x55ec849493b0 2024-05-27T09:50:30.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.322+0000 7f37e72acb80 10 client.17925 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:30.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.322+0000 7f37e72acb80 10 client.17925 resolve_mds: validated gid 24542 aka daemon mds.b 2024-05-27T09:50:30.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.322+0000 7f37e72acb80 20 client.17925 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:30.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.322+0000 7f37e72acb80 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ec849ce680 0x55ec849d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:30.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.322+0000 7f37e72acb80 4 client.17925 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-05-27T09:50:30.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.322+0000 7f37e72acb80 1 -- 172.21.15.44:0/718023204 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55ec848725e0 con 0x55ec849ce680 2024-05-27T09:50:30.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.322+0000 7f37d5e00700 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ec849ce680 0x55ec849d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:30.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.323+0000 7f37d5e00700 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ec849ce680 0x55ec849d0aa0 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a141062c62f6ff50 server_cookie=2cdc00d58c062f87 in_seq=0 out_seq=0 2024-05-27T09:50:30.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.323+0000 7f37b3e00700 10 client.17925 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:30.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.323+0000 7f37b3e00700 1 -- 172.21.15.44:0/718023204 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55ec848725e0 con 0x55ec849ce680 2024-05-27T09:50:30.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.323+0000 7f37b3e00700 10 client.17925 handle_command_reply: tid=0 2024-05-27T09:50:30.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.323+0000 7f37e72acb80 1 client.17925 shutdown 2024-05-27T09:50:30.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37e72acb80 1 -- 172.21.15.44:0/718023204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 msgr2=0x55ec84957c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:30.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37e72acb80 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec84957c40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x55ec8494f7d0 tx=0x7f37c0009310 comp rx=0 tx=0).stop 2024-05-27T09:50:30.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37e72acb80 1 -- 172.21.15.44:0/718023204 shutdown_connections 2024-05-27T09:50:30.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37e72acb80 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55ec849ce680 0x55ec849d0aa0 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37e72acb80 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ec8494f260 0x55ec84958180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37e72acb80 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec847c9eb0 0x55ec8495b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37e72acb80 1 --2- 172.21.15.44:0/718023204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ec849493b0 0x55ec84957c40 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37b3e00700 0 client.17925 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:30.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37e72acb80 1 -- 172.21.15.44:0/718023204 >> 172.21.15.44:0/718023204 conn(0x55ec8482a030 msgr2=0x55ec8494cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:30.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37e72acb80 1 -- 172.21.15.44:0/718023204 shutdown_connections 2024-05-27T09:50:30.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37e72acb80 1 -- 172.21.15.44:0/718023204 wait complete. 2024-05-27T09:50:30.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.324+0000 7f37e72acb80 20 client.17925 trim_cache size 0 max 16384 2024-05-27T09:50:30.586 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:50:30.586 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 43, 2024-05-27T09:50:30.586 INFO:teuthology.orchestra.run.smithi044.stdout: "leader": 24542, 2024-05-27T09:50:30.586 INFO:teuthology.orchestra.run.smithi044.stdout: "set_version": 39, 2024-05-27T09:50:30.587 INFO:teuthology.orchestra.run.smithi044.stdout: "sets": {} 2024-05-27T09:50:30.587 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:50:30.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 -- 172.21.15.44:0/3502801335 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37b80b2a70 msgr2=0x7f37b80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:30.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37b80b2a70 0x7f37b80b4f80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55ec8494a060 tx=0x7f37cc00b040 comp rx=0 tx=0).stop 2024-05-27T09:50:30.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 -- 172.21.15.44:0/3502801335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d010b2d0 msgr2=0x7f37d01a19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:30.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d010b2d0 0x7f37d01a19f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f37c0000c00 tx=0x7f37c000bfd0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 -- 172.21.15.44:0/3502801335 shutdown_connections 2024-05-27T09:50:30.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37b80b2a70 0x7f37b80b4f80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37d0110960 0x7f37d01a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37d010a980 0x7f37d01a1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 --2- 172.21.15.44:0/3502801335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d010b2d0 0x7f37d01a19f0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:30.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 -- 172.21.15.44:0/3502801335 >> 172.21.15.44:0/3502801335 conn(0x7f37d0077c70 msgr2=0x7f37d007b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:30.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 -- 172.21.15.44:0/3502801335 shutdown_connections 2024-05-27T09:50:30.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:30.325+0000 7f37d8800700 1 -- 172.21.15.44:0/3502801335 wait complete. 2024-05-27T09:50:30.594 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:50:32.065 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:32.207 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:32 smithi044 ceph-mon[31618]: pgmap v481: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-05-27T09:50:32.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:32 smithi189 ceph-mon[36671]: pgmap v481: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-05-27T09:50:32.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:32 smithi050 ceph-mon[36371]: pgmap v481: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-05-27T09:50:32.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.878+0000 7f326a400700 1 -- 172.21.15.44:0/20348732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3264108f20 msgr2=0x7f3264109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:32.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.878+0000 7f326a400700 1 --2- 172.21.15.44:0/20348732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3264108f20 0x7f3264109330 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3250009920 tx=0x7f3250009c30 comp rx=0 tx=0).stop 2024-05-27T09:50:32.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.878+0000 7f326a400700 1 -- 172.21.15.44:0/20348732 shutdown_connections 2024-05-27T09:50:32.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.878+0000 7f326a400700 1 --2- 172.21.15.44:0/20348732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3264109870 0x7f32641111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.878+0000 7f326a400700 1 --2- 172.21.15.44:0/20348732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3264104640 0x7f3264104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.878+0000 7f326a400700 1 --2- 172.21.15.44:0/20348732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3264108f20 0x7f3264109330 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.878+0000 7f326a400700 1 -- 172.21.15.44:0/20348732 >> 172.21.15.44:0/20348732 conn(0x7f3264079ef0 msgr2=0x7f326407a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:32.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.878+0000 7f326a400700 1 -- 172.21.15.44:0/20348732 shutdown_connections 2024-05-27T09:50:32.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.878+0000 7f326a400700 1 -- 172.21.15.44:0/20348732 wait complete. 2024-05-27T09:50:32.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.880+0000 7f326a400700 1 Processor -- start 2024-05-27T09:50:32.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.880+0000 7f326a400700 1 -- start start 2024-05-27T09:50:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.881+0000 7f326a400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3264104640 0x7f326419d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.881+0000 7f326a400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3264108f20 0x7f326419d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.881+0000 7f326a400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3264109870 0x7f32641a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.881+0000 7f326a400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3264114480 con 0x7f3264109870 2024-05-27T09:50:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.881+0000 7f326a400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3264114300 con 0x7f3264108f20 2024-05-27T09:50:32.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.881+0000 7f326a400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3264114600 con 0x7f3264104640 2024-05-27T09:50:32.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.881+0000 7f3263400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3264104640 0x7f326419d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.881+0000 7f3263400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3264104640 0x7f326419d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49712/0 (socket says 172.21.15.44:49712) 2024-05-27T09:50:32.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.881+0000 7f3263400700 1 -- 172.21.15.44:0/3003626292 learned_addr learned my addr 172.21.15.44:0/3003626292 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:32.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.881+0000 7f3262a00700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3264108f20 0x7f326419d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.881+0000 7f3263e00700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3264109870 0x7f32641a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.882+0000 7f3263400700 1 -- 172.21.15.44:0/3003626292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3264108f20 msgr2=0x7f326419d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:32.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.882+0000 7f3263400700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3264108f20 0x7f326419d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.882+0000 7f3263400700 1 -- 172.21.15.44:0/3003626292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3264109870 msgr2=0x7f32641a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:32.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.882+0000 7f3263400700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3264109870 0x7f32641a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.882+0000 7f3263400700 1 -- 172.21.15.44:0/3003626292 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32641a1fc0 con 0x7f3264104640 2024-05-27T09:50:32.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.882+0000 7f3263400700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3264104640 0x7f326419d0f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f325400ec10 tx=0x7f325400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:32.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.883+0000 7f325be00700 1 -- 172.21.15.44:0/3003626292 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325400cc90 con 0x7f3264104640 2024-05-27T09:50:32.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.883+0000 7f325be00700 1 -- 172.21.15.44:0/3003626292 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f325400ce30 con 0x7f3264104640 2024-05-27T09:50:32.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.883+0000 7f325be00700 1 -- 172.21.15.44:0/3003626292 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254010430 con 0x7f3264104640 2024-05-27T09:50:32.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.883+0000 7f326a400700 1 -- 172.21.15.44:0/3003626292 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32641a22b0 con 0x7f3264104640 2024-05-27T09:50:32.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.883+0000 7f326a400700 1 -- 172.21.15.44:0/3003626292 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32641a9bd0 con 0x7f3264104640 2024-05-27T09:50:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.888+0000 7f325be00700 1 -- 172.21.15.44:0/3003626292 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f3254004000 con 0x7f3264104640 2024-05-27T09:50:32.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.888+0000 7f3278fc3b80 1 Processor -- start 2024-05-27T09:50:32.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.888+0000 7f3278fc3b80 1 -- start start 2024-05-27T09:50:32.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.889+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406b9f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.889+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406a20e40 0x55a406b9f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.889+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406b9fe70 0x55a406bac110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.889+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a406acde40 con 0x55a406b9fe70 2024-05-27T09:50:32.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.889+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a406abd9e0 con 0x55a406a20e40 2024-05-27T09:50:32.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.889+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a406ac1a20 con 0x55a406b0c9f0 2024-05-27T09:50:32.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.890+0000 7f3262a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406b9fe70 0x55a406bac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.890+0000 7f3262a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406b9fe70 0x55a406bac110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44838/0 (socket says 172.21.15.44:44838) 2024-05-27T09:50:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.890+0000 7f3262a00700 1 -- 172.21.15.44:0/4125292908 learned_addr learned my addr 172.21.15.44:0/4125292908 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.890+0000 7f3263400700 1 --2- 172.21.15.44:0/4125292908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406a20e40 0x55a406b9f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.890+0000 7f3263e00700 1 --2- 172.21.15.44:0/4125292908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406b9f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.890+0000 7f3262a00700 1 -- 172.21.15.44:0/4125292908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 msgr2=0x55a406b9f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.890+0000 7f3262a00700 1 --2- 172.21.15.44:0/4125292908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406b9f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.890+0000 7f325be00700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32480b2a70 0x7f32480b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.890+0000 7f3262a00700 1 -- 172.21.15.44:0/4125292908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406a20e40 msgr2=0x55a406b9f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.890+0000 7f3262a00700 1 --2- 172.21.15.44:0/4125292908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406a20e40 0x55a406b9f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.890+0000 7f3262a00700 1 -- 172.21.15.44:0/4125292908 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a406a86420 con 0x55a406b9fe70 2024-05-27T09:50:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.891+0000 7f325be00700 1 -- 172.21.15.44:0/3003626292 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f32540e2940 con 0x7f3264104640 2024-05-27T09:50:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.891+0000 7f3263e00700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32480b2a70 0x7f32480b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.891+0000 7f3262a00700 1 --2- 172.21.15.44:0/4125292908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406b9fe70 0x55a406bac110 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3250009920 tx=0x7f325000bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760eaa6e21754f23 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.892+0000 7f3258c00700 1 -- 172.21.15.44:0/4125292908 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325000fe40 con 0x55a406b9fe70 2024-05-27T09:50:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.892+0000 7f3258c00700 1 -- 172.21.15.44:0/4125292908 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f325000f910 con 0x55a406b9fe70 2024-05-27T09:50:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.892+0000 7f3258c00700 1 -- 172.21.15.44:0/4125292908 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250003880 con 0x55a406b9fe70 2024-05-27T09:50:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.893+0000 7f3278fc3b80 1 -- 172.21.15.44:0/4125292908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406b9fe70 msgr2=0x55a406bac110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.893+0000 7f3263e00700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32480b2a70 0x7f32480b4f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55a406ba1ff0 tx=0x7f325c00b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.893+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/4125292908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406b9fe70 0x55a406bac110 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3250009920 tx=0x7f325000bfa0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.893+0000 7f3278fc3b80 1 -- 172.21.15.44:0/4125292908 shutdown_connections 2024-05-27T09:50:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.893+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/4125292908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406b9fe70 0x55a406bac110 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.893+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/4125292908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406b9f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.893+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/4125292908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406a20e40 0x55a406b9f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.893+0000 7f3278fc3b80 1 -- 172.21.15.44:0/4125292908 >> 172.21.15.44:0/4125292908 conn(0x55a406aba390 msgr2=0x55a406b9b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.893+0000 7f3278fc3b80 1 -- 172.21.15.44:0/4125292908 shutdown_connections 2024-05-27T09:50:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.893+0000 7f3278fc3b80 1 -- 172.21.15.44:0/4125292908 wait complete. 2024-05-27T09:50:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3278fc3b80 1 -- start start 2024-05-27T09:50:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3278fc3b80 1 Processor -- start 2024-05-27T09:50:32.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bb8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bb87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b9fe70 0x55a406bb2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a4069c3fa0 con 0x55a406a20e40 2024-05-27T09:50:32.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a406a5e760 con 0x55a406b9fe70 2024-05-27T09:50:32.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a406bacb70 con 0x55a406b0c9f0 2024-05-27T09:50:32.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3278fc3b80 10 client.0 _ll_register_callbacks cb 0x55a406a67c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:32.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3262a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bb87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3262a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bb87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49744/0 (socket says 172.21.15.44:49744) 2024-05-27T09:50:32.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3263400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bb8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.895+0000 7f3262a00700 1 -- 172.21.15.44:0/1736327323 learned_addr learned my addr 172.21.15.44:0/1736327323 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:32.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.896+0000 7f3263e00700 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b9fe70 0x55a406bb2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.896+0000 7f3262a00700 1 -- 172.21.15.44:0/1736327323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b9fe70 msgr2=0x55a406bb2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:32.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.896+0000 7f3262a00700 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b9fe70 0x55a406bb2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.896+0000 7f3262a00700 1 -- 172.21.15.44:0/1736327323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 msgr2=0x55a406bb8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:32.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.896+0000 7f3262a00700 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bb8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.896+0000 7f3262a00700 1 -- 172.21.15.44:0/1736327323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a406a0d8d0 con 0x55a406b0c9f0 2024-05-27T09:50:32.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.896+0000 7f3263400700 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bb8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:32.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.896+0000 7f3262a00700 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bb87a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3250000c00 tx=0x7f32500052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df157c9f7aff19ec server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:32.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.896+0000 7f324d600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:50:32.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.897+0000 7f324d600700 1 -- 172.21.15.44:0/1736327323 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325002e070 con 0x55a406b0c9f0 2024-05-27T09:50:32.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.897+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1736327323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f325000f910 con 0x55a406b0c9f0 2024-05-27T09:50:32.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.897+0000 7f324d600700 1 -- 172.21.15.44:0/1736327323 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f3250003a90 con 0x55a406b0c9f0 2024-05-27T09:50:32.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.898+0000 7f324d600700 1 -- 172.21.15.44:0/1736327323 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f325003a050 con 0x55a406b0c9f0 2024-05-27T09:50:32.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.898+0000 7f3278fc3b80 10 client.28249 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:32.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.898+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1736327323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55a406a8e060 con 0x55a406b0c9f0 2024-05-27T09:50:32.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.898+0000 7f324d600700 1 -- 172.21.15.44:0/1736327323 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f325000f910 con 0x55a406b0c9f0 2024-05-27T09:50:32.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.898+0000 7f324d600700 1 -- 172.21.15.44:0/1736327323 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f3250020070 con 0x55a406b0c9f0 2024-05-27T09:50:32.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.899+0000 7f3278fc3b80 10 client.28249 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:32.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.899+0000 7f3278fc3b80 10 client.28249 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:32.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.899+0000 7f3278fc3b80 20 client.28249 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:32.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.899+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a406bc2180 0x55a406bc45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.899+0000 7f3278fc3b80 4 client.28249 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:32.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.899+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1736327323 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a406acfd90 con 0x55a406bc2180 2024-05-27T09:50:32.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.899+0000 7f3263400700 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a406bc2180 0x55a406bc45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.900+0000 7f324d600700 10 client.28249 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:32.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.900+0000 7f3263400700 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a406bc2180 0x55a406bc45a0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71e55fe04c5f5578 server_cookie=c7468f6135c771ef in_seq=0 out_seq=0 2024-05-27T09:50:32.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f324d600700 1 -- 172.21.15.44:0/1736327323 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a406acfd90 con 0x55a406bc2180 2024-05-27T09:50:32.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f324d600700 10 client.28249 handle_command_reply: tid=0 2024-05-27T09:50:32.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f3278fc3b80 1 client.28249 shutdown 2024-05-27T09:50:32.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1736327323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 msgr2=0x55a406bb87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:32.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bb87a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3250000c00 tx=0x7f32500052f0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1736327323 shutdown_connections 2024-05-27T09:50:32.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a406bc2180 0x55a406bc45a0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b9fe70 0x55a406bb2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bb8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1736327323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bb87a0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1736327323 >> 172.21.15.44:0/1736327323 conn(0x55a406aba390 msgr2=0x55a406b9f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:32.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.901+0000 7f324d600700 0 client.28249 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:32.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.902+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1736327323 shutdown_connections 2024-05-27T09:50:32.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.902+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1736327323 wait complete. 2024-05-27T09:50:32.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.902+0000 7f3278fc3b80 20 client.28249 trim_cache size 0 max 16384 2024-05-27T09:50:32.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.920+0000 7f3278fc3b80 1 Processor -- start 2024-05-27T09:50:32.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.920+0000 7f3278fc3b80 1 -- start start 2024-05-27T09:50:32.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bb2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b0c9f0 0x55a406bab840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b9fe70 0x55a406babd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f32641af810 con 0x55a406a20e40 2024-05-27T09:50:32.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f32641ad040 con 0x55a406b0c9f0 2024-05-27T09:50:32.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f32641aafb0 con 0x55a406b9fe70 2024-05-27T09:50:32.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3263e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b9fe70 0x55a406babd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3262a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b0c9f0 0x55a406bab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3263400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bb2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3262a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b0c9f0 0x55a406bab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:57952/0 (socket says 172.21.15.44:57952) 2024-05-27T09:50:32.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3263e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b9fe70 0x55a406babd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49758/0 (socket says 172.21.15.44:49758) 2024-05-27T09:50:32.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3263400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bb2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44866/0 (socket says 172.21.15.44:44866) 2024-05-27T09:50:32.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.921+0000 7f3262a00700 1 -- 172.21.15.44:0/4289329505 learned_addr learned my addr 172.21.15.44:0/4289329505 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:32.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.922+0000 7f3263e00700 1 -- 172.21.15.44:0/4289329505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b0c9f0 msgr2=0x55a406bab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.922+0000 7f3263e00700 1 --2- 172.21.15.44:0/4289329505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b0c9f0 0x55a406bab840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.922+0000 7f3263e00700 1 -- 172.21.15.44:0/4289329505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 msgr2=0x55a406bb2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.922+0000 7f3263e00700 1 --2- 172.21.15.44:0/4289329505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bb2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.922+0000 7f3263e00700 1 -- 172.21.15.44:0/4289329505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a406b97240 con 0x55a406b9fe70 2024-05-27T09:50:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.922+0000 7f3263400700 1 --2- 172.21.15.44:0/4289329505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bb2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.922+0000 7f3263e00700 1 --2- 172.21.15.44:0/4289329505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b9fe70 0x55a406babd80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f325c007940 tx=0x7f325c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a07c10f1e0929bf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.923+0000 7f324fe00700 1 -- 172.21.15.44:0/4289329505 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c016da0 con 0x55a406b9fe70 2024-05-27T09:50:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.923+0000 7f324fe00700 1 -- 172.21.15.44:0/4289329505 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f325c01c030 con 0x55a406b9fe70 2024-05-27T09:50:32.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.923+0000 7f324fe00700 1 -- 172.21.15.44:0/4289329505 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c024630 con 0x55a406b9fe70 2024-05-27T09:50:32.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.924+0000 7f3278fc3b80 1 -- 172.21.15.44:0/4289329505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b9fe70 msgr2=0x55a406babd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:32.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.924+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/4289329505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b9fe70 0x55a406babd80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f325c007940 tx=0x7f325c016000 comp rx=0 tx=0).stop 2024-05-27T09:50:32.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.924+0000 7f3278fc3b80 1 -- 172.21.15.44:0/4289329505 shutdown_connections 2024-05-27T09:50:32.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.924+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/4289329505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b9fe70 0x55a406babd80 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.924+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/4289329505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bb2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.924+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/4289329505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b0c9f0 0x55a406bab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.924+0000 7f3278fc3b80 1 -- 172.21.15.44:0/4289329505 >> 172.21.15.44:0/4289329505 conn(0x55a406aba390 msgr2=0x55a406bb2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:32.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.924+0000 7f3278fc3b80 1 -- 172.21.15.44:0/4289329505 shutdown_connections 2024-05-27T09:50:32.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.924+0000 7f3278fc3b80 1 -- 172.21.15.44:0/4289329505 wait complete. 2024-05-27T09:50:32.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.925+0000 7f3278fc3b80 1 -- start start 2024-05-27T09:50:32.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.925+0000 7f3278fc3b80 1 Processor -- start 2024-05-27T09:50:32.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bbb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bbbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3278fc3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b9fe70 0x55a406c2a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f32641ab500 con 0x55a406a20e40 2024-05-27T09:50:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f32641ac970 con 0x55a406b9fe70 2024-05-27T09:50:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3278fc3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f32641abbd0 con 0x55a406b0c9f0 2024-05-27T09:50:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3278fc3b80 10 client.0 _ll_register_callbacks cb 0x55a406a67c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3262a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bbbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3263400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bbb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3262a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bbbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49766/0 (socket says 172.21.15.44:49766) 2024-05-27T09:50:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3262a00700 1 -- 172.21.15.44:0/1200279125 learned_addr learned my addr 172.21.15.44:0/1200279125 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.926+0000 7f3263e00700 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b9fe70 0x55a406c2a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.927+0000 7f3262a00700 1 -- 172.21.15.44:0/1200279125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b9fe70 msgr2=0x55a406c2a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:32.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.927+0000 7f3262a00700 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b9fe70 0x55a406c2a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.927+0000 7f3262a00700 1 -- 172.21.15.44:0/1200279125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 msgr2=0x55a406bbb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:32.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.927+0000 7f3262a00700 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bbb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.927+0000 7f3262a00700 1 -- 172.21.15.44:0/1200279125 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a406c2af20 con 0x55a406b0c9f0 2024-05-27T09:50:32.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.927+0000 7f3263400700 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bbb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T09:50:32.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.927+0000 7f3262a00700 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bbbac0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3250000c00 tx=0x7f325000b350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cfbd27bfd82144a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:32.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.927+0000 7f324d600700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:50:32.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.927+0000 7f324d600700 1 -- 172.21.15.44:0/1200279125 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325002e040 con 0x55a406b0c9f0 2024-05-27T09:50:32.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.927+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1200279125 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f325c01c030 con 0x55a406b0c9f0 2024-05-27T09:50:32.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.927+0000 7f324d600700 1 -- 172.21.15.44:0/1200279125 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8651+0+0 (secure 0 0 0) 0x7f325000b760 con 0x55a406b0c9f0 2024-05-27T09:50:32.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.928+0000 7f324d600700 1 -- 172.21.15.44:0/1200279125 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f325000b9a0 con 0x55a406b0c9f0 2024-05-27T09:50:32.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.928+0000 7f3278fc3b80 10 client.28255 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:32.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.928+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1200279125 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55a406c2c010 con 0x55a406b0c9f0 2024-05-27T09:50:32.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.928+0000 7f324d600700 1 -- 172.21.15.44:0/1200279125 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f325c01c030 con 0x55a406b0c9f0 2024-05-27T09:50:32.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.928+0000 7f324d600700 1 -- 172.21.15.44:0/1200279125 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f3250020040 con 0x55a406b0c9f0 2024-05-27T09:50:32.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.928+0000 7f3278fc3b80 10 client.28255 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:32.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.928+0000 7f3278fc3b80 10 client.28255 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:32.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.928+0000 7f3278fc3b80 20 client.28255 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:32.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.929+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a406c2c680 0x55a406c2eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:32.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.929+0000 7f3278fc3b80 4 client.28255 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:50:32.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.929+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1200279125 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a406b9ab10 con 0x55a406c2c680 2024-05-27T09:50:32.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.929+0000 7f3263400700 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a406c2c680 0x55a406c2eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:32.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f324d600700 10 client.28255 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:32.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f3263400700 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a406c2c680 0x55a406c2eaa0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb1c7db72a951142 server_cookie=eba4f4f6070c74f5 in_seq=0 out_seq=0 2024-05-27T09:50:32.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f324d600700 1 -- 172.21.15.44:0/1200279125 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a406b9ab10 con 0x55a406c2c680 2024-05-27T09:50:32.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f324d600700 10 client.28255 handle_command_reply: tid=0 2024-05-27T09:50:32.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f3278fc3b80 1 client.28255 shutdown 2024-05-27T09:50:32.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1200279125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 msgr2=0x55a406bbbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:32.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bbbac0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3250000c00 tx=0x7f325000b350 comp rx=0 tx=0).stop 2024-05-27T09:50:32.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1200279125 shutdown_connections 2024-05-27T09:50:32.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x55a406c2c680 0x55a406c2eaa0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a406b9fe70 0x55a406c2a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a406a20e40 0x55a406bbb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f3278fc3b80 1 --2- 172.21.15.44:0/1200279125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a406b0c9f0 0x55a406bbbac0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f324d600700 0 client.28255 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:32.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.930+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1200279125 >> 172.21.15.44:0/1200279125 conn(0x55a406aba390 msgr2=0x55a406ba0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:32.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.931+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1200279125 shutdown_connections 2024-05-27T09:50:32.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.931+0000 7f3278fc3b80 1 -- 172.21.15.44:0/1200279125 wait complete. 2024-05-27T09:50:32.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.931+0000 7f3278fc3b80 20 client.28255 trim_cache size 0 max 16384 2024-05-27T09:50:32.937 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:50:32.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 -- 172.21.15.44:0/3003626292 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32480b2a70 msgr2=0x7f32480b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:32.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32480b2a70 0x7f32480b4f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55a406ba1ff0 tx=0x7f325c00b040 comp rx=0 tx=0).stop 2024-05-27T09:50:32.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 -- 172.21.15.44:0/3003626292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3264104640 msgr2=0x7f326419d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:32.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3264104640 0x7f326419d0f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f325400ec10 tx=0x7f325400ef20 comp rx=0 tx=0).stop 2024-05-27T09:50:32.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 -- 172.21.15.44:0/3003626292 shutdown_connections 2024-05-27T09:50:32.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f32480b2a70 0x7f32480b4f80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3264109870 0x7f32641a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3264104640 0x7f326419d0f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 --2- 172.21.15.44:0/3003626292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3264108f20 0x7f326419d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:32.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 -- 172.21.15.44:0/3003626292 >> 172.21.15.44:0/3003626292 conn(0x7f3264079ef0 msgr2=0x7f326410f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:32.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 -- 172.21.15.44:0/3003626292 shutdown_connections 2024-05-27T09:50:32.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:32.932+0000 7f326a400700 1 -- 172.21.15.44:0/3003626292 wait complete. 2024-05-27T09:50:34.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:34 smithi044 ceph-mon[31618]: pgmap v482: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2024-05-27T09:50:34.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:34 smithi044 ceph-mon[31618]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T09:50:34.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:34 smithi050 ceph-mon[36371]: pgmap v482: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2024-05-27T09:50:34.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:34 smithi050 ceph-mon[36371]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T09:50:34.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:34 smithi189 ceph-mon[36671]: pgmap v482: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2024-05-27T09:50:34.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:34 smithi189 ceph-mon[36671]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T09:50:34.873 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T09:50:34.874 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T09:50:35.730 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.772+0000 7f7bd4c00700 1 -- 172.21.15.44:0/773051599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bd01054f0 msgr2=0x7f7bd0105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.772+0000 7f7bd4c00700 1 --2- 172.21.15.44:0/773051599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bd01054f0 0x7f7bd0105900 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0009990 tx=0x7f7bc0009ca0 comp rx=0 tx=0).stop 2024-05-27T09:50:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.772+0000 7f7bd4c00700 1 -- 172.21.15.44:0/773051599 shutdown_connections 2024-05-27T09:50:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.772+0000 7f7bd4c00700 1 --2- 172.21.15.44:0/773051599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7bd010e920 0x7f7bd0110e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.772+0000 7f7bd4c00700 1 --2- 172.21.15.44:0/773051599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bd01054f0 0x7f7bd0105900 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.772+0000 7f7bd4c00700 1 --2- 172.21.15.44:0/773051599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd0105e40 0x7f7bd010e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:35.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.772+0000 7f7bd4c00700 1 -- 172.21.15.44:0/773051599 >> 172.21.15.44:0/773051599 conn(0x7f7bd00fef20 msgr2=0x7f7bd0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:35.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.772+0000 7f7bd4c00700 1 -- 172.21.15.44:0/773051599 shutdown_connections 2024-05-27T09:50:35.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.772+0000 7f7bd4c00700 1 -- 172.21.15.44:0/773051599 wait complete. 2024-05-27T09:50:35.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.773+0000 7f7bd4c00700 1 Processor -- start 2024-05-27T09:50:35.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.773+0000 7f7bd4c00700 1 -- start start 2024-05-27T09:50:35.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.774+0000 7f7bd4c00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7bd01054f0 0x7f7bd019d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:35.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.774+0000 7f7bd4c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bd0105e40 0x7f7bd019d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:35.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.774+0000 7f7bd4c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd010e920 0x7f7bd01a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:35.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.774+0000 7f7bd4c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd0114440 con 0x7f7bd010e920 2024-05-27T09:50:35.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.774+0000 7f7bd4c00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd01142c0 con 0x7f7bd01054f0 2024-05-27T09:50:35.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.774+0000 7f7bd4c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd01145c0 con 0x7f7bd0105e40 2024-05-27T09:50:35.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.774+0000 7f7bcd400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bd0105e40 0x7f7bd019d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:35.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.774+0000 7f7bcd400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bd0105e40 0x7f7bd019d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49786/0 (socket says 172.21.15.44:49786) 2024-05-27T09:50:35.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.774+0000 7f7bcd400700 1 -- 172.21.15.44:0/2580632808 learned_addr learned my addr 172.21.15.44:0/2580632808 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:35.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.775+0000 7f7bcde00700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7bd01054f0 0x7f7bd019d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:35.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.775+0000 7f7bce800700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd010e920 0x7f7bd01a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:35.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.775+0000 7f7bcd400700 1 -- 172.21.15.44:0/2580632808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7bd01054f0 msgr2=0x7f7bd019d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:35.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.775+0000 7f7bcd400700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7bd01054f0 0x7f7bd019d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:35.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.775+0000 7f7bcd400700 1 -- 172.21.15.44:0/2580632808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd010e920 msgr2=0x7f7bd01a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:35.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.775+0000 7f7bcd400700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd010e920 0x7f7bd01a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:35.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.775+0000 7f7bcd400700 1 -- 172.21.15.44:0/2580632808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd01a1ee0 con 0x7f7bd0105e40 2024-05-27T09:50:35.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.775+0000 7f7bce800700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd010e920 0x7f7bd01a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:35.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.776+0000 7f7bcd400700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bd0105e40 0x7f7bd019d5b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00e9a0 tx=0x7f7bbc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:35.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.776+0000 7f7bc6a00700 1 -- 172.21.15.44:0/2580632808 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00cbd0 con 0x7f7bd0105e40 2024-05-27T09:50:35.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.776+0000 7f7bc6a00700 1 -- 172.21.15.44:0/2580632808 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bbc00cd70 con 0x7f7bd0105e40 2024-05-27T09:50:35.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.776+0000 7f7bc6a00700 1 -- 172.21.15.44:0/2580632808 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc018680 con 0x7f7bd0105e40 2024-05-27T09:50:35.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.776+0000 7f7bd4c00700 1 -- 172.21.15.44:0/2580632808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd01a21d0 con 0x7f7bd0105e40 2024-05-27T09:50:35.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.776+0000 7f7bd4c00700 1 -- 172.21.15.44:0/2580632808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd01a9af0 con 0x7f7bd0105e40 2024-05-27T09:50:35.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.778+0000 7f7bd4c00700 1 -- 172.21.15.44:0/2580632808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd00512b0 con 0x7f7bd0105e40 2024-05-27T09:50:35.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.781+0000 7f7bc6a00700 1 -- 172.21.15.44:0/2580632808 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f7bbc010430 con 0x7f7bd0105e40 2024-05-27T09:50:35.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.783+0000 7f7bc6a00700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7bb40b2a70 0x7f7bb40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:35.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.783+0000 7f7bc6a00700 1 -- 172.21.15.44:0/2580632808 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f7bbc0e28e0 con 0x7f7bd0105e40 2024-05-27T09:50:35.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.783+0000 7f7bcde00700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7bb40b2a70 0x7f7bb40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:35.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.784+0000 7f7bcde00700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7bb40b2a70 0x7f7bb40b4f80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000b2e0 tx=0x7f7bc0005960 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:35.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:35.787+0000 7f7bc6a00700 1 -- 172.21.15.44:0/2580632808 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7bbc0aa210 con 0x7f7bd0105e40 2024-05-27T09:50:36.031 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:36 smithi044 ceph-mon[31618]: pgmap v484: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 9.2 KiB/s wr, 9 op/s 2024-05-27T09:50:36.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.087+0000 7f7bd4c00700 1 -- 172.21.15.44:0/2580632808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bd01a9cc0 con 0x7f7bd0105e40 2024-05-27T09:50:36.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.088+0000 7f7bc6a00700 1 -- 172.21.15.44:0/2580632808 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11870 (secure 0 0 0) 0x7f7bbc0a9b10 con 0x7f7bd0105e40 2024-05-27T09:50:36.090 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:50:36.090 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:48:16.864365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24503]},"id":1}]} 2024-05-27T09:50:36.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 -- 172.21.15.44:0/2580632808 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7bb40b2a70 msgr2=0x7f7bb40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:36.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7bb40b2a70 0x7f7bb40b4f80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000b2e0 tx=0x7f7bc0005960 comp rx=0 tx=0).stop 2024-05-27T09:50:36.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 -- 172.21.15.44:0/2580632808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bd0105e40 msgr2=0x7f7bd019d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:36.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bd0105e40 0x7f7bd019d5b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00e9a0 tx=0x7f7bbc00ecb0 comp rx=0 tx=0).stop 2024-05-27T09:50:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 -- 172.21.15.44:0/2580632808 shutdown_connections 2024-05-27T09:50:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f7bb40b2a70 0x7f7bb40b4f80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd010e920 0x7f7bd01a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:36.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7bd01054f0 0x7f7bd019d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:36.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 --2- 172.21.15.44:0/2580632808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bd0105e40 0x7f7bd019d5b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:36.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 -- 172.21.15.44:0/2580632808 >> 172.21.15.44:0/2580632808 conn(0x7f7bd00fef20 msgr2=0x7f7bd01031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:36.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 -- 172.21.15.44:0/2580632808 shutdown_connections 2024-05-27T09:50:36.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:36.092+0000 7f7bd4c00700 1 -- 172.21.15.44:0/2580632808 wait complete. 2024-05-27T09:50:36.095 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 43 2024-05-27T09:50:36.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:36 smithi050 ceph-mon[36371]: pgmap v484: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 9.2 KiB/s wr, 9 op/s 2024-05-27T09:50:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:36 smithi189 ceph-mon[36671]: pgmap v484: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 9.2 KiB/s wr, 9 op/s 2024-05-27T09:50:37.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:37 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2580632808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:37.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:37 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2580632808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:37.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:37 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2580632808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:37.398 INFO:tasks.quiescer:done joining Quiescers 2024-05-27T09:50:37.398 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-05-27T09:50:37.409 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi044.front.sepia.ceph.com 2024-05-27T09:50:37.436 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-27T09:50:37.447 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-27T09:50:37.447 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T09:50:37.448 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-05-27T09:50:37.448 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-27T09:50:37.602 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-27T09:50:37.704 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-27T09:50:37.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.983+0000 7f62a9e00640 1 -- 172.21.15.44:0/4079522804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62a4109c80 msgr2=0x7f62a4103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:37.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.983+0000 7f62a9e00640 1 --2- 172.21.15.44:0/4079522804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62a4109c80 0x7f62a4103820 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6294009900 tx=0x7f629402a3f0 comp rx=0 tx=0).stop 2024-05-27T09:50:37.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.984+0000 7f62a9e00640 1 -- 172.21.15.44:0/4079522804 shutdown_connections 2024-05-27T09:50:37.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.984+0000 7f62a9e00640 1 --2- 172.21.15.44:0/4079522804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a41043d0 0x7f62a41107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:37.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.984+0000 7f62a9e00640 1 --2- 172.21.15.44:0/4079522804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62a410a120 0x7f62a4103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:37.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.984+0000 7f62a9e00640 1 --2- 172.21.15.44:0/4079522804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62a4109c80 0x7f62a4103820 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:37.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.984+0000 7f62a9e00640 1 -- 172.21.15.44:0/4079522804 >> 172.21.15.44:0/4079522804 conn(0x7f62a40ff340 msgr2=0x7f62a4101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:37.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.984+0000 7f62a9e00640 1 -- 172.21.15.44:0/4079522804 shutdown_connections 2024-05-27T09:50:37.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.984+0000 7f62a9e00640 1 -- 172.21.15.44:0/4079522804 wait complete. 2024-05-27T09:50:37.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.985+0000 7f62a9e00640 1 Processor -- start 2024-05-27T09:50:37.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.985+0000 7f62a9e00640 1 -- start start 2024-05-27T09:50:37.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.985+0000 7f62a9e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62a41043d0 0x7f62a41a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:37.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.985+0000 7f62a9e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62a4109c80 0x7f62a41a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:37.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.985+0000 7f62a9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a410a120 0x7f62a419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:37.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.985+0000 7f62a9e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f62a4113ea0 con 0x7f62a41043d0 2024-05-27T09:50:37.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.985+0000 7f62a9e00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f62a4113d20 con 0x7f62a4109c80 2024-05-27T09:50:37.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.985+0000 7f62a9e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f62a4114020 con 0x7f62a410a120 2024-05-27T09:50:37.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62a41043d0 0x7f62a41a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:37.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62a41043d0 0x7f62a41a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44938/0 (socket says 172.21.15.44:44938) 2024-05-27T09:50:37.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a2a00640 1 -- 172.21.15.44:0/1398315766 learned_addr learned my addr 172.21.15.44:0/1398315766 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:37.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a2000640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62a4109c80 0x7f62a41a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:37.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a3400640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a410a120 0x7f62a419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:37.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a2a00640 1 -- 172.21.15.44:0/1398315766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a410a120 msgr2=0x7f62a419a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:37.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a2a00640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a410a120 0x7f62a419a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:37.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a2a00640 1 -- 172.21.15.44:0/1398315766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62a4109c80 msgr2=0x7f62a41a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:37.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a2a00640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62a4109c80 0x7f62a41a0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:37.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a2a00640 1 -- 172.21.15.44:0/1398315766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a419ae30 con 0x7f62a41043d0 2024-05-27T09:50:37.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a3400640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a410a120 0x7f62a419a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:37.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.986+0000 7f62a2000640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62a4109c80 0x7f62a41a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:37.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.987+0000 7f62a2a00640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62a41043d0 0x7f62a41a0580 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6294005ab0 tx=0x7f6294003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:37.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.987+0000 7f62a8a00640 1 -- 172.21.15.44:0/1398315766 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629403f070 con 0x7f62a41043d0 2024-05-27T09:50:37.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.987+0000 7f62a8a00640 1 -- 172.21.15.44:0/1398315766 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6294031b50 con 0x7f62a41043d0 2024-05-27T09:50:37.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.987+0000 7f62a9e00640 1 -- 172.21.15.44:0/1398315766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a419b060 con 0x7f62a41043d0 2024-05-27T09:50:37.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.987+0000 7f62a8a00640 1 -- 172.21.15.44:0/1398315766 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6294035030 con 0x7f62a41043d0 2024-05-27T09:50:37.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.987+0000 7f62a9e00640 1 -- 172.21.15.44:0/1398315766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a419b550 con 0x7f62a41043d0 2024-05-27T09:50:37.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.991+0000 7f62a8a00640 1 -- 172.21.15.44:0/1398315766 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f6294042070 con 0x7f62a41043d0 2024-05-27T09:50:37.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.993+0000 7f62a8a00640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f62880b2950 0x7f62880b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:37.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.993+0000 7f62a8a00640 1 -- 172.21.15.44:0/1398315766 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f6294102d80 con 0x7f62a41043d0 2024-05-27T09:50:37.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.993+0000 7f62a2000640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f62880b2950 0x7f62880b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:38.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:37.988+0000 7f62a9e00640 1 -- 172.21.15.44:0/1398315766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62a4104f60 con 0x7f62a41043d0 2024-05-27T09:50:38.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.002+0000 7f62a2000640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f62880b2950 0x7f62880b4e60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f62a419bea0 tx=0x7f6290005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:38.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.008+0000 7f62a8a00640 1 -- 172.21.15.44:0/1398315766 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f62940ca6b0 con 0x7f62a41043d0 2024-05-27T09:50:38.204 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:38 smithi044 ceph-mon[31618]: pgmap v485: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-05-27T09:50:38.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.264+0000 7f62a9e00640 1 -- 172.21.15.44:0/1398315766 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f62a40ffe60 con 0x7f62880b2950 2024-05-27T09:50:38.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.282+0000 7f62a8a00640 1 -- 172.21.15.44:0/1398315766 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f62a40ffe60 con 0x7f62880b2950 2024-05-27T09:50:38.283 INFO:teuthology.orchestra.run.smithi044.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-27T00:00:00", "created": "2024-05-27T09:35:33", "first": "2024-05-27T09:36:00", "last": "2024-05-27T09:50:00", "last_pruned": "2024-05-27T09:50:01", "created_count": 15, "pruned_count": 9, "active": true} 2024-05-27T09:50:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 -- 172.21.15.44:0/1398315766 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f62880b2950 msgr2=0x7f62880b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f62880b2950 0x7f62880b4e60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f62a419bea0 tx=0x7f6290005f90 comp rx=0 tx=0).stop 2024-05-27T09:50:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 -- 172.21.15.44:0/1398315766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62a41043d0 msgr2=0x7f62a41a0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62a41043d0 0x7f62a41a0580 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6294005ab0 tx=0x7f6294003b90 comp rx=0 tx=0).stop 2024-05-27T09:50:38.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 -- 172.21.15.44:0/1398315766 shutdown_connections 2024-05-27T09:50:38.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a410a120 0x7f62a419a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62a4109c80 0x7f62a41a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f62880b2950 0x7f62880b4e60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 --2- 172.21.15.44:0/1398315766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62a41043d0 0x7f62a41a0580 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 -- 172.21.15.44:0/1398315766 >> 172.21.15.44:0/1398315766 conn(0x7f62a40ff340 msgr2=0x7f62a40ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:38.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 -- 172.21.15.44:0/1398315766 shutdown_connections 2024-05-27T09:50:38.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.285+0000 7f62a9e00640 1 -- 172.21.15.44:0/1398315766 wait complete. 2024-05-27T09:50:38.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:38 smithi050 ceph-mon[36371]: pgmap v485: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-05-27T09:50:38.316 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-27T09:50:38.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:38 smithi189 ceph-mon[36671]: pgmap v485: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-05-27T09:50:38.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.592+0000 7fe366000640 1 -- 172.21.15.44:0/2741717070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe36010a150 msgr2=0x7fe360103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:38.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.592+0000 7fe366000640 1 --2- 172.21.15.44:0/2741717070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe36010a150 0x7fe360103d90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe348009900 tx=0x7fe34802a3f0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.592+0000 7fe366000640 1 -- 172.21.15.44:0/2741717070 shutdown_connections 2024-05-27T09:50:38.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.592+0000 7fe366000640 1 --2- 172.21.15.44:0/2741717070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe360104400 0x7fe3601107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.592+0000 7fe366000640 1 --2- 172.21.15.44:0/2741717070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe36010a150 0x7fe360103d90 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.592+0000 7fe366000640 1 --2- 172.21.15.44:0/2741717070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe360109cb0 0x7fe360103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.592+0000 7fe366000640 1 -- 172.21.15.44:0/2741717070 >> 172.21.15.44:0/2741717070 conn(0x7fe3600ff480 msgr2=0x7fe3601018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:38.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.593+0000 7fe366000640 1 -- 172.21.15.44:0/2741717070 shutdown_connections 2024-05-27T09:50:38.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.593+0000 7fe366000640 1 -- 172.21.15.44:0/2741717070 wait complete. 2024-05-27T09:50:38.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.594+0000 7fe366000640 1 Processor -- start 2024-05-27T09:50:38.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.594+0000 7fe366000640 1 -- start start 2024-05-27T09:50:38.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.594+0000 7fe366000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe360104400 0x7fe36019d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:38.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.594+0000 7fe366000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe360109cb0 0x7fe36019dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:38.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.594+0000 7fe366000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe36010a150 0x7fe3601a8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:38.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.594+0000 7fe366000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe360113ef0 con 0x7fe360104400 2024-05-27T09:50:38.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.594+0000 7fe366000640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe360113d70 con 0x7fe36010a150 2024-05-27T09:50:38.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.594+0000 7fe366000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe360114070 con 0x7fe360109cb0 2024-05-27T09:50:38.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.595+0000 7fe35f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe360104400 0x7fe36019d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:38.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.595+0000 7fe35f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe360104400 0x7fe36019d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44952/0 (socket says 172.21.15.44:44952) 2024-05-27T09:50:38.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.595+0000 7fe35f400640 1 -- 172.21.15.44:0/1671078368 learned_addr learned my addr 172.21.15.44:0/1671078368 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:38.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.595+0000 7fe35fe00640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe36010a150 0x7fe3601a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:38.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.595+0000 7fe356a00640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe360109cb0 0x7fe36019dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:38.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.595+0000 7fe35f400640 1 -- 172.21.15.44:0/1671078368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe360109cb0 msgr2=0x7fe36019dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:38.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.595+0000 7fe35f400640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe360109cb0 0x7fe36019dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.595+0000 7fe35f400640 1 -- 172.21.15.44:0/1671078368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe36010a150 msgr2=0x7fe3601a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:38.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.595+0000 7fe35f400640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe36010a150 0x7fe3601a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.595+0000 7fe35f400640 1 -- 172.21.15.44:0/1671078368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3601a8d90 con 0x7fe360104400 2024-05-27T09:50:38.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.596+0000 7fe35f400640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe360104400 0x7fe36019d7d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe35000e9a0 tx=0x7fe35000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:38.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.596+0000 7fe35cc00640 1 -- 172.21.15.44:0/1671078368 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35000cd80 con 0x7fe360104400 2024-05-27T09:50:38.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.596+0000 7fe35cc00640 1 -- 172.21.15.44:0/1671078368 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3500035f0 con 0x7fe360104400 2024-05-27T09:50:38.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.596+0000 7fe366000640 1 -- 172.21.15.44:0/1671078368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3601a8f60 con 0x7fe360104400 2024-05-27T09:50:38.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.596+0000 7fe35cc00640 1 -- 172.21.15.44:0/1671078368 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe350010690 con 0x7fe360104400 2024-05-27T09:50:38.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.596+0000 7fe366000640 1 -- 172.21.15.44:0/1671078368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3601a93c0 con 0x7fe360104400 2024-05-27T09:50:38.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.597+0000 7fe366000640 1 -- 172.21.15.44:0/1671078368 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe36006a230 con 0x7fe360104400 2024-05-27T09:50:38.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.600+0000 7fe35cc00640 1 -- 172.21.15.44:0/1671078368 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fe350004000 con 0x7fe360104400 2024-05-27T09:50:38.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.601+0000 7fe35cc00640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe3400b29d0 0x7fe3400b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:38.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.602+0000 7fe35cc00640 1 -- 172.21.15.44:0/1671078368 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7fe3500e29c0 con 0x7fe360104400 2024-05-27T09:50:38.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.602+0000 7fe356a00640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe3400b29d0 0x7fe3400b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:38.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.603+0000 7fe356a00640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe3400b29d0 0x7fe3400b4ee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe34802a9e0 tx=0x7fe348031400 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.604+0000 7fe35cc00640 1 -- 172.21.15.44:0/1671078368 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe3500042f0 con 0x7fe360104400 2024-05-27T09:50:38.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.846+0000 7fe366000640 1 -- 172.21.15.44:0/1671078368 --> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fe360075600 con 0x7fe3400b29d0 2024-05-27T09:50:38.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.862+0000 7fe35cc00640 1 -- 172.21.15.44:0/1671078368 <== mgr.25072 v2:172.21.15.44:6800/3776762031 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fe360075600 con 0x7fe3400b29d0 2024-05-27T09:50:38.864 INFO:teuthology.orchestra.run.smithi044.stdout:/ 1m 6m3h 2024-05-27T09:50:38.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.865+0000 7fe366000640 1 -- 172.21.15.44:0/1671078368 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe3400b29d0 msgr2=0x7fe3400b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:38.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.865+0000 7fe366000640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe3400b29d0 0x7fe3400b4ee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe34802a9e0 tx=0x7fe348031400 comp rx=0 tx=0).stop 2024-05-27T09:50:38.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.865+0000 7fe366000640 1 -- 172.21.15.44:0/1671078368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe360104400 msgr2=0x7fe36019d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:38.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.865+0000 7fe366000640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe360104400 0x7fe36019d7d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe35000e9a0 tx=0x7fe35000ee60 comp rx=0 tx=0).stop 2024-05-27T09:50:38.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.865+0000 7fe366000640 1 -- 172.21.15.44:0/1671078368 shutdown_connections 2024-05-27T09:50:38.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.865+0000 7fe366000640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe36010a150 0x7fe3601a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.865+0000 7fe366000640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe360109cb0 0x7fe36019dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.865+0000 7fe366000640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fe3400b29d0 0x7fe3400b4ee0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.865+0000 7fe366000640 1 --2- 172.21.15.44:0/1671078368 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe360104400 0x7fe36019d7d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:38.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.865+0000 7fe366000640 1 -- 172.21.15.44:0/1671078368 >> 172.21.15.44:0/1671078368 conn(0x7fe3600ff480 msgr2=0x7fe360074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:38.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.865+0000 7fe366000640 1 -- 172.21.15.44:0/1671078368 shutdown_connections 2024-05-27T09:50:38.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:38.866+0000 7fe366000640 1 -- 172.21.15.44:0/1671078368 wait complete. 2024-05-27T09:50:38.894 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-27T09:50:38.905 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-27T09:50:39.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.169+0000 7f687f600700 1 -- 172.21.15.44:0/4257252540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6878103bb0 msgr2=0x7f68781080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:39.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.169+0000 7f687f600700 1 --2- 172.21.15.44:0/4257252540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6878103bb0 0x7f68781080e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6860009990 tx=0x7f6860009ca0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.169+0000 7f687f600700 1 -- 172.21.15.44:0/4257252540 shutdown_connections 2024-05-27T09:50:39.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.169+0000 7f687f600700 1 --2- 172.21.15.44:0/4257252540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68781086b0 0x7f6878108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.169+0000 7f687f600700 1 --2- 172.21.15.44:0/4257252540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6878103260 0x7f6878103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.169+0000 7f687f600700 1 --2- 172.21.15.44:0/4257252540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6878103bb0 0x7f68781080e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.169+0000 7f687f600700 1 -- 172.21.15.44:0/4257252540 >> 172.21.15.44:0/4257252540 conn(0x7f68780fef20 msgr2=0x7f6878101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:39.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.170+0000 7f687f600700 1 -- 172.21.15.44:0/4257252540 shutdown_connections 2024-05-27T09:50:39.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.170+0000 7f687f600700 1 -- 172.21.15.44:0/4257252540 wait complete. 2024-05-27T09:50:39.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.171+0000 7f687f600700 1 Processor -- start 2024-05-27T09:50:39.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.171+0000 7f687f600700 1 -- start start 2024-05-27T09:50:39.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.172+0000 7f687f600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6878103260 0x7f68781a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.172+0000 7f687f600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878103bb0 0x7f68781a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.172+0000 7f687f600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68781086b0 0x7f68781a5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.172+0000 7f687f600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f687811aa30 con 0x7f6878103260 2024-05-27T09:50:39.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.172+0000 7f687f600700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f687811a8b0 con 0x7f68781086b0 2024-05-27T09:50:39.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.172+0000 7f687f600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f687811abb0 con 0x7f6878103bb0 2024-05-27T09:50:39.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.172+0000 7f687cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6878103260 0x7f68781a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.173+0000 7f686fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878103bb0 0x7f68781a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.173+0000 7f687d600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68781086b0 0x7f68781a5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.173+0000 7f687d600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68781086b0 0x7f68781a5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:58036/0 (socket says 172.21.15.44:58036) 2024-05-27T09:50:39.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.173+0000 7f686fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878103bb0 0x7f68781a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49832/0 (socket says 172.21.15.44:49832) 2024-05-27T09:50:39.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.173+0000 7f687d600700 1 -- 172.21.15.44:0/1225507703 learned_addr learned my addr 172.21.15.44:0/1225507703 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:39.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.173+0000 7f686fe00700 1 -- 172.21.15.44:0/1225507703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68781086b0 msgr2=0x7f68781a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:39.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.173+0000 7f686fe00700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68781086b0 0x7f68781a5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.173+0000 7f686fe00700 1 -- 172.21.15.44:0/1225507703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6878103260 msgr2=0x7f68781a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:39.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.173+0000 7f686fe00700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6878103260 0x7f68781a1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.173+0000 7f686fe00700 1 -- 172.21.15.44:0/1225507703 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68781a62f0 con 0x7f6878103bb0 2024-05-27T09:50:39.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.173+0000 7f686fe00700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878103bb0 0x7f68781a19c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6860006010 tx=0x7f68600057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:39.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.174+0000 7f6876000700 1 -- 172.21.15.44:0/1225507703 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6860027070 con 0x7f6878103bb0 2024-05-27T09:50:39.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.174+0000 7f6876000700 1 -- 172.21.15.44:0/1225507703 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6860022470 con 0x7f6878103bb0 2024-05-27T09:50:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.174+0000 7f6876000700 1 -- 172.21.15.44:0/1225507703 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6860017650 con 0x7f6878103bb0 2024-05-27T09:50:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.174+0000 7f687f600700 1 -- 172.21.15.44:0/1225507703 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68781a6580 con 0x7f6878103bb0 2024-05-27T09:50:39.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.174+0000 7f687f600700 1 -- 172.21.15.44:0/1225507703 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68781a6a10 con 0x7f6878103bb0 2024-05-27T09:50:39.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.178+0000 7f6876000700 1 -- 172.21.15.44:0/1225507703 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f686002a070 con 0x7f6878103bb0 2024-05-27T09:50:39.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.179+0000 7f688e17fb80 1 Processor -- start 2024-05-27T09:50:39.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.179+0000 7f688e17fb80 1 -- start start 2024-05-27T09:50:39.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.179+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3e10a70 0x5587e3ea33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.179+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3d24ec0 0x5587e3ea38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.179+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eb0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.179+0000 7f688e17fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5587e3dd1ec0 con 0x5587e3ea3ef0 2024-05-27T09:50:39.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.179+0000 7f688e17fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5587e3dc1a60 con 0x5587e3e10a70 2024-05-27T09:50:39.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.179+0000 7f688e17fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5587e3dc5aa0 con 0x5587e3d24ec0 2024-05-27T09:50:39.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.180+0000 7f6876000700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68580b2a70 0x7f68580b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.180+0000 7f687cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3d24ec0 0x5587e3ea38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.180+0000 7f687d600700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3e10a70 0x5587e3ea33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.180+0000 7f687cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3d24ec0 0x5587e3ea38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49848/0 (socket says 172.21.15.44:49848) 2024-05-27T09:50:39.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.180+0000 7f686fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eb0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.180+0000 7f687cc00700 1 -- 172.21.15.44:0/3104553545 learned_addr learned my addr 172.21.15.44:0/3104553545 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:39.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.180+0000 7f687d600700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68580b2a70 0x7f68580b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.180+0000 7f6876000700 1 -- 172.21.15.44:0/1225507703 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f68600e3290 con 0x7f6878103bb0 2024-05-27T09:50:39.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.181+0000 7f687cc00700 1 -- 172.21.15.44:0/3104553545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3e10a70 msgr2=0x5587e3ea33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:39.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.181+0000 7f687cc00700 1 --2- 172.21.15.44:0/3104553545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3e10a70 0x5587e3ea33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.181+0000 7f687cc00700 1 -- 172.21.15.44:0/3104553545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 msgr2=0x5587e3eb0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:39.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.181+0000 7f687cc00700 1 --2- 172.21.15.44:0/3104553545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eb0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.181+0000 7f687cc00700 1 -- 172.21.15.44:0/3104553545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587e3d8a4a0 con 0x5587e3d24ec0 2024-05-27T09:50:39.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.181+0000 7f687d600700 1 --2- 172.21.15.44:0/3104553545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3e10a70 0x5587e3ea33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:39.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.181+0000 7f687cc00700 1 --2- 172.21.15.44:0/3104553545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3d24ec0 0x5587e3ea38f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6868009d80 tx=0x7f6868011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=403daa65200638b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:39.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.181+0000 7f686e000700 1 -- 172.21.15.44:0/3104553545 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6868004230 con 0x5587e3d24ec0 2024-05-27T09:50:39.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.181+0000 7f686e000700 1 -- 172.21.15.44:0/3104553545 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6868009e10 con 0x5587e3d24ec0 2024-05-27T09:50:39.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.181+0000 7f687d600700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68580b2a70 0x7f68580b4f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6870009d30 tx=0x7f687000b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:39.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.181+0000 7f686e000700 1 -- 172.21.15.44:0/3104553545 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6868019810 con 0x5587e3d24ec0 2024-05-27T09:50:39.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.182+0000 7f688e17fb80 1 -- 172.21.15.44:0/3104553545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3d24ec0 msgr2=0x5587e3ea38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:39.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.182+0000 7f688e17fb80 1 --2- 172.21.15.44:0/3104553545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3d24ec0 0x5587e3ea38f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6868009d80 tx=0x7f6868011720 comp rx=0 tx=0).stop 2024-05-27T09:50:39.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.182+0000 7f688e17fb80 1 -- 172.21.15.44:0/3104553545 shutdown_connections 2024-05-27T09:50:39.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.182+0000 7f688e17fb80 1 --2- 172.21.15.44:0/3104553545 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eb0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.182+0000 7f688e17fb80 1 --2- 172.21.15.44:0/3104553545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3e10a70 0x5587e3ea33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.182+0000 7f688e17fb80 1 --2- 172.21.15.44:0/3104553545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3d24ec0 0x5587e3ea38f0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.182+0000 7f688e17fb80 1 -- 172.21.15.44:0/3104553545 >> 172.21.15.44:0/3104553545 conn(0x5587e3dbe410 msgr2=0x5587e3e9f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:39.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.183+0000 7f688e17fb80 1 -- 172.21.15.44:0/3104553545 shutdown_connections 2024-05-27T09:50:39.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.183+0000 7f688e17fb80 1 -- 172.21.15.44:0/3104553545 wait complete. 2024-05-27T09:50:39.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.183+0000 7f688e17fb80 1 -- start start 2024-05-27T09:50:39.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.184+0000 7f688e17fb80 1 Processor -- start 2024-05-27T09:50:39.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.184+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3ead680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.184+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 0x5587e3eadbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.184+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.184+0000 7f688e17fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5587e3cc8020 con 0x5587e3ea3ef0 2024-05-27T09:50:39.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.184+0000 7f688e17fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5587e3d627e0 con 0x5587e3d24ec0 2024-05-27T09:50:39.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.184+0000 7f688e17fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5587e3eb0c00 con 0x5587e3e10a70 2024-05-27T09:50:39.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.184+0000 7f688e17fb80 10 client.0 _ll_register_callbacks cb 0x5587e3d6bcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:39.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.185+0000 7f687d600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.185+0000 7f686fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 0x5587e3eadbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.185+0000 7f687d600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44994/0 (socket says 172.21.15.44:44994) 2024-05-27T09:50:39.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.185+0000 7f687d600700 1 -- 172.21.15.44:0/2427186833 learned_addr learned my addr 172.21.15.44:0/2427186833 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:39.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.185+0000 7f687cc00700 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3ead680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.185+0000 7f686fe00700 1 -- 172.21.15.44:0/2427186833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 msgr2=0x5587e3ead680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:39.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.185+0000 7f686fe00700 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3ead680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.185+0000 7f686fe00700 1 -- 172.21.15.44:0/2427186833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 msgr2=0x5587e3eae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:39.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.185+0000 7f686fe00700 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.185+0000 7f686fe00700 1 -- 172.21.15.44:0/2427186833 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587e3d11950 con 0x5587e3e10a70 2024-05-27T09:50:39.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.185+0000 7f687d600700 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:39.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.186+0000 7f686fe00700 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 0x5587e3eadbc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f68600e0930 tx=0x7f68600e0e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ffdd395c2a59fe3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.186+0000 7f6866a00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:50:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.186+0000 7f6866a00700 1 -- 172.21.15.44:0/2427186833 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68600a9cc0 con 0x5587e3e10a70 2024-05-27T09:50:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.186+0000 7f6866a00700 1 -- 172.21.15.44:0/2427186833 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f68600b2e90 con 0x5587e3e10a70 2024-05-27T09:50:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.186+0000 7f688e17fb80 1 -- 172.21.15.44:0/2427186833 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6868009e10 con 0x5587e3e10a70 2024-05-27T09:50:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.186+0000 7f6866a00700 1 -- 172.21.15.44:0/2427186833 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f68600aa060 con 0x5587e3e10a70 2024-05-27T09:50:39.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.186+0000 7f688e17fb80 10 client.28285 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.186+0000 7f688e17fb80 1 -- 172.21.15.44:0/2427186833 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5587e3d920e0 con 0x5587e3e10a70 2024-05-27T09:50:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.186+0000 7f6866a00700 1 -- 172.21.15.44:0/2427186833 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6868009e10 con 0x5587e3e10a70 2024-05-27T09:50:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.187+0000 7f6866a00700 1 -- 172.21.15.44:0/2427186833 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f68600050f0 con 0x5587e3e10a70 2024-05-27T09:50:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.187+0000 7f688e17fb80 10 client.28285 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.187+0000 7f688e17fb80 10 client.28285 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.187+0000 7f688e17fb80 20 client.28285 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:39.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.187+0000 7f688e17fb80 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5587e3ebdb30 0x5587e3ebff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.187+0000 7f688e17fb80 4 client.28285 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.187+0000 7f688e17fb80 1 -- 172.21.15.44:0/2427186833 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587e3dd3e10 con 0x5587e3ebdb30 2024-05-27T09:50:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.187+0000 7f687cc00700 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5587e3ebdb30 0x5587e3ebff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.188+0000 7f687cc00700 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5587e3ebdb30 0x5587e3ebff80 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43c3c46dc41c1c42 server_cookie=7542fef52dd713fe in_seq=0 out_seq=0 2024-05-27T09:50:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.188+0000 7f6866a00700 10 client.28285 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f6866a00700 1 -- 172.21.15.44:0/2427186833 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5587e3dd3e10 con 0x5587e3ebdb30 2024-05-27T09:50:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f6866a00700 10 client.28285 handle_command_reply: tid=0 2024-05-27T09:50:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f688e17fb80 1 client.28285 shutdown 2024-05-27T09:50:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f688e17fb80 1 -- 172.21.15.44:0/2427186833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 msgr2=0x5587e3eadbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f688e17fb80 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 0x5587e3eadbc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f68600e0930 tx=0x7f68600e0e90 comp rx=0 tx=0).stop 2024-05-27T09:50:39.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f688e17fb80 1 -- 172.21.15.44:0/2427186833 shutdown_connections 2024-05-27T09:50:39.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f688e17fb80 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5587e3ebdb30 0x5587e3ebff80 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f688e17fb80 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f688e17fb80 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3ead680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f6866a00700 0 client.28285 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:39.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f688e17fb80 1 --2- 172.21.15.44:0/2427186833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 0x5587e3eadbc0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.189+0000 7f688e17fb80 1 -- 172.21.15.44:0/2427186833 >> 172.21.15.44:0/2427186833 conn(0x5587e3dbe410 msgr2=0x5587e3ea3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.190+0000 7f688e17fb80 1 -- 172.21.15.44:0/2427186833 shutdown_connections 2024-05-27T09:50:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.190+0000 7f688e17fb80 1 -- 172.21.15.44:0/2427186833 wait complete. 2024-05-27T09:50:39.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.190+0000 7f688e17fb80 20 client.28285 trim_cache size 0 max 16384 2024-05-27T09:50:39.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.213+0000 7f688e17fb80 1 Processor -- start 2024-05-27T09:50:39.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.213+0000 7f688e17fb80 1 -- start start 2024-05-27T09:50:39.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.213+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3eb9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.213+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3e10a70 0x5587e3eaf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.213+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3ea3ef0 0x5587e3eaf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.213+0000 7f688e17fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f68781b7db0 con 0x5587e3e10a70 2024-05-27T09:50:39.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.213+0000 7f688e17fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f68781b55e0 con 0x5587e3d24ec0 2024-05-27T09:50:39.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.213+0000 7f688e17fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f68781b3450 con 0x5587e3ea3ef0 2024-05-27T09:50:39.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.214+0000 7f686fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3e10a70 0x5587e3eaf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.214+0000 7f687d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3ea3ef0 0x5587e3eaf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.214+0000 7f686fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3e10a70 0x5587e3eaf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45004/0 (socket says 172.21.15.44:45004) 2024-05-27T09:50:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.214+0000 7f686fe00700 1 -- 172.21.15.44:0/3623384855 learned_addr learned my addr 172.21.15.44:0/3623384855 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.214+0000 7f687cc00700 1 --2- 172.21.15.44:0/3623384855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3eb9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.214+0000 7f687d600700 1 -- 172.21.15.44:0/3623384855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 msgr2=0x5587e3eb9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.214+0000 7f687d600700 1 --2- 172.21.15.44:0/3623384855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3eb9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.214+0000 7f687d600700 1 -- 172.21.15.44:0/3623384855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3e10a70 msgr2=0x5587e3eaf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:39.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.214+0000 7f687d600700 1 --2- 172.21.15.44:0/3623384855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3e10a70 0x5587e3eaf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.214+0000 7f687d600700 1 -- 172.21.15.44:0/3623384855 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587e3e9b2c0 con 0x5587e3ea3ef0 2024-05-27T09:50:39.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.214+0000 7f686fe00700 1 --2- 172.21.15.44:0/3623384855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3e10a70 0x5587e3eaf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:39.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.215+0000 7f687d600700 1 --2- 172.21.15.44:0/3623384855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3ea3ef0 0x5587e3eaf830 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6870008fd0 tx=0x7f6870009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=952c8148e5081ea9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:39.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.215+0000 7f686d600700 1 -- 172.21.15.44:0/3623384855 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870029030 con 0x5587e3ea3ef0 2024-05-27T09:50:39.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.215+0000 7f686d600700 1 -- 172.21.15.44:0/3623384855 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6870004bb0 con 0x5587e3ea3ef0 2024-05-27T09:50:39.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.215+0000 7f686d600700 1 -- 172.21.15.44:0/3623384855 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870007500 con 0x5587e3ea3ef0 2024-05-27T09:50:39.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.216+0000 7f688e17fb80 1 -- 172.21.15.44:0/3623384855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3ea3ef0 msgr2=0x5587e3eaf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:39.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.216+0000 7f688e17fb80 1 --2- 172.21.15.44:0/3623384855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3ea3ef0 0x5587e3eaf830 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6870008fd0 tx=0x7f6870009920 comp rx=0 tx=0).stop 2024-05-27T09:50:39.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.216+0000 7f688e17fb80 1 -- 172.21.15.44:0/3623384855 shutdown_connections 2024-05-27T09:50:39.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.216+0000 7f688e17fb80 1 --2- 172.21.15.44:0/3623384855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3ea3ef0 0x5587e3eaf830 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.216+0000 7f688e17fb80 1 --2- 172.21.15.44:0/3623384855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3eb9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.216+0000 7f688e17fb80 1 --2- 172.21.15.44:0/3623384855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3e10a70 0x5587e3eaf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.216+0000 7f688e17fb80 1 -- 172.21.15.44:0/3623384855 >> 172.21.15.44:0/3623384855 conn(0x5587e3dbe410 msgr2=0x5587e3eaebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:39.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.216+0000 7f688e17fb80 1 -- 172.21.15.44:0/3623384855 shutdown_connections 2024-05-27T09:50:39.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.216+0000 7f688e17fb80 1 -- 172.21.15.44:0/3623384855 wait complete. 2024-05-27T09:50:39.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.217+0000 7f688e17fb80 1 -- start start 2024-05-27T09:50:39.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.217+0000 7f688e17fb80 1 Processor -- start 2024-05-27T09:50:39.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3f20980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 0x5587e3f20ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f688e17fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eb24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f688e17fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f68781b3aa0 con 0x5587e3ea3ef0 2024-05-27T09:50:39.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f688e17fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f68781b4f10 con 0x5587e3d24ec0 2024-05-27T09:50:39.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f688e17fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f68781b4170 con 0x5587e3e10a70 2024-05-27T09:50:39.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f688e17fb80 10 client.0 _ll_register_callbacks cb 0x5587e3d6bcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:39.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f687d600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eb24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f686fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 0x5587e3f20ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f687d600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eb24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45012/0 (socket says 172.21.15.44:45012) 2024-05-27T09:50:39.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f687d600700 1 -- 172.21.15.44:0/1046297437 learned_addr learned my addr 172.21.15.44:0/1046297437 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:39.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.218+0000 7f687cc00700 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3f20980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f687cc00700 1 -- 172.21.15.44:0/1046297437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 msgr2=0x5587e3f20ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:39.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f687cc00700 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 0x5587e3f20ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f687cc00700 1 -- 172.21.15.44:0/1046297437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 msgr2=0x5587e3eb24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:39.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f687cc00700 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eb24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f687cc00700 1 -- 172.21.15.44:0/1046297437 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587e3eb2b50 con 0x5587e3d24ec0 2024-05-27T09:50:39.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f687d600700 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eb24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:39.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f686fe00700 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 0x5587e3f20ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f687cc00700 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3f20980 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6868000c00 tx=0x7f68680058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2b028ae8bc0768b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f6866a00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f6866a00700 1 -- 172.21.15.44:0/1046297437 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6868003b40 con 0x5587e3d24ec0 2024-05-27T09:50:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f688e17fb80 1 -- 172.21.15.44:0/1046297437 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6870004bb0 con 0x5587e3d24ec0 2024-05-27T09:50:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.219+0000 7f6866a00700 1 -- 172.21.15.44:0/1046297437 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f68680050b0 con 0x5587e3d24ec0 2024-05-27T09:50:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.220+0000 7f6866a00700 1 -- 172.21.15.44:0/1046297437 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f686803b050 con 0x5587e3d24ec0 2024-05-27T09:50:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.220+0000 7f688e17fb80 10 client.28283 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:39.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.220+0000 7f688e17fb80 1 -- 172.21.15.44:0/1046297437 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5587e3f29100 con 0x5587e3d24ec0 2024-05-27T09:50:39.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.220+0000 7f6866a00700 1 -- 172.21.15.44:0/1046297437 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6870004bb0 con 0x5587e3d24ec0 2024-05-27T09:50:39.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.220+0000 7f6866a00700 1 -- 172.21.15.44:0/1046297437 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f686802b970 con 0x5587e3d24ec0 2024-05-27T09:50:39.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.220+0000 7f688e17fb80 10 client.28283 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:39.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.220+0000 7f688e17fb80 10 client.28283 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:39.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.220+0000 7f688e17fb80 20 client.28283 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:39.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.220+0000 7f688e17fb80 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5587e3f29650 0x5587e3f2ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:39.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.221+0000 7f688e17fb80 4 client.28283 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T09:50:39.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.221+0000 7f688e17fb80 1 -- 172.21.15.44:0/1046297437 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5587e3dcd5f0 con 0x5587e3f29650 2024-05-27T09:50:39.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.221+0000 7f686fe00700 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5587e3f29650 0x5587e3f2ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:39.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.221+0000 7f686fe00700 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5587e3f29650 0x5587e3f2ba70 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc35d13fb1a5aedf server_cookie=4fb4393843649e6 in_seq=0 out_seq=0 2024-05-27T09:50:39.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.221+0000 7f6866a00700 10 client.28283 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:39.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f6866a00700 1 -- 172.21.15.44:0/1046297437 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5587e3dcd5f0 con 0x5587e3f29650 2024-05-27T09:50:39.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f6866a00700 10 client.28283 handle_command_reply: tid=0 2024-05-27T09:50:39.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f688e17fb80 1 client.28283 shutdown 2024-05-27T09:50:39.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f688e17fb80 1 -- 172.21.15.44:0/1046297437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 msgr2=0x5587e3f20980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:39.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f688e17fb80 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3f20980 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6868000c00 tx=0x7f68680058e0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f688e17fb80 1 -- 172.21.15.44:0/1046297437 shutdown_connections 2024-05-27T09:50:39.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f688e17fb80 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5587e3f29650 0x5587e3f2ba70 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f688e17fb80 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5587e3ea3ef0 0x5587e3eb24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f688e17fb80 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5587e3d24ec0 0x5587e3f20980 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f688e17fb80 1 --2- 172.21.15.44:0/1046297437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587e3e10a70 0x5587e3f20ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f6866a00700 0 client.28283 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:39.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f688e17fb80 1 -- 172.21.15.44:0/1046297437 >> 172.21.15.44:0/1046297437 conn(0x5587e3dbe410 msgr2=0x5587e3eac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:39.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f688e17fb80 1 -- 172.21.15.44:0/1046297437 shutdown_connections 2024-05-27T09:50:39.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.223+0000 7f688e17fb80 1 -- 172.21.15.44:0/1046297437 wait complete. 2024-05-27T09:50:39.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.224+0000 7f688e17fb80 20 client.28283 trim_cache size 0 max 16384 2024-05-27T09:50:39.233 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:50:39.233 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-05-27T09:50:39.233 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "2ecde5bb-e9d0-4093-ade8-eba70383a4f1", 2024-05-27T09:50:39.233 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-05-27T09:50:39.233 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:50:39.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.224+0000 7f687f600700 1 -- 172.21.15.44:0/1225507703 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68580b2a70 msgr2=0x7f68580b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:39.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.224+0000 7f687f600700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68580b2a70 0x7f68580b4f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6870009d30 tx=0x7f687000b040 comp rx=0 tx=0).stop 2024-05-27T09:50:39.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.224+0000 7f687f600700 1 -- 172.21.15.44:0/1225507703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878103bb0 msgr2=0x7f68781a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:39.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.224+0000 7f687f600700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878103bb0 0x7f68781a19c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6860006010 tx=0x7f68600057c0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.225+0000 7f687f600700 1 -- 172.21.15.44:0/1225507703 shutdown_connections 2024-05-27T09:50:39.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.225+0000 7f687f600700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f68580b2a70 0x7f68580b4f80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.225+0000 7f687f600700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68781086b0 0x7f68781a5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.225+0000 7f687f600700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6878103260 0x7f68781a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.225+0000 7f687f600700 1 --2- 172.21.15.44:0/1225507703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878103bb0 0x7f68781a19c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:39.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.225+0000 7f687f600700 1 -- 172.21.15.44:0/1225507703 >> 172.21.15.44:0/1225507703 conn(0x7f68780fef20 msgr2=0x7f6878106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:39.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.225+0000 7f687f600700 1 -- 172.21.15.44:0/1225507703 shutdown_connections 2024-05-27T09:50:39.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:39.225+0000 7f687f600700 1 -- 172.21.15.44:0/1225507703 wait complete. 2024-05-27T09:50:40.088 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T09:50:40.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:40 smithi050 ceph-mon[36371]: from='client.17952 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:50:40.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:40 smithi050 ceph-mon[36371]: from='client.17958 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:50:40.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:40 smithi050 ceph-mon[36371]: pgmap v486: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-05-27T09:50:40.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:40 smithi050 ceph-mon[36371]: scrub queued for path: / 2024-05-27T09:50:40.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:40 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:40.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:40 smithi050 ceph-mon[36371]: scrub summary: active paths [/] 2024-05-27T09:50:40.295 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:40 smithi050 ceph-mon[36371]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:40.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:40 smithi189 ceph-mon[36671]: from='client.17952 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:50:40.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:40 smithi189 ceph-mon[36671]: from='client.17958 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:50:40.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:40 smithi189 ceph-mon[36671]: pgmap v486: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-05-27T09:50:40.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:40 smithi189 ceph-mon[36671]: scrub queued for path: / 2024-05-27T09:50:40.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:40 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:40.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:40 smithi189 ceph-mon[36671]: scrub summary: active paths [/] 2024-05-27T09:50:40.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:40 smithi189 ceph-mon[36671]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:40 smithi044 ceph-mon[31618]: from='client.17952 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:50:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:40 smithi044 ceph-mon[31618]: from='client.17958 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T09:50:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:40 smithi044 ceph-mon[31618]: pgmap v486: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-05-27T09:50:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:40 smithi044 ceph-mon[31618]: scrub queued for path: / 2024-05-27T09:50:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:40 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:40 smithi044 ceph-mon[31618]: scrub summary: active paths [/] 2024-05-27T09:50:40.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:40 smithi044 ceph-mon[31618]: scrub summary: idle+waiting paths [/] 2024-05-27T09:50:40.574 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:42.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:42 smithi044 ceph-mon[31618]: pgmap v487: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T09:50:42.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:42 smithi050 ceph-mon[36371]: pgmap v487: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T09:50:42.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:42 smithi189 ceph-mon[36671]: pgmap v487: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T09:50:42.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.871+0000 7f0a42a00700 1 -- 172.21.15.44:0/2117256609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a3c108ef0 msgr2=0x7f0a3c109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:42.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.871+0000 7f0a42a00700 1 --2- 172.21.15.44:0/2117256609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a3c108ef0 0x7f0a3c109300 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009990 tx=0x7f0a24009ca0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.872+0000 7f0a42a00700 1 -- 172.21.15.44:0/2117256609 shutdown_connections 2024-05-27T09:50:42.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.872+0000 7f0a42a00700 1 --2- 172.21.15.44:0/2117256609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a3c109840 0x7f0a3c1111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.872+0000 7f0a42a00700 1 --2- 172.21.15.44:0/2117256609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c1045c0 0x7f0a3c1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.872+0000 7f0a42a00700 1 --2- 172.21.15.44:0/2117256609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a3c108ef0 0x7f0a3c109300 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.872+0000 7f0a42a00700 1 -- 172.21.15.44:0/2117256609 >> 172.21.15.44:0/2117256609 conn(0x7f0a3c0ffe90 msgr2=0x7f0a3c1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:42.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.872+0000 7f0a42a00700 1 -- 172.21.15.44:0/2117256609 shutdown_connections 2024-05-27T09:50:42.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.872+0000 7f0a42a00700 1 -- 172.21.15.44:0/2117256609 wait complete. 2024-05-27T09:50:42.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.873+0000 7f0a42a00700 1 Processor -- start 2024-05-27T09:50:42.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.873+0000 7f0a42a00700 1 -- start start 2024-05-27T09:50:42.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a42a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a3c1045c0 0x7f0a3c19add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a42a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a3c108ef0 0x7f0a3c19b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a42a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c109840 0x7f0a3c19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a42a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c1144a0 con 0x7f0a3c109840 2024-05-27T09:50:42.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a42a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c114320 con 0x7f0a3c108ef0 2024-05-27T09:50:42.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a42a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c114620 con 0x7f0a3c1045c0 2024-05-27T09:50:42.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a3be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a3c1045c0 0x7f0a3c19add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a33400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a3c108ef0 0x7f0a3c19b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a40a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c109840 0x7f0a3c19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a3be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a3c1045c0 0x7f0a3c19add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60352/0 (socket says 172.21.15.44:60352) 2024-05-27T09:50:42.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a33400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a3c108ef0 0x7f0a3c19b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:41784/0 (socket says 172.21.15.44:41784) 2024-05-27T09:50:42.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.874+0000 7f0a3be00700 1 -- 172.21.15.44:0/4131267269 learned_addr learned my addr 172.21.15.44:0/4131267269 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:42.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.875+0000 7f0a33400700 1 -- 172.21.15.44:0/4131267269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a3c1045c0 msgr2=0x7f0a3c19add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:42.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.875+0000 7f0a33400700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a3c1045c0 0x7f0a3c19add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.875+0000 7f0a33400700 1 -- 172.21.15.44:0/4131267269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c109840 msgr2=0x7f0a3c19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:42.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.875+0000 7f0a33400700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c109840 0x7f0a3c19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.875+0000 7f0a33400700 1 -- 172.21.15.44:0/4131267269 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3c19fca0 con 0x7f0a3c108ef0 2024-05-27T09:50:42.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.875+0000 7f0a3be00700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a3c1045c0 0x7f0a3c19add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:42.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.875+0000 7f0a33400700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a3c108ef0 0x7f0a3c19b370 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0a240098e0 tx=0x7f0a240057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:42.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.875+0000 7f0a39600700 1 -- 172.21.15.44:0/4131267269 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24027070 con 0x7f0a3c108ef0 2024-05-27T09:50:42.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.875+0000 7f0a39600700 1 -- 172.21.15.44:0/4131267269 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a2400b610 con 0x7f0a3c108ef0 2024-05-27T09:50:42.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.875+0000 7f0a39600700 1 -- 172.21.15.44:0/4131267269 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a240173f0 con 0x7f0a3c108ef0 2024-05-27T09:50:42.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.875+0000 7f0a42a00700 1 -- 172.21.15.44:0/4131267269 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3c19ff30 con 0x7f0a3c108ef0 2024-05-27T09:50:42.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.876+0000 7f0a42a00700 1 -- 172.21.15.44:0/4131267269 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a3c1a03c0 con 0x7f0a3c108ef0 2024-05-27T09:50:42.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.881+0000 7f0a39600700 1 -- 172.21.15.44:0/4131267269 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0a2402a070 con 0x7f0a3c108ef0 2024-05-27T09:50:42.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.882+0000 7f0a514b3b80 1 Processor -- start 2024-05-27T09:50:42.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.882+0000 7f0a514b3b80 1 -- start start 2024-05-27T09:50:42.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a39600700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a200b29f0 0x7f0a200b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c055a9f0 0x5627c05ed330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05ed870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c05ede70 0x5627c05fa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5627c051be40 con 0x5627c046ee40 2024-05-27T09:50:42.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5627c050b9e0 con 0x5627c05ede70 2024-05-27T09:50:42.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5627c050fa20 con 0x5627c055a9f0 2024-05-27T09:50:42.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a39600700 1 -- 172.21.15.44:0/4131267269 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f0a240e35e0 con 0x7f0a3c108ef0 2024-05-27T09:50:42.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a40a00700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a200b29f0 0x7f0a200b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a33400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05ed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a33400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05ed870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44058/0 (socket says 172.21.15.44:44058) 2024-05-27T09:50:42.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.883+0000 7f0a33400700 1 -- 172.21.15.44:0/1353475059 learned_addr learned my addr 172.21.15.44:0/1353475059 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:42.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.884+0000 7f0a3be00700 1 --2- 172.21.15.44:0/1353475059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c055a9f0 0x5627c05ed330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.884+0000 7f0a40a00700 1 --2- 172.21.15.44:0/1353475059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c05ede70 0x5627c05fa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.884+0000 7f0a33400700 1 -- 172.21.15.44:0/1353475059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c055a9f0 msgr2=0x5627c05ed330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:42.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.884+0000 7f0a33400700 1 --2- 172.21.15.44:0/1353475059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c055a9f0 0x5627c05ed330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.884+0000 7f0a33400700 1 -- 172.21.15.44:0/1353475059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c05ede70 msgr2=0x5627c05fa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:42.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.884+0000 7f0a33400700 1 --2- 172.21.15.44:0/1353475059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c05ede70 0x5627c05fa110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.884+0000 7f0a33400700 1 -- 172.21.15.44:0/1353475059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627c04d4420 con 0x5627c046ee40 2024-05-27T09:50:42.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.884+0000 7f0a40a00700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a200b29f0 0x7f0a200b4f00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0a340097a0 tx=0x7f0a3401c880 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:42.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.884+0000 7f0a33400700 1 --2- 172.21.15.44:0/1353475059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05ed870 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0a240aa5a0 tx=0x7f0a240aa6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e201a2f52ae033c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:42.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.885+0000 7f0a31600700 1 -- 172.21.15.44:0/1353475059 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a240aa010 con 0x5627c046ee40 2024-05-27T09:50:42.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.885+0000 7f0a31600700 1 -- 172.21.15.44:0/1353475059 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a240aa3b0 con 0x5627c046ee40 2024-05-27T09:50:42.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.885+0000 7f0a31600700 1 -- 172.21.15.44:0/1353475059 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a240831a0 con 0x5627c046ee40 2024-05-27T09:50:42.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.886+0000 7f0a514b3b80 1 -- 172.21.15.44:0/1353475059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 msgr2=0x5627c05ed870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:42.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.886+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/1353475059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05ed870 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0a240aa5a0 tx=0x7f0a240aa6f0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.886+0000 7f0a514b3b80 1 -- 172.21.15.44:0/1353475059 shutdown_connections 2024-05-27T09:50:42.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.886+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/1353475059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c05ede70 0x5627c05fa110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.886+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/1353475059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c055a9f0 0x5627c05ed330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.886+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/1353475059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05ed870 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.886+0000 7f0a514b3b80 1 -- 172.21.15.44:0/1353475059 >> 172.21.15.44:0/1353475059 conn(0x5627c0508390 msgr2=0x5627c05e94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:42.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.886+0000 7f0a514b3b80 1 -- 172.21.15.44:0/1353475059 shutdown_connections 2024-05-27T09:50:42.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.886+0000 7f0a514b3b80 1 -- 172.21.15.44:0/1353475059 wait complete. 2024-05-27T09:50:42.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.886+0000 7f0a514b3b80 1 -- start start 2024-05-27T09:50:42.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.887+0000 7f0a514b3b80 1 Processor -- start 2024-05-27T09:50:42.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.887+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c046ee40 0x5627c05f7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.887+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c055a9f0 0x5627c05f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.887+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c05f8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.887+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5627c0411fa0 con 0x5627c055a9f0 2024-05-27T09:50:42.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.887+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5627c04ac760 con 0x5627c046ee40 2024-05-27T09:50:42.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.887+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5627c05fa940 con 0x5627c05ede70 2024-05-27T09:50:42.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a514b3b80 10 client.0 _ll_register_callbacks cb 0x5627c04b5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:42.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a33400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c055a9f0 0x5627c05f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a3be00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c046ee40 0x5627c05f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a33400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c055a9f0 0x5627c05f7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44064/0 (socket says 172.21.15.44:44064) 2024-05-27T09:50:42.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a33400700 1 -- 172.21.15.44:0/3202759335 learned_addr learned my addr 172.21.15.44:0/3202759335 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:42.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a40a00700 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c05f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a3be00700 1 -- 172.21.15.44:0/3202759335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 msgr2=0x5627c05f8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:42.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a3be00700 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c05f8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a3be00700 1 -- 172.21.15.44:0/3202759335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c055a9f0 msgr2=0x5627c05f7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:42.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a3be00700 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c055a9f0 0x5627c05f7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a3be00700 1 -- 172.21.15.44:0/3202759335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627c045b8d0 con 0x5627c046ee40 2024-05-27T09:50:42.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a40a00700 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c05f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.888+0000 7f0a33400700 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c055a9f0 0x5627c05f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.889+0000 7f0a3be00700 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c046ee40 0x5627c05f7580 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009c10 tx=0x7f0a2c007b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1b0e107004af831 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.889+0000 7f0a2a000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.889+0000 7f0a2a000700 1 -- 172.21.15.44:0/3202759335 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c01a070 con 0x5627c046ee40 2024-05-27T09:50:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.889+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3202759335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a240aa3b0 con 0x5627c046ee40 2024-05-27T09:50:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.889+0000 7f0a2a000700 1 -- 172.21.15.44:0/3202759335 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f0a2c004510 con 0x5627c046ee40 2024-05-27T09:50:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.890+0000 7f0a2a000700 1 -- 172.21.15.44:0/3202759335 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0a2c004710 con 0x5627c046ee40 2024-05-27T09:50:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.890+0000 7f0a514b3b80 10 client.28295 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:42.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.890+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3202759335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5627c04dc060 con 0x5627c046ee40 2024-05-27T09:50:42.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.890+0000 7f0a2a000700 1 -- 172.21.15.44:0/3202759335 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a240aa3b0 con 0x5627c046ee40 2024-05-27T09:50:42.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.890+0000 7f0a2a000700 1 -- 172.21.15.44:0/3202759335 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f0a2c01e520 con 0x5627c046ee40 2024-05-27T09:50:42.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.890+0000 7f0a514b3b80 10 client.28295 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:42.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.890+0000 7f0a514b3b80 10 client.28295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:42.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.890+0000 7f0a514b3b80 20 client.28295 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:42.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.890+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5627c0607780 0x5627c0609ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.890+0000 7f0a514b3b80 4 client.28295 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:42.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.890+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3202759335 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627c051dd90 con 0x5627c0607780 2024-05-27T09:50:42.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.891+0000 7f0a33400700 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5627c0607780 0x5627c0609ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.891+0000 7f0a33400700 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5627c0607780 0x5627c0609ba0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f6c1ab5e6ed49ae server_cookie=9cf35112e0d49507 in_seq=0 out_seq=0 2024-05-27T09:50:42.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.891+0000 7f0a2a000700 10 client.28295 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:42.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.892+0000 7f0a2a000700 1 -- 172.21.15.44:0/3202759335 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5627c051dd90 con 0x5627c0607780 2024-05-27T09:50:42.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.892+0000 7f0a2a000700 10 client.28295 handle_command_reply: tid=0 2024-05-27T09:50:42.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.892+0000 7f0a514b3b80 1 client.28295 shutdown 2024-05-27T09:50:42.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.892+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3202759335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c046ee40 msgr2=0x5627c05f7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:42.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.892+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c046ee40 0x5627c05f7580 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009c10 tx=0x7f0a2c007b40 comp rx=0 tx=0).stop 2024-05-27T09:50:42.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.892+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3202759335 shutdown_connections 2024-05-27T09:50:42.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.892+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5627c0607780 0x5627c0609ba0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.892+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c05f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.892+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c046ee40 0x5627c05f7580 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.892+0000 7f0a2a000700 0 client.28295 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:42.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.892+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3202759335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c055a9f0 0x5627c05f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.893+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3202759335 >> 172.21.15.44:0/3202759335 conn(0x5627c0508390 msgr2=0x5627c05ed310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:42.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.893+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3202759335 shutdown_connections 2024-05-27T09:50:42.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.893+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3202759335 wait complete. 2024-05-27T09:50:42.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.893+0000 7f0a514b3b80 20 client.28295 trim_cache size 0 max 16384 2024-05-27T09:50:42.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.915+0000 7f0a514b3b80 1 Processor -- start 2024-05-27T09:50:42.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.915+0000 7f0a514b3b80 1 -- start start 2024-05-27T09:50:42.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.916+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.916+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c055a9f0 0x5627c05f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.916+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c06039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.916+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c1af530 con 0x5627c046ee40 2024-05-27T09:50:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.916+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c1acd60 con 0x5627c055a9f0 2024-05-27T09:50:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.916+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c1aabd0 con 0x5627c05ede70 2024-05-27T09:50:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.916+0000 7f0a3be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.916+0000 7f0a3be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05f2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44078/0 (socket says 172.21.15.44:44078) 2024-05-27T09:50:42.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.916+0000 7f0a3be00700 1 -- 172.21.15.44:0/1459799563 learned_addr learned my addr 172.21.15.44:0/1459799563 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:42.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.916+0000 7f0a33400700 1 --2- 172.21.15.44:0/1459799563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c055a9f0 0x5627c05f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.916+0000 7f0a40a00700 1 --2- 172.21.15.44:0/1459799563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c06039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.917+0000 7f0a3be00700 1 -- 172.21.15.44:0/1459799563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 msgr2=0x5627c06039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:42.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.917+0000 7f0a3be00700 1 --2- 172.21.15.44:0/1459799563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c06039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.917+0000 7f0a3be00700 1 -- 172.21.15.44:0/1459799563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c055a9f0 msgr2=0x5627c05f15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:42.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.917+0000 7f0a3be00700 1 --2- 172.21.15.44:0/1459799563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c055a9f0 0x5627c05f15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.917+0000 7f0a3be00700 1 -- 172.21.15.44:0/1459799563 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627c0605250 con 0x5627c046ee40 2024-05-27T09:50:42.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.917+0000 7f0a33400700 1 --2- 172.21.15.44:0/1459799563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c055a9f0 0x5627c05f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:42.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.917+0000 7f0a3be00700 1 --2- 172.21.15.44:0/1459799563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05f2df0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0094d0 tx=0x7f0a2c01ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a222ae269d5299 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:42.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.917+0000 7f0a30c00700 1 -- 172.21.15.44:0/1459799563 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c033070 con 0x5627c046ee40 2024-05-27T09:50:42.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.917+0000 7f0a30c00700 1 -- 172.21.15.44:0/1459799563 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a2c0048b0 con 0x5627c046ee40 2024-05-27T09:50:42.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.918+0000 7f0a30c00700 1 -- 172.21.15.44:0/1459799563 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c01a070 con 0x5627c046ee40 2024-05-27T09:50:42.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.918+0000 7f0a514b3b80 1 -- 172.21.15.44:0/1459799563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 msgr2=0x5627c05f2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:42.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.918+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/1459799563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05f2df0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0094d0 tx=0x7f0a2c01ed90 comp rx=0 tx=0).stop 2024-05-27T09:50:42.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.918+0000 7f0a514b3b80 1 -- 172.21.15.44:0/1459799563 shutdown_connections 2024-05-27T09:50:42.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.918+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/1459799563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c06039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.918+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/1459799563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c05f2df0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.918+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/1459799563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c055a9f0 0x5627c05f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.918+0000 7f0a514b3b80 1 -- 172.21.15.44:0/1459799563 >> 172.21.15.44:0/1459799563 conn(0x5627c0508390 msgr2=0x5627c05f0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:42.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.919+0000 7f0a514b3b80 1 -- 172.21.15.44:0/1459799563 shutdown_connections 2024-05-27T09:50:42.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.919+0000 7f0a514b3b80 1 -- 172.21.15.44:0/1459799563 wait complete. 2024-05-27T09:50:42.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.919+0000 7f0a514b3b80 1 -- start start 2024-05-27T09:50:42.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.919+0000 7f0a514b3b80 1 Processor -- start 2024-05-27T09:50:42.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c06000d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c055a9f0 0x5627c05fb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a514b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c05fb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c1ab220 con 0x5627c046ee40 2024-05-27T09:50:42.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c1ac690 con 0x5627c055a9f0 2024-05-27T09:50:42.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a514b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c1ab8f0 con 0x5627c05ede70 2024-05-27T09:50:42.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a514b3b80 10 client.0 _ll_register_callbacks cb 0x5627c04b5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:42.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a3be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c06000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a40a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c05fb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a40a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c05fb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60388/0 (socket says 172.21.15.44:60388) 2024-05-27T09:50:42.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a40a00700 1 -- 172.21.15.44:0/3190603043 learned_addr learned my addr 172.21.15.44:0/3190603043 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:42.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.920+0000 7f0a33400700 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c055a9f0 0x5627c05fb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.921+0000 7f0a40a00700 1 -- 172.21.15.44:0/3190603043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c055a9f0 msgr2=0x5627c05fb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:42.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.921+0000 7f0a40a00700 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c055a9f0 0x5627c05fb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.921+0000 7f0a40a00700 1 -- 172.21.15.44:0/3190603043 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 msgr2=0x5627c06000d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.921+0000 7f0a40a00700 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c06000d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.921+0000 7f0a40a00700 1 -- 172.21.15.44:0/3190603043 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627c05fbf30 con 0x5627c05ede70 2024-05-27T09:50:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.921+0000 7f0a40a00700 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c05fb970 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0a34004350 tx=0x7f0a340047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4a40276a4b1341 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.921+0000 7f0a2a000700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:50:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.921+0000 7f0a2a000700 1 -- 172.21.15.44:0/3190603043 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3402c030 con 0x5627c05ede70 2024-05-27T09:50:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.921+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3190603043 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a2c0048b0 con 0x5627c05ede70 2024-05-27T09:50:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.921+0000 7f0a2a000700 1 -- 172.21.15.44:0/3190603043 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f0a34009040 con 0x5627c05ede70 2024-05-27T09:50:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.922+0000 7f0a2a000700 1 -- 172.21.15.44:0/3190603043 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0a34038050 con 0x5627c05ede70 2024-05-27T09:50:42.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.922+0000 7f0a514b3b80 10 client.28309 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:42.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.922+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3190603043 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5627c0671fd0 con 0x5627c05ede70 2024-05-27T09:50:42.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.922+0000 7f0a2a000700 1 -- 172.21.15.44:0/3190603043 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a2c0048b0 con 0x5627c05ede70 2024-05-27T09:50:42.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.922+0000 7f0a2a000700 1 -- 172.21.15.44:0/3190603043 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f0a34025070 con 0x5627c05ede70 2024-05-27T09:50:42.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.922+0000 7f0a514b3b80 10 client.28309 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:42.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.922+0000 7f0a514b3b80 10 client.28309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:42.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.922+0000 7f0a514b3b80 20 client.28309 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:42.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.923+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5627c06726c0 0x5627c0674ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:42.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.923+0000 7f0a514b3b80 4 client.28309 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T09:50:42.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.923+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3190603043 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5627c0517570 con 0x5627c06726c0 2024-05-27T09:50:42.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.923+0000 7f0a3be00700 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5627c06726c0 0x5627c0674ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:42.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.923+0000 7f0a3be00700 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5627c06726c0 0x5627c0674ae0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fd0a12eb4ca6958 server_cookie=e2680eac3199fc17 in_seq=0 out_seq=0 2024-05-27T09:50:42.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.923+0000 7f0a2a000700 10 client.28309 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:42.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a2a000700 1 -- 172.21.15.44:0/3190603043 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5627c0517570 con 0x5627c06726c0 2024-05-27T09:50:42.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a2a000700 10 client.28309 handle_command_reply: tid=0 2024-05-27T09:50:42.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 1 client.28309 shutdown 2024-05-27T09:50:42.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3190603043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 msgr2=0x5627c05fb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:42.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c05fb970 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0a34004350 tx=0x7f0a340047c0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3190603043 shutdown_connections 2024-05-27T09:50:42.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x5627c06726c0 0x5627c0674ae0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627c05ede70 0x5627c05fb970 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5627c046ee40 0x5627c06000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 1 --2- 172.21.15.44:0/3190603043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5627c055a9f0 0x5627c05fb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3190603043 >> 172.21.15.44:0/3190603043 conn(0x5627c0508390 msgr2=0x5627c05f7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:42.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a2a000700 0 client.28309 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:42.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3190603043 shutdown_connections 2024-05-27T09:50:42.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 1 -- 172.21.15.44:0/3190603043 wait complete. 2024-05-27T09:50:42.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.924+0000 7f0a514b3b80 20 client.28309 trim_cache size 0 max 16384 2024-05-27T09:50:42.932 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:50:42.933 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-05-27T09:50:42.933 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-05-27T09:50:42.933 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:50:42.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 -- 172.21.15.44:0/4131267269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a200b29f0 msgr2=0x7f0a200b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:42.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a200b29f0 0x7f0a200b4f00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0a340097a0 tx=0x7f0a3401c880 comp rx=0 tx=0).stop 2024-05-27T09:50:42.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 -- 172.21.15.44:0/4131267269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a3c108ef0 msgr2=0x7f0a3c19b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:42.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a3c108ef0 0x7f0a3c19b370 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0a240098e0 tx=0x7f0a240057c0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 -- 172.21.15.44:0/4131267269 shutdown_connections 2024-05-27T09:50:42.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0a200b29f0 0x7f0a200b4f00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c109840 0x7f0a3c19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a3c1045c0 0x7f0a3c19add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 --2- 172.21.15.44:0/4131267269 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a3c108ef0 0x7f0a3c19b370 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:42.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 -- 172.21.15.44:0/4131267269 >> 172.21.15.44:0/4131267269 conn(0x7f0a3c0ffe90 msgr2=0x7f0a3c10a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:42.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 -- 172.21.15.44:0/4131267269 shutdown_connections 2024-05-27T09:50:42.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:42.925+0000 7f0a42a00700 1 -- 172.21.15.44:0/4131267269 wait complete. 2024-05-27T09:50:43.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:43 smithi044 ceph-mon[31618]: scrub summary: idle 2024-05-27T09:50:43.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:43 smithi189 ceph-mon[36671]: scrub summary: idle 2024-05-27T09:50:43.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:43 smithi050 ceph-mon[36371]: scrub summary: idle 2024-05-27T09:50:43.899 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T09:50:43.899 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:50:44.142 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:44 smithi044 ceph-mon[31618]: pgmap v488: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T09:50:44.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:44 smithi189 ceph-mon[36671]: pgmap v488: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T09:50:44.380 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:44.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:44 smithi050 ceph-mon[36371]: pgmap v488: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T09:50:46.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:46 smithi189 ceph-mon[36671]: pgmap v489: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.0 KiB/s wr, 9 op/s 2024-05-27T09:50:46.367 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:46 smithi044 ceph-mon[31618]: pgmap v489: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.0 KiB/s wr, 9 op/s 2024-05-27T09:50:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:46 smithi050 ceph-mon[36371]: pgmap v489: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.0 KiB/s wr, 9 op/s 2024-05-27T09:50:46.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.614+0000 7fa666800700 1 -- 172.21.15.44:0/4093531803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa660106ee0 msgr2=0x7fa660113370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:46.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.614+0000 7fa666800700 1 --2- 172.21.15.44:0/4093531803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa660106ee0 0x7fa660113370 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa65800b0a0 tx=0x7fa65800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:50:46.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.614+0000 7fa666800700 1 -- 172.21.15.44:0/4093531803 shutdown_connections 2024-05-27T09:50:46.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.614+0000 7fa666800700 1 --2- 172.21.15.44:0/4093531803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa660106ee0 0x7fa660113370 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:46.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.614+0000 7fa666800700 1 --2- 172.21.15.44:0/4093531803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa66010c5d0 0x7fa66010c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:46.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.614+0000 7fa666800700 1 --2- 172.21.15.44:0/4093531803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa660106590 0x7fa6601069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.614+0000 7fa666800700 1 -- 172.21.15.44:0/4093531803 >> 172.21.15.44:0/4093531803 conn(0x7fa660102070 msgr2=0x7fa6601044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.614+0000 7fa666800700 1 -- 172.21.15.44:0/4093531803 shutdown_connections 2024-05-27T09:50:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.614+0000 7fa666800700 1 -- 172.21.15.44:0/4093531803 wait complete. 2024-05-27T09:50:46.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.615+0000 7fa666800700 1 Processor -- start 2024-05-27T09:50:46.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.615+0000 7fa666800700 1 -- start start 2024-05-27T09:50:46.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.616+0000 7fa666800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa660106590 0x7fa66019d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:46.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.616+0000 7fa666800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa660106ee0 0x7fa66019d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:46.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.616+0000 7fa666800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa66010c5d0 0x7fa6601a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:46.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.616+0000 7fa666800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa660116750 con 0x7fa66010c5d0 2024-05-27T09:50:46.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.616+0000 7fa666800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa6601165d0 con 0x7fa660106ee0 2024-05-27T09:50:46.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.616+0000 7fa666800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa6601168d0 con 0x7fa660106590 2024-05-27T09:50:46.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.616+0000 7fa65fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa66010c5d0 0x7fa6601a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:46.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.616+0000 7fa65fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa66010c5d0 0x7fa6601a1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44100/0 (socket says 172.21.15.44:44100) 2024-05-27T09:50:46.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.616+0000 7fa65f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa660106590 0x7fa66019d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:46.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.616+0000 7fa65fe00700 1 -- 172.21.15.44:0/4029941419 learned_addr learned my addr 172.21.15.44:0/4029941419 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:46.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.616+0000 7fa65ea00700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa660106ee0 0x7fa66019d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:46.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.617+0000 7fa65fe00700 1 -- 172.21.15.44:0/4029941419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa660106590 msgr2=0x7fa66019d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:46.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.617+0000 7fa65fe00700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa660106590 0x7fa66019d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:46.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.617+0000 7fa65fe00700 1 -- 172.21.15.44:0/4029941419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa660106ee0 msgr2=0x7fa66019d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:46.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.617+0000 7fa65fe00700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa660106ee0 0x7fa66019d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:46.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.617+0000 7fa65fe00700 1 -- 172.21.15.44:0/4029941419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6601a1ef0 con 0x7fa66010c5d0 2024-05-27T09:50:46.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.617+0000 7fa65ea00700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa660106ee0 0x7fa66019d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.617+0000 7fa65fe00700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa66010c5d0 0x7fa6601a1850 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa65800b070 tx=0x7fa658009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.618+0000 7fa657e00700 1 -- 172.21.15.44:0/4029941419 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa658021070 con 0x7fa66010c5d0 2024-05-27T09:50:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.618+0000 7fa657e00700 1 -- 172.21.15.44:0/4029941419 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa658004e40 con 0x7fa66010c5d0 2024-05-27T09:50:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.618+0000 7fa666800700 1 -- 172.21.15.44:0/4029941419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6601a2180 con 0x7fa66010c5d0 2024-05-27T09:50:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.618+0000 7fa657e00700 1 -- 172.21.15.44:0/4029941419 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa658004630 con 0x7fa66010c5d0 2024-05-27T09:50:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.618+0000 7fa666800700 1 -- 172.21.15.44:0/4029941419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6601a9b00 con 0x7fa66010c5d0 2024-05-27T09:50:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.619+0000 7fa666800700 1 -- 172.21.15.44:0/4029941419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa66010e990 con 0x7fa66010c5d0 2024-05-27T09:50:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.622+0000 7fa657e00700 1 -- 172.21.15.44:0/4029941419 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa658007980 con 0x7fa66010c5d0 2024-05-27T09:50:46.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.624+0000 7fa657e00700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa6440b29f0 0x7fa6440b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.625+0000 7fa657e00700 1 -- 172.21.15.44:0/4029941419 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7fa65800f020 con 0x7fa66010c5d0 2024-05-27T09:50:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.625+0000 7fa65f400700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa6440b29f0 0x7fa6440b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.625+0000 7fa65f400700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa6440b29f0 0x7fa6440b4f00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa660107a00 tx=0x7fa650009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:46.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.627+0000 7fa657e00700 1 -- 172.21.15.44:0/4029941419 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa65802a020 con 0x7fa66010c5d0 2024-05-27T09:50:46.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.919+0000 7fa666800700 1 -- 172.21.15.44:0/4029941419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa66019e380 con 0x7fa66010c5d0 2024-05-27T09:50:46.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.921+0000 7fa657e00700 1 -- 172.21.15.44:0/4029941419 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11870 (secure 0 0 0) 0x7fa65802e3e0 con 0x7fa66010c5d0 2024-05-27T09:50:46.923 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:50:46.923 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:48:16.864365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24520,24542,24503]},"id":1}]} 2024-05-27T09:50:46.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.924+0000 7fa666800700 1 -- 172.21.15.44:0/4029941419 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa6440b29f0 msgr2=0x7fa6440b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:46.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.924+0000 7fa666800700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa6440b29f0 0x7fa6440b4f00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa660107a00 tx=0x7fa650009260 comp rx=0 tx=0).stop 2024-05-27T09:50:46.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.924+0000 7fa666800700 1 -- 172.21.15.44:0/4029941419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa66010c5d0 msgr2=0x7fa6601a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:46.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.924+0000 7fa666800700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa66010c5d0 0x7fa6601a1850 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa65800b070 tx=0x7fa658009c90 comp rx=0 tx=0).stop 2024-05-27T09:50:46.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.924+0000 7fa666800700 1 -- 172.21.15.44:0/4029941419 shutdown_connections 2024-05-27T09:50:46.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.924+0000 7fa666800700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa6440b29f0 0x7fa6440b4f00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:46.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.924+0000 7fa666800700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa66010c5d0 0x7fa6601a1850 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:46.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.924+0000 7fa666800700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa660106590 0x7fa66019d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:46.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.924+0000 7fa666800700 1 --2- 172.21.15.44:0/4029941419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa660106ee0 0x7fa66019d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:46.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.924+0000 7fa666800700 1 -- 172.21.15.44:0/4029941419 >> 172.21.15.44:0/4029941419 conn(0x7fa660102070 msgr2=0x7fa660103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:46.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.925+0000 7fa666800700 1 -- 172.21.15.44:0/4029941419 shutdown_connections 2024-05-27T09:50:46.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:46.925+0000 7fa666800700 1 -- 172.21.15.44:0/4029941419 wait complete. 2024-05-27T09:50:46.928 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 43 2024-05-27T09:50:47.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:47 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/4029941419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:47.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:47 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/4029941419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:47.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:47 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/4029941419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:50:47.805 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T09:50:48.296 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:48.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:48 smithi189 ceph-mon[36671]: pgmap v490: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-05-27T09:50:48.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:48 smithi044 ceph-mon[31618]: pgmap v490: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-05-27T09:50:48.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:48 smithi050 ceph-mon[36371]: pgmap v490: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-05-27T09:50:50.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:50 smithi044 ceph-mon[31618]: pgmap v491: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-27T09:50:50.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:50 smithi189 ceph-mon[36671]: pgmap v491: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-27T09:50:50.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:50 smithi050 ceph-mon[36371]: pgmap v491: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-27T09:50:50.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.633+0000 7f751f000700 1 -- 172.21.15.44:0/3042038050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f751810a980 msgr2=0x7f751810ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:50.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.633+0000 7f751f000700 1 --2- 172.21.15.44:0/3042038050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f751810a980 0x7f751810ad90 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f75040098f0 tx=0x7f7504009c00 comp rx=0 tx=0).stop 2024-05-27T09:50:50.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.633+0000 7f751f000700 1 -- 172.21.15.44:0/3042038050 shutdown_connections 2024-05-27T09:50:50.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.633+0000 7f751f000700 1 --2- 172.21.15.44:0/3042038050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f751810b2d0 0x7f7518117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.633+0000 7f751f000700 1 --2- 172.21.15.44:0/3042038050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7518110960 0x7f7518110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.633+0000 7f751f000700 1 --2- 172.21.15.44:0/3042038050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f751810a980 0x7f751810ad90 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.633+0000 7f751f000700 1 -- 172.21.15.44:0/3042038050 >> 172.21.15.44:0/3042038050 conn(0x7f7518077c70 msgr2=0x7f75180780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.634+0000 7f751f000700 1 -- 172.21.15.44:0/3042038050 shutdown_connections 2024-05-27T09:50:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.634+0000 7f751f000700 1 -- 172.21.15.44:0/3042038050 wait complete. 2024-05-27T09:50:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.635+0000 7f751f000700 1 Processor -- start 2024-05-27T09:50:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.636+0000 7f751f000700 1 -- start start 2024-05-27T09:50:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.636+0000 7f751f000700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f751810a980 0x7f75181a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.636+0000 7f751f000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f751810b2d0 0x7f75181a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.636+0000 7f751f000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7518110960 0x7f75181a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.636+0000 7f751f000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f751811aad0 con 0x7f751810b2d0 2024-05-27T09:50:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.636+0000 7f751f000700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f751811a950 con 0x7f751810a980 2024-05-27T09:50:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.636+0000 7f751f000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f751811ac50 con 0x7f7518110960 2024-05-27T09:50:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.637+0000 7f7517e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f751810a980 0x7f75181a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.637+0000 7f7517e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f751810a980 0x7f75181a1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:41842/0 (socket says 172.21.15.44:41842) 2024-05-27T09:50:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.637+0000 7f7517400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f751810b2d0 0x7f75181a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.637+0000 7f7517e00700 1 -- 172.21.15.44:0/1884157725 learned_addr learned my addr 172.21.15.44:0/1884157725 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.637+0000 7f751d000700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7518110960 0x7f75181a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.637+0000 7f7517400700 1 -- 172.21.15.44:0/1884157725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7518110960 msgr2=0x7f75181a5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:50.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.637+0000 7f7517400700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7518110960 0x7f75181a5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.637+0000 7f7517400700 1 -- 172.21.15.44:0/1884157725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f751810a980 msgr2=0x7f75181a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:50.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.637+0000 7f7517400700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f751810a980 0x7f75181a1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.637+0000 7f7517400700 1 -- 172.21.15.44:0/1884157725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75181a63d0 con 0x7f751810b2d0 2024-05-27T09:50:50.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.637+0000 7f7517e00700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f751810a980 0x7f75181a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:50.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.638+0000 7f7517400700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f751810b2d0 0x7f75181a1aa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7504000c00 tx=0x7f750400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:50.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.638+0000 7f7514c00700 1 -- 172.21.15.44:0/1884157725 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75040051c0 con 0x7f751810b2d0 2024-05-27T09:50:50.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.638+0000 7f7514c00700 1 -- 172.21.15.44:0/1884157725 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f750400b560 con 0x7f751810b2d0 2024-05-27T09:50:50.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.638+0000 7f751f000700 1 -- 172.21.15.44:0/1884157725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75181a6660 con 0x7f751810b2d0 2024-05-27T09:50:50.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.638+0000 7f7514c00700 1 -- 172.21.15.44:0/1884157725 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75040039b0 con 0x7f751810b2d0 2024-05-27T09:50:50.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.638+0000 7f751f000700 1 -- 172.21.15.44:0/1884157725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75181a6b10 con 0x7f751810b2d0 2024-05-27T09:50:50.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.643+0000 7f752daa7b80 1 Processor -- start 2024-05-27T09:50:50.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.643+0000 7f7514c00700 1 -- 172.21.15.44:0/1884157725 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f750400b8c0 con 0x7f751810b2d0 2024-05-27T09:50:50.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.643+0000 7f752daa7b80 1 -- start start 2024-05-27T09:50:50.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.644+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be13ae40 0x5566be2bf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.644+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 0x5566be2b9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.644+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2ba1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.644+0000 7f752daa7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5566be1e7e40 con 0x5566be2b9db0 2024-05-27T09:50:50.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.644+0000 7f752daa7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5566be1d79e0 con 0x5566be13ae40 2024-05-27T09:50:50.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.644+0000 7f752daa7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5566be1dba20 con 0x5566be2bfd30 2024-05-27T09:50:50.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.644+0000 7f7517e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 0x5566be2b9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.644+0000 7f7517e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 0x5566be2b9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60412/0 (socket says 172.21.15.44:60412) 2024-05-27T09:50:50.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.644+0000 7f7517e00700 1 -- 172.21.15.44:0/2954148610 learned_addr learned my addr 172.21.15.44:0/2954148610 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.644+0000 7f7517400700 1 --2- 172.21.15.44:0/2954148610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2ba1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.644+0000 7f751d000700 1 --2- 172.21.15.44:0/2954148610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be13ae40 0x5566be2bf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f7517e00700 1 -- 172.21.15.44:0/2954148610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be13ae40 msgr2=0x5566be2bf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f7517e00700 1 --2- 172.21.15.44:0/2954148610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be13ae40 0x5566be2bf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f7517e00700 1 -- 172.21.15.44:0/2954148610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 msgr2=0x5566be2ba1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f7517e00700 1 --2- 172.21.15.44:0/2954148610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2ba1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f7517e00700 1 -- 172.21.15.44:0/2954148610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566be1a0420 con 0x5566be2bfd30 2024-05-27T09:50:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f7517400700 1 --2- 172.21.15.44:0/2954148610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2ba1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:50.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f7517e00700 1 --2- 172.21.15.44:0/2954148610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 0x5566be2b9870 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f75080115d0 tx=0x7f75080118e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b93ef0e2c17e3a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:50.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f7514c00700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74fc0b29f0 0x7f74fc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f750d600700 1 -- 172.21.15.44:0/2954148610 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750801e070 con 0x5566be2bfd30 2024-05-27T09:50:50.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f750d600700 1 -- 172.21.15.44:0/2954148610 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7508019ae0 con 0x5566be2bfd30 2024-05-27T09:50:50.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f750d600700 1 -- 172.21.15.44:0/2954148610 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750800ec30 con 0x5566be2bfd30 2024-05-27T09:50:50.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.645+0000 7f7514c00700 1 -- 172.21.15.44:0/1884157725 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f75040e3700 con 0x7f751810b2d0 2024-05-27T09:50:50.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.646+0000 7f751d000700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74fc0b29f0 0x7f74fc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.646+0000 7f752daa7b80 1 -- 172.21.15.44:0/2954148610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 msgr2=0x5566be2b9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:50.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.646+0000 7f751d000700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74fc0b29f0 0x7f74fc0b4f00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x5566be2bab30 tx=0x7f751000b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:50.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.646+0000 7f752daa7b80 1 --2- 172.21.15.44:0/2954148610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 0x5566be2b9870 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f75080115d0 tx=0x7f75080118e0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.647+0000 7f752daa7b80 1 -- 172.21.15.44:0/2954148610 shutdown_connections 2024-05-27T09:50:50.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.647+0000 7f752daa7b80 1 --2- 172.21.15.44:0/2954148610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2ba1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.647+0000 7f752daa7b80 1 --2- 172.21.15.44:0/2954148610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be13ae40 0x5566be2bf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.647+0000 7f752daa7b80 1 --2- 172.21.15.44:0/2954148610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 0x5566be2b9870 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.647+0000 7f752daa7b80 1 -- 172.21.15.44:0/2954148610 >> 172.21.15.44:0/2954148610 conn(0x5566be19afc0 msgr2=0x5566be2b48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.647+0000 7f752daa7b80 1 -- 172.21.15.44:0/2954148610 shutdown_connections 2024-05-27T09:50:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.647+0000 7f752daa7b80 1 -- 172.21.15.44:0/2954148610 wait complete. 2024-05-27T09:50:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.647+0000 7f752daa7b80 1 -- start start 2024-05-27T09:50:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.648+0000 7f752daa7b80 1 Processor -- start 2024-05-27T09:50:50.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.648+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be13ae40 0x5566be2c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.648+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2b9db0 0x5566be2c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.648+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be2bfd30 0x5566be2c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.648+0000 7f752daa7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5566be0ddfa0 con 0x5566be13ae40 2024-05-27T09:50:50.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.648+0000 7f752daa7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5566be178760 con 0x5566be2bfd30 2024-05-27T09:50:50.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.648+0000 7f752daa7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5566be2c6800 con 0x5566be2b9db0 2024-05-27T09:50:50.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.648+0000 7f752daa7b80 10 client.0 _ll_register_callbacks cb 0x5566be181c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:50.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.649+0000 7f7517e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be13ae40 0x5566be2c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.649+0000 7f7517e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be13ae40 0x5566be2c34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44148/0 (socket says 172.21.15.44:44148) 2024-05-27T09:50:50.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.649+0000 7f7517e00700 1 -- 172.21.15.44:0/3229640226 learned_addr learned my addr 172.21.15.44:0/3229640226 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:50.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.649+0000 7f751d000700 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be2bfd30 0x5566be2c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.649+0000 7f7517400700 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2b9db0 0x5566be2c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.649+0000 7f7517e00700 1 -- 172.21.15.44:0/3229640226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2b9db0 msgr2=0x5566be2c39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:50.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.649+0000 7f7517e00700 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2b9db0 0x5566be2c39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.649+0000 7f7517e00700 1 -- 172.21.15.44:0/3229640226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be2bfd30 msgr2=0x5566be2c3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:50.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.649+0000 7f7517e00700 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be2bfd30 0x5566be2c3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.649+0000 7f7517e00700 1 -- 172.21.15.44:0/3229640226 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566be1278d0 con 0x5566be13ae40 2024-05-27T09:50:50.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.650+0000 7f7517e00700 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be13ae40 0x5566be2c34b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f750800e730 tx=0x7f750800e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498960130de0d085 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:50.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.650+0000 7f7502000700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:50:50.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.650+0000 7f7502000700 1 -- 172.21.15.44:0/3229640226 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7508026020 con 0x5566be13ae40 2024-05-27T09:50:50.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.650+0000 7f7502000700 1 -- 172.21.15.44:0/3229640226 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f7508004030 con 0x5566be13ae40 2024-05-27T09:50:50.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.650+0000 7f752daa7b80 1 -- 172.21.15.44:0/3229640226 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7508019ae0 con 0x5566be13ae40 2024-05-27T09:50:50.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.651+0000 7f7502000700 1 -- 172.21.15.44:0/3229640226 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f750801e040 con 0x5566be13ae40 2024-05-27T09:50:50.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.651+0000 7f752daa7b80 10 client.18012 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:50.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.651+0000 7f752daa7b80 1 -- 172.21.15.44:0/3229640226 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5566be1a8060 con 0x5566be13ae40 2024-05-27T09:50:50.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.651+0000 7f7502000700 1 -- 172.21.15.44:0/3229640226 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7508019ae0 con 0x5566be13ae40 2024-05-27T09:50:50.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.651+0000 7f7502000700 1 -- 172.21.15.44:0/3229640226 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f7508005040 con 0x5566be13ae40 2024-05-27T09:50:50.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.651+0000 7f752daa7b80 10 client.18012 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:50.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.651+0000 7f752daa7b80 10 client.18012 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T09:50:50.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.651+0000 7f752daa7b80 20 client.18012 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:50.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.651+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x5566be2d3590 0x5566be2d59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.651+0000 7f752daa7b80 4 client.18012 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:50.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.651+0000 7f752daa7b80 1 -- 172.21.15.44:0/3229640226 --> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566be1e9d90 con 0x5566be2d3590 2024-05-27T09:50:50.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.652+0000 7f7517400700 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x5566be2d3590 0x5566be2d59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.652+0000 7f7517400700 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x5566be2d3590 0x5566be2d59b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7f9e8c8e85c3997d server_cookie=adf32bfdb7d02f65 in_seq=0 out_seq=0 2024-05-27T09:50:50.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.652+0000 7f7502000700 10 client.18012 ms_handle_connect on v2:172.21.15.50:6836/3540954143 2024-05-27T09:50:50.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.653+0000 7f7502000700 1 -- 172.21.15.44:0/3229640226 <== mds.1 v2:172.21.15.50:6836/3540954143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5566be1e9d90 con 0x5566be2d3590 2024-05-27T09:50:50.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.653+0000 7f7502000700 10 client.18012 handle_command_reply: tid=0 2024-05-27T09:50:50.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.653+0000 7f752daa7b80 1 client.18012 shutdown 2024-05-27T09:50:50.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.653+0000 7f752daa7b80 1 -- 172.21.15.44:0/3229640226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be13ae40 msgr2=0x5566be2c34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:50.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.653+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be13ae40 0x5566be2c34b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f750800e730 tx=0x7f750800e570 comp rx=0 tx=0).stop 2024-05-27T09:50:50.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.653+0000 7f752daa7b80 1 -- 172.21.15.44:0/3229640226 shutdown_connections 2024-05-27T09:50:50.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.653+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x5566be2d3590 0x5566be2d59b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.653+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be2bfd30 0x5566be2c3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.653+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be13ae40 0x5566be2c34b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.653+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3229640226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2b9db0 0x5566be2c39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.653+0000 7f752daa7b80 1 -- 172.21.15.44:0/3229640226 >> 172.21.15.44:0/3229640226 conn(0x5566be19afc0 msgr2=0x5566be2b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:50.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.654+0000 7f7502000700 0 client.18012 ms_handle_reset on v2:172.21.15.50:6836/3540954143 2024-05-27T09:50:50.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.654+0000 7f752daa7b80 1 -- 172.21.15.44:0/3229640226 shutdown_connections 2024-05-27T09:50:50.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.654+0000 7f752daa7b80 1 -- 172.21.15.44:0/3229640226 wait complete. 2024-05-27T09:50:50.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.654+0000 7f752daa7b80 20 client.18012 trim_cache size 0 max 16384 2024-05-27T09:50:50.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.671+0000 7f752daa7b80 1 Processor -- start 2024-05-27T09:50:50.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.671+0000 7f752daa7b80 1 -- start start 2024-05-27T09:50:50.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.671+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be13ae40 0x5566be2b9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.671+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.671+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be2bfd30 0x5566be2bd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.671+0000 7f752daa7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75181b7e20 con 0x5566be2b9db0 2024-05-27T09:50:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.671+0000 7f752daa7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f75181b5650 con 0x5566be2bfd30 2024-05-27T09:50:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.671+0000 7f752daa7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75181b34c0 con 0x5566be13ae40 2024-05-27T09:50:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.672+0000 7f7517e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be13ae40 0x5566be2b9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.672+0000 7f7517e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be13ae40 0x5566be2b9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:60432/0 (socket says 172.21.15.44:60432) 2024-05-27T09:50:50.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.672+0000 7f7517e00700 1 -- 172.21.15.44:0/968697176 learned_addr learned my addr 172.21.15.44:0/968697176 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:50.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.672+0000 7f7517400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.672+0000 7f751d000700 1 --2- 172.21.15.44:0/968697176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be2bfd30 0x5566be2bd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.672+0000 7f7517e00700 1 -- 172.21.15.44:0/968697176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be2bfd30 msgr2=0x5566be2bd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:50.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.672+0000 7f7517e00700 1 --2- 172.21.15.44:0/968697176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be2bfd30 0x5566be2bd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.672+0000 7f7517e00700 1 -- 172.21.15.44:0/968697176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 msgr2=0x5566be2cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:50.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.672+0000 7f7517e00700 1 --2- 172.21.15.44:0/968697176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.672+0000 7f7517e00700 1 -- 172.21.15.44:0/968697176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566be2c42a0 con 0x5566be13ae40 2024-05-27T09:50:50.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.673+0000 7f7517400700 1 --2- 172.21.15.44:0/968697176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:50.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.673+0000 7f751d000700 1 --2- 172.21.15.44:0/968697176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be2bfd30 0x5566be2bd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:50.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.673+0000 7f7517e00700 1 --2- 172.21.15.44:0/968697176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be13ae40 0x5566be2b9870 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f750802d040 tx=0x7f7508005810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbbcef879332f524 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:50.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.673+0000 7f750cc00700 1 -- 172.21.15.44:0/968697176 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7508026070 con 0x5566be13ae40 2024-05-27T09:50:50.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.673+0000 7f750cc00700 1 -- 172.21.15.44:0/968697176 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7508038070 con 0x5566be13ae40 2024-05-27T09:50:50.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.673+0000 7f750cc00700 1 -- 172.21.15.44:0/968697176 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7508019d40 con 0x5566be13ae40 2024-05-27T09:50:50.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.674+0000 7f752daa7b80 1 -- 172.21.15.44:0/968697176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be13ae40 msgr2=0x5566be2b9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:50.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.674+0000 7f752daa7b80 1 --2- 172.21.15.44:0/968697176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be13ae40 0x5566be2b9870 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f750802d040 tx=0x7f7508005810 comp rx=0 tx=0).stop 2024-05-27T09:50:50.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.674+0000 7f752daa7b80 1 -- 172.21.15.44:0/968697176 shutdown_connections 2024-05-27T09:50:50.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.674+0000 7f752daa7b80 1 --2- 172.21.15.44:0/968697176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be2bfd30 0x5566be2bd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.674+0000 7f752daa7b80 1 --2- 172.21.15.44:0/968697176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be13ae40 0x5566be2b9870 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.674+0000 7f752daa7b80 1 --2- 172.21.15.44:0/968697176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.674+0000 7f752daa7b80 1 -- 172.21.15.44:0/968697176 >> 172.21.15.44:0/968697176 conn(0x5566be19afc0 msgr2=0x5566be2bd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.674+0000 7f752daa7b80 1 -- 172.21.15.44:0/968697176 shutdown_connections 2024-05-27T09:50:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.674+0000 7f752daa7b80 1 -- 172.21.15.44:0/968697176 wait complete. 2024-05-27T09:50:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.675+0000 7f752daa7b80 1 -- start start 2024-05-27T09:50:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.675+0000 7f752daa7b80 1 Processor -- start 2024-05-27T09:50:50.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be13ae40 0x5566be2cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2cc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f752daa7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 0x5566be2da000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f752daa7b80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75181b3b10 con 0x5566be2b9db0 2024-05-27T09:50:50.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f752daa7b80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f75181b4f80 con 0x5566be13ae40 2024-05-27T09:50:50.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f752daa7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75181b41e0 con 0x5566be2bfd30 2024-05-27T09:50:50.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f752daa7b80 10 client.0 _ll_register_callbacks cb 0x5566be181c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:50.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f7517400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f7517400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44160/0 (socket says 172.21.15.44:44160) 2024-05-27T09:50:50.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f7517400700 1 -- 172.21.15.44:0/3251468154 learned_addr learned my addr 172.21.15.44:0/3251468154 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:50.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f7517e00700 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be13ae40 0x5566be2cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f751d000700 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 0x5566be2da000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f7517e00700 1 -- 172.21.15.44:0/3251468154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 msgr2=0x5566be2da000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:50.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f7517e00700 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 0x5566be2da000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f7517e00700 1 -- 172.21.15.44:0/3251468154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 msgr2=0x5566be2cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:50.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.676+0000 7f7517e00700 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.677+0000 7f7517e00700 1 -- 172.21.15.44:0/3251468154 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566be2dba60 con 0x5566be13ae40 2024-05-27T09:50:50.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.677+0000 7f7517400700 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:50.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.677+0000 7f7517e00700 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be13ae40 0x5566be2cbaf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7508011920 tx=0x7f7508004700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf82f6de60320973 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:50.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.677+0000 7f7502000700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:50.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.677+0000 7f7502000700 1 -- 172.21.15.44:0/3251468154 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7508026040 con 0x5566be13ae40 2024-05-27T09:50:50.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.677+0000 7f7502000700 1 -- 172.21.15.44:0/3251468154 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f7508019bd0 con 0x5566be13ae40 2024-05-27T09:50:50.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.677+0000 7f752daa7b80 1 -- 172.21.15.44:0/3251468154 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7508038070 con 0x5566be13ae40 2024-05-27T09:50:50.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.678+0000 7f7502000700 1 -- 172.21.15.44:0/3251468154 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f750804a050 con 0x5566be13ae40 2024-05-27T09:50:50.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.678+0000 7f752daa7b80 10 client.28325 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:50.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.678+0000 7f752daa7b80 1 -- 172.21.15.44:0/3251468154 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5566be33d320 con 0x5566be13ae40 2024-05-27T09:50:50.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.678+0000 7f7502000700 1 -- 172.21.15.44:0/3251468154 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7508038070 con 0x5566be13ae40 2024-05-27T09:50:50.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.678+0000 7f7502000700 1 -- 172.21.15.44:0/3251468154 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f750800ebc0 con 0x5566be13ae40 2024-05-27T09:50:50.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.678+0000 7f752daa7b80 10 client.28325 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:50.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.678+0000 7f752daa7b80 10 client.28325 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T09:50:50.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.678+0000 7f752daa7b80 20 client.28325 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:50.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.678+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x5566be33d8c0 0x5566be33fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:50.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.678+0000 7f752daa7b80 4 client.28325 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:50:50.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.678+0000 7f752daa7b80 1 -- 172.21.15.44:0/3251468154 --> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5566be1e3570 con 0x5566be33d8c0 2024-05-27T09:50:50.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.679+0000 7f7517400700 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x5566be33d8c0 0x5566be33fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:50.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.679+0000 7f7517400700 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x5566be33d8c0 0x5566be33fce0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=88c1397a6847ab6 server_cookie=5d33ccd372cad5d6 in_seq=0 out_seq=0 2024-05-27T09:50:50.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.679+0000 7f7502000700 10 client.28325 ms_handle_connect on v2:172.21.15.50:6836/3540954143 2024-05-27T09:50:50.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.679+0000 7f7502000700 1 -- 172.21.15.44:0/3251468154 <== mds.1 v2:172.21.15.50:6836/3540954143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5566be1e3570 con 0x5566be33d8c0 2024-05-27T09:50:50.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.679+0000 7f7502000700 10 client.28325 handle_command_reply: tid=0 2024-05-27T09:50:50.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.679+0000 7f752daa7b80 1 client.28325 shutdown 2024-05-27T09:50:50.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f752daa7b80 1 -- 172.21.15.44:0/3251468154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be13ae40 msgr2=0x5566be2cbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:50.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be13ae40 0x5566be2cbaf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7508011920 tx=0x7f7508004700 comp rx=0 tx=0).stop 2024-05-27T09:50:50.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f752daa7b80 1 -- 172.21.15.44:0/3251468154 shutdown_connections 2024-05-27T09:50:50.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566be2bfd30 0x5566be2da000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.50:6836/3540954143,v1:172.21.15.50:6837/3540954143] conn(0x5566be33d8c0 0x5566be33fce0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5566be13ae40 0x5566be2cbaf0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f752daa7b80 1 --2- 172.21.15.44:0/3251468154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566be2b9db0 0x5566be2cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f7502000700 0 client.28325 ms_handle_reset on v2:172.21.15.50:6836/3540954143 2024-05-27T09:50:50.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f752daa7b80 1 -- 172.21.15.44:0/3251468154 >> 172.21.15.44:0/3251468154 conn(0x5566be19afc0 msgr2=0x5566be2bed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:50.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f752daa7b80 1 -- 172.21.15.44:0/3251468154 shutdown_connections 2024-05-27T09:50:50.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f752daa7b80 1 -- 172.21.15.44:0/3251468154 wait complete. 2024-05-27T09:50:50.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.680+0000 7f752daa7b80 20 client.28325 trim_cache size 0 max 16384 2024-05-27T09:50:50.689 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:50:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 -- 172.21.15.44:0/1884157725 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74fc0b29f0 msgr2=0x7f74fc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74fc0b29f0 0x7f74fc0b4f00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x5566be2bab30 tx=0x7f751000b040 comp rx=0 tx=0).stop 2024-05-27T09:50:50.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 -- 172.21.15.44:0/1884157725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f751810b2d0 msgr2=0x7f75181a1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:50.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f751810b2d0 0x7f75181a1aa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7504000c00 tx=0x7f750400fa00 comp rx=0 tx=0).stop 2024-05-27T09:50:50.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 -- 172.21.15.44:0/1884157725 shutdown_connections 2024-05-27T09:50:50.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f74fc0b29f0 0x7f74fc0b4f00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7518110960 0x7f75181a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f751810a980 0x7f75181a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 --2- 172.21.15.44:0/1884157725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f751810b2d0 0x7f75181a1aa0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:50.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 -- 172.21.15.44:0/1884157725 >> 172.21.15.44:0/1884157725 conn(0x7f7518077c70 msgr2=0x7f751807b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:50.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 -- 172.21.15.44:0/1884157725 shutdown_connections 2024-05-27T09:50:50.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:50.681+0000 7f751f000700 1 -- 172.21.15.44:0/1884157725 wait complete. 2024-05-27T09:50:51.775 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T09:50:52.261 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:52 smithi189 ceph-mon[36671]: pgmap v492: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T09:50:52.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:52 smithi044 ceph-mon[31618]: pgmap v492: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T09:50:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:52 smithi050 ceph-mon[36371]: pgmap v492: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T09:50:54.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:54 smithi189 ceph-mon[36671]: pgmap v493: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:50:54.409 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:54 smithi044 ceph-mon[31618]: pgmap v493: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:50:54.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:54 smithi050 ceph-mon[36371]: pgmap v493: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:50:54.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.664+0000 7f3732400700 1 -- 172.21.15.44:0/2026185171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c076670 msgr2=0x7f372c110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:54.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.664+0000 7f3732400700 1 --2- 172.21.15.44:0/2026185171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c076670 0x7f372c110640 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3714009990 tx=0x7f3714009ca0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.665+0000 7f3732400700 1 -- 172.21.15.44:0/2026185171 shutdown_connections 2024-05-27T09:50:54.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.665+0000 7f3732400700 1 --2- 172.21.15.44:0/2026185171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f372c110b80 0x7f372c112fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.665+0000 7f3732400700 1 --2- 172.21.15.44:0/2026185171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f372c075d20 0x7f372c076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.665+0000 7f3732400700 1 --2- 172.21.15.44:0/2026185171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c076670 0x7f372c110640 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.665+0000 7f3732400700 1 -- 172.21.15.44:0/2026185171 >> 172.21.15.44:0/2026185171 conn(0x7f372c0ffe60 msgr2=0x7f372c1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:54.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.665+0000 7f3732400700 1 -- 172.21.15.44:0/2026185171 shutdown_connections 2024-05-27T09:50:54.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.665+0000 7f3732400700 1 -- 172.21.15.44:0/2026185171 wait complete. 2024-05-27T09:50:54.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.666+0000 7f3732400700 1 Processor -- start 2024-05-27T09:50:54.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.667+0000 7f3732400700 1 -- start start 2024-05-27T09:50:54.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.667+0000 7f3732400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c075d20 0x7f372c19d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.667+0000 7f3732400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f372c076670 0x7f372c19d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.667+0000 7f3732400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f372c110b80 0x7f372c1a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.667+0000 7f3732400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1166f0 con 0x7f372c076670 2024-05-27T09:50:54.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.667+0000 7f3732400700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f372c116570 con 0x7f372c075d20 2024-05-27T09:50:54.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.667+0000 7f3732400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f372c116870 con 0x7f372c110b80 2024-05-27T09:50:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f3722a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f372c076670 0x7f372c19d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f372b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c075d20 0x7f372c19d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f372be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f372c110b80 0x7f372c1a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f3722a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f372c076670 0x7f372c19d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42758/0 (socket says 172.21.15.44:42758) 2024-05-27T09:50:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f3722a00700 1 -- 172.21.15.44:0/3063646486 learned_addr learned my addr 172.21.15.44:0/3063646486 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f372b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c075d20 0x7f372c19d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37166/0 (socket says 172.21.15.44:37166) 2024-05-27T09:50:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f372b400700 1 -- 172.21.15.44:0/3063646486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f372c110b80 msgr2=0x7f372c1a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f372b400700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f372c110b80 0x7f372c1a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f372b400700 1 -- 172.21.15.44:0/3063646486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f372c076670 msgr2=0x7f372c19d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f372b400700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f372c076670 0x7f372c19d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f372b400700 1 -- 172.21.15.44:0/3063646486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372c1a1f70 con 0x7f372c075d20 2024-05-27T09:50:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.668+0000 7f3722a00700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f372c076670 0x7f372c19d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.669+0000 7f372b400700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c075d20 0x7f372c19d0a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f371c00efc0 tx=0x7f371c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.669+0000 7f3728c00700 1 -- 172.21.15.44:0/3063646486 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c0092e0 con 0x7f372c075d20 2024-05-27T09:50:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.669+0000 7f3728c00700 1 -- 172.21.15.44:0/3063646486 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f371c004bb0 con 0x7f372c075d20 2024-05-27T09:50:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.669+0000 7f3732400700 1 -- 172.21.15.44:0/3063646486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f372c1a2200 con 0x7f372c075d20 2024-05-27T09:50:54.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.669+0000 7f3728c00700 1 -- 172.21.15.44:0/3063646486 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c004040 con 0x7f372c075d20 2024-05-27T09:50:54.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.669+0000 7f3732400700 1 -- 172.21.15.44:0/3063646486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f372c1a26f0 con 0x7f372c075d20 2024-05-27T09:50:54.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.673+0000 7f3728c00700 1 -- 172.21.15.44:0/3063646486 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f371c01b030 con 0x7f372c075d20 2024-05-27T09:50:54.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.674+0000 7f374102bb80 1 Processor -- start 2024-05-27T09:50:54.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.674+0000 7f374102bb80 1 -- start start 2024-05-27T09:50:54.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.675+0000 7f3728c00700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37100b29f0 0x7f37100b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.675+0000 7f374102bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1150330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.675+0000 7f374102bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 0x55c2f1150870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.675+0000 7f374102bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f1150e70 0x55c2f115d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.675+0000 7f3728c00700 1 -- 172.21.15.44:0/3063646486 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f371c0e2240 con 0x7f372c075d20 2024-05-27T09:50:54.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.675+0000 7f374102bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c2f107ee40 con 0x55c2f0fd1e40 2024-05-27T09:50:54.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.675+0000 7f374102bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c2f106e9e0 con 0x55c2f1150e70 2024-05-27T09:50:54.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.675+0000 7f374102bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c2f1072a20 con 0x55c2f10bd9f0 2024-05-27T09:50:54.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.675+0000 7f372be00700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37100b29f0 0x7f37100b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.675+0000 7f3722a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 0x55c2f1150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.675+0000 7f372b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f3722a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 0x55c2f1150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42772/0 (socket says 172.21.15.44:42772) 2024-05-27T09:50:54.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f3722a00700 1 -- 172.21.15.44:0/3949848596 learned_addr learned my addr 172.21.15.44:0/3949848596 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:54.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f372b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1150330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49768/0 (socket says 172.21.15.44:49768) 2024-05-27T09:50:54.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f372be00700 1 --2- 172.21.15.44:0/3949848596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f1150e70 0x55c2f115d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f372b400700 1 -- 172.21.15.44:0/3949848596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f1150e70 msgr2=0x55c2f115d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:54.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f372b400700 1 --2- 172.21.15.44:0/3949848596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f1150e70 0x55c2f115d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f372b400700 1 -- 172.21.15.44:0/3949848596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 msgr2=0x55c2f1150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f372b400700 1 --2- 172.21.15.44:0/3949848596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 0x55c2f1150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f372b400700 1 -- 172.21.15.44:0/3949848596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2f1037420 con 0x55c2f10bd9f0 2024-05-27T09:50:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f372be00700 1 --2- 172.21.15.44:0/3949848596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f1150e70 0x55c2f115d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:50:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f372b400700 1 --2- 172.21.15.44:0/3949848596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1150330 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f371c0df8e0 tx=0x7f371c00ca70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ebc6cc48e220a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.676+0000 7f372be00700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37100b29f0 0x7f37100b4f00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f372c077bc0 tx=0x7f37240137f0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.677+0000 7f3720c00700 1 -- 172.21.15.44:0/3949848596 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c0910d0 con 0x55c2f10bd9f0 2024-05-27T09:50:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.677+0000 7f3720c00700 1 -- 172.21.15.44:0/3949848596 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f371c0a9500 con 0x55c2f10bd9f0 2024-05-27T09:50:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.677+0000 7f3720c00700 1 -- 172.21.15.44:0/3949848596 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c0b0ba0 con 0x55c2f10bd9f0 2024-05-27T09:50:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.678+0000 7f374102bb80 1 -- 172.21.15.44:0/3949848596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 msgr2=0x55c2f1150330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.678+0000 7f374102bb80 1 --2- 172.21.15.44:0/3949848596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1150330 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f371c0df8e0 tx=0x7f371c00ca70 comp rx=0 tx=0).stop 2024-05-27T09:50:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.678+0000 7f374102bb80 1 -- 172.21.15.44:0/3949848596 shutdown_connections 2024-05-27T09:50:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.678+0000 7f374102bb80 1 --2- 172.21.15.44:0/3949848596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f1150e70 0x55c2f115d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.678+0000 7f374102bb80 1 --2- 172.21.15.44:0/3949848596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1150330 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.678+0000 7f374102bb80 1 --2- 172.21.15.44:0/3949848596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 0x55c2f1150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.678+0000 7f374102bb80 1 -- 172.21.15.44:0/3949848596 >> 172.21.15.44:0/3949848596 conn(0x55c2f106b390 msgr2=0x55c2f114c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.678+0000 7f374102bb80 1 -- 172.21.15.44:0/3949848596 shutdown_connections 2024-05-27T09:50:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.678+0000 7f374102bb80 1 -- 172.21.15.44:0/3949848596 wait complete. 2024-05-27T09:50:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.679+0000 7f374102bb80 1 -- start start 2024-05-27T09:50:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.679+0000 7f374102bb80 1 Processor -- start 2024-05-27T09:50:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f374102bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 0x55c2f115a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f374102bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f10bd9f0 0x55c2f115abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f374102bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f1150e70 0x55c2f115b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f374102bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c2f0f74fa0 con 0x55c2f0fd1e40 2024-05-27T09:50:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f374102bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c2f100f760 con 0x55c2f10bd9f0 2024-05-27T09:50:54.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f374102bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c2f115db80 con 0x55c2f1150e70 2024-05-27T09:50:54.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f374102bb80 10 client.0 _ll_register_callbacks cb 0x55c2f1018c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:54.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f372b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 0x55c2f115a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f372b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 0x55c2f115a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42778/0 (socket says 172.21.15.44:42778) 2024-05-27T09:50:54.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f372b400700 1 -- 172.21.15.44:0/4251498667 learned_addr learned my addr 172.21.15.44:0/4251498667 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:54.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f3722a00700 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f10bd9f0 0x55c2f115abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f372be00700 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f1150e70 0x55c2f115b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f372b400700 1 -- 172.21.15.44:0/4251498667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f1150e70 msgr2=0x55c2f115b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:54.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f372b400700 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f1150e70 0x55c2f115b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f372b400700 1 -- 172.21.15.44:0/4251498667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f10bd9f0 msgr2=0x55c2f115abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:54.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f372b400700 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f10bd9f0 0x55c2f115abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.680+0000 7f372b400700 1 -- 172.21.15.44:0/4251498667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2f0fbe8d0 con 0x55c2f0fd1e40 2024-05-27T09:50:54.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.681+0000 7f3722a00700 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f10bd9f0 0x55c2f115abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:54.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.681+0000 7f372b400700 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 0x55c2f115a680 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f371c00ef90 tx=0x7f371c0a9130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df243dfa8c34d0ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:54.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.681+0000 7f3719600700 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-05-27T09:50:54.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.681+0000 7f3719600700 1 -- 172.21.15.44:0/4251498667 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c0ad8b0 con 0x55c2f0fd1e40 2024-05-27T09:50:54.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.681+0000 7f3719600700 1 -- 172.21.15.44:0/4251498667 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f371c0b0db0 con 0x55c2f0fd1e40 2024-05-27T09:50:54.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.681+0000 7f374102bb80 1 -- 172.21.15.44:0/4251498667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f371c0a9500 con 0x55c2f0fd1e40 2024-05-27T09:50:54.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.681+0000 7f3719600700 1 -- 172.21.15.44:0/4251498667 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f371c0b22d0 con 0x55c2f0fd1e40 2024-05-27T09:50:54.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.682+0000 7f374102bb80 10 client.18030 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:54.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.682+0000 7f374102bb80 1 -- 172.21.15.44:0/4251498667 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55c2f103f060 con 0x55c2f0fd1e40 2024-05-27T09:50:54.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.682+0000 7f3719600700 1 -- 172.21.15.44:0/4251498667 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f371c0a9500 con 0x55c2f0fd1e40 2024-05-27T09:50:54.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.682+0000 7f3719600700 1 -- 172.21.15.44:0/4251498667 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f371c0a4f90 con 0x55c2f0fd1e40 2024-05-27T09:50:54.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.682+0000 7f374102bb80 10 client.18030 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:54.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.682+0000 7f374102bb80 10 client.18030 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:50:54.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.682+0000 7f374102bb80 20 client.18030 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:54.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.682+0000 7f374102bb80 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55c2f116a9a0 0x55c2f116cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.682+0000 7f374102bb80 4 client.18030 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:54.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.682+0000 7f374102bb80 1 -- 172.21.15.44:0/4251498667 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2f1080d90 con 0x55c2f116a9a0 2024-05-27T09:50:54.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.683+0000 7f3722a00700 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55c2f116a9a0 0x55c2f116cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.683+0000 7f3722a00700 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55c2f116a9a0 0x55c2f116cdc0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e2db866e6942e0a1 server_cookie=548ee37b8fb75188 in_seq=0 out_seq=0 2024-05-27T09:50:54.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.683+0000 7f3719600700 10 client.18030 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:50:54.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f3719600700 1 -- 172.21.15.44:0/4251498667 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c2f1080d90 con 0x55c2f116a9a0 2024-05-27T09:50:54.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f3719600700 10 client.18030 handle_command_reply: tid=0 2024-05-27T09:50:54.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 1 client.18030 shutdown 2024-05-27T09:50:54.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 1 -- 172.21.15.44:0/4251498667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 msgr2=0x55c2f115a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:54.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 0x55c2f115a680 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f371c00ef90 tx=0x7f371c0a9130 comp rx=0 tx=0).stop 2024-05-27T09:50:54.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 1 -- 172.21.15.44:0/4251498667 shutdown_connections 2024-05-27T09:50:54.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f1150e70 0x55c2f115b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55c2f116a9a0 0x55c2f116cdc0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f0fd1e40 0x55c2f115a680 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 1 --2- 172.21.15.44:0/4251498667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f10bd9f0 0x55c2f115abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 1 -- 172.21.15.44:0/4251498667 >> 172.21.15.44:0/4251498667 conn(0x55c2f106b390 msgr2=0x55c2f1150310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:54.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f3719600700 0 client.18030 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:50:54.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 1 -- 172.21.15.44:0/4251498667 shutdown_connections 2024-05-27T09:50:54.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 1 -- 172.21.15.44:0/4251498667 wait complete. 2024-05-27T09:50:54.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.685+0000 7f374102bb80 20 client.18030 trim_cache size 0 max 16384 2024-05-27T09:50:54.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.703+0000 7f374102bb80 1 Processor -- start 2024-05-27T09:50:54.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.703+0000 7f374102bb80 1 -- start start 2024-05-27T09:50:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.703+0000 7f374102bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 0x55c2f11595e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.703+0000 7f374102bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.703+0000 7f374102bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f1150e70 0x55c2f1166e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.703+0000 7f374102bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1b3af0 con 0x55c2f1150e70 2024-05-27T09:50:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.703+0000 7f374102bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1b1320 con 0x55c2f0fd1e40 2024-05-27T09:50:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.703+0000 7f374102bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1af190 con 0x55c2f10bd9f0 2024-05-27T09:50:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.703+0000 7f3722a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.704+0000 7f372b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 0x55c2f11595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.704+0000 7f3722a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49794/0 (socket says 172.21.15.44:49794) 2024-05-27T09:50:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.704+0000 7f3722a00700 1 -- 172.21.15.44:0/3915879134 learned_addr learned my addr 172.21.15.44:0/3915879134 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.704+0000 7f372be00700 1 --2- 172.21.15.44:0/3915879134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f1150e70 0x55c2f1166e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.704+0000 7f3722a00700 1 -- 172.21.15.44:0/3915879134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 msgr2=0x55c2f11595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.704+0000 7f3722a00700 1 --2- 172.21.15.44:0/3915879134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 0x55c2f11595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.704+0000 7f3722a00700 1 -- 172.21.15.44:0/3915879134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f1150e70 msgr2=0x55c2f1166e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.704+0000 7f3722a00700 1 --2- 172.21.15.44:0/3915879134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f1150e70 0x55c2f1166e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.704+0000 7f3722a00700 1 -- 172.21.15.44:0/3915879134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2f1148240 con 0x55c2f10bd9f0 2024-05-27T09:50:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.704+0000 7f372b400700 1 --2- 172.21.15.44:0/3915879134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 0x55c2f11595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.704+0000 7f3722a00700 1 --2- 172.21.15.44:0/3915879134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1155e50 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x55c2f114d790 tx=0x7f371400b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=266090af01f40a40 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.705+0000 7f371be00700 1 -- 172.21.15.44:0/3915879134 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714003680 con 0x55c2f10bd9f0 2024-05-27T09:50:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.705+0000 7f371be00700 1 -- 172.21.15.44:0/3915879134 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3714003cd0 con 0x55c2f10bd9f0 2024-05-27T09:50:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.705+0000 7f371be00700 1 -- 172.21.15.44:0/3915879134 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371400fc50 con 0x55c2f10bd9f0 2024-05-27T09:50:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.706+0000 7f374102bb80 1 -- 172.21.15.44:0/3915879134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 msgr2=0x55c2f1155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.706+0000 7f374102bb80 1 --2- 172.21.15.44:0/3915879134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1155e50 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x55c2f114d790 tx=0x7f371400b260 comp rx=0 tx=0).stop 2024-05-27T09:50:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.706+0000 7f374102bb80 1 -- 172.21.15.44:0/3915879134 shutdown_connections 2024-05-27T09:50:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.706+0000 7f374102bb80 1 --2- 172.21.15.44:0/3915879134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f1150e70 0x55c2f1166e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.706+0000 7f374102bb80 1 --2- 172.21.15.44:0/3915879134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 0x55c2f11595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.706+0000 7f374102bb80 1 --2- 172.21.15.44:0/3915879134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f10bd9f0 0x55c2f1155e50 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.706+0000 7f374102bb80 1 -- 172.21.15.44:0/3915879134 >> 172.21.15.44:0/3915879134 conn(0x55c2f106b390 msgr2=0x55c2f1153ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.706+0000 7f374102bb80 1 -- 172.21.15.44:0/3915879134 shutdown_connections 2024-05-27T09:50:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.706+0000 7f374102bb80 1 -- 172.21.15.44:0/3915879134 wait complete. 2024-05-27T09:50:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.706+0000 7f374102bb80 1 -- start start 2024-05-27T09:50:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.707+0000 7f374102bb80 1 Processor -- start 2024-05-27T09:50:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.707+0000 7f374102bb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 0x55c2f116ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.707+0000 7f374102bb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f10bd9f0 0x55c2f116cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.707+0000 7f374102bb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f1150e70 0x55c2f116d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.707+0000 7f374102bb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1af7e0 con 0x55c2f10bd9f0 2024-05-27T09:50:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.707+0000 7f374102bb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1b0c50 con 0x55c2f0fd1e40 2024-05-27T09:50:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.707+0000 7f374102bb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1afeb0 con 0x55c2f1150e70 2024-05-27T09:50:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.707+0000 7f374102bb80 10 client.0 _ll_register_callbacks cb 0x55c2f1018c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f372b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 0x55c2f116ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f372b400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 0x55c2f116ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37204/0 (socket says 172.21.15.44:37204) 2024-05-27T09:50:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f372b400700 1 -- 172.21.15.44:0/3870616557 learned_addr learned my addr 172.21.15.44:0/3870616557 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f3722a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f10bd9f0 0x55c2f116cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f372be00700 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f1150e70 0x55c2f116d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f372b400700 1 -- 172.21.15.44:0/3870616557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f1150e70 msgr2=0x55c2f116d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:54.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f372b400700 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f1150e70 0x55c2f116d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f372b400700 1 -- 172.21.15.44:0/3870616557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f10bd9f0 msgr2=0x55c2f116cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:54.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f372b400700 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f10bd9f0 0x55c2f116cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f372b400700 1 -- 172.21.15.44:0/3870616557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2f11d3460 con 0x55c2f0fd1e40 2024-05-27T09:50:54.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f372b400700 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 0x55c2f116ca00 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f371c0a6770 tx=0x7f371c0df9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=539a2c010f1697a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:54.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.708+0000 7f3719600700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:54.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.709+0000 7f3719600700 1 -- 172.21.15.44:0/3870616557 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c0a9500 con 0x55c2f0fd1e40 2024-05-27T09:50:54.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.709+0000 7f3719600700 1 -- 172.21.15.44:0/3870616557 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f371c0a4e70 con 0x55c2f0fd1e40 2024-05-27T09:50:54.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.709+0000 7f374102bb80 1 -- 172.21.15.44:0/3870616557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3714003cd0 con 0x55c2f0fd1e40 2024-05-27T09:50:54.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.709+0000 7f3719600700 1 -- 172.21.15.44:0/3870616557 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f371c0ad8b0 con 0x55c2f0fd1e40 2024-05-27T09:50:54.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.709+0000 7f374102bb80 10 client.28343 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:54.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.709+0000 7f374102bb80 1 -- 172.21.15.44:0/3870616557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55c2f11d46c0 con 0x55c2f0fd1e40 2024-05-27T09:50:54.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.709+0000 7f3719600700 1 -- 172.21.15.44:0/3870616557 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3714003cd0 con 0x55c2f0fd1e40 2024-05-27T09:50:54.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.709+0000 7f3719600700 1 -- 172.21.15.44:0/3870616557 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f371c0a8760 con 0x55c2f0fd1e40 2024-05-27T09:50:54.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.710+0000 7f374102bb80 10 client.28343 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:54.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.710+0000 7f374102bb80 10 client.28343 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-27T09:50:54.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.710+0000 7f374102bb80 20 client.28343 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:54.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.710+0000 7f374102bb80 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55c2f11d4b80 0x55c2f11d6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:54.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.710+0000 7f374102bb80 4 client.28343 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:50:54.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.710+0000 7f374102bb80 1 -- 172.21.15.44:0/3870616557 --> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2f107a570 con 0x55c2f11d4b80 2024-05-27T09:50:54.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.710+0000 7f3722a00700 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55c2f11d4b80 0x55c2f11d6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:54.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f3722a00700 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55c2f11d4b80 0x55c2f11d6fa0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e0634b912c62c337 server_cookie=1c9228780cf579a5 in_seq=0 out_seq=0 2024-05-27T09:50:54.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f3719600700 10 client.28343 ms_handle_connect on v2:172.21.15.189:6836/2265427173 2024-05-27T09:50:54.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f3719600700 1 -- 172.21.15.44:0/3870616557 <== mds.2 v2:172.21.15.189:6836/2265427173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2f107a570 con 0x55c2f11d4b80 2024-05-27T09:50:54.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f3719600700 10 client.28343 handle_command_reply: tid=0 2024-05-27T09:50:54.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f374102bb80 1 client.28343 shutdown 2024-05-27T09:50:54.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f374102bb80 1 -- 172.21.15.44:0/3870616557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 msgr2=0x55c2f116ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:54.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f374102bb80 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 0x55c2f116ca00 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f371c0a6770 tx=0x7f371c0df9c0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f374102bb80 1 -- 172.21.15.44:0/3870616557 shutdown_connections 2024-05-27T09:50:54.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f374102bb80 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2f1150e70 0x55c2f116d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f374102bb80 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2f0fd1e40 0x55c2f116ca00 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f374102bb80 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.189:6836/2265427173,v1:172.21.15.189:6837/2265427173] conn(0x55c2f11d4b80 0x55c2f11d6fa0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f374102bb80 1 --2- 172.21.15.44:0/3870616557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c2f10bd9f0 0x55c2f116cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f374102bb80 1 -- 172.21.15.44:0/3870616557 >> 172.21.15.44:0/3870616557 conn(0x55c2f106b390 msgr2=0x55c2f115ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:54.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.711+0000 7f3719600700 0 client.28343 ms_handle_reset on v2:172.21.15.189:6836/2265427173 2024-05-27T09:50:54.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.712+0000 7f374102bb80 1 -- 172.21.15.44:0/3870616557 shutdown_connections 2024-05-27T09:50:54.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.712+0000 7f374102bb80 1 -- 172.21.15.44:0/3870616557 wait complete. 2024-05-27T09:50:54.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.712+0000 7f374102bb80 20 client.28343 trim_cache size 0 max 16384 2024-05-27T09:50:54.719 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:50:54.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.712+0000 7f3732400700 1 -- 172.21.15.44:0/3063646486 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37100b29f0 msgr2=0x7f37100b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:54.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.712+0000 7f3732400700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37100b29f0 0x7f37100b4f00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f372c077bc0 tx=0x7f37240137f0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.712+0000 7f3732400700 1 -- 172.21.15.44:0/3063646486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c075d20 msgr2=0x7f372c19d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:54.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.712+0000 7f3732400700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c075d20 0x7f372c19d0a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f371c00efc0 tx=0x7f371c00c3a0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.712+0000 7f3732400700 1 -- 172.21.15.44:0/3063646486 shutdown_connections 2024-05-27T09:50:54.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.713+0000 7f3732400700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f37100b29f0 0x7f37100b4f00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.713+0000 7f3732400700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f372c110b80 0x7f372c1a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.713+0000 7f3732400700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c075d20 0x7f372c19d0a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.713+0000 7f3732400700 1 --2- 172.21.15.44:0/3063646486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f372c076670 0x7f372c19d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:54.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.713+0000 7f3732400700 1 -- 172.21.15.44:0/3063646486 >> 172.21.15.44:0/3063646486 conn(0x7f372c0ffe60 msgr2=0x7f372c100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:54.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.713+0000 7f3732400700 1 -- 172.21.15.44:0/3063646486 shutdown_connections 2024-05-27T09:50:54.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:54.713+0000 7f3732400700 1 -- 172.21.15.44:0/3063646486 wait complete. 2024-05-27T09:50:55.709 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T09:50:56.171 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:50:56.354 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:56 smithi044 ceph-mon[31618]: pgmap v494: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:50:56.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:56 smithi050 ceph-mon[36371]: pgmap v494: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:50:56.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:56 smithi189 ceph-mon[36671]: pgmap v494: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:50:58.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:50:58 smithi044 ceph-mon[31618]: pgmap v495: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-27T09:50:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.498+0000 7f413cc00700 1 -- 172.21.15.44:0/597217574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4138106570 msgr2=0x7f4138106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.498+0000 7f413cc00700 1 --2- 172.21.15.44:0/597217574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4138106570 0x7f4138106980 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4120009990 tx=0x7f4120009ca0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.499+0000 7f413cc00700 1 -- 172.21.15.44:0/597217574 shutdown_connections 2024-05-27T09:50:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.499+0000 7f413cc00700 1 --2- 172.21.15.44:0/597217574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4138106ec0 0x7f4138113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.499+0000 7f413cc00700 1 --2- 172.21.15.44:0/597217574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f413810c5b0 0x7f413810c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.499+0000 7f413cc00700 1 --2- 172.21.15.44:0/597217574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4138106570 0x7f4138106980 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.499+0000 7f413cc00700 1 -- 172.21.15.44:0/597217574 >> 172.21.15.44:0/597217574 conn(0x7f4138102070 msgr2=0x7f41381044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.499+0000 7f413cc00700 1 -- 172.21.15.44:0/597217574 shutdown_connections 2024-05-27T09:50:58.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.499+0000 7f413cc00700 1 -- 172.21.15.44:0/597217574 wait complete. 2024-05-27T09:50:58.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.500+0000 7f413cc00700 1 Processor -- start 2024-05-27T09:50:58.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.500+0000 7f413cc00700 1 -- start start 2024-05-27T09:50:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.500+0000 7f413cc00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4138106570 0x7f413819d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.500+0000 7f413cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4138106ec0 0x7f413819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f413cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f413810c5b0 0x7f41381a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f413cc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f41381166f0 con 0x7f4138106ec0 2024-05-27T09:50:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f413cc00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4138116570 con 0x7f4138106570 2024-05-27T09:50:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f413cc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4138116870 con 0x7f413810c5b0 2024-05-27T09:50:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f412fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4138106ec0 0x7f413819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f412fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4138106ec0 0x7f413819d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42826/0 (socket says 172.21.15.44:42826) 2024-05-27T09:50:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f4135e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4138106570 0x7f413819d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f412fe00700 1 -- 172.21.15.44:0/1288354710 learned_addr learned my addr 172.21.15.44:0/1288354710 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f4136800700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f413810c5b0 0x7f41381a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f412fe00700 1 -- 172.21.15.44:0/1288354710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f413810c5b0 msgr2=0x7f41381a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f412fe00700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f413810c5b0 0x7f41381a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f412fe00700 1 -- 172.21.15.44:0/1288354710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4138106570 msgr2=0x7f413819d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f412fe00700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4138106570 0x7f413819d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.501+0000 7f412fe00700 1 -- 172.21.15.44:0/1288354710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41381a1f10 con 0x7f4138106ec0 2024-05-27T09:50:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.502+0000 7f412fe00700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4138106ec0 0x7f413819d5e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4120000c00 tx=0x7f41200047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.502+0000 7f412ea00700 1 -- 172.21.15.44:0/1288354710 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120026070 con 0x7f4138106ec0 2024-05-27T09:50:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.502+0000 7f413cc00700 1 -- 172.21.15.44:0/1288354710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41381a21a0 con 0x7f4138106ec0 2024-05-27T09:50:58.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.502+0000 7f412ea00700 1 -- 172.21.15.44:0/1288354710 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4120021410 con 0x7f4138106ec0 2024-05-27T09:50:58.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.502+0000 7f412ea00700 1 -- 172.21.15.44:0/1288354710 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120017650 con 0x7f4138106ec0 2024-05-27T09:50:58.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.502+0000 7f413cc00700 1 -- 172.21.15.44:0/1288354710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41381a9b20 con 0x7f4138106ec0 2024-05-27T09:50:58.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.508+0000 7f412ea00700 1 -- 172.21.15.44:0/1288354710 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f4120029070 con 0x7f4138106ec0 2024-05-27T09:50:58.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.509+0000 7f414b77fb80 1 Processor -- start 2024-05-27T09:50:58.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.509+0000 7f414b77fb80 1 -- start start 2024-05-27T09:50:58.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.509+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c306330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.509+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c306870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.509+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c306e70 0x559a2c313110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.509+0000 7f414b77fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559a2c234e40 con 0x559a2c2739f0 2024-05-27T09:50:58.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.509+0000 7f414b77fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559a2c2249e0 con 0x559a2c306e70 2024-05-27T09:50:58.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.509+0000 7f414b77fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559a2c228a20 con 0x559a2c187e40 2024-05-27T09:50:58.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f412fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c306e70 0x559a2c313110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f4136800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c306330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f4135e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c306870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f4135e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c306870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49824/0 (socket says 172.21.15.44:49824) 2024-05-27T09:50:58.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f412fe00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c306e70 0x559a2c313110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37246/0 (socket says 172.21.15.44:37246) 2024-05-27T09:50:58.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f4135e00700 1 -- 172.21.15.44:0/1502320708 learned_addr learned my addr 172.21.15.44:0/1502320708 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:58.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f412ea00700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f411c0b29f0 0x7f411c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f4135e00700 1 -- 172.21.15.44:0/1502320708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c306e70 msgr2=0x559a2c313110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:58.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f4135e00700 1 --2- 172.21.15.44:0/1502320708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c306e70 0x559a2c313110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f4135e00700 1 -- 172.21.15.44:0/1502320708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 msgr2=0x559a2c306330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:58.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f4135e00700 1 --2- 172.21.15.44:0/1502320708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c306330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f4135e00700 1 -- 172.21.15.44:0/1502320708 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a2c1ed420 con 0x559a2c187e40 2024-05-27T09:50:58.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.510+0000 7f412ea00700 1 -- 172.21.15.44:0/1288354710 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f41200e3240 con 0x7f4138106ec0 2024-05-27T09:50:58.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.511+0000 7f4136800700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f411c0b29f0 0x7f411c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.511+0000 7f4135e00700 1 --2- 172.21.15.44:0/1502320708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c306870 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4128009d20 tx=0x7f4128011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f86994c655aa65d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:58.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.511+0000 7f4127400700 1 -- 172.21.15.44:0/1502320708 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412801e070 con 0x559a2c187e40 2024-05-27T09:50:58.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.511+0000 7f4127400700 1 -- 172.21.15.44:0/1502320708 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4128019ae0 con 0x559a2c187e40 2024-05-27T09:50:58.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.511+0000 7f4127400700 1 -- 172.21.15.44:0/1502320708 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412800ea30 con 0x559a2c187e40 2024-05-27T09:50:58.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.512+0000 7f414b77fb80 1 -- 172.21.15.44:0/1502320708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 msgr2=0x559a2c306870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:58.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.512+0000 7f414b77fb80 1 --2- 172.21.15.44:0/1502320708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c306870 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4128009d20 tx=0x7f4128011720 comp rx=0 tx=0).stop 2024-05-27T09:50:58.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.512+0000 7f4136800700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f411c0b29f0 0x7f411c0b4f00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x559a2c308ff0 tx=0x7f4130009040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:58.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.513+0000 7f414b77fb80 1 -- 172.21.15.44:0/1502320708 shutdown_connections 2024-05-27T09:50:58.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.513+0000 7f414b77fb80 1 --2- 172.21.15.44:0/1502320708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c306e70 0x559a2c313110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.513+0000 7f414b77fb80 1 --2- 172.21.15.44:0/1502320708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c306330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.513+0000 7f414b77fb80 1 --2- 172.21.15.44:0/1502320708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c306870 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.513+0000 7f414b77fb80 1 -- 172.21.15.44:0/1502320708 >> 172.21.15.44:0/1502320708 conn(0x559a2c221390 msgr2=0x559a2c3024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:58.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.513+0000 7f414b77fb80 1 -- 172.21.15.44:0/1502320708 shutdown_connections 2024-05-27T09:50:58.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.513+0000 7f414b77fb80 1 -- 172.21.15.44:0/1502320708 wait complete. 2024-05-27T09:50:58.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.513+0000 7f414b77fb80 1 -- start start 2024-05-27T09:50:58.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.513+0000 7f414b77fb80 1 Processor -- start 2024-05-27T09:50:58.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.514+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 0x559a2c310680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.514+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c310bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.514+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c306e70 0x559a2c311100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.514+0000 7f414b77fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559a2c12afa0 con 0x559a2c2739f0 2024-05-27T09:50:58.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.514+0000 7f414b77fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559a2c1c5760 con 0x559a2c187e40 2024-05-27T09:50:58.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.514+0000 7f414b77fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559a2c313b80 con 0x559a2c306e70 2024-05-27T09:50:58.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.514+0000 7f414b77fb80 10 client.0 _ll_register_callbacks cb 0x559a2c1cec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:58.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.514+0000 7f412fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c310bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.514+0000 7f4136800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c306e70 0x559a2c311100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f4135e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 0x559a2c310680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f4135e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 0x559a2c310680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37252/0 (socket says 172.21.15.44:37252) 2024-05-27T09:50:58.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f4136800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c306e70 0x559a2c311100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49828/0 (socket says 172.21.15.44:49828) 2024-05-27T09:50:58.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f4135e00700 1 -- 172.21.15.44:0/3886135299 learned_addr learned my addr 172.21.15.44:0/3886135299 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:58.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f412fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c310bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42838/0 (socket says 172.21.15.44:42838) 2024-05-27T09:50:58.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f4135e00700 1 -- 172.21.15.44:0/3886135299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c306e70 msgr2=0x559a2c311100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:58.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f4135e00700 1 --2- 172.21.15.44:0/3886135299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c306e70 0x559a2c311100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f4135e00700 1 -- 172.21.15.44:0/3886135299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 msgr2=0x559a2c310bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:58.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f4135e00700 1 --2- 172.21.15.44:0/3886135299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c310bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f4135e00700 1 -- 172.21.15.44:0/3886135299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a2c1748d0 con 0x559a2c187e40 2024-05-27T09:50:58.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f4135e00700 1 --2- 172.21.15.44:0/3886135299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 0x559a2c310680 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4128009920 tx=0x7f4128009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e5e5d74511b6b0b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:58.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.515+0000 7f411be00700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-05-27T09:50:58.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.516+0000 7f411be00700 1 -- 172.21.15.44:0/3886135299 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412801e040 con 0x559a2c187e40 2024-05-27T09:50:58.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.516+0000 7f414b77fb80 1 -- 172.21.15.44:0/3886135299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4128019ae0 con 0x559a2c187e40 2024-05-27T09:50:58.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.516+0000 7f411be00700 1 -- 172.21.15.44:0/3886135299 <== mon.2 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f41280198c0 con 0x559a2c187e40 2024-05-27T09:50:58.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.516+0000 7f411be00700 1 -- 172.21.15.44:0/3886135299 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f4128032050 con 0x559a2c187e40 2024-05-27T09:50:58.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.516+0000 7f414b77fb80 10 client.28355 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:58.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.516+0000 7f414b77fb80 1 -- 172.21.15.44:0/3886135299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559a2c1f5060 con 0x559a2c187e40 2024-05-27T09:50:58.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.517+0000 7f411be00700 1 -- 172.21.15.44:0/3886135299 <== mon.2 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4128019ae0 con 0x559a2c187e40 2024-05-27T09:50:58.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.517+0000 7f411be00700 1 -- 172.21.15.44:0/3886135299 <== mon.2 v2:172.21.15.50:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f4128004750 con 0x559a2c187e40 2024-05-27T09:50:58.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.517+0000 7f414b77fb80 10 client.28355 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:58.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.517+0000 7f414b77fb80 10 client.28355 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:58.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.517+0000 7f414b77fb80 20 client.28355 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:58.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.517+0000 7f414b77fb80 1 --2- 172.21.15.44:0/3886135299 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559a2c320a00 0x559a2c322e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.517+0000 7f414b77fb80 4 client.28355 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T09:50:58.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.517+0000 7f414b77fb80 1 -- 172.21.15.44:0/3886135299 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a2c236d90 con 0x559a2c320a00 2024-05-27T09:50:58.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.518+0000 7f412fe00700 1 --2- 172.21.15.44:0/3886135299 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559a2c320a00 0x559a2c322e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.518+0000 7f412fe00700 1 --2- 172.21.15.44:0/3886135299 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559a2c320a00 0x559a2c322e20 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f9e71bf38034889 server_cookie=9f110de3b50c605 in_seq=0 out_seq=0 2024-05-27T09:50:58.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.518+0000 7f411be00700 10 client.28355 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:58.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f411be00700 1 -- 172.21.15.44:0/3886135299 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559a2c236d90 con 0x559a2c320a00 2024-05-27T09:50:58.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f411be00700 10 client.28355 handle_command_reply: tid=0 2024-05-27T09:50:58.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f414b77fb80 1 client.28355 shutdown 2024-05-27T09:50:58.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f414b77fb80 1 -- 172.21.15.44:0/3886135299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 msgr2=0x559a2c310680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:58.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f414b77fb80 1 --2- 172.21.15.44:0/3886135299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 0x559a2c310680 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4128009920 tx=0x7f4128009490 comp rx=0 tx=0).stop 2024-05-27T09:50:58.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f414b77fb80 1 -- 172.21.15.44:0/3886135299 shutdown_connections 2024-05-27T09:50:58.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f414b77fb80 1 --2- 172.21.15.44:0/3886135299 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559a2c320a00 0x559a2c322e20 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f414b77fb80 1 --2- 172.21.15.44:0/3886135299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c306e70 0x559a2c311100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f414b77fb80 1 --2- 172.21.15.44:0/3886135299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 0x559a2c310680 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f414b77fb80 1 --2- 172.21.15.44:0/3886135299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c310bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f411be00700 0 client.28355 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:58.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.519+0000 7f414b77fb80 1 -- 172.21.15.44:0/3886135299 >> 172.21.15.44:0/3886135299 conn(0x559a2c221390 msgr2=0x559a2c306310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:58.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.520+0000 7f414b77fb80 1 -- 172.21.15.44:0/3886135299 shutdown_connections 2024-05-27T09:50:58.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.520+0000 7f414b77fb80 1 -- 172.21.15.44:0/3886135299 wait complete. 2024-05-27T09:50:58.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.520+0000 7f414b77fb80 20 client.28355 trim_cache size 0 max 16384 2024-05-27T09:50:58.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.537+0000 7f414b77fb80 1 Processor -- start 2024-05-27T09:50:58.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.537+0000 7f414b77fb80 1 -- start start 2024-05-27T09:50:58.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.537+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 0x559a2c30be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.537+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c2739f0 0x559a2c3024f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.537+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c306e70 0x559a2c308900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.537+0000 7f414b77fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f41381b3a10 con 0x559a2c306e70 2024-05-27T09:50:58.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.537+0000 7f414b77fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f41381b1240 con 0x559a2c187e40 2024-05-27T09:50:58.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.537+0000 7f414b77fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f41381af0b0 con 0x559a2c2739f0 2024-05-27T09:50:58.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f4136800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c306e70 0x559a2c308900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f4135e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 0x559a2c30be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f4136800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c306e70 0x559a2c308900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42842/0 (socket says 172.21.15.44:42842) 2024-05-27T09:50:58.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f4136800700 1 -- 172.21.15.44:0/420920382 learned_addr learned my addr 172.21.15.44:0/420920382 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:58.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f412fe00700 1 --2- 172.21.15.44:0/420920382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c2739f0 0x559a2c3024f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f4136800700 1 -- 172.21.15.44:0/420920382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c2739f0 msgr2=0x559a2c3024f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:58.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f4136800700 1 --2- 172.21.15.44:0/420920382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c2739f0 0x559a2c3024f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f4136800700 1 -- 172.21.15.44:0/420920382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 msgr2=0x559a2c30be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:58.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f4136800700 1 --2- 172.21.15.44:0/420920382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 0x559a2c30be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f4136800700 1 -- 172.21.15.44:0/420920382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a2c2fe240 con 0x559a2c306e70 2024-05-27T09:50:58.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f4135e00700 1 --2- 172.21.15.44:0/420920382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 0x559a2c30be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:50:58.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.538+0000 7f4136800700 1 --2- 172.21.15.44:0/420920382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c306e70 0x559a2c308900 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4130017a60 tx=0x7f4130004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359673177a63058f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:58.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.539+0000 7f4126a00700 1 -- 172.21.15.44:0/420920382 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130024840 con 0x559a2c306e70 2024-05-27T09:50:58.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.539+0000 7f4126a00700 1 -- 172.21.15.44:0/420920382 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4130029070 con 0x559a2c306e70 2024-05-27T09:50:58.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.539+0000 7f4126a00700 1 -- 172.21.15.44:0/420920382 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130016bd0 con 0x559a2c306e70 2024-05-27T09:50:58.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.540+0000 7f414b77fb80 1 -- 172.21.15.44:0/420920382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c306e70 msgr2=0x559a2c308900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:58.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.540+0000 7f414b77fb80 1 --2- 172.21.15.44:0/420920382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c306e70 0x559a2c308900 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4130017a60 tx=0x7f4130004b40 comp rx=0 tx=0).stop 2024-05-27T09:50:58.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.540+0000 7f414b77fb80 1 -- 172.21.15.44:0/420920382 shutdown_connections 2024-05-27T09:50:58.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.540+0000 7f414b77fb80 1 --2- 172.21.15.44:0/420920382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c306e70 0x559a2c308900 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.540+0000 7f414b77fb80 1 --2- 172.21.15.44:0/420920382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c187e40 0x559a2c30be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.540+0000 7f414b77fb80 1 --2- 172.21.15.44:0/420920382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c2739f0 0x559a2c3024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.540+0000 7f414b77fb80 1 -- 172.21.15.44:0/420920382 >> 172.21.15.44:0/420920382 conn(0x559a2c221390 msgr2=0x559a2c310460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:58.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.540+0000 7f414b77fb80 1 -- 172.21.15.44:0/420920382 shutdown_connections 2024-05-27T09:50:58.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.540+0000 7f414b77fb80 1 -- 172.21.15.44:0/420920382 wait complete. 2024-05-27T09:50:58.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.541+0000 7f414b77fb80 1 -- start start 2024-05-27T09:50:58.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.541+0000 7f414b77fb80 1 Processor -- start 2024-05-27T09:50:58.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.541+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c30a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.541+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c30aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.541+0000 7f414b77fb80 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c306e70 0x559a2c30af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.541+0000 7f414b77fb80 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f41381af700 con 0x559a2c2739f0 2024-05-27T09:50:58.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.541+0000 7f414b77fb80 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f41381b0b70 con 0x559a2c306e70 2024-05-27T09:50:58.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.541+0000 7f414b77fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f41381afdd0 con 0x559a2c187e40 2024-05-27T09:50:58.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.542+0000 7f414b77fb80 10 client.0 _ll_register_callbacks cb 0x559a2c1cec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T09:50:58.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.542+0000 7f4135e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c30a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.542+0000 7f4135e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c30a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:49834/0 (socket says 172.21.15.44:49834) 2024-05-27T09:50:58.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.542+0000 7f4135e00700 1 -- 172.21.15.44:0/2226290450 learned_addr learned my addr 172.21.15.44:0/2226290450 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:58.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.542+0000 7f412fe00700 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c30aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.542+0000 7f4136800700 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c306e70 0x559a2c30af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.542+0000 7f4135e00700 1 -- 172.21.15.44:0/2226290450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c306e70 msgr2=0x559a2c30af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:58.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.542+0000 7f4135e00700 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c306e70 0x559a2c30af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.542+0000 7f4135e00700 1 -- 172.21.15.44:0/2226290450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 msgr2=0x559a2c30aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T09:50:58.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.542+0000 7f4135e00700 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c30aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.542+0000 7f4135e00700 1 -- 172.21.15.44:0/2226290450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a2c317f10 con 0x559a2c187e40 2024-05-27T09:50:58.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.543+0000 7f4135e00700 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c30a4d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4128000c00 tx=0x7f4128003960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f369549542e2231 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:58.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.543+0000 7f411be00700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-05-27T09:50:58.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.543+0000 7f411be00700 1 -- 172.21.15.44:0/2226290450 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412803a020 con 0x559a2c187e40 2024-05-27T09:50:58.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.543+0000 7f414b77fb80 1 -- 172.21.15.44:0/2226290450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4130029070 con 0x559a2c187e40 2024-05-27T09:50:58.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.543+0000 7f411be00700 1 -- 172.21.15.44:0/2226290450 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f41280198c0 con 0x559a2c187e40 2024-05-27T09:50:58.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.543+0000 7f411be00700 1 -- 172.21.15.44:0/2226290450 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f4128045050 con 0x559a2c187e40 2024-05-27T09:50:58.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.543+0000 7f414b77fb80 10 client.28369 fetch_fsmap learned FSMap version 43 2024-05-27T09:50:58.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.543+0000 7f414b77fb80 1 -- 172.21.15.44:0/2226290450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559a2c319130 con 0x559a2c187e40 2024-05-27T09:50:58.548 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:50:58 smithi050 ceph-mon[36371]: pgmap v495: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-27T09:50:58.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.544+0000 7f411be00700 1 -- 172.21.15.44:0/2226290450 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4130029070 con 0x559a2c187e40 2024-05-27T09:50:58.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.544+0000 7f411be00700 1 -- 172.21.15.44:0/2226290450 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f4128005490 con 0x559a2c187e40 2024-05-27T09:50:58.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.544+0000 7f414b77fb80 10 client.28369 fetch_fsmap finished waiting for FSMap version 43 2024-05-27T09:50:58.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.544+0000 7f414b77fb80 10 client.28369 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T09:50:58.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.544+0000 7f414b77fb80 20 client.28369 populate_metadata read hostname 'smithi044' 2024-05-27T09:50:58.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.544+0000 7f414b77fb80 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559a2c38b390 0x559a2c38d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:58.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.544+0000 7f414b77fb80 4 client.28369 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T09:50:58.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.544+0000 7f414b77fb80 1 -- 172.21.15.44:0/2226290450 --> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559a2c230570 con 0x559a2c38b390 2024-05-27T09:50:58.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.544+0000 7f412fe00700 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559a2c38b390 0x559a2c38d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:58.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f412fe00700 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559a2c38b390 0x559a2c38d7b0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce6af7a2e7a76257 server_cookie=395c0bfaaebda5b1 in_seq=0 out_seq=0 2024-05-27T09:50:58.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f411be00700 10 client.28369 ms_handle_connect on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:58.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f411be00700 1 -- 172.21.15.44:0/2226290450 <== mds.0 v2:172.21.15.50:6838/1124366265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559a2c230570 con 0x559a2c38b390 2024-05-27T09:50:58.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f411be00700 10 client.28369 handle_command_reply: tid=0 2024-05-27T09:50:58.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f414b77fb80 1 client.28369 shutdown 2024-05-27T09:50:58.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f414b77fb80 1 -- 172.21.15.44:0/2226290450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 msgr2=0x559a2c30a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:58.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f414b77fb80 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c30a4d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4128000c00 tx=0x7f4128003960 comp rx=0 tx=0).stop 2024-05-27T09:50:58.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f414b77fb80 1 -- 172.21.15.44:0/2226290450 shutdown_connections 2024-05-27T09:50:58.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f414b77fb80 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.50:6838/1124366265,v1:172.21.15.50:6839/1124366265] conn(0x559a2c38b390 0x559a2c38d7b0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f414b77fb80 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559a2c306e70 0x559a2c30af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f414b77fb80 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559a2c187e40 0x559a2c30a4d0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f414b77fb80 1 --2- 172.21.15.44:0/2226290450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559a2c2739f0 0x559a2c30aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f414b77fb80 1 -- 172.21.15.44:0/2226290450 >> 172.21.15.44:0/2226290450 conn(0x559a2c221390 msgr2=0x559a2c308660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:58.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.545+0000 7f411be00700 0 client.28369 ms_handle_reset on v2:172.21.15.50:6838/1124366265 2024-05-27T09:50:58.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.546+0000 7f414b77fb80 1 -- 172.21.15.44:0/2226290450 shutdown_connections 2024-05-27T09:50:58.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.546+0000 7f414b77fb80 1 -- 172.21.15.44:0/2226290450 wait complete. 2024-05-27T09:50:58.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.546+0000 7f414b77fb80 20 client.28369 trim_cache size 0 max 16384 2024-05-27T09:50:58.553 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-05-27T09:50:58.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.546+0000 7f413cc00700 1 -- 172.21.15.44:0/1288354710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f411c0b29f0 msgr2=0x7f411c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:58.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.546+0000 7f413cc00700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f411c0b29f0 0x7f411c0b4f00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x559a2c308ff0 tx=0x7f4130009040 comp rx=0 tx=0).stop 2024-05-27T09:50:58.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.546+0000 7f413cc00700 1 -- 172.21.15.44:0/1288354710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4138106ec0 msgr2=0x7f413819d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:58.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.546+0000 7f413cc00700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4138106ec0 0x7f413819d5e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4120000c00 tx=0x7f41200047b0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.546+0000 7f413cc00700 1 -- 172.21.15.44:0/1288354710 shutdown_connections 2024-05-27T09:50:58.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.546+0000 7f413cc00700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f411c0b29f0 0x7f411c0b4f00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.547+0000 7f413cc00700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f413810c5b0 0x7f41381a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.547+0000 7f413cc00700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4138106570 0x7f413819d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.547+0000 7f413cc00700 1 --2- 172.21.15.44:0/1288354710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4138106ec0 0x7f413819d5e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:58.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.547+0000 7f413cc00700 1 -- 172.21.15.44:0/1288354710 >> 172.21.15.44:0/1288354710 conn(0x7f4138102070 msgr2=0x7f4138104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:58.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.547+0000 7f413cc00700 1 -- 172.21.15.44:0/1288354710 shutdown_connections 2024-05-27T09:50:58.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:58.547+0000 7f413cc00700 1 -- 172.21.15.44:0/1288354710 wait complete. 2024-05-27T09:50:58.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:50:58 smithi189 ceph-mon[36671]: pgmap v495: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-27T09:50:59.409 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph-c_a6507388dd5057528934822c0163b0c347ef1d5d/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-05-27T09:50:59.417 INFO:tasks.fwd_scrub:done joining 2024-05-27T09:50:59.418 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-27T09:50:59.429 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T09:50:59.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.729+0000 7f73db200640 1 -- 172.21.15.44:0/2992367677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73dc104180 msgr2=0x7f73dc1104b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:59.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.729+0000 7f73db200640 1 --2- 172.21.15.44:0/2992367677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73dc104180 0x7f73dc1104b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f73d4009be0 tx=0x7f73d402a6d0 comp rx=0 tx=0).stop 2024-05-27T09:50:59.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.730+0000 7f73db200640 1 -- 172.21.15.44:0/2992367677 shutdown_connections 2024-05-27T09:50:59.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.730+0000 7f73db200640 1 --2- 172.21.15.44:0/2992367677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73dc104180 0x7f73dc1104b0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:59.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.730+0000 7f73db200640 1 --2- 172.21.15.44:0/2992367677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73dc109ec0 0x7f73dc103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:59.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.730+0000 7f73db200640 1 --2- 172.21.15.44:0/2992367677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73dc109a20 0x7f73dc1035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:59.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.730+0000 7f73db200640 1 -- 172.21.15.44:0/2992367677 >> 172.21.15.44:0/2992367677 conn(0x7f73dc07a210 msgr2=0x7f73dc07a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:59.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.730+0000 7f73db200640 1 -- 172.21.15.44:0/2992367677 shutdown_connections 2024-05-27T09:50:59.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.730+0000 7f73db200640 1 -- 172.21.15.44:0/2992367677 wait complete. 2024-05-27T09:50:59.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.731+0000 7f73db200640 1 Processor -- start 2024-05-27T09:50:59.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.731+0000 7f73db200640 1 -- start start 2024-05-27T09:50:59.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.732+0000 7f73db200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73dc104180 0x7f73dc074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:59.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.732+0000 7f73db200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73dc109a20 0x7f73dc075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:59.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.732+0000 7f73db200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73dc109ec0 0x7f73dc075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:59.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.732+0000 7f73db200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc113c40 con 0x7f73dc109ec0 2024-05-27T09:50:59.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.732+0000 7f73db200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc113ac0 con 0x7f73dc104180 2024-05-27T09:50:59.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.732+0000 7f73db200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc113dc0 con 0x7f73dc109a20 2024-05-27T09:50:59.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.732+0000 7f73d9e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73dc104180 0x7f73dc074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:59.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.732+0000 7f73d9e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73dc104180 0x7f73dc074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37276/0 (socket says 172.21.15.44:37276) 2024-05-27T09:50:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.732+0000 7f73da800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73dc109ec0 0x7f73dc075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.732+0000 7f73d9e00640 1 -- 172.21.15.44:0/2420458056 learned_addr learned my addr 172.21.15.44:0/2420458056 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:50:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.732+0000 7f73d9400640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73dc109a20 0x7f73dc075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.733+0000 7f73d9e00640 1 -- 172.21.15.44:0/2420458056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73dc109a20 msgr2=0x7f73dc075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.733+0000 7f73d9e00640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73dc109a20 0x7f73dc075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.733+0000 7f73d9e00640 1 -- 172.21.15.44:0/2420458056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73dc109ec0 msgr2=0x7f73dc075a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:59.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.733+0000 7f73d9e00640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73dc109ec0 0x7f73dc075a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.733+0000 7f73d9e00640 1 -- 172.21.15.44:0/2420458056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc1a88e0 con 0x7f73dc104180 2024-05-27T09:50:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.733+0000 7f73d9e00640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73dc104180 0x7f73dc074fc0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00e9c0 tx=0x7f73cc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.733+0000 7f73d2a00640 1 -- 172.21.15.44:0/2420458056 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc00ccc0 con 0x7f73dc104180 2024-05-27T09:50:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.733+0000 7f73d2a00640 1 -- 172.21.15.44:0/2420458056 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73cc0108e0 con 0x7f73dc104180 2024-05-27T09:50:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.733+0000 7f73d2a00640 1 -- 172.21.15.44:0/2420458056 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc0187e0 con 0x7f73dc104180 2024-05-27T09:50:59.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.733+0000 7f73db200640 1 -- 172.21.15.44:0/2420458056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73dc1a8bd0 con 0x7f73dc104180 2024-05-27T09:50:59.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.734+0000 7f73db200640 1 -- 172.21.15.44:0/2420458056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73dc1a90d0 con 0x7f73dc104180 2024-05-27T09:50:59.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.735+0000 7f73db200640 1 -- 172.21.15.44:0/2420458056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73dc0764f0 con 0x7f73dc104180 2024-05-27T09:50:59.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.737+0000 7f73d2a00640 1 -- 172.21.15.44:0/2420458056 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f73cc010430 con 0x7f73dc104180 2024-05-27T09:50:59.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.739+0000 7f73d2a00640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73c00b29d0 0x7f73c00b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:50:59.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.739+0000 7f73d2a00640 1 -- 172.21.15.44:0/2420458056 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f73cc0e2870 con 0x7f73dc104180 2024-05-27T09:50:59.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.739+0000 7f73d9400640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73c00b29d0 0x7f73c00b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:50:59.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.740+0000 7f73d9400640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73c00b29d0 0x7f73c00b4ee0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f73dc101270 tx=0x7f73c8005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:50:59.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.741+0000 7f73d2a00640 1 -- 172.21.15.44:0/2420458056 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f73cc0a9b50 con 0x7f73dc104180 2024-05-27T09:50:59.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.958+0000 7f73db200640 1 -- 172.21.15.44:0/2420458056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73dc02dc20 con 0x7f73dc104180 2024-05-27T09:50:59.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.960+0000 7f73d2a00640 1 -- 172.21.15.44:0/2420458056 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+23214 (secure 0 0 0) 0x7f73cc0b2220 con 0x7f73dc104180 2024-05-27T09:50:59.961 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:50:59.961 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":103,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:50:32.999007+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/4077502151":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3716545176":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.189:6838/4109192600":"2024-05-28T09:47:25.864371+0000","172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/408258823":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/4102964244":"2024-05-28T09:35:14.029409+0000","172.21.15.189:6839/4109192600":"2024-05-28T09:47:25.864371+0000","172.21.15.44:0/2278564126":"2024-05-28T09:35:14.029409+0000","172.21.15.44:6800/4077502151":"2024-05-28T09:35:14.029409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:50:59.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.963+0000 7f73db200640 1 -- 172.21.15.44:0/2420458056 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73c00b29d0 msgr2=0x7f73c00b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:59.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.963+0000 7f73db200640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73c00b29d0 0x7f73c00b4ee0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f73dc101270 tx=0x7f73c8005f90 comp rx=0 tx=0).stop 2024-05-27T09:50:59.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.963+0000 7f73db200640 1 -- 172.21.15.44:0/2420458056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73dc104180 msgr2=0x7f73dc074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:50:59.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.963+0000 7f73db200640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73dc104180 0x7f73dc074fc0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00e9c0 tx=0x7f73cc00ee80 comp rx=0 tx=0).stop 2024-05-27T09:50:59.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.963+0000 7f73db200640 1 -- 172.21.15.44:0/2420458056 shutdown_connections 2024-05-27T09:50:59.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.963+0000 7f73db200640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f73c00b29d0 0x7f73c00b4ee0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:59.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.963+0000 7f73db200640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73dc109ec0 0x7f73dc075a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:59.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.963+0000 7f73db200640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73dc109a20 0x7f73dc075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:59.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.963+0000 7f73db200640 1 --2- 172.21.15.44:0/2420458056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73dc104180 0x7f73dc074fc0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:50:59.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.963+0000 7f73db200640 1 -- 172.21.15.44:0/2420458056 >> 172.21.15.44:0/2420458056 conn(0x7f73dc07a210 msgr2=0x7f73dc101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:50:59.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.964+0000 7f73db200640 1 -- 172.21.15.44:0/2420458056 shutdown_connections 2024-05-27T09:50:59.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:50:59.964+0000 7f73db200640 1 -- 172.21.15.44:0/2420458056 wait complete. 2024-05-27T09:50:59.995 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T09:25:46.636320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T09:31:19.459994+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T09:31:23.749576+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 25, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T09:50:59.996 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T09:51:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.311+0000 7fc89d000640 1 -- 172.21.15.44:0/3777689927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc898109c30 msgr2=0x7fc8981037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.311+0000 7fc89d000640 1 --2- 172.21.15.44:0/3777689927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc898109c30 0x7fc8981037d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc888009ca0 tx=0x7fc88802a8c0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.312+0000 7fc89d000640 1 -- 172.21.15.44:0/3777689927 shutdown_connections 2024-05-27T09:51:00.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.312+0000 7fc89d000640 1 --2- 172.21.15.44:0/3777689927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc898104380 0x7fc898110750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.312+0000 7fc89d000640 1 --2- 172.21.15.44:0/3777689927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc89810a0d0 0x7fc898103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.312+0000 7fc89d000640 1 --2- 172.21.15.44:0/3777689927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc898109c30 0x7fc8981037d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.312+0000 7fc89d000640 1 -- 172.21.15.44:0/3777689927 >> 172.21.15.44:0/3777689927 conn(0x7fc8980ff2f0 msgr2=0x7fc898101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:00.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.312+0000 7fc89d000640 1 -- 172.21.15.44:0/3777689927 shutdown_connections 2024-05-27T09:51:00.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.312+0000 7fc89d000640 1 -- 172.21.15.44:0/3777689927 wait complete. 2024-05-27T09:51:00.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.313+0000 7fc89d000640 1 Processor -- start 2024-05-27T09:51:00.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.313+0000 7fc89d000640 1 -- start start 2024-05-27T09:51:00.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.314+0000 7fc89d000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc898104380 0x7fc89819d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:00.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.314+0000 7fc89d000640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc898109c30 0x7fc89819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:00.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.314+0000 7fc89d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89810a0d0 0x7fc8981a8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:00.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.314+0000 7fc89d000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc898113e20 con 0x7fc898104380 2024-05-27T09:51:00.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.314+0000 7fc89d000640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc898113ca0 con 0x7fc898109c30 2024-05-27T09:51:00.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.314+0000 7fc89d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc898113fa0 con 0x7fc89810a0d0 2024-05-27T09:51:00.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.314+0000 7fc895e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc898104380 0x7fc89819d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.314+0000 7fc896800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89810a0d0 0x7fc8981a8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.314+0000 7fc895e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc898104380 0x7fc89819d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42894/0 (socket says 172.21.15.44:42894) 2024-05-27T09:51:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.314+0000 7fc895e00640 1 -- 172.21.15.44:0/2907867658 learned_addr learned my addr 172.21.15.44:0/2907867658 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:51:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.315+0000 7fc88d400640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc898109c30 0x7fc89819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.315+0000 7fc896800640 1 -- 172.21.15.44:0/2907867658 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc898109c30 msgr2=0x7fc89819de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.315+0000 7fc896800640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc898109c30 0x7fc89819de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.315+0000 7fc896800640 1 -- 172.21.15.44:0/2907867658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc898104380 msgr2=0x7fc89819d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.315+0000 7fc896800640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc898104380 0x7fc89819d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.315+0000 7fc896800640 1 -- 172.21.15.44:0/2907867658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8981a8ca0 con 0x7fc89810a0d0 2024-05-27T09:51:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.315+0000 7fc895e00640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc898104380 0x7fc89819d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:51:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.315+0000 7fc896800640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89810a0d0 0x7fc8981a8760 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc89000e970 tx=0x7fc89000ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.316+0000 7fc88f400640 1 -- 172.21.15.44:0/2907867658 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89000ccc0 con 0x7fc89810a0d0 2024-05-27T09:51:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.316+0000 7fc88f400640 1 -- 172.21.15.44:0/2907867658 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8900035f0 con 0x7fc89810a0d0 2024-05-27T09:51:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.316+0000 7fc88f400640 1 -- 172.21.15.44:0/2907867658 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89000ce60 con 0x7fc89810a0d0 2024-05-27T09:51:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.316+0000 7fc89d000640 1 -- 172.21.15.44:0/2907867658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8981a8f90 con 0x7fc89810a0d0 2024-05-27T09:51:00.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.316+0000 7fc89d000640 1 -- 172.21.15.44:0/2907867658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8981a93f0 con 0x7fc89810a0d0 2024-05-27T09:51:00.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.317+0000 7fc89d000640 1 -- 172.21.15.44:0/2907867658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc898104f10 con 0x7fc89810a0d0 2024-05-27T09:51:00.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.319+0000 7fc88f400640 1 -- 172.21.15.44:0/2907867658 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc890004020 con 0x7fc89810a0d0 2024-05-27T09:51:00.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.321+0000 7fc88f400640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc87c0b2960 0x7fc87c0b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:00.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.321+0000 7fc88f400640 1 -- 172.21.15.44:0/2907867658 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7fc8900e28d0 con 0x7fc89810a0d0 2024-05-27T09:51:00.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.321+0000 7fc895e00640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc87c0b2960 0x7fc87c0b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:00.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.322+0000 7fc895e00640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc87c0b2960 0x7fc87c0b4e70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc888033040 tx=0x7fc888032eb0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:00.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.323+0000 7fc88f400640 1 -- 172.21.15.44:0/2907867658 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc8900aa200 con 0x7fc89810a0d0 2024-05-27T09:51:00.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:00 smithi044 ceph-mon[31618]: pgmap v496: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:00.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:00 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2420458056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:51:00.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.537+0000 7fc89d000640 1 -- 172.21.15.44:0/2907867658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc8981a9750 con 0x7fc89810a0d0 2024-05-27T09:51:00.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.538+0000 7fc88f400640 1 -- 172.21.15.44:0/2907867658 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7fc8900a9b00 con 0x7fc89810a0d0 2024-05-27T09:51:00.538 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-05-27T09:51:00.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.540+0000 7fc89d000640 1 -- 172.21.15.44:0/2907867658 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc87c0b2960 msgr2=0x7fc87c0b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:00.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.540+0000 7fc89d000640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc87c0b2960 0x7fc87c0b4e70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc888033040 tx=0x7fc888032eb0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.540+0000 7fc89d000640 1 -- 172.21.15.44:0/2907867658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89810a0d0 msgr2=0x7fc8981a8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:00.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.540+0000 7fc89d000640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89810a0d0 0x7fc8981a8760 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc89000e970 tx=0x7fc89000ee30 comp rx=0 tx=0).stop 2024-05-27T09:51:00.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.541+0000 7fc89d000640 1 -- 172.21.15.44:0/2907867658 shutdown_connections 2024-05-27T09:51:00.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.541+0000 7fc89d000640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc89810a0d0 0x7fc8981a8760 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.541+0000 7fc89d000640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc898109c30 0x7fc89819de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.541+0000 7fc89d000640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc87c0b2960 0x7fc87c0b4e70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.541+0000 7fc89d000640 1 --2- 172.21.15.44:0/2907867658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc898104380 0x7fc89819d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.541+0000 7fc89d000640 1 -- 172.21.15.44:0/2907867658 >> 172.21.15.44:0/2907867658 conn(0x7fc8980ff2f0 msgr2=0x7fc898076000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:00.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.541+0000 7fc89d000640 1 -- 172.21.15.44:0/2907867658 shutdown_connections 2024-05-27T09:51:00.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.541+0000 7fc89d000640 1 -- 172.21.15.44:0/2907867658 wait complete. 2024-05-27T09:51:00.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:00 smithi050 ceph-mon[36371]: pgmap v496: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:00.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:00 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2420458056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:51:00.571 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T09:51:00.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:00 smithi189 ceph-mon[36671]: pgmap v496: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:00.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:00 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2420458056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:51:00.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.867+0000 7f293b200640 1 -- 172.21.15.44:0/3390249790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f293c075480 msgr2=0x7f293c075890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:00.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.867+0000 7f293b200640 1 --2- 172.21.15.44:0/3390249790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f293c075480 0x7f293c075890 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f292c009900 tx=0x7f292c02a480 comp rx=0 tx=0).stop 2024-05-27T09:51:00.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.867+0000 7f293b200640 1 -- 172.21.15.44:0/3390249790 shutdown_connections 2024-05-27T09:51:00.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.867+0000 7f293b200640 1 --2- 172.21.15.44:0/3390249790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f293c114a30 0x7f293c116e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.867+0000 7f293b200640 1 --2- 172.21.15.44:0/3390249790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f293c1120d0 0x7f293c1144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.867+0000 7f293b200640 1 --2- 172.21.15.44:0/3390249790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f293c075480 0x7f293c075890 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.867+0000 7f293b200640 1 -- 172.21.15.44:0/3390249790 >> 172.21.15.44:0/3390249790 conn(0x7f293c07a1e0 msgr2=0x7f293c07a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:00.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.867+0000 7f293b200640 1 -- 172.21.15.44:0/3390249790 shutdown_connections 2024-05-27T09:51:00.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.867+0000 7f293b200640 1 -- 172.21.15.44:0/3390249790 wait complete. 2024-05-27T09:51:00.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.868+0000 7f293b200640 1 Processor -- start 2024-05-27T09:51:00.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.868+0000 7f293b200640 1 -- start start 2024-05-27T09:51:00.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.869+0000 7f293b200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f293c075480 0x7f293c1a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:00.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.869+0000 7f293b200640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f293c1120d0 0x7f293c1a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:00.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.869+0000 7f293b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f293c114a30 0x7f293c1a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:00.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.869+0000 7f293b200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f293c11a220 con 0x7f293c075480 2024-05-27T09:51:00.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.869+0000 7f293b200640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f293c11a0a0 con 0x7f293c1120d0 2024-05-27T09:51:00.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.869+0000 7f293b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f293c11a3a0 con 0x7f293c114a30 2024-05-27T09:51:00.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.869+0000 7f2939e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f293c075480 0x7f293c1a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:00.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.869+0000 7f2939e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f293c075480 0x7f293c1a6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42916/0 (socket says 172.21.15.44:42916) 2024-05-27T09:51:00.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.869+0000 7f2939e00640 1 -- 172.21.15.44:0/3447476804 learned_addr learned my addr 172.21.15.44:0/3447476804 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:51:00.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.869+0000 7f293a800640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f293c114a30 0x7f293c1a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:00.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.869+0000 7f2939400640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f293c1120d0 0x7f293c1a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:00.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.870+0000 7f2939e00640 1 -- 172.21.15.44:0/3447476804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f293c114a30 msgr2=0x7f293c1a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:00.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.870+0000 7f2939e00640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f293c114a30 0x7f293c1a0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.870+0000 7f2939e00640 1 -- 172.21.15.44:0/3447476804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f293c1120d0 msgr2=0x7f293c1a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:00.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.870+0000 7f2939e00640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f293c1120d0 0x7f293c1a6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:00.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.870+0000 7f2939e00640 1 -- 172.21.15.44:0/3447476804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293c1a1140 con 0x7f293c075480 2024-05-27T09:51:00.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.870+0000 7f2939e00640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f293c075480 0x7f293c1a6890 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f292c005940 tx=0x7f292c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:00.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.870+0000 7f2932a00640 1 -- 172.21.15.44:0/3447476804 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c03f030 con 0x7f293c075480 2024-05-27T09:51:00.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.870+0000 7f2932a00640 1 -- 172.21.15.44:0/3447476804 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f292c0044c0 con 0x7f293c075480 2024-05-27T09:51:00.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.870+0000 7f293b200640 1 -- 172.21.15.44:0/3447476804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f293c1a13d0 con 0x7f293c075480 2024-05-27T09:51:00.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.870+0000 7f2932a00640 1 -- 172.21.15.44:0/3447476804 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c035070 con 0x7f293c075480 2024-05-27T09:51:00.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.870+0000 7f293b200640 1 -- 172.21.15.44:0/3447476804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f293c1a18c0 con 0x7f293c075480 2024-05-27T09:51:00.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.872+0000 7f293b200640 1 -- 172.21.15.44:0/3447476804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f293c0767a0 con 0x7f293c075480 2024-05-27T09:51:00.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.875+0000 7f2932a00640 1 -- 172.21.15.44:0/3447476804 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f292c042030 con 0x7f293c075480 2024-05-27T09:51:00.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.876+0000 7f2932a00640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f291c0b2950 0x7f291c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:00.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.876+0000 7f2932a00640 1 -- 172.21.15.44:0/3447476804 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f292c102c60 con 0x7f293c075480 2024-05-27T09:51:00.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.877+0000 7f2939400640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f291c0b2950 0x7f291c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:00.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.877+0000 7f2939400640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f291c0b2950 0x7f291c0b4e60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f293c113e70 tx=0x7f2924009500 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:00.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:00.879+0000 7f2932a00640 1 -- 172.21.15.44:0/3447476804 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f292c0c9f40 con 0x7f293c075480 2024-05-27T09:51:01.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.101+0000 7f293b200640 1 -- 172.21.15.44:0/3447476804 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f293c1ad7d0 con 0x7f293c075480 2024-05-27T09:51:01.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.106+0000 7f2932a00640 1 -- 172.21.15.44:0/3447476804 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) ==== 104+0+11 (secure 0 0 0) 0x7f292c0ce240 con 0x7f293c075480 2024-05-27T09:51:01.107 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-05-27T09:51:01.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 -- 172.21.15.44:0/3447476804 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f291c0b2950 msgr2=0x7f291c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:01.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f291c0b2950 0x7f291c0b4e60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f293c113e70 tx=0x7f2924009500 comp rx=0 tx=0).stop 2024-05-27T09:51:01.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 -- 172.21.15.44:0/3447476804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f293c075480 msgr2=0x7f293c1a6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:01.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f293c075480 0x7f293c1a6890 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f292c005940 tx=0x7f292c0040c0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 -- 172.21.15.44:0/3447476804 shutdown_connections 2024-05-27T09:51:01.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f293c114a30 0x7f293c1a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f293c1120d0 0x7f293c1a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f291c0b2950 0x7f291c0b4e60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 --2- 172.21.15.44:0/3447476804 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f293c075480 0x7f293c1a6890 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 -- 172.21.15.44:0/3447476804 >> 172.21.15.44:0/3447476804 conn(0x7f293c07a1e0 msgr2=0x7f293c1124e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:01.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 -- 172.21.15.44:0/3447476804 shutdown_connections 2024-05-27T09:51:01.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.110+0000 7f293b200640 1 -- 172.21.15.44:0/3447476804 wait complete. 2024-05-27T09:51:01.139 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T09:51:01.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.432+0000 7f8113e00640 1 -- 172.21.15.44:0/2380174900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811410c110 msgr2=0x7f8114105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:01.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.432+0000 7f8113e00640 1 --2- 172.21.15.44:0/2380174900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811410c110 0x7f8114105d50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8100009900 tx=0x7f810002a3f0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.432+0000 7f8113e00640 1 -- 172.21.15.44:0/2380174900 shutdown_connections 2024-05-27T09:51:01.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.432+0000 7f8113e00640 1 --2- 172.21.15.44:0/2380174900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81141063c0 0x7f8114112790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.432+0000 7f8113e00640 1 --2- 172.21.15.44:0/2380174900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811410c110 0x7f8114105d50 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.432+0000 7f8113e00640 1 --2- 172.21.15.44:0/2380174900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f811410bc70 0x7f8114105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.432+0000 7f8113e00640 1 -- 172.21.15.44:0/2380174900 >> 172.21.15.44:0/2380174900 conn(0x7f811407d590 msgr2=0x7f811407d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:01.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.432+0000 7f8113e00640 1 -- 172.21.15.44:0/2380174900 shutdown_connections 2024-05-27T09:51:01.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.432+0000 7f8113e00640 1 -- 172.21.15.44:0/2380174900 wait complete. 2024-05-27T09:51:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.433+0000 7f8113e00640 1 Processor -- start 2024-05-27T09:51:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8113e00640 1 -- start start 2024-05-27T09:51:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8113e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81141063c0 0x7f81141a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8113e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811410bc70 0x7f81141a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8113e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f811410c110 0x7f811419c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:01.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8113e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8114115eb0 con 0x7f81141063c0 2024-05-27T09:51:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8113e00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8114115d30 con 0x7f811410c110 2024-05-27T09:51:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8113e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8114116030 con 0x7f811410bc70 2024-05-27T09:51:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8112a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81141063c0 0x7f81141a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8112a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81141063c0 0x7f81141a2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42936/0 (socket says 172.21.15.44:42936) 2024-05-27T09:51:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8112a00640 1 -- 172.21.15.44:0/1509925310 learned_addr learned my addr 172.21.15.44:0/1509925310 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:51:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8112000640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811410bc70 0x7f81141a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.434+0000 7f8113400640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f811410c110 0x7f811419c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:01.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.435+0000 7f8112000640 1 -- 172.21.15.44:0/1509925310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f811410c110 msgr2=0x7f811419c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:01.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.435+0000 7f8112000640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f811410c110 0x7f811419c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.435+0000 7f8112000640 1 -- 172.21.15.44:0/1509925310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81141063c0 msgr2=0x7f81141a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:01.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.435+0000 7f8112000640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81141063c0 0x7f81141a2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.435+0000 7f8112000640 1 -- 172.21.15.44:0/1509925310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811419ce90 con 0x7f811410bc70 2024-05-27T09:51:01.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.435+0000 7f8112a00640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81141063c0 0x7f81141a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:51:01.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.435+0000 7f8112000640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811410bc70 0x7f81141a2ac0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8100009230 tx=0x7f8100003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:01.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.436+0000 7f810b400640 1 -- 172.21.15.44:0/1509925310 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810003f070 con 0x7f811410bc70 2024-05-27T09:51:01.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.436+0000 7f810b400640 1 -- 172.21.15.44:0/1509925310 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8100031b60 con 0x7f811410bc70 2024-05-27T09:51:01.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.436+0000 7f8113e00640 1 -- 172.21.15.44:0/1509925310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811419d120 con 0x7f811410bc70 2024-05-27T09:51:01.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.436+0000 7f810b400640 1 -- 172.21.15.44:0/1509925310 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8100035030 con 0x7f811410bc70 2024-05-27T09:51:01.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.436+0000 7f8113e00640 1 -- 172.21.15.44:0/1509925310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f811419d5e0 con 0x7f811410bc70 2024-05-27T09:51:01.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.437+0000 7f8113e00640 1 -- 172.21.15.44:0/1509925310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8114106f50 con 0x7f811410bc70 2024-05-27T09:51:01.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.440+0000 7f810b400640 1 -- 172.21.15.44:0/1509925310 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f8100042070 con 0x7f811410bc70 2024-05-27T09:51:01.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.441+0000 7f810b400640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f80f80b2950 0x7f80f80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:01.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.441+0000 7f810b400640 1 -- 172.21.15.44:0/1509925310 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7f81001033c0 con 0x7f811410bc70 2024-05-27T09:51:01.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.441+0000 7f8112a00640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f80f80b2950 0x7f80f80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:01.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.442+0000 7f8112a00640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f80f80b2950 0x7f80f80b4e60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f811419dae0 tx=0x7f8104005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:01.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.443+0000 7f810b400640 1 -- 172.21.15.44:0/1509925310 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f81000ca6a0 con 0x7f811410bc70 2024-05-27T09:51:01.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:01 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2907867658' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T09:51:01.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:01 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2907867658' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T09:51:01.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:01 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2907867658' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T09:51:01.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.657+0000 7f8113e00640 1 -- 172.21.15.44:0/1509925310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f811406a150 con 0x7f811410bc70 2024-05-27T09:51:01.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.658+0000 7f810b400640 1 -- 172.21.15.44:0/1509925310 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7f811406a150 con 0x7f811410bc70 2024-05-27T09:51:01.659 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-05-27T09:51:01.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 -- 172.21.15.44:0/1509925310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f80f80b2950 msgr2=0x7f80f80b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:01.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f80f80b2950 0x7f80f80b4e60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f811419dae0 tx=0x7f8104005f90 comp rx=0 tx=0).stop 2024-05-27T09:51:01.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 -- 172.21.15.44:0/1509925310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811410bc70 msgr2=0x7f81141a2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:01.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811410bc70 0x7f81141a2ac0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8100009230 tx=0x7f8100003c10 comp rx=0 tx=0).stop 2024-05-27T09:51:01.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 -- 172.21.15.44:0/1509925310 shutdown_connections 2024-05-27T09:51:01.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f811410c110 0x7f811419c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811410bc70 0x7f81141a2ac0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f80f80b2950 0x7f80f80b4e60 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 --2- 172.21.15.44:0/1509925310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81141063c0 0x7f81141a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 -- 172.21.15.44:0/1509925310 >> 172.21.15.44:0/1509925310 conn(0x7f811407d590 msgr2=0x7f8114104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:01.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 -- 172.21.15.44:0/1509925310 shutdown_connections 2024-05-27T09:51:01.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.661+0000 7f8113e00640 1 -- 172.21.15.44:0/1509925310 wait complete. 2024-05-27T09:51:01.690 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-27T09:51:01.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.975+0000 7fc42ac00640 1 -- 172.21.15.44:0/3600696710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc42410e870 msgr2=0x7fc42410eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:01.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.975+0000 7fc42ac00640 1 --2- 172.21.15.44:0/3600696710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc42410e870 0x7fc42410eca0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc410009900 tx=0x7fc41002a3f0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.975+0000 7fc42ac00640 1 -- 172.21.15.44:0/3600696710 shutdown_connections 2024-05-27T09:51:01.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.975+0000 7fc42ac00640 1 --2- 172.21.15.44:0/3600696710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc42410f330 0x7fc424116ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.975+0000 7fc42ac00640 1 --2- 172.21.15.44:0/3600696710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc42410e870 0x7fc42410eca0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.975+0000 7fc42ac00640 1 --2- 172.21.15.44:0/3600696710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc42410a2e0 0x7fc42410a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.975+0000 7fc42ac00640 1 -- 172.21.15.44:0/3600696710 >> 172.21.15.44:0/3600696710 conn(0x7fc424076c80 msgr2=0x7fc4240770c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:01.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.975+0000 7fc42ac00640 1 -- 172.21.15.44:0/3600696710 shutdown_connections 2024-05-27T09:51:01.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.975+0000 7fc42ac00640 1 -- 172.21.15.44:0/3600696710 wait complete. 2024-05-27T09:51:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.977+0000 7fc42ac00640 1 Processor -- start 2024-05-27T09:51:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.977+0000 7fc42ac00640 1 -- start start 2024-05-27T09:51:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.977+0000 7fc42ac00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc42410a2e0 0x7fc4241a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.977+0000 7fc42ac00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc42410e870 0x7fc4241a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.977+0000 7fc42ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc42410f330 0x7fc4241a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.977+0000 7fc42ac00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc42411a260 con 0x7fc42410e870 2024-05-27T09:51:01.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.977+0000 7fc42ac00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc42411a0e0 con 0x7fc42410a2e0 2024-05-27T09:51:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.977+0000 7fc42ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc42411a3e0 con 0x7fc42410f330 2024-05-27T09:51:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc428e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc42410e870 0x7fc4241a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc429800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc42410a2e0 0x7fc4241a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc429800640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc42410a2e0 0x7fc4241a6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:37324/0 (socket says 172.21.15.44:37324) 2024-05-27T09:51:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc429800640 1 -- 172.21.15.44:0/900504659 learned_addr learned my addr 172.21.15.44:0/900504659 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:51:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc428e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc42410e870 0x7fc4241a6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42956/0 (socket says 172.21.15.44:42956) 2024-05-27T09:51:01.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc42a200640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc42410f330 0x7fc4241a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:01.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc428e00640 1 -- 172.21.15.44:0/900504659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc42410f330 msgr2=0x7fc4241a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:01.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc428e00640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc42410f330 0x7fc4241a0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc428e00640 1 -- 172.21.15.44:0/900504659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc42410a2e0 msgr2=0x7fc4241a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:01.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc428e00640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc42410a2e0 0x7fc4241a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:01.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc428e00640 1 -- 172.21.15.44:0/900504659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4241a1140 con 0x7fc42410e870 2024-05-27T09:51:01.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.978+0000 7fc429800640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc42410a2e0 0x7fc4241a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:51:01.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.979+0000 7fc428e00640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc42410e870 0x7fc4241a6dd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc41002a910 tx=0x7fc410003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:01.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.979+0000 7fc422000640 1 -- 172.21.15.44:0/900504659 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41003f070 con 0x7fc42410e870 2024-05-27T09:51:01.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.979+0000 7fc422000640 1 -- 172.21.15.44:0/900504659 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc410031b60 con 0x7fc42410e870 2024-05-27T09:51:01.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.979+0000 7fc42ac00640 1 -- 172.21.15.44:0/900504659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4241a13d0 con 0x7fc42410e870 2024-05-27T09:51:01.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.979+0000 7fc422000640 1 -- 172.21.15.44:0/900504659 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc410035030 con 0x7fc42410e870 2024-05-27T09:51:01.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.979+0000 7fc42ac00640 1 -- 172.21.15.44:0/900504659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4241a1890 con 0x7fc42410e870 2024-05-27T09:51:01.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.980+0000 7fc42ac00640 1 -- 172.21.15.44:0/900504659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc42406a150 con 0x7fc42410e870 2024-05-27T09:51:01.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.983+0000 7fc422000640 1 -- 172.21.15.44:0/900504659 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc410042070 con 0x7fc42410e870 2024-05-27T09:51:01.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.985+0000 7fc422000640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc40c0b2950 0x7fc40c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:01.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.985+0000 7fc422000640 1 -- 172.21.15.44:0/900504659 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8607+0+0 (secure 0 0 0) 0x7fc410103230 con 0x7fc42410e870 2024-05-27T09:51:01.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.985+0000 7fc429800640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc40c0b2950 0x7fc40c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:01.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.986+0000 7fc429800640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc40c0b2950 0x7fc40c0b4e60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc4241a1be0 tx=0x7fc418005f90 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:01.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:01.988+0000 7fc422000640 1 -- 172.21.15.44:0/900504659 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc4100ca510 con 0x7fc42410e870 2024-05-27T09:51:02.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.252+0000 7fc42ac00640 1 -- 172.21.15.44:0/900504659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fc4241ad830 con 0x7fc42410e870 2024-05-27T09:51:02.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.272+0000 7fc422000640 1 -- 172.21.15.44:0/900504659 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+816174 (secure 0 0 0) 0x7fc4100ce810 con 0x7fc42410e870 2024-05-27T09:51:02.282 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:51:02.282 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:51:02.282 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 27, 2024-05-27T09:51:02.282 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.283 INFO:teuthology.orchestra.run.smithi044.stdout: "active_gid": 25072, 2024-05-27T09:51:02.283 INFO:teuthology.orchestra.run.smithi044.stdout: "active_name": "x", 2024-05-27T09:51:02.283 INFO:teuthology.orchestra.run.smithi044.stdout: "active_addrs": { 2024-05-27T09:51:02.283 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-05-27T09:51:02.283 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.283 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-05-27T09:51:02.283 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:6800", 2024-05-27T09:51:02.284 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3776762031 2024-05-27T09:51:02.284 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.284 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.284 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-05-27T09:51:02.284 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:6801", 2024-05-27T09:51:02.284 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3776762031 2024-05-27T09:51:02.284 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.284 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:51:02.285 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.285 INFO:teuthology.orchestra.run.smithi044.stdout: "active_addr": "172.21.15.44:6801/3776762031", 2024-05-27T09:51:02.285 INFO:teuthology.orchestra.run.smithi044.stdout: "active_change": "2024-05-27T09:35:14.029930+0000", 2024-05-27T09:51:02.285 INFO:teuthology.orchestra.run.smithi044.stdout: "active_mgr_features": 4540701547738038271, 2024-05-27T09:51:02.285 INFO:teuthology.orchestra.run.smithi044.stdout: "available": true, 2024-05-27T09:51:02.285 INFO:teuthology.orchestra.run.smithi044.stdout: "standbys": [ 2024-05-27T09:51:02.285 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.285 INFO:teuthology.orchestra.run.smithi044.stdout: "gid": 25058, 2024-05-27T09:51:02.286 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "y", 2024-05-27T09:51:02.286 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr_features": 4540701547738038271, 2024-05-27T09:51:02.286 INFO:teuthology.orchestra.run.smithi044.stdout: "available_modules": [ 2024-05-27T09:51:02.286 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.286 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alerts", 2024-05-27T09:51:02.286 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.286 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.287 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.287 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:02.287 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:02.287 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.287 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.287 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.287 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:02.287 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.287 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.288 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.288 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T09:51:02.288 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.288 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.288 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.288 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.288 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.288 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.289 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.289 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.289 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.289 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.289 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.289 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.289 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.289 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.290 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.290 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.290 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.290 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.290 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.290 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.290 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.291 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.291 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.291 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.291 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.291 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.291 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.291 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.291 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.292 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.292 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.292 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.292 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.292 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.292 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.292 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.292 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.292 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.293 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.293 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.293 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.293 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.293 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.293 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.293 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.293 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.294 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.294 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.294 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.294 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.294 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.294 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.294 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.294 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.295 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.295 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.295 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.295 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.295 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.295 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.295 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.296 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.296 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.296 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.296 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.296 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.296 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.296 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.296 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.297 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.297 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.297 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.297 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.297 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.297 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_destination": { 2024-05-27T09:51:02.297 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_destination", 2024-05-27T09:51:02.297 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.298 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.298 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.298 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.298 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.298 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.298 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.298 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email address to send alerts to", 2024-05-27T09:51:02.299 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.299 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.299 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.299 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.299 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_from_name": { 2024-05-27T09:51:02.299 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_from_name", 2024-05-27T09:51:02.299 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.299 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.300 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.300 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "Ceph", 2024-05-27T09:51:02.300 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.300 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.300 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.300 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email From: name", 2024-05-27T09:51:02.300 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.300 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.301 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.301 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.301 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_host": { 2024-05-27T09:51:02.301 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_host", 2024-05-27T09:51:02.301 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.301 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.301 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.301 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.301 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.302 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.302 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.302 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP server", 2024-05-27T09:51:02.302 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.302 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.302 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.302 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.303 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_password": { 2024-05-27T09:51:02.303 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_password", 2024-05-27T09:51:02.303 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.303 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.303 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.303 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.303 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.304 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.304 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.304 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Password to authenticate with", 2024-05-27T09:51:02.304 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.304 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.304 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.304 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.304 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_port": { 2024-05-27T09:51:02.305 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_port", 2024-05-27T09:51:02.305 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.305 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.305 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.305 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "465", 2024-05-27T09:51:02.305 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.305 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.305 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.305 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP port", 2024-05-27T09:51:02.306 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.306 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.306 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.306 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.306 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_sender": { 2024-05-27T09:51:02.306 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_sender", 2024-05-27T09:51:02.306 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.307 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.307 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.307 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.307 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.307 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.307 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.307 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP envelope sender", 2024-05-27T09:51:02.307 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.308 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.308 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.308 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.308 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_ssl": { 2024-05-27T09:51:02.308 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_ssl", 2024-05-27T09:51:02.308 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.308 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.309 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.309 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.309 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.309 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.309 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.309 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T09:51:02.309 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.309 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.310 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.310 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.310 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_user": { 2024-05-27T09:51:02.310 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_user", 2024-05-27T09:51:02.310 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.310 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.310 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.311 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.311 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.311 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.311 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.311 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "User to authenticate as", 2024-05-27T09:51:02.311 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.311 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.312 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.312 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.312 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.312 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.312 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.312 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.312 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.312 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.313 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.313 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.313 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.313 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.313 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.313 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.313 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.313 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.314 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.314 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.314 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.314 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "balancer", 2024-05-27T09:51:02.314 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.314 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.314 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.314 INFO:teuthology.orchestra.run.smithi044.stdout: "active": { 2024-05-27T09:51:02.315 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "active", 2024-05-27T09:51:02.315 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.315 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.315 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.315 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.315 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.315 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.315 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.316 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T09:51:02.316 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.316 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.316 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.316 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.316 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_time": { 2024-05-27T09:51:02.316 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_time", 2024-05-27T09:51:02.316 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.317 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.317 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.317 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0000", 2024-05-27T09:51:02.317 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.317 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.317 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.317 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T09:51:02.317 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T09:51:02.318 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.318 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.318 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.318 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_weekday": { 2024-05-27T09:51:02.318 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_weekday", 2024-05-27T09:51:02.318 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:02.318 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.318 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.319 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.319 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-05-27T09:51:02.319 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-05-27T09:51:02.319 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.319 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T09:51:02.319 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T09:51:02.319 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.319 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.320 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.320 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_max_iterations": { 2024-05-27T09:51:02.320 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_max_iterations", 2024-05-27T09:51:02.320 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:02.320 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.320 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.320 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "25", 2024-05-27T09:51:02.320 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:02.321 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "250", 2024-05-27T09:51:02.321 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.321 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T09:51:02.321 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.321 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.321 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.321 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.321 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_metrics": { 2024-05-27T09:51:02.322 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_metrics", 2024-05-27T09:51:02.322 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.322 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.322 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.322 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T09:51:02.322 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.322 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.323 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.323 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T09:51:02.323 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T09:51:02.323 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.323 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.323 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.323 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_step": { 2024-05-27T09:51:02.323 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_step", 2024-05-27T09:51:02.324 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:02.324 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.324 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.324 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.5", 2024-05-27T09:51:02.324 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0.001", 2024-05-27T09:51:02.324 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "0.999", 2024-05-27T09:51:02.324 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.325 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "aggressiveness of optimization", 2024-05-27T09:51:02.325 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T09:51:02.325 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.325 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.325 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.325 INFO:teuthology.orchestra.run.smithi044.stdout: "end_time": { 2024-05-27T09:51:02.325 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_time", 2024-05-27T09:51:02.325 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.326 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.326 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.326 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2359", 2024-05-27T09:51:02.326 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.326 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.326 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.326 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T09:51:02.327 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T09:51:02.327 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.327 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.327 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.327 INFO:teuthology.orchestra.run.smithi044.stdout: "end_weekday": { 2024-05-27T09:51:02.327 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_weekday", 2024-05-27T09:51:02.328 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:02.328 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.328 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.328 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.328 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-05-27T09:51:02.328 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-05-27T09:51:02.328 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.328 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T09:51:02.329 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T09:51:02.329 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.329 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.329 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.329 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.329 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.329 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.330 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.330 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.330 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.330 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.330 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.330 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.330 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.330 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.331 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.331 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.331 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.331 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.331 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.331 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.331 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.332 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.332 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.332 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.332 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.332 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.332 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.332 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.332 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.333 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.333 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.333 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.333 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.333 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.333 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.333 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.333 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.333 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.334 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.334 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.334 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.334 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.334 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.334 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.335 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.335 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.335 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.335 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.335 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.335 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.335 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.335 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.336 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.336 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.336 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.336 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.336 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.336 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.336 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.336 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.337 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.337 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.337 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.337 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.337 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.337 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.337 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.337 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.338 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.338 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.338 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.338 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.338 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.338 INFO:teuthology.orchestra.run.smithi044.stdout: "min_score": { 2024-05-27T09:51:02.338 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_score", 2024-05-27T09:51:02.339 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:02.339 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.339 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.339 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.339 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.339 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.339 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.339 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T09:51:02.340 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.340 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.340 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.340 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.340 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-05-27T09:51:02.340 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-05-27T09:51:02.340 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.340 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.340 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.341 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "upmap", 2024-05-27T09:51:02.341 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.341 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.341 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.341 INFO:teuthology.orchestra.run.smithi044.stdout: "crush-compat", 2024-05-27T09:51:02.341 INFO:teuthology.orchestra.run.smithi044.stdout: "none", 2024-05-27T09:51:02.341 INFO:teuthology.orchestra.run.smithi044.stdout: "read", 2024-05-27T09:51:02.341 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap", 2024-05-27T09:51:02.342 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap-read" 2024-05-27T09:51:02.342 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.342 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Balancer mode", 2024-05-27T09:51:02.342 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.342 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.342 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.342 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.342 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_ids": { 2024-05-27T09:51:02.343 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_ids", 2024-05-27T09:51:02.343 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.343 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.343 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.343 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.343 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.343 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.343 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.344 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T09:51:02.344 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.344 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.344 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.344 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.344 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:02.344 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:02.344 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.344 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.345 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.345 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:02.345 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.345 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.345 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.345 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T09:51:02.345 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.345 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.346 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.346 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.346 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.346 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.346 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.346 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.346 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.347 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.347 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.347 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.347 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.347 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.347 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.347 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.347 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.348 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.348 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_deviation": { 2024-05-27T09:51:02.348 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_deviation", 2024-05-27T09:51:02.348 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.348 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.348 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.348 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:02.348 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:02.348 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.349 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.349 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T09:51:02.349 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T09:51:02.349 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.349 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.349 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.349 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_optimizations": { 2024-05-27T09:51:02.349 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_optimizations", 2024-05-27T09:51:02.350 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:02.350 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.350 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.350 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:02.350 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.350 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.350 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.350 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T09:51:02.351 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.351 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.351 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.351 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.351 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.351 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.351 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.351 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm", 2024-05-27T09:51:02.352 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.352 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.352 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.352 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_down_multiplier": { 2024-05-27T09:51:02.352 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_down_multiplier", 2024-05-27T09:51:02.352 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:02.352 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.352 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.353 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-05-27T09:51:02.353 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.353 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.353 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.353 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T09:51:02.353 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.353 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.354 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.354 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.354 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_refresh_rate": { 2024-05-27T09:51:02.354 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_refresh_rate", 2024-05-27T09:51:02.354 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.354 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.354 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.355 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "20", 2024-05-27T09:51:02.355 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.355 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.355 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.355 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T09:51:02.355 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.355 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.355 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.356 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.356 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_starting_port": { 2024-05-27T09:51:02.356 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_starting_port", 2024-05-27T09:51:02.356 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.356 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.356 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.356 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4721", 2024-05-27T09:51:02.356 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.357 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.357 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.357 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T09:51:02.357 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.357 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.357 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.357 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.357 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_ptrace": { 2024-05-27T09:51:02.357 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_ptrace", 2024-05-27T09:51:02.358 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.358 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.358 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.358 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.358 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.358 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.358 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.359 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T09:51:02.359 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T09:51:02.359 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.359 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.359 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.359 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_interval": { 2024-05-27T09:51:02.359 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_interval", 2024-05-27T09:51:02.359 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.360 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.360 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.360 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:02.360 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.360 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.360 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.360 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T09:51:02.360 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.361 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.361 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.361 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.361 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_memory_target_ratio": { 2024-05-27T09:51:02.361 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T09:51:02.361 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:02.361 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.361 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.362 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.7", 2024-05-27T09:51:02.362 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.362 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.362 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.362 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T09:51:02.362 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.362 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.363 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.363 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.363 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm_log_destination": { 2024-05-27T09:51:02.363 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm_log_destination", 2024-05-27T09:51:02.363 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.363 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.363 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.363 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.364 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.364 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.364 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.364 INFO:teuthology.orchestra.run.smithi044.stdout: "file", 2024-05-27T09:51:02.364 INFO:teuthology.orchestra.run.smithi044.stdout: "file,syslog", 2024-05-27T09:51:02.364 INFO:teuthology.orchestra.run.smithi044.stdout: "syslog" 2024-05-27T09:51:02.364 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.364 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T09:51:02.364 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.365 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.365 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.365 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.365 INFO:teuthology.orchestra.run.smithi044.stdout: "cgroups_split": { 2024-05-27T09:51:02.365 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cgroups_split", 2024-05-27T09:51:02.365 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.365 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.365 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.366 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.366 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.366 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.366 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.366 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T09:51:02.366 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.366 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.367 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.367 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.367 INFO:teuthology.orchestra.run.smithi044.stdout: "config_checks_enabled": { 2024-05-27T09:51:02.367 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_checks_enabled", 2024-05-27T09:51:02.367 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.367 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.367 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.367 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.368 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.368 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.368 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.368 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T09:51:02.368 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.368 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.368 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.368 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.369 INFO:teuthology.orchestra.run.smithi044.stdout: "config_dashboard": { 2024-05-27T09:51:02.369 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_dashboard", 2024-05-27T09:51:02.369 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.369 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.369 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.369 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.369 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.369 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.370 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.370 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T09:51:02.370 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.370 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.370 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.370 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.370 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_alertmanager": { 2024-05-27T09:51:02.370 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_alertmanager", 2024-05-27T09:51:02.371 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.373 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.373 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.374 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T09:51:02.374 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.374 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.374 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.374 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:02.374 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.374 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.374 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.375 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.375 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_base": { 2024-05-27T09:51:02.375 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_base", 2024-05-27T09:51:02.375 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.375 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.375 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.375 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T09:51:02.375 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.376 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.376 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.376 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Container image name, without the tag", 2024-05-27T09:51:02.376 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.376 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.376 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.376 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.376 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_elasticsearch": { 2024-05-27T09:51:02.377 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_elasticsearch", 2024-05-27T09:51:02.377 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.377 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.377 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.377 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T09:51:02.377 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.377 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.377 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.378 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "elasticsearch container image", 2024-05-27T09:51:02.378 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.378 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.378 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.378 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.378 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_grafana": { 2024-05-27T09:51:02.378 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_grafana", 2024-05-27T09:51:02.379 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.379 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.379 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.379 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T09:51:02.379 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.379 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.379 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.380 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:02.380 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.380 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.380 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.380 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.380 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_haproxy": { 2024-05-27T09:51:02.380 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_haproxy", 2024-05-27T09:51:02.380 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.380 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.381 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.381 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T09:51:02.381 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.381 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.381 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.381 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "HAproxy container image", 2024-05-27T09:51:02.381 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.381 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.382 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.382 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.382 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_agent": { 2024-05-27T09:51:02.382 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_agent", 2024-05-27T09:51:02.382 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.382 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.382 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.382 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T09:51:02.383 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.383 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.383 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.383 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger agent container image", 2024-05-27T09:51:02.383 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.383 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.383 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.383 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.384 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_collector": { 2024-05-27T09:51:02.384 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_collector", 2024-05-27T09:51:02.384 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.384 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.384 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.384 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T09:51:02.384 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.384 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.385 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.385 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger collector container image", 2024-05-27T09:51:02.385 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.385 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.385 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.385 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.385 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_query": { 2024-05-27T09:51:02.385 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_query", 2024-05-27T09:51:02.386 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.386 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.386 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.386 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T09:51:02.386 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.386 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.386 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.386 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger query container image", 2024-05-27T09:51:02.387 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.387 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.387 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.387 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.387 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_keepalived": { 2024-05-27T09:51:02.387 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_keepalived", 2024-05-27T09:51:02.387 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.387 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.387 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.388 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T09:51:02.388 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.388 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.388 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.388 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Keepalived container image", 2024-05-27T09:51:02.388 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.388 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.388 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.389 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.389 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_loki": { 2024-05-27T09:51:02.389 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_loki", 2024-05-27T09:51:02.389 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.389 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.389 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.389 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T09:51:02.389 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.390 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.390 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.390 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Loki container image", 2024-05-27T09:51:02.390 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.390 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.390 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.390 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.390 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_node_exporter": { 2024-05-27T09:51:02.391 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_node_exporter", 2024-05-27T09:51:02.391 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.391 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.391 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.391 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T09:51:02.391 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.391 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.391 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.392 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:02.392 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.392 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.392 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.392 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.392 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_nvmeof": { 2024-05-27T09:51:02.392 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_nvmeof", 2024-05-27T09:51:02.392 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.393 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.393 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.393 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T09:51:02.393 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.393 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.393 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.393 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Nvme-of container image", 2024-05-27T09:51:02.393 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.394 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.394 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.394 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.394 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_prometheus": { 2024-05-27T09:51:02.394 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_prometheus", 2024-05-27T09:51:02.394 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.394 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.394 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.395 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T09:51:02.395 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.395 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.395 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.395 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:02.395 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.395 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.395 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.396 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.396 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_promtail": { 2024-05-27T09:51:02.396 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_promtail", 2024-05-27T09:51:02.396 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.396 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.396 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.396 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T09:51:02.396 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.397 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.397 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.397 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Promtail container image", 2024-05-27T09:51:02.397 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.397 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.397 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.397 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.397 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_samba": { 2024-05-27T09:51:02.398 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_samba", 2024-05-27T09:51:02.398 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.398 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.398 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.398 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T09:51:02.398 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.398 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.398 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.399 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Samba/SMB container image", 2024-05-27T09:51:02.399 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.399 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.399 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.399 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.399 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_snmp_gateway": { 2024-05-27T09:51:02.399 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_snmp_gateway", 2024-05-27T09:51:02.400 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.400 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.400 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.400 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T09:51:02.400 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.400 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.400 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.400 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SNMP Gateway container image", 2024-05-27T09:51:02.401 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.401 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.401 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.401 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.401 INFO:teuthology.orchestra.run.smithi044.stdout: "container_init": { 2024-05-27T09:51:02.401 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_init", 2024-05-27T09:51:02.401 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.401 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.402 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.402 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.402 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.402 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.402 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.402 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T09:51:02.402 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.402 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.403 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.403 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.403 INFO:teuthology.orchestra.run.smithi044.stdout: "daemon_cache_timeout": { 2024-05-27T09:51:02.403 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "daemon_cache_timeout", 2024-05-27T09:51:02.403 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.403 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.403 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.403 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:02.404 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.404 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.404 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.404 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T09:51:02.404 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.404 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.404 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.404 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.405 INFO:teuthology.orchestra.run.smithi044.stdout: "default_cephadm_command_timeout": { 2024-05-27T09:51:02.405 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T09:51:02.405 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.405 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.405 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.405 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "900", 2024-05-27T09:51:02.405 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.405 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.405 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.406 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T09:51:02.406 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.406 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.406 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.406 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.406 INFO:teuthology.orchestra.run.smithi044.stdout: "default_registry": { 2024-05-27T09:51:02.406 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_registry", 2024-05-27T09:51:02.406 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.407 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.407 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.407 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io", 2024-05-27T09:51:02.407 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.407 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.407 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.407 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T09:51:02.407 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.408 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.408 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.408 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.408 INFO:teuthology.orchestra.run.smithi044.stdout: "device_cache_timeout": { 2024-05-27T09:51:02.408 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_cache_timeout", 2024-05-27T09:51:02.408 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.408 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.409 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.409 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1800", 2024-05-27T09:51:02.409 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.409 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.409 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.409 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache device inventory", 2024-05-27T09:51:02.409 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.409 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.410 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.410 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.410 INFO:teuthology.orchestra.run.smithi044.stdout: "device_enhanced_scan": { 2024-05-27T09:51:02.410 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_enhanced_scan", 2024-05-27T09:51:02.410 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.410 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.410 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.411 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.411 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.411 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.411 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.411 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T09:51:02.411 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.411 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.411 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.412 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.412 INFO:teuthology.orchestra.run.smithi044.stdout: "facts_cache_timeout": { 2024-05-27T09:51:02.412 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "facts_cache_timeout", 2024-05-27T09:51:02.412 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.412 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.412 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.412 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:02.412 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.412 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.413 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.413 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache host facts data", 2024-05-27T09:51:02.413 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.413 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.413 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.413 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.413 INFO:teuthology.orchestra.run.smithi044.stdout: "grafana_dashboards_path": { 2024-05-27T09:51:02.413 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "grafana_dashboards_path", 2024-05-27T09:51:02.414 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.414 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.414 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.414 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T09:51:02.414 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.414 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.414 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.414 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T09:51:02.415 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.415 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.415 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.415 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.415 INFO:teuthology.orchestra.run.smithi044.stdout: "host_check_interval": { 2024-05-27T09:51:02.415 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "host_check_interval", 2024-05-27T09:51:02.415 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.416 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.416 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.416 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:02.416 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.416 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.416 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.416 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to perform a host check", 2024-05-27T09:51:02.416 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.416 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.417 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.417 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.417 INFO:teuthology.orchestra.run.smithi044.stdout: "hw_monitoring": { 2024-05-27T09:51:02.417 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hw_monitoring", 2024-05-27T09:51:02.417 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.417 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.417 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.417 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.418 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.418 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.418 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.418 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T09:51:02.418 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.418 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.418 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.418 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.419 INFO:teuthology.orchestra.run.smithi044.stdout: "inventory_list_all": { 2024-05-27T09:51:02.419 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "inventory_list_all", 2024-05-27T09:51:02.419 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.419 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.419 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.419 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.419 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.419 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.420 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.420 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T09:51:02.420 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.420 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.420 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.420 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.420 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.420 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.421 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.421 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.421 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.421 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.421 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.421 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.421 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.421 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.422 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.422 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.422 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.422 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.422 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.422 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.422 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.422 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.423 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.423 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.423 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.423 INFO:teuthology.orchestra.run.smithi044.stdout: "log_refresh_metadata": { 2024-05-27T09:51:02.423 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_refresh_metadata", 2024-05-27T09:51:02.423 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.423 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.423 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.423 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.424 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.424 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.424 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.424 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T09:51:02.424 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.424 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.424 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.424 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.425 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.425 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.425 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.425 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.425 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.425 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.425 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.425 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.426 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.426 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T09:51:02.426 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.426 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.426 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.426 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.426 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.427 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.427 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.427 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.427 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.427 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.427 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.427 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.427 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.427 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.428 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.428 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.428 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.428 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.428 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.428 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.428 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.428 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.429 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.429 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.429 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.429 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.429 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.429 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.429 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.429 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.430 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.430 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.430 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.430 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.430 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.430 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.430 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.430 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.431 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.431 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T09:51:02.431 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T09:51:02.431 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.431 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.431 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.431 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.432 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.432 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.432 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.432 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T09:51:02.432 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.432 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.432 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.432 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.433 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T09:51:02.433 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T09:51:02.433 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.433 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.433 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.433 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "*", 2024-05-27T09:51:02.433 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.433 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.434 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.434 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T09:51:02.434 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.434 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.434 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.434 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.434 INFO:teuthology.orchestra.run.smithi044.stdout: "max_count_per_host": { 2024-05-27T09:51:02.434 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_count_per_host", 2024-05-27T09:51:02.435 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.435 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.435 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.435 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:02.435 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.435 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.435 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.435 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of daemons per service per host", 2024-05-27T09:51:02.436 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.436 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.436 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.436 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.436 INFO:teuthology.orchestra.run.smithi044.stdout: "max_osd_draining_count": { 2024-05-27T09:51:02.436 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_osd_draining_count", 2024-05-27T09:51:02.436 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.436 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.437 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.437 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:02.437 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.437 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.437 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.437 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T09:51:02.437 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.437 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.437 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.438 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.438 INFO:teuthology.orchestra.run.smithi044.stdout: "migration_current": { 2024-05-27T09:51:02.438 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "migration_current", 2024-05-27T09:51:02.438 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.438 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.438 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.438 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.439 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.439 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.439 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.439 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "internal - do not modify", 2024-05-27T09:51:02.439 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.439 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.439 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.439 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.440 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-05-27T09:51:02.440 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-05-27T09:51:02.440 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.440 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.440 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.440 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "root", 2024-05-27T09:51:02.440 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.440 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.440 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.441 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm-package", 2024-05-27T09:51:02.441 INFO:teuthology.orchestra.run.smithi044.stdout: "root" 2024-05-27T09:51:02.441 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.441 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T09:51:02.441 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.441 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.441 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.441 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.442 INFO:teuthology.orchestra.run.smithi044.stdout: "oob_default_addr": { 2024-05-27T09:51:02.442 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "oob_default_addr", 2024-05-27T09:51:02.442 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.442 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.442 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.442 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "169.254.1.1", 2024-05-27T09:51:02.442 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.443 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.443 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.443 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T09:51:02.443 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.443 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.443 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.444 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.444 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_alerts_path": { 2024-05-27T09:51:02.444 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_alerts_path", 2024-05-27T09:51:02.444 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.444 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.444 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.444 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T09:51:02.445 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.445 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.445 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.445 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T09:51:02.445 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.445 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.445 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.446 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.446 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_insecure": { 2024-05-27T09:51:02.446 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_insecure", 2024-05-27T09:51:02.446 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.446 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.446 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.446 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.447 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.447 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.447 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.447 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T09:51:02.447 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.447 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.447 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.447 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.448 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_password": { 2024-05-27T09:51:02.448 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_password", 2024-05-27T09:51:02.448 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.448 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.448 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.448 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.448 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.448 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.449 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.449 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T09:51:02.449 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.449 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.449 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.449 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.449 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_url": { 2024-05-27T09:51:02.449 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_url", 2024-05-27T09:51:02.450 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.450 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.450 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.450 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.450 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.450 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.450 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.450 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T09:51:02.451 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.451 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.451 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.451 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.451 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_username": { 2024-05-27T09:51:02.451 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_username", 2024-05-27T09:51:02.451 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.451 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.451 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.452 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.452 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.452 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.452 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.452 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T09:51:02.452 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.452 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.452 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.453 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.453 INFO:teuthology.orchestra.run.smithi044.stdout: "secure_monitoring_stack": { 2024-05-27T09:51:02.453 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "secure_monitoring_stack", 2024-05-27T09:51:02.453 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.453 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.453 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.453 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.453 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.453 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.454 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.454 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T09:51:02.454 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.454 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.454 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.454 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.454 INFO:teuthology.orchestra.run.smithi044.stdout: "service_discovery_port": { 2024-05-27T09:51:02.454 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "service_discovery_port", 2024-05-27T09:51:02.455 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.455 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.455 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.455 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8765", 2024-05-27T09:51:02.455 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.455 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.455 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.455 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "cephadm service discovery port", 2024-05-27T09:51:02.456 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.456 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.456 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.456 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.456 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.456 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.456 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.456 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.457 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.457 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.457 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.457 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.457 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.457 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.457 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.457 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.458 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.458 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.458 INFO:teuthology.orchestra.run.smithi044.stdout: "ssh_config_file": { 2024-05-27T09:51:02.458 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssh_config_file", 2024-05-27T09:51:02.458 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.458 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.458 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.458 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.459 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.459 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.459 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.459 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T09:51:02.459 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.459 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.459 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.459 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.459 INFO:teuthology.orchestra.run.smithi044.stdout: "use_agent": { 2024-05-27T09:51:02.460 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_agent", 2024-05-27T09:51:02.460 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.460 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.460 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.460 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.460 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.460 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.460 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.461 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T09:51:02.461 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.461 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.461 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.461 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.461 INFO:teuthology.orchestra.run.smithi044.stdout: "use_repo_digest": { 2024-05-27T09:51:02.461 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_repo_digest", 2024-05-27T09:51:02.461 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.462 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.462 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.462 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.462 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.462 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.462 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.462 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T09:51:02.463 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.463 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.463 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.463 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.463 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_failed_host_check": { 2024-05-27T09:51:02.463 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_failed_host_check", 2024-05-27T09:51:02.463 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.463 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.463 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.464 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.464 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.464 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.464 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.464 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T09:51:02.464 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.464 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.465 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.465 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.465 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_daemons": { 2024-05-27T09:51:02.465 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_daemons", 2024-05-27T09:51:02.465 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.465 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.465 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.465 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.466 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.466 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.466 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.466 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T09:51:02.466 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.466 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.466 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.466 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.467 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_hosts": { 2024-05-27T09:51:02.467 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_hosts", 2024-05-27T09:51:02.467 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.467 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.467 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.467 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.467 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.467 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.468 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.468 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T09:51:02.468 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.468 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.468 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.468 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.468 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.468 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.469 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.469 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crash", 2024-05-27T09:51:02.469 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.469 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.469 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.469 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.469 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.469 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.470 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.470 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.470 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.470 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.470 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.470 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.470 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.470 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.471 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.471 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.471 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.471 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.471 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.471 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.471 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.471 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.472 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.472 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.472 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.472 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.472 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.472 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.472 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.472 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.473 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.473 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.473 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.473 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.473 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.473 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.473 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.473 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.473 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.474 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.474 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.474 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.474 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.474 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.474 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.474 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.474 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.475 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.475 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.475 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.475 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.475 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.475 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.475 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.475 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.476 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.476 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.476 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.476 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.476 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.476 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.476 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.476 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.477 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.477 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.477 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.477 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.477 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.477 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.477 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.477 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.477 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.478 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.478 INFO:teuthology.orchestra.run.smithi044.stdout: "retain_interval": { 2024-05-27T09:51:02.478 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retain_interval", 2024-05-27T09:51:02.478 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.478 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.478 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.478 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "31536000", 2024-05-27T09:51:02.479 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.479 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.479 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.479 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T09:51:02.479 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.479 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.479 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.479 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.480 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.480 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.480 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.480 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.480 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.480 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.480 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.480 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.481 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.481 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.481 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.481 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.481 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.481 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.481 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_recent_interval": { 2024-05-27T09:51:02.481 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_recent_interval", 2024-05-27T09:51:02.481 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.482 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.482 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.482 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1209600", 2024-05-27T09:51:02.482 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.482 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.482 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.482 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T09:51:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dashboard", 2024-05-27T09:51:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.484 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.484 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.484 INFO:teuthology.orchestra.run.smithi044.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T09:51:02.484 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T09:51:02.484 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.484 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.484 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.484 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:02.485 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.485 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.485 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.485 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.485 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.485 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.485 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.485 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.486 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T09:51:02.486 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T09:51:02.486 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.486 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.486 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.486 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.486 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.486 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.487 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.487 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.487 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.487 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.487 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.487 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.487 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T09:51:02.487 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T09:51:02.488 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.488 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.488 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.488 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.488 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.488 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.488 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.488 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.489 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.489 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.489 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.489 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.489 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_ENABLED": { 2024-05-27T09:51:02.489 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T09:51:02.489 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.489 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.490 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.490 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.490 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.490 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.490 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.490 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.490 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.491 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.491 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.491 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.491 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T09:51:02.491 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T09:51:02.491 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.491 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.491 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.492 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.492 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.492 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.492 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.492 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.492 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.492 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.492 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.493 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.493 INFO:teuthology.orchestra.run.smithi044.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T09:51:02.493 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T09:51:02.493 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.493 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.493 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.493 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.494 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.494 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.494 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.494 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.494 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.494 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.494 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.494 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.495 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T09:51:02.495 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T09:51:02.495 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.495 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.495 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.495 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.495 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.496 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.496 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.496 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.496 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.496 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.496 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.496 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.496 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T09:51:02.497 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T09:51:02.497 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.497 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.497 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.497 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.497 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.497 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.497 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.498 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.498 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.498 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.498 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.498 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.498 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T09:51:02.498 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T09:51:02.498 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.499 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.499 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.499 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.499 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.499 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.499 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.499 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.499 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.500 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.500 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.500 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.500 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T09:51:02.500 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T09:51:02.500 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.500 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.500 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.501 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.501 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.501 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.501 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.501 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.501 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.501 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.501 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.501 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.502 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T09:51:02.502 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T09:51:02.502 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.502 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.502 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.502 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.502 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.503 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.503 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.503 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.503 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.503 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.503 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.503 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.503 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T09:51:02.504 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T09:51:02.504 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.504 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.504 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.504 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.504 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.504 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.504 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.504 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.505 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.505 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.505 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.505 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.505 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T09:51:02.505 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T09:51:02.505 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.506 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.506 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.506 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.506 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.506 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.506 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.506 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.506 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.507 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.507 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.507 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.507 INFO:teuthology.orchestra.run.smithi044.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T09:51:02.507 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T09:51:02.507 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.507 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.507 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.508 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.508 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.508 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.508 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.508 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.508 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.508 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.508 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.508 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.509 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T09:51:02.509 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T09:51:02.509 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.509 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.509 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.509 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-05-27T09:51:02.509 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.509 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.509 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.510 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.510 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.510 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.510 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.510 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.510 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T09:51:02.510 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T09:51:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.511 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_URL": { 2024-05-27T09:51:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_URL", 2024-05-27T09:51:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.512 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.513 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.514 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.514 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.514 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T09:51:02.514 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T09:51:02.514 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.514 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.514 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.514 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-05-27T09:51:02.515 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.515 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.515 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.515 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.515 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.515 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.515 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.515 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.516 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T09:51:02.516 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T09:51:02.516 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.516 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.516 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.516 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.516 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.516 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.517 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.517 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.517 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.517 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.517 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.517 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.517 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T09:51:02.517 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T09:51:02.518 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.518 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.518 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.518 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.518 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.518 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.518 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.518 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.519 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.519 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.519 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.519 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.519 INFO:teuthology.orchestra.run.smithi044.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T09:51:02.519 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T09:51:02.519 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.519 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.520 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.520 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.520 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.520 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.520 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.520 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.520 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.520 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.521 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.521 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.521 INFO:teuthology.orchestra.run.smithi044.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T09:51:02.521 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T09:51:02.521 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.521 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.521 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.521 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.522 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.522 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.522 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.522 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.522 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.522 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.522 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.522 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.523 INFO:teuthology.orchestra.run.smithi044.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T09:51:02.523 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T09:51:02.523 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.523 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.523 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.523 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "[]", 2024-05-27T09:51:02.523 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.523 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.524 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.524 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.524 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.524 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.524 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.524 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.524 INFO:teuthology.orchestra.run.smithi044.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T09:51:02.524 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T09:51:02.524 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.525 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.525 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.525 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "{}", 2024-05-27T09:51:02.525 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.525 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.525 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.525 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.525 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.526 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.526 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.526 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.526 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T09:51:02.526 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T09:51:02.526 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.526 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.526 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.527 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.527 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.527 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.527 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.527 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.527 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.527 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.527 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.527 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.528 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T09:51:02.528 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T09:51:02.528 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.528 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.528 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.528 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.528 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.528 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.529 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.529 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.529 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.529 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.529 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.529 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.529 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T09:51:02.529 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T09:51:02.529 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.530 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.530 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.530 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.530 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.530 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.530 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.530 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.530 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.531 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.531 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.531 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.531 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T09:51:02.531 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T09:51:02.531 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.531 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.531 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.532 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.532 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.532 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.532 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.532 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.532 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.532 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.532 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.533 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.533 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T09:51:02.533 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T09:51:02.533 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.533 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.533 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.533 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.534 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.534 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.534 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.534 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.534 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.534 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.534 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.534 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.535 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T09:51:02.535 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T09:51:02.535 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.535 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.535 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.535 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.535 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.535 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.536 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.536 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.536 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.536 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.536 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.536 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.536 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T09:51:02.536 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T09:51:02.536 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.537 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.537 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.537 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.537 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.537 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.537 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.537 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.537 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.538 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.538 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.538 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.538 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T09:51:02.538 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T09:51:02.538 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.538 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.538 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.539 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.539 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.539 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.539 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.539 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.539 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.539 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.539 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.540 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.540 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T09:51:02.540 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T09:51:02.540 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.540 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.540 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.540 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.540 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.541 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.541 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.541 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.541 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.541 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.541 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.541 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.541 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T09:51:02.541 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T09:51:02.542 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.542 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.542 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.542 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.542 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.542 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.542 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.542 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.543 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.543 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.543 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.543 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.543 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T09:51:02.543 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T09:51:02.543 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.543 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.544 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.544 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T09:51:02.544 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.544 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.544 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.544 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.544 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.544 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.545 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.545 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.545 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T09:51:02.545 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T09:51:02.545 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.545 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.545 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.545 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:02.546 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.546 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.546 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.546 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.546 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.546 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.546 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.546 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.547 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T09:51:02.547 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T09:51:02.547 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.547 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.547 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.547 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8", 2024-05-27T09:51:02.547 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.547 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.548 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.548 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.548 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.548 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.548 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.548 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.548 INFO:teuthology.orchestra.run.smithi044.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T09:51:02.548 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T09:51:02.549 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.549 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.549 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.549 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "45", 2024-05-27T09:51:02.549 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.549 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.549 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.549 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.550 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.550 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.550 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.550 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.550 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T09:51:02.550 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T09:51:02.550 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.550 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.551 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.551 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.551 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.551 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.551 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.551 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.551 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.551 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.551 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.552 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.552 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T09:51:02.552 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T09:51:02.552 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.552 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.552 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.552 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-05-27T09:51:02.552 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.552 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.553 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.553 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.553 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.553 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.553 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.553 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.553 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T09:51:02.553 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T09:51:02.553 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.554 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.554 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.554 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.554 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.554 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.554 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.554 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.554 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.554 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.555 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.555 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.555 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T09:51:02.555 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T09:51:02.555 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.555 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.555 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.555 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.556 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.556 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.556 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.556 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.556 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.556 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.556 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.556 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.557 INFO:teuthology.orchestra.run.smithi044.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T09:51:02.557 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T09:51:02.557 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.557 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.557 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.557 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.557 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.557 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.558 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.558 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.558 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.558 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.558 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.558 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.558 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T09:51:02.558 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T09:51:02.558 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.559 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.559 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.559 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.559 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.559 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.559 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.559 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.559 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.559 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.560 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.560 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.560 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T09:51:02.560 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T09:51:02.560 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.560 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.560 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.560 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:02.561 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.561 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.561 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.561 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.561 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.561 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.561 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.561 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.561 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T09:51:02.562 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T09:51:02.562 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.562 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.562 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.562 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:02.562 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.562 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.562 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.562 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.563 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.563 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.563 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.563 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.563 INFO:teuthology.orchestra.run.smithi044.stdout: "cross_origin_url": { 2024-05-27T09:51:02.563 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cross_origin_url", 2024-05-27T09:51:02.563 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.563 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.563 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.564 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.564 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.564 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.564 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.564 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.564 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.564 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.564 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.564 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.565 INFO:teuthology.orchestra.run.smithi044.stdout: "crt_file": { 2024-05-27T09:51:02.565 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crt_file", 2024-05-27T09:51:02.565 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.565 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.565 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.565 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.565 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.565 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.566 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.566 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.566 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.566 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.566 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.566 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.566 INFO:teuthology.orchestra.run.smithi044.stdout: "debug": { 2024-05-27T09:51:02.566 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "debug", 2024-05-27T09:51:02.567 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.567 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.567 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.567 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.567 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.567 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.567 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.567 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable/disable debug options", 2024-05-27T09:51:02.568 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.568 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.568 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.568 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.568 INFO:teuthology.orchestra.run.smithi044.stdout: "jwt_token_ttl": { 2024-05-27T09:51:02.568 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "jwt_token_ttl", 2024-05-27T09:51:02.568 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.568 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.568 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.569 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "28800", 2024-05-27T09:51:02.569 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.569 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.569 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.569 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.569 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.569 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.569 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.570 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.570 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-05-27T09:51:02.570 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-05-27T09:51:02.570 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.570 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.570 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.570 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.570 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.571 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.571 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.571 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.571 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.571 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.571 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.571 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.571 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.572 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.572 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.572 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.572 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.572 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.572 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.572 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.572 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.573 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.573 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.573 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.573 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.573 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.573 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.573 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.573 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.573 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.574 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.574 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.574 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.574 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.574 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.574 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.574 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.575 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.575 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.575 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.575 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.575 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.575 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.575 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.575 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.576 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.576 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.576 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.576 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.576 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.576 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.576 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.576 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.577 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.577 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.577 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.577 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.577 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.577 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.577 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.577 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.578 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.578 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.578 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.578 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.578 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.578 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.578 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.578 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.579 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.579 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.579 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.579 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.579 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.579 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.579 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.579 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.580 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.580 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.580 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.580 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.580 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.580 INFO:teuthology.orchestra.run.smithi044.stdout: "motd": { 2024-05-27T09:51:02.580 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "motd", 2024-05-27T09:51:02.580 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.581 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.581 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.581 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.581 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.581 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.581 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.581 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "The message of the day", 2024-05-27T09:51:02.581 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.582 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.582 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.582 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.582 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect_resolve_ip_addr": { 2024-05-27T09:51:02.582 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T09:51:02.582 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.582 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.582 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.583 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.583 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.583 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.583 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.583 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.583 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.583 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.583 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.584 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-05-27T09:51:02.584 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-05-27T09:51:02.584 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.584 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.584 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.584 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-05-27T09:51:02.584 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.585 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.585 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.585 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.585 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.585 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.585 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.585 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.585 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-05-27T09:51:02.586 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-05-27T09:51:02.586 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.586 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.586 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.586 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8080", 2024-05-27T09:51:02.586 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.586 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.586 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.587 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.587 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.587 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.587 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.587 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.587 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.587 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.587 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.588 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.588 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.588 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.588 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.588 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.588 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.588 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.588 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.589 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.589 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.589 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.589 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-05-27T09:51:02.589 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-05-27T09:51:02.589 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.589 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.589 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.589 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.590 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.590 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.590 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.590 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.590 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.590 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.590 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.590 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.591 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl_server_port": { 2024-05-27T09:51:02.591 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl_server_port", 2024-05-27T09:51:02.591 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.591 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.591 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.591 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8443", 2024-05-27T09:51:02.591 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.591 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.591 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.592 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.592 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.592 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.592 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.592 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.592 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-05-27T09:51:02.592 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-05-27T09:51:02.592 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.593 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.593 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.593 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "redirect", 2024-05-27T09:51:02.593 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.593 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.593 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.593 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.593 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect" 2024-05-27T09:51:02.593 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.594 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.594 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.594 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.594 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.594 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.594 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-05-27T09:51:02.594 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-05-27T09:51:02.595 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.595 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.595 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.595 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-05-27T09:51:02.595 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-05-27T09:51:02.595 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-05-27T09:51:02.595 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.595 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.595 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.596 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.596 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.596 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.596 INFO:teuthology.orchestra.run.smithi044.stdout: "url_prefix": { 2024-05-27T09:51:02.596 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url_prefix", 2024-05-27T09:51:02.596 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.596 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.596 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.597 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.597 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.597 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.597 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.597 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.597 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.597 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.598 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.598 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.598 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.598 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.598 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.598 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2024-05-27T09:51:02.598 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.598 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.599 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.599 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_monitoring": { 2024-05-27T09:51:02.599 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_monitoring", 2024-05-27T09:51:02.599 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.599 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.599 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.599 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.599 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.599 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.600 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.600 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "monitor device health metrics", 2024-05-27T09:51:02.600 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.600 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.600 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.600 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.600 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.601 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.601 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.601 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.601 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.601 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.601 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.601 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.602 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.602 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.602 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.602 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.602 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.602 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.602 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.602 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.603 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.603 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.603 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.603 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.603 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.603 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.603 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.603 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.604 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.604 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.604 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.604 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.604 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.604 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.604 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.604 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.605 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.605 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.605 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.605 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.605 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.605 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.605 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.605 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.606 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.606 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.606 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.606 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.606 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.606 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.606 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.606 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.607 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.607 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.607 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.607 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.607 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.607 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.607 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.607 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.608 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.608 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.608 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.608 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.608 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.608 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.608 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.608 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.608 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.609 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.609 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.609 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.609 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.609 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.609 INFO:teuthology.orchestra.run.smithi044.stdout: "mark_out_threshold": { 2024-05-27T09:51:02.609 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mark_out_threshold", 2024-05-27T09:51:02.609 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.610 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.610 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.610 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2419200", 2024-05-27T09:51:02.610 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.610 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.610 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.610 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T09:51:02.611 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.611 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.611 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.611 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.611 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_name": { 2024-05-27T09:51:02.611 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_name", 2024-05-27T09:51:02.611 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.611 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.612 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.612 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "device_health_metrics", 2024-05-27T09:51:02.612 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.612 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.612 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.612 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T09:51:02.612 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.612 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.613 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.613 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.613 INFO:teuthology.orchestra.run.smithi044.stdout: "retention_period": { 2024-05-27T09:51:02.613 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retention_period", 2024-05-27T09:51:02.613 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.613 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.613 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.613 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15552000", 2024-05-27T09:51:02.614 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.614 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.614 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.614 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain device health metrics", 2024-05-27T09:51:02.614 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.614 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.614 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.614 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.615 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_frequency": { 2024-05-27T09:51:02.615 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_frequency", 2024-05-27T09:51:02.615 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.615 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.615 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.615 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-05-27T09:51:02.615 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.615 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.616 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.616 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T09:51:02.616 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.616 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.616 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.616 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.616 INFO:teuthology.orchestra.run.smithi044.stdout: "self_heal": { 2024-05-27T09:51:02.616 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "self_heal", 2024-05-27T09:51:02.617 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.617 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.617 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.617 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.617 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.617 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.617 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.617 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T09:51:02.618 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.618 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.618 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.618 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.618 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:02.618 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:02.618 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.619 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.619 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.619 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:02.619 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.619 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.619 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.619 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T09:51:02.620 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.620 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.620 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.620 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.620 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.620 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.620 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.620 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.620 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.621 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.621 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.621 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.621 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.621 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.621 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.621 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.621 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.622 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.622 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_threshold": { 2024-05-27T09:51:02.622 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_threshold", 2024-05-27T09:51:02.622 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.622 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.622 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.622 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7257600", 2024-05-27T09:51:02.622 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.623 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.623 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.623 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T09:51:02.623 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.623 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.623 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.623 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.623 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.623 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.624 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.624 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "diskprediction_local", 2024-05-27T09:51:02.624 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.624 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.624 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.624 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.624 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.624 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.625 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.625 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.625 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.625 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.625 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.625 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.625 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.626 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.626 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.626 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.626 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.626 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.626 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.626 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.626 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.626 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.627 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.627 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.627 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.627 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.627 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.627 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.627 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.627 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.628 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.628 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.628 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.628 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.628 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.628 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.628 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.628 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.629 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.629 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.629 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.629 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.629 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.629 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.629 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.629 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.630 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.630 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.630 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.630 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.630 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.630 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.630 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.630 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.631 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.631 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.631 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.631 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.631 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.631 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.631 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.632 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.632 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.632 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.632 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.632 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.632 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.632 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.632 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.633 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.633 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.633 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.633 INFO:teuthology.orchestra.run.smithi044.stdout: "predict_interval": { 2024-05-27T09:51:02.633 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predict_interval", 2024-05-27T09:51:02.633 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.633 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.633 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.633 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-05-27T09:51:02.634 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.634 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.634 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.634 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.634 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.634 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.634 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.635 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.635 INFO:teuthology.orchestra.run.smithi044.stdout: "predictor_model": { 2024-05-27T09:51:02.635 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predictor_model", 2024-05-27T09:51:02.635 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.635 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.635 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.635 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "prophetstor", 2024-05-27T09:51:02.635 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.636 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.636 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.636 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.636 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.636 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.636 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.636 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.636 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:02.636 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:02.637 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.637 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.637 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.637 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:02.637 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.637 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.637 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.637 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.638 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.638 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.638 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.638 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.638 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.638 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.638 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.638 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.639 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.639 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.639 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.639 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.639 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.639 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.639 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.639 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.639 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.640 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.640 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.640 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.640 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.640 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "influx", 2024-05-27T09:51:02.640 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": false, 2024-05-27T09:51:02.640 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "influxdb python module not found", 2024-05-27T09:51:02.640 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.641 INFO:teuthology.orchestra.run.smithi044.stdout: "batch_size": { 2024-05-27T09:51:02.641 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "batch_size", 2024-05-27T09:51:02.641 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.641 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.641 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.641 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5000", 2024-05-27T09:51:02.641 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.641 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.642 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.642 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T09:51:02.642 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.642 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.642 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.642 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.642 INFO:teuthology.orchestra.run.smithi044.stdout: "database": { 2024-05-27T09:51:02.642 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "database", 2024-05-27T09:51:02.643 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.643 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.643 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.643 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "ceph", 2024-05-27T09:51:02.643 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.643 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.643 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.643 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T09:51:02.644 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.644 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.644 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.644 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.644 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": { 2024-05-27T09:51:02.644 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hostname", 2024-05-27T09:51:02.644 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.644 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.645 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.645 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.645 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.645 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.645 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.645 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server hostname", 2024-05-27T09:51:02.645 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.645 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.645 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.646 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.646 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:02.646 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:02.646 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.646 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.646 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.646 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "30", 2024-05-27T09:51:02.647 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "5", 2024-05-27T09:51:02.647 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.647 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.647 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T09:51:02.647 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.647 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.647 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.647 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.648 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.648 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.648 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.648 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.648 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.648 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.648 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.648 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.649 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.649 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.649 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.649 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.649 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.649 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.649 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.649 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.650 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.650 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.650 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.650 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.650 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.650 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.650 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.651 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.651 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.651 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.651 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.651 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.651 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.651 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.651 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.652 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.652 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.652 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.652 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.652 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.652 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.652 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.652 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.653 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.653 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.653 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.653 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.653 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.653 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.653 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.653 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.654 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.654 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.654 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.654 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.654 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.654 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.654 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.655 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.655 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.655 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.655 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.655 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.655 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.655 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.655 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.656 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.656 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.656 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.656 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.656 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.656 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.656 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.656 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.656 INFO:teuthology.orchestra.run.smithi044.stdout: "password": { 2024-05-27T09:51:02.657 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "password", 2024-05-27T09:51:02.657 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.657 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.657 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.657 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.657 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.657 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.657 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.658 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "password of InfluxDB server user", 2024-05-27T09:51:02.658 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.658 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.658 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.658 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.658 INFO:teuthology.orchestra.run.smithi044.stdout: "port": { 2024-05-27T09:51:02.658 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "port", 2024-05-27T09:51:02.658 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.659 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.659 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.659 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8086", 2024-05-27T09:51:02.659 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.659 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.659 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.659 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server port", 2024-05-27T09:51:02.659 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.660 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.660 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.660 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.660 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.660 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.660 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.660 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.660 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.661 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.661 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.661 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.661 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.661 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.661 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.661 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.661 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.661 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.662 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-05-27T09:51:02.662 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-05-27T09:51:02.662 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.662 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.662 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.662 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "false", 2024-05-27T09:51:02.662 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.663 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.663 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.663 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T09:51:02.663 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.663 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.663 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.663 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.663 INFO:teuthology.orchestra.run.smithi044.stdout: "threads": { 2024-05-27T09:51:02.663 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threads", 2024-05-27T09:51:02.664 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.664 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.664 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.664 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:02.664 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:02.664 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "32", 2024-05-27T09:51:02.664 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.664 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T09:51:02.665 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.665 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.665 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.665 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.665 INFO:teuthology.orchestra.run.smithi044.stdout: "username": { 2024-05-27T09:51:02.665 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "username", 2024-05-27T09:51:02.665 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.665 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.666 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.666 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.666 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.666 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.666 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.666 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "username of InfluxDB server user", 2024-05-27T09:51:02.666 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.666 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.667 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.667 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.667 INFO:teuthology.orchestra.run.smithi044.stdout: "verify_ssl": { 2024-05-27T09:51:02.667 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "verify_ssl", 2024-05-27T09:51:02.667 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.667 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.667 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.667 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "true", 2024-05-27T09:51:02.668 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.668 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.668 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.668 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T09:51:02.668 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.668 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.668 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.668 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.669 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.669 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.669 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.669 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "insights", 2024-05-27T09:51:02.669 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.669 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.669 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.669 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.670 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.670 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.670 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.670 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.670 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.670 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.670 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.670 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.671 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.671 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.671 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.671 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.671 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.671 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.671 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.671 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.672 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.672 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.672 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.672 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.672 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.672 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.672 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.672 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.673 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.673 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.673 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.673 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.673 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.673 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.673 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.673 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.674 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.674 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.674 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.674 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.674 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.674 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.674 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.674 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.675 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.675 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.675 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.675 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.675 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.675 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.675 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.675 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.676 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.676 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.676 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.676 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.676 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.676 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.676 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.676 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.677 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.677 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.677 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.677 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.677 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.677 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.677 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.677 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.677 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.678 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.678 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.678 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.678 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.678 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.678 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.678 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.679 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.679 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.679 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.679 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.679 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.679 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.679 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.679 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.679 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.680 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.680 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.680 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.680 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.680 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.680 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "iostat", 2024-05-27T09:51:02.680 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.680 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.681 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.681 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.681 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.681 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.681 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.681 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.681 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.681 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.682 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.682 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.682 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.682 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.682 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.682 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.682 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.682 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.683 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.683 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.683 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.683 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.683 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.683 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.683 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.683 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.684 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.684 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.684 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.684 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.684 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.684 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.684 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.684 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.684 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.685 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.685 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.685 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.685 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.685 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.685 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.685 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.686 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.686 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.686 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.686 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.686 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.686 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.686 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.686 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.687 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.687 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.687 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.687 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.687 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.687 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.687 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.687 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.687 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.688 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.688 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.688 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.688 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.688 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.688 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.688 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.688 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.689 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.689 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.689 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.689 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.689 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.689 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.689 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.689 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.689 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.690 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.690 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.690 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.690 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.690 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.690 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.690 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.691 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.691 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.691 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.691 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.691 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.691 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.691 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.691 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "k8sevents", 2024-05-27T09:51:02.692 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.692 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.692 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.692 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_event_retention_days": { 2024-05-27T09:51:02.692 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ceph_event_retention_days", 2024-05-27T09:51:02.692 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.692 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.692 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.692 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7", 2024-05-27T09:51:02.693 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.693 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.693 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.693 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T09:51:02.693 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.693 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.693 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.693 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.694 INFO:teuthology.orchestra.run.smithi044.stdout: "config_check_secs": { 2024-05-27T09:51:02.694 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_check_secs", 2024-05-27T09:51:02.694 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.694 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.694 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.694 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:02.694 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "10", 2024-05-27T09:51:02.694 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.695 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.695 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T09:51:02.695 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.695 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.695 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.695 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.695 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.696 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.696 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.696 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.696 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.696 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.696 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.696 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.696 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.697 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.697 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.697 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.697 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.697 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.697 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.697 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.697 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.698 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.698 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.698 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.698 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.698 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.698 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.698 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.698 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.699 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.699 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.699 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.699 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.699 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.699 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.699 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.699 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.699 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.700 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.700 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.700 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.700 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.700 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.700 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.700 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.700 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.701 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.701 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.701 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.701 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.701 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.701 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.701 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.701 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.702 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.702 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.702 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.702 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.702 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.702 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.702 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.702 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.703 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.703 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.703 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.703 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.703 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.703 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.703 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.703 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.703 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.704 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.704 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.704 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.704 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.704 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.704 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.704 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.704 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.705 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.705 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.705 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.705 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.705 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.705 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.705 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.705 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.706 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.706 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.706 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.706 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.706 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "localpool", 2024-05-27T09:51:02.706 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.706 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.707 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.707 INFO:teuthology.orchestra.run.smithi044.stdout: "failure_domain": { 2024-05-27T09:51:02.707 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "failure_domain", 2024-05-27T09:51:02.707 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.707 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.707 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.707 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "host", 2024-05-27T09:51:02.707 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.707 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.708 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.708 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "failure domain for any created local pool", 2024-05-27T09:51:02.708 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T09:51:02.708 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.708 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.708 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.708 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.708 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.709 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.709 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.709 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.709 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.709 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.709 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.709 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.709 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.709 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.710 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.710 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.710 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.710 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.710 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.710 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.710 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.710 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.711 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.711 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.711 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.711 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.711 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.711 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.711 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.711 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.712 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.712 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.712 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.712 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.712 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.712 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.712 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.712 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.713 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.713 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.713 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.713 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.713 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.713 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.713 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.713 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.713 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.714 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.714 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.714 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.714 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.714 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.714 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.714 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.714 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.715 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.715 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.715 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.715 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.715 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.715 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.715 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.715 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.716 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.716 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.716 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.716 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.716 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.716 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.716 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.716 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.717 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.717 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.717 INFO:teuthology.orchestra.run.smithi044.stdout: "min_size": { 2024-05-27T09:51:02.717 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_size", 2024-05-27T09:51:02.717 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.717 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.717 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.717 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.718 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.718 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.718 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.718 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default min_size for any created local pool", 2024-05-27T09:51:02.718 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T09:51:02.718 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.718 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.718 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.719 INFO:teuthology.orchestra.run.smithi044.stdout: "num_rep": { 2024-05-27T09:51:02.719 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "num_rep", 2024-05-27T09:51:02.719 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.719 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.719 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.719 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3", 2024-05-27T09:51:02.719 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.720 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.720 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.720 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default replica count for any created local pool", 2024-05-27T09:51:02.720 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.720 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.720 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.720 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.720 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_num": { 2024-05-27T09:51:02.720 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_num", 2024-05-27T09:51:02.721 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.721 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.721 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.721 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "128", 2024-05-27T09:51:02.721 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.721 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.721 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.722 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T09:51:02.722 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.722 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.722 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.722 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.722 INFO:teuthology.orchestra.run.smithi044.stdout: "prefix": { 2024-05-27T09:51:02.722 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prefix", 2024-05-27T09:51:02.722 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.723 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.723 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.723 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.723 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.723 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.723 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.723 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name prefix for any created local pool", 2024-05-27T09:51:02.723 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.723 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.724 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.724 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.724 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.724 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.724 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.724 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.724 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.724 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.725 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.725 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.725 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.725 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.725 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.725 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.725 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.725 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.726 INFO:teuthology.orchestra.run.smithi044.stdout: "subtree": { 2024-05-27T09:51:02.726 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "subtree", 2024-05-27T09:51:02.726 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.726 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.726 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.726 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "rack", 2024-05-27T09:51:02.726 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.726 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.727 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.727 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T09:51:02.727 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T09:51:02.727 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.727 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.727 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.727 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.727 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.727 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.728 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mds_autoscaler", 2024-05-27T09:51:02.728 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.728 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.728 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.728 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.728 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.728 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.728 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.729 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.729 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.729 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.729 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.729 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.729 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.729 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.729 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.729 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.730 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.730 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.730 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.730 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.730 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.730 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.730 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.730 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.731 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.731 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.731 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.731 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.731 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.731 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.731 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.731 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.731 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.732 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.732 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.732 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.732 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.732 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.732 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.732 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.732 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.736 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.736 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.736 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.736 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.736 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.736 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.736 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.736 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.737 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.737 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.737 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.737 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.737 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.737 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.737 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.737 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.738 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.738 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.738 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.738 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.738 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.738 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.738 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.738 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.739 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirroring", 2024-05-27T09:51:02.739 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.739 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.739 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.739 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.739 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.739 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.739 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.740 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.740 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.740 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.740 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.740 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.740 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.740 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.740 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.741 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.741 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.741 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.741 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.741 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.741 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.741 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.741 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.742 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.742 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.742 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.742 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.742 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.742 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.742 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.742 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.743 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.743 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.743 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.743 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.743 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.743 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.743 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.743 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.744 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.744 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.744 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.744 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.744 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.744 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.744 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.744 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.745 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.745 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.745 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.745 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.745 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.745 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.745 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.745 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.746 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.746 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.746 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.746 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.746 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.746 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.746 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.746 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.747 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.747 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.747 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.747 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.747 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.747 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.747 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.747 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.748 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.748 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.748 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.748 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.748 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.748 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.748 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.748 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.748 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.749 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.749 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.749 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.749 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.749 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.749 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.749 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.749 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.750 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.750 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.750 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "nfs", 2024-05-27T09:51:02.750 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.750 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.750 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.750 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.750 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.751 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.751 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.751 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.751 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.751 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.751 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.751 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.751 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.752 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.752 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.752 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.752 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.752 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.752 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.752 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.752 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.753 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.753 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.753 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.753 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.753 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.753 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.753 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.753 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.753 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.754 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.754 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.754 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.754 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.754 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.754 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.754 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.754 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.755 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.755 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.755 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.755 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.755 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.755 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.755 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.756 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.756 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.756 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.756 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.756 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.756 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.756 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.756 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.757 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.757 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.757 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.757 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.757 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.757 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.757 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.757 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.757 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.758 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.758 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.758 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.758 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.758 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.758 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.758 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.758 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.759 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.759 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.759 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.759 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.759 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.759 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.759 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.759 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.760 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.760 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.760 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.760 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.760 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.760 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.760 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.760 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.761 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.761 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.761 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.761 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.761 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-05-27T09:51:02.761 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.761 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.761 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.761 INFO:teuthology.orchestra.run.smithi044.stdout: "fail_fs": { 2024-05-27T09:51:02.762 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "fail_fs", 2024-05-27T09:51:02.762 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.762 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.762 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.762 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.762 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.762 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.762 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.763 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T09:51:02.763 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.763 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.763 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.763 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.763 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.763 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.763 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.764 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.764 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.764 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.764 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.764 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.764 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.764 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.764 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.765 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.765 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.765 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.765 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.765 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.765 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.765 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.765 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.766 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.766 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.766 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.766 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.766 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.766 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.766 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.767 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.767 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.767 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.767 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.767 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.767 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.767 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.768 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.768 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.768 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.768 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.768 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.768 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.768 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.768 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.768 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.769 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.769 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.769 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.769 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.769 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.769 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.769 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.769 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.769 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.770 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.770 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.770 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.770 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.770 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.770 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.770 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.771 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.771 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.771 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.771 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.771 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.771 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.771 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.771 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.772 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.772 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.772 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.772 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.772 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator": { 2024-05-27T09:51:02.772 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-05-27T09:51:02.772 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.772 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.772 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.773 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.773 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.773 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.773 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.773 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2024-05-27T09:51:02.773 INFO:teuthology.orchestra.run.smithi044.stdout: "rook", 2024-05-27T09:51:02.773 INFO:teuthology.orchestra.run.smithi044.stdout: "test_orchestrator" 2024-05-27T09:51:02.773 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.774 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Orchestrator backend", 2024-05-27T09:51:02.774 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.774 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.774 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.774 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.774 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.774 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.774 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.775 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.775 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.775 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.775 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.775 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.775 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.775 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.775 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.776 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.776 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.776 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.776 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.776 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.776 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.776 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_perf_query", 2024-05-27T09:51:02.776 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.777 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.777 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.777 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.777 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.777 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.777 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.777 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.777 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.777 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.778 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.778 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.778 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.778 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.778 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.778 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.778 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.779 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.779 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.779 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.779 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.779 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.779 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.779 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.779 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.779 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.780 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.780 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.780 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.780 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.780 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.780 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.780 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.780 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.781 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.781 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.781 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.781 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.781 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.781 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.781 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.781 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.782 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.782 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.782 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.782 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.782 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.782 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.782 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.782 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.783 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.783 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.783 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.783 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.783 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.783 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.783 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.783 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.783 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.784 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.784 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.784 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.784 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.784 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.784 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.784 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.784 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.785 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.785 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.785 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.785 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.785 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.785 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.785 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.785 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.786 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.786 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.786 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.786 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.786 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.786 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.786 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.787 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.787 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.787 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.787 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.787 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.787 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.787 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.787 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.787 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_support", 2024-05-27T09:51:02.788 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.788 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.788 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.788 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.788 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.788 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.788 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.788 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.789 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.789 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.789 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.789 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.789 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.789 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.789 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.789 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.790 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.790 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.790 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.790 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.790 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.790 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.790 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.790 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.791 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.791 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.791 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.791 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.791 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.791 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.791 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.792 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.792 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.792 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.792 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.792 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.792 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.792 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.792 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.793 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.793 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.793 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.793 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.793 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.793 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.793 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.793 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.794 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.794 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.794 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.794 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.794 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.794 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.794 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.795 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.795 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.795 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.795 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.795 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.795 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.795 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.796 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.796 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.796 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.796 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.796 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.796 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.796 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.796 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.797 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.797 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.797 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.797 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.797 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.797 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.797 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.797 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.798 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.798 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.798 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.798 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.798 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.798 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.798 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.798 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.799 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.799 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.799 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.799 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.799 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.799 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_autoscaler", 2024-05-27T09:51:02.799 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.799 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.800 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.800 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.800 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.800 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.800 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.800 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.800 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.800 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.801 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.801 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.801 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.801 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.801 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.801 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.801 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.801 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.802 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.802 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.802 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.802 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.802 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.802 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.802 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.802 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.803 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.803 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.803 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.803 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.803 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.803 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.803 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.804 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.804 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.804 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.804 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.804 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.804 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.804 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.804 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.805 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.805 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.805 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.805 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.805 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.805 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.805 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.805 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.806 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.806 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.806 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.806 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.806 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.806 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.806 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.807 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.807 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.807 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.807 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.807 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.807 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.807 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.808 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.808 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.808 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.808 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.808 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.808 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.808 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.808 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.809 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.809 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.809 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:02.809 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:02.809 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.809 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.809 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.810 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:02.810 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.810 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.810 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.810 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.810 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.810 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.810 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.811 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.811 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.811 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.811 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.811 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.811 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.811 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.811 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.812 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.812 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.812 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.812 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.812 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.812 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.812 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.812 INFO:teuthology.orchestra.run.smithi044.stdout: "threshold": { 2024-05-27T09:51:02.813 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threshold", 2024-05-27T09:51:02.813 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:02.813 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.813 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.813 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-05-27T09:51:02.813 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1.0", 2024-05-27T09:51:02.813 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.813 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.814 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "scaling threshold", 2024-05-27T09:51:02.814 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T09:51:02.814 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.814 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.814 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.814 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.814 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.814 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.815 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "progress", 2024-05-27T09:51:02.815 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.815 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.815 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.815 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_pg_recovery_event": { 2024-05-27T09:51:02.815 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_pg_recovery_event", 2024-05-27T09:51:02.815 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.815 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.816 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.816 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.816 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.816 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.816 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.816 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T09:51:02.816 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.817 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.817 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.817 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.817 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-05-27T09:51:02.817 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-05-27T09:51:02.817 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.817 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.817 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.818 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.818 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.818 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.818 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.818 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.818 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.818 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.818 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.819 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.819 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.819 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.819 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.819 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.819 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.819 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.819 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.820 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.820 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.820 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.820 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.820 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.820 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.820 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.820 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.821 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.821 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.821 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.821 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.821 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.821 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.821 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.822 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.822 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.822 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.822 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.822 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.822 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.822 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.822 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.823 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.823 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.823 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.823 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.823 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.823 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.823 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.823 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.824 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.824 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.824 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.824 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.824 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.824 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.824 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.824 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.825 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.825 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.825 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.825 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.825 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.825 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.825 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.826 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.826 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.826 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.826 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.826 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.826 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.826 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.826 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.827 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.827 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.827 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.827 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.827 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.827 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.827 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.827 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.828 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.828 INFO:teuthology.orchestra.run.smithi044.stdout: "max_completed_events": { 2024-05-27T09:51:02.828 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_completed_events", 2024-05-27T09:51:02.828 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.828 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.828 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.828 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "50", 2024-05-27T09:51:02.828 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.829 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.829 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.829 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "number of past completed events to remember", 2024-05-27T09:51:02.829 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.829 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.829 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.829 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.829 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:02.830 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:02.830 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:02.830 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.830 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.830 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:02.830 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.830 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.830 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.831 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long the module is going to sleep", 2024-05-27T09:51:02.831 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.831 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.831 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.831 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.831 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.831 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.831 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.831 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.832 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.832 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.832 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.832 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.832 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.832 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.832 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.832 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.833 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.833 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.833 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.833 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.833 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.833 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus", 2024-05-27T09:51:02.833 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.833 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.834 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.834 INFO:teuthology.orchestra.run.smithi044.stdout: "cache": { 2024-05-27T09:51:02.834 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cache", 2024-05-27T09:51:02.834 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.834 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.834 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.834 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.835 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.835 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.835 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.835 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.835 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.835 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.835 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.835 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.836 INFO:teuthology.orchestra.run.smithi044.stdout: "exclude_perf_counters": { 2024-05-27T09:51:02.836 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "exclude_perf_counters", 2024-05-27T09:51:02.836 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.836 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.836 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.836 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.836 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.837 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.837 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.837 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T09:51:02.837 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T09:51:02.837 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.837 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.837 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.837 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.837 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.838 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.838 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.838 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.838 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.838 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.838 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.838 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.838 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.839 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.839 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.839 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.839 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.839 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.840 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.840 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.840 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.840 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.840 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.840 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.840 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.840 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.841 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.841 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.841 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.841 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.841 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.841 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.841 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.841 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.842 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.842 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.842 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.842 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.842 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.842 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.842 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.842 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.843 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.843 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.843 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.843 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.843 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.843 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.843 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.843 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.844 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.844 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.844 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.844 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.844 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.844 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.844 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.844 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.845 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.845 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.845 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.845 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.845 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.845 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.845 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.845 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.846 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.846 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.846 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.846 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.846 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.846 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.846 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.846 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools": { 2024-05-27T09:51:02.847 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools", 2024-05-27T09:51:02.847 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.847 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.847 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.847 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.847 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.847 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.847 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T09:51:02.848 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T09:51:02.849 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.849 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.849 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.849 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300", 2024-05-27T09:51:02.849 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.849 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.849 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.849 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.849 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.850 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.850 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.850 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.850 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_interval": { 2024-05-27T09:51:02.850 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_interval", 2024-05-27T09:51:02.850 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:02.850 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.850 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.851 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15.0", 2024-05-27T09:51:02.851 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.851 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.851 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.851 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.851 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.851 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.851 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.852 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.852 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-05-27T09:51:02.852 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-05-27T09:51:02.852 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.852 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.852 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.852 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-05-27T09:51:02.852 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.853 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.853 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.853 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T09:51:02.853 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.853 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.853 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.853 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.853 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-05-27T09:51:02.854 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-05-27T09:51:02.854 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.854 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.854 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.854 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "9283", 2024-05-27T09:51:02.854 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.854 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.854 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.855 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T09:51:02.855 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.855 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.855 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.855 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.855 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.855 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.855 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.856 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.856 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.856 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.856 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.856 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.856 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.856 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.856 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.857 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.857 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.857 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.857 INFO:teuthology.orchestra.run.smithi044.stdout: "stale_cache_strategy": { 2024-05-27T09:51:02.857 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stale_cache_strategy", 2024-05-27T09:51:02.857 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.857 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.858 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.858 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "log", 2024-05-27T09:51:02.858 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.858 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.858 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.858 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.858 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.858 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.859 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.859 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.859 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-05-27T09:51:02.859 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-05-27T09:51:02.859 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.859 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.859 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.859 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "default", 2024-05-27T09:51:02.860 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.860 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.860 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.860 INFO:teuthology.orchestra.run.smithi044.stdout: "default", 2024-05-27T09:51:02.860 INFO:teuthology.orchestra.run.smithi044.stdout: "error" 2024-05-27T09:51:02.860 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.860 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.861 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.861 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.861 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.861 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.861 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-05-27T09:51:02.861 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-05-27T09:51:02.861 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.861 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.862 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.862 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-05-27T09:51:02.862 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-05-27T09:51:02.862 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-05-27T09:51:02.862 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.862 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.862 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.862 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.863 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.863 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.863 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.863 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.863 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.863 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2024-05-27T09:51:02.863 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.864 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.864 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.864 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.864 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.864 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.864 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.864 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.865 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.865 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.865 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.865 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.865 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.865 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.865 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.866 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.866 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.866 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.866 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.866 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.866 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.866 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.867 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.867 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.867 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.867 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.867 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.867 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.867 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.867 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.868 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.868 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.868 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.868 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.868 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.868 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.868 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.868 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.869 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.869 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.869 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.869 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.869 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.869 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.869 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.870 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.870 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.870 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.870 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.870 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.870 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.870 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.870 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.871 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.871 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.871 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.871 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.871 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.871 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.871 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.872 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.872 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.872 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.872 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.872 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.872 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.872 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.872 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.873 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.873 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.873 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.873 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.873 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.873 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_snap_create": { 2024-05-27T09:51:02.873 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_snap_create", 2024-05-27T09:51:02.873 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.874 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.874 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.874 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:02.874 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.874 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.874 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.877 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.878 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.878 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.878 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.878 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.878 INFO:teuthology.orchestra.run.smithi044.stdout: "mirror_snapshot_schedule": { 2024-05-27T09:51:02.878 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T09:51:02.878 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.879 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.879 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.879 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.879 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.879 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.879 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.879 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.879 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.879 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.880 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.880 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.880 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.880 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.880 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.880 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.880 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.880 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.881 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.881 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.881 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.881 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.881 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.881 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.881 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.881 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.881 INFO:teuthology.orchestra.run.smithi044.stdout: "trash_purge_schedule": { 2024-05-27T09:51:02.882 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "trash_purge_schedule", 2024-05-27T09:51:02.882 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.882 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.882 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.882 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.882 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.882 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.882 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.883 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.883 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.883 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.883 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.883 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.883 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.883 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.883 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.884 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "restful", 2024-05-27T09:51:02.884 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.884 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.884 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.884 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_auth": { 2024-05-27T09:51:02.884 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_auth", 2024-05-27T09:51:02.884 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.884 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.885 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.885 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.885 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.885 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.885 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.885 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.885 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.885 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.886 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.886 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.886 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-05-27T09:51:02.886 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-05-27T09:51:02.886 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.886 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.886 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.886 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.887 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.887 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.887 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.887 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.887 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.887 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.887 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.887 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.888 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.888 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.888 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.888 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.888 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.888 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.888 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.888 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.888 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.889 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.889 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.889 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.889 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.889 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.889 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.889 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.889 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.890 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.890 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.890 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.890 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.890 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.890 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.890 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.890 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.891 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.891 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.891 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.891 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.891 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.891 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.891 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.892 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.892 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.892 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.892 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.892 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.892 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.892 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.892 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.893 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.893 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.893 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.893 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.893 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.893 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.893 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.893 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.894 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.894 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.894 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.894 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.894 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.894 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.894 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.894 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.895 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.895 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.895 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.895 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.895 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.895 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.895 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.895 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.896 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.896 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.896 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.896 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.896 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.896 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.896 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-05-27T09:51:02.896 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-05-27T09:51:02.897 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.897 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.897 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.897 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.897 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.897 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.897 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.897 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.898 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.898 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.898 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.898 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.898 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-05-27T09:51:02.898 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-05-27T09:51:02.898 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.898 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.899 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.899 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.899 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.899 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.899 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.899 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.899 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.899 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.900 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.900 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.900 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.900 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.900 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.900 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.900 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.900 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.901 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.901 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.901 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.901 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.901 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.901 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.901 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.902 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.902 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.902 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.902 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.902 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rgw", 2024-05-27T09:51:02.902 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.902 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.902 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.903 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.903 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.903 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.903 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.903 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.903 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.903 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.903 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.904 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.904 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.904 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.904 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.904 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.904 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.904 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.904 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.905 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.905 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.905 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.905 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.905 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.905 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.905 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.905 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.905 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.906 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.906 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.906 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.906 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.906 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.906 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.906 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.906 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.907 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.907 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.907 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.907 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.907 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.907 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.907 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.907 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.908 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.908 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.908 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.908 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.908 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.908 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.908 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.908 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.908 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.908 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.909 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.909 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.909 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.909 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.909 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.909 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.909 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.909 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.909 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.910 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.910 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.910 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.910 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.910 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.910 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.910 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.911 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.911 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.911 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.911 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.911 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.911 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.911 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.911 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.911 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.912 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.912 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.912 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.912 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.912 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.912 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.912 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.912 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.913 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.913 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.913 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.913 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rook", 2024-05-27T09:51:02.913 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.913 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.913 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.913 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.914 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.914 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.914 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.914 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.914 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.914 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.914 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.914 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.915 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.915 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.915 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.915 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.915 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.915 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.915 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.916 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.916 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.916 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.916 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.916 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.916 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.916 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.916 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.917 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.917 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.917 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.917 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.917 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.917 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.917 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.917 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.918 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.918 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.918 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.918 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.918 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.918 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.918 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.918 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.921 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.921 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.921 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.921 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.921 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.921 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.921 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.921 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.921 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.922 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.922 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.922 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.922 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.922 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.922 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.922 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.922 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.923 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.923 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.923 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.923 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.923 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.923 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.923 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.924 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.924 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.924 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.924 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.924 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.924 INFO:teuthology.orchestra.run.smithi044.stdout: "storage_class": { 2024-05-27T09:51:02.924 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "storage_class", 2024-05-27T09:51:02.925 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.925 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.925 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.925 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "local", 2024-05-27T09:51:02.925 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.925 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.925 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.925 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T09:51:02.926 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.926 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.926 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.926 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.926 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.926 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.926 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.926 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "selftest", 2024-05-27T09:51:02.927 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.927 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.927 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.927 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.927 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.927 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.927 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.927 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.928 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.928 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.928 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.928 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.928 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.928 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.928 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.928 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.929 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.929 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.929 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.929 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.929 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.929 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.929 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.929 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.930 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.930 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.930 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.930 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.930 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.930 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.930 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.930 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.931 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.931 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.931 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.931 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.931 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.931 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.931 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.931 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.932 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.932 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.932 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.932 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.932 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.932 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.932 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.932 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.933 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.933 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.933 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.933 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.933 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.933 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.933 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.933 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.934 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.934 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.934 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.934 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.934 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.934 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.934 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.934 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.935 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.935 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.935 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.935 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.935 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.935 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.935 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.936 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.936 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.936 INFO:teuthology.orchestra.run.smithi044.stdout: "roption1": { 2024-05-27T09:51:02.936 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption1", 2024-05-27T09:51:02.936 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.936 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.936 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.936 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.936 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.937 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.937 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.937 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.937 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.937 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.937 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.937 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.937 INFO:teuthology.orchestra.run.smithi044.stdout: "roption2": { 2024-05-27T09:51:02.938 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption2", 2024-05-27T09:51:02.938 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.938 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.938 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.938 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "xyz", 2024-05-27T09:51:02.938 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.938 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.938 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.939 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.939 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.939 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.939 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.939 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.939 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption1": { 2024-05-27T09:51:02.939 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption1", 2024-05-27T09:51:02.939 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.939 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.940 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.940 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.940 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.940 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.940 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.940 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.940 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.940 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.941 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.941 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.941 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption2": { 2024-05-27T09:51:02.941 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption2", 2024-05-27T09:51:02.941 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.941 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.941 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.941 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.942 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.942 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.942 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.942 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.942 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.942 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.942 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.942 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.943 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption3": { 2024-05-27T09:51:02.943 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption3", 2024-05-27T09:51:02.943 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:02.943 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.943 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.943 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.943 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.943 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.943 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.944 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.944 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.944 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.944 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.944 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.944 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption4": { 2024-05-27T09:51:02.944 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption4", 2024-05-27T09:51:02.944 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.945 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.945 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.945 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.945 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.945 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.945 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.945 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.945 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.946 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.946 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.946 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.946 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption5": { 2024-05-27T09:51:02.946 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption5", 2024-05-27T09:51:02.946 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.946 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.946 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.947 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.947 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.947 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.947 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.947 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.947 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.947 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.947 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.948 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.948 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption6": { 2024-05-27T09:51:02.948 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption6", 2024-05-27T09:51:02.948 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.948 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.948 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.948 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.948 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.949 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.949 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.949 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.949 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.949 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.949 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.949 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.949 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption7": { 2024-05-27T09:51:02.950 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption7", 2024-05-27T09:51:02.950 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.950 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.950 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.950 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.950 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:02.950 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "42", 2024-05-27T09:51:02.950 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.951 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.951 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.951 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.951 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.951 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.951 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.951 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.951 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.952 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.952 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.952 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.952 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.952 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.952 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.952 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.952 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.953 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.953 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.953 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.953 INFO:teuthology.orchestra.run.smithi044.stdout: "testkey": { 2024-05-27T09:51:02.953 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testkey", 2024-05-27T09:51:02.953 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.953 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.953 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.954 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.954 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.954 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.954 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.954 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.954 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.954 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.954 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.955 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.955 INFO:teuthology.orchestra.run.smithi044.stdout: "testlkey": { 2024-05-27T09:51:02.955 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testlkey", 2024-05-27T09:51:02.955 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.955 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.955 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.955 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.955 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.956 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.956 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.956 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.956 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.956 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.956 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.956 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.956 INFO:teuthology.orchestra.run.smithi044.stdout: "testnewline": { 2024-05-27T09:51:02.957 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testnewline", 2024-05-27T09:51:02.957 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.957 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.957 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.957 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.957 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.957 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.957 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.957 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.958 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.958 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.958 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.958 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.958 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.958 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.958 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.959 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smb", 2024-05-27T09:51:02.959 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.959 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.959 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.959 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.959 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.959 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.959 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.960 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.960 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.960 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.960 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.960 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.960 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.960 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.960 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.961 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.961 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.961 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.961 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.961 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.961 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.961 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.961 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.962 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.962 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.962 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.962 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.962 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.962 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.962 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.962 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.963 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.963 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.963 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.963 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.963 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.963 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.963 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.963 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.963 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.964 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.964 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.964 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.964 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.964 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.964 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.964 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.965 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.965 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.965 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.965 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.965 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.965 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.965 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.965 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.966 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.966 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.966 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.966 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.966 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.966 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.966 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.966 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.967 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.967 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.967 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.967 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.967 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.967 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.967 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.967 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.968 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.968 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.968 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.968 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.968 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.968 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.968 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.968 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.969 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.969 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.969 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.969 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.969 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.969 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.969 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.969 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.970 INFO:teuthology.orchestra.run.smithi044.stdout: "update_orchestration": { 2024-05-27T09:51:02.970 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "update_orchestration", 2024-05-27T09:51:02.970 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.970 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.970 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:02.970 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:02.970 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.970 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.971 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.971 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T09:51:02.971 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.971 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.971 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.971 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.971 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.971 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.971 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.972 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2024-05-27T09:51:02.972 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.972 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.972 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.972 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_m_granularity": { 2024-05-27T09:51:02.972 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_m_granularity", 2024-05-27T09:51:02.972 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.972 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.973 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.973 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.973 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.973 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.973 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.973 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T09:51:02.973 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.973 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.974 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.974 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.974 INFO:teuthology.orchestra.run.smithi044.stdout: "dump_on_update": { 2024-05-27T09:51:02.974 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dump_on_update", 2024-05-27T09:51:02.974 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.974 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.974 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.974 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.975 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.975 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.975 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.975 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "dump database to debug log on update", 2024-05-27T09:51:02.975 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.975 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.975 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.975 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.976 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.976 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.976 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.976 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.976 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.976 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.976 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.976 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.977 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.977 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.977 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.977 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.977 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.977 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.977 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.977 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.977 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.978 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.978 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.978 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.978 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.978 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.978 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.978 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.978 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.979 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.979 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.979 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.979 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.979 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.979 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.979 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.979 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.980 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.980 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.980 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.980 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.980 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.980 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.980 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.980 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.981 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.981 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.981 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.981 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.981 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.981 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.981 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.981 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.981 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.982 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.982 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.982 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.982 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.982 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.982 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.982 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.983 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.983 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.983 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.983 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.983 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.983 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.983 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.983 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.984 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.984 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.984 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.984 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.984 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.984 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.984 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.984 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.985 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.985 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.985 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.985 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.985 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.985 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.985 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.985 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.986 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.986 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.986 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.986 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.986 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.986 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.986 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stats", 2024-05-27T09:51:02.986 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.987 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.987 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.987 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.987 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.987 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.987 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.987 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.987 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.987 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.988 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.988 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.988 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.988 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.988 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.988 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.988 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.988 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.989 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.989 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.989 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.989 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.989 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.989 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.989 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:02.989 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:02.990 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.990 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.990 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.990 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.990 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.990 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.990 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.990 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.991 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.991 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.991 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.991 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.991 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:02.991 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:02.991 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.991 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.992 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.992 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:02.992 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.992 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.992 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.992 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.992 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.992 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.993 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:02.993 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:02.993 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:02.993 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:02.993 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.993 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.993 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.993 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.994 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.994 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:02.994 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:02.994 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:02.994 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.994 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.994 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:02.994 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.995 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.995 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.995 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.995 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.995 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.995 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.995 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.995 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:02.996 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:02.996 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:02.996 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:02.996 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.996 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:02.996 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.996 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.996 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:02.997 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:02.997 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:02.997 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:02.997 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:02.997 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.997 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:02.997 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:02.997 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:02.997 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "status", 2024-05-27T09:51:02.998 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:02.998 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:02.998 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:02.998 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:02.998 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:02.998 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:02.998 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:02.998 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:02.999 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:02.999 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:02.999 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:02.999 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:02.999 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:02.999 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:02.999 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:02.999 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.000 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.000 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.000 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.000 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.000 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.000 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.000 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.000 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.001 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.001 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.001 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.001 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.001 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.001 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.001 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.001 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.002 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.002 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.002 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.002 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.002 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.002 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.002 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.002 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.003 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.003 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.003 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.003 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.003 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.003 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.003 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.003 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.004 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.004 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.004 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.004 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.004 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.004 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.004 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.004 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.005 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.005 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.005 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.005 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.005 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.005 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.005 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.006 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.006 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.006 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.006 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.006 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.006 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.006 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.006 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.006 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.007 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.007 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.007 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.007 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.007 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.007 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.007 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.007 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.008 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.008 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.008 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.008 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.008 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.008 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.008 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.008 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.008 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.009 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.009 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telegraf", 2024-05-27T09:51:03.009 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.009 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.009 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.009 INFO:teuthology.orchestra.run.smithi044.stdout: "address": { 2024-05-27T09:51:03.009 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "address", 2024-05-27T09:51:03.009 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.010 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.010 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.010 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T09:51:03.010 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.010 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.010 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.010 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.010 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.011 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15", 2024-05-27T09:51:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.012 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.013 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.013 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.013 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.013 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.013 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.013 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.013 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.013 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.014 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.014 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.014 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.014 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.014 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.014 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.014 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.016 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.016 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.016 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.016 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.016 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.016 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.016 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.016 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.017 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.017 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.017 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.017 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.017 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.017 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.017 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.017 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.018 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.018 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.018 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.018 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.018 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.018 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.018 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.018 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.019 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.019 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.019 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.019 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.019 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.019 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.019 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.019 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.020 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.020 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.020 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.020 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.020 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.020 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.020 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.020 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.021 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.021 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.021 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.021 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.021 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.021 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.021 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.021 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.022 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.022 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.022 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.022 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.022 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.022 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.022 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.022 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.023 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.023 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.023 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.023 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.023 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.023 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.023 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.023 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.024 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telemetry", 2024-05-27T09:51:03.024 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.024 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.024 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.024 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_basic": { 2024-05-27T09:51:03.024 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_basic", 2024-05-27T09:51:03.024 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.024 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.024 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.025 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.025 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.025 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.025 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.025 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T09:51:03.025 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.025 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.025 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.026 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.026 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_crash": { 2024-05-27T09:51:03.026 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_crash", 2024-05-27T09:51:03.026 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.026 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.026 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.026 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.026 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.027 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.027 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.027 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T09:51:03.027 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.027 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.027 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.027 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.027 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_device": { 2024-05-27T09:51:03.028 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_device", 2024-05-27T09:51:03.028 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.028 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.028 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.028 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.028 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.028 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.028 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.029 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T09:51:03.029 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.029 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.029 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.029 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.029 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_ident": { 2024-05-27T09:51:03.029 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_ident", 2024-05-27T09:51:03.029 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.029 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.030 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.030 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.030 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.030 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.030 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.030 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T09:51:03.030 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.030 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.031 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.031 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.031 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_perf": { 2024-05-27T09:51:03.031 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_perf", 2024-05-27T09:51:03.031 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.031 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.031 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.031 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.032 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.032 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.032 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.032 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T09:51:03.032 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.032 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.032 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.032 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.033 INFO:teuthology.orchestra.run.smithi044.stdout: "contact": { 2024-05-27T09:51:03.033 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "contact", 2024-05-27T09:51:03.033 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.033 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.033 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.033 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.033 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.033 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.034 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.034 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.034 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.034 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.034 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.034 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.034 INFO:teuthology.orchestra.run.smithi044.stdout: "description": { 2024-05-27T09:51:03.034 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "description", 2024-05-27T09:51:03.035 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.035 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.035 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.035 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.035 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.035 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.035 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.035 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.036 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.036 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.036 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.036 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.036 INFO:teuthology.orchestra.run.smithi044.stdout: "device_url": { 2024-05-27T09:51:03.036 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_url", 2024-05-27T09:51:03.036 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.036 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.037 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.037 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T09:51:03.037 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.037 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.037 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.037 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.037 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.037 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.038 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.038 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.038 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-05-27T09:51:03.038 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-05-27T09:51:03.038 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.038 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.038 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.038 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.039 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.039 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.039 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.039 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.039 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.039 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.039 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.039 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.039 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:03.040 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:03.040 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.040 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.040 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.040 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "24", 2024-05-27T09:51:03.040 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "8", 2024-05-27T09:51:03.040 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.040 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.041 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.041 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.041 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.041 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.041 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.041 INFO:teuthology.orchestra.run.smithi044.stdout: "last_opt_revision": { 2024-05-27T09:51:03.041 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "last_opt_revision", 2024-05-27T09:51:03.041 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.041 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.042 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.042 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1", 2024-05-27T09:51:03.042 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.042 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.042 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.042 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.042 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.042 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.043 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.043 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.043 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard": { 2024-05-27T09:51:03.043 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard", 2024-05-27T09:51:03.043 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.043 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.043 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.043 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.044 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.044 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.044 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.044 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.044 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.044 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.044 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.044 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.045 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard_description": { 2024-05-27T09:51:03.045 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard_description", 2024-05-27T09:51:03.045 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.045 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.045 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.045 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.045 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.045 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.046 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.046 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.046 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.046 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.046 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.046 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.046 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.046 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.047 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.047 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.047 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.047 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.047 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.047 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.047 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.047 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.047 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.048 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.048 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.048 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.048 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.048 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.048 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.048 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.048 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.048 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.049 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.049 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.049 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.049 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.049 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.049 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.049 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.049 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.050 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.050 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.050 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.050 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.050 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.050 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.050 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.050 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.051 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.051 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.051 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.051 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.051 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.051 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.051 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.051 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.051 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.052 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.052 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.052 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.052 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.052 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.052 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.052 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.052 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.053 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.053 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.053 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.053 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.053 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.053 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.053 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.053 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.054 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.054 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.054 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.054 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.054 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.054 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.054 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.054 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.055 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.055 INFO:teuthology.orchestra.run.smithi044.stdout: "organization": { 2024-05-27T09:51:03.055 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "organization", 2024-05-27T09:51:03.055 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.055 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.055 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.055 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.055 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: "proxy": { 2024-05-27T09:51:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "proxy", 2024-05-27T09:51:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.058 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.058 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.058 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.058 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.058 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.058 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.058 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.058 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.059 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.059 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.059 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.059 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.059 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.059 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.059 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.059 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.060 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.060 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.060 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.060 INFO:teuthology.orchestra.run.smithi044.stdout: "url": { 2024-05-27T09:51:03.060 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url", 2024-05-27T09:51:03.060 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.060 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.060 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.061 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T09:51:03.061 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.061 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.061 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.061 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.061 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.061 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.061 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.062 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.062 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.062 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.062 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.062 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "test_orchestrator", 2024-05-27T09:51:03.062 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.062 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.062 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.063 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.063 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.063 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.063 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.063 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.063 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.063 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.063 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.063 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.064 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.064 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.064 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.064 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.064 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.064 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.064 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.064 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.065 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.065 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.065 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.065 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.065 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.065 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.065 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.065 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.066 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.066 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.066 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.066 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.066 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.066 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.066 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.066 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.067 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.067 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.067 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.067 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.067 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.067 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.067 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.067 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.068 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.068 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.068 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.068 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.068 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.068 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.068 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.068 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.068 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.069 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.069 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.069 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.069 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.069 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.069 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.069 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.069 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.070 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.070 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.070 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.070 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.070 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.070 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.070 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.070 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.071 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.071 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.071 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.071 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.071 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.071 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.071 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.071 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.071 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.072 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.072 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.072 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.072 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.072 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.072 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.072 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.072 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.073 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.073 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.073 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.073 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.073 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2024-05-27T09:51:03.073 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.073 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.073 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.074 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.074 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.074 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.074 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.074 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.074 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.074 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.074 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.075 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.075 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.075 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.075 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.075 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.075 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.075 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.075 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.076 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.076 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.076 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.076 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.076 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.076 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.076 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.076 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.077 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.077 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.077 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.077 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.077 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.077 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.077 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.077 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.078 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.078 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.078 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.078 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.078 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.078 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.078 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.078 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.081 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.082 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.082 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.082 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.082 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.082 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.082 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.082 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.082 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.083 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.083 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.083 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.083 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.083 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.083 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.083 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.083 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.084 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.084 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.084 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.084 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.084 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.084 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.084 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.084 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.085 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.085 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.085 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.085 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.085 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.085 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_clones": { 2024-05-27T09:51:03.085 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_clones", 2024-05-27T09:51:03.085 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.086 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.086 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.086 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4", 2024-05-27T09:51:03.086 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.086 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.086 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.086 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T09:51:03.087 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.087 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.087 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.087 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.087 INFO:teuthology.orchestra.run.smithi044.stdout: "periodic_async_work": { 2024-05-27T09:51:03.087 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "periodic_async_work", 2024-05-27T09:51:03.087 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.087 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.088 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.088 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.088 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.088 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.088 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.088 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Periodically check for async work", 2024-05-27T09:51:03.088 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.088 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.089 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.089 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.089 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_delay": { 2024-05-27T09:51:03.089 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_delay", 2024-05-27T09:51:03.089 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.089 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.089 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.089 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.090 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.090 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.090 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.090 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T09:51:03.090 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.090 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.090 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.090 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.091 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_no_wait": { 2024-05-27T09:51:03.091 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T09:51:03.091 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.091 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.091 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.091 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.091 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.091 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.092 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.092 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T09:51:03.092 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.092 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.092 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.092 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.092 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.092 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.093 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.093 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.093 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.093 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.093 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.093 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.093 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.093 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.094 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.094 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.094 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.094 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.094 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.094 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.094 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.094 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix", 2024-05-27T09:51:03.095 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.095 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.095 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.095 INFO:teuthology.orchestra.run.smithi044.stdout: "discovery_interval": { 2024-05-27T09:51:03.095 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "discovery_interval", 2024-05-27T09:51:03.095 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:03.095 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.095 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.096 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "100", 2024-05-27T09:51:03.096 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.096 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.096 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.096 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.096 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.096 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.096 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.097 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.097 INFO:teuthology.orchestra.run.smithi044.stdout: "identifier": { 2024-05-27T09:51:03.097 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "identifier", 2024-05-27T09:51:03.097 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.097 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.097 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.097 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.097 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.098 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.098 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.098 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.098 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.098 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.098 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.098 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.099 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:03.099 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:03.099 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.099 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.099 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.099 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:03.099 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.099 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.100 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.100 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.100 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.100 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.100 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.100 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.100 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.100 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.101 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.101 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.101 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.101 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.101 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.101 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.101 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.101 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.102 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.102 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.102 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.102 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.102 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.102 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.102 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.102 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.103 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.103 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.103 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.103 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.103 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.103 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.103 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.103 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.104 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.104 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.104 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.104 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.104 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.104 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.104 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.104 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.105 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.105 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.105 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.105 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.105 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.105 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.105 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.106 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.106 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.106 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.106 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.106 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.106 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.106 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.106 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.107 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.107 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.107 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.107 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.107 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.107 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.107 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.107 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.108 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.108 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.108 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.108 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.108 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.108 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.108 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.108 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.109 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.109 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.109 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.109 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.109 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.109 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.109 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.110 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.110 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.110 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.110 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.110 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.110 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.110 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.111 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.111 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.111 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.111 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.111 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.111 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_host": { 2024-05-27T09:51:03.111 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_host", 2024-05-27T09:51:03.111 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.112 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.112 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.112 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.112 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.112 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.112 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.112 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.112 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.113 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.113 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.113 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.113 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_port": { 2024-05-27T09:51:03.113 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_port", 2024-05-27T09:51:03.113 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.113 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.113 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.114 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10051", 2024-05-27T09:51:03.114 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.114 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.114 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.114 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.114 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.114 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.115 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.115 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.115 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_sender": { 2024-05-27T09:51:03.115 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_sender", 2024-05-27T09:51:03.115 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.115 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.115 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.115 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T09:51:03.116 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.116 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.116 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.116 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.116 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.116 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.116 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.116 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.117 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.117 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.117 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:51:03.117 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.117 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.117 INFO:teuthology.orchestra.run.smithi044.stdout: "gid": 25066, 2024-05-27T09:51:03.117 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "z", 2024-05-27T09:51:03.117 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr_features": 4540701547738038271, 2024-05-27T09:51:03.117 INFO:teuthology.orchestra.run.smithi044.stdout: "available_modules": [ 2024-05-27T09:51:03.118 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.118 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alerts", 2024-05-27T09:51:03.118 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.118 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.118 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.118 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:03.118 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:03.119 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.119 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.119 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.119 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:03.119 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.119 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.119 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.119 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T09:51:03.119 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.120 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.120 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.120 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.120 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.120 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.120 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.120 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.120 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.121 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.121 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.121 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.121 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.121 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.121 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.121 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.121 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.122 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.122 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.122 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.122 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.122 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.122 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.122 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.122 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.123 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.123 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.123 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.123 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.123 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.123 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.123 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.123 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.124 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.124 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.124 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.124 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.124 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.124 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.124 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.124 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.124 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.125 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.125 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.125 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.125 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.125 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.125 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.125 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.125 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.125 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.126 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.126 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.126 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.126 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.126 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.126 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.126 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.126 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.127 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.127 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.127 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.127 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.127 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.127 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.127 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.127 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.128 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.128 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.128 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.128 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.128 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.128 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.128 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.128 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_destination": { 2024-05-27T09:51:03.129 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_destination", 2024-05-27T09:51:03.129 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.129 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.129 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.129 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.129 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.129 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.129 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.129 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email address to send alerts to", 2024-05-27T09:51:03.130 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.130 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.130 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.130 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.130 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_from_name": { 2024-05-27T09:51:03.130 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_from_name", 2024-05-27T09:51:03.130 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.131 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.131 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.131 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "Ceph", 2024-05-27T09:51:03.131 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.131 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.131 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.131 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email From: name", 2024-05-27T09:51:03.131 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.131 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.132 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.132 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.132 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_host": { 2024-05-27T09:51:03.132 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_host", 2024-05-27T09:51:03.132 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.132 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.132 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.132 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.133 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.133 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.133 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.133 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP server", 2024-05-27T09:51:03.133 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.133 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.133 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.133 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.134 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_password": { 2024-05-27T09:51:03.134 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_password", 2024-05-27T09:51:03.134 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.134 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.134 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.134 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.134 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.134 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.135 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.135 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Password to authenticate with", 2024-05-27T09:51:03.135 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.135 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.135 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.135 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.135 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_port": { 2024-05-27T09:51:03.135 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_port", 2024-05-27T09:51:03.136 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.136 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.136 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.136 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "465", 2024-05-27T09:51:03.136 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.136 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.136 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.136 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP port", 2024-05-27T09:51:03.137 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.137 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.137 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.137 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.137 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_sender": { 2024-05-27T09:51:03.137 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_sender", 2024-05-27T09:51:03.137 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.137 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.138 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.138 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.138 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.138 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.138 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.138 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP envelope sender", 2024-05-27T09:51:03.138 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.138 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.139 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.139 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.139 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_ssl": { 2024-05-27T09:51:03.139 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_ssl", 2024-05-27T09:51:03.139 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.139 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.139 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.139 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.140 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.140 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.140 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.140 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T09:51:03.140 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.140 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.140 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.140 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.140 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_user": { 2024-05-27T09:51:03.141 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_user", 2024-05-27T09:51:03.141 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.141 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.141 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.141 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.141 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.141 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.141 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.142 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "User to authenticate as", 2024-05-27T09:51:03.142 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.142 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.142 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.142 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.142 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.142 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.143 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.143 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.143 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.143 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.143 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.143 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.143 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.143 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.143 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.144 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.144 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.144 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.144 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.144 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.144 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.144 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "balancer", 2024-05-27T09:51:03.144 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.145 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.145 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.145 INFO:teuthology.orchestra.run.smithi044.stdout: "active": { 2024-05-27T09:51:03.145 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "active", 2024-05-27T09:51:03.145 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.145 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.145 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.145 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.146 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.146 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.146 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.146 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T09:51:03.146 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.146 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.146 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.146 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.147 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_time": { 2024-05-27T09:51:03.147 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_time", 2024-05-27T09:51:03.147 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.147 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.147 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.147 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0000", 2024-05-27T09:51:03.147 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.147 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.148 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.148 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T09:51:03.148 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T09:51:03.148 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.148 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.148 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.148 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_weekday": { 2024-05-27T09:51:03.148 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_weekday", 2024-05-27T09:51:03.149 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:03.149 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.149 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.149 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.149 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-05-27T09:51:03.149 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-05-27T09:51:03.149 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.149 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T09:51:03.150 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T09:51:03.150 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.150 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.150 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.150 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_max_iterations": { 2024-05-27T09:51:03.150 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_max_iterations", 2024-05-27T09:51:03.150 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:03.150 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.151 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.151 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "25", 2024-05-27T09:51:03.151 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:03.151 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "250", 2024-05-27T09:51:03.151 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.151 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T09:51:03.151 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.151 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.151 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.152 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.152 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_metrics": { 2024-05-27T09:51:03.152 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_metrics", 2024-05-27T09:51:03.152 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.152 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.152 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.152 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T09:51:03.152 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.153 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.153 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.153 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T09:51:03.153 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T09:51:03.153 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.153 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.153 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.154 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_step": { 2024-05-27T09:51:03.154 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_step", 2024-05-27T09:51:03.154 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:03.154 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.154 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.154 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.5", 2024-05-27T09:51:03.154 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0.001", 2024-05-27T09:51:03.154 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "0.999", 2024-05-27T09:51:03.154 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.155 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "aggressiveness of optimization", 2024-05-27T09:51:03.155 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T09:51:03.155 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.155 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.155 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.155 INFO:teuthology.orchestra.run.smithi044.stdout: "end_time": { 2024-05-27T09:51:03.155 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_time", 2024-05-27T09:51:03.155 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.156 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.156 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.156 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2359", 2024-05-27T09:51:03.156 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.156 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.156 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.156 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T09:51:03.156 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T09:51:03.157 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.157 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.157 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.157 INFO:teuthology.orchestra.run.smithi044.stdout: "end_weekday": { 2024-05-27T09:51:03.157 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_weekday", 2024-05-27T09:51:03.157 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:03.157 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.158 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.158 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.158 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-05-27T09:51:03.158 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-05-27T09:51:03.158 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.158 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T09:51:03.158 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T09:51:03.158 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.159 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.159 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.159 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.159 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.159 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.159 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.159 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.159 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.160 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.160 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.160 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.160 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.160 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.160 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.160 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.160 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.161 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.161 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.161 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.161 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.161 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.161 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.161 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.161 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.162 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.162 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.162 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.162 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.162 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.162 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.162 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.162 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.163 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.163 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.163 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.163 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.163 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.163 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.163 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.163 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.164 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.164 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.164 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.164 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.164 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.164 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.164 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.164 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.165 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.165 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.165 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.165 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.165 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.165 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.165 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.165 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.166 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.166 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.166 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.166 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.166 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.166 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.166 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.166 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.167 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.167 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.167 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.167 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.167 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.167 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.167 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.167 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.168 INFO:teuthology.orchestra.run.smithi044.stdout: "min_score": { 2024-05-27T09:51:03.168 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_score", 2024-05-27T09:51:03.168 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:03.168 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.168 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.168 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.168 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.168 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.168 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.169 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T09:51:03.169 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.169 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.169 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.169 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.169 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-05-27T09:51:03.169 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-05-27T09:51:03.169 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.170 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.170 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.170 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "upmap", 2024-05-27T09:51:03.170 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.170 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.170 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.170 INFO:teuthology.orchestra.run.smithi044.stdout: "crush-compat", 2024-05-27T09:51:03.170 INFO:teuthology.orchestra.run.smithi044.stdout: "none", 2024-05-27T09:51:03.171 INFO:teuthology.orchestra.run.smithi044.stdout: "read", 2024-05-27T09:51:03.171 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap", 2024-05-27T09:51:03.171 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap-read" 2024-05-27T09:51:03.171 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.171 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Balancer mode", 2024-05-27T09:51:03.171 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.171 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.171 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.172 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.172 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_ids": { 2024-05-27T09:51:03.172 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_ids", 2024-05-27T09:51:03.172 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.172 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.172 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.172 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.172 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.173 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.173 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.173 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T09:51:03.173 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.173 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.173 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.173 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.173 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:03.174 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:03.174 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.174 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.174 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.174 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:03.174 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.174 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.175 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.175 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T09:51:03.175 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.175 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.175 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.175 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.175 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.175 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.176 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.176 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.176 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.176 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.176 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.176 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.176 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.176 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.176 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.177 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.177 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.177 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.177 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_deviation": { 2024-05-27T09:51:03.177 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_deviation", 2024-05-27T09:51:03.177 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.177 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.177 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.177 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:03.178 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:03.178 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.178 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.178 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T09:51:03.178 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T09:51:03.178 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.178 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.178 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.179 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_optimizations": { 2024-05-27T09:51:03.179 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_optimizations", 2024-05-27T09:51:03.179 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:03.179 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.179 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.179 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:03.179 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.179 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.180 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.180 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T09:51:03.180 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.180 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.180 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.180 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.180 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.180 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.181 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.181 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm", 2024-05-27T09:51:03.181 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.181 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.181 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.181 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_down_multiplier": { 2024-05-27T09:51:03.181 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_down_multiplier", 2024-05-27T09:51:03.181 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:03.181 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.182 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.182 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-05-27T09:51:03.182 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.182 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.182 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.182 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T09:51:03.182 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.182 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.183 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.183 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.183 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_refresh_rate": { 2024-05-27T09:51:03.183 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_refresh_rate", 2024-05-27T09:51:03.183 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.183 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.183 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.183 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "20", 2024-05-27T09:51:03.184 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.184 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.184 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.184 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T09:51:03.184 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.184 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.184 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.184 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.185 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_starting_port": { 2024-05-27T09:51:03.185 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_starting_port", 2024-05-27T09:51:03.185 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.185 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.185 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.185 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4721", 2024-05-27T09:51:03.185 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.185 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.186 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.186 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T09:51:03.186 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.186 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.186 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.186 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.186 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_ptrace": { 2024-05-27T09:51:03.186 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_ptrace", 2024-05-27T09:51:03.186 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.187 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.187 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.187 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.187 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.187 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.187 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.187 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T09:51:03.187 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T09:51:03.187 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.188 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.188 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.188 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_interval": { 2024-05-27T09:51:03.188 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_interval", 2024-05-27T09:51:03.188 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.188 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.188 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.188 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:03.188 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.189 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.189 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.189 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T09:51:03.189 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.189 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.189 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.189 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.189 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_memory_target_ratio": { 2024-05-27T09:51:03.189 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T09:51:03.190 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:03.190 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.190 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.190 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.7", 2024-05-27T09:51:03.190 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.190 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.190 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.190 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T09:51:03.190 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.191 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.191 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.191 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.191 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm_log_destination": { 2024-05-27T09:51:03.191 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm_log_destination", 2024-05-27T09:51:03.191 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.191 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.191 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.192 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.192 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.192 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.192 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.192 INFO:teuthology.orchestra.run.smithi044.stdout: "file", 2024-05-27T09:51:03.192 INFO:teuthology.orchestra.run.smithi044.stdout: "file,syslog", 2024-05-27T09:51:03.192 INFO:teuthology.orchestra.run.smithi044.stdout: "syslog" 2024-05-27T09:51:03.192 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.193 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T09:51:03.193 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.193 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.193 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.193 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.193 INFO:teuthology.orchestra.run.smithi044.stdout: "cgroups_split": { 2024-05-27T09:51:03.193 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cgroups_split", 2024-05-27T09:51:03.193 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.194 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.194 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.194 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.194 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.194 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.194 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.194 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T09:51:03.194 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.195 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.195 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.195 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.195 INFO:teuthology.orchestra.run.smithi044.stdout: "config_checks_enabled": { 2024-05-27T09:51:03.195 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_checks_enabled", 2024-05-27T09:51:03.195 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.195 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.196 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.196 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.196 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.196 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.196 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.196 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T09:51:03.196 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.196 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.197 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.197 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.197 INFO:teuthology.orchestra.run.smithi044.stdout: "config_dashboard": { 2024-05-27T09:51:03.197 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_dashboard", 2024-05-27T09:51:03.197 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.197 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.197 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.197 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.198 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.198 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.198 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.198 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T09:51:03.198 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.198 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.199 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.199 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.199 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_alertmanager": { 2024-05-27T09:51:03.199 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_alertmanager", 2024-05-27T09:51:03.199 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.199 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.199 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.199 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T09:51:03.200 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.200 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.200 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.200 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:03.200 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.200 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.200 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.200 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.201 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_base": { 2024-05-27T09:51:03.201 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_base", 2024-05-27T09:51:03.201 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.201 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.201 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.201 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T09:51:03.201 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.201 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.202 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.202 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Container image name, without the tag", 2024-05-27T09:51:03.202 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.202 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.202 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.202 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.202 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_elasticsearch": { 2024-05-27T09:51:03.202 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_elasticsearch", 2024-05-27T09:51:03.203 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.203 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.203 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.203 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T09:51:03.203 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.203 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.203 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.203 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "elasticsearch container image", 2024-05-27T09:51:03.203 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.204 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.204 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.204 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.204 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_grafana": { 2024-05-27T09:51:03.204 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_grafana", 2024-05-27T09:51:03.204 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.204 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.204 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.205 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T09:51:03.205 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.205 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.205 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.205 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:03.205 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.205 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.205 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.206 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.206 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_haproxy": { 2024-05-27T09:51:03.206 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_haproxy", 2024-05-27T09:51:03.206 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.206 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.206 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.206 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T09:51:03.207 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.207 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.207 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.207 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "HAproxy container image", 2024-05-27T09:51:03.207 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.207 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.207 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.207 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.208 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_agent": { 2024-05-27T09:51:03.208 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_agent", 2024-05-27T09:51:03.208 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.208 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.208 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.208 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T09:51:03.208 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.209 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.209 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.209 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger agent container image", 2024-05-27T09:51:03.209 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.209 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.209 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.209 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.209 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_collector": { 2024-05-27T09:51:03.210 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_collector", 2024-05-27T09:51:03.210 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.210 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.210 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.210 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T09:51:03.210 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.210 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.210 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.211 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger collector container image", 2024-05-27T09:51:03.211 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.211 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.211 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.211 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.211 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_query": { 2024-05-27T09:51:03.211 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_query", 2024-05-27T09:51:03.211 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.212 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.212 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.212 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T09:51:03.212 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.212 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.212 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.212 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger query container image", 2024-05-27T09:51:03.213 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.213 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.213 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.213 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.213 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_keepalived": { 2024-05-27T09:51:03.213 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_keepalived", 2024-05-27T09:51:03.213 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.213 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.214 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.214 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T09:51:03.214 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.214 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.214 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.214 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Keepalived container image", 2024-05-27T09:51:03.214 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.214 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.215 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.215 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.215 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_loki": { 2024-05-27T09:51:03.215 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_loki", 2024-05-27T09:51:03.215 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.215 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.215 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.215 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T09:51:03.216 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.216 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.216 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.216 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Loki container image", 2024-05-27T09:51:03.216 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.216 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.216 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.216 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.216 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_node_exporter": { 2024-05-27T09:51:03.217 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_node_exporter", 2024-05-27T09:51:03.217 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.217 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.217 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.217 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T09:51:03.217 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.217 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.217 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.218 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:03.218 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.218 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.218 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.218 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.218 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_nvmeof": { 2024-05-27T09:51:03.218 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_nvmeof", 2024-05-27T09:51:03.218 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.219 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.219 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.219 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T09:51:03.219 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.219 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.219 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.219 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Nvme-of container image", 2024-05-27T09:51:03.219 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.220 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.220 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.220 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.220 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_prometheus": { 2024-05-27T09:51:03.220 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_prometheus", 2024-05-27T09:51:03.220 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.220 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.221 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.221 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T09:51:03.221 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.221 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.221 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.221 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:03.221 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.221 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.221 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.222 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.222 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_promtail": { 2024-05-27T09:51:03.222 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_promtail", 2024-05-27T09:51:03.222 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.222 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.222 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.222 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T09:51:03.222 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.223 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.223 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.223 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Promtail container image", 2024-05-27T09:51:03.223 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.223 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.223 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.223 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.223 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_samba": { 2024-05-27T09:51:03.223 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_samba", 2024-05-27T09:51:03.224 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.224 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.224 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.224 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T09:51:03.224 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.224 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.224 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.224 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Samba/SMB container image", 2024-05-27T09:51:03.225 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.225 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.225 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.225 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.225 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_snmp_gateway": { 2024-05-27T09:51:03.225 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_snmp_gateway", 2024-05-27T09:51:03.225 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.225 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.226 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.226 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T09:51:03.226 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.226 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.226 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.226 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SNMP Gateway container image", 2024-05-27T09:51:03.226 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.226 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.227 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.227 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.227 INFO:teuthology.orchestra.run.smithi044.stdout: "container_init": { 2024-05-27T09:51:03.227 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_init", 2024-05-27T09:51:03.227 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.227 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.227 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.227 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.227 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.228 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.228 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.228 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T09:51:03.228 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.228 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.228 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.228 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.229 INFO:teuthology.orchestra.run.smithi044.stdout: "daemon_cache_timeout": { 2024-05-27T09:51:03.229 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "daemon_cache_timeout", 2024-05-27T09:51:03.229 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.229 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.229 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.229 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:03.229 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.229 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.230 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.230 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T09:51:03.230 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.230 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.230 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.230 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.230 INFO:teuthology.orchestra.run.smithi044.stdout: "default_cephadm_command_timeout": { 2024-05-27T09:51:03.230 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T09:51:03.231 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.231 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.231 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.231 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "900", 2024-05-27T09:51:03.231 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.231 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.231 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.232 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T09:51:03.232 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.232 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.232 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.232 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.232 INFO:teuthology.orchestra.run.smithi044.stdout: "default_registry": { 2024-05-27T09:51:03.232 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_registry", 2024-05-27T09:51:03.232 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.233 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.233 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.233 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io", 2024-05-27T09:51:03.233 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.233 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.233 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.233 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T09:51:03.233 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.234 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.234 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.234 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.234 INFO:teuthology.orchestra.run.smithi044.stdout: "device_cache_timeout": { 2024-05-27T09:51:03.234 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_cache_timeout", 2024-05-27T09:51:03.234 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.234 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.234 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.235 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1800", 2024-05-27T09:51:03.235 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.235 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.235 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.235 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache device inventory", 2024-05-27T09:51:03.235 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.235 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.235 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.236 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.236 INFO:teuthology.orchestra.run.smithi044.stdout: "device_enhanced_scan": { 2024-05-27T09:51:03.236 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_enhanced_scan", 2024-05-27T09:51:03.236 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.236 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.236 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.236 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.236 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.237 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.237 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.237 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T09:51:03.237 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.237 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.237 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.237 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.237 INFO:teuthology.orchestra.run.smithi044.stdout: "facts_cache_timeout": { 2024-05-27T09:51:03.237 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "facts_cache_timeout", 2024-05-27T09:51:03.238 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.238 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.238 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.238 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:03.238 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.238 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.238 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.238 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache host facts data", 2024-05-27T09:51:03.239 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.239 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.239 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.239 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.239 INFO:teuthology.orchestra.run.smithi044.stdout: "grafana_dashboards_path": { 2024-05-27T09:51:03.239 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "grafana_dashboards_path", 2024-05-27T09:51:03.239 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.239 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.239 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.240 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T09:51:03.240 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.240 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.240 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.240 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T09:51:03.240 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.240 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.240 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.241 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.241 INFO:teuthology.orchestra.run.smithi044.stdout: "host_check_interval": { 2024-05-27T09:51:03.241 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "host_check_interval", 2024-05-27T09:51:03.241 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.241 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.241 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.241 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:03.241 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.242 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.242 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.242 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to perform a host check", 2024-05-27T09:51:03.242 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.242 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.242 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.242 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.242 INFO:teuthology.orchestra.run.smithi044.stdout: "hw_monitoring": { 2024-05-27T09:51:03.243 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hw_monitoring", 2024-05-27T09:51:03.243 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.243 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.243 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.243 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.243 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.243 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.243 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.244 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T09:51:03.244 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.244 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.244 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.244 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.244 INFO:teuthology.orchestra.run.smithi044.stdout: "inventory_list_all": { 2024-05-27T09:51:03.244 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "inventory_list_all", 2024-05-27T09:51:03.244 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.245 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.245 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.245 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.245 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.245 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.245 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.245 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T09:51:03.246 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.246 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.246 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.246 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.246 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.246 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.246 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.246 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.246 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.247 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.247 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.247 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.247 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.247 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.247 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.247 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.247 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.247 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.248 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.248 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.248 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.248 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.248 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.248 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.248 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.248 INFO:teuthology.orchestra.run.smithi044.stdout: "log_refresh_metadata": { 2024-05-27T09:51:03.249 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_refresh_metadata", 2024-05-27T09:51:03.249 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.249 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.249 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.249 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.249 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.249 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.249 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.249 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T09:51:03.250 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.250 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.250 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.250 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.250 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.250 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.250 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.250 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.251 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.251 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.251 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.251 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.251 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.251 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T09:51:03.251 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.251 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.251 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.252 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.252 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.252 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.252 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.252 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.252 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.252 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.252 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.253 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.253 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.253 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.253 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.253 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.253 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.253 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.253 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.253 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.254 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.254 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.254 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.254 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.254 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.254 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.254 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.254 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.255 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.255 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.255 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.255 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.255 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.255 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.255 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.255 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.256 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.256 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.256 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.256 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T09:51:03.256 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T09:51:03.256 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.256 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.257 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.257 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.257 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.257 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.257 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.257 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T09:51:03.257 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.257 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.257 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.258 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.258 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T09:51:03.258 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T09:51:03.258 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.258 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.258 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.258 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "*", 2024-05-27T09:51:03.258 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.258 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.259 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.259 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T09:51:03.259 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.259 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.259 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.259 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.259 INFO:teuthology.orchestra.run.smithi044.stdout: "max_count_per_host": { 2024-05-27T09:51:03.259 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_count_per_host", 2024-05-27T09:51:03.259 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.260 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.260 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.260 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:03.260 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.260 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.260 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.260 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of daemons per service per host", 2024-05-27T09:51:03.260 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.261 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.261 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.261 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.261 INFO:teuthology.orchestra.run.smithi044.stdout: "max_osd_draining_count": { 2024-05-27T09:51:03.261 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_osd_draining_count", 2024-05-27T09:51:03.261 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.261 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.261 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.262 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:03.262 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.262 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.262 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.262 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T09:51:03.262 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.262 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.262 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.263 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.263 INFO:teuthology.orchestra.run.smithi044.stdout: "migration_current": { 2024-05-27T09:51:03.263 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "migration_current", 2024-05-27T09:51:03.263 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.263 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.263 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.263 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.264 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.264 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.264 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.264 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "internal - do not modify", 2024-05-27T09:51:03.264 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.264 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.264 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.264 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.264 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-05-27T09:51:03.265 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-05-27T09:51:03.265 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.265 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.265 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.265 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "root", 2024-05-27T09:51:03.265 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.265 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.265 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.266 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm-package", 2024-05-27T09:51:03.266 INFO:teuthology.orchestra.run.smithi044.stdout: "root" 2024-05-27T09:51:03.266 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.266 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T09:51:03.266 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.266 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.266 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.266 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.267 INFO:teuthology.orchestra.run.smithi044.stdout: "oob_default_addr": { 2024-05-27T09:51:03.267 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "oob_default_addr", 2024-05-27T09:51:03.267 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.267 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.267 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.267 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "169.254.1.1", 2024-05-27T09:51:03.267 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.267 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.268 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.268 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T09:51:03.268 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.268 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.268 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.268 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.268 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_alerts_path": { 2024-05-27T09:51:03.269 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_alerts_path", 2024-05-27T09:51:03.269 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.269 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.269 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.269 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T09:51:03.269 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.269 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.269 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.269 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T09:51:03.270 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.270 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.270 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.270 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.270 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_insecure": { 2024-05-27T09:51:03.270 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_insecure", 2024-05-27T09:51:03.270 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.271 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.271 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.271 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.271 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.271 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.271 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.271 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T09:51:03.271 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.272 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.272 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.272 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.272 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_password": { 2024-05-27T09:51:03.272 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_password", 2024-05-27T09:51:03.272 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.272 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.272 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.272 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.273 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.273 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.273 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.273 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T09:51:03.273 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.273 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.273 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.273 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.274 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_url": { 2024-05-27T09:51:03.274 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_url", 2024-05-27T09:51:03.274 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.274 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.274 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.274 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.274 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.274 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.275 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.275 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T09:51:03.275 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.275 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.275 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.275 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.275 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_username": { 2024-05-27T09:51:03.275 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_username", 2024-05-27T09:51:03.275 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.276 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.276 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.276 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.276 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.276 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.276 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.276 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T09:51:03.276 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.276 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.277 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.277 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.277 INFO:teuthology.orchestra.run.smithi044.stdout: "secure_monitoring_stack": { 2024-05-27T09:51:03.277 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "secure_monitoring_stack", 2024-05-27T09:51:03.277 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.277 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.277 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.277 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.277 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.278 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.278 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.278 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T09:51:03.278 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.278 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.278 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.278 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.278 INFO:teuthology.orchestra.run.smithi044.stdout: "service_discovery_port": { 2024-05-27T09:51:03.279 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "service_discovery_port", 2024-05-27T09:51:03.279 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.279 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.279 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.279 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8765", 2024-05-27T09:51:03.279 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.279 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.279 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.280 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "cephadm service discovery port", 2024-05-27T09:51:03.280 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.280 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.280 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.280 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.280 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.280 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.280 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.281 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.281 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.281 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.281 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.281 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.281 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.281 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.281 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.282 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.282 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.282 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.282 INFO:teuthology.orchestra.run.smithi044.stdout: "ssh_config_file": { 2024-05-27T09:51:03.282 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssh_config_file", 2024-05-27T09:51:03.282 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.282 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.282 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.283 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.283 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.283 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.286 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.286 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T09:51:03.286 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.286 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.286 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.286 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.286 INFO:teuthology.orchestra.run.smithi044.stdout: "use_agent": { 2024-05-27T09:51:03.286 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_agent", 2024-05-27T09:51:03.287 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.287 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.287 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.287 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.287 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.287 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.287 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.287 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T09:51:03.287 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.288 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.288 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.288 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.288 INFO:teuthology.orchestra.run.smithi044.stdout: "use_repo_digest": { 2024-05-27T09:51:03.288 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_repo_digest", 2024-05-27T09:51:03.288 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.288 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.288 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.288 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.289 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.289 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.289 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.289 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T09:51:03.289 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.289 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.289 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.289 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.290 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_failed_host_check": { 2024-05-27T09:51:03.290 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_failed_host_check", 2024-05-27T09:51:03.290 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.290 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.290 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.290 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.290 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.290 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.290 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.291 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T09:51:03.291 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.291 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.291 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.291 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.291 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_daemons": { 2024-05-27T09:51:03.291 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_daemons", 2024-05-27T09:51:03.291 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.291 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.292 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.292 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.292 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.292 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.292 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.292 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T09:51:03.292 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.293 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.293 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.293 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.293 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_hosts": { 2024-05-27T09:51:03.293 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_hosts", 2024-05-27T09:51:03.293 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.293 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.293 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.294 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.294 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.294 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.294 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.294 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T09:51:03.294 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.294 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.295 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.295 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.295 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.295 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.295 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.295 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crash", 2024-05-27T09:51:03.295 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.295 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.295 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.296 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.296 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.296 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.296 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.296 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.296 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.296 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.297 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.297 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.297 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.297 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.297 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.297 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.297 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.297 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.298 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.298 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.298 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.298 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.298 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.298 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.298 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.298 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.299 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.299 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.299 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.299 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.299 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.299 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.299 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.299 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.300 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.301 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.302 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.303 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.304 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: "retain_interval": { 2024-05-27T09:51:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retain_interval", 2024-05-27T09:51:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "31536000", 2024-05-27T09:51:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.305 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T09:51:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.306 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.307 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_recent_interval": { 2024-05-27T09:51:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_recent_interval", 2024-05-27T09:51:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.308 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1209600", 2024-05-27T09:51:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T09:51:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.309 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dashboard", 2024-05-27T09:51:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.310 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T09:51:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T09:51:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.311 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T09:51:03.312 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T09:51:03.313 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.313 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.313 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.313 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.313 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.313 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.313 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.313 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.313 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T09:51:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T09:51:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.314 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.315 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_ENABLED": { 2024-05-27T09:51:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T09:51:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.316 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.317 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.317 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.317 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.317 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.317 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.317 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.317 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.317 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.318 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T09:51:03.318 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T09:51:03.318 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.318 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.318 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.318 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.318 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.318 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.319 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.319 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.319 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.319 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.319 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.319 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.319 INFO:teuthology.orchestra.run.smithi044.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T09:51:03.319 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T09:51:03.320 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.320 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.320 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.320 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.320 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.320 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.320 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.320 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.321 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.321 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.321 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.321 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.321 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T09:51:03.321 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T09:51:03.321 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.321 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.322 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.322 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.322 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.322 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.322 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.322 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.322 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.322 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.322 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.323 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.323 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T09:51:03.323 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T09:51:03.323 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.323 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.323 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.323 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.323 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.323 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.324 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.324 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.324 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.324 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.324 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.324 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.324 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T09:51:03.324 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T09:51:03.324 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.325 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.325 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.325 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.325 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.325 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.325 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.325 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.325 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.326 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.326 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.326 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.326 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T09:51:03.326 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T09:51:03.326 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.326 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.327 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.327 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.327 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.327 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.327 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.327 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.327 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.327 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.327 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.328 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.328 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T09:51:03.328 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T09:51:03.328 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.328 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.328 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.328 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.329 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.329 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.329 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.329 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.329 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.329 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.329 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.329 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.330 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T09:51:03.330 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T09:51:03.330 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.330 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.330 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.330 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.330 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.330 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.330 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.331 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.331 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.331 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.331 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.331 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.331 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T09:51:03.331 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T09:51:03.332 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.332 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.332 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.332 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.332 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.332 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.332 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.332 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.333 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.333 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.333 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.333 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.333 INFO:teuthology.orchestra.run.smithi044.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T09:51:03.333 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T09:51:03.333 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.333 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.334 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.334 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.334 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.334 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.334 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.334 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.334 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.335 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.335 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.335 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.335 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T09:51:03.335 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T09:51:03.335 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.335 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.335 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.336 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-05-27T09:51:03.336 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.336 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.336 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.336 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.336 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.336 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.336 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.337 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.337 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T09:51:03.337 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T09:51:03.337 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.337 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.337 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.337 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.337 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.338 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.338 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.338 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.338 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.338 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.338 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.338 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.338 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_URL": { 2024-05-27T09:51:03.339 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_URL", 2024-05-27T09:51:03.339 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.339 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.339 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.339 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.339 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.339 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.339 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.340 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.340 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.340 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.340 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.340 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.340 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T09:51:03.340 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T09:51:03.340 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.341 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.341 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.341 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-05-27T09:51:03.341 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.341 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.341 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.341 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.341 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.342 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.342 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.342 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.342 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T09:51:03.342 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T09:51:03.342 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.342 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.343 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.343 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.343 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.343 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.343 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.343 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.343 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.343 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.344 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.344 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.344 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T09:51:03.344 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T09:51:03.344 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.344 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.344 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.344 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.345 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.345 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.345 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.345 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.345 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.345 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.345 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.346 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.346 INFO:teuthology.orchestra.run.smithi044.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T09:51:03.346 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T09:51:03.346 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.346 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.346 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.346 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.346 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.347 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.347 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.347 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.347 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.347 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.347 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.347 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.347 INFO:teuthology.orchestra.run.smithi044.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T09:51:03.348 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T09:51:03.348 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.348 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.348 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.348 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.348 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.348 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.348 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.348 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.349 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.349 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.349 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.349 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.349 INFO:teuthology.orchestra.run.smithi044.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T09:51:03.349 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T09:51:03.349 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.349 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.349 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.350 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "[]", 2024-05-27T09:51:03.350 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.350 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.350 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.350 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.350 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.350 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.350 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.351 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.351 INFO:teuthology.orchestra.run.smithi044.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T09:51:03.351 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T09:51:03.351 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.351 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.351 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.351 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "{}", 2024-05-27T09:51:03.351 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.351 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.352 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.352 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.352 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.352 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.352 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.352 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.352 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T09:51:03.352 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T09:51:03.352 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.353 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.353 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.353 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.353 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.353 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.353 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.353 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.354 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.354 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.354 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.354 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.354 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T09:51:03.354 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T09:51:03.354 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.354 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.355 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.355 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.355 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.355 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.355 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.355 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.355 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.355 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.356 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.356 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.356 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T09:51:03.356 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T09:51:03.356 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.356 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.356 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.356 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.356 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.357 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.357 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.357 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.357 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.357 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.357 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.357 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.357 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T09:51:03.357 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T09:51:03.358 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.358 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.358 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.358 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.358 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.358 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.358 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.358 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.358 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.359 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.359 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.359 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.359 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T09:51:03.359 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T09:51:03.359 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.359 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.359 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.359 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.360 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.360 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.360 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.360 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.360 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.360 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.360 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.360 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.360 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T09:51:03.361 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T09:51:03.361 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.361 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.361 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.361 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.361 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.361 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.361 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.362 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.362 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.362 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.362 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.362 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.362 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T09:51:03.362 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T09:51:03.362 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.362 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.363 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.363 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.363 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.363 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.363 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.363 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.363 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.363 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.364 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.364 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.364 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T09:51:03.364 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T09:51:03.364 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.364 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.364 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.364 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.365 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.365 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.365 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.365 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.365 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.365 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.365 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.365 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.366 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T09:51:03.366 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T09:51:03.366 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.366 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.366 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.366 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.366 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.366 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.367 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.367 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.367 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.367 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.367 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.367 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.367 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T09:51:03.367 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T09:51:03.368 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.368 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.368 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.368 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.368 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.368 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.368 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.368 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.368 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.369 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.369 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.369 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.369 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T09:51:03.369 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T09:51:03.369 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.369 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.369 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.369 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T09:51:03.370 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.370 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.370 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.370 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.370 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.370 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.370 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.370 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.371 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T09:51:03.371 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T09:51:03.371 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.371 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.371 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.371 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:03.371 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.372 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.372 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.372 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.372 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.372 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.372 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.372 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.372 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T09:51:03.373 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T09:51:03.373 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.373 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.373 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.373 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8", 2024-05-27T09:51:03.373 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.373 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.373 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.374 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.374 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.374 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.374 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.374 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.374 INFO:teuthology.orchestra.run.smithi044.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T09:51:03.374 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T09:51:03.374 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.375 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.375 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.375 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "45", 2024-05-27T09:51:03.375 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.375 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.375 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.375 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.375 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.376 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.376 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.376 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.376 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T09:51:03.376 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T09:51:03.376 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.376 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.376 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.377 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.377 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.377 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.377 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.377 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.377 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.377 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.378 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.378 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.378 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T09:51:03.378 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T09:51:03.378 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.378 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.378 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.378 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-05-27T09:51:03.379 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.379 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.379 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.379 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.379 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.379 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.379 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.380 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.380 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T09:51:03.380 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T09:51:03.380 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.380 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.380 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.380 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.380 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.381 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.381 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.381 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.381 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.381 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.381 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.381 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.382 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T09:51:03.382 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T09:51:03.382 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.382 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.382 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.382 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.382 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.383 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.383 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.383 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.383 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.383 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.383 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.383 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.383 INFO:teuthology.orchestra.run.smithi044.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T09:51:03.383 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T09:51:03.384 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.384 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.384 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.384 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.384 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.384 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.384 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.385 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.385 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.385 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.385 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.385 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.385 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T09:51:03.385 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T09:51:03.386 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.386 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.386 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.386 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.386 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.386 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.386 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.387 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.387 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.387 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.387 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.387 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.387 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T09:51:03.387 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T09:51:03.387 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.387 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.388 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.388 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:03.388 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.388 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.388 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.388 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.388 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.388 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.389 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.389 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.389 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T09:51:03.389 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T09:51:03.389 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.389 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.389 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.389 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:03.389 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.390 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.390 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.390 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.390 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.390 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.390 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.391 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.391 INFO:teuthology.orchestra.run.smithi044.stdout: "cross_origin_url": { 2024-05-27T09:51:03.391 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cross_origin_url", 2024-05-27T09:51:03.391 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.391 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.391 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.391 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.391 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.392 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.392 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.392 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.392 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.392 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.392 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.392 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.392 INFO:teuthology.orchestra.run.smithi044.stdout: "crt_file": { 2024-05-27T09:51:03.393 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crt_file", 2024-05-27T09:51:03.393 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.393 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.393 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.393 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.393 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.393 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.393 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.394 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.394 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.394 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.394 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.394 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.394 INFO:teuthology.orchestra.run.smithi044.stdout: "debug": { 2024-05-27T09:51:03.394 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "debug", 2024-05-27T09:51:03.395 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.395 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.395 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.395 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.395 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.395 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.395 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.396 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable/disable debug options", 2024-05-27T09:51:03.396 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.396 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.396 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.396 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.396 INFO:teuthology.orchestra.run.smithi044.stdout: "jwt_token_ttl": { 2024-05-27T09:51:03.396 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "jwt_token_ttl", 2024-05-27T09:51:03.396 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.396 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.397 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.397 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "28800", 2024-05-27T09:51:03.397 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.397 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.397 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.397 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.397 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.397 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.398 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.398 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.398 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-05-27T09:51:03.398 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-05-27T09:51:03.398 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.398 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.398 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.398 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.399 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.399 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.399 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.399 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.399 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.399 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.399 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.399 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.400 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.400 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.400 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.400 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.400 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.400 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.400 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.400 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.401 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.401 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.401 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.401 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.401 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.401 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.401 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.401 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.402 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.402 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.402 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.402 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.402 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.402 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.402 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.403 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.403 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.403 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.403 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.403 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.403 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.403 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.403 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.404 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.404 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.404 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.404 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.404 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.404 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.404 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.404 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.405 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.405 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.405 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.405 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.405 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.405 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.405 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.406 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.406 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.406 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.406 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.406 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.406 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.406 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.407 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.407 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.407 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.407 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.407 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.407 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.407 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.408 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.408 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.408 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.408 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.408 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.408 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.408 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.408 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.409 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.409 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.409 INFO:teuthology.orchestra.run.smithi044.stdout: "motd": { 2024-05-27T09:51:03.409 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "motd", 2024-05-27T09:51:03.409 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.409 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.409 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.409 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.409 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.410 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.410 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.410 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "The message of the day", 2024-05-27T09:51:03.410 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.410 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.410 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.410 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.411 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect_resolve_ip_addr": { 2024-05-27T09:51:03.411 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T09:51:03.411 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.411 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.411 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.411 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.411 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.411 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.411 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.412 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.412 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.412 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.412 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.412 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.412 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-05-27T09:51:03.412 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-05-27T09:51:03.412 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.413 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.413 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.413 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-05-27T09:51:03.413 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.413 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.413 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.413 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.413 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.413 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.414 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.414 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.414 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-05-27T09:51:03.414 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-05-27T09:51:03.414 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.414 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.414 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8080", 2024-05-27T09:51:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.417 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.417 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.417 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.417 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.417 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.417 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.417 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.417 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.418 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-05-27T09:51:03.418 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-05-27T09:51:03.418 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.418 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.418 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.418 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.418 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.418 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.419 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.419 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.419 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.419 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.419 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.419 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.419 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl_server_port": { 2024-05-27T09:51:03.420 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl_server_port", 2024-05-27T09:51:03.420 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.420 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.420 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.420 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8443", 2024-05-27T09:51:03.420 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.420 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.420 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.421 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.421 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.421 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.421 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.421 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.421 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-05-27T09:51:03.421 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-05-27T09:51:03.421 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.422 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.422 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.422 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "redirect", 2024-05-27T09:51:03.422 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.422 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.422 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.422 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.422 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect" 2024-05-27T09:51:03.423 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.423 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.423 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.423 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.423 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.423 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.423 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-05-27T09:51:03.423 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-05-27T09:51:03.424 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.424 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.424 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.424 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-05-27T09:51:03.424 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-05-27T09:51:03.424 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-05-27T09:51:03.424 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.424 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.425 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.425 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.425 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.425 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.425 INFO:teuthology.orchestra.run.smithi044.stdout: "url_prefix": { 2024-05-27T09:51:03.425 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url_prefix", 2024-05-27T09:51:03.425 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.425 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.426 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.426 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.426 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.426 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.426 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.426 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.426 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.427 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.427 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.427 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.427 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.427 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.427 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.427 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2024-05-27T09:51:03.427 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.428 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.428 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.428 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_monitoring": { 2024-05-27T09:51:03.428 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_monitoring", 2024-05-27T09:51:03.428 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.428 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.428 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.428 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.429 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.429 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.429 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.429 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "monitor device health metrics", 2024-05-27T09:51:03.429 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.429 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.429 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.429 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.430 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.430 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.430 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.430 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.430 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.430 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.430 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.430 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.430 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.431 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.431 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.431 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.431 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.431 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.431 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.431 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.432 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.432 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.432 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.432 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.432 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.432 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.432 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.432 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.433 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.433 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.433 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.433 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.433 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.433 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.433 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.433 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.434 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.434 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.434 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.434 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.434 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.434 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.434 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.434 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.435 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.435 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.435 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.435 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.435 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.435 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.435 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.436 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.436 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.436 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.436 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.436 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.436 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.436 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.436 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.437 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.437 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.437 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.437 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.437 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.437 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.437 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.437 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.437 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.438 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.438 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.438 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.438 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.438 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.438 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.438 INFO:teuthology.orchestra.run.smithi044.stdout: "mark_out_threshold": { 2024-05-27T09:51:03.439 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mark_out_threshold", 2024-05-27T09:51:03.439 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.439 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.439 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.439 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2419200", 2024-05-27T09:51:03.439 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.439 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.439 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.440 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T09:51:03.440 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.440 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.440 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.440 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.440 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_name": { 2024-05-27T09:51:03.440 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_name", 2024-05-27T09:51:03.440 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.441 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.441 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.441 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "device_health_metrics", 2024-05-27T09:51:03.441 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.441 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.441 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.441 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T09:51:03.441 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.442 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.442 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.442 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.442 INFO:teuthology.orchestra.run.smithi044.stdout: "retention_period": { 2024-05-27T09:51:03.442 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retention_period", 2024-05-27T09:51:03.442 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.442 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.443 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.443 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15552000", 2024-05-27T09:51:03.443 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.443 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.443 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.443 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain device health metrics", 2024-05-27T09:51:03.443 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.443 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.444 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.444 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.444 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_frequency": { 2024-05-27T09:51:03.444 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_frequency", 2024-05-27T09:51:03.444 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.444 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.444 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.444 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-05-27T09:51:03.445 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.445 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.445 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.445 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T09:51:03.445 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.445 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.445 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.445 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.446 INFO:teuthology.orchestra.run.smithi044.stdout: "self_heal": { 2024-05-27T09:51:03.446 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "self_heal", 2024-05-27T09:51:03.446 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.446 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.446 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.446 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.446 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.447 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.447 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.447 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T09:51:03.447 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.447 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.447 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.447 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.447 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:03.448 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:03.448 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.448 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.448 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.448 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:03.448 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.448 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.448 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.449 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T09:51:03.449 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.449 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.449 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.449 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.449 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.450 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.450 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.450 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.450 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.450 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.450 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.451 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.451 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.451 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.451 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.451 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.451 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.451 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.452 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_threshold": { 2024-05-27T09:51:03.452 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_threshold", 2024-05-27T09:51:03.452 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.452 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.452 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.452 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7257600", 2024-05-27T09:51:03.452 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.452 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.453 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.453 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T09:51:03.453 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.453 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.453 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.453 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.453 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.453 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.454 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.454 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "diskprediction_local", 2024-05-27T09:51:03.454 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.454 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.454 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.454 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.454 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.454 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.455 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.455 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.455 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.455 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.455 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.455 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.455 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.455 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.456 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.456 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.456 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.456 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.456 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.456 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.456 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.456 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.457 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.457 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.457 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.457 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.457 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.457 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.457 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.457 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.457 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.458 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.458 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.458 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.458 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.458 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.458 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.458 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.458 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.459 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.459 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.459 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.459 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.459 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.459 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.459 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.459 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.460 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.460 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.460 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.460 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.460 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.460 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.460 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.460 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.461 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.461 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.461 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.461 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.461 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.461 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.461 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.461 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.461 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.462 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.462 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.462 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.462 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.462 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.462 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.462 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.463 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.463 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.463 INFO:teuthology.orchestra.run.smithi044.stdout: "predict_interval": { 2024-05-27T09:51:03.463 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predict_interval", 2024-05-27T09:51:03.463 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.463 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.463 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.463 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-05-27T09:51:03.463 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.464 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.464 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.464 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.464 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.464 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.464 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.464 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.464 INFO:teuthology.orchestra.run.smithi044.stdout: "predictor_model": { 2024-05-27T09:51:03.464 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predictor_model", 2024-05-27T09:51:03.465 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.465 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.465 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.465 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "prophetstor", 2024-05-27T09:51:03.465 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.465 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.465 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.465 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.466 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.466 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.466 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.466 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.466 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:03.466 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:03.466 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.466 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.466 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.467 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:03.467 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.467 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.467 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.467 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.467 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.467 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.467 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.467 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.468 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.468 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.468 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.468 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.468 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.468 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.468 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.468 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.469 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.469 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.469 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.469 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.469 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.469 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.469 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.469 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.470 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.470 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "influx", 2024-05-27T09:51:03.470 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": false, 2024-05-27T09:51:03.470 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "influxdb python module not found", 2024-05-27T09:51:03.470 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.470 INFO:teuthology.orchestra.run.smithi044.stdout: "batch_size": { 2024-05-27T09:51:03.470 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "batch_size", 2024-05-27T09:51:03.470 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.471 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.471 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.471 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5000", 2024-05-27T09:51:03.471 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.471 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.471 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.471 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T09:51:03.471 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.472 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.472 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.472 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.472 INFO:teuthology.orchestra.run.smithi044.stdout: "database": { 2024-05-27T09:51:03.472 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "database", 2024-05-27T09:51:03.472 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.472 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.472 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.473 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "ceph", 2024-05-27T09:51:03.473 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.473 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.473 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.473 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T09:51:03.473 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.473 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.474 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.474 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.474 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": { 2024-05-27T09:51:03.474 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hostname", 2024-05-27T09:51:03.474 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.474 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.474 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.474 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.475 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.475 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.475 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.475 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server hostname", 2024-05-27T09:51:03.475 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.475 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.475 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.475 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.476 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:03.476 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:03.476 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.476 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.476 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.476 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "30", 2024-05-27T09:51:03.476 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "5", 2024-05-27T09:51:03.476 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.477 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.477 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T09:51:03.477 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.477 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.477 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.477 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.477 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.477 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.478 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.478 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.478 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.478 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.478 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.478 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.478 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.478 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.479 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.479 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.479 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.479 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.479 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.479 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.479 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.479 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.480 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.480 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.480 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.480 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.480 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.480 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.480 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.480 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.481 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.481 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.481 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.481 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.481 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.481 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.481 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.481 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.481 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.482 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.482 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.482 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.482 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.482 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.482 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.482 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.483 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.483 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.483 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.483 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.483 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.483 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.483 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.483 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.483 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.484 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.484 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.484 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.484 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.484 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.484 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.484 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.484 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.485 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.485 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.485 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.485 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.485 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.485 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.485 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.485 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.486 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.486 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.486 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.486 INFO:teuthology.orchestra.run.smithi044.stdout: "password": { 2024-05-27T09:51:03.486 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "password", 2024-05-27T09:51:03.486 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.486 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.487 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.487 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.487 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.487 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.487 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.487 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "password of InfluxDB server user", 2024-05-27T09:51:03.487 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.487 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.488 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.488 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.488 INFO:teuthology.orchestra.run.smithi044.stdout: "port": { 2024-05-27T09:51:03.488 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "port", 2024-05-27T09:51:03.488 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.488 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.488 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.488 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8086", 2024-05-27T09:51:03.488 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.489 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.489 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.489 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server port", 2024-05-27T09:51:03.489 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.489 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.489 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.489 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.489 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.489 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.490 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.490 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.490 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.490 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.490 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.490 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.490 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.490 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.491 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.491 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.491 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.491 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.491 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-05-27T09:51:03.491 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-05-27T09:51:03.491 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.491 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.491 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.492 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "false", 2024-05-27T09:51:03.492 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.492 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.492 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.492 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T09:51:03.492 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.492 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.492 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.492 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.493 INFO:teuthology.orchestra.run.smithi044.stdout: "threads": { 2024-05-27T09:51:03.493 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threads", 2024-05-27T09:51:03.493 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.493 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.493 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.493 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:03.493 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:03.493 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "32", 2024-05-27T09:51:03.493 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.494 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T09:51:03.494 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.494 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.494 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.494 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.494 INFO:teuthology.orchestra.run.smithi044.stdout: "username": { 2024-05-27T09:51:03.494 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "username", 2024-05-27T09:51:03.494 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.495 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.495 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.495 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.495 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.495 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.495 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.495 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "username of InfluxDB server user", 2024-05-27T09:51:03.495 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.496 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.496 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.496 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.496 INFO:teuthology.orchestra.run.smithi044.stdout: "verify_ssl": { 2024-05-27T09:51:03.496 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "verify_ssl", 2024-05-27T09:51:03.496 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.496 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.496 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.497 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "true", 2024-05-27T09:51:03.497 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.497 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.497 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.497 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T09:51:03.497 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.497 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.497 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.497 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.498 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.498 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.498 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.498 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "insights", 2024-05-27T09:51:03.498 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.498 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.498 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.498 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.499 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.499 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.499 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.499 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.499 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.499 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.499 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.499 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.500 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.500 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.500 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.500 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.500 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.500 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.500 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.500 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.501 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.501 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.501 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.501 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.501 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.501 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.501 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.501 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.502 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.502 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.502 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.502 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.502 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.502 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.502 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.503 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.503 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.503 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.503 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.503 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.503 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.503 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.503 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.504 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.504 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.504 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.504 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.504 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.504 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.504 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.504 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.504 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.505 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.505 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.505 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.505 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.505 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.505 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.505 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.505 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.506 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.506 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.506 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.506 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.506 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.506 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.506 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.506 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.507 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.507 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.507 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.507 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.507 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.507 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.507 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.507 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.508 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.508 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.508 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.508 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.508 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.508 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.508 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.508 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.509 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.509 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.509 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.509 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.509 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.509 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.509 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "iostat", 2024-05-27T09:51:03.509 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.509 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.510 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.510 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.510 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.510 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.510 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.510 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.510 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.510 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.511 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.511 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.511 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.511 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.511 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.511 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.511 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.512 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.512 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.512 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.512 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.512 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.512 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.512 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.512 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.513 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.513 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.513 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.513 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.513 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.513 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.513 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.513 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.514 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.514 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.514 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.514 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.514 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.514 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.514 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.514 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.515 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.515 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.515 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.515 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.515 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.515 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.515 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.516 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.516 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.516 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.516 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.516 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.516 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.516 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.516 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.517 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.517 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.517 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.517 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.517 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.517 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.517 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.517 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.518 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.518 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.518 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.518 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.518 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.518 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.518 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.518 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.519 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.519 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.519 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.519 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.519 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.519 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.519 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.519 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.519 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.520 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.520 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.520 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.520 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.520 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.520 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.520 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.520 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.521 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.521 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "k8sevents", 2024-05-27T09:51:03.521 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.521 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.521 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.521 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_event_retention_days": { 2024-05-27T09:51:03.521 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ceph_event_retention_days", 2024-05-27T09:51:03.521 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.522 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.522 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.522 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7", 2024-05-27T09:51:03.522 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.522 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.522 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.522 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T09:51:03.523 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.523 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.523 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.523 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.523 INFO:teuthology.orchestra.run.smithi044.stdout: "config_check_secs": { 2024-05-27T09:51:03.523 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_check_secs", 2024-05-27T09:51:03.523 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.523 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.524 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.524 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:03.524 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "10", 2024-05-27T09:51:03.524 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.524 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.524 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T09:51:03.524 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.524 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.524 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.525 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.525 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.525 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.525 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.525 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.525 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.525 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.525 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.525 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.526 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.526 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.526 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.526 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.526 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.526 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.526 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.527 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.527 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.527 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.527 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.527 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.527 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.527 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.527 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.527 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.528 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.528 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.528 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.528 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.528 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.528 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.528 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.528 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.529 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.529 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.529 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.529 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.529 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.529 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.529 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.529 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.529 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.530 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.530 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.530 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.530 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.530 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.530 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.530 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.530 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.531 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.531 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.531 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.531 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.531 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.531 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.531 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.532 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.532 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.532 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.532 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.532 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.532 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.532 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.532 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.533 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.533 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.533 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.533 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.533 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.533 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.533 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.533 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.533 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.534 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.534 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.534 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.534 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.534 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.534 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.534 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.535 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.535 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.535 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.535 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.535 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.535 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.535 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.535 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "localpool", 2024-05-27T09:51:03.536 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.536 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.536 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.536 INFO:teuthology.orchestra.run.smithi044.stdout: "failure_domain": { 2024-05-27T09:51:03.536 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "failure_domain", 2024-05-27T09:51:03.536 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.536 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.536 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.537 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "host", 2024-05-27T09:51:03.537 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.537 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.537 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.537 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "failure domain for any created local pool", 2024-05-27T09:51:03.537 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T09:51:03.537 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.537 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.538 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.538 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.538 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.538 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.538 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.538 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.538 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.538 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.539 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.539 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.539 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.539 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.539 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.539 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.539 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.540 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.540 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.540 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.540 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.540 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.540 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.540 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.540 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.541 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.541 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.541 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.541 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.541 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.541 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.541 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.541 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.542 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.542 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.542 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.542 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.542 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.542 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.542 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.542 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.543 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.543 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.543 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.543 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.543 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.543 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.543 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.543 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.544 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.544 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.544 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.544 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.544 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.544 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.544 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.544 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.545 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.545 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.545 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.545 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.545 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.545 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.545 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.545 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.546 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.546 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.546 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.546 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.546 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.546 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.546 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.546 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.546 INFO:teuthology.orchestra.run.smithi044.stdout: "min_size": { 2024-05-27T09:51:03.547 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_size", 2024-05-27T09:51:03.547 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.547 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.547 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.547 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.547 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.547 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.547 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.548 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default min_size for any created local pool", 2024-05-27T09:51:03.548 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T09:51:03.548 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.548 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.548 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.548 INFO:teuthology.orchestra.run.smithi044.stdout: "num_rep": { 2024-05-27T09:51:03.548 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "num_rep", 2024-05-27T09:51:03.548 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.549 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.549 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.549 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3", 2024-05-27T09:51:03.549 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.549 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.549 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.549 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default replica count for any created local pool", 2024-05-27T09:51:03.549 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.549 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.550 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.550 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.550 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_num": { 2024-05-27T09:51:03.550 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_num", 2024-05-27T09:51:03.550 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.550 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.550 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.550 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "128", 2024-05-27T09:51:03.551 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.551 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.551 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.551 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T09:51:03.551 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.551 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.551 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.551 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.551 INFO:teuthology.orchestra.run.smithi044.stdout: "prefix": { 2024-05-27T09:51:03.552 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prefix", 2024-05-27T09:51:03.552 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.552 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.552 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.552 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.552 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.552 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.552 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.553 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name prefix for any created local pool", 2024-05-27T09:51:03.553 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.553 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.553 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.553 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.553 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.553 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.553 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.554 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.554 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.554 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.554 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.554 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.554 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.554 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.554 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.555 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.555 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.555 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.555 INFO:teuthology.orchestra.run.smithi044.stdout: "subtree": { 2024-05-27T09:51:03.555 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "subtree", 2024-05-27T09:51:03.555 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.555 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.555 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.556 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "rack", 2024-05-27T09:51:03.556 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.556 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.556 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.556 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T09:51:03.556 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T09:51:03.556 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.557 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.557 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.557 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.557 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.557 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.557 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mds_autoscaler", 2024-05-27T09:51:03.557 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.557 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.558 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.558 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.558 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.558 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.558 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.558 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.558 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.558 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.559 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.559 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.559 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.559 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.559 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.559 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.559 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.559 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.559 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.560 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.560 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.560 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.560 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.560 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.560 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.560 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.560 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.561 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.561 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.561 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.561 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.561 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.561 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.561 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.561 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.562 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.562 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.562 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.562 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.562 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.562 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.562 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.563 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.563 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.563 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.563 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.563 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.563 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.563 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.563 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.563 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.564 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.564 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.564 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.564 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.564 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.564 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.564 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.564 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.565 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.565 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.565 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.565 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.565 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.565 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.565 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.565 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.566 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.566 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.566 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.566 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.566 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.566 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.566 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.566 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.567 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.567 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.567 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.567 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.567 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.567 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.567 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.567 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.567 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.568 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.568 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.568 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.568 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.568 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.568 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.568 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirroring", 2024-05-27T09:51:03.568 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.568 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.569 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.569 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.569 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.569 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.569 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.569 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.569 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.569 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.570 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.570 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.570 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.570 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.570 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.570 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.570 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.570 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.571 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.571 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.571 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.571 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.571 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.571 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.571 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.571 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.572 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.572 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.572 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.572 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.572 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.572 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.572 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.572 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.573 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.573 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.573 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.573 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.573 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.573 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.573 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.573 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.574 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.574 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.574 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.574 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.574 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.574 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.574 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.574 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.575 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.575 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.575 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.575 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.575 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.575 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.575 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.575 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.576 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.576 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.576 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.576 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.576 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.576 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.576 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.576 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.577 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.577 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.577 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.577 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.577 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.577 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.577 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.577 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.578 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.578 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.578 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.578 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.578 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.578 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.578 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.578 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.579 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.579 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.579 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.579 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.579 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.579 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.579 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.580 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.580 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "nfs", 2024-05-27T09:51:03.580 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.580 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.580 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.580 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.580 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.580 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.580 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.581 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.581 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.581 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.581 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.581 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.581 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.581 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.581 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.582 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.582 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.582 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.582 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.582 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.582 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.582 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.583 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.583 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.583 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.583 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.583 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.583 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.583 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.583 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.584 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.584 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.584 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.584 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.584 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.584 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.584 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.585 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.585 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.585 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.585 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.585 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.585 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.585 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.586 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.586 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.586 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.586 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.586 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.586 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.586 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.586 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.587 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.587 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.587 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.587 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.587 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.587 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.587 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.587 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.588 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.588 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.588 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.588 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.588 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.588 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.588 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.588 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.589 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.589 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.589 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.589 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.589 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.589 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.589 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.589 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.589 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.590 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.590 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.590 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.590 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.590 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.590 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.590 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.590 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.591 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.591 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.591 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.591 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.591 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-05-27T09:51:03.591 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.591 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.591 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.591 INFO:teuthology.orchestra.run.smithi044.stdout: "fail_fs": { 2024-05-27T09:51:03.592 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "fail_fs", 2024-05-27T09:51:03.592 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.592 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.592 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.592 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.592 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.592 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.592 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.592 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T09:51:03.593 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.593 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.593 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.593 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.593 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.593 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.593 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.593 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.593 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.594 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.594 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.594 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.594 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.594 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.594 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.594 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.594 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.595 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.595 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.595 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.595 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.595 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.595 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.595 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.595 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.596 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.596 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.596 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.596 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.596 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.596 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.596 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.596 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.597 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.597 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.597 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.597 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.597 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.597 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.597 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.598 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.598 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.598 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.598 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.598 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.598 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.598 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.598 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.598 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.599 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.599 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.599 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.599 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.599 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.599 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.599 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.599 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.600 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.600 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.600 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.600 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.600 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.600 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.600 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.600 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.600 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.601 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.601 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.601 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.601 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.601 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.601 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.601 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.601 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.601 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator": { 2024-05-27T09:51:03.602 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-05-27T09:51:03.602 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.602 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.602 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.602 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.602 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.602 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.602 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.603 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2024-05-27T09:51:03.603 INFO:teuthology.orchestra.run.smithi044.stdout: "rook", 2024-05-27T09:51:03.603 INFO:teuthology.orchestra.run.smithi044.stdout: "test_orchestrator" 2024-05-27T09:51:03.603 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.603 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Orchestrator backend", 2024-05-27T09:51:03.603 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.603 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.603 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.604 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.604 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.604 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.604 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.604 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.604 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.604 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.604 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.605 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.605 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.605 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.605 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.605 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.605 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.605 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.605 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.606 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.606 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.606 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_perf_query", 2024-05-27T09:51:03.606 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.606 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.606 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.606 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.606 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.607 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.607 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.607 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.607 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.607 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.607 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.607 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.607 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.608 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.608 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.608 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.608 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.608 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.608 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.608 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.608 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.608 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.609 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.609 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.609 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.609 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.609 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.609 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.609 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.609 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.610 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.610 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.610 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.610 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.610 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.610 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.610 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.610 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.611 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.611 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.611 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.611 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.611 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.611 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.611 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.611 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.612 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.612 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.612 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.612 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.612 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.612 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.612 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.612 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.613 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.613 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.613 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.613 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.613 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.613 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.613 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.613 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.613 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.614 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.614 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.614 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.614 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.614 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.614 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.614 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.614 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.615 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.615 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.615 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.615 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.615 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.615 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.615 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.615 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.616 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.616 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.616 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.616 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.616 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.616 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.616 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.617 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.617 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.617 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.617 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.617 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_support", 2024-05-27T09:51:03.617 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.617 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.617 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.618 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.618 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.618 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.618 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.618 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.618 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.618 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.618 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.618 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.619 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.619 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.619 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.619 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.619 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.619 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.619 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.620 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.620 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.620 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.620 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.620 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.620 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.620 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.620 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.620 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.621 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.621 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.621 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.621 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.621 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.621 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.621 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.621 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.622 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.622 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.622 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.622 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.622 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.622 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.622 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.622 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.623 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.623 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.623 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.623 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.623 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.623 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.623 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.623 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.623 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.624 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.624 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.624 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.624 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.624 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.624 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.624 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.624 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.624 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.625 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.625 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.625 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.625 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.625 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.625 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.625 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.625 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.625 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.626 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.626 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.626 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.626 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.626 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.626 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.626 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.626 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.627 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.627 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.627 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.627 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.627 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.627 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.627 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.627 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.627 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.628 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.628 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.628 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_autoscaler", 2024-05-27T09:51:03.628 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.628 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.628 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.628 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.628 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.629 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.629 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.629 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.629 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.629 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.629 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.629 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.629 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.630 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.630 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.630 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.630 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.630 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.630 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.630 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.630 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.631 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.631 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.631 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.631 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.631 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.631 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.632 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.632 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.632 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.632 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.632 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.632 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.632 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.632 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.633 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.633 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.633 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.633 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.633 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.633 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.633 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.633 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.634 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.634 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.634 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.634 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.634 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.634 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.634 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.635 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.635 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.635 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.635 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.635 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.635 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.635 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.636 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.636 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.636 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.636 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.636 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.636 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.636 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.636 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.637 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.637 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.637 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.637 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.637 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.637 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.637 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.637 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:03.638 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:03.638 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.638 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.638 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.638 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:03.638 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.638 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.638 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.639 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.639 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.639 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.639 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.639 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.639 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.639 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.640 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.640 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.640 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.640 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.640 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.640 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.640 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.640 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.640 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.641 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.641 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.641 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.641 INFO:teuthology.orchestra.run.smithi044.stdout: "threshold": { 2024-05-27T09:51:03.641 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threshold", 2024-05-27T09:51:03.641 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:03.641 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.641 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.642 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-05-27T09:51:03.642 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1.0", 2024-05-27T09:51:03.642 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.642 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.642 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "scaling threshold", 2024-05-27T09:51:03.642 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T09:51:03.642 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.642 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.643 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.643 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.643 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.643 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.643 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "progress", 2024-05-27T09:51:03.643 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.643 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.643 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.644 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_pg_recovery_event": { 2024-05-27T09:51:03.644 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_pg_recovery_event", 2024-05-27T09:51:03.644 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.644 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.644 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.644 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.644 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.644 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.645 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.645 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T09:51:03.645 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.645 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.645 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.645 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.645 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-05-27T09:51:03.646 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-05-27T09:51:03.646 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.646 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.646 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.646 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.646 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.646 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.646 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.647 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.647 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.647 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.647 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.647 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.647 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.647 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.648 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.648 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.648 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.648 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.648 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.648 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.648 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.648 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.649 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.649 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.649 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.649 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.649 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.649 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.649 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.649 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.650 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.650 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.650 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.650 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.650 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.650 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.650 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.651 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.651 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.651 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.651 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.651 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.651 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.651 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.651 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.652 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.652 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.652 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.652 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.652 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.652 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.652 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.652 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.653 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.653 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.653 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.653 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.653 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.653 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.653 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.653 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.653 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.654 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.654 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.654 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.654 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.654 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.654 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.654 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.655 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.655 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.655 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.655 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.655 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.655 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.655 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.655 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.656 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.656 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.656 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.656 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.656 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.656 INFO:teuthology.orchestra.run.smithi044.stdout: "max_completed_events": { 2024-05-27T09:51:03.656 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_completed_events", 2024-05-27T09:51:03.656 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.657 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.657 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.657 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "50", 2024-05-27T09:51:03.657 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.657 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.657 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.657 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "number of past completed events to remember", 2024-05-27T09:51:03.657 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.657 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.658 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.658 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.658 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:03.658 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:03.658 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.658 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.658 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.659 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:03.659 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.659 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.659 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.659 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long the module is going to sleep", 2024-05-27T09:51:03.659 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.659 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.659 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.660 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.660 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.660 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.660 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.660 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.660 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.660 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.660 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.660 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.661 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.661 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.661 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.661 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.661 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.661 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.661 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.661 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.661 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.662 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus", 2024-05-27T09:51:03.662 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.662 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.662 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.662 INFO:teuthology.orchestra.run.smithi044.stdout: "cache": { 2024-05-27T09:51:03.662 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cache", 2024-05-27T09:51:03.662 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.662 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.663 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.663 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.663 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.663 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.663 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.663 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.663 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.663 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.663 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.664 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.664 INFO:teuthology.orchestra.run.smithi044.stdout: "exclude_perf_counters": { 2024-05-27T09:51:03.664 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "exclude_perf_counters", 2024-05-27T09:51:03.664 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.664 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.664 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.664 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.664 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.664 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.665 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.665 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T09:51:03.665 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T09:51:03.665 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.665 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.665 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.665 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.665 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.666 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.666 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.666 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.666 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.666 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.666 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.666 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.666 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.666 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.667 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.667 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.667 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.667 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.667 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.667 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.667 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.667 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.667 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.668 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.668 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.668 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.668 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.668 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.668 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.668 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.668 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.669 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.669 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.669 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.669 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.669 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.669 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.669 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.669 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.669 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.670 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.670 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.670 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.670 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.670 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.670 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.670 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.670 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.670 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.671 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.671 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.671 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.671 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.671 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.671 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.671 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.671 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.672 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.672 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.672 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.672 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.672 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.672 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.672 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.673 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.673 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.673 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.673 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.673 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.673 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.673 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.673 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.674 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.674 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools": { 2024-05-27T09:51:03.674 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools", 2024-05-27T09:51:03.674 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.674 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.674 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.674 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.674 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.675 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.675 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.675 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.675 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.675 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.675 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.675 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.675 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T09:51:03.676 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T09:51:03.676 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.676 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.676 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.676 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300", 2024-05-27T09:51:03.676 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.676 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.676 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.677 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.677 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.677 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.677 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.677 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.677 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_interval": { 2024-05-27T09:51:03.677 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_interval", 2024-05-27T09:51:03.677 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:03.678 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.678 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.678 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15.0", 2024-05-27T09:51:03.678 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.678 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.678 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.678 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.679 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.679 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.679 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.679 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.679 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-05-27T09:51:03.679 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-05-27T09:51:03.679 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.679 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.679 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.680 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-05-27T09:51:03.680 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.680 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.680 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.680 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T09:51:03.680 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.680 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.680 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.681 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.681 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-05-27T09:51:03.681 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-05-27T09:51:03.681 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.681 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.681 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.681 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "9283", 2024-05-27T09:51:03.681 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.682 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.682 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.682 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T09:51:03.682 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.682 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.682 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.682 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.683 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.683 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.683 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.683 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.683 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.683 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.683 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.683 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.683 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.684 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.684 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.684 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.684 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.684 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.684 INFO:teuthology.orchestra.run.smithi044.stdout: "stale_cache_strategy": { 2024-05-27T09:51:03.684 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stale_cache_strategy", 2024-05-27T09:51:03.684 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.685 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.685 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.685 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "log", 2024-05-27T09:51:03.685 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.685 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.685 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.685 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.685 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.686 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.686 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.686 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.686 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-05-27T09:51:03.686 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-05-27T09:51:03.686 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.686 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.687 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.687 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "default", 2024-05-27T09:51:03.687 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.687 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.687 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.687 INFO:teuthology.orchestra.run.smithi044.stdout: "default", 2024-05-27T09:51:03.687 INFO:teuthology.orchestra.run.smithi044.stdout: "error" 2024-05-27T09:51:03.687 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.687 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.688 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.688 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.688 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.688 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.688 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-05-27T09:51:03.688 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-05-27T09:51:03.688 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.688 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.689 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.689 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-05-27T09:51:03.689 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-05-27T09:51:03.689 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-05-27T09:51:03.689 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.689 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.689 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.689 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.690 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.690 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.690 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.690 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.690 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.690 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2024-05-27T09:51:03.690 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.691 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.691 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.691 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.691 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.691 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.694 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.694 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.694 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.694 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.694 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.694 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.694 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.695 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.695 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.695 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.695 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.695 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.695 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.695 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.695 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.696 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.696 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.696 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.696 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.696 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.696 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.696 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.696 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.697 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.697 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.697 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.697 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.697 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.697 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.697 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.697 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.697 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.698 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.698 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.698 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.698 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.698 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.698 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.698 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.698 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.699 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.699 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.699 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.699 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.699 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.699 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.699 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.699 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.700 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.700 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.700 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.700 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.700 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.700 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.700 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.700 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.700 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.701 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.701 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.701 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.701 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.701 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.701 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.701 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.701 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.702 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.702 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.702 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_snap_create": { 2024-05-27T09:51:03.702 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_snap_create", 2024-05-27T09:51:03.702 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.702 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.702 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.702 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:03.703 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.703 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.703 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.703 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.703 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.703 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.703 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.703 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.704 INFO:teuthology.orchestra.run.smithi044.stdout: "mirror_snapshot_schedule": { 2024-05-27T09:51:03.704 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T09:51:03.704 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.704 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.704 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.704 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.704 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.704 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.704 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.705 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.705 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.705 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.705 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.705 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.705 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.705 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.706 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.706 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.706 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.706 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.706 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.706 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.706 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.706 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.707 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.707 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.707 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.707 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.707 INFO:teuthology.orchestra.run.smithi044.stdout: "trash_purge_schedule": { 2024-05-27T09:51:03.707 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "trash_purge_schedule", 2024-05-27T09:51:03.707 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.708 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.708 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.708 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.708 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.708 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.708 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.708 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.708 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.708 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.709 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.709 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.709 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.709 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.709 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.709 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "restful", 2024-05-27T09:51:03.709 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.709 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.710 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.710 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_auth": { 2024-05-27T09:51:03.710 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_auth", 2024-05-27T09:51:03.710 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.710 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.710 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.710 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.710 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.711 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.711 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.711 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.711 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.711 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.711 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.711 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.711 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-05-27T09:51:03.712 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-05-27T09:51:03.712 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.712 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.712 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.712 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.712 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.712 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.713 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.713 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.713 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.713 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.713 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.713 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.713 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.713 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.714 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.714 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.714 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.714 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.714 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.714 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.714 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.715 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.715 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.715 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.715 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.715 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.715 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.715 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.715 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.716 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.716 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.716 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.716 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.716 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.716 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.716 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.716 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.716 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.717 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.717 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.717 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.717 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.717 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.717 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.717 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.717 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.718 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.718 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.718 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.718 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.718 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.718 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.718 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.719 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.719 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.719 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.719 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.719 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.719 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.719 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.719 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.720 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.720 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.720 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.720 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.720 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.720 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.720 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.720 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.721 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.721 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.721 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.721 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.721 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.721 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.721 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.721 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.722 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.722 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.722 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.722 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.722 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.722 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-05-27T09:51:03.722 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-05-27T09:51:03.722 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.723 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.723 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.723 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.723 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.723 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.723 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.723 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.723 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.724 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.724 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.724 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.724 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-05-27T09:51:03.724 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-05-27T09:51:03.724 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.724 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.724 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.725 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.725 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.725 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.725 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.725 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.725 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.725 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.725 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.726 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.726 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.726 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.726 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.726 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.726 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.726 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.726 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.727 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.727 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.727 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.727 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.727 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.727 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.727 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.727 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.727 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rgw", 2024-05-27T09:51:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.729 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.729 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.729 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.729 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.729 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.729 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.729 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.729 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.730 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.730 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.730 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.730 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.730 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.730 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.730 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.730 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.730 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.731 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.731 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.731 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.731 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.731 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.731 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.731 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.731 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.732 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.732 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.732 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.732 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.732 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.732 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.732 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.732 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.733 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.733 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.733 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.733 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.733 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.733 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.733 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.733 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.734 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.734 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.734 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.734 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.734 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.734 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.734 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.734 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.734 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.735 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.735 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.735 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.735 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.735 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.735 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.735 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.735 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.736 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.736 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.736 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.736 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.736 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.736 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.736 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.736 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.736 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.737 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.737 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.737 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.737 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.737 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.737 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.737 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.737 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.738 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.738 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.738 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.738 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.738 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.738 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.738 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.738 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.739 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.739 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rook", 2024-05-27T09:51:03.739 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.739 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.739 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.739 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.739 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.739 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.740 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.740 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.740 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.740 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.740 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.740 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.740 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.740 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.741 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.741 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.741 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.741 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.741 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.741 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.741 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.741 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.742 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.742 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.742 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.742 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.742 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.742 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.742 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.742 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.743 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.743 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.743 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.743 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.743 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.743 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.743 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.743 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.744 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.744 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.744 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.744 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.744 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.744 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.744 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.744 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.745 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.745 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.745 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.745 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.745 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.745 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.745 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.745 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.746 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.746 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.746 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.746 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.746 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.746 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.746 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.746 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.747 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.747 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.747 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.747 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.747 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.747 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.747 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.747 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.748 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.748 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.748 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.748 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.748 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.748 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.748 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.748 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.749 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.749 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.749 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.749 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.749 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.749 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.749 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.749 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.750 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.750 INFO:teuthology.orchestra.run.smithi044.stdout: "storage_class": { 2024-05-27T09:51:03.750 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "storage_class", 2024-05-27T09:51:03.750 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.750 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.750 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.750 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "local", 2024-05-27T09:51:03.750 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.751 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.751 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.751 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T09:51:03.751 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.751 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.751 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.751 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.751 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.752 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.752 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.752 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "selftest", 2024-05-27T09:51:03.752 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.752 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.752 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.752 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.752 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.753 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.753 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.753 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.753 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.753 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.753 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.753 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.753 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.754 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.754 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.754 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.754 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.754 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.754 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.754 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.754 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.755 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.755 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.755 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.755 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.755 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.755 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.755 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.755 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.756 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.756 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.756 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.756 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.756 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.756 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.756 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.756 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.757 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.757 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.757 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.757 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.757 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.757 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.757 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.757 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.757 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.758 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.758 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.758 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.758 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.758 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.758 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.758 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.758 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.759 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.759 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.759 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.759 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.759 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.759 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.759 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.759 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.759 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.760 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.760 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.760 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.760 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.760 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.760 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.760 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.760 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.760 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.761 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.761 INFO:teuthology.orchestra.run.smithi044.stdout: "roption1": { 2024-05-27T09:51:03.761 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption1", 2024-05-27T09:51:03.761 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.761 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.761 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.761 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.761 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.762 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.762 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.762 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.762 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.762 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.762 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.762 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.762 INFO:teuthology.orchestra.run.smithi044.stdout: "roption2": { 2024-05-27T09:51:03.763 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption2", 2024-05-27T09:51:03.763 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.763 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.763 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.763 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "xyz", 2024-05-27T09:51:03.763 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.763 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.763 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.763 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.764 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.764 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.764 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.764 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.764 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption1": { 2024-05-27T09:51:03.764 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption1", 2024-05-27T09:51:03.764 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.764 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.764 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.765 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.765 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.765 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.765 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.765 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.765 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.765 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.765 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.765 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.766 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption2": { 2024-05-27T09:51:03.766 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption2", 2024-05-27T09:51:03.766 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.766 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.766 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.766 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.766 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.766 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.767 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.767 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.767 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.767 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.767 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.767 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.767 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption3": { 2024-05-27T09:51:03.767 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption3", 2024-05-27T09:51:03.768 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:03.768 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.768 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.768 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.768 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.768 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.768 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.768 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.769 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.769 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.769 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.769 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.769 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption4": { 2024-05-27T09:51:03.769 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption4", 2024-05-27T09:51:03.769 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.769 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.770 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.770 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.770 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.770 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.770 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.770 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.770 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.771 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.771 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.771 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.771 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption5": { 2024-05-27T09:51:03.771 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption5", 2024-05-27T09:51:03.771 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.771 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.771 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.772 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.772 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.772 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.772 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.772 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.772 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.772 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.772 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.773 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.773 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption6": { 2024-05-27T09:51:03.773 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption6", 2024-05-27T09:51:03.773 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.773 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.773 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.773 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.773 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.774 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.774 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.774 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.774 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.774 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.774 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.774 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption7": { 2024-05-27T09:51:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption7", 2024-05-27T09:51:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "42", 2024-05-27T09:51:03.776 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.776 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.776 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.776 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.776 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.776 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.776 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.777 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.777 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.777 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.777 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.777 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.777 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.777 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.777 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.778 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.778 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.778 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.778 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.778 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.778 INFO:teuthology.orchestra.run.smithi044.stdout: "testkey": { 2024-05-27T09:51:03.778 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testkey", 2024-05-27T09:51:03.778 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.779 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.779 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.779 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.779 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.779 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.779 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.779 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.779 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.779 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.780 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.780 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.780 INFO:teuthology.orchestra.run.smithi044.stdout: "testlkey": { 2024-05-27T09:51:03.780 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testlkey", 2024-05-27T09:51:03.780 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.780 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.780 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.780 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.781 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.781 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.781 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.781 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.781 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.781 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.781 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.781 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.782 INFO:teuthology.orchestra.run.smithi044.stdout: "testnewline": { 2024-05-27T09:51:03.782 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testnewline", 2024-05-27T09:51:03.782 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.782 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.782 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.782 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.782 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.783 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.783 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.783 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.783 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.783 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.783 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.783 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.783 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.784 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.784 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.784 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smb", 2024-05-27T09:51:03.784 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.784 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.784 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.784 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.784 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.785 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.785 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.785 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.785 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.785 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.785 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.785 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.785 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.786 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.786 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.786 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.786 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.786 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.786 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.786 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.787 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.787 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.787 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.787 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.787 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.787 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.787 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.787 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.788 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.788 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.788 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.788 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.788 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.788 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.788 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.788 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.789 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.789 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.789 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.789 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.789 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.789 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.789 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.789 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.790 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.790 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.790 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.790 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.790 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.790 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.790 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.790 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.791 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.791 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.791 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.791 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.791 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.791 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.791 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.791 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.792 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.792 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.792 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.792 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.792 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.792 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.792 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.792 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.792 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.793 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.793 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.793 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.793 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.793 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.793 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.793 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.793 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.794 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.794 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.794 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.794 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.794 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.794 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.794 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.794 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.795 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.795 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.795 INFO:teuthology.orchestra.run.smithi044.stdout: "update_orchestration": { 2024-05-27T09:51:03.795 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "update_orchestration", 2024-05-27T09:51:03.795 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.795 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.795 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.795 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.796 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.796 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.796 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.796 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T09:51:03.796 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.796 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.796 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.796 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2024-05-27T09:51:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_m_granularity": { 2024-05-27T09:51:03.797 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_m_granularity", 2024-05-27T09:51:03.798 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.798 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.798 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.798 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.798 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.798 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.798 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.798 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T09:51:03.799 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.799 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.799 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.799 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.799 INFO:teuthology.orchestra.run.smithi044.stdout: "dump_on_update": { 2024-05-27T09:51:03.799 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dump_on_update", 2024-05-27T09:51:03.799 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.799 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.799 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.800 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.800 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.800 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.800 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.800 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "dump database to debug log on update", 2024-05-27T09:51:03.800 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.800 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.800 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.801 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.801 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.801 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.801 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.801 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.801 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.801 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.801 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.801 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.802 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.802 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.802 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.802 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.802 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.802 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.802 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.802 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.802 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.803 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.803 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.803 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.803 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.803 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.803 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.803 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.803 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.804 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.804 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.804 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.804 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.804 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.804 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.804 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.804 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.804 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.804 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.805 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.805 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.805 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.805 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.805 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.805 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.805 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.806 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.806 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.806 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.806 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.806 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.806 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.806 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.806 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.807 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.807 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.807 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.807 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.807 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.807 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.807 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.807 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.808 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.808 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.808 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.808 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.808 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.808 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.808 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.808 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.808 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.809 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.809 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.809 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.809 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.809 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.809 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.809 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.809 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.810 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.810 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.810 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.810 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.810 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.810 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.810 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.810 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.811 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.811 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.811 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.811 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.811 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stats", 2024-05-27T09:51:03.811 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.811 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.811 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.811 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.812 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.812 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.812 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.812 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.812 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.812 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.812 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.812 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.813 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.813 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.813 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.813 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.813 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.813 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.813 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.813 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.814 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.814 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.814 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.814 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.814 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.814 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.814 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.814 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.815 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.815 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.815 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.815 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.815 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.815 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.815 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.815 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.816 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.816 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.816 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.816 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.816 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.816 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.816 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.817 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.817 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.817 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.817 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.817 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.817 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.817 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.817 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.818 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.818 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.818 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.818 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.818 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.818 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.818 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.818 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.819 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.819 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.819 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.819 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.819 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.819 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.819 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.819 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.820 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.820 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.820 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.820 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.820 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.820 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.820 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.820 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.821 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.821 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.821 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.821 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.821 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.821 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.821 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.821 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.822 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.822 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.822 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.822 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.822 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.822 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.822 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.822 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "status", 2024-05-27T09:51:03.823 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.823 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.823 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.823 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.823 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.823 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.823 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.823 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.824 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.824 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.824 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.824 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.824 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.824 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.824 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.825 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.825 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.825 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.825 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.825 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.825 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.825 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.825 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.826 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.826 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.826 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.826 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.826 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.826 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.826 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.826 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.826 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.827 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.827 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.827 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.827 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.827 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.827 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.827 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.827 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.827 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.828 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.828 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.828 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.828 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.828 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.828 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.828 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.828 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.828 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.829 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.829 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.829 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.829 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.829 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.829 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.829 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.829 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.830 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.830 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.830 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.830 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.830 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.830 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.830 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.831 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.831 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.831 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.831 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.831 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.831 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.831 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.831 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.832 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.832 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.832 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.832 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.832 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.832 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.832 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.832 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.833 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.833 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.833 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.833 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.833 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.833 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.833 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.833 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.833 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.834 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telegraf", 2024-05-27T09:51:03.834 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.834 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.834 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.834 INFO:teuthology.orchestra.run.smithi044.stdout: "address": { 2024-05-27T09:51:03.834 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "address", 2024-05-27T09:51:03.834 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.834 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.835 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.835 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T09:51:03.835 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.835 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.835 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.835 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.835 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.836 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.836 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.836 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.836 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:03.836 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:03.836 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.836 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.836 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.836 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15", 2024-05-27T09:51:03.837 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.837 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.837 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.837 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.837 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.837 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.837 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.837 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.837 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.838 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.838 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.838 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.838 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.838 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.838 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.838 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.838 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.839 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.839 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.839 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.839 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.839 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.839 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.839 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.839 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.839 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.840 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.840 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.840 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.840 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.840 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.840 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.840 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.840 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.841 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.841 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.841 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.841 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.841 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.841 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.841 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.842 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.842 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.842 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.842 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.842 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.842 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.842 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.842 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.843 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.843 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.843 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.843 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.843 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.843 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.843 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.843 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.844 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.844 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.844 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.844 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.844 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.844 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.844 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.844 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.844 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.845 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.845 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.845 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.845 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.845 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.845 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.845 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.845 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.846 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.846 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.846 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.846 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.846 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.846 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.846 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.846 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.847 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.847 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.847 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.847 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.847 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.847 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.847 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.847 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.847 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.848 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.848 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.848 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.848 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.848 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telemetry", 2024-05-27T09:51:03.848 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.848 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.848 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.849 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_basic": { 2024-05-27T09:51:03.849 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_basic", 2024-05-27T09:51:03.849 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.849 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.849 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.849 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.849 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.849 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.850 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.850 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T09:51:03.850 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.850 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.850 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.850 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.850 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_crash": { 2024-05-27T09:51:03.851 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_crash", 2024-05-27T09:51:03.851 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.851 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.851 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.851 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.851 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.851 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.851 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.851 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T09:51:03.852 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.852 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.852 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.852 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.852 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_device": { 2024-05-27T09:51:03.852 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_device", 2024-05-27T09:51:03.852 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.852 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.852 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.853 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.853 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.853 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.853 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.853 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T09:51:03.853 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.853 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.853 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.854 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.854 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_ident": { 2024-05-27T09:51:03.854 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_ident", 2024-05-27T09:51:03.854 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.854 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.854 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.855 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.855 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.855 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.855 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.855 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T09:51:03.855 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.855 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.855 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.856 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.856 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_perf": { 2024-05-27T09:51:03.856 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_perf", 2024-05-27T09:51:03.856 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.856 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.856 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.856 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.856 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.857 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.857 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.857 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T09:51:03.857 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.857 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.857 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.857 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.857 INFO:teuthology.orchestra.run.smithi044.stdout: "contact": { 2024-05-27T09:51:03.857 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "contact", 2024-05-27T09:51:03.858 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.858 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.858 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.858 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.858 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.858 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.858 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.859 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.859 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.859 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.859 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.859 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.859 INFO:teuthology.orchestra.run.smithi044.stdout: "description": { 2024-05-27T09:51:03.859 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "description", 2024-05-27T09:51:03.859 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.859 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.860 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.860 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.860 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.860 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.860 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.860 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.860 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.860 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.861 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.861 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.861 INFO:teuthology.orchestra.run.smithi044.stdout: "device_url": { 2024-05-27T09:51:03.861 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_url", 2024-05-27T09:51:03.861 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.861 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.861 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.861 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T09:51:03.862 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.862 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.862 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.862 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.862 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.862 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.862 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.862 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.863 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-05-27T09:51:03.863 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-05-27T09:51:03.863 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.863 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.863 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.863 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.863 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.863 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.864 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.864 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.864 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.864 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.864 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.864 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.864 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:03.864 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:03.865 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.865 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.865 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.865 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "24", 2024-05-27T09:51:03.865 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "8", 2024-05-27T09:51:03.865 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.865 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.866 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.866 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.866 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.866 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.866 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.866 INFO:teuthology.orchestra.run.smithi044.stdout: "last_opt_revision": { 2024-05-27T09:51:03.866 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "last_opt_revision", 2024-05-27T09:51:03.866 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.867 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.867 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.867 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1", 2024-05-27T09:51:03.867 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.867 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.867 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.867 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.867 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.867 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.868 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.868 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.868 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard": { 2024-05-27T09:51:03.868 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard", 2024-05-27T09:51:03.868 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.868 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.868 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.868 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.868 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.869 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.869 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.869 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.869 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.869 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.869 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.869 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.870 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard_description": { 2024-05-27T09:51:03.870 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard_description", 2024-05-27T09:51:03.870 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.870 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.870 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.870 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.870 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.870 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.871 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.871 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.871 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.871 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.871 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.871 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.871 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.871 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.871 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.872 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.872 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.872 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.872 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.872 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.872 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.872 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.872 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.872 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.873 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.873 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.873 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.873 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.873 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.873 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.873 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.873 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.874 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.874 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.874 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.874 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.874 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.874 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.874 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.874 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.875 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.875 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.875 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.875 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.875 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.875 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.875 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.875 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.876 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.876 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.876 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.876 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.876 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.876 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.876 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.876 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.876 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.877 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.877 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.877 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.877 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.877 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.877 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.877 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.877 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.878 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.878 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.878 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.878 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.878 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.878 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.878 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.878 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.879 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.879 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.880 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:02 smithi189 ceph-mon[36671]: pgmap v497: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-27T09:51:03.881 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:02 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3447476804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T09:51:03.881 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:02 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1509925310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T09:51:03.881 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:03 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/900504659' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T09:51:03.881 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:02 smithi050 ceph-mon[36371]: pgmap v497: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-27T09:51:03.881 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:02 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3447476804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T09:51:03.881 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:02 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1509925310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T09:51:03.881 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:03 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/900504659' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T09:51:03.884 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.884 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.884 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.884 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.885 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.885 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.885 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.885 INFO:teuthology.orchestra.run.smithi044.stdout: "organization": { 2024-05-27T09:51:03.885 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "organization", 2024-05-27T09:51:03.885 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.885 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.885 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.886 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.886 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.886 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.886 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.886 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.886 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.886 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.887 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.887 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.887 INFO:teuthology.orchestra.run.smithi044.stdout: "proxy": { 2024-05-27T09:51:03.887 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "proxy", 2024-05-27T09:51:03.887 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.887 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.887 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.887 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.888 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.888 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.888 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.888 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.888 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.888 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.888 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.888 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.889 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.889 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.889 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.889 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.889 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.889 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.889 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.889 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.890 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.890 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.890 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.890 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.890 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.890 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.890 INFO:teuthology.orchestra.run.smithi044.stdout: "url": { 2024-05-27T09:51:03.891 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url", 2024-05-27T09:51:03.891 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.891 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.891 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.891 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T09:51:03.891 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.891 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.891 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.892 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.892 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.892 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.892 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.892 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.892 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.892 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.892 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.893 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "test_orchestrator", 2024-05-27T09:51:03.893 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.893 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.893 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.893 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.893 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.893 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.893 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.894 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.894 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.894 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.894 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.894 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.894 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.894 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.894 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.895 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.895 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.895 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.895 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.895 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.895 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.895 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.895 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.896 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.896 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.896 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.896 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.896 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.896 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.896 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.896 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.896 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.897 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.897 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.897 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.897 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.897 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.897 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.897 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.897 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.898 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.898 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.898 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.898 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.898 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.898 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.898 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.898 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.898 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.899 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.899 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.899 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.899 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.899 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.899 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.899 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.899 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.899 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.900 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.900 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.900 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.900 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.900 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.900 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.900 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.900 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.900 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.901 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.901 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.901 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.901 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.901 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.901 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.901 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.901 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.901 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.902 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.902 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.902 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.902 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.902 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.902 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.902 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.902 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.903 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.903 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.903 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.903 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.903 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.903 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.903 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2024-05-27T09:51:03.903 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.903 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.904 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.904 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.904 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.904 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.904 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.904 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.904 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.904 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.905 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.905 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.905 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.905 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.905 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.905 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.905 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.905 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.906 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.906 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.906 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.906 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.906 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.906 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.906 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.906 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.906 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.907 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.907 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.907 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.907 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.907 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.907 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.907 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.907 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.907 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.908 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.908 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.908 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.908 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.908 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.908 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.908 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.908 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.909 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.909 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.909 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.909 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.909 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.909 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.909 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.909 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.910 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.910 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.910 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.910 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.910 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.910 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.910 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.910 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.911 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.911 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.911 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.911 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.911 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.911 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.911 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.911 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.912 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.912 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.912 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.912 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.912 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.912 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_clones": { 2024-05-27T09:51:03.912 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_clones", 2024-05-27T09:51:03.912 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.913 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.913 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.913 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4", 2024-05-27T09:51:03.913 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.913 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.913 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.913 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T09:51:03.913 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.914 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.914 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.914 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.914 INFO:teuthology.orchestra.run.smithi044.stdout: "periodic_async_work": { 2024-05-27T09:51:03.914 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "periodic_async_work", 2024-05-27T09:51:03.914 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.914 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.914 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.915 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.915 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.915 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.915 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.915 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Periodically check for async work", 2024-05-27T09:51:03.915 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.915 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.916 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.916 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.916 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_delay": { 2024-05-27T09:51:03.916 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_delay", 2024-05-27T09:51:03.916 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.916 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.916 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.916 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.917 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.917 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.917 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.917 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T09:51:03.917 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.917 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.917 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.917 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.918 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_no_wait": { 2024-05-27T09:51:03.918 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T09:51:03.918 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.918 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.918 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.918 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.918 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.919 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.919 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.919 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T09:51:03.919 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.919 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.919 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.919 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.919 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.920 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.920 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.920 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.920 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.920 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.920 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.920 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.920 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.921 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.921 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.921 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.921 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.921 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.921 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.921 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.921 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.922 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix", 2024-05-27T09:51:03.922 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.922 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.922 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.922 INFO:teuthology.orchestra.run.smithi044.stdout: "discovery_interval": { 2024-05-27T09:51:03.922 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "discovery_interval", 2024-05-27T09:51:03.922 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:03.922 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.923 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.923 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "100", 2024-05-27T09:51:03.923 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.923 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.923 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.923 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.923 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.923 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.924 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.924 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.924 INFO:teuthology.orchestra.run.smithi044.stdout: "identifier": { 2024-05-27T09:51:03.924 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "identifier", 2024-05-27T09:51:03.924 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.924 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.924 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.924 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.925 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.925 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.925 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.925 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.925 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.925 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.925 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.925 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.926 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:03.926 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:03.926 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.926 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.926 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.926 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:03.926 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.927 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.927 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.927 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.927 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.927 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.927 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.927 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.927 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.928 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.928 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.928 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.928 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.928 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.928 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.928 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.928 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.929 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.929 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.929 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.929 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.929 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.929 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.929 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.930 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.930 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.930 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.930 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.930 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.930 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.930 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.930 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.931 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.931 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.931 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.931 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.931 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.931 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.931 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.931 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.932 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.932 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.932 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.932 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.932 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.932 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.932 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.932 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.933 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.933 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.933 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.933 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.933 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.933 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.933 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.933 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.934 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.934 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.934 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.934 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.934 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.934 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.934 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.934 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.935 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.935 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.935 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.935 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.935 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.935 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.935 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.935 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.935 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.936 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.936 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.936 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.936 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.936 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.936 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.936 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.936 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.937 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.937 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.937 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.937 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.937 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.937 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.937 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.937 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.937 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.938 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.938 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.938 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_host": { 2024-05-27T09:51:03.938 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_host", 2024-05-27T09:51:03.938 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.938 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.938 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.938 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.938 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.939 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.939 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.939 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.939 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.939 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.939 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.939 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.939 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_port": { 2024-05-27T09:51:03.939 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_port", 2024-05-27T09:51:03.940 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.940 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.940 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.940 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10051", 2024-05-27T09:51:03.940 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.940 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.940 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.940 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.941 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.941 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.941 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.941 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.941 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_sender": { 2024-05-27T09:51:03.941 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_sender", 2024-05-27T09:51:03.941 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.941 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.941 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:03.942 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T09:51:03.942 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.942 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.942 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.942 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.942 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.942 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.943 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.943 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.943 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.943 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.943 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:51:03.943 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.943 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.943 INFO:teuthology.orchestra.run.smithi044.stdout: "modules": [ 2024-05-27T09:51:03.944 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2024-05-27T09:51:03.944 INFO:teuthology.orchestra.run.smithi044.stdout: "dashboard", 2024-05-27T09:51:03.944 INFO:teuthology.orchestra.run.smithi044.stdout: "iostat", 2024-05-27T09:51:03.944 INFO:teuthology.orchestra.run.smithi044.stdout: "nfs", 2024-05-27T09:51:03.944 INFO:teuthology.orchestra.run.smithi044.stdout: "restful", 2024-05-27T09:51:03.944 INFO:teuthology.orchestra.run.smithi044.stdout: "snap_schedule" 2024-05-27T09:51:03.944 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.944 INFO:teuthology.orchestra.run.smithi044.stdout: "available_modules": [ 2024-05-27T09:51:03.945 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.945 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alerts", 2024-05-27T09:51:03.945 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.945 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.945 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.945 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:03.945 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:03.945 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:03.946 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.946 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.946 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:03.946 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.946 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.946 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.946 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T09:51:03.946 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.947 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.947 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.947 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.947 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.947 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.947 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.947 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.947 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.947 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.948 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.948 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.948 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.948 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.948 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.948 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.948 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.948 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.949 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.949 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.949 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.949 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.949 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.949 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.949 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.949 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.950 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.950 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.950 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.950 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.950 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.950 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.950 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.950 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.951 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.951 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.951 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.951 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.951 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.951 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.951 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.951 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.952 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.952 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.952 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.952 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.952 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.952 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.952 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.952 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.953 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.953 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.953 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.953 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.953 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.953 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.953 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.953 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.954 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.954 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.954 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.954 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.954 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.954 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.954 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.954 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.955 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.955 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.955 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.955 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.955 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.955 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.955 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.955 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.956 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_destination": { 2024-05-27T09:51:03.956 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_destination", 2024-05-27T09:51:03.956 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.956 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.956 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.956 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.956 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.956 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.957 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.957 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email address to send alerts to", 2024-05-27T09:51:03.957 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.957 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.957 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.957 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.957 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_from_name": { 2024-05-27T09:51:03.957 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_from_name", 2024-05-27T09:51:03.957 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.958 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.958 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.958 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "Ceph", 2024-05-27T09:51:03.958 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.958 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.958 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.958 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email From: name", 2024-05-27T09:51:03.959 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.959 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.959 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.959 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.959 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_host": { 2024-05-27T09:51:03.959 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_host", 2024-05-27T09:51:03.959 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.959 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.960 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.960 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.960 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.960 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.960 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.960 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP server", 2024-05-27T09:51:03.960 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.960 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.961 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.961 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.961 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_password": { 2024-05-27T09:51:03.961 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_password", 2024-05-27T09:51:03.961 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.961 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.961 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.961 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.961 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.962 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.962 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.962 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Password to authenticate with", 2024-05-27T09:51:03.962 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.962 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.962 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.962 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.962 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_port": { 2024-05-27T09:51:03.963 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_port", 2024-05-27T09:51:03.963 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.963 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.963 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.963 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "465", 2024-05-27T09:51:03.963 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.963 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.963 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.963 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP port", 2024-05-27T09:51:03.964 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.964 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.964 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.964 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.964 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_sender": { 2024-05-27T09:51:03.964 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_sender", 2024-05-27T09:51:03.964 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.964 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.965 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.965 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.965 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.965 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.965 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.965 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP envelope sender", 2024-05-27T09:51:03.965 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.966 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.966 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.966 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.966 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_ssl": { 2024-05-27T09:51:03.966 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_ssl", 2024-05-27T09:51:03.966 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.966 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.966 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.966 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.967 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.967 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.967 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.967 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T09:51:03.967 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.967 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.967 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.967 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.967 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_user": { 2024-05-27T09:51:03.968 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_user", 2024-05-27T09:51:03.968 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.968 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.968 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.968 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.968 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.968 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.968 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.968 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "User to authenticate as", 2024-05-27T09:51:03.969 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.969 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.969 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.969 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.969 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:03.969 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:03.969 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:03.969 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:03.970 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.970 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.970 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.970 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.970 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.970 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.970 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.970 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.971 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.971 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.971 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:03.971 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.971 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:03.971 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "balancer", 2024-05-27T09:51:03.971 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:03.971 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:03.972 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:03.972 INFO:teuthology.orchestra.run.smithi044.stdout: "active": { 2024-05-27T09:51:03.972 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "active", 2024-05-27T09:51:03.972 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.972 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.972 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.972 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:03.972 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.972 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.973 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.973 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T09:51:03.973 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.973 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.973 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.973 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.973 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_time": { 2024-05-27T09:51:03.973 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_time", 2024-05-27T09:51:03.974 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.974 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.974 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.974 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0000", 2024-05-27T09:51:03.974 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.974 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.974 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.974 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T09:51:03.975 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T09:51:03.975 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.975 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.975 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.975 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_weekday": { 2024-05-27T09:51:03.975 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_weekday", 2024-05-27T09:51:03.975 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:03.975 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.976 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.976 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.976 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-05-27T09:51:03.976 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-05-27T09:51:03.976 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.976 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T09:51:03.976 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T09:51:03.976 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.977 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.977 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.977 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_max_iterations": { 2024-05-27T09:51:03.977 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_max_iterations", 2024-05-27T09:51:03.977 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:03.977 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.977 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.977 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "25", 2024-05-27T09:51:03.977 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:03.978 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "250", 2024-05-27T09:51:03.978 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.978 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T09:51:03.978 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.978 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.978 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.978 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.979 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_metrics": { 2024-05-27T09:51:03.979 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_metrics", 2024-05-27T09:51:03.979 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.979 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.979 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.979 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T09:51:03.979 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.979 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.980 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.980 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T09:51:03.980 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T09:51:03.980 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.980 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.980 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.980 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_step": { 2024-05-27T09:51:03.980 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_step", 2024-05-27T09:51:03.980 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:03.981 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.981 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.981 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.5", 2024-05-27T09:51:03.981 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0.001", 2024-05-27T09:51:03.981 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "0.999", 2024-05-27T09:51:03.981 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.981 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "aggressiveness of optimization", 2024-05-27T09:51:03.981 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T09:51:03.982 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.982 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.982 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.982 INFO:teuthology.orchestra.run.smithi044.stdout: "end_time": { 2024-05-27T09:51:03.982 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_time", 2024-05-27T09:51:03.982 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.982 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.983 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.983 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2359", 2024-05-27T09:51:03.983 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.983 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.983 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.983 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T09:51:03.984 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T09:51:03.984 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.984 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.984 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.984 INFO:teuthology.orchestra.run.smithi044.stdout: "end_weekday": { 2024-05-27T09:51:03.984 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_weekday", 2024-05-27T09:51:03.984 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:03.985 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.985 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.985 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.985 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-05-27T09:51:03.985 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-05-27T09:51:03.985 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.985 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T09:51:03.986 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T09:51:03.986 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.986 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.986 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.986 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:03.986 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:03.986 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.986 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.987 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.987 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:03.987 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.987 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.987 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.987 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.987 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.987 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.988 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.988 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.988 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.988 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.988 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.988 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.988 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.988 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.989 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.989 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:03.989 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:03.989 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.989 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.989 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.989 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.989 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.990 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.990 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.990 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.990 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.990 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.990 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.990 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.990 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:03.991 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:03.991 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.991 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.991 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.991 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:03.991 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.991 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.992 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.992 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:03.992 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:03.992 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:03.992 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:03.992 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:03.992 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:03.992 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.993 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.993 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.993 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.993 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.993 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.993 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:03.993 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:03.993 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:03.994 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.994 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.994 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:03.994 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.994 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.994 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.994 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:03.994 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.995 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.995 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.995 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.995 INFO:teuthology.orchestra.run.smithi044.stdout: "min_score": { 2024-05-27T09:51:03.995 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_score", 2024-05-27T09:51:03.995 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:03.995 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.995 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.996 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:03.996 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.996 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.996 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:03.996 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T09:51:03.996 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.996 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.996 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.997 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.997 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-05-27T09:51:03.997 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-05-27T09:51:03.997 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:03.997 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:03.997 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:03.997 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "upmap", 2024-05-27T09:51:03.997 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:03.998 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:03.998 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:03.998 INFO:teuthology.orchestra.run.smithi044.stdout: "crush-compat", 2024-05-27T09:51:03.998 INFO:teuthology.orchestra.run.smithi044.stdout: "none", 2024-05-27T09:51:03.998 INFO:teuthology.orchestra.run.smithi044.stdout: "read", 2024-05-27T09:51:03.998 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap", 2024-05-27T09:51:03.998 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap-read" 2024-05-27T09:51:03.998 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:03.999 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Balancer mode", 2024-05-27T09:51:03.999 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:03.999 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:03.999 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:03.999 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:03.999 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_ids": { 2024-05-27T09:51:03.999 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_ids", 2024-05-27T09:51:03.999 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.000 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.000 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.000 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.000 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.000 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.000 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.000 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T09:51:04.001 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.001 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.001 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.001 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.001 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:04.001 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:04.001 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.001 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.001 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.002 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:04.002 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.002 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.002 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.002 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T09:51:04.002 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.002 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.002 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.003 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.003 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.003 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.003 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.003 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.003 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.003 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.003 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.003 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.004 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.004 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.004 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.004 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.004 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.004 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.004 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_deviation": { 2024-05-27T09:51:04.004 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_deviation", 2024-05-27T09:51:04.004 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.005 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.005 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.005 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:04.005 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:04.005 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.005 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.005 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T09:51:04.005 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T09:51:04.006 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.006 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.006 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.006 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_optimizations": { 2024-05-27T09:51:04.006 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_optimizations", 2024-05-27T09:51:04.006 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:04.006 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.006 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.007 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:04.007 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.007 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.007 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.007 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T09:51:04.007 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.007 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.007 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.007 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.008 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.008 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.008 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.008 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm", 2024-05-27T09:51:04.008 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.008 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.008 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.008 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_down_multiplier": { 2024-05-27T09:51:04.008 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_down_multiplier", 2024-05-27T09:51:04.009 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:04.009 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.009 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.009 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-05-27T09:51:04.009 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.009 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.009 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.009 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T09:51:04.010 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.010 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.010 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.010 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.010 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_refresh_rate": { 2024-05-27T09:51:04.010 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_refresh_rate", 2024-05-27T09:51:04.010 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.010 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.011 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.011 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "20", 2024-05-27T09:51:04.011 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.011 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.011 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.011 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T09:51:04.011 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.011 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.012 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.012 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.012 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_starting_port": { 2024-05-27T09:51:04.012 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_starting_port", 2024-05-27T09:51:04.012 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.012 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.012 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.013 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4721", 2024-05-27T09:51:04.013 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.013 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.013 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.013 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T09:51:04.013 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.013 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.014 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.014 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.014 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_ptrace": { 2024-05-27T09:51:04.014 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_ptrace", 2024-05-27T09:51:04.014 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.014 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.014 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.014 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.015 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.015 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.015 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.015 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T09:51:04.015 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T09:51:04.015 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.015 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.015 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.016 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_interval": { 2024-05-27T09:51:04.016 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_interval", 2024-05-27T09:51:04.016 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.016 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.016 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.016 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:04.016 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.017 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.017 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.017 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T09:51:04.017 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.017 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.017 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.017 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.017 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_memory_target_ratio": { 2024-05-27T09:51:04.018 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T09:51:04.018 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:04.018 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.018 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.018 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.7", 2024-05-27T09:51:04.018 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.018 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.018 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.019 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T09:51:04.019 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.019 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.019 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.019 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.019 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm_log_destination": { 2024-05-27T09:51:04.019 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm_log_destination", 2024-05-27T09:51:04.019 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.020 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.020 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.020 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.020 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.020 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.020 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.020 INFO:teuthology.orchestra.run.smithi044.stdout: "file", 2024-05-27T09:51:04.020 INFO:teuthology.orchestra.run.smithi044.stdout: "file,syslog", 2024-05-27T09:51:04.021 INFO:teuthology.orchestra.run.smithi044.stdout: "syslog" 2024-05-27T09:51:04.021 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.021 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T09:51:04.021 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.021 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.021 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.021 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.021 INFO:teuthology.orchestra.run.smithi044.stdout: "cgroups_split": { 2024-05-27T09:51:04.022 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cgroups_split", 2024-05-27T09:51:04.022 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.022 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.022 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.022 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.022 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.022 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.023 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.023 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T09:51:04.023 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.023 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.023 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.023 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.023 INFO:teuthology.orchestra.run.smithi044.stdout: "config_checks_enabled": { 2024-05-27T09:51:04.023 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_checks_enabled", 2024-05-27T09:51:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T09:51:04.025 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.025 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.025 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.025 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.025 INFO:teuthology.orchestra.run.smithi044.stdout: "config_dashboard": { 2024-05-27T09:51:04.025 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_dashboard", 2024-05-27T09:51:04.025 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.026 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.026 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.026 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.026 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.026 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.026 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.026 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T09:51:04.026 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.027 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.027 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.027 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.027 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_alertmanager": { 2024-05-27T09:51:04.027 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_alertmanager", 2024-05-27T09:51:04.027 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.027 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.027 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.028 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T09:51:04.028 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.028 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.028 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.028 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:04.028 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.028 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.028 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.029 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.029 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_base": { 2024-05-27T09:51:04.029 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_base", 2024-05-27T09:51:04.029 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.029 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.029 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.029 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T09:51:04.029 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.029 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.030 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.030 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Container image name, without the tag", 2024-05-27T09:51:04.030 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.030 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.030 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.030 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.030 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_elasticsearch": { 2024-05-27T09:51:04.030 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_elasticsearch", 2024-05-27T09:51:04.030 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.031 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.031 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.031 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T09:51:04.031 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.031 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.031 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.031 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "elasticsearch container image", 2024-05-27T09:51:04.031 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.032 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.032 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.032 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.032 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_grafana": { 2024-05-27T09:51:04.032 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_grafana", 2024-05-27T09:51:04.032 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.032 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.032 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.032 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T09:51:04.033 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.033 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.033 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.033 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:04.033 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.033 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.033 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.033 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.033 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_haproxy": { 2024-05-27T09:51:04.034 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_haproxy", 2024-05-27T09:51:04.034 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.034 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.034 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.034 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T09:51:04.034 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.034 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.034 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.035 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "HAproxy container image", 2024-05-27T09:51:04.035 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.035 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.035 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.035 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.035 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_agent": { 2024-05-27T09:51:04.035 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_agent", 2024-05-27T09:51:04.035 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.036 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.036 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.036 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T09:51:04.036 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.036 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.036 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.036 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger agent container image", 2024-05-27T09:51:04.036 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.036 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.037 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.037 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.037 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_collector": { 2024-05-27T09:51:04.037 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_collector", 2024-05-27T09:51:04.037 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.037 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.037 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.037 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T09:51:04.038 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.038 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.038 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.038 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger collector container image", 2024-05-27T09:51:04.038 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.038 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.038 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.038 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.039 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_query": { 2024-05-27T09:51:04.039 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_query", 2024-05-27T09:51:04.039 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.039 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.039 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.039 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T09:51:04.039 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.039 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.039 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.040 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger query container image", 2024-05-27T09:51:04.040 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.040 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.040 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.040 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.040 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_keepalived": { 2024-05-27T09:51:04.040 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_keepalived", 2024-05-27T09:51:04.040 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.040 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.041 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.041 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T09:51:04.041 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.041 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.041 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.041 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Keepalived container image", 2024-05-27T09:51:04.041 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.041 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.041 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.042 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.042 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_loki": { 2024-05-27T09:51:04.042 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_loki", 2024-05-27T09:51:04.042 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.042 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.042 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.042 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T09:51:04.042 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.042 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.043 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.043 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Loki container image", 2024-05-27T09:51:04.043 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.043 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.043 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.043 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.043 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_node_exporter": { 2024-05-27T09:51:04.043 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_node_exporter", 2024-05-27T09:51:04.043 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.044 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.044 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.044 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T09:51:04.044 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.044 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.044 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.044 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:04.044 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.044 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.045 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.045 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.045 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_nvmeof": { 2024-05-27T09:51:04.045 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_nvmeof", 2024-05-27T09:51:04.045 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.045 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.045 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.045 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T09:51:04.045 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.046 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.046 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.046 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Nvme-of container image", 2024-05-27T09:51:04.046 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.046 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.046 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.046 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.046 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_prometheus": { 2024-05-27T09:51:04.047 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_prometheus", 2024-05-27T09:51:04.047 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.047 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.047 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.047 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T09:51:04.047 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.047 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.047 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.048 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-05-27T09:51:04.048 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.048 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.048 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.048 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.048 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_promtail": { 2024-05-27T09:51:04.048 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_promtail", 2024-05-27T09:51:04.048 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.048 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.049 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.049 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T09:51:04.049 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.049 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.049 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.049 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Promtail container image", 2024-05-27T09:51:04.049 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.049 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.050 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.050 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.050 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_samba": { 2024-05-27T09:51:04.050 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_samba", 2024-05-27T09:51:04.050 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.050 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.050 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.050 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T09:51:04.051 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.051 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.051 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.051 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Samba/SMB container image", 2024-05-27T09:51:04.051 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.051 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.051 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.051 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.052 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_snmp_gateway": { 2024-05-27T09:51:04.052 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_snmp_gateway", 2024-05-27T09:51:04.052 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.052 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.052 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.052 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T09:51:04.052 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.052 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.053 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.053 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SNMP Gateway container image", 2024-05-27T09:51:04.053 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.053 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.053 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.053 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.053 INFO:teuthology.orchestra.run.smithi044.stdout: "container_init": { 2024-05-27T09:51:04.053 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_init", 2024-05-27T09:51:04.053 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.054 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.054 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.054 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.054 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.054 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.054 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.054 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T09:51:04.054 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.055 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.055 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.055 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.055 INFO:teuthology.orchestra.run.smithi044.stdout: "daemon_cache_timeout": { 2024-05-27T09:51:04.055 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "daemon_cache_timeout", 2024-05-27T09:51:04.055 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.055 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.055 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.056 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:04.056 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.056 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.056 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.056 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T09:51:04.056 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.056 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.056 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.057 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.057 INFO:teuthology.orchestra.run.smithi044.stdout: "default_cephadm_command_timeout": { 2024-05-27T09:51:04.057 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T09:51:04.057 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.057 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.057 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.057 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "900", 2024-05-27T09:51:04.057 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.057 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.058 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.058 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T09:51:04.058 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.058 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.058 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.058 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.058 INFO:teuthology.orchestra.run.smithi044.stdout: "default_registry": { 2024-05-27T09:51:04.058 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_registry", 2024-05-27T09:51:04.059 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.059 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.059 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.059 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io", 2024-05-27T09:51:04.059 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.059 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.059 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.059 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T09:51:04.059 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.060 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.060 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.060 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.060 INFO:teuthology.orchestra.run.smithi044.stdout: "device_cache_timeout": { 2024-05-27T09:51:04.060 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_cache_timeout", 2024-05-27T09:51:04.060 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.060 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.060 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.061 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1800", 2024-05-27T09:51:04.061 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.061 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.061 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.061 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache device inventory", 2024-05-27T09:51:04.061 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.061 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.061 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.062 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.062 INFO:teuthology.orchestra.run.smithi044.stdout: "device_enhanced_scan": { 2024-05-27T09:51:04.062 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_enhanced_scan", 2024-05-27T09:51:04.062 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.062 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.062 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.062 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.062 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.063 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.063 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.063 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T09:51:04.063 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.063 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.063 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.063 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.063 INFO:teuthology.orchestra.run.smithi044.stdout: "facts_cache_timeout": { 2024-05-27T09:51:04.063 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "facts_cache_timeout", 2024-05-27T09:51:04.064 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.064 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.064 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.064 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:04.064 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.064 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.064 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.064 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache host facts data", 2024-05-27T09:51:04.065 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.065 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.065 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.065 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.065 INFO:teuthology.orchestra.run.smithi044.stdout: "grafana_dashboards_path": { 2024-05-27T09:51:04.065 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "grafana_dashboards_path", 2024-05-27T09:51:04.065 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.065 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.066 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.066 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T09:51:04.066 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.066 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.066 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.066 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T09:51:04.066 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.066 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.066 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.067 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.067 INFO:teuthology.orchestra.run.smithi044.stdout: "host_check_interval": { 2024-05-27T09:51:04.067 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "host_check_interval", 2024-05-27T09:51:04.067 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.067 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.067 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.067 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:04.067 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.067 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.068 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.068 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to perform a host check", 2024-05-27T09:51:04.068 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.068 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.068 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.068 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.068 INFO:teuthology.orchestra.run.smithi044.stdout: "hw_monitoring": { 2024-05-27T09:51:04.068 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hw_monitoring", 2024-05-27T09:51:04.069 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.069 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.069 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.069 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.069 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.069 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.069 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.069 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T09:51:04.069 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.070 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.070 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.070 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.070 INFO:teuthology.orchestra.run.smithi044.stdout: "inventory_list_all": { 2024-05-27T09:51:04.070 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "inventory_list_all", 2024-05-27T09:51:04.070 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.070 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.070 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.070 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.071 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.071 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.071 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.071 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T09:51:04.071 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.071 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.071 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.071 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.072 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.072 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.072 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.072 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.072 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.072 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.072 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.072 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.072 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.072 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.073 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.073 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.073 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.073 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.073 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.073 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.073 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.073 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.074 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.074 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.074 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.074 INFO:teuthology.orchestra.run.smithi044.stdout: "log_refresh_metadata": { 2024-05-27T09:51:04.074 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_refresh_metadata", 2024-05-27T09:51:04.074 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.074 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.074 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.075 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.075 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.075 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.075 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.075 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T09:51:04.075 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.075 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.075 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.076 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.076 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.076 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.076 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.076 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.076 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.076 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.076 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.077 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.077 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.077 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T09:51:04.077 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.077 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.077 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.077 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.077 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.077 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.078 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.078 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.078 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.078 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.078 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.078 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.078 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.078 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.079 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.079 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.079 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.079 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.079 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.079 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.079 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.079 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.079 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.080 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.080 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.080 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.080 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.080 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.080 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.080 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.080 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.080 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.081 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.081 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.081 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.081 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.081 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.081 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.081 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.081 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T09:51:04.082 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T09:51:04.082 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.082 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.082 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.082 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.082 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.082 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.082 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.083 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T09:51:04.083 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.083 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.083 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.083 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.083 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T09:51:04.083 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T09:51:04.083 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.084 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.084 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.084 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "*", 2024-05-27T09:51:04.084 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.084 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.084 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.084 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T09:51:04.084 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.085 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.085 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.085 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.085 INFO:teuthology.orchestra.run.smithi044.stdout: "max_count_per_host": { 2024-05-27T09:51:04.085 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_count_per_host", 2024-05-27T09:51:04.085 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.085 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.085 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.085 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:04.086 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.086 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.086 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.086 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of daemons per service per host", 2024-05-27T09:51:04.086 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.086 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.086 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.086 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.087 INFO:teuthology.orchestra.run.smithi044.stdout: "max_osd_draining_count": { 2024-05-27T09:51:04.087 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_osd_draining_count", 2024-05-27T09:51:04.087 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.087 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.087 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.087 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:04.088 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.088 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.088 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.088 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T09:51:04.088 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.088 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.088 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.088 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.088 INFO:teuthology.orchestra.run.smithi044.stdout: "migration_current": { 2024-05-27T09:51:04.089 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "migration_current", 2024-05-27T09:51:04.089 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.089 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.089 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.089 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.089 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.089 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.089 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.090 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "internal - do not modify", 2024-05-27T09:51:04.090 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.090 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.090 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.090 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.090 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-05-27T09:51:04.090 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-05-27T09:51:04.090 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.091 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.091 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.091 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "root", 2024-05-27T09:51:04.091 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.091 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.091 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.091 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm-package", 2024-05-27T09:51:04.091 INFO:teuthology.orchestra.run.smithi044.stdout: "root" 2024-05-27T09:51:04.092 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.092 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T09:51:04.092 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.092 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.092 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.092 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.092 INFO:teuthology.orchestra.run.smithi044.stdout: "oob_default_addr": { 2024-05-27T09:51:04.092 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "oob_default_addr", 2024-05-27T09:51:04.092 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.093 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.093 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.093 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "169.254.1.1", 2024-05-27T09:51:04.093 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.093 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.093 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.093 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T09:51:04.093 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.093 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.094 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.094 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.094 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_alerts_path": { 2024-05-27T09:51:04.094 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_alerts_path", 2024-05-27T09:51:04.094 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.094 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.094 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.095 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T09:51:04.095 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.095 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.095 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.095 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T09:51:04.095 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.095 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.095 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.096 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.096 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_insecure": { 2024-05-27T09:51:04.096 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_insecure", 2024-05-27T09:51:04.096 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.096 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.096 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.096 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.096 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.096 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.097 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.097 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T09:51:04.097 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.097 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.097 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.097 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.097 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_password": { 2024-05-27T09:51:04.097 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_password", 2024-05-27T09:51:04.098 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.098 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.098 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.098 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.098 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.098 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.098 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.098 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T09:51:04.099 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.099 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.099 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.099 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.099 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_url": { 2024-05-27T09:51:04.099 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_url", 2024-05-27T09:51:04.099 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.099 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.100 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.100 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.100 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.100 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.100 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.100 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T09:51:04.100 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.100 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.100 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.101 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.101 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_username": { 2024-05-27T09:51:04.101 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_username", 2024-05-27T09:51:04.101 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.101 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.101 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.101 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.101 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.102 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.102 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.102 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T09:51:04.102 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.102 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.102 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.102 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.102 INFO:teuthology.orchestra.run.smithi044.stdout: "secure_monitoring_stack": { 2024-05-27T09:51:04.103 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "secure_monitoring_stack", 2024-05-27T09:51:04.103 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.103 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.103 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.103 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.103 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.103 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.103 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.103 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T09:51:04.104 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.104 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.104 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.104 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.104 INFO:teuthology.orchestra.run.smithi044.stdout: "service_discovery_port": { 2024-05-27T09:51:04.104 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "service_discovery_port", 2024-05-27T09:51:04.104 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.104 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.104 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.105 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8765", 2024-05-27T09:51:04.105 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.105 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.105 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.105 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "cephadm service discovery port", 2024-05-27T09:51:04.105 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.105 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.105 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.106 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.106 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.106 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.106 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.106 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.106 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.106 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.106 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.106 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.107 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.107 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.107 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.107 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.107 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.107 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.107 INFO:teuthology.orchestra.run.smithi044.stdout: "ssh_config_file": { 2024-05-27T09:51:04.107 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssh_config_file", 2024-05-27T09:51:04.107 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.108 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.108 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.108 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.108 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.108 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.108 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.108 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T09:51:04.108 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: "use_agent": { 2024-05-27T09:51:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_agent", 2024-05-27T09:51:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.109 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.110 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.110 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.110 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.110 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T09:51:04.110 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.110 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.110 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.110 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.111 INFO:teuthology.orchestra.run.smithi044.stdout: "use_repo_digest": { 2024-05-27T09:51:04.111 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_repo_digest", 2024-05-27T09:51:04.111 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.111 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.111 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.111 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.111 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.111 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.111 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.112 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T09:51:04.112 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.112 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.112 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.112 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.112 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_failed_host_check": { 2024-05-27T09:51:04.112 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_failed_host_check", 2024-05-27T09:51:04.113 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.113 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.113 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.113 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.113 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.113 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.113 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.113 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T09:51:04.114 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.114 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.114 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.114 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.114 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_daemons": { 2024-05-27T09:51:04.114 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_daemons", 2024-05-27T09:51:04.114 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.114 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.115 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.115 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.115 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.115 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.115 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.115 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T09:51:04.115 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.115 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.116 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.116 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.116 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_hosts": { 2024-05-27T09:51:04.116 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_hosts", 2024-05-27T09:51:04.116 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.116 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.116 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.116 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.117 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.117 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.117 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.117 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T09:51:04.117 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.117 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.117 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.117 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.118 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.118 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.118 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.118 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crash", 2024-05-27T09:51:04.118 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.118 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.118 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.118 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.119 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.119 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.119 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.119 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.119 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.119 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.119 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.119 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.120 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.120 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.120 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.120 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.120 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.120 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.120 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.120 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.121 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.121 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.121 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.121 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.121 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.121 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.121 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.121 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.122 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.122 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.122 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.122 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.122 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.122 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.122 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.122 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.123 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.123 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.123 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.123 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.123 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.123 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.123 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.123 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.124 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.124 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.124 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.124 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.124 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.124 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.124 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.124 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.125 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.125 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.125 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.125 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.125 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.125 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.125 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.125 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.126 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.126 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.126 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.126 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.126 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.126 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.126 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.126 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.126 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.127 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.127 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.127 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.127 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.127 INFO:teuthology.orchestra.run.smithi044.stdout: "retain_interval": { 2024-05-27T09:51:04.127 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retain_interval", 2024-05-27T09:51:04.127 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.127 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.128 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.128 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "31536000", 2024-05-27T09:51:04.128 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.128 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.128 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.128 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T09:51:04.128 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.128 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.129 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.129 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.129 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.129 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.129 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.129 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.129 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.129 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.130 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.130 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.130 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.130 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.130 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.130 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.130 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.130 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.130 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_recent_interval": { 2024-05-27T09:51:04.131 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_recent_interval", 2024-05-27T09:51:04.131 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.131 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.131 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.131 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1209600", 2024-05-27T09:51:04.131 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.131 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.131 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.132 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T09:51:04.132 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.132 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.132 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.132 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.132 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.132 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.132 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.132 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dashboard", 2024-05-27T09:51:04.133 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.133 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.133 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.133 INFO:teuthology.orchestra.run.smithi044.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T09:51:04.133 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T09:51:04.133 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.133 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.133 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.133 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:04.134 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.134 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.134 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.134 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.134 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.134 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.134 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.134 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.135 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T09:51:04.135 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T09:51:04.135 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.135 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.135 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.135 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.135 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.136 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.136 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.136 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.136 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.136 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.136 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.136 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.136 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T09:51:04.136 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T09:51:04.137 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.137 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.137 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.137 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.137 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.137 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.137 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.137 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.137 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.138 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.138 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.138 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.138 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_ENABLED": { 2024-05-27T09:51:04.138 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T09:51:04.138 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.138 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.138 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.139 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.139 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.139 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.139 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.139 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.139 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.139 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.139 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.139 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.140 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T09:51:04.140 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T09:51:04.140 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.140 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.140 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.140 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.140 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.140 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.141 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.141 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.141 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.141 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.141 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.141 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.141 INFO:teuthology.orchestra.run.smithi044.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T09:51:04.141 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T09:51:04.141 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.142 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.142 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.142 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.142 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.142 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.142 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.142 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.142 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.143 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.143 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.143 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.143 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T09:51:04.143 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T09:51:04.143 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.143 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.143 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.143 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.144 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.144 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.144 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.144 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.144 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.144 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.144 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.145 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.145 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T09:51:04.145 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T09:51:04.145 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.145 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.145 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.145 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.145 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.146 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.146 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.146 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.146 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.146 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.146 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.146 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.147 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T09:51:04.147 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T09:51:04.147 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.147 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.147 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.147 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.147 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.147 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.148 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.148 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.148 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.148 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.148 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.148 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.148 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T09:51:04.148 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T09:51:04.149 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.149 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.149 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.149 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.149 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.149 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.149 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.149 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.149 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.150 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.150 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.150 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.150 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T09:51:04.150 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T09:51:04.150 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.150 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.150 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.151 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.151 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.151 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.151 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.151 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.151 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.151 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.151 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.152 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.152 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T09:51:04.152 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T09:51:04.152 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.152 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.152 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.152 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.153 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.153 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.153 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.153 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.153 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.153 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.153 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.153 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.154 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T09:51:04.154 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T09:51:04.154 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.154 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.154 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.154 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.154 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.154 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.155 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.155 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.155 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.155 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.155 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.155 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.155 INFO:teuthology.orchestra.run.smithi044.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T09:51:04.155 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T09:51:04.156 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.156 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.156 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.156 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.156 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.156 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.156 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.156 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.157 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.157 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.157 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.157 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.157 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T09:51:04.157 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T09:51:04.157 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.157 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.158 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.158 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-05-27T09:51:04.158 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.158 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.158 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.158 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.158 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.159 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.159 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.159 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.159 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T09:51:04.159 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T09:51:04.159 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.159 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.159 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.159 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.160 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.161 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_URL": { 2024-05-27T09:51:04.161 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_URL", 2024-05-27T09:51:04.161 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.161 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.161 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.161 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.161 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.161 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.161 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.162 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.162 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.162 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.162 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.162 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.162 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T09:51:04.162 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T09:51:04.162 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.163 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.163 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.163 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-05-27T09:51:04.163 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.163 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.163 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.163 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.163 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.163 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.164 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.164 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.164 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T09:51:04.164 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T09:51:04.164 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.164 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.164 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.164 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.165 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.165 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.165 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.165 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.165 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.165 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.165 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.165 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.165 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T09:51:04.166 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T09:51:04.166 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.166 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.166 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.166 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.166 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.166 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.166 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.167 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.167 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.167 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.167 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.167 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.167 INFO:teuthology.orchestra.run.smithi044.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T09:51:04.167 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T09:51:04.167 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.167 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.168 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.168 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.168 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.168 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.168 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.168 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.168 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.168 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.168 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.168 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.169 INFO:teuthology.orchestra.run.smithi044.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T09:51:04.169 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T09:51:04.169 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.169 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.169 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.169 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.169 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.169 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.169 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.170 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.170 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.170 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.170 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.170 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.170 INFO:teuthology.orchestra.run.smithi044.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T09:51:04.170 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T09:51:04.170 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.171 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.171 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.171 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "[]", 2024-05-27T09:51:04.171 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.171 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.171 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.171 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.171 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.171 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.172 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.172 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.172 INFO:teuthology.orchestra.run.smithi044.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T09:51:04.172 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T09:51:04.172 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.172 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.172 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.172 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "{}", 2024-05-27T09:51:04.172 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.173 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.173 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.173 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.173 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.173 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.173 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.173 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.173 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T09:51:04.174 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T09:51:04.174 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.174 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.174 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.174 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.174 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.174 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.174 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.174 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.175 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.175 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.175 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.175 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.175 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T09:51:04.175 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T09:51:04.175 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.175 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.176 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.176 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.176 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.176 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.176 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.176 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.176 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.176 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.176 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.177 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.177 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T09:51:04.177 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T09:51:04.177 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.177 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.177 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.177 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.177 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.178 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.178 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.178 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.178 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.178 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.178 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.178 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.178 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T09:51:04.179 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T09:51:04.179 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.179 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.179 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.179 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.179 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.179 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.179 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.180 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.180 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.180 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.180 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.180 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.180 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T09:51:04.180 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T09:51:04.180 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.180 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.181 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.181 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.181 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.181 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.181 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.181 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.181 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.181 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.181 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.182 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.182 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T09:51:04.182 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T09:51:04.182 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.182 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.182 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.182 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.182 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.183 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.183 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.183 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.183 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.183 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.183 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.183 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.183 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T09:51:04.184 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T09:51:04.184 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.184 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.184 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.184 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.184 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.184 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.184 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.185 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.185 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.185 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.185 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.185 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.185 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T09:51:04.185 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T09:51:04.185 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.186 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.186 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.186 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.186 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.186 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.186 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.186 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.187 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.187 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.187 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.187 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.187 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T09:51:04.187 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T09:51:04.187 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.187 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.188 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.188 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.188 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.188 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.188 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.188 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.188 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.188 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.189 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.189 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.189 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T09:51:04.189 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T09:51:04.189 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.189 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.189 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.189 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.189 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.190 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.190 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.190 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.190 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.190 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.190 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.190 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.191 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T09:51:04.191 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T09:51:04.191 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.191 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.191 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.191 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T09:51:04.191 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.191 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.191 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.192 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.192 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.192 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.192 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.192 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.192 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T09:51:04.192 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T09:51:04.192 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.193 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.193 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.193 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:04.193 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.193 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.193 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.193 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.193 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.194 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.194 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.194 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.194 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T09:51:04.194 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T09:51:04.194 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.194 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.195 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.195 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8", 2024-05-27T09:51:04.195 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.195 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.195 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.195 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.195 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.195 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.196 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.196 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.196 INFO:teuthology.orchestra.run.smithi044.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T09:51:04.196 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T09:51:04.196 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.196 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.196 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.196 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "45", 2024-05-27T09:51:04.197 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.197 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.197 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.197 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.197 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.197 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.197 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.197 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.197 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T09:51:04.198 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T09:51:04.198 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.198 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.198 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.198 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.198 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.198 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.198 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.199 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.199 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.199 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.199 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.199 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.199 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T09:51:04.199 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T09:51:04.199 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.200 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.200 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.200 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-05-27T09:51:04.200 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.200 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.200 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.200 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.201 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.201 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.201 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.201 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.201 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T09:51:04.201 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T09:51:04.201 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.201 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.202 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.202 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.202 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.202 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.202 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.202 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.202 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.202 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.202 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.203 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.203 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T09:51:04.203 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T09:51:04.203 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.203 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.203 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.203 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.203 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.204 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.204 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.204 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.204 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.204 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.204 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.204 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.204 INFO:teuthology.orchestra.run.smithi044.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T09:51:04.204 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T09:51:04.205 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.205 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.205 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.205 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.205 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.205 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.205 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.205 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.206 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.206 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.206 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.206 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.206 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T09:51:04.206 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T09:51:04.206 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.206 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.206 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.207 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.207 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.207 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.207 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.207 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.207 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.207 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.207 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.208 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.208 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T09:51:04.208 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T09:51:04.208 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.208 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.208 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.208 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:04.208 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.209 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.209 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.209 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.209 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.209 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.209 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.209 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.209 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T09:51:04.209 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T09:51:04.210 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.210 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.210 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.210 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:04.210 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.210 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.210 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.210 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.210 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.211 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.211 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.211 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.211 INFO:teuthology.orchestra.run.smithi044.stdout: "cross_origin_url": { 2024-05-27T09:51:04.211 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cross_origin_url", 2024-05-27T09:51:04.211 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.211 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.211 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.211 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.212 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.212 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.212 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.212 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.212 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.212 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.212 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.213 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.213 INFO:teuthology.orchestra.run.smithi044.stdout: "crt_file": { 2024-05-27T09:51:04.213 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crt_file", 2024-05-27T09:51:04.213 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.213 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.213 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.213 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.213 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.214 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.214 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.214 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.214 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.214 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.214 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.214 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.214 INFO:teuthology.orchestra.run.smithi044.stdout: "debug": { 2024-05-27T09:51:04.215 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "debug", 2024-05-27T09:51:04.215 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.215 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.215 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.215 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.215 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.215 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.215 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.216 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable/disable debug options", 2024-05-27T09:51:04.216 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.216 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.216 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.216 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.216 INFO:teuthology.orchestra.run.smithi044.stdout: "jwt_token_ttl": { 2024-05-27T09:51:04.216 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "jwt_token_ttl", 2024-05-27T09:51:04.216 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.217 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.217 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.217 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "28800", 2024-05-27T09:51:04.217 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.217 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.217 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.217 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.217 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.218 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.218 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.218 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.218 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-05-27T09:51:04.218 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-05-27T09:51:04.218 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.218 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.218 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.219 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.219 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.219 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.219 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.219 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.219 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.219 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.219 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.220 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.220 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.220 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.220 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.220 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.220 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.220 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.221 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.221 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.221 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.221 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.221 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.221 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.221 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.221 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.222 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.222 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.222 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.222 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.222 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.222 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.222 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.223 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.223 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.223 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.223 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.223 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.223 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.223 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.223 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.223 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.224 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.224 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.224 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.224 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.224 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.224 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.224 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.224 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.225 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.225 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.225 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.225 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.225 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.225 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.225 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.226 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.226 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.226 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.226 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.226 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.226 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.226 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.226 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.227 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.227 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.227 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.227 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.227 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.227 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.227 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.227 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.228 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.228 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.228 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.228 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.228 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.228 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.228 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.228 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.228 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.229 INFO:teuthology.orchestra.run.smithi044.stdout: "motd": { 2024-05-27T09:51:04.229 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "motd", 2024-05-27T09:51:04.229 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.229 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.229 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.229 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.229 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.229 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.230 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.230 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "The message of the day", 2024-05-27T09:51:04.230 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.230 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.230 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.230 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.230 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect_resolve_ip_addr": { 2024-05-27T09:51:04.231 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T09:51:04.231 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.231 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.231 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.231 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.231 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.231 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.231 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.231 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.232 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.232 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.232 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.232 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.232 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-05-27T09:51:04.232 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-05-27T09:51:04.232 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.232 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.233 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.233 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-05-27T09:51:04.233 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.233 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.233 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.233 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.233 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.233 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.233 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.234 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.234 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-05-27T09:51:04.234 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-05-27T09:51:04.234 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.234 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.234 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.234 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8080", 2024-05-27T09:51:04.234 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.235 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.235 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.235 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.235 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.235 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.235 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.235 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.235 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.235 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.236 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.236 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.236 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.236 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.236 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.236 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.236 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.236 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.236 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-05-27T09:51:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-05-27T09:51:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.238 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.238 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.238 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.238 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.238 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.238 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.238 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.238 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.238 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl_server_port": { 2024-05-27T09:51:04.239 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl_server_port", 2024-05-27T09:51:04.239 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.239 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.239 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.239 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8443", 2024-05-27T09:51:04.239 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.239 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.239 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.239 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.240 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.240 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.240 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.240 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.240 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-05-27T09:51:04.240 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-05-27T09:51:04.240 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.240 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.240 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.241 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "redirect", 2024-05-27T09:51:04.241 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.241 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.241 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.241 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.241 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect" 2024-05-27T09:51:04.241 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.241 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.241 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.242 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.242 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.242 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.242 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-05-27T09:51:04.242 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-05-27T09:51:04.242 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.242 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.242 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.242 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-05-27T09:51:04.243 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-05-27T09:51:04.243 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-05-27T09:51:04.243 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.243 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.243 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.243 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.243 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.243 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.243 INFO:teuthology.orchestra.run.smithi044.stdout: "url_prefix": { 2024-05-27T09:51:04.244 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url_prefix", 2024-05-27T09:51:04.244 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.244 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.244 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.244 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.244 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.244 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.244 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.245 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.245 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.245 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.245 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.245 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.245 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.245 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.245 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.246 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2024-05-27T09:51:04.246 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.246 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.246 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.246 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_monitoring": { 2024-05-27T09:51:04.246 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_monitoring", 2024-05-27T09:51:04.246 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.246 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.247 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.247 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.247 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.247 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.247 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.247 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "monitor device health metrics", 2024-05-27T09:51:04.247 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.247 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.247 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.248 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.248 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.248 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.248 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.248 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.248 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.248 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.248 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.248 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.249 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.249 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.249 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.249 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.249 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.249 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.249 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.249 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.249 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.250 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.250 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.250 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.250 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.250 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.250 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.250 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.250 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.251 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.251 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.251 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.251 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.251 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.251 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.251 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.251 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.251 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.252 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.252 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.252 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.252 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.252 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.252 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.252 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.252 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.253 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.253 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.253 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.253 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.253 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.253 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.253 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.253 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.254 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.254 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.254 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.254 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.254 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.254 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.254 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.254 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.255 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.255 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.255 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.255 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.255 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.255 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.255 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.255 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.256 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.256 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.256 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.256 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.256 INFO:teuthology.orchestra.run.smithi044.stdout: "mark_out_threshold": { 2024-05-27T09:51:04.256 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mark_out_threshold", 2024-05-27T09:51:04.256 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.257 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.257 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.257 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2419200", 2024-05-27T09:51:04.257 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.257 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.257 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.257 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T09:51:04.257 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.258 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.258 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.258 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.258 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_name": { 2024-05-27T09:51:04.258 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_name", 2024-05-27T09:51:04.258 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.258 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.258 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.259 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "device_health_metrics", 2024-05-27T09:51:04.259 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.259 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.259 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.259 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T09:51:04.259 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.259 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.259 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.260 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.260 INFO:teuthology.orchestra.run.smithi044.stdout: "retention_period": { 2024-05-27T09:51:04.260 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retention_period", 2024-05-27T09:51:04.260 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.260 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.260 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.260 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15552000", 2024-05-27T09:51:04.260 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.261 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.261 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.261 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain device health metrics", 2024-05-27T09:51:04.261 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.261 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.261 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.261 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.261 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_frequency": { 2024-05-27T09:51:04.262 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_frequency", 2024-05-27T09:51:04.262 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.262 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.262 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.262 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-05-27T09:51:04.262 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.262 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.262 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.263 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T09:51:04.263 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.263 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.263 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.263 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.263 INFO:teuthology.orchestra.run.smithi044.stdout: "self_heal": { 2024-05-27T09:51:04.263 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "self_heal", 2024-05-27T09:51:04.263 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.264 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.264 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.264 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.264 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.264 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.264 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.264 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T09:51:04.264 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.265 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.265 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.265 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.265 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:04.265 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:04.265 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.265 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.265 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.265 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:04.266 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.266 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.266 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.266 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T09:51:04.266 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.266 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.266 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.266 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.267 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.267 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.267 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.267 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.267 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.267 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.267 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.267 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.268 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.268 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.268 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.268 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.268 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.268 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.268 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_threshold": { 2024-05-27T09:51:04.268 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_threshold", 2024-05-27T09:51:04.268 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.269 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.269 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.269 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7257600", 2024-05-27T09:51:04.269 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.269 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.269 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.269 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T09:51:04.269 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.270 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.270 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.270 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.270 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.270 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.270 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.270 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "diskprediction_local", 2024-05-27T09:51:04.270 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.271 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.271 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.271 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.271 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.271 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.271 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.271 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.271 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.271 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.272 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.272 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.272 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.272 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.272 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.272 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.272 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.272 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.272 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.273 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.273 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.273 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.273 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.273 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.273 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.273 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.273 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.273 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.274 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.274 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.274 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.274 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.274 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.274 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.274 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.274 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.274 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.275 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.275 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.275 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.275 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.275 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.275 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.275 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.276 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.276 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.276 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.276 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.276 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.276 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.276 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.276 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.277 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.277 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.277 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.277 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.277 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.277 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.277 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.277 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.278 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.278 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.278 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.278 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.278 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.278 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.278 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.278 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.279 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.279 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.279 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.279 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.279 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.279 INFO:teuthology.orchestra.run.smithi044.stdout: "predict_interval": { 2024-05-27T09:51:04.279 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predict_interval", 2024-05-27T09:51:04.280 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.280 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.280 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.280 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-05-27T09:51:04.280 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.280 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.280 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.281 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.281 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.281 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.281 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.281 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.281 INFO:teuthology.orchestra.run.smithi044.stdout: "predictor_model": { 2024-05-27T09:51:04.281 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predictor_model", 2024-05-27T09:51:04.281 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.282 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.282 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.282 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "prophetstor", 2024-05-27T09:51:04.282 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.282 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.282 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.282 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.282 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.283 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.283 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.283 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.283 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:04.283 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:04.283 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.283 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.283 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.284 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-05-27T09:51:04.284 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.284 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.284 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.284 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.284 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.284 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.285 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.285 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.285 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.285 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.285 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.285 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.285 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.285 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.286 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.286 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.286 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.286 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.286 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.286 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.286 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.286 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.287 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.287 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.287 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.287 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "influx", 2024-05-27T09:51:04.287 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": false, 2024-05-27T09:51:04.287 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "influxdb python module not found", 2024-05-27T09:51:04.287 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.287 INFO:teuthology.orchestra.run.smithi044.stdout: "batch_size": { 2024-05-27T09:51:04.287 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "batch_size", 2024-05-27T09:51:04.288 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.288 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.288 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.288 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5000", 2024-05-27T09:51:04.288 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.288 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.288 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.288 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T09:51:04.288 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.289 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.289 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.289 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.289 INFO:teuthology.orchestra.run.smithi044.stdout: "database": { 2024-05-27T09:51:04.289 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "database", 2024-05-27T09:51:04.289 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.289 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.289 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.289 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "ceph", 2024-05-27T09:51:04.290 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.290 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.290 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.290 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T09:51:04.290 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.290 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.290 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.290 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.291 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": { 2024-05-27T09:51:04.291 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hostname", 2024-05-27T09:51:04.291 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.291 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.291 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.291 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.291 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.291 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.291 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.292 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server hostname", 2024-05-27T09:51:04.292 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.292 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.292 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.292 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.292 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:04.292 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:04.293 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.293 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.293 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.293 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "30", 2024-05-27T09:51:04.293 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "5", 2024-05-27T09:51:04.293 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.293 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.293 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T09:51:04.293 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.294 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.294 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.294 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.294 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.294 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.294 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.294 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.295 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.295 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.295 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.295 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.295 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.295 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.295 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.295 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.296 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.296 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.296 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.296 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.296 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.296 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.296 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.296 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.297 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.297 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.297 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.297 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.297 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.297 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.297 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.298 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.298 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.298 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.298 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.298 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.298 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.298 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.298 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.299 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.299 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.299 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.299 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.299 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.299 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.299 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.299 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.300 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.300 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.300 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.300 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.300 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.300 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.300 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.300 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.301 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.301 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.301 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.301 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.301 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.301 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.301 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.301 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.302 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.302 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.302 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.302 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.302 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.302 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.302 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.303 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.303 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.303 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.303 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.303 INFO:teuthology.orchestra.run.smithi044.stdout: "password": { 2024-05-27T09:51:04.303 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "password", 2024-05-27T09:51:04.303 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.303 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.304 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.304 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.304 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.304 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.304 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.304 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "password of InfluxDB server user", 2024-05-27T09:51:04.304 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.304 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.305 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.305 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.305 INFO:teuthology.orchestra.run.smithi044.stdout: "port": { 2024-05-27T09:51:04.305 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "port", 2024-05-27T09:51:04.305 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.305 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.305 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.305 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8086", 2024-05-27T09:51:04.306 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.306 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.306 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.306 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server port", 2024-05-27T09:51:04.306 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.306 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.306 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.307 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.307 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.307 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.307 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.307 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.307 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.307 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.307 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.308 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.308 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.308 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.308 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.308 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.308 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.308 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.308 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-05-27T09:51:04.309 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-05-27T09:51:04.309 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.309 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.309 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.309 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "false", 2024-05-27T09:51:04.309 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.309 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.309 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.310 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T09:51:04.310 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.310 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.310 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.310 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.310 INFO:teuthology.orchestra.run.smithi044.stdout: "threads": { 2024-05-27T09:51:04.310 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threads", 2024-05-27T09:51:04.310 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.311 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.311 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.311 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:04.311 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:04.311 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "32", 2024-05-27T09:51:04.311 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.311 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T09:51:04.311 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.311 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.312 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.312 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.312 INFO:teuthology.orchestra.run.smithi044.stdout: "username": { 2024-05-27T09:51:04.312 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "username", 2024-05-27T09:51:04.312 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.312 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.312 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.312 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.312 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.313 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.313 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.313 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "username of InfluxDB server user", 2024-05-27T09:51:04.313 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.313 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.313 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.313 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.313 INFO:teuthology.orchestra.run.smithi044.stdout: "verify_ssl": { 2024-05-27T09:51:04.313 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "verify_ssl", 2024-05-27T09:51:04.314 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.314 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.314 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.314 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "true", 2024-05-27T09:51:04.314 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.314 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.314 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.315 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T09:51:04.315 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.315 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.315 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.315 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.315 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.315 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.316 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.316 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "insights", 2024-05-27T09:51:04.316 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.316 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.316 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.316 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.316 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.316 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.317 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.317 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.317 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.317 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.317 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.317 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.317 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.317 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.317 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.318 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.318 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.318 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.318 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.318 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.318 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.318 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.319 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.319 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.319 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.319 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.319 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.319 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.319 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.319 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.319 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.320 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.320 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.320 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.320 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.320 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.320 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.320 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.320 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.321 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.321 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.321 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.321 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.321 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.321 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.321 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.321 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.322 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.322 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.322 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.322 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.322 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.322 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.322 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.322 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.323 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.323 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.323 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.323 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.323 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.323 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.323 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.323 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.324 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.324 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.324 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.324 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.324 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.324 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.324 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.324 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.324 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.325 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.325 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.325 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.325 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.325 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.325 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.325 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.325 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.326 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.326 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.326 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.326 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.326 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.326 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.326 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.327 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.327 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.327 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.327 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "iostat", 2024-05-27T09:51:04.327 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.327 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.327 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.327 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.327 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.328 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.328 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.328 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.328 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.328 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.328 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.328 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.328 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.329 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.329 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.329 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.329 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.329 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.329 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.329 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.329 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.329 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.330 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.330 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.330 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.330 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.330 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.330 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.330 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.330 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.331 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.331 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.331 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.331 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.331 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.331 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.331 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.331 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.332 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.332 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.332 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.332 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.332 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.332 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.332 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.332 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.332 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.333 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.333 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.333 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.333 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.333 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.333 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.333 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.333 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.333 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.334 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.334 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.334 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.334 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.334 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.334 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.334 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.334 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.334 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.335 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.335 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.335 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.335 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.335 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.335 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.335 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.336 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.336 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.336 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.336 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.336 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.336 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.336 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.336 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.336 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.337 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.337 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.337 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.337 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.337 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.337 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.337 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.337 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.337 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.338 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "k8sevents", 2024-05-27T09:51:04.338 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.338 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.338 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.338 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_event_retention_days": { 2024-05-27T09:51:04.338 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ceph_event_retention_days", 2024-05-27T09:51:04.338 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.338 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.339 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.339 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7", 2024-05-27T09:51:04.339 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.339 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.339 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.339 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T09:51:04.339 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.339 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.339 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.340 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.340 INFO:teuthology.orchestra.run.smithi044.stdout: "config_check_secs": { 2024-05-27T09:51:04.340 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_check_secs", 2024-05-27T09:51:04.340 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.340 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.340 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.340 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:04.340 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "10", 2024-05-27T09:51:04.340 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.341 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.341 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T09:51:04.341 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.341 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.341 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.341 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.341 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.341 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.342 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.342 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.342 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.342 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.342 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.342 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.342 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.342 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.343 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.343 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.343 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.343 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.343 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.343 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.343 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.343 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.343 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.344 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.344 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.344 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.344 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.344 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.344 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.344 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.344 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.345 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.345 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.345 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.345 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.345 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.345 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.345 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.345 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.346 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.346 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.346 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.346 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.346 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.346 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.346 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.346 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.347 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.347 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.347 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.347 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.347 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.347 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.347 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.347 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.347 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.348 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.348 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.348 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.348 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.348 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.348 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.348 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.348 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.349 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.349 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.349 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.349 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.349 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.349 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.349 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.349 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.349 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.350 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.350 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.350 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.350 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.350 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.350 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.350 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.351 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.351 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.351 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.351 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.351 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.351 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.351 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.351 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.352 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.352 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.352 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.352 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "localpool", 2024-05-27T09:51:04.352 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.352 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.352 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.352 INFO:teuthology.orchestra.run.smithi044.stdout: "failure_domain": { 2024-05-27T09:51:04.352 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "failure_domain", 2024-05-27T09:51:04.353 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.353 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.353 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.353 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "host", 2024-05-27T09:51:04.353 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.353 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.353 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.353 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "failure domain for any created local pool", 2024-05-27T09:51:04.354 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T09:51:04.354 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.354 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.354 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.354 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.354 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.354 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.354 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.355 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.355 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.355 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.355 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.355 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.355 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.355 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.356 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.356 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.356 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.356 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.356 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.356 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.356 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.356 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.356 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.357 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.357 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.357 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.357 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.357 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.357 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.357 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.357 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.358 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.358 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.358 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.358 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.358 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.358 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.358 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.358 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.359 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.359 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.359 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.359 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.359 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.359 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.359 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.359 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.360 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.360 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.360 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.360 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.360 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.360 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.360 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.360 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.360 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.361 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.361 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.361 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.361 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.361 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.361 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.361 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.361 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.362 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.362 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.362 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.362 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.362 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.362 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.362 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.362 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.363 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.363 INFO:teuthology.orchestra.run.smithi044.stdout: "min_size": { 2024-05-27T09:51:04.363 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_size", 2024-05-27T09:51:04.363 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.363 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.363 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.364 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.364 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.364 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.364 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.364 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default min_size for any created local pool", 2024-05-27T09:51:04.364 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T09:51:04.364 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.365 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.365 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.365 INFO:teuthology.orchestra.run.smithi044.stdout: "num_rep": { 2024-05-27T09:51:04.365 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "num_rep", 2024-05-27T09:51:04.365 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.365 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.365 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.366 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3", 2024-05-27T09:51:04.366 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.366 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.366 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.366 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default replica count for any created local pool", 2024-05-27T09:51:04.366 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.366 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.367 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.367 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.367 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_num": { 2024-05-27T09:51:04.367 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_num", 2024-05-27T09:51:04.367 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.367 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.367 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.367 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "128", 2024-05-27T09:51:04.368 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.368 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.368 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.368 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T09:51:04.368 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.368 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.368 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.368 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.369 INFO:teuthology.orchestra.run.smithi044.stdout: "prefix": { 2024-05-27T09:51:04.369 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prefix", 2024-05-27T09:51:04.369 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.369 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.369 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.369 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.369 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.369 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.370 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.370 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name prefix for any created local pool", 2024-05-27T09:51:04.370 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.370 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.370 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.370 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.370 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.370 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.371 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.371 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.371 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.371 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.371 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.371 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.371 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.371 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.372 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.372 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.372 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.372 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.372 INFO:teuthology.orchestra.run.smithi044.stdout: "subtree": { 2024-05-27T09:51:04.372 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "subtree", 2024-05-27T09:51:04.372 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.372 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.372 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.373 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "rack", 2024-05-27T09:51:04.373 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.373 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.373 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.373 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T09:51:04.373 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T09:51:04.373 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.373 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.373 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.374 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.374 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.374 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.374 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mds_autoscaler", 2024-05-27T09:51:04.374 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.374 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.374 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.374 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.375 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.375 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.375 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.375 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.375 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.375 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.375 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.375 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.376 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.376 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.376 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.376 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.376 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.376 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.376 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.376 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.377 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.377 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.377 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.377 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.377 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.377 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.377 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.377 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.377 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.378 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.378 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.378 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.378 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.378 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.378 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.378 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.379 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.379 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.379 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.379 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.379 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.379 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.379 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.379 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.379 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.380 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.380 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.380 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.380 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.380 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.380 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.380 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.380 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.380 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.381 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.381 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.381 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.381 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.381 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.381 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.381 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.381 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.381 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.382 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.382 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.382 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.382 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.382 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.382 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.382 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.382 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.382 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.383 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.383 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.383 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.385 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.386 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.386 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.386 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.386 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.386 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.386 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.386 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.386 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.387 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.387 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.387 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.387 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.387 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.387 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.387 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirroring", 2024-05-27T09:51:04.387 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.388 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.388 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.388 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.388 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.388 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.388 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.388 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.388 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.388 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.389 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.389 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.389 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.389 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.389 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.389 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.389 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.389 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.389 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.390 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.390 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.390 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.390 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.390 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.390 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.390 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.390 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.391 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.391 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.391 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.391 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.391 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.391 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.391 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.391 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.391 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.392 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.392 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.392 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.392 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.392 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.392 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.392 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.392 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.392 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.393 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.393 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.393 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.393 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.393 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.393 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.393 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.393 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.394 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.394 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.394 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.394 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.394 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.394 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.394 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.394 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.394 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.395 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.395 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.395 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.395 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.395 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.395 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.395 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.395 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.396 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.396 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.396 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.396 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.396 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.396 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.396 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.397 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.397 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.397 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.397 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.397 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.397 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.397 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.397 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.398 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.398 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.398 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.398 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.398 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.398 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "nfs", 2024-05-27T09:51:04.398 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.399 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.399 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.399 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.399 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.399 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.399 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.399 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.399 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.400 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.400 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.400 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.400 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.400 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.400 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.400 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.400 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.401 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.401 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.401 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.401 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.401 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.401 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.401 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.401 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.402 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.402 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.402 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.402 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.402 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.402 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.402 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.402 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.403 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.403 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.403 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.403 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.403 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.403 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.403 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.403 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.404 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.404 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.404 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.404 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.404 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.404 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.404 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.404 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.405 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.405 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.405 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.405 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.405 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.405 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.405 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.405 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.406 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.406 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.406 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.406 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.406 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.406 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.406 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.407 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.407 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.407 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.407 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.407 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.408 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.408 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.408 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.408 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.408 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.408 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.409 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.409 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.409 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.409 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.409 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.410 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.410 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.410 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.410 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.410 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.411 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.411 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.411 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.411 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.412 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.412 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-05-27T09:51:04.412 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.412 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.412 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.413 INFO:teuthology.orchestra.run.smithi044.stdout: "fail_fs": { 2024-05-27T09:51:04.413 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "fail_fs", 2024-05-27T09:51:04.413 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.413 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.414 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.414 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.414 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.414 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.415 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.415 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T09:51:04.415 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.415 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.415 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.416 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.416 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.416 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.416 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.416 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.416 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.416 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.416 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.416 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.417 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.417 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.417 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.417 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.417 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.418 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.418 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.418 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.418 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.418 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.418 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.418 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.418 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.419 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.419 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.419 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.419 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.419 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.419 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.419 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.419 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.419 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.420 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.420 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.420 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.420 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.420 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.420 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.420 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.420 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.420 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.421 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.421 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.421 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.421 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.421 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.421 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.421 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.421 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.421 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.422 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.422 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.422 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.422 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.422 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.422 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.422 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.422 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.423 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.423 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.423 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.423 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.423 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.423 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.423 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.423 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.423 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.424 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.424 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.424 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.424 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.424 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.424 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator": { 2024-05-27T09:51:04.424 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-05-27T09:51:04.424 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.425 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.425 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.425 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.425 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.425 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.425 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.425 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2024-05-27T09:51:04.425 INFO:teuthology.orchestra.run.smithi044.stdout: "rook", 2024-05-27T09:51:04.425 INFO:teuthology.orchestra.run.smithi044.stdout: "test_orchestrator" 2024-05-27T09:51:04.426 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.426 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Orchestrator backend", 2024-05-27T09:51:04.426 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.426 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.426 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.426 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.426 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.426 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.427 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.427 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.427 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.427 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.427 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.427 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.427 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.428 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.428 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.428 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.428 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.428 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.429 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.429 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.429 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.429 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_perf_query", 2024-05-27T09:51:04.429 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.430 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.430 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.430 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.430 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.431 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.431 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.431 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.431 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.431 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.432 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.432 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.432 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.432 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.432 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.433 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.433 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.433 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.433 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.434 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.434 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.434 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.434 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.434 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.434 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.435 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.435 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.435 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.435 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.435 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.435 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.436 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.436 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.436 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.436 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.436 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.436 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.436 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.436 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.437 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.437 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.437 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.437 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.437 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.437 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.437 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.437 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.437 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.438 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.438 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.438 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.438 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.438 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.438 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.438 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.438 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.439 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.439 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.439 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.439 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.439 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.439 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.439 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.439 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.440 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.440 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.440 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.440 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.440 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.440 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.440 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.440 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.441 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.441 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.441 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.441 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.441 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.441 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.441 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.441 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.441 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.442 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.442 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.442 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.442 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.442 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.442 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.442 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.442 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.443 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.443 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_support", 2024-05-27T09:51:04.443 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.443 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.443 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.443 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.443 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.443 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.443 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.444 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.444 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.444 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.444 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.444 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.444 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.444 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.444 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.444 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.445 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.445 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.445 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.445 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.445 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.445 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.445 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.445 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.446 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.446 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.446 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.446 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.446 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.446 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.446 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.446 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.446 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.447 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.447 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.447 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.447 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.447 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.447 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.447 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.447 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.447 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.448 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.448 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.448 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.448 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.448 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.448 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.448 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.448 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.449 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.449 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.449 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.449 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.449 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.449 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.449 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.449 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.450 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.450 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.450 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.450 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.450 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.450 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.450 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.450 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.451 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.451 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.451 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.451 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.451 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.451 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.451 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.451 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.452 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.452 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.452 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.452 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.452 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.452 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.452 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.452 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.452 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.453 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.453 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.453 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.453 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.453 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.453 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.453 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.453 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_autoscaler", 2024-05-27T09:51:04.453 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.454 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.454 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.454 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.454 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.454 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.454 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.454 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.454 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.455 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.455 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.455 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.455 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.455 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.455 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.455 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.455 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.455 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.456 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.456 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.456 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.456 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.456 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.456 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.456 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.456 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.457 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.457 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.457 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.457 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.457 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.457 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.457 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.457 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.458 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.458 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.458 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.458 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.458 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.458 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.458 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.458 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.459 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.459 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.459 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.459 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.459 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.459 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.459 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.459 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.460 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.460 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.460 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.460 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.460 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.460 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.460 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.460 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.461 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.461 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.461 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.461 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.461 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.461 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.461 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.461 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.462 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.462 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.462 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.462 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.462 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.462 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.462 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.462 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:04.463 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:04.463 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.463 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.463 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.463 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:04.463 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.463 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.464 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.464 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.464 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.464 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.464 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.464 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.464 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.464 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.465 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.465 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.465 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.465 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.465 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.465 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.465 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.465 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.466 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.466 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.466 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.466 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.466 INFO:teuthology.orchestra.run.smithi044.stdout: "threshold": { 2024-05-27T09:51:04.466 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threshold", 2024-05-27T09:51:04.466 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:04.467 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.467 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.467 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-05-27T09:51:04.467 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1.0", 2024-05-27T09:51:04.467 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.467 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.467 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "scaling threshold", 2024-05-27T09:51:04.467 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T09:51:04.468 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.468 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.468 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.468 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.468 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.468 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.468 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "progress", 2024-05-27T09:51:04.469 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.469 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.469 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.469 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_pg_recovery_event": { 2024-05-27T09:51:04.469 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_pg_recovery_event", 2024-05-27T09:51:04.469 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.469 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.469 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.470 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.470 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.470 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.470 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.470 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T09:51:04.470 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.470 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.471 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.471 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.471 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-05-27T09:51:04.471 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-05-27T09:51:04.471 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.471 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.471 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.471 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.471 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.472 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.472 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.472 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.472 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.472 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.472 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.472 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.472 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.472 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.473 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.473 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.473 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.473 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.473 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.473 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.473 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.473 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.474 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.474 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.474 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.474 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.474 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.474 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.474 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.475 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.475 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.475 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.475 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.475 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.475 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.475 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.475 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.475 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.476 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.476 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.476 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.476 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.476 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.476 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.476 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.476 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.477 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.477 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.477 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.477 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.477 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.477 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.477 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.477 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.478 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.478 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.478 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.478 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.478 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.478 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.478 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.479 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.479 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.479 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.479 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.479 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.479 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.479 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.479 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.480 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.480 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.480 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.480 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.480 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.480 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.480 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.480 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.480 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.481 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.481 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.481 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.481 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.481 INFO:teuthology.orchestra.run.smithi044.stdout: "max_completed_events": { 2024-05-27T09:51:04.481 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_completed_events", 2024-05-27T09:51:04.481 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.481 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.481 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.482 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "50", 2024-05-27T09:51:04.482 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.482 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.482 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.482 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "number of past completed events to remember", 2024-05-27T09:51:04.482 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.482 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.482 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.483 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.483 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-05-27T09:51:04.483 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-05-27T09:51:04.483 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.483 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.483 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.483 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-05-27T09:51:04.483 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.484 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.484 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.484 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long the module is going to sleep", 2024-05-27T09:51:04.484 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.484 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.484 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.484 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.484 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.484 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.485 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.485 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.485 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.485 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.485 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.485 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.485 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.485 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.485 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.486 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.486 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.486 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.486 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.486 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.486 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.486 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus", 2024-05-27T09:51:04.486 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.487 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.487 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.487 INFO:teuthology.orchestra.run.smithi044.stdout: "cache": { 2024-05-27T09:51:04.487 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cache", 2024-05-27T09:51:04.487 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.487 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.487 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.487 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.488 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.488 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.488 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.488 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.488 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.488 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.488 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.488 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.489 INFO:teuthology.orchestra.run.smithi044.stdout: "exclude_perf_counters": { 2024-05-27T09:51:04.489 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "exclude_perf_counters", 2024-05-27T09:51:04.489 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.489 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.489 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.489 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.489 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.489 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.490 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.490 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T09:51:04.490 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T09:51:04.490 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.490 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.490 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.490 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.490 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.491 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.491 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.491 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.491 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.491 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.491 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.491 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.491 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.491 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.492 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.492 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.492 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.492 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.492 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.492 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.492 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.492 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.492 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.493 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.493 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.493 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.493 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.493 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.493 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.493 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.493 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.494 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.494 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.494 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.494 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.494 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.494 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.494 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.494 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.495 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.495 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.495 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.495 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.495 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.495 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.495 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.495 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.495 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.496 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.496 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.496 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.496 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.496 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.496 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.496 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.496 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.497 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.497 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.497 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.497 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.497 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.497 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.497 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.497 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.498 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.498 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.498 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.498 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.498 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.498 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.498 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.499 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.499 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.499 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools": { 2024-05-27T09:51:04.499 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools", 2024-05-27T09:51:04.499 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.499 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.499 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.499 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.500 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.500 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.500 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.500 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.500 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.500 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.500 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.500 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.501 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T09:51:04.501 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T09:51:04.501 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.501 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.501 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.501 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300", 2024-05-27T09:51:04.501 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.501 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.502 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.502 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.502 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.502 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.502 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.502 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.502 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_interval": { 2024-05-27T09:51:04.502 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_interval", 2024-05-27T09:51:04.503 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:04.503 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.503 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.503 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15.0", 2024-05-27T09:51:04.503 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.503 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.503 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.503 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.503 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.504 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.504 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.504 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.504 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-05-27T09:51:04.504 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-05-27T09:51:04.504 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.504 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.504 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.504 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-05-27T09:51:04.505 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.505 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.505 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.505 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T09:51:04.505 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.505 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.505 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.506 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.506 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-05-27T09:51:04.506 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-05-27T09:51:04.506 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.506 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.506 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.506 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "9283", 2024-05-27T09:51:04.506 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.507 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.507 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.507 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T09:51:04.507 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.507 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.507 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.507 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.507 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.508 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.508 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.508 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.508 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.508 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.508 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.508 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.508 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.508 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.509 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.509 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.509 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.509 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.509 INFO:teuthology.orchestra.run.smithi044.stdout: "stale_cache_strategy": { 2024-05-27T09:51:04.509 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stale_cache_strategy", 2024-05-27T09:51:04.509 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.510 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.510 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.510 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "log", 2024-05-27T09:51:04.510 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.510 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.510 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.510 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.513 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.513 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.514 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.514 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.514 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-05-27T09:51:04.514 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-05-27T09:51:04.514 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.514 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.514 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.515 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "default", 2024-05-27T09:51:04.515 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.515 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.515 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.515 INFO:teuthology.orchestra.run.smithi044.stdout: "default", 2024-05-27T09:51:04.515 INFO:teuthology.orchestra.run.smithi044.stdout: "error" 2024-05-27T09:51:04.515 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.515 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.516 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.516 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.516 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.516 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.516 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-05-27T09:51:04.516 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-05-27T09:51:04.516 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.516 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.517 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.517 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-05-27T09:51:04.517 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-05-27T09:51:04.517 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-05-27T09:51:04.517 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.517 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.517 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.517 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.518 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.518 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.518 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.518 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.518 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.518 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2024-05-27T09:51:04.518 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.518 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.519 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.519 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.519 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.519 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.519 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.519 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.519 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.519 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.520 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.520 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.520 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.520 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.520 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.520 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.520 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.520 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.520 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.521 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.521 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.521 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.521 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.521 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.521 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.521 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.521 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.522 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.522 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.522 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.522 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.522 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.522 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.522 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.522 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.523 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.523 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.523 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.523 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.523 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.523 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.523 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.523 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.524 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.524 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.524 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.524 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.524 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.524 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.524 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.524 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.524 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.525 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.525 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.525 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.525 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.525 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.525 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.525 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.525 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.525 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.526 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.526 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.526 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.526 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.526 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.526 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.526 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.527 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.527 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.527 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.527 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.527 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.527 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_snap_create": { 2024-05-27T09:51:04.527 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_snap_create", 2024-05-27T09:51:04.527 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.528 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.528 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.528 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-05-27T09:51:04.528 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.528 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.528 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.528 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.528 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.529 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.529 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.529 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.529 INFO:teuthology.orchestra.run.smithi044.stdout: "mirror_snapshot_schedule": { 2024-05-27T09:51:04.529 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T09:51:04.529 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.529 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.529 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.529 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.530 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.530 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.530 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.530 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.530 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.530 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.530 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.530 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.531 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.531 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.531 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.531 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.531 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.531 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.531 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.531 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.532 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.532 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.532 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.532 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.532 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.532 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.532 INFO:teuthology.orchestra.run.smithi044.stdout: "trash_purge_schedule": { 2024-05-27T09:51:04.532 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "trash_purge_schedule", 2024-05-27T09:51:04.533 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.533 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.533 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.533 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.533 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.533 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.533 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.533 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.534 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.534 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.534 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.534 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.534 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.534 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.534 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.534 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "restful", 2024-05-27T09:51:04.535 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.535 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.535 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.535 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_auth": { 2024-05-27T09:51:04.535 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_auth", 2024-05-27T09:51:04.535 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.535 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.535 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.536 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.536 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.536 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.536 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.536 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.536 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.536 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.536 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.536 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.537 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-05-27T09:51:04.537 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-05-27T09:51:04.537 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.537 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.537 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.537 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.537 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.537 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.538 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.538 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.538 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.538 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.538 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.538 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.538 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.538 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.539 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.539 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.539 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.539 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.539 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.539 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.539 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.539 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.539 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.540 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.540 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.540 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.540 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.540 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.540 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.540 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.540 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.541 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.541 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.541 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.541 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.541 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.541 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.541 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.541 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.542 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.542 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.542 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.542 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.542 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.542 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.542 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.542 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.543 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.543 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.543 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.543 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.543 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.543 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.543 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.543 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.544 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.544 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.544 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.544 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.544 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.544 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.544 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.544 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.544 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.545 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.545 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.545 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.545 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.545 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.545 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.545 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.545 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.546 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.546 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.546 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.546 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.546 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.546 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.546 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.546 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.547 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.547 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.547 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-05-27T09:51:04.547 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-05-27T09:51:04.547 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.547 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.547 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.547 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.548 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.548 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.548 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.548 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.548 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.548 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.548 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.548 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.549 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-05-27T09:51:04.549 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-05-27T09:51:04.549 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.549 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.549 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.549 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.549 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.549 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.550 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.550 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.550 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.550 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.550 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.550 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.550 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.550 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.551 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.551 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.551 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.551 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.551 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.551 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.551 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.551 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.551 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.552 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.552 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.552 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.552 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.552 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.552 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.552 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rgw", 2024-05-27T09:51:04.552 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.552 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.553 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.553 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.553 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.553 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.553 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.553 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.553 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.553 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.553 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.554 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.554 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.554 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.554 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.554 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.554 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.554 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.554 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.555 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.555 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.555 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.555 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.555 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.555 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.555 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.555 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.555 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.556 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.556 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.556 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.556 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.556 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.556 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.556 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.556 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.556 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.557 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.557 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.557 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.557 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.557 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.557 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.557 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.557 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.557 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.558 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.558 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.558 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.558 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.558 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.558 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.558 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.558 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.559 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.559 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.559 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.559 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.559 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.559 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.559 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.559 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.560 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.560 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.560 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.560 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.560 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.560 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.560 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.560 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.560 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.561 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.561 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.561 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.561 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.561 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.561 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.561 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.561 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.561 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.562 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.562 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.562 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.562 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.562 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.562 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.562 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.562 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.562 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.563 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.563 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rook", 2024-05-27T09:51:04.563 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.563 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.563 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.563 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.563 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.563 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.564 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.564 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.564 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.564 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.564 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.564 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.564 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.564 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.564 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.565 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.565 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.565 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.565 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.565 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.565 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.565 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.565 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.565 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.566 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.566 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.566 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.566 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.566 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.566 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.566 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.566 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.567 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.567 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.567 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.567 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.567 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.567 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.567 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.567 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.567 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.568 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.568 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.568 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.568 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.568 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.568 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.568 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.568 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.568 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.569 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.569 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.569 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.569 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.569 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.569 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.569 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.569 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.570 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.570 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.570 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.570 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.570 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.570 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.570 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.570 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.571 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.571 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.571 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.571 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.571 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.571 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.571 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.571 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.571 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.572 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.572 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.572 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.572 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.572 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.572 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.572 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.572 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.572 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.573 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.573 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.573 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.573 INFO:teuthology.orchestra.run.smithi044.stdout: "storage_class": { 2024-05-27T09:51:04.573 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "storage_class", 2024-05-27T09:51:04.573 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.573 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.573 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.573 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "local", 2024-05-27T09:51:04.574 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.574 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.574 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.574 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T09:51:04.574 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.574 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.574 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.574 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.575 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.575 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.575 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.575 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "selftest", 2024-05-27T09:51:04.575 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.575 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.575 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.575 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.575 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.576 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.576 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.576 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.576 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.576 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.576 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.576 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.576 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.577 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.577 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.577 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.577 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.577 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.577 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.577 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.577 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.577 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.578 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.578 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.578 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.578 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.578 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.578 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.578 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.579 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.579 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.579 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.579 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.579 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.579 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.579 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.579 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.580 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.580 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.580 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.580 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.580 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.580 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.580 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.580 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.580 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.581 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.581 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.581 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.581 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.581 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.581 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.581 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.581 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.582 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.582 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.582 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.582 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.582 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.582 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.582 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.583 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.583 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.583 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.583 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.583 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.583 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.583 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.583 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.584 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.584 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.584 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.584 INFO:teuthology.orchestra.run.smithi044.stdout: "roption1": { 2024-05-27T09:51:04.584 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption1", 2024-05-27T09:51:04.584 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.584 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.584 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.585 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.585 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.585 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.585 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.585 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.585 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.585 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.585 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.585 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.586 INFO:teuthology.orchestra.run.smithi044.stdout: "roption2": { 2024-05-27T09:51:04.586 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption2", 2024-05-27T09:51:04.586 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.586 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.586 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.586 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "xyz", 2024-05-27T09:51:04.586 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.586 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.587 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.587 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.587 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.587 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.587 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.587 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.587 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption1": { 2024-05-27T09:51:04.587 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption1", 2024-05-27T09:51:04.588 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.588 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.588 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.588 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.588 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.588 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.588 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.588 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.588 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.589 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.589 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.589 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.589 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption2": { 2024-05-27T09:51:04.589 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption2", 2024-05-27T09:51:04.589 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.589 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.589 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.590 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.590 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.590 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.590 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.590 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.590 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.590 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.591 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.591 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.591 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption3": { 2024-05-27T09:51:04.591 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption3", 2024-05-27T09:51:04.591 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-05-27T09:51:04.591 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.591 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.591 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.592 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.592 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.592 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.592 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.592 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.592 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.592 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.592 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.592 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption4": { 2024-05-27T09:51:04.593 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption4", 2024-05-27T09:51:04.593 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.593 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.593 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.593 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.593 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.593 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.593 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.593 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.594 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.594 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.594 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.594 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.594 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption5": { 2024-05-27T09:51:04.594 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption5", 2024-05-27T09:51:04.594 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.594 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.595 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.595 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.595 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.595 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.595 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.595 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.595 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.595 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.595 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.596 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.596 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption6": { 2024-05-27T09:51:04.596 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption6", 2024-05-27T09:51:04.596 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.596 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.596 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.596 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.596 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.597 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.597 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.597 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.597 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.597 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.597 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.597 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.597 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption7": { 2024-05-27T09:51:04.597 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption7", 2024-05-27T09:51:04.598 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.598 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.598 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.598 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.598 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-05-27T09:51:04.598 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "42", 2024-05-27T09:51:04.598 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.599 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.599 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.599 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.599 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.599 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.599 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.599 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.599 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.599 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.600 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.600 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.600 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.600 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.600 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.600 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.600 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.600 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.601 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.601 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.601 INFO:teuthology.orchestra.run.smithi044.stdout: "testkey": { 2024-05-27T09:51:04.601 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testkey", 2024-05-27T09:51:04.601 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.601 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.601 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.601 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.602 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.602 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.602 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.602 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.602 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.602 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.602 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.603 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.603 INFO:teuthology.orchestra.run.smithi044.stdout: "testlkey": { 2024-05-27T09:51:04.603 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testlkey", 2024-05-27T09:51:04.603 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.603 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.603 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.603 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.603 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.603 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.604 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.604 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.604 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.604 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.604 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.604 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.604 INFO:teuthology.orchestra.run.smithi044.stdout: "testnewline": { 2024-05-27T09:51:04.604 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testnewline", 2024-05-27T09:51:04.605 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.605 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.605 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.605 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.605 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.605 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.605 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.605 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.605 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.606 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.606 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.606 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.606 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.606 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.606 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.606 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smb", 2024-05-27T09:51:04.606 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.607 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.607 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.607 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.607 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.607 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.607 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.607 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.607 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.607 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.608 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.608 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.608 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.608 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.608 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.608 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.608 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.608 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.609 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.609 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.609 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.609 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.609 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.609 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.609 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.609 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.609 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.610 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.610 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.610 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.610 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.610 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.610 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.610 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.610 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.611 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.611 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.611 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.611 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.611 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.611 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.611 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.611 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.611 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.612 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.612 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.612 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.612 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.612 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.612 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.612 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.612 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.612 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.613 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.613 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.613 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.613 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.613 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.613 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.613 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.613 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.614 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.614 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.614 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.614 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.614 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.614 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.614 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.615 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.615 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.615 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.615 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.615 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.615 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.615 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.615 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.616 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.616 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.616 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.616 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.616 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.616 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.616 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.616 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.617 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.617 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.617 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.617 INFO:teuthology.orchestra.run.smithi044.stdout: "update_orchestration": { 2024-05-27T09:51:04.617 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "update_orchestration", 2024-05-27T09:51:04.617 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.617 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.617 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.617 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.618 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.618 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.618 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.618 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T09:51:04.618 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.618 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.618 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.618 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.619 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.619 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.619 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.619 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2024-05-27T09:51:04.619 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.619 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.619 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.619 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_m_granularity": { 2024-05-27T09:51:04.620 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_m_granularity", 2024-05-27T09:51:04.620 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.620 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.620 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.620 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.620 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.620 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.620 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.621 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T09:51:04.621 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.621 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.621 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.621 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.621 INFO:teuthology.orchestra.run.smithi044.stdout: "dump_on_update": { 2024-05-27T09:51:04.621 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dump_on_update", 2024-05-27T09:51:04.621 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.621 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.622 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.622 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.622 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.622 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.622 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.622 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "dump database to debug log on update", 2024-05-27T09:51:04.622 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.622 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.623 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.623 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.623 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.623 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.623 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.623 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.623 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.623 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.623 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.624 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.624 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.624 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.624 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.624 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.624 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.624 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.624 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.624 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.625 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.625 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.625 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.625 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.625 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.625 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.625 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.625 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.625 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.626 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.626 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.626 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.626 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.626 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.627 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.627 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.627 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.627 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.628 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.628 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.628 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.628 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.629 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.629 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.629 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.629 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.629 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.630 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.630 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.630 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.630 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.631 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.631 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.631 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.631 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.631 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.632 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.632 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.632 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.632 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.632 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.632 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.633 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.633 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.633 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.633 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.633 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.633 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.634 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.634 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.634 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.634 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.634 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.635 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.635 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.635 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.635 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.635 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.635 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.636 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.636 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.636 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.636 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.636 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.637 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.637 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.637 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.637 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.637 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.638 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.638 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.638 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stats", 2024-05-27T09:51:04.638 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.638 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.639 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.639 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.639 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.639 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.640 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.640 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.640 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.640 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.640 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.640 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.641 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.641 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.641 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.641 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.641 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.641 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.641 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.641 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.642 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.642 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.642 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.642 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.642 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.643 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.643 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.643 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.643 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.643 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.643 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.643 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.644 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.644 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.644 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.644 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.644 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.644 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.644 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.645 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.645 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.645 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.645 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.645 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.645 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.645 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.645 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.646 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.646 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.646 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.646 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.646 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.646 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.646 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.646 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.647 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.647 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.647 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.647 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.647 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.647 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.647 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.648 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.648 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.648 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.648 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.648 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.648 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.648 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.648 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.649 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.649 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.649 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.649 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.649 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.649 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.649 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.650 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.650 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.650 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.650 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.650 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.650 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.650 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.650 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.651 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.651 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.651 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.651 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.651 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.651 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "status", 2024-05-27T09:51:04.651 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.651 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.652 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.652 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.652 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.652 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.652 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.652 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.652 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.652 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.652 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.653 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.653 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.653 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.653 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.653 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.653 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.653 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.653 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.654 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.654 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.654 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.654 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.654 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.654 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.654 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.654 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.655 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.655 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.655 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.655 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.655 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.655 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.655 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.655 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.655 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.656 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.656 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.656 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.656 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.656 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.656 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.656 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.656 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.657 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.657 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.657 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.657 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.657 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.657 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.657 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.657 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.657 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.658 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.658 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.658 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.658 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.658 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.658 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.658 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.658 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.659 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.659 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.659 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.659 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.659 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.659 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.659 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.659 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.659 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.660 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.660 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.660 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.660 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.660 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.660 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.660 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.660 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.661 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.661 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.661 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.661 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.661 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.661 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.661 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.661 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.661 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.662 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.662 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.662 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.662 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telegraf", 2024-05-27T09:51:04.662 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.662 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.662 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.662 INFO:teuthology.orchestra.run.smithi044.stdout: "address": { 2024-05-27T09:51:04.662 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "address", 2024-05-27T09:51:04.663 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.663 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.663 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.663 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T09:51:04.663 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.663 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.663 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.663 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.664 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.664 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.664 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.664 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.664 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:04.664 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:04.664 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.664 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.664 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.665 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15", 2024-05-27T09:51:04.665 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.665 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.665 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.665 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.665 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.665 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.665 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.666 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.666 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.666 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.666 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.666 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.666 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.666 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.666 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.666 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.667 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.667 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.667 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.667 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.667 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.667 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.667 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.667 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.667 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.668 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.668 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.668 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.668 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.668 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.668 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.668 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.668 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.668 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.669 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.669 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.669 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.669 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.669 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.669 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.669 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.669 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.669 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.670 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.670 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.670 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.670 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.670 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.670 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.670 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.670 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.670 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.671 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.671 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.671 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.671 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.671 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.671 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.671 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.671 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.671 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.672 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.672 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.672 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.672 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.672 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.672 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.672 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.672 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.673 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.673 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.673 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.673 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.673 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.673 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.673 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.673 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.673 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.674 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.674 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.674 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.674 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.674 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.674 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.674 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.674 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.675 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.675 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.675 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.675 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.675 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.675 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.675 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.675 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.675 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.676 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telemetry", 2024-05-27T09:51:04.676 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.676 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.676 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.676 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_basic": { 2024-05-27T09:51:04.676 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_basic", 2024-05-27T09:51:04.676 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.676 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.677 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.677 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.677 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.677 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.677 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.677 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T09:51:04.677 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.677 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.677 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.678 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.678 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_crash": { 2024-05-27T09:51:04.678 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_crash", 2024-05-27T09:51:04.678 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.678 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.678 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.678 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.679 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.679 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.679 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.679 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T09:51:04.679 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.679 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.679 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.679 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.680 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_device": { 2024-05-27T09:51:04.680 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_device", 2024-05-27T09:51:04.680 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.680 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.680 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.680 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.680 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.680 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.680 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.681 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T09:51:04.681 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.681 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.681 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.681 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.681 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_ident": { 2024-05-27T09:51:04.681 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_ident", 2024-05-27T09:51:04.681 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.681 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.682 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.682 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.682 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.682 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.682 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.682 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T09:51:04.682 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.682 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.683 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.683 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.683 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_perf": { 2024-05-27T09:51:04.683 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_perf", 2024-05-27T09:51:04.683 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.683 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.683 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.683 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.684 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.684 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.684 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.684 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T09:51:04.684 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.684 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.684 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.684 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.685 INFO:teuthology.orchestra.run.smithi044.stdout: "contact": { 2024-05-27T09:51:04.685 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "contact", 2024-05-27T09:51:04.685 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.685 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.685 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.685 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.685 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.685 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.685 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.686 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.686 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.686 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.686 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.686 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.686 INFO:teuthology.orchestra.run.smithi044.stdout: "description": { 2024-05-27T09:51:04.686 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "description", 2024-05-27T09:51:04.686 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.687 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.687 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.687 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.687 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.687 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.687 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.687 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.687 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.688 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.688 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.688 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.688 INFO:teuthology.orchestra.run.smithi044.stdout: "device_url": { 2024-05-27T09:51:04.688 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_url", 2024-05-27T09:51:04.688 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.688 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.688 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.689 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T09:51:04.689 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.689 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.689 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.689 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.689 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.689 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.689 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.690 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.690 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-05-27T09:51:04.690 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-05-27T09:51:04.690 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.690 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.690 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.690 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.690 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.691 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.691 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.691 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.691 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.691 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.691 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.691 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.691 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:04.691 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:04.692 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.692 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.692 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.692 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "24", 2024-05-27T09:51:04.692 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "8", 2024-05-27T09:51:04.692 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.692 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.692 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.693 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.693 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.693 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.693 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.693 INFO:teuthology.orchestra.run.smithi044.stdout: "last_opt_revision": { 2024-05-27T09:51:04.693 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "last_opt_revision", 2024-05-27T09:51:04.693 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.693 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.693 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.693 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1", 2024-05-27T09:51:04.694 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.694 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.694 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.694 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.694 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.694 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.694 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.695 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.695 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard": { 2024-05-27T09:51:04.695 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard", 2024-05-27T09:51:04.695 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.695 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.695 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.695 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.695 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.695 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.696 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.696 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.696 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.696 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.696 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.696 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.696 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard_description": { 2024-05-27T09:51:04.696 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard_description", 2024-05-27T09:51:04.697 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.697 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.697 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.697 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.697 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.697 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.697 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.697 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.698 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.698 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.698 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.698 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.698 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.698 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.698 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.698 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.699 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.700 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.700 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.700 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.700 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.700 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.700 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.700 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.700 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.701 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.701 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.701 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.701 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.701 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.701 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.701 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.701 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.701 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.702 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.702 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.702 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.702 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.702 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.702 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.702 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.703 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.703 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.703 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.703 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.703 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.703 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.703 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.703 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.703 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.704 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.704 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.704 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.704 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.704 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.704 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.704 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.704 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.705 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.705 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.705 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.705 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.705 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.705 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.705 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.705 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.706 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.706 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.706 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.706 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.706 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.706 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.706 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.707 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.707 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.707 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.707 INFO:teuthology.orchestra.run.smithi044.stdout: "organization": { 2024-05-27T09:51:04.707 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "organization", 2024-05-27T09:51:04.707 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.707 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.707 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.707 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.708 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.708 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.708 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.708 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.708 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.708 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.708 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.708 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.708 INFO:teuthology.orchestra.run.smithi044.stdout: "proxy": { 2024-05-27T09:51:04.709 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "proxy", 2024-05-27T09:51:04.709 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.709 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.709 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.709 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.709 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.709 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.709 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.709 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.710 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.710 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.710 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.710 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.710 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.710 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.710 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.710 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.711 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.711 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.711 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.711 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.711 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.711 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.711 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.711 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.711 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.712 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.712 INFO:teuthology.orchestra.run.smithi044.stdout: "url": { 2024-05-27T09:51:04.712 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url", 2024-05-27T09:51:04.712 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.712 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.712 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.712 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T09:51:04.712 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.712 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.713 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.713 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.713 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.713 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.713 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.713 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.713 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.713 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.713 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.714 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "test_orchestrator", 2024-05-27T09:51:04.714 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.714 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.714 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.714 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.714 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.714 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.715 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.715 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.715 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.715 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.715 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.715 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.715 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.715 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.716 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.716 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.716 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.716 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.716 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.716 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.716 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.716 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.717 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.717 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.717 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.717 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.717 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.717 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.717 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.717 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.718 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.718 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.718 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.718 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.718 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.718 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.718 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.718 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.719 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.719 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.719 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.719 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.719 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.719 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.719 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.719 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.719 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.720 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.721 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.721 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.721 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.721 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.721 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.721 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.721 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.721 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.722 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.722 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.722 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.722 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.722 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.722 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.722 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.722 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.723 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.723 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.723 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.723 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.723 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.723 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.723 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.723 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.723 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.724 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.724 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.724 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.724 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.724 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.724 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.724 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.724 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.724 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.725 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2024-05-27T09:51:04.725 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.725 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.725 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.725 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.725 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.725 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.725 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.725 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.726 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.726 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.726 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.726 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.726 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.726 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.726 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.726 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.726 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.727 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.727 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.727 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.727 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.727 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.727 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.727 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.727 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.727 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.728 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.728 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.728 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.728 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.728 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.728 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.728 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.728 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.728 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.729 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.729 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.729 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.729 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.729 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.729 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.729 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.729 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.730 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.730 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.730 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.730 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.730 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.730 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.730 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.730 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.730 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.731 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.731 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.731 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.731 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.731 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.731 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.731 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.731 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.731 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.732 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.732 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.732 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.732 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.732 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.732 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.732 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.732 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.732 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.733 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.733 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.733 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.733 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_clones": { 2024-05-27T09:51:04.733 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_clones", 2024-05-27T09:51:04.733 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.733 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.733 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.733 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4", 2024-05-27T09:51:04.734 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.734 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.734 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.734 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T09:51:04.734 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.734 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.734 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.735 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.735 INFO:teuthology.orchestra.run.smithi044.stdout: "periodic_async_work": { 2024-05-27T09:51:04.735 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "periodic_async_work", 2024-05-27T09:51:04.735 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.735 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.735 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.735 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.735 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.735 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.736 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.736 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Periodically check for async work", 2024-05-27T09:51:04.736 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.736 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.736 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.736 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.736 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_delay": { 2024-05-27T09:51:04.736 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_delay", 2024-05-27T09:51:04.736 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.737 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.737 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.737 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.737 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.737 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.737 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.737 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T09:51:04.737 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.737 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.738 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.738 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.738 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_no_wait": { 2024-05-27T09:51:04.738 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T09:51:04.738 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.738 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.738 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.739 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-05-27T09:51:04.739 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.739 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.739 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.739 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T09:51:04.739 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.739 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.739 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.739 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.740 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.740 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.740 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.740 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.740 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.740 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.740 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.740 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.740 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.741 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.741 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.741 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.741 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.741 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.741 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.741 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.741 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.742 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix", 2024-05-27T09:51:04.742 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-05-27T09:51:04.742 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-05-27T09:51:04.742 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-05-27T09:51:04.742 INFO:teuthology.orchestra.run.smithi044.stdout: "discovery_interval": { 2024-05-27T09:51:04.742 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "discovery_interval", 2024-05-27T09:51:04.742 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-05-27T09:51:04.742 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.743 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.743 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "100", 2024-05-27T09:51:04.743 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.743 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.743 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.743 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.743 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.743 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.744 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.744 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.744 INFO:teuthology.orchestra.run.smithi044.stdout: "identifier": { 2024-05-27T09:51:04.744 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "identifier", 2024-05-27T09:51:04.744 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.744 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.744 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.744 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.744 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.745 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.745 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.745 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.745 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.745 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.745 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.745 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.745 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-05-27T09:51:04.746 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-05-27T09:51:04.746 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-05-27T09:51:04.746 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.746 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.746 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-05-27T09:51:04.746 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.746 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.746 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.747 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.747 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.747 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.747 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.747 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.747 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-05-27T09:51:04.747 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-05-27T09:51:04.747 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.747 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.749 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.749 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.749 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.749 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.749 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.749 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.749 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.749 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.749 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-05-27T09:51:04.750 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-05-27T09:51:04.750 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.750 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.750 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.750 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.750 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.750 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.750 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.750 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.751 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.751 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.751 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.751 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.751 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-05-27T09:51:04.751 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-05-27T09:51:04.751 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.751 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.752 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.752 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-05-27T09:51:04.752 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.752 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.752 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-05-27T09:51:04.752 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-05-27T09:51:04.752 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-05-27T09:51:04.752 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-05-27T09:51:04.752 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-05-27T09:51:04.753 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-05-27T09:51:04.753 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-05-27T09:51:04.753 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.753 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.753 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.753 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.753 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.753 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.753 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-05-27T09:51:04.753 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-05-27T09:51:04.754 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-05-27T09:51:04.754 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.754 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.754 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-05-27T09:51:04.754 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.754 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.754 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.754 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.755 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.755 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.755 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.755 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.755 INFO:teuthology.orchestra.run.smithi044.stdout: "sqlite3_killpoint": { 2024-05-27T09:51:04.755 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sqlite3_killpoint", 2024-05-27T09:51:04.755 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.755 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "dev", 2024-05-27T09:51:04.755 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-05-27T09:51:04.756 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-05-27T09:51:04.756 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.756 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.756 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.756 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.756 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.756 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.756 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.756 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.757 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_host": { 2024-05-27T09:51:04.757 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_host", 2024-05-27T09:51:04.757 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.757 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.757 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.757 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-05-27T09:51:04.757 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.757 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.757 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.758 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.758 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.758 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.758 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.758 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.758 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_port": { 2024-05-27T09:51:04.758 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_port", 2024-05-27T09:51:04.759 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-05-27T09:51:04.759 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.759 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.759 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10051", 2024-05-27T09:51:04.759 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.759 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.759 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.759 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.759 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.760 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.760 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.760 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.760 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_sender": { 2024-05-27T09:51:04.760 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_sender", 2024-05-27T09:51:04.760 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-05-27T09:51:04.760 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-05-27T09:51:04.760 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-05-27T09:51:04.760 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T09:51:04.761 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-05-27T09:51:04.761 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-05-27T09:51:04.761 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-05-27T09:51:04.761 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-05-27T09:51:04.761 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-05-27T09:51:04.761 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-05-27T09:51:04.761 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-05-27T09:51:04.761 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.762 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.762 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.762 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.762 INFO:teuthology.orchestra.run.smithi044.stdout: "services": { 2024-05-27T09:51:04.762 INFO:teuthology.orchestra.run.smithi044.stdout: "dashboard": "https://172.21.15.44:8443/" 2024-05-27T09:51:04.762 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.762 INFO:teuthology.orchestra.run.smithi044.stdout: "always_on_modules": { 2024-05-27T09:51:04.762 INFO:teuthology.orchestra.run.smithi044.stdout: "octopus": [ 2024-05-27T09:51:04.763 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2024-05-27T09:51:04.763 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-05-27T09:51:04.763 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2024-05-27T09:51:04.763 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2024-05-27T09:51:04.763 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2024-05-27T09:51:04.763 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2024-05-27T09:51:04.763 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2024-05-27T09:51:04.763 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2024-05-27T09:51:04.763 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2024-05-27T09:51:04.764 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2024-05-27T09:51:04.764 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.764 INFO:teuthology.orchestra.run.smithi044.stdout: "pacific": [ 2024-05-27T09:51:04.764 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2024-05-27T09:51:04.764 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-05-27T09:51:04.764 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2024-05-27T09:51:04.764 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2024-05-27T09:51:04.764 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2024-05-27T09:51:04.765 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2024-05-27T09:51:04.765 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2024-05-27T09:51:04.765 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2024-05-27T09:51:04.765 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2024-05-27T09:51:04.765 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2024-05-27T09:51:04.765 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.765 INFO:teuthology.orchestra.run.smithi044.stdout: "quincy": [ 2024-05-27T09:51:04.765 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2024-05-27T09:51:04.765 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-05-27T09:51:04.766 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2024-05-27T09:51:04.766 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2024-05-27T09:51:04.766 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2024-05-27T09:51:04.766 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2024-05-27T09:51:04.766 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2024-05-27T09:51:04.766 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2024-05-27T09:51:04.766 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2024-05-27T09:51:04.766 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2024-05-27T09:51:04.767 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.767 INFO:teuthology.orchestra.run.smithi044.stdout: "reef": [ 2024-05-27T09:51:04.767 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2024-05-27T09:51:04.767 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-05-27T09:51:04.767 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2024-05-27T09:51:04.767 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2024-05-27T09:51:04.767 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2024-05-27T09:51:04.767 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2024-05-27T09:51:04.768 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2024-05-27T09:51:04.768 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2024-05-27T09:51:04.768 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2024-05-27T09:51:04.768 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2024-05-27T09:51:04.768 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-05-27T09:51:04.768 INFO:teuthology.orchestra.run.smithi044.stdout: "squid": [ 2024-05-27T09:51:04.768 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2024-05-27T09:51:04.768 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-05-27T09:51:04.768 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2024-05-27T09:51:04.769 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2024-05-27T09:51:04.769 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2024-05-27T09:51:04.769 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2024-05-27T09:51:04.769 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2024-05-27T09:51:04.769 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2024-05-27T09:51:04.769 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2024-05-27T09:51:04.769 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2024-05-27T09:51:04.769 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:51:04.769 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.770 INFO:teuthology.orchestra.run.smithi044.stdout: "last_failure_osd_epoch": 82, 2024-05-27T09:51:04.770 INFO:teuthology.orchestra.run.smithi044.stdout: "active_clients": [ 2024-05-27T09:51:04.770 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.770 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2024-05-27T09:51:04.770 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-05-27T09:51:04.770 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.770 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-05-27T09:51:04.770 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-05-27T09:51:04.770 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2594372831 2024-05-27T09:51:04.771 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.771 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:51:04.771 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.771 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.771 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "libcephsqlite", 2024-05-27T09:51:04.771 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-05-27T09:51:04.771 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.771 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-05-27T09:51:04.771 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-05-27T09:51:04.772 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3203834116 2024-05-27T09:51:04.772 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.772 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:51:04.772 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.772 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.772 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2024-05-27T09:51:04.772 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-05-27T09:51:04.772 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.772 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-05-27T09:51:04.773 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-05-27T09:51:04.773 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2778199352 2024-05-27T09:51:04.773 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.773 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:51:04.773 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.773 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.773 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2024-05-27T09:51:04.773 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-05-27T09:51:04.773 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.774 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-05-27T09:51:04.774 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-05-27T09:51:04.774 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 349174012 2024-05-27T09:51:04.774 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.774 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:51:04.774 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.774 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.774 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2024-05-27T09:51:04.775 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-05-27T09:51:04.775 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.775 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-05-27T09:51:04.775 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-05-27T09:51:04.775 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2218285324 2024-05-27T09:51:04.775 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.775 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:51:04.775 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:04.775 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.776 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2024-05-27T09:51:04.776 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-05-27T09:51:04.776 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-05-27T09:51:04.776 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-05-27T09:51:04.776 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-05-27T09:51:04.776 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3401189288 2024-05-27T09:51:04.776 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.776 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:51:04.776 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:04.777 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-05-27T09:51:04.777 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:51:04.778 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:04 smithi189 ceph-mon[36671]: pgmap v498: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-27T09:51:04.778 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:04 smithi050 ceph-mon[36371]: pgmap v498: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-27T09:51:04.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 -- 172.21.15.44:0/900504659 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc40c0b2950 msgr2=0x7fc40c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:04.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc40c0b2950 0x7fc40c0b4e60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc4241a1be0 tx=0x7fc418005f90 comp rx=0 tx=0).stop 2024-05-27T09:51:04.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 -- 172.21.15.44:0/900504659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc42410e870 msgr2=0x7fc4241a6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:04.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc42410e870 0x7fc4241a6dd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc41002a910 tx=0x7fc410003c10 comp rx=0 tx=0).stop 2024-05-27T09:51:04.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 -- 172.21.15.44:0/900504659 shutdown_connections 2024-05-27T09:51:04.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc42410f330 0x7fc4241a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:04.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc40c0b2950 0x7fc40c0b4e60 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:04.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc42410e870 0x7fc4241a6dd0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:04.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 --2- 172.21.15.44:0/900504659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc42410a2e0 0x7fc4241a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:04.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 -- 172.21.15.44:0/900504659 >> 172.21.15.44:0/900504659 conn(0x7fc424076c80 msgr2=0x7fc424107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:04.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 -- 172.21.15.44:0/900504659 shutdown_connections 2024-05-27T09:51:04.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:02.287+0000 7fc42ac00640 1 -- 172.21.15.44:0/900504659 wait complete. 2024-05-27T09:51:04.787 DEBUG:tasks.check_counter:Getting stats from a 2024-05-27T09:51:04.788 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-27T09:51:04.788 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:02 smithi044 ceph-mon[31618]: pgmap v497: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-27T09:51:04.788 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:02 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3447476804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T09:51:04.789 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:02 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1509925310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T09:51:04.789 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:03 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/900504659' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T09:51:04.789 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:04 smithi044 ceph-mon[31618]: pgmap v498: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-27T09:51:05.253 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:51:06.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:06 smithi044 ceph-mon[31618]: pgmap v499: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-27T09:51:06.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:06 smithi050 ceph-mon[36371]: pgmap v499: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-27T09:51:06.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:06 smithi189 ceph-mon[36671]: pgmap v499: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-27T09:51:07.774 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:51:07.775 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:07.775 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 333, 2024-05-27T09:51:07.775 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 321, 2024-05-27T09:51:07.775 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1892176, 2024-05-27T09:51:07.775 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 198320, 2024-05-27T09:51:07.775 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2024-05-27T09:51:07.776 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2024-05-27T09:51:07.776 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.141744856, 2024-05-27T09:51:07.776 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.046426883, 2024-05-27T09:51:07.776 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.108437218, 2024-05-27T09:51:07.776 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.014040074, 2024-05-27T09:51:07.776 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:07.776 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 320, 2024-05-27T09:51:07.776 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.031414875, 2024-05-27T09:51:07.777 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000098171 2024-05-27T09:51:07.777 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:07.777 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:07.777 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:07.777 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:07.777 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:07.777 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:07.777 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1892176, 2024-05-27T09:51:07.778 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 198320 2024-05-27T09:51:07.778 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:07.778 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:07.778 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 7, 2024-05-27T09:51:07.778 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 248, 2024-05-27T09:51:07.778 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 3520, 2024-05-27T09:51:07.778 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 280240, 2024-05-27T09:51:07.778 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 6, 2024-05-27T09:51:07.779 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2024-05-27T09:51:07.779 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.048163580, 2024-05-27T09:51:07.779 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.032586495, 2024-05-27T09:51:07.779 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.008925807, 2024-05-27T09:51:07.779 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:07.779 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:07.779 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 246, 2024-05-27T09:51:07.779 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.059818684, 2024-05-27T09:51:07.780 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000243165 2024-05-27T09:51:07.780 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:07.780 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:07.780 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:07.780 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:07.780 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:07.780 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:07.780 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 3520, 2024-05-27T09:51:07.781 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 280240 2024-05-27T09:51:07.781 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:07.781 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:07.781 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 0, 2024-05-27T09:51:07.781 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 2, 2024-05-27T09:51:07.781 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 0, 2024-05-27T09:51:07.781 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 0, 2024-05-27T09:51:07.781 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2024-05-27T09:51:07.782 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2024-05-27T09:51:07.782 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.001168223, 2024-05-27T09:51:07.782 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000000000, 2024-05-27T09:51:07.782 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.000936988, 2024-05-27T09:51:07.782 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:07.782 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:07.782 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:07.782 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:07.783 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:07.783 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:07.783 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:07.783 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:07.783 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:07.783 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:07.783 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:07.783 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-27T09:51:07.784 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 0 2024-05-27T09:51:07.784 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:07.784 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2024-05-27T09:51:07.784 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2024-05-27T09:51:07.784 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2024-05-27T09:51:07.784 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:07.784 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2024-05-27T09:51:07.785 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2024-05-27T09:51:07.785 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2024-05-27T09:51:07.785 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:07.785 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:07.785 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:07.785 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:07.785 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:07.785 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:07.786 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:07.786 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:07.786 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:07.786 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:07.786 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:07.786 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:07.786 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:07.787 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:07.787 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:07.787 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:07.787 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:07.787 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:07.787 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:07.787 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:07.787 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:07.788 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:07.788 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:07.788 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:07.788 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:07.788 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:07.788 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:07.788 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:07.789 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:07.789 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2024-05-27T09:51:07.789 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:07.789 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:07.789 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:07.789 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:07.789 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 149992, 2024-05-27T09:51:07.789 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 30, 2024-05-27T09:51:07.790 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2024-05-27T09:51:07.790 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2024-05-27T09:51:07.790 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2024-05-27T09:51:07.790 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2024-05-27T09:51:07.790 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:07.790 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:07.790 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:07.791 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:07.791 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2024-05-27T09:51:07.791 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2024-05-27T09:51:07.791 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:07.791 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:07.791 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:07.791 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2024-05-27T09:51:07.791 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2024-05-27T09:51:07.792 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2024-05-27T09:51:07.792 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:07.792 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2024-05-27T09:51:07.792 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:07.792 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2024-05-27T09:51:07.792 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:07.792 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:07.792 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2024-05-27T09:51:07.793 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2024-05-27T09:51:07.793 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2024-05-27T09:51:07.793 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 336, 2024-05-27T09:51:07.793 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1845170, 2024-05-27T09:51:07.793 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:07.793 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 336, 2024-05-27T09:51:07.793 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2024-05-27T09:51:07.794 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2024-05-27T09:51:07.794 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 336, 2024-05-27T09:51:07.794 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1845170, 2024-05-27T09:51:07.794 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2024-05-27T09:51:07.794 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:07.794 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:07.794 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:07.794 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:07.795 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:07.795 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:51:08.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:08 smithi044 ceph-mon[31618]: pgmap v500: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-27T09:51:08.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:08 smithi044 ceph-mon[31618]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T09:51:08.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:08 smithi050 ceph-mon[36371]: pgmap v500: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-27T09:51:08.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:08 smithi050 ceph-mon[36371]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T09:51:08.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:08 smithi189 ceph-mon[36671]: pgmap v500: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-27T09:51:08.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:08 smithi189 ceph-mon[36671]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T09:51:08.685 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-05-27T09:51:08.685 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-05-27T09:51:08.686 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-05-27T09:51:08.686 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-05-27T09:51:08.686 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-05-27T09:51:08.686 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-05-27T09:51:08.686 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-05-27T09:51:08.686 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-05-27T09:51:08.686 DEBUG:tasks.check_counter:Getting stats from d 2024-05-27T09:51:08.687 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.d perf dump 2024-05-27T09:51:09.365 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:51:10.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:10 smithi044 ceph-mon[31618]: pgmap v502: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-27T09:51:10.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:10 smithi050 ceph-mon[36371]: pgmap v502: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-27T09:51:10.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:10 smithi189 ceph-mon[36671]: pgmap v502: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-27T09:51:11.954 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:51:11.955 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:11.955 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 0, 2024-05-27T09:51:11.955 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 2, 2024-05-27T09:51:11.955 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 0, 2024-05-27T09:51:11.955 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 0, 2024-05-27T09:51:11.955 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2024-05-27T09:51:11.956 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2024-05-27T09:51:11.956 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.004477926, 2024-05-27T09:51:11.956 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000000000, 2024-05-27T09:51:11.956 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.001159080, 2024-05-27T09:51:11.956 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:11.956 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:11.956 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:11.956 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:11.957 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:11.957 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:11.957 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:11.957 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:11.957 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:11.957 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:11.957 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:11.957 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-27T09:51:11.958 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 0 2024-05-27T09:51:11.958 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:11.958 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:11.958 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 343, 2024-05-27T09:51:11.958 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 329, 2024-05-27T09:51:11.958 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1896848, 2024-05-27T09:51:11.958 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 202608, 2024-05-27T09:51:11.958 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2024-05-27T09:51:11.959 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2024-05-27T09:51:11.959 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.147991003, 2024-05-27T09:51:11.959 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.050290461, 2024-05-27T09:51:11.959 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.118325335, 2024-05-27T09:51:11.959 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.014083303, 2024-05-27T09:51:11.959 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:11.959 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 329, 2024-05-27T09:51:11.959 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.037606527, 2024-05-27T09:51:11.960 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000114305 2024-05-27T09:51:11.960 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:11.960 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:11.960 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:11.960 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:11.960 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:11.960 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:11.960 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1896848, 2024-05-27T09:51:11.961 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 202608 2024-05-27T09:51:11.961 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:11.961 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:11.961 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 2, 2024-05-27T09:51:11.961 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 250, 2024-05-27T09:51:11.961 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 320, 2024-05-27T09:51:11.961 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 201488, 2024-05-27T09:51:11.961 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 5, 2024-05-27T09:51:11.962 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2024-05-27T09:51:11.962 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.047474947, 2024-05-27T09:51:11.962 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.033477115, 2024-05-27T09:51:11.962 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.006842517, 2024-05-27T09:51:11.962 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:11.962 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:11.962 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 247, 2024-05-27T09:51:11.962 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.040165129, 2024-05-27T09:51:11.963 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000162611 2024-05-27T09:51:11.963 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:11.963 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:11.963 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:11.963 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:11.963 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:11.963 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:11.964 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-27T09:51:11.964 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 201488 2024-05-27T09:51:11.964 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:11.964 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2024-05-27T09:51:11.964 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2024-05-27T09:51:11.964 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2024-05-27T09:51:11.964 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:11.964 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2024-05-27T09:51:11.965 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2024-05-27T09:51:11.965 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2024-05-27T09:51:11.965 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:11.965 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:11.965 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:11.965 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:11.965 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:11.966 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:11.966 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:11.966 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:11.966 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:11.966 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:11.966 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:11.966 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:11.966 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:11.967 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:11.967 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:11.967 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:11.967 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:11.967 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:11.967 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:11.967 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:11.968 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:11.968 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:11.968 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:11.968 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:11.968 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:11.968 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:11.968 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:11.968 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:11.968 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:11.969 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2024-05-27T09:51:11.969 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:11.969 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:11.969 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:11.969 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:11.969 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 150776, 2024-05-27T09:51:11.969 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 32, 2024-05-27T09:51:11.969 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2024-05-27T09:51:11.970 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2024-05-27T09:51:11.970 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2024-05-27T09:51:11.970 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2024-05-27T09:51:11.970 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:11.970 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:11.970 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:11.970 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:11.970 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2024-05-27T09:51:11.971 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2024-05-27T09:51:11.971 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:11.971 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:11.971 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:11.971 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2024-05-27T09:51:11.971 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2024-05-27T09:51:11.971 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2024-05-27T09:51:11.972 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:11.972 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2024-05-27T09:51:11.972 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:11.972 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2024-05-27T09:51:11.972 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:11.972 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:11.972 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2024-05-27T09:51:11.973 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2024-05-27T09:51:11.973 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2024-05-27T09:51:11.973 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 341, 2024-05-27T09:51:11.973 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1845923, 2024-05-27T09:51:11.973 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:11.973 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 341, 2024-05-27T09:51:11.973 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2024-05-27T09:51:11.974 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2024-05-27T09:51:11.974 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 341, 2024-05-27T09:51:11.974 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1845923, 2024-05-27T09:51:11.974 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2024-05-27T09:51:11.974 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:11.974 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:11.974 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:11.974 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:11.975 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:11.975 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:51:12.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:12 smithi044 ceph-mon[31618]: pgmap v503: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:12.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:12 smithi050 ceph-mon[36371]: pgmap v503: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:12 smithi189 ceph-mon[36671]: pgmap v503: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:12.772 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-05-27T09:51:12.773 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-05-27T09:51:12.773 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-05-27T09:51:12.773 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-05-27T09:51:12.773 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-05-27T09:51:12.773 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-05-27T09:51:12.773 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-05-27T09:51:12.774 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-05-27T09:51:12.774 DEBUG:tasks.check_counter:Getting stats from g 2024-05-27T09:51:12.774 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.g perf dump 2024-05-27T09:51:13.412 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:51:14.311 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:14 smithi044 ceph-mon[31618]: pgmap v504: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:14.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:14 smithi050 ceph-mon[36371]: pgmap v504: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:14.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:14 smithi189 ceph-mon[36671]: pgmap v504: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:14.899 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:51:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:51:14.641+0000 7f49fc600700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-27T09:51:16.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:16 smithi044 ceph-mon[31618]: pgmap v505: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:16.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:16 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:51:16.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:16 smithi050 ceph-mon[36371]: pgmap v505: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:16.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:16 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:51:16.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:16 smithi189 ceph-mon[36671]: pgmap v505: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:16.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:16 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:51:16.804 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:51:16.804 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:16.805 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 4, 2024-05-27T09:51:16.805 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 4, 2024-05-27T09:51:16.805 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 3040, 2024-05-27T09:51:16.805 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 448, 2024-05-27T09:51:16.805 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2024-05-27T09:51:16.805 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2024-05-27T09:51:16.805 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.005134726, 2024-05-27T09:51:16.805 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000070033, 2024-05-27T09:51:16.806 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.001661129, 2024-05-27T09:51:16.806 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:16.806 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:16.806 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 3, 2024-05-27T09:51:16.806 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.001551647, 2024-05-27T09:51:16.806 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000517215 2024-05-27T09:51:16.806 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:16.807 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:16.807 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:16.807 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:16.807 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:16.807 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:16.807 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-05-27T09:51:16.807 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 448 2024-05-27T09:51:16.807 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:16.808 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:16.808 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 336, 2024-05-27T09:51:16.808 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 324, 2024-05-27T09:51:16.808 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1892992, 2024-05-27T09:51:16.808 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 200224, 2024-05-27T09:51:16.808 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2024-05-27T09:51:16.808 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2024-05-27T09:51:16.808 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.148395683, 2024-05-27T09:51:16.809 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.050697598, 2024-05-27T09:51:16.809 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.121330679, 2024-05-27T09:51:16.809 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.013080660, 2024-05-27T09:51:16.809 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:16.809 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 323, 2024-05-27T09:51:16.809 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.034787854, 2024-05-27T09:51:16.809 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000107702 2024-05-27T09:51:16.809 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:16.810 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:16.810 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:16.810 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:16.810 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:16.810 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:16.810 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1892992, 2024-05-27T09:51:16.810 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 200224 2024-05-27T09:51:16.810 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:16.811 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:16.811 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 2, 2024-05-27T09:51:16.811 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 248, 2024-05-27T09:51:16.811 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 320, 2024-05-27T09:51:16.811 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 201136, 2024-05-27T09:51:16.811 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 5, 2024-05-27T09:51:16.811 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2024-05-27T09:51:16.811 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.048089846, 2024-05-27T09:51:16.812 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.035157802, 2024-05-27T09:51:16.812 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.006196982, 2024-05-27T09:51:16.812 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:16.812 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:16.812 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 245, 2024-05-27T09:51:16.812 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.038320479, 2024-05-27T09:51:16.812 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000156410 2024-05-27T09:51:16.812 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:16.812 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:16.813 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:16.813 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:16.813 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:16.813 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:16.813 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-27T09:51:16.813 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 201136 2024-05-27T09:51:16.813 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:16.813 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2024-05-27T09:51:16.814 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2024-05-27T09:51:16.814 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2024-05-27T09:51:16.814 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:16.814 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2024-05-27T09:51:16.814 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2024-05-27T09:51:16.814 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2024-05-27T09:51:16.814 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:16.814 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:16.815 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:16.815 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:16.815 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:16.815 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:16.815 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:16.815 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:16.815 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:16.815 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:16.815 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:16.816 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:16.816 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:16.816 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:16.816 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:16.816 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:16.816 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:16.817 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:16.817 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:16.817 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:16.817 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:16.817 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:16.817 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:16.817 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:16.817 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:16.818 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:16.818 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:16.818 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:16.818 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:16.818 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2024-05-27T09:51:16.818 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:16.818 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:16.818 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:16.819 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:16.819 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 150776, 2024-05-27T09:51:16.819 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 32, 2024-05-27T09:51:16.819 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2024-05-27T09:51:16.819 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2024-05-27T09:51:16.819 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2024-05-27T09:51:16.819 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2024-05-27T09:51:16.819 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:16.820 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:16.820 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:16.820 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:16.820 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2024-05-27T09:51:16.820 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2024-05-27T09:51:16.820 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:16.820 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:16.820 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:16.821 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2024-05-27T09:51:16.821 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2024-05-27T09:51:16.821 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2024-05-27T09:51:16.821 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:16.821 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2024-05-27T09:51:16.821 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:16.821 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2024-05-27T09:51:16.821 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:16.822 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:16.822 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2024-05-27T09:51:16.822 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2024-05-27T09:51:16.822 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2024-05-27T09:51:16.822 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 338, 2024-05-27T09:51:16.822 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1845533, 2024-05-27T09:51:16.822 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:16.823 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 338, 2024-05-27T09:51:16.823 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2024-05-27T09:51:16.823 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2024-05-27T09:51:16.823 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 338, 2024-05-27T09:51:16.823 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1845533, 2024-05-27T09:51:16.823 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2024-05-27T09:51:16.823 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:16.823 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:16.823 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:16.824 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:16.824 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:16.824 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:51:17.946 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-05-27T09:51:17.946 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-05-27T09:51:17.946 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-05-27T09:51:17.947 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-05-27T09:51:17.947 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-05-27T09:51:17.947 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-05-27T09:51:17.947 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-05-27T09:51:17.947 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-05-27T09:51:17.947 DEBUG:tasks.check_counter:Getting stats from j 2024-05-27T09:51:17.947 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.j perf dump 2024-05-27T09:51:18.177 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:18 smithi044 ceph-mon[31618]: pgmap v506: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:18.177 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:18 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:51:18.177 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:18 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:51:18.177 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:18 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:51:18.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:18 smithi050 ceph-mon[36371]: pgmap v506: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:18 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:51:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:18 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:51:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:18 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:51:18.584 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:51:18.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:18 smithi189 ceph-mon[36671]: pgmap v506: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-27T09:51:18.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:18 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:51:18.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:18 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:51:18.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:18 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:51:20.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:20 smithi044 ceph-mon[31618]: pgmap v507: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 174 B/s wr, 5 op/s 2024-05-27T09:51:20.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:20 smithi050 ceph-mon[36371]: pgmap v507: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 174 B/s wr, 5 op/s 2024-05-27T09:51:20.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:20 smithi189 ceph-mon[36671]: pgmap v507: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 174 B/s wr, 5 op/s 2024-05-27T09:51:22.193 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-05-27T09:51:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:22.193 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 0, 2024-05-27T09:51:22.194 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 2, 2024-05-27T09:51:22.194 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 0, 2024-05-27T09:51:22.194 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 0, 2024-05-27T09:51:22.194 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2024-05-27T09:51:22.194 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2024-05-27T09:51:22.194 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.003824220, 2024-05-27T09:51:22.194 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000000000, 2024-05-27T09:51:22.195 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.000740641, 2024-05-27T09:51:22.195 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:22.195 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:22.195 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:22.195 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:22.195 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:22.195 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:22.195 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:22.196 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:22.196 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:22.196 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:22.196 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:22.196 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-27T09:51:22.196 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 0 2024-05-27T09:51:22.196 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:22.196 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:22.196 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 347, 2024-05-27T09:51:22.197 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 333, 2024-05-27T09:51:22.197 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1897936, 2024-05-27T09:51:22.197 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 205152, 2024-05-27T09:51:22.197 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2024-05-27T09:51:22.197 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2024-05-27T09:51:22.197 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.158236652, 2024-05-27T09:51:22.197 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.050163238, 2024-05-27T09:51:22.197 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.139512839, 2024-05-27T09:51:22.198 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.013805941, 2024-05-27T09:51:22.198 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:22.198 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 333, 2024-05-27T09:51:22.198 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.030755022, 2024-05-27T09:51:22.198 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000092357 2024-05-27T09:51:22.198 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:22.198 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:22.198 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:22.198 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:22.199 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:22.199 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:22.199 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1897936, 2024-05-27T09:51:22.199 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 205152 2024-05-27T09:51:22.199 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:22.199 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:22.199 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 2, 2024-05-27T09:51:22.199 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 253, 2024-05-27T09:51:22.199 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 320, 2024-05-27T09:51:22.200 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 202016, 2024-05-27T09:51:22.200 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 5, 2024-05-27T09:51:22.200 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2024-05-27T09:51:22.200 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.046917860, 2024-05-27T09:51:22.200 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.034211846, 2024-05-27T09:51:22.200 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.005738628, 2024-05-27T09:51:22.200 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:22.200 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 250, 2024-05-27T09:51:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.043259829, 2024-05-27T09:51:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000173039 2024-05-27T09:51:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:22.201 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-27T09:51:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 202016 2024-05-27T09:51:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2024-05-27T09:51:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2024-05-27T09:51:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2024-05-27T09:51:22.202 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:22.203 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2024-05-27T09:51:22.203 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2024-05-27T09:51:22.203 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2024-05-27T09:51:22.203 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:22.203 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:22.203 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:22.203 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:22.203 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:22.203 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:22.204 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:22.204 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:22.204 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:22.204 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:22.204 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:22.204 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:22.204 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:22.205 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:22.205 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:22.205 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:22.205 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:22.205 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:22.205 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:22.205 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:22.205 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:22.206 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2024-05-27T09:51:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 148714, 2024-05-27T09:51:22.207 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 30, 2024-05-27T09:51:22.208 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2024-05-27T09:51:22.208 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2024-05-27T09:51:22.208 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2024-05-27T09:51:22.208 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2024-05-27T09:51:22.208 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:22.208 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:22.208 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:22.208 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:22.209 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2024-05-27T09:51:22.209 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2024-05-27T09:51:22.209 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:22.209 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:22.209 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:22.209 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2024-05-27T09:51:22.209 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2024-05-27T09:51:22.209 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2024-05-27T09:51:22.210 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:22.210 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2024-05-27T09:51:22.210 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:22.210 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2024-05-27T09:51:22.210 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-05-27T09:51:22.210 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:22.210 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2024-05-27T09:51:22.210 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2024-05-27T09:51:22.211 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2024-05-27T09:51:22.211 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 345, 2024-05-27T09:51:22.211 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1846443, 2024-05-27T09:51:22.211 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:22.211 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 345, 2024-05-27T09:51:22.211 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2024-05-27T09:51:22.211 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2024-05-27T09:51:22.212 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 345, 2024-05-27T09:51:22.212 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1846443, 2024-05-27T09:51:22.212 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2024-05-27T09:51:22.212 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-05-27T09:51:22.212 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-05-27T09:51:22.212 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-05-27T09:51:22.213 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:22.213 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-05-27T09:51:22.213 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-05-27T09:51:22.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:22 smithi044 ceph-mon[31618]: pgmap v508: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-05-27T09:51:22.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:22 smithi050 ceph-mon[36371]: pgmap v508: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-05-27T09:51:22.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:22 smithi189 ceph-mon[36671]: pgmap v508: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-05-27T09:51:23.109 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-05-27T09:51:23.110 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-05-27T09:51:23.110 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-05-27T09:51:23.110 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-05-27T09:51:23.110 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-05-27T09:51:23.110 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-05-27T09:51:23.110 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-05-27T09:51:23.110 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-05-27T09:51:23.111 DEBUG:tasks.check_counter:Getting stats from b 2024-05-27T09:51:23.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.b perf dump 2024-05-27T09:51:23.538 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:51:24.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:24 smithi050 ceph-mon[36371]: pgmap v509: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:51:24.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:24 smithi044 ceph-mon[31618]: pgmap v509: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:51:24.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:24 smithi189 ceph-mon[36671]: pgmap v509: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:51:26.329 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:26 smithi050 ceph-mon[36371]: pgmap v510: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:51:26.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:26 smithi044 ceph-mon[31618]: pgmap v510: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:51:26.581 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-05-27T09:51:26.581 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:26.581 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 56281, 2024-05-27T09:51:26.582 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 39702, 2024-05-27T09:51:26.582 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 115178759, 2024-05-27T09:51:26.582 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 122465938, 2024-05-27T09:51:26.582 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 85, 2024-05-27T09:51:26.582 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 4, 2024-05-27T09:51:26.582 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 4.041599857, 2024-05-27T09:51:26.582 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 1.392028010, 2024-05-27T09:51:26.583 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 2.522069190, 2024-05-27T09:51:26.583 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.635867651, 2024-05-27T09:51:26.583 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:26.583 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 39690, 2024-05-27T09:51:26.583 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 10.198499608, 2024-05-27T09:51:26.583 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000256953 2024-05-27T09:51:26.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.584 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:26.584 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 58842, 2024-05-27T09:51:26.584 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.011360927, 2024-05-27T09:51:26.584 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000193 2024-05-27T09:51:26.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.584 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 2568032, 2024-05-27T09:51:26.584 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 391904 2024-05-27T09:51:26.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.585 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:26.585 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 98343, 2024-05-27T09:51:26.585 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 146787, 2024-05-27T09:51:26.585 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 36386134, 2024-05-27T09:51:26.585 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 146502404, 2024-05-27T09:51:26.585 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 183, 2024-05-27T09:51:26.585 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 5, 2024-05-27T09:51:26.585 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 6.268160301, 2024-05-27T09:51:26.585 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 2.770921247, 2024-05-27T09:51:26.586 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 3.579171958, 2024-05-27T09:51:26.586 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.720296868, 2024-05-27T09:51:26.586 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:26.586 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 146785, 2024-05-27T09:51:26.586 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 17.559175023, 2024-05-27T09:51:26.586 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000119625 2024-05-27T09:51:26.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.587 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:26.587 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 92781, 2024-05-27T09:51:26.587 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.030231200, 2024-05-27T09:51:26.587 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000325 2024-05-27T09:51:26.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.587 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 105584, 2024-05-27T09:51:26.587 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 643824 2024-05-27T09:51:26.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.588 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:26.588 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 47437, 2024-05-27T09:51:26.588 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 43677, 2024-05-27T09:51:26.588 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 83450851, 2024-05-27T09:51:26.588 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 251302476, 2024-05-27T09:51:26.588 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 41, 2024-05-27T09:51:26.588 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 7, 2024-05-27T09:51:26.588 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 3.828035977, 2024-05-27T09:51:26.589 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 1.674486402, 2024-05-27T09:51:26.589 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 1.552314193, 2024-05-27T09:51:26.589 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.682459293, 2024-05-27T09:51:26.589 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:26.589 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 43675, 2024-05-27T09:51:26.589 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 15.865881439, 2024-05-27T09:51:26.589 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000363271 2024-05-27T09:51:26.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.589 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:26.590 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 40470, 2024-05-27T09:51:26.590 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.007702426, 2024-05-27T09:51:26.590 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000190 2024-05-27T09:51:26.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.590 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 43472, 2024-05-27T09:51:26.590 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 4416 2024-05-27T09:51:26.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.591 INFO:teuthology.orchestra.run.smithi050.stdout: "cct": { 2024-05-27T09:51:26.591 INFO:teuthology.orchestra.run.smithi050.stdout: "total_workers": 1, 2024-05-27T09:51:26.591 INFO:teuthology.orchestra.run.smithi050.stdout: "unhealthy_workers": 0 2024-05-27T09:51:26.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.591 INFO:teuthology.orchestra.run.smithi050.stdout: "finisher-MDSRank": { 2024-05-27T09:51:26.591 INFO:teuthology.orchestra.run.smithi050.stdout: "queue_len": 0, 2024-05-27T09:51:26.591 INFO:teuthology.orchestra.run.smithi050.stdout: "complete_latency": { 2024-05-27T09:51:26.591 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 29340, 2024-05-27T09:51:26.592 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 47.669141869, 2024-05-27T09:51:26.592 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.001624715 2024-05-27T09:51:26.592 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:26.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.592 INFO:teuthology.orchestra.run.smithi050.stdout: "finisher-PurgeQueue": { 2024-05-27T09:51:26.592 INFO:teuthology.orchestra.run.smithi050.stdout: "queue_len": 0, 2024-05-27T09:51:26.592 INFO:teuthology.orchestra.run.smithi050.stdout: "complete_latency": { 2024-05-27T09:51:26.592 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 18019, 2024-05-27T09:51:26.593 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 2.296830218, 2024-05-27T09:51:26.593 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000127467 2024-05-27T09:51:26.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:26.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.593 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": { 2024-05-27T09:51:26.593 INFO:teuthology.orchestra.run.smithi050.stdout: "request": 49038, 2024-05-27T09:51:26.593 INFO:teuthology.orchestra.run.smithi050.stdout: "reply": 34474, 2024-05-27T09:51:26.593 INFO:teuthology.orchestra.run.smithi050.stdout: "reply_latency": { 2024-05-27T09:51:26.594 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 34474, 2024-05-27T09:51:26.594 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 281.289350953, 2024-05-27T09:51:26.594 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.008159463 2024-05-27T09:51:26.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.594 INFO:teuthology.orchestra.run.smithi050.stdout: "slow_reply": 0, 2024-05-27T09:51:26.594 INFO:teuthology.orchestra.run.smithi050.stdout: "forward": 14564, 2024-05-27T09:51:26.594 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_fetch_complete": 100, 2024-05-27T09:51:26.594 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_fetch_keys": 1, 2024-05-27T09:51:26.595 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_commit": 8959, 2024-05-27T09:51:26.595 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_split": 77, 2024-05-27T09:51:26.595 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_merge": 0, 2024-05-27T09:51:26.595 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes": 31461, 2024-05-27T09:51:26.595 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_top": 22005, 2024-05-27T09:51:26.595 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_bottom": 9441, 2024-05-27T09:51:26.595 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_pin_tail": 15, 2024-05-27T09:51:26.595 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_pinned": 22, 2024-05-27T09:51:26.596 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_expired": 2450, 2024-05-27T09:51:26.596 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_with_caps": 3, 2024-05-27T09:51:26.596 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": 4, 2024-05-27T09:51:26.596 INFO:teuthology.orchestra.run.smithi050.stdout: "subtrees": 3, 2024-05-27T09:51:26.596 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse": 58732, 2024-05-27T09:51:26.596 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_hit": 43853, 2024-05-27T09:51:26.596 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_forward": 14563, 2024-05-27T09:51:26.596 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_discover": 0, 2024-05-27T09:51:26.596 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_dir_fetch": 0, 2024-05-27T09:51:26.597 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_remote_ino": 0, 2024-05-27T09:51:26.597 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_lock": 0, 2024-05-27T09:51:26.597 INFO:teuthology.orchestra.run.smithi050.stdout: "load_cent": 0, 2024-05-27T09:51:26.597 INFO:teuthology.orchestra.run.smithi050.stdout: "q": 0, 2024-05-27T09:51:26.597 INFO:teuthology.orchestra.run.smithi050.stdout: "exported": 8, 2024-05-27T09:51:26.597 INFO:teuthology.orchestra.run.smithi050.stdout: "exported_inodes": 2407, 2024-05-27T09:51:26.597 INFO:teuthology.orchestra.run.smithi050.stdout: "imported": 512, 2024-05-27T09:51:26.597 INFO:teuthology.orchestra.run.smithi050.stdout: "imported_inodes": 24794, 2024-05-27T09:51:26.598 INFO:teuthology.orchestra.run.smithi050.stdout: "openino_dir_fetch": 0, 2024-05-27T09:51:26.598 INFO:teuthology.orchestra.run.smithi050.stdout: "openino_backtrace_fetch": 30132, 2024-05-27T09:51:26.598 INFO:teuthology.orchestra.run.smithi050.stdout: "openino_peer_discover": 0, 2024-05-27T09:51:26.598 INFO:teuthology.orchestra.run.smithi050.stdout: "root_rfiles": 4, 2024-05-27T09:51:26.598 INFO:teuthology.orchestra.run.smithi050.stdout: "root_rbytes": 234, 2024-05-27T09:51:26.598 INFO:teuthology.orchestra.run.smithi050.stdout: "root_rsnaps": 6, 2024-05-27T09:51:26.598 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_backtrace_fetch": 30132, 2024-05-27T09:51:26.599 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_set_tag": 0, 2024-05-27T09:51:26.599 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_backtrace_repaired": 0, 2024-05-27T09:51:26.599 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_inotable_repaired": 0, 2024-05-27T09:51:26.599 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_dir_inodes": 137, 2024-05-27T09:51:26.599 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_dir_base_inodes": 13, 2024-05-27T09:51:26.599 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_dirfrag_rstats": 124, 2024-05-27T09:51:26.599 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_file_inodes": 29995, 2024-05-27T09:51:26.599 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_inode_file_caps": 18, 2024-05-27T09:51:26.600 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_revoke": 11327, 2024-05-27T09:51:26.600 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_grant": 22597, 2024-05-27T09:51:26.600 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_trunc": 1, 2024-05-27T09:51:26.600 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-27T09:51:26.600 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-27T09:51:26.600 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_caps": 24259, 2024-05-27T09:51:26.600 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_caps_dirty": 14803, 2024-05-27T09:51:26.600 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_cap_release": 226, 2024-05-27T09:51:26.601 INFO:teuthology.orchestra.run.smithi050.stdout: "process_request_cap_release": 12230 2024-05-27T09:51:26.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.601 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache": { 2024-05-27T09:51:26.601 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_update": 983, 2024-05-27T09:51:26.601 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_update_receipt": 0, 2024-05-27T09:51:26.601 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_try_discover": 0, 2024-05-27T09:51:26.601 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_send_discover": 0, 2024-05-27T09:51:26.601 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_handle_discover": 2052, 2024-05-27T09:51:26.602 INFO:teuthology.orchestra.run.smithi050.stdout: "num_strays": 9, 2024-05-27T09:51:26.602 INFO:teuthology.orchestra.run.smithi050.stdout: "num_strays_delayed": 0, 2024-05-27T09:51:26.602 INFO:teuthology.orchestra.run.smithi050.stdout: "num_strays_enqueuing": 0, 2024-05-27T09:51:26.602 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_created": 10121, 2024-05-27T09:51:26.602 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_enqueued": 10112, 2024-05-27T09:51:26.602 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_reintegrated": 0, 2024-05-27T09:51:26.602 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_migrated": 0, 2024-05-27T09:51:26.602 INFO:teuthology.orchestra.run.smithi050.stdout: "num_recovering_processing": 0, 2024-05-27T09:51:26.603 INFO:teuthology.orchestra.run.smithi050.stdout: "num_recovering_enqueued": 0, 2024-05-27T09:51:26.603 INFO:teuthology.orchestra.run.smithi050.stdout: "num_recovering_prioritized": 0, 2024-05-27T09:51:26.603 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_started": 0, 2024-05-27T09:51:26.603 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_completed": 0, 2024-05-27T09:51:26.603 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_quiesce_path": 10, 2024-05-27T09:51:26.603 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_quiesce_inode": 20241, 2024-05-27T09:51:26.603 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_enqueue_scrub": 13, 2024-05-27T09:51:26.603 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_exportdir": 8, 2024-05-27T09:51:26.604 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_flush": 0, 2024-05-27T09:51:26.604 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_fragmentdir": 81, 2024-05-27T09:51:26.604 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_fragstats": 0, 2024-05-27T09:51:26.604 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_inodestats": 0 2024-05-27T09:51:26.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.604 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log": { 2024-05-27T09:51:26.604 INFO:teuthology.orchestra.run.smithi050.stdout: "evlrg": 22, 2024-05-27T09:51:26.604 INFO:teuthology.orchestra.run.smithi050.stdout: "evadd": 49994, 2024-05-27T09:51:26.605 INFO:teuthology.orchestra.run.smithi050.stdout: "evex": 49980, 2024-05-27T09:51:26.605 INFO:teuthology.orchestra.run.smithi050.stdout: "evtrm": 49980, 2024-05-27T09:51:26.605 INFO:teuthology.orchestra.run.smithi050.stdout: "ev": 14, 2024-05-27T09:51:26.605 INFO:teuthology.orchestra.run.smithi050.stdout: "evexg": 0, 2024-05-27T09:51:26.605 INFO:teuthology.orchestra.run.smithi050.stdout: "evexd": 9, 2024-05-27T09:51:26.605 INFO:teuthology.orchestra.run.smithi050.stdout: "segadd": 135, 2024-05-27T09:51:26.605 INFO:teuthology.orchestra.run.smithi050.stdout: "segex": 134, 2024-05-27T09:51:26.605 INFO:teuthology.orchestra.run.smithi050.stdout: "segtrm": 134, 2024-05-27T09:51:26.606 INFO:teuthology.orchestra.run.smithi050.stdout: "seg": 1, 2024-05-27T09:51:26.606 INFO:teuthology.orchestra.run.smithi050.stdout: "segmjr": 1, 2024-05-27T09:51:26.606 INFO:teuthology.orchestra.run.smithi050.stdout: "segexg": 0, 2024-05-27T09:51:26.606 INFO:teuthology.orchestra.run.smithi050.stdout: "segexd": 1, 2024-05-27T09:51:26.606 INFO:teuthology.orchestra.run.smithi050.stdout: "expos": 223422806, 2024-05-27T09:51:26.606 INFO:teuthology.orchestra.run.smithi050.stdout: "wrpos": 223475164, 2024-05-27T09:51:26.606 INFO:teuthology.orchestra.run.smithi050.stdout: "rdpos": 0, 2024-05-27T09:51:26.606 INFO:teuthology.orchestra.run.smithi050.stdout: "jlat": { 2024-05-27T09:51:26.607 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 3439, 2024-05-27T09:51:26.607 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 129.352039299, 2024-05-27T09:51:26.607 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.037613271 2024-05-27T09:51:26.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.607 INFO:teuthology.orchestra.run.smithi050.stdout: "replayed": 0 2024-05-27T09:51:26.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.607 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_mem": { 2024-05-27T09:51:26.607 INFO:teuthology.orchestra.run.smithi050.stdout: "ino": 29417, 2024-05-27T09:51:26.608 INFO:teuthology.orchestra.run.smithi050.stdout: "ino+": 41958, 2024-05-27T09:51:26.608 INFO:teuthology.orchestra.run.smithi050.stdout: "ino-": 12541, 2024-05-27T09:51:26.608 INFO:teuthology.orchestra.run.smithi050.stdout: "dir": 1063, 2024-05-27T09:51:26.608 INFO:teuthology.orchestra.run.smithi050.stdout: "dir+": 3115, 2024-05-27T09:51:26.608 INFO:teuthology.orchestra.run.smithi050.stdout: "dir-": 2052, 2024-05-27T09:51:26.608 INFO:teuthology.orchestra.run.smithi050.stdout: "dn": 31461, 2024-05-27T09:51:26.608 INFO:teuthology.orchestra.run.smithi050.stdout: "dn+": 61995, 2024-05-27T09:51:26.608 INFO:teuthology.orchestra.run.smithi050.stdout: "dn-": 30534, 2024-05-27T09:51:26.609 INFO:teuthology.orchestra.run.smithi050.stdout: "cap": 4, 2024-05-27T09:51:26.609 INFO:teuthology.orchestra.run.smithi050.stdout: "cap+": 23786, 2024-05-27T09:51:26.609 INFO:teuthology.orchestra.run.smithi050.stdout: "cap-": 23782, 2024-05-27T09:51:26.609 INFO:teuthology.orchestra.run.smithi050.stdout: "rss": 309536, 2024-05-27T09:51:26.609 INFO:teuthology.orchestra.run.smithi050.stdout: "heap": 231824 2024-05-27T09:51:26.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.609 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_server": { 2024-05-27T09:51:26.609 INFO:teuthology.orchestra.run.smithi050.stdout: "dispatch_client_request": 50827, 2024-05-27T09:51:26.610 INFO:teuthology.orchestra.run.smithi050.stdout: "dispatch_server_request": 20194, 2024-05-27T09:51:26.610 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_request": 49038, 2024-05-27T09:51:26.610 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_session": 51, 2024-05-27T09:51:26.610 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_peer_request": 40511, 2024-05-27T09:51:26.610 INFO:teuthology.orchestra.run.smithi050.stdout: "req_create_latency": { 2024-05-27T09:51:26.610 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 12416, 2024-05-27T09:51:26.610 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 120.470243231, 2024-05-27T09:51:26.610 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.009702822 2024-05-27T09:51:26.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.611 INFO:teuthology.orchestra.run.smithi050.stdout: "req_getattr_latency": { 2024-05-27T09:51:26.611 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 181, 2024-05-27T09:51:26.611 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.920919089, 2024-05-27T09:51:26.611 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.005087950 2024-05-27T09:51:26.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.611 INFO:teuthology.orchestra.run.smithi050.stdout: "req_getfilelock_latency": { 2024-05-27T09:51:26.611 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.612 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.612 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.612 INFO:teuthology.orchestra.run.smithi050.stdout: "req_link_latency": { 2024-05-27T09:51:26.612 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.612 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.612 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.613 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookup_latency": { 2024-05-27T09:51:26.613 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 9491, 2024-05-27T09:51:26.613 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 6.357764209, 2024-05-27T09:51:26.613 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000669872 2024-05-27T09:51:26.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.613 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookuphash_latency": { 2024-05-27T09:51:26.613 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.613 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.614 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.614 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupino_latency": { 2024-05-27T09:51:26.614 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 1, 2024-05-27T09:51:26.614 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000332995, 2024-05-27T09:51:26.614 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000332995 2024-05-27T09:51:26.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.614 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupname_latency": { 2024-05-27T09:51:26.615 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.615 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.615 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.615 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupparent_latency": { 2024-05-27T09:51:26.615 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.615 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.615 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.616 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupsnap_latency": { 2024-05-27T09:51:26.616 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:26 smithi189 ceph-mon[36671]: pgmap v510: 97 pgs: 42 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 33 active+clean; 342 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T09:51:26.617 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.617 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.617 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lssnap_latency": { 2024-05-27T09:51:26.617 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 16, 2024-05-27T09:51:26.618 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.004352594, 2024-05-27T09:51:26.618 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000272037 2024-05-27T09:51:26.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.618 INFO:teuthology.orchestra.run.smithi050.stdout: "req_mkdir_latency": { 2024-05-27T09:51:26.618 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 125, 2024-05-27T09:51:26.618 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 11.404022065, 2024-05-27T09:51:26.618 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.091232176 2024-05-27T09:51:26.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.619 INFO:teuthology.orchestra.run.smithi050.stdout: "req_mknod_latency": { 2024-05-27T09:51:26.619 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.619 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.619 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.619 INFO:teuthology.orchestra.run.smithi050.stdout: "req_mksnap_latency": { 2024-05-27T09:51:26.619 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 16, 2024-05-27T09:51:26.620 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 1.815156315, 2024-05-27T09:51:26.620 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.113447269 2024-05-27T09:51:26.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.620 INFO:teuthology.orchestra.run.smithi050.stdout: "req_open_latency": { 2024-05-27T09:51:26.620 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 1, 2024-05-27T09:51:26.620 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000262421, 2024-05-27T09:51:26.620 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000262421 2024-05-27T09:51:26.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.621 INFO:teuthology.orchestra.run.smithi050.stdout: "req_readdir_latency": { 2024-05-27T09:51:26.621 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 2057, 2024-05-27T09:51:26.621 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 20.025303950, 2024-05-27T09:51:26.621 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.009735198 2024-05-27T09:51:26.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.621 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rename_latency": { 2024-05-27T09:51:26.621 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 2, 2024-05-27T09:51:26.621 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.002533299, 2024-05-27T09:51:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.001266649 2024-05-27T09:51:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: "req_renamesnap_latency": { 2024-05-27T09:51:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: "req_snapdiff_latency": { 2024-05-27T09:51:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rmdir_latency": { 2024-05-27T09:51:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 113, 2024-05-27T09:51:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 18.142925530, 2024-05-27T09:51:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.160556863 2024-05-27T09:51:26.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.624 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rmsnap_latency": { 2024-05-27T09:51:26.624 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 10, 2024-05-27T09:51:26.624 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.467350258, 2024-05-27T09:51:26.624 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.046735025 2024-05-27T09:51:26.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.624 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rmxattr_latency": { 2024-05-27T09:51:26.624 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 1, 2024-05-27T09:51:26.625 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000240043, 2024-05-27T09:51:26.625 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000240043 2024-05-27T09:51:26.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.625 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setattr_latency": { 2024-05-27T09:51:26.625 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 3, 2024-05-27T09:51:26.625 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.008747485, 2024-05-27T09:51:26.625 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.002915828 2024-05-27T09:51:26.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.626 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setdirlayout_latency": { 2024-05-27T09:51:26.626 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.626 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.626 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.626 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setfilelock_latency": { 2024-05-27T09:51:26.626 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 22, 2024-05-27T09:51:26.627 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 10.020078354, 2024-05-27T09:51:26.627 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.455458107 2024-05-27T09:51:26.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.627 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setlayout_latency": { 2024-05-27T09:51:26.627 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.627 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.627 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.628 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setxattr_latency": { 2024-05-27T09:51:26.628 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 11, 2024-05-27T09:51:26.628 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.027351469, 2024-05-27T09:51:26.628 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.002486497 2024-05-27T09:51:26.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.628 INFO:teuthology.orchestra.run.smithi050.stdout: "req_symlink_latency": { 2024-05-27T09:51:26.628 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.628 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.629 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.629 INFO:teuthology.orchestra.run.smithi050.stdout: "req_unlink_latency": { 2024-05-27T09:51:26.629 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 10008, 2024-05-27T09:51:26.629 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 91.621767646, 2024-05-27T09:51:26.629 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.009154852 2024-05-27T09:51:26.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.629 INFO:teuthology.orchestra.run.smithi050.stdout: "cap_revoke_eviction": 0, 2024-05-27T09:51:26.630 INFO:teuthology.orchestra.run.smithi050.stdout: "cap_acquisition_throttle": 0, 2024-05-27T09:51:26.630 INFO:teuthology.orchestra.run.smithi050.stdout: "req_getvxattr_latency": { 2024-05-27T09:51:26.630 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.630 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.630 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:26.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.630 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_sessions": { 2024-05-27T09:51:26.631 INFO:teuthology.orchestra.run.smithi050.stdout: "session_count": 2, 2024-05-27T09:51:26.631 INFO:teuthology.orchestra.run.smithi050.stdout: "session_add": 6, 2024-05-27T09:51:26.631 INFO:teuthology.orchestra.run.smithi050.stdout: "session_remove": 4, 2024-05-27T09:51:26.631 INFO:teuthology.orchestra.run.smithi050.stdout: "sessions_open": 2, 2024-05-27T09:51:26.631 INFO:teuthology.orchestra.run.smithi050.stdout: "sessions_stale": 0, 2024-05-27T09:51:26.631 INFO:teuthology.orchestra.run.smithi050.stdout: "total_load": 1295, 2024-05-27T09:51:26.631 INFO:teuthology.orchestra.run.smithi050.stdout: "average_load": 647, 2024-05-27T09:51:26.631 INFO:teuthology.orchestra.run.smithi050.stdout: "avg_session_uptime": 945, 2024-05-27T09:51:26.632 INFO:teuthology.orchestra.run.smithi050.stdout: "mdthresh_evicted": 0 2024-05-27T09:51:26.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.632 INFO:teuthology.orchestra.run.smithi050.stdout: "mempool": { 2024-05-27T09:51:26.632 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_bytes": 360, 2024-05-27T09:51:26.632 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_items": 360, 2024-05-27T09:51:26.632 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:26.632 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:26.632 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:26.633 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:26.633 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:26.633 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:26.633 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:26.633 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:26.633 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:26.633 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:26.633 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:26.634 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:26.634 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:26.634 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:26.634 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:26.634 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:26.634 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:26.634 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:26.634 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:26.635 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:26.635 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:26.635 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:26.635 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:26.635 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:26.635 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:26.635 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:26.635 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:26.636 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:26.636 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:26.636 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_items": 0, 2024-05-27T09:51:26.636 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:26.636 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:26.636 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:26.636 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:26.636 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_bytes": 1275073, 2024-05-27T09:51:26.636 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_items": 25063, 2024-05-27T09:51:26.637 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_bytes": 88, 2024-05-27T09:51:26.637 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_items": 1, 2024-05-27T09:51:26.637 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bytes": 0, 2024-05-27T09:51:26.637 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_items": 0, 2024-05-27T09:51:26.637 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:26.637 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:26.637 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:26.637 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:26.638 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_bytes": 8448, 2024-05-27T09:51:26.638 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_items": 193, 2024-05-27T09:51:26.638 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:26.638 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:26.638 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:26.638 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_items": 0, 2024-05-27T09:51:26.638 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_bytes": 90749305, 2024-05-27T09:51:26.638 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_items": 761654, 2024-05-27T09:51:26.639 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:26.639 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_items": 0, 2024-05-27T09:51:26.639 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:26.639 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_items": 0 2024-05-27T09:51:26.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.639 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter": { 2024-05-27T09:51:26.639 INFO:teuthology.orchestra.run.smithi050.stdout: "op_active": 0, 2024-05-27T09:51:26.639 INFO:teuthology.orchestra.run.smithi050.stdout: "op_laggy": 0, 2024-05-27T09:51:26.640 INFO:teuthology.orchestra.run.smithi050.stdout: "op_send": 66799, 2024-05-27T09:51:26.640 INFO:teuthology.orchestra.run.smithi050.stdout: "op_send_bytes": 265545265, 2024-05-27T09:51:26.640 INFO:teuthology.orchestra.run.smithi050.stdout: "op_resend": 37, 2024-05-27T09:51:26.640 INFO:teuthology.orchestra.run.smithi050.stdout: "op_reply": 66799, 2024-05-27T09:51:26.640 INFO:teuthology.orchestra.run.smithi050.stdout: "op_latency": { 2024-05-27T09:51:26.640 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 66799, 2024-05-27T09:51:26.640 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 3039.458258848, 2024-05-27T09:51:26.640 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.045501553 2024-05-27T09:51:26.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.641 INFO:teuthology.orchestra.run.smithi050.stdout: "op_inflight": 0, 2024-05-27T09:51:26.641 INFO:teuthology.orchestra.run.smithi050.stdout: "oplen_avg": { 2024-05-27T09:51:26.641 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 66799, 2024-05-27T09:51:26.641 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 105327 2024-05-27T09:51:26.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.641 INFO:teuthology.orchestra.run.smithi050.stdout: "op": 66799, 2024-05-27T09:51:26.641 INFO:teuthology.orchestra.run.smithi050.stdout: "op_r": 30478, 2024-05-27T09:51:26.642 INFO:teuthology.orchestra.run.smithi050.stdout: "op_w": 36321, 2024-05-27T09:51:26.642 INFO:teuthology.orchestra.run.smithi050.stdout: "op_rmw": 0, 2024-05-27T09:51:26.642 INFO:teuthology.orchestra.run.smithi050.stdout: "op_pg": 0, 2024-05-27T09:51:26.642 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_stat": 8199, 2024-05-27T09:51:26.642 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_create": 12147, 2024-05-27T09:51:26.642 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_read": 232, 2024-05-27T09:51:26.642 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_write": 3658, 2024-05-27T09:51:26.642 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_writefull": 346, 2024-05-27T09:51:26.643 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_writesame": 0, 2024-05-27T09:51:26.643 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_append": 0, 2024-05-27T09:51:26.643 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_zero": 0, 2024-05-27T09:51:26.643 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_truncate": 5, 2024-05-27T09:51:26.643 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_delete": 10800, 2024-05-27T09:51:26.643 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_mapext": 0, 2024-05-27T09:51:26.643 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_sparse_read": 0, 2024-05-27T09:51:26.643 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_clonerange": 0, 2024-05-27T09:51:26.644 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_getxattr": 30151, 2024-05-27T09:51:26.644 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_setxattr": 24294, 2024-05-27T09:51:26.644 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_cmpxattr": 0, 2024-05-27T09:51:26.644 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_rmxattr": 0, 2024-05-27T09:51:26.644 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_resetxattrs": 0, 2024-05-27T09:51:26.644 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_call": 0, 2024-05-27T09:51:26.644 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_watch": 0, 2024-05-27T09:51:26.644 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_notify": 0, 2024-05-27T09:51:26.645 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_src_cmpxattr": 0, 2024-05-27T09:51:26.645 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_pgls": 0, 2024-05-27T09:51:26.645 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_pgls_filter": 0, 2024-05-27T09:51:26.645 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_other": 1, 2024-05-27T09:51:26.645 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_active": 0, 2024-05-27T09:51:26.645 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_send": 0, 2024-05-27T09:51:26.645 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_resend": 0, 2024-05-27T09:51:26.645 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_ping": 0, 2024-05-27T09:51:26.646 INFO:teuthology.orchestra.run.smithi050.stdout: "poolop_active": 0, 2024-05-27T09:51:26.646 INFO:teuthology.orchestra.run.smithi050.stdout: "poolop_send": 0, 2024-05-27T09:51:26.646 INFO:teuthology.orchestra.run.smithi050.stdout: "poolop_resend": 0, 2024-05-27T09:51:26.646 INFO:teuthology.orchestra.run.smithi050.stdout: "poolstat_active": 0, 2024-05-27T09:51:26.646 INFO:teuthology.orchestra.run.smithi050.stdout: "poolstat_send": 0, 2024-05-27T09:51:26.646 INFO:teuthology.orchestra.run.smithi050.stdout: "poolstat_resend": 0, 2024-05-27T09:51:26.646 INFO:teuthology.orchestra.run.smithi050.stdout: "statfs_active": 0, 2024-05-27T09:51:26.646 INFO:teuthology.orchestra.run.smithi050.stdout: "statfs_send": 0, 2024-05-27T09:51:26.647 INFO:teuthology.orchestra.run.smithi050.stdout: "statfs_resend": 0, 2024-05-27T09:51:26.647 INFO:teuthology.orchestra.run.smithi050.stdout: "command_active": 0, 2024-05-27T09:51:26.647 INFO:teuthology.orchestra.run.smithi050.stdout: "command_send": 0, 2024-05-27T09:51:26.647 INFO:teuthology.orchestra.run.smithi050.stdout: "command_resend": 0, 2024-05-27T09:51:26.647 INFO:teuthology.orchestra.run.smithi050.stdout: "map_epoch": 104, 2024-05-27T09:51:26.647 INFO:teuthology.orchestra.run.smithi050.stdout: "map_full": 0, 2024-05-27T09:51:26.647 INFO:teuthology.orchestra.run.smithi050.stdout: "map_inc": 25, 2024-05-27T09:51:26.647 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_sessions": 12, 2024-05-27T09:51:26.647 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_session_open": 12, 2024-05-27T09:51:26.648 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_session_close": 0, 2024-05-27T09:51:26.648 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_laggy": 0, 2024-05-27T09:51:26.648 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_wr": 14117, 2024-05-27T09:51:26.648 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_rd": 202, 2024-05-27T09:51:26.648 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_del": 1175 2024-05-27T09:51:26.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.648 INFO:teuthology.orchestra.run.smithi050.stdout: "oft": { 2024-05-27T09:51:26.648 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_objs": 1, 2024-05-27T09:51:26.649 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_kv_pairs": 0, 2024-05-27T09:51:26.649 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_updates": 0, 2024-05-27T09:51:26.649 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_removes": 0 2024-05-27T09:51:26.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.649 INFO:teuthology.orchestra.run.smithi050.stdout: "osd-slow-ops": { 2024-05-27T09:51:26.649 INFO:teuthology.orchestra.run.smithi050.stdout: "slow_ops_count": 20141 2024-05-27T09:51:26.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.649 INFO:teuthology.orchestra.run.smithi050.stdout: "purge_queue": { 2024-05-27T09:51:26.650 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing_ops": 0, 2024-05-27T09:51:26.650 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing_ops_high_water": 10, 2024-05-27T09:51:26.650 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing": 0, 2024-05-27T09:51:26.650 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing_high_water": 10, 2024-05-27T09:51:26.650 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executed_ops": 10112, 2024-05-27T09:51:26.650 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executed": 10112, 2024-05-27T09:51:26.650 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_item_in_journal": 0 2024-05-27T09:51:26.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.651 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:26.651 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:26.651 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 104857600, 2024-05-27T09:51:26.651 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:26.651 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 202057, 2024-05-27T09:51:26.651 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 217915387, 2024-05-27T09:51:26.651 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:26.651 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 202057, 2024-05-27T09:51:26.652 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-05-27T09:51:26.652 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-05-27T09:51:26.652 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 202057, 2024-05-27T09:51:26.652 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 217915387, 2024-05-27T09:51:26.652 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:26.652 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.652 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.652 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.653 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:26.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.653 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-objecter_bytes": { 2024-05-27T09:51:26.653 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:26.653 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 104857600, 2024-05-27T09:51:26.653 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:26.653 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 0, 2024-05-27T09:51:26.653 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 0, 2024-05-27T09:51:26.654 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:26.654 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:26.654 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 66799, 2024-05-27T09:51:26.654 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 266841181, 2024-05-27T09:51:26.654 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 66799, 2024-05-27T09:51:26.654 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 266841181, 2024-05-27T09:51:26.654 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:26.654 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.654 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.655 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.655 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:26.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.655 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-objecter_ops": { 2024-05-27T09:51:26.655 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:26.655 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 1024, 2024-05-27T09:51:26.655 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:26.655 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 0, 2024-05-27T09:51:26.656 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 0, 2024-05-27T09:51:26.656 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:26.656 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:26.656 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 66799, 2024-05-27T09:51:26.656 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 66799, 2024-05-27T09:51:26.656 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 66799, 2024-05-27T09:51:26.656 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 66799, 2024-05-27T09:51:26.656 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:26.657 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.657 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.657 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.657 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:26.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.657 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-write_buf_throttle": { 2024-05-27T09:51:26.657 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:26.657 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 3758096384, 2024-05-27T09:51:26.658 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:26.658 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 10112, 2024-05-27T09:51:26.658 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 1297520, 2024-05-27T09:51:26.658 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:26.658 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 10112, 2024-05-27T09:51:26.658 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-05-27T09:51:26.658 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-05-27T09:51:26.658 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 219, 2024-05-27T09:51:26.659 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 1297520, 2024-05-27T09:51:26.659 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:26.659 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.659 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.659 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.659 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:26.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:26.659 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-write_buf_throttle-0x5623286e6b60": { 2024-05-27T09:51:26.660 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:26.660 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 3758096384, 2024-05-27T09:51:26.660 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:26.660 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 49994, 2024-05-27T09:51:26.660 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 219280860, 2024-05-27T09:51:26.660 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:26.660 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 49994, 2024-05-27T09:51:26.660 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-05-27T09:51:26.661 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-05-27T09:51:26.661 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 3439, 2024-05-27T09:51:26.661 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 219280860, 2024-05-27T09:51:26.661 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:26.661 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:26.661 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:26.661 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:26.661 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:26.662 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:26.662 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-05-27T09:51:27.651 INFO:tasks.check_counter:Daemon mds.b mds.exported=8 2024-05-27T09:51:27.651 INFO:tasks.check_counter:Daemon mds.b mds.imported=512 2024-05-27T09:51:27.651 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=983 2024-05-27T09:51:27.651 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-05-27T09:51:27.651 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-05-27T09:51:27.651 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=16 2024-05-27T09:51:27.651 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=10 2024-05-27T09:51:27.652 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=77 2024-05-27T09:51:27.652 DEBUG:tasks.check_counter:Getting stats from e 2024-05-27T09:51:27.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.e perf dump 2024-05-27T09:51:28.080 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:51:28.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:28 smithi044 ceph-mon[31618]: pgmap v511: 97 pgs: 30 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 52 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-27T09:51:28.507 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:28 smithi050 ceph-mon[36371]: pgmap v511: 97 pgs: 30 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 52 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-27T09:51:28.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:28 smithi189 ceph-mon[36671]: pgmap v511: 97 pgs: 30 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 52 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-27T09:51:30.153 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-05-27T09:51:30.153 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:30.153 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 393, 2024-05-27T09:51:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 362, 2024-05-27T09:51:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 2002578, 2024-05-27T09:51:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 210276, 2024-05-27T09:51:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 6, 2024-05-27T09:51:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 1, 2024-05-27T09:51:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.073073200, 2024-05-27T09:51:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.021229444, 2024-05-27T09:51:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.056418065, 2024-05-27T09:51:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.008391511, 2024-05-27T09:51:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 361, 2024-05-27T09:51:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.021329976, 2024-05-27T09:51:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000059085 2024-05-27T09:51:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 2001744, 2024-05-27T09:51:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 209952 2024-05-27T09:51:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 1088, 2024-05-27T09:51:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 1337, 2024-05-27T09:51:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 269554, 2024-05-27T09:51:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 1022257, 2024-05-27T09:51:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 8, 2024-05-27T09:51:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 5, 2024-05-27T09:51:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.117279777, 2024-05-27T09:51:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.041514821, 2024-05-27T09:51:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.025092063, 2024-05-27T09:51:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.037516994, 2024-05-27T09:51:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 1335, 2024-05-27T09:51:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.086716194, 2024-05-27T09:51:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000064955 2024-05-27T09:51:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 3360, 2024-05-27T09:51:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 678176 2024-05-27T09:51:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 2369, 2024-05-27T09:51:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 2371, 2024-05-27T09:51:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 710834, 2024-05-27T09:51:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 753400, 2024-05-27T09:51:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 4, 2024-05-27T09:51:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 3, 2024-05-27T09:51:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.245439189, 2024-05-27T09:51:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.086036872, 2024-05-27T09:51:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.063498184, 2024-05-27T09:51:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.072824607, 2024-05-27T09:51:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 2369, 2024-05-27T09:51:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.101385505, 2024-05-27T09:51:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000042796 2024-05-27T09:51:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-27T09:51:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 0 2024-05-27T09:51:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: "cct": { 2024-05-27T09:51:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: "total_workers": 1, 2024-05-27T09:51:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: "unhealthy_workers": 0 2024-05-27T09:51:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: "finisher-MDSRank": { 2024-05-27T09:51:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: "queue_len": 0, 2024-05-27T09:51:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: "complete_latency": { 2024-05-27T09:51:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 5806, 2024-05-27T09:51:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.170970099, 2024-05-27T09:51:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000029447 2024-05-27T09:51:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: "finisher-PurgeQueue": { 2024-05-27T09:51:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: "queue_len": 0, 2024-05-27T09:51:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: "complete_latency": { 2024-05-27T09:51:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": { 2024-05-27T09:51:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "request": 0, 2024-05-27T09:51:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "reply": 0, 2024-05-27T09:51:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "reply_latency": { 2024-05-27T09:51:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: "slow_reply": 0, 2024-05-27T09:51:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: "forward": 0, 2024-05-27T09:51:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_fetch_complete": 0, 2024-05-27T09:51:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_fetch_keys": 0, 2024-05-27T09:51:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_commit": 0, 2024-05-27T09:51:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_split": 0, 2024-05-27T09:51:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_merge": 0, 2024-05-27T09:51:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes": 0, 2024-05-27T09:51:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_top": 0, 2024-05-27T09:51:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_bottom": 0, 2024-05-27T09:51:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_pin_tail": 0, 2024-05-27T09:51:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_pinned": 0, 2024-05-27T09:51:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_expired": 0, 2024-05-27T09:51:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_with_caps": 0, 2024-05-27T09:51:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": 0, 2024-05-27T09:51:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: "subtrees": 1, 2024-05-27T09:51:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse": 0, 2024-05-27T09:51:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_hit": 0, 2024-05-27T09:51:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_forward": 0, 2024-05-27T09:51:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_discover": 0, 2024-05-27T09:51:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_dir_fetch": 0, 2024-05-27T09:51:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_remote_ino": 0, 2024-05-27T09:51:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_lock": 0, 2024-05-27T09:51:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "load_cent": 0, 2024-05-27T09:51:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "q": 0, 2024-05-27T09:51:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "exported": 0, 2024-05-27T09:51:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "exported_inodes": 0, 2024-05-27T09:51:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "imported": 0, 2024-05-27T09:51:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "imported_inodes": 0, 2024-05-27T09:51:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "openino_dir_fetch": 0, 2024-05-27T09:51:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "openino_backtrace_fetch": 0, 2024-05-27T09:51:30.171 INFO:teuthology.orchestra.run.smithi050.stdout: "openino_peer_discover": 0, 2024-05-27T09:51:30.171 INFO:teuthology.orchestra.run.smithi050.stdout: "root_rfiles": 0, 2024-05-27T09:51:30.171 INFO:teuthology.orchestra.run.smithi050.stdout: "root_rbytes": 0, 2024-05-27T09:51:30.171 INFO:teuthology.orchestra.run.smithi050.stdout: "root_rsnaps": 0, 2024-05-27T09:51:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_backtrace_fetch": 0, 2024-05-27T09:51:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_set_tag": 0, 2024-05-27T09:51:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_backtrace_repaired": 0, 2024-05-27T09:51:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_inotable_repaired": 0, 2024-05-27T09:51:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_dir_inodes": 0, 2024-05-27T09:51:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_dir_base_inodes": 0, 2024-05-27T09:51:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_dirfrag_rstats": 0, 2024-05-27T09:51:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_file_inodes": 0, 2024-05-27T09:51:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_inode_file_caps": 0, 2024-05-27T09:51:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_revoke": 0, 2024-05-27T09:51:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_grant": 0, 2024-05-27T09:51:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_trunc": 0, 2024-05-27T09:51:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-27T09:51:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-27T09:51:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_caps": 0, 2024-05-27T09:51:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_caps_dirty": 0, 2024-05-27T09:51:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_cap_release": 0, 2024-05-27T09:51:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: "process_request_cap_release": 0 2024-05-27T09:51:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache": { 2024-05-27T09:51:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_update": 0, 2024-05-27T09:51:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_update_receipt": 0, 2024-05-27T09:51:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_try_discover": 0, 2024-05-27T09:51:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_send_discover": 0, 2024-05-27T09:51:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_handle_discover": 0, 2024-05-27T09:51:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "num_strays": 0, 2024-05-27T09:51:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "num_strays_delayed": 0, 2024-05-27T09:51:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "num_strays_enqueuing": 0, 2024-05-27T09:51:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_created": 0, 2024-05-27T09:51:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_enqueued": 0, 2024-05-27T09:51:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_reintegrated": 0, 2024-05-27T09:51:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_migrated": 0, 2024-05-27T09:51:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "num_recovering_processing": 0, 2024-05-27T09:51:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "num_recovering_enqueued": 0, 2024-05-27T09:51:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "num_recovering_prioritized": 0, 2024-05-27T09:51:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_started": 0, 2024-05-27T09:51:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_completed": 0, 2024-05-27T09:51:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_quiesce_path": 0, 2024-05-27T09:51:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_quiesce_inode": 0, 2024-05-27T09:51:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_enqueue_scrub": 0, 2024-05-27T09:51:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_exportdir": 0, 2024-05-27T09:51:30.178 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_flush": 0, 2024-05-27T09:51:30.178 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_fragmentdir": 0, 2024-05-27T09:51:30.178 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_fragstats": 0, 2024-05-27T09:51:30.178 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_inodestats": 0 2024-05-27T09:51:30.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.178 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log": { 2024-05-27T09:51:30.178 INFO:teuthology.orchestra.run.smithi050.stdout: "evlrg": 0, 2024-05-27T09:51:30.178 INFO:teuthology.orchestra.run.smithi050.stdout: "evadd": 0, 2024-05-27T09:51:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: "evex": 0, 2024-05-27T09:51:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: "evtrm": 0, 2024-05-27T09:51:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: "ev": 10, 2024-05-27T09:51:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: "evexg": 0, 2024-05-27T09:51:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: "evexd": 0, 2024-05-27T09:51:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: "segadd": 0, 2024-05-27T09:51:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: "segex": 0, 2024-05-27T09:51:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: "segtrm": 0, 2024-05-27T09:51:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "seg": 2, 2024-05-27T09:51:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "segmjr": 2, 2024-05-27T09:51:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "segexg": 0, 2024-05-27T09:51:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "segexd": 0, 2024-05-27T09:51:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "expos": 4194304, 2024-05-27T09:51:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "wrpos": 4197185, 2024-05-27T09:51:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "rdpos": 4197044, 2024-05-27T09:51:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "jlat": { 2024-05-27T09:51:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: "replayed": 10 2024-05-27T09:51:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_mem": { 2024-05-27T09:51:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: "ino": 3, 2024-05-27T09:51:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: "ino+": 3, 2024-05-27T09:51:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "ino-": 0, 2024-05-27T09:51:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "dir": 1, 2024-05-27T09:51:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "dir+": 1, 2024-05-27T09:51:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "dir-": 0, 2024-05-27T09:51:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "dn": 0, 2024-05-27T09:51:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "dn+": 0, 2024-05-27T09:51:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "dn-": 0, 2024-05-27T09:51:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: "cap": 0, 2024-05-27T09:51:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: "cap+": 0, 2024-05-27T09:51:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: "cap-": 0, 2024-05-27T09:51:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: "rss": 121932, 2024-05-27T09:51:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: "heap": 215440 2024-05-27T09:51:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_server": { 2024-05-27T09:51:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "dispatch_client_request": 0, 2024-05-27T09:51:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "dispatch_server_request": 0, 2024-05-27T09:51:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_request": 0, 2024-05-27T09:51:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_session": 0, 2024-05-27T09:51:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_peer_request": 0, 2024-05-27T09:51:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "req_create_latency": { 2024-05-27T09:51:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: "req_getattr_latency": { 2024-05-27T09:51:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: "req_getfilelock_latency": { 2024-05-27T09:51:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: "req_link_latency": { 2024-05-27T09:51:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookup_latency": { 2024-05-27T09:51:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookuphash_latency": { 2024-05-27T09:51:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupino_latency": { 2024-05-27T09:51:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupname_latency": { 2024-05-27T09:51:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupparent_latency": { 2024-05-27T09:51:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupsnap_latency": { 2024-05-27T09:51:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lssnap_latency": { 2024-05-27T09:51:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: "req_mkdir_latency": { 2024-05-27T09:51:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: "req_mknod_latency": { 2024-05-27T09:51:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: "req_mksnap_latency": { 2024-05-27T09:51:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: "req_open_latency": { 2024-05-27T09:51:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: "req_readdir_latency": { 2024-05-27T09:51:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rename_latency": { 2024-05-27T09:51:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: "req_renamesnap_latency": { 2024-05-27T09:51:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: "req_snapdiff_latency": { 2024-05-27T09:51:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rmdir_latency": { 2024-05-27T09:51:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rmsnap_latency": { 2024-05-27T09:51:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rmxattr_latency": { 2024-05-27T09:51:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setattr_latency": { 2024-05-27T09:51:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setdirlayout_latency": { 2024-05-27T09:51:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setfilelock_latency": { 2024-05-27T09:51:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setlayout_latency": { 2024-05-27T09:51:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setxattr_latency": { 2024-05-27T09:51:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "req_symlink_latency": { 2024-05-27T09:51:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "req_unlink_latency": { 2024-05-27T09:51:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "cap_revoke_eviction": 0, 2024-05-27T09:51:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "cap_acquisition_throttle": 0, 2024-05-27T09:51:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "req_getvxattr_latency": { 2024-05-27T09:51:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_sessions": { 2024-05-27T09:51:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "session_count": 2, 2024-05-27T09:51:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "session_add": 5, 2024-05-27T09:51:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "session_remove": 3, 2024-05-27T09:51:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "sessions_open": 4, 2024-05-27T09:51:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "sessions_stale": 0, 2024-05-27T09:51:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "total_load": 0, 2024-05-27T09:51:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "average_load": 0, 2024-05-27T09:51:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "avg_session_uptime": 4309, 2024-05-27T09:51:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "mdthresh_evicted": 0 2024-05-27T09:51:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "mempool": { 2024-05-27T09:51:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_bytes": 0, 2024-05-27T09:51:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_items": 0, 2024-05-27T09:51:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_items": 0, 2024-05-27T09:51:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_bytes": 190923, 2024-05-27T09:51:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_items": 54, 2024-05-27T09:51:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_bytes": 0, 2024-05-27T09:51:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_items": 0, 2024-05-27T09:51:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bytes": 0, 2024-05-27T09:51:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_items": 0, 2024-05-27T09:51:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_bytes": 8448, 2024-05-27T09:51:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_items": 193, 2024-05-27T09:51:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_items": 0, 2024-05-27T09:51:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_bytes": 8512, 2024-05-27T09:51:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_items": 13, 2024-05-27T09:51:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_items": 0, 2024-05-27T09:51:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_items": 0 2024-05-27T09:51:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter": { 2024-05-27T09:51:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "op_active": 0, 2024-05-27T09:51:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "op_laggy": 0, 2024-05-27T09:51:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "op_send": 3452, 2024-05-27T09:51:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "op_send_bytes": 21, 2024-05-27T09:51:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "op_resend": 3, 2024-05-27T09:51:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "op_reply": 3452, 2024-05-27T09:51:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "op_latency": { 2024-05-27T09:51:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 3452, 2024-05-27T09:51:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 4.011898807, 2024-05-27T09:51:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.001162195 2024-05-27T09:51:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "op_inflight": 0, 2024-05-27T09:51:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "oplen_avg": { 2024-05-27T09:51:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 3452, 2024-05-27T09:51:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 3453 2024-05-27T09:51:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "op": 3452, 2024-05-27T09:51:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "op_r": 3452, 2024-05-27T09:51:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "op_w": 0, 2024-05-27T09:51:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "op_rmw": 0, 2024-05-27T09:51:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "op_pg": 0, 2024-05-27T09:51:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_stat": 2260, 2024-05-27T09:51:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_create": 0, 2024-05-27T09:51:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_read": 1190, 2024-05-27T09:51:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_write": 0, 2024-05-27T09:51:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_writefull": 0, 2024-05-27T09:51:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_writesame": 0, 2024-05-27T09:51:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_append": 0, 2024-05-27T09:51:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_zero": 0, 2024-05-27T09:51:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_truncate": 0, 2024-05-27T09:51:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_delete": 0, 2024-05-27T09:51:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_mapext": 0, 2024-05-27T09:51:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_sparse_read": 0, 2024-05-27T09:51:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_clonerange": 0, 2024-05-27T09:51:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_getxattr": 1, 2024-05-27T09:51:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_setxattr": 0, 2024-05-27T09:51:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_cmpxattr": 0, 2024-05-27T09:51:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_rmxattr": 0, 2024-05-27T09:51:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_resetxattrs": 0, 2024-05-27T09:51:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_call": 0, 2024-05-27T09:51:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_watch": 0, 2024-05-27T09:51:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_notify": 0, 2024-05-27T09:51:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_src_cmpxattr": 0, 2024-05-27T09:51:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_pgls": 0, 2024-05-27T09:51:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_pgls_filter": 0, 2024-05-27T09:51:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_other": 0, 2024-05-27T09:51:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_active": 0, 2024-05-27T09:51:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_send": 0, 2024-05-27T09:51:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_resend": 0, 2024-05-27T09:51:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_ping": 0, 2024-05-27T09:51:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "poolop_active": 0, 2024-05-27T09:51:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "poolop_send": 0, 2024-05-27T09:51:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "poolop_resend": 0, 2024-05-27T09:51:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "poolstat_active": 0, 2024-05-27T09:51:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "poolstat_send": 0, 2024-05-27T09:51:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "poolstat_resend": 0, 2024-05-27T09:51:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "statfs_active": 0, 2024-05-27T09:51:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "statfs_send": 0, 2024-05-27T09:51:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "statfs_resend": 0, 2024-05-27T09:51:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "command_active": 0, 2024-05-27T09:51:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "command_send": 0, 2024-05-27T09:51:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "command_resend": 0, 2024-05-27T09:51:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "map_epoch": 104, 2024-05-27T09:51:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "map_full": 0, 2024-05-27T09:51:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "map_inc": 25, 2024-05-27T09:51:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_sessions": 5, 2024-05-27T09:51:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_session_open": 6, 2024-05-27T09:51:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_session_close": 1, 2024-05-27T09:51:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_laggy": 0, 2024-05-27T09:51:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_wr": 0, 2024-05-27T09:51:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_rd": 2, 2024-05-27T09:51:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_del": 0 2024-05-27T09:51:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "oft": { 2024-05-27T09:51:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_objs": 0, 2024-05-27T09:51:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_kv_pairs": 0, 2024-05-27T09:51:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_updates": 0, 2024-05-27T09:51:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_removes": 0 2024-05-27T09:51:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: "osd-slow-ops": { 2024-05-27T09:51:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: "slow_ops_count": 0 2024-05-27T09:51:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "purge_queue": { 2024-05-27T09:51:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing_ops": 0, 2024-05-27T09:51:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing_ops_high_water": 0, 2024-05-27T09:51:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing": 0, 2024-05-27T09:51:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing_high_water": 0, 2024-05-27T09:51:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executed_ops": 0, 2024-05-27T09:51:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executed": 0, 2024-05-27T09:51:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_item_in_journal": 0 2024-05-27T09:51:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 104857600, 2024-05-27T09:51:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 3846, 2024-05-27T09:51:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 2613774, 2024-05-27T09:51:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 3846, 2024-05-27T09:51:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-05-27T09:51:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-05-27T09:51:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 3846, 2024-05-27T09:51:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 2613774, 2024-05-27T09:51:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-objecter_bytes": { 2024-05-27T09:51:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 104857600, 2024-05-27T09:51:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 0, 2024-05-27T09:51:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 0, 2024-05-27T09:51:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 3452, 2024-05-27T09:51:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 2886, 2024-05-27T09:51:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 3452, 2024-05-27T09:51:30.231 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 2886, 2024-05-27T09:51:30.231 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:30.231 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.231 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.231 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:30.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.231 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-objecter_ops": { 2024-05-27T09:51:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 1024, 2024-05-27T09:51:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 0, 2024-05-27T09:51:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 0, 2024-05-27T09:51:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 3452, 2024-05-27T09:51:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 3452, 2024-05-27T09:51:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 3452, 2024-05-27T09:51:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 3452, 2024-05-27T09:51:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-write_buf_throttle": { 2024-05-27T09:51:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 3758096384, 2024-05-27T09:51:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 0, 2024-05-27T09:51:30.235 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 0, 2024-05-27T09:51:30.235 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:30.235 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:30.235 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-05-27T09:51:30.235 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-05-27T09:51:30.235 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 0, 2024-05-27T09:51:30.235 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 0, 2024-05-27T09:51:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-write_buf_throttle-0x55a21577ec30": { 2024-05-27T09:51:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 3758096384, 2024-05-27T09:51:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 0, 2024-05-27T09:51:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 0, 2024-05-27T09:51:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-05-27T09:51:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-05-27T09:51:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 0, 2024-05-27T09:51:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 0, 2024-05-27T09:51:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:30.239 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-05-27T09:51:30.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:30 smithi044 ceph-mon[31618]: pgmap v512: 97 pgs: 30 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 52 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:30.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:30 smithi050 ceph-mon[36371]: pgmap v512: 97 pgs: 30 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 52 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:30.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:30 smithi189 ceph-mon[36671]: pgmap v512: 97 pgs: 30 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 52 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:31.086 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-05-27T09:51:31.086 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-05-27T09:51:31.086 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-05-27T09:51:31.086 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-05-27T09:51:31.087 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-05-27T09:51:31.087 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-05-27T09:51:31.087 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-05-27T09:51:31.087 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-05-27T09:51:31.087 DEBUG:tasks.check_counter:Getting stats from h 2024-05-27T09:51:31.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.h perf dump 2024-05-27T09:51:31.431 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:51:32.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:32 smithi044 ceph-mon[31618]: pgmap v513: 97 pgs: 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 80 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:51:32.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:32 smithi050 ceph-mon[36371]: pgmap v513: 97 pgs: 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 80 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:51:32.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:32 smithi189 ceph-mon[36671]: pgmap v513: 97 pgs: 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 80 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-27T09:51:33.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:33 smithi050 ceph-mon[36371]: pgmap v514: 97 pgs: 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 80 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:51:33.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:33 smithi189 ceph-mon[36671]: pgmap v514: 97 pgs: 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 80 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:51:33.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:33 smithi044 ceph-mon[31618]: pgmap v514: 97 pgs: 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 80 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:51:34.043 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-05-27T09:51:34.043 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:34.043 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 18796, 2024-05-27T09:51:34.043 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 18808, 2024-05-27T09:51:34.043 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 254993736, 2024-05-27T09:51:34.044 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 67749948, 2024-05-27T09:51:34.044 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 11, 2024-05-27T09:51:34.044 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 4, 2024-05-27T09:51:34.044 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 1.846391422, 2024-05-27T09:51:34.044 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.619473183, 2024-05-27T09:51:34.044 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.493735097, 2024-05-27T09:51:34.044 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.608102218, 2024-05-27T09:51:34.044 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:34.045 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 18806, 2024-05-27T09:51:34.045 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 6.373108047, 2024-05-27T09:51:34.045 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000338886 2024-05-27T09:51:34.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.045 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:34.045 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 18, 2024-05-27T09:51:34.045 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000005017, 2024-05-27T09:51:34.045 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000278 2024-05-27T09:51:34.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.046 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 720, 2024-05-27T09:51:34.046 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 3648 2024-05-27T09:51:34.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.046 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:34.046 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 21726, 2024-05-27T09:51:34.046 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 28236, 2024-05-27T09:51:34.046 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 259683063, 2024-05-27T09:51:34.047 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 144814010, 2024-05-27T09:51:34.047 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 17, 2024-05-27T09:51:34.047 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 6, 2024-05-27T09:51:34.047 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 2.627905441, 2024-05-27T09:51:34.047 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 1.114485112, 2024-05-27T09:51:34.047 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 1.032064689, 2024-05-27T09:51:34.047 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.403125214, 2024-05-27T09:51:34.047 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:34.047 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 28236, 2024-05-27T09:51:34.048 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 4.795411785, 2024-05-27T09:51:34.048 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000169833 2024-05-27T09:51:34.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.048 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:34.048 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 27889, 2024-05-27T09:51:34.048 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.008074597, 2024-05-27T09:51:34.048 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000289 2024-05-27T09:51:34.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.049 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 2006448, 2024-05-27T09:51:34.049 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 214592 2024-05-27T09:51:34.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.049 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:34.049 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 44715, 2024-05-27T09:51:34.049 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 77358, 2024-05-27T09:51:34.050 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 140023637, 2024-05-27T09:51:34.050 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 122504243, 2024-05-27T09:51:34.050 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 12, 2024-05-27T09:51:34.050 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 5, 2024-05-27T09:51:34.050 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 3.360633673, 2024-05-27T09:51:34.050 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 1.550676012, 2024-05-27T09:51:34.050 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 1.619473030, 2024-05-27T09:51:34.051 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.371091477, 2024-05-27T09:51:34.051 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:34.051 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 77355, 2024-05-27T09:51:34.051 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 6.052019363, 2024-05-27T09:51:34.051 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000078236 2024-05-27T09:51:34.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.051 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:34.051 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 50290, 2024-05-27T09:51:34.052 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.014855846, 2024-05-27T09:51:34.052 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000295 2024-05-27T09:51:34.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.052 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-27T09:51:34.052 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 681264 2024-05-27T09:51:34.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.053 INFO:teuthology.orchestra.run.smithi050.stdout: "cct": { 2024-05-27T09:51:34.053 INFO:teuthology.orchestra.run.smithi050.stdout: "total_workers": 1, 2024-05-27T09:51:34.053 INFO:teuthology.orchestra.run.smithi050.stdout: "unhealthy_workers": 0 2024-05-27T09:51:34.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.053 INFO:teuthology.orchestra.run.smithi050.stdout: "finisher-MDSRank": { 2024-05-27T09:51:34.053 INFO:teuthology.orchestra.run.smithi050.stdout: "queue_len": 0, 2024-05-27T09:51:34.053 INFO:teuthology.orchestra.run.smithi050.stdout: "complete_latency": { 2024-05-27T09:51:34.053 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 17362, 2024-05-27T09:51:34.054 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 46.783613443, 2024-05-27T09:51:34.054 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.002694598 2024-05-27T09:51:34.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:34.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.054 INFO:teuthology.orchestra.run.smithi050.stdout: "finisher-PurgeQueue": { 2024-05-27T09:51:34.055 INFO:teuthology.orchestra.run.smithi050.stdout: "queue_len": 0, 2024-05-27T09:51:34.055 INFO:teuthology.orchestra.run.smithi050.stdout: "complete_latency": { 2024-05-27T09:51:34.055 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 25878, 2024-05-27T09:51:34.055 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 16.294484064, 2024-05-27T09:51:34.055 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000629665 2024-05-27T09:51:34.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:34.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.056 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": { 2024-05-27T09:51:34.056 INFO:teuthology.orchestra.run.smithi050.stdout: "request": 32154, 2024-05-27T09:51:34.056 INFO:teuthology.orchestra.run.smithi050.stdout: "reply": 32147, 2024-05-27T09:51:34.056 INFO:teuthology.orchestra.run.smithi050.stdout: "reply_latency": { 2024-05-27T09:51:34.056 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 32147, 2024-05-27T09:51:34.057 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 250.939991791, 2024-05-27T09:51:34.057 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.007806015 2024-05-27T09:51:34.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.057 INFO:teuthology.orchestra.run.smithi050.stdout: "slow_reply": 0, 2024-05-27T09:51:34.057 INFO:teuthology.orchestra.run.smithi050.stdout: "forward": 3, 2024-05-27T09:51:34.057 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_fetch_complete": 2072, 2024-05-27T09:51:34.058 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_fetch_keys": 0, 2024-05-27T09:51:34.058 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_commit": 1662, 2024-05-27T09:51:34.058 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_split": 0, 2024-05-27T09:51:34.058 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_merge": 9, 2024-05-27T09:51:34.058 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes": 87159, 2024-05-27T09:51:34.058 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_top": 47116, 2024-05-27T09:51:34.059 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_bottom": 40040, 2024-05-27T09:51:34.059 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_pin_tail": 3, 2024-05-27T09:51:34.059 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_pinned": 19849, 2024-05-27T09:51:34.059 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_expired": 19527, 2024-05-27T09:51:34.059 INFO:teuthology.orchestra.run.smithi050.stdout: "inodes_with_caps": 1, 2024-05-27T09:51:34.060 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": 1, 2024-05-27T09:51:34.060 INFO:teuthology.orchestra.run.smithi050.stdout: "subtrees": 3, 2024-05-27T09:51:34.060 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse": 53817, 2024-05-27T09:51:34.060 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_hit": 47711, 2024-05-27T09:51:34.060 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_forward": 1, 2024-05-27T09:51:34.060 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_discover": 0, 2024-05-27T09:51:34.061 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_dir_fetch": 548, 2024-05-27T09:51:34.061 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_remote_ino": 0, 2024-05-27T09:51:34.061 INFO:teuthology.orchestra.run.smithi050.stdout: "traverse_lock": 0, 2024-05-27T09:51:34.061 INFO:teuthology.orchestra.run.smithi050.stdout: "load_cent": 0, 2024-05-27T09:51:34.061 INFO:teuthology.orchestra.run.smithi050.stdout: "q": 0, 2024-05-27T09:51:34.061 INFO:teuthology.orchestra.run.smithi050.stdout: "exported": 288, 2024-05-27T09:51:34.062 INFO:teuthology.orchestra.run.smithi050.stdout: "exported_inodes": 13952, 2024-05-27T09:51:34.062 INFO:teuthology.orchestra.run.smithi050.stdout: "imported": 0, 2024-05-27T09:51:34.062 INFO:teuthology.orchestra.run.smithi050.stdout: "imported_inodes": 0, 2024-05-27T09:51:34.062 INFO:teuthology.orchestra.run.smithi050.stdout: "openino_dir_fetch": 0, 2024-05-27T09:51:34.062 INFO:teuthology.orchestra.run.smithi050.stdout: "openino_backtrace_fetch": 13281, 2024-05-27T09:51:34.063 INFO:teuthology.orchestra.run.smithi050.stdout: "openino_peer_discover": 0, 2024-05-27T09:51:34.063 INFO:teuthology.orchestra.run.smithi050.stdout: "root_rfiles": 4, 2024-05-27T09:51:34.063 INFO:teuthology.orchestra.run.smithi050.stdout: "root_rbytes": 234, 2024-05-27T09:51:34.063 INFO:teuthology.orchestra.run.smithi050.stdout: "root_rsnaps": 6, 2024-05-27T09:51:34.063 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_backtrace_fetch": 13281, 2024-05-27T09:51:34.063 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_set_tag": 0, 2024-05-27T09:51:34.063 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_backtrace_repaired": 0, 2024-05-27T09:51:34.063 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_inotable_repaired": 0, 2024-05-27T09:51:34.064 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_dir_inodes": 6, 2024-05-27T09:51:34.064 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_dir_base_inodes": 0, 2024-05-27T09:51:34.064 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_dirfrag_rstats": 6, 2024-05-27T09:51:34.064 INFO:teuthology.orchestra.run.smithi050.stdout: "scrub_file_inodes": 13275, 2024-05-27T09:51:34.064 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_inode_file_caps": 0, 2024-05-27T09:51:34.064 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_revoke": 26, 2024-05-27T09:51:34.064 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_grant": 15590, 2024-05-27T09:51:34.065 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_trunc": 0, 2024-05-27T09:51:34.065 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-27T09:51:34.065 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-27T09:51:34.065 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_caps": 26, 2024-05-27T09:51:34.065 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_caps_dirty": 0, 2024-05-27T09:51:34.066 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_cap_release": 115, 2024-05-27T09:51:34.066 INFO:teuthology.orchestra.run.smithi050.stdout: "process_request_cap_release": 22031 2024-05-27T09:51:34.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.066 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache": { 2024-05-27T09:51:34.066 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_update": 278, 2024-05-27T09:51:34.067 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_update_receipt": 4, 2024-05-27T09:51:34.067 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_try_discover": 2, 2024-05-27T09:51:34.067 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_send_discover": 2, 2024-05-27T09:51:34.067 INFO:teuthology.orchestra.run.smithi050.stdout: "dir_handle_discover": 289, 2024-05-27T09:51:34.067 INFO:teuthology.orchestra.run.smithi050.stdout: "num_strays": 3, 2024-05-27T09:51:34.067 INFO:teuthology.orchestra.run.smithi050.stdout: "num_strays_delayed": 0, 2024-05-27T09:51:34.067 INFO:teuthology.orchestra.run.smithi050.stdout: "num_strays_enqueuing": 0, 2024-05-27T09:51:34.068 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_created": 15568, 2024-05-27T09:51:34.068 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_enqueued": 15583, 2024-05-27T09:51:34.068 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_reintegrated": 0, 2024-05-27T09:51:34.068 INFO:teuthology.orchestra.run.smithi050.stdout: "strays_migrated": 0, 2024-05-27T09:51:34.068 INFO:teuthology.orchestra.run.smithi050.stdout: "num_recovering_processing": 0, 2024-05-27T09:51:34.068 INFO:teuthology.orchestra.run.smithi050.stdout: "num_recovering_enqueued": 0, 2024-05-27T09:51:34.068 INFO:teuthology.orchestra.run.smithi050.stdout: "num_recovering_prioritized": 0, 2024-05-27T09:51:34.068 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_started": 0, 2024-05-27T09:51:34.068 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_completed": 0, 2024-05-27T09:51:34.069 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_quiesce_path": 3, 2024-05-27T09:51:34.069 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_quiesce_inode": 12982, 2024-05-27T09:51:34.069 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_enqueue_scrub": 0, 2024-05-27T09:51:34.069 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_exportdir": 312, 2024-05-27T09:51:34.069 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_flush": 0, 2024-05-27T09:51:34.069 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_fragmentdir": 9, 2024-05-27T09:51:34.069 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_fragstats": 0, 2024-05-27T09:51:34.069 INFO:teuthology.orchestra.run.smithi050.stdout: "ireq_inodestats": 0 2024-05-27T09:51:34.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.070 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log": { 2024-05-27T09:51:34.070 INFO:teuthology.orchestra.run.smithi050.stdout: "evlrg": 2, 2024-05-27T09:51:34.070 INFO:teuthology.orchestra.run.smithi050.stdout: "evadd": 36193, 2024-05-27T09:51:34.070 INFO:teuthology.orchestra.run.smithi050.stdout: "evex": 46452, 2024-05-27T09:51:34.070 INFO:teuthology.orchestra.run.smithi050.stdout: "evtrm": 36870, 2024-05-27T09:51:34.070 INFO:teuthology.orchestra.run.smithi050.stdout: "ev": 173248, 2024-05-27T09:51:34.070 INFO:teuthology.orchestra.run.smithi050.stdout: "evexg": 0, 2024-05-27T09:51:34.070 INFO:teuthology.orchestra.run.smithi050.stdout: "evexd": 9582, 2024-05-27T09:51:34.071 INFO:teuthology.orchestra.run.smithi050.stdout: "segadd": 53, 2024-05-27T09:51:34.071 INFO:teuthology.orchestra.run.smithi050.stdout: "segex": 66, 2024-05-27T09:51:34.071 INFO:teuthology.orchestra.run.smithi050.stdout: "segtrm": 54, 2024-05-27T09:51:34.071 INFO:teuthology.orchestra.run.smithi050.stdout: "seg": 140, 2024-05-27T09:51:34.071 INFO:teuthology.orchestra.run.smithi050.stdout: "segmjr": 9, 2024-05-27T09:51:34.071 INFO:teuthology.orchestra.run.smithi050.stdout: "segexg": 0, 2024-05-27T09:51:34.071 INFO:teuthology.orchestra.run.smithi050.stdout: "segexd": 12, 2024-05-27T09:51:34.071 INFO:teuthology.orchestra.run.smithi050.stdout: "expos": 315509816, 2024-05-27T09:51:34.072 INFO:teuthology.orchestra.run.smithi050.stdout: "wrpos": 668405433, 2024-05-27T09:51:34.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rdpos": 526221318, 2024-05-27T09:51:34.072 INFO:teuthology.orchestra.run.smithi050.stdout: "jlat": { 2024-05-27T09:51:34.072 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 685, 2024-05-27T09:51:34.072 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 52.303354414, 2024-05-27T09:51:34.072 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.076355261 2024-05-27T09:51:34.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.072 INFO:teuthology.orchestra.run.smithi050.stdout: "replayed": 173925 2024-05-27T09:51:34.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.073 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_mem": { 2024-05-27T09:51:34.073 INFO:teuthology.orchestra.run.smithi050.stdout: "ino": 49659, 2024-05-27T09:51:34.073 INFO:teuthology.orchestra.run.smithi050.stdout: "ino+": 109776, 2024-05-27T09:51:34.073 INFO:teuthology.orchestra.run.smithi050.stdout: "ino-": 60117, 2024-05-27T09:51:34.073 INFO:teuthology.orchestra.run.smithi050.stdout: "dir": 1564, 2024-05-27T09:51:34.073 INFO:teuthology.orchestra.run.smithi050.stdout: "dir+": 2516, 2024-05-27T09:51:34.073 INFO:teuthology.orchestra.run.smithi050.stdout: "dir-": 952, 2024-05-27T09:51:34.073 INFO:teuthology.orchestra.run.smithi050.stdout: "dn": 87159, 2024-05-27T09:51:34.073 INFO:teuthology.orchestra.run.smithi050.stdout: "dn+": 149781, 2024-05-27T09:51:34.074 INFO:teuthology.orchestra.run.smithi050.stdout: "dn-": 62622, 2024-05-27T09:51:34.074 INFO:teuthology.orchestra.run.smithi050.stdout: "cap": 1, 2024-05-27T09:51:34.074 INFO:teuthology.orchestra.run.smithi050.stdout: "cap+": 15576, 2024-05-27T09:51:34.074 INFO:teuthology.orchestra.run.smithi050.stdout: "cap-": 15575, 2024-05-27T09:51:34.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rss": 420352, 2024-05-27T09:51:34.074 INFO:teuthology.orchestra.run.smithi050.stdout: "heap": 240016 2024-05-27T09:51:34.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.074 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_server": { 2024-05-27T09:51:34.075 INFO:teuthology.orchestra.run.smithi050.stdout: "dispatch_client_request": 39490, 2024-05-27T09:51:34.075 INFO:teuthology.orchestra.run.smithi050.stdout: "dispatch_server_request": 24, 2024-05-27T09:51:34.075 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_request": 32154, 2024-05-27T09:51:34.075 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_client_session": 6, 2024-05-27T09:51:34.075 INFO:teuthology.orchestra.run.smithi050.stdout: "handle_peer_request": 5838, 2024-05-27T09:51:34.075 INFO:teuthology.orchestra.run.smithi050.stdout: "req_create_latency": { 2024-05-27T09:51:34.075 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.075 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.076 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.076 INFO:teuthology.orchestra.run.smithi050.stdout: "req_getattr_latency": { 2024-05-27T09:51:34.076 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.076 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.076 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.076 INFO:teuthology.orchestra.run.smithi050.stdout: "req_getfilelock_latency": { 2024-05-27T09:51:34.076 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.076 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.077 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.077 INFO:teuthology.orchestra.run.smithi050.stdout: "req_link_latency": { 2024-05-27T09:51:34.077 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.077 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.077 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.077 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookup_latency": { 2024-05-27T09:51:34.078 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 15554, 2024-05-27T09:51:34.078 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 72.934702094, 2024-05-27T09:51:34.078 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.004689128 2024-05-27T09:51:34.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.078 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookuphash_latency": { 2024-05-27T09:51:34.078 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.078 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.078 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.079 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupino_latency": { 2024-05-27T09:51:34.079 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.079 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.079 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.079 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupname_latency": { 2024-05-27T09:51:34.079 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.079 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.080 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.080 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupparent_latency": { 2024-05-27T09:51:34.080 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.080 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.080 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.080 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lookupsnap_latency": { 2024-05-27T09:51:34.081 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.081 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.081 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.082 INFO:teuthology.orchestra.run.smithi050.stdout: "req_lssnap_latency": { 2024-05-27T09:51:34.082 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.082 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.082 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.082 INFO:teuthology.orchestra.run.smithi050.stdout: "req_mkdir_latency": { 2024-05-27T09:51:34.082 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.083 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.083 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.083 INFO:teuthology.orchestra.run.smithi050.stdout: "req_mknod_latency": { 2024-05-27T09:51:34.083 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.083 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.083 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.083 INFO:teuthology.orchestra.run.smithi050.stdout: "req_mksnap_latency": { 2024-05-27T09:51:34.084 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.084 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.084 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.084 INFO:teuthology.orchestra.run.smithi050.stdout: "req_open_latency": { 2024-05-27T09:51:34.084 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.084 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.084 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.085 INFO:teuthology.orchestra.run.smithi050.stdout: "req_readdir_latency": { 2024-05-27T09:51:34.085 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 1025, 2024-05-27T09:51:34.085 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 6.845394931, 2024-05-27T09:51:34.085 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.006678434 2024-05-27T09:51:34.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.085 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rename_latency": { 2024-05-27T09:51:34.085 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.085 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.085 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.086 INFO:teuthology.orchestra.run.smithi050.stdout: "req_renamesnap_latency": { 2024-05-27T09:51:34.086 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.086 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.086 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.086 INFO:teuthology.orchestra.run.smithi050.stdout: "req_snapdiff_latency": { 2024-05-27T09:51:34.086 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.087 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.087 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.087 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rmdir_latency": { 2024-05-27T09:51:34.087 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 3, 2024-05-27T09:51:34.087 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.096477290, 2024-05-27T09:51:34.087 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.032159096 2024-05-27T09:51:34.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.088 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rmsnap_latency": { 2024-05-27T09:51:34.088 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.088 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.088 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.088 INFO:teuthology.orchestra.run.smithi050.stdout: "req_rmxattr_latency": { 2024-05-27T09:51:34.088 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.088 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.089 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.089 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setattr_latency": { 2024-05-27T09:51:34.089 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.089 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.089 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.089 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setdirlayout_latency": { 2024-05-27T09:51:34.089 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.090 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.090 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.090 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setfilelock_latency": { 2024-05-27T09:51:34.090 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.090 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.090 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.091 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setlayout_latency": { 2024-05-27T09:51:34.091 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.091 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.091 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.091 INFO:teuthology.orchestra.run.smithi050.stdout: "req_setxattr_latency": { 2024-05-27T09:51:34.091 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.091 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.092 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.092 INFO:teuthology.orchestra.run.smithi050.stdout: "req_symlink_latency": { 2024-05-27T09:51:34.092 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.092 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.092 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.092 INFO:teuthology.orchestra.run.smithi050.stdout: "req_unlink_latency": { 2024-05-27T09:51:34.093 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 15565, 2024-05-27T09:51:34.093 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 171.063417476, 2024-05-27T09:51:34.093 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.010990261 2024-05-27T09:51:34.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.093 INFO:teuthology.orchestra.run.smithi050.stdout: "cap_revoke_eviction": 0, 2024-05-27T09:51:34.093 INFO:teuthology.orchestra.run.smithi050.stdout: "cap_acquisition_throttle": 0, 2024-05-27T09:51:34.093 INFO:teuthology.orchestra.run.smithi050.stdout: "req_getvxattr_latency": { 2024-05-27T09:51:34.093 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.093 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.094 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.094 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:34.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.094 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_sessions": { 2024-05-27T09:51:34.094 INFO:teuthology.orchestra.run.smithi050.stdout: "session_count": 2, 2024-05-27T09:51:34.094 INFO:teuthology.orchestra.run.smithi050.stdout: "session_add": 4, 2024-05-27T09:51:34.094 INFO:teuthology.orchestra.run.smithi050.stdout: "session_remove": 2, 2024-05-27T09:51:34.095 INFO:teuthology.orchestra.run.smithi050.stdout: "sessions_open": 3, 2024-05-27T09:51:34.095 INFO:teuthology.orchestra.run.smithi050.stdout: "sessions_stale": 0, 2024-05-27T09:51:34.095 INFO:teuthology.orchestra.run.smithi050.stdout: "total_load": 14508, 2024-05-27T09:51:34.095 INFO:teuthology.orchestra.run.smithi050.stdout: "average_load": 7254, 2024-05-27T09:51:34.095 INFO:teuthology.orchestra.run.smithi050.stdout: "avg_session_uptime": 3479, 2024-05-27T09:51:34.095 INFO:teuthology.orchestra.run.smithi050.stdout: "mdthresh_evicted": 0 2024-05-27T09:51:34.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.095 INFO:teuthology.orchestra.run.smithi050.stdout: "mempool": { 2024-05-27T09:51:34.095 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_bytes": 0, 2024-05-27T09:51:34.096 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_items": 0, 2024-05-27T09:51:34.096 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:34.096 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:34.096 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:34.096 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:34.096 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:34.096 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:34.096 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:34.096 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:34.097 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:34.097 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:34.097 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:34.097 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:34.097 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:34.097 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:34.097 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:34.097 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:34.097 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:34.098 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:34.098 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:34.098 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:34.098 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:34.098 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:34.098 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:34.098 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:34.098 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:34.098 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:34.099 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:34.099 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:34.099 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:34.099 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_items": 0, 2024-05-27T09:51:34.099 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:34.099 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:34.099 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:34.099 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:34.100 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_bytes": 2167405, 2024-05-27T09:51:34.100 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_items": 49771, 2024-05-27T09:51:34.100 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_bytes": 0, 2024-05-27T09:51:34.100 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_items": 0, 2024-05-27T09:51:34.100 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bytes": 0, 2024-05-27T09:51:34.100 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_items": 0, 2024-05-27T09:51:34.100 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:34.100 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:34.101 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:34.101 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:34.101 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_bytes": 8448, 2024-05-27T09:51:34.101 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_items": 193, 2024-05-27T09:51:34.101 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:34.101 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:34.101 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:34.101 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_items": 0, 2024-05-27T09:51:34.102 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_bytes": 178462176, 2024-05-27T09:51:34.102 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_items": 1545816, 2024-05-27T09:51:34.102 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:34.102 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_items": 0, 2024-05-27T09:51:34.102 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:34.102 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_items": 0 2024-05-27T09:51:34.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.102 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter": { 2024-05-27T09:51:34.103 INFO:teuthology.orchestra.run.smithi050.stdout: "op_active": 0, 2024-05-27T09:51:34.103 INFO:teuthology.orchestra.run.smithi050.stdout: "op_laggy": 0, 2024-05-27T09:51:34.103 INFO:teuthology.orchestra.run.smithi050.stdout: "op_send": 40084, 2024-05-27T09:51:34.103 INFO:teuthology.orchestra.run.smithi050.stdout: "op_send_bytes": 168109323, 2024-05-27T09:51:34.103 INFO:teuthology.orchestra.run.smithi050.stdout: "op_resend": 3, 2024-05-27T09:51:34.103 INFO:teuthology.orchestra.run.smithi050.stdout: "op_reply": 40084, 2024-05-27T09:51:34.103 INFO:teuthology.orchestra.run.smithi050.stdout: "op_latency": { 2024-05-27T09:51:34.103 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 40084, 2024-05-27T09:51:34.103 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 644.884737606, 2024-05-27T09:51:34.104 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.016088332 2024-05-27T09:51:34.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.104 INFO:teuthology.orchestra.run.smithi050.stdout: "op_inflight": 0, 2024-05-27T09:51:34.104 INFO:teuthology.orchestra.run.smithi050.stdout: "oplen_avg": { 2024-05-27T09:51:34.104 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 40084, 2024-05-27T09:51:34.104 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 50217 2024-05-27T09:51:34.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.104 INFO:teuthology.orchestra.run.smithi050.stdout: "op": 40084, 2024-05-27T09:51:34.105 INFO:teuthology.orchestra.run.smithi050.stdout: "op_r": 19270, 2024-05-27T09:51:34.105 INFO:teuthology.orchestra.run.smithi050.stdout: "op_w": 20814, 2024-05-27T09:51:34.105 INFO:teuthology.orchestra.run.smithi050.stdout: "op_rmw": 0, 2024-05-27T09:51:34.105 INFO:teuthology.orchestra.run.smithi050.stdout: "op_pg": 0, 2024-05-27T09:51:34.105 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_stat": 2892, 2024-05-27T09:51:34.105 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_create": 2360, 2024-05-27T09:51:34.105 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_read": 1484, 2024-05-27T09:51:34.105 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_write": 901, 2024-05-27T09:51:34.105 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_writefull": 78, 2024-05-27T09:51:34.106 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_writesame": 0, 2024-05-27T09:51:34.106 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_append": 0, 2024-05-27T09:51:34.106 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_zero": 2, 2024-05-27T09:51:34.106 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_truncate": 0, 2024-05-27T09:51:34.106 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_delete": 15726, 2024-05-27T09:51:34.106 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_mapext": 0, 2024-05-27T09:51:34.106 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_sparse_read": 0, 2024-05-27T09:51:34.106 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_clonerange": 0, 2024-05-27T09:51:34.106 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_getxattr": 13283, 2024-05-27T09:51:34.107 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_setxattr": 4720, 2024-05-27T09:51:34.107 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_cmpxattr": 0, 2024-05-27T09:51:34.107 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_rmxattr": 0, 2024-05-27T09:51:34.107 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_resetxattrs": 0, 2024-05-27T09:51:34.107 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_call": 0, 2024-05-27T09:51:34.107 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_watch": 0, 2024-05-27T09:51:34.107 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_notify": 0, 2024-05-27T09:51:34.107 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_src_cmpxattr": 0, 2024-05-27T09:51:34.107 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_pgls": 0, 2024-05-27T09:51:34.108 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_pgls_filter": 0, 2024-05-27T09:51:34.108 INFO:teuthology.orchestra.run.smithi050.stdout: "osdop_other": 0, 2024-05-27T09:51:34.108 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_active": 0, 2024-05-27T09:51:34.108 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_send": 0, 2024-05-27T09:51:34.108 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_resend": 0, 2024-05-27T09:51:34.108 INFO:teuthology.orchestra.run.smithi050.stdout: "linger_ping": 0, 2024-05-27T09:51:34.108 INFO:teuthology.orchestra.run.smithi050.stdout: "poolop_active": 0, 2024-05-27T09:51:34.108 INFO:teuthology.orchestra.run.smithi050.stdout: "poolop_send": 0, 2024-05-27T09:51:34.108 INFO:teuthology.orchestra.run.smithi050.stdout: "poolop_resend": 0, 2024-05-27T09:51:34.109 INFO:teuthology.orchestra.run.smithi050.stdout: "poolstat_active": 0, 2024-05-27T09:51:34.109 INFO:teuthology.orchestra.run.smithi050.stdout: "poolstat_send": 0, 2024-05-27T09:51:34.109 INFO:teuthology.orchestra.run.smithi050.stdout: "poolstat_resend": 0, 2024-05-27T09:51:34.109 INFO:teuthology.orchestra.run.smithi050.stdout: "statfs_active": 0, 2024-05-27T09:51:34.109 INFO:teuthology.orchestra.run.smithi050.stdout: "statfs_send": 0, 2024-05-27T09:51:34.109 INFO:teuthology.orchestra.run.smithi050.stdout: "statfs_resend": 0, 2024-05-27T09:51:34.109 INFO:teuthology.orchestra.run.smithi050.stdout: "command_active": 0, 2024-05-27T09:51:34.109 INFO:teuthology.orchestra.run.smithi050.stdout: "command_send": 0, 2024-05-27T09:51:34.109 INFO:teuthology.orchestra.run.smithi050.stdout: "command_resend": 0, 2024-05-27T09:51:34.110 INFO:teuthology.orchestra.run.smithi050.stdout: "map_epoch": 104, 2024-05-27T09:51:34.110 INFO:teuthology.orchestra.run.smithi050.stdout: "map_full": 0, 2024-05-27T09:51:34.110 INFO:teuthology.orchestra.run.smithi050.stdout: "map_inc": 25, 2024-05-27T09:51:34.110 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_sessions": 12, 2024-05-27T09:51:34.110 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_session_open": 12, 2024-05-27T09:51:34.110 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_session_close": 0, 2024-05-27T09:51:34.110 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_laggy": 0, 2024-05-27T09:51:34.110 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_wr": 3472, 2024-05-27T09:51:34.111 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_rd": 4148, 2024-05-27T09:51:34.111 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_del": 1151 2024-05-27T09:51:34.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.111 INFO:teuthology.orchestra.run.smithi050.stdout: "oft": { 2024-05-27T09:51:34.111 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_objs": 0, 2024-05-27T09:51:34.111 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_kv_pairs": 0, 2024-05-27T09:51:34.111 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_updates": 0, 2024-05-27T09:51:34.111 INFO:teuthology.orchestra.run.smithi050.stdout: "omap_total_removes": 0 2024-05-27T09:51:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: "osd-slow-ops": { 2024-05-27T09:51:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: "slow_ops_count": 12665 2024-05-27T09:51:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: "purge_queue": { 2024-05-27T09:51:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing_ops": 0, 2024-05-27T09:51:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing_ops_high_water": 10, 2024-05-27T09:51:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing": 0, 2024-05-27T09:51:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executing_high_water": 10, 2024-05-27T09:51:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executed_ops": 15583, 2024-05-27T09:51:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_executed": 15583, 2024-05-27T09:51:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "pq_item_in_journal": 0 2024-05-27T09:51:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 104857600, 2024-05-27T09:51:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 85233, 2024-05-27T09:51:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 647522518, 2024-05-27T09:51:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 85233, 2024-05-27T09:51:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-05-27T09:51:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-05-27T09:51:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 85233, 2024-05-27T09:51:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 647522518, 2024-05-27T09:51:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-objecter_bytes": { 2024-05-27T09:51:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 104857600, 2024-05-27T09:51:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 0, 2024-05-27T09:51:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 0, 2024-05-27T09:51:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 40084, 2024-05-27T09:51:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 692416163, 2024-05-27T09:51:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 40084, 2024-05-27T09:51:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 692416163, 2024-05-27T09:51:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-objecter_ops": { 2024-05-27T09:51:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 1024, 2024-05-27T09:51:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 0, 2024-05-27T09:51:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 0, 2024-05-27T09:51:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 40084, 2024-05-27T09:51:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 40084, 2024-05-27T09:51:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 40084, 2024-05-27T09:51:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 40084, 2024-05-27T09:51:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-write_buf_throttle": { 2024-05-27T09:51:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 3758096384, 2024-05-27T09:51:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 15583, 2024-05-27T09:51:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 2228227, 2024-05-27T09:51:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 15583, 2024-05-27T09:51:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-05-27T09:51:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-05-27T09:51:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 216, 2024-05-27T09:51:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 2228227, 2024-05-27T09:51:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-write_buf_throttle-0x565415a2a9c0": { 2024-05-27T09:51:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 3758096384, 2024-05-27T09:51:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 36193, 2024-05-27T09:51:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 142183516, 2024-05-27T09:51:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 36193, 2024-05-27T09:51:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-05-27T09:51:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-05-27T09:51:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 685, 2024-05-27T09:51:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 142183516, 2024-05-27T09:51:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:34.123 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-05-27T09:51:34.930 INFO:tasks.check_counter:Daemon mds.h mds.exported=288 2024-05-27T09:51:34.931 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-05-27T09:51:34.931 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=278 2024-05-27T09:51:34.931 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=4 2024-05-27T09:51:34.931 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-05-27T09:51:34.931 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-05-27T09:51:34.931 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-05-27T09:51:34.931 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-05-27T09:51:34.931 DEBUG:tasks.check_counter:Getting stats from k 2024-05-27T09:51:34.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.k perf dump 2024-05-27T09:51:35.184 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T09:51:36.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:35 smithi044 ceph-mon[31618]: pgmap v515: 97 pgs: 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 80 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:36.255 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:35 smithi050 ceph-mon[36371]: pgmap v515: 97 pgs: 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 80 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:36.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:35 smithi189 ceph-mon[36671]: pgmap v515: 97 pgs: 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 80 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:36.811 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-05-27T09:51:36.811 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:36.812 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 0, 2024-05-27T09:51:36.812 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 2, 2024-05-27T09:51:36.812 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 0, 2024-05-27T09:51:36.812 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 0, 2024-05-27T09:51:36.812 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 4, 2024-05-27T09:51:36.812 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 0, 2024-05-27T09:51:36.812 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.001117456, 2024-05-27T09:51:36.812 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.000000000, 2024-05-27T09:51:36.813 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.000118512, 2024-05-27T09:51:36.813 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:36.813 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:36.813 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:36.813 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:36.813 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:36.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:36.814 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:36.814 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:36.814 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:36.814 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:36.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:36.814 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-27T09:51:36.814 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 0 2024-05-27T09:51:36.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:36.815 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:36.815 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 3, 2024-05-27T09:51:36.815 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 258, 2024-05-27T09:51:36.815 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 480, 2024-05-27T09:51:36.815 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 281904, 2024-05-27T09:51:36.815 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 6, 2024-05-27T09:51:36.815 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 3, 2024-05-27T09:51:36.815 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.017179575, 2024-05-27T09:51:36.815 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.011705734, 2024-05-27T09:51:36.816 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.002447665, 2024-05-27T09:51:36.816 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:36.816 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:36.816 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 255, 2024-05-27T09:51:36.816 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.043015303, 2024-05-27T09:51:36.816 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000168687 2024-05-27T09:51:36.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:36.816 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:36.817 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:36.817 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:36.817 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:36.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:36.817 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-27T09:51:36.817 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 281904 2024-05-27T09:51:36.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:36.817 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:36.817 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 353, 2024-05-27T09:51:36.818 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 339, 2024-05-27T09:51:36.818 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 1900192, 2024-05-27T09:51:36.818 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 208704, 2024-05-27T09:51:36.818 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 2, 2024-05-27T09:51:36.818 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 1, 2024-05-27T09:51:36.818 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.069940488, 2024-05-27T09:51:36.818 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.020676692, 2024-05-27T09:51:36.818 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.053529253, 2024-05-27T09:51:36.819 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.008033099, 2024-05-27T09:51:36.819 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:36.819 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 339, 2024-05-27T09:51:36.819 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.018020812, 2024-05-27T09:51:36.819 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000053158 2024-05-27T09:51:36.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:36.819 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:36.819 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:36.820 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:36.820 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:36.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:36.820 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 1900192, 2024-05-27T09:51:36.820 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 208704 2024-05-27T09:51:36.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:36.820 INFO:teuthology.orchestra.run.smithi050.stdout: "cct": { 2024-05-27T09:51:36.820 INFO:teuthology.orchestra.run.smithi050.stdout: "total_workers": 0, 2024-05-27T09:51:36.821 INFO:teuthology.orchestra.run.smithi050.stdout: "unhealthy_workers": 0 2024-05-27T09:51:36.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:36.821 INFO:teuthology.orchestra.run.smithi050.stdout: "mempool": { 2024-05-27T09:51:36.821 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_bytes": 0, 2024-05-27T09:51:36.821 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_items": 0, 2024-05-27T09:51:36.821 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:36.821 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:36.821 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:36.821 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:36.822 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:36.822 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:36.822 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:36.822 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:36.822 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:36.822 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:36.822 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:36.822 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:36.823 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:36.823 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:36.823 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:36.823 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:36.823 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:36.823 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:36.823 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:36.823 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:36.824 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:36.824 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:36.824 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:36.824 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:36.824 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:36.824 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:36.824 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:36.824 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:36.824 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:36.825 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_items": 0, 2024-05-27T09:51:36.825 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:36.825 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:36.825 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:36.825 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:36.825 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_bytes": 148732, 2024-05-27T09:51:36.825 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_items": 31, 2024-05-27T09:51:36.825 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_bytes": 0, 2024-05-27T09:51:36.826 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_items": 0, 2024-05-27T09:51:36.826 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bytes": 0, 2024-05-27T09:51:36.826 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_items": 0, 2024-05-27T09:51:36.826 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:36.826 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:36.826 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:36.826 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:36.826 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_bytes": 0, 2024-05-27T09:51:36.827 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_items": 0, 2024-05-27T09:51:36.827 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:36.827 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:36.827 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:36.827 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_items": 0, 2024-05-27T09:51:36.827 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_bytes": 928, 2024-05-27T09:51:36.827 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_items": 2, 2024-05-27T09:51:36.827 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:36.828 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_items": 0, 2024-05-27T09:51:36.828 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:36.828 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_items": 0 2024-05-27T09:51:36.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-27T09:51:36.828 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:36.828 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-05-27T09:51:36.828 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 104857600, 2024-05-27T09:51:36.828 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-05-27T09:51:36.828 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 352, 2024-05-27T09:51:36.829 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 1847887, 2024-05-27T09:51:36.829 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:36.829 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 352, 2024-05-27T09:51:36.829 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-05-27T09:51:36.829 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-05-27T09:51:36.829 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 352, 2024-05-27T09:51:36.829 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 1847887, 2024-05-27T09:51:36.829 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-05-27T09:51:36.830 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-05-27T09:51:36.830 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-05-27T09:51:36.830 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-05-27T09:51:36.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:36.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-27T09:51:36.830 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-05-27T09:51:37.484 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-05-27T09:51:37.484 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-05-27T09:51:37.484 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-05-27T09:51:37.484 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-05-27T09:51:37.484 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-05-27T09:51:37.484 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-05-27T09:51:37.485 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-05-27T09:51:37.485 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-05-27T09:51:37.485 DEBUG:tasks.check_counter:Getting stats from c 2024-05-27T09:51:37.485 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.c perf dump 2024-05-27T09:51:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:51:38.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:37 smithi189 ceph-mon[36671]: pgmap v516: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T09:51:38.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:37 smithi189 ceph-mon[36671]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:51:38.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:37 smithi050 ceph-mon[36371]: pgmap v516: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T09:51:38.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:37 smithi050 ceph-mon[36371]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:51:38.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:37 smithi044 ceph-mon[31618]: pgmap v516: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T09:51:38.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:37 smithi044 ceph-mon[31618]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T09:51:39.251 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:39 smithi189 ceph-mon[36671]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T09:51:39.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:39 smithi050 ceph-mon[36371]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T09:51:39.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:39 smithi044 ceph-mon[31618]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T09:51:39.854 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-05-27T09:51:39.854 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:39.854 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 10771, 2024-05-27T09:51:39.854 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 6615, 2024-05-27T09:51:39.854 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 148543650, 2024-05-27T09:51:39.855 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 1534818, 2024-05-27T09:51:39.855 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 10, 2024-05-27T09:51:39.855 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 2, 2024-05-27T09:51:39.855 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.662055432, 2024-05-27T09:51:39.855 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.213751330, 2024-05-27T09:51:39.855 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.605722986, 2024-05-27T09:51:39.855 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.006999078, 2024-05-27T09:51:39.855 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:39.856 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 6617, 2024-05-27T09:51:39.856 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.293600728, 2024-05-27T09:51:39.856 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000044370 2024-05-27T09:51:39.856 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.856 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:39.856 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 16928, 2024-05-27T09:51:39.856 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.003111153, 2024-05-27T09:51:39.856 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000183 2024-05-27T09:51:39.856 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.857 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2208, 2024-05-27T09:51:39.857 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 9152 2024-05-27T09:51:39.857 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.857 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:39.857 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 8287, 2024-05-27T09:51:39.857 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 4460, 2024-05-27T09:51:39.857 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 88800067, 2024-05-27T09:51:39.857 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 1147837, 2024-05-27T09:51:39.857 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 35, 2024-05-27T09:51:39.858 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 7, 2024-05-27T09:51:39.858 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.440492158, 2024-05-27T09:51:39.858 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.179432222, 2024-05-27T09:51:39.858 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.350912345, 2024-05-27T09:51:39.858 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.008812148, 2024-05-27T09:51:39.858 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:39.858 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4457, 2024-05-27T09:51:39.858 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.110879825, 2024-05-27T09:51:39.859 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000024877 2024-05-27T09:51:39.859 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.859 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:39.859 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 11625, 2024-05-27T09:51:39.859 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.001657320, 2024-05-27T09:51:39.859 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000142 2024-05-27T09:51:39.859 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.859 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2020640, 2024-05-27T09:51:39.860 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 217184 2024-05-27T09:51:39.860 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.860 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:39.860 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 14825, 2024-05-27T09:51:39.860 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 15125, 2024-05-27T09:51:39.860 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 4178717, 2024-05-27T09:51:39.860 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 2042457, 2024-05-27T09:51:39.860 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 23, 2024-05-27T09:51:39.861 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 6, 2024-05-27T09:51:39.861 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.818959808, 2024-05-27T09:51:39.861 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.349199755, 2024-05-27T09:51:39.861 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.656880424, 2024-05-27T09:51:39.861 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000109188, 2024-05-27T09:51:39.861 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:39.861 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 15121, 2024-05-27T09:51:39.861 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.337700585, 2024-05-27T09:51:39.861 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000022333 2024-05-27T09:51:39.862 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.862 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:39.862 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 14798, 2024-05-27T09:51:39.862 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.007103487, 2024-05-27T09:51:39.862 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000480 2024-05-27T09:51:39.862 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.863 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 5376, 2024-05-27T09:51:39.863 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 684912 2024-05-27T09:51:39.863 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.863 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-05-27T09:51:39.863 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-05-27T09:51:39.863 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-05-27T09:51:39.863 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.863 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-05-27T09:51:39.864 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-27T09:51:39.864 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-27T09:51:39.864 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 71, 2024-05-27T09:51:39.864 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.001665373, 2024-05-27T09:51:39.864 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000023455 2024-05-27T09:51:39.864 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:39.864 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.864 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-05-27T09:51:39.865 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-27T09:51:39.865 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-27T09:51:39.865 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-05-27T09:51:39.865 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000007701, 2024-05-27T09:51:39.865 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000003850 2024-05-27T09:51:39.865 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:39.865 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.865 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-05-27T09:51:39.866 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 14771, 2024-05-27T09:51:39.866 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 3, 2024-05-27T09:51:39.866 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-05-27T09:51:39.866 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 3, 2024-05-27T09:51:39.866 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 10.763489412, 2024-05-27T09:51:39.866 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 3.587829804 2024-05-27T09:51:39.866 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.867 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-05-27T09:51:39.867 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 14768, 2024-05-27T09:51:39.867 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-05-27T09:51:39.867 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-05-27T09:51:39.867 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 11, 2024-05-27T09:51:39.867 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-05-27T09:51:39.867 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-05-27T09:51:39.867 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 21, 2024-05-27T09:51:39.867 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 3, 2024-05-27T09:51:39.868 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 18, 2024-05-27T09:51:39.868 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-05-27T09:51:39.868 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 15, 2024-05-27T09:51:39.868 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 23, 2024-05-27T09:51:39.868 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-05-27T09:51:39.868 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-05-27T09:51:39.868 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 4, 2024-05-27T09:51:39.868 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 17921, 2024-05-27T09:51:39.869 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 3108, 2024-05-27T09:51:39.869 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 14768, 2024-05-27T09:51:39.869 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 36, 2024-05-27T09:51:39.869 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-05-27T09:51:39.869 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-05-27T09:51:39.869 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 3, 2024-05-27T09:51:39.869 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-05-27T09:51:39.869 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-05-27T09:51:39.870 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-05-27T09:51:39.870 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-05-27T09:51:39.870 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-05-27T09:51:39.870 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-05-27T09:51:39.870 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-05-27T09:51:39.870 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-05-27T09:51:39.870 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-05-27T09:51:39.870 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 4, 2024-05-27T09:51:39.871 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 234, 2024-05-27T09:51:39.871 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 6, 2024-05-27T09:51:39.871 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-05-27T09:51:39.871 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-05-27T09:51:39.871 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-05-27T09:51:39.871 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-05-27T09:51:39.871 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-05-27T09:51:39.871 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-05-27T09:51:39.872 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-05-27T09:51:39.872 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-05-27T09:51:39.872 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-05-27T09:51:39.872 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 6, 2024-05-27T09:51:39.872 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 7, 2024-05-27T09:51:39.872 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-05-27T09:51:39.872 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-27T09:51:39.872 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-27T09:51:39.873 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 8, 2024-05-27T09:51:39.873 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-05-27T09:51:39.873 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 2, 2024-05-27T09:51:39.873 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-05-27T09:51:39.873 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.873 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-05-27T09:51:39.873 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-05-27T09:51:39.873 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 6187, 2024-05-27T09:51:39.874 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 3100, 2024-05-27T09:51:39.874 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 4173, 2024-05-27T09:51:39.874 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-05-27T09:51:39.874 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-05-27T09:51:39.874 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-05-27T09:51:39.874 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-05-27T09:51:39.874 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-05-27T09:51:39.874 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-05-27T09:51:39.875 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-05-27T09:51:39.875 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-05-27T09:51:39.875 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-05-27T09:51:39.875 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-05-27T09:51:39.875 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-05-27T09:51:39.875 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-05-27T09:51:39.875 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-05-27T09:51:39.875 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 10, 2024-05-27T09:51:39.876 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 102, 2024-05-27T09:51:39.876 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-05-27T09:51:39.876 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-05-27T09:51:39.876 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-05-27T09:51:39.876 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-05-27T09:51:39.876 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-05-27T09:51:39.876 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-05-27T09:51:39.876 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.877 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-05-27T09:51:39.877 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-05-27T09:51:39.877 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 10, 2024-05-27T09:51:39.877 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-05-27T09:51:39.877 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-05-27T09:51:39.877 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 10, 2024-05-27T09:51:39.877 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-05-27T09:51:39.877 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-05-27T09:51:39.878 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 2, 2024-05-27T09:51:39.878 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-05-27T09:51:39.878 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-05-27T09:51:39.878 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-05-27T09:51:39.878 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 2, 2024-05-27T09:51:39.878 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-05-27T09:51:39.878 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-05-27T09:51:39.878 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-05-27T09:51:39.879 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 4197185, 2024-05-27T09:51:39.879 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 0, 2024-05-27T09:51:39.879 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-05-27T09:51:39.879 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 9, 2024-05-27T09:51:39.879 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.019050087, 2024-05-27T09:51:39.879 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.002116676 2024-05-27T09:51:39.879 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.879 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 0 2024-05-27T09:51:39.880 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.880 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-05-27T09:51:39.880 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 26, 2024-05-27T09:51:39.880 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 41, 2024-05-27T09:51:39.880 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 15, 2024-05-27T09:51:39.880 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 26, 2024-05-27T09:51:39.880 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 3138, 2024-05-27T09:51:39.881 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 3112, 2024-05-27T09:51:39.881 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 21, 2024-05-27T09:51:39.881 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 44, 2024-05-27T09:51:39.881 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 23, 2024-05-27T09:51:39.881 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-05-27T09:51:39.881 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 2, 2024-05-27T09:51:39.881 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 2, 2024-05-27T09:51:39.881 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 132480, 2024-05-27T09:51:39.881 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 223632 2024-05-27T09:51:39.882 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.882 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-05-27T09:51:39.882 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 14797, 2024-05-27T09:51:39.882 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 1, 2024-05-27T09:51:39.882 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 14771, 2024-05-27T09:51:39.882 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 37, 2024-05-27T09:51:39.882 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 104, 2024-05-27T09:51:39.883 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-05-27T09:51:39.883 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.883 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.883 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.883 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.883 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-05-27T09:51:39.883 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.883 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.884 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.884 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.884 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-05-27T09:51:39.884 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.884 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.884 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.884 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.884 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-05-27T09:51:39.885 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.885 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.885 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.885 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.885 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-05-27T09:51:39.885 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 3, 2024-05-27T09:51:39.885 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 10.763489412, 2024-05-27T09:51:39.886 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 3.587829804 2024-05-27T09:51:39.886 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.886 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-05-27T09:51:39.886 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.886 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.886 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.886 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.886 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-05-27T09:51:39.887 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.887 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.887 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.887 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.887 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-05-27T09:51:39.887 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.887 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.887 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.888 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.888 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-05-27T09:51:39.888 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.888 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.888 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.888 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.888 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-05-27T09:51:39.888 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.889 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.889 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.889 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.889 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-05-27T09:51:39.889 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.889 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.890 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.890 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.890 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-05-27T09:51:39.890 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.890 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.891 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.891 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.891 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-05-27T09:51:39.891 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.891 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.891 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.891 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.891 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-05-27T09:51:39.892 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.892 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.892 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.892 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.892 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-05-27T09:51:39.892 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.892 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.892 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.893 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.893 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-05-27T09:51:39.893 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.893 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.893 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.893 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.893 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-05-27T09:51:39.893 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.894 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.894 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.894 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.894 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-05-27T09:51:39.894 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.894 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.894 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.895 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.895 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-05-27T09:51:39.895 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.895 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.895 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.895 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.895 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-05-27T09:51:39.895 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.896 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.896 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.896 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.896 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-05-27T09:51:39.896 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.896 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.896 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.896 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.896 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-05-27T09:51:39.897 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.897 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.897 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.897 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.897 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-05-27T09:51:39.897 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.897 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.897 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.898 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.898 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-05-27T09:51:39.898 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.898 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.898 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.898 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.898 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-05-27T09:51:39.899 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.899 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.899 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.899 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.899 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-05-27T09:51:39.899 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.899 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.899 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.899 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.900 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-05-27T09:51:39.900 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.900 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.900 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.900 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.900 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-05-27T09:51:39.900 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.900 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.901 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.901 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.901 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-05-27T09:51:39.901 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.901 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.901 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.901 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.901 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-05-27T09:51:39.902 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-05-27T09:51:39.902 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-05-27T09:51:39.902 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.902 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.902 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.902 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:39.902 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.902 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-05-27T09:51:39.903 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 2, 2024-05-27T09:51:39.903 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 5, 2024-05-27T09:51:39.903 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 3, 2024-05-27T09:51:39.903 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 2, 2024-05-27T09:51:39.903 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-05-27T09:51:39.903 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 1, 2024-05-27T09:51:39.903 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 0, 2024-05-27T09:51:39.903 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 962, 2024-05-27T09:51:39.904 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-05-27T09:51:39.904 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.904 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-05-27T09:51:39.904 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-05-27T09:51:39.904 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-05-27T09:51:39.904 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:39.904 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:39.904 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:39.905 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:39.905 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:39.905 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:39.905 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:39.905 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:39.905 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:39.905 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:39.905 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:39.905 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:39.906 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:39.906 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:39.906 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:39.906 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:39.906 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:39.906 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:39.906 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:39.906 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:39.907 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:39.907 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:39.907 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:39.907 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:39.907 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:39.907 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:39.907 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:39.907 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:39.908 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:39.908 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-05-27T09:51:39.908 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:39.908 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:39.908 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:39.908 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:39.908 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 248308, 2024-05-27T09:51:39.908 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 88, 2024-05-27T09:51:39.909 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-05-27T09:51:39.909 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-05-27T09:51:39.909 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-05-27T09:51:39.909 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-05-27T09:51:39.909 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:39.909 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:39.909 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:39.910 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:39.910 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 8328, 2024-05-27T09:51:39.910 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 190, 2024-05-27T09:51:39.910 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:39.910 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:39.910 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:39.910 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-05-27T09:51:39.910 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 117816, 2024-05-27T09:51:39.911 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 485, 2024-05-27T09:51:39.911 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:39.911 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-05-27T09:51:39.911 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:39.911 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-05-27T09:51:39.911 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.911 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-05-27T09:51:39.911 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-05-27T09:51:39.912 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-05-27T09:51:39.912 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 53, 2024-05-27T09:51:39.912 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 14549, 2024-05-27T09:51:39.912 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 33, 2024-05-27T09:51:39.912 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 53, 2024-05-27T09:51:39.912 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-05-27T09:51:39.912 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 53, 2024-05-27T09:51:39.912 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.356993051, 2024-05-27T09:51:39.913 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.006735717 2024-05-27T09:51:39.913 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.913 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-05-27T09:51:39.913 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-05-27T09:51:39.913 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 53, 2024-05-27T09:51:39.913 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 81 2024-05-27T09:51:39.913 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.913 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 53, 2024-05-27T09:51:39.914 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 0, 2024-05-27T09:51:39.914 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 53, 2024-05-27T09:51:39.914 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-05-27T09:51:39.914 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-05-27T09:51:39.914 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 0, 2024-05-27T09:51:39.914 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 10, 2024-05-27T09:51:39.914 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 0, 2024-05-27T09:51:39.914 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 9, 2024-05-27T09:51:39.915 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 9, 2024-05-27T09:51:39.915 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-05-27T09:51:39.915 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-05-27T09:51:39.915 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-05-27T09:51:39.915 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-05-27T09:51:39.915 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 6, 2024-05-27T09:51:39.915 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-05-27T09:51:39.915 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-05-27T09:51:39.916 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-05-27T09:51:39.916 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 0, 2024-05-27T09:51:39.916 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 20, 2024-05-27T09:51:39.916 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-05-27T09:51:39.916 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-05-27T09:51:39.916 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-05-27T09:51:39.916 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-05-27T09:51:39.916 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-05-27T09:51:39.917 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-05-27T09:51:39.917 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-05-27T09:51:39.917 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-05-27T09:51:39.917 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-05-27T09:51:39.917 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-05-27T09:51:39.917 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-05-27T09:51:39.917 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-05-27T09:51:39.917 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-05-27T09:51:39.918 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-05-27T09:51:39.918 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-05-27T09:51:39.918 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-05-27T09:51:39.918 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-05-27T09:51:39.918 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-05-27T09:51:39.918 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-05-27T09:51:39.918 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-05-27T09:51:39.918 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-05-27T09:51:39.919 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-05-27T09:51:39.919 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-05-27T09:51:39.919 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-05-27T09:51:39.919 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-05-27T09:51:39.919 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-05-27T09:51:39.919 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 105, 2024-05-27T09:51:39.919 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-05-27T09:51:39.919 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 26, 2024-05-27T09:51:39.920 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 8, 2024-05-27T09:51:39.920 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 12, 2024-05-27T09:51:39.920 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 4, 2024-05-27T09:51:39.920 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-05-27T09:51:39.920 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 23, 2024-05-27T09:51:39.920 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 0, 2024-05-27T09:51:39.920 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 4 2024-05-27T09:51:39.920 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.921 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-05-27T09:51:39.921 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 1, 2024-05-27T09:51:39.921 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-05-27T09:51:39.921 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-05-27T09:51:39.921 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-05-27T09:51:39.921 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.921 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-05-27T09:51:39.921 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 103 2024-05-27T09:51:39.922 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.922 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-05-27T09:51:39.922 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-05-27T09:51:39.922 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-05-27T09:51:39.922 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-05-27T09:51:39.922 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-05-27T09:51:39.922 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-05-27T09:51:39.923 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-05-27T09:51:39.923 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-05-27T09:51:39.923 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.923 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:39.923 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:39.923 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-27T09:51:39.923 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:39.923 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 33879, 2024-05-27T09:51:39.924 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 238684234, 2024-05-27T09:51:39.924 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:39.924 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 33879, 2024-05-27T09:51:39.924 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-27T09:51:39.924 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-27T09:51:39.924 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 33879, 2024-05-27T09:51:39.924 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 238684234, 2024-05-27T09:51:39.924 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:39.925 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.925 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.925 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.925 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:39.925 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.925 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-05-27T09:51:39.925 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:39.925 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-27T09:51:39.926 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:39.926 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-27T09:51:39.926 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-27T09:51:39.926 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:39.926 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:39.926 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 53, 2024-05-27T09:51:39.926 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 14549, 2024-05-27T09:51:39.926 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 53, 2024-05-27T09:51:39.927 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 14549, 2024-05-27T09:51:39.927 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:39.927 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.927 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.927 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.927 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:39.927 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.927 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-05-27T09:51:39.928 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:39.928 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-05-27T09:51:39.928 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:39.928 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-27T09:51:39.928 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-27T09:51:39.928 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:39.928 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:39.928 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 53, 2024-05-27T09:51:39.928 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 53, 2024-05-27T09:51:39.929 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 53, 2024-05-27T09:51:39.929 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 53, 2024-05-27T09:51:39.929 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:39.929 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.929 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.929 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.929 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:39.929 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.930 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-05-27T09:51:39.930 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:39.930 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-27T09:51:39.930 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:39.930 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-27T09:51:39.930 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-27T09:51:39.930 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:39.930 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:39.931 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-27T09:51:39.931 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-27T09:51:39.931 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-05-27T09:51:39.931 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-05-27T09:51:39.931 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:39.931 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.931 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.931 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.932 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:39.932 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:39.932 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x557295fcc9c0": { 2024-05-27T09:51:39.932 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:39.932 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-27T09:51:39.932 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:39.932 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 10, 2024-05-27T09:51:39.932 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 2881, 2024-05-27T09:51:39.933 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:39.933 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 10, 2024-05-27T09:51:39.933 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-27T09:51:39.933 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-27T09:51:39.933 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 9, 2024-05-27T09:51:39.933 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 2881, 2024-05-27T09:51:39.933 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:39.933 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:39.934 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:39.934 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:39.934 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:39.934 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:39.934 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-05-27T09:51:40.324 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:40 smithi189 ceph-mon[36671]: pgmap v518: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:40.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:40 smithi044 ceph-mon[31618]: pgmap v518: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:40.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:40 smithi050 ceph-mon[36371]: pgmap v518: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:40.611 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-05-27T09:51:40.611 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-05-27T09:51:40.611 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-05-27T09:51:40.611 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=6187 2024-05-27T09:51:40.611 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-05-27T09:51:40.611 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-05-27T09:51:40.611 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-05-27T09:51:40.612 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-05-27T09:51:40.612 DEBUG:tasks.check_counter:Getting stats from f 2024-05-27T09:51:40.612 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.f perf dump 2024-05-27T09:51:40.855 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:51:42.239 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:42 smithi189 ceph-mon[36671]: pgmap v519: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:42.346 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:42 smithi050 ceph-mon[36371]: pgmap v519: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:42.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:42 smithi044 ceph-mon[31618]: pgmap v519: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:42.816 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-05-27T09:51:42.817 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:42.817 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 998, 2024-05-27T09:51:42.817 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 984, 2024-05-27T09:51:42.817 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 42472656, 2024-05-27T09:51:42.817 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 347010, 2024-05-27T09:51:42.817 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 7, 2024-05-27T09:51:42.817 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 3, 2024-05-27T09:51:42.818 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.119803949, 2024-05-27T09:51:42.818 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.028357867, 2024-05-27T09:51:42.818 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.056861883, 2024-05-27T09:51:42.818 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.027901578, 2024-05-27T09:51:42.818 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:42.818 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 983, 2024-05-27T09:51:42.818 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.035902357, 2024-05-27T09:51:42.818 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000036523 2024-05-27T09:51:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 266592, 2024-05-27T09:51:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 82112 2024-05-27T09:51:42.820 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.820 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:42.820 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 484, 2024-05-27T09:51:42.820 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 579, 2024-05-27T09:51:42.820 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 47481165, 2024-05-27T09:51:42.820 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 411540, 2024-05-27T09:51:42.820 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-05-27T09:51:42.821 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-05-27T09:51:42.821 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.070214413, 2024-05-27T09:51:42.821 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.016865442, 2024-05-27T09:51:42.821 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.036319141, 2024-05-27T09:51:42.821 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.012707448, 2024-05-27T09:51:42.821 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:42.821 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 578, 2024-05-27T09:51:42.821 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.029339328, 2024-05-27T09:51:42.822 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000050760 2024-05-27T09:51:42.822 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.822 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:42.822 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.822 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.822 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.822 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.822 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 3200, 2024-05-27T09:51:42.823 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 259152 2024-05-27T09:51:42.823 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.823 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:42.823 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 389, 2024-05-27T09:51:42.823 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 391, 2024-05-27T09:51:42.823 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 37142984, 2024-05-27T09:51:42.823 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 123709, 2024-05-27T09:51:42.823 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-05-27T09:51:42.823 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-05-27T09:51:42.824 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.055231132, 2024-05-27T09:51:42.824 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.010871699, 2024-05-27T09:51:42.824 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.029468020, 2024-05-27T09:51:42.824 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.011858332, 2024-05-27T09:51:42.824 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:42.824 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 389, 2024-05-27T09:51:42.824 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.015543006, 2024-05-27T09:51:42.824 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000039956 2024-05-27T09:51:42.825 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.825 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:42.825 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.825 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.825 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.825 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.825 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-27T09:51:42.825 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 0 2024-05-27T09:51:42.826 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.826 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-05-27T09:51:42.826 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-05-27T09:51:42.826 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-05-27T09:51:42.826 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.826 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-05-27T09:51:42.826 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-27T09:51:42.826 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-27T09:51:42.827 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2811, 2024-05-27T09:51:42.827 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.343599763, 2024-05-27T09:51:42.827 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000122233 2024-05-27T09:51:42.827 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:42.827 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.827 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-05-27T09:51:42.827 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-27T09:51:42.827 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-27T09:51:42.828 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.828 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.828 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.828 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:42.828 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.828 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-05-27T09:51:42.828 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 0, 2024-05-27T09:51:42.828 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 0, 2024-05-27T09:51:42.829 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-05-27T09:51:42.829 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.829 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.829 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.829 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.829 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-05-27T09:51:42.829 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-05-27T09:51:42.829 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-05-27T09:51:42.829 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-05-27T09:51:42.830 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 0, 2024-05-27T09:51:42.830 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-05-27T09:51:42.830 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-05-27T09:51:42.830 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 54831, 2024-05-27T09:51:42.830 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 38364, 2024-05-27T09:51:42.830 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 16467, 2024-05-27T09:51:42.830 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-05-27T09:51:42.830 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 21, 2024-05-27T09:51:42.831 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-05-27T09:51:42.831 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-05-27T09:51:42.831 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-05-27T09:51:42.831 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 2, 2024-05-27T09:51:42.831 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 0, 2024-05-27T09:51:42.831 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 0, 2024-05-27T09:51:42.831 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-05-27T09:51:42.831 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 0, 2024-05-27T09:51:42.832 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-05-27T09:51:42.832 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-05-27T09:51:42.832 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-05-27T09:51:42.832 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-05-27T09:51:42.832 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-05-27T09:51:42.832 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-05-27T09:51:42.832 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-05-27T09:51:42.832 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-05-27T09:51:42.833 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-05-27T09:51:42.833 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-05-27T09:51:42.833 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-05-27T09:51:42.833 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-05-27T09:51:42.833 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 4, 2024-05-27T09:51:42.833 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 234, 2024-05-27T09:51:42.833 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 6, 2024-05-27T09:51:42.833 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-05-27T09:51:42.834 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-05-27T09:51:42.834 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-05-27T09:51:42.834 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-05-27T09:51:42.834 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-05-27T09:51:42.834 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-05-27T09:51:42.834 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-05-27T09:51:42.834 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-05-27T09:51:42.834 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-05-27T09:51:42.835 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-05-27T09:51:42.835 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-05-27T09:51:42.835 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-05-27T09:51:42.835 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-27T09:51:42.835 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-27T09:51:42.835 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-05-27T09:51:42.835 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-05-27T09:51:42.835 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-05-27T09:51:42.836 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-05-27T09:51:42.836 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.836 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-05-27T09:51:42.836 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-05-27T09:51:42.836 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-05-27T09:51:42.836 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-05-27T09:51:42.836 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 0, 2024-05-27T09:51:42.836 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-05-27T09:51:42.837 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-05-27T09:51:42.837 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-05-27T09:51:42.837 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-05-27T09:51:42.837 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-05-27T09:51:42.837 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-05-27T09:51:42.837 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-05-27T09:51:42.837 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-05-27T09:51:42.837 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-05-27T09:51:42.838 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-05-27T09:51:42.838 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-05-27T09:51:42.838 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-05-27T09:51:42.838 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-05-27T09:51:42.838 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-05-27T09:51:42.838 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-05-27T09:51:42.838 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-05-27T09:51:42.838 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-05-27T09:51:42.839 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-05-27T09:51:42.839 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-05-27T09:51:42.839 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-05-27T09:51:42.839 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-05-27T09:51:42.839 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.839 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-05-27T09:51:42.839 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-05-27T09:51:42.839 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 0, 2024-05-27T09:51:42.840 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-05-27T09:51:42.840 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-05-27T09:51:42.840 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 22304, 2024-05-27T09:51:42.840 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-05-27T09:51:42.840 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-05-27T09:51:42.840 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 0, 2024-05-27T09:51:42.840 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-05-27T09:51:42.840 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-05-27T09:51:42.840 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-05-27T09:51:42.841 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 29, 2024-05-27T09:51:42.841 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-05-27T09:51:42.841 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-05-27T09:51:42.841 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 223422806, 2024-05-27T09:51:42.841 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 223475164, 2024-05-27T09:51:42.841 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 223475052, 2024-05-27T09:51:42.841 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-05-27T09:51:42.841 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.842 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.842 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.842 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.842 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 22304 2024-05-27T09:51:42.842 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.842 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-05-27T09:51:42.842 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 24818, 2024-05-27T09:51:42.843 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 34823, 2024-05-27T09:51:42.843 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 10005, 2024-05-27T09:51:42.843 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 1046, 2024-05-27T09:51:42.843 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 1047, 2024-05-27T09:51:42.843 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 1, 2024-05-27T09:51:42.843 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 54831, 2024-05-27T09:51:42.843 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 54834, 2024-05-27T09:51:42.843 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 3, 2024-05-27T09:51:42.843 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-05-27T09:51:42.844 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 0, 2024-05-27T09:51:42.844 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 0, 2024-05-27T09:51:42.844 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 231004, 2024-05-27T09:51:42.844 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 215416 2024-05-27T09:51:42.844 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.844 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-05-27T09:51:42.844 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 0, 2024-05-27T09:51:42.844 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-05-27T09:51:42.845 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 0, 2024-05-27T09:51:42.845 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 0, 2024-05-27T09:51:42.845 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 0, 2024-05-27T09:51:42.845 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-05-27T09:51:42.845 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.845 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.845 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.845 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.846 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-05-27T09:51:42.846 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.846 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.846 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.846 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.846 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-05-27T09:51:42.846 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.846 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.847 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.847 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.847 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-05-27T09:51:42.847 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.847 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.847 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.847 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.847 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-05-27T09:51:42.848 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.848 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.848 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.848 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.848 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-05-27T09:51:42.848 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.848 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.848 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.849 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.849 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-05-27T09:51:42.849 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.849 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.849 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.849 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.849 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-05-27T09:51:42.849 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.850 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.850 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.850 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.850 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-05-27T09:51:42.850 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.850 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.850 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.850 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.850 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-05-27T09:51:42.851 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.851 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.851 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.851 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.851 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-05-27T09:51:42.851 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.851 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.851 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.852 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.852 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-05-27T09:51:42.852 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.853 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.853 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.853 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.853 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-05-27T09:51:42.853 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.853 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.853 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.853 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.854 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-05-27T09:51:42.854 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.854 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.854 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.854 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.854 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-05-27T09:51:42.854 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.855 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.855 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.855 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.855 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-05-27T09:51:42.855 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.855 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.855 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.855 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.855 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-05-27T09:51:42.856 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.856 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.856 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.856 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.856 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-05-27T09:51:42.856 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.856 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.856 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.857 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.857 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-05-27T09:51:42.857 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.857 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.857 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.857 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.857 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-05-27T09:51:42.857 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.858 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.858 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.858 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.858 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-05-27T09:51:42.858 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.858 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.858 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.858 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.859 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-05-27T09:51:42.859 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.859 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.859 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.859 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.859 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-05-27T09:51:42.859 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.859 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.860 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.860 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.860 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-05-27T09:51:42.860 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.860 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.860 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.860 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.860 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-05-27T09:51:42.861 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.861 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.861 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.861 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.861 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-05-27T09:51:42.861 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.861 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.861 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.862 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.862 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-05-27T09:51:42.862 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.862 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.862 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.862 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.862 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-05-27T09:51:42.862 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.863 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.863 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.863 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.863 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-05-27T09:51:42.863 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.863 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.863 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.863 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.864 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-05-27T09:51:42.864 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-05-27T09:51:42.864 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-05-27T09:51:42.864 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.864 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.864 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.864 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:42.864 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.865 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-05-27T09:51:42.865 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 2, 2024-05-27T09:51:42.865 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-05-27T09:51:42.865 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 0, 2024-05-27T09:51:42.865 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 0, 2024-05-27T09:51:42.865 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-05-27T09:51:42.865 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 0, 2024-05-27T09:51:42.865 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 0, 2024-05-27T09:51:42.866 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 2689, 2024-05-27T09:51:42.866 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-05-27T09:51:42.866 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.866 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-05-27T09:51:42.866 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-05-27T09:51:42.866 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-05-27T09:51:42.866 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:42.866 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:42.867 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:42.867 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:42.867 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:42.867 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:42.867 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:42.867 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:42.867 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:42.867 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:42.868 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:42.868 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:42.868 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:42.868 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:42.868 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:42.868 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:42.868 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:42.868 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:42.868 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:42.869 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:42.869 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:42.869 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:42.869 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:42.869 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:42.869 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:42.869 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:42.869 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:42.870 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:42.870 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:42.870 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-05-27T09:51:42.870 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:42.870 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:42.870 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:42.870 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:42.870 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 1148460, 2024-05-27T09:51:42.871 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 24901, 2024-05-27T09:51:42.871 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-05-27T09:51:42.871 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-05-27T09:51:42.871 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-05-27T09:51:42.871 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-05-27T09:51:42.871 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:42.871 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:42.871 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:42.872 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:42.872 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 8328, 2024-05-27T09:51:42.872 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 190, 2024-05-27T09:51:42.872 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:42.872 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:42.872 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:42.872 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-05-27T09:51:42.872 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 96609260, 2024-05-27T09:51:42.873 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 745255, 2024-05-27T09:51:42.873 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:42.873 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-05-27T09:51:42.873 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:42.873 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-05-27T09:51:42.873 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.873 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-05-27T09:51:42.873 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-05-27T09:51:42.874 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-05-27T09:51:42.874 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 1701, 2024-05-27T09:51:42.874 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 26, 2024-05-27T09:51:42.874 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 4, 2024-05-27T09:51:42.874 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 1701, 2024-05-27T09:51:42.874 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-05-27T09:51:42.874 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1701, 2024-05-27T09:51:42.874 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1.774966054, 2024-05-27T09:51:42.875 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001043483 2024-05-27T09:51:42.875 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.875 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-05-27T09:51:42.875 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-05-27T09:51:42.875 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1701, 2024-05-27T09:51:42.875 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1702 2024-05-27T09:51:42.875 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.875 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 1701, 2024-05-27T09:51:42.876 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 1701, 2024-05-27T09:51:42.876 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 0, 2024-05-27T09:51:42.876 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-05-27T09:51:42.876 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-05-27T09:51:42.876 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 999, 2024-05-27T09:51:42.876 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 0, 2024-05-27T09:51:42.876 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 699, 2024-05-27T09:51:42.876 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 0, 2024-05-27T09:51:42.877 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 0, 2024-05-27T09:51:42.877 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-05-27T09:51:42.877 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-05-27T09:51:42.877 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-05-27T09:51:42.877 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-05-27T09:51:42.877 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 0, 2024-05-27T09:51:42.877 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-05-27T09:51:42.877 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-05-27T09:51:42.878 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-05-27T09:51:42.878 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 2, 2024-05-27T09:51:42.878 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 0, 2024-05-27T09:51:42.878 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-05-27T09:51:42.878 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-05-27T09:51:42.878 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-05-27T09:51:42.878 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-05-27T09:51:42.878 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-05-27T09:51:42.879 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-05-27T09:51:42.879 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-05-27T09:51:42.879 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-05-27T09:51:42.879 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-05-27T09:51:42.879 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-05-27T09:51:42.879 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-05-27T09:51:42.879 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-05-27T09:51:42.879 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-05-27T09:51:42.879 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-05-27T09:51:42.880 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-05-27T09:51:42.880 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-05-27T09:51:42.880 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-05-27T09:51:42.880 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-05-27T09:51:42.880 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-05-27T09:51:42.880 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-05-27T09:51:42.880 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-05-27T09:51:42.880 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-05-27T09:51:42.881 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-05-27T09:51:42.881 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-05-27T09:51:42.881 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-05-27T09:51:42.881 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-05-27T09:51:42.881 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 105, 2024-05-27T09:51:42.881 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-05-27T09:51:42.881 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 17, 2024-05-27T09:51:42.881 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 10, 2024-05-27T09:51:42.882 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 10, 2024-05-27T09:51:42.882 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-05-27T09:51:42.882 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-05-27T09:51:42.882 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 0, 2024-05-27T09:51:42.882 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 2, 2024-05-27T09:51:42.882 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 0 2024-05-27T09:51:42.882 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.882 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-05-27T09:51:42.883 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 0, 2024-05-27T09:51:42.883 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-05-27T09:51:42.883 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-05-27T09:51:42.883 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-05-27T09:51:42.883 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.883 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-05-27T09:51:42.883 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-05-27T09:51:42.883 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.884 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-05-27T09:51:42.884 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-05-27T09:51:42.884 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-05-27T09:51:42.884 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-05-27T09:51:42.884 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-05-27T09:51:42.884 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-05-27T09:51:42.884 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-05-27T09:51:42.884 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-05-27T09:51:42.885 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.885 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:42.885 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:42.885 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-27T09:51:42.885 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:42.885 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 1867, 2024-05-27T09:51:42.885 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 126917361, 2024-05-27T09:51:42.885 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:42.886 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 1867, 2024-05-27T09:51:42.886 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-27T09:51:42.886 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-27T09:51:42.886 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1867, 2024-05-27T09:51:42.886 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 126917361, 2024-05-27T09:51:42.886 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:42.886 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.886 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.887 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.887 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:42.887 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.887 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-05-27T09:51:42.887 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:42.887 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-27T09:51:42.887 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:42.887 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-27T09:51:42.888 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-27T09:51:42.888 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:42.888 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:42.888 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 1701, 2024-05-27T09:51:42.888 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 126343468, 2024-05-27T09:51:42.888 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1701, 2024-05-27T09:51:42.888 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 126343468, 2024-05-27T09:51:42.888 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:42.889 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.889 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.889 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.889 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:42.889 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.889 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-05-27T09:51:42.889 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:42.889 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-05-27T09:51:42.890 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:42.890 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-27T09:51:42.890 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-27T09:51:42.890 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:42.890 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:42.890 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 1701, 2024-05-27T09:51:42.890 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 1701, 2024-05-27T09:51:42.890 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1701, 2024-05-27T09:51:42.891 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1701, 2024-05-27T09:51:42.891 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:42.891 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.891 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.891 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.891 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:42.891 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.891 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-05-27T09:51:42.892 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:42.892 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-27T09:51:42.892 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:42.892 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-27T09:51:42.892 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-27T09:51:42.892 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:42.892 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:42.892 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-27T09:51:42.893 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-27T09:51:42.893 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-05-27T09:51:42.893 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-05-27T09:51:42.893 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:42.893 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.893 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.893 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.893 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:42.893 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:42.894 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x565413833520": { 2024-05-27T09:51:42.894 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:42.894 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-27T09:51:42.894 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:42.894 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-27T09:51:42.894 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-27T09:51:42.894 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:42.895 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:42.895 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-27T09:51:42.895 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-27T09:51:42.895 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-05-27T09:51:42.895 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-05-27T09:51:42.895 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:42.895 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:42.895 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:42.895 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:42.896 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:42.896 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:42.896 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-05-27T09:51:43.650 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-05-27T09:51:43.650 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-05-27T09:51:43.650 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-05-27T09:51:43.650 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-05-27T09:51:43.650 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-05-27T09:51:43.651 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-05-27T09:51:43.651 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-05-27T09:51:43.651 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-05-27T09:51:43.651 DEBUG:tasks.check_counter:Getting stats from i 2024-05-27T09:51:43.651 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.i perf dump 2024-05-27T09:51:43.907 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:51:44.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:44 smithi189 ceph-mon[36671]: pgmap v520: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:44.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:44 smithi044 ceph-mon[31618]: pgmap v520: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:44.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:44 smithi050 ceph-mon[36371]: pgmap v520: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:45.469 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-05-27T09:51:45.469 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:45.469 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 475, 2024-05-27T09:51:45.469 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 469, 2024-05-27T09:51:45.469 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 134554785, 2024-05-27T09:51:45.469 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 167874, 2024-05-27T09:51:45.470 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 7, 2024-05-27T09:51:45.470 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 3, 2024-05-27T09:51:45.470 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.125331973, 2024-05-27T09:51:45.470 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.016123540, 2024-05-27T09:51:45.470 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.088877664, 2024-05-27T09:51:45.470 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.015931623, 2024-05-27T09:51:45.470 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:45.471 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 469, 2024-05-27T09:51:45.471 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.026863770, 2024-05-27T09:51:45.471 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000057278 2024-05-27T09:51:45.471 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.471 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:45.471 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.471 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.471 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.471 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.472 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 207824, 2024-05-27T09:51:45.472 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 46080 2024-05-27T09:51:45.472 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.472 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:45.472 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 282, 2024-05-27T09:51:45.472 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 336, 2024-05-27T09:51:45.472 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 130123271, 2024-05-27T09:51:45.472 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 269452, 2024-05-27T09:51:45.473 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 7, 2024-05-27T09:51:45.473 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 6, 2024-05-27T09:51:45.473 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.089069369, 2024-05-27T09:51:45.473 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.010741871, 2024-05-27T09:51:45.473 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.066076768, 2024-05-27T09:51:45.473 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.009115406, 2024-05-27T09:51:45.473 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:45.473 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 334, 2024-05-27T09:51:45.473 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.016358301, 2024-05-27T09:51:45.474 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000048976 2024-05-27T09:51:45.474 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:45.474 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.474 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.474 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.474 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 160, 2024-05-27T09:51:45.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 180032 2024-05-27T09:51:45.475 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.475 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:45.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 389, 2024-05-27T09:51:45.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 391, 2024-05-27T09:51:45.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 196064533, 2024-05-27T09:51:45.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 123709, 2024-05-27T09:51:45.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-05-27T09:51:45.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-05-27T09:51:45.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.120924663, 2024-05-27T09:51:45.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.011837178, 2024-05-27T09:51:45.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.095403921, 2024-05-27T09:51:45.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.010146812, 2024-05-27T09:51:45.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:45.476 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 389, 2024-05-27T09:51:45.476 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.014130079, 2024-05-27T09:51:45.476 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000036324 2024-05-27T09:51:45.477 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:45.477 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.477 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.477 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.477 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-27T09:51:45.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 0 2024-05-27T09:51:45.477 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.477 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-05-27T09:51:45.478 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-05-27T09:51:45.478 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-05-27T09:51:45.478 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.478 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-05-27T09:51:45.478 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-27T09:51:45.478 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-27T09:51:45.478 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1575, 2024-05-27T09:51:45.478 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.119579647, 2024-05-27T09:51:45.479 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000075923 2024-05-27T09:51:45.479 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:45.479 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.479 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-05-27T09:51:45.479 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-05-27T09:51:45.479 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-05-27T09:51:45.479 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.480 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.480 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.480 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:45.480 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.480 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-05-27T09:51:45.480 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 0, 2024-05-27T09:51:45.480 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 0, 2024-05-27T09:51:45.480 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-05-27T09:51:45.480 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.481 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.481 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.481 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.481 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-05-27T09:51:45.481 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-05-27T09:51:45.481 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-05-27T09:51:45.481 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-05-27T09:51:45.481 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 0, 2024-05-27T09:51:45.481 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-05-27T09:51:45.482 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-05-27T09:51:45.482 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 109952, 2024-05-27T09:51:45.482 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 7064, 2024-05-27T09:51:45.482 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 102888, 2024-05-27T09:51:45.482 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-05-27T09:51:45.482 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 99860, 2024-05-27T09:51:45.482 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-05-27T09:51:45.482 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-05-27T09:51:45.483 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-05-27T09:51:45.483 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 1, 2024-05-27T09:51:45.483 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 0, 2024-05-27T09:51:45.483 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 0, 2024-05-27T09:51:45.483 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-05-27T09:51:45.483 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 0, 2024-05-27T09:51:45.483 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-05-27T09:51:45.483 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-05-27T09:51:45.483 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-05-27T09:51:45.484 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-05-27T09:51:45.484 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-05-27T09:51:45.484 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-05-27T09:51:45.484 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-05-27T09:51:45.484 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-05-27T09:51:45.484 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-05-27T09:51:45.484 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-05-27T09:51:45.484 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-05-27T09:51:45.484 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-05-27T09:51:45.485 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 0, 2024-05-27T09:51:45.485 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 0, 2024-05-27T09:51:45.485 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-05-27T09:51:45.485 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-05-27T09:51:45.485 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-05-27T09:51:45.485 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-05-27T09:51:45.485 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-05-27T09:51:45.485 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-05-27T09:51:45.485 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-05-27T09:51:45.486 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-05-27T09:51:45.486 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-05-27T09:51:45.486 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-05-27T09:51:45.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-05-27T09:51:45.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-05-27T09:51:45.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-05-27T09:51:45.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-27T09:51:45.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-27T09:51:45.487 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-05-27T09:51:45.487 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-05-27T09:51:45.487 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-05-27T09:51:45.487 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-05-27T09:51:45.487 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.487 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-05-27T09:51:45.487 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-05-27T09:51:45.487 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-05-27T09:51:45.488 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-05-27T09:51:45.488 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 0, 2024-05-27T09:51:45.488 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-05-27T09:51:45.488 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-05-27T09:51:45.488 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-05-27T09:51:45.488 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-05-27T09:51:45.488 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-05-27T09:51:45.489 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-05-27T09:51:45.489 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-05-27T09:51:45.489 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-05-27T09:51:45.489 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-05-27T09:51:45.489 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-05-27T09:51:45.489 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-05-27T09:51:45.489 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-05-27T09:51:45.489 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-05-27T09:51:45.490 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-05-27T09:51:45.490 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-05-27T09:51:45.490 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-05-27T09:51:45.490 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-05-27T09:51:45.490 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-05-27T09:51:45.490 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-05-27T09:51:45.490 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-05-27T09:51:45.491 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-05-27T09:51:45.491 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.491 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-05-27T09:51:45.491 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-05-27T09:51:45.491 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 0, 2024-05-27T09:51:45.491 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-05-27T09:51:45.491 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-05-27T09:51:45.491 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 136377, 2024-05-27T09:51:45.492 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-05-27T09:51:45.492 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-05-27T09:51:45.492 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 0, 2024-05-27T09:51:45.492 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-05-27T09:51:45.492 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-05-27T09:51:45.492 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 140, 2024-05-27T09:51:45.492 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 12, 2024-05-27T09:51:45.492 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-05-27T09:51:45.492 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-05-27T09:51:45.493 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 315509816, 2024-05-27T09:51:45.493 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 668405433, 2024-05-27T09:51:45.493 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 668401453, 2024-05-27T09:51:45.493 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-05-27T09:51:45.493 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.493 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.493 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.493 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.494 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 136377 2024-05-27T09:51:45.494 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.494 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-05-27T09:51:45.494 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 39951, 2024-05-27T09:51:45.494 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 92273, 2024-05-27T09:51:45.494 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 52322, 2024-05-27T09:51:45.494 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 1554, 2024-05-27T09:51:45.494 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 2160, 2024-05-27T09:51:45.495 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 606, 2024-05-27T09:51:45.495 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 109952, 2024-05-27T09:51:45.495 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 132276, 2024-05-27T09:51:45.495 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 22324, 2024-05-27T09:51:45.495 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-05-27T09:51:45.495 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 0, 2024-05-27T09:51:45.495 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 0, 2024-05-27T09:51:45.495 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 326296, 2024-05-27T09:51:45.496 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 215440 2024-05-27T09:51:45.496 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.496 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-05-27T09:51:45.496 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 0, 2024-05-27T09:51:45.496 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-05-27T09:51:45.496 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 0, 2024-05-27T09:51:45.496 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 0, 2024-05-27T09:51:45.496 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 0, 2024-05-27T09:51:45.496 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-05-27T09:51:45.497 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.497 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.497 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.497 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.497 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-05-27T09:51:45.497 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.497 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.498 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.498 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.498 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-05-27T09:51:45.498 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.498 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.498 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.498 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.498 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-05-27T09:51:45.499 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.499 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.499 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.499 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.499 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-05-27T09:51:45.499 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.499 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.499 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.500 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.500 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-05-27T09:51:45.500 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.500 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.500 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.500 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.500 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-05-27T09:51:45.500 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.501 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.501 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.501 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.501 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-05-27T09:51:45.501 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.501 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.501 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.501 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.502 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-05-27T09:51:45.502 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.502 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.502 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.502 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.502 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-05-27T09:51:45.502 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.502 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.503 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.503 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.503 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-05-27T09:51:45.503 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.503 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.503 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.503 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.504 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-05-27T09:51:45.504 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.504 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.504 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.504 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.505 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-05-27T09:51:45.505 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.505 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.505 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.505 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.505 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-05-27T09:51:45.505 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.505 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.506 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.506 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.506 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-05-27T09:51:45.506 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.506 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.506 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.506 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.506 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-05-27T09:51:45.507 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.507 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.507 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.507 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.507 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-05-27T09:51:45.507 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.507 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.507 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.507 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.508 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-05-27T09:51:45.508 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.508 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.508 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.508 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.508 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-05-27T09:51:45.508 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.509 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.509 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.509 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.509 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-05-27T09:51:45.509 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.509 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.509 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.509 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.509 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-05-27T09:51:45.510 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.510 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.510 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.510 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.510 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-05-27T09:51:45.510 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.510 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.511 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.511 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.511 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-05-27T09:51:45.511 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.511 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.511 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.511 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.511 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-05-27T09:51:45.512 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.512 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.512 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.512 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.512 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-05-27T09:51:45.512 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.512 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.512 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.513 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.513 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-05-27T09:51:45.513 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.513 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.513 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.513 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.513 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-05-27T09:51:45.513 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.513 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.514 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.514 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.514 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-05-27T09:51:45.514 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.514 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.514 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.514 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.514 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-05-27T09:51:45.515 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.515 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.515 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.515 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.515 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-05-27T09:51:45.515 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-05-27T09:51:45.515 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-05-27T09:51:45.515 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.516 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.516 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.516 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:45.516 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.516 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-05-27T09:51:45.516 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 2, 2024-05-27T09:51:45.516 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-05-27T09:51:45.517 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 0, 2024-05-27T09:51:45.517 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 0, 2024-05-27T09:51:45.517 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-05-27T09:51:45.517 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 0, 2024-05-27T09:51:45.517 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 0, 2024-05-27T09:51:45.517 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 2689, 2024-05-27T09:51:45.517 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-05-27T09:51:45.517 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.518 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-05-27T09:51:45.518 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-05-27T09:51:45.518 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-05-27T09:51:45.518 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:45.518 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:45.518 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:45.518 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:45.518 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:45.519 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:45.519 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:45.519 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:45.519 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:45.519 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:45.519 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:45.519 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:45.519 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:45.520 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:45.520 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:45.520 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:45.520 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:45.520 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:45.520 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:45.520 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:45.520 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:45.521 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:45.521 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:45.521 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:45.521 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:45.521 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:45.521 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:45.521 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:45.522 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:45.522 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-05-27T09:51:45.522 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:45.522 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:45.522 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:45.522 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:45.522 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 1716160, 2024-05-27T09:51:45.522 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 40031, 2024-05-27T09:51:45.523 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-05-27T09:51:45.523 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-05-27T09:51:45.523 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-05-27T09:51:45.523 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-05-27T09:51:45.523 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:45.523 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:45.523 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:45.523 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:45.524 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 8328, 2024-05-27T09:51:45.524 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 190, 2024-05-27T09:51:45.524 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:45.524 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:45.524 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:45.524 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-05-27T09:51:45.524 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 170927115, 2024-05-27T09:51:45.524 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 1317405, 2024-05-27T09:51:45.524 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:45.525 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-05-27T09:51:45.525 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:45.525 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-05-27T09:51:45.525 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.525 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-05-27T09:51:45.525 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-05-27T09:51:45.525 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-05-27T09:51:45.525 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 1053, 2024-05-27T09:51:45.526 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 21, 2024-05-27T09:51:45.526 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 0, 2024-05-27T09:51:45.526 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 1053, 2024-05-27T09:51:45.526 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-05-27T09:51:45.526 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1053, 2024-05-27T09:51:45.526 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 2.039960700, 2024-05-27T09:51:45.526 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001937284 2024-05-27T09:51:45.526 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.527 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-05-27T09:51:45.527 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-05-27T09:51:45.527 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1053, 2024-05-27T09:51:45.527 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1054 2024-05-27T09:51:45.527 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.527 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 1053, 2024-05-27T09:51:45.527 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 1053, 2024-05-27T09:51:45.527 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 0, 2024-05-27T09:51:45.528 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-05-27T09:51:45.528 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-05-27T09:51:45.528 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 636, 2024-05-27T09:51:45.528 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 0, 2024-05-27T09:51:45.528 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 415, 2024-05-27T09:51:45.528 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 0, 2024-05-27T09:51:45.528 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 0, 2024-05-27T09:51:45.528 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-05-27T09:51:45.529 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-05-27T09:51:45.529 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-05-27T09:51:45.529 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-05-27T09:51:45.529 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 0, 2024-05-27T09:51:45.529 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-05-27T09:51:45.529 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-05-27T09:51:45.530 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-05-27T09:51:45.530 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 1, 2024-05-27T09:51:45.530 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 0, 2024-05-27T09:51:45.530 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-05-27T09:51:45.530 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-05-27T09:51:45.530 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-05-27T09:51:45.530 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-05-27T09:51:45.530 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-05-27T09:51:45.531 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-05-27T09:51:45.531 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-05-27T09:51:45.531 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-05-27T09:51:45.531 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-05-27T09:51:45.531 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-05-27T09:51:45.531 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-05-27T09:51:45.531 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-05-27T09:51:45.532 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-05-27T09:51:45.532 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-05-27T09:51:45.532 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-05-27T09:51:45.532 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-05-27T09:51:45.532 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-05-27T09:51:45.532 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-05-27T09:51:45.532 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-05-27T09:51:45.532 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-05-27T09:51:45.532 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-05-27T09:51:45.533 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-05-27T09:51:45.533 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-05-27T09:51:45.533 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-05-27T09:51:45.533 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-05-27T09:51:45.533 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-05-27T09:51:45.533 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 105, 2024-05-27T09:51:45.533 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-05-27T09:51:45.533 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 8, 2024-05-27T09:51:45.534 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 11, 2024-05-27T09:51:45.534 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 11, 2024-05-27T09:51:45.534 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-05-27T09:51:45.534 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-05-27T09:51:45.534 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 0, 2024-05-27T09:51:45.534 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 2, 2024-05-27T09:51:45.534 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 0 2024-05-27T09:51:45.535 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.535 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-05-27T09:51:45.535 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 0, 2024-05-27T09:51:45.535 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-05-27T09:51:45.535 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-05-27T09:51:45.535 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-05-27T09:51:45.535 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.535 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-05-27T09:51:45.535 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-05-27T09:51:45.536 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.536 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-05-27T09:51:45.536 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-05-27T09:51:45.536 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-05-27T09:51:45.536 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-05-27T09:51:45.536 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-05-27T09:51:45.536 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-05-27T09:51:45.536 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-05-27T09:51:45.537 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-05-27T09:51:45.537 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.537 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:45.537 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:45.537 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-27T09:51:45.537 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:45.537 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 1142, 2024-05-27T09:51:45.537 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 460632270, 2024-05-27T09:51:45.537 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:45.538 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 1142, 2024-05-27T09:51:45.538 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-27T09:51:45.538 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-27T09:51:45.538 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1142, 2024-05-27T09:51:45.538 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 460632270, 2024-05-27T09:51:45.538 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:45.538 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.538 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.539 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.539 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:45.539 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.539 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-05-27T09:51:45.539 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:45.539 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-27T09:51:45.539 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:45.539 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-27T09:51:45.539 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-27T09:51:45.540 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:45.540 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:45.540 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 1053, 2024-05-27T09:51:45.540 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 460222604, 2024-05-27T09:51:45.540 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1053, 2024-05-27T09:51:45.540 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 460222604, 2024-05-27T09:51:45.540 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:45.540 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.540 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.541 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.541 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:45.541 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.541 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-05-27T09:51:45.541 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:45.541 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-05-27T09:51:45.541 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:45.541 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-27T09:51:45.542 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-27T09:51:45.542 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:45.542 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:45.542 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 1053, 2024-05-27T09:51:45.542 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 1053, 2024-05-27T09:51:45.542 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1053, 2024-05-27T09:51:45.542 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1053, 2024-05-27T09:51:45.542 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:45.543 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.543 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.543 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.543 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:45.543 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.543 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-05-27T09:51:45.543 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:45.543 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-27T09:51:45.544 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:45.544 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-27T09:51:45.544 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-27T09:51:45.544 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:45.544 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:45.544 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-27T09:51:45.544 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-27T09:51:45.544 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-05-27T09:51:45.544 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-05-27T09:51:45.545 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:45.545 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.545 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.545 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.545 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:45.545 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:45.545 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x56519384f110": { 2024-05-27T09:51:45.545 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:45.546 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-05-27T09:51:45.546 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:45.546 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-05-27T09:51:45.546 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-05-27T09:51:45.546 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:45.546 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-05-27T09:51:45.546 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-27T09:51:45.546 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-27T09:51:45.547 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-05-27T09:51:45.547 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-05-27T09:51:45.547 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:45.547 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:45.547 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:45.547 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:45.547 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:45.547 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:45.547 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-05-27T09:51:46.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:46 smithi189 ceph-mon[36671]: pgmap v521: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:46.223 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-05-27T09:51:46.224 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-05-27T09:51:46.224 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-05-27T09:51:46.224 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=0 2024-05-27T09:51:46.224 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=0 2024-05-27T09:51:46.224 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-05-27T09:51:46.224 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-05-27T09:51:46.225 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-05-27T09:51:46.225 DEBUG:tasks.check_counter:Getting stats from l 2024-05-27T09:51:46.225 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph daemon mds.l perf dump 2024-05-27T09:51:46.465 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T09:51:46.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:46 smithi044 ceph-mon[31618]: pgmap v521: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:46.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:46 smithi050 ceph-mon[36371]: pgmap v521: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T09:51:48.087 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:48 smithi189 ceph-mon[36671]: pgmap v522: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-27T09:51:48.119 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-05-27T09:51:48.120 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-05-27T09:51:48.120 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 0, 2024-05-27T09:51:48.120 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 2, 2024-05-27T09:51:48.120 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 0, 2024-05-27T09:51:48.120 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 0, 2024-05-27T09:51:48.120 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 4, 2024-05-27T09:51:48.120 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 0, 2024-05-27T09:51:48.120 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.001055031, 2024-05-27T09:51:48.121 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.000000000, 2024-05-27T09:51:48.121 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.000133692, 2024-05-27T09:51:48.121 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:48.121 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:48.121 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:48.121 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:48.121 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:48.121 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:48.122 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:48.122 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:48.122 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:48.122 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:48.122 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:48.122 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-27T09:51:48.122 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 0 2024-05-27T09:51:48.122 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:48.123 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-05-27T09:51:48.123 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 357, 2024-05-27T09:51:48.123 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 343, 2024-05-27T09:51:48.123 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 1901280, 2024-05-27T09:51:48.123 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 211264, 2024-05-27T09:51:48.123 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 2, 2024-05-27T09:51:48.123 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 1, 2024-05-27T09:51:48.123 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.067601705, 2024-05-27T09:51:48.124 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.021158864, 2024-05-27T09:51:48.124 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.050522370, 2024-05-27T09:51:48.124 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.008194203, 2024-05-27T09:51:48.124 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:48.124 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 343, 2024-05-27T09:51:48.124 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.017789243, 2024-05-27T09:51:48.124 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000051863 2024-05-27T09:51:48.124 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:48.126 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:48.126 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:48.126 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:48.126 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:48.126 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:48.127 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 1901280, 2024-05-27T09:51:48.127 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 211264 2024-05-27T09:51:48.127 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:48.127 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-05-27T09:51:48.127 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 2, 2024-05-27T09:51:48.127 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 261, 2024-05-27T09:51:48.127 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 320, 2024-05-27T09:51:48.127 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 282432, 2024-05-27T09:51:48.127 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-05-27T09:51:48.128 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 3, 2024-05-27T09:51:48.128 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.015565497, 2024-05-27T09:51:48.128 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.010793164, 2024-05-27T09:51:48.128 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.001736445, 2024-05-27T09:51:48.128 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-27T09:51:48.128 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-05-27T09:51:48.128 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 258, 2024-05-27T09:51:48.128 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.043817490, 2024-05-27T09:51:48.129 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000169835 2024-05-27T09:51:48.129 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:48.129 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-05-27T09:51:48.129 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:48.129 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:48.129 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:48.129 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:48.129 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-27T09:51:48.130 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 282432 2024-05-27T09:51:48.130 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:48.130 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-05-27T09:51:48.130 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 0, 2024-05-27T09:51:48.130 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-05-27T09:51:48.130 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:48.130 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-05-27T09:51:48.130 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-05-27T09:51:48.131 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-05-27T09:51:48.131 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-05-27T09:51:48.131 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-05-27T09:51:48.131 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-05-27T09:51:48.131 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-05-27T09:51:48.131 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-27T09:51:48.131 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-05-27T09:51:48.131 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-27T09:51:48.131 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-05-27T09:51:48.132 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-05-27T09:51:48.132 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-05-27T09:51:48.132 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-27T09:51:48.132 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-05-27T09:51:48.132 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-05-27T09:51:48.132 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-05-27T09:51:48.132 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-05-27T09:51:48.132 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-05-27T09:51:48.133 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-27T09:51:48.133 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-05-27T09:51:48.133 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-27T09:51:48.133 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-05-27T09:51:48.133 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-05-27T09:51:48.133 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-05-27T09:51:48.133 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-05-27T09:51:48.133 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-05-27T09:51:48.134 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-27T09:51:48.134 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-05-27T09:51:48.134 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-05-27T09:51:48.134 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-05-27T09:51:48.134 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-05-27T09:51:48.134 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-05-27T09:51:48.134 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-05-27T09:51:48.134 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-05-27T09:51:48.135 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-05-27T09:51:48.135 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-05-27T09:51:48.135 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 148692, 2024-05-27T09:51:48.135 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 30, 2024-05-27T09:51:48.135 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-05-27T09:51:48.135 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-05-27T09:51:48.135 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-05-27T09:51:48.135 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-05-27T09:51:48.136 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-05-27T09:51:48.136 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-05-27T09:51:48.136 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-05-27T09:51:48.136 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-05-27T09:51:48.136 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 0, 2024-05-27T09:51:48.136 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 0, 2024-05-27T09:51:48.136 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-05-27T09:51:48.136 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-05-27T09:51:48.136 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-05-27T09:51:48.137 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-05-27T09:51:48.137 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 928, 2024-05-27T09:51:48.137 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 2, 2024-05-27T09:51:48.137 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-05-27T09:51:48.137 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-05-27T09:51:48.137 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-05-27T09:51:48.137 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-05-27T09:51:48.137 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-05-27T09:51:48.138 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-27T09:51:48.138 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-05-27T09:51:48.138 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-05-27T09:51:48.138 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-05-27T09:51:48.138 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 355, 2024-05-27T09:51:48.138 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1848380, 2024-05-27T09:51:48.138 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-05-27T09:51:48.138 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 355, 2024-05-27T09:51:48.139 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-05-27T09:51:48.139 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-05-27T09:51:48.139 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 355, 2024-05-27T09:51:48.139 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1848380, 2024-05-27T09:51:48.139 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-05-27T09:51:48.139 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-05-27T09:51:48.139 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-05-27T09:51:48.139 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-05-27T09:51:48.140 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:48.140 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-05-27T09:51:48.140 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-05-27T09:51:48.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:48 smithi044 ceph-mon[31618]: pgmap v522: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-27T09:51:48.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:48 smithi050 ceph-mon[36371]: pgmap v522: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-27T09:51:48.661 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-05-27T09:51:48.662 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-05-27T09:51:48.662 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-05-27T09:51:48.662 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-05-27T09:51:48.662 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-05-27T09:51:48.662 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-05-27T09:51:48.662 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-05-27T09:51:48.662 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-05-27T09:51:48.663 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-27T09:51:48.674 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-27T09:51:48.675 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:51:48.675 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T09:51:48.751 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-05-27T09:51:48.751 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T09:51:48.812 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T09:51:49.325 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:51:50.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:50 smithi189 ceph-mon[36671]: pgmap v523: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T09:51:50.382 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:50 smithi044 ceph-mon[31618]: pgmap v523: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T09:51:50.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:50 smithi050 ceph-mon[36371]: pgmap v523: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T09:51:51.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.716+0000 7f0f6de00700 1 -- 172.21.15.44:0/2526654336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68109860 msgr2=0x7f0f681111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:51.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.716+0000 7f0f6de00700 1 --2- 172.21.15.44:0/2526654336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68109860 0x7f0f681111c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000b0a0 tx=0x7f0f6000b3b0 comp rx=0 tx=0).stop 2024-05-27T09:51:51.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.716+0000 7f0f6de00700 1 -- 172.21.15.44:0/2526654336 shutdown_connections 2024-05-27T09:51:51.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.716+0000 7f0f6de00700 1 --2- 172.21.15.44:0/2526654336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68109860 0x7f0f681111c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:51.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.716+0000 7f0f6de00700 1 --2- 172.21.15.44:0/2526654336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f681045e0 0x7f0f681049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:51.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.716+0000 7f0f6de00700 1 --2- 172.21.15.44:0/2526654336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f68108f10 0x7f0f68109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:51.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.716+0000 7f0f6de00700 1 -- 172.21.15.44:0/2526654336 >> 172.21.15.44:0/2526654336 conn(0x7f0f68100030 msgr2=0x7f0f68102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:51.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.716+0000 7f0f6de00700 1 -- 172.21.15.44:0/2526654336 shutdown_connections 2024-05-27T09:51:51.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.716+0000 7f0f6de00700 1 -- 172.21.15.44:0/2526654336 wait complete. 2024-05-27T09:51:51.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.718+0000 7f0f6de00700 1 Processor -- start 2024-05-27T09:51:51.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.718+0000 7f0f6de00700 1 -- start start 2024-05-27T09:51:51.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.719+0000 7f0f6de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f681045e0 0x7f0f6819adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:51.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.719+0000 7f0f6de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f68108f10 0x7f0f6819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:51.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.719+0000 7f0f6de00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68109860 0x7f0f6819f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:51.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.719+0000 7f0f6de00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0f681144e0 con 0x7f0f68108f10 2024-05-27T09:51:51.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.719+0000 7f0f6de00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0f68114360 con 0x7f0f68109860 2024-05-27T09:51:51.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.719+0000 7f0f6de00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0f68114660 con 0x7f0f681045e0 2024-05-27T09:51:51.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.719+0000 7f0f66a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f681045e0 0x7f0f6819adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:51.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.719+0000 7f0f66a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f681045e0 0x7f0f6819adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:47636/0 (socket says 172.21.15.44:47636) 2024-05-27T09:51:51.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.719+0000 7f0f66a00700 1 -- 172.21.15.44:0/1062339169 learned_addr learned my addr 172.21.15.44:0/1062339169 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:51:51.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.719+0000 7f0f5e000700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f68108f10 0x7f0f6819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:51.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.719+0000 7f0f67400700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68109860 0x7f0f6819f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:51.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.720+0000 7f0f66a00700 1 -- 172.21.15.44:0/1062339169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68109860 msgr2=0x7f0f6819f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:51.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.720+0000 7f0f66a00700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68109860 0x7f0f6819f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:51.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.720+0000 7f0f66a00700 1 -- 172.21.15.44:0/1062339169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f68108f10 msgr2=0x7f0f6819b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:51.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.720+0000 7f0f66a00700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f68108f10 0x7f0f6819b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:51.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.720+0000 7f0f66a00700 1 -- 172.21.15.44:0/1062339169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f6819fcc0 con 0x7f0f681045e0 2024-05-27T09:51:51.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.720+0000 7f0f5e000700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f68108f10 0x7f0f6819b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:51:51.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.720+0000 7f0f66a00700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f681045e0 0x7f0f6819adf0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0f5800e9a0 tx=0x7f0f5800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:51.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.720+0000 7f0f6ca00700 1 -- 172.21.15.44:0/1062339169 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5800cbd0 con 0x7f0f681045e0 2024-05-27T09:51:51.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.720+0000 7f0f6de00700 1 -- 172.21.15.44:0/1062339169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f6819ffb0 con 0x7f0f681045e0 2024-05-27T09:51:51.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.721+0000 7f0f6de00700 1 -- 172.21.15.44:0/1062339169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f681a78d0 con 0x7f0f681045e0 2024-05-27T09:51:51.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.721+0000 7f0f6ca00700 1 -- 172.21.15.44:0/1062339169 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f5800cd70 con 0x7f0f681045e0 2024-05-27T09:51:51.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.721+0000 7f0f6ca00700 1 -- 172.21.15.44:0/1062339169 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f58018680 con 0x7f0f681045e0 2024-05-27T09:51:51.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.722+0000 7f0f6de00700 1 -- 172.21.15.44:0/1062339169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f44005360 con 0x7f0f681045e0 2024-05-27T09:51:51.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.724+0000 7f0f6ca00700 1 -- 172.21.15.44:0/1062339169 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f0f58018820 con 0x7f0f681045e0 2024-05-27T09:51:51.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.726+0000 7f0f6ca00700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f4c0b29f0 0x7f0f4c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:51.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.726+0000 7f0f6ca00700 1 -- 172.21.15.44:0/1062339169 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8607+0+0 (secure 0 0 0) 0x7f0f580e2740 con 0x7f0f681045e0 2024-05-27T09:51:51.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.726+0000 7f0f5e000700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f4c0b29f0 0x7f0f4c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:51.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.727+0000 7f0f5e000700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f4c0b29f0 0x7f0f4c0b4f00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0f68103a30 tx=0x7f0f50009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:51.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.728+0000 7f0f6ca00700 1 -- 172.21.15.44:0/1062339169 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0f580aa180 con 0x7f0f681045e0 2024-05-27T09:51:51.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.989+0000 7f0f6de00700 1 -- 172.21.15.44:0/1062339169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0f44005650 con 0x7f0f681045e0 2024-05-27T09:51:51.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.990+0000 7f0f6ca00700 1 -- 172.21.15.44:0/1062339169 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 18 entries v105) ==== 81+0+1068 (secure 0 0 0) 0x7f0f580b2070 con 0x7f0f681045e0 2024-05-27T09:51:51.992 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/4254617371 2024-05-28T09:22:56.451569+0000 2024-05-27T09:51:51.993 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3022591733 2024-05-28T09:22:20.128618+0000 2024-05-27T09:51:51.993 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/2049000735 2024-05-28T09:22:20.128618+0000 2024-05-27T09:51:51.993 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/4077502151 2024-05-28T09:35:14.029409+0000 2024-05-27T09:51:51.993 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/340631364 2024-05-28T09:22:20.128618+0000 2024-05-27T09:51:51.993 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2194243272 2024-05-28T09:22:20.128618+0000 2024-05-27T09:51:51.994 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/2049000735 2024-05-28T09:22:20.128618+0000 2024-05-27T09:51:51.994 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/4254617371 2024-05-28T09:22:56.451569+0000 2024-05-27T09:51:51.994 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2234109234 2024-05-28T09:22:56.451569+0000 2024-05-27T09:51:51.995 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3716545176 2024-05-28T09:35:14.029409+0000 2024-05-27T09:51:51.995 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3788109140 2024-05-28T09:22:56.451569+0000 2024-05-27T09:51:51.995 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.189:6838/4109192600 2024-05-28T09:47:25.864371+0000 2024-05-27T09:51:51.995 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1366147831 2024-05-28T09:22:56.451569+0000 2024-05-27T09:51:51.995 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/408258823 2024-05-28T09:35:14.029409+0000 2024-05-27T09:51:51.995 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/4102964244 2024-05-28T09:35:14.029409+0000 2024-05-27T09:51:51.996 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.189:6839/4109192600 2024-05-28T09:47:25.864371+0000 2024-05-27T09:51:51.996 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2278564126 2024-05-28T09:35:14.029409+0000 2024-05-27T09:51:51.997 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/4077502151 2024-05-28T09:35:14.029409+0000 2024-05-27T09:51:51.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 -- 172.21.15.44:0/1062339169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f4c0b29f0 msgr2=0x7f0f4c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:51.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f4c0b29f0 0x7f0f4c0b4f00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0f68103a30 tx=0x7f0f50009260 comp rx=0 tx=0).stop 2024-05-27T09:51:51.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 -- 172.21.15.44:0/1062339169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f681045e0 msgr2=0x7f0f6819adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:52.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f681045e0 0x7f0f6819adf0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0f5800e9a0 tx=0x7f0f5800ecb0 comp rx=0 tx=0).stop 2024-05-27T09:51:52.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 -- 172.21.15.44:0/1062339169 shutdown_connections 2024-05-27T09:51:52.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f0f4c0b29f0 0x7f0f4c0b4f00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:52.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68109860 0x7f0f6819f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:52.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f681045e0 0x7f0f6819adf0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:52.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 --2- 172.21.15.44:0/1062339169 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f68108f10 0x7f0f6819b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:52.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 -- 172.21.15.44:0/1062339169 >> 172.21.15.44:0/1062339169 conn(0x7f0f68100030 msgr2=0x7f0f68108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:52.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 -- 172.21.15.44:0/1062339169 shutdown_connections 2024-05-27T09:51:52.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:51.994+0000 7f0f6de00700 1 -- 172.21.15.44:0/1062339169 wait complete. 2024-05-27T09:51:52.001 INFO:teuthology.orchestra.run.smithi044.stderr:listed 18 entries 2024-05-27T09:51:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:52 smithi189 ceph-mon[36671]: pgmap v524: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T09:51:52.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:52 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1062339169' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T09:51:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:52 smithi044 ceph-mon[31618]: pgmap v524: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T09:51:52.381 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:52 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1062339169' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T09:51:52.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:52 smithi050 ceph-mon[36371]: pgmap v524: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T09:51:52.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:52 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1062339169' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T09:51:52.815 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-27T09:51:52.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:51:52.816 DEBUG:teuthology.orchestra.run.smithi044:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-27T09:51:52.957 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-05-27T09:51:52.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:51:52.957 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-27T09:51:53.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:51:53.062 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-05-27T09:51:53.196 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-27T09:51:53.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:51:53.198 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T09:51:53.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T09:51:53.271 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link delete ceph-brx 2024-05-27T09:51:53.371 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-27T09:51:53.383 INFO:tasks.cephadm:Teardown begin 2024-05-27T09:51:53.383 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-27T09:51:53.383 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:51:53.834 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:51:54.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:54 smithi044 ceph-mon[31618]: pgmap v525: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:51:54.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:54 smithi189 ceph-mon[36671]: pgmap v525: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:51:54.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:54 smithi050 ceph-mon[36371]: pgmap v525: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T09:51:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.048+0000 7fb97ec00700 1 -- 172.21.15.44:0/606981551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978108ef0 msgr2=0x7fb978109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.048+0000 7fb97ec00700 1 --2- 172.21.15.44:0/606981551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978108ef0 0x7fb978109300 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb9640099c0 tx=0x7fb964009cd0 comp rx=0 tx=0).stop 2024-05-27T09:51:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.048+0000 7fb97ec00700 1 -- 172.21.15.44:0/606981551 shutdown_connections 2024-05-27T09:51:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.048+0000 7fb97ec00700 1 --2- 172.21.15.44:0/606981551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb978109840 0x7fb9781111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.048+0000 7fb97ec00700 1 --2- 172.21.15.44:0/606981551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb9781045c0 0x7fb9781049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.048+0000 7fb97ec00700 1 --2- 172.21.15.44:0/606981551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978108ef0 0x7fb978109300 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.048+0000 7fb97ec00700 1 -- 172.21.15.44:0/606981551 >> 172.21.15.44:0/606981551 conn(0x7fb9780ffe90 msgr2=0x7fb9781022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.048+0000 7fb97ec00700 1 -- 172.21.15.44:0/606981551 shutdown_connections 2024-05-27T09:51:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.048+0000 7fb97ec00700 1 -- 172.21.15.44:0/606981551 wait complete. 2024-05-27T09:51:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.050+0000 7fb97ec00700 1 Processor -- start 2024-05-27T09:51:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.050+0000 7fb97ec00700 1 -- start start 2024-05-27T09:51:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.051+0000 7fb97ec00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9781045c0 0x7fb97819d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:56.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.051+0000 7fb97ec00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978108ef0 0x7fb97819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:56.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.051+0000 7fb97ec00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb978109840 0x7fb9781a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:56.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.051+0000 7fb97ec00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb9781144a0 con 0x7fb978109840 2024-05-27T09:51:56.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.051+0000 7fb97ec00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb978114320 con 0x7fb9781045c0 2024-05-27T09:51:56.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.051+0000 7fb97ec00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb978114620 con 0x7fb978108ef0 2024-05-27T09:51:56.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.051+0000 7fb977e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9781045c0 0x7fb97819d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:56.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.051+0000 7fb977e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9781045c0 0x7fb97819d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36544/0 (socket says 172.21.15.44:36544) 2024-05-27T09:51:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.051+0000 7fb977e00700 1 -- 172.21.15.44:0/1881548004 learned_addr learned my addr 172.21.15.44:0/1881548004 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:51:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.051+0000 7fb97cc00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb978109840 0x7fb9781a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.051+0000 7fb977400700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978108ef0 0x7fb97819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.052+0000 7fb977e00700 1 -- 172.21.15.44:0/1881548004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978108ef0 msgr2=0x7fb97819d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.052+0000 7fb977e00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978108ef0 0x7fb97819d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.052+0000 7fb977e00700 1 -- 172.21.15.44:0/1881548004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb978109840 msgr2=0x7fb9781a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.052+0000 7fb977e00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb978109840 0x7fb9781a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:56.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.052+0000 7fb977e00700 1 -- 172.21.15.44:0/1881548004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9781a1f20 con 0x7fb9781045c0 2024-05-27T09:51:56.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.052+0000 7fb97cc00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb978109840 0x7fb9781a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:51:56.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.052+0000 7fb977e00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9781045c0 0x7fb97819d050 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb96800e9a0 tx=0x7fb96800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:56.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.052+0000 7fb974c00700 1 -- 172.21.15.44:0/1881548004 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96800cbd0 con 0x7fb9781045c0 2024-05-27T09:51:56.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.052+0000 7fb97ec00700 1 -- 172.21.15.44:0/1881548004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9781a2210 con 0x7fb9781045c0 2024-05-27T09:51:56.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.053+0000 7fb974c00700 1 -- 172.21.15.44:0/1881548004 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb96800cd70 con 0x7fb9781045c0 2024-05-27T09:51:56.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.053+0000 7fb974c00700 1 -- 172.21.15.44:0/1881548004 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968018680 con 0x7fb9781045c0 2024-05-27T09:51:56.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.053+0000 7fb97ec00700 1 -- 172.21.15.44:0/1881548004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9781a9b30 con 0x7fb9781045c0 2024-05-27T09:51:56.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.054+0000 7fb97ec00700 1 -- 172.21.15.44:0/1881548004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9780512b0 con 0x7fb9781045c0 2024-05-27T09:51:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.057+0000 7fb974c00700 1 -- 172.21.15.44:0/1881548004 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fb968010430 con 0x7fb9781045c0 2024-05-27T09:51:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.058+0000 7fb974c00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb95c0b29f0 0x7fb95c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:56.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.058+0000 7fb974c00700 1 -- 172.21.15.44:0/1881548004 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8607+0+0 (secure 0 0 0) 0x7fb9680e2ba0 con 0x7fb9781045c0 2024-05-27T09:51:56.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.058+0000 7fb977400700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb95c0b29f0 0x7fb95c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:56.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.059+0000 7fb977400700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb95c0b29f0 0x7fb95c0b4f00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb964000c00 tx=0x7fb96401a040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.060+0000 7fb974c00700 1 -- 172.21.15.44:0/1881548004 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb9680aa4d0 con 0x7fb9781045c0 2024-05-27T09:51:56.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.352+0000 7fb97ec00700 1 -- 172.21.15.44:0/1881548004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9781a9d00 con 0x7fb9781045c0 2024-05-27T09:51:56.353 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:56 smithi044 ceph-mon[31618]: pgmap v526: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.354+0000 7fb974c00700 1 -- 172.21.15.44:0/1881548004 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11868 (secure 0 0 0) 0x7fb9680a9dd0 con 0x7fb9781045c0 2024-05-27T09:51:56.355 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:51:56.356 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":44,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:51:36.888303+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24503,24520,24542]},"id":1}]} 2024-05-27T09:51:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.357+0000 7fb97ec00700 1 -- 172.21.15.44:0/1881548004 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb95c0b29f0 msgr2=0x7fb95c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.357+0000 7fb97ec00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb95c0b29f0 0x7fb95c0b4f00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb964000c00 tx=0x7fb96401a040 comp rx=0 tx=0).stop 2024-05-27T09:51:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.357+0000 7fb97ec00700 1 -- 172.21.15.44:0/1881548004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9781045c0 msgr2=0x7fb97819d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.357+0000 7fb97ec00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9781045c0 0x7fb97819d050 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb96800e9a0 tx=0x7fb96800ecb0 comp rx=0 tx=0).stop 2024-05-27T09:51:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.357+0000 7fb97ec00700 1 -- 172.21.15.44:0/1881548004 shutdown_connections 2024-05-27T09:51:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.357+0000 7fb97ec00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fb95c0b29f0 0x7fb95c0b4f00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.357+0000 7fb97ec00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb978109840 0x7fb9781a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.357+0000 7fb97ec00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9781045c0 0x7fb97819d050 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.357+0000 7fb97ec00700 1 --2- 172.21.15.44:0/1881548004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978108ef0 0x7fb97819d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.357+0000 7fb97ec00700 1 -- 172.21.15.44:0/1881548004 >> 172.21.15.44:0/1881548004 conn(0x7fb9780ffe90 msgr2=0x7fb97810a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.358+0000 7fb97ec00700 1 -- 172.21.15.44:0/1881548004 shutdown_connections 2024-05-27T09:51:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:56.358+0000 7fb97ec00700 1 -- 172.21.15.44:0/1881548004 wait complete. 2024-05-27T09:51:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 44 2024-05-27T09:51:56.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:56 smithi050 ceph-mon[36371]: pgmap v526: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:56.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:56 smithi189 ceph-mon[36671]: pgmap v526: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T09:51:57.113 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T09:51:57.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:57 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1881548004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:51:57.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:57 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1881548004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:51:57.583 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:51:57.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:57 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1881548004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:51:58.464 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:51:58 smithi044 ceph-mon[31618]: pgmap v527: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 8 op/s 2024-05-27T09:51:58.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:51:58 smithi050 ceph-mon[36371]: pgmap v527: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 8 op/s 2024-05-27T09:51:58.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:51:58 smithi189 ceph-mon[36671]: pgmap v527: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 8 op/s 2024-05-27T09:51:59.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.620+0000 7fc596e00700 1 -- 172.21.15.44:0/2549040204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc590105dc0 msgr2=0x7fc59010e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:59.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.620+0000 7fc596e00700 1 --2- 172.21.15.44:0/2549040204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc590105dc0 0x7fc59010e310 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009990 tx=0x7fc57c009ca0 comp rx=0 tx=0).stop 2024-05-27T09:51:59.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.620+0000 7fc596e00700 1 -- 172.21.15.44:0/2549040204 shutdown_connections 2024-05-27T09:51:59.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.620+0000 7fc596e00700 1 --2- 172.21.15.44:0/2549040204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc59010e8e0 0x7fc590110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:59.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.620+0000 7fc596e00700 1 --2- 172.21.15.44:0/2549040204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc590105470 0x7fc590105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:59.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.620+0000 7fc596e00700 1 --2- 172.21.15.44:0/2549040204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc590105dc0 0x7fc59010e310 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:59.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.620+0000 7fc596e00700 1 -- 172.21.15.44:0/2549040204 >> 172.21.15.44:0/2549040204 conn(0x7fc5900fef20 msgr2=0x7fc590101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:59.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.621+0000 7fc596e00700 1 -- 172.21.15.44:0/2549040204 shutdown_connections 2024-05-27T09:51:59.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.621+0000 7fc596e00700 1 -- 172.21.15.44:0/2549040204 wait complete. 2024-05-27T09:51:59.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.622+0000 7fc596e00700 1 Processor -- start 2024-05-27T09:51:59.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.622+0000 7fc596e00700 1 -- start start 2024-05-27T09:51:59.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.623+0000 7fc596e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc590105470 0x7fc59019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:59.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.623+0000 7fc596e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc590105dc0 0x7fc59019d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:59.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.623+0000 7fc596e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59010e8e0 0x7fc5901a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:59.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.623+0000 7fc596e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc590114440 con 0x7fc59010e8e0 2024-05-27T09:51:59.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.623+0000 7fc596e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901142c0 con 0x7fc590105dc0 2024-05-27T09:51:59.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.623+0000 7fc596e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901145c0 con 0x7fc590105470 2024-05-27T09:51:59.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.623+0000 7fc58f400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc590105dc0 0x7fc59019d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:59.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.623+0000 7fc58fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc590105470 0x7fc59019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:59.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.623+0000 7fc594e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59010e8e0 0x7fc5901a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:59.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.623+0000 7fc58f400700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc590105dc0 0x7fc59019d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:36564/0 (socket says 172.21.15.44:36564) 2024-05-27T09:51:59.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.623+0000 7fc58f400700 1 -- 172.21.15.44:0/2261383430 learned_addr learned my addr 172.21.15.44:0/2261383430 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:51:59.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.624+0000 7fc58f400700 1 -- 172.21.15.44:0/2261383430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc590105470 msgr2=0x7fc59019cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:59.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.624+0000 7fc58f400700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc590105470 0x7fc59019cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:59.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.624+0000 7fc58f400700 1 -- 172.21.15.44:0/2261383430 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59010e8e0 msgr2=0x7fc5901a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:59.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.624+0000 7fc58f400700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59010e8e0 0x7fc5901a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:59.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.624+0000 7fc58f400700 1 -- 172.21.15.44:0/2261383430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5901a1ea0 con 0x7fc590105dc0 2024-05-27T09:51:59.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.624+0000 7fc58fe00700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc590105470 0x7fc59019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:51:59.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.624+0000 7fc58f400700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc590105dc0 0x7fc59019d570 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc57c0057c0 tx=0x7fc57c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:59.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.624+0000 7fc58cc00700 1 -- 172.21.15.44:0/2261383430 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c027070 con 0x7fc590105dc0 2024-05-27T09:51:59.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.625+0000 7fc58cc00700 1 -- 172.21.15.44:0/2261383430 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc57c022470 con 0x7fc590105dc0 2024-05-27T09:51:59.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.625+0000 7fc596e00700 1 -- 172.21.15.44:0/2261383430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5901a2130 con 0x7fc590105dc0 2024-05-27T09:51:59.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.625+0000 7fc58cc00700 1 -- 172.21.15.44:0/2261383430 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c017650 con 0x7fc590105dc0 2024-05-27T09:51:59.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.625+0000 7fc596e00700 1 -- 172.21.15.44:0/2261383430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5901a9ab0 con 0x7fc590105dc0 2024-05-27T09:51:59.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.626+0000 7fc596e00700 1 -- 172.21.15.44:0/2261383430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5900512b0 con 0x7fc590105dc0 2024-05-27T09:51:59.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.630+0000 7fc58cc00700 1 -- 172.21.15.44:0/2261383430 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc57c02a070 con 0x7fc590105dc0 2024-05-27T09:51:59.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.631+0000 7fc58cc00700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc5740b2a70 0x7fc5740b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:51:59.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.631+0000 7fc58cc00700 1 -- 172.21.15.44:0/2261383430 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8607+0+0 (secure 0 0 0) 0x7fc57c0e3840 con 0x7fc590105dc0 2024-05-27T09:51:59.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.632+0000 7fc58fe00700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc5740b2a70 0x7fc5740b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:51:59.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.632+0000 7fc58fe00700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc5740b2a70 0x7fc5740b4f80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc580011490 tx=0x7fc580009260 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:51:59.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.633+0000 7fc58cc00700 1 -- 172.21.15.44:0/2261383430 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc57c013070 con 0x7fc590105dc0 2024-05-27T09:51:59.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.919+0000 7fc596e00700 1 -- 172.21.15.44:0/2261383430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc59019e330 con 0x7fc590105dc0 2024-05-27T09:51:59.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.920+0000 7fc58cc00700 1 -- 172.21.15.44:0/2261383430 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11868 (secure 0 0 0) 0x7fc59019e330 con 0x7fc590105dc0 2024-05-27T09:51:59.921 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:51:59.921 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14367,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/907929206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":907929206},{"type":"v1","addr":"172.21.15.44:6835","nonce":907929206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3467964892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3467964892},{"type":"v1","addr":"172.21.15.44:6837","nonce":3467964892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/285214695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":285214695},{"type":"v1","addr":"172.21.15.44:6839","nonce":285214695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/3128909996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":3128909996},{"type":"v1","addr":"172.21.15.44:6841","nonce":3128909996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4273379740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4273379740},{"type":"v1","addr":"172.21.15.50:6833","nonce":4273379740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/742901973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":742901973},{"type":"v1","addr":"172.21.15.189:6833","nonce":742901973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":44,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T09:31:27.747151+0000","modified":"2024-05-27T09:51:36.888303+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24542,"mds_1":24503,"mds_2":24520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/1092513236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1092513236},{"type":"v1","addr":"172.21.15.50:6835","nonce":1092513236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":36,"state":"up:active","state_seq":253,"addr":"172.21.15.50:6837/3540954143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3540954143},{"type":"v1","addr":"172.21.15.50:6837","nonce":3540954143}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6837/2265427173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2265427173},{"type":"v1","addr":"172.21.15.189:6837","nonce":2265427173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6839/1124366265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":1124366265},{"type":"v1","addr":"172.21.15.50:6839","nonce":1124366265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26833":{"gid":26833,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/3188237380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3188237380},{"type":"v1","addr":"172.21.15.189:6835","nonce":3188237380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27484":{"gid":27484,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6839/2373131774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2373131774},{"type":"v1","addr":"172.21.15.189:6839","nonce":2373131774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24542,"qdb_cluster":[24503,24520,24542]},"id":1}]} 2024-05-27T09:51:59.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.923+0000 7fc596e00700 1 -- 172.21.15.44:0/2261383430 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc5740b2a70 msgr2=0x7fc5740b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:59.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.923+0000 7fc596e00700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc5740b2a70 0x7fc5740b4f80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc580011490 tx=0x7fc580009260 comp rx=0 tx=0).stop 2024-05-27T09:51:59.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.923+0000 7fc596e00700 1 -- 172.21.15.44:0/2261383430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc590105dc0 msgr2=0x7fc59019d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:51:59.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.924+0000 7fc596e00700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc590105dc0 0x7fc59019d570 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc57c0057c0 tx=0x7fc57c0057f0 comp rx=0 tx=0).stop 2024-05-27T09:51:59.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.924+0000 7fc596e00700 1 -- 172.21.15.44:0/2261383430 shutdown_connections 2024-05-27T09:51:59.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.924+0000 7fc596e00700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc5740b2a70 0x7fc5740b4f80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:59.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.924+0000 7fc596e00700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59010e8e0 0x7fc5901a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:59.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.924+0000 7fc596e00700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc590105470 0x7fc59019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:59.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.924+0000 7fc596e00700 1 --2- 172.21.15.44:0/2261383430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc590105dc0 0x7fc59019d570 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:51:59.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.924+0000 7fc596e00700 1 -- 172.21.15.44:0/2261383430 >> 172.21.15.44:0/2261383430 conn(0x7fc5900fef20 msgr2=0x7fc5901030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:51:59.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.924+0000 7fc596e00700 1 -- 172.21.15.44:0/2261383430 shutdown_connections 2024-05-27T09:51:59.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:51:59.924+0000 7fc596e00700 1 -- 172.21.15.44:0/2261383430 wait complete. 2024-05-27T09:51:59.926 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 44 2024-05-27T09:52:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:00 smithi044 ceph-mon[31618]: pgmap v528: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-27T09:52:00.224 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:00 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2261383430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:52:00.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:00 smithi050 ceph-mon[36371]: pgmap v528: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-27T09:52:00.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:00 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2261383430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:52:00.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:00 smithi189 ceph-mon[36671]: pgmap v528: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-27T09:52:00.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:00 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2261383430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T09:52:00.839 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T09:52:01.327 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:52:02.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:02 smithi044 ceph-mon[31618]: pgmap v529: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T09:52:02.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:02 smithi050 ceph-mon[36371]: pgmap v529: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T09:52:02.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:02 smithi189 ceph-mon[36671]: pgmap v529: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T09:52:03.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.542+0000 7fc718e00700 1 -- 172.21.15.44:0/4055927775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714076670 msgr2=0x7fc714110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:03.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.542+0000 7fc718e00700 1 --2- 172.21.15.44:0/4055927775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714076670 0x7fc714110650 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc7000098f0 tx=0x7fc700009c00 comp rx=0 tx=0).stop 2024-05-27T09:52:03.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.542+0000 7fc718e00700 1 -- 172.21.15.44:0/4055927775 shutdown_connections 2024-05-27T09:52:03.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.542+0000 7fc718e00700 1 --2- 172.21.15.44:0/4055927775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc714110b90 0x7fc714112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:03.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.542+0000 7fc718e00700 1 --2- 172.21.15.44:0/4055927775 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc714075d20 0x7fc714076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:03.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.542+0000 7fc718e00700 1 --2- 172.21.15.44:0/4055927775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714076670 0x7fc714110650 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:03.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.542+0000 7fc718e00700 1 -- 172.21.15.44:0/4055927775 >> 172.21.15.44:0/4055927775 conn(0x7fc714100030 msgr2=0x7fc714102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:52:03.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.542+0000 7fc718e00700 1 -- 172.21.15.44:0/4055927775 shutdown_connections 2024-05-27T09:52:03.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.542+0000 7fc718e00700 1 -- 172.21.15.44:0/4055927775 wait complete. 2024-05-27T09:52:03.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.544+0000 7fc718e00700 1 Processor -- start 2024-05-27T09:52:03.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.544+0000 7fc718e00700 1 -- start start 2024-05-27T09:52:03.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.545+0000 7fc718e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714075d20 0x7fc71419d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:03.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.545+0000 7fc718e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc714076670 0x7fc71419d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:03.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.545+0000 7fc718e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc714110b90 0x7fc7141a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:03.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.545+0000 7fc718e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc714116690 con 0x7fc714076670 2024-05-27T09:52:03.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.545+0000 7fc718e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc714116510 con 0x7fc714110b90 2024-05-27T09:52:03.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.545+0000 7fc718e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc714116810 con 0x7fc714075d20 2024-05-27T09:52:03.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.545+0000 7fc711e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714075d20 0x7fc71419d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:03.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.545+0000 7fc711e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714075d20 0x7fc71419d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:40974/0 (socket says 172.21.15.44:40974) 2024-05-27T09:52:03.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.545+0000 7fc711400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc714076670 0x7fc71419d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:03.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.545+0000 7fc711e00700 1 -- 172.21.15.44:0/1969243995 learned_addr learned my addr 172.21.15.44:0/1969243995 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:52:03.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.545+0000 7fc712800700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc714110b90 0x7fc7141a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:03.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.546+0000 7fc711e00700 1 -- 172.21.15.44:0/1969243995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc714110b90 msgr2=0x7fc7141a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:03.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.546+0000 7fc711e00700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc714110b90 0x7fc7141a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:03.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.546+0000 7fc711e00700 1 -- 172.21.15.44:0/1969243995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc714076670 msgr2=0x7fc71419d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:03.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.546+0000 7fc711e00700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc714076670 0x7fc71419d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:03.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.546+0000 7fc711e00700 1 -- 172.21.15.44:0/1969243995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7141a1f50 con 0x7fc714075d20 2024-05-27T09:52:03.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.546+0000 7fc711e00700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714075d20 0x7fc71419d080 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc70400e9a0 tx=0x7fc70400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:52:03.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.546+0000 7fc70aa00700 1 -- 172.21.15.44:0/1969243995 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70400cbd0 con 0x7fc714075d20 2024-05-27T09:52:03.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.547+0000 7fc718e00700 1 -- 172.21.15.44:0/1969243995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7141a2240 con 0x7fc714075d20 2024-05-27T09:52:03.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.547+0000 7fc70aa00700 1 -- 172.21.15.44:0/1969243995 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc70400cd70 con 0x7fc714075d20 2024-05-27T09:52:03.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.547+0000 7fc70aa00700 1 -- 172.21.15.44:0/1969243995 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704018680 con 0x7fc714075d20 2024-05-27T09:52:03.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.547+0000 7fc718e00700 1 -- 172.21.15.44:0/1969243995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7141a9b60 con 0x7fc714075d20 2024-05-27T09:52:03.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.548+0000 7fc718e00700 1 -- 172.21.15.44:0/1969243995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7140512b0 con 0x7fc714075d20 2024-05-27T09:52:03.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.551+0000 7fc70aa00700 1 -- 172.21.15.44:0/1969243995 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fc704010430 con 0x7fc714075d20 2024-05-27T09:52:03.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.552+0000 7fc70aa00700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc6f80b29f0 0x7fc6f80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:03.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.553+0000 7fc70aa00700 1 -- 172.21.15.44:0/1969243995 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8607+0+0 (secure 0 0 0) 0x7fc7040e2ba0 con 0x7fc714075d20 2024-05-27T09:52:03.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.553+0000 7fc711400700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc6f80b29f0 0x7fc6f80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:03.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.553+0000 7fc711400700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc6f80b29f0 0x7fc6f80b4f00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc700009fd0 tx=0x7fc700019040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:52:03.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.554+0000 7fc70aa00700 1 -- 172.21.15.44:0/1969243995 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc7040aa4d0 con 0x7fc714075d20 2024-05-27T09:52:03.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.797+0000 7fc718e00700 1 -- 172.21.15.44:0/1969243995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc71402e5e0 con 0x7fc714075d20 2024-05-27T09:52:03.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.799+0000 7fc70aa00700 1 -- 172.21.15.44:0/1969243995 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+23214 (secure 0 0 0) 0x7fc7040a9dd0 con 0x7fc714075d20 2024-05-27T09:52:03.801 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-05-27T09:52:03.801 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":105,"fsid":"604b015a-1c0a-11ef-bc9b-c7b262605968","created":"2024-05-27T09:21:36.258511+0000","modified":"2024-05-27T09:51:38.027909+0000","last_up_change":"2024-05-27T09:30:09.010194+0000","last_in_change":"2024-05-27T09:29:48.273873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T09:25:46.636320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T09:31:19.459994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T09:31:23.749576+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8959d30f-ab21-41e8-b900-1f5ef3426892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6803","nonce":3103307534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6805","nonce":3103307534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6809","nonce":3103307534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3103307534},{"type":"v1","addr":"172.21.15.44:6807","nonce":3103307534}]},"public_addr":"172.21.15.44:6803/3103307534","cluster_addr":"172.21.15.44:6805/3103307534","heartbeat_back_addr":"172.21.15.44:6809/3103307534","heartbeat_front_addr":"172.21.15.44:6807/3103307534","state":["exists","up"]},{"osd":1,"uuid":"55f7adef-a968-40d3-9722-d17f5d4f9670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6801","nonce":1520188354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6803","nonce":1520188354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6807","nonce":1520188354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1520188354},{"type":"v1","addr":"172.21.15.50:6805","nonce":1520188354}]},"public_addr":"172.21.15.50:6801/1520188354","cluster_addr":"172.21.15.50:6803/1520188354","heartbeat_back_addr":"172.21.15.50:6807/1520188354","heartbeat_front_addr":"172.21.15.50:6805/1520188354","state":["exists","up"]},{"osd":2,"uuid":"6b3e59b2-65cf-40df-ab3c-eee28b4157d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6801","nonce":1488616374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6803","nonce":1488616374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6807","nonce":1488616374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1488616374},{"type":"v1","addr":"172.21.15.189:6805","nonce":1488616374}]},"public_addr":"172.21.15.189:6801/1488616374","cluster_addr":"172.21.15.189:6803/1488616374","heartbeat_back_addr":"172.21.15.189:6807/1488616374","heartbeat_front_addr":"172.21.15.189:6805/1488616374","state":["exists","up"]},{"osd":3,"uuid":"4c9928a4-b7d8-436e-893b-5146bbbd7f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6811","nonce":2748877697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6813","nonce":2748877697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6817","nonce":2748877697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2748877697},{"type":"v1","addr":"172.21.15.44:6815","nonce":2748877697}]},"public_addr":"172.21.15.44:6811/2748877697","cluster_addr":"172.21.15.44:6813/2748877697","heartbeat_back_addr":"172.21.15.44:6817/2748877697","heartbeat_front_addr":"172.21.15.44:6815/2748877697","state":["exists","up"]},{"osd":4,"uuid":"88aa8a99-e01b-45d4-bac3-df51aaf8b94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6809","nonce":299847660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6811","nonce":299847660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6815","nonce":299847660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":299847660},{"type":"v1","addr":"172.21.15.50:6813","nonce":299847660}]},"public_addr":"172.21.15.50:6809/299847660","cluster_addr":"172.21.15.50:6811/299847660","heartbeat_back_addr":"172.21.15.50:6815/299847660","heartbeat_front_addr":"172.21.15.50:6813/299847660","state":["exists","up"]},{"osd":5,"uuid":"b075f197-da86-4cca-b088-d6912121fbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6809","nonce":2756976881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6811","nonce":2756976881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6815","nonce":2756976881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2756976881},{"type":"v1","addr":"172.21.15.189:6813","nonce":2756976881}]},"public_addr":"172.21.15.189:6809/2756976881","cluster_addr":"172.21.15.189:6811/2756976881","heartbeat_back_addr":"172.21.15.189:6815/2756976881","heartbeat_front_addr":"172.21.15.189:6813/2756976881","state":["exists","up"]},{"osd":6,"uuid":"24e70958-59a8-415b-95f9-1b52a7dd8263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6819","nonce":44684643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6821","nonce":44684643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6825","nonce":44684643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":44684643},{"type":"v1","addr":"172.21.15.44:6823","nonce":44684643}]},"public_addr":"172.21.15.44:6819/44684643","cluster_addr":"172.21.15.44:6821/44684643","heartbeat_back_addr":"172.21.15.44:6825/44684643","heartbeat_front_addr":"172.21.15.44:6823/44684643","state":["exists","up"]},{"osd":7,"uuid":"cd1d6765-ace5-45f8-a07b-6cd583f32f7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6817","nonce":1616906446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6819","nonce":1616906446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6823","nonce":1616906446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1616906446},{"type":"v1","addr":"172.21.15.50:6821","nonce":1616906446}]},"public_addr":"172.21.15.50:6817/1616906446","cluster_addr":"172.21.15.50:6819/1616906446","heartbeat_back_addr":"172.21.15.50:6823/1616906446","heartbeat_front_addr":"172.21.15.50:6821/1616906446","state":["exists","up"]},{"osd":8,"uuid":"2602577e-c4b3-433e-a443-a71e4aa793a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6817","nonce":2379603694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6819","nonce":2379603694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6823","nonce":2379603694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2379603694},{"type":"v1","addr":"172.21.15.189:6821","nonce":2379603694}]},"public_addr":"172.21.15.189:6817/2379603694","cluster_addr":"172.21.15.189:6819/2379603694","heartbeat_back_addr":"172.21.15.189:6823/2379603694","heartbeat_front_addr":"172.21.15.189:6821/2379603694","state":["exists","up"]},{"osd":9,"uuid":"0307f939-76d2-4b3d-89ae-31581649f5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6827","nonce":503459032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6829","nonce":503459032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6833","nonce":503459032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":503459032},{"type":"v1","addr":"172.21.15.44:6831","nonce":503459032}]},"public_addr":"172.21.15.44:6827/503459032","cluster_addr":"172.21.15.44:6829/503459032","heartbeat_back_addr":"172.21.15.44:6833/503459032","heartbeat_front_addr":"172.21.15.44:6831/503459032","state":["exists","up"]},{"osd":10,"uuid":"f094a1ca-1b0a-47a4-974e-5620cd898ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6825","nonce":1371655028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6827","nonce":1371655028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6831","nonce":1371655028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1371655028},{"type":"v1","addr":"172.21.15.50:6829","nonce":1371655028}]},"public_addr":"172.21.15.50:6825/1371655028","cluster_addr":"172.21.15.50:6827/1371655028","heartbeat_back_addr":"172.21.15.50:6831/1371655028","heartbeat_front_addr":"172.21.15.50:6829/1371655028","state":["exists","up"]},{"osd":11,"uuid":"4b9794fa-856f-4343-945e-4be444c0cd9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6825","nonce":2712606633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6827","nonce":2712606633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6831","nonce":2712606633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2712606633},{"type":"v1","addr":"172.21.15.189:6829","nonce":2712606633}]},"public_addr":"172.21.15.189:6825/2712606633","cluster_addr":"172.21.15.189:6827/2712606633","heartbeat_back_addr":"172.21.15.189:6831/2712606633","heartbeat_front_addr":"172.21.15.189:6829/2712606633","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:24:53.474730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:19.395845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:25:43.572336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:18.785234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:26:45.643088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:10.403130+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:27:45.895395+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:12.677694+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:28:37.272113+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:13.755574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:29:41.168627+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T09:30:06.034237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:6800/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3022591733":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/4077502151":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/340631364":"2024-05-28T09:22:20.128618+0000","172.21.15.44:0/2194243272":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6800/2049000735":"2024-05-28T09:22:20.128618+0000","172.21.15.44:6801/4254617371":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/2234109234":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/3716545176":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/3788109140":"2024-05-28T09:22:56.451569+0000","172.21.15.189:6838/4109192600":"2024-05-28T09:47:25.864371+0000","172.21.15.44:0/1366147831":"2024-05-28T09:22:56.451569+0000","172.21.15.44:0/408258823":"2024-05-28T09:35:14.029409+0000","172.21.15.44:0/4102964244":"2024-05-28T09:35:14.029409+0000","172.21.15.189:6839/4109192600":"2024-05-28T09:47:25.864371+0000","172.21.15.44:0/2278564126":"2024-05-28T09:35:14.029409+0000","172.21.15.44:6800/4077502151":"2024-05-28T09:35:14.029409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T09:52:03.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.802+0000 7fc718e00700 1 -- 172.21.15.44:0/1969243995 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc6f80b29f0 msgr2=0x7fc6f80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:03.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.803+0000 7fc718e00700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc6f80b29f0 0x7fc6f80b4f00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc700009fd0 tx=0x7fc700019040 comp rx=0 tx=0).stop 2024-05-27T09:52:03.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.803+0000 7fc718e00700 1 -- 172.21.15.44:0/1969243995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714075d20 msgr2=0x7fc71419d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:03.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.803+0000 7fc718e00700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714075d20 0x7fc71419d080 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc70400e9a0 tx=0x7fc70400ecb0 comp rx=0 tx=0).stop 2024-05-27T09:52:03.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.803+0000 7fc718e00700 1 -- 172.21.15.44:0/1969243995 shutdown_connections 2024-05-27T09:52:03.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.803+0000 7fc718e00700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fc6f80b29f0 0x7fc6f80b4f00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:03.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.803+0000 7fc718e00700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc714110b90 0x7fc7141a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:03.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.803+0000 7fc718e00700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714075d20 0x7fc71419d080 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:03.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.803+0000 7fc718e00700 1 --2- 172.21.15.44:0/1969243995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc714076670 0x7fc71419d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:03.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.803+0000 7fc718e00700 1 -- 172.21.15.44:0/1969243995 >> 172.21.15.44:0/1969243995 conn(0x7fc714100030 msgr2=0x7fc714100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:52:03.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.803+0000 7fc718e00700 1 -- 172.21.15.44:0/1969243995 shutdown_connections 2024-05-27T09:52:03.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:03.803+0000 7fc718e00700 1 -- 172.21.15.44:0/1969243995 wait complete. 2024-05-27T09:52:04.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:04 smithi044 ceph-mon[31618]: pgmap v530: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-05-27T09:52:04.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:04 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1969243995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:52:04.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:04 smithi050 ceph-mon[36371]: pgmap v530: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-05-27T09:52:04.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:04 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1969243995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:52:04.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:04 smithi189 ceph-mon[36671]: pgmap v530: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-05-27T09:52:04.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:04 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1969243995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T09:52:04.705 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs fail cephfs 2024-05-27T09:52:05.200 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:52:06.429 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:06 smithi044 ceph-mon[31618]: pgmap v531: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-05-27T09:52:06.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:06 smithi050 ceph-mon[36371]: pgmap v531: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-05-27T09:52:06.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:06 smithi189 ceph-mon[36671]: pgmap v531: 97 pgs: 97 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-05-27T09:52:07.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.725+0000 7fd43ea00700 1 -- 172.21.15.44:0/1713365824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd438109860 msgr2=0x7fd4381111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:07.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.725+0000 7fd43ea00700 1 --2- 172.21.15.44:0/1713365824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd438109860 0x7fd4381111c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd4240099c0 tx=0x7fd424009cd0 comp rx=0 tx=0).stop 2024-05-27T09:52:07.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.725+0000 7fd43ea00700 1 -- 172.21.15.44:0/1713365824 shutdown_connections 2024-05-27T09:52:07.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.725+0000 7fd43ea00700 1 --2- 172.21.15.44:0/1713365824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd438109860 0x7fd4381111c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:07.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.725+0000 7fd43ea00700 1 --2- 172.21.15.44:0/1713365824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4381045e0 0x7fd4381049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:07.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.725+0000 7fd43ea00700 1 --2- 172.21.15.44:0/1713365824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd438108f10 0x7fd438109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:07.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.725+0000 7fd43ea00700 1 -- 172.21.15.44:0/1713365824 >> 172.21.15.44:0/1713365824 conn(0x7fd438100030 msgr2=0x7fd438102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:52:07.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.726+0000 7fd43ea00700 1 -- 172.21.15.44:0/1713365824 shutdown_connections 2024-05-27T09:52:07.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.726+0000 7fd43ea00700 1 -- 172.21.15.44:0/1713365824 wait complete. 2024-05-27T09:52:07.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.728+0000 7fd43ea00700 1 Processor -- start 2024-05-27T09:52:07.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.728+0000 7fd43ea00700 1 -- start start 2024-05-27T09:52:07.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.731+0000 7fd43ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4381045e0 0x7fd438075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:07.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.731+0000 7fd43ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd438108f10 0x7fd438075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:07.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.731+0000 7fd43ea00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd438109860 0x7fd438076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:07.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.731+0000 7fd43ea00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd438114550 con 0x7fd438108f10 2024-05-27T09:52:07.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.731+0000 7fd43ea00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd4381143d0 con 0x7fd438109860 2024-05-27T09:52:07.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.731+0000 7fd43ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd4381146d0 con 0x7fd4381045e0 2024-05-27T09:52:07.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.732+0000 7fd43ca00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd438109860 0x7fd438076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:07.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.732+0000 7fd437400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd438108f10 0x7fd438075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:07.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.732+0000 7fd43ca00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd438109860 0x7fd438076530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47130/0 (socket says 172.21.15.44:47130) 2024-05-27T09:52:07.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.732+0000 7fd43ca00700 1 -- 172.21.15.44:0/2512169438 learned_addr learned my addr 172.21.15.44:0/2512169438 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:52:07.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.732+0000 7fd437e00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4381045e0 0x7fd438075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:07.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.733+0000 7fd437400700 1 -- 172.21.15.44:0/2512169438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4381045e0 msgr2=0x7fd438075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:07.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.733+0000 7fd437400700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4381045e0 0x7fd438075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:07.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.733+0000 7fd437400700 1 -- 172.21.15.44:0/2512169438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd438109860 msgr2=0x7fd438076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:07.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.733+0000 7fd437400700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd438109860 0x7fd438076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:07.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.733+0000 7fd437400700 1 -- 172.21.15.44:0/2512169438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4381a9220 con 0x7fd438108f10 2024-05-27T09:52:07.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.733+0000 7fd437e00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4381045e0 0x7fd438075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T09:52:07.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.733+0000 7fd43ca00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd438109860 0x7fd438076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T09:52:07.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.734+0000 7fd437400700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd438108f10 0x7fd438075ff0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd43806c670 tx=0x7fd42800b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:52:07.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.734+0000 7fd434c00700 1 -- 172.21.15.44:0/2512169438 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd428015020 con 0x7fd438108f10 2024-05-27T09:52:07.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.734+0000 7fd434c00700 1 -- 172.21.15.44:0/2512169438 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd428011ce0 con 0x7fd438108f10 2024-05-27T09:52:07.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.734+0000 7fd434c00700 1 -- 172.21.15.44:0/2512169438 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd428010c60 con 0x7fd438108f10 2024-05-27T09:52:07.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.735+0000 7fd43ea00700 1 -- 172.21.15.44:0/2512169438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4381a9510 con 0x7fd438108f10 2024-05-27T09:52:07.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.737+0000 7fd43ea00700 1 -- 172.21.15.44:0/2512169438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4381a9a00 con 0x7fd438108f10 2024-05-27T09:52:07.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.738+0000 7fd43ea00700 1 -- 172.21.15.44:0/2512169438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4380512b0 con 0x7fd438108f10 2024-05-27T09:52:07.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.744+0000 7fd434c00700 1 -- 172.21.15.44:0/2512169438 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fd428018020 con 0x7fd438108f10 2024-05-27T09:52:07.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.746+0000 7fd434c00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd41c0b2bf0 0x7fd41c0b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:07.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.747+0000 7fd437e00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd41c0b2bf0 0x7fd41c0b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:07.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.748+0000 7fd434c00700 1 -- 172.21.15.44:0/2512169438 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8651+0+0 (secure 0 0 0) 0x7fd428010a00 con 0x7fd438108f10 2024-05-27T09:52:07.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.749+0000 7fd437e00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd41c0b2bf0 0x7fd41c0b5100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd4300093a0 tx=0x7fd43000b040 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:52:07.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:07.762+0000 7fd434c00700 1 -- 172.21.15.44:0/2512169438 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd42801f030 con 0x7fd438108f10 2024-05-27T09:52:08.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.289+0000 7fd43ea00700 1 -- 172.21.15.44:0/2512169438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd43802e5e0 con 0x7fd438108f10 2024-05-27T09:52:08.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.298+0000 7fd434c00700 1 -- 172.21.15.44:0/2512169438 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) ==== 161+0+0 (secure 0 0 0) 0x7fd4280a92c0 con 0x7fd438108f10 2024-05-27T09:52:08.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.302+0000 7fd43ea00700 1 -- 172.21.15.44:0/2512169438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd41c0b2bf0 msgr2=0x7fd41c0b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:08.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.302+0000 7fd43ea00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd41c0b2bf0 0x7fd41c0b5100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd4300093a0 tx=0x7fd43000b040 comp rx=0 tx=0).stop 2024-05-27T09:52:08.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.302+0000 7fd43ea00700 1 -- 172.21.15.44:0/2512169438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd438108f10 msgr2=0x7fd438075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:08.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.302+0000 7fd43ea00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd438108f10 0x7fd438075ff0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd43806c670 tx=0x7fd42800b730 comp rx=0 tx=0).stop 2024-05-27T09:52:08.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.303+0000 7fd43ea00700 1 -- 172.21.15.44:0/2512169438 shutdown_connections 2024-05-27T09:52:08.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.303+0000 7fd43ea00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fd41c0b2bf0 0x7fd41c0b5100 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:08.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.303+0000 7fd43ea00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd438109860 0x7fd438076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:08.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.303+0000 7fd43ea00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4381045e0 0x7fd438075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:08.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.303+0000 7fd43ea00700 1 --2- 172.21.15.44:0/2512169438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd438108f10 0x7fd438075ff0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:08.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.303+0000 7fd43ea00700 1 -- 172.21.15.44:0/2512169438 >> 172.21.15.44:0/2512169438 conn(0x7fd438100030 msgr2=0x7fd438108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:52:08.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.305+0000 7fd43ea00700 1 -- 172.21.15.44:0/2512169438 shutdown_connections 2024-05-27T09:52:08.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:08.305+0000 7fd43ea00700 1 -- 172.21.15.44:0/2512169438 wait complete. 2024-05-27T09:52:08.313 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-27T09:52:08.408 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:08 smithi189 ceph-mon[36671]: pgmap v532: 97 pgs: 97 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-05-27T09:52:08.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:08 smithi189 ceph-mon[36671]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T09:52:08.411 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:08 smithi050 ceph-mon[36371]: pgmap v532: 97 pgs: 97 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-05-27T09:52:08.411 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:08 smithi050 ceph-mon[36371]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T09:52:08.444 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:08 smithi044 ceph-mon[31618]: pgmap v532: 97 pgs: 97 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-05-27T09:52:08.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:08 smithi044 ceph-mon[31618]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T09:52:08.445 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:08 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a[31614]: 2024-05-27T09:52:08.293+0000 7f58c4c00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T09:52:08.794 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:52:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e[66101]: ignoring --setuser ceph since I am not root 2024-05-27T09:52:08.794 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:52:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e[66101]: ignoring --setgroup ceph since I am not root 2024-05-27T09:52:08.794 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:52:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e[66101]: starting mds.e at 2024-05-27T09:52:08.795 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:52:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h[66406]: ignoring --setuser ceph since I am not root 2024-05-27T09:52:08.795 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:52:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h[66406]: ignoring --setgroup ceph since I am not root 2024-05-27T09:52:08.795 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:52:08 smithi050 ceph-mds[66410]: -- 172.21.15.50:0/1866984580 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564320ab5a40 con 0x564320afd600 2024-05-27T09:52:08.795 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:52:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h[66406]: starting mds.h at 2024-05-27T09:52:08.795 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:52:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b[66721]: ignoring --setuser ceph since I am not root 2024-05-27T09:52:08.796 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:52:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b[66721]: ignoring --setgroup ceph since I am not root 2024-05-27T09:52:08.796 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:52:08 smithi050 ceph-mds[66725]: -- 172.21.15.50:0/2067438468 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c34e1804e0 con 0x55c34e1cb600 2024-05-27T09:52:08.796 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:52:08 smithi050 ceph-mds[66725]: -- 172.21.15.50:0/2067438468 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c34e183a40 con 0x55c34e1cb600 2024-05-27T09:52:08.796 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:52:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b[66721]: starting mds.b at 2024-05-27T09:52:08.837 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:52:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c[67552]: ignoring --setuser ceph since I am not root 2024-05-27T09:52:08.838 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:52:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c[67552]: ignoring --setgroup ceph since I am not root 2024-05-27T09:52:08.838 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:52:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c[67552]: starting mds.c at 2024-05-27T09:52:08.838 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:52:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[72375]: ignoring --setuser ceph since I am not root 2024-05-27T09:52:08.839 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:52:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[72375]: ignoring --setgroup ceph since I am not root 2024-05-27T09:52:08.839 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:52:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[72375]: starting mds.f at 2024-05-27T09:52:08.839 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:52:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[74173]: ignoring --setuser ceph since I am not root 2024-05-27T09:52:08.839 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:52:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[74173]: ignoring --setgroup ceph since I am not root 2024-05-27T09:52:08.839 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:52:08 smithi189 ceph-mds[74177]: -- 172.21.15.189:0/2636750495 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5599c1a15a40 con 0x5599c1a5d600 2024-05-27T09:52:08.839 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:52:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[74173]: starting mds.i at 2024-05-27T09:52:09.227 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-05-27T09:52:09.320 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:09 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2512169438' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T09:52:09.320 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:09 smithi044 ceph-mon[31618]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T09:52:09.320 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:09 smithi044 ceph-mon[31618]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T09:52:09.320 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:09 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/2512169438' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-27T09:52:09.320 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:09 smithi044 ceph-mon[31618]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T09:52:09.320 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:09 smithi044 ceph-mon[31618]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-27T09:52:09.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:09 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2512169438' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T09:52:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:09 smithi050 ceph-mon[36371]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T09:52:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:09 smithi050 ceph-mon[36371]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T09:52:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:09 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/2512169438' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-27T09:52:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:09 smithi050 ceph-mon[36371]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T09:52:09.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:09 smithi050 ceph-mon[36371]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-27T09:52:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:09 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2512169438' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T09:52:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:09 smithi189 ceph-mon[36671]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T09:52:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:09 smithi189 ceph-mon[36671]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T09:52:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:09 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/2512169438' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-27T09:52:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:09 smithi189 ceph-mon[36671]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T09:52:09.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:09 smithi189 ceph-mon[36671]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-27T09:52:09.905 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:52:10.448 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: pgmap v535: 97 pgs: 97 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 34 KiB/s wr, 7 op/s 2024-05-27T09:52:10.449 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6834/924573183,v1:172.21.15.50:6835/924573183] up:boot 2024-05-27T09:52:10.449 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6836/2476201943,v1:172.21.15.50:6837/2476201943] up:boot 2024-05-27T09:52:10.449 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6836/4035715790,v1:172.21.15.189:6837/4035715790] up:boot 2024-05-27T09:52:10.449 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6834/770771025,v1:172.21.15.189:6835/770771025] up:boot 2024-05-27T09:52:10.449 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T09:52:10.449 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.189:6838/2266824431,v1:172.21.15.189:6839/2266824431] up:boot 2024-05-27T09:52:10.450 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: mds.? [v2:172.21.15.50:6838/1677433023,v1:172.21.15.50:6839/1677433023] up:boot 2024-05-27T09:52:10.450 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-05-27T09:52:10.450 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:52:10.450 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T09:52:10.450 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:52:10.450 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:52:10.450 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:10 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T09:52:10.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: pgmap v535: 97 pgs: 97 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 34 KiB/s wr, 7 op/s 2024-05-27T09:52:10.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6834/924573183,v1:172.21.15.50:6835/924573183] up:boot 2024-05-27T09:52:10.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6836/2476201943,v1:172.21.15.50:6837/2476201943] up:boot 2024-05-27T09:52:10.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6836/4035715790,v1:172.21.15.189:6837/4035715790] up:boot 2024-05-27T09:52:10.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6834/770771025,v1:172.21.15.189:6835/770771025] up:boot 2024-05-27T09:52:10.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T09:52:10.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.189:6838/2266824431,v1:172.21.15.189:6839/2266824431] up:boot 2024-05-27T09:52:10.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: mds.? [v2:172.21.15.50:6838/1677433023,v1:172.21.15.50:6839/1677433023] up:boot 2024-05-27T09:52:10.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-05-27T09:52:10.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:52:10.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T09:52:10.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:52:10.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:52:10.545 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:10 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T09:52:10.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: pgmap v535: 97 pgs: 97 active+clean; 342 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 34 KiB/s wr, 7 op/s 2024-05-27T09:52:10.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6834/924573183,v1:172.21.15.50:6835/924573183] up:boot 2024-05-27T09:52:10.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6836/2476201943,v1:172.21.15.50:6837/2476201943] up:boot 2024-05-27T09:52:10.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6836/4035715790,v1:172.21.15.189:6837/4035715790] up:boot 2024-05-27T09:52:10.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6834/770771025,v1:172.21.15.189:6835/770771025] up:boot 2024-05-27T09:52:10.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T09:52:10.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.189:6838/2266824431,v1:172.21.15.189:6839/2266824431] up:boot 2024-05-27T09:52:10.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: mds.? [v2:172.21.15.50:6838/1677433023,v1:172.21.15.50:6839/1677433023] up:boot 2024-05-27T09:52:10.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-05-27T09:52:10.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T09:52:10.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T09:52:10.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T09:52:10.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T09:52:10.589 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:10 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T09:52:12.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:12 smithi044 ceph-mon[31618]: pgmap v536: 97 pgs: 20 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 40 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 33 KiB/s wr, 8 op/s 2024-05-27T09:52:12.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:12 smithi050 ceph-mon[36371]: pgmap v536: 97 pgs: 20 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 40 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 33 KiB/s wr, 8 op/s 2024-05-27T09:52:12.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:12 smithi189 ceph-mon[36671]: pgmap v536: 97 pgs: 20 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 40 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 33 KiB/s wr, 8 op/s 2024-05-27T09:52:12.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.803+0000 7f5b88800700 1 -- 172.21.15.44:0/1426025138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b80103b90 msgr2=0x7f5b801080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.803+0000 7f5b88800700 1 --2- 172.21.15.44:0/1426025138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b80103b90 0x7f5b801080c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009990 tx=0x7f5b70009ca0 comp rx=0 tx=0).stop 2024-05-27T09:52:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.803+0000 7f5b88800700 1 -- 172.21.15.44:0/1426025138 shutdown_connections 2024-05-27T09:52:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.803+0000 7f5b88800700 1 --2- 172.21.15.44:0/1426025138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b80108690 0x7f5b80108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.803+0000 7f5b88800700 1 --2- 172.21.15.44:0/1426025138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b80103240 0x7f5b80103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.803+0000 7f5b88800700 1 --2- 172.21.15.44:0/1426025138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b80103b90 0x7f5b801080c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.803+0000 7f5b88800700 1 -- 172.21.15.44:0/1426025138 >> 172.21.15.44:0/1426025138 conn(0x7f5b800fedb0 msgr2=0x7f5b801011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:52:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.803+0000 7f5b88800700 1 -- 172.21.15.44:0/1426025138 shutdown_connections 2024-05-27T09:52:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.803+0000 7f5b88800700 1 -- 172.21.15.44:0/1426025138 wait complete. 2024-05-27T09:52:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.805+0000 7f5b88800700 1 Processor -- start 2024-05-27T09:52:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.807+0000 7f5b88800700 1 -- start start 2024-05-27T09:52:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.807+0000 7f5b88800700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b80103240 0x7f5b801a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.807+0000 7f5b88800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b80108690 0x7f5b801a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.807+0000 7f5b88800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b801a5ef0 0x7f5b801a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.807+0000 7f5b88800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8011abf0 con 0x7f5b801a5ef0 2024-05-27T09:52:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.807+0000 7f5b88800700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8011aa70 con 0x7f5b80103240 2024-05-27T09:52:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.807+0000 7f5b88800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8011ad70 con 0x7f5b80108690 2024-05-27T09:52:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.808+0000 7f5b85e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b80103240 0x7f5b801a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.808+0000 7f5b85e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b80103240 0x7f5b801a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.44:47154/0 (socket says 172.21.15.44:47154) 2024-05-27T09:52:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.808+0000 7f5b85e00700 1 -- 172.21.15.44:0/1264290614 learned_addr learned my addr 172.21.15.44:0/1264290614 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:52:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.808+0000 7f5b85400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b80108690 0x7f5b801a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.808+0000 7f5b86800700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b801a5ef0 0x7f5b801a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.808+0000 7f5b85e00700 1 -- 172.21.15.44:0/1264290614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b80108690 msgr2=0x7f5b801a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.808+0000 7f5b85e00700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b80108690 0x7f5b801a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:12.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.808+0000 7f5b85e00700 1 -- 172.21.15.44:0/1264290614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b801a5ef0 msgr2=0x7f5b801a6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:12.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.808+0000 7f5b85e00700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b801a5ef0 0x7f5b801a6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:12.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.808+0000 7f5b85e00700 1 -- 172.21.15.44:0/1264290614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b801a6a40 con 0x7f5b80103240 2024-05-27T09:52:12.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.810+0000 7f5b85e00700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b80103240 0x7f5b801a1780 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400b840 tx=0x7f5b7400bb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:52:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.811+0000 7f5b7aa00700 1 -- 172.21.15.44:0/1264290614 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74014070 con 0x7f5b80103240 2024-05-27T09:52:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.811+0000 7f5b7aa00700 1 -- 172.21.15.44:0/1264290614 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b74010cb0 con 0x7f5b80103240 2024-05-27T09:52:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.811+0000 7f5b7aa00700 1 -- 172.21.15.44:0/1264290614 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74007da0 con 0x7f5b80103240 2024-05-27T09:52:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.813+0000 7f5b88800700 1 -- 172.21.15.44:0/1264290614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b801a6d30 con 0x7f5b80103240 2024-05-27T09:52:12.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.813+0000 7f5b88800700 1 -- 172.21.15.44:0/1264290614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b8006d4c0 con 0x7f5b80103240 2024-05-27T09:52:12.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.815+0000 7f5b88800700 1 -- 172.21.15.44:0/1264290614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b800512b0 con 0x7f5b80103240 2024-05-27T09:52:12.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.819+0000 7f5b7aa00700 1 -- 172.21.15.44:0/1264290614 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7f5b74017070 con 0x7f5b80103240 2024-05-27T09:52:12.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.822+0000 7f5b7aa00700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b680b2b80 0x7f5b680b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:12.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.823+0000 7f5b7aa00700 1 -- 172.21.15.44:0/1264290614 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8865+0+0 (secure 0 0 0) 0x7f5b740e1ff0 con 0x7f5b80103240 2024-05-27T09:52:12.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.824+0000 7f5b85400700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b680b2b80 0x7f5b680b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:12.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.826+0000 7f5b85400700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b680b2b80 0x7f5b680b5090 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5b70006010 tx=0x7f5b7000b540 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:52:12.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:12.829+0000 7f5b7aa00700 1 -- 172.21.15.44:0/1264290614 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5b7401e030 con 0x7f5b80103240 2024-05-27T09:52:13.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:13.349+0000 7f5b88800700 1 -- 172.21.15.44:0/1264290614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f5b8006d7e0 con 0x7f5b80103240 2024-05-27T09:52:14.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.170+0000 7f5b7aa00700 1 -- 172.21.15.44:0/1264290614 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v47) ==== 104+0+0 (secure 0 0 0) 0x7f5b740a9810 con 0x7f5b80103240 2024-05-27T09:52:14.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.175+0000 7f5b88800700 1 -- 172.21.15.44:0/1264290614 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b680b2b80 msgr2=0x7f5b680b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:14.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.175+0000 7f5b88800700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b680b2b80 0x7f5b680b5090 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5b70006010 tx=0x7f5b7000b540 comp rx=0 tx=0).stop 2024-05-27T09:52:14.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.175+0000 7f5b88800700 1 -- 172.21.15.44:0/1264290614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b80103240 msgr2=0x7f5b801a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:14.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.175+0000 7f5b88800700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b80103240 0x7f5b801a1780 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400b840 tx=0x7f5b7400bb50 comp rx=0 tx=0).stop 2024-05-27T09:52:14.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.175+0000 7f5b88800700 1 -- 172.21.15.44:0/1264290614 shutdown_connections 2024-05-27T09:52:14.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.175+0000 7f5b88800700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7f5b680b2b80 0x7f5b680b5090 secure :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5b70006010 tx=0x7f5b7000b540 comp rx=0 tx=0).stop 2024-05-27T09:52:14.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.175+0000 7f5b88800700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b801a5ef0 0x7f5b801a6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:14.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.175+0000 7f5b88800700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b80103240 0x7f5b801a1780 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:14.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.175+0000 7f5b88800700 1 --2- 172.21.15.44:0/1264290614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b80108690 0x7f5b801a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:14.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.175+0000 7f5b88800700 1 -- 172.21.15.44:0/1264290614 >> 172.21.15.44:0/1264290614 conn(0x7f5b800fedb0 msgr2=0x7f5b80106280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:52:14.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.175+0000 7f5b88800700 1 -- 172.21.15.44:0/1264290614 shutdown_connections 2024-05-27T09:52:14.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:14.176+0000 7f5b88800700 1 -- 172.21.15.44:0/1264290614 wait complete. 2024-05-27T09:52:14.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:14 smithi044 ceph-mon[31618]: pgmap v537: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 33 KiB/s wr, 8 op/s 2024-05-27T09:52:14.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:14 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1264290614' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:52:14.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:14 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:52:14.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:14 smithi050 ceph-mon[36371]: pgmap v537: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 33 KiB/s wr, 8 op/s 2024-05-27T09:52:14.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:14 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1264290614' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:52:14.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:14 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:52:14.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:14 smithi189 ceph-mon[36671]: pgmap v537: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 33 KiB/s wr, 8 op/s 2024-05-27T09:52:14.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:14 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1264290614' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:52:14.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:14 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T09:52:14.974 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:52:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:52:14.671+0000 7f49fc600700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-27T09:52:15.156 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-27T09:52:15.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:15 smithi044 ceph-mon[31618]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T09:52:15.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:15 smithi044 ceph-mon[31618]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-27T09:52:15.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:15 smithi044 ceph-mon[31618]: Cluster is now healthy 2024-05-27T09:52:15.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:15 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-27T09:52:15.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:15 smithi044 ceph-mon[31618]: fsmap 12 up:standby 2024-05-27T09:52:15.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:15 smithi050 ceph-mon[36371]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T09:52:15.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:15 smithi050 ceph-mon[36371]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-27T09:52:15.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:15 smithi050 ceph-mon[36371]: Cluster is now healthy 2024-05-27T09:52:15.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:15 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-27T09:52:15.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:15 smithi050 ceph-mon[36371]: fsmap 12 up:standby 2024-05-27T09:52:15.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:15 smithi189 ceph-mon[36671]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T09:52:15.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:15 smithi189 ceph-mon[36671]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-27T09:52:15.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:15 smithi189 ceph-mon[36671]: Cluster is now healthy 2024-05-27T09:52:15.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:15 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-27T09:52:15.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:15 smithi189 ceph-mon[36671]: fsmap 12 up:standby 2024-05-27T09:52:15.821 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:52:16.339 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:16 smithi044 ceph-mon[31618]: pgmap v538: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-05-27T09:52:16.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:16 smithi050 ceph-mon[36371]: pgmap v538: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-05-27T09:52:16.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:16 smithi189 ceph-mon[36671]: pgmap v538: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-05-27T09:52:18.245 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:18 smithi044 ceph-mon[31618]: pgmap v539: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 104 B/s wr, 1 op/s 2024-05-27T09:52:18.245 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:18 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:52:18.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:18 smithi050 ceph-mon[36371]: pgmap v539: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 104 B/s wr, 1 op/s 2024-05-27T09:52:18.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:18 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:52:18.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:18 smithi189 ceph-mon[36671]: pgmap v539: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 104 B/s wr, 1 op/s 2024-05-27T09:52:18.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:18 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T09:52:18.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.738+0000 7ff6a0a00700 1 -- 172.21.15.44:0/3666640632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff69c1032e0 msgr2=0x7ff69c1036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:18.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.738+0000 7ff6a0a00700 1 --2- 172.21.15.44:0/3666640632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff69c1032e0 0x7ff69c1036f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff69400b0a0 tx=0x7ff69400b3b0 comp rx=0 tx=0).stop 2024-05-27T09:52:18.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.738+0000 7ff6a0a00700 1 -- 172.21.15.44:0/3666640632 shutdown_connections 2024-05-27T09:52:18.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.738+0000 7ff6a0a00700 1 --2- 172.21.15.44:0/3666640632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff69c110cd0 0x7ff69c1130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:18.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.738+0000 7ff6a0a00700 1 --2- 172.21.15.44:0/3666640632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff69c1032e0 0x7ff69c1036f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:18.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.738+0000 7ff6a0a00700 1 --2- 172.21.15.44:0/3666640632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff69c103c30 0x7ff69c110790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:18.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.738+0000 7ff6a0a00700 1 -- 172.21.15.44:0/3666640632 >> 172.21.15.44:0/3666640632 conn(0x7ff69c0fedb0 msgr2=0x7ff69c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:52:18.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.738+0000 7ff6a0a00700 1 -- 172.21.15.44:0/3666640632 shutdown_connections 2024-05-27T09:52:18.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.738+0000 7ff6a0a00700 1 -- 172.21.15.44:0/3666640632 wait complete. 2024-05-27T09:52:18.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.740+0000 7ff6a0a00700 1 Processor -- start 2024-05-27T09:52:18.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.740+0000 7ff6a0a00700 1 -- start start 2024-05-27T09:52:18.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.741+0000 7ff6a0a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff69c1032e0 0x7ff69c113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:18.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.742+0000 7ff6a0a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff69c103c30 0x7ff69c114480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:18.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.742+0000 7ff6a0a00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff69c110cd0 0x7ff69c1b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:18.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.742+0000 7ff6a0a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff69c10d420 con 0x7ff69c1032e0 2024-05-27T09:52:18.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.742+0000 7ff6a0a00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff69c10d2a0 con 0x7ff69c110cd0 2024-05-27T09:52:18.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.742+0000 7ff699400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff69c103c30 0x7ff69c114480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:18.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.742+0000 7ff6a0a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff69c10d5a0 con 0x7ff69c103c30 2024-05-27T09:52:18.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.742+0000 7ff699400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff69c103c30 0x7ff69c114480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:43752/0 (socket says 172.21.15.44:43752) 2024-05-27T09:52:18.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.742+0000 7ff699400700 1 -- 172.21.15.44:0/1774944649 learned_addr learned my addr 172.21.15.44:0/1774944649 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:52:18.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.743+0000 7ff69a800700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff69c110cd0 0x7ff69c1b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:18.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.743+0000 7ff69a800700 1 -- 172.21.15.44:0/1774944649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff69c103c30 msgr2=0x7ff69c114480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:18.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.743+0000 7ff69a800700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff69c103c30 0x7ff69c114480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:18.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.744+0000 7ff69a800700 1 -- 172.21.15.44:0/1774944649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff69c1032e0 msgr2=0x7ff69c113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:18.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.744+0000 7ff69a800700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff69c1032e0 0x7ff69c113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:18.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.744+0000 7ff69a800700 1 -- 172.21.15.44:0/1774944649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69c114d70 con 0x7ff69c110cd0 2024-05-27T09:52:18.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.745+0000 7ff69a800700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff69c110cd0 0x7ff69c1b5c80 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00b3c0 tx=0x7ff68c00b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:52:18.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.745+0000 7ff692a00700 1 -- 172.21.15.44:0/1774944649 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c015020 con 0x7ff69c110cd0 2024-05-27T09:52:18.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.745+0000 7ff692a00700 1 -- 172.21.15.44:0/1774944649 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff68c011ce0 con 0x7ff69c110cd0 2024-05-27T09:52:18.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.745+0000 7ff692a00700 1 -- 172.21.15.44:0/1774944649 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c010c60 con 0x7ff69c110cd0 2024-05-27T09:52:18.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.746+0000 7ff6a0a00700 1 -- 172.21.15.44:0/1774944649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff69c1b62e0 con 0x7ff69c110cd0 2024-05-27T09:52:18.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.746+0000 7ff6a0a00700 1 -- 172.21.15.44:0/1774944649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff69c1b67d0 con 0x7ff69c110cd0 2024-05-27T09:52:18.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.747+0000 7ff6a0a00700 1 -- 172.21.15.44:0/1774944649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff69c06be40 con 0x7ff69c110cd0 2024-05-27T09:52:18.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.756+0000 7ff692a00700 1 -- 172.21.15.44:0/1774944649 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7ff68c018020 con 0x7ff69c110cd0 2024-05-27T09:52:18.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.758+0000 7ff692a00700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6800b2a70 0x7ff6800b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:18.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.759+0000 7ff692a00700 1 -- 172.21.15.44:0/1774944649 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8865+0+0 (secure 0 0 0) 0x7ff68c0e1c60 con 0x7ff69c110cd0 2024-05-27T09:52:18.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.760+0000 7ff699e00700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6800b2a70 0x7ff6800b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:18.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.763+0000 7ff699e00700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6800b2a70 0x7ff6800b4f80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff694009e60 tx=0x7ff6940096c0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:52:18.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:18.765+0000 7ff692a00700 1 -- 172.21.15.44:0/1774944649 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff68c01f020 con 0x7ff69c110cd0 2024-05-27T09:52:19.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.186+0000 7ff6a0a00700 1 -- 172.21.15.44:0/1774944649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff69c1b6af0 con 0x7ff69c110cd0 2024-05-27T09:52:19.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.239+0000 7ff692a00700 1 -- 172.21.15.44:0/1774944649 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v108) ==== 181+0+0 (secure 0 0 0) 0x7ff69c114d70 con 0x7ff69c110cd0 2024-05-27T09:52:19.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 -- 172.21.15.44:0/1774944649 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6800b2a70 msgr2=0x7ff6800b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:19.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6800b2a70 0x7ff6800b4f80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff694009e60 tx=0x7ff6940096c0 comp rx=0 tx=0).stop 2024-05-27T09:52:19.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 -- 172.21.15.44:0/1774944649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff69c110cd0 msgr2=0x7ff69c1b5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:19.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff69c110cd0 0x7ff69c1b5c80 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00b3c0 tx=0x7ff68c00b6d0 comp rx=0 tx=0).stop 2024-05-27T09:52:19.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 -- 172.21.15.44:0/1774944649 shutdown_connections 2024-05-27T09:52:19.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7ff6800b2a70 0x7ff6800b4f80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:19.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff69c110cd0 0x7ff69c1b5c80 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:19.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff69c1032e0 0x7ff69c113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:19.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 --2- 172.21.15.44:0/1774944649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff69c103c30 0x7ff69c114480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:19.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 -- 172.21.15.44:0/1774944649 >> 172.21.15.44:0/1774944649 conn(0x7ff69c0fedb0 msgr2=0x7ff69c10f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:52:19.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 -- 172.21.15.44:0/1774944649 shutdown_connections 2024-05-27T09:52:19.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:19.244+0000 7ff6a0a00700 1 -- 172.21.15.44:0/1774944649 wait complete. 2024-05-27T09:52:19.247 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' removed 2024-05-27T09:52:19.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:19 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/1774944649' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:19.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:19 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:19.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:19 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/1774944649' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:19.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:19 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:19.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:19 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/1774944649' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:19.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:19 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:20.150 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d shell --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-27T09:52:20.479 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:20 smithi044 ceph-mon[31618]: pgmap v540: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 964 B/s rd, 96 B/s wr, 1 op/s 2024-05-27T09:52:20.479 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:20 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T09:52:20.480 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:20 smithi044 ceph-mon[31618]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T09:52:20.480 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:20 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:52:20.480 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:20 smithi044 ceph-mon[31618]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:52:20.480 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:20 smithi044 ceph-mon[31618]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:52:20.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:20 smithi050 ceph-mon[36371]: pgmap v540: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 964 B/s rd, 96 B/s wr, 1 op/s 2024-05-27T09:52:20.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:20 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T09:52:20.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:20 smithi050 ceph-mon[36371]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T09:52:20.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:20 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:52:20.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:20 smithi050 ceph-mon[36371]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:52:20.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:20 smithi050 ceph-mon[36371]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:52:20.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:20 smithi189 ceph-mon[36671]: pgmap v540: 97 pgs: 22 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 33 active+clean; 342 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 964 B/s rd, 96 B/s wr, 1 op/s 2024-05-27T09:52:20.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:20 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T09:52:20.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:20 smithi189 ceph-mon[36671]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T09:52:20.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:20 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T09:52:20.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:20 smithi189 ceph-mon[36671]: from='mgr.25072 172.21.15.44:0/1449042402' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T09:52:20.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:20 smithi189 ceph-mon[36671]: from='mgr.25072 ' entity='mgr.x' 2024-05-27T09:52:20.804 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T09:52:21.497 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:21 smithi044 ceph-mon[31618]: pgmap v542: 65 pgs: 7 active+clean, 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait; 577 KiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:52:21.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:21 smithi050 ceph-mon[36371]: pgmap v542: 65 pgs: 7 active+clean, 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait; 577 KiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:52:21.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:21 smithi189 ceph-mon[36671]: pgmap v542: 65 pgs: 7 active+clean, 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait; 577 KiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:52:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.728+0000 7fa427e00700 1 -- 172.21.15.44:0/149836139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4201092a0 msgr2=0x7fa420117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:23.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.728+0000 7fa427e00700 1 --2- 172.21.15.44:0/149836139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4201092a0 0x7fa420117480 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa41800b0a0 tx=0x7fa41800b3b0 comp rx=0 tx=0).stop 2024-05-27T09:52:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.728+0000 7fa427e00700 1 -- 172.21.15.44:0/149836139 shutdown_connections 2024-05-27T09:52:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.728+0000 7fa427e00700 1 --2- 172.21.15.44:0/149836139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4201092a0 0x7fa420117480 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.728+0000 7fa427e00700 1 --2- 172.21.15.44:0/149836139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4201030d0 0x7fa4201034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.728+0000 7fa427e00700 1 --2- 172.21.15.44:0/149836139 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa420103a20 0x7fa420114f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.728+0000 7fa427e00700 1 -- 172.21.15.44:0/149836139 >> 172.21.15.44:0/149836139 conn(0x7fa4200fede0 msgr2=0x7fa420101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:52:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.729+0000 7fa427e00700 1 -- 172.21.15.44:0/149836139 shutdown_connections 2024-05-27T09:52:23.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.729+0000 7fa427e00700 1 -- 172.21.15.44:0/149836139 wait complete. 2024-05-27T09:52:23.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.734+0000 7fa427e00700 1 Processor -- start 2024-05-27T09:52:23.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.735+0000 7fa427e00700 1 -- start start 2024-05-27T09:52:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.735+0000 7fa427e00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4201030d0 0x7fa420117450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.735+0000 7fa427e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa420103a20 0x7fa420111570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.735+0000 7fa427e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4201092a0 0x7fa420111bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.736+0000 7fa427e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa42010d020 con 0x7fa420103a20 2024-05-27T09:52:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.736+0000 7fa427e00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa42010cea0 con 0x7fa4201030d0 2024-05-27T09:52:23.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.736+0000 7fa427e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa42010d1a0 con 0x7fa4201092a0 2024-05-27T09:52:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.736+0000 7fa425e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4201092a0 0x7fa420111bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.736+0000 7fa425e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4201092a0 0x7fa420111bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.44:38674/0 (socket says 172.21.15.44:38674) 2024-05-27T09:52:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.736+0000 7fa425e00700 1 -- 172.21.15.44:0/3380878297 learned_addr learned my addr 172.21.15.44:0/3380878297 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-05-27T09:52:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.737+0000 7fa424a00700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa420103a20 0x7fa420111570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.737+0000 7fa425e00700 1 -- 172.21.15.44:0/3380878297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4201030d0 msgr2=0x7fa420117450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.737+0000 7fa425e00700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4201030d0 0x7fa420117450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:23.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.737+0000 7fa425e00700 1 -- 172.21.15.44:0/3380878297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa420103a20 msgr2=0x7fa420111570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:23.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.737+0000 7fa425e00700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa420103a20 0x7fa420111570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:23.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.737+0000 7fa425e00700 1 -- 172.21.15.44:0/3380878297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4201b5f70 con 0x7fa4201092a0 2024-05-27T09:52:23.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.737+0000 7fa425e00700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4201092a0 0x7fa420111bc0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa41800d010 tx=0x7fa418004da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:52:23.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.738+0000 7fa41e000700 1 -- 172.21.15.44:0/3380878297 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418021070 con 0x7fa4201092a0 2024-05-27T09:52:23.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.738+0000 7fa427e00700 1 -- 172.21.15.44:0/3380878297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4201b6140 con 0x7fa4201092a0 2024-05-27T09:52:23.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.738+0000 7fa41e000700 1 -- 172.21.15.44:0/3380878297 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa418007cc0 con 0x7fa4201092a0 2024-05-27T09:52:23.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.738+0000 7fa427e00700 1 -- 172.21.15.44:0/3380878297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4201b6590 con 0x7fa4201092a0 2024-05-27T09:52:23.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.739+0000 7fa41e000700 1 -- 172.21.15.44:0/3380878297 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418021070 con 0x7fa4201092a0 2024-05-27T09:52:23.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.743+0000 7fa427e00700 1 -- 172.21.15.44:0/3380878297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa420076260 con 0x7fa4201092a0 2024-05-27T09:52:23.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.747+0000 7fa41e000700 1 -- 172.21.15.44:0/3380878297 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 149783+0+0 (secure 0 0 0) 0x7fa418004520 con 0x7fa4201092a0 2024-05-27T09:52:23.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.751+0000 7fa41e000700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa4080b2e70 0x7fa4080b5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T09:52:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.751+0000 7fa41e000700 1 -- 172.21.15.44:0/3380878297 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8311+0+0 (secure 0 0 0) 0x7fa41800f020 con 0x7fa4201092a0 2024-05-27T09:52:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.751+0000 7fa425400700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa4080b2e70 0x7fa4080b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T09:52:23.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.752+0000 7fa425400700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa4080b2e70 0x7fa4080b5380 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa40c005950 tx=0x7fa40c0058e0 comp rx=0 tx=0).ready entity=mgr.25072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T09:52:23.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:23.756+0000 7fa41e000700 1 -- 172.21.15.44:0/3380878297 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa41802a020 con 0x7fa4201092a0 2024-05-27T09:52:24.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:24.196+0000 7fa427e00700 1 -- 172.21.15.44:0/3380878297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa420112aa0 con 0x7fa4201092a0 2024-05-27T09:52:24.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:23 smithi050 ceph-mon[36371]: pgmap v543: 65 pgs: 7 active+clean, 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:52:24.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:23 smithi189 ceph-mon[36671]: pgmap v543: 65 pgs: 7 active+clean, 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:52:24.473 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:23 smithi044 ceph-mon[31618]: pgmap v543: 65 pgs: 7 active+clean, 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:52:25.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.001+0000 7fa41e000700 1 -- 172.21.15.44:0/3380878297 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v109) ==== 169+0+0 (secure 0 0 0) 0x7fa4180ab240 con 0x7fa4201092a0 2024-05-27T09:52:25.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.009+0000 7fa427e00700 1 -- 172.21.15.44:0/3380878297 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa4080b2e70 msgr2=0x7fa4080b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:25.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.009+0000 7fa427e00700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa4080b2e70 0x7fa4080b5380 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa40c005950 tx=0x7fa40c0058e0 comp rx=0 tx=0).stop 2024-05-27T09:52:25.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.009+0000 7fa427e00700 1 -- 172.21.15.44:0/3380878297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4201092a0 msgr2=0x7fa420111bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T09:52:25.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.009+0000 7fa427e00700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4201092a0 0x7fa420111bc0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa41800d010 tx=0x7fa418004da0 comp rx=0 tx=0).stop 2024-05-27T09:52:25.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.009+0000 7fa427e00700 1 -- 172.21.15.44:0/3380878297 shutdown_connections 2024-05-27T09:52:25.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.009+0000 7fa427e00700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.44:6800/3776762031,v1:172.21.15.44:6801/3776762031] conn(0x7fa4080b2e70 0x7fa4080b5380 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:25.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.009+0000 7fa427e00700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4201092a0 0x7fa420111bc0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:25.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.009+0000 7fa427e00700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4201030d0 0x7fa420117450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:25.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.009+0000 7fa427e00700 1 --2- 172.21.15.44:0/3380878297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa420103a20 0x7fa420111570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T09:52:25.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.009+0000 7fa427e00700 1 -- 172.21.15.44:0/3380878297 >> 172.21.15.44:0/3380878297 conn(0x7fa4200fede0 msgr2=0x7fa420101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T09:52:25.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.011+0000 7fa427e00700 1 -- 172.21.15.44:0/3380878297 shutdown_connections 2024-05-27T09:52:25.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-05-27T09:52:25.011+0000 7fa427e00700 1 -- 172.21.15.44:0/3380878297 wait complete. 2024-05-27T09:52:25.013 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' removed 2024-05-27T09:52:25.293 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:24 smithi050 ceph-mon[36371]: from='client.? 172.21.15.44:0/3380878297' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:25.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:24 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:25.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:24 smithi189 ceph-mon[36671]: from='client.? 172.21.15.44:0/3380878297' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:25.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:24 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:25.422 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:24 smithi044 ceph-mon[31618]: from='client.? 172.21.15.44:0/3380878297' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:25.422 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:24 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T09:52:25.891 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T09:52:25.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T09:52:25.998 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T09:52:26.039 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-27T09:52:26.039 DEBUG:teuthology.orchestra.run.smithi044:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T09:52:26.115 DEBUG:teuthology.orchestra.run.smithi050:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T09:52:26.131 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T09:52:26.146 INFO:tasks.cephadm:Stopping all daemons... 2024-05-27T09:52:26.147 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-27T09:52:26.147 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.a 2024-05-27T09:52:26.219 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:25 smithi044 ceph-mon[31618]: pgmap v544: 65 pgs: 7 active+clean, 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:52:26.219 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:25 smithi044 ceph-mon[31618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T09:52:26.220 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:25 smithi044 ceph-mon[31618]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T09:52:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:25 smithi050 ceph-mon[36371]: pgmap v544: 65 pgs: 7 active+clean, 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:52:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:25 smithi050 ceph-mon[36371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T09:52:26.294 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:25 smithi050 ceph-mon[36371]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T09:52:26.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:25 smithi189 ceph-mon[36671]: pgmap v544: 65 pgs: 7 active+clean, 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T09:52:26.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:25 smithi189 ceph-mon[36671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T09:52:26.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:25 smithi189 ceph-mon[36671]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T09:52:26.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:26 smithi044 systemd[1]: Stopping Ceph mon.a for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:52:27.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:27 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a[31614]: 2024-05-27T09:52:27.216+0000 7f58cba00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:52:27.475 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:27 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a[31614]: 2024-05-27T09:52:27.216+0000 7f58cba00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-27T09:52:27.857 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:27 smithi044 podman[225801]: 2024-05-27 09:52:27.557194168 +0000 UTC m=+0.550688723 container died b84ccfae169a68c88fdcfff87c89693b8d29e829b2905953c90493f103ffe83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-05-27T09:52:28.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:28 smithi044 podman[225801]: 2024-05-27 09:52:28.109268859 +0000 UTC m=+1.102763395 container cleanup b84ccfae169a68c88fdcfff87c89693b8d29e829b2905953c90493f103ffe83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:52:28.397 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:28 smithi044 bash[225801]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a 2024-05-27T09:52:28.723 INFO:journalctl@ceph.mon.a.smithi044.stdout:May 27 09:52:28 smithi044 podman[225814]: 2024-05-27 09:52:28.394037312 +0000 UTC m=+0.819136110 container remove b84ccfae169a68c88fdcfff87c89693b8d29e829b2905953c90493f103ffe83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:52:28.943 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.a.service' 2024-05-27T09:52:29.123 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:52:29.124 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-27T09:52:29.124 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-27T09:52:29.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.b 2024-05-27T09:52:29.442 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:29 smithi050 systemd[1]: Stopping Ceph mon.b for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:52:29.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:29 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-b[36367]: 2024-05-27T09:52:29.441+0000 7f2759954700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:52:29.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:29 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-b[36367]: 2024-05-27T09:52:29.441+0000 7f2759954700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-27T09:52:29.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:29 smithi050 podman[73922]: 2024-05-27 09:52:29.605182539 +0000 UTC m=+0.284464705 container died c51b05fd7b42cf0e1610a373598af8b2330671bd24c118dd93fadf4fd654103b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T09:52:30.285 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:30 smithi050 podman[73922]: 2024-05-27 09:52:30.033531349 +0000 UTC m=+0.712813461 container cleanup c51b05fd7b42cf0e1610a373598af8b2330671bd24c118dd93fadf4fd654103b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-b, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T09:52:30.285 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:30 smithi050 bash[73922]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-b 2024-05-27T09:52:30.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.b.service' 2024-05-27T09:52:30.543 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:30 smithi050 podman[73934]: 2024-05-27 09:52:30.285034746 +0000 UTC m=+0.671880933 container remove c51b05fd7b42cf0e1610a373598af8b2330671bd24c118dd93fadf4fd654103b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True) 2024-05-27T09:52:30.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:30 smithi050 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.b.service: Deactivated successfully. 2024-05-27T09:52:30.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:30 smithi050 systemd[1]: Stopped Ceph mon.b for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:52:30.544 INFO:journalctl@ceph.mon.b.smithi050.stdout:May 27 09:52:30 smithi050 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.b.service: Consumed 20.096s CPU time. 2024-05-27T09:52:30.574 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:52:30.575 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-27T09:52:30.575 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-27T09:52:30.575 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.c 2024-05-27T09:52:31.003 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:30 smithi189 systemd[1]: Stopping Ceph mon.c for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:52:31.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:31 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-c[36665]: 2024-05-27T09:52:31.002+0000 7fa95d0d4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:52:31.337 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:31 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-c[36665]: 2024-05-27T09:52:31.002+0000 7fa95d0d4700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-27T09:52:31.338 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:31 smithi189 podman[75723]: 2024-05-27 09:52:31.226908384 +0000 UTC m=+0.348521247 container died f21d35a4925a4020a75b9b8dfaeae524a1c71fef71c5d734aa256abfba19d2ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-27T09:52:31.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:31 smithi189 podman[75723]: 2024-05-27 09:52:31.728239053 +0000 UTC m=+0.849851908 container cleanup f21d35a4925a4020a75b9b8dfaeae524a1c71fef71c5d734aa256abfba19d2ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:52:31.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:May 27 09:52:31 smithi189 bash[75723]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mon-c 2024-05-27T09:52:32.121 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mon.c.service' 2024-05-27T09:52:32.165 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:52:32.165 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-27T09:52:32.165 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-27T09:52:32.166 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.x 2024-05-27T09:52:32.724 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:52:32 smithi044 systemd[1]: Stopping Ceph mgr.x for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:52:33.124 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:52:32 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:52:32.730+0000 7f4a63000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:52:33.125 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:52:32 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x[31856]: 2024-05-27T09:52:32.730+0000 7f4a63000700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-27T09:52:33.125 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:52:32 smithi044 podman[225921]: 2024-05-27 09:52:32.833607773 +0000 UTC m=+0.249775597 container died 310da0308fc5104ca2b057b9aea4d1e5c20e666a22cd17b534a1b279374bd34f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:52:33.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:52:33 smithi044 podman[225921]: 2024-05-27 09:52:33.339560251 +0000 UTC m=+0.755728039 container cleanup 310da0308fc5104ca2b057b9aea4d1e5c20e666a22cd17b534a1b279374bd34f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T09:52:33.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:52:33 smithi044 bash[225921]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x 2024-05-27T09:52:33.973 INFO:journalctl@ceph.mgr.x.smithi044.stdout:May 27 09:52:33 smithi044 podman[225933]: 2024-05-27 09:52:33.669272587 +0000 UTC m=+0.816820495 container remove 310da0308fc5104ca2b057b9aea4d1e5c20e666a22cd17b534a1b279374bd34f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a6507388dd5057528934822c0163b0c347ef1d5d, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-27T09:52:34.229 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.x.service' 2024-05-27T09:52:34.336 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:52:34.336 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-27T09:52:34.336 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-27T09:52:34.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.y 2024-05-27T09:52:34.743 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:52:34 smithi050 systemd[1]: Stopping Ceph mgr.y for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:52:35.043 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:52:34 smithi050 podman[74045]: 2024-05-27 09:52:34.743602624 +0000 UTC m=+0.203952325 container died 5211fc522376d1889f81f5b0b67a6c2a82dfb2c39eff84e8e7afc96a2132346a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-27T09:52:35.306 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:52:35 smithi050 podman[74045]: 2024-05-27 09:52:35.146412853 +0000 UTC m=+0.606762555 container cleanup 5211fc522376d1889f81f5b0b67a6c2a82dfb2c39eff84e8e7afc96a2132346a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T09:52:35.306 INFO:journalctl@ceph.mgr.y.smithi050.stdout:May 27 09:52:35 smithi050 bash[74045]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-y 2024-05-27T09:52:35.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.y.service' 2024-05-27T09:52:35.560 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:52:35.560 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-27T09:52:35.561 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-27T09:52:35.561 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.z 2024-05-27T09:52:35.837 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:52:35 smithi189 systemd[1]: Stopping Ceph mgr.z for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:52:36.185 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:52:35 smithi189 podman[75846]: 2024-05-27 09:52:35.906164608 +0000 UTC m=+0.157212998 container died 672bcb45c16631b6b9c4dbc1629ade9eadbaaf33f1598a491ffd2723bd85705e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-27T09:52:36.451 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:52:36 smithi189 podman[75846]: 2024-05-27 09:52:36.249943222 +0000 UTC m=+0.500991594 container cleanup 672bcb45c16631b6b9c4dbc1629ade9eadbaaf33f1598a491ffd2723bd85705e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T09:52:36.451 INFO:journalctl@ceph.mgr.z.smithi189.stdout:May 27 09:52:36 smithi189 bash[75846]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mgr-z 2024-05-27T09:52:36.671 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mgr.z.service' 2024-05-27T09:52:36.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:52:36.705 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-27T09:52:36.705 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-27T09:52:36.705 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.0 2024-05-27T09:52:37.223 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:36 smithi044 systemd[1]: Stopping Ceph osd.0 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:52:37.724 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:37 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0[46734]: 2024-05-27T09:52:37.325+0000 7fc7e4a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:52:37.725 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:37 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0[46734]: 2024-05-27T09:52:37.325+0000 7fc7e4a00700 -1 osd.0 109 *** Got signal Terminated *** 2024-05-27T09:52:37.725 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:37 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0[46734]: 2024-05-27T09:52:37.325+0000 7fc7e4a00700 -1 osd.0 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:52:43.444 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:43 smithi044 podman[226039]: 2024-05-27 09:52:43.128279153 +0000 UTC m=+5.959604608 container died 97e508c5001046e74cfe6a2e35324f9840c8a20ebd38c3aa4a84acbee480cdd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:52:43.714 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:43 smithi044 podman[226039]: 2024-05-27 09:52:43.686976813 +0000 UTC m=+6.518302273 container cleanup 97e508c5001046e74cfe6a2e35324f9840c8a20ebd38c3aa4a84acbee480cdd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T09:52:43.715 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:43 smithi044 bash[226039]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0 2024-05-27T09:52:43.974 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:43 smithi044 podman[226057]: 2024-05-27 09:52:43.947120869 +0000 UTC m=+0.810921435 container remove 97e508c5001046e74cfe6a2e35324f9840c8a20ebd38c3aa4a84acbee480cdd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-27T09:52:45.102 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:44 smithi044 podman[226157]: 2024-05-27 09:52:44.673356616 +0000 UTC m=+0.032925221 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:52:45.103 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:44 smithi044 podman[226157]: 2024-05-27 09:52:44.781182845 +0000 UTC m=+0.140751486 container create 687d5a6bee65eba2914ebd750cc6ee054613d0a89b4860d316213a0a316446cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:52:45.474 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:45 smithi044 podman[226157]: 2024-05-27 09:52:45.102024157 +0000 UTC m=+0.461592838 container init 687d5a6bee65eba2914ebd750cc6ee054613d0a89b4860d316213a0a316446cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-05-27T09:52:45.474 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:45 smithi044 podman[226157]: 2024-05-27 09:52:45.114985375 +0000 UTC m=+0.474554022 container start 687d5a6bee65eba2914ebd750cc6ee054613d0a89b4860d316213a0a316446cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T09:52:45.475 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:45 smithi044 podman[226157]: 2024-05-27 09:52:45.217219018 +0000 UTC m=+0.576787666 container attach 687d5a6bee65eba2914ebd750cc6ee054613d0a89b4860d316213a0a316446cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T09:52:47.974 INFO:journalctl@ceph.osd.0.smithi044.stdout:May 27 09:52:47 smithi044 podman[226157]: 2024-05-27 09:52:47.677367958 +0000 UTC m=+3.036936639 container died 687d5a6bee65eba2914ebd750cc6ee054613d0a89b4860d316213a0a316446cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-05-27T09:52:48.622 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.0.service' 2024-05-27T09:52:48.744 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:52:48.744 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-27T09:52:48.745 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-27T09:52:48.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.1 2024-05-27T09:52:49.043 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:48 smithi050 systemd[1]: Stopping Ceph osd.1 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:52:49.293 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:49 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1[40952]: 2024-05-27T09:52:49.041+0000 7fb9d8dd3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:52:49.294 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:49 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1[40952]: 2024-05-27T09:52:49.041+0000 7fb9d8dd3700 -1 osd.1 109 *** Got signal Terminated *** 2024-05-27T09:52:49.294 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:49 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1[40952]: 2024-05-27T09:52:49.041+0000 7fb9d8dd3700 -1 osd.1 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:52:54.794 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:54 smithi050 podman[74182]: 2024-05-27 09:52:54.526628604 +0000 UTC m=+5.573880940 container died 52651e44b02c5d27d2da3caf99346661b705c8f2c02638468b0a92e6ffc7c011 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:52:55.202 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:54 smithi050 podman[74182]: 2024-05-27 09:52:54.959479176 +0000 UTC m=+6.006731513 container cleanup 52651e44b02c5d27d2da3caf99346661b705c8f2c02638468b0a92e6ffc7c011 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:52:55.203 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:54 smithi050 bash[74182]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1 2024-05-27T09:52:55.543 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:55 smithi050 podman[74198]: 2024-05-27 09:52:55.202523779 +0000 UTC m=+0.675469821 container remove 52651e44b02c5d27d2da3caf99346661b705c8f2c02638468b0a92e6ffc7c011 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T09:52:55.956 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:55 smithi050 podman[74287]: 2024-05-27 09:52:55.522315757 +0000 UTC m=+0.017880663 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:52:55.957 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:55 smithi050 podman[74287]: 2024-05-27 09:52:55.629034874 +0000 UTC m=+0.124599801 container create 52e49d81ccb5a4614b9b4654821ab4101faed86c2daedcd93f57efd4be037628 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-05-27T09:52:55.957 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:55 smithi050 podman[74287]: 2024-05-27 09:52:55.956717367 +0000 UTC m=+0.452282258 container init 52e49d81ccb5a4614b9b4654821ab4101faed86c2daedcd93f57efd4be037628 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:52:56.262 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:55 smithi050 podman[74287]: 2024-05-27 09:52:55.960478614 +0000 UTC m=+0.456043508 container start 52e49d81ccb5a4614b9b4654821ab4101faed86c2daedcd93f57efd4be037628 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521) 2024-05-27T09:52:56.263 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:56 smithi050 podman[74287]: 2024-05-27 09:52:56.039583844 +0000 UTC m=+0.535148735 container attach 52e49d81ccb5a4614b9b4654821ab4101faed86c2daedcd93f57efd4be037628 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-27T09:52:56.544 INFO:journalctl@ceph.osd.1.smithi050.stdout:May 27 09:52:56 smithi050 podman[74287]: 2024-05-27 09:52:56.263262977 +0000 UTC m=+0.758827863 container died 52e49d81ccb5a4614b9b4654821ab4101faed86c2daedcd93f57efd4be037628 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-27T09:52:56.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.1.service' 2024-05-27T09:52:56.880 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:52:56.880 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-27T09:52:56.880 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-27T09:52:56.881 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.2 2024-05-27T09:52:57.194 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:52:56 smithi189 systemd[1]: Stopping Ceph osd.2 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:52:57.587 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:52:57 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2[41765]: 2024-05-27T09:52:57.193+0000 7f979fbf4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:52:57.587 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:52:57 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2[41765]: 2024-05-27T09:52:57.193+0000 7f979fbf4700 -1 osd.2 109 *** Got signal Terminated *** 2024-05-27T09:52:57.588 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:52:57 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2[41765]: 2024-05-27T09:52:57.193+0000 7f979fbf4700 -1 osd.2 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:53:03.083 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:53:02 smithi189 podman[75971]: 2024-05-27 09:53:02.751903856 +0000 UTC m=+5.678139507 container died a162344dbf316faac39e7ab9e7dc7b6c3115c437a2b0cae8e6c9f2c147dadc7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T09:53:03.338 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:53:03 smithi189 podman[75971]: 2024-05-27 09:53:03.170059697 +0000 UTC m=+6.096295342 container cleanup a162344dbf316faac39e7ab9e7dc7b6c3115c437a2b0cae8e6c9f2c147dadc7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T09:53:03.338 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:53:03 smithi189 bash[75971]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2 2024-05-27T09:53:03.719 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:53:03 smithi189 podman[75997]: 2024-05-27 09:53:03.379692823 +0000 UTC m=+0.629639678 container remove a162344dbf316faac39e7ab9e7dc7b6c3115c437a2b0cae8e6c9f2c147dadc7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-27T09:53:04.017 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:53:03 smithi189 podman[76092]: 2024-05-27 09:53:03.622867728 +0000 UTC m=+0.017055169 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:53:04.017 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:53:03 smithi189 podman[76092]: 2024-05-27 09:53:03.730428206 +0000 UTC m=+0.124615638 container create 99cccdcc097b95150b3c032f4badd941b2703dd8824d96cc964c3bee838ee254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-27T09:53:04.017 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:53:04 smithi189 podman[76092]: 2024-05-27 09:53:04.016936211 +0000 UTC m=+0.411123641 container init 99cccdcc097b95150b3c032f4badd941b2703dd8824d96cc964c3bee838ee254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T09:53:04.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:04.177+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:04.338 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:53:04 smithi189 podman[76092]: 2024-05-27 09:53:04.020361295 +0000 UTC m=+0.414548732 container start 99cccdcc097b95150b3c032f4badd941b2703dd8824d96cc964c3bee838ee254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T09:53:04.338 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:53:04 smithi189 podman[76092]: 2024-05-27 09:53:04.11713668 +0000 UTC m=+0.511324112 container attach 99cccdcc097b95150b3c032f4badd941b2703dd8824d96cc964c3bee838ee254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T09:53:04.686 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:53:04 smithi189 podman[76092]: 2024-05-27 09:53:04.388232321 +0000 UTC m=+0.782419754 container died 99cccdcc097b95150b3c032f4badd941b2703dd8824d96cc964c3bee838ee254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-05-27T09:53:04.726 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:04 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:04.387+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:04.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:04.381+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:04.946 INFO:journalctl@ceph.osd.2.smithi189.stdout:May 27 09:53:04 smithi189 podman[76117]: 2024-05-27 09:53:04.929243433 +0000 UTC m=+0.530936474 container remove 99cccdcc097b95150b3c032f4badd941b2703dd8824d96cc964c3bee838ee254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True) 2024-05-27T09:53:04.946 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:04.720+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:04.951 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.2.service' 2024-05-27T09:53:04.989 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:53:04.990 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-27T09:53:04.990 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-27T09:53:04.990 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.3 2024-05-27T09:53:05.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:05 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:05.150+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:05.374 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:05 smithi044 systemd[1]: Stopping Ceph osd.3 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:53:05.375 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:05 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:05.243+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:05.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:05 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:05.372+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:05.724 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:05 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3[53483]: 2024-05-27T09:53:05.589+0000 7f62f3000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:53:05.725 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:05 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3[53483]: 2024-05-27T09:53:05.589+0000 7f62f3000700 -1 osd.3 109 *** Got signal Terminated *** 2024-05-27T09:53:05.725 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:05 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3[53483]: 2024-05-27T09:53:05.589+0000 7f62f3000700 -1 osd.3 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:53:05.725 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:05 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3[53483]: 2024-05-27T09:53:05.621+0000 7f62ee800700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.059054+0000 front 2024-05-27T09:52:40.059264+0000 (oldest deadline 2024-05-27T09:53:04.759357+0000) 2024-05-27T09:53:05.793 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:05 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:05.403+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:06.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:05 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:05.736+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:06.587 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:06.113+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:06.595 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:06 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:06.366+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:06.596 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:06 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:06.230+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:06.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:06.435+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:06.794 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:06.620+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:06.974 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:06 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3[53483]: 2024-05-27T09:53:06.595+0000 7f62ee800700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.059054+0000 front 2024-05-27T09:52:40.059264+0000 (oldest deadline 2024-05-27T09:53:04.759357+0000) 2024-05-27T09:53:07.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:06.746+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:07.586 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:07 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:07.385+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:07.586 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:07 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:07.258+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:07.587 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:07.117+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:07.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:07.387+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:07.794 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:07.592+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:07.974 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:07 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3[53483]: 2024-05-27T09:53:07.585+0000 7f62ee800700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.059054+0000 front 2024-05-27T09:52:40.059264+0000 (oldest deadline 2024-05-27T09:53:04.759357+0000) 2024-05-27T09:53:08.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:07.747+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:08.556 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:08 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:08.348+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:08.557 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:08 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:08.282+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:08.587 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:08.105+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:08.793 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:08.630+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:08.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:08.423+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:08.973 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:08 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3[53483]: 2024-05-27T09:53:08.555+0000 7f62ee800700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.059054+0000 front 2024-05-27T09:52:40.059264+0000 (oldest deadline 2024-05-27T09:53:04.759357+0000) 2024-05-27T09:53:09.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:08.708+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:09.529 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:09 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:09.348+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:09.530 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:09 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:09.263+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:09.587 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:09.109+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:09.793 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:09.603+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:09.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:09.449+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:09.973 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:09 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3[53483]: 2024-05-27T09:53:09.528+0000 7f62ee800700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.059054+0000 front 2024-05-27T09:52:40.059264+0000 (oldest deadline 2024-05-27T09:53:04.759357+0000) 2024-05-27T09:53:10.067 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:09.733+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:10.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:10.066+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:10.571 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:10 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:10.325+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:10.571 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:10 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:10.273+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:10.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:10.451+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:10.794 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:10.553+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:10.973 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:10 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3[53483]: 2024-05-27T09:53:10.570+0000 7f62ee800700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.059054+0000 front 2024-05-27T09:52:40.059264+0000 (oldest deadline 2024-05-27T09:53:04.759357+0000) 2024-05-27T09:53:11.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:10.776+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:11.538 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:11 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:11.374+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:11.538 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:11 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:11.273+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:11.538 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:11 smithi044 podman[226264]: 2024-05-27 09:53:11.255519359 +0000 UTC m=+5.800889233 container stop 0831d46132e9b121964c5796f185d458b42bab8ddd9f9a47692d835a900efa93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T09:53:11.539 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:11 smithi044 podman[226264]: 2024-05-27 09:53:11.282308487 +0000 UTC m=+5.827678354 container died 0831d46132e9b121964c5796f185d458b42bab8ddd9f9a47692d835a900efa93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T09:53:11.587 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:11 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:11.087+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:11.793 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:11 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:11.495+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:11.794 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:11 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:11.578+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:11.960 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:11 smithi044 podman[226264]: 2024-05-27 09:53:11.740271149 +0000 UTC m=+6.285641019 container cleanup 0831d46132e9b121964c5796f185d458b42bab8ddd9f9a47692d835a900efa93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:53:11.961 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:11 smithi044 bash[226264]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3 2024-05-27T09:53:12.038 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:11 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:11.739+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:12.223 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:11 smithi044 podman[226284]: 2024-05-27 09:53:11.958303893 +0000 UTC m=+0.675507092 container remove 0831d46132e9b121964c5796f185d458b42bab8ddd9f9a47692d835a900efa93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:53:12.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:12 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:12.038+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:12.723 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:12 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:12.378+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:12.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:12 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:12.275+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:12.793 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:12 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:12.608+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:12.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:12 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:12.534+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:13.077 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:12 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:12.691+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:13.121 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:12 smithi044 podman[226372]: 2024-05-27 09:53:12.728436973 +0000 UTC m=+0.035415667 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:53:13.122 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:12 smithi044 podman[226372]: 2024-05-27 09:53:12.834298055 +0000 UTC m=+0.141276766 container create 8e8ed1622d0e3adc411a85adfce6fc4b1abbb2fd39fc82c9300aec2d27a25c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux ) 2024-05-27T09:53:13.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:13 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:13.076+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:13.474 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:13 smithi044 podman[226372]: 2024-05-27 09:53:13.121115645 +0000 UTC m=+0.428094395 container init 8e8ed1622d0e3adc411a85adfce6fc4b1abbb2fd39fc82c9300aec2d27a25c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-27T09:53:13.474 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:13 smithi044 podman[226372]: 2024-05-27 09:53:13.132918676 +0000 UTC m=+0.439897402 container start 8e8ed1622d0e3adc411a85adfce6fc4b1abbb2fd39fc82c9300aec2d27a25c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738) 2024-05-27T09:53:13.474 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:13 smithi044 podman[226372]: 2024-05-27 09:53:13.220020268 +0000 UTC m=+0.526999042 container attach 8e8ed1622d0e3adc411a85adfce6fc4b1abbb2fd39fc82c9300aec2d27a25c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:53:13.475 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:13 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:13.329+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:13.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:13 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:13.319+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:13.793 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:13 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:13.563+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:13.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:13 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:13.521+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:13.848 INFO:journalctl@ceph.osd.3.smithi044.stdout:May 27 09:53:13 smithi044 podman[226372]: 2024-05-27 09:53:13.606871171 +0000 UTC m=+0.913849858 container died 8e8ed1622d0e3adc411a85adfce6fc4b1abbb2fd39fc82c9300aec2d27a25c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T09:53:14.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:13 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:13.679+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:14.388 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.3.service' 2024-05-27T09:53:14.454 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:14.092+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:14.486 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:14.330+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:14.486 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:14 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:14.334+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:14.490 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:53:14.490 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-27T09:53:14.490 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-27T09:53:14.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.4 2024-05-27T09:53:14.793 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:14 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:14.544+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:14.794 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:14 smithi050 systemd[1]: Stopping Ceph osd.4 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:53:14.794 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:14 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:14.557+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:14.837 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:14.696+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:15.293 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:14 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:14.804+0000 7f247bcfa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:53:15.294 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:14 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:14.804+0000 7f247bcfa700 -1 osd.4 109 *** Got signal Terminated *** 2024-05-27T09:53:15.294 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:14 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:14.804+0000 7f247bcfa700 -1 osd.4 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:53:15.587 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:15 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:15.094+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:15.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:15 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:15.366+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:15.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:15 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:15.371+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:15.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:15 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:15.371+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:15.793 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:15 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:15.501+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:15.794 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:15 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:15.584+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:15.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:15 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:15.695+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:16.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:15 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:15.662+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:16.587 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:16.133+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:16.587 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:16.133+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:16.723 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:16 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:16.330+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:16.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:16 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:16.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:16.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:16 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:16.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:16.745 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:16 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:16.584+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:16.746 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:16 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:16.466+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:17.043 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:16 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:16.745+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:17.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:16.699+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:17.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:16.699+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:17.441 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:17.095+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:17.442 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:17.095+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:17.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:17 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:17.356+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:17.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:17 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:17.386+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:17.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:17 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:17.386+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:17.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:17 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:17.701+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:17.794 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:17 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:17.543+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:17.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:17 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:17.493+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:17.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:17 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:17.493+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:17.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:17.440+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:17.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:17.678+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:17.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:17.678+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:18.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:18.142+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:18.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:18.142+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:18.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:18 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:18.313+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:18.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:18 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:18.429+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:18.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:18 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:18.429+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:18.793 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:18 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:18.732+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:18.794 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:18 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:18.522+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:18.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:18 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:18.490+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:18.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:18 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:18.490+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:18.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:18.460+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:18.837 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:18.660+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:18.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:18.660+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:19.416 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:19 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:19.096+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:19.416 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:19 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:19.096+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:19.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:19 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:19.356+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:19.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:19 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:19.356+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:19.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:19 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:19.431+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:19.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:19 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:19.431+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:19.793 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:19 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4[47465]: 2024-05-27T09:53:19.556+0000 7f24782b0700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.189111+0000 front 2024-05-27T09:52:41.189572+0000 (oldest deadline 2024-05-27T09:53:05.888913+0000) 2024-05-27T09:53:19.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:19 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:19.692+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:19.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:19 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:19.482+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:19.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:19 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:19.482+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:19.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:19 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:19.416+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:19.837 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:19 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:19.665+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:19.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:19 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:19.665+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:20.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:20 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:20.059+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:20.338 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:20 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:20.059+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:20.648 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:20 smithi050 podman[74414]: 2024-05-27 09:53:20.392397223 +0000 UTC m=+5.707907838 container died a872924d2ef5eb5c79d8628506eb34d112c4416a9e4805e7ec9c03a6d0380164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:53:20.648 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:20 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:20.437+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:20.648 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:20 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:20.437+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:20.723 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:20 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:20.322+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:20.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:20 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:20.322+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:20.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:20 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:20.436+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:20.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:20 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:20.436+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:20.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:20 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:20.420+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:20.837 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:20 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:20.635+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:20.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:20 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:20.635+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:20.997 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:20 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:20.665+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:20.997 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:20 smithi050 podman[74414]: 2024-05-27 09:53:20.737867926 +0000 UTC m=+6.053378537 container cleanup a872924d2ef5eb5c79d8628506eb34d112c4416a9e4805e7ec9c03a6d0380164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T09:53:20.998 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:20 smithi050 bash[74414]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4 2024-05-27T09:53:21.293 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:20 smithi050 podman[74430]: 2024-05-27 09:53:20.997454911 +0000 UTC m=+0.605578231 container remove a872924d2ef5eb5c79d8628506eb34d112c4416a9e4805e7ec9c03a6d0380164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-27T09:53:21.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:21 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:21.019+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:21.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:21 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:21.019+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:21.642 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:21 smithi050 podman[74519]: 2024-05-27 09:53:21.364978598 +0000 UTC m=+0.108982543 container create 1d3209cbb050a71106fdd0acfd8a703fdac9ce01246ab7cf2ba6565e84db5326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-05-27T09:53:21.642 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:21 smithi050 podman[74519]: 2024-05-27 09:53:21.273611949 +0000 UTC m=+0.017615919 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:53:21.643 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:21 smithi050 podman[74519]: 2024-05-27 09:53:21.542182484 +0000 UTC m=+0.286186438 container init 1d3209cbb050a71106fdd0acfd8a703fdac9ce01246ab7cf2ba6565e84db5326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-27T09:53:21.643 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:21 smithi050 podman[74519]: 2024-05-27 09:53:21.545365636 +0000 UTC m=+0.289369597 container start 1d3209cbb050a71106fdd0acfd8a703fdac9ce01246ab7cf2ba6565e84db5326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T09:53:21.643 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:21 smithi050 podman[74519]: 2024-05-27 09:53:21.642535003 +0000 UTC m=+0.386538972 container attach 1d3209cbb050a71106fdd0acfd8a703fdac9ce01246ab7cf2ba6565e84db5326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T09:53:21.643 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:21 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:21.419+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:21.643 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:21 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:21.419+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:21.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:21 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:21.365+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:21.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:21 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:21.365+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:21.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:21 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:21.434+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:21.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:21 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:21.434+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:21.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:21 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:21.427+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:21.837 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:21 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:21.627+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:21.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:21 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:21.627+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:22.044 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:21 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:21.693+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:22.044 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:21 smithi050 podman[74519]: 2024-05-27 09:53:21.85627003 +0000 UTC m=+0.600273988 container died 1d3209cbb050a71106fdd0acfd8a703fdac9ce01246ab7cf2ba6565e84db5326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-27T09:53:22.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:22.003+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:22.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:22.003+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:22.387 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:22 smithi050 podman[74545]: 2024-05-27 09:53:22.362517278 +0000 UTC m=+0.500088161 container remove 1d3209cbb050a71106fdd0acfd8a703fdac9ce01246ab7cf2ba6565e84db5326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T09:53:22.387 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:22 smithi050 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.4.service: Deactivated successfully. 2024-05-27T09:53:22.387 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:22 smithi050 systemd[1]: Stopped Ceph osd.4 for 604b015a-1c0a-11ef-bc9b-c7b262605968. 2024-05-27T09:53:22.388 INFO:journalctl@ceph.osd.4.smithi050.stdout:May 27 09:53:22 smithi050 systemd[1]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.4.service: Consumed 2min 52.948s CPU time. 2024-05-27T09:53:22.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.4.service' 2024-05-27T09:53:22.428 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:53:22.428 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-27T09:53:22.429 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-27T09:53:22.429 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.5 2024-05-27T09:53:22.680 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:22 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:22.440+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:22.680 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:22 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:22.440+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:22.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:22 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:22.366+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:22.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:22 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:22.366+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:22.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:22 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:22.427+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:22.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:22 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:22.427+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:22.837 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:22 smithi189 systemd[1]: Stopping Ceph osd.5 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:53:22.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:22.476+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:22.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:22.676+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:22.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:22.676+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:23.043 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:22 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:22.680+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:23.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:22.881+0000 7f391cc1f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:53:23.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:22.881+0000 7f391cc1f700 -1 osd.5 109 *** Got signal Terminated *** 2024-05-27T09:53:23.338 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:22.881+0000 7f391cc1f700 -1 osd.5 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:53:23.338 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:22.966+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:23.338 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:22 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:22.966+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:23.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:23 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:23.386+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:23.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:23 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:23.386+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:23.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:23 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:23.403+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:23.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:23 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:23.403+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:23.725 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:23 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:23.403+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:23.793 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:23 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:23.451+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:23.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:23 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:23.451+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:23.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:23 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:23.451+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:23.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:23 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:23.654+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:23.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:23 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:23.654+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:23.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:23 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:23.478+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:23.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:23 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:23.478+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:23.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:23 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:23.715+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:23.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:23 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:23.715+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:24.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:23 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:23.983+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:24.338 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:23 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:23.983+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:24.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:24 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:24.374+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:24.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:24 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:24.374+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:24.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:24 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:24.374+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:24.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:24 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:24.434+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:24.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:24 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:24.434+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:24.734 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:24.477+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:24.734 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:24.477+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:24.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:24 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:24.646+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:24.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:24 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:24.646+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:24.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:24 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:24.415+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:24.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:24 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:24.416+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:24.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:24 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:24.416+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:25.087 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:24.952+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:25.087 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:24.952+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:25.088 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:24.952+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.897783+0000 front 2024-05-27T09:53:00.897822+0000 (oldest deadline 2024-05-27T09:53:24.397894+0000) 2024-05-27T09:53:25.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:24.733+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:25.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:24.733+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:25.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:24 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:24.733+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:25.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:25 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:25.338+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:25.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:25 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:25.338+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:25.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:25 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:25.338+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:25.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:25 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:25.448+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:25.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:25 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:25.448+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:25.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:25 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:25.448+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:25.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:25 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:25.639+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:25.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:25 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:25.639+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:25.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:25 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:25.412+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:25.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:25 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:25.412+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:25.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:25 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:25.412+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:25.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:25 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:25.466+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:25.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:25 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:25.466+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:25.837 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:25 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:25.687+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:25.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:25 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:25.687+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:25.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:25 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:25.687+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:26.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:25 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:25.958+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:26.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:25 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:25.958+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:26.338 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:25 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:25.958+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.897783+0000 front 2024-05-27T09:53:00.897822+0000 (oldest deadline 2024-05-27T09:53:24.397894+0000) 2024-05-27T09:53:26.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:26.320+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:26.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:26.320+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:26.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:26.320+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:26.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:26.417+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:26.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:26.417+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:26.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:26 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:26.417+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:26.793 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:26 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:26.591+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:26.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:26 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:26.591+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:26.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:26 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:26.436+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:26.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:26 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:26.436+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:26.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:26 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:26.436+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:26.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:26 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:26.515+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:26.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:26 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:26.515+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:26.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:26 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:26.643+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:26.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:26 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:26.643+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:26.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:26 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:26.643+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:27.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:26 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:26.909+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.594877+0000 front 2024-05-27T09:52:40.594429+0000 (oldest deadline 2024-05-27T09:53:03.494321+0000) 2024-05-27T09:53:27.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:26 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:26.909+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:49.196520+0000 front 2024-05-27T09:52:49.196620+0000 (oldest deadline 2024-05-27T09:53:15.096611+0000) 2024-05-27T09:53:27.338 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:26 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5[48291]: 2024-05-27T09:53:26.909+0000 7f39151cd700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.897783+0000 front 2024-05-27T09:53:00.897822+0000 (oldest deadline 2024-05-27T09:53:24.397894+0000) 2024-05-27T09:53:27.691 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:27 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:27.548+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:27.691 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:27 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:27.548+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:27.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:27 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:27.342+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:27.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:27 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:27.342+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:27.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:27 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:27.342+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:27.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:27 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:27.405+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:27.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:27 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:27.405+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:27.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:27 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:27.405+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:27.793 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:27 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:27.582+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:27.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:27 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:27.582+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:27.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:27 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:27.473+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:27.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:27 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:27.473+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:27.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:27 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:27.473+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:28.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:27 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:27.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:28.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:27 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:27.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:28.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:27 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:27.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:28.720 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:28 smithi189 podman[76198]: 2024-05-27 09:53:28.469302053 +0000 UTC m=+5.841723438 container died 03afd35c8e2d27819e558a61bbf19f432094e001f17b2c57cf23f1cad14e3098 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738) 2024-05-27T09:53:28.720 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:28 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:28.547+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:28.720 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:28 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:28.547+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:28.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:28 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:28.342+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:28.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:28 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:28.342+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:28.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:28 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:28.342+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:28.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:28 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:28.414+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:28.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:28 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:28.414+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:28.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:28 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:28.414+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:28.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:28 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:28.534+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:28.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:28 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:28.534+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:28.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:28 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:28.516+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:28.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:28 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:28.516+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:28.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:28 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:28.516+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:29.087 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:28 smithi189 podman[76198]: 2024-05-27 09:53:28.823094307 +0000 UTC m=+6.195515675 container cleanup 03afd35c8e2d27819e558a61bbf19f432094e001f17b2c57cf23f1cad14e3098 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-27T09:53:29.088 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:28 smithi189 bash[76198]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5 2024-05-27T09:53:29.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:28 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:28.720+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:29.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:28 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:28.720+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:29.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:28 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:28.720+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:29.440 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:29 smithi189 podman[76214]: 2024-05-27 09:53:29.107988051 +0000 UTC m=+0.633520920 container remove 03afd35c8e2d27819e558a61bbf19f432094e001f17b2c57cf23f1cad14e3098 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True) 2024-05-27T09:53:29.691 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:29 smithi189 podman[76302]: 2024-05-27 09:53:29.343873967 +0000 UTC m=+0.016899664 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:53:29.691 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:29 smithi189 podman[76302]: 2024-05-27 09:53:29.466663672 +0000 UTC m=+0.139689345 container create 85962bfd33d615c7667551b8ef9e96bc11e485da9b9f74a0f7b63ec1db2cae3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-27T09:53:29.692 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:29 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:29.584+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:29.692 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:29 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:29.584+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:29.693 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:29 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:29.691+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:29.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:29 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:29.302+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:29.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:29 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:29.302+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:29.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:29 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:29.302+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:29.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:29 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:29.399+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:29.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:29 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:29.399+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:29.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:29 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:29.399+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:29.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:29 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:29.530+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:29.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:29 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:29.530+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:29.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:29 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:29.497+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:29.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:29 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:29.497+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:29.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:29 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:29.497+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:29.987 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:29 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:29.691+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:29.987 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:29 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:29.691+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:29.987 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:29 smithi189 podman[76302]: 2024-05-27 09:53:29.694621356 +0000 UTC m=+0.367647036 container init 85962bfd33d615c7667551b8ef9e96bc11e485da9b9f74a0f7b63ec1db2cae3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:53:29.988 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:29 smithi189 podman[76302]: 2024-05-27 09:53:29.697768805 +0000 UTC m=+0.370794489 container start 85962bfd33d615c7667551b8ef9e96bc11e485da9b9f74a0f7b63ec1db2cae3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-05-27T09:53:29.988 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:29 smithi189 podman[76302]: 2024-05-27 09:53:29.777210772 +0000 UTC m=+0.450236454 container attach 85962bfd33d615c7667551b8ef9e96bc11e485da9b9f74a0f7b63ec1db2cae3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-27T09:53:30.297 INFO:journalctl@ceph.osd.5.smithi189.stdout:May 27 09:53:29 smithi189 podman[76302]: 2024-05-27 09:53:29.987632325 +0000 UTC m=+0.660658011 container died 85962bfd33d615c7667551b8ef9e96bc11e485da9b9f74a0f7b63ec1db2cae3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T09:53:30.585 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.5.service' 2024-05-27T09:53:30.621 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:53:30.621 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-27T09:53:30.622 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-27T09:53:30.622 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.6 2024-05-27T09:53:30.699 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:30 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:30.324+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:30.700 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:30 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:30.324+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:30.700 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:30 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:30.324+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:30.700 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:30 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:30.381+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:30.700 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:30 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:30.381+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:30.700 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:30 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:30.381+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:30.793 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:30 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:30.542+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:30.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:30 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:30.542+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:30.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:30 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:30.525+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:30.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:30 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:30.525+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:30.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:30 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:30.525+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:30.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:30 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:30.563+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:30.838 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:30 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:30.563+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:30.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:30 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:30.730+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:30.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:30 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:30.730+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:30.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:30 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:30.730+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:30.974 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:30 smithi044 systemd[1]: Stopping Ceph osd.6 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:53:31.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:31 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:31.378+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:31.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:31 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:31.378+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:31.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:31 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:31.378+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:31.725 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:31 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:31.230+0000 7f3284c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:53:31.725 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:31 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:31.230+0000 7f3284c00700 -1 osd.6 109 *** Got signal Terminated *** 2024-05-27T09:53:31.725 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:31 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:31.230+0000 7f3284c00700 -1 osd.6 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:53:31.725 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:31 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:31.276+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:31.725 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:31 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:31.276+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:31.726 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:31 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:31.276+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:31.793 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:31 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:31.587+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:31.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:31 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:31.587+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:31.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:31 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:31.488+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:31.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:31 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:31.488+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:31.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:31 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:31.488+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:31.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:31 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:31.585+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:31.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:31 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:31.585+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:31.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:31 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:31.717+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:31.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:31 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:31.717+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:31.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:31 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:31.717+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:32.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:32 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:32.260+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:32.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:32 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:32.260+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:32.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:32 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:32.260+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:32.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:32 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:32.414+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:32.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:32 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:32.414+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:32.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:32 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:32.414+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:32.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:32 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:32.550+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:32.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:32 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:32.550+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:32.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:32 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:32.550+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:32.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:32 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:32.502+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:32.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:32 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:32.502+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:32.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:32 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:32.502+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:33.087 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:32 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:32.592+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:33.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:32 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:32.592+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:33.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:32 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:32.683+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:33.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:32 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:32.683+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:33.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:32 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:32.683+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:33.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:33 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:33.267+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:33.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:33 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:33.267+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:33.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:33 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:33.267+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:33.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:33 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:33.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:33.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:33 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:33.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:33.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:33 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:33.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:33.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:33 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:33.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:33.793 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:33 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:33.599+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:33.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:33 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:33.599+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:33.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:33 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:33.599+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:33.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:33 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:33.460+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:33.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:33 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:33.460+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:33.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:33 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:33.460+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:34.087 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:33 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:33.607+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:34.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:33 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:33.607+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:34.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:33 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:33.672+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:34.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:33 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:33.672+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:34.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:33 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:33.672+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:34.474 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:34 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:34.220+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:34.474 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:34 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:34.220+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:34.474 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:34 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:34.220+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:34.474 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:34 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:34.404+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:34.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:34 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:34.404+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:34.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:34 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:34.404+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:34.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:34 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:34.404+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:34.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:34 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:34.646+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:34.795 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:34 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:34.646+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:34.795 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:34 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:34.646+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:34.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:34 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:34.437+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:34.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:34 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:34.437+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:34.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:34 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:34.437+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:34.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:34 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:34.572+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:34.838 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:34 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:34.572+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:34.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:34 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:34.701+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:34.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:34 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:34.701+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:34.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:34 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:34.701+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:35.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:35 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:35.241+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.346598+0000 front 2024-05-27T09:52:41.346503+0000 (oldest deadline 2024-05-27T09:53:03.646566+0000) 2024-05-27T09:53:35.724 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:35 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:35.241+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.450113+0000 front 2024-05-27T09:52:52.450193+0000 (oldest deadline 2024-05-27T09:53:18.350377+0000) 2024-05-27T09:53:35.725 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:35 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:35.241+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.252445+0000 front 2024-05-27T09:53:01.252870+0000 (oldest deadline 2024-05-27T09:53:22.952708+0000) 2024-05-27T09:53:35.725 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:35 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6[59997]: 2024-05-27T09:53:35.241+0000 7f327b400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:10.456820+0000 front 2024-05-27T09:53:10.457571+0000 (oldest deadline 2024-05-27T09:53:34.556719+0000) 2024-05-27T09:53:35.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:35 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:35.432+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:35.726 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:35 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:35.432+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:35.726 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:35 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:35.432+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:35.726 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:35 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:35.432+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:35.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:35 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:35.660+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:35.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:35 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:35.660+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:35.795 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:35 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:35.660+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:35.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:35 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:35.453+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:35.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:35 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:35.453+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:35.796 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:35 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:35.453+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:35.796 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:35 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:35.453+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:35.837 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:35 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:35.561+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:35.838 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:35 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:35.561+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:35.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:35 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:35.700+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:35.839 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:35 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:35.700+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:35.839 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:35 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:35.700+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:36.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:36 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:36.396+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:36.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:36 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:36.396+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:36.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:36 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:36.396+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:36.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:36 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:36.396+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:36.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:36 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:36.622+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:36.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:36 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:36.622+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:36.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:36 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:36.622+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:36.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:36 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:36.441+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:36.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:36 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:36.441+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:36.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:36 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:36.441+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:36.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:36 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:36.441+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:37.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:36 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:36.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:37.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:36 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:36.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:37.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:36 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:36.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:37.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:36 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:36.595+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:37.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:36 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:36.595+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:37.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:36 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:36.595+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:37.187 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:36 smithi044 podman[226490]: 2024-05-27 09:53:36.895612954 +0000 UTC m=+5.798013382 container died ef24bd158be48340c3b6e8b662be7ad8414618e041d58c09d9cca2a2f5548759 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:53:37.473 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:37 smithi044 podman[226490]: 2024-05-27 09:53:37.404983303 +0000 UTC m=+6.307383734 container cleanup ef24bd158be48340c3b6e8b662be7ad8414618e041d58c09d9cca2a2f5548759 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:53:37.474 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:37 smithi044 bash[226490]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6 2024-05-27T09:53:37.474 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:37 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:37.368+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:37.474 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:37 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:37.368+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:37.474 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:37 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:37.368+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:37.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:37 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:37.368+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:37.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:37 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:37.574+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:37.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:37 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:37.574+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:37.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:37 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:37.574+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:37.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:37 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:37.394+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:37.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:37 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:37.394+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:37.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:37 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:37.394+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:37.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:37 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:37.394+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:37.974 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:37 smithi044 podman[226509]: 2024-05-27 09:53:37.707167958 +0000 UTC m=+0.794027767 container remove ef24bd158be48340c3b6e8b662be7ad8414618e041d58c09d9cca2a2f5548759 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:53:38.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:37 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:37.646+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:38.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:37 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:37.646+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:38.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:37 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:37.646+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:38.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:37 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:37.645+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:38.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:37 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:37.645+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:38.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:37 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:37.645+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:38.633 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:38 smithi044 podman[226597]: 2024-05-27 09:53:38.495861537 +0000 UTC m=+0.037028395 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:53:38.634 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:38 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:38.370+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:38.634 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:38 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:38.370+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:38.634 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:38 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:38.370+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:38.634 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:38 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:38.370+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:38.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:38 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:38.549+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:38.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:38 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:38.549+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:38.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:38 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:38.549+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:38.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:38 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:38.348+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:38.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:38 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:38.348+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:38.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:38 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:38.348+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:38.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:38 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:38.348+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:38.892 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:38 smithi044 podman[226597]: 2024-05-27 09:53:38.633178971 +0000 UTC m=+0.174345867 container create e7537130515622a14bea53299b3070fc36ecfa8cf3f2bf4244d3be3482e0ba1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738) 2024-05-27T09:53:38.892 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:38 smithi044 podman[226597]: 2024-05-27 09:53:38.879141821 +0000 UTC m=+0.420308737 container init e7537130515622a14bea53299b3070fc36ecfa8cf3f2bf4244d3be3482e0ba1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, ceph=True) 2024-05-27T09:53:38.893 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:38 smithi044 podman[226597]: 2024-05-27 09:53:38.891922665 +0000 UTC m=+0.433089531 container start e7537130515622a14bea53299b3070fc36ecfa8cf3f2bf4244d3be3482e0ba1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:53:39.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:38 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:38.608+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:39.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:38 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:38.608+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:39.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:38 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:38.608+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:39.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:38 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:38.611+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:39.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:38 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:38.611+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:39.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:38 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:38.611+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:39.223 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:39 smithi044 podman[226597]: 2024-05-27 09:53:39.019544207 +0000 UTC m=+0.560711150 container attach e7537130515622a14bea53299b3070fc36ecfa8cf3f2bf4244d3be3482e0ba1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:53:39.564 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:39 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:39.304+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:39.564 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:39 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:39.304+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:39.564 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:39 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:39.304+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:39.564 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:39 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:39.304+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:39.653 INFO:journalctl@ceph.osd.6.smithi044.stdout:May 27 09:53:39 smithi044 podman[226597]: 2024-05-27 09:53:39.357580556 +0000 UTC m=+0.898747482 container died e7537130515622a14bea53299b3070fc36ecfa8cf3f2bf4244d3be3482e0ba1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:53:39.654 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:39 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:39.361+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:39.654 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:39 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:39.361+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:39.654 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:39 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:39.361+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:39.654 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:39 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:39.361+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:40.044 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:39 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:39.563+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:40.044 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:39 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:39.563+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:40.044 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:39 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:39.563+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:40.087 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:39 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:39.655+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:40.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:39 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:39.655+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:40.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:39 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:39.655+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:40.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:39 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:39.622+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:40.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:39 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:39.622+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:40.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:39 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:39.622+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:40.119 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.6.service' 2024-05-27T09:53:40.237 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:53:40.237 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-27T09:53:40.237 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-27T09:53:40.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.7 2024-05-27T09:53:40.324 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:40 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:40.322+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:40.325 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:40 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:40.322+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:40.325 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:40 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:40.322+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:40.325 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:40 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:40.322+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:40.522 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:40 smithi050 systemd[1]: Stopping Ceph osd.7 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:53:40.522 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:40 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:40.268+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:40.522 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:40 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:40.268+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:40.522 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:40 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:40.268+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:40.523 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:40 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:40.268+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:40.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:40 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:40.521+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:40.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:40 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:40.521+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:40.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:40 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:40.521+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:40.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:40 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:40.672+0000 7fa611b2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:53:40.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:40 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:40.672+0000 7fa611b2d700 -1 osd.7 109 *** Got signal Terminated *** 2024-05-27T09:53:40.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:40 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:40.672+0000 7fa611b2d700 -1 osd.7 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:53:41.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:40 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:40.625+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:41.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:40 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:40.625+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:41.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:40 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:40.625+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:41.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:40 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:40.693+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:41.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:40 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:40.693+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:41.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:40 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:40.693+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:41.520 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:41 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:41.251+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:41.520 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:41 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:41.251+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:41.521 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:41 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:41.251+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:41.521 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:41 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:41.251+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:41.521 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:41 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:41.251+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:41.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:41 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:41.299+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:41.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:41 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:41.299+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:41.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:41 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:41.299+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:41.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:41 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:41.299+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:41.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:41 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:41.520+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:41.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:41 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:41.520+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:41.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:41 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:41.520+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:41.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:41 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:41.520+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.440146+0000 front 2024-05-27T09:53:16.440191+0000 (oldest deadline 2024-05-27T09:53:41.040877+0000) 2024-05-27T09:53:42.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:41 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:41.642+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:42.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:41 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:41.642+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:42.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:41 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:41.642+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:42.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:41 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:41.678+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:42.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:41 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:41.678+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:42.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:41 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:41.678+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:42.543 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:42 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:42.292+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:42.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:42 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:42.292+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:42.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:42 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:42.292+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:42.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:42 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:42.292+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:42.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:42 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:42.292+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:42.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:42 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:42.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:42.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:42 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:42.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:42.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:42 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:42.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:42.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:42 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:42.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:42.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:42 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:42.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:43.044 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:42 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:42.562+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:43.044 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:42 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:42.562+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:43.044 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:42 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:42.562+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:43.044 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:42 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:42.562+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.440146+0000 front 2024-05-27T09:53:16.440191+0000 (oldest deadline 2024-05-27T09:53:41.040877+0000) 2024-05-27T09:53:43.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:42 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:42.618+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:43.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:42 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:42.618+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:43.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:42 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:42.618+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:43.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:42 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:42.641+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:43.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:42 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:42.641+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:43.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:42 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:42.641+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:43.089 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:42 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:42.641+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:43.335 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:43 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:43.272+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:43.335 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:43 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:43.272+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:43.335 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:43 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:43.272+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:43.335 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:43 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:43.272+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:43.335 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:43 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:43.272+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:43.626 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:43 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:43.327+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:43.626 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:43 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:43.327+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:43.626 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:43 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:43.327+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:43.626 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:43 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:43.327+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:43.627 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:43 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:43.327+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:43.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:43 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:43.561+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:43.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:43 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:43.561+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:43.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:43 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:43.561+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:43.795 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:43 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:43.561+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.440146+0000 front 2024-05-27T09:53:16.440191+0000 (oldest deadline 2024-05-27T09:53:41.040877+0000) 2024-05-27T09:53:44.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:43 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:43.603+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:44.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:43 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:43.603+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:44.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:43 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:43.603+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:44.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:43 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:43.608+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:44.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:43 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:43.608+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:44.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:43 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:43.608+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:44.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:43 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:43.608+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:44.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:44 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:44.283+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:44.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:44 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:44.283+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:44.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:44 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:44.283+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:44.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:44 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:44.283+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:44.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:44 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:44.283+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:44.793 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:44 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:44.522+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:44.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:44 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:44.522+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:44.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:44 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:44.522+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:44.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:44 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:44.522+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.440146+0000 front 2024-05-27T09:53:16.440191+0000 (oldest deadline 2024-05-27T09:53:41.040877+0000) 2024-05-27T09:53:44.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:44 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:44.308+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:44.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:44 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:44.308+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:44.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:44 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:44.308+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:44.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:44 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:44.308+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:44.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:44 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:44.308+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:45.087 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:44 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:44.619+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:45.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:44 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:44.619+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:45.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:44 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:44.619+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:45.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:44 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:44.619+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:45.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:44 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:44.626+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:45.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:44 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:44.626+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:45.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:44 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:44.626+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:45.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:44 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:44.626+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:45.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:45.289+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:45.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:45.289+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:45.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:45.289+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:45.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:45.289+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:45.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:45.289+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:45.545 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:45.508+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.935175+0000 front 2024-05-27T09:52:50.935027+0000 (oldest deadline 2024-05-27T09:53:15.035141+0000) 2024-05-27T09:53:45.545 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:45.508+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:02.036742+0000 front 2024-05-27T09:53:02.036718+0000 (oldest deadline 2024-05-27T09:53:23.136784+0000) 2024-05-27T09:53:45.546 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:45.508+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:06.538682+0000 front 2024-05-27T09:53:06.538396+0000 (oldest deadline 2024-05-27T09:53:31.838280+0000) 2024-05-27T09:53:45.546 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7[54472]: 2024-05-27T09:53:45.508+0000 7fa60e0e3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.440146+0000 front 2024-05-27T09:53:16.440191+0000 (oldest deadline 2024-05-27T09:53:41.040877+0000) 2024-05-27T09:53:45.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:45 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:45.296+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:45.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:45 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:45.296+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:45.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:45 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:45.296+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:45.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:45 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:45.296+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:45.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:45 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:45.296+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:45.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:45 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:45.650+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:45.839 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:45 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:45.650+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:45.839 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:45 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:45.650+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:45.839 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:45 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:45.650+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:45.839 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:45 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:45.584+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:45.839 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:45 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:45.584+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:45.840 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:45 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:45.584+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:45.840 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:45 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:45.584+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:46.541 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:46 smithi050 podman[74626]: 2024-05-27 09:53:46.25037564 +0000 UTC m=+5.812706930 container died 412cb68def25258b7623874a2038a5288c9e26eae8eabe8f24f7dda17c1db45c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7, ceph=True, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738) 2024-05-27T09:53:46.541 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:46 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:46.316+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:46.542 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:46 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:46.316+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:46.542 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:46 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:46.316+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:46.542 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:46 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:46.316+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:46.542 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:46 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:46.316+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:46.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:46 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:46.265+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:46.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:46 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:46.265+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:46.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:46 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:46.265+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:46.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:46 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:46.265+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:46.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:46 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:46.265+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:46.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:46 smithi050 podman[74626]: 2024-05-27 09:53:46.614940815 +0000 UTC m=+6.177272096 container cleanup 412cb68def25258b7623874a2038a5288c9e26eae8eabe8f24f7dda17c1db45c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD) 2024-05-27T09:53:46.795 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:46 smithi050 bash[74626]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7 2024-05-27T09:53:47.087 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:46 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:46.621+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:47.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:46 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:46.621+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:47.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:46 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:46.621+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:47.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:46 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:46.621+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:47.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:46 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:46.697+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:47.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:46 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:46.697+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:47.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:46 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:46.697+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:47.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:46 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:46.697+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:47.258 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:46 smithi050 podman[74655]: 2024-05-27 09:53:46.899744368 +0000 UTC m=+0.645422335 container remove 412cb68def25258b7623874a2038a5288c9e26eae8eabe8f24f7dda17c1db45c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, ceph=True) 2024-05-27T09:53:47.533 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:47 smithi050 podman[74741]: 2024-05-27 09:53:47.258849973 +0000 UTC m=+0.111377449 container create 089e34873a2073d3afafc8b25e05bcf161355da25989d6922239e4518d3138a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738) 2024-05-27T09:53:47.533 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:47 smithi050 podman[74741]: 2024-05-27 09:53:47.165419069 +0000 UTC m=+0.017946562 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:53:47.534 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:47 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:47.266+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:47.534 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:47 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:47.266+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:47.534 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:47 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:47.266+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:47.534 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:47 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:47.266+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:47.534 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:47 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:47.266+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:47.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:47.315+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:47.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:47.315+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:47.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:47.315+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:47.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:47.315+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:47.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:47 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:47.315+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:47.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:47 smithi050 podman[74741]: 2024-05-27 09:53:47.644431234 +0000 UTC m=+0.496958705 container init 089e34873a2073d3afafc8b25e05bcf161355da25989d6922239e4518d3138a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:53:47.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:47 smithi050 podman[74741]: 2024-05-27 09:53:47.647543514 +0000 UTC m=+0.500070984 container start 089e34873a2073d3afafc8b25e05bcf161355da25989d6922239e4518d3138a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:53:47.794 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:47 smithi050 podman[74741]: 2024-05-27 09:53:47.744529692 +0000 UTC m=+0.597057171 container attach 089e34873a2073d3afafc8b25e05bcf161355da25989d6922239e4518d3138a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-27T09:53:48.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:47 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:47.669+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:48.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:47 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:47.669+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:48.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:47 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:47.669+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:48.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:47 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:47.669+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:48.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:47 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:47.706+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:48.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:47 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:47.706+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:48.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:47 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:47.706+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:48.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:47 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:47.706+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:48.247 INFO:journalctl@ceph.osd.7.smithi050.stdout:May 27 09:53:47 smithi050 podman[74741]: 2024-05-27 09:53:47.953263091 +0000 UTC m=+0.805790562 container died 089e34873a2073d3afafc8b25e05bcf161355da25989d6922239e4518d3138a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:53:48.515 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:48 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:48.248+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:48.515 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:48 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:48.248+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:48.515 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:48 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:48.248+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:48.516 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:48 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:48.248+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:48.516 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:48 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:48.248+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:48.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.7.service' 2024-05-27T09:53:48.590 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:53:48.591 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-27T09:53:48.591 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-27T09:53:48.591 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.8 2024-05-27T09:53:48.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:48 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:48.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:48.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:48 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:48.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:48.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:48 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:48.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:48.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:48 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:48.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:48.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:48 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:48.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:48.891 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:48 smithi189 systemd[1]: Stopping Ceph osd.8 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:53:48.892 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:48.710+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:48.892 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:48.710+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:48.892 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:48.710+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:48.892 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:48.710+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:48.892 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:48.710+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.401504+0000 front 2024-05-27T09:53:27.401533+0000 (oldest deadline 2024-05-27T09:53:48.501822+0000) 2024-05-27T09:53:48.893 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:48.890+0000 7fa876492700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:53:48.893 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:48.890+0000 7fa876492700 -1 osd.8 109 *** Got signal Terminated *** 2024-05-27T09:53:48.893 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:48.890+0000 7fa876492700 -1 osd.8 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:53:48.893 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:48.679+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:48.894 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:48.679+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:48.894 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:48.679+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:48.894 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:48 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:48.679+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:49.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:49 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:49.272+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:49.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:49 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:49.272+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:49.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:49 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:49.272+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:49.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:49 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:49.272+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:49.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:49 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:49.272+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:49.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:49 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:49.298+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:49.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:49 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:49.298+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:49.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:49 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:49.298+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:49.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:49 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:49.298+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:49.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:49 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:49.298+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:50.087 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:49 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:49.691+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:50.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:49 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:49.691+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:50.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:49 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:49.691+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:50.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:49 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:49.691+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:50.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:49 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:49.691+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.401504+0000 front 2024-05-27T09:53:27.401533+0000 (oldest deadline 2024-05-27T09:53:48.501822+0000) 2024-05-27T09:53:50.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:49 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:49.634+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:50.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:49 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:49.634+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:50.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:49 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:49.634+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:50.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:49 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:49.634+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:50.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:49 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:49.634+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:53:50.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:50 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:50.314+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:50.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:50 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:50.314+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:50.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:50 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:50.314+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:50.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:50 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:50.314+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:50.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:50 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:50.314+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:50.793 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:50 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:50.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:50.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:50 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:50.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:50.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:50 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:50.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:50.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:50 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:50.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:50.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:50 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:50.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:51.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:50 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:50.590+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:51.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:50 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:50.590+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:51.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:50 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:50.590+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:51.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:50 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:50.590+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:51.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:50 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:50.590+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:53:51.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:50 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:50.702+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:51.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:50 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:50.702+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:51.089 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:50 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:50.702+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:51.089 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:50 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:50.702+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:51.089 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:50 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:50.702+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.401504+0000 front 2024-05-27T09:53:27.401533+0000 (oldest deadline 2024-05-27T09:53:48.501822+0000) 2024-05-27T09:53:51.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:51 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:51.267+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:51.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:51 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:51.267+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:51.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:51 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:51.267+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:51.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:51 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:51.267+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:51.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:51 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:51.267+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:51.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:51 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:51.299+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:51.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:51 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:51.299+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:51.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:51 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:51.299+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:51.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:51 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:51.299+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:51.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:51 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:51.299+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:51.837 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:51.584+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:51.837 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:51.584+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:51.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:51.584+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:51.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:51.584+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:51.838 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:51.584+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:53:51.838 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:51.739+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:51.838 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:51.739+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:51.838 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:51.739+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:51.839 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:51.739+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:51.839 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:51 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:51.739+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.401504+0000 front 2024-05-27T09:53:27.401533+0000 (oldest deadline 2024-05-27T09:53:48.501822+0000) 2024-05-27T09:53:52.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:52 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:52.304+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:52.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:52 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:52.304+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:52.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:52 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:52.304+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:52.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:52 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:52.304+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:52.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:52 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:52.304+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:52.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:52 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:52.304+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.849212+0000 front 2024-05-27T09:53:27.849278+0000 (oldest deadline 2024-05-27T09:53:51.349065+0000) 2024-05-27T09:53:52.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:52 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:52.295+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:52.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:52 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:52.295+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:52.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:52 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:52.295+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:52.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:52 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:52.295+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:52.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:52 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:52.295+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:53.087 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:52 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:52.732+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:53.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:52 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:52.732+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:53.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:52 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:52.732+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:53.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:52 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:52.732+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:53.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:52 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:52.732+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.401504+0000 front 2024-05-27T09:53:27.401533+0000 (oldest deadline 2024-05-27T09:53:48.501822+0000) 2024-05-27T09:53:53.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:52 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:52.614+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:53.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:52 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:52.614+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:53.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:52 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:52.614+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:53.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:52 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:52.614+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:53.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:52 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:52.614+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:53:53.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:53 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:53.289+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:53.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:53 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:53.289+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:53.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:53 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:53.289+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:53.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:53 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:53.289+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:53.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:53 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:53.289+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:53.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:53 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:53.289+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:53:53.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:53 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:53.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:53.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:53 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:53.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:53.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:53 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:53.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:53.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:53 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:53.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:53.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:53 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:53.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:53.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:53 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:53.303+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.849212+0000 front 2024-05-27T09:53:27.849278+0000 (oldest deadline 2024-05-27T09:53:51.349065+0000) 2024-05-27T09:53:54.087 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:53 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:53.775+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.595932+0000 front 2024-05-27T09:52:50.595867+0000 (oldest deadline 2024-05-27T09:53:16.495905+0000) 2024-05-27T09:53:54.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:53 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:53.775+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:56.996916+0000 front 2024-05-27T09:52:56.996880+0000 (oldest deadline 2024-05-27T09:53:22.897031+0000) 2024-05-27T09:53:54.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:53 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:53.775+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:07.498864+0000 front 2024-05-27T09:53:07.498945+0000 (oldest deadline 2024-05-27T09:53:36.299167+0000) 2024-05-27T09:53:54.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:53 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:53.775+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.299819+0000 front 2024-05-27T09:53:16.299754+0000 (oldest deadline 2024-05-27T09:53:42.199754+0000) 2024-05-27T09:53:54.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:53 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8[55272]: 2024-05-27T09:53:53.775+0000 7fa872a48700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.401504+0000 front 2024-05-27T09:53:27.401533+0000 (oldest deadline 2024-05-27T09:53:48.501822+0000) 2024-05-27T09:53:54.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:53 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:53.620+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:54.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:53 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:53.620+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:54.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:53 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:53.620+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:54.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:53 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:53.620+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:54.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:53 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:53.620+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:53:54.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:54 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:54.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:54.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:54 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:54.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:54.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:54 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:54.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:54.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:54 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:54.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:54.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:54 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:54.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:54.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:54 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:54.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:53:54.666 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:54 smithi189 podman[76408]: 2024-05-27 09:53:54.409299704 +0000 UTC m=+5.627391326 container died 240f01128c5a865d94ba392be10ca749dd63aaf45f1e0b6ed1f8cb2597fa2b23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-05-27T09:53:54.667 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:54 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:54.634+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:54.667 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:54 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:54.634+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:54.667 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:54 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:54.634+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:54.667 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:54 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:54.634+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:54.667 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:54 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:54.634+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:53:54.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:54.308+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:54.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:54.308+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:54.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:54.308+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:54.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:54.308+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:54.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:54.308+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:54.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:54 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:54.308+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.849212+0000 front 2024-05-27T09:53:27.849278+0000 (oldest deadline 2024-05-27T09:53:51.349065+0000) 2024-05-27T09:53:55.087 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:54 smithi189 podman[76408]: 2024-05-27 09:53:54.766108784 +0000 UTC m=+5.984200409 container cleanup 240f01128c5a865d94ba392be10ca749dd63aaf45f1e0b6ed1f8cb2597fa2b23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T09:53:55.088 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:54 smithi189 bash[76408]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8 2024-05-27T09:53:55.417 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:55 smithi189 podman[76422]: 2024-05-27 09:53:55.10902397 +0000 UTC m=+0.697083838 container remove 240f01128c5a865d94ba392be10ca749dd63aaf45f1e0b6ed1f8cb2597fa2b23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:53:55.543 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:55 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:55.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:55.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:55 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:55.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:55.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:55 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:55.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:55.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:55 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:55.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:55.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:55 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:55.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:55.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:55 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:55.270+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:53:55.720 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:55 smithi189 podman[76514]: 2024-05-27 09:53:55.418178318 +0000 UTC m=+0.106936376 container create 3c15b87667aaa1d6a7b249a2f242f25df218f6e12e35bc53fc05bf1a00fd1ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T09:53:55.720 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:55 smithi189 podman[76514]: 2024-05-27 09:53:55.32839267 +0000 UTC m=+0.017150713 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:53:55.721 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:55 smithi189 podman[76514]: 2024-05-27 09:53:55.612549031 +0000 UTC m=+0.301307068 container init 3c15b87667aaa1d6a7b249a2f242f25df218f6e12e35bc53fc05bf1a00fd1ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T09:53:55.721 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:55 smithi189 podman[76514]: 2024-05-27 09:53:55.615830178 +0000 UTC m=+0.304588220 container start 3c15b87667aaa1d6a7b249a2f242f25df218f6e12e35bc53fc05bf1a00fd1ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux ) 2024-05-27T09:53:55.721 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:55 smithi189 podman[76514]: 2024-05-27 09:53:55.720248685 +0000 UTC m=+0.409006730 container attach 3c15b87667aaa1d6a7b249a2f242f25df218f6e12e35bc53fc05bf1a00fd1ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-27T09:53:55.721 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:55 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:55.600+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:55.721 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:55 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:55.600+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:55.721 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:55 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:55.600+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:55.722 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:55 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:55.600+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:55.722 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:55 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:55.600+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:53:55.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:55 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:55.330+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:55.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:55 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:55.330+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:55.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:55 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:55.330+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:55.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:55 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:55.330+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:55.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:55 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:55.330+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:55.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:55 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:55.330+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.849212+0000 front 2024-05-27T09:53:27.849278+0000 (oldest deadline 2024-05-27T09:53:51.349065+0000) 2024-05-27T09:53:56.087 INFO:journalctl@ceph.osd.8.smithi189.stdout:May 27 09:53:55 smithi189 podman[76514]: 2024-05-27 09:53:55.908501816 +0000 UTC m=+0.597259860 container died 3c15b87667aaa1d6a7b249a2f242f25df218f6e12e35bc53fc05bf1a00fd1ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T09:53:56.479 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.8.service' 2024-05-27T09:53:56.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:53:56.517 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-27T09:53:56.517 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-27T09:53:56.517 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.9 2024-05-27T09:53:56.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:56 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:56.263+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:56.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:56 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:56.263+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:56.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:56 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:56.263+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:56.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:56 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:56.264+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:56.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:56 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:56.264+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:56.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:56 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:56.264+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:53:56.595 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:56 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:56.288+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:56.596 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:56 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:56.288+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:56.596 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:56 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:56.288+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:56.596 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:56 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:56.288+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:56.596 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:56 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:56.288+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:56.596 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:56 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:56.288+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.849212+0000 front 2024-05-27T09:53:27.849278+0000 (oldest deadline 2024-05-27T09:53:51.349065+0000) 2024-05-27T09:53:56.733 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:56 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:56.563+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:56.733 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:56 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:56.563+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:56.733 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:56 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:56.563+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:56.733 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:56 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:56.563+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:56.734 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:56 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:56.563+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:53:56.974 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:56 smithi044 systemd[1]: Stopping Ceph osd.9 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:53:57.474 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:57 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:57.085+0000 7f8a84400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:53:57.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:57 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:57.085+0000 7f8a84400700 -1 osd.9 109 *** Got signal Terminated *** 2024-05-27T09:53:57.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:57 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:57.085+0000 7f8a84400700 -1 osd.9 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:53:57.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:57 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:57.333+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:57.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:57 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:57.333+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:57.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:57 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:57.333+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:57.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:57 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:57.333+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:57.475 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:57 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:57.333+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:57.476 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:57 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:57.333+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.849212+0000 front 2024-05-27T09:53:27.849278+0000 (oldest deadline 2024-05-27T09:53:51.349065+0000) 2024-05-27T09:53:57.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:57 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:57.252+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:57.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:57 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:57.252+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:57.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:57 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:57.252+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:57.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:57 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:57.252+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:57.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:57 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:57.252+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:57.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:57 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:57.252+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:53:58.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:57 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:57.597+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:58.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:57 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:57.597+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:58.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:57 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:57.597+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:58.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:57 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:57.597+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:58.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:57 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:57.597+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:53:58.543 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:58 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:58.273+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:58.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:58 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:58.273+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:58.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:58 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:58.273+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:58.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:58 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:58.273+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:58.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:58 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:58.273+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:58.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:58 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:58.273+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:53:58.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:58 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:58.273+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:53:58.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:58 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:58.367+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:58.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:58 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:58.367+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:58.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:58 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:58.367+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:58.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:58 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:58.367+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:58.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:58 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:58.367+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:58.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:58 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:58.367+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.849212+0000 front 2024-05-27T09:53:27.849278+0000 (oldest deadline 2024-05-27T09:53:51.349065+0000) 2024-05-27T09:53:59.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:58 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:58.646+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:53:59.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:58 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:58.646+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:53:59.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:58 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:58.646+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:53:59.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:58 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:58.646+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:53:59.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:58 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:58.646+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:53:59.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:59 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:59.243+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:53:59.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:59 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:59.243+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:53:59.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:59 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:59.243+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:53:59.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:59 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:59.243+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:53:59.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:59 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:59.243+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:53:59.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:59 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:59.243+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:53:59.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:53:59 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:53:59.243+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:53:59.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:59 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:59.349+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:53:59.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:59 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:59.349+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:53:59.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:59 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:59.349+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:53:59.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:59 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:59.349+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:53:59.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:59 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:59.349+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:53:59.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:59 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:59.349+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.849212+0000 front 2024-05-27T09:53:27.849278+0000 (oldest deadline 2024-05-27T09:53:51.349065+0000) 2024-05-27T09:53:59.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:53:59 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:53:59.349+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:33.651456+0000 front 2024-05-27T09:53:33.651517+0000 (oldest deadline 2024-05-27T09:53:58.951060+0000) 2024-05-27T09:54:00.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:59 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:59.654+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:00.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:59 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:59.654+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:00.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:59 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:59.654+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:00.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:59 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:59.654+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:00.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:53:59 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:53:59.654+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:00.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:00 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:00.280+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:54:00.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:00 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:00.280+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:54:00.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:00 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:00.280+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:54:00.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:00 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:00.280+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:54:00.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:00 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:00.280+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:54:00.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:00 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:00.280+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:54:00.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:00 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:00.280+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:54:00.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:00.398+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:54:00.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:00.398+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:54:00.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:00.398+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:54:00.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:00.398+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:54:00.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:00.398+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:54:00.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:00.398+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.849212+0000 front 2024-05-27T09:53:27.849278+0000 (oldest deadline 2024-05-27T09:53:51.349065+0000) 2024-05-27T09:54:00.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:00 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:00.398+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:33.651456+0000 front 2024-05-27T09:53:33.651517+0000 (oldest deadline 2024-05-27T09:53:58.951060+0000) 2024-05-27T09:54:01.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:00 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:00.657+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:01.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:00 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:00.657+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:01.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:00 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:00.657+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:01.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:00 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:00.657+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:01.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:00 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:00.657+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:01.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:00 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:00.657+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:01.543 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:01 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:01.230+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:54:01.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:01 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:01.230+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:54:01.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:01 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:01.230+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:54:01.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:01 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:01.230+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:54:01.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:01 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:01.230+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:54:01.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:01 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:01.230+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:54:01.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:01 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:01.230+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:54:01.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:01 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:01.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:40.929868+0000 front 2024-05-27T09:52:40.930257+0000 (oldest deadline 2024-05-27T09:53:05.029980+0000) 2024-05-27T09:54:01.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:01 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:01.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.035810+0000 front 2024-05-27T09:52:50.035899+0000 (oldest deadline 2024-05-27T09:53:14.736037+0000) 2024-05-27T09:54:01.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:01 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:01.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:01.139021+0000 front 2024-05-27T09:53:01.139094+0000 (oldest deadline 2024-05-27T09:53:25.239320+0000) 2024-05-27T09:54:01.724 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:01 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:01.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:09.242639+0000 front 2024-05-27T09:53:09.242579+0000 (oldest deadline 2024-05-27T09:53:33.342408+0000) 2024-05-27T09:54:01.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:01 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:01.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:16.145875+0000 front 2024-05-27T09:53:16.145823+0000 (oldest deadline 2024-05-27T09:53:42.046096+0000) 2024-05-27T09:54:01.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:01 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:01.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:27.849212+0000 front 2024-05-27T09:53:27.849278+0000 (oldest deadline 2024-05-27T09:53:51.349065+0000) 2024-05-27T09:54:01.725 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:01 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9[66707]: 2024-05-27T09:54:01.412+0000 7f8a7fc00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:33.651456+0000 front 2024-05-27T09:53:33.651517+0000 (oldest deadline 2024-05-27T09:53:58.951060+0000) 2024-05-27T09:54:02.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:01 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:01.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:02.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:01 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:01.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:02.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:01 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:01.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:02.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:01 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:01.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:02.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:01 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:01.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:02.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:01 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:01.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:02.543 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:02 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:02.219+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:54:02.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:02 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:02.219+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:54:02.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:02 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:02.219+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:54:02.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:02 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:02.219+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:54:02.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:02 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:02.219+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:54:02.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:02 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:02.219+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:54:02.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:02 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:02.219+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:54:03.061 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:02 smithi044 podman[226702]: 2024-05-27 09:54:02.730094243 +0000 UTC m=+5.785248320 container died 011df4158fc9b78c4267e4d11c27d2fdb26e75bcf302ade47f966d6fe436ee60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-27T09:54:03.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:02 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:02.735+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:03.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:02 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:02.735+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:03.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:02 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:02.735+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:03.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:02 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:02.735+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:03.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:02 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:02.735+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:03.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:02 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:02.735+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:03.474 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:03 smithi044 podman[226702]: 2024-05-27 09:54:03.269673084 +0000 UTC m=+6.324827197 container cleanup 011df4158fc9b78c4267e4d11c27d2fdb26e75bcf302ade47f966d6fe436ee60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-05-27T09:54:03.474 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:03 smithi044 bash[226702]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9 2024-05-27T09:54:03.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:03 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:03.261+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:54:03.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:03 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:03.261+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:54:03.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:03 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:03.261+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:54:03.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:03 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:03.261+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:54:03.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:03 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:03.261+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:54:03.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:03 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:03.261+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:54:03.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:03 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:03.261+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:54:03.974 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:03 smithi044 podman[226721]: 2024-05-27 09:54:03.529222674 +0000 UTC m=+0.779438144 container remove 011df4158fc9b78c4267e4d11c27d2fdb26e75bcf302ade47f966d6fe436ee60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, ceph=True) 2024-05-27T09:54:04.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:03 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:03.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:04.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:03 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:03.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:04.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:03 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:03.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:04.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:03 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:03.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:04.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:03 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:03.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:04.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:03 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:03.690+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:04.626 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:04 smithi044 podman[226813]: 2024-05-27 09:54:04.261883707 +0000 UTC m=+0.037638711 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:54:04.626 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:04 smithi044 podman[226813]: 2024-05-27 09:54:04.371991793 +0000 UTC m=+0.147746822 container create 7eb85cfafe0866cc45e57c4bd99536afec111d0eefbe0e23480fb16a5a3d0a76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-27T09:54:04.793 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:04.304+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:54:04.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:04.304+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:54:04.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:04.304+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:54:04.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:04.304+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:54:04.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:04.304+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:54:04.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:04.304+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:54:04.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:04 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:04.304+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:54:04.974 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:04 smithi044 podman[226813]: 2024-05-27 09:54:04.625473343 +0000 UTC m=+0.401228390 container init 7eb85cfafe0866cc45e57c4bd99536afec111d0eefbe0e23480fb16a5a3d0a76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738) 2024-05-27T09:54:04.974 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:04 smithi044 podman[226813]: 2024-05-27 09:54:04.637861686 +0000 UTC m=+0.413616717 container start 7eb85cfafe0866cc45e57c4bd99536afec111d0eefbe0e23480fb16a5a3d0a76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T09:54:04.975 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:04 smithi044 podman[226813]: 2024-05-27 09:54:04.739026888 +0000 UTC m=+0.514781956 container attach 7eb85cfafe0866cc45e57c4bd99536afec111d0eefbe0e23480fb16a5a3d0a76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:54:05.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:04.706+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:05.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:04.706+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:05.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:04.706+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:05.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:04.706+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:05.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:04.706+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:05.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:04.706+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:05.466 INFO:journalctl@ceph.osd.9.smithi044.stdout:May 27 09:54:05 smithi044 podman[226813]: 2024-05-27 09:54:05.117565622 +0000 UTC m=+0.893320643 container died 7eb85cfafe0866cc45e57c4bd99536afec111d0eefbe0e23480fb16a5a3d0a76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T09:54:05.780 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.9.service' 2024-05-27T09:54:05.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:05 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:05.339+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:54:05.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:05 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:05.339+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:54:05.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:05 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:05.339+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:54:05.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:05 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:05.339+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:54:05.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:05 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:05.339+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:54:05.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:05 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:05.339+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:54:05.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:05 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:05.339+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:54:05.884 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:54:05.884 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-27T09:54:05.885 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-27T09:54:05.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.10 2024-05-27T09:54:06.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:05 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:05.698+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:06.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:05 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:05.698+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:06.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:05 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:05.698+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:06.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:05 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:05.698+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:06.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:05 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:05.698+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:06.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:05 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:05.698+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:06.184 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:05 smithi050 systemd[1]: Stopping Ceph osd.10 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:54:06.184 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:06.183+0000 7fc4823f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:54:06.184 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:06.183+0000 7fc4823f1700 -1 osd.10 109 *** Got signal Terminated *** 2024-05-27T09:54:06.184 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:06.183+0000 7fc4823f1700 -1 osd.10 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:54:06.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:06.315+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:54:06.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:06.315+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:54:06.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:06.315+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:54:06.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:06.315+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:54:06.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:06.315+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:54:06.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:06.315+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:54:06.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:06 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:06.315+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:54:07.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:06.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:07.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:06.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:07.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:06.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:07.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:06.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:07.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:06.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:07.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:06.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:07.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:06 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:06.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:07.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:07.317+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:54:07.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:07.317+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:54:07.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:07.317+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:54:07.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:07.317+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:54:07.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:07.317+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:54:07.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:07.317+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:54:07.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:07 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:07.317+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:54:08.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:07.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:08.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:07.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:08.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:07.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:08.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:07.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:08.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:07.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:08.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:07.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:08.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:07 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:07.677+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:08.793 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:08.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:54:08.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:08.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:54:08.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:08.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:54:08.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:08.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:54:08.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:08.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:54:08.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:08.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:54:08.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:08 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:08.310+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:54:09.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:08.710+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:09.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:08.710+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:09.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:08.710+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:09.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:08.710+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:09.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:08.710+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:09.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:08.710+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:09.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:08 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:08.710+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:09.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:09.324+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:54:09.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:09.324+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:54:09.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:09.324+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:54:09.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:09.324+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:54:09.794 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:09.324+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:54:09.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:09.324+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:54:09.795 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:09 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:09.324+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:54:10.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:09.747+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:10.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:09.747+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:10.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:09.747+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:10.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:09.747+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:10.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:09.747+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:10.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:09.747+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:10.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:09.747+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:10.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:10.287+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:41.398263+0000 front 2024-05-27T09:52:41.398566+0000 (oldest deadline 2024-05-27T09:53:03.697952+0000) 2024-05-27T09:54:10.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:10.287+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:52.400383+0000 front 2024-05-27T09:52:52.400249+0000 (oldest deadline 2024-05-27T09:53:17.100421+0000) 2024-05-27T09:54:10.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:10.287+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:52:58.201362+0000 front 2024-05-27T09:52:58.201377+0000 (oldest deadline 2024-05-27T09:53:22.901486+0000) 2024-05-27T09:54:10.544 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:10.287+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-05-27T09:53:05.202695+0000 front 2024-05-27T09:53:05.202780+0000 (oldest deadline 2024-05-27T09:53:35.202942+0000) 2024-05-27T09:54:10.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:10.287+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:18.103829+0000 front 2024-05-27T09:53:18.103776+0000 (oldest deadline 2024-05-27T09:53:41.003916+0000) 2024-05-27T09:54:10.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:10.287+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:26.006397+0000 front 2024-05-27T09:53:26.006537+0000 (oldest deadline 2024-05-27T09:53:52.406853+0000) 2024-05-27T09:54:10.545 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:10 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10[61354]: 2024-05-27T09:54:10.287+0000 7fc47a99f700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:32.407480+0000 front 2024-05-27T09:53:32.407586+0000 (oldest deadline 2024-05-27T09:53:57.707299+0000) 2024-05-27T09:54:11.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:10.777+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:11.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:10.777+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:11.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:10.777+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:11.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:10.777+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:11.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:10.777+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:11.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:10.777+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:11.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:10 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:10.777+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:11.952 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:11 smithi050 podman[74847]: 2024-05-27 09:54:11.588413034 +0000 UTC m=+5.505479761 container died 5406196eefce17f49fb66c222dda48f42fd48b56df893af889f7ea7e759afc65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T09:54:12.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:11 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:11.825+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:12.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:11 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:11.825+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:12.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:11 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:11.825+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:12.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:11 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:11.825+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:12.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:11 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:11.825+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:12.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:11 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:11.825+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:12.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:11 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:11.825+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:12.294 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:12 smithi050 podman[74847]: 2024-05-27 09:54:12.072718385 +0000 UTC m=+5.989785096 container cleanup 5406196eefce17f49fb66c222dda48f42fd48b56df893af889f7ea7e759afc65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-27T09:54:12.294 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:12 smithi050 bash[74847]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10 2024-05-27T09:54:12.634 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:12 smithi050 podman[74862]: 2024-05-27 09:54:12.301134853 +0000 UTC m=+0.707366290 container remove 5406196eefce17f49fb66c222dda48f42fd48b56df893af889f7ea7e759afc65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2) 2024-05-27T09:54:12.912 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:12 smithi050 podman[74949]: 2024-05-27 09:54:12.635172783 +0000 UTC m=+0.114325818 container create 04f84a979a1afd895b05a9852844e47b588b4672752d3a3d2bda6389f0caf78f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521) 2024-05-27T09:54:12.912 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:12 smithi050 podman[74949]: 2024-05-27 09:54:12.538741667 +0000 UTC m=+0.017894720 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:54:12.913 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:12 smithi050 podman[74949]: 2024-05-27 09:54:12.837884239 +0000 UTC m=+0.317037276 container init 04f84a979a1afd895b05a9852844e47b588b4672752d3a3d2bda6389f0caf78f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10-deactivate, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T09:54:12.913 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:12 smithi050 podman[74949]: 2024-05-27 09:54:12.841074076 +0000 UTC m=+0.320227114 container start 04f84a979a1afd895b05a9852844e47b588b4672752d3a3d2bda6389f0caf78f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T09:54:12.913 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:12 smithi050 podman[74949]: 2024-05-27 09:54:12.912659217 +0000 UTC m=+0.391812259 container attach 04f84a979a1afd895b05a9852844e47b588b4672752d3a3d2bda6389f0caf78f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:54:13.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:12 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:12.791+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:13.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:12 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:12.791+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:13.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:12 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:12.791+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:13.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:12 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:12.791+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:13.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:12 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:12.791+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:13.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:12 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:12.791+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:13.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:12 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:12.791+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:13.294 INFO:journalctl@ceph.osd.10.smithi050.stdout:May 27 09:54:13 smithi050 podman[74949]: 2024-05-27 09:54:13.150358796 +0000 UTC m=+0.629511841 container died 04f84a979a1afd895b05a9852844e47b588b4672752d3a3d2bda6389f0caf78f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-27T09:54:13.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.10.service' 2024-05-27T09:54:13.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:54:13.719 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-27T09:54:13.719 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-27T09:54:13.719 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.11 2024-05-27T09:54:14.027 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:13 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:13.750+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:14.028 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:13 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:13.750+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:14.028 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:13 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:13.750+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:14.028 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:13 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:13.750+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:14.028 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:13 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:13.751+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:14.028 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:13 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:13.751+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:14.028 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:13 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:13.751+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:14.029 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:13 smithi189 systemd[1]: Stopping Ceph osd.11 for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:54:14.029 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:14.027+0000 7ffa81b10700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:54:14.029 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:14.027+0000 7ffa81b10700 -1 osd.11 109 *** Got signal Terminated *** 2024-05-27T09:54:14.029 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:14.027+0000 7ffa81b10700 -1 osd.11 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T09:54:15.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:14.762+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:15.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:14.762+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:15.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:14.762+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:15.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:14.762+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:15.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:14.762+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:15.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:14.762+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:15.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:14 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:14.762+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:16.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:15 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:15.787+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:16.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:15 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:15.787+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:16.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:15 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:15.787+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:16.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:15 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:15.787+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:16.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:15 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:15.787+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:16.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:15 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:15.787+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:16.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:15 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:15.787+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:16.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:15 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:15.787+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6820 osd.8 since back 2024-05-27T09:53:51.135587+0000 front 2024-05-27T09:53:51.135627+0000 (oldest deadline 2024-05-27T09:54:15.235602+0000) 2024-05-27T09:54:17.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:16.773+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:17.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:16.773+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:17.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:16.773+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:17.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:16.773+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:17.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:16.773+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:17.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:16.773+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:17.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:16.773+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:17.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:16 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:16.773+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6820 osd.8 since back 2024-05-27T09:53:51.135587+0000 front 2024-05-27T09:53:51.135627+0000 (oldest deadline 2024-05-27T09:54:15.235602+0000) 2024-05-27T09:54:18.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:17.734+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:18.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:17.734+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:18.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:17.734+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:18.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:17.734+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:18.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:17.734+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:18.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:17.734+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:18.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:17.734+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:18.089 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:17 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:17.734+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6820 osd.8 since back 2024-05-27T09:53:51.135587+0000 front 2024-05-27T09:53:51.135627+0000 (oldest deadline 2024-05-27T09:54:15.235602+0000) 2024-05-27T09:54:19.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:18.704+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-05-27T09:52:38.725321+0000 front 2024-05-27T09:52:38.725298+0000 (oldest deadline 2024-05-27T09:53:04.025242+0000) 2024-05-27T09:54:19.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:18.704+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-05-27T09:52:50.327723+0000 front 2024-05-27T09:52:50.327507+0000 (oldest deadline 2024-05-27T09:53:16.227580+0000) 2024-05-27T09:54:19.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:18.704+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-05-27T09:53:00.828780+0000 front 2024-05-27T09:53:00.828985+0000 (oldest deadline 2024-05-27T09:53:24.328908+0000) 2024-05-27T09:54:19.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:18.704+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-05-27T09:53:19.432481+0000 front 2024-05-27T09:53:19.432522+0000 (oldest deadline 2024-05-27T09:53:44.132571+0000) 2024-05-27T09:54:19.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:18.704+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-05-27T09:53:24.132987+0000 front 2024-05-27T09:53:24.133034+0000 (oldest deadline 2024-05-27T09:53:49.433117+0000) 2024-05-27T09:54:19.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:18.704+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-05-27T09:53:34.634766+0000 front 2024-05-27T09:53:34.634601+0000 (oldest deadline 2024-05-27T09:54:00.534320+0000) 2024-05-27T09:54:19.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:18.704+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-05-27T09:53:40.534926+0000 front 2024-05-27T09:53:40.534942+0000 (oldest deadline 2024-05-27T09:54:06.434743+0000) 2024-05-27T09:54:19.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:18 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11[62128]: 2024-05-27T09:54:18.704+0000 7ffa7a0be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.189:6820 osd.8 since back 2024-05-27T09:53:51.135587+0000 front 2024-05-27T09:53:51.135627+0000 (oldest deadline 2024-05-27T09:54:15.235602+0000) 2024-05-27T09:54:19.805 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:19 smithi189 podman[76633]: 2024-05-27 09:54:19.523634117 +0000 UTC m=+5.602067102 container died 5b38665e141f6e8b5b3195d90beba72ade04c425953b4dbf719ef3c85cf44e01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-27T09:54:20.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:19 smithi189 podman[76633]: 2024-05-27 09:54:19.893841228 +0000 UTC m=+5.972274214 container cleanup 5b38665e141f6e8b5b3195d90beba72ade04c425953b4dbf719ef3c85cf44e01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T09:54:20.088 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:19 smithi189 bash[76633]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11 2024-05-27T09:54:20.522 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:20 smithi189 podman[76651]: 2024-05-27 09:54:20.195556484 +0000 UTC m=+0.671678986 container remove 5b38665e141f6e8b5b3195d90beba72ade04c425953b4dbf719ef3c85cf44e01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux ) 2024-05-27T09:54:20.807 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:20 smithi189 podman[76739]: 2024-05-27 09:54:20.425392187 +0000 UTC m=+0.017497118 image pull 850a06a84d28fb71dec1431a907aeab5c3ccf2f0fcd791cd2cbd1ea45413ef76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6 2024-05-27T09:54:20.807 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:20 smithi189 podman[76739]: 2024-05-27 09:54:20.537939584 +0000 UTC m=+0.130044516 container create 0107c4cd2b0f6c498b08e2dd002e4bd156262993113c9328c77a050e8ced8fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:54:20.807 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:20 smithi189 podman[76739]: 2024-05-27 09:54:20.731908537 +0000 UTC m=+0.324013458 container init 0107c4cd2b0f6c498b08e2dd002e4bd156262993113c9328c77a050e8ced8fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:54:20.807 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:20 smithi189 podman[76739]: 2024-05-27 09:54:20.735110734 +0000 UTC m=+0.327215673 container start 0107c4cd2b0f6c498b08e2dd002e4bd156262993113c9328c77a050e8ced8fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:54:20.807 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:20 smithi189 podman[76739]: 2024-05-27 09:54:20.806763079 +0000 UTC m=+0.398868010 container attach 0107c4cd2b0f6c498b08e2dd002e4bd156262993113c9328c77a050e8ced8fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-27T09:54:21.087 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:21 smithi189 podman[76739]: 2024-05-27 09:54:21.037309248 +0000 UTC m=+0.629414184 container died 0107c4cd2b0f6c498b08e2dd002e4bd156262993113c9328c77a050e8ced8fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11-deactivate, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-05-27T09:54:21.603 INFO:journalctl@ceph.osd.11.smithi189.stdout:May 27 09:54:21 smithi189 podman[76765]: 2024-05-27 09:54:21.58465562 +0000 UTC m=+0.539821197 container remove 0107c4cd2b0f6c498b08e2dd002e4bd156262993113c9328c77a050e8ced8fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:54:21.609 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@osd.11.service' 2024-05-27T09:54:21.649 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:54:21.650 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-27T09:54:21.650 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-27T09:54:21.650 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.a 2024-05-27T09:54:22.224 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:54:21 smithi044 systemd[1]: Stopping Ceph mds.a for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:54:22.724 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:54:22 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-a[73653]: 2024-05-27T09:54:22.296+0000 7f3329c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:54:22.725 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:54:22 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-a[73653]: 2024-05-27T09:54:22.296+0000 7f3329c00700 -1 mds.a *** got signal Terminated *** 2024-05-27T09:54:26.685 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:54:26 smithi044 podman[226945]: 2024-05-27 09:54:26.331203303 +0000 UTC m=+4.192208358 container died 7658c9f94714258bd2b5523be4ddf958efce2f979936dd90311552ec16c552e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:54:26.974 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:54:26 smithi044 podman[226945]: 2024-05-27 09:54:26.796752351 +0000 UTC m=+4.657757344 container cleanup 7658c9f94714258bd2b5523be4ddf958efce2f979936dd90311552ec16c552e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-27T09:54:26.975 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:54:26 smithi044 bash[226945]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-a 2024-05-27T09:54:27.428 INFO:journalctl@ceph.mds.a.smithi044.stdout:May 27 09:54:26 smithi044 podman[226957]: 2024-05-27 09:54:26.993279879 +0000 UTC m=+0.643160032 container remove 7658c9f94714258bd2b5523be4ddf958efce2f979936dd90311552ec16c552e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-27T09:54:27.482 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.a.service' 2024-05-27T09:54:27.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:54:27.594 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-27T09:54:27.594 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-27T09:54:27.594 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.d 2024-05-27T09:54:27.974 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:54:27 smithi044 systemd[1]: Stopping Ceph mds.d for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:54:28.474 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:54:28 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-d[72491]: 2024-05-27T09:54:28.216+0000 7f63a9800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:54:28.474 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:54:28 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-d[72491]: 2024-05-27T09:54:28.216+0000 7f63a9800700 -1 mds.d *** got signal Terminated *** 2024-05-27T09:54:32.590 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:54:32 smithi044 podman[227063]: 2024-05-27 09:54:32.244493698 +0000 UTC m=+4.178025956 container died dd646d323f4b969fa42a95f101fd5fcc58c98431e9cde0b4b18cd803d5d6d4c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-d, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T09:54:32.974 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:54:32 smithi044 podman[227063]: 2024-05-27 09:54:32.770615708 +0000 UTC m=+4.704147965 container cleanup dd646d323f4b969fa42a95f101fd5fcc58c98431e9cde0b4b18cd803d5d6d4c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738) 2024-05-27T09:54:32.974 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:54:32 smithi044 bash[227063]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-d 2024-05-27T09:54:33.473 INFO:journalctl@ceph.mds.d.smithi044.stdout:May 27 09:54:33 smithi044 podman[227077]: 2024-05-27 09:54:33.055496771 +0000 UTC m=+0.792334894 container remove dd646d323f4b969fa42a95f101fd5fcc58c98431e9cde0b4b18cd803d5d6d4c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738) 2024-05-27T09:54:33.518 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.d.service' 2024-05-27T09:54:33.633 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:54:33.634 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-27T09:54:33.634 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-27T09:54:33.634 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.g 2024-05-27T09:54:34.223 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:54:33 smithi044 systemd[1]: Stopping Ceph mds.g for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:54:34.724 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:54:34 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-g[73258]: 2024-05-27T09:54:34.306+0000 7f0e2f600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:54:34.724 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:54:34 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-g[73258]: 2024-05-27T09:54:34.306+0000 7f0e2f600700 -1 mds.g *** got signal Terminated *** 2024-05-27T09:54:38.649 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:54:38 smithi044 podman[227182]: 2024-05-27 09:54:38.350181408 +0000 UTC m=+4.158100713 container died 35e81d080ca225250602f940cb6dafe2ba812955b6eee09bcdf7bb8405937d1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:54:39.224 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:54:38 smithi044 podman[227182]: 2024-05-27 09:54:38.902533976 +0000 UTC m=+4.710453291 container cleanup 35e81d080ca225250602f940cb6dafe2ba812955b6eee09bcdf7bb8405937d1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-27T09:54:39.224 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:54:38 smithi044 bash[227182]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-g 2024-05-27T09:54:39.225 INFO:journalctl@ceph.mds.g.smithi044.stdout:May 27 09:54:39 smithi044 podman[227195]: 2024-05-27 09:54:39.07920177 +0000 UTC m=+0.713285786 container remove 35e81d080ca225250602f940cb6dafe2ba812955b6eee09bcdf7bb8405937d1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-g, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-27T09:54:39.645 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.g.service' 2024-05-27T09:54:39.756 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:54:39.756 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-27T09:54:39.756 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-27T09:54:39.757 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.j 2024-05-27T09:54:40.223 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:54:39 smithi044 systemd[1]: Stopping Ceph mds.j for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:54:40.724 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:54:40 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-j[72070]: 2024-05-27T09:54:40.396+0000 7f8584a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:54:40.724 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:54:40 smithi044 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-j[72070]: 2024-05-27T09:54:40.396+0000 7f8584a00700 -1 mds.j *** got signal Terminated *** 2024-05-27T09:54:44.474 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:54:44 smithi044 podman[227302]: 2024-05-27 09:54:44.076483634 +0000 UTC m=+3.792969517 container died b861e9014d39d2a93efcde4fd5685d818f3fdd98ae82ce6999ad371cfe8516ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T09:54:44.869 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:54:44 smithi044 podman[227302]: 2024-05-27 09:54:44.675422634 +0000 UTC m=+4.391908521 container cleanup b861e9014d39d2a93efcde4fd5685d818f3fdd98ae82ce6999ad371cfe8516ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T09:54:44.869 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:54:44 smithi044 bash[227302]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-j 2024-05-27T09:54:45.224 INFO:journalctl@ceph.mds.j.smithi044.stdout:May 27 09:54:44 smithi044 podman[227328]: 2024-05-27 09:54:44.868175641 +0000 UTC m=+0.773264166 container remove b861e9014d39d2a93efcde4fd5685d818f3fdd98ae82ce6999ad371cfe8516ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-j, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-05-27T09:54:45.347 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.j.service' 2024-05-27T09:54:45.459 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:54:45.459 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-27T09:54:45.459 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-27T09:54:45.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.b 2024-05-27T09:54:45.792 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:54:45 smithi050 systemd[1]: Stopping Ceph mds.b for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:54:46.043 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:54:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b[66721]: 2024-05-27T09:54:45.792+0000 7f15df963700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:54:46.044 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:54:45 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b[66721]: 2024-05-27T09:54:45.792+0000 7f15df963700 -1 mds.b *** got signal Terminated *** 2024-05-27T09:54:48.750 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:54:48 smithi050 podman[75068]: 2024-05-27 09:54:48.459969757 +0000 UTC m=+2.776890552 container died 8b7b22588b732a688b4845dd8457da3babc752f80dd4c3ed3d6e9e0ca05050c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-27T09:54:49.043 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:54:48 smithi050 podman[75068]: 2024-05-27 09:54:48.850337781 +0000 UTC m=+3.167258597 container cleanup 8b7b22588b732a688b4845dd8457da3babc752f80dd4c3ed3d6e9e0ca05050c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T09:54:49.044 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:54:48 smithi050 bash[75068]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b 2024-05-27T09:54:49.424 INFO:journalctl@ceph.mds.b.smithi050.stdout:May 27 09:54:49 smithi050 podman[75094]: 2024-05-27 09:54:49.134701586 +0000 UTC m=+0.671941636 container remove 8b7b22588b732a688b4845dd8457da3babc752f80dd4c3ed3d6e9e0ca05050c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-27T09:54:49.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.b.service' 2024-05-27T09:54:49.498 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:54:49.499 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-27T09:54:49.499 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-27T09:54:49.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.e 2024-05-27T09:54:49.793 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:54:49 smithi050 systemd[1]: Stopping Ceph mds.e for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:54:50.293 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:54:49 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e[66101]: 2024-05-27T09:54:49.944+0000 7fc2f4cf8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:54:50.294 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:54:49 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e[66101]: 2024-05-27T09:54:49.944+0000 7fc2f4cf8700 -1 mds.e *** got signal Terminated *** 2024-05-27T09:54:52.794 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:54:52 smithi050 podman[75201]: 2024-05-27 09:54:52.46103902 +0000 UTC m=+2.749163207 container died 9e88f59da4bece838578c28d8f3285bfce288600bf52ab36adbae7018259f203 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T09:54:53.188 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:54:52 smithi050 podman[75201]: 2024-05-27 09:54:52.912267893 +0000 UTC m=+3.200392085 container cleanup 9e88f59da4bece838578c28d8f3285bfce288600bf52ab36adbae7018259f203 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T09:54:53.188 INFO:journalctl@ceph.mds.e.smithi050.stdout:May 27 09:54:52 smithi050 bash[75201]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-e 2024-05-27T09:54:53.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.e.service' 2024-05-27T09:54:53.393 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:54:53.394 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-27T09:54:53.394 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-27T09:54:53.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.h 2024-05-27T09:54:53.779 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:54:53 smithi050 systemd[1]: Stopping Ceph mds.h for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:54:54.043 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:54:53 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h[66406]: 2024-05-27T09:54:53.775+0000 7fb7faca5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:54:54.044 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:54:53 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h[66406]: 2024-05-27T09:54:53.775+0000 7fb7faca5700 -1 mds.h *** got signal Terminated *** 2024-05-27T09:54:56.794 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:54:56 smithi050 podman[75322]: 2024-05-27 09:54:56.473760207 +0000 UTC m=+2.859356900 container died 9d1fd8d5f05aab4d3a98300187dc17e53241c6a5b3735a48b3ea54e601aa2f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-27T09:54:57.192 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:54:56 smithi050 podman[75322]: 2024-05-27 09:54:56.924697891 +0000 UTC m=+3.310294578 container cleanup 9d1fd8d5f05aab4d3a98300187dc17e53241c6a5b3735a48b3ea54e601aa2f59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-27T09:54:57.193 INFO:journalctl@ceph.mds.h.smithi050.stdout:May 27 09:54:56 smithi050 bash[75322]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-h 2024-05-27T09:54:57.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.h.service' 2024-05-27T09:54:57.367 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:54:57.368 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-27T09:54:57.368 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-27T09:54:57.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.k 2024-05-27T09:54:57.793 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:54:57 smithi050 systemd[1]: Stopping Ceph mds.k for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:54:57.794 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:54:57 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-k[65788]: 2024-05-27T09:54:57.685+0000 7f48428a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:54:57.794 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:54:57 smithi050 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-k[65788]: 2024-05-27T09:54:57.685+0000 7f48428a7700 -1 mds.k *** got signal Terminated *** 2024-05-27T09:54:59.043 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:54:58 smithi050 podman[75443]: 2024-05-27 09:54:58.659142064 +0000 UTC m=+1.069669470 container died 166e6764c810be15ce17ed86f563136d99faa547986e59c9abee82b072e5754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-k, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, ceph=True) 2024-05-27T09:54:59.387 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:54:59 smithi050 podman[75443]: 2024-05-27 09:54:59.161440791 +0000 UTC m=+1.571968196 container cleanup 166e6764c810be15ce17ed86f563136d99faa547986e59c9abee82b072e5754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-k, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T09:54:59.387 INFO:journalctl@ceph.mds.k.smithi050.stdout:May 27 09:54:59 smithi050 bash[75443]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-k 2024-05-27T09:54:59.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.k.service' 2024-05-27T09:54:59.559 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:54:59.559 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-27T09:54:59.559 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-27T09:54:59.559 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.c 2024-05-27T09:54:59.837 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:54:59 smithi189 systemd[1]: Stopping Ceph mds.c for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:55:00.099 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:54:59 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c[67552]: 2024-05-27T09:54:59.841+0000 7f9f44d45700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:55:00.100 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:54:59 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c[67552]: 2024-05-27T09:54:59.841+0000 7f9f44d45700 -1 mds.c *** got signal Terminated *** 2024-05-27T09:55:04.158 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:55:03 smithi189 podman[76845]: 2024-05-27 09:55:03.857124945 +0000 UTC m=+4.108039680 container died 9889fd83bbe81c8c6788dcb7d2ef6aabf433e84410a5cad19776ffc2e461df8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T09:55:04.159 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:55:04 smithi189 podman[76845]: 2024-05-27 09:55:04.158166626 +0000 UTC m=+4.409081349 container cleanup 9889fd83bbe81c8c6788dcb7d2ef6aabf433e84410a5cad19776ffc2e461df8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T09:55:04.498 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:55:04 smithi189 bash[76845]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c 2024-05-27T09:55:04.499 INFO:journalctl@ceph.mds.c.smithi189.stdout:May 27 09:55:04 smithi189 podman[76870]: 2024-05-27 09:55:04.384464358 +0000 UTC m=+0.522069913 container remove 9889fd83bbe81c8c6788dcb7d2ef6aabf433e84410a5cad19776ffc2e461df8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-27T09:55:04.527 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.c.service' 2024-05-27T09:55:04.561 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:55:04.562 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-27T09:55:04.562 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-27T09:55:04.562 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.f 2024-05-27T09:55:04.837 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:55:04 smithi189 systemd[1]: Stopping Ceph mds.f for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:55:05.337 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:55:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[72375]: 2024-05-27T09:55:04.895+0000 7fe356700700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:55:05.337 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:55:04 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f[72375]: 2024-05-27T09:55:04.895+0000 7fe356700700 -1 mds.f *** got signal Terminated *** 2024-05-27T09:55:08.730 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:55:08 smithi189 podman[76979]: 2024-05-27 09:55:08.460388762 +0000 UTC m=+3.672777342 container died a1d538aae68166ff48a930b9af49dccd5a3ef2114af0dee624850f8f659ad3e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-27T09:55:09.004 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:55:08 smithi189 podman[76979]: 2024-05-27 09:55:08.803359545 +0000 UTC m=+4.015748116 container cleanup a1d538aae68166ff48a930b9af49dccd5a3ef2114af0dee624850f8f659ad3e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-27T09:55:09.004 INFO:journalctl@ceph.mds.f.smithi189.stdout:May 27 09:55:08 smithi189 bash[76979]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-f 2024-05-27T09:55:09.226 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.f.service' 2024-05-27T09:55:09.260 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:55:09.260 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-27T09:55:09.260 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-27T09:55:09.260 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.i 2024-05-27T09:55:09.574 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:55:09 smithi189 systemd[1]: Stopping Ceph mds.i for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:55:09.837 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:55:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[74173]: 2024-05-27T09:55:09.574+0000 7fd4c3ded700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:55:09.837 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:55:09 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i[74173]: 2024-05-27T09:55:09.574+0000 7fd4c3ded700 -1 mds.i *** got signal Terminated *** 2024-05-27T09:55:12.837 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:55:12 smithi189 podman[77101]: 2024-05-27 09:55:12.469666307 +0000 UTC m=+2.994055079 container died b0bc0afa505177c0523859016e6665eb04d04d9e8fb53ef64b151753b0b78b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-27T09:55:13.193 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:55:12 smithi189 podman[77101]: 2024-05-27 09:55:12.924311541 +0000 UTC m=+3.448700329 container cleanup b0bc0afa505177c0523859016e6665eb04d04d9e8fb53ef64b151753b0b78b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-05-27T09:55:13.194 INFO:journalctl@ceph.mds.i.smithi189.stdout:May 27 09:55:12 smithi189 bash[77101]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-i 2024-05-27T09:55:13.358 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.i.service' 2024-05-27T09:55:13.389 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:55:13.390 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-27T09:55:13.390 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-27T09:55:13.390 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.l 2024-05-27T09:55:13.711 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:55:13 smithi189 systemd[1]: Stopping Ceph mds.l for 604b015a-1c0a-11ef-bc9b-c7b262605968... 2024-05-27T09:55:14.087 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:55:13 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-l[66930]: 2024-05-27T09:55:13.710+0000 7fb6320dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T09:55:14.088 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:55:13 smithi189 ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-l[66930]: 2024-05-27T09:55:13.710+0000 7fb6320dd700 -1 mds.l *** got signal Terminated *** 2024-05-27T09:55:16.120 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:55:15 smithi189 podman[77221]: 2024-05-27 09:55:15.841562151 +0000 UTC m=+2.224864096 container died 2f8c7b3cbe9889510e721163688d567f97631a9c30da43ad66bdf33bb59bd15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-l, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-05-27T09:55:16.412 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:55:16 smithi189 podman[77221]: 2024-05-27 09:55:16.203363516 +0000 UTC m=+2.586665413 container cleanup 2f8c7b3cbe9889510e721163688d567f97631a9c30da43ad66bdf33bb59bd15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:40667fa40b383802316dd30b2fb5cc99041435f0e277cde147e3c3dc5c88e9a6, name=ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-l, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.105740-debug-a650738, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True) 2024-05-27T09:55:16.412 INFO:journalctl@ceph.mds.l.smithi189.stdout:May 27 09:55:16 smithi189 bash[77221]: ceph-604b015a-1c0a-11ef-bc9b-c7b262605968-mds-l 2024-05-27T09:55:16.622 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-604b015a-1c0a-11ef-bc9b-c7b262605968@mds.l.service' 2024-05-27T09:55:16.653 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T09:55:16.653 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-27T09:55:16.653 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T09:55:16.988 INFO:teuthology.orchestra.run.smithi044.stdout:Deleting cluster with fsid: 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:55:23.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T09:55:23.467 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:55:26.021 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T09:55:26.217 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:55:28.769 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T09:55:28.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T09:55:28.896 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T09:55:28.922 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-27T09:55:28.924 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/remote/smithi044/crash 2024-05-27T09:55:28.925 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T09:55:29.029 INFO:teuthology.orchestra.run.smithi044.stderr:tar: /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T09:55:29.029 INFO:teuthology.orchestra.run.smithi044.stderr:tar: Error is not recoverable: exiting now 2024-05-27T09:55:29.034 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/remote/smithi050/crash 2024-05-27T09:55:29.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T09:55:29.079 INFO:teuthology.orchestra.run.smithi050.stderr:tar: /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T09:55:29.079 INFO:teuthology.orchestra.run.smithi050.stderr:tar: Error is not recoverable: exiting now 2024-05-27T09:55:29.081 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/remote/smithi189/crash 2024-05-27T09:55:29.082 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T09:55:29.113 INFO:teuthology.orchestra.run.smithi189.stderr:tar: /var/lib/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T09:55:29.113 INFO:teuthology.orchestra.run.smithi189.stderr:tar: Error is not recoverable: exiting now 2024-05-27T09:55:29.114 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-27T09:55:29.114 DEBUG:teuthology.orchestra.run.smithi044:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-27T09:55:29.269 INFO:tasks.cephadm:Compressing logs... 2024-05-27T09:55:29.269 DEBUG:teuthology.orchestra.run.smithi044:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T09:55:29.274 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T09:55:29.276 DEBUG:teuthology.orchestra.run.smithi189:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T09:55:29.302 INFO:teuthology.orchestra.run.smithi050.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-27T09:55:29.303 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T09:55:29.303 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T09:55:29.304 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T09:55:29.304 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.log: -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.7.log 2024-05-27T09:55:29.304 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.b.log 2024-05-27T09:55:29.304 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-05-27T09:55:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.7.log: /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.b.log: gzip 92.1% -5 --verbose -- replaced with /var/log/ceph/cephadm.log.gz -- 2024-05-27T09:55:29.305 INFO:teuthology.orchestra.run.smithi050.stderr: /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-05-27T09:55:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mgr.y.log 2024-05-27T09:55:29.307 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.h.log 2024-05-27T09:55:29.307 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.10.log 2024-05-27T09:55:29.307 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.h.log: 90.3% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T09:55:29.308 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 87.4% --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T09:55:29.308 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T09:55:29.310 INFO:teuthology.orchestra.run.smithi189.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-27T09:55:29.310 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T09:55:29.311 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T09:55:29.313 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-05-27T09:55:29.313 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.f.log 2024-05-27T09:55:29.314 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.log: /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T09:55:29.314 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-05-27T09:55:29.314 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.c.log 2024-05-27T09:55:29.314 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.5.log: -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mon.c.log 2024-05-27T09:55:29.314 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mgr.z.log 2024-05-27T09:55:29.314 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.i.log 2024-05-27T09:55:29.314 INFO:teuthology.orchestra.run.smithi189.stderr:gzip/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mgr.z.log: -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T09:55:29.314 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.11.log 2024-05-27T09:55:29.315 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.8.log 2024-05-27T09:55:29.315 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T09:55:29.315 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.l.log 2024-05-27T09:55:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.e.log 2024-05-27T09:55:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.cephadm.log: 83.3% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T09:55:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T09:55:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.e.log: 92.3% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mgr.y.log.gz 2024-05-27T09:55:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mon.b.log 2024-05-27T09:55:29.340 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.k.log 2024-05-27T09:55:29.342 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T09:55:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.k.log: 94.4% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.k.log.gz 2024-05-27T09:55:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-volume.log: /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.l.log: 83.2% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T09:55:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-27T09:55:29.400 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T09:55:29.406 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T09:55:29.408 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T09:55:29.409 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.g.log 2024-05-27T09:55:29.411 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.j.log 2024-05-27T09:55:29.413 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.g.log: 87.4% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T09:55:29.414 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-05-27T09:55:29.415 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-05-27T09:55:29.416 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mgr.x.log 2024-05-27T09:55:29.418 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mon.a.log 2024-05-27T09:55:29.419 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.6.log 2024-05-27T09:55:29.420 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T09:55:29.422 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.a.log 2024-05-27T09:55:29.423 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T09:55:29.423 INFO:teuthology.orchestra.run.smithi050.stderr: 94.5% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.e.log.gz 2024-05-27T09:55:29.424 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.d.log 2024-05-27T09:55:29.425 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.9.log 2024-05-27T09:55:29.446 INFO:teuthology.orchestra.run.smithi189.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T09:55:29.476 INFO:teuthology.orchestra.run.smithi189.stderr: 92.4% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mgr.z.log.gz 2024-05-27T09:55:29.482 INFO:teuthology.orchestra.run.smithi189.stderr: 94.4% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.l.log.gz 2024-05-27T09:55:29.497 INFO:teuthology.orchestra.run.smithi189.stderr: 91.0% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T09:55:29.521 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.d.log: /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.9.log: 83.7% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T09:55:29.535 INFO:teuthology.orchestra.run.smithi189.stderr: 90.4% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T09:55:29.535 INFO:teuthology.orchestra.run.smithi189.stderr: 87.4% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T09:55:29.567 INFO:teuthology.orchestra.run.smithi044.stderr: 94.4% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.d.log.gz 2024-05-27T09:55:29.623 INFO:teuthology.orchestra.run.smithi044.stderr: 94.4% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.g.log.gz 2024-05-27T09:55:29.649 INFO:teuthology.orchestra.run.smithi044.stderr: 90.9% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T09:55:29.680 INFO:teuthology.orchestra.run.smithi044.stderr: 90.2% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T09:55:29.694 INFO:teuthology.orchestra.run.smithi044.stderr: 94.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T09:55:29.712 INFO:teuthology.orchestra.run.smithi044.stderr: 94.4% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.j.log.gz 2024-05-27T09:55:29.755 INFO:teuthology.orchestra.run.smithi044.stderr: 94.3% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.a.log.gz 2024-05-27T09:55:30.053 INFO:teuthology.orchestra.run.smithi050.stderr: 91.6% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mon.b.log.gz 2024-05-27T09:55:30.511 INFO:teuthology.orchestra.run.smithi189.stderr: 91.5% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mon.c.log.gz 2024-05-27T09:55:31.517 INFO:teuthology.orchestra.run.smithi044.stderr: 90.1% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mgr.x.log.gz 2024-05-27T09:55:34.600 INFO:teuthology.orchestra.run.smithi044.stderr: 90.3% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mon.a.log.gz 2024-05-27T09:55:36.090 INFO:teuthology.orchestra.run.smithi050.stderr: 94.2% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-05-27T09:55:36.440 INFO:teuthology.orchestra.run.smithi189.stderr: 95.0% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.f.log.gz 2024-05-27T09:55:37.138 INFO:teuthology.orchestra.run.smithi189.stderr: 94.1% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.c.log.gz 2024-05-27T09:55:38.889 INFO:teuthology.orchestra.run.smithi189.stderr: 93.8% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.8.log.gz 2024-05-27T09:55:41.756 INFO:teuthology.orchestra.run.smithi050.stderr: 93.9% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.7.log.gz 2024-05-27T09:55:41.998 INFO:teuthology.orchestra.run.smithi189.stderr: 93.9% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-05-27T09:55:42.793 INFO:teuthology.orchestra.run.smithi050.stderr: 93.8% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-05-27T09:55:43.678 INFO:teuthology.orchestra.run.smithi050.stderr: 93.7% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.10.log.gz 2024-05-27T09:55:43.869 INFO:teuthology.orchestra.run.smithi044.stderr: 93.8% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.9.log.gz 2024-05-27T09:55:44.830 INFO:teuthology.orchestra.run.smithi189.stderr: 93.8% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-05-27T09:55:45.129 INFO:teuthology.orchestra.run.smithi044.stderr: 93.8% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-05-27T09:55:45.687 INFO:teuthology.orchestra.run.smithi044.stderr: 93.8% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.6.log.gz 2024-05-27T09:55:46.445 INFO:teuthology.orchestra.run.smithi044.stderr: 93.6% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-05-27T09:55:46.451 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-05-27T09:55:46.452 INFO:teuthology.orchestra.run.smithi044.stderr:real 0m17.138s 2024-05-27T09:55:46.452 INFO:teuthology.orchestra.run.smithi044.stderr:user 0m52.893s 2024-05-27T09:55:46.452 INFO:teuthology.orchestra.run.smithi044.stderr:sys 0m4.734s 2024-05-27T09:55:54.835 INFO:teuthology.orchestra.run.smithi189.stderr: 94.0% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-osd.11.log.gz 2024-05-27T09:56:27.622 INFO:teuthology.orchestra.run.smithi050.stderr: 93.5% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.h.log.gz 2024-05-27T09:56:33.926 INFO:teuthology.orchestra.run.smithi050.stderr: 93.8% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.b.log.gz 2024-05-27T09:56:33.928 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-05-27T09:56:33.929 INFO:teuthology.orchestra.run.smithi050.stderr:real 1m4.637s 2024-05-27T09:56:33.929 INFO:teuthology.orchestra.run.smithi050.stderr:user 2m45.712s 2024-05-27T09:56:33.929 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m4.858s 2024-05-27T09:58:29.032 INFO:teuthology.orchestra.run.smithi189.stderr: 93.6% -- replaced with /var/log/ceph/604b015a-1c0a-11ef-bc9b-c7b262605968/ceph-mds.i.log.gz 2024-05-27T09:58:29.034 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-05-27T09:58:29.034 INFO:teuthology.orchestra.run.smithi189.stderr:real 2m59.744s 2024-05-27T09:58:29.035 INFO:teuthology.orchestra.run.smithi189.stderr:user 3m31.294s 2024-05-27T09:58:29.035 INFO:teuthology.orchestra.run.smithi189.stderr:sys 0m7.292s 2024-05-27T09:58:29.036 INFO:tasks.cephadm:Archiving logs... 2024-05-27T09:58:29.036 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/remote/smithi044/log 2024-05-27T09:58:29.037 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T09:58:35.745 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/remote/smithi050/log 2024-05-27T09:58:35.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T09:58:57.088 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/remote/smithi189/log 2024-05-27T09:58:57.089 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T09:59:27.660 INFO:tasks.cephadm:Removing cluster... 2024-05-27T09:59:27.662 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 --force 2024-05-27T09:59:28.001 INFO:teuthology.orchestra.run.smithi044.stdout:Deleting cluster with fsid: 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:59:28.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 --force 2024-05-27T09:59:28.906 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:59:29.358 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 604b015a-1c0a-11ef-bc9b-c7b262605968 --force 2024-05-27T09:59:29.551 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 604b015a-1c0a-11ef-bc9b-c7b262605968 2024-05-27T09:59:30.040 INFO:tasks.cephadm:Removing cephadm ... 2024-05-27T09:59:30.040 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T09:59:30.101 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T09:59:30.118 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T09:59:30.133 INFO:tasks.cephadm:Teardown complete 2024-05-27T09:59:30.134 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-27T09:59:30.148 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-27T09:59:30.148 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T09:59:30.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T09:59:30.160 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T09:59:30.316 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-27T09:59:30.328 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-27T09:59:30.329 DEBUG:teuthology.orchestra.run.smithi044:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T09:59:30.337 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T09:59:30.339 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T09:59:30.351 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-05-27T09:59:30.353 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-05-27T09:59:30.356 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T09:59:30.356 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-05-27T09:59:30.356 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 2 +891us[ +891us] +/- 52ms 2024-05-27T09:59:30.356 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -1654us[-1508us] +/- 71ms 2024-05-27T09:59:30.357 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +549us[ +695us] +/- 80ms 2024-05-27T09:59:30.357 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:59:30.364 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T09:59:30.364 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-05-27T09:59:30.364 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 57 +905us[+1042us] +/- 52ms 2024-05-27T09:59:30.365 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -1650us[-1514us] +/- 71ms 2024-05-27T09:59:30.365 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +552us[ +689us] +/- 80ms 2024-05-27T09:59:30.365 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:59:30.386 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-05-27T09:59:30.398 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T09:59:30.398 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-05-27T09:59:30.398 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 62 +889us[+1034us] +/- 52ms 2024-05-27T09:59:30.398 INFO:teuthology.orchestra.run.smithi044.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -1664us[-1519us] +/- 71ms 2024-05-27T09:59:30.398 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 55 +555us[ +555us] +/- 80ms 2024-05-27T09:59:30.398 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T09:59:30.399 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-27T09:59:30.412 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-27T09:59:30.412 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-27T09:59:30.426 DEBUG:teuthology.orchestra.run.smithi044:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T09:59:30.757 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T09:59:30.825 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T09:59:30.942 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T09:59:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:59:31.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T09:59:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:59:31.069 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T09:59:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:59:31.096 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-27T09:59:31.108 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-27T09:59:31.130 INFO:teuthology.task.internal:Duration was 2846.712199 seconds 2024-05-27T09:59:31.131 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-27T09:59:31.143 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-27T09:59:31.144 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T09:59:31.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T09:59:31.154 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T09:59:31.189 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T09:59:31.223 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T09:59:31.334 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T09:59:31.856 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-27T09:59:31.856 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi044.front.sepia.ceph.com 2024-05-27T09:59:31.856 DEBUG:teuthology.orchestra.run.smithi044:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T09:59:31.930 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-05-27T09:59:31.930 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T09:59:31.958 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-05-27T09:59:31.958 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T09:59:32.003 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-27T09:59:32.003 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T09:59:32.011 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T09:59:32.013 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T09:59:32.102 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-27T09:59:32.102 DEBUG:teuthology.orchestra.run.smithi044:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T09:59:32.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T09:59:32.111 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T09:59:33.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-27T09:59:33.522 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-27T09:59:33.522 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T09:59:33.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T09:59:33.635 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T09:59:33.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-27T09:59:33.693 DEBUG:teuthology.orchestra.run.smithi044:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T09:59:33.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T09:59:33.705 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T09:59:33.729 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-05-27T09:59:33.731 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-05-27T09:59:33.817 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = core 2024-05-27T09:59:33.868 DEBUG:teuthology.orchestra.run.smithi044:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T09:59:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:59:33.923 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T09:59:33.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:59:33.942 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T09:59:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T09:59:33.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-27T09:59:33.979 INFO:teuthology.task.internal:Transferring archived files... 2024-05-27T09:59:33.981 DEBUG:teuthology.misc:Transferring archived files from smithi044:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/remote/smithi044 2024-05-27T09:59:33.981 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T09:59:34.113 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/remote/smithi050 2024-05-27T09:59:34.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T09:59:34.156 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_06:46:53-fs-wip-vshankar-testing-20240521.105740-debug-testing-default-smithi/7727679/remote/smithi189 2024-05-27T09:59:34.157 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T09:59:34.196 INFO:teuthology.task.internal:Removing archive directory... 2024-05-27T09:59:34.197 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T09:59:34.207 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T09:59:34.209 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T09:59:34.258 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-27T09:59:34.272 INFO:teuthology.task.internal:Not uploading archives. 2024-05-27T09:59:34.272 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-27T09:59:34.284 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-27T09:59:34.284 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T09:59:34.289 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T09:59:34.292 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T09:59:34.307 INFO:teuthology.orchestra.run.smithi050.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 09:59 /home/ubuntu/cephtest 2024-05-27T09:59:34.308 INFO:teuthology.orchestra.run.smithi189.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 09:59 /home/ubuntu/cephtest 2024-05-27T09:59:34.340 INFO:teuthology.orchestra.run.smithi044.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 09:59 /home/ubuntu/cephtest 2024-05-27T09:59:34.344 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-27T09:59:34.359 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-27T09:59:34.390 INFO:teuthology.run:Summary data: client.0-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/fs/misc}} duration: 2846.7121987342834 failure_reason: 'Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi044 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6507388dd5057528934822c0163b0c347ef1d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh''' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e2b4eb7192c3476a8ab4bbf2d943712e status: fail success: false 2024-05-27T09:59:34.391 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T09:59:34.479 INFO:teuthology.run:FAIL